Why: API keys live in Postgres as plaintext JSON. A DB read exposes every credential in the system. Before centralising more secrets (LLM keys, etc.) we want to be able to point at an external KV store and drop DB access to sensitive rows. New model: - `SecretBackend` resource (CRUD + isDefault invariant) owns how a secret is stored. `Secret` gains `backendId` FK and `externalRef`. Reads/writes dispatch through a driver. - `plaintext` driver (near-noop, uses existing Secret.data column) is seeded as the `default` row at startup. Acts as trust root / bootstrap. - `openbao` driver (also HashiCorp Vault KV v2 compatible) talks plain HTTP, no SDK dependency. Auth via static token pulled from a plaintext-backed `Secret` through the injected SecretRefResolver. Caches resolved token. - `SecretMigrateService` moves secrets one-at-a-time: read → write dest → flip row → best-effort source delete. Interrupted runs are idempotent (skips secrets already on destination). CLI surface: - `mcpctl create|get|describe|delete secretbackend` + `--default` on create. - `mcpctl migrate secrets --from X --to Y [--names a,b] [--keep-source] [--dry-run]` - `apply -f` round-trips secretbackends (yaml/json multi-doc + grouped). - RBAC: `secretbackends` resource + `run:migrate-secrets` operation. - Fish + bash completions regenerated. docs/secret-backends.md covers the OpenBao policy, chicken-and-egg auth flow, and the migration semantics. Broke the circular dep (OpenBao needs SecretService to resolve its own token, SecretService needs SecretBackendService) with a deferred-resolver bridge in mcpd startup. 11 new driver unit tests; existing env-resolver/secret-route/ backup tests updated for the new service signatures. Full suite: 1792/1792. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
6.1 KiB
Secret backends
mcpctl stores the raw data for Secret resources in a pluggable backend.
The default is plaintext — the secret payload lives in Postgres as plain JSON
— which is fine for laptop development but a poor fit for shared clusters. For
production, point at an external KV store and delete secrets from the DB after
migration.
This guide covers the model, the shipped drivers, and how to migrate without downtime.
Model
- A
SecretBackendresource is a single named driver instance (e.g. a pointer at one OpenBao deployment). - Every
Secretrow carries abackendIdFK — the backend that owns its data. - Exactly one
SecretBackendhasisDefault: true. New secrets created through the API/CLI land on that backend. - The
plaintextbackend is seeded at startup and nameddefault. It cannot be deleted — there needs to always be one row where the driver's own credentials can bootstrap from (see below).
CLI
mcpctl get secretbackends # list backends
mcpctl describe secretbackend <name> # inspect config (credentials masked)
mcpctl create secretbackend <name> --type plaintext [--default] [--description ...]
mcpctl create secretbackend <name> --type openbao \
--url http://bao.example:8200 \
--token-secret bao-creds/token \
[--namespace <ns>] [--mount secret] [--path-prefix mcpctl] \
[--default]
mcpctl delete secretbackend <name> # blocked if any secret still points at it
mcpctl migrate secrets --from default --to bao
mcpctl migrate secrets --from default --to bao --names a,b --keep-source
mcpctl migrate secrets --from default --to bao --dry-run
Anything you can do with create secretbackend also works via apply -f:
kind: secretbackend
name: bao
type: openbao
description: "shared cluster OpenBao"
isDefault: true
config:
url: http://bao.svc.cluster.local:8200
tokenSecretRef: { name: bao-creds, key: token }
namespace: platform
Drivers
plaintext
Trivial. Secret.data holds the JSON, externalRef is empty.
- Storage: Postgres column.
- Bootstrap: seeded as
defaultat startup. - Cost: zero setup, zero encryption at rest, full access for any DB reader.
Use for development, CI, or single-tenant self-hosts where the DB itself is treated as sensitive.
openbao
Talks HTTP to an OpenBao (MPL 2.0 Vault fork) KV v2 mount. Also compatible with HashiCorp Vault KV v2 — the wire protocol is the same.
| Config key | Required? | Description |
|---|---|---|
url |
yes | Base URL, e.g. http://bao.svc.cluster.local:8200. |
tokenSecretRef |
yes | { name, key } pointing at a Secret on the plaintext backend that holds the bootstrap token. |
mount |
no | KV v2 mount name. Default secret. |
pathPrefix |
no | Path prefix under the mount. Default mcpctl. Secrets land at <mount>/<pathPrefix>/<secretName>. |
namespace |
no | X-Vault-Namespace header for OpenBao/Vault Enterprise namespaces. |
The driver only stores a reference in Secret.externalRef (mount/path). The
Secret.data column is left empty for openbao-backed rows — you can safely
drop DB-level access to secrets after migration.
Required OpenBao policy
Minimum token policy for a backend that lives at secret/mcpctl/:
path "secret/data/mcpctl/*" {
capabilities = ["create", "read", "update"]
}
path "secret/metadata/mcpctl/*" {
capabilities = ["list", "delete"]
}
path "secret/metadata/mcpctl/" {
capabilities = ["list"]
}
Grant delete on metadata/... only if you need mcpctl to fully remove
secrets — OpenBao soft-deletes until the metadata is gone.
Chicken-and-egg: where does the OpenBao token live?
mcpd reads the OpenBao token from a Secret on the plaintext backend.
That's the whole point of keeping plaintext around — it's the trust root:
- Operator creates a plaintext
Secretholding the bootstrap token. - Operator creates the
openbaobackend, pointing at that secret viatokenSecretRef. - Operator runs
mcpctl migrate secrets --from default --to baoto move all other secrets off plaintext. - After migration, the only sensitive row left on plaintext is the OpenBao token itself. DB access is now equivalent to OpenBao token access (a single key), not equivalent to all API keys in the system.
Follow-up work (not shipped yet) replaces static token auth with Kubernetes ServiceAccount auth so no bootstrap token is needed at all.
Migration — mcpctl migrate secrets
Atomicity is per secret, not per batch. Remote writes can't roll back, so we don't pretend. For each secret the service:
- Reads the plaintext from the source driver.
- Writes it to the destination driver.
- Updates the
Secretrow: flipsbackendId, sets newexternalRef, clearsdata. - Deletes from source (skipped with
--keep-source).
If the command is interrupted between step 2 and 3, the destination has an orphan entry but the source still owns the row. Re-running is idempotent — the service skips secrets that are already on the destination and picks up the rest.
# Dry-run first: see what would move.
mcpctl migrate secrets --from default --to bao --dry-run
# Migrate everything.
mcpctl migrate secrets --from default --to bao
# Migrate a subset only.
mcpctl migrate secrets --from default --to bao --names api-keys,oauth-client
# Leave the source copy in place (useful for A/B validation).
mcpctl migrate secrets --from default --to bao --keep-source
The command prints a per-secret summary (migrated / skipped / failed) and exits non-zero if any secret failed. Ctrl-C during the run is safe — restart when you want, no duplicate writes.
RBAC
resource: secretbackends— gated like any other resource (view,create,edit,delete).role: run, action: migrate-secrets— required to callPOST /api/v1/secrets/migrate.
Describe output masks config values whose keys look like credentials
(token, secret, password, key), so mcpctl describe secretbackend is
safe to paste into tickets.