mcpctl/docs/secret-backends.md

Secret backends

mcpctl stores the raw data for Secret resources in a pluggable backend. The default is plaintext — the secret payload lives in Postgres as plain JSON — which is fine for laptop development but a poor fit for shared clusters. For production, point at an external KV store and delete secrets from the DB after migration.

This guide covers the model, the shipped drivers, and how to migrate without downtime.

Model

• A SecretBackend resource is a single named driver instance (e.g. a pointer at one OpenBao deployment).
• Every Secret row carries a backendId FK — the backend that owns its data.
• Exactly one SecretBackend has isDefault: true. New secrets created through the API/CLI land on that backend.
• The plaintext backend is seeded at startup and named default. It cannot be deleted — there needs to always be one row where the driver's own credentials can bootstrap from (see below).

CLI

mcpctl get secretbackends              # list backends
mcpctl describe secretbackend <name>   # inspect config (credentials masked)
mcpctl create secretbackend <name> --type plaintext [--default] [--description ...]
mcpctl create secretbackend <name> --type openbao \
  --url http://bao.example:8200 \
  --token-secret bao-creds/token \
  [--namespace <ns>] [--mount secret] [--path-prefix mcpctl] \
  [--default]
mcpctl delete secretbackend <name>     # blocked if any secret still points at it

mcpctl migrate secrets --from default --to bao
mcpctl migrate secrets --from default --to bao --names a,b --keep-source
mcpctl migrate secrets --from default --to bao --dry-run

Anything you can do with create secretbackend also works via apply -f:

kind: secretbackend
name: bao
type: openbao
description: "shared cluster OpenBao"
isDefault: true
config:
  url: http://bao.svc.cluster.local:8200
  tokenSecretRef: { name: bao-creds, key: token }
  namespace: platform

Drivers

plaintext

Trivial. Secret.data holds the JSON, externalRef is empty.

• Storage: Postgres column.
• Bootstrap: seeded as default at startup.
• Cost: zero setup, zero encryption at rest, full access for any DB reader.

Use for development, CI, or single-tenant self-hosts where the DB itself is treated as sensitive.

openbao

Talks HTTP to an OpenBao (MPL 2.0 Vault fork) KV v2 mount. Also compatible with HashiCorp Vault KV v2 — the wire protocol is the same.

Config key	Required?	Description
url	yes	Base URL, e.g. http://bao.svc.cluster.local:8200.
tokenSecretRef	yes	{ name, key } pointing at a Secret on the plaintext backend that holds the bootstrap token.
mount	no	KV v2 mount name. Default secret.
pathPrefix	no	Path prefix under the mount. Default mcpctl. Secrets land at <mount>/<pathPrefix>/<secretName>.
namespace	no	X-Vault-Namespace header for OpenBao/Vault Enterprise namespaces.

The driver only stores a reference in Secret.externalRef (mount/path). The Secret.data column is left empty for openbao-backed rows — you can safely drop DB-level access to secrets after migration.

Required OpenBao policy

Minimum token policy for a backend that lives at secret/mcpctl/:

path "secret/data/mcpctl/*" {
  capabilities = ["create", "read", "update"]
}

path "secret/metadata/mcpctl/*" {
  capabilities = ["list", "delete"]
}

path "secret/metadata/mcpctl/" {
  capabilities = ["list"]
}

Grant delete on metadata/... only if you need mcpctl to fully remove secrets — OpenBao soft-deletes until the metadata is gone.

Chicken-and-egg: where does the OpenBao token live?

mcpd reads the OpenBao token from a Secret on the plaintext backend. That's the whole point of keeping plaintext around — it's the trust root:

1. Operator creates a plaintext Secret holding the bootstrap token.
2. Operator creates the openbao backend, pointing at that secret via tokenSecretRef.
3. Operator runs mcpctl migrate secrets --from default --to bao to move all other secrets off plaintext.
4. After migration, the only sensitive row left on plaintext is the OpenBao token itself. DB access is now equivalent to OpenBao token access (a single key), not equivalent to all API keys in the system.

Kubernetes ServiceAccount auth (no bootstrap token)

auth: kubernetes removes the chicken-and-egg entirely: mcpd exchanges its projected ServiceAccount JWT for an OpenBao token at auth/<authMount>/role/<role>, so there is no static credential in the database at all. The token is cached for its lease and re-minted lazily with a 60s grace window.

kind: secretbackend
name: bao-k8s
type: openbao
isDefault: true
config:
  url: https://bao.example
  auth: kubernetes
  role: mcpctl
  authMount: kubernetes-worker0   # defaults to `kubernetes`

Note that the daily rotator does not apply to these backends — there is no stored token to rotate. That has a consequence for monitoring, see below.

Reliability

Remote backends are network dependencies on the critical path of nearly everything: server env resolution, LLM api keys, chat, git providers, code repos, webhooks. Three mechanisms keep an outage from cascading.

Request hardening

Every call carries a timeout (default 5s) and retries 5xx/429/network failures with full-jitter exponential backoff (3 attempts). A sealed OpenBao answers 503, so this covers unseal windows and failovers.

The 403 path is separate and deliberately single-shot: the driver purges its cached token, re-authenticates and retries once. That is a credential refresh, not a backend-unavailable condition — looping on it would hide a genuinely revoked grant.

Value cache with stale-while-error

Resolved values are cached per backend (default TTL 5 minutes, LRU-bounded). Past the TTL the backend is always consulted; if it fails as a transport failure, the last known-good value is served instead of throwing.

Failure	Behaviour
Backend unreachable / timeout / exhausted 5xx	Serve last known-good, mark degraded, log BACKEND_UNREACHABLE once
Secret deleted (404)	Evict and throw. Never served stale — that would resurrect a revoked credential
403 after a token refresh	Throw. Revoked grants must stay loud
Nothing cached yet	Throw

The stale window is unbounded on purpose: a cap would mean a long outage eventually takes mcpd down anyway.

plaintext backends are not cached — their read() is an identity function over the row the caller already supplied.

Cold cache is the known gap. If mcpd restarts while the backend is unreachable, nothing has a last-known-good value and secret-bearing servers fail to start. That is deliberate: booting a server with an empty credential is worse (gitea-mcp once ran for weeks with an empty GITEA_ACCESS_TOKEN, answering tools/list and reporting healthy while every authenticated call failed). mcpd mitigates it by warming the cache at boot — one read per referenced secret — so an outage that starts after startup is fully absorbed.

Health: live vs ready

curl $MCPD/api/v1/secretbackends/<id>/health

{ "live": true, "liveDetail": "active",
  "ready": false, "readyDetail": "OpenBao list: HTTP 403 permission denied",
  "cache": { "entries": 9, "servingStale": 0 },
  "rotation": { "rotatable": false, "lastRotationError": null } }

• live — unauthenticated sys/health. Distinguishes down from sealed from standby.
• ready — a real read with our credentials.

The two are separate because live && !ready is a distinct, important state: a re-initialised OpenBao hands back valid-looking tokens that grant nothing. Collapsing them into one boolean is what let that go unnoticed for four days.

mcpctl status renders the probe directly:

Secrets:    bao-k8s* ✓ reachable, default ✓ reachable
Secrets:    bao-k8s* ⚠ degraded — serving 7 cached secret(s)
Secrets:    bao-k8s* ✗ unreachable: sealed
Secrets:    bao-k8s* ✗ auth failed: HTTP 403 permission denied
Secrets:    bao-k8s* ? unknown

Historical note. This verdict used to come solely from tokenMeta.lastRotationError, which only the rotator writes — and the rotator skips auth: kubernetes backends. The Secrets line was therefore incapable of going red for a k8s-auth backend, and reported OpenBao healthy while it was unreachable. ? (probe failed) renders yellow, never green: not knowing is not health.

Migration — mcpctl migrate secrets

Atomicity is per secret, not per batch. Remote writes can't roll back, so we don't pretend. For each secret the service:

1. Reads the plaintext from the source driver.
2. Writes it to the destination driver.
3. Updates the Secret row: flips backendId, sets new externalRef, clears data.
4. Deletes from source (skipped with --keep-source).

If the command is interrupted between step 2 and 3, the destination has an orphan entry but the source still owns the row. Re-running is idempotent — the service skips secrets that are already on the destination and picks up the rest.

# Dry-run first: see what would move.
mcpctl migrate secrets --from default --to bao --dry-run

# Migrate everything.
mcpctl migrate secrets --from default --to bao

# Migrate a subset only.
mcpctl migrate secrets --from default --to bao --names api-keys,oauth-client

# Leave the source copy in place (useful for A/B validation).
mcpctl migrate secrets --from default --to bao --keep-source

The command prints a per-secret summary (migrated / skipped / failed) and exits non-zero if any secret failed. Ctrl-C during the run is safe — restart when you want, no duplicate writes.

RBAC

• resource: secretbackends — gated like any other resource (view, create, edit, delete).
• role: run, action: migrate-secrets — required to call POST /api/v1/secrets/migrate.

Describe output masks config values whose keys look like credentials (token, secret, password, key), so mcpctl describe secretbackend is safe to paste into tickets.