Merge pull request 'feat(secrets): survive OpenBao outages and report real backend health' (#115) from feat/openbao-resilience into main

docs/secret-backends.md

@@ -118,8 +118,110 @@ That's the whole point of keeping plaintext around — it's the trust root:
    token itself. DB access is now equivalent to OpenBao token access (a single
    key), not equivalent to all API keys in the system.
 
-Follow-up work (not shipped yet) replaces static token auth with Kubernetes
-ServiceAccount auth so no bootstrap token is needed at all.
+#### Kubernetes ServiceAccount auth (no bootstrap token)
+
+`auth: kubernetes` removes the chicken-and-egg entirely: mcpd exchanges its
+projected ServiceAccount JWT for an OpenBao token at
+`auth/<authMount>/role/<role>`, so there is no static credential in the database
+at all. The token is cached for its lease and re-minted lazily with a 60s grace
+window.
+
+```yaml
+kind: secretbackend
+name: bao-k8s
+type: openbao
+isDefault: true
+config:
+  url: https://bao.example
+  auth: kubernetes
+  role: mcpctl
+  authMount: kubernetes-worker0   # defaults to `kubernetes`
+```
+
+Note that the daily **rotator does not apply** to these backends — there is no
+stored token to rotate. That has a consequence for monitoring, see below.
+
+## Reliability
+
+Remote backends are network dependencies on the critical path of nearly
+everything: server env resolution, LLM api keys, chat, git providers, code
+repos, webhooks. Three mechanisms keep an outage from cascading.
+
+### Request hardening
+
+Every call carries a timeout (default 5s) and retries `5xx`/`429`/network
+failures with full-jitter exponential backoff (3 attempts). A **sealed** OpenBao
+answers `503`, so this covers unseal windows and failovers.
+
+The `403` path is separate and deliberately single-shot: the driver purges its
+cached token, re-authenticates and retries **once**. That is a credential
+refresh, not a backend-unavailable condition — looping on it would hide a
+genuinely revoked grant.
+
+### Value cache with stale-while-error
+
+Resolved values are cached per backend (default TTL 5 minutes, LRU-bounded).
+Past the TTL the backend is always consulted; if it fails *as a transport
+failure*, the last known-good value is served instead of throwing.
+
+| Failure | Behaviour |
+|---|---|
+| Backend unreachable / timeout / exhausted 5xx | Serve last known-good, mark degraded, log `BACKEND_UNREACHABLE` once |
+| Secret deleted (404) | **Evict and throw.** Never served stale — that would resurrect a revoked credential |
+| 403 after a token refresh | Throw. Revoked grants must stay loud |
+| Nothing cached yet | Throw |
+
+The stale window is unbounded on purpose: a cap would mean a long outage
+eventually takes mcpd down anyway.
+
+`plaintext` backends are not cached — their `read()` is an identity function
+over the row the caller already supplied.
+
+**Cold cache is the known gap.** If mcpd restarts *while* the backend is
+unreachable, nothing has a last-known-good value and secret-bearing servers fail
+to start. That is deliberate: booting a server with an empty credential is worse
+(gitea-mcp once ran for weeks with an empty `GITEA_ACCESS_TOKEN`, answering
+`tools/list` and reporting healthy while every authenticated call failed). mcpd
+mitigates it by warming the cache at boot — one read per referenced secret — so
+an outage that starts *after* startup is fully absorbed.
+
+### Health: `live` vs `ready`
+
+```bash
+curl $MCPD/api/v1/secretbackends/<id>/health
+```
+
+```json
+{ "live": true, "liveDetail": "active",
+  "ready": false, "readyDetail": "OpenBao list: HTTP 403 permission denied",
+  "cache": { "entries": 9, "servingStale": 0 },
+  "rotation": { "rotatable": false, "lastRotationError": null } }
+```
+
+- **`live`** — unauthenticated `sys/health`. Distinguishes *down* from *sealed*
+  from *standby*.
+- **`ready`** — a real read with our credentials.
+
+The two are separate because `live && !ready` is a distinct, important state: a
+re-initialised OpenBao hands back valid-looking tokens that grant nothing.
+Collapsing them into one boolean is what let that go unnoticed for four days.
+
+`mcpctl status` renders the probe directly:
+
+```
+Secrets:    bao-k8s* ✓ reachable, default ✓ reachable
+Secrets:    bao-k8s* ⚠ degraded — serving 7 cached secret(s)
+Secrets:    bao-k8s* ✗ unreachable: sealed
+Secrets:    bao-k8s* ✗ auth failed: HTTP 403 permission denied
+Secrets:    bao-k8s* ? unknown
+```
+
+> **Historical note.** This verdict used to come solely from
+> `tokenMeta.lastRotationError`, which only the rotator writes — and the rotator
+> skips `auth: kubernetes` backends. The Secrets line was therefore *incapable*
+> of going red for a k8s-auth backend, and reported OpenBao healthy while it was
+> unreachable. `?` (probe failed) renders yellow, never green: not knowing is not
+> health.
 
 ## Migration — `mcpctl migrate secrets`