mcpd re-read the secret backend on every use — server env resolution, LLM
api keys, chat, git providers, code repos, webhooks — with no value cache,
no request timeout, and a retry that only fired on HTTP 403. A few seconds
of OpenBao unavailability therefore turned into minutes of degraded
service: instances that restarted during the blip failed env resolution,
got marked ERROR, and entered the 30s x5 then 5min backoff.
Three changes, in dependency order:
1. Typed errors. `SecretNotFoundError` (definitive) vs
`SecretBackendUnavailableError` (transport). The distinction has to be
typed rather than string-matched — a mis-classified "not found" would
resurrect deleted secrets, and a mis-classified auth failure would
paper over revoked grants, which is how an upstream re-init once broke
every secret write for four days (e51b924).
2. Driver resilience. Every request now carries an AbortSignal timeout
(there was none, so an unreachable backend hung its caller) and retries
5xx/429/network with full-jitter backoff — 503 is what a sealed
OpenBao returns and used to be an immediate hard failure. The 403
purge-and-retry stays single-shot and outside the retry budget: it is a
credential refresh, not a backend-unavailable condition, and looping on
it would hide a genuinely revoked grant.
`healthCheck()` no longer routes through the authenticated path, so an
expired role stops reporting as "OpenBao is down"; it maps OpenBao's
status codes (sealed/standby/uninitialised) instead. New `authCheck()`
covers the readiness half via list(), which exercises the capability we
actually depend on — unlike lookup-self, which only proves the token
exists.
3. CachingSecretBackendDriver. Fresh reads inside the TTL never touch the
network; past it we always try the backend, and on a transport failure
serve the last known-good value instead of throwing. That is what stops
the ERROR storm. Deleted secrets evict and rethrow — serving those
stale would resurrect a revoked credential, strictly worse than an
outage — and non-transport errors rethrow untouched. plaintext is not
wrapped: its read() is an identity function over the row handed in.
A cold cache during an outage still fails, loudly and by design (e6cd735).
Also routes BACKEND_TOKEN_DEAD / BACKEND_ROTATION_FAILED through pino
rather than bare console.error. They bypassed the multistream feeding
ErrorLogBuffer, so the one failure `mcpctl errors` exists to surface was
the one it never showed.
Tests: 20 new. The two load-bearing guards (never serve a deleted secret
stale; never serve stale for a non-transport error) were confirmed to fail
against deliberately broken code before being kept. The 403 purge-retry
path had no coverage at all until now.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018vybEitX4FykeMatKe5Xki
bd3e1134c9