mcpctl/docs/reliability.md

Reliability: don't let a bad LLM take mcpctl down

The homelab model changes often (fast ↔ thinking, model swaps, backends that drift or go down). mcpctl must stay responsive and honest through all of it.

Principle

LLM-optional operations must be time-bounded, fall back deterministically, and report the degradation — never hang and never degrade silently.

• Bounded: every optional LLM call is wrapped in withTimeout (Promise.race + an AbortSignal so fetch-based providers actually cancel). A thinking model that streams for minutes can never block the caller.
• Deterministic fallback: when the LLM times out or errors, use the non-LLM path (priority/keyword ordering, byte-range pages).
• Loud, not silent: log the reason ([gate] …, [pagination] …) and tell the user. begin_session prepends ⚠ Smart prompt-selection unavailable (<reason>)… and sets degraded: true + degradedReason on the audit gate_decision event.

Applied in: the gate's begin_session prompt selection (proxymodel/plugins/gate.ts, cap MCPCTL_GATE_LLM_TIMEOUT_MS, default 8s) and pagination's smart index (llm/pagination.ts, MCPCTL_PAGINATION_LLM_TIMEOUT_MS, default 10s). read_prompts is LLM-free by design.

Note: the gate's prompt-ranking uses the heavy client provider's own model — it deliberately does not force the project's vLLM model onto it (doing so made every selection fail silently when the model wasn't anthropic-servable).

LLM-essential operations — failover chain

Chat needs an LLM but not a specific one. Instead of failing when the pinned model is down, chat fails over across an ordered chain and reports which model actually answered.

• Chain: an Llm declares fallbacks in extraConfig.fallbacks: string[] (Llm names, in order). The dispatcher builds an ordered candidate list — the primary's pool, then each fallback's pool — and tries them in order.
• Fails over on real failures, not just transport: a non-2xx status (e.g. a drifted model's 400) or an empty/invalid completion now advances to the next candidate (chat.service.ts runOneInference). Streaming fails over pre-first-chunk.
• Transparency: ChatResult / the SSE final frame carry llm, model, and failedOver. The CLI prints model: <llm> (<model>) per turn, and ⚠ failed over → answered by <llm> (<model>) when a fallback was used.
• Exhaustion is clear: if every candidate fails, the error names the last model + upstream status/body (not "no choice").

Homelab chain: vllm-current (the served vLLM) → an anthropic-fallback server Llm as the always-up last resort. The chain is owned declaratively by Pulumi (kubernetes-deployment/deployments/mcpctl/llm-target.ts, fallbacks arg) so a pulumi up can't wipe it — mcpd replaces extraConfig on update.

A real last resort needs an independent cloud credential. LiteLLM only fronts the single local vLLM, so any local model shares the same GPU-box failure as the primary. anthropic-fallback points at api.anthropic.com and only serves once mcpd Secret anthropic-key holds a genuine sk-ant-api03 API key (from the Anthropic Console). A Claude subscription OAuth token (sk-ant-oat…, what claude setup-token mints) does not work: Anthropic gates those to the Claude Code client — probed directly they 404 on older model ids and 429 on current ones. mcpd's anthropic adapter does send OAuth tokens via Authorization: Bearer (so auth passes), but the gating is server-side and unavoidable. Until a real key is set, the fallback is wired-but-inert: an outage fails over to it and surfaces a clear anthropic-fallback (model) HTTP 4xx error rather than improving uptime.