mcpctl/docs/reliability.md

# Reliability: don't let a bad LLM take mcpctl down

The homelab model changes often (fast ↔ thinking, model swaps, backends that
drift or go down). mcpctl must stay responsive and honest through all of it.

## Principle

**LLM-*optional* operations must be time-bounded, fall back deterministically,
and report the degradation — never hang and never degrade silently.**

- **Bounded:** every optional LLM call is wrapped in
  [`withTimeout`](../src/mcplocal/src/util/with-timeout.ts) (Promise.race + an
  `AbortSignal` so fetch-based providers actually cancel). A thinking model that
  streams for minutes can never block the caller.
- **Deterministic fallback:** when the LLM times out or errors, use the
  non-LLM path (priority/keyword ordering, byte-range pages).
- **Loud, not silent:** log the reason (`[gate] …`, `[pagination] …`) and tell
  the user. `begin_session` prepends `⚠ Smart prompt-selection unavailable
  (<reason>)…` and sets `degraded: true` + `degradedReason` on the audit
  `gate_decision` event.

Applied in: the gate's `begin_session` prompt selection
(`proxymodel/plugins/gate.ts`, cap `MCPCTL_GATE_LLM_TIMEOUT_MS`, default 8s) and
pagination's smart index (`llm/pagination.ts`, `MCPCTL_PAGINATION_LLM_TIMEOUT_MS`,
default 10s). `read_prompts` is LLM-free by design.

Note: the gate's prompt-ranking uses the **heavy client provider's own model** —
it deliberately does *not* force the project's vLLM model onto it (doing so made
every selection fail silently when the model wasn't anthropic-servable).

## LLM-*essential* operations — failover chain

Chat needs *an* LLM but not a *specific* one. Instead of failing when the pinned
model is down, chat **fails over across an ordered chain** and **reports which
model actually answered**.

- **Chain:** an `Llm` declares fallbacks in `extraConfig.fallbacks: string[]`
  (Llm names, in order). The dispatcher builds an ordered candidate list —
  the primary's pool, then each fallback's pool — and tries them in order.
- **Fails over on real failures**, not just transport: a non-2xx status
  (e.g. a drifted model's `400`) or an empty/invalid completion now advances to
  the next candidate (`chat.service.ts` `runOneInference`). Streaming fails over
  pre-first-chunk.
- **Transparency:** `ChatResult` / the SSE `final` frame carry `llm`, `model`,
  and `failedOver`. The CLI prints `model: <llm> (<model>)` per turn, and
  `⚠ failed over → answered by <llm> (<model>)` when a fallback was used.
- **Exhaustion is clear:** if every candidate fails, the error names the last
  model + upstream status/body (not "no choice").

Homelab chain: `vllm-current` (the served vLLM) → an `anthropic-fallback` server
Llm as the always-up last resort. The chain is owned declaratively by Pulumi
(`kubernetes-deployment/deployments/mcpctl/llm-target.ts`, `fallbacks` arg) so a
`pulumi up` can't wipe it — mcpd *replaces* `extraConfig` on update.

**A real last resort needs an independent cloud credential.** LiteLLM only fronts
the single local vLLM, so any local model shares the same GPU-box failure as the
primary. `anthropic-fallback` points at `api.anthropic.com` and only serves once
mcpd Secret `anthropic-key` holds a genuine **`sk-ant-api03` API key** (from the
Anthropic Console). A Claude *subscription* OAuth token (`sk-ant-oat…`, what
`claude setup-token` mints) does **not** work: Anthropic gates those to the Claude
Code client — probed directly they 404 on older model ids and 429 on current ones.
mcpd's anthropic adapter does send OAuth tokens via `Authorization: Bearer` (so
auth *passes*), but the gating is server-side and unavoidable. Until a real key is
set, the fallback is wired-but-inert: an outage fails over to it and surfaces a
clear `anthropic-fallback (model) HTTP 4xx` error rather than improving uptime.