88025d0ef5
Some checks failed
CI/CD / typecheck (pull_request) Successful in 1m14s
CI/CD / lint (pull_request) Successful in 2m15s
CI/CD / test (pull_request) Successful in 1m27s
CI/CD / smoke (pull_request) Failing after 2m47s
CI/CD / build (pull_request) Successful in 2m23s
CI/CD / publish (pull_request) Has been skipped
Record the empirical finding: a Claude subscription OAuth token is gated by Anthropic (404/429 as a raw API) and can't serve as the server-side fallback, even though mcpd's adapter now sends it via Bearer. The anthropic-fallback is wired-but-inert until mcpd Secret anthropic-key holds a real console API key. Also note the chain is now Pulumi-owned so pulumi up won't wipe it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
66 lines
3.7 KiB
Markdown
66 lines
3.7 KiB
Markdown
# Reliability: don't let a bad LLM take mcpctl down
|
|
|
|
The homelab model changes often (fast ↔ thinking, model swaps, backends that
|
|
drift or go down). mcpctl must stay responsive and honest through all of it.
|
|
|
|
## Principle
|
|
|
|
**LLM-*optional* operations must be time-bounded, fall back deterministically,
|
|
and report the degradation — never hang and never degrade silently.**
|
|
|
|
- **Bounded:** every optional LLM call is wrapped in
|
|
[`withTimeout`](../src/mcplocal/src/util/with-timeout.ts) (Promise.race + an
|
|
`AbortSignal` so fetch-based providers actually cancel). A thinking model that
|
|
streams for minutes can never block the caller.
|
|
- **Deterministic fallback:** when the LLM times out or errors, use the
|
|
non-LLM path (priority/keyword ordering, byte-range pages).
|
|
- **Loud, not silent:** log the reason (`[gate] …`, `[pagination] …`) and tell
|
|
the user. `begin_session` prepends `⚠ Smart prompt-selection unavailable
|
|
(<reason>)…` and sets `degraded: true` + `degradedReason` on the audit
|
|
`gate_decision` event.
|
|
|
|
Applied in: the gate's `begin_session` prompt selection
|
|
(`proxymodel/plugins/gate.ts`, cap `MCPCTL_GATE_LLM_TIMEOUT_MS`, default 8s) and
|
|
pagination's smart index (`llm/pagination.ts`, `MCPCTL_PAGINATION_LLM_TIMEOUT_MS`,
|
|
default 10s). `read_prompts` is LLM-free by design.
|
|
|
|
Note: the gate's prompt-ranking uses the **heavy client provider's own model** —
|
|
it deliberately does *not* force the project's vLLM model onto it (doing so made
|
|
every selection fail silently when the model wasn't anthropic-servable).
|
|
|
|
## LLM-*essential* operations — failover chain
|
|
|
|
Chat needs *an* LLM but not a *specific* one. Instead of failing when the pinned
|
|
model is down, chat **fails over across an ordered chain** and **reports which
|
|
model actually answered**.
|
|
|
|
- **Chain:** an `Llm` declares fallbacks in `extraConfig.fallbacks: string[]`
|
|
(Llm names, in order). The dispatcher builds an ordered candidate list —
|
|
the primary's pool, then each fallback's pool — and tries them in order.
|
|
- **Fails over on real failures**, not just transport: a non-2xx status
|
|
(e.g. a drifted model's `400`) or an empty/invalid completion now advances to
|
|
the next candidate (`chat.service.ts` `runOneInference`). Streaming fails over
|
|
pre-first-chunk.
|
|
- **Transparency:** `ChatResult` / the SSE `final` frame carry `llm`, `model`,
|
|
and `failedOver`. The CLI prints `model: <llm> (<model>)` per turn, and
|
|
`⚠ failed over → answered by <llm> (<model>)` when a fallback was used.
|
|
- **Exhaustion is clear:** if every candidate fails, the error names the last
|
|
model + upstream status/body (not "no choice").
|
|
|
|
Homelab chain: `vllm-current` (the served vLLM) → an `anthropic-fallback` server
|
|
Llm as the always-up last resort. The chain is owned declaratively by Pulumi
|
|
(`kubernetes-deployment/deployments/mcpctl/llm-target.ts`, `fallbacks` arg) so a
|
|
`pulumi up` can't wipe it — mcpd *replaces* `extraConfig` on update.
|
|
|
|
**A real last resort needs an independent cloud credential.** LiteLLM only fronts
|
|
the single local vLLM, so any local model shares the same GPU-box failure as the
|
|
primary. `anthropic-fallback` points at `api.anthropic.com` and only serves once
|
|
mcpd Secret `anthropic-key` holds a genuine **`sk-ant-api03` API key** (from the
|
|
Anthropic Console). A Claude *subscription* OAuth token (`sk-ant-oat…`, what
|
|
`claude setup-token` mints) does **not** work: Anthropic gates those to the Claude
|
|
Code client — probed directly they 404 on older model ids and 429 on current ones.
|
|
mcpd's anthropic adapter does send OAuth tokens via `Authorization: Bearer` (so
|
|
auth *passes*), but the gating is server-side and unavoidable. Until a real key is
|
|
set, the fallback is wired-but-inert: an outage fails over to it and surfaces a
|
|
clear `anthropic-fallback (model) HTTP 4xx` error rather than improving uptime.
|