38 lines
1.9 KiB
Markdown
38 lines
1.9 KiB
Markdown
|
# Reliability: don't let a bad LLM take mcpctl down
|
||
|
|
|
||
|
|
The homelab model changes often (fast ↔ thinking, model swaps, backends that
|
||
|
|
drift or go down). mcpctl must stay responsive and honest through all of it.
|
||
|
|
|
||
|
|
## Principle
|
||
|
|
|
||
|
|
**LLM-*optional* operations must be time-bounded, fall back deterministically,
|
||
|
|
and report the degradation — never hang and never degrade silently.**
|
||
|
|
|
||
|
|
- **Bounded:** every optional LLM call is wrapped in
|
||
|
|
[`withTimeout`](../src/mcplocal/src/util/with-timeout.ts) (Promise.race + an
|
||
|
|
`AbortSignal` so fetch-based providers actually cancel). A thinking model that
|
||
|
|
streams for minutes can never block the caller.
|
||
|
|
- **Deterministic fallback:** when the LLM times out or errors, use the
|
||
|
|
non-LLM path (priority/keyword ordering, byte-range pages).
|
||
|
|
- **Loud, not silent:** log the reason (`[gate] …`, `[pagination] …`) and tell
|
||
|
|
the user. `begin_session` prepends `⚠ Smart prompt-selection unavailable
|
||
|
|
(<reason>)…` and sets `degraded: true` + `degradedReason` on the audit
|
||
|
|
`gate_decision` event.
|
||
|
|
|
||
|
|
Applied in: the gate's `begin_session` prompt selection
|
||
|
|
(`proxymodel/plugins/gate.ts`, cap `MCPCTL_GATE_LLM_TIMEOUT_MS`, default 8s) and
|
||
|
|
pagination's smart index (`llm/pagination.ts`, `MCPCTL_PAGINATION_LLM_TIMEOUT_MS`,
|
||
|
|
default 10s). `read_prompts` is LLM-free by design.
|
||
|
|
|
||
|
|
Note: the gate's prompt-ranking uses the **heavy client provider's own model** —
|
||
|
|
it deliberately does *not* force the project's vLLM model onto it (doing so made
|
||
|
|
every selection fail silently when the model wasn't anthropic-servable).
|
||
|
|
|
||
|
|
## LLM-*essential* operations
|
||
|
|
|
||
|
|
Chat needs the LLM — it can't fall back. It must **fail fast with a clear,
|
||
|
|
actionable message** instead of a vague one. Chat surfaces the upstream status +
|
||
|
|
body (which names the model + reason), e.g.
|
||
|
|
`LLM returned no completion (HTTP 400): {"error":"model deepseek-v4-flash not found"}`,
|
||
|
|
rather than "Adapter returned no choice".
|