485b8f613f
mcpctl hung on prompt-reading whenever the LLM misbehaved (thinking model = minutes; drifted model = silent failure). Root cause: the gate's begin_session prompt-selection called the LLM with no timeout, its fallback only fired on error and was silent, and it forced the project's vLLM model onto the anthropic heavy provider (so selection failed silently every time). - New withTimeout(run, ms, label): Promise.race + AbortSignal (fetch-based providers cancel). CompletionOptions.signal threaded into anthropic/openai. - Gate begin_session: LLM selection is time-bounded (MCPCTL_GATE_LLM_TIMEOUT_MS, 8s); on timeout/error it falls back to deterministic tag matching, logs [gate] loudly, prepends a ⚠ degraded note to the response, and sets degraded/degradedReason on the audit gate_decision. - Gate selector no longer forces the project vLLM model — uses the heavy provider's own model (fixes the always-silent-fail bug). - Pagination smart-index is time-bounded too (falls back to byte-range pages). - Chat (LLM-essential) surfaces the upstream status+body (names model+reason) instead of "Adapter returned no choice". - docs/reliability.md documents the principle. Tests: with-timeout unit tests; gate degradation test (error → visible ⚠ + deterministic prompts, no hang). mcplocal 737 + mcpd 945 green; tsc clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1.9 KiB
Reliability: don't let a bad LLM take mcpctl down
The homelab model changes often (fast ↔ thinking, model swaps, backends that drift or go down). mcpctl must stay responsive and honest through all of it.
Principle
LLM-optional operations must be time-bounded, fall back deterministically, and report the degradation — never hang and never degrade silently.
- Bounded: every optional LLM call is wrapped in
withTimeout(Promise.race + anAbortSignalso fetch-based providers actually cancel). A thinking model that streams for minutes can never block the caller. - Deterministic fallback: when the LLM times out or errors, use the non-LLM path (priority/keyword ordering, byte-range pages).
- Loud, not silent: log the reason (
[gate] …,[pagination] …) and tell the user.begin_sessionprepends⚠ Smart prompt-selection unavailable (<reason>)…and setsdegraded: true+degradedReasonon the auditgate_decisionevent.
Applied in: the gate's begin_session prompt selection
(proxymodel/plugins/gate.ts, cap MCPCTL_GATE_LLM_TIMEOUT_MS, default 8s) and
pagination's smart index (llm/pagination.ts, MCPCTL_PAGINATION_LLM_TIMEOUT_MS,
default 10s). read_prompts is LLM-free by design.
Note: the gate's prompt-ranking uses the heavy client provider's own model — it deliberately does not force the project's vLLM model onto it (doing so made every selection fail silently when the model wasn't anthropic-servable).
LLM-essential operations
Chat needs the LLM — it can't fall back. It must fail fast with a clear,
actionable message instead of a vague one. Chat surfaces the upstream status +
body (which names the model + reason), e.g.
LLM returned no completion (HTTP 400): {"error":"model deepseek-v4-flash not found"},
rather than "Adapter returned no choice".