mcpctl/docs/reliability.md

Reliability: don't let a bad LLM take mcpctl down

The homelab model changes often (fast ↔ thinking, model swaps, backends that drift or go down). mcpctl must stay responsive and honest through all of it.

Principle

LLM-optional operations must be time-bounded, fall back deterministically, and report the degradation — never hang and never degrade silently.

• Bounded: every optional LLM call is wrapped in withTimeout (Promise.race + an AbortSignal so fetch-based providers actually cancel). A thinking model that streams for minutes can never block the caller.
• Deterministic fallback: when the LLM times out or errors, use the non-LLM path (priority/keyword ordering, byte-range pages).
• Loud, not silent: log the reason ([gate] …, [pagination] …) and tell the user. begin_session prepends ⚠ Smart prompt-selection unavailable (<reason>)… and sets degraded: true + degradedReason on the audit gate_decision event.

One implementation: util/degrade.ts's bounded(). It cannot throw — the caller always gets a value or a reason — so the deterministic fallback is unconditional rather than something a catch block has to remember. degradationNotice() produces the ⚠ … unavailable (reason). hint wording, and degradationAudit() the degraded/degradedReason payload, so every degradation reads the same whichever subsystem produced it.

Nothing optional is unbounded by construction. The budget lives in LLMProviderAdapter.complete() (proxymodel/llm-adapter.ts), not at each call site, so a stage that passes no options is still bounded and a stage written next year inherits the guarantee. One budget spans the whole failover chain — a per-provider timeout would make the worst case N × timeout.

Knob	Default	Bounds
MCPCTL_LLM_CALL_BUDGET_MS	20s	one ctx.llm.complete(), failover included
MCPCTL_LLM_PROVIDER_TIMEOUT_MS	10s	a single provider attempt inside that budget
MCPCTL_STAGE_LLM_BUDGET_MS	30s	all LLM work in one stage invocation
MCPCTL_GATE_LLM_TIMEOUT_MS	8s	the gate's begin_session prompt selection
MCPCTL_PAGINATION_LLM_TIMEOUT_MS	10s	pagination's smart index (llm/pagination.ts)

The stage budget exists because a per-call timeout multiplies rather than bounds when a stage loops: summarize-tree recurses to maxDepth (3) and loops per section at every level, so hundreds of sequential calls are reachable. One budget is shared across the whole recursion; when it is gone the stage switches to first-line excerpts and says so. A warm cache is never budget-gated — a cached summary costs nothing, so an exhausted budget must not degrade a result we already hold.

read_prompts is LLM-free by design.

Why this is written down twice

This document stated the principle while proxymodel/stages/paginate.ts awaited ctx.llm.complete() with no timeout at all. When a provider hung rather than erroring, the promise never settled, the tool call never returned, and the client waited out its own 1800s timeout — three such requests in production, misdiagnosed twice as an upstream "transport fault". The doc named the gate and llm/pagination.ts as the compliant sites, and the newer stages simply never joined the list. Naming one helper here, rather than a list of call sites, is what stops that drift recurring.

Note: the gate's prompt-ranking uses the heavy client provider's own model — it deliberately does not force the project's vLLM model onto it (doing so made every selection fail silently when the model wasn't anthropic-servable).

Instance health: live is not healthy

An MCP server answers tools/list from a static, in-process table. It costs a few milliseconds, needs no credentials, and reaches no upstream — so it stays green while the thing the server exists to talk to is unreachable. Treating that as a health signal is how mcpctl get instances showed eight healthy servers while the UniFi one had never once reached its controller.

So the probe reports two different passes:

Status	Probe	Means
healthy	readiness — tools/call on healthCheck.tool	The upstream answered. The server can do its job.
live	liveness — tools/list only	The process is up and speaks MCP. Its upstream is unverified.
degraded	either, failing	Failing, but under failureThreshold.
unhealthy	either, failing	Failed failureThreshold times in a row.

live is the default for any server with no healthCheck.tool. It is not a warning — it is an admission that nothing is watching that server's upstream.

Configure a readiness probe on every server. Pick a read-only tool that genuinely round-trips to the upstream, and verify it passes before configuring it — a probe naming a local-only tool (get_..._version) or a tool the server doesn't expose reproduces the same false green it was meant to remove.

mcpctl create server unifi-network --health-check-tool list_sites \
  --health-check-interval 60 --health-check-timeout 15 --force

or declaratively — healthCheck round-trips through get -o yaml | apply -f:

healthCheck:
  tool: list_sites
  arguments: {}
  intervalSeconds: 60
  timeoutSeconds: 15
  failureThreshold: 3

Omit tool to keep liveness while still tuning the timings.

Latency is the tell: a probe answering in single-digit milliseconds is reading a local table, not crossing a network. The UniFi probe went from 3ms (tools/list, lying) to 1847ms on its first real list_sites — login, TLS, controller round trip — and ~40ms once the session was warm.

Where a failing readiness probe usually points

Turning these probes on for the first time took the fleet from "8/8 healthy" to three genuine failures in under a minute. All three were network shape, not code — check these before suspecting the server:

1. Egress port. MCP server pods default to TCP 80/443 only (servers-allow-external-egress). Any upstream on another port — the UniFi controller on :8443 — times out on every call. Declare it in Pulumi's mcpctl.serverEgressTargets; don't widen the blanket rule.
2. Ingress hairpin. A co-located service reached over its public hostname goes out and back through the per-host Envoy L7 policy, which doesn't reliably carry the caller's identity and replies with a bare Access denied. Grafana 403'd on every call this way while the identical token succeeded from a laptop. The tell is the error shape: plain text, not the upstream's own JSON error. Use the ClusterIP (serverEgressTargets with namespace:).
3. Address reachability. A pod cannot reach a Tailscale 100.64.0.0/10 address. Config pointing at one connect-timeouts forever. Use LAN IPs. (This one turned out to be a retired service, which is its own kind of answer.)

Also check the dialect: UniFi's controller_type must be classic for a self-hosted controller (login /api/login, no /proxy/network prefix). unifi_os sends every request to a path that 404s.

A server that restarts instead of erroring is out of memory

Server pods get 512 MiB by default (DEFAULT_MEMORY_LIMIT). That is ample for a server that proxies an API and nowhere near enough for one that drives a browser or holds an index in memory.

An OOMKill is the quietest failure in the fleet, because nothing reports an error. The kernel kills the container, Kubernetes restarts it, the readiness probe passes again, and mcpctl get instances reads healthy. Whatever the server was doing is simply gone — for docs, six scrape jobs whose queue lived in memory, leaving an index with 17 pages in it and no failed job to look at.

The tells, in order of how fast they answer the question:

kubectl -n mcpctl-servers get pods | grep <server>     # RESTARTS climbing
kubectl -n mcpctl-servers get pod <pod> -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}'

OOMKilled there is conclusive. mcpctl logs will not show it: the process never got to say anything.

Raise the ceiling per server rather than for the fleet — most servers do not need it, and a bigger default wastes real memory on every node:

mcpctl create server docs --memory-limit-mb 2048 --force

Declared in MiB, stored on the server, converted to bytes in the container spec. Null keeps the 512 MiB default, so existing servers are unchanged. Sizing rule of thumb: measure idle first (/sys/fs/cgroup/memory.current inside the pod), then leave headroom for the peak — docs idles at ~228 MiB and crosses 512 MiB within seconds of a scrape, because each page render is a Chromium process.

LLM-essential operations — failover chain

Chat needs an LLM but not a specific one. Instead of failing when the pinned model is down, chat fails over across an ordered chain and reports which model actually answered.

• Chain: an Llm declares fallbacks in extraConfig.fallbacks: string[] (Llm names, in order). The dispatcher builds an ordered candidate list — the primary's pool, then each fallback's pool — and tries them in order.
• Fails over on real failures, not just transport: a non-2xx status (e.g. a drifted model's 400) or an empty/invalid completion now advances to the next candidate (chat.service.ts runOneInference). Streaming fails over pre-first-chunk.
• Transparency: ChatResult / the SSE final frame carry llm, model, and failedOver. The CLI prints model: <llm> (<model>) per turn, and ⚠ failed over → answered by <llm> (<model>) when a fallback was used.
• Exhaustion is clear: if every candidate fails, the error names the last model + upstream status/body (not "no choice").

Homelab chain: vllm-current (the served vLLM) → an anthropic-fallback server Llm as the always-up last resort. The chain is owned declaratively by Pulumi (kubernetes-deployment/deployments/mcpctl/llm-target.ts, fallbacks arg) so a pulumi up can't wipe it — mcpd replaces extraConfig on update.

A real last resort needs an independent cloud credential. LiteLLM only fronts the single local vLLM, so any local model shares the same GPU-box failure as the primary. anthropic-fallback points at api.anthropic.com and only serves once mcpd Secret anthropic-key holds a genuine sk-ant-api03 API key (from the Anthropic Console). A Claude subscription OAuth token (sk-ant-oat…, what claude setup-token mints) does not work: Anthropic gates those to the Claude Code client — probed directly they 404 on older model ids and 429 on current ones. mcpd's anthropic adapter does send OAuth tokens via Authorization: Bearer (so auth passes), but the gating is server-side and unavoidable. Until a real key is set, the fallback is wired-but-inert: an outage fails over to it and surfaces a clear anthropic-fallback (model) HTTP 4xx error rather than improving uptime.