mcpctl/docs/reliability.md

# Reliability: don't let a bad LLM take mcpctl down

The homelab model changes often (fast ↔ thinking, model swaps, backends that
drift or go down). mcpctl must stay responsive and honest through all of it.

## Principle

**LLM-*optional* operations must be time-bounded, fall back deterministically,
and report the degradation — never hang and never degrade silently.**

- **Bounded:** every optional LLM call is wrapped in
  [`withTimeout`](../src/mcplocal/src/util/with-timeout.ts) (Promise.race + an
  `AbortSignal` so fetch-based providers actually cancel). A thinking model that
  streams for minutes can never block the caller.
- **Deterministic fallback:** when the LLM times out or errors, use the
  non-LLM path (priority/keyword ordering, byte-range pages).
- **Loud, not silent:** log the reason (`[gate] …`, `[pagination] …`) and tell
  the user. `begin_session` prepends `⚠ Smart prompt-selection unavailable
  (<reason>)…` and sets `degraded: true` + `degradedReason` on the audit
  `gate_decision` event.

**One implementation:** [`util/degrade.ts`](../src/mcplocal/src/util/degrade.ts)'s
`bounded()`. It cannot throw — the caller always gets a value or a reason — so
the deterministic fallback is unconditional rather than something a `catch`
block has to remember. `degradationNotice()` produces the `⚠ … unavailable
(reason). hint` wording, and `degradationAudit()` the `degraded`/`degradedReason`
payload, so every degradation reads the same whichever subsystem produced it.

**Nothing optional is unbounded by construction.** The budget lives in
`LLMProviderAdapter.complete()` (`proxymodel/llm-adapter.ts`), not at each call
site, so a stage that passes no options is still bounded and a stage written
next year inherits the guarantee. One budget spans the whole failover chain —
a per-provider timeout would make the worst case N × timeout.

| Knob | Default | Bounds |
|---|---|---|
| `MCPCTL_LLM_CALL_BUDGET_MS` | 20s | one `ctx.llm.complete()`, failover included |
| `MCPCTL_LLM_PROVIDER_TIMEOUT_MS` | 10s | a single provider attempt inside that budget |
| `MCPCTL_STAGE_LLM_BUDGET_MS` | 30s | all LLM work in one stage invocation |
| `MCPCTL_GATE_LLM_TIMEOUT_MS` | 8s | the gate's `begin_session` prompt selection |
| `MCPCTL_PAGINATION_LLM_TIMEOUT_MS` | 10s | pagination's smart index (`llm/pagination.ts`) |

The **stage** budget exists because a per-call timeout multiplies rather than
bounds when a stage loops: `summarize-tree` recurses to `maxDepth` (3) and loops
per section at every level, so hundreds of sequential calls are reachable. One
budget is shared across the whole recursion; when it is gone the stage switches
to first-line excerpts and says so. A **warm cache is never budget-gated** — a
cached summary costs nothing, so an exhausted budget must not degrade a result
we already hold.

`read_prompts` is LLM-free by design.

### Why this is written down twice

This document stated the principle while `proxymodel/stages/paginate.ts` awaited
`ctx.llm.complete()` with no timeout at all. When a provider hung rather than
erroring, the promise never settled, the tool call never returned, and the
client waited out its own 1800s timeout — three such requests in production,
misdiagnosed twice as an upstream "transport fault". The doc named the gate and
`llm/pagination.ts` as the compliant sites, and the newer stages simply never
joined the list. Naming **one** helper here, rather than a list of call sites,
is what stops that drift recurring.

Note: the gate's prompt-ranking uses the **heavy client provider's own model** —
it deliberately does *not* force the project's vLLM model onto it (doing so made
every selection fail silently when the model wasn't anthropic-servable).

## Instance health: `live` is not `healthy`

An MCP server answers `tools/list` from a **static, in-process table**. It costs
a few milliseconds, needs no credentials, and reaches no upstream — so it stays
green while the thing the server exists to talk to is unreachable. Treating that
as a health signal is how `mcpctl get instances` showed eight healthy servers
while the UniFi one had never once reached its controller.

So the probe reports two different passes:

| Status | Probe | Means |
|---|---|---|
| `healthy` | **readiness** — `tools/call` on `healthCheck.tool` | The upstream answered. The server can do its job. |
| `live` | **liveness** — `tools/list` only | The process is up and speaks MCP. Its upstream is **unverified**. |
| `degraded` | either, failing | Failing, but under `failureThreshold`. |
| `unhealthy` | either, failing | Failed `failureThreshold` times in a row. |

`live` is the default for any server with no `healthCheck.tool`. It is not a
warning — it is an admission that nothing is watching that server's upstream.

**Configure a readiness probe on every server.** Pick a read-only tool that
genuinely round-trips to the upstream, and verify it passes before configuring
it — a probe naming a local-only tool (`get_..._version`) or a tool the server
doesn't expose reproduces the same false green it was meant to remove.

```bash
mcpctl create server unifi-network --health-check-tool list_sites \
  --health-check-interval 60 --health-check-timeout 15 --force
```

or declaratively — `healthCheck` round-trips through `get -o yaml | apply -f`:

```yaml
healthCheck:
  tool: list_sites
  arguments: {}
  intervalSeconds: 60
  timeoutSeconds: 15
  failureThreshold: 3
```

Omit `tool` to keep liveness while still tuning the timings.

Latency is the tell: a probe answering in single-digit milliseconds is reading a
local table, not crossing a network. The UniFi probe went from 3ms (`tools/list`,
lying) to 1847ms on its first real `list_sites` — login, TLS, controller round
trip — and ~40ms once the session was warm.

### Where a failing readiness probe usually points

Turning these probes on for the first time took the fleet from "8/8 healthy" to
three genuine failures in under a minute. All three were network shape, not
code — check these before suspecting the server:

1. **Egress port.** MCP server pods default to TCP 80/443 only
   (`servers-allow-external-egress`). Any upstream on another port — the UniFi
   controller on `:8443` — times out on every call. Declare it in Pulumi's
   `mcpctl.serverEgressTargets`; don't widen the blanket rule.
2. **Ingress hairpin.** A co-located service reached over its *public* hostname
   goes out and back through the per-host Envoy L7 policy, which doesn't
   reliably carry the caller's identity and replies with a bare `Access denied`.
   Grafana 403'd on every call this way while the identical token succeeded from
   a laptop. The tell is the error *shape*: plain text, not the upstream's own
   JSON error. Use the ClusterIP (`serverEgressTargets` with `namespace:`).
3. **Address reachability.** A pod cannot reach a **Tailscale** `100.64.0.0/10`
   address. Config pointing at one connect-timeouts forever. Use LAN IPs. (This
   one turned out to be a retired service, which is its own kind of answer.)

Also check the *dialect*: UniFi's `controller_type` must be `classic` for a
self-hosted controller (login `/api/login`, no `/proxy/network` prefix).
`unifi_os` sends every request to a path that 404s.

### A server that restarts instead of erroring is out of memory

Server pods get **512 MiB** by default (`DEFAULT_MEMORY_LIMIT`). That is ample
for a server that proxies an API and nowhere near enough for one that drives a
browser or holds an index in memory.

An OOMKill is the quietest failure in the fleet, because **nothing reports an
error**. The kernel kills the container, Kubernetes restarts it, the readiness
probe passes again, and `mcpctl get instances` reads `healthy`. Whatever the
server was doing is simply gone — for `docs`, six scrape jobs whose queue lived
in memory, leaving an index with 17 pages in it and no failed job to look at.

The tells, in order of how fast they answer the question:

```bash
kubectl -n mcpctl-servers get pods | grep <server>     # RESTARTS climbing
kubectl -n mcpctl-servers get pod <pod> -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}'
```

`OOMKilled` there is conclusive. `mcpctl logs` will not show it: the process
never got to say anything.

Raise the ceiling per server rather than for the fleet — most servers do not
need it, and a bigger default wastes real memory on every node:

```bash
mcpctl create server docs --memory-limit-mb 2048 --force
```

Declared in MiB, stored on the server, converted to bytes in the container
spec. Null keeps the 512 MiB default, so existing servers are unchanged. Sizing
rule of thumb: measure idle first (`/sys/fs/cgroup/memory.current` inside the
pod), then leave headroom for the peak — `docs` idles at ~228 MiB and crosses
512 MiB within seconds of a scrape, because each page render is a Chromium
process.

## LLM-*essential* operations — failover chain

Chat needs *an* LLM but not a *specific* one. Instead of failing when the pinned
model is down, chat **fails over across an ordered chain** and **reports which
model actually answered**.

- **Chain:** an `Llm` declares fallbacks in `extraConfig.fallbacks: string[]`
  (Llm names, in order). The dispatcher builds an ordered candidate list —
  the primary's pool, then each fallback's pool — and tries them in order.
- **Fails over on real failures**, not just transport: a non-2xx status
  (e.g. a drifted model's `400`) or an empty/invalid completion now advances to
  the next candidate (`chat.service.ts` `runOneInference`). Streaming fails over
  pre-first-chunk.
- **Transparency:** `ChatResult` / the SSE `final` frame carry `llm`, `model`,
  and `failedOver`. The CLI prints `model: <llm> (<model>)` per turn, and
  `⚠ failed over → answered by <llm> (<model>)` when a fallback was used.
- **Exhaustion is clear:** if every candidate fails, the error names the last
  model + upstream status/body (not "no choice").

Homelab chain: `vllm-current` (the served vLLM) → an `anthropic-fallback` server
Llm as the always-up last resort. The chain is owned declaratively by Pulumi
(`kubernetes-deployment/deployments/mcpctl/llm-target.ts`, `fallbacks` arg) so a
`pulumi up` can't wipe it — mcpd *replaces* `extraConfig` on update.

**A real last resort needs an independent cloud credential.** LiteLLM only fronts
the single local vLLM, so any local model shares the same GPU-box failure as the
primary. `anthropic-fallback` points at `api.anthropic.com` and only serves once
mcpd Secret `anthropic-key` holds a genuine **`sk-ant-api03` API key** (from the
Anthropic Console). A Claude *subscription* OAuth token (`sk-ant-oat…`, what
`claude setup-token` mints) does **not** work: Anthropic gates those to the Claude
Code client — probed directly they 404 on older model ids and 429 on current ones.
mcpd's anthropic adapter does send OAuth tokens via `Authorization: Bearer` (so
auth *passes*), but the gating is server-side and unavoidable. Until a real key is
set, the fallback is wired-but-inert: an outage fails over to it and surfaces a
clear `anthropic-fallback (model) HTTP 4xx` error rather than improving uptime.