fix(health): a passing tools/list is live, not healthy
Some checks failed
CI/CD / lint (pull_request) Successful in 1m16s
CI/CD / test (pull_request) Successful in 1m29s
CI/CD / typecheck (pull_request) Successful in 3m15s
CI/CD / smoke (pull_request) Failing after 2m0s
CI/CD / build (pull_request) Successful in 5m5s
CI/CD / publish (pull_request) Has been skipped

`mcpctl get instances` showed all eight servers healthy while the UniFi one
had never once reached its controller. The default probe is `tools/list`,
which MCP servers answer from a static in-process table — no credentials, no
upstream, ~3ms. It cannot fail for any reason the user cares about, so it was
reporting `healthy` for every process that managed to start.

Split the two passes:

  healthy  — readiness: `tools/call` on `healthCheck.tool`. The upstream
             answered, so the server can actually do its job.
  live     — liveness: `tools/list` only. Process up, upstream unverified.

`live` is now the default for any server without a `healthCheck.tool`. It is
not a warning; it is an admission that nothing is watching that server. Probe
events name which probe ran and which tool ("Readiness check (list_sites)
passed"), so the events log distinguishes the two after the fact.

Also:
- `healthCheck.tool` is optional now, so the timings can be tuned without
  inventing a readiness probe.
- `create server --health-check-tool/-args/-interval/-timeout/
  -failure-threshold`, per the rule that everything applyable is a create
  flag. Merges over a `--from-template` healthCheck rather than replacing it.
- `describe instance` explains a `live` verdict instead of leaving it cryptic.
- create.ts held a raw NUL byte in a string literal, which made grep treat the
  whole file as binary and silently skip it. Escaped as `\0`.

Verified against the live fleet: with readiness probes configured, my-grafana
went unhealthy (Grafana API 403) and my-node-red degraded (connect timeout to
a Tailscale address) — both had read healthy for months.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0114dg56YmVacyqhp5fitcTb
This commit is contained in:
Michal
2026-08-09 21:23:33 +01:00
parent ff0e71da05
commit b2547429ca
11 changed files with 495 additions and 9 deletions

View File

@@ -28,6 +28,66 @@ Note: the gate's prompt-ranking uses the **heavy client provider's own model**
it deliberately does *not* force the project's vLLM model onto it (doing so made
every selection fail silently when the model wasn't anthropic-servable).
## Instance health: `live` is not `healthy`
An MCP server answers `tools/list` from a **static, in-process table**. It costs
a few milliseconds, needs no credentials, and reaches no upstream — so it stays
green while the thing the server exists to talk to is unreachable. Treating that
as a health signal is how `mcpctl get instances` showed eight healthy servers
while the UniFi one had never once reached its controller.
So the probe reports two different passes:
| Status | Probe | Means |
|---|---|---|
| `healthy` | **readiness**`tools/call` on `healthCheck.tool` | The upstream answered. The server can do its job. |
| `live` | **liveness**`tools/list` only | The process is up and speaks MCP. Its upstream is **unverified**. |
| `degraded` | either, failing | Failing, but under `failureThreshold`. |
| `unhealthy` | either, failing | Failed `failureThreshold` times in a row. |
`live` is the default for any server with no `healthCheck.tool`. It is not a
warning — it is an admission that nothing is watching that server's upstream.
**Configure a readiness probe on every server.** Pick a read-only tool that
genuinely round-trips to the upstream, and verify it passes before configuring
it — a probe naming a local-only tool (`get_..._version`) or a tool the server
doesn't expose reproduces the same false green it was meant to remove.
```bash
mcpctl create server unifi-network --health-check-tool list_sites \
--health-check-interval 60 --health-check-timeout 15 --force
```
or declaratively — `healthCheck` round-trips through `get -o yaml | apply -f`:
```yaml
healthCheck:
tool: list_sites
arguments: {}
intervalSeconds: 60
timeoutSeconds: 15
failureThreshold: 3
```
Omit `tool` to keep liveness while still tuning the timings.
Latency is the tell: a probe answering in single-digit milliseconds is reading a
local table, not crossing a network. The UniFi probe went from 3ms (`tools/list`,
lying) to 1847ms on its first real `list_sites` — login, TLS, controller round
trip — and ~40ms once the session was warm.
### Two failure modes the probe cannot see for you
The healthy-looking UniFi server was broken **twice over**, and both are worth
checking first when a readiness probe starts failing:
1. **Egress.** MCP server pods default to TCP 80/443 only
(`servers-allow-external-egress`). Any upstream on another port — the UniFi
controller on `:8443` — times out on every call. Declare it in Pulumi's
`mcpctl.serverEgressTargets` (name + `/32` + ports); don't widen the blanket rule.
2. **Address reachability.** A pod cannot reach a **Tailscale** `100.64.0.0/10`
address. Config pointing at one connect-timeouts forever. Use LAN IPs.
## LLM-*essential* operations — failover chain
Chat needs *an* LLM but not a *specific* one. Instead of failing when the pinned