fix(health): a passing tools/list is live, not healthy
Some checks failed
CI/CD / lint (pull_request) Successful in 1m16s
CI/CD / test (pull_request) Successful in 1m29s
CI/CD / typecheck (pull_request) Successful in 3m15s
CI/CD / smoke (pull_request) Failing after 2m0s
CI/CD / build (pull_request) Successful in 5m5s
CI/CD / publish (pull_request) Has been skipped
Some checks failed
CI/CD / lint (pull_request) Successful in 1m16s
CI/CD / test (pull_request) Successful in 1m29s
CI/CD / typecheck (pull_request) Successful in 3m15s
CI/CD / smoke (pull_request) Failing after 2m0s
CI/CD / build (pull_request) Successful in 5m5s
CI/CD / publish (pull_request) Has been skipped
`mcpctl get instances` showed all eight servers healthy while the UniFi one
had never once reached its controller. The default probe is `tools/list`,
which MCP servers answer from a static in-process table — no credentials, no
upstream, ~3ms. It cannot fail for any reason the user cares about, so it was
reporting `healthy` for every process that managed to start.
Split the two passes:
healthy — readiness: `tools/call` on `healthCheck.tool`. The upstream
answered, so the server can actually do its job.
live — liveness: `tools/list` only. Process up, upstream unverified.
`live` is now the default for any server without a `healthCheck.tool`. It is
not a warning; it is an admission that nothing is watching that server. Probe
events name which probe ran and which tool ("Readiness check (list_sites)
passed"), so the events log distinguishes the two after the fact.
Also:
- `healthCheck.tool` is optional now, so the timings can be tuned without
inventing a readiness probe.
- `create server --health-check-tool/-args/-interval/-timeout/
-failure-threshold`, per the rule that everything applyable is a create
flag. Merges over a `--from-template` healthCheck rather than replacing it.
- `describe instance` explains a `live` verdict instead of leaving it cryptic.
- create.ts held a raw NUL byte in a string literal, which made grep treat the
whole file as binary and silently skip it. Escaped as `\0`.
Verified against the live fleet: with readiness probes configured, my-grafana
went unhealthy (Grafana API 403) and my-node-red degraded (connect timeout to
a Tailscale address) — both had read healthy for months.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0114dg56YmVacyqhp5fitcTb
This commit is contained in:
@@ -28,6 +28,66 @@ Note: the gate's prompt-ranking uses the **heavy client provider's own model**
|
||||
it deliberately does *not* force the project's vLLM model onto it (doing so made
|
||||
every selection fail silently when the model wasn't anthropic-servable).
|
||||
|
||||
## Instance health: `live` is not `healthy`
|
||||
|
||||
An MCP server answers `tools/list` from a **static, in-process table**. It costs
|
||||
a few milliseconds, needs no credentials, and reaches no upstream — so it stays
|
||||
green while the thing the server exists to talk to is unreachable. Treating that
|
||||
as a health signal is how `mcpctl get instances` showed eight healthy servers
|
||||
while the UniFi one had never once reached its controller.
|
||||
|
||||
So the probe reports two different passes:
|
||||
|
||||
| Status | Probe | Means |
|
||||
|---|---|---|
|
||||
| `healthy` | **readiness** — `tools/call` on `healthCheck.tool` | The upstream answered. The server can do its job. |
|
||||
| `live` | **liveness** — `tools/list` only | The process is up and speaks MCP. Its upstream is **unverified**. |
|
||||
| `degraded` | either, failing | Failing, but under `failureThreshold`. |
|
||||
| `unhealthy` | either, failing | Failed `failureThreshold` times in a row. |
|
||||
|
||||
`live` is the default for any server with no `healthCheck.tool`. It is not a
|
||||
warning — it is an admission that nothing is watching that server's upstream.
|
||||
|
||||
**Configure a readiness probe on every server.** Pick a read-only tool that
|
||||
genuinely round-trips to the upstream, and verify it passes before configuring
|
||||
it — a probe naming a local-only tool (`get_..._version`) or a tool the server
|
||||
doesn't expose reproduces the same false green it was meant to remove.
|
||||
|
||||
```bash
|
||||
mcpctl create server unifi-network --health-check-tool list_sites \
|
||||
--health-check-interval 60 --health-check-timeout 15 --force
|
||||
```
|
||||
|
||||
or declaratively — `healthCheck` round-trips through `get -o yaml | apply -f`:
|
||||
|
||||
```yaml
|
||||
healthCheck:
|
||||
tool: list_sites
|
||||
arguments: {}
|
||||
intervalSeconds: 60
|
||||
timeoutSeconds: 15
|
||||
failureThreshold: 3
|
||||
```
|
||||
|
||||
Omit `tool` to keep liveness while still tuning the timings.
|
||||
|
||||
Latency is the tell: a probe answering in single-digit milliseconds is reading a
|
||||
local table, not crossing a network. The UniFi probe went from 3ms (`tools/list`,
|
||||
lying) to 1847ms on its first real `list_sites` — login, TLS, controller round
|
||||
trip — and ~40ms once the session was warm.
|
||||
|
||||
### Two failure modes the probe cannot see for you
|
||||
|
||||
The healthy-looking UniFi server was broken **twice over**, and both are worth
|
||||
checking first when a readiness probe starts failing:
|
||||
|
||||
1. **Egress.** MCP server pods default to TCP 80/443 only
|
||||
(`servers-allow-external-egress`). Any upstream on another port — the UniFi
|
||||
controller on `:8443` — times out on every call. Declare it in Pulumi's
|
||||
`mcpctl.serverEgressTargets` (name + `/32` + ports); don't widen the blanket rule.
|
||||
2. **Address reachability.** A pod cannot reach a **Tailscale** `100.64.0.0/10`
|
||||
address. Config pointing at one connect-timeouts forever. Use LAN IPs.
|
||||
|
||||
## LLM-*essential* operations — failover chain
|
||||
|
||||
Chat needs *an* LLM but not a *specific* one. Instead of failing when the pinned
|
||||
|
||||
Reference in New Issue
Block a user