fix(health): a passing tools/list is live, not healthy
Some checks failed
CI/CD / lint (pull_request) Successful in 1m16s
CI/CD / test (pull_request) Successful in 1m29s
CI/CD / typecheck (pull_request) Successful in 3m15s
CI/CD / smoke (pull_request) Failing after 2m0s
CI/CD / build (pull_request) Successful in 5m5s
CI/CD / publish (pull_request) Has been skipped

`mcpctl get instances` showed all eight servers healthy while the UniFi one
had never once reached its controller. The default probe is `tools/list`,
which MCP servers answer from a static in-process table — no credentials, no
upstream, ~3ms. It cannot fail for any reason the user cares about, so it was
reporting `healthy` for every process that managed to start.

Split the two passes:

  healthy  — readiness: `tools/call` on `healthCheck.tool`. The upstream
             answered, so the server can actually do its job.
  live     — liveness: `tools/list` only. Process up, upstream unverified.

`live` is now the default for any server without a `healthCheck.tool`. It is
not a warning; it is an admission that nothing is watching that server. Probe
events name which probe ran and which tool ("Readiness check (list_sites)
passed"), so the events log distinguishes the two after the fact.

Also:
- `healthCheck.tool` is optional now, so the timings can be tuned without
  inventing a readiness probe.
- `create server --health-check-tool/-args/-interval/-timeout/
  -failure-threshold`, per the rule that everything applyable is a create
  flag. Merges over a `--from-template` healthCheck rather than replacing it.
- `describe instance` explains a `live` verdict instead of leaving it cryptic.
- create.ts held a raw NUL byte in a string literal, which made grep treat the
  whole file as binary and silently skip it. Escaped as `\0`.

Verified against the live fleet: with readiness probes configured, my-grafana
went unhealthy (Grafana API 403) and my-node-red degraded (connect timeout to
a Tailscale address) — both had read healthy for months.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0114dg56YmVacyqhp5fitcTb
This commit is contained in:
Michal
2026-08-09 21:23:33 +01:00
parent ff0e71da05
commit b2547429ca
11 changed files with 495 additions and 9 deletions

View File

@@ -123,13 +123,76 @@ describe('HealthProbeRunner', () => {
// No exec fallback — liveness goes through mcpProxyService
expect(orchestrator.execInContainer).not.toHaveBeenCalled();
expect(mcpProxyService.execute).toHaveBeenCalledWith({ serverId: 'srv-1', method: 'tools/list' });
// A passing liveness probe is `live`, never `healthy` — `tools/list` is
// answered in-process and proves nothing about the server's upstream.
expect(instanceRepo.updateStatus).toHaveBeenCalledWith(
'inst-1',
'RUNNING',
expect.objectContaining({ healthStatus: 'healthy' }),
expect.objectContaining({ healthStatus: 'live' }),
);
});
it('reports `live` (not `healthy`) even when the upstream is dead, and says so in the event', async () => {
// The regression this guards: a UniFi server whose controller port was
// firewalled off sat at "healthy" for months because `tools/list` kept
// answering from the in-process tool table.
const instance = makeInstance();
const server = makeServer({ healthCheck: null });
vi.mocked(instanceRepo.findAll).mockResolvedValue([instance]);
vi.mocked(serverRepo.findById).mockResolvedValue(server);
const result = await runner.probeInstance(instance, server, { intervalSeconds: 0 });
expect(result.healthy).toBe(true);
expect(result.probe).toBe('liveness');
const fields = vi.mocked(instanceRepo.updateStatus).mock.calls[0]?.[2];
expect(fields?.healthStatus).toBe('live');
const events = fields?.events as Array<{ message: string }>;
expect(events[events.length - 1]?.message).toContain('Liveness check (tools/list) passed');
});
it('a passing readiness probe earns `healthy` and names the tool in the event', async () => {
const instance = makeInstance();
const server = makeServer({
healthCheck: { tool: 'list_sites', intervalSeconds: 0 } as McpServer['healthCheck'],
});
vi.mocked(instanceRepo.findAll).mockResolvedValue([instance]);
vi.mocked(serverRepo.findById).mockResolvedValue(server);
vi.mocked(mcpProxyService.execute).mockResolvedValue({ jsonrpc: '2.0', id: 1, result: {} });
const result = await runner.probeInstance(instance, server, { tool: 'list_sites' });
expect(result.probe).toBe('readiness');
const fields = vi.mocked(instanceRepo.updateStatus).mock.calls[0]?.[2];
expect(fields?.healthStatus).toBe('healthy');
const events = fields?.events as Array<{ message: string }>;
expect(events[events.length - 1]?.message).toContain('Readiness check (list_sites) passed');
});
it('a readiness probe whose tool call fails reports the upstream error, not `live`', async () => {
const instance = makeInstance();
const server = makeServer({
healthCheck: { tool: 'list_sites', failureThreshold: 1 } as McpServer['healthCheck'],
});
vi.mocked(mcpProxyService.execute).mockResolvedValue({
jsonrpc: '2.0',
id: 1,
error: { code: -32000, message: 'connect ETIMEDOUT 192.168.1.5:8443' },
});
await runner.probeInstance(instance, server, { tool: 'list_sites', failureThreshold: 1 });
const fields = vi.mocked(instanceRepo.updateStatus).mock.calls[0]?.[2];
expect(fields?.healthStatus).toBe('unhealthy');
const events = fields?.events as Array<{ message: string }>;
expect(events[events.length - 1]?.message).toContain('Readiness check (list_sites) failed');
expect(events[events.length - 1]?.message).toContain('ETIMEDOUT');
});
it('default liveness probe marks unhealthy when tools/list returns JSON-RPC error', async () => {
const instance = makeInstance();
const server = makeServer({