michal/mcpctl
1
0
Fork 0
Code Issues Pull Requests 7 Actions Packages Projects Releases Wiki Activity

fix(health): a passing tools/list is live, not healthy #104

Merged
michal merged 3 commits from worktree-fix-healthcheck-unifi into main 2026-08-09 23:01:02 +00:00
Conversation 0 Commits 3 Files Changed 21 +19 -7
1 changed files with 19 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Showing only changes of commit 732ca98ccc - Show all commits

26
docs/reliability.md
View File

@@ -76,17 +76,29 @@ local table, not crossing a network. The UniFi probe went from 3ms (`tools/list`
lying) to 1847ms on its first real `list_sites` — login, TLS, controller round
trip — and ~40ms once the session was warm.
### Two failure modes the probe cannot see for you
### Where a failing readiness probe usually points
The healthy-looking UniFi server was broken **twice over**, and both are worth
checking first when a readiness probe starts failing:
Turning these probes on for the first time took the fleet from "8/8 healthy" to
three genuine failures in under a minute. All three were network shape, not
code — check these before suspecting the server:
1. **Egress.** MCP server pods default to TCP 80/443 only
1. **Egress port.** MCP server pods default to TCP 80/443 only
(`servers-allow-external-egress`). Any upstream on another port — the UniFi
controller on `:8443` — times out on every call. Declare it in Pulumi's
`mcpctl.serverEgressTargets` (name + `/32` + ports); don't widen the blanket rule.
2. **Address reachability.** A pod cannot reach a **Tailscale** `100.64.0.0/10`
address. Config pointing at one connect-timeouts forever. Use LAN IPs.
`mcpctl.serverEgressTargets`; don't widen the blanket rule.
2. **Ingress hairpin.** A co-located service reached over its *public* hostname
goes out and back through the per-host Envoy L7 policy, which doesn't
reliably carry the caller's identity and replies with a bare `Access denied`.
Grafana 403'd on every call this way while the identical token succeeded from
a laptop. The tell is the error *shape*: plain text, not the upstream's own
JSON error. Use the ClusterIP (`serverEgressTargets` with `namespace:`).
3. **Address reachability.** A pod cannot reach a **Tailscale** `100.64.0.0/10`
address. Config pointing at one connect-timeouts forever. Use LAN IPs. (This
one turned out to be a retired service, which is its own kind of answer.)
Also check the *dialect*: UniFi's `controller_type` must be `classic` for a
self-hosted controller (login `/api/login`, no `/proxy/network` prefix).
`unifi_os` sends every request to a path that 404s.
## LLM-*essential* operations — failover chain