michal/mcpctl
1
0
Fork 0
Code Issues Pull Requests 7 Actions Packages Projects Releases Wiki Activity

docs(reliability): record the three shapes a failing readiness probe takes
Some checks failed
CI/CD / lint (pull_request) Successful in 1m12s
Details
CI/CD / test (pull_request) Successful in 1m23s
Details
CI/CD / typecheck (pull_request) Successful in 3m9s
Details
CI/CD / smoke (pull_request) Failing after 1m58s
Details
CI/CD / build (pull_request) Successful in 4m44s
Details
CI/CD / publish (pull_request) Has been skipped
Details

Browse Source 
Turning readiness probes on took the fleet from 8/8 healthy to three real
failures in under a minute, and all three were network shape rather than code:
an egress port (UniFi :8443), an ingress hairpin through the Envoy L7 policy
(Grafana 403 `Access denied` with a token that worked from a laptop), and a
Tailscale address a pod can never reach (Node-RED, since retired).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0114dg56YmVacyqhp5fitcTb
This commit is contained in:
Michal
2026-08-09 22:54:33 +01:00
parent b2547429ca
commit 732ca98ccc
1 changed files with 19 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files

26
docs/reliability.md
View File

@@ -76,17 +76,29 @@ local table, not crossing a network. The UniFi probe went from 3ms (`tools/list`
lying) to 1847ms on its first real `list_sites` — login, TLS, controller round
trip — and ~40ms once the session was warm.
### Two failure modes the probe cannot see for you
### Where a failing readiness probe usually points
The healthy-looking UniFi server was broken **twice over**, and both are worth
checking first when a readiness probe starts failing:
Turning these probes on for the first time took the fleet from "8/8 healthy" to
three genuine failures in under a minute. All three were network shape, not
code — check these before suspecting the server:
1. **Egress.** MCP server pods default to TCP 80/443 only
1. **Egress port.** MCP server pods default to TCP 80/443 only
(`servers-allow-external-egress`). Any upstream on another port — the UniFi
controller on `:8443` — times out on every call. Declare it in Pulumi's
`mcpctl.serverEgressTargets` (name + `/32` + ports); don't widen the blanket rule.
2. **Address reachability.** A pod cannot reach a **Tailscale** `100.64.0.0/10`
address. Config pointing at one connect-timeouts forever. Use LAN IPs.
`mcpctl.serverEgressTargets`; don't widen the blanket rule.
2. **Ingress hairpin.** A co-located service reached over its *public* hostname
goes out and back through the per-host Envoy L7 policy, which doesn't
reliably carry the caller's identity and replies with a bare `Access denied`.
Grafana 403'd on every call this way while the identical token succeeded from
a laptop. The tell is the error *shape*: plain text, not the upstream's own
JSON error. Use the ClusterIP (`serverEgressTargets` with `namespace:`).
3. **Address reachability.** A pod cannot reach a **Tailscale** `100.64.0.0/10`
address. Config pointing at one connect-timeouts forever. Use LAN IPs. (This
one turned out to be a retired service, which is its own kind of answer.)
Also check the *dialect*: UniFi's `controller_type` must be `classic` for a
self-hosted controller (login `/api/login`, no `/proxy/network` prefix).
`unifi_os` sends every request to a path that 404s.
## LLM-*essential* operations — failover chain