diff --git a/docs/reliability.md b/docs/reliability.md index 5c1f2c5..30423c8 100644 --- a/docs/reliability.md +++ b/docs/reliability.md @@ -76,17 +76,29 @@ local table, not crossing a network. The UniFi probe went from 3ms (`tools/list` lying) to 1847ms on its first real `list_sites` — login, TLS, controller round trip — and ~40ms once the session was warm. -### Two failure modes the probe cannot see for you +### Where a failing readiness probe usually points -The healthy-looking UniFi server was broken **twice over**, and both are worth -checking first when a readiness probe starts failing: +Turning these probes on for the first time took the fleet from "8/8 healthy" to +three genuine failures in under a minute. All three were network shape, not +code — check these before suspecting the server: -1. **Egress.** MCP server pods default to TCP 80/443 only +1. **Egress port.** MCP server pods default to TCP 80/443 only (`servers-allow-external-egress`). Any upstream on another port — the UniFi controller on `:8443` — times out on every call. Declare it in Pulumi's - `mcpctl.serverEgressTargets` (name + `/32` + ports); don't widen the blanket rule. -2. **Address reachability.** A pod cannot reach a **Tailscale** `100.64.0.0/10` - address. Config pointing at one connect-timeouts forever. Use LAN IPs. + `mcpctl.serverEgressTargets`; don't widen the blanket rule. +2. **Ingress hairpin.** A co-located service reached over its *public* hostname + goes out and back through the per-host Envoy L7 policy, which doesn't + reliably carry the caller's identity and replies with a bare `Access denied`. + Grafana 403'd on every call this way while the identical token succeeded from + a laptop. The tell is the error *shape*: plain text, not the upstream's own + JSON error. Use the ClusterIP (`serverEgressTargets` with `namespace:`). +3. **Address reachability.** A pod cannot reach a **Tailscale** `100.64.0.0/10` + address. Config pointing at one connect-timeouts forever. Use LAN IPs. (This + one turned out to be a retired service, which is its own kind of answer.) + +Also check the *dialect*: UniFi's `controller_type` must be `classic` for a +self-hosted controller (login `/api/login`, no `/proxy/network` prefix). +`unifi_os` sends every request to a path that 404s. ## LLM-*essential* operations — failover chain