From 732ca98ccc8c243e277d8efbbebafb960ba8f750 Mon Sep 17 00:00:00 2001 From: Michal Date: Sun, 9 Aug 2026 22:54:33 +0100 Subject: [PATCH] docs(reliability): record the three shapes a failing readiness probe takes Turning readiness probes on took the fleet from 8/8 healthy to three real failures in under a minute, and all three were network shape rather than code: an egress port (UniFi :8443), an ingress hairpin through the Envoy L7 policy (Grafana 403 `Access denied` with a token that worked from a laptop), and a Tailscale address a pod can never reach (Node-RED, since retired). Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_0114dg56YmVacyqhp5fitcTb --- docs/reliability.md | 26 +++++++++++++++++++------- 1 file changed, 19 insertions(+), 7 deletions(-) diff --git a/docs/reliability.md b/docs/reliability.md index 5c1f2c5..30423c8 100644 --- a/docs/reliability.md +++ b/docs/reliability.md @@ -76,17 +76,29 @@ local table, not crossing a network. The UniFi probe went from 3ms (`tools/list` lying) to 1847ms on its first real `list_sites` — login, TLS, controller round trip — and ~40ms once the session was warm. -### Two failure modes the probe cannot see for you +### Where a failing readiness probe usually points -The healthy-looking UniFi server was broken **twice over**, and both are worth -checking first when a readiness probe starts failing: +Turning these probes on for the first time took the fleet from "8/8 healthy" to +three genuine failures in under a minute. All three were network shape, not +code — check these before suspecting the server: -1. **Egress.** MCP server pods default to TCP 80/443 only +1. **Egress port.** MCP server pods default to TCP 80/443 only (`servers-allow-external-egress`). Any upstream on another port — the UniFi controller on `:8443` — times out on every call. Declare it in Pulumi's - `mcpctl.serverEgressTargets` (name + `/32` + ports); don't widen the blanket rule. -2. **Address reachability.** A pod cannot reach a **Tailscale** `100.64.0.0/10` - address. Config pointing at one connect-timeouts forever. Use LAN IPs. + `mcpctl.serverEgressTargets`; don't widen the blanket rule. +2. **Ingress hairpin.** A co-located service reached over its *public* hostname + goes out and back through the per-host Envoy L7 policy, which doesn't + reliably carry the caller's identity and replies with a bare `Access denied`. + Grafana 403'd on every call this way while the identical token succeeded from + a laptop. The tell is the error *shape*: plain text, not the upstream's own + JSON error. Use the ClusterIP (`serverEgressTargets` with `namespace:`). +3. **Address reachability.** A pod cannot reach a **Tailscale** `100.64.0.0/10` + address. Config pointing at one connect-timeouts forever. Use LAN IPs. (This + one turned out to be a retired service, which is its own kind of answer.) + +Also check the *dialect*: UniFi's `controller_type` must be `classic` for a +self-hosted controller (login `/api/login`, no `/proxy/network` prefix). +`unifi_os` sends every request to a path that 404s. ## LLM-*essential* operations — failover chain