feat(servers): per-server memory ceiling, because 512Mi OOMKills silently
Some checks failed
CI/CD / typecheck (pull_request) Successful in 1m16s
CI/CD / lint (pull_request) Successful in 2m16s
CI/CD / test (pull_request) Successful in 1m24s
CI/CD / build (pull_request) Successful in 2m10s
CI/CD / smoke (pull_request) Failing after 3m15s
CI/CD / publish (pull_request) Has been skipped
Some checks failed
CI/CD / typecheck (pull_request) Successful in 1m16s
CI/CD / lint (pull_request) Successful in 2m16s
CI/CD / test (pull_request) Successful in 1m24s
CI/CD / build (pull_request) Successful in 2m10s
CI/CD / smoke (pull_request) Failing after 3m15s
CI/CD / publish (pull_request) Has been skipped
Server pods have always had a hardcoded 512Mi limit. That is right for a server that proxies an API and fatal for one that drives a browser: the `docs` server (docs-mcp-server, which scrapes with headless Chromium) idles at ~228Mi and crosses 512Mi within seconds of its first scrape. An OOMKill is the quietest failure we have. The kernel kills it, the pod restarts, the readiness probe passes, and the instance reads `healthy` again — while the six scrape jobs whose queue lived in memory are gone and the index has 17 pages in it. Nothing is logged, because the process never got to say anything. memoryLimitMb is declared per server, in MiB, and converted to bytes in the container spec. NULL keeps DEFAULT_MEMORY_LIMIT, so every existing server is bit-for-bit unchanged — a bigger default would have cost real memory on every node for servers that do not need it. mcpctl create server docs --memory-limit-mb 2048 --force Tests assert the two places a new column dies quietly: the repository's field-by-field mapping (a column not mapped there returns "patched" and changes nothing) and the spec built in instance.service. docs/reliability gains the OOMKill section — the `lastState.terminated.reason` check is what turns "it restarted again" into an answer. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JPjtnE6Gd343oRNtMU9Bcd
This commit is contained in:
@@ -137,6 +137,42 @@ Also check the *dialect*: UniFi's `controller_type` must be `classic` for a
|
||||
self-hosted controller (login `/api/login`, no `/proxy/network` prefix).
|
||||
`unifi_os` sends every request to a path that 404s.
|
||||
|
||||
### A server that restarts instead of erroring is out of memory
|
||||
|
||||
Server pods get **512 MiB** by default (`DEFAULT_MEMORY_LIMIT`). That is ample
|
||||
for a server that proxies an API and nowhere near enough for one that drives a
|
||||
browser or holds an index in memory.
|
||||
|
||||
An OOMKill is the quietest failure in the fleet, because **nothing reports an
|
||||
error**. The kernel kills the container, Kubernetes restarts it, the readiness
|
||||
probe passes again, and `mcpctl get instances` reads `healthy`. Whatever the
|
||||
server was doing is simply gone — for `docs`, six scrape jobs whose queue lived
|
||||
in memory, leaving an index with 17 pages in it and no failed job to look at.
|
||||
|
||||
The tells, in order of how fast they answer the question:
|
||||
|
||||
```bash
|
||||
kubectl -n mcpctl-servers get pods | grep <server> # RESTARTS climbing
|
||||
kubectl -n mcpctl-servers get pod <pod> -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}'
|
||||
```
|
||||
|
||||
`OOMKilled` there is conclusive. `mcpctl logs` will not show it: the process
|
||||
never got to say anything.
|
||||
|
||||
Raise the ceiling per server rather than for the fleet — most servers do not
|
||||
need it, and a bigger default wastes real memory on every node:
|
||||
|
||||
```bash
|
||||
mcpctl create server docs --memory-limit-mb 2048 --force
|
||||
```
|
||||
|
||||
Declared in MiB, stored on the server, converted to bytes in the container
|
||||
spec. Null keeps the 512 MiB default, so existing servers are unchanged. Sizing
|
||||
rule of thumb: measure idle first (`/sys/fs/cgroup/memory.current` inside the
|
||||
pod), then leave headroom for the peak — `docs` idles at ~228 MiB and crosses
|
||||
512 MiB within seconds of a scrape, because each page render is a Chromium
|
||||
process.
|
||||
|
||||
## LLM-*essential* operations — failover chain
|
||||
|
||||
Chat needs *an* LLM but not a *specific* one. Instead of failing when the pinned
|
||||
|
||||
Reference in New Issue
Block a user