michal/mcpctl
1
0
Fork 0
Code Issues Pull Requests 7 Actions Packages Projects Releases Wiki Activity

feat(servers): per-server memory ceiling, because 512Mi OOMKills silently #132

Merged
michal merged 1 commits from feat/server-memory-limit into main 2026-09-16 23:24:32 +00:00
Conversation 0 Commits 1 Files Changed 12 +136 -2
michal commented 2026-09-16 23:24:19 +00:00
Owner
Copy Link

MCP server pods have a hardcoded 512Mi limit. Fine for an API proxy, fatal for a server that drives a browser: docs (docs-mcp-server, scrapes with headless Chromium) idles at ~228Mi and crosses 512Mi seconds into its first scrape.

Why this was invisible. An OOMKill reports nothing: kernel kills it, pod restarts, readiness passes, instance reads healthy. Six scrape jobs (queue held in memory) vanished and the index kept 17 pages. mcpctl logs shows nothing, because the process never got to speak.

Change: memoryLimitMb on the server (MiB), converted to bytes in the ContainerSpec. NULL keeps DEFAULT_MEMORY_LIMIT, so existing servers are unchanged — raising the fleet default would waste memory on every node for servers that do not need it.

mcpctl create server docs --memory-limit-mb 2048 --force

Threaded through the places a new column dies quietly: Prisma migration, create/update validation, the repository field-by-field mapping, apply.ts server spec, the CLI flag, and shell completions.

Tests: repository mapping (create + update + absent), and instance.service converting MiB to bytes / leaving the spec untouched when unset. mcpd 1159 passed, cli 740 passed, tsc clean.

Docs: docs/reliability.md gains "A server that restarts instead of erroring is out of memory" — the lastState.terminated.reason check that turns "it restarted again" into an answer.

🤖 Generated with Claude Code

https://claude.ai/code/session_01JPjtnE6Gd343oRNtMU9Bcd

MCP server pods have a hardcoded 512Mi limit. Fine for an API proxy, fatal for a server that drives a browser: `docs` (docs-mcp-server, scrapes with headless Chromium) idles at ~228Mi and crosses 512Mi seconds into its first scrape. **Why this was invisible.** An OOMKill reports nothing: kernel kills it, pod restarts, readiness passes, instance reads `healthy`. Six scrape jobs (queue held in memory) vanished and the index kept 17 pages. `mcpctl logs` shows nothing, because the process never got to speak. **Change:** `memoryLimitMb` on the server (MiB), converted to bytes in the ContainerSpec. NULL keeps `DEFAULT_MEMORY_LIMIT`, so existing servers are unchanged — raising the fleet default would waste memory on every node for servers that do not need it. ``` mcpctl create server docs --memory-limit-mb 2048 --force ``` Threaded through the places a new column dies quietly: Prisma migration, create/update validation, the repository field-by-field mapping, apply.ts server spec, the CLI flag, and shell completions. **Tests:** repository mapping (create + update + absent), and instance.service converting MiB to bytes / leaving the spec untouched when unset. mcpd 1159 passed, cli 740 passed, tsc clean. **Docs:** docs/reliability.md gains "A server that restarts instead of erroring is out of memory" — the `lastState.terminated.reason` check that turns "it restarted again" into an answer. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01JPjtnE6Gd343oRNtMU9Bcd
michal added 1 commit 2026-09-16 23:24:20 +00:00
feat(servers): per-server memory ceiling, because 512Mi OOMKills silently
Some checks failed
CI/CD / typecheck (pull_request) Successful in 1m16s
Details
CI/CD / lint (pull_request) Successful in 2m16s
Details
CI/CD / test (pull_request) Successful in 1m24s
Details
CI/CD / build (pull_request) Successful in 2m10s
Details
CI/CD / smoke (pull_request) Failing after 3m15s
Details
CI/CD / publish (pull_request) Has been skipped
Details
859150770f 
Server pods have always had a hardcoded 512Mi limit. That is right for a
server that proxies an API and fatal for one that drives a browser: the
`docs` server (docs-mcp-server, which scrapes with headless Chromium)
idles at ~228Mi and crosses 512Mi within seconds of its first scrape.

An OOMKill is the quietest failure we have. The kernel kills it, the pod
restarts, the readiness probe passes, and the instance reads `healthy`
again — while the six scrape jobs whose queue lived in memory are gone
and the index has 17 pages in it. Nothing is logged, because the process
never got to say anything.

memoryLimitMb is declared per server, in MiB, and converted to bytes in
the container spec. NULL keeps DEFAULT_MEMORY_LIMIT, so every existing
server is bit-for-bit unchanged — a bigger default would have cost real
memory on every node for servers that do not need it.

  mcpctl create server docs --memory-limit-mb 2048 --force

Tests assert the two places a new column dies quietly: the repository's
field-by-field mapping (a column not mapped there returns "patched" and
changes nothing) and the spec built in instance.service. docs/reliability
gains the OOMKill section — the `lastState.terminated.reason` check is
what turns "it restarted again" into an answer.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JPjtnE6Gd343oRNtMU9Bcd
michal merged commit 0076fa0d37 into main 2026-09-16 23:24:32 +00:00
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
michal
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: michal/mcpctl#132
Delete Branch "feat/server-memory-limit"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?