A container restart (OOMKill, crash, transient npx failure) used to strand
the instance in ERROR forever while its pod sat 1/1 Running, because five
gaps lined up (#114):
- the exec/attach websocket died without ending the stdout PassThrough
(client-node installs no onclose), so PersistentStdioClient — whose only
death signal was stdout 'end' — kept believing it was connected and every
request rode the 120s timeout into a dead pipe;
- the stdioClients cache is keyed by pod name, which survives a restart, so
nothing ever evicted the corpse;
- syncStatus never re-inspected ERROR rows and never read restartCount, so
neither the recovery nor the in-place restart was visible;
- its ERROR writes clobbered retry metadata, making the row instantly
dueForRetry, and the retry recreated the pod under the SAME name — an
uncaught 409 that looped ERROR against a healthy pod;
- the stuck row consumed the whole replica budget, blocking a fresh-id
replacement.
The fix, layer by layer:
- ws 'close'/'error' now end stdout in both k8s interactive paths (and the
docker interactive path mirrors its own one-shot handlers), funneling into
an identity-guarded teardown in PersistentStdioClient that also listens
for stream 'close'/'error' — a late event from a previous session cannot
clobber a reconnected one;
- syncStatus re-inspects ERROR rows (pod running again → back to RUNNING
with retry metadata cleared), tracks restartCount in instance metadata to
catch restarts between polls, MERGES retry metadata instead of clobbering
it, and evicts the cached stdio client through a setter-injected hook
whenever the pipe is known-dead;
- createContainer adopts an alive pod on 409 instead of throwing (and only
replaces a genuinely dead one, waiting out the deletion grace period);
- attach mode retries once through a fresh client before surfacing an error,
so the first call after a detected death succeeds;
- the health probe evicts the cached client when failures cross the
threshold, and shutdown finally calls closeAll().
Closes#114
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JaFvfHrQyUKCGv6o3N2Wir
@kubernetes/client-node v1 raises ApiException, which carries the HTTP status
on `.code`. Every check in the orchestrator read only `.statusCode` — the
pre-1.0 HttpError field — so `status` came back undefined and each "expected"
404/409 was rethrown instead of handled.
Found in production: the first start of a server with a volume read a
not-yet-existing PVC, got a 404 that should have meant "create it", and failed
the instance instead. The same latent bug sat in removeContainer, where a
missing pod failed the delete rather than being treated as already gone.
httpStatusOf() checks .code, .statusCode and .response.statusCode, and falls
back to parsing the rendered "HTTP-Code: NNN" message for wrappers that keep
none of them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017BMXdb2qZbPSh8Q7XpTyjB
Two bugs fixed:
1. Backup completeness: JSON backup API now includes prompts and
templates. Previously these were silently dropped during
backup/restore, causing data loss on migration.
2. STDIO proxy for docker-image servers: servers with dockerImage
but no packageName/command (like docmost) now use k8s Attach
to connect to the container's PID 1 stdin/stdout instead of
exec. This fixes "has no packageName or command" errors.
Changes:
- backup-service.ts: add BackupPrompt/BackupTemplate types, export them
- restore-service.ts: restore prompts (with project FK) and templates
- mcp-proxy-service.ts: sendViaPersistentAttach for docker-image STDIO
- orchestrator.ts: add attachInteractive to McpOrchestrator interface
- kubernetes-orchestrator.ts: implement attachInteractive via k8s Attach
- k8s-client-official.ts: expose Attach client
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
mcpd can now deploy MCP server instances as Kubernetes pods instead of
Docker containers. Set MCPD_ORCHESTRATOR=kubernetes to enable.
- Add @kubernetes/client-node with thin wrapper (context enforcement
via MCPD_K8S_CONTEXT to prevent multi-cluster mishaps)
- Rewrite KubernetesOrchestrator: pod CRUD, pod IP extraction,
exec via SPDY (one-shot + interactive), log streaming
- Manifest generator: stdin:true for STDIO servers, args (not command)
to preserve runner image entrypoint, security hardening
- Orchestrator selection in main.ts via MCPD_ORCHESTRATOR env var
- 25 unit tests for k8s orchestrator, all 624 tests pass
Tested end-to-end on local k3s:
- mcpd deployed via Pulumi, creates pods in mcpctl-servers namespace
- NetworkPolicy verified: only mcpd can reach MCP server pods
- Python runner (uvx) successfully runs aws-documentation-mcp-server
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
KubernetesOrchestrator implements McpOrchestrator interface with K8s API
client, manifest generation (Pod/Deployment), namespace management,
resource limits, and security contexts. 39 new tests.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
