Michal b022f322f0 fix(mcpd): recover k8s STDIO instances after in-place container restarts ...

A container restart (OOMKill, crash, transient npx failure) used to strand
the instance in ERROR forever while its pod sat 1/1 Running, because five
gaps lined up (#114):

- the exec/attach websocket died without ending the stdout PassThrough
  (client-node installs no onclose), so PersistentStdioClient — whose only
  death signal was stdout 'end' — kept believing it was connected and every
  request rode the 120s timeout into a dead pipe;
- the stdioClients cache is keyed by pod name, which survives a restart, so
  nothing ever evicted the corpse;
- syncStatus never re-inspected ERROR rows and never read restartCount, so
  neither the recovery nor the in-place restart was visible;
- its ERROR writes clobbered retry metadata, making the row instantly
  dueForRetry, and the retry recreated the pod under the SAME name — an
  uncaught 409 that looped ERROR against a healthy pod;
- the stuck row consumed the whole replica budget, blocking a fresh-id
  replacement.

The fix, layer by layer:
- ws 'close'/'error' now end stdout in both k8s interactive paths (and the
  docker interactive path mirrors its own one-shot handlers), funneling into
  an identity-guarded teardown in PersistentStdioClient that also listens
  for stream 'close'/'error' — a late event from a previous session cannot
  clobber a reconnected one;
- syncStatus re-inspects ERROR rows (pod running again → back to RUNNING
  with retry metadata cleared), tracks restartCount in instance metadata to
  catch restarts between polls, MERGES retry metadata instead of clobbering
  it, and evicts the cached stdio client through a setter-injected hook
  whenever the pipe is known-dead;
- createContainer adopts an alive pod on 409 instead of throwing (and only
  replaces a genuinely dead one, waiting out the deletion grace period);
- attach mode retries once through a fresh client before surfacing an error,
  so the first call after a detected death succeeds;
- the health probe evicts the cached client when failures cross the
  threshold, and shutdown finally calls closeAll().

Closes #114

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JaFvfHrQyUKCGv6o3N2Wir

2026-08-15 23:32:23 +01:00

8.1 KiB

Raw Blame History

View Raw