A freshly provisioned PVC mounts root:root, so any image that drops privileges
cannot write to it. docs-mcp-server runs as uid 1000 and died on first start
with SQLITE_CANTOPEN; its own Dockerfile says to chown the volume 1000:1000.
Pods with volumes now carry securityContext.fsGroup, defaulting to 1000 and
overridable per volume. fsGroupChangePolicy is OnRootMismatch so Kubernetes
does not walk and re-chown the whole volume on every start.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017BMXdb2qZbPSh8Q7XpTyjB
Instances are immutable and get recreated on any server edit, so anything an
MCP server wrote to its container filesystem was lost at exactly that point.
That ruled out every stateful MCP server, docs-mcp among them: its index is a
SQLite file (better-sqlite3 + sqlite-vec) and it has no external-database mode,
so no amount of Postgres helps.
A server or template can now declare volumes. The backing store is keyed on the
server, not the instance — `mcpctl-<server>-<name>` — which is the whole point:
an instance-scoped claim would be destroyed precisely when the data needs to
survive. On Kubernetes that is a PVC ensured in the servers namespace before the
pod is created and never deleted with it; on Docker, a named volume (named, not
anonymous, so `removeContainer`'s `v: true` leaves it alone).
Claims are ReadWriteOnce, so volumes and replicas > 1 are mutually exclusive;
validation rejects that pair instead of leaving the extra replicas
unschedulable. storageClassName is omitted rather than sent empty when no class
is configured — to Kubernetes those mean different things.
Also fixes a pre-existing bug in the same path: seedTemplates dropped `runtime`,
so every PyPI-backed template seeded from YAML silently defaulted to node and
would run `npx` against a package that only exists on PyPI. `unifi-network`
declares `runtime: python` and had been seeding with runtime unset.
Templates added, all self-hosted and none needing an API key:
- duckduckgo — no backing service at all
- searxng — needs a SearXNG engine (compose profile in stack/)
- docs-mcp — open-source Context7/Ref alternative, uses the new volume
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017BMXdb2qZbPSh8Q7XpTyjB
Three connected issues with how instances came up + got reported as
healthy when their secret backend was unreachable. The motivating
case: gitea-mcp-server starts when mcpd can't read the
gitea-creds secret from OpenBao, runs with an empty
GITEA_ACCESS_TOKEN, replies fine to tools/list (so liveness passes),
but every authed call fails with "token is required" — and
`mcpctl get instances` cheerfully reports the instance as healthy.
## What changed
### 1. Env resolution failures are now fatal for the start attempt
`src/mcpd/src/services/instance.service.ts`
The previous behaviour swallowed `resolveServerEnv` failures and let
the container start anyway with whatever env survived ("non-fatal —
container may still work if env vars are optional"). That's the bug:
the gitea container started with no token, ran for weeks, and was
reported healthy.
The catch now calls `markInstanceError(instance, "secret resolution
failed: <reason>")` and returns. Optional/missing env vars should be
modelled as `value: ""` entries on the server, not as silent
secret-resolution failures.
### 2. ERROR instances retry with backoff, not blind churn
Adds Kubernetes-style escalation: 30 s × 5 attempts, then 5 min
pauses thereafter. Retry state lives on `McpInstance.metadata` (no
schema migration) — `attemptCount`, `lastAttemptAt`, `nextRetryAt`,
`error`.
The reconciler no longer tears down ERROR instances and creates
fresh replacements (which would reset attemptCount and effectively
loop at 30 s forever). Instead:
- ERROR rows whose `nextRetryAt` is in the future are LEFT ALONE
and counted against the replica budget — preventing tight create-
fail-create churn while a previous attempt is in its backoff window.
- ERROR rows whose `nextRetryAt` has elapsed are retried IN-PLACE
via a new `retryInstance` method, which preserves attemptCount on
the same row so the schedule actually escalates.
The work has been factored into `startOne` (creates + initial attempt)
+ `attemptStart` (env + container) + `retryInstance` (re-attempt the
same row) + `markInstanceError` (write retry metadata).
### 3. STDIO readiness probe goes through mcpProxyService
`src/mcpd/src/services/health-probe.service.ts`
The legacy `probeStdio` (a `docker exec node -e '... spawn(packageName)
...'` invocation) only worked for packageName-based servers. Image-
based STDIO servers like gitea-mcp-server fell through with "No
packageName or command for STDIO server" and were reported unhealthy
for the WRONG reason — they have no packageName because they are an
image, not because anything's wrong.
New `probeReadinessViaProxy`: sends `tools/call` through the live
running container via `mcpProxyService.execute`. Same code path as
production traffic, so probe failures match real failures. Picks up:
- JSON-RPC errors (e.g. "token is required" when env is empty).
- Tool-level errors expressed as `result.isError: true`.
- Connection failures wrapped as exceptions.
- Hard timeouts via the deadline race.
After this PR, configuring `gitea` with
`healthCheck: { tool: get_me, intervalSeconds: 60 }` makes
`mcpctl get instances` report it as `unhealthy` whenever the auth
token is missing or wrong — which is honest.
The dead `probeStdio` (~120 LOC) is removed; HTTP/SSE bespoke probe
paths are kept for now (they work and the diff stays minimal).
## Tests
`src/mcpd/tests/instance-service.test.ts`:
- Replaces "cleans up ERROR instances and creates replacements" with
"retries ERROR instances in-place when their backoff has elapsed".
- Adds "leaves ERROR instances alone while their nextRetryAt is in
the future" and "escalates the backoff: attemptCount + nextRetryAt
persist on retry failures".
`src/mcpd/tests/services/health-probe.test.ts`:
- Swaps STDIO probe mocks from `orchestrator.execInContainer` →
`mcpProxyService.execute`.
- Adds "marks unhealthy when proxy returns a JSON-RPC error
(e.g. broken-secret auth failure)" — explicitly the gitea case.
- Adds "marks unhealthy when proxy returns a tool-level error in
result.isError" — covers servers that report tool failures as
isError instead of as JSON-RPC errors.
- Renames "handles exec timeout" → "handles probe timeout" and
exercises the deadline race rather than an exec rejection.
Full suite: 162 test files / 2161 tests green (+4 new).
## Manual verification step (post-deploy)
```bash
mcpctl edit server gitea
# → add healthCheck:
# tool: get_me
# intervalSeconds: 60
# timeoutSeconds: 10
# failureThreshold: 3
```
If OpenBao is still down: gitea instance enters ERROR with
attemptCount + nextRetryAt visible in `mcpctl describe instance`.
Otherwise: gitea env resolves at next start, probe passes, instance
is honestly healthy.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Why: API keys live in Postgres as plaintext JSON. A DB read exposes every
credential in the system. Before centralising more secrets (LLM keys, etc.)
we want to be able to point at an external KV store and drop DB access to
sensitive rows.
New model:
- `SecretBackend` resource (CRUD + isDefault invariant) owns how a secret is
stored. `Secret` gains `backendId` FK and `externalRef`. Reads/writes
dispatch through a driver.
- `plaintext` driver (near-noop, uses existing Secret.data column) is seeded
as the `default` row at startup. Acts as trust root / bootstrap.
- `openbao` driver (also HashiCorp Vault KV v2 compatible) talks plain HTTP,
no SDK dependency. Auth via static token pulled from a plaintext-backed
`Secret` through the injected SecretRefResolver. Caches resolved token.
- `SecretMigrateService` moves secrets one-at-a-time: read → write dest →
flip row → best-effort source delete. Interrupted runs are idempotent
(skips secrets already on destination).
CLI surface:
- `mcpctl create|get|describe|delete secretbackend` + `--default` on create.
- `mcpctl migrate secrets --from X --to Y [--names a,b] [--keep-source] [--dry-run]`
- `apply -f` round-trips secretbackends (yaml/json multi-doc + grouped).
- RBAC: `secretbackends` resource + `run:migrate-secrets` operation.
- Fish + bash completions regenerated.
docs/secret-backends.md covers the OpenBao policy, chicken-and-egg auth flow,
and the migration semantics.
Broke the circular dep (OpenBao needs SecretService to resolve its own token,
SecretService needs SecretBackendService) with a deferred-resolver bridge in
mcpd startup. 11 new driver unit tests; existing env-resolver/secret-route/
backup tests updated for the new service signatures. Full suite: 1792/1792.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Instance status now reflects actual container state:
- startOne() sets STARTING (not RUNNING) after container creation
- syncStatus() promotes STARTING→RUNNING when pod is ready
- syncStatus() demotes RUNNING→STARTING if pod restarts (CrashLoop)
- External servers still get RUNNING immediately (no container)
Previously, CrashLooping pods showed as RUNNING in mcpctl get instances.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
mcpd now runs a periodic reconcileAll() every 30s that:
- Detects crashed/missing containers (syncStatus)
- Cleans up ERROR instances
- Creates replacement pods to match desired replica count
This replaces the old syncStatus-only timer. Servers migrated
from another deployment or recovering from node failures will
automatically get their instances recreated.
6 new tests for reconcileAll covering: missing instances, skip
replicas=0, already-at-count, ERROR cleanup, multi-server,
error isolation.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add warmup() to LlmProvider interface for eager subprocess startup
- ManagedVllmProvider.warmup() starts vLLM in background on project load
- ProviderRegistry.warmupAll() triggers all managed providers
- NamedProvider proxies warmup() to inner provider
- paginate stage generates LLM-powered descriptive page titles when
available, cached by content hash, falls back to generic "Page N"
- project-mcp-endpoint calls warmupAll() on router creation so vLLM
is loading while the session initializes
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add syncStatus() to InstanceService: detects crashed/stopped containers,
marks them ERROR with last log line as context
- Reconcile now syncs container status first (detect dead before counting)
- Add 30s periodic sync loop in main.ts
- Switch node-runner from alpine to slim (Debian) for npm compatibility
(fixes home-assistant-mcp-server binary not found on Alpine)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
STDIO servers with packageName (e.g. @leval/mcp-grafana) need a Node.js
container that runs `npx -y <package>`. Previously, packageName was used
as a Docker image reference causing "invalid reference format" errors.
- Add Dockerfile.node-runner: minimal node:20-alpine with npx entrypoint
- Update instance.service.ts: detect npm-based servers and use node-runner
image with npx command instead of treating packageName as image name
- Fix NanoCPUs: only set when explicitly provided (kernel CFS not available
on all hosts)
- Add mcp-servers network with explicit name for container isolation
- Configure MCPD_NODE_RUNNER_IMAGE and MCPD_MCP_NETWORK env vars
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace the confused Profile abstraction with a dedicated Secret resource
following Kubernetes conventions. Servers now have env entries with inline
values or secretRef references. Env vars are resolved and passed to
containers at startup (fixes existing gap).
- Add Secret CRUD (model, repo, service, routes, CLI commands)
- Server env: {name, value} or {name, valueFrom: {secretRef: {name, key}}}
- Add env-resolver utility shared by instance startup and config generation
- Remove all profile-related code (models, services, routes, CLI, tests)
- Update backup/restore for secrets instead of profiles
- describe secret masks values by default, --show-values to reveal
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Server = Deployment (defines what to run + desired replicas)
Instance = Pod (ephemeral, auto-created by reconciliation)
Backend:
- Add replicas field to McpServer schema
- Add reconcile() to InstanceService (scales instances to match replicas)
- Remove manual start/stop/restart - instances are auto-managed
- Cascade: deleting server stops all containers then cascades DB
- Server create/update auto-triggers reconciliation
CLI:
- Add top-level delete command (servers, instances, profiles, projects)
- Add top-level logs command
- Remove instance compound command (use get/delete/logs instead)
- Clean up project command (list/show/delete → top-level get/describe/delete)
- Enhance describe for instances with container inspect info
- Add replicas to apply command's ServerSpec
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Support non-containerized MCP servers via externalUrl field and add
streamable-http session management for HA MCP proof of concept.
- Add externalUrl, command, containerPort fields to McpServer schema
- Skip Docker orchestration for external servers (virtual instances)
- Implement streamable-http proxy with Mcp-Session-Id session management
- Parse SSE-framed responses from streamable-http endpoints
- Add command passthrough to Docker container creation
- Create HA MCP example manifest (examples/ha-mcp.yaml)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds restart/inspect methods to InstanceService, state validation for stop,
REST endpoints for restart and inspect, and full CLI command suite for
instance list/start/stop/restart/remove/logs/inspect.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
McpOrchestrator interface with DockerContainerManager implementation,
instance service for lifecycle management, instance API routes,
and docker-compose with mcpd service. 127 tests passing.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
