Three smoke tests predated PR #137 (ungated-global pre-gate servers) and
the ambient-context clock, and were never exercised against a prod that
had calcmcp seeded — the 5415060 fix was cut over via kubectl, bypassing
the suite. They failed on today's a783869 deploy:
- proxy-pipeline: a fresh gated session legitimately sees begin_session
plus the ungated-global project's tools. Now discovers the gate-exempt
server set and asserts nothing ELSE leaks pre-gate.
- tool-drilldown: get_current_time is a router-dispatched virtual tool
(ambient clock) that never paginates; added to the exempt set.
- backup: command is optional for dockerImage STDIO servers (entrypoint
images — calcmcp, dav-michal, jmap-michal run with none). The real
contract is fidelity: every live server WITH a command must keep it
through backup. Asserted that instead.
- audit: 'Gated (only begin_session)' label said something the >=1 check
never verified; renamed.
Full suite against live a783869: 32 files, 181/181 green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The four Secret tests in models.test.ts created rows via raw
prisma.secret.create without a backendId, so the schema default ''
violated Secret_backendId_fkey (that default only exists so 'db push'
can add the column to pre-existing rows; real writes always carry a
valid FK). Seed a plaintext backend per test, the same shape
bootstrapSecretBackends creates, and clear SecretBackend rows in
clearAllTables so nothing leaks between tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Conflict in src/mcpd/src/main.ts: both sides touched the import block.
Kept this branch's mapUrlToPermission plus main's bootstrapUngatedGlobal;
dropped main's SET_OWN_PASSWORD_OPERATION import — this branch moved that
usage into middleware/url-permission.ts, which imports it itself.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
bootstrapUngatedGlobal wrote its seed marker into the mcpctl-system-settings
Secret, creating the row with the default backendId '' when it did not exist.
The Secret_backendId_fkey constraint is enforced at insert time, so on a DB
where that secret was absent, mcpd crash-looped on startup (P2003).
Fix: run the seed after bootstrapSecretBackends (so the default plaintext
backend exists) and set backendId to the resolved default on create. If no
backend exists at all, skip writing the marker rather than crash.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
New seed-once system project `ungated-global`: every server attached to it
is merged into every project's MCP endpoint in front of the gate (listed
and callable before begin_session), and its project prompt is prepended to
every project's instructions.
- mcpd: bootstrapUngatedGlobal seeds the project once-ever (marker key
`ungatedGlobalSeeded` in mcpctl-system-settings), with default server
`calcmcp` (patched CalcMCP fork, mysources.co.uk/michal/calcmcp:1.1.0)
and a "never do math yourself, always use calcmcp" prompt. Deletions are
durable: `mcpctl delete server calcmcp -p ungated-global` stays deleted.
- mcpd: GET /projects/:id/servers appends global servers flagged
`ungatedGlobal: true` (direct attachment wins, no self-merge);
GET /projects/:id/instructions prepends the global prompt and lists the
global servers.
- mcplocal: discovery surfaces the flagged names; the gate plugin exempts
them — gated tools/list returns begin_session + exempt tools, calls to
them pass through without ungating, and initialize instructions list
them under "Available immediately (no begin_session required)".
- templates/calcmcp.yaml so a deleted default can be re-created by hand.
- docs/ungated-global.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Models consuming projects have no idea what day it is. Three always-on
layers from mcplocal plus one from mcpd:
- initialize instructions and the begin_session briefing carry
'[context] now: <date/time> — project: <name>' (frozen at session
start by design);
- a get_current_time virtual tool (ungated catalogs only, same pattern
as read_prompts);
- every tools/call result gets a trailing '[context] now: …' content
item at the endpoint send site — the only chokepoint that also covers
virtual tools, gate intercepts and failure results, and the layer
that keeps a days-old chat session current;
- mcpd agent/project chat system blocks append the same line each turn.
Time renders as UTC plus an optional local timezone from
MCPLOCAL_AMBIENT_TZ (IANA name; both services read it; invalid or unset
means UTC only). ~15 tokens per stamp, minute precision.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
dav-mcp declares tsdav/tsdav-utils as GitHub-URL deps — npx on the
git-less node-runner dies with 'spawn git ENOENT' (observed: dav-michal
CrashLoopBackOff on first deploy). Upstream's ghcr image PID-1s the HTTP
server, so mysources.co.uk/michal/dav-mcp bakes the install and execs
src/server-stdio.js.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Stalwart's session advertises publicUrl-absolute endpoints; over the
ClusterIP those hairpin through Envoy and 403. Verified in-cluster:
apiUrl came back as https://smtp.ad.itaz.eu/jmap/.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
jmap and ttrss are custom images (mysources.co.uk/michal/{jmap-mcp,ttrss-mcp});
dav-mcp and @karakeep/mcp run on the node runner. Health checks are real
backend calls. Applied live 2026-09-30; these seeds keep the next mcpd
image rebuild from reverting them.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A cached project router outlives the caller's bearer token. syncUpstreams
only created a McpdUpstream (which binds the token at construction) for
server names not already registered, so after a token rotation every
pre-rotation upstream kept posting the old bearer: six attached, healthy
servers 401'd on tools/list forever — no error anywhere the client could
see — while the two servers attached after the rotation worked.
- discovery.ts: rebuild every upstream on every sync, so it always carries
the current caller token and the current server id (also fixes a stale
serverId after delete+recreate under the same name).
- router.ts: record per-server discovery failures with their messages and
expose them (getToolDiscoveryFailures).
- gate.ts: a partial catalog is now loud — initialize instructions and the
begin_session response name every attached server that contributed no
tools and the error that silenced it.
- tests: regression tests for token rebinding and serverId refresh.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Server pods have always had a hardcoded 512Mi limit. That is right for a
server that proxies an API and fatal for one that drives a browser: the
`docs` server (docs-mcp-server, which scrapes with headless Chromium)
idles at ~228Mi and crosses 512Mi within seconds of its first scrape.
An OOMKill is the quietest failure we have. The kernel kills it, the pod
restarts, the readiness probe passes, and the instance reads `healthy`
again — while the six scrape jobs whose queue lived in memory are gone
and the index has 17 pages in it. Nothing is logged, because the process
never got to say anything.
memoryLimitMb is declared per server, in MiB, and converted to bytes in
the container spec. NULL keeps DEFAULT_MEMORY_LIMIT, so every existing
server is bit-for-bit unchanged — a bigger default would have cost real
memory on every node for servers that do not need it.
mcpctl create server docs --memory-limit-mb 2048 --force
Tests assert the two places a new column dies quietly: the repository's
field-by-field mapping (a column not mapped there returns "patched" and
changes nothing) and the spec built in instance.service. docs/reliability
gains the OOMKill section — the `lastState.terminated.reason` check is
what turns "it restarted again" into an answer.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JPjtnE6Gd343oRNtMU9Bcd
Readiness probes ran real searches every interval: searxng's
searxng_web_search probe fanned a query out to every engine 1,440 times a
day, billing API-key engines (braveapi, kagi), and duckduckgo's `search`
probe scraped DuckDuckGo from the same IP agents search from.
- searxng: probe searxng_instance_info, which reads /config only
- duckduckgo: probe fetch_content on example.com every 300s
- firecrawl (new): firecrawl-mcp against a self-hosted or cloud
Firecrawl, for reading pages as main-content markdown; probe
firecrawl_scrape on example.com every 300s
Every probe still names a readiness tool, as templates.test.ts requires;
each was run against the real package before being written down.
docs/web-search.md now pairs searxng (search) with firecrawl (reading)
and says why duckduckgo does not hold up under agent traffic.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JPjtnE6Gd343oRNtMU9Bcd
124 commits of main against 33 of agentic-teams. 22 files overlapped, 7
conflicted. Three needed real reconciliation rather than a side-pick:
**The `volumes` collision.** Both branches independently added a `volumes`
field to ContainerSpec meaning different things: main's is persistent storage
keyed by a server-derived `claimName` that deliberately outlives an instance;
mine is emptyDir scratch shared between an init container and the main
container. Worse, each emitted the pod-level `volumes` list from its own
helper, so spreading both would have silently kept only the last — a server
with a PVC would have lost it the moment a session scratch volume appeared.
Resolved by renaming the unreleased side: mine is now `scratchVolumes`, main's
keeps `volumes` since it is already shipped and in use. `buildPodVolumes` now
owns the whole pod-level list and emits both kinds; `buildPodExtras` is init
containers only. fsGroup/securityContext stays tied to the PVC case, because
an emptyDir is created writable by the container's own user and chowning it
is work with no purpose. Five tests pin the coexistence so a later change
cannot drop one kind again.
**The extracted permission mapper.** I moved `mapUrlToPermission` out of
main.ts into middleware/url-permission.ts so it could be tested; main kept it
inline and kept editing it. Kept the extraction and ported main's one change
across — the `/api/v1/audit/events/purge` mapping onto the existing
`audit-purge` operation. Verified rule-by-rule that the extracted version is a
superset: all 15 of main's rules plus the 5 agentic-teams ones.
**The NUL byte in create.ts.** Both branches fixed the same defect
independently, mine as `''` and main's as `'\0'`. Took main's.
The rest were one-liners: .gitignore comment wording, and an add/add on
proxy-long-running.test.ts where main's copy carries a progressive-delivery
test mine lacked — took main's in full.
Not changed, but worth flagging: main's generateDeploymentSpec hardcodes
`automountServiceAccountToken: false` while generatePodSpec honours
`spec.automountServiceAccountToken`. That asymmetry is main's own and predates
this merge, so it is left exactly as-is rather than altered inside a merge
commit.
Verified: 3138 tests pass (237 files), build clean, prisma schema valid.
Lint 809/80 vs 794/77 pre-merge — the delta is main's own debt; every file
touched during conflict resolution was checked and introduces no new errors.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N4wNHWf7xSwnZCWpJcyv9p
The drill-down checks matched /_resultId:\s*(\S+)/ — the legacy paginator's
JSON spelling. The content pipeline writes _resultId="pm-abc", so the regex
found nothing: "Has _resultId for drill-down" reported a false failure on
every run, and the drill-down tests that used the same regex to extract an id
skipped themselves as "not large enough for section-split". Both were
invisible in a green suite.
One shared extractResultId() in the smoke harness now accepts either
spelling and returns the bare id, so the ad-hoc character strip on the
captured group goes away too. proxy-pipeline's check block reports 21/21
instead of 20 with a phantom failure.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JQr5Z9gYrqBQXTGBBuemZ2
A paginated tool result tells the model to call the same tool again with
only _resultId/_section. The advertised schema still carried the upstream's
own `required` list, so a client that validates arguments rejected exactly
that call — "Received tool input did not match expected schema" — and the
model concluded pagination was broken and gave up
(websearch/fetch_content requires `url`; verified live on llm-model-tester).
`required` is ANDed with everything else in a schema, so no property
declaration could rescue it. It now becomes an alternation: either the
upstream's requirements (a fresh call) or `_resultId` (a re-read, which is
answered from cache and never reaches the upstream). Validators that
understand anyOf enforce that; naive ones that only read top-level
`required` now find none and accept both. Schemas that already carry a
combinator get the alternation appended to `allOf`, and a schema with no
required params is left untouched.
Covered by unit tests on a required-bearing upstream schema plus a smoke
assertion that no advertised tool keeps a drill-down-blocking `required`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JQr5Z9gYrqBQXTGBBuemZ2
Auto-following to the newest Opus moved the failure rather than removing it.
The deploy's own smoke output showed it: 404 "model not found" became
HTTP 400: `temperature` is deprecated for this model.
Anthropic removed temperature/top_p/top_k on the current generation (Opus 5,
Sonnet 5, Opus 4.7/4.8, Fable 5), and the adapter sends temperature: 0
unconditionally. So the gate's prompt-selection was still degrading on every
call -- just with a different status code.
A list of which models accept sampling would rot exactly the way the pinned
model ids did, which is the whole thing this branch is trying to stop. So the
provider learns it instead: the first 400 naming a sampling parameter drops it
and retries, and remembers the model so every later call omits it up front.
One wasted call, once, rather than a hardcoded table to maintain.
The match is deliberately narrow -- a 400 must actually name temperature/top_p/
top_k. An unrelated 400 (missing max_tokens, bad schema) propagates untouched;
there is a test for that, because a broad match here would silently swallow
real request errors and retry them pointlessly.
Verified live: claude-opus-latest -> claude-opus-5, first attempt 400 on
temperature, retry succeeds past it. The retry then hit HTTP 429 -- the
personal OAuth token's rate limit, which is the pre-existing credential-tiering
issue, not this path.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GqMidYEGUJG5fxeoTELBu2
Completes the plan's last item. The deadline shipped with a global default;
this makes it overridable per server, where the knowledge actually lives -- a
server with genuinely slow tools declares its own budget instead of forcing the
global up for everyone.
Source of truth is the server resource, following healthCheck exactly:
Prisma column + migration (NULL keeps today's behaviour, so no existing server
changes), zod validation on create and update, the repository, an
`--tool-call-timeout` flag mirroring `--health-check-timeout`, and the apply
schema so `apply -f` accepts what `get server -o yaml` emits.
That round-trip needed care: get emits `toolCallTimeoutSeconds: null` for every
server without an override, so the apply schema is nullable, not merely
optional -- otherwise the very first `get -o yaml | apply -f` on an untouched
server would have failed validation.
Bounded at one hour. A deadline exists so a wedged call answers instead of
hanging; a value beyond an hour is indistinguishable from having none.
It reaches mcplocal through server discovery rather than the project-scoped
serverOverrides map, and is applied on EVERY sync rather than only at first
registration -- raising a server's timeout should take effect at the next
refresh, not require the upstream to be dropped and rebuilt.
The endpoint sees the wire name (`docmost_search`), so it decodes to the
canonical `server/tool` before resolving. The override is keyed by server, so
every tool on that server inherits it without being enumerated, and the
failure message quotes the deadline actually applied rather than the global.
Smoke tests (tests/smoke/bounded-failures.smoke.test.ts) cover the two
production faults against the live proxy: a stale session recovers instead of
stranding the client on an uncorrelatable 404, every request is answered, and
`mcpctl trace` responds for an unknown code.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GqMidYEGUJG5fxeoTELBu2
`claude-opus-4-20250514` was pinned as the heavy provider and had been
returning 404 on every gate ranking and every pagination title. The only
visible symptom was a fallback that looked like an ordinary one -- it surfaced
here because the degradation notice added earlier in this branch finally
printed the reason.
Checked against GET /v1/models: BOTH pins were dead. The fast tier's
claude-haiku-3-5-20241022 is gone too, so that tier had been silently 404ing
as well.
The provider's listModels() asserted "Anthropic doesn't have a models listing
endpoint" and returned four hardcoded dated ids. That endpoint does exist and
answers fine with the OAuth token this deployment uses; the hardcoded list was
simply stale, and a test pinned it in place.
So: `claude-<family>-latest` (or a bare `opus` / `sonnet` / `haiku` / `fable`)
now resolves against the live list. Exact ids pass through untouched, so
pinning still works when someone wants it.
Newest is decided by `created_at`, never by parsing the version out of the id.
That is not incidental: `claude-opus-4-5` sorts ABOVE `claude-opus-5` as a
string, and "4-5" parses as a larger minor than "5". There is a test for
exactly that trap.
Resolution is cached (12h, MCPCTL_ANTHROPIC_MODEL_TTL_MS) so it is not a
per-call network hop, and shared across instances since the model list is
account-wide. When the endpoint is unreachable it falls back to a pinned
known-good id per family and says so on stderr -- the map going stale can then
only cost availability, never correctness.
The constructor default was `claude-sonnet-4-20250514`, also retired; it now
tracks the family too.
Local config: heavy -> claude-opus-latest, fast -> claude-haiku-latest.
Verified live: opus-latest -> claude-opus-5, haiku-latest ->
claude-haiku-4-5-20251001, sonnet-latest -> claude-sonnet-5, and an exact id
passes through.
Also found while there: both anthropic entries were named "anthropic", and the
registry keys by name, so the second silently OVERWROTE the first and one
tier's model was discarded entirely. Renamed to anthropic-fast /
anthropic-heavy so NamedProvider keeps them distinct and the tier split is
real.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GqMidYEGUJG5fxeoTELBu2
tool_call_trace and pipeline_execution are aggregates OF the stages, so letting
them compete always named the total and told you nothing. Verified live: now
reports 'slowest: paginate (252ms)' rather than 'tool_call_trace (509ms)'.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GqMidYEGUJG5fxeoTELBu2
The trace codes are now emitted in every deadline message and degradation
notice, so they need somewhere to go.
`mcpctl trace <code>` queries the audit events mcpd already stores and renders
the request as a waterfall: each step with its offset, duration, byte delta,
and any ⚠ degradation or ✗ error, then a summary naming the slowest step. No
new endpoint was needed -- GET /api/v1/audit/events?correlationId= already
existed and correlationId is already an indexed column; it just had nothing
writing meaningful values into it until this branch, and nothing reading it.
--strict exits non-zero when the trace contains an error or a degraded step, so
it composes into scripts. An empty result explains itself rather than printing
nothing: batches flush up to 5s late, old requests predate trace codes, and the
codes exclude I/L/O/U so a mistyped one is worth calling out.
Retention: AuditEvent had no prune at all. That was defensible while the table
was write-only; it is not now that a command reads it. Mirrors the AuditLog
convention exactly -- POST /api/v1/audit/events/purge, triggered rather than
scheduled, guarded by the existing audit-purge RBAC operation so it cannot be
granted by accident separately from the log purge. 30 days by default rather
than AuditLog's 90: these are several rows per MCP call, not a record of
administrative mutations.
Note for the plan's sake: I had assumed AuditLog retention was a scheduled job
registered in main.ts. It is not -- it is a manual endpoint. Mirroring what the
codebase actually does beat inventing a scheduler that exists for neither table.
completions are generated, so `trace` is registered in PROJECT_SCOPED_COMMANDS
and both shells regenerated; the freshness test passes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GqMidYEGUJG5fxeoTELBu2
The LLM budgets bound the failure we actually hit. This is the backstop for the
ones they cannot see: a wedged plugin hook, a virtual-tool handler, an upstream
path without its own timeout.
transport.onmessage IS the whole request pipeline, and it writes to a socket
that has already been hijacked from Fastify. The SDK does not await it, so a
throw became an unhandled rejection and a hang became silence -- in both cases
the client got no response at all and waited out its own timeout. onmessage is
now structured so that reaching transport.send() is unconditional: the route
call is raced against a deadline, and the notification flush and the send each
carry their own catch, so a failure while flushing can no longer cost the client
its response.
transport.onerror was never assigned, so SDK-level transport errors were
swallowed entirely. It is now.
Shape of the answer is deliberate. A tools/call comes back as a SUCCESSFUL
result with isError and readable text naming the trace code -- the same shape
router.ts already uses for an expired _resultId, because a transport error tends
to surface as a hard failure while a tool error is something the model reads and
acts on. It also says the upstream may still be running, which is true and
matters. Other methods get a JSON-RPC error (-32001).
MCPLOCAL_TOOLCALL_DEADLINE_MS defaults to TOOLCALL_TIMEOUT_MS + 30s rather than
120s. The watchdog arms earlier in the request than mcpd's fetch does, so at
equal values it would always fire first, masking mcpd's specific
UpstreamTimeoutError with a generic message and cutting off pagination and the
rest of the post-processing. There is a test asserting the two stay ordered.
The pause queue is exempt, but not bypassed. It blocks until a human operator
releases, edits or drops a response, so the deadline SUSPENDS -- clock stopped,
then re-armed with whatever was left. A test asserts the re-arm, because an
exemption that forgot to re-arm would silently make every paused request
immortal.
End-to-end: an upstream that never settles now yields a response in under a
second carrying ⚠, the deadline, and `mcpctl trace <code>`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GqMidYEGUJG5fxeoTELBu2
Second of the two faults that both presented as a 1800s hang.
mcplocal keeps MCP sessions in memory only, so any restart -- a deploy, an RPM
install, scripts/release.sh -- invalidates every connected client's session id.
The response was a bare 404 {"error":"Session not found"}: 3.5ms server-side,
but it lands OUTSIDE the JSON-RPC envelope with no id, so the client cannot
correlate it to its request and simply waits out its own timeout. Every mcpctl
tool call in that session became a 30-minute stall while the server was
perfectly healthy and answering other traffic in milliseconds.
Now the session is recreated, keeping the id the client already holds -- a
re-handshake would change it and invalidate everything the client has cached.
The SDK assigns sessionId/_initialized only while handling an `initialize`
(webStandardStreamableHttp.js:419-420), so adoptSession() replays exactly those
two assignments. It reaches into SDK internals, so it fails LOUDLY: the guard
throws with a pointer to itself, and session-adopt.test.ts is the canary that
goes red at `pnpm test` rather than in production on the next SDK bump.
Recreation is not silent. Doing it quietly would hand an agent an ungated tool
list with no signal that its gate state had vanished, so the first tools/call
result carries the established "⚠ Session state unavailable (...)" notice
telling it to call begin_session again.
onsessioninitialized does NOT fire on the adopt path, so its body is factored
into registerSession() and called explicitly. That is the subtle part: skipping
it would have produced recreated sessions with null userName on every audit
event, silently.
Also fixed, and it is the CAUSE rather than the symptom: MCP clients close a
session with DELETE + Content-Type: application/json and an empty body, which
Fastify's default parser rejected with FST_ERR_CTP_EMPTY_JSON_BODY before the
route handler ran. sessions.delete() therefore never happened and every
"closed" session leaked -- manufacturing the stale-session condition this
commit handles. Fixing recreation without fixing teardown would have shipped
the workaround and kept the cause.
project-mcp-endpoint.test.ts's "returns 404 for unknown session ID" inverts to
"recreates an unknown session instead of 404ing it", and asserts the id comes
back unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GqMidYEGUJG5fxeoTELBu2
The plumbing for request tracing was all present and almost entirely inert.
correlationId was generated at the endpoint and threaded into traffic events,
but:
- emitTrace() dropped it, so every durable tool_call_trace row had
correlationId = null and could not be joined to anything;
- ExecuteOptions.correlationId existed but its only production caller never
passed it, making every stage_execution and pipeline_execution row
unjoinable too -- dead code that looked like a feature;
- plugin events (all three gate emits) had no access to it at all;
- and the value itself was `<session-uuid>:<jsonrpc-id>`, which nobody can
retype from a screenshot.
Also fixed: onToolCallBefore intercepts returned without emitTrace, so every
tool call made while a session was gated produced no trace whatsoever.
The trace code now IS the correlationId. Nothing parses the old format --
checked across mcpd's audit repository and route, mcplocal's router and traffic
capture, and the CLI console -- sessionId is its own audit column and the
JSON-RPC id is in the traffic body, so nothing is lost. AuditEvent.correlationId
is already an indexed Postgres column, so lookup works with no migration.
8 chars of Crockford base32 without I/L/O/U: readable aloud, typeable from a
screenshot, 40 bits (a collision shows two traces, it corrupts nothing).
Getting it to the plugins and the executor needed AsyncLocalStorage, not a
parameter. Both are constructed per SESSION, not per request -- processContent
is a closure with no RouteContext -- so threading an argument would have touched
PluginSessionContext, every plugin, ExecuteOptions and StageContext. ALS also
stays correct with two requests in flight on one session, which a mutable field
on the session-scoped context would not; there is a test for exactly that.
PluginContextImpl.emitAuditEvent auto-fills it the way it already auto-fills
sessionId, so this covers every gate event and every plugin written later
without a per-callsite edit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GqMidYEGUJG5fxeoTELBu2
A per-call timeout bounds one call. It does not bound a stage that calls the
LLM in a loop -- and summarize-tree does worse than loop: buildTree recurses to
maxDepth (3 by default) and iterates per section at every level, then
groupSections iterates again. Hundreds of sequential calls are reachable, so a
10s per-call cap is a multiplier, not a bound.
StageBudget is one wall-clock budget shared across everything a single stage
invocation does. Once it is gone the stage takes its deterministic path --
first-line excerpts, numbered pages -- and says so. The executor builds one per
stage from config.budgetMs ?? MCPCTL_STAGE_LLM_BUDGET_MS (30s) and disposes it
in a finally, so a long-lived mcplocal never accumulates timers.
Two details that matter more than they look:
- A warm cache is never budget-gated. cachedSummarize and generatePageTitles
used ctx.cache.getOrCompute, which makes the budget check impossible to
place correctly; split into get/set so a cached summary -- which costs
nothing -- is still returned when the budget is spent.
- "No LLM configured" is NOT a degradation. It is a deliberate choice, and
numbered pages are the expected output there, so it gets no warning. Only
failures, timeouts and exhausted budgets do. Crying wolf on a working
configuration would train people to ignore the notice.
summarize-tree's single-block path previously had no try/catch at all, so a
failure escaped the stage entirely and executor.ts discarded the whole stage's
work. It now degrades like the others.
Degradation is reported three ways, all from util/degrade.ts: the reason in the
content behind the established "⚠ <feature> unavailable (<reason>)" prefix, the
count of affected sections (the recursion can hit one exhausted budget dozens of
times, so the user needs the reason once plus what it cost), and
degraded/degradedReason on the stage_execution audit event -- previously a
degradation was invisible in the trace.
docs/reliability.md now names ONE helper rather than a list of compliant call
sites. That list is exactly how this drifted: the doc named the gate and
llm/pagination.ts, and the newer stages never joined it.
Tests: the 20-section recursion completes in ~300ms against a 300ms budget;
a warm cache still serves real titles with the budget exhausted.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GqMidYEGUJG5fxeoTELBu2
docs/reliability.md has stated the rule since it was written: "LLM-optional
operations must be time-bounded, fall back deterministically, and report the
degradation". The gate and llm/pagination.ts obey it. The newer proxymodel
stages never adopted it -- stages/paginate.ts:72 awaited ctx.llm.complete()
with no timeout and no signal, and its try/catch caught errors, not hangs.
So when a provider hung rather than erroring, the promise never settled, the
tool call never returned, mcplocal never wrote to the already-hijacked socket,
and the client waited out its own 1800s timeout. In the journal that is three
POST /projects/docmost/mcp requests accepted and never answered -- the only
three such requests in 9,600 across every project, each followed immediately
by [llm-adapter] "trying next" and [paginate] "Smart page titles failed".
It read as a Docmost transport fault for two sessions. It was neither: Docmost
answers /search in 203ms, and through mcplocal on a fresh session in 461ms.
docmost_search returns ~14KB and crosses the pagination threshold; update_page
returns a small ack and does not. One tool paginated, one did not, and only the
paginating one could hang.
Bound it centrally in the adapter rather than at each call site, so a stage
that passes no options is still bounded and a stage written next year inherits
the guarantee:
- LLMCompleteOptions gains optional budgetMs/perCallTimeoutMs/signal. Optional
matters -- seven inline stubs implement LLMProvider structurally and a
required member would break every one.
- LLMProviderAdapter.complete() runs one budget with a deadline across the
whole failover chain, each attempt capped at min(perCall, remaining). A
per-provider timeout would have made the worst case N x timeout; wrapping
the method would have killed failover mid-chain.
- Every attempt now gets a real AbortSignal. anthropic and openai (which back
vllm) honour it; deepseek/ollama/gemini ignore it, and withTimeout's race
unblocks the caller anyway -- exactly what its doc comment anticipated.
- A caller abort stops failover: if nobody is waiting, don't burn the chain.
util/degrade.ts generalises the gate's bounded -> fallback -> loud log ->
degradedReason pattern into bounded(), which cannot throw, so the deterministic
fallback is unconditional rather than something a catch block must remember.
gate.ts is refactored onto it; plugin-gate and router-gate pass UNMODIFIED (61
tests), which is the proof the generalisation is faithful.
anySignal() rather than AbortSignal.any: that landed in Node 20.3 and
package.json declares >=20.0.0.
proxymodel-llm-adapter.test.ts now asserts signal: expect.any(AbortSignal) --
every provider attempt being cancellable is the guarantee, so it is pinned.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GqMidYEGUJG5fxeoTELBu2
The smoke suite talks to the live mcplocal through the HTTP boundary, which
serves wire names since #124 — assertions expecting `favourite/`, `all/` and
`smoke-aws-docs/` prefixes now check the underscore wire form (config pins
stay canonical). These two files were the release-gate failures after the
bbd3188 rollout; 166/166 green against the live fleet after the update.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JaFvfHrQyUKCGv6o3N2Wir
Follow-up to #124: the boundary WireNameCodec serves underscore-joined
names, but the gate plugin's tool inventories (initialize instructions and
begin_session response) still listed canonical `server/tool`, and the
favourite-index instruction told the model to prefer `favourite/<tool>` —
prose naming functions the model cannot call. Inventories now run through
sanitizeWireName and the instruction describes the favourite_/all_ prefixes.
Cosmetic for routing (slash names still pass through the codec) but
load-bearing for tool selection: models copy names out of prose.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JaFvfHrQyUKCGv6o3N2Wir
Drives /projects/:name/mcp over the real Streamable HTTP transport with a
fake websearch upstream: tools/list must serve `websearch_fetch_content`
(every name matching the OpenAI function-name charset), calling that wire
name must reach the upstream as bare `fetch_content`, and a legacy client
echoing the canonical `websearch/fetch_content` must still route.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JaFvfHrQyUKCGv6o3N2Wir
The proxy namespaces tools as `server/tool` (and favourite-index presents
`favourite/<tool>` / `all/<server>/<tool>`). A `/` is not a valid character
in an OpenAI-style function name, so hosts that forward MCP tool names
verbatim as LLM function names depend on the model faithfully echoing an
illegal name. LibreChat did exactly that: deepseek-v4-flash intermittently
dropped the `websearch/` prefix, LibreChat's registry lookup failed, and it
reported "This tool's MCP server is temporarily unavailable" while nothing
was down — the calls never reached mcplocal at all (confirmed against the
AuditEvent table, 2026-08-25). Claude Code and the pi extension only dodge
this because they sanitize names client-side.
Fix at the HTTP boundary only: a WireNameCodec rewrites tools/list responses
to wire-safe names (`/` -> `_`, exact-match reverse map, deterministic
suffix on collision) and maps tools/call names back before routing. Wired
into both /mcp and /projects/:name/mcp. Everything inside the proxy —
routing maps, plugins, favourites config, audit events — keeps canonical
names, and unknown inbound names (legacy clients echoing slash names,
virtual tools) pass through unchanged, so existing clients keep working.
Codecs are keyed per project and outlive the router cache TTL so a client
can call a tool it listed minutes earlier; after a restart the client's
initialize-time tools/list repopulates the map.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JaFvfHrQyUKCGv6o3N2Wir
Two corrections to the shell-bearing rebuild.
**Pin by digest.** It copied from `:latest`, so a rebuild silently ships
whatever upstream has moved to. When a probe started failing right after a
rebuild I could not tell a version change from a broken build, and burned
time on the wrong one — the binary's own `--version` prints 1.1.0 while
the image label says 1.6.0, so that was a red herring too. Now pinned to
sha256:dda8d56e…, which IS the running 1.6.0.
**CMD, not ENTRYPOINT.** Upstream sets `Cmd: ["/app/gitea-mcp"]` with no
entrypoint. mcpd maps a server's `command` to k8s `args`, which REPLACES
Cmd but only APPENDS to an ENTRYPOINT — so the ENTRYPOINT form would have
changed how the binary is invoked for any server that sets a command.
Matching upstream's shape keeps the non-injected path byte-identical.
gitea also needs `entrypoint` on its server row: its `command` is [], so
there is nothing for the injector wrapper to wrap without it. Set to
["/usr/local/bin/gitea-mcp"] via apply -f (patch cannot express an array).
Verified live: gitea RUNNING/healthy on secretDelivery: injector, with
vault-agent-init present and the command wrapped as
["/bin/sh","-c",". /vault/secrets/gitea-creds; exec \"$0\" \"$@\"",
"/usr/local/bin/gitea-mcp"]
`get_me` — which needs read:user, the scope that started this whole
session — returns the real account. Plaintext credentials across all
mcpctl server pod specs: 4 -> 0.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018vybEitX4FykeMatKe5Xki
Upstream docker.gitea.com/gitea-mcp-server is distroless — `Cmd` is
["/app/gitea-mcp"] and there is no /bin/sh at any path (verified by
exec'ing every candidate against the running pod).
That is fine until the server wants secretDelivery: injector. The OpenBao
agent renders secrets to a FILE, so mcpd wraps the container command as
`sh -c '. /vault/secrets/<name>; exec "$0" "$@"'`, which needs a shell.
gitea was the only server in the fleet blocked on this, and so the only
one whose token had to stay inline in its pod spec.
Copying one static Go binary onto debian:stable-slim is cheaper than
building and maintaining a static envexec shim, and follows the precedent
in deploy/Dockerfile.docmost-mcp — this repo already rebuilds third-party
MCP servers when it needs to change how they run.
ca-certificates is required rather than incidental: the binary talks HTTPS
to mysources.co.uk and the distroless base shipped a trust store we are
leaving behind. Verified in the built image: shell present, binary runs,
ca-certificates.crt present.
ENTRYPOINT is kept so the plain (non-injected) path behaves exactly like
upstream; mcpd replaces it with the sourcing wrapper only when the server
opts in.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018vybEitX4FykeMatKe5Xki
