mcpctl/docs/inference-tasks.md

Inference Tasks (v5)

A durable inference task queue. Every inference call lands as a row in the database; mcpd's previous in-memory request map is gone. mcpd restart no longer drops in-flight work, worker disconnects no longer fail callers, and pools that have no live workers can still accept work that drains when one shows up.

The queue is the same persistence layer underneath two API surfaces:

• Sync (existing): POST /api/v1/llms/<name>/infer and POST /api/v1/agents/<name>/chat. Caller's HTTP request stays open until the task finishes, exactly like pre-v5. Internally it's now a DB row with a 10-min wait. Used by mcpctl chat / mcpctl chat-llm.
• Async (new — this doc): POST /api/v1/inference-tasks. Caller gets a task id immediately and polls / streams / cancels separately. Useful for fire-and-forget scripts, queued work for offline pools, and operator visibility into in-flight jobs.

Lifecycle

                ┌──────► claimed ──first chunk──► running ──┐
   pending ──┐  │             │                              │
             ├──┘ on dispatch  │ on revert (worker disconnect)│
             │                  └─────────► pending           │
             ▼                                                ▼
       cancelled ◄───────────── DELETE ──────────────► completed | error

Status	Meaning
pending	In queue, no worker has it yet (or claim was reverted)
claimed	A worker has it; dispatch frame went down the SSE channel
running	First chunk came back; streaming is in flight
completed	Worker POSTed terminal result
error	Permanent failure (auth, bad request, queue timeout)
cancelled	Caller cancelled via DELETE

A worker disconnecting mid-task reverts claimed/running rows back to pending (the row's claimedBy is cleared); a sibling worker on the same pool drains it. Tasks survive mcpd restart — the row is the source of truth, so a fresh mcpd reads the queue and either matches a worker or waits.

Async API

POST   /api/v1/inference-tasks                  → 201 + { id, status, poolName, llmName, streaming, createdAt }
GET    /api/v1/inference-tasks?...               → list (owner-scoped by default; ?status, ?poolName, ?agentId, ?limit)
GET    /api/v1/inference-tasks/<id>             → poll one row
GET    /api/v1/inference-tasks/<id>/stream      → SSE: `event: chunk` + `event: terminal`
DELETE /api/v1/inference-tasks/<id>             → cancel pending/claimed/running; no-op on terminal

Foreign-owner ids return 404 (not 403) on every endpoint — keeps id-enumeration from leaking task existence across users.

Enqueue

curl -X POST https://mcpd/api/v1/inference-tasks \
  -H 'Authorization: Bearer ...' \
  -H 'Content-Type: application/json' \
  -d '{
    "llmName": "vllm-local-qwen3",
    "request": {
      "messages": [{"role":"user","content":"summarize the changelog"}],
      "max_tokens": 500
    },
    "streaming": false
  }'

# 201 Created
# {
#   "id": "cmoip...",
#   "status": "pending",
#   "poolName": "vllm-local-qwen3",
#   "llmName": "vllm-local-qwen3",
#   "streaming": false,
#   "createdAt": "2026-04-28T..."
# }

Rejects with 400 for public Llms — the existing POST /api/v1/llms/<name>/infer is the right tool there. Rejects with 404 for missing Llm names.

Poll

curl https://mcpd/api/v1/inference-tasks/cmoip... \
  -H 'Authorization: Bearer ...'

# 200
# {
#   "id": "cmoip...",
#   "status": "completed",
#   "responseBody": { "choices": [{"message":{"role":"assistant","content":"..."}}] },
#   "claimedBy": "<providerSessionId>",
#   "createdAt": "...", "claimedAt": "...", "completedAt": "..."
# }

Stream live updates

curl -N https://mcpd/api/v1/inference-tasks/cmoip.../stream \
  -H 'Authorization: Bearer ...'

# event: chunk
# data: {"data":"hello "}
#
# event: chunk
# data: {"data":"world","done":true}
#
# event: terminal
# data: { ...full row... }

Already-terminal tasks emit one terminal event and close. Caller disconnect cleans up server-side.

Cancel

curl -X DELETE https://mcpd/api/v1/inference-tasks/cmoip... \
  -H 'Authorization: Bearer ...'

Returns 200 + the cancelled row. Cancelling a completed / error / already-cancelled task is a no-op (returns the current row). The worker that picked the task up may still execute the underlying inference call (no remote-cancel protocol yet); the result is just discarded server-side.

CLI

mcpctl get tasks                # table view: ID, STATUS, POOL, LLM, MODEL, STREAM, AGE, WORKER
mcpctl get task <id>            # describe one
mcpctl get task <id> -o yaml    # full row, yaml format

# Enqueue async via chat-llm:
mcpctl chat-llm vllm-local-qwen3 --async -m "summarize the changelog"
# stdout: cmoip... (just the id, pipeable)
# stderr: (task cmoip... enqueued for pool 'vllm-local-qwen3', status=pending)

# Wire to other commands:
ID=$(mcpctl chat-llm vllm-local-qwen3 --async -m "long task")
sleep 30
mcpctl get task $ID

RBAC

A new tasks resource. Default action mapping:

Method	Action
GET	view
POST	create
DELETE	delete

Aliases: task, inference-task, inference-tasks all normalize to tasks so mcpctl create rbac-binding --resource task --role view ... works regardless of which form the operator types.

Owner scoping is enforced at the route layer ON TOP of the RBAC hook:

• A caller with only the implicit role sees and can cancel their own tasks.
• view:tasks resource-wide is required to see / list / cancel another user's tasks.

GC

mcpd runs a 5-min ticker that:

• Flips pending tasks older than the queue timeout (default 1 h) to error with a clear message ("Task expired in pending state after ...s"). Any caller still polling sees a clean failure.
• Deletes terminal tasks (completed/error/cancelled) past the retention window (default 7 d). Indexes back the WHERE status IN (...) AND completedAt < cutoff predicate so the sweep is cheap even with millions of rows.

Both timeouts are constants in main.ts; making them per-deployment configurable is a v6 concern.

Limitations / Out-of-scope for v5

• Streaming chunks are not persisted. The DB stores only the final assembled body (or null for streaming-only consumers). A caller who connects to /stream AFTER the task completed gets one terminal event with no chunk replay. Persisting deltas would be a per-chunk INSERT — too expensive at SSE granularity.
• Public-Llm queueing is not exposed. Public Llms are HTTP adapters with no SSE worker, so the queue concept doesn't apply. Direct /llms/<name>/infer against a public Llm stays the right call.
• Multi-instance mcpd: the in-process EventEmitter that wakes blocked HTTP handlers fires only on the mcpd that holds the row's blocked handler. With multiple mcpd replicas behind a load balancer the wakeup wouldn't cross instances. v6 swap: pg LISTEN/NOTIFY, no schema change.
• Worker capacity is one task per session at a time (today). A per-session capacity hint + scheduler-level concurrency control is a v6+ concern.
• Remote cancel protocol: cancelling a running task marks the row as cancelled but the worker keeps executing. The result POST is discarded server-side. Real cancellation (an SSE message that tells the worker to abort its in-flight call) is a v6 concern.

See also

• virtual-llms.md — the Llm-side primitives (kind, lifecycle, pools) that the queue routes around.
• agents.md — how the chat path uses the same queue underneath.