mcpctl/docs/llm-tiers.md

LLM tiers — tracking the served model without hardcoding it

The homelab's served model changes. glm-4.6-reap became deepseek-v4-flash; something else will replace it. Every place mcpctl writes a model id down is a place that silently rots when that happens — a row left requesting a suspended model gets HTTP 400 model not found on the next call, and the failure surfaces somewhere unhelpful (a gate that quietly stops ranking prompts, an agent that 500s).

mcpctl's answer is that it never names a model, and prefers not to name a row. Consumers ask for a role; the registry says who currently fills it.

The three layers

LiteLLM (llm.ad.itaz.eu)      deepseek-v4-flash, -fast, -low, -think, -max
        ▲                     served-model ids — change on every model swap
        │  Pulumi owns this mapping
mcpd Llm rows                 vllm-current (tier: fast)
        ▲                     vllm-think   (tier: heavy)
        │                     stable identities; `model` follows the deployment
mcpctl consumers              "give me the fast one"
                              no model id, no row name

Layer 1 → 2 is Pulumi's job. deployments/mcpctl/llm-target.ts in the kubernetes-deployment repo declares both rows via the @mcpctl/pulumi provider and feeds them the active served-model name. Swap the model, run a targeted pulumi up, and the rows follow. See pulumi-provider-llm-autopoint.

Layer 2 → 3 is src/mcplocal/src/server-llm.ts. tier is already a first-class field on the Llm resource and is set by the same Pulumi resource that sets model, so the two cannot drift.

Resolution rules

resolveServerLlmByTier(client, tier) returns the row filling a tier, or null. A row is eligible when:

• its tier matches exactly;
• its status is active — selecting a row with no live backend just moves the failure to the first inference call;
• its kind is not virtual. Virtual rows are backed by some user's mcplocal over the SSE control channel, i.e. by a machine and personal credentials nobody chose deliberately. An explicit pin may still name one; automatic resolution must not route a project's traffic through a laptop.

When a tier holds several rows — heavy holds both vllm-think and the cloud anthropic-fallback — an ordered well-known name list breaks the tie:

tier	preference order
fast	vllm-fast, vllm-current
heavy	vllm-think, vllm-thinking

Names that match nothing are inert. That is deliberate: the list carries both the current names and the symmetric names the rows might be renamed to, so a rename stays a Pulumi-only change and neither deploy order breaks the other. Rows absent from the list are still eligible — they just sort last, then by name, so the choice is stable across calls regardless of mcpd's list order.

Override the order without a release:

MCPCTL_LLM_PREFER_FAST=spare-row,vllm-current
MCPCTL_LLM_PREFER_HEAVY=vllm-think

Resolution never throws. mcpd being unreachable is a normal degraded state for mcplocal, and every caller has a fallback path; returning null keeps that fallback intact instead of failing session setup.

Who uses it

Gate prompt-selection (project-mcp-endpoint.ts). Order:

1. MCPCTL_GATE_SELECTION_LLM — a global pin, so selection can sit on a fast no-think Llm while chat keeps a thinking model;
2. the project's own llmProvider (none disables);
3. whichever row currently fills the fast tier.

Step 3 is what makes an unpinned project work. Without it, a project with no llmProvider fell through to the local personal-token provider — and on a machine whose personal key is stale, that means every gated session silently ran on priority-ordered prompts instead of LLM ranking:

[gate] LLM prompt-selection failed: Anthropic HTTP 404: model: claude-opus-4-20250514
       — falling back to priority-ordered prompts

The gate prints a ⚠ Smart prompt-selection unavailable banner in the begin_session response whenever it degrades. Absence of that banner on a project with no pin is what the smoke test asserts.

Explicit pins still win everywhere — this only changes what happens when nothing is pinned.

Why not rename vllm-current to vllm-fast?

name is immutable in mcpd (agents and projects reference it), so a rename is a data migration plus a Pulumi resource replace, not an edit. Once resolution is tier-based nothing in mcpctl reads the names, so the migration buys nothing. vllm-fast would also be actively confusing: the fast row points at deepseek-v4-flash (the base route, which carries no forced extra_body), while LiteLLM separately serves a deepseek-v4-fast route that pins thinking: false. Two different things, one name apart.

Testing

• src/mcplocal/tests/server-llm.test.ts — selection rules, tiebreak, env override, degraded paths.
• src/mcplocal/tests/smoke/llm-tier.smoke.test.ts — against the live stack: both tiers resolve, the resolved fast row answers real inference (the drift check — a row pointing at a suspended model fails here), and an unpinned gated project gets LLM-ranked selection.