mcpctl/docs/llm-tiers.md

# LLM tiers — tracking the served model without hardcoding it

The homelab's served model changes. `glm-4.6-reap` became `deepseek-v4-flash`;
something else will replace it. Every place mcpctl writes a model id down is a
place that silently rots when that happens — a row left requesting a suspended
model gets `HTTP 400 model not found` on the next call, and the failure surfaces
somewhere unhelpful (a gate that quietly stops ranking prompts, an agent that
500s).

mcpctl's answer is that **it never names a model, and prefers not to name a
row**. Consumers ask for a *role*; the registry says who currently fills it.

## The three layers

```
LiteLLM (llm.ad.itaz.eu)      deepseek-v4-flash, -fast, -low, -think, -max
        ▲                     served-model ids — change on every model swap
        │  Pulumi owns this mapping
mcpd Llm rows                 vllm-current (tier: fast)
        ▲                     vllm-think   (tier: heavy)
        │                     stable identities; `model` follows the deployment
mcpctl consumers              "give me the fast one"
                              no model id, no row name
```

**Layer 1 → 2 is Pulumi's job.** `deployments/mcpctl/llm-target.ts` in the
`kubernetes-deployment` repo declares both rows via the `@mcpctl/pulumi`
provider and feeds them the active served-model name. Swap the model, run a
targeted `pulumi up`, and the rows follow. See
[pulumi-provider-llm-autopoint](../src/pulumi/README.md).

**Layer 2 → 3 is `src/mcplocal/src/server-llm.ts`.** `tier` is already a
first-class field on the `Llm` resource and is set by the same Pulumi resource
that sets `model`, so the two cannot drift.

## Resolution rules

`resolveServerLlmByTier(client, tier)` returns the row filling a tier, or
`null`. A row is eligible when:

- its `tier` matches exactly;
- its `status` is `active` — selecting a row with no live backend just moves
  the failure to the first inference call;
- its `kind` is **not** `virtual`. Virtual rows are backed by some user's
  `mcplocal` over the SSE control channel, i.e. by a machine and personal
  credentials nobody chose deliberately. An explicit pin may still name one;
  automatic resolution must not route a project's traffic through a laptop.

When a tier holds several rows — `heavy` holds both `vllm-think` and the cloud
`anthropic-fallback` — an ordered **well-known name** list breaks the tie:

| tier | preference order |
|------|------------------|
| `fast` | `vllm-fast`, `vllm-current` |
| `heavy` | `vllm-think`, `vllm-thinking` |

Names that match nothing are inert. That is deliberate: the list carries both
the current names and the symmetric names the rows might be renamed to, so a
rename stays a Pulumi-only change and neither deploy order breaks the other.
Rows absent from the list are still eligible — they just sort last, then by
name, so the choice is stable across calls regardless of mcpd's list order.

Override the order without a release:

```bash
MCPCTL_LLM_PREFER_FAST=spare-row,vllm-current
MCPCTL_LLM_PREFER_HEAVY=vllm-think
```

Resolution never throws. mcpd being unreachable is a normal degraded state for
`mcplocal`, and every caller has a fallback path; returning `null` keeps that
fallback intact instead of failing session setup.

## Who uses it

**Gate prompt-selection** (`project-mcp-endpoint.ts`). Order:

1. `MCPCTL_GATE_SELECTION_LLM` — a global pin, so selection can sit on a fast
   no-think Llm while chat keeps a thinking model;
2. the project's own `llmProvider` (`none` disables);
3. whichever row currently fills the `fast` tier.

Step 3 is what makes an unpinned project work. Without it, a project with no
`llmProvider` fell through to the local personal-token provider — and on a
machine whose personal key is stale, that means every gated session silently
ran on priority-ordered prompts instead of LLM ranking:

```
[gate] LLM prompt-selection failed: Anthropic HTTP 404: model: claude-opus-4-20250514
       — falling back to priority-ordered prompts
```

The gate prints a `⚠ Smart prompt-selection unavailable` banner in the
`begin_session` response whenever it degrades. Absence of that banner on a
project with no pin is what the smoke test asserts.

Explicit pins still win everywhere — this only changes what happens when
nothing is pinned.

## Why not rename `vllm-current` to `vllm-fast`?

`name` is immutable in mcpd (agents and projects reference it), so a rename is
a data migration plus a Pulumi resource replace, not an edit. Once resolution
is tier-based nothing in mcpctl reads the names, so the migration buys nothing.
`vllm-fast` would also be actively confusing: the fast row points at
`deepseek-v4-flash` (the base route, which carries no forced `extra_body`),
while LiteLLM separately serves a `deepseek-v4-fast` route that pins
`thinking: false`. Two different things, one name apart.

## Testing

- `src/mcplocal/tests/server-llm.test.ts` — selection rules, tiebreak, env
  override, degraded paths.
- `src/mcplocal/tests/smoke/llm-tier.smoke.test.ts` — against the live stack:
  both tiers resolve, the resolved fast row answers real inference (the drift
  check — a row pointing at a suspended model fails here), and an unpinned
  gated project gets LLM-ranked selection.