93 lines
4.3 KiB
Markdown
93 lines
4.3 KiB
Markdown
|
# `claude-vllm` — Claude Code on the homelab LLM
|
||
|
|
|
||
|
|
## The short version
|
||
|
|
|
||
|
|
```bash
|
||
|
|
claude-vllm # default provider + model
|
||
|
|
claude-vllm --model deepseek-v4-max
|
||
|
|
claude-vllm --list # what would it use?
|
||
|
|
claude-vllm --print-env # the exports, without running claude
|
||
|
|
claude-vllm -- -p "summarise this repo" # anything after -- goes to claude
|
||
|
|
```
|
||
|
|
|
||
|
|
## Why no bridge is needed
|
||
|
|
|
||
|
|
The homelab gateway at `llm.ad.itaz.eu` is **LiteLLM in front of vLLM**, and
|
||
|
|
LiteLLM already serves the Anthropic Messages API:
|
||
|
|
|
||
|
|
```console
|
||
|
|
$ curl -s https://llm.ad.itaz.eu/v1/messages -X POST \
|
||
|
|
-H 'x-api-key: <key>' -H 'anthropic-version: 2023-06-01' \
|
||
|
|
-d '{"model":"deepseek-v4-fast","max_tokens":32,
|
||
|
|
"messages":[{"role":"user","content":"say only: ok"}]}'
|
||
|
|
{"type":"message","role":"assistant","model":"deepseek-v4-fast",
|
||
|
|
"content":[{"type":"text","text":"ok"}],"stop_reason":"end_turn", …}
|
||
|
|
```
|
||
|
|
|
||
|
|
That is the exact protocol Claude Code speaks. So pointing `ANTHROPIC_BASE_URL`
|
||
|
|
at the gateway is the whole integration — no translation layer, no proxy, no
|
||
|
|
mcpctl in the request path.
|
||
|
|
|
||
|
|
### Why not route it through mcpctl
|
||
|
|
|
||
|
|
mcpctl's LLM layer is a **client**, not a server: `mcpd` exposes
|
||
|
|
`/api/v1/llms` (management) and the adapters in
|
||
|
|
`src/mcpd/src/services/llm/adapters/` *call out* to Anthropic/OpenAI for gating,
|
||
|
|
prompt selection and agent chat. Nothing in mcpd or mcplocal serves
|
||
|
|
`/v1/messages`. A `claude-mcpctl` would therefore mean adding an
|
||
|
|
Anthropic-shaped passthrough to mcpd that re-wraps what LiteLLM already does —
|
||
|
|
new surface, new failure mode, no new capability.
|
||
|
|
|
||
|
|
It would only be worth it if you wanted mcpctl *in* the LLM path for its own
|
||
|
|
sake: per-project gating of model calls, audit of prompts, or budget
|
||
|
|
enforcement. Those are real features, but they are a passthrough endpoint in
|
||
|
|
mcpd, not a wrapper script.
|
||
|
|
|
||
|
|
## What it discovers, and from where
|
||
|
|
|
||
|
|
`claude-vllm` exists only so you don't paste four exports each time. It reuses
|
||
|
|
whatever you already configured for another agent — first hit wins:
|
||
|
|
|
||
|
|
| Order | Source | Supplies |
|
||
|
|
|-------|--------|----------|
|
||
|
|
| 1 | environment (`ANTHROPIC_BASE_URL`, `ANTHROPIC_AUTH_TOKEN`, `ANTHROPIC_MODEL`) | anything already set is respected |
|
||
|
|
| 2 | `--provider` / `--model` flags | provider, model |
|
||
|
|
| 3 | `~/.pi/agent` | `settings.json` → default provider/model · `models.json` → base URL, context window · `auth.json` → key |
|
||
|
|
| 4 | `~/.prime/agent` | same shape |
|
||
|
|
| 5 | `~/.config/opencode/opencode.jsonc` | `provider.<name>.options.{baseURL,apiKey}` |
|
||
|
|
|
||
|
|
The base URL and the credential always come from the *same* source, so one
|
||
|
|
gateway's URL is never paired with another's key.
|
||
|
|
|
||
|
|
## What it sets, and why each one
|
||
|
|
|
||
|
|
| Variable | Reason |
|
||
|
|
|----------|--------|
|
||
|
|
| `ANTHROPIC_BASE_URL` | The gateway. A stored OpenAI-style `…/v1` base has the suffix stripped — Claude Code appends `/v1/messages` itself, and `/v1/v1/messages` 404s. |
|
||
|
|
| `ANTHROPIC_AUTH_TOKEN` + `ANTHROPIC_API_KEY` | The gateway takes either; which one Claude Code sends has changed between releases. |
|
||
|
|
| `ANTHROPIC_MODEL` | The model to drive the session. |
|
||
|
|
| `ANTHROPIC_SMALL_FAST_MODEL`, `ANTHROPIC_DEFAULT_HAIKU_MODEL` | Background/summarisation calls otherwise ask for a real Haiku the gateway does not serve, and every one 404s. |
|
||
|
|
| `CLAUDE_CODE_MAX_CONTEXT_TOKENS` | Claude Code assumes 200k for models it has no table for. `deepseek-v4-*` is 393k, so without this it auto-compacts at half capacity. Taken from `contextWindow` in the provider config. |
|
||
|
|
| `CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS` | Beta headers the gateway does not implement make it reject otherwise-fine requests. |
|
||
|
|
| `CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC` | No telemetry about a non-Anthropic endpoint. |
|
||
|
|
|
||
|
|
## Expected noise
|
||
|
|
|
||
|
|
```
|
||
|
|
⚠ claude.ai connectors are disabled because ANTHROPIC_API_KEY or another auth
|
||
|
|
source is set and takes precedence over your claude.ai login
|
||
|
|
```
|
||
|
|
|
||
|
|
Unavoidable and harmless: it fires for *any* non-claude.ai auth, including
|
||
|
|
`ANTHROPIC_AUTH_TOKEN` alone (verified). It means your claude.ai org connectors
|
||
|
|
are not loaded for this session — which is the point of running against the
|
||
|
|
homelab.
|
||
|
|
|
||
|
|
## Verified
|
||
|
|
|
||
|
|
```console
|
||
|
|
$ claude-vllm --model deepseek-v4-fast -- -p "Reply with exactly: PARITY-OK"
|
||
|
|
claude-vllm: itaz · deepseek-v4-fast · https://llm.ad.itaz.eu
|
||
|
|
PARITY-OK
|
||
|
|
```
|