michal/mcpctl
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(llm): probe upstream auth at registration time

Browse Source 
mcpd now runs a cheap auth probe whenever an Llm is created (or its
apiKeyRef/url is updated). Catches misconfigured tokens / wrong URLs at
registration with a 422 + structured error message, instead of silently
500-ing on first chat with a generic "fetch failed". Caught in the wild
today: the homelab Pulumi config exposed `MCPCTL_GATEWAY_TOKEN` (which
is mcpctl_pat_-prefixed, intended for LiteLLM→mcplocal direction) where
LiteLLM expects `LITELLM_MASTER_KEY` (sk-prefixed). The probe makes
this immediate.

Probe shape (LlmAdapter.verifyAuth):
  - OpenAI passthrough → GET <url>/v1/models. Cheap, idempotent, gated
    by the same auth as chat/completions.
  - Anthropic → POST /v1/messages with max_tokens:1, "ping". Anthropic
    has no list-models endpoint; this is the cheapest auth-exercising
    call.
  - Returns one of:
      { ok: true }
      { ok: false, reason: "auth", status, body }    — 401/403, fail hard
      { ok: false, reason: "unreachable", error }    — network, warn-only
      { ok: false, reason: "unexpected", status, body } — non-auth 4xx, warn-only

Behavior:
  - LlmService.create()/update() runs the probe after resolveApiKey.
    Throws LlmAuthVerificationError on `auth`, logs warn for
    unreachable/unexpected, swallows for offline registration.
  - Probe is skipped when there's no apiKeyRef (nothing to verify) or
    when the caller passes skipAuthCheck=true.
  - update() probes only when apiKeyRef OR url changes — pure
    description/tier updates don't trigger upstream calls.
  - Routes catch LlmAuthVerificationError and return 422 with
    `{ error, status }`. The CLI surfaces the message verbatim via
    ApiError.

Opt-out:
  - CLI: `mcpctl create llm ... --skip-auth-check` for offline
    registration before the upstream is reachable.
  - HTTP: side-channel body field `_skipAuthCheck: true` (stripped
    before validation, never persisted on the row).

Side fix in same commit (caught while testing): src/cli/src/index.ts
read `program.opts()` BEFORE `program.parse()`, so `--direct` was a
no-op for ApiClient — every command went to mcplocal regardless. Some
commands accidentally still worked because mcplocal forwards plain
`/api/v1/*` to mcpd, but flows that need direct SSE streaming (e.g.
`mcpctl chat`) couldn't reach mcpd. Fixed by peeking at process.argv
directly for the two global flags before Commander's parse runs.

Tests:
  - llm-adapters.test.ts (+8): OpenAI 200/401/403/404/network, Anthropic
    200/401/400 (typo'd model = unexpected, NOT auth — registration
    shouldn't block on bad model names that surface at chat time).
  - llm-service.test.ts (+6): create-throws-on-auth-fail (no row
    written), warn-only on unreachable/unexpected, skipAuthCheck
    bypass, no-key skip, update-only-probes-on-auth-affecting-change.

mcpd 775/775, mcplocal 715/715, cli 430/430.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Michal
2026-04-26 16:51:55 +01:00
parent 1f0be8a5c1
commit cc225eb70f
12 changed files with 495 additions and 14 deletions
Download Patch File Download Diff File Expand all files Collapse all files

99
src/mcpd/tests/llm-adapters.test.ts
View File

@@ -208,3 +208,102 @@ describe('LlmAdapterRegistry', () => {
expect(() => reg.get('bogus')).toThrow(UnsupportedProviderError);
});
});
describe('verifyAuth — registration-time probe', () => {
it('OpenAI passthrough: 200 from /v1/models → ok', async () => {
const fetchImpl = mockFetch([
{ match: /\/v1\/models$/, status: 200, body: { data: [{ id: 'gpt-4o-mini' }] } },
]);
const adapter = new OpenAiPassthroughAdapter('openai', { fetch: fetchImpl as unknown as typeof fetch });
const result = await adapter.verifyAuth(makeCtx({ url: 'http://lite:4000', apiKey: 'sk-good' }));
expect(result).toEqual({ ok: true });
expect(fetchImpl).toHaveBeenCalledWith('http://lite:4000/v1/models', expect.objectContaining({ method: 'GET' }));
const callInit = fetchImpl.mock.calls[0][1] as RequestInit;
expect((callInit.headers as Record<string, string>)['Authorization']).toBe('Bearer sk-good');
});
it('OpenAI passthrough: 401 → reason=auth (caller throws)', async () => {
const fetchImpl = mockFetch([
{ match: /\/v1\/models$/, status: 401, text: '{"error":"invalid_api_key"}' },
]);
const adapter = new OpenAiPassthroughAdapter('openai', { fetch: fetchImpl as unknown as typeof fetch });
const result = await adapter.verifyAuth(makeCtx({ url: 'http://lite:4000', apiKey: 'sk-bad' }));
expect(result.ok).toBe(false);
if (!result.ok) {
expect(result.reason).toBe('auth');
if (result.reason === 'auth') {
expect(result.status).toBe(401);
expect(result.body).toContain('invalid_api_key');
}
}
});
it('OpenAI passthrough: 403 → reason=auth', async () => {
const fetchImpl = mockFetch([
{ match: /\/v1\/models$/, status: 403, text: 'forbidden' },
]);
const adapter = new OpenAiPassthroughAdapter('openai', { fetch: fetchImpl as unknown as typeof fetch });
const result = await adapter.verifyAuth(makeCtx({ url: 'http://lite:4000', apiKey: 'k' }));
expect(result.ok).toBe(false);
if (!result.ok) expect(result.reason).toBe('auth');
});
it('OpenAI passthrough: 404 (proxy without /v1/models) → reason=unexpected (warn-only)', async () => {
const fetchImpl = mockFetch([
{ match: /\/v1\/models$/, status: 404, text: 'not found' },
]);
const adapter = new OpenAiPassthroughAdapter('openai', { fetch: fetchImpl as unknown as typeof fetch });
const result = await adapter.verifyAuth(makeCtx({ url: 'http://lite:4000', apiKey: 'k' }));
expect(result.ok).toBe(false);
if (!result.ok) expect(result.reason).toBe('unexpected');
});
it('OpenAI passthrough: network error → reason=unreachable (warn-only)', async () => {
const fetchImpl = vi.fn(async () => { throw new Error('ECONNREFUSED 127.0.0.1:9999'); });
const adapter = new OpenAiPassthroughAdapter('openai', { fetch: fetchImpl as unknown as typeof fetch });
const result = await adapter.verifyAuth(makeCtx({ url: 'http://localhost:9999', apiKey: 'k' }));
expect(result.ok).toBe(false);
if (!result.ok) {
expect(result.reason).toBe('unreachable');
if (result.reason === 'unreachable') {
expect(result.error).toContain('ECONNREFUSED');
}
}
});
it('Anthropic: 200 from /v1/messages probe → ok', async () => {
const fetchImpl = mockFetch([
{ match: /\/v1\/messages$/, status: 200, body: { id: 'msg_x', content: [{ type: 'text', text: 'pong' }] } },
]);
const adapter = new AnthropicAdapter({ fetch: fetchImpl as unknown as typeof fetch });
const result = await adapter.verifyAuth(makeCtx({ url: 'https://api.anthropic.com', apiKey: 'sk-ant-good' }));
expect(result.ok).toBe(true);
const callInit = fetchImpl.mock.calls[0][1] as RequestInit;
expect((callInit.headers as Record<string, string>)['x-api-key']).toBe('sk-ant-good');
const reqBody = JSON.parse(callInit.body as string) as { max_tokens: number };
expect(reqBody.max_tokens).toBe(1);
});
it('Anthropic: 401 → reason=auth', async () => {
const fetchImpl = mockFetch([
{ match: /\/v1\/messages$/, status: 401, text: '{"type":"authentication_error"}' },
]);
const adapter = new AnthropicAdapter({ fetch: fetchImpl as unknown as typeof fetch });
const result = await adapter.verifyAuth(makeCtx({ apiKey: 'bad' }));
expect(result.ok).toBe(false);
if (!result.ok) expect(result.reason).toBe('auth');
});
it('Anthropic: 400 (typo\'d model) → reason=unexpected, NOT auth', async () => {
// Auth was fine; the request was rejected for a different reason. We
// don't want to block registration on bad model names — that error
// surfaces at chat time when the user actually picks a model.
const fetchImpl = mockFetch([
{ match: /\/v1\/messages$/, status: 400, text: '{"error":"model not found"}' },
]);
const adapter = new AnthropicAdapter({ fetch: fetchImpl as unknown as typeof fetch });
const result = await adapter.verifyAuth(makeCtx({ apiKey: 'sk-ant-x', modelOverride: 'claude-fake' }));
expect(result.ok).toBe(false);
if (!result.ok) expect(result.reason).toBe('unexpected');
});
});

123
src/mcpd/tests/llm-service.test.ts
View File

@@ -1,5 +1,5 @@
import { describe, it, expect, vi } from 'vitest';
import { LlmService } from '../src/services/llm.service.js';
import { LlmService, LlmAuthVerificationError } from '../src/services/llm.service.js';
import type { ILlmRepository } from '../src/repositories/llm.repository.js';
import type { Llm, Secret } from '@prisma/client';
@@ -229,4 +229,125 @@ describe('LlmService', () => {
name: 'x', type: 'openai', model: 'gpt-4', tier: 'warp-speed',
})).rejects.toThrow();
});
// ── Auth verification at registration time ────────────────────────────
// Catches misconfigured tokens / wrong URLs at create/update, not at
// first chat. The actual upstream-probe logic lives in each adapter's
// verifyAuth(); these tests exercise the service's reaction to the
// probe result.
it('create: throws LlmAuthVerificationError when adapter probe returns reason=auth', async () => {
const repo = mockRepo();
const sec = makeSecret({ id: 'sec-bad', name: 'bad-key' });
const secrets = mockSecrets({ 'bad-key': sec }, { token: 'sk-bad' });
const adapters = {
get: vi.fn(() => ({
kind: 'openai',
verifyAuth: vi.fn(async () => ({ ok: false, reason: 'auth', status: 401, body: '{"error":"invalid_api_key"}' })),
})),
} as unknown as Parameters<typeof LlmService>[2]['adapters'];
// eslint-disable-next-line @typescript-eslint/no-explicit-any
const svc = new LlmService(repo, secrets as any, { adapters });
await expect(svc.create({
name: 'wrong-key', type: 'openai', model: 'gpt-4o',
apiKeyRef: { name: 'bad-key', key: 'token' },
})).rejects.toThrow(LlmAuthVerificationError);
// Repo.create should NOT have been called — no row written.
expect(repo.create).not.toHaveBeenCalled();
});
it('create: warn-only when probe returns reason=unreachable (still creates row)', async () => {
const repo = mockRepo();
const sec = makeSecret({ id: 'sec-x', name: 'k' });
const secrets = mockSecrets({ k: sec }, { token: 'k' });
const log = { warn: vi.fn() };
const adapters = {
get: vi.fn(() => ({
kind: 'openai',
verifyAuth: vi.fn(async () => ({ ok: false, reason: 'unreachable', error: 'ECONNREFUSED' })),
})),
} as unknown as Parameters<typeof LlmService>[2]['adapters'];
// eslint-disable-next-line @typescript-eslint/no-explicit-any
const svc = new LlmService(repo, secrets as any, { adapters, log });
const view = await svc.create({
name: 'offline', type: 'openai', model: 'gpt-4o',
url: 'http://localhost:9999',
apiKeyRef: { name: 'k', key: 'token' },
});
expect(view.name).toBe('offline');
expect(repo.create).toHaveBeenCalledOnce();
expect(log.warn).toHaveBeenCalledWith(expect.stringContaining('unreachable'));
});
it('create: warn-only when probe returns reason=unexpected (404 from a stripped proxy)', async () => {
const repo = mockRepo();
const sec = makeSecret({ id: 'sec-x', name: 'k' });
const secrets = mockSecrets({ k: sec }, { token: 'k' });
const log = { warn: vi.fn() };
const adapters = {
get: vi.fn(() => ({
kind: 'openai',
verifyAuth: vi.fn(async () => ({ ok: false, reason: 'unexpected', status: 404, body: 'not found' })),
})),
} as unknown as Parameters<typeof LlmService>[2]['adapters'];
// eslint-disable-next-line @typescript-eslint/no-explicit-any
const svc = new LlmService(repo, secrets as any, { adapters, log });
const view = await svc.create({
name: 'stripped-proxy', type: 'openai', model: 'gpt-4o',
apiKeyRef: { name: 'k', key: 'token' },
});
expect(view.name).toBe('stripped-proxy');
expect(log.warn).toHaveBeenCalledWith(expect.stringContaining('HTTP 404'));
});
it('create: skipAuthCheck=true bypasses the probe', async () => {
const repo = mockRepo();
const sec = makeSecret({ id: 'sec-x', name: 'k' });
const secrets = mockSecrets({ k: sec }, { token: 'k' });
const verifyAuth = vi.fn(async () => ({ ok: false, reason: 'auth', status: 401, body: 'no' }));
const adapters = {
get: vi.fn(() => ({ kind: 'openai', verifyAuth })),
} as unknown as Parameters<typeof LlmService>[2]['adapters'];
// eslint-disable-next-line @typescript-eslint/no-explicit-any
const svc = new LlmService(repo, secrets as any, { adapters });
const view = await svc.create({
name: 'offline-staging', type: 'openai', model: 'gpt-4o',
apiKeyRef: { name: 'k', key: 'token' },
}, { skipAuthCheck: true });
expect(view.name).toBe('offline-staging');
expect(verifyAuth).not.toHaveBeenCalled();
});
it('create: probe is skipped when no apiKeyRef (nothing to verify)', async () => {
const repo = mockRepo();
const verifyAuth = vi.fn();
const adapters = {
get: vi.fn(() => ({ kind: 'openai', verifyAuth })),
} as unknown as Parameters<typeof LlmService>[2]['adapters'];
// eslint-disable-next-line @typescript-eslint/no-explicit-any
const svc = new LlmService(repo, mockSecrets({}) as any, { adapters });
await svc.create({ name: 'no-key', type: 'ollama', model: 'llama3', url: 'http://localhost:11434' });
expect(verifyAuth).not.toHaveBeenCalled();
});
it('update: probes only when apiKeyRef or url changes', async () => {
const existing = makeLlm({ id: 'llm-up', name: 'up', apiKeySecretId: 'sec-x', apiKeySecretKey: 'token' });
const repo = mockRepo([existing]);
const sec = makeSecret({ id: 'sec-x', name: 'k' });
const secrets = mockSecrets({ k: sec }, { token: 'k' });
const verifyAuth = vi.fn(async () => ({ ok: true }));
const adapters = {
get: vi.fn(() => ({ kind: 'openai', verifyAuth })),
} as unknown as Parameters<typeof LlmService>[2]['adapters'];
// eslint-disable-next-line @typescript-eslint/no-explicit-any
const svc = new LlmService(repo, secrets as any, { adapters });
// Description-only update — no probe.
await svc.update('llm-up', { description: 'new' });
expect(verifyAuth).not.toHaveBeenCalled();
// URL change — probe runs.
await svc.update('llm-up', { url: 'http://new-host:4000' });
expect(verifyAuth).toHaveBeenCalledOnce();
});
});