feat(chat): LLM failover chain + show which model answered #83

michal · 2026-07-21T22:40:18Z

michal commented

2026-07-21 22:40:18 +00:00

Chat now fails over across an ordered LLM chain instead of failing when the pinned model is down/drifted, and shows which model actually answered.

Why

You saw vllm-current ✗ HTTP 400 (drifted off the served model) with anthropic available, and expected failover. Chat only failed over on transport errors, not 4xx, and there was no fallback target.

Changes

Ordered fallback chain: an Llm declares extraConfig.fallbacks: string[]; the dispatcher builds primary-pool → fallback-pool(s) and tries them in order (resolveCandidatesWithFallbacks).
Fail over on real failures: runOneInference advances on a non-2xx status (e.g. a drifted-model 400) or an empty/invalid completion — not just thrown transport errors. Streaming fails over pre-first-chunk.
Show which model answered: ChatResult + the SSE final frame carry {llm, model, failedOver}; the CLI prints model: <llm> (<model>) each turn and ⚠ failed over → answered by … on a fallback.
Exhaustion names the last model + upstream body (not "no choice").

Tests

3 failover unit tests (primary 400 → fallback answers + model reported; primary answers → failedOver=false; all fail → clear error). mcpd 948 + CLI 508 green; tsc + lint clean. docs/reliability.md updated.

Follow-up (live, this session): provision the homelab chain vllm-current → anthropic-fallback (anthropic key as an mcpd secret + provider wiring).

🤖 Generated with Claude Code

Chat now fails over across an ordered LLM chain instead of failing when the pinned model is down/drifted, and shows which model actually answered. ## Why You saw `vllm-current ✗ HTTP 400` (drifted off the served model) with anthropic available, and expected failover. Chat only failed over on *transport* errors, not 4xx, and there was no fallback target. ## Changes - **Ordered fallback chain**: an `Llm` declares `extraConfig.fallbacks: string[]`; the dispatcher builds primary-pool → fallback-pool(s) and tries them in order (`resolveCandidatesWithFallbacks`). - **Fail over on real failures**: `runOneInference` advances on a non-2xx status (e.g. a drifted-model 400) or an empty/invalid completion — not just thrown transport errors. Streaming fails over pre-first-chunk. - **Show which model answered**: `ChatResult` + the SSE `final` frame carry `{llm, model, failedOver}`; the CLI prints `model: <llm> (<model>)` each turn and `⚠ failed over → answered by …` on a fallback. - **Exhaustion** names the last model + upstream body (not "no choice"). ## Tests 3 failover unit tests (primary 400 → fallback answers + model reported; primary answers → failedOver=false; all fail → clear error). **mcpd 948 + CLI 508 green; tsc + lint clean.** docs/reliability.md updated. Follow-up (live, this session): provision the homelab chain vllm-current → anthropic-fallback (anthropic key as an mcpd secret + provider wiring). 🤖 Generated with [Claude Code](https://claude.com/claude-code)

michal added 1 commit 2026-07-21 22:40:18 +00:00

feat(chat): LLM failover chain + show which model answered

CI/CD / typecheck (pull_request) Successful in 1m4s

Details

CI/CD / lint (pull_request) Successful in 2m14s

Details

CI/CD / test (pull_request) Successful in 1m18s

Details

CI/CD / build (pull_request) Successful in 2m30s

Details

CI/CD / smoke (pull_request) Failing after 3m18s

Details

CI/CD / publish (pull_request) Has been skipped

Details

18524c1c57

Chat is LLM-essential but not model-specific — instead of failing when the
pinned model is down/drifted, it now fails over across an ordered chain and
reports which model actually answered.

- Ordered fallback: an Llm declares `extraConfig.fallbacks: string[]`; the
  dispatcher builds primary-pool → fallback-pool(s) candidates and tries them
  in order (resolveCandidatesWithFallbacks).
- Fail over on real failures, not just transport: runOneInference now advances
  on a non-2xx status (e.g. a 400 from a drifted model) or an empty/invalid
  completion, not only thrown transport errors. Streaming fails over
  pre-first-chunk (already threw on 4xx).
- Transparency: ChatResult + the SSE `final` frame carry {llm, model,
  failedOver}; the CLI prints `model: <llm> (<model>)` each turn and
  `⚠ failed over → answered by …` when a fallback was used.
- Exhaustion names the last model + upstream body (not "no choice").

Tests: 3 failover unit tests (primary 400 → fallback answers + model reported;
primary answers → failedOver=false; all fail → clear aggregated error).
mcpd 948 + CLI 508 green; tsc + lint clean. docs/reliability.md updated.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

michal merged commit efbe99832b into main

2026-07-21 22:40:30 +00:00

michal deleted branch feat/llm-failover-chain

2026-07-21 22:40:30 +00:00

michal referenced this issue from a commit

2026-07-21 22:40:31 +00:00

Merge pull request 'feat(chat): LLM failover chain + show which model answered' (#83) from feat/llm-failover-chain into main

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: michal/mcpctl#83