feat(chat): LLM failover chain + show which model answered
Some checks failed
CI/CD / typecheck (pull_request) Successful in 1m4s
CI/CD / lint (pull_request) Successful in 2m14s
CI/CD / test (pull_request) Successful in 1m18s
CI/CD / build (pull_request) Successful in 2m30s
CI/CD / smoke (pull_request) Failing after 3m18s
CI/CD / publish (pull_request) Has been skipped
Some checks failed
CI/CD / typecheck (pull_request) Successful in 1m4s
CI/CD / lint (pull_request) Successful in 2m14s
CI/CD / test (pull_request) Successful in 1m18s
CI/CD / build (pull_request) Successful in 2m30s
CI/CD / smoke (pull_request) Failing after 3m18s
CI/CD / publish (pull_request) Has been skipped
Chat is LLM-essential but not model-specific — instead of failing when the
pinned model is down/drifted, it now fails over across an ordered chain and
reports which model actually answered.
- Ordered fallback: an Llm declares `extraConfig.fallbacks: string[]`; the
dispatcher builds primary-pool → fallback-pool(s) candidates and tries them
in order (resolveCandidatesWithFallbacks).
- Fail over on real failures, not just transport: runOneInference now advances
on a non-2xx status (e.g. a 400 from a drifted model) or an empty/invalid
completion, not only thrown transport errors. Streaming fails over
pre-first-chunk (already threw on 4xx).
- Transparency: ChatResult + the SSE `final` frame carry {llm, model,
failedOver}; the CLI prints `model: <llm> (<model>)` each turn and
`⚠ failed over → answered by …` when a fallback was used.
- Exhaustion names the last model + upstream body (not "no choice").
Tests: 3 failover unit tests (primary 400 → fallback answers + model reported;
primary answers → failedOver=false; all fail → clear aggregated error).
mcpd 948 + CLI 508 green; tsc + lint clean. docs/reliability.md updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -28,10 +28,24 @@ Note: the gate's prompt-ranking uses the **heavy client provider's own model**
|
||||
it deliberately does *not* force the project's vLLM model onto it (doing so made
|
||||
every selection fail silently when the model wasn't anthropic-servable).
|
||||
|
||||
## LLM-*essential* operations
|
||||
## LLM-*essential* operations — failover chain
|
||||
|
||||
Chat needs the LLM — it can't fall back. It must **fail fast with a clear,
|
||||
actionable message** instead of a vague one. Chat surfaces the upstream status +
|
||||
body (which names the model + reason), e.g.
|
||||
`LLM returned no completion (HTTP 400): {"error":"model deepseek-v4-flash not found"}`,
|
||||
rather than "Adapter returned no choice".
|
||||
Chat needs *an* LLM but not a *specific* one. Instead of failing when the pinned
|
||||
model is down, chat **fails over across an ordered chain** and **reports which
|
||||
model actually answered**.
|
||||
|
||||
- **Chain:** an `Llm` declares fallbacks in `extraConfig.fallbacks: string[]`
|
||||
(Llm names, in order). The dispatcher builds an ordered candidate list —
|
||||
the primary's pool, then each fallback's pool — and tries them in order.
|
||||
- **Fails over on real failures**, not just transport: a non-2xx status
|
||||
(e.g. a drifted model's `400`) or an empty/invalid completion now advances to
|
||||
the next candidate (`chat.service.ts` `runOneInference`). Streaming fails over
|
||||
pre-first-chunk.
|
||||
- **Transparency:** `ChatResult` / the SSE `final` frame carry `llm`, `model`,
|
||||
and `failedOver`. The CLI prints `model: <llm> (<model>)` per turn, and
|
||||
`⚠ failed over → answered by <llm> (<model>)` when a fallback was used.
|
||||
- **Exhaustion is clear:** if every candidate fails, the error names the last
|
||||
model + upstream status/body (not "no choice").
|
||||
|
||||
Homelab chain: `vllm-current` (the served vLLM) → an `anthropic-fallback` server
|
||||
Llm as the always-up last resort.
|
||||
|
||||
Reference in New Issue
Block a user