feat(gate): fast no-think selection + dedicated selection Llm #88

michal · 2026-07-24T13:29:12Z

michal commented

2026-07-24 13:29:12 +00:00

Two knobs so gate prompt-selection is fast: (1) selection requests send no-think hints (enable_thinking=false, reasoning_effort=low), (2) MCPCTL_GATE_SELECTION_LLM pins a dedicated fast selection Llm independent of the project chat model. Pairs with the litellm qwen3-fast alias + a vllm-fast mcpd Llm. Safe/no-op until wired + backend up. 75 gate/selector tests green.

🤖 Generated with Claude Code

Two knobs so gate prompt-selection is fast: (1) selection requests send no-think hints (enable_thinking=false, reasoning_effort=low), (2) MCPCTL_GATE_SELECTION_LLM pins a dedicated fast selection Llm independent of the project chat model. Pairs with the litellm qwen3-fast alias + a vllm-fast mcpd Llm. Safe/no-op until wired + backend up. 75 gate/selector tests green. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

michal added 2 commits 2026-07-24 13:29:13 +00:00

feat(gate): ask reasoning models for a fast, no-think selection response 27480181bf

Prompt selection is mechanical classification — it gains nothing from chain-of-
thought, and a reasoning model (qwen3-thinking) otherwise burns its whole budget
reasoning (~40s, 6k+ chars) and blows the gate's 8s timeout. The gate's server
selection request now includes thinking-suppression hints, forwarded verbatim by
mcpd's passthrough adapter to litellm/vLLM:
  chat_template_kwargs.enable_thinking=false  (Qwen3 hard-off)
  reasoning_effort=low                         (o-series / newer vLLM)
Harmless for models that ignore them; if a backend rejects them the selector
falls back to the local provider. Override via MCPCTL_GATE_SELECT_EXTRA_BODY
(JSON), or '' to disable.

NOT yet validated live — the qwen3-thinking vLLM is crashlooping again (0/1,
HTTP 500), and whether litellm forwards chat_template_kwargs to vLLM is unconfirmed
(the /no_think prompt directive was NOT honored). Validate when the backend
recovers. mcpld gate/selector tests green (75).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(gate): MCPCTL_GATE_SELECTION_LLM — pin a dedicated fast selection Llm

CI/CD / lint (pull_request) Successful in 1m2s

Details

CI/CD / typecheck (pull_request) Successful in 2m7s

Details

CI/CD / test (pull_request) Successful in 1m19s

Details

CI/CD / build (pull_request) Successful in 2m11s

Details

CI/CD / smoke (pull_request) Failing after 2m47s

Details

CI/CD / publish (pull_request) Has been skipped

Details

5225b54901

Lets ops route gate prompt-selection at a fast (no-think) server Llm independent
of the project's chat llmProvider — so selection stays ~1-2s while chat keeps its
thinking model. Overrides the project llmProvider for the gate's server-selection
path; unset → prior behavior (use the project's llmProvider). Pairs with the
litellm qwen3-fast no-think alias (kubernetes-deployment) + a `vllm-fast` mcpd Llm.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

michal merged commit 3b96b0399e into main

2026-07-24 13:29:59 +00:00

michal referenced this issue from a commit

2026-07-24 13:30:01 +00:00

feat(gate): fast no-think selection + dedicated selection Llm (#88)

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: michal/mcpctl#88