feat(gate): terse selection prompt so a no-think model answers in ~2s
Some checks failed
CI/CD / lint (pull_request) Successful in 1m3s
CI/CD / typecheck (pull_request) Successful in 2m10s
CI/CD / test (pull_request) Successful in 1m21s
CI/CD / build (pull_request) Successful in 2m14s
CI/CD / smoke (pull_request) Failing after 3m16s
CI/CD / publish (pull_request) Has been skipped

With the fast (no-think) route, gate-selection latency is now proportional to the
ANSWER length (a no-think model runs ~10-13 tok/s), not hidden thinking. A verbose
"reasoning" field pushed a selection to ~9s (over the 8s budget). Request compact
JSON with a <=8-word reasoning → ~2s (vs ~9s prose, or ~1.2s with no reasoning).
Validated live against vllm-fast (glm-4.6-reap-fast): 2.0s, valid selection JSON,
zero reasoning tokens.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Michal
2026-07-24 21:38:08 +01:00
parent 3b96b0399e
commit a432e9e19f

View File

@@ -95,7 +95,11 @@ export class LlmPromptSelector {
): Promise<LlmSelectionResult> {
const { getSystemPromptFn, signal, serverInfer } = opts;
const DEFAULT_SYSTEM_PROMPT = `You are a context selection assistant. Given a developer's task keywords and a list of available project prompts, select which prompts are relevant to their work. Return a JSON object with "selectedNames" (array of prompt names) and "reasoning" (brief explanation). Priority 10 prompts must always be included.`;
// Keep the answer SHORT: latency is proportional to output length (a
// no-think model runs ~10-13 tok/s), and a verbose reasoning field pushes a
// selection past the gate's responsiveness budget. Ask for compact JSON with
// a <=8-word reasoning — ~2s vs ~9s for a prose reasoning.
const DEFAULT_SYSTEM_PROMPT = `You are a context selection assistant. Given a developer's task keywords and available project prompts, select the relevant ones. Return ONLY compact JSON: {"selectedNames":[...],"reasoning":"<=8 words"}. No prose outside the JSON. Priority 10 prompts must always be included.`;
const systemPrompt = getSystemPromptFn
? await getSystemPromptFn('llm-gate-context-selector', DEFAULT_SYSTEM_PROMPT)
: DEFAULT_SYSTEM_PROMPT;