feat(gate): drop the reasoning field from selection → ~1.5s (comfortable margin)
Some checks failed
CI/CD / lint (pull_request) Successful in 1m6s
CI/CD / typecheck (pull_request) Successful in 2m12s
CI/CD / test (pull_request) Successful in 1m20s
CI/CD / build (pull_request) Successful in 2m11s
CI/CD / smoke (pull_request) Failing after 2m47s
CI/CD / publish (pull_request) Has been skipped
Some checks failed
CI/CD / lint (pull_request) Successful in 1m6s
CI/CD / typecheck (pull_request) Successful in 2m12s
CI/CD / test (pull_request) Successful in 1m20s
CI/CD / build (pull_request) Successful in 2m11s
CI/CD / smoke (pull_request) Failing after 2m47s
CI/CD / publish (pull_request) Has been skipped
Even "<=8 words" was ignored — the model wrote a full-sentence reasoning, pushing
begin_session to ~7s (thin under the 8s budget). The gate only needs the names, so
request just {"selectedNames":[...]}. Validated live vs vllm-current
(glm-4.6-reap-fast): 1.5s, valid JSON. (reasoning defaults to '' in extractSelection.)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -95,11 +95,11 @@ export class LlmPromptSelector {
|
||||
): Promise<LlmSelectionResult> {
|
||||
const { getSystemPromptFn, signal, serverInfer } = opts;
|
||||
|
||||
// Keep the answer SHORT: latency is proportional to output length (a
|
||||
// no-think model runs ~10-13 tok/s), and a verbose reasoning field pushes a
|
||||
// selection past the gate's responsiveness budget. Ask for compact JSON with
|
||||
// a <=8-word reasoning — ~2s vs ~9s for a prose reasoning.
|
||||
const DEFAULT_SYSTEM_PROMPT = `You are a context selection assistant. Given a developer's task keywords and available project prompts, select the relevant ones. Return ONLY compact JSON: {"selectedNames":[...],"reasoning":"<=8 words"}. No prose outside the JSON. Priority 10 prompts must always be included.`;
|
||||
// Keep the answer as SHORT as possible: latency is proportional to output
|
||||
// length (a no-think model runs ~10-18 tok/s). Even an "<=8 words" reasoning
|
||||
// got ignored (models write a full sentence → ~7s). Drop the reasoning field
|
||||
// entirely — the gate only needs the names → just {selectedNames} → ~1-2s.
|
||||
const DEFAULT_SYSTEM_PROMPT = `You are a context selection assistant. Given a developer's task keywords and available project prompts, select the relevant ones. Return ONLY this JSON and nothing else: {"selectedNames":[...]}. No reasoning, no prose, no explanation. Priority 10 prompts must always be included.`;
|
||||
const systemPrompt = getSystemPromptFn
|
||||
? await getSystemPromptFn('llm-gate-context-selector', DEFAULT_SYSTEM_PROMPT)
|
||||
: DEFAULT_SYSTEM_PROMPT;
|
||||
@@ -109,7 +109,7 @@ export class LlmPromptSelector {
|
||||
Available prompts:
|
||||
${promptIndex.map((p) => `- ${p.name} (priority: ${p.priority}): ${p.summary ?? 'No summary'}${p.chapters?.length ? `\n Chapters: ${p.chapters.join(', ')}` : ''}`).join('\n')}
|
||||
|
||||
Select the relevant prompts. Return JSON: { "selectedNames": [...], "reasoning": "..." }`;
|
||||
Select the relevant prompts. Return ONLY: {"selectedNames":[...]}`;
|
||||
|
||||
const messages: ChatMessage[] = [
|
||||
{ role: 'system', content: systemPrompt },
|
||||
|
||||
Reference in New Issue
Block a user