Michal
2c90f5971d
docs/reliability.md has stated the rule since it was written: "LLM-optional
operations must be time-bounded, fall back deterministically, and report the
degradation". The gate and llm/pagination.ts obey it. The newer proxymodel
stages never adopted it -- stages/paginate.ts:72 awaited ctx.llm.complete()
with no timeout and no signal, and its try/catch caught errors, not hangs.
So when a provider hung rather than erroring, the promise never settled, the
tool call never returned, mcplocal never wrote to the already-hijacked socket,
and the client waited out its own 1800s timeout. In the journal that is three
POST /projects/docmost/mcp requests accepted and never answered -- the only
three such requests in 9,600 across every project, each followed immediately
by [llm-adapter] "trying next" and [paginate] "Smart page titles failed".
It read as a Docmost transport fault for two sessions. It was neither: Docmost
answers /search in 203ms, and through mcplocal on a fresh session in 461ms.
docmost_search returns ~14KB and crosses the pagination threshold; update_page
returns a small ack and does not. One tool paginated, one did not, and only the
paginating one could hang.
Bound it centrally in the adapter rather than at each call site, so a stage
that passes no options is still bounded and a stage written next year inherits
the guarantee:
- LLMCompleteOptions gains optional budgetMs/perCallTimeoutMs/signal. Optional
matters -- seven inline stubs implement LLMProvider structurally and a
required member would break every one.
- LLMProviderAdapter.complete() runs one budget with a deadline across the
whole failover chain, each attempt capped at min(perCall, remaining). A
per-provider timeout would have made the worst case N x timeout; wrapping
the method would have killed failover mid-chain.
- Every attempt now gets a real AbortSignal. anthropic and openai (which back
vllm) honour it; deepseek/ollama/gemini ignore it, and withTimeout's race
unblocks the caller anyway -- exactly what its doc comment anticipated.
- A caller abort stops failover: if nobody is waiting, don't burn the chain.
util/degrade.ts generalises the gate's bounded -> fallback -> loud log ->
degradedReason pattern into bounded(), which cannot throw, so the deterministic
fallback is unconditional rather than something a catch block must remember.
gate.ts is refactored onto it; plugin-gate and router-gate pass UNMODIFIED (61
tests), which is the proof the generalisation is faithful.
anySignal() rather than AbortSignal.any: that landed in Node 20.3 and
package.json declares >=20.0.0.
proxymodel-llm-adapter.test.ts now asserts signal: expect.any(AbortSignal) --
every provider attempt being cancellable is the guarantee, so it is pinned.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GqMidYEGUJG5fxeoTELBu2
64 lines
2.3 KiB
TypeScript
64 lines
2.3 KiB
TypeScript
/**
|
|
* Bound an async operation with a timeout.
|
|
*
|
|
* Reliability principle: any LLM-*optional* operation (gate prompt-selection,
|
|
* pagination summaries) must be time-bounded so a slow/hung LLM (e.g. a
|
|
* thinking model that streams for minutes) can never block the caller. On
|
|
* timeout the returned promise rejects with a {@link TimeoutError}; callers are
|
|
* expected to fall back deterministically and report the degradation.
|
|
*
|
|
* The passed `AbortSignal` is aborted on timeout so fetch-based providers can
|
|
* cancel the in-flight request. Providers that ignore the signal are still
|
|
* unblocked immediately by the race (their orphaned request settles harmlessly
|
|
* in the background).
|
|
*/
|
|
export class TimeoutError extends Error {
|
|
constructor(label: string, ms: number) {
|
|
super(`${label} timed out after ${ms}ms`);
|
|
this.name = 'TimeoutError';
|
|
}
|
|
}
|
|
|
|
export async function withTimeout<T>(
|
|
run: (signal: AbortSignal) => Promise<T>,
|
|
ms: number,
|
|
label: string,
|
|
): Promise<T> {
|
|
const controller = new AbortController();
|
|
let timer: ReturnType<typeof setTimeout> | undefined;
|
|
const timeout = new Promise<never>((_, reject) => {
|
|
timer = setTimeout(() => {
|
|
controller.abort();
|
|
reject(new TimeoutError(label, ms));
|
|
}, ms);
|
|
});
|
|
try {
|
|
return await Promise.race([run(controller.signal), timeout]);
|
|
} finally {
|
|
if (timer !== undefined) clearTimeout(timer);
|
|
}
|
|
}
|
|
|
|
/**
|
|
* Combine a timeout signal with an optional caller signal.
|
|
*
|
|
* `AbortSignal.any` landed in Node 20.3 but `package.json` declares `>=20.0.0`,
|
|
* so we cannot rely on it being present. Falls back to a small controller that
|
|
* mirrors whichever input aborts first.
|
|
*/
|
|
export function anySignal(timeoutSignal: AbortSignal, callerSignal?: AbortSignal): AbortSignal {
|
|
if (!callerSignal) return timeoutSignal;
|
|
const anyFn = (AbortSignal as { any?: (signals: AbortSignal[]) => AbortSignal }).any;
|
|
if (typeof anyFn === 'function') return anyFn([timeoutSignal, callerSignal]);
|
|
|
|
const controller = new AbortController();
|
|
if (timeoutSignal.aborted || callerSignal.aborted) {
|
|
controller.abort();
|
|
return controller.signal;
|
|
}
|
|
const onAbort = (): void => { controller.abort(); };
|
|
timeoutSignal.addEventListener('abort', onAbort, { once: true });
|
|
callerSignal.addEventListener('abort', onAbort, { once: true });
|
|
return controller.signal;
|
|
}
|