Files
mcpctl/src/mcplocal/tests/toolcall-deadline.test.ts

102 lines
3.9 KiB
TypeScript
Raw Normal View History

feat(mcplocal): every request answers, even when the pipeline wedges The LLM budgets bound the failure we actually hit. This is the backstop for the ones they cannot see: a wedged plugin hook, a virtual-tool handler, an upstream path without its own timeout. transport.onmessage IS the whole request pipeline, and it writes to a socket that has already been hijacked from Fastify. The SDK does not await it, so a throw became an unhandled rejection and a hang became silence -- in both cases the client got no response at all and waited out its own timeout. onmessage is now structured so that reaching transport.send() is unconditional: the route call is raced against a deadline, and the notification flush and the send each carry their own catch, so a failure while flushing can no longer cost the client its response. transport.onerror was never assigned, so SDK-level transport errors were swallowed entirely. It is now. Shape of the answer is deliberate. A tools/call comes back as a SUCCESSFUL result with isError and readable text naming the trace code -- the same shape router.ts already uses for an expired _resultId, because a transport error tends to surface as a hard failure while a tool error is something the model reads and acts on. It also says the upstream may still be running, which is true and matters. Other methods get a JSON-RPC error (-32001). MCPLOCAL_TOOLCALL_DEADLINE_MS defaults to TOOLCALL_TIMEOUT_MS + 30s rather than 120s. The watchdog arms earlier in the request than mcpd's fetch does, so at equal values it would always fire first, masking mcpd's specific UpstreamTimeoutError with a generic message and cutting off pagination and the rest of the post-processing. There is a test asserting the two stay ordered. The pause queue is exempt, but not bypassed. It blocks until a human operator releases, edits or drops a response, so the deadline SUSPENDS -- clock stopped, then re-armed with whatever was left. A test asserts the re-arm, because an exemption that forgot to re-arm would silently make every paused request immortal. End-to-end: an upstream that never settles now yields a response in under a second carrying ⚠, the deadline, and `mcpctl trace <code>`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GqMidYEGUJG5fxeoTELBu2
2026-08-25 23:41:09 +01:00
import { describe, it, expect, vi, afterEach } from 'vitest';
import {
createRequestDeadline,
DeadlineExceededError,
TOOLCALL_DEADLINE_MS,
} from '../src/http/deadline.js';
import { TOOLCALL_TIMEOUT_MS } from '../src/http/mcpd-client.js';
afterEach(() => { vi.restoreAllMocks(); });
describe('TOOLCALL_DEADLINE_MS', () => {
it('exceeds mcpd\'s own tool-call timeout', () => {
// The watchdog arms earlier in the request than mcpd's fetch does. At equal
// values it would always fire first, masking mcpd's specific
// UpstreamTimeoutError with a generic deadline message and cutting off
// pagination and the rest of the post-processing.
expect(TOOLCALL_DEADLINE_MS).toBeGreaterThan(TOOLCALL_TIMEOUT_MS);
});
it('stays well below a typical client timeout so our error arrives first', () => {
expect(TOOLCALL_DEADLINE_MS).toBeLessThan(600_000);
});
});
describe('createRequestDeadline', () => {
it('rejects with DeadlineExceededError when the budget runs out', async () => {
const d = createRequestDeadline('tools/call', 40);
await expect(Promise.race([
new Promise<never>(() => { /* the hang we are protecting against */ }),
d.expiry,
])).rejects.toBeInstanceOf(DeadlineExceededError);
d.dispose();
});
it('does not fire once disposed', async () => {
const d = createRequestDeadline('tools/call', 30);
d.dispose();
const outcome = await Promise.race([
d.expiry.then(() => 'fired', () => 'fired'),
new Promise((r) => setTimeout(() => { r('quiet'); }, 120)),
]);
expect(outcome).toBe('quiet');
});
it('stops the clock while suspended, so an operator pause cannot time out', async () => {
const d = createRequestDeadline('tools/call', 100);
// A pause queue wait far longer than the whole deadline.
const held = await d.suspended(async () => {
await new Promise((r) => setTimeout(r, 250));
return 'released';
});
expect(held).toBe('released');
// Still alive afterwards, with roughly the original budget left.
const outcome = await Promise.race([
d.expiry.then(() => 'fired', () => 'fired'),
new Promise((r) => setTimeout(() => { r('still-running'); }, 40)),
]);
expect(outcome).toBe('still-running');
d.dispose();
});
it('re-arms after suspension rather than becoming immortal', async () => {
const d = createRequestDeadline('tools/call', 60);
await d.suspended(async () => { await new Promise((r) => setTimeout(r, 100)); });
// The remaining budget must still expire — suspending is not a bypass.
await expect(Promise.race([
new Promise<never>(() => { /* hang */ }),
d.expiry,
])).rejects.toBeInstanceOf(DeadlineExceededError);
d.dispose();
});
});
feat: per-server tool-call timeout, on the server resource Completes the plan's last item. The deadline shipped with a global default; this makes it overridable per server, where the knowledge actually lives -- a server with genuinely slow tools declares its own budget instead of forcing the global up for everyone. Source of truth is the server resource, following healthCheck exactly: Prisma column + migration (NULL keeps today's behaviour, so no existing server changes), zod validation on create and update, the repository, an `--tool-call-timeout` flag mirroring `--health-check-timeout`, and the apply schema so `apply -f` accepts what `get server -o yaml` emits. That round-trip needed care: get emits `toolCallTimeoutSeconds: null` for every server without an override, so the apply schema is nullable, not merely optional -- otherwise the very first `get -o yaml | apply -f` on an untouched server would have failed validation. Bounded at one hour. A deadline exists so a wedged call answers instead of hanging; a value beyond an hour is indistinguishable from having none. It reaches mcplocal through server discovery rather than the project-scoped serverOverrides map, and is applied on EVERY sync rather than only at first registration -- raising a server's timeout should take effect at the next refresh, not require the upstream to be dropped and rebuilt. The endpoint sees the wire name (`docmost_search`), so it decodes to the canonical `server/tool` before resolving. The override is keyed by server, so every tool on that server inherits it without being enumerated, and the failure message quotes the deadline actually applied rather than the global. Smoke tests (tests/smoke/bounded-failures.smoke.test.ts) cover the two production faults against the live proxy: a stale session recovers instead of stranding the client on an uncorrelatable 404, every request is answered, and `mcpctl trace` responds for an unknown code. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GqMidYEGUJG5fxeoTELBu2
2026-08-26 00:03:06 +01:00
describe('per-server deadline override', () => {
it('falls back to the global when a server declares nothing', async () => {
const { McpRouter } = await import('../src/router.js');
const router = new McpRouter();
expect(router.getToolCallDeadlineMs('docmost/search')).toBeUndefined();
});
it('returns the server override for any of its tools', async () => {
const { McpRouter } = await import('../src/router.js');
const router = new McpRouter();
router.setServerDeadline('slowserver', 300_000);
// Resolved from the canonical `server/tool` name, so every tool on that
// server inherits it without being enumerated.
expect(router.getToolCallDeadlineMs('slowserver/render')).toBe(300_000);
expect(router.getToolCallDeadlineMs('slowserver/export')).toBe(300_000);
expect(router.getToolCallDeadlineMs('other/tool')).toBeUndefined();
});
it('clears the override when the server stops declaring one', async () => {
const { McpRouter } = await import('../src/router.js');
const router = new McpRouter();
router.setServerDeadline('slowserver', 300_000);
router.setServerDeadline('slowserver', undefined);
expect(router.getToolCallDeadlineMs('slowserver/render')).toBeUndefined();
});
});