fix(smoke): actually clean up smoke-test resources

341 of 356 projects on the shared mcpd were smoke-test leftovers, plus 11
never-expiring `smoke-*` mcptokens sitting on the real `mcpctl-development`
and `sre` projects. Enough to make the project picker unusable.

Two causes, both silent:

- `delete project X --force` — `--force` is valid on `create project` but NOT
  on `delete`, so every cleanup call exited non-zero and deleted nothing.
  project-llm-ref's afterAll looked correct and had never worked, which is why
  there were 84 each of smoke-proj-{ok,orphan,none}-*.
- mcptoken.smoke had no afterAll at all; its cleanup was an `it()` at the end
  of the file, so it was skipped whenever an earlier assertion failed.

Cleanup now lives in `afterAll` (runs on failure too) and is no longer gated
on the health probe: the project is created through mcpd, which can be up when
the gateway probe is not, and deleting a project that was never created is a
no-op.

Adds `pnpm smoke:clean` for the case afterAll cannot cover — a killed process
(CI timeout, Ctrl-C, OOM). Dry-run by default, `--yes` to apply. Only touches
`smoke-*` names and protects the shared `smoke-data` / `smoke-aws-docs`
fixture, which five suites depend on and which must survive a concurrent run.

Verified end to end: created a throwaway smoke project, confirmed the dry run
lists without deleting, then `--yes` removed it and left all 15 real projects
untouched.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017BMXdb2qZbPSh8Q7XpTyjB
This commit is contained in:
Michal
2026-08-08 18:49:42 +01:00
parent e638f81456
commit a90dd091f0
4 changed files with 130 additions and 13 deletions

View File

@@ -14,7 +14,7 @@
*
* Run with: pnpm test:smoke
*/
import { describe, it, expect, beforeAll } from 'vitest';
import { describe, it, expect, beforeAll, afterAll } from 'vitest';
import http from 'node:http';
import https from 'node:https';
import { execSync } from 'node:child_process';
@@ -87,9 +87,23 @@ describe('mcptoken smoke', () => {
}
}, 20_000);
// Cleanup belongs in afterAll, not an `it()`: a failing assertion earlier in
// the file must not strand server-side projects. Deliberately NOT gated on
// `gatewayUp` — the project is created through mcpd, which can be reachable
// even when the gateway healthz probe is not, and deleting something that was
// never created is a harmless no-op.
//
// `delete project` takes no --force (only `create project` does); passing it
// made every one of these calls exit non-zero and silently do nothing, which
// is how hundreds of smoke-* projects accumulated on the shared mcpd.
afterAll(() => {
run(`delete project ${PROJECT_NAME}`);
run(`delete project ${OTHER_PROJECT}`);
});
it('creates the project and a project-scoped mcptoken', () => {
if (!gatewayUp) return;
run(`delete project ${PROJECT_NAME} --force`); // cleanup leftovers — best-effort
run(`delete project ${PROJECT_NAME}`); // cleanup leftovers — best-effort
const createProj = run(`create project ${PROJECT_NAME} --force`);
expect(createProj.code).toBe(0);
@@ -149,10 +163,8 @@ describe('mcptoken smoke', () => {
expect(report.error ?? '').toMatch(/401|revoked|Invalid token/i);
}, 20_000);
it('cleans up test fixtures', () => {
it('recorded a tool name for the gated catalogue', () => {
if (!gatewayUp) return;
run(`delete project ${PROJECT_NAME} --force`);
run(`delete project ${OTHER_PROJECT} --force`);
expect(knownToolName === undefined || typeof knownToolName === 'string').toBe(true);
});
});

View File

@@ -81,17 +81,20 @@ describe('project-llm-ref smoke', () => {
}
}, 30_000);
// Not gated on `mcpdUp`: if creation got far enough to leave a project
// behind, cleanup must still run. Deleting a non-existent project is a no-op.
// (`delete project` takes no --force — passing it made these calls fail
// silently and leak a project per run.)
afterAll(() => {
if (!mcpdUp) return;
run(`delete project ${PROJ_OK} --force`);
run(`delete project ${PROJ_ORPHAN} --force`);
run(`delete project ${PROJ_NONE} --force`);
run(`delete project ${PROJ_OK}`);
run(`delete project ${PROJ_ORPHAN}`);
run(`delete project ${PROJ_NONE}`);
run(`delete llm ${LLM_NAME}`);
});
it('project with --llm pointing at a registered Llm describes without warning', () => {
if (!mcpdUp) return;
run(`delete project ${PROJ_OK} --force`);
run(`delete project ${PROJ_OK}`);
const created = run(`create project ${PROJ_OK} --llm ${LLM_NAME}`);
expect(created.code, created.stderr || created.stdout).toBe(0);
@@ -104,7 +107,7 @@ describe('project-llm-ref smoke', () => {
it('project with --llm naming an unregistered Llm shows the warning line', () => {
if (!mcpdUp) return;
run(`delete project ${PROJ_ORPHAN} --force`);
run(`delete project ${PROJ_ORPHAN}`);
const created = run(`create project ${PROJ_ORPHAN} --llm claude-ghost-${SUFFIX}`);
expect(created.code, created.stderr || created.stdout).toBe(0);
@@ -117,7 +120,7 @@ describe('project-llm-ref smoke', () => {
it('project with --llm none treats it as an explicit disable (no warning)', () => {
if (!mcpdUp) return;
run(`delete project ${PROJ_NONE} --force`);
run(`delete project ${PROJ_NONE}`);
const created = run(`create project ${PROJ_NONE} --llm none`);
expect(created.code).toBe(0);