build: fail the release when smoke tests fail #108

Merged

michal merged 1 commits from fix/release-fails-on-smoke into main 2026-08-10 16:16:06 +00:00

Conversation 0 Commits 1 Files Changed 3 +52 -8

michal commented 2026-08-10 16:15:52 +00:00

Owner

Copy Link

Problem

release.sh ran the smoke suite, printed WARNING: Smoke tests failed! and exited 0.

That is exactly how four broken readiness probes shipped unnoticed on 2026-08-10 — the warning scrolled past in the build log and the release reported success. docs and the smoke fixture had no probe at all, my-home-assistant pointed at a tool the upstream had renamed, and docmost was 401'ing on a stale password. Nothing failed.

Fix

Exit 1, with MCPCTL_ALLOW_SMOKE_FAILURE=1 as the escape hatch.

The message is deliberately explicit about what the gate does not do: smoke runs last, against the installed binary, so the package is already published and installed by the time it fails. It reports fleet breakage rather than preventing a bad artifact — so investigate the fleet, don't assume the build is bad and re-run.

The hang that had to be fixed first

Turning the gate on would have blocked every release, because one smoke test hung.

security.test.ts > /inspect SSE endpoint is accessible without authentication waited for a response body that by design never ends. It could only settle via the socket's inactivity timeout — and /inspect relays every project's MCP traffic, so during a full smoke run it is never idle. Run alone it passed and looked flaky; run with the suite it failed every time.

httpRequest gains a headersOnly option that resolves on the response headers and hangs up. The assertion only ever needed the status line, so this is both correct and load-independent.

Verification

• Full smoke suite: 158/158, 27/27 files (was 157/158, with this test timing out at 10s).
• The gate block, lifted verbatim out of release.sh and run with a stubbed failing smoke command: exits 1 and stops the script.
• Same block with MCPCTL_ALLOW_SMOKE_FAILURE=1: exits 0 and reaches the code after the gate.

## Problem `release.sh` ran the smoke suite, printed `WARNING: Smoke tests failed!` and **exited 0**. That is exactly how four broken readiness probes shipped unnoticed on 2026-08-10 — the warning scrolled past in the build log and the release reported success. `docs` and the smoke fixture had no probe at all, `my-home-assistant` pointed at a tool the upstream had renamed, and `docmost` was 401'ing on a stale password. Nothing failed. ## Fix Exit 1, with `MCPCTL_ALLOW_SMOKE_FAILURE=1` as the escape hatch. The message is deliberately explicit about what the gate does *not* do: smoke runs **last**, against the installed binary, so the package is already published and installed by the time it fails. It reports fleet breakage rather than preventing a bad artifact — so investigate the fleet, don't assume the build is bad and re-run. ## The hang that had to be fixed first Turning the gate on would have blocked every release, because one smoke test hung. `security.test.ts > /inspect SSE endpoint is accessible without authentication` waited for a response body that **by design never ends**. It could only settle via the socket's *inactivity* timeout — and `/inspect` relays every project's MCP traffic, so during a full smoke run it is never idle. Run alone it passed and looked flaky; run with the suite it failed every time. `httpRequest` gains a `headersOnly` option that resolves on the response headers and hangs up. The assertion only ever needed the status line, so this is both correct and load-independent. ## Verification - Full smoke suite: **158/158, 27/27 files** (was 157/158, with this test timing out at 10s). - The gate block, lifted verbatim out of `release.sh` and run with a stubbed failing smoke command: **exits 1** and stops the script. - Same block with `MCPCTL_ALLOW_SMOKE_FAILURE=1`: exits 0 and reaches the code after the gate.

michal added 1 commit 2026-08-10 16:15:53 +00:00

build: fail the release when smoke tests fail, and fix the SSE test that hung ... 822c1bb047

release.sh printed `WARNING: Smoke tests failed!` and exited 0. That is how four
broken readiness probes shipped unnoticed on 2026-08-10 — the warning scrolled
past in the build log and the release reported success.

It now exits 1, with `MCPCTL_ALLOW_SMOKE_FAILURE=1` as the escape hatch. The
message is explicit that smoke runs LAST, against the installed binary: the
package is already published and installed, so the failure reports fleet
breakage rather than preventing a bad artifact.

Turning the gate on required fixing a latent hang first, or every release would
have blocked on it. `security.test.ts > /inspect SSE endpoint …` waited for a
response body that by design never ends, so it could only settle via the
socket's *inactivity* timeout — and /inspect relays every project's MCP traffic,
so during a full smoke run it is never idle. Run alone it passed and looked
flaky; run with the suite it failed every time.

httpRequest gains `headersOnly`, which resolves on the response headers and
hangs up. The assertion only ever needed the status line.

Verified: full smoke suite 158/158 (was 157/158 with this test timing out); the
gate block lifted verbatim from release.sh exits 1 with a stubbed failing smoke
run, and exits 0 reaching subsequent code under MCPCTL_ALLOW_SMOKE_FAILURE=1.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019wUmrfkVQR6CKcYKxENq7k

michal merged commit ae5a6203f8 into main 2026-08-10 16:16:06 +00:00

michal referenced this issue from a commit 2026-08-10 16:16:08 +00:00

Merge PR #108: fail the release when smoke tests fail

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

No items

No Label

Milestone

No items

No Milestone

Projects

Clear projects

No project

Assignees

Clear assignees

michal

No Assignees

1 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: michal/mcpctl#108