Step 7 called release.sh — which restarts mcplocal and runs the smoke suite
against the binary it just installed — and then restarted mcplocal and ran the
whole suite again. Two full suites inside a minute trips mcpd's rate limiter,
so the second run came back with six 429s and printed
SMOKE TESTS FAILED — system may be unhealthy. Consider rollback
over a deploy that was fine. Seen for real on 35d506d: the first suite was
162/162 green, the second failed only on `mcpd returned 429: Rate limit
exceeded`, and a clean re-run afterwards was 162/162 again.
A deploy script that recommends rolling back a healthy release is worse than
one that says nothing. One run, one verdict — release.sh's exit status.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JNXFvxanvM6uiFcb4Mp3xU
Same hazard as the package build, with the cluster on the receiving end: a
branch behind main builds images missing whatever landed there, and deploy-k8s.sh
pins that sha in Pulumi — making the stale build the cluster's source of truth.
build-mcpd.sh gets its own call because it is run standalone as well as from
deploy-k8s.sh, so neither can rely on the other having checked.
`--dry-run` is exempt. It builds and cuts over nothing, and blocking a read-only
inspection is exactly what teaches people to export MCPCTL_ALLOW_BEHIND_MAIN=1
permanently — which would disable the gate for the real deploys too.
The failure text is now artifact-agnostic ("produce an artifact" / "shipping
it"), since one helper now speaks for packages, images and deploys.
Verified against a synthetic ref one commit ahead: build-mcpd.sh exits 1 before
any docker work, and deploy-k8s.sh exits 1 before the test gate, the pg_dump,
the image build and pulumi. Neither the working tree nor HEAD was moved to test
this — the ref was built with git commit-tree and deleted afterwards.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019wUmrfkVQR6CKcYKxENq7k
passwd smoke read ~/.config/mcpctl/credentials and silently skipped; the CLI
stores creds at ~/.mcpctl/credentials. Now verified live against prod (4/4).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
scripts/deploy-k8s.sh replaces fulldeploy.sh's rollout-restart-:latest pattern
(which bypassed Pulumi and left no rollback target). It:
- gates on pnpm test:run
- captures the current prod images as immutable rollback tags (skopeo) + records digests
- pg_dumps the prod DB before the destructive-capable `prisma db push`
- builds/pushes mcpd+mcplocal tagged with the git short-sha
- pins the sha in ../kubernetes-deployment/Pulumi.homelab.yaml and runs
`pulumi up --target` the mcpd/mcplocal Deployments only (avoids the SOGo
docker-image resource that needs a local docker daemon)
- waits for rollout + /healthz, installs the CLI RPM, runs smoke tests
- prints an exact rollback recipe on post-cutover failure
--dry-run validated: tests/pg_dump/targeted preview run read-only.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
