`provision debug` is the lab's recovery tool of last resort and had no
integration coverage on any architecture. Adding it -- x86_64 on KVM so
it runs in ~15 minutes, plus the aarch64 equivalent -- showed the rescue
environment coming up correctly but nothing ever listening on port 22.
Reproduced on both architectures, so it is neither ARM-specific nor an
emulation artefact, and it is orthogonal to the multi-arch work: x86_64
is unchanged by that. Documented in ARCHITECTURE.md with the leads worth
checking, rather than left as a silent gap.
Also restructures the ARM rescue suite to seed the machine into state
instead of discovering it first. That mirrors the DGX Spark situation --
SSH-onboarded, never PXE-discovered, architecture known only from its
record -- and holds the test to one emulated boot, since each spends
~15 of its ~18 minutes fetching Anaconda's stage2 under TCG.
KEEP_VM=1 leaves the VM up on failure; half-hour emulated runs are too
expensive to pay twice just to see what happened.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015nRFZXpKwUVE4SRSHw6GjF
Records the two things that are easy to get wrong and expensive to
rediscover: the DHCP option 93 value table (19 is arm64 UEFI HTTP boot,
20 is PC/AT BIOS), and that arm64 needs iPXE with LoadFile2 or the
kernel panics with unknown-block(0,0) and looks like a disk fault.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015nRFZXpKwUVE4SRSHw6GjF
Covers all components (bastion, labd, labctl, agent, modules),
data flow, machine lifecycle, disk layout, kickstart features,
deployment, testing, security, known issues, and planned work.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Documents the 2026-03-30 debugging session: root cause (console=ttyS0
on UART-less hardware), what was tried, what was fixed, and remaining
work items.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When using `labctl provision debug <target> --sshd`, the rescue
kickstart generates host keys, starts sshd (pw: debug) and nc
listener (port 2323), and reports the IP back to bastion via
/api/progress callback. Fully self-contained, no mounted FS needed.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
