The previous wording claimed the failure was "reproduced on x86_64", which
overstates it. The aarch64 observation is direct -- port 22 probed for 30
minutes with the Anaconda environment demonstrably up. The x86_64 run only
shows SSH not becoming available inside a 15-minute budget; that VM was
never observed reaching the rescue environment, because vitest's reporter
discards the streamed log.
The distinction decides where the next person looks, so it should not rest
on an inference presented as an observation.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015nRFZXpKwUVE4SRSHw6GjF
`provision debug` is the lab's recovery tool of last resort and had no
integration coverage on any architecture. Adding it -- x86_64 on KVM so
it runs in ~15 minutes, plus the aarch64 equivalent -- showed the rescue
environment coming up correctly but nothing ever listening on port 22.
Reproduced on both architectures, so it is neither ARM-specific nor an
emulation artefact, and it is orthogonal to the multi-arch work: x86_64
is unchanged by that. Documented in ARCHITECTURE.md with the leads worth
checking, rather than left as a silent gap.
Also restructures the ARM rescue suite to seed the machine into state
instead of discovering it first. That mirrors the DGX Spark situation --
SSH-onboarded, never PXE-discovered, architecture known only from its
record -- and holds the test to one emulated boot, since each spends
~15 of its ~18 minutes fetching Anaconda's stage2 under TCG.
KEEP_VM=1 leaves the VM up on failure; half-hour emulated runs are too
expensive to pay twice just to see what happened.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015nRFZXpKwUVE4SRSHw6GjF
Records the two things that are easy to get wrong and expensive to
rediscover: the DHCP option 93 value table (19 is arm64 UEFI HTTP boot,
20 is PC/AT BIOS), and that arm64 needs iPXE with LoadFile2 or the
kernel panics with unknown-block(0,0) and looks like a disk fault.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015nRFZXpKwUVE4SRSHw6GjF
The install guard catches an unsupported OS/architecture pair when the
machine's architecture is already known, but a machine queued before it
was ever discovered reaches dispatch with nothing having checked. Serving
it the x86-only Ubuntu kernel is exactly the failure this work exists to
fix, so stop with a legible reason on the console instead -- a machine
handed a kernel it cannot execute fails later and far less clearly.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015nRFZXpKwUVE4SRSHw6GjF
Covers what the boot-ISO ARM test never did: DHCP option 93 handing an
arm64 client an arm64 iPXE binary, dispatch serving an aarch64 kernel,
and `provision debug` reaching a rescue shell over SSH.
Split by cost. `arm-pxe` runs NBP handoff, discovery and rescue in about
20-30 minutes -- that is the path the DGX Sparks need. The full install
is another hour on top and only runs with ARM_PXE_FULL=1; an hour-plus
test that runs by default is a test nobody runs.
The suite fails fast if the arm64 iPXE build lacks LoadFile2, because the
symptom otherwise is a 30-minute boot ending in unknown-block(0,0) --
byte-identical to the DGX Spark bug this all started with, and easy to
misdiagnose as a reproduction of it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015nRFZXpKwUVE4SRSHw6GjF
Includes the case that motivated all of this: a machine recorded as
aarch64 and queued for rescue is served an aarch64 kernel, and installs
targeting either Spark are refused while rescue still works.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015nRFZXpKwUVE4SRSHw6GjF
--pxe-boot boots the installed system with a kernel and initrd from the
network, so it needs a root= for that machine. It used to hardcode our
Fedora LVM layout, which is wrong for anything else -- including both
DGX Sparks.
Where the root device comes from, in order of preference:
- already recorded on the machine (provision recheck now collects it)
- probed over SSH when --pxe-boot is requested and the machine answers
- reported from the rescue shell by debug-setup.sh, which mounts each
candidate read-only and picks the one with /etc/fstab and /usr
The rescue image cannot report it unprompted -- %pre/%post do not run in
rescue mode -- so the probe lives in the script the operator curls, which
already existed for the nc listener.
/api/discover now preserves fields a report omits. The probe posts only a
root device, and blanking a machine's inventory as a side effect of that
would be silent data loss.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015nRFZXpKwUVE4SRSHw6GjF
The DGX Sparks run DGX OS with a proprietary NVIDIA driver and firmware
stack. No image in our pipeline restores it, so an install destroys the
machine's software permanently -- and `labctl provision install` would
happily do it.
Machines carry an `onboard` classification and the `vendor_os` they must
keep running. Both install entry points (the HTTP route and the labd
command handler) refuse, naming the machine, what it runs, and pointing
at `provision debug` instead. Rescue is deliberately never guarded: being
unable to reinstall is exactly when you need a rescue shell.
Classification is a fact about the machine, not a blocklist. When a DGX
OS image joins the pipeline, teaching the installer about that vendor_os
is what unblocks these boxes. It is keyed on DMI identity, with the two
known Sparks also matched by MAC -- neither has DMI in bastion state
today, so a DMI-only rule would fail open on exactly the machines this
protects.
Also refuses an OS/architecture combination with no netboot artifacts,
which is Ubuntu on aarch64.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015nRFZXpKwUVE4SRSHw6GjF
Per the IANA Processor Architecture Types registry, 0x0013 (19) is
"arm uefi 64 boot from http". 20 is "pc/at bios boot from http", so an
arm64 machine using UEFI HTTP Boot never matched the tag and was never
offered the arm64 iPXE binary.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015nRFZXpKwUVE4SRSHw6GjF
The bastion served one x86_64 kernel to every machine regardless of
architecture, so an ARM64 box was handed a binary its UEFI will not
execute. This is why `labctl provision debug` could not rescue the DGX
Sparks during the 2026-08-11 kernel panic.
Architecture is resolved, never typed by an operator: the tracked machine
record first, then iPXE's ${buildarch} reported on the dispatch URL, then
the configured default. boot.ipxe gains &arch=${buildarch} so that signal
reaches the HTTP endpoint -- DHCP option 93 only ever reaches dnsmasq.
One script covers network PXE, UEFI HTTP boot and the boot ISO alike.
x86_64 keeps its unsuffixed /vmlinuz and /initrd.img so its rendered
scripts are byte-identical; aarch64 gets suffixed paths, its own Fedora
mirror, and console=ttyAMA0 instead of nomodeset, which does not mean the
same thing on arm64 and can leave a headless machine with no console.
--pxe-boot no longer hardcodes the Fedora LVM layout: root device and
dracut args come from the machine's record, and dispatch falls back to
rescue rather than guessing a root= that would leave a machine unbootable.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015nRFZXpKwUVE4SRSHw6GjF
The aarch64 PXE work touches every iPXE template. Capture what an x86_64
machine is served today, straight from the templates rather than by hand,
so any unintended change to that path fails a test instead of a machine.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015nRFZXpKwUVE4SRSHw6GjF
