feat(bastion): first-class aarch64 support in the network-PXE path #17
Reference in New Issue
Block a user
Delete Branch "feat/arm64-pxe-support"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Why
On 2026-08-11
spark-2935(DGX Spark, aarch64) panicked withVFS: Unable to mount root fs on unknown-block(0,0). The obvious recovery tool —labctl provision debug, "PXE boot into Fedora rescue mode" — was useless: the bastion served a single x86_64 kernel to every machine regardless of architecture, so an ARM64 box got a binary its UEFI will not execute.The boot-ISO path had already solved this. This brings the network PXE path to parity.
What changed
Architecture is resolved, never flagged. No
--archanywhere./dispatchresolves in order: tracked machine record → iPXE's${buildarch}reported at boot → configured default.boot.ipxegained&arch=${buildarch}because DHCP option 93 only ever reaches dnsmasq, never the HTTP endpoint; one script then covers network PXE, UEFI HTTP boot and the boot ISO alike.Per-arch artifacts. x86_64 keeps unsuffixed
/vmlinuz+/initrd.img; aarch64 gets/vmlinuz-aarch64+/initrd-aarch64.img, its own Fedora mirror, andconsole=tty0 console=ttyAMA0,115200instead ofnomodeset— which does not mean the same thing on arm64 and can leave a headless machine with no console at all.--pxe-bootno longer hardcodeslabvg. Root device and dracut args come from the machine's record, observed three ways:provision recheckcollects them,provision debug --pxe-bootprobes over SSH when the machine answers, anddebug-setup.shreports them from a rescue shell (mounting each candidate read-only, picking the one with/etc/fstaband/usr). When nothing is known, dispatch serves rescue rather than guessing aroot=that would leave a machine unbootable.The Sparks are guarded. Machines carry
onboard: "pxe" | "ssh"and thevendor_osthey must keep running. Installs are refused at both entry points, naming the machine and pointing atprovision debug; rescue is never guarded — being unable to reinstall is exactly when you need a rescue shell. Modelled as a fact about the machine, not a blocklist: the refusal follows from "no image in our pipeline restoresdgx-os", so adding a SparkOS image is what unblocks these boxes.Two corrections to the brief
Option 93 handling already existed in
dnsmasq.conf.ts. The real bug washttpboot-arm64matching client-arch 20, which per IANA is pc/at bios boot from http; arm64 UEFI HTTP boot is 19. This is load-bearing, not cosmetic — AAVMF prefers HTTP Boot, and the VM run shows it sendingHTTPClient:Arch:00019.LoadFile2 is supported — no GRUB2 chainload needed.
EFI_LOAD_FILE2_PROTOCOL_GUID(4006c0c1-fcb3-403e-996d-4a6c8724e06d) is at offset0x3bbf0ofipxe-bootimgs-aarch64-20240119'ssnponly.efi, with the Linux initrd media device path at0x3e530. Confirmed empirically below.Also: Ubuntu publishes no arm64 netboot artifacts, so that combination is refused rather than served an x86 kernel. (Separately and pre-existing: the Ubuntu URL the code already fetches 404s even for amd64, so that path looks dead today. Left alone as out of scope.)
VM evidence
Emulated aarch64 on an isolated libvirt network:
Recorded state:
{"arch": "aarch64", "onboard": "pxe", ...}.LoadFile2 confirmed working: the kernel reached systemd inside the initramfs and went on to run Anaconda. Had iPXE dropped the initrd, that is precisely where it would have panicked with
unknown-block(0,0).x86_64 regression
The four dispatch-served scripts are byte-identical, pinned by a golden fixture dumped from the templates before any change (
tests/fixtures/ipxe-x86_64-golden.json). The single intended diff is&arch=inboot.ipxe, asserted explicitly.Known gap — rescue SSH
Adding the first-ever integration coverage for
provision debugexposed that rescue SSH has never been observed working: the environment boots correctly and Anaconda comes up, but nothing listens on port 22.Evidence, stated at its actual strength because it decides where to look next:
KEEP_VM=1and probe port 22 directly to settle it.If the x86_64 result holds, this is orthogonal to the multi-arch work, since x86_64 is untouched by it. Prime suspect is that
inst.sshddoes not start sshd ininst.rescuemode at all — an auth failure would still leave the port open. Leads documented inARCHITECTURE.md. Thedebug-setup.shnc-listener workaround is unaffected.Consequence:
labctl provision debug spark-2935will now boot a Spark into a rescue environment on its console — which it could not do at all before — but not hand you an SSH shell until this is resolved.Testing
arm-pxe-fullhas not been run yet.KEEP_VM=1leaves the VM up for inspection.Not deployed, and neither Spark has been touched.
The bastion served one x86_64 kernel to every machine regardless of architecture, so an ARM64 box was handed a binary its UEFI will not execute. This is why `labctl provision debug` could not rescue the DGX Sparks during the 2026-08-11 kernel panic. Architecture is resolved, never typed by an operator: the tracked machine record first, then iPXE's ${buildarch} reported on the dispatch URL, then the configured default. boot.ipxe gains &arch=${buildarch} so that signal reaches the HTTP endpoint -- DHCP option 93 only ever reaches dnsmasq. One script covers network PXE, UEFI HTTP boot and the boot ISO alike. x86_64 keeps its unsuffixed /vmlinuz and /initrd.img so its rendered scripts are byte-identical; aarch64 gets suffixed paths, its own Fedora mirror, and console=ttyAMA0 instead of nomodeset, which does not mean the same thing on arm64 and can leave a headless machine with no console. --pxe-boot no longer hardcodes the Fedora LVM layout: root device and dracut args come from the machine's record, and dispatch falls back to rescue rather than guessing a root= that would leave a machine unbootable. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015nRFZXpKwUVE4SRSHw6GjF--pxe-boot boots the installed system with a kernel and initrd from the network, so it needs a root= for that machine. It used to hardcode our Fedora LVM layout, which is wrong for anything else -- including both DGX Sparks. Where the root device comes from, in order of preference: - already recorded on the machine (provision recheck now collects it) - probed over SSH when --pxe-boot is requested and the machine answers - reported from the rescue shell by debug-setup.sh, which mounts each candidate read-only and picks the one with /etc/fstab and /usr The rescue image cannot report it unprompted -- %pre/%post do not run in rescue mode -- so the probe lives in the script the operator curls, which already existed for the nc listener. /api/discover now preserves fields a report omits. The probe posts only a root device, and blanking a machine's inventory as a side effect of that would be silent data loss. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015nRFZXpKwUVE4SRSHw6GjFView command line instructions
Checkout
From your project repository, check out a new branch and test the changes.