test(bastion): cover the rescue boot path and record what it exposed

`provision debug` is the lab's recovery tool of last resort and had no
integration coverage on any architecture. Adding it -- x86_64 on KVM so
it runs in ~15 minutes, plus the aarch64 equivalent -- showed the rescue
environment coming up correctly but nothing ever listening on port 22.

Reproduced on both architectures, so it is neither ARM-specific nor an
emulation artefact, and it is orthogonal to the multi-arch work: x86_64
is unchanged by that. Documented in ARCHITECTURE.md with the leads worth
checking, rather than left as a silent gap.

Also restructures the ARM rescue suite to seed the machine into state
instead of discovering it first. That mirrors the DGX Spark situation --
SSH-onboarded, never PXE-discovered, architecture known only from its
record -- and holds the test to one emulated boot, since each spends
~15 of its ~18 minutes fetching Anaconda's stage2 under TCG.

KEEP_VM=1 leaves the VM up on failure; half-hour emulated runs are too
expensive to pay twice just to see what happened.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015nRFZXpKwUVE4SRSHw6GjF

bastion/docs/ARCHITECTURE.md

@@ -89,6 +89,83 @@ Side paths:
 
 ---
 
+## Multi-architecture PXE
+
+The bastion serves both `x86_64` and `aarch64` over the network. Nothing about this is
+operator-configured -- there is no `--arch` flag, by design.
+
+### How a client's architecture is decided
+
+1. **DHCP option 93** (Client System Architecture) picks the *bootloader*. dnsmasq matches
+   it and hands out a matching iPXE binary:
+
+   | Option 93 | Client | Served |
+   |---|---|---|
+   | `0` | x86 BIOS | `undionly.kpxe` (TFTP) |
+   | `7`, `9` | x64 UEFI | `ipxe.efi` (TFTP) |
+   | `11` | **ARM64 UEFI** | `ipxe-arm64.efi` (TFTP) |
+   | `16` | x64 UEFI HTTP Boot | `http://…/ipxe.efi` |
+   | `19` | **ARM64 UEFI HTTP Boot** | `http://…/ipxe-arm64.efi` |
+
+   Values come from the IANA Processor Architecture Types registry. Note `19`, not `20` --
+   `20` is *pc/at bios boot from http*. EDK2/AAVMF prefers HTTP Boot over TFTP PXE, so the
+   iPXE binaries are staged in **both** `tftpDir` and `httpDir` (symlinked by `main.ts`).
+
+2. **`/dispatch` picks the kernel.** Option 93 never reaches the HTTP endpoint, so
+   `boot.ipxe` passes iPXE's own `${buildarch}` as `?arch=`. `resolveArch()` prefers, in
+   order: the tracked machine record → the reported `?arch=` → the configured default.
+   The record wins because it is what we observed on the machine itself.
+
+### Artifact naming
+
+`x86_64` keeps the original unsuffixed paths so its rendered iPXE scripts are unchanged;
+everything else is suffixed. `kernelPath()` / `initrdPath()` in `templates/boot.ipxe.ts`
+are the single source of truth, used by both the templates and `main.ts` staging.
+
+| arch | kernel | initrd |
+|---|---|---|
+| `x86_64` | `/vmlinuz` | `/initrd.img` |
+| `aarch64` | `/vmlinuz-aarch64` | `/initrd-aarch64.img` |
+
+`tests/ipxe-x86-regression.test.ts` pins the x86_64 output against a golden fixture.
+
+### arm64 gotchas
+
+- **LoadFile2 is mandatory.** arm64 has no `HdrS` boot protocol; the kernel's EFI stub
+  fetches the initrd over the UEFI `EFI_LOAD_FILE2_PROTOCOL`. An iPXE build without it
+  accepts the `initrd` line, silently drops it, and the kernel panics with
+  `VFS: Unable to mount root fs on unknown-block(0,0)`. Fedora's
+  `ipxe-bootimgs-aarch64` implements it; the integration test asserts this up front so
+  the failure names itself instead of looking like a disk problem.
+- **`nomodeset` is x86-only.** On arm64 there is no VGA path to fall back to. aarch64 gets
+  `console=tty0 console=ttyAMA0,115200` instead — the last `console=` wins for
+  `/dev/console`, so serial is the interactive one.
+- **Ubuntu is x86_64-only.** `releases.ubuntu.com` publishes no arm64 netboot artifacts.
+  `osSupportsArch()` encodes this, and both the install guard and `/dispatch` refuse the
+  combination rather than serving an x86 kernel to an ARM machine.
+
+---
+
+## Onboarding classification (vendor OS)
+
+Machines carry an `onboard` field: `"pxe"` (default) or `"ssh"`, plus `vendor_os` naming
+what they run. `classifyOnboard()` in `@lab/shared` sets it from DMI identity, with known
+hardware also matched by MAC — a machine can sit in state for a long time with no DMI, and
+a DMI-only rule would fail open exactly where it matters.
+
+`onboard: "ssh"` means *we cannot rebuild this machine's OS*. Installs are refused at both
+entry points (`/api/install` and the labd `command-install` handler) with an error naming
+the machine and pointing at `provision debug`. **Rescue is never guarded** — being unable
+to reinstall a machine is precisely when a rescue shell is needed.
+
+This is a fact about the machine, not a blocklist. The refusal follows from "no image in
+our pipeline restores `vendor_os`", so adding a DGX OS image to the pipeline is what
+unblocks the DGX Sparks — no entry needs deleting.
+
+Current classifications: NVIDIA DGX Spark (`spark-2935`, `spark-3a1c`) → `dgx-os`.
+
+---
+
 ## Packages
 
 ### Monorepo Structure
@@ -404,6 +481,28 @@ Hardcoded `/dev/sda` default broke NVMe-only machines. Fix: default to empty str
 ### Anaconda Rescue Mode Limitations
 `%pre` and `%post` sections do not execute in `inst.rescue` mode. SSH in rescue mode is provided by Anaconda's `inst.sshd` kernel parameter + `sshpw` kickstart directive. Manual setup via `curl bastion:8080/debug-setup.sh | bash` for nc listener.
 
+**Unresolved (2026-08-11): rescue SSH has never been observed working.** Adding the first
+integration coverage for `provision debug` (`tests/integration/pxe-rescue.test.ts`) showed the
+rescue environment coming up correctly — the bastion serves the kernel and initrd, Anaconda
+boots, fetches `debug.ks`, and reaches its installer environment — but **nothing ever listens on
+port 22**. Confirmed on aarch64 by probing the port for 30 minutes while the Anaconda environment
+was demonstrably running (NetworkManager, polkitd, rsyslog all up), and reproduced on x86_64 with
+KVM, so it is not architecture-specific and not an emulation artefact.
+
+This is orthogonal to the multi-architecture work: the same failure occurs on x86_64, which that
+work does not touch. Leads worth checking, in order:
+- Does `inst.sshd` actually start `sshd` in `inst.rescue` mode, or only in install mode? The
+  port never opens, so this is the prime suspect — an auth problem would still show an open port.
+- `sshkey` may apply only to the *installed* system, leaving the installer environment
+  password-only via `sshpw`. That would matter once sshd does listen: the test authenticates
+  key-only (`BatchMode=yes`).
+- The `%anaconda`-context directives in `debug.ks` may be skipped entirely when a kickstart is
+  supplied alongside `inst.rescue`.
+
+Until this is resolved, `provision debug` gets you a booted rescue environment on the console
+(including on arm64), but not an SSH shell. The `debug-setup.sh` nc-listener path is the
+documented workaround and is unaffected.
+
 ---
 
 ## Planned Work (Taskmaster)

bastion/package.json

@@ -23,6 +23,12 @@
     "test:integration:iso:host": "sudo -E $(which npx) vitest run -c tests/integration/vitest.config.ts -t 'ISO boot'",
     "test:integration:arm-iso": "vitest run -c tests/integration/vitest.config.ts -t 'ARM ISO'",
     "test:integration:arm-iso:host": "sudo -E $(which npx) vitest run -c tests/integration/vitest.config.ts -t 'ARM ISO'",
+    "test:integration:rescue": "vitest run -c tests/integration/vitest.config.ts -t 'x86 rescue boot'",
+    "test:integration:rescue:host": "sudo -E $(which npx) vitest run -c tests/integration/vitest.config.ts -t 'x86 rescue boot'",
+    "test:integration:arm-pxe": "vitest run -c tests/integration/vitest.config.ts -t 'ARM PXE rescue'",
+    "test:integration:arm-pxe:host": "sudo -E $(which npx) vitest run -c tests/integration/vitest.config.ts -t 'ARM PXE rescue'",
+    "test:integration:arm-pxe-full": "ARM_PXE_FULL=1 vitest run -c tests/integration/vitest.config.ts -t 'ARM PXE'",
+    "test:integration:arm-pxe-full:host": "sudo -E ARM_PXE_FULL=1 $(which npx) vitest run -c tests/integration/vitest.config.ts -t 'ARM PXE'",
     "test:integration:asahi": "vitest run -c tests/integration/vitest.config.ts -t 'asahi firstboot'",
     "test:integration:asahi:host": "sudo -E $(which npx) vitest run -c tests/integration/vitest.config.ts -t 'asahi firstboot'",
     "test:integration:asahi-validate": "vitest run -c tests/integration/vitest.config.ts -t 'asahi.*validation'",

bastion/scripts/test-provision.sh

@@ -2,16 +2,19 @@
 # Run PXE and/or ISO boot integration tests.
 #
 # Usage:
-#   sudo ./scripts/test-provision.sh          # run PXE + ISO (x86_64)
+#   sudo ./scripts/test-provision.sh              # run PXE + ISO (x86_64)
-#   sudo ./scripts/test-provision.sh pxe      # PXE only
+#   sudo ./scripts/test-provision.sh pxe          # PXE only
-#   sudo ./scripts/test-provision.sh iso      # ISO only (x86_64)
+#   sudo ./scripts/test-provision.sh iso          # ISO only (x86_64)
-#   sudo ./scripts/test-provision.sh arm      # ARM ISO boot (emulated, SLOW ~60min)
+#   sudo ./scripts/test-provision.sh rescue       # x86_64 Anaconda rescue boot + SSH (~15min)
-#   sudo ./scripts/test-provision.sh all      # all tests including ARM
+#   sudo ./scripts/test-provision.sh arm          # ARM ISO boot (emulated, SLOW ~60min)
+#   sudo ./scripts/test-provision.sh arm-pxe      # ARM network PXE rescue: NBP + rescue over SSH (~25-30min)
+#   sudo ./scripts/test-provision.sh arm-pxe-full # ARM network PXE incl. discover + full install (~75-95min)
+#   sudo ./scripts/test-provision.sh all          # all tests including ARM
 #
 # Prerequisites:
 #   libvirtd, OVMF (edk2-ovmf), iPXE (ipxe-bootimgs-x86),
 #   dnsmasq, xorriso, mtools, virt-install, qemu-img
-#   ARM: qemu-system-aarch64, edk2-aarch64
+#   ARM: qemu-system-aarch64, edk2-aarch64, ipxe-bootimgs-aarch64
 set -e
 
 SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
@@ -58,6 +61,10 @@ if [ ! -f /usr/share/edk2/ovmf/OVMF_CODE.fd ]; then
     exit 1
 fi
 
+MODE="${1:-both}"
+
+# iPXE binaries are per-architecture. x86_64 is always required (the dnsmasq config
+# references it); arm64 only for the ARM network-PXE modes.
 IPXE_EFI=""
 for f in /usr/share/ipxe/ipxe-snponly-x86_64.efi /usr/share/ipxe/ipxe-snp-x86_64.efi /usr/share/ipxe/ipxe-x86_64.efi; do
     [ -f "$f" ] && IPXE_EFI="$f" && break
@@ -67,6 +74,20 @@ if [ -z "$IPXE_EFI" ]; then
     exit 1
 fi
 
+IPXE_EFI_ARM64=""
+for f in /usr/share/ipxe/arm64-efi/snponly.efi /usr/share/ipxe/arm64-efi/ipxe.efi; do
+    [ -f "$f" ] && IPXE_EFI_ARM64="$f" && break
+done
+
+case "$MODE" in
+    arm-pxe|arm-pxe-full|all)
+        if [ -z "$IPXE_EFI_ARM64" ] && [ "$MODE" != "all" ]; then
+            echo -e "${RED}arm64 iPXE binary not found.${RESET} Install: sudo dnf install ipxe-bootimgs-aarch64"
+            exit 1
+        fi
+        ;;
+esac
+
 # Find SSH key
 SSH_KEY=""
 for name in id_ed25519 id_ecdsa id_rsa; do
@@ -83,10 +104,19 @@ fi
 echo -e "  User:    ${BOLD}$REAL_USER${RESET}"
 echo -e "  SSH key: ${BOLD}$SSH_KEY${RESET}"
 echo -e "  iPXE:    ${BOLD}$IPXE_EFI${RESET}"
+echo -e "  iPXE a64:${BOLD} ${IPXE_EFI_ARM64:-not installed}${RESET}"
 echo ""
 
-# --- Determine which tests to run ---
+require_arm_emulation() {
-MODE="${1:-both}"
+    if ! command -v qemu-system-aarch64 &>/dev/null; then
+        echo -e "${RED}qemu-system-aarch64 not found.${RESET} Install: sudo dnf install qemu-system-aarch64 edk2-aarch64"
+        exit 1
+    fi
+    if [ ! -f /usr/share/edk2/aarch64/QEMU_EFI.fd ]; then
+        echo -e "${RED}AAVMF firmware not found.${RESET} Install: sudo dnf install edk2-aarch64"
+        exit 1
+    fi
+}
 
 run_test() {
     local name="$1" pattern="$2"
@@ -116,13 +146,26 @@ case "$MODE" in
         run_test "ISO boot" "ISO boot" || FAILED=1
         ;;
     arm|arm-iso)
-        if ! command -v qemu-system-aarch64 &>/dev/null; then
+        require_arm_emulation
-            echo -e "${RED}qemu-system-aarch64 not found.${RESET} Install: sudo dnf install qemu-system-aarch64 edk2-aarch64"
-            exit 1
-        fi
         echo -e "${YELLOW}ARM emulation is ~10x slower than native. Expect 30-60 minutes.${RESET}"
         run_test "ARM ISO boot" "ARM ISO" || FAILED=1
         ;;
+    rescue)
+        echo -e "${YELLOW}x86_64 rescue boot (KVM). Expect ~15 minutes.${RESET}"
+        run_test "x86 rescue boot" "x86 rescue boot" || FAILED=1
+        ;;
+    arm-pxe)
+        require_arm_emulation
+        echo -e "${YELLOW}ARM emulation is ~10x slower than native. Expect 25-30 minutes.${RESET}"
+        echo -e "${YELLOW}Covers option 93 -> arm64 NBP, arch resolution, and rescue over SSH.${RESET}"
+        echo -e "${YELLOW}For the full install too, use: $0 arm-pxe-full${RESET}"
+        run_test "ARM PXE rescue" "ARM PXE rescue" || FAILED=1
+        ;;
+    arm-pxe-full)
+        require_arm_emulation
+        echo -e "${YELLOW}ARM emulation is ~10x slower than native. Expect 75-95 minutes.${RESET}"
+        ARM_PXE_FULL=1 run_test "ARM PXE (rescue + install)" "ARM PXE" || FAILED=1
+        ;;
     both)
         run_test "PXE boot" "PXE boot" || FAILED=1
         run_test "ISO boot" "ISO boot" || FAILED=1
@@ -133,12 +176,17 @@ case "$MODE" in
         if command -v qemu-system-aarch64 &>/dev/null; then
             echo -e "${YELLOW}ARM emulation is ~10x slower than native.${RESET}"
             run_test "ARM ISO boot" "ARM ISO" || FAILED=1
+            if [ -n "$IPXE_EFI_ARM64" ]; then
+                run_test "ARM PXE rescue" "ARM PXE rescue" || FAILED=1
+            else
+                echo -e "${YELLOW}Skipping ARM PXE test (ipxe-bootimgs-aarch64 not installed)${RESET}"
+            fi
         else
-            echo -e "${YELLOW}Skipping ARM test (qemu-system-aarch64 not installed)${RESET}"
+            echo -e "${YELLOW}Skipping ARM tests (qemu-system-aarch64 not installed)${RESET}"
         fi
         ;;
     *)
-        echo "Usage: $0 [pxe|iso|arm|both|all]"
+        echo "Usage: $0 [pxe|iso|rescue|arm|arm-pxe|arm-pxe-full|both|all]"
         exit 1
         ;;
 esac

bastion/src/bastion/src/main.ts

@@ -3,7 +3,9 @@
 
 import { mkdirSync, writeFileSync, readFileSync, existsSync, copyFileSync, symlinkSync, unlinkSync } from "node:fs";
 import { execSync } from "node:child_process";
-import type { BastionConfig } from "@lab/shared";
+import type { Arch, BastionConfig } from "@lab/shared";
+import { SUPPORTED_ARCHES, fedoraMirrorFor, classifyOnboard } from "@lab/shared";
+import { kernelPath, initrdPath } from "./templates/boot.ipxe.js";
 import { loadConfig } from "./config.js";
 import { populateNetworkConfig } from "./services/network.js";
 import { createApp } from "./server.js";
@@ -13,6 +15,7 @@ import { renderBootIpxe } from "./templates/boot.ipxe.js";
 import { logger } from "./services/logger.js";
 import { BastionConnection } from "./services/labd-connection.js";
 import { progressBus } from "./services/progress-events.js";
+import { checkInstallAllowed } from "./services/install-guard.js";
 import { ensureBootIso } from "./routes/boot-iso.js";
 
 function copyIfMissing(src: string, dest: string, label: string): void {
@@ -130,9 +133,14 @@ export async function startBastion(overrides: Partial<BastionConfig> = {}): Prom
   mkdirSync(config.tftpDir, { recursive: true });
   mkdirSync(config.httpDir, { recursive: true });
 
+  // Architectures we can actually network boot, reported in the banner so a missing
+  // arm64 payload is visible at startup instead of at 2am when a rescue is needed.
+  const bootArches: Arch[] = [];
+  let ipxeArm64Ready = false;
+
   // Prepare boot artifacts
   if (config.skipArtifacts !== true) {
-    logger.info(`Preparing boot artifacts (Fedora ${config.fedoraVersion} ${config.arch})...`);
+    logger.info(`Preparing boot artifacts (Fedora ${config.fedoraVersion}, ${SUPPORTED_ARCHES.join(" + ")})...`);
 
     copyIfMissing(
       "/usr/share/ipxe/undionly.kpxe",
@@ -150,20 +158,41 @@ export async function startBastion(overrides: Partial<BastionConfig> = {}): Prom
         `${config.tftpDir}/ipxe-arm64.efi`,
         "iPXE UEFI arm64",
       );
+      ipxeArm64Ready = true;
     } catch {
-      logger.warn("arm64 iPXE not available -- skipping");
+      logger.warn("arm64 iPXE not available -- arm64 machines cannot network boot.");
+      logger.warn("  Install with: sudo dnf install ipxe-bootimgs-aarch64");
     }
 
-    download(
+    // Fedora pxeboot kernel + initrd per architecture. x86_64 keeps the unsuffixed
-      `${config.fedoraMirror}/images/pxeboot/vmlinuz`,
+    // names it has always used; other architectures are suffixed. The iPXE templates
-      `${config.httpDir}/vmlinuz`,
+    // resolve the same paths via kernelPath()/initrdPath().
-      "Fedora kernel",
+    for (const arch of SUPPORTED_ARCHES) {
-    );
+      const mirror = fedoraMirrorFor(config.fedoraVersion, arch);
-    download(
+      try {
-      `${config.fedoraMirror}/images/pxeboot/initrd.img`,
+        download(
-      `${config.httpDir}/initrd.img`,
+          `${mirror}/images/pxeboot/vmlinuz`,
-      "Fedora initrd",
+          `${config.httpDir}${kernelPath(arch)}`,
-    );
+          `Fedora ${arch} kernel`,
+        );
+        download(
+          `${mirror}/images/pxeboot/initrd.img`,
+          `${config.httpDir}${initrdPath(arch)}`,
+          `Fedora ${arch} initrd`,
+        );
+        bootArches.push(arch);
+      } catch (err) {
+        // Non-fatal: a bastion with no arm64 artifacts still serves x86_64 fine.
+        // Failing startup over an unreachable mirror for an architecture that may not
+        // even be present on this network would be worse.
+        logger.warn(`Fedora ${arch} kernel/initrd unavailable -- ${arch} PXE disabled`);
+        logger.warn(`  ${err instanceof Error ? err.message : String(err)}`);
+      }
+    }
+
+    if (!bootArches.includes("x86_64")) {
+      throw new Error("Fedora x86_64 kernel/initrd could not be staged -- cannot serve PXE");
+    }
 
     // Ubuntu netboot artifacts (non-fatal — Ubuntu version may not be released yet)
     try {
@@ -254,6 +283,13 @@ export async function startBastion(overrides: Partial<BastionConfig> = {}): Prom
     // Wire up command handlers so labd can send install/forget/role commands
     labdConn.onCommand("command-install", async (msg) => {
       if (msg.type !== "command-install") throw new Error("unexpected");
+      const installMac = msg.mac.toLowerCase().replace(/-/g, ":");
+      const osId = (msg.os as import("@lab/shared").OsId | undefined) ?? "fedora-43";
+      const check = checkInstallAllowed(state.load(), installMac, osId);
+      if (check.allowed === false) {
+        logger.warn(`INSTALL REFUSED: ${installMac} -- ${check.error}`);
+        return { status: "error", error: check.error };
+      }
       state.update((s) => {
         s.install_queue[msg.mac] = {
           hostname: msg.hostname,
@@ -314,13 +350,24 @@ export async function startBastion(overrides: Partial<BastionConfig> = {}): Prom
       const mac = (msg.mac as string).toLowerCase();
       const now = new Date().toISOString();
       const existing = state.load().discovered[mac];
+      const identity = {
+        mac,
+        manufacturer: (msg.manufacturer as string) ?? "unknown",
+        product: (msg.product as string) ?? "unknown",
+        board: (msg.board as string) ?? "unknown",
+        ...(existing?.onboard !== undefined ? { onboard: existing.onboard } : {}),
+        ...(existing?.vendor_os !== undefined ? { vendor_os: existing.vendor_os } : {}),
+      };
+      const onboarding = classifyOnboard(identity);
+      const rootDevice = msg.root_device ?? existing?.root_device;
+      const rootArgs = msg.root_args ?? existing?.root_args;
       state.update((s) => {
         s.discovered[mac] = {
           mac,
-          product: (msg.product as string) ?? "unknown",
+          product: identity.product,
-          board: (msg.board as string) ?? "unknown",
+          board: identity.board,
           serial: (msg.serial as string) ?? "unknown",
-          manufacturer: (msg.manufacturer as string) ?? "unknown",
+          manufacturer: identity.manufacturer,
           cpu_model: (msg.cpu_model as string) ?? "unknown",
           cpu_cores: (msg.cpu_cores as number) ?? 0,
           memory_gb: (msg.memory_gb as number) ?? 0,
@@ -329,7 +376,20 @@ export async function startBastion(overrides: Partial<BastionConfig> = {}): Prom
           nics: (msg.nics as Array<{ name: string; mac: string; state: string }>) ?? [],
           first_seen: existing?.first_seen ?? now,
           last_seen: now,
+          onboard: onboarding.onboard,
+          ...(onboarding.vendor_os !== undefined ? { vendor_os: onboarding.vendor_os } : {}),
+          ...(rootDevice !== undefined ? { root_device: rootDevice } : {}),
+          ...(rootArgs !== undefined ? { root_args: rootArgs } : {}),
         };
+        // Keep the installed record in step -- the guard and --pxe-boot both read it.
+        const inst = s.installed[mac];
+        if (inst) {
+          inst.arch = (msg.arch as string) ?? inst.arch;
+          inst.onboard = onboarding.onboard;
+          if (onboarding.vendor_os !== undefined) inst.vendor_os = onboarding.vendor_os;
+          if (rootDevice !== undefined) inst.root_device = rootDevice;
+          if (rootArgs !== undefined) inst.root_args = rootArgs;
+        }
       });
       logger.info(`HARDWARE UPDATED: ${mac} -- ${msg.manufacturer ?? "?"} ${msg.product ?? "?"} (${msg.cpu_model ?? "?"}, ${msg.cpu_cores ?? "?"} cores, ${msg.memory_gb ?? "?"}GB RAM)`);
       return { status: "ok", data: { mac } };
@@ -364,7 +424,7 @@ export async function startBastion(overrides: Partial<BastionConfig> = {}): Prom
   }
 
   // Print banner
-  printBanner(config);
+  printBanner(config, bootArches, ipxeArm64Ready);
 
   // Graceful shutdown
   const shutdown = async (): Promise<void> => {
@@ -386,11 +446,22 @@ export async function startBastion(overrides: Partial<BastionConfig> = {}): Prom
   await new Promise(() => {});
 }
 
-function printBanner(config: BastionConfig): void {
+function printBanner(config: BastionConfig, bootArches: Arch[], ipxeArm64Ready: boolean): void {
   const dhcpInfo = config.dhcpMode === "full"
     ? `full (${config.dhcpRangeStart}-${config.dhcpRangeEnd})`
     : "proxy (alongside existing DHCP)";
 
+  // arm64 needs both an iPXE binary (DHCP hands it out on option 93 = 0x0b) and a
+  // kernel/initrd pair. Report the combination, since either missing breaks it.
+  const archInfo = config.skipArtifacts === true
+    ? "(artifacts skipped)"
+    : SUPPORTED_ARCHES
+        .map((a) => {
+          const ready = bootArches.includes(a) && (a !== "aarch64" || ipxeArm64Ready);
+          return ready ? a : `${a} (unavailable)`;
+        })
+        .join(", ");
+
   console.log("");
   console.log("\x1b[36m\x1b[1m" + "=".repeat(60) + "\x1b[0m");
   console.log("\x1b[36m\x1b[1m  Lab PXE Bastion -- Discovery Mode\x1b[0m");
@@ -399,7 +470,8 @@ function printBanner(config: BastionConfig): void {
   console.log(`  Network:   \x1b[1m${config.network}/24\x1b[0m via \x1b[1m${config.iface}\x1b[0m`);
   console.log(`  DHCP:      \x1b[1m${dhcpInfo}\x1b[0m`);
   console.log(`  HTTP:      \x1b[1mhttp://${config.serverIp}:${config.httpPort}/\x1b[0m`);
-  console.log(`  OS:        \x1b[1mFedora ${config.fedoraVersion} (${config.arch})\x1b[0m`);
+  console.log(`  OS:        \x1b[1mFedora ${config.fedoraVersion}\x1b[0m`);
+  console.log(`  Net boot:  \x1b[1m${archInfo}\x1b[0m`);
   console.log(`  Domain:    \x1b[1m${config.domain}\x1b[0m`);
   console.log(`  State:     \x1b[1m${config.stateFile}\x1b[0m`);
   console.log("");

bastion/src/bastion/src/routes/api.ts

@@ -6,10 +6,11 @@
 
 import type { FastifyInstance } from "fastify";
 import type { HardwareInfo, InstalledInfo, Role } from "@lab/shared";
-import { isValidOsId, SUPPORTED_ROLES } from "@lab/shared";
+import { isValidOsId, SUPPORTED_ROLES, classifyOnboard } from "@lab/shared";
 import type { StateManager } from "../services/state.js";
 import { logger } from "../services/logger.js";
 import { triggerPostProvisionK3s } from "../services/post-provision.js";
+import { checkInstallAllowed } from "../services/install-guard.js";
 import { progressBus } from "../services/progress-events.js";
 import type { ProgressEvent } from "../services/progress-events.js";
 import type { InstallLogBuffer } from "../services/install-log.js";
@@ -53,6 +54,12 @@ export function registerApiRoutes(
       return reply.status(400).send({ error: `invalid os: '${osId}'. Supported: fedora-43, ubuntu-26.04` });
     }
 
+    const check = checkInstallAllowed(state.load(), mac, osId);
+    if (check.allowed === false) {
+      logger.warn(`INSTALL REFUSED: ${mac} -- ${check.error}`);
+      return reply.status(409).send({ error: check.error });
+    }
+
     state.update((s) => {
       s.install_queue[mac] = {
         hostname: hostname ?? "lab-node",
@@ -284,6 +291,10 @@ export function registerApiRoutes(
       arch?: string;
       disks?: Array<{ name: string; size_gb: number; model: string }>;
       nics?: Array<{ name: string; mac: string; state: string }>;
+      // Root filesystem, when the reporter could observe it (recheck over SSH, or the
+      // probe script run from a rescue shell). Used by --pxe-boot.
+      root_device?: string;
+      root_args?: string;
     };
   }>("/api/discover", async (request, reply) => {
     const data = request.body;
@@ -298,22 +309,53 @@ export function registerApiRoutes(
 
     state.update((s) => {
       const existing = s.discovered[mac];
+      // Classify onboarding from the DMI identity we just received. An explicit
+      // classification already on the record wins (see classifyOnboard).
+      const onboarding = classifyOnboard({
+        mac,
+        manufacturer: data.manufacturer ?? existing?.manufacturer ?? "unknown",
+        product: data.product ?? existing?.product ?? "unknown",
+        board: data.board ?? existing?.board ?? "unknown",
+        ...(existing?.onboard !== undefined ? { onboard: existing.onboard } : {}),
+        ...(existing?.vendor_os !== undefined ? { vendor_os: existing.vendor_os } : {}),
+      });
+      const rootDevice = data.root_device ?? existing?.root_device;
+      const rootArgs = data.root_args ?? existing?.root_args;
+
+      // Absent fields keep whatever we already knew. Reporters are not all the full
+      // discovery kickstart: the rescue-shell probe posts only a root device, and
+      // blanking a machine's hardware inventory as a side effect of that would be
+      // silent data loss.
       const hwInfo: HardwareInfo = {
         mac,
-        product: data.product ?? "unknown",
+        product: data.product ?? existing?.product ?? "unknown",
-        board: data.board ?? "unknown",
+        board: data.board ?? existing?.board ?? "unknown",
-        serial: data.serial ?? "unknown",
+        serial: data.serial ?? existing?.serial ?? "unknown",
-        manufacturer: data.manufacturer ?? "unknown",
+        manufacturer: data.manufacturer ?? existing?.manufacturer ?? "unknown",
-        cpu_model: data.cpu_model ?? "unknown",
+        cpu_model: data.cpu_model ?? existing?.cpu_model ?? "unknown",
-        cpu_cores: data.cpu_cores ?? 0,
+        cpu_cores: data.cpu_cores ?? existing?.cpu_cores ?? 0,
-        memory_gb: data.memory_gb ?? 0,
+        memory_gb: data.memory_gb ?? existing?.memory_gb ?? 0,
-        arch: data.arch ?? "unknown",
+        arch: data.arch ?? existing?.arch ?? "unknown",
-        disks: data.disks ?? [],
+        disks: data.disks ?? existing?.disks ?? [],
-        nics: data.nics ?? [],
+        nics: data.nics ?? existing?.nics ?? [],
         first_seen: existing?.first_seen ?? now,
         last_seen: now,
+        onboard: onboarding.onboard,
+        ...(onboarding.vendor_os !== undefined ? { vendor_os: onboarding.vendor_os } : {}),
+        ...(rootDevice !== undefined ? { root_device: rootDevice } : {}),
+        ...(rootArgs !== undefined ? { root_args: rootArgs } : {}),
       };
       s.discovered[mac] = hwInfo;
+
+      // Keep the installed record in step -- the install guard and --pxe-boot read it.
+      const inst = s.installed[mac];
+      if (inst) {
+        if (data.arch !== undefined) inst.arch = data.arch;
+        inst.onboard = onboarding.onboard;
+        if (onboarding.vendor_os !== undefined) inst.vendor_os = onboarding.vendor_os;
+        if (rootDevice !== undefined) inst.root_device = rootDevice;
+        if (rootArgs !== undefined) inst.root_args = rootArgs;
+      }
     });
 
     const label = isNew ? "NEW MACHINE DISCOVERED" : "MACHINE RE-DISCOVERED";

bastion/src/bastion/src/routes/dispatch.ts

@@ -5,7 +5,8 @@
 //   - unknown       -> discovery mode (collect hardware, POST to bastion)
 
 import type { FastifyInstance } from "fastify";
-import type { BastionConfig } from "@lab/shared";
+import type { Arch, BastionConfig, BastionState, OsId } from "@lab/shared";
+import { normalizeArch, fedoraMirrorFor, osSupportsArch } from "@lab/shared";
 import type { StateManager } from "../services/state.js";
 import {
   renderDiscoverIpxe,
@@ -13,11 +14,51 @@ import {
   renderDebugIpxe,
   renderPxeBootDebugIpxe,
   renderLocalBootIpxe,
+  renderUnsupportedIpxe,
 } from "../templates/boot.ipxe.js";
 import { renderUbuntuInstallIpxe } from "../templates/ubuntu-boot.ipxe.js";
 import { renderDebugKickstart } from "../templates/debug.ks.js";
 import { logger } from "../services/logger.js";
 
+/**
+ * Resolve a booting machine's architecture.
+ *
+ * Order matters. The tracked record is what we actually observed on the machine, so it
+ * wins. `reported` is iPXE's ${buildarch}, which is only as good as the binary DHCP
+ * handed the client -- correct in practice, but a misconfigured option 93 mapping would
+ * make it lie. The configured default is the last resort.
+ *
+ * There is deliberately no operator-supplied architecture anywhere in this path.
+ */
+export function resolveArch(
+  state: BastionState,
+  mac: string,
+  reported: string | undefined,
+  config: BastionConfig,
+): Arch {
+  return normalizeArch(state.installed[mac]?.arch)
+    ?? normalizeArch(state.install_queue[mac]?.arch)
+    ?? normalizeArch(state.discovered[mac]?.arch)
+    ?? normalizeArch(reported)
+    ?? normalizeArch(config.arch)
+    ?? "x86_64";
+}
+
+/** The root filesystem to boot for --pxe-boot, if the machine's record carries one. */
+function resolveRoot(
+  state: BastionState,
+  mac: string,
+): { rootDevice: string; rootArgs?: string } | null {
+  const installed = state.installed[mac];
+  const discovered = state.discovered[mac];
+  const rootDevice = installed?.root_device ?? discovered?.root_device;
+  if (rootDevice === undefined || rootDevice === "") return null;
+  const rootArgs = installed?.root_args ?? discovered?.root_args;
+  return rootArgs !== undefined && rootArgs !== ""
+    ? { rootDevice, rootArgs }
+    : { rootDevice };
+}
+
 export function registerDispatchRoutes(
   app: FastifyInstance,
   config: BastionConfig,
@@ -52,18 +93,68 @@ curl -sf -X POST "http://${config.serverIp}:${config.httpPort}/api/progress" \\
     -H "Content-Type: application/json" \\
     -d "{\\"mac\\":\\"$MAC_ADDR\\",\\"stage\\":\\"debug-ready\\",\\"detail\\":\\"nc $IP_ADDR 2323\\"}" 2>/dev/null || true
 
+# --- Find the installed root filesystem and report it ---
+# This is what 'labctl provision debug --pxe-boot' needs. The rescue image cannot
+# report it by itself: %pre/%post do not run in rescue mode, so it happens here.
+vgchange -ay >/dev/null 2>&1 || true
+
+ROOT_DEVICE=""
+ROOT_ARGS=""
+PROBE_MNT=/tmp/lab-rootprobe
+mkdir -p "$PROBE_MNT"
+
+# Candidates: every LVM logical volume plus every non-LVM partition with a filesystem.
+for CAND in $(lvs --noheadings -o lv_path 2>/dev/null) \\
+            $(blkid -o device 2>/dev/null | grep -v '^/dev/mapper/'); do
+    [ -b "$CAND" ] || continue
+    mount -o ro "$CAND" "$PROBE_MNT" >/dev/null 2>&1 || continue
+    # A root filesystem has both of these; /boot and /home do not.
+    if [ -f "$PROBE_MNT/etc/fstab" ] && [ -d "$PROBE_MNT/usr" ]; then
+        ROOT_DEVICE="$CAND"
+        PRETTY=$(. "$PROBE_MNT/etc/os-release" 2>/dev/null && echo "$PRETTY_NAME")
+        echo "  found root: $CAND \${PRETTY:+($PRETTY)}"
+        if [ "$(lsblk -no TYPE "$CAND" 2>/dev/null | head -1)" = "lvm" ]; then
+            VGLV=$(lvs --noheadings -o vg_name,lv_name "$CAND" 2>/dev/null | awk '{print $1"/"$2}')
+            [ -n "$VGLV" ] && ROOT_ARGS="rd.lvm.lv=$VGLV"
+            # Swap comes from fstab here — /proc/swaps is the rescue image's, not the host's.
+            SWLV=$(awk '$3=="swap" && $1 ~ /^\\/dev\\// {print $1; exit}' "$PROBE_MNT/etc/fstab" 2>/dev/null)
+            if [ -n "$SWLV" ]; then
+                SWVGLV=$(lvs --noheadings -o vg_name,lv_name "$SWLV" 2>/dev/null | awk '{print $1"/"$2}')
+                [ -n "$SWVGLV" ] && [ "$SWVGLV" != "$VGLV" ] && ROOT_ARGS="$ROOT_ARGS rd.lvm.lv=$SWVGLV"
+            fi
+        fi
+        umount "$PROBE_MNT" >/dev/null 2>&1 || true
+        break
+    fi
+    umount "$PROBE_MNT" >/dev/null 2>&1 || true
+done
+
+if [ -n "$ROOT_DEVICE" ]; then
+    curl -sf -X POST "http://${config.serverIp}:${config.httpPort}/api/discover" \\
+        -H "Content-Type: application/json" \\
+        -d "{\\"mac\\":\\"$MAC_ADDR\\",\\"root_device\\":\\"$ROOT_DEVICE\\",\\"root_args\\":\\"$ROOT_ARGS\\"}" 2>/dev/null \\
+        && echo "  reported to bastion — 'labctl provision debug --pxe-boot' will work now"
+else
+    echo "  no root filesystem found — --pxe-boot cannot be used on this machine"
+fi
+
 echo ""
 echo "=== Debug environment ready ==="
 echo "  nc $IP_ADDR 2323    (remote shell)"
 echo "  ssh root@$IP_ADDR   (password: debug)"
+if [ -n "$ROOT_DEVICE" ]; then
+    echo "  root: $ROOT_DEVICE $ROOT_ARGS"
+fi
 echo "==============================="
 `;
     return reply.type("text/plain").send(script);
   });
 
-  app.get<{ Querystring: { mac?: string } }>("/dispatch", async (request, reply) => {
+  app.get<{ Querystring: { mac?: string; arch?: string } }>("/dispatch", async (request, reply) => {
     const mac = (request.query.mac ?? "").toLowerCase().replace(/-/g, ":");
     const currentState = state.load();
+    const arch = resolveArch(currentState, mac, request.query.arch, config);
+    const fedoraMirror = fedoraMirrorFor(config.fedoraVersion, arch);
 
     // Debug mode takes highest priority — auto-clear after serving once
     const debugEntry = currentState.debug[mac];
@@ -72,22 +163,48 @@ echo "==============================="
       state.update((s) => { delete s.debug[mac]; });
 
       let script: string;
-      if (debugEntry.pxeBoot) {
+      const wantsPxeBoot = debugEntry.pxeBoot === true;
-        logger.info(`PXE BOOT DEBUG: ${mac} -> ${hostname} (kernel+initrd from PXE, root from NVMe)`);
+      const root = wantsPxeBoot ? resolveRoot(currentState, mac) : null;
+
+      if (root !== null) {
+        logger.info(`PXE BOOT DEBUG: ${mac} -> ${hostname} (${arch}, root=${root.rootDevice})`);
         script = renderPxeBootDebugIpxe({
           mac,
           hostname,
           serverIp: config.serverIp,
           httpPort: config.httpPort,
+          arch,
+          ...root,
         });
       } else {
-        logger.info(`DEBUG BOOT: ${mac} -> ${hostname} (rescue mode)`);
+        // --pxe-boot without a known root device falls back to rescue rather than
+        // guessing. A wrong root= leaves the machine unbootable, and rescue is where
+        // the operator can find the real one (curl /debug-setup.sh reports it back).
+        const notice = wantsPxeBoot
+          ? [
+              "",
+              "NOTE: --pxe-boot requested, but no root device is recorded",
+              "      for this machine. Booting rescue instead.",
+              "      From the rescue shell, run:",
+              // No pipe or && here: iPXE treats || and && as command separators, so keep
+              // the printed command free of anything its parser might claim.
+              `        curl -s http://${config.serverIp}:${config.httpPort}/debug-setup.sh -o /tmp/s.sh ; sh /tmp/s.sh`,
+              "      then retry --pxe-boot.",
+            ]
+          : undefined;
+        if (wantsPxeBoot) {
+          logger.warn(`PXE BOOT DEBUG: ${mac} -> ${hostname} has no recorded root device -- serving rescue instead`);
+        } else {
+          logger.info(`DEBUG BOOT: ${mac} -> ${hostname} (${arch}, rescue mode)`);
+        }
         script = renderDebugIpxe({
           mac,
           hostname,
           serverIp: config.serverIp,
           httpPort: config.httpPort,
-          fedoraMirror: config.fedoraMirror,
+          fedoraMirror,
+          arch,
+          ...(notice ? { notice } : {}),
         });
       }
       return reply.type("text/plain").send(script);
@@ -97,10 +214,24 @@ echo "==============================="
     if (queueEntry) {
       const hostname = queueEntry.hostname ?? "lab-node";
       const os = queueEntry.os ?? "fedora-43";
-      logger.info(`INSTALL STARTED: ${mac} -> ${hostname} (${os})`);
+      logger.info(`INSTALL STARTED: ${mac} -> ${hostname} (${os}, ${arch})`);
 
       let script: string;
       if (os.startsWith("ubuntu")) {
+        // Last line of defence. The install guard refuses this combination when the
+        // machine's architecture is already known, but a machine queued before it was
+        // discovered can reach here. Serving the x86-only Ubuntu kernel to an arm64
+        // client is precisely the bug this work exists to fix, so stop instead.
+        if (!osSupportsArch(os as OsId, arch)) {
+          logger.error(`INSTALL BLOCKED: ${mac} -> ${hostname} -- ${os} has no ${arch} artifacts`);
+          script = renderUnsupportedIpxe({
+            hostname,
+            mac,
+            reason: `${os} publishes no ${arch} netboot artifacts`,
+            action: `labctl provision install ${mac} ${hostname} --os fedora-43`,
+          });
+          return reply.type("text/plain").send(script);
+        }
         script = renderUbuntuInstallIpxe({
           mac,
           hostname,
@@ -115,7 +246,8 @@ echo "==============================="
           serverIp: config.serverIp,
           httpPort: config.httpPort,
           fedoraVersion: config.fedoraVersion,
-          fedoraMirror: config.fedoraMirror,
+          fedoraMirror,
+          arch,
         });
       }
 
@@ -132,13 +264,14 @@ echo "==============================="
     }
 
     // Unknown MAC -> discovery mode
-    logger.info(`PXE request from ${mac} -> discovery mode`);
+    logger.info(`PXE request from ${mac} (${arch}) -> discovery mode`);
 
     const script = renderDiscoverIpxe({
       mac,
       serverIp: config.serverIp,
       httpPort: config.httpPort,
-      fedoraMirror: config.fedoraMirror,
+      fedoraMirror,
+      arch,
     });
 
     return reply.type("text/plain").send(script);

bastion/src/bastion/src/services/install-guard.ts

@@ -0,0 +1,94 @@
+// Pre-flight checks for queuing an OS install.
+//
+// Both entry points -- the HTTP /api/install route and the labd command-install handler
+// -- run this, so `labctl provision install` and `provision reprovision` are covered
+// whichever way the request arrives.
+//
+// Rescue/debug is deliberately NOT guarded. Being unable to reinstall a machine is
+// exactly when you most need to boot it into a rescue shell.
+
+import type { Arch, BastionState, OsId } from "@lab/shared";
+import { classifyOnboard, normalizeArch, osSupportsArch, vendorOsDescription, archesForOs } from "@lab/shared";
+
+export type InstallCheck =
+  | { allowed: true }
+  | { allowed: false; error: string };
+
+interface MachineIdentity {
+  hostname: string;
+  arch: Arch | undefined;
+  identity: Parameters<typeof classifyOnboard>[0];
+}
+
+/** Best-known identity for a MAC, merged across the three state maps. */
+function identify(state: BastionState, mac: string): MachineIdentity {
+  const discovered = state.discovered[mac];
+  const installed = state.installed[mac];
+  const queued = state.install_queue[mac];
+
+  const manufacturer = discovered?.manufacturer ?? installed?.manufacturer;
+  const product = discovered?.product ?? installed?.product;
+  const board = discovered?.board;
+  const onboard = installed?.onboard ?? discovered?.onboard;
+  const vendorOs = installed?.vendor_os ?? discovered?.vendor_os;
+
+  return {
+    hostname: installed?.hostname ?? queued?.hostname ?? discovered?.product ?? mac,
+    arch: normalizeArch(installed?.arch ?? queued?.arch ?? discovered?.arch),
+    identity: {
+      mac,
+      ...(manufacturer !== undefined ? { manufacturer } : {}),
+      ...(product !== undefined ? { product } : {}),
+      ...(board !== undefined ? { board } : {}),
+      ...(onboard !== undefined ? { onboard } : {}),
+      ...(vendorOs !== undefined ? { vendor_os: vendorOs } : {}),
+    },
+  };
+}
+
+/**
+ * Decide whether `mac` may be queued for an install of `os`.
+ *
+ * Refusals name the machine and the reason, and point at the action that is available
+ * instead. An operator hitting this at 2am should not have to read the source to work
+ * out what happened.
+ */
+export function checkInstallAllowed(
+  state: BastionState,
+  mac: string,
+  os: OsId,
+): InstallCheck {
+  const machine = identify(state, mac);
+  const { onboard, vendor_os } = classifyOnboard(machine.identity);
+
+  // 1. Machines running a vendor OS we cannot rebuild.
+  if (onboard === "ssh") {
+    const what = vendorOsDescription(vendor_os);
+    return {
+      allowed: false,
+      error:
+        `Refusing to install ${machine.hostname} (${mac}): it runs ${what}. ` +
+        `No image in our pipeline can restore it, so installing ${os} would destroy that ` +
+        `driver and firmware stack permanently. This machine is SSH-onboard: we manage its ` +
+        `userspace, not its OS. ` +
+        `To boot it into a rescue shell instead, run: labctl provision debug ${machine.hostname}`,
+      // TODO: when a DGX OS / SparkOS image joins the pipeline, an install targeting a
+      // machine whose vendor_os matches that image should be allowed through here.
+    };
+  }
+
+  // 2. Architecture the OS has no netboot artifacts for.
+  if (machine.arch !== undefined && !osSupportsArch(os, machine.arch)) {
+    const supported = archesForOs(os);
+    return {
+      allowed: false,
+      error:
+        `Refusing to install ${os} on ${machine.hostname} (${mac}): ` +
+        `${os} has no ${machine.arch} netboot artifacts` +
+        (supported.length > 0 ? ` (only ${supported.join(", ")})` : "") +
+        `. Use an OS that supports ${machine.arch}.`,
+    };
+  }
+
+  return { allowed: true };
+}

bastion/src/bastion/src/templates/boot.ipxe.ts

@@ -1,4 +1,55 @@
 // iPXE boot script templates for dispatch routing.
+//
+// Architecture handling: the bastion serves one kernel/initrd pair per architecture.
+// x86_64 keeps the original unsuffixed paths so its output is unchanged; every other
+// architecture gets an arch-suffixed pair. See stageBootArtifacts() in main.ts for the
+// matching staging side, and boot-iso.ts for the same scheme on the ISO path.
+
+import type { Arch } from "@lab/shared";
+
+/** Kernel/initrd URL paths, keyed by architecture. */
+export function kernelPath(arch: Arch): string {
+  return arch === "x86_64" ? "/vmlinuz" : `/vmlinuz-${arch}`;
+}
+
+export function initrdPath(arch: Arch): string {
+  return arch === "x86_64" ? "/initrd.img" : `/initrd-${arch}.img`;
+}
+
+/**
+ * Console arguments per architecture.
+ *
+ * arm64 has no VGA text console: a headless machine only talks over the SoC UART, so
+ * ttyAMA0 must be listed as well. The last console= wins for /dev/console, so serial
+ * is the interactive one while tty0 still receives boot output on machines with a
+ * display attached.
+ */
+const CONSOLE_ARGS: Record<Arch, string> = {
+  x86_64: "console=tty0",
+  aarch64: "console=tty0 console=ttyAMA0,115200",
+};
+
+/**
+ * Anaconda arguments for the graphical-suppression / console setup.
+ *
+ * `nomodeset` disables kernel mode setting, which on x86 forces the generic VGA path
+ * and makes flaky GPU drivers survive the installer. On arm64 it does not mean the
+ * same thing -- there is no VGA fallback to drop back to, and it can leave the machine
+ * with no usable console at all -- so arm64 gets explicit console arguments instead.
+ */
+function installerArgs(arch: Arch): string {
+  return arch === "x86_64" ? "inst.text nomodeset" : `inst.text ${CONSOLE_ARGS[arch]}`;
+}
+
+/** Extra console arguments appended to templates that don't already set them. */
+function extraConsoleArgs(arch: Arch): string {
+  return arch === "x86_64" ? "" : ` ${CONSOLE_ARGS[arch]}`;
+}
+
+/** Join kernel arguments, dropping empties so callers can pass optional groups. */
+function joinArgs(...parts: Array<string | undefined>): string {
+  return parts.filter((p) => p !== undefined && p !== "").join(" ");
+}
 
 export interface BootIpxeParams {
   serverIp: string;
@@ -8,6 +59,11 @@ export interface BootIpxeParams {
 /**
  * Initial iPXE boot script that chains to the dispatch endpoint.
  * This is what dnsmasq serves to iPXE clients via HTTP.
+ *
+ * `${buildarch}` is iPXE's own build architecture ("x86_64" or "arm64"), which is the
+ * one architecture signal available on every path -- network PXE, UEFI HTTP boot and
+ * the boot ISO alike. DHCP option 93 only reaches dnsmasq, never this HTTP endpoint.
+ * dispatch prefers the tracked machine record and falls back to this.
  */
 export function renderBootIpxe(params: BootIpxeParams): string {
   return `#!ipxe
@@ -19,7 +75,7 @@ echo   Contacting server for instructions...
 echo ============================================
 echo
 
-chain http://${params.serverIp}:${params.httpPort}/dispatch?mac=\${net0/mac}
+chain http://${params.serverIp}:${params.httpPort}/dispatch?mac=\${net0/mac}&arch=\${buildarch}
 `;
 }
 
@@ -31,7 +87,9 @@ export function renderDiscoverIpxe(params: {
   serverIp: string;
   httpPort: number;
   fedoraMirror: string;
+  arch: Arch;
 }): string {
+  const base = `http://${params.serverIp}:${params.httpPort}`;
   return `#!ipxe
 
 echo
@@ -42,8 +100,8 @@ echo   Collecting hardware info...
 echo =============================================
 echo
 
-kernel http://${params.serverIp}:${params.httpPort}/vmlinuz inst.ks=http://${params.serverIp}:${params.httpPort}/discover.ks inst.stage2=${params.fedoraMirror} inst.text nomodeset
+kernel ${base}${kernelPath(params.arch)} inst.ks=${base}/discover.ks inst.stage2=${params.fedoraMirror} ${installerArgs(params.arch)}
-initrd http://${params.serverIp}:${params.httpPort}/initrd.img
+initrd ${base}${initrdPath(params.arch)}
 boot
 `;
 }
@@ -58,7 +116,9 @@ export function renderInstallIpxe(params: {
   httpPort: number;
   fedoraVersion: string;
   fedoraMirror: string;
+  arch: Arch;
 }): string {
+  const base = `http://${params.serverIp}:${params.httpPort}`;
   return `#!ipxe
 
 echo
@@ -69,8 +129,8 @@ echo   MAC:    ${params.mac}
 echo =============================================
 echo
 
-kernel http://${params.serverIp}:${params.httpPort}/vmlinuz inst.ks=http://${params.serverIp}:${params.httpPort}/ks?mac=${params.mac} inst.repo=${params.fedoraMirror} inst.text nomodeset
+kernel ${base}${kernelPath(params.arch)} inst.ks=${base}/ks?mac=${params.mac} inst.repo=${params.fedoraMirror} ${installerArgs(params.arch)}
-initrd http://${params.serverIp}:${params.httpPort}/initrd.img
+initrd ${base}${initrdPath(params.arch)}
 boot
 `;
 }
@@ -78,6 +138,9 @@ boot
 /**
  * iPXE script for debug/rescue mode -- boots Fedora installer in rescue mode.
  * Provides a shell with LVM tools, network, and SSH for inspecting installed systems.
+ *
+ * `notice` is shown before the boot line. dispatch uses it to explain why a requested
+ * --pxe-boot fell back to rescue.
  */
 export function renderDebugIpxe(params: {
   mac: string;
@@ -85,7 +148,11 @@ export function renderDebugIpxe(params: {
   serverIp: string;
   httpPort: number;
   fedoraMirror: string;
+  arch: Arch;
+  notice?: string[];
 }): string {
+  const base = `http://${params.serverIp}:${params.httpPort}`;
+  const notice = (params.notice ?? []).map((line) => `echo   ${line}\n`).join("");
   return `#!ipxe
 
 echo
@@ -93,11 +160,11 @@ echo =============================================
 echo   Lab PXE Bastion - DEBUG/RESCUE MODE
 echo   Target: ${params.hostname}
 echo   MAC:    ${params.mac}
-echo =============================================
+${notice}echo =============================================
 echo
 
-kernel http://${params.serverIp}:${params.httpPort}/vmlinuz inst.rescue inst.text inst.sshd inst.ks=http://${params.serverIp}:${params.httpPort}/debug.ks?mac=${params.mac} inst.stage2=${params.fedoraMirror}
+kernel ${base}${kernelPath(params.arch)} inst.rescue inst.text inst.sshd inst.ks=${base}/debug.ks?mac=${params.mac} inst.stage2=${params.fedoraMirror}${extraConsoleArgs(params.arch)}
-initrd http://${params.serverIp}:${params.httpPort}/initrd.img
+initrd ${base}${initrdPath(params.arch)}
 boot
 `;
 }
@@ -106,13 +173,28 @@ boot
  * iPXE script for PXE-boot debug mode -- boots the installed system's root
  * filesystem using the bastion's PXE kernel+initrd instead of local GRUB.
  * Workaround for UEFI firmware bugs that make local disk boot slow.
+ *
+ * rootDevice/rootArgs come from the machine's record -- they are not assumed. Our
+ * Fedora installs use an LVM layout, but nothing guarantees any given machine does,
+ * and a wrong root= here means an unbootable machine. dispatch refuses to render this
+ * script without them.
  */
 export function renderPxeBootDebugIpxe(params: {
   mac: string;
   hostname: string;
   serverIp: string;
   httpPort: number;
+  arch: Arch;
+  rootDevice: string;
+  rootArgs?: string;
 }): string {
+  const base = `http://${params.serverIp}:${params.httpPort}`;
+  const cmdline = joinArgs(
+    `root=${params.rootDevice}`,
+    "ro",
+    params.rootArgs,
+    CONSOLE_ARGS[params.arch],
+  );
   return `#!ipxe
 
 echo
@@ -124,12 +206,40 @@ echo   Kernel+initrd from PXE, root from NVMe
 echo =============================================
 echo
 
-kernel http://${params.serverIp}:${params.httpPort}/vmlinuz root=/dev/mapper/labvg-root ro rd.lvm.lv=labvg/root rd.lvm.lv=labvg/swap console=tty0
+kernel ${base}${kernelPath(params.arch)} ${cmdline}
-initrd http://${params.serverIp}:${params.httpPort}/initrd.img
+initrd ${base}${initrdPath(params.arch)}
 boot
 `;
 }
 
+/**
+ * iPXE script for a request we refuse to serve.
+ *
+ * Better a machine that stops with a legible reason on its console than one handed a
+ * kernel it cannot execute, which fails much later and much less clearly.
+ */
+export function renderUnsupportedIpxe(params: {
+  mac: string;
+  hostname: string;
+  reason: string;
+  action?: string;
+}): string {
+  return `#!ipxe
+
+echo
+echo =============================================
+echo   Lab PXE Bastion - CANNOT BOOT THIS MACHINE
+echo   Target: ${params.hostname}
+echo   MAC:    ${params.mac}
+echo
+echo   ${params.reason}
+${params.action !== undefined ? `echo\necho   Try: ${params.action}\n` : ""}echo =============================================
+echo
+sleep 10
+exit 1
+`;
+}
+
 /**
  * iPXE script for already-installed machines -- exits to boot from local disk.
  */

bastion/src/bastion/src/templates/dnsmasq.conf.ts

@@ -48,15 +48,20 @@ enable-tftp
 tftp-root=${tftpDir}
 tftp-no-blocksize
 
-# Detect client architecture -- PXE (TFTP) clients
+# Detect client architecture -- PXE (TFTP) clients.
+# Values are DHCP option 93 (Client System Architecture), IANA "Processor Architecture
+# Types". Getting these wrong means the machine is handed a bootloader its firmware
+# cannot execute, and it loops or hangs with no console output.
 dhcp-match=set:bios,option:client-arch,0
 dhcp-match=set:efi-x86_64,option:client-arch,7
 dhcp-match=set:efi-x86_64,option:client-arch,9
 dhcp-match=set:efi-arm64,option:client-arch,11
 
-# Detect client architecture -- UEFI HTTP Boot clients (no TFTP size limit)
+# Detect client architecture -- UEFI HTTP Boot clients (no TFTP size limit).
+# 16 = x64 uefi boot from http, 19 = arm uefi 64 boot from http.
+# (20 is pc/at bios boot from http -- not arm64.)
 dhcp-match=set:httpboot-x86_64,option:client-arch,16
-dhcp-match=set:httpboot-arm64,option:client-arch,20
+dhcp-match=set:httpboot-arm64,option:client-arch,19
 
 # Detect iPXE clients (already chainloaded)
 dhcp-userclass=set:ipxe,iPXE

bastion/src/bastion/tests/arch-dispatch.test.ts

@@ -0,0 +1,291 @@
+// aarch64 support in the PXE dispatch path.
+//
+// The x86_64 side is pinned separately by ipxe-x86-regression.test.ts.
+
+import { describe, it, expect, beforeEach, afterEach } from "vitest";
+import { mkdirSync, rmSync } from "node:fs";
+import { join } from "node:path";
+import { tmpdir } from "node:os";
+import type { BastionConfig, BastionState, HardwareInfo } from "@lab/shared";
+import { createApp } from "../src/server.js";
+import { resolveArch } from "../src/routes/dispatch.js";
+import { renderDnsmasqConf } from "../src/templates/dnsmasq.conf.js";
+import type { FastifyInstance } from "fastify";
+import type { StateManager } from "../src/services/state.js";
+
+function createTestConfig(testDir: string): BastionConfig {
+  return {
+    fedoraVersion: "43",
+    arch: "x86_64",
+    httpPort: 0,
+    timezone: "Europe/London",
+    locale: "en_GB.UTF-8",
+    bastionDir: testDir,
+    domain: "test.local",
+    dhcpMode: "proxy",
+    dhcpRangeStart: "",
+    dhcpRangeEnd: "",
+    ubuntuVersion: "26.04",
+    ubuntuMirror: "https://releases.ubuntu.com/26.04",
+    iface: "eth0",
+    serverIp: "10.0.0.1",
+    network: "10.0.0.0",
+    gateway: "10.0.0.1",
+    sshKeys: ["ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAITEST test@test"],
+    adminUser: "testadmin",
+    syslogPort: 15514,
+    skipDnsmasq: true,
+    skipArtifacts: true,
+    fedoraMirror: "https://download.fedoraproject.org/pub/fedora/linux/releases/43/Everything/x86_64/os",
+    tftpDir: join(testDir, "tftp"),
+    httpDir: join(testDir, "http"),
+    stateFile: join(testDir, "state.json"),
+  };
+}
+
+function hardware(mac: string, over: Partial<HardwareInfo> = {}): HardwareInfo {
+  return {
+    mac,
+    product: "TestBox",
+    board: "TestBoard",
+    serial: "SN123",
+    manufacturer: "TestCorp",
+    cpu_model: "Test CPU",
+    cpu_cores: 4,
+    memory_gb: 16,
+    arch: "x86_64",
+    disks: [],
+    nics: [],
+    first_seen: new Date().toISOString(),
+    last_seen: new Date().toISOString(),
+    ...over,
+  };
+}
+
+const emptyState = (): BastionState => ({
+  discovered: {}, install_queue: {}, installed: {}, debug: {},
+});
+
+describe("architecture resolution", () => {
+  const config = createTestConfig("/tmp/unused");
+  const mac = "aa:bb:cc:dd:ee:ff";
+
+  it("prefers the tracked record over what the client reports", () => {
+    const state = emptyState();
+    state.discovered[mac] = hardware(mac, { arch: "aarch64" });
+    // Client claims x86_64; the machine record says otherwise and wins.
+    expect(resolveArch(state, mac, "x86_64", config)).toBe("aarch64");
+  });
+
+  it("falls back to the architecture reported at boot", () => {
+    expect(resolveArch(emptyState(), mac, "arm64", config)).toBe("aarch64");
+  });
+
+  it("normalises iPXE's arm64 spelling to aarch64", () => {
+    expect(resolveArch(emptyState(), mac, "arm64", config)).toBe("aarch64");
+    expect(resolveArch(emptyState(), mac, "x86_64", config)).toBe("x86_64");
+  });
+
+  it("falls back to the configured default for unknown architectures", () => {
+    expect(resolveArch(emptyState(), mac, "riscv64", config)).toBe("x86_64");
+    expect(resolveArch(emptyState(), mac, undefined, config)).toBe("x86_64");
+  });
+
+  it("reads arch from the installed record for already-provisioned machines", () => {
+    const state = emptyState();
+    state.installed[mac] = {
+      hostname: "spark", role: "worker", ip: "10.0.0.5",
+      installed_at: new Date().toISOString(), arch: "aarch64",
+    };
+    expect(resolveArch(state, mac, undefined, config)).toBe("aarch64");
+  });
+});
+
+describe("aarch64 dispatch", () => {
+  let testDir: string;
+  let app: FastifyInstance;
+  let state: StateManager;
+  const mac = "aa:bb:cc:dd:ee:ff";
+
+  beforeEach(() => {
+    testDir = join(tmpdir(), `bastion-arch-test-${Date.now()}-${Math.random().toString(36).slice(2)}`);
+    mkdirSync(join(testDir, "http"), { recursive: true });
+    mkdirSync(join(testDir, "tftp"), { recursive: true });
+    const result = createApp(createTestConfig(testDir));
+    app = result.app;
+    state = result.state;
+  });
+
+  afterEach(async () => {
+    await app.close();
+    rmSync(testDir, { recursive: true, force: true });
+  });
+
+  it("serves the aarch64 kernel and initrd to an arm64 client", async () => {
+    const res = await app.inject({ method: "GET", url: `/dispatch?mac=${mac}&arch=arm64` });
+    expect(res.statusCode).toBe(200);
+    expect(res.body).toContain("/vmlinuz-aarch64");
+    expect(res.body).toContain("/initrd-aarch64.img");
+    expect(res.body).not.toContain("/vmlinuz ");
+  });
+
+  it("points an arm64 client at the aarch64 Fedora mirror", async () => {
+    const res = await app.inject({ method: "GET", url: `/dispatch?mac=${mac}&arch=arm64` });
+    expect(res.body).toContain("Everything/aarch64/os");
+    expect(res.body).not.toContain("Everything/x86_64/os");
+  });
+
+  it("uses serial console arguments and not nomodeset on arm64", async () => {
+    const res = await app.inject({ method: "GET", url: `/dispatch?mac=${mac}&arch=arm64` });
+    expect(res.body).toContain("console=ttyAMA0,115200");
+    expect(res.body).not.toContain("nomodeset");
+  });
+
+  it("refuses to serve the x86-only Ubuntu kernel to an arm64 client", async () => {
+    // A machine queued for Ubuntu before it was discovered as aarch64 reaches dispatch
+    // with no guard having run. Serving it /ubuntu-vmlinuz is the original bug.
+    state.update((s) => {
+      s.install_queue[mac] = {
+        hostname: "arm-node", disk: "", role: "worker",
+        os: "ubuntu-26.04", queued_at: new Date().toISOString(),
+      };
+    });
+
+    const res = await app.inject({ method: "GET", url: `/dispatch?mac=${mac}&arch=arm64` });
+    expect(res.statusCode).toBe(200);
+    expect(res.body).toContain("CANNOT BOOT THIS MACHINE");
+    expect(res.body).toContain("no aarch64 netboot artifacts");
+    expect(res.body).not.toContain("ubuntu-vmlinuz");
+  });
+
+  it("still serves Ubuntu to an x86_64 client", async () => {
+    state.update((s) => {
+      s.install_queue[mac] = {
+        hostname: "x86-node", disk: "", role: "worker",
+        os: "ubuntu-26.04", queued_at: new Date().toISOString(),
+      };
+    });
+
+    const res = await app.inject({ method: "GET", url: `/dispatch?mac=${mac}&arch=x86_64` });
+    expect(res.body).toContain("ubuntu-vmlinuz");
+    expect(res.body).not.toContain("CANNOT BOOT");
+  });
+
+  it("serves a rescue kernel for the recorded architecture, not the requester's", async () => {
+    // The Spark case: machine known to be aarch64, queued for rescue.
+    state.update((s) => {
+      s.discovered[mac] = hardware(mac, { arch: "aarch64" });
+      s.debug[mac] = { hostname: "spark-2935", queued_at: new Date().toISOString() };
+    });
+
+    const res = await app.inject({ method: "GET", url: `/dispatch?mac=${mac}` });
+    expect(res.body).toContain("DEBUG/RESCUE MODE");
+    expect(res.body).toContain("/vmlinuz-aarch64");
+    expect(res.body).toContain("inst.rescue");
+    expect(res.body).toContain("inst.sshd");
+  });
+});
+
+describe("--pxe-boot root device", () => {
+  let testDir: string;
+  let app: FastifyInstance;
+  let state: StateManager;
+  const mac = "aa:bb:cc:dd:ee:ff";
+
+  beforeEach(() => {
+    testDir = join(tmpdir(), `bastion-root-test-${Date.now()}-${Math.random().toString(36).slice(2)}`);
+    mkdirSync(join(testDir, "http"), { recursive: true });
+    mkdirSync(join(testDir, "tftp"), { recursive: true });
+    const result = createApp(createTestConfig(testDir));
+    app = result.app;
+    state = result.state;
+  });
+
+  afterEach(async () => {
+    await app.close();
+    rmSync(testDir, { recursive: true, force: true });
+  });
+
+  it("uses the root device recorded on the machine", async () => {
+    state.update((s) => {
+      s.installed[mac] = {
+        hostname: "worker-1", role: "worker", ip: "10.0.0.50",
+        installed_at: new Date().toISOString(),
+        root_device: "/dev/mapper/otherVG-root",
+        root_args: "rd.lvm.lv=otherVG/root",
+      };
+      s.debug[mac] = { hostname: "worker-1", queued_at: new Date().toISOString(), pxeBoot: true };
+    });
+
+    const res = await app.inject({ method: "GET", url: `/dispatch?mac=${mac}` });
+    expect(res.body).toContain("PXE BOOT (debug)");
+    expect(res.body).toContain("root=/dev/mapper/otherVG-root");
+    expect(res.body).toContain("rd.lvm.lv=otherVG/root");
+    // The old hardcoded layout must not leak back in.
+    expect(res.body).not.toContain("labvg");
+  });
+
+  it("falls back to rescue rather than guessing when no root device is known", async () => {
+    state.update((s) => {
+      s.installed[mac] = {
+        hostname: "spark-2935", role: "worker", ip: "192.168.8.12",
+        installed_at: new Date().toISOString(), arch: "aarch64",
+      };
+      s.debug[mac] = { hostname: "spark-2935", queued_at: new Date().toISOString(), pxeBoot: true };
+    });
+
+    const res = await app.inject({ method: "GET", url: `/dispatch?mac=${mac}` });
+    expect(res.body).toContain("DEBUG/RESCUE MODE");
+    expect(res.body).toContain("no root device is recorded");
+    expect(res.body).toContain("debug-setup.sh");
+    expect(res.body).not.toContain("root=");
+    // And it is still the right architecture.
+    expect(res.body).toContain("/vmlinuz-aarch64");
+  });
+
+  it("records a root device reported from a rescue shell without erasing hardware info", async () => {
+    state.update((s) => {
+      s.discovered[mac] = hardware(mac, { product: "DGX Spark", manufacturer: "NVIDIA", arch: "aarch64" });
+    });
+
+    const res = await app.inject({
+      method: "POST",
+      url: "/api/discover",
+      headers: { "Content-Type": "application/json" },
+      body: JSON.stringify({ mac, root_device: "/dev/nvme0n1p2" }),
+    });
+    expect(res.statusCode).toBe(200);
+
+    const hw = state.load().discovered[mac];
+    expect(hw?.root_device).toBe("/dev/nvme0n1p2");
+    // The partial report must not blank what we already knew.
+    expect(hw?.product).toBe("DGX Spark");
+    expect(hw?.cpu_cores).toBe(4);
+    expect(hw?.arch).toBe("aarch64");
+  });
+});
+
+describe("dnsmasq architecture detection", () => {
+  const conf = renderDnsmasqConf(createTestConfig("/tmp/unused"));
+
+  it("maps DHCP option 93 values to per-architecture bootloaders", () => {
+    // 11 = ARM 64-bit UEFI
+    expect(conf).toContain("dhcp-match=set:efi-arm64,option:client-arch,11");
+    expect(conf).toContain("dhcp-boot=tag:efi-arm64,tag:!ipxe,ipxe-arm64.efi");
+    // 7 / 9 = x64 UEFI, 0 = x86 BIOS
+    expect(conf).toContain("dhcp-match=set:efi-x86_64,option:client-arch,7");
+    expect(conf).toContain("dhcp-match=set:efi-x86_64,option:client-arch,9");
+    expect(conf).toContain("dhcp-match=set:bios,option:client-arch,0");
+  });
+
+  it("matches arm64 UEFI HTTP boot on 19, not 20", () => {
+    // IANA: 19 = arm uefi 64 boot from http, 20 = pc/at bios boot from http.
+    expect(conf).toContain("dhcp-match=set:httpboot-arm64,option:client-arch,19");
+    expect(conf).not.toContain("dhcp-match=set:httpboot-arm64,option:client-arch,20");
+    expect(conf).toContain("dhcp-match=set:httpboot-x86_64,option:client-arch,16");
+  });
+
+  it("offers an arm64 PXE service directive in proxy mode", () => {
+    expect(conf).toContain('pxe-service=tag:!ipxe,ARM64_EFI,"PXE Boot",ipxe-arm64.efi');
+  });
+});

bastion/src/bastion/tests/fixtures/ipxe-x86_64-golden.json

@@ -0,0 +1,8 @@
+{
+  "boot": "#!ipxe\n\necho\necho ============================================\necho   Lab PXE Bastion\necho   Contacting server for instructions...\necho ============================================\necho\n\nchain http://10.0.0.1:8080/dispatch?mac=${net0/mac}\n",
+  "discover": "#!ipxe\n\necho\necho =============================================\necho   Lab PXE Bastion - DISCOVERY MODE\necho   MAC: aa:bb:cc:dd:ee:ff\necho   Collecting hardware info...\necho =============================================\necho\n\nkernel http://10.0.0.1:8080/vmlinuz inst.ks=http://10.0.0.1:8080/discover.ks inst.stage2=https://download.fedoraproject.org/pub/fedora/linux/releases/43/Everything/x86_64/os inst.text nomodeset\ninitrd http://10.0.0.1:8080/initrd.img\nboot\n",
+  "install": "#!ipxe\n\necho\necho =============================================\necho   Lab PXE Bastion - INSTALLING Fedora 43\necho   Target: worker-1\necho   MAC:    aa:bb:cc:dd:ee:ff\necho =============================================\necho\n\nkernel http://10.0.0.1:8080/vmlinuz inst.ks=http://10.0.0.1:8080/ks?mac=aa:bb:cc:dd:ee:ff inst.repo=https://download.fedoraproject.org/pub/fedora/linux/releases/43/Everything/x86_64/os inst.text nomodeset\ninitrd http://10.0.0.1:8080/initrd.img\nboot\n",
+  "debug": "#!ipxe\n\necho\necho =============================================\necho   Lab PXE Bastion - DEBUG/RESCUE MODE\necho   Target: worker-1\necho   MAC:    aa:bb:cc:dd:ee:ff\necho =============================================\necho\n\nkernel http://10.0.0.1:8080/vmlinuz inst.rescue inst.text inst.sshd inst.ks=http://10.0.0.1:8080/debug.ks?mac=aa:bb:cc:dd:ee:ff inst.stage2=https://download.fedoraproject.org/pub/fedora/linux/releases/43/Everything/x86_64/os\ninitrd http://10.0.0.1:8080/initrd.img\nboot\n",
+  "pxeBoot": "#!ipxe\n\necho\necho =============================================\necho   Lab PXE Bastion - PXE BOOT (debug)\necho   Target: worker-1\necho   MAC:    aa:bb:cc:dd:ee:ff\necho   Kernel+initrd from PXE, root from NVMe\necho =============================================\necho\n\nkernel http://10.0.0.1:8080/vmlinuz root=/dev/mapper/labvg-root ro rd.lvm.lv=labvg/root rd.lvm.lv=labvg/swap console=tty0\ninitrd http://10.0.0.1:8080/initrd.img\nboot\n",
+  "localBoot": "#!ipxe\n\necho\necho =============================================\necho   Lab PXE Bastion - worker-1\necho   Already installed, booting from local disk\necho =============================================\necho\nsleep 3\nexit 1\n"
+}

bastion/src/bastion/tests/install-guard.test.ts

@@ -0,0 +1,194 @@
+// Installs must never reach a machine running a vendor OS we cannot restore.
+//
+// This is the guardrail that stops someone reinstalling a DGX Spark at 2am. Rescue is
+// deliberately still allowed for the same machines -- that is the whole point.
+
+import { describe, it, expect, beforeEach, afterEach } from "vitest";
+import { mkdirSync, rmSync } from "node:fs";
+import { join } from "node:path";
+import { tmpdir } from "node:os";
+import type { BastionConfig, BastionState, HardwareInfo } from "@lab/shared";
+import { classifyOnboard } from "@lab/shared";
+import { createApp } from "../src/server.js";
+import { checkInstallAllowed } from "../src/services/install-guard.js";
+import type { FastifyInstance } from "fastify";
+import type { StateManager } from "../src/services/state.js";
+
+// The real machines this exists to protect.
+const SPARK_2935 = "4c:bb:47:7f:29:35";
+const SPARK_3A1C = "48:21:0b:96:3a:1c";
+const ORDINARY = "aa:bb:cc:dd:ee:ff";
+
+function createTestConfig(testDir: string): BastionConfig {
+  return {
+    fedoraVersion: "43", arch: "x86_64", httpPort: 0,
+    timezone: "Europe/London", locale: "en_GB.UTF-8", bastionDir: testDir,
+    domain: "test.local", dhcpMode: "proxy", dhcpRangeStart: "", dhcpRangeEnd: "",
+    ubuntuVersion: "26.04", ubuntuMirror: "https://releases.ubuntu.com/26.04",
+    iface: "eth0", serverIp: "10.0.0.1", network: "10.0.0.0", gateway: "10.0.0.1",
+    sshKeys: [], adminUser: "testadmin", syslogPort: 15514,
+    skipDnsmasq: true, skipArtifacts: true,
+    fedoraMirror: "https://download.fedoraproject.org/pub/fedora/linux/releases/43/Everything/x86_64/os",
+    tftpDir: join(testDir, "tftp"), httpDir: join(testDir, "http"),
+    stateFile: join(testDir, "state.json"),
+  };
+}
+
+function hardware(mac: string, over: Partial<HardwareInfo> = {}): HardwareInfo {
+  return {
+    mac, product: "TestBox", board: "TestBoard", serial: "SN1",
+    manufacturer: "TestCorp", cpu_model: "Test CPU", cpu_cores: 4, memory_gb: 16,
+    arch: "x86_64", disks: [], nics: [],
+    first_seen: new Date().toISOString(), last_seen: new Date().toISOString(),
+    ...over,
+  };
+}
+
+const emptyState = (): BastionState => ({
+  discovered: {}, install_queue: {}, installed: {}, debug: {},
+});
+
+describe("classifyOnboard", () => {
+  it("recognises a DGX Spark from its DMI identity", () => {
+    expect(classifyOnboard({
+      mac: ORDINARY, manufacturer: "NVIDIA", product: "NVIDIA DGX Spark", board: "GB10",
+    })).toEqual({ onboard: "ssh", vendor_os: "dgx-os" });
+  });
+
+  it("recognises the known Sparks even with no DMI recorded", () => {
+    // Neither Spark has hardware info in bastion state today. A DMI-only rule would
+    // fail open on exactly the machines this protects.
+    expect(classifyOnboard({ mac: SPARK_2935 }).onboard).toBe("ssh");
+    expect(classifyOnboard({ mac: SPARK_3A1C }).onboard).toBe("ssh");
+  });
+
+  it("treats ordinary hardware as PXE-installable", () => {
+    expect(classifyOnboard({
+      mac: ORDINARY, manufacturer: "Beelink", product: "SER9", board: "SER9",
+    })).toEqual({ onboard: "pxe" });
+  });
+
+  it("does not override an explicit classification already on the record", () => {
+    expect(classifyOnboard({
+      mac: SPARK_2935, onboard: "pxe",
+    })).toEqual({ onboard: "pxe" });
+  });
+});
+
+describe("checkInstallAllowed", () => {
+  it("refuses a DGX Spark and explains why", () => {
+    const state = emptyState();
+    state.installed[SPARK_2935] = {
+      hostname: "spark-2935", role: "worker", ip: "192.168.8.12",
+      installed_at: new Date().toISOString(), arch: "aarch64",
+    };
+
+    const result = checkInstallAllowed(state, SPARK_2935, "fedora-43");
+    expect(result.allowed).toBe(false);
+    if (result.allowed === false) {
+      expect(result.error).toContain("spark-2935");
+      expect(result.error).toContain("DGX OS");
+      expect(result.error).toContain("provision debug");
+    }
+  });
+
+  it("refuses a Spark that is only known by MAC", () => {
+    expect(checkInstallAllowed(emptyState(), SPARK_3A1C, "fedora-43").allowed).toBe(false);
+  });
+
+  it("allows an ordinary discovered machine", () => {
+    const state = emptyState();
+    state.discovered[ORDINARY] = hardware(ORDINARY);
+    expect(checkInstallAllowed(state, ORDINARY, "fedora-43").allowed).toBe(true);
+  });
+
+  it("allows Fedora on aarch64", () => {
+    const state = emptyState();
+    state.discovered[ORDINARY] = hardware(ORDINARY, { arch: "aarch64" });
+    expect(checkInstallAllowed(state, ORDINARY, "fedora-43").allowed).toBe(true);
+  });
+
+  it("refuses Ubuntu on aarch64 -- no netboot artifacts are published", () => {
+    const state = emptyState();
+    state.discovered[ORDINARY] = hardware(ORDINARY, { arch: "aarch64" });
+    const result = checkInstallAllowed(state, ORDINARY, "ubuntu-26.04");
+    expect(result.allowed).toBe(false);
+    if (result.allowed === false) {
+      expect(result.error).toContain("aarch64");
+    }
+  });
+
+  it("allows Ubuntu on x86_64", () => {
+    const state = emptyState();
+    state.discovered[ORDINARY] = hardware(ORDINARY, { arch: "x86_64" });
+    expect(checkInstallAllowed(state, ORDINARY, "ubuntu-26.04").allowed).toBe(true);
+  });
+});
+
+describe("install route enforces the guard", () => {
+  let testDir: string;
+  let app: FastifyInstance;
+  let state: StateManager;
+
+  beforeEach(() => {
+    testDir = join(tmpdir(), `bastion-guard-test-${Date.now()}-${Math.random().toString(36).slice(2)}`);
+    mkdirSync(join(testDir, "http"), { recursive: true });
+    mkdirSync(join(testDir, "tftp"), { recursive: true });
+    const result = createApp(createTestConfig(testDir));
+    app = result.app;
+    state = result.state;
+  });
+
+  afterEach(async () => {
+    await app.close();
+    rmSync(testDir, { recursive: true, force: true });
+  });
+
+  it("rejects POST /api/install for a Spark and queues nothing", async () => {
+    const res = await app.inject({
+      method: "POST",
+      url: "/api/install",
+      headers: { "Content-Type": "application/json" },
+      body: JSON.stringify({ mac: SPARK_2935, hostname: "spark-2935", role: "worker" }),
+    });
+
+    expect(res.statusCode).toBe(409);
+    expect(JSON.parse(res.body).error).toContain("Refusing to install");
+    expect(state.load().install_queue[SPARK_2935]).toBeUndefined();
+  });
+
+  it("still serves rescue to a Spark -- debug is never guarded", async () => {
+    state.update((s) => {
+      s.installed[SPARK_2935] = {
+        hostname: "spark-2935", role: "worker", ip: "192.168.8.12",
+        installed_at: new Date().toISOString(), arch: "aarch64",
+      };
+      s.debug[SPARK_2935] = { hostname: "spark-2935", queued_at: new Date().toISOString() };
+    });
+
+    const res = await app.inject({ method: "GET", url: `/dispatch?mac=${SPARK_2935}` });
+    expect(res.statusCode).toBe(200);
+    expect(res.body).toContain("DEBUG/RESCUE MODE");
+    expect(res.body).toContain("/vmlinuz-aarch64");
+  });
+
+  it("a Spark that PXE boots unqueued gets discovery, never an install", async () => {
+    const res = await app.inject({ method: "GET", url: `/dispatch?mac=${SPARK_2935}&arch=arm64` });
+    expect(res.body).toContain("DISCOVERY MODE");
+    expect(res.body).not.toContain("INSTALLING");
+  });
+
+  it("still accepts an ordinary machine", async () => {
+    state.update((s) => { s.discovered[ORDINARY] = hardware(ORDINARY); });
+
+    const res = await app.inject({
+      method: "POST",
+      url: "/api/install",
+      headers: { "Content-Type": "application/json" },
+      body: JSON.stringify({ mac: ORDINARY, hostname: "worker-1", role: "worker" }),
+    });
+
+    expect(res.statusCode).toBe(200);
+    expect(state.load().install_queue[ORDINARY]).toBeDefined();
+  });
+});

bastion/src/bastion/tests/ipxe-x86-regression.test.ts

@@ -0,0 +1,89 @@
+// x86_64 iPXE output regression gate.
+//
+// The aarch64 PXE work must not change what an x86_64 machine is served. The golden
+// fixture was dumped from the templates as they stood before that work started, so
+// any diff here is a regression, not an improvement.
+//
+// The one deliberate exception is renderBootIpxe: its chain URL gained
+// `&arch=${buildarch}` so the dispatch endpoint can observe the client's
+// architecture at boot time. That single change is asserted explicitly below
+// rather than being allowed to slip through the byte-for-byte comparison.
+
+import { describe, it, expect } from "vitest";
+import { readFileSync } from "node:fs";
+import { join } from "node:path";
+import { fileURLToPath } from "node:url";
+import { dirname } from "node:path";
+import {
+  renderBootIpxe,
+  renderDiscoverIpxe,
+  renderInstallIpxe,
+  renderDebugIpxe,
+  renderPxeBootDebugIpxe,
+  renderLocalBootIpxe,
+} from "../src/templates/boot.ipxe.js";
+
+const here = dirname(fileURLToPath(import.meta.url));
+const golden = JSON.parse(
+  readFileSync(join(here, "fixtures", "ipxe-x86_64-golden.json"), "utf-8"),
+) as Record<string, string>;
+
+// Exactly the parameters used to dump the fixture.
+const serverIp = "10.0.0.1";
+const httpPort = 8080;
+const mac = "aa:bb:cc:dd:ee:ff";
+const hostname = "worker-1";
+const fedoraVersion = "43";
+const fedoraMirror =
+  "https://download.fedoraproject.org/pub/fedora/linux/releases/43/Everything/x86_64/os";
+
+// The x86_64 LVM layout the fixture was captured with. Before this work the values
+// were hardcoded in the template; they are now supplied by the caller from machine
+// state, so the fixture pins the rendering, not the defaults.
+const x86Root = {
+  rootDevice: "/dev/mapper/labvg-root",
+  rootArgs: "rd.lvm.lv=labvg/root rd.lvm.lv=labvg/swap",
+};
+
+describe("x86_64 iPXE output is unchanged", () => {
+  it("discover script is byte-identical", () => {
+    const rendered = renderDiscoverIpxe({
+      mac, serverIp, httpPort, fedoraMirror, arch: "x86_64",
+    });
+    expect(rendered).toBe(golden["discover"]);
+  });
+
+  it("install script is byte-identical", () => {
+    const rendered = renderInstallIpxe({
+      mac, hostname, serverIp, httpPort, fedoraVersion, fedoraMirror, arch: "x86_64",
+    });
+    expect(rendered).toBe(golden["install"]);
+  });
+
+  it("debug/rescue script is byte-identical", () => {
+    const rendered = renderDebugIpxe({
+      mac, hostname, serverIp, httpPort, fedoraMirror, arch: "x86_64",
+    });
+    expect(rendered).toBe(golden["debug"]);
+  });
+
+  it("--pxe-boot script is byte-identical when state carries the Fedora LVM layout", () => {
+    const rendered = renderPxeBootDebugIpxe({
+      mac, hostname, serverIp, httpPort, arch: "x86_64", ...x86Root,
+    });
+    expect(rendered).toBe(golden["pxeBoot"]);
+  });
+
+  it("local boot script is byte-identical", () => {
+    expect(renderLocalBootIpxe(hostname)).toBe(golden["localBoot"]);
+  });
+
+  it("boot.ipxe differs only by the &arch= chain parameter", () => {
+    const rendered = renderBootIpxe({ serverIp, httpPort });
+    // The sole intended difference.
+    expect(rendered).toBe(golden["boot"].replace(
+      "/dispatch?mac=${net0/mac}",
+      "/dispatch?mac=${net0/mac}&arch=${buildarch}",
+    ));
+  });
+});

bastion/src/cli/src/api/client.ts

@@ -110,6 +110,7 @@ export class LabdClient {
     memory_gb?: number; arch?: string;
     disks?: Array<{ name: string; size_gb: number; model: string }>;
     nics?: Array<{ name: string; mac: string; state: string }>;
+    root_device?: string; root_args?: string;
   }): Promise<{ status: string; error?: string }> {
     return this.request("POST", "/api/machines/discover", { body: data });
   }

bastion/src/cli/src/commands/debug.ts

@@ -8,6 +8,7 @@ import { join } from "node:path";
 import { Command } from "commander";
 import type { BastionState } from "@lab/shared";
 import { getLabdClient } from "../api/config.js";
+import { ROOT_DEVICE_PROBE, parseRootProbe } from "../utils/hardware-probe.js";
 
 /** Resolve a target (hostname, MAC, or IP) to {mac, hostname, ip} from state. */
 function resolveTarget(
@@ -44,6 +45,54 @@ function resolveTarget(
   return null;
 }
 
+/** The local admin account to SSH as (root is not usable — it has no key here). */
+function sshUser(): string {
+  const adminUser = process.env["SUDO_USER"] ?? process.env["USER"] ?? "";
+  return adminUser === "root" ? "" : adminUser;
+}
+
+/** Common ssh arguments, ending with user@host. Null when there is no usable user. */
+function sshBaseArgs(ip: string): string[] | null {
+  const user = sshUser();
+  if (user === "") return null;
+
+  const sudoUser = process.env["SUDO_USER"];
+  const realHome = sudoUser !== undefined ? join("/home", sudoUser) : homedir();
+  const sshKey = ["id_ed25519", "id_rsa", "id_ecdsa"]
+    .map((name) => join(realHome, ".ssh", name))
+    .find((k) => existsSync(k));
+
+  return [
+    "-o", "StrictHostKeyChecking=no",
+    "-o", "UserKnownHostsFile=/dev/null",
+    "-o", "ConnectTimeout=10",
+    ...(sshKey !== undefined ? ["-i", sshKey] : []),
+    `${user}@${ip}`,
+  ];
+}
+
+/**
+ * Run a shell script on the target as root and return its stdout, or null.
+ *
+ * The script goes over stdin rather than the command line so it can contain quotes
+ * without a second round of shell escaping. `sudo -n` fails fast instead of hanging on
+ * a password prompt that would then eat the script.
+ */
+function sshCapture(ip: string, script: string): string | null {
+  const base = sshBaseArgs(ip);
+  if (base === null) return null;
+  try {
+    return execFileSync("ssh", [...base, "sudo", "-n", "sh", "-s"], {
+      input: script,
+      encoding: "utf-8",
+      stdio: ["pipe", "pipe", "pipe"],
+      timeout: 30_000,
+    });
+  } catch {
+    return null;
+  }
+}
+
 export function registerDebugCommand(parent: Command): void {
   parent
     .command("debug <target>")
@@ -71,6 +120,31 @@ export function registerDebugCommand(parent: Command): void {
       }
 
       const { mac, hostname, ip } = resolved;
+
+      // --pxe-boot needs a root= for the installed system. If the machine is still
+      // reachable, observe it now rather than assuming a disk layout: a wrong root=
+      // leaves the machine unbootable. If it isn't reachable, dispatch falls back to
+      // rescue and the operator reports the real one from there.
+      if (opts.pxeBoot === true && ip !== "") {
+        const known = state.installed[mac]?.root_device ?? state.discovered[mac]?.root_device;
+        if (known === undefined || known === "") {
+          console.log(`No root device recorded for ${hostname}. Probing over SSH...`);
+          const probe = sshCapture(ip, ROOT_DEVICE_PROBE);
+          const root = probe === null ? {} : parseRootProbe(probe);
+          if (root.root_device !== undefined) {
+            console.log(`  root=${root.root_device}${root.root_args !== undefined ? ` ${root.root_args}` : ""}`);
+            try {
+              await client.discoverMachine({ mac, ...root });
+            } catch (err) {
+              console.error(`  Could not record it: ${err instanceof Error ? err.message : String(err)}`);
+            }
+          } else {
+            console.log("  Probe failed. Booting rescue instead; report the root device with:");
+            console.log("    curl http://<bastion>:8080/debug-setup.sh | bash");
+          }
+        }
+      }
+
       console.log(`Queuing debug mode for ${hostname} (${mac})...`);
 
       try {
@@ -86,32 +160,15 @@ export function registerDebugCommand(parent: Command): void {
 
       // Try SSH reboot into PXE
       if (ip !== "") {
-        const adminUser = process.env["SUDO_USER"] ?? process.env["USER"] ?? "";
+        const base = sshBaseArgs(ip);
-        const effectiveUser = adminUser === "root" ? "" : adminUser;
+        if (base !== null) {
-
+          console.log(`\nAttempting SSH reboot into PXE (${sshUser()}@${ip})...`);
-        if (effectiveUser !== "") {
-          console.log(`\nAttempting SSH reboot into PXE (${effectiveUser}@${ip})...`);
-
-          const sudoUser = process.env["SUDO_USER"];
-          const realHome = sudoUser !== undefined ? join("/home", sudoUser) : homedir();
-          const keyPaths = [
-            join(realHome, ".ssh", "id_ed25519"),
-            join(realHome, ".ssh", "id_rsa"),
-            join(realHome, ".ssh", "id_ecdsa"),
-          ];
-          const sshKey = keyPaths.find(k => existsSync(k));
-
-          const sshArgs = [
-            "-o", "StrictHostKeyChecking=no",
-            "-o", "UserKnownHostsFile=/dev/null",
-            "-o", "ConnectTimeout=10",
-            ...(sshKey !== undefined ? ["-i", sshKey] : []),
-            `${effectiveUser}@${ip}`,
-            'PXE_ENTRY=$(sudo efibootmgr | grep -iE "pxe|network|ipv4" | head -1 | grep -oP "Boot\\K[0-9A-F]+"); if [ -n "$PXE_ENTRY" ]; then sudo efibootmgr --bootnext "$PXE_ENTRY" && echo "PXE set as next boot" && sudo reboot; else echo "No PXE boot entry found, rebooting anyway..." && sudo reboot; fi',
-          ];
 
           try {
-            execFileSync("ssh", sshArgs, { stdio: "inherit" });
+            execFileSync("ssh", [
+              ...base,
+              'PXE_ENTRY=$(sudo efibootmgr | grep -iE "pxe|network|ipv4" | head -1 | grep -oP "Boot\\K[0-9A-F]+"); if [ -n "$PXE_ENTRY" ]; then sudo efibootmgr --bootnext "$PXE_ENTRY" && echo "PXE set as next boot" && sudo reboot; else echo "No PXE boot entry found, rebooting anyway..." && sudo reboot; fi',
+            ], { stdio: "inherit" });
           } catch {
             // SSH connection closing during reboot is expected
           }

bastion/src/cli/src/commands/recheck.ts

@@ -4,6 +4,7 @@
 import type { Command } from "commander";
 import { sshExec } from "@lab/modules";
 import { getLabdClient } from "../api/config.js";
+import { ROOT_DEVICE_PROBE } from "../utils/hardware-probe.js";
 
 const BOLD = "\x1b[1m";
 const GREEN = "\x1b[0;32m";
@@ -24,7 +25,9 @@ const HW_COLLECT_SCRIPT = [
   'N=$(grep -c "^processor" /proc/cpuinfo 2>/dev/null || echo 0)',
   'R=$(awk "/MemTotal/ {printf \\"%d\\", \\$2/1024/1024}" /proc/meminfo 2>/dev/null || echo 0)',
   'A=$(uname -m)',
-  'printf \'{"product":"%s","board":"%s","serial":"%s","manufacturer":"%s","cpu_model":"%s","cpu_cores":%s,"memory_gb":%s,"arch":"%s"}\\n\' "$P" "$B" "$S" "$M" "$C" "$N" "$R" "$A"',
+  // Root filesystem, so --pxe-boot has a root= to use instead of assuming our layout.
+  ROOT_DEVICE_PROBE,
+  'printf \'{"product":"%s","board":"%s","serial":"%s","manufacturer":"%s","cpu_model":"%s","cpu_cores":%s,"memory_gb":%s,"arch":"%s","root_device":"%s","root_args":"%s"}\\n\' "$P" "$B" "$S" "$M" "$C" "$N" "$R" "$A" "$RD" "$RA"',
 ].join("; ");
 
 export function registerRecheckCommand(parent: Command): void {
@@ -81,7 +84,10 @@ export function registerRecheckCommand(parent: Command): void {
           const cpu = hwData.cpu_model || "?";
           const cores = hwData.cpu_cores || "?";
           const mem = hwData.memory_gb || "?";
-          console.log(`${GREEN}OK${RESET} ${DIM}${cpu}, ${cores} cores, ${mem}GB${RESET}`);
+          const root = typeof hwData.root_device === "string" && hwData.root_device !== ""
+            ? `, root=${hwData.root_device}`
+            : "";
+          console.log(`${GREEN}OK${RESET} ${DIM}${cpu}, ${cores} cores, ${mem}GB${root}${RESET}`);
           updated++;
         } catch (err) {
           console.log(`${RED}FAIL${RESET} ${DIM}${err instanceof Error ? err.message : String(err)}${RESET}`);

bastion/src/cli/src/utils/hardware-probe.ts

@@ -0,0 +1,59 @@
+// Shell snippets for observing a machine's hardware over SSH.
+//
+// Pure shell + awk, no Python: these run on whatever the target happens to be,
+// including a minimal rescue environment.
+
+/**
+ * Report the root filesystem and any dracut arguments needed to assemble it.
+ *
+ * Emits two lines:
+ *   ROOT_DEVICE=<device>
+ *   ROOT_ARGS=<args>
+ *
+ * Used by `--pxe-boot`, which boots the installed system with a kernel and initrd from
+ * the network. Getting root= wrong there leaves the machine unbootable, so this observes
+ * the machine rather than assuming our Fedora LVM layout.
+ *
+ * Device form is chosen for stability across reboots: LVM logical volumes keep their
+ * /dev/mapper path, anything else is reported by UUID, which survives device renumbering.
+ */
+export const ROOT_DEVICE_PROBE = [
+  'RD=$(findmnt -no SOURCE / 2>/dev/null | head -1)',
+  'RA=""',
+  'RT=$(lsblk -no TYPE "$RD" 2>/dev/null | head -1)',
+  'if [ "$RT" = "lvm" ]; then',
+  '  VGLV=$(lvs --noheadings -o vg_name,lv_name "$RD" 2>/dev/null | awk \'{print $1"/"$2}\')',
+  '  [ -n "$VGLV" ] && RA="rd.lvm.lv=$VGLV"',
+  // Swap must be assembled too or resume= stalls the boot waiting for it.
+  '  SW=$(awk \'NR>1 {print $1; exit}\' /proc/swaps 2>/dev/null)',
+  '  if [ -n "$SW" ] && [ "$(lsblk -no TYPE "$SW" 2>/dev/null | head -1)" = "lvm" ]; then',
+  '    SWVGLV=$(lvs --noheadings -o vg_name,lv_name "$SW" 2>/dev/null | awk \'{print $1"/"$2}\')',
+  '    [ -n "$SWVGLV" ] && [ "$SWVGLV" != "$VGLV" ] && RA="$RA rd.lvm.lv=$SWVGLV"',
+  '  fi',
+  'elif [ -n "$RD" ]; then',
+  '  U=$(findmnt -no UUID / 2>/dev/null | head -1)',
+  '  [ -n "$U" ] && RD="UUID=$U"',
+  'fi',
+  'printf \'ROOT_DEVICE=%s\\nROOT_ARGS=%s\\n\' "$RD" "$RA"',
+].join("; ");
+
+export interface RootInfo {
+  root_device?: string;
+  root_args?: string;
+}
+
+/** Parse the ROOT_DEVICE/ROOT_ARGS lines emitted by ROOT_DEVICE_PROBE. */
+export function parseRootProbe(stdout: string): RootInfo {
+  const out: RootInfo = {};
+  for (const line of stdout.split("\n")) {
+    const trimmed = line.trim();
+    if (trimmed.startsWith("ROOT_DEVICE=")) {
+      const v = trimmed.slice("ROOT_DEVICE=".length).trim();
+      if (v !== "") out.root_device = v;
+    } else if (trimmed.startsWith("ROOT_ARGS=")) {
+      const v = trimmed.slice("ROOT_ARGS=".length).trim();
+      if (v !== "") out.root_args = v;
+    }
+  }
+  return out;
+}

bastion/src/labd/src/routes/bastions.ts

@@ -299,6 +299,7 @@ export function registerBastionRoutes(app: FastifyInstance, db: DbClient): void
       memory_gb?: number; arch?: string;
       disks?: Array<{ name: string; size_gb: number; model: string }>;
       nics?: Array<{ name: string; mac: string; state: string }>;
+      root_device?: string; root_args?: string;
     };
   }>("/api/machines/discover", async (request, reply) => {
     const data = request.body ?? {};

bastion/src/shared/src/hardware/index.ts

@@ -0,0 +1,154 @@
+// Architecture normalisation and machine classification.
+//
+// Both are derived from what the system already observes about a machine -- never from
+// an operator-supplied flag.
+
+import type { Arch, HardwareInfo, OnboardMethod, OsId } from "../types/index.js";
+
+export const SUPPORTED_ARCHES: readonly Arch[] = ["x86_64", "aarch64"] as const;
+
+/**
+ * Normalise an architecture string to one we serve boot artifacts for.
+ *
+ * Sources and their spellings:
+ *   uname -m         -> "x86_64" / "aarch64"
+ *   iPXE ${buildarch}-> "x86_64" / "arm64"
+ *   dpkg/Debian      -> "amd64"  / "arm64"
+ *
+ * Returns undefined for anything we don't serve, so callers fall back rather than
+ * inventing a kernel path that would 404.
+ */
+export function normalizeArch(value: string | undefined | null): Arch | undefined {
+  switch ((value ?? "").trim().toLowerCase()) {
+    case "x86_64":
+    case "x86-64":
+    case "amd64":
+      return "x86_64";
+    case "aarch64":
+    case "arm64":
+      return "aarch64";
+    default:
+      return undefined;
+  }
+}
+
+/** Fedora pxeboot artifact base URL for an architecture. */
+export function fedoraMirrorFor(fedoraVersion: string, arch: Arch): string {
+  return `https://download.fedoraproject.org/pub/fedora/linux/releases/${fedoraVersion}/Everything/${arch}/os`;
+}
+
+/**
+ * Which architectures each OS in the pipeline can actually be installed on.
+ *
+ * Fedora publishes pxeboot vmlinuz/initrd for both. Ubuntu does not: as of 26.04,
+ * releases.ubuntu.com publishes amd64 artifacts only, so there is nothing to netboot an
+ * arm64 machine with. Claiming support would fail at download time with a 404 instead
+ * of a useful message.
+ */
+const OS_ARCH_SUPPORT: Record<OsId, readonly Arch[]> = {
+  "fedora-43": ["x86_64", "aarch64"],
+  "ubuntu-26.04": ["x86_64"],
+};
+
+export function osSupportsArch(os: OsId, arch: Arch): boolean {
+  return (OS_ARCH_SUPPORT[os] ?? []).includes(arch);
+}
+
+export function archesForOs(os: OsId): readonly Arch[] {
+  return OS_ARCH_SUPPORT[os] ?? [];
+}
+
+/**
+ * Machines that run a vendor OS we have no image for.
+ *
+ * These are SSH-onboard: we manage userspace, but reinstalling destroys a driver and
+ * firmware stack our pipeline cannot rebuild. Matched on DMI identity, which is what
+ * discovery and `provision recheck` both collect.
+ *
+ * This is deliberately a property of the machine ("it runs DGX OS"), not a blocklist
+ * ("never install this MAC"). When a DGX OS image joins the pipeline, teaching the
+ * installer about vendor_os "dgx-os" is what unblocks these machines -- no entry here
+ * needs deleting.
+ */
+interface VendorOsRule {
+  vendorOs: string;
+  description: string;
+  matches: (hw: DmiIdentity) => boolean;
+}
+
+interface DmiIdentity {
+  manufacturer: string;
+  product: string;
+  board: string;
+}
+
+const VENDOR_OS_RULES: readonly VendorOsRule[] = [
+  {
+    vendorOs: "dgx-os",
+    description: "NVIDIA DGX OS (proprietary driver + firmware stack, no image in our pipeline)",
+    matches: ({ manufacturer, product, board }) =>
+      (manufacturer.includes("nvidia") || product.includes("nvidia")) &&
+      (product.includes("dgx") || product.includes("spark") ||
+       board.includes("gb10") || product.includes("gb10")),
+  },
+];
+
+/**
+ * Machines known to run a vendor OS, by MAC.
+ *
+ * The DMI rules above only fire once discovery or `provision recheck` has populated a
+ * hardware record. Machines onboarded over SSH may sit in state for a long time with no
+ * DMI at all -- which is exactly the state both DGX Sparks are in today -- so a
+ * DMI-only classifier would fail open on the machines this guard exists to protect.
+ *
+ * This is a statement of fact about known hardware ("this box runs DGX OS"), not an
+ * install policy. Whether that means "refuse" is decided by whether the pipeline has an
+ * image for that vendor OS.
+ */
+const KNOWN_VENDOR_OS_MACS: Record<string, string> = {
+  "4c:bb:47:7f:29:35": "dgx-os",  // spark-2935
+  "48:21:0b:96:3a:1c": "dgx-os",  // spark-3a1c
+};
+
+/**
+ * Classify how a machine should be onboarded, from its hardware record.
+ *
+ * An explicit `onboard` already on the record wins: it may have been set by an operator
+ * or by a rule that has since changed, and silently overriding it would be worse than
+ * leaving it.
+ */
+export function classifyOnboard(
+  hw: Partial<Pick<HardwareInfo, "mac" | "manufacturer" | "product" | "board">>
+    & { onboard?: OnboardMethod; vendor_os?: string },
+): { onboard: OnboardMethod; vendor_os?: string } {
+  if (hw.onboard !== undefined) {
+    return hw.vendor_os !== undefined
+      ? { onboard: hw.onboard, vendor_os: hw.vendor_os }
+      : { onboard: hw.onboard };
+  }
+
+  const knownVendorOs = KNOWN_VENDOR_OS_MACS[(hw.mac ?? "").toLowerCase().replace(/-/g, ":")];
+  if (knownVendorOs !== undefined) {
+    return { onboard: "ssh", vendor_os: knownVendorOs };
+  }
+
+  const identity: DmiIdentity = {
+    manufacturer: (hw.manufacturer ?? "").toLowerCase(),
+    product: (hw.product ?? "").toLowerCase(),
+    board: (hw.board ?? "").toLowerCase(),
+  };
+
+  for (const rule of VENDOR_OS_RULES) {
+    if (rule.matches(identity)) {
+      return { onboard: "ssh", vendor_os: rule.vendorOs };
+    }
+  }
+
+  return { onboard: "pxe" };
+}
+
+/** Human-readable reason a vendor-OS machine must not be reinstalled. */
+export function vendorOsDescription(vendorOs: string | undefined): string {
+  const rule = VENDOR_OS_RULES.find((r) => r.vendorOs === vendorOs);
+  return rule?.description ?? "a vendor OS with no image in our pipeline";
+}

bastion/src/shared/src/index.ts

@@ -1,6 +1,8 @@
 export type {
   OsId,
   Arch,
+  OnboardMethod,
+  RootCandidate,
   Role,
   HardwareInfo,
   InstallConfig,
@@ -10,6 +12,16 @@ export type {
   BastionConfig,
 } from "./types/index.js";
 
+export {
+  SUPPORTED_ARCHES,
+  normalizeArch,
+  fedoraMirrorFor,
+  osSupportsArch,
+  archesForOs,
+  classifyOnboard,
+  vendorOsDescription,
+} from "./hardware/index.js";
+
 export { SUPPORTED_OS, SUPPORTED_ROLES, ROLE_REGISTRY, isValidOsId } from "./types/index.js";
 export type { RoleInfo } from "./types/index.js";
 

bastion/src/shared/src/protocol/index.ts

@@ -113,7 +113,7 @@ export type LabdBastionMessage =
   | { type: "command-role-update"; requestId: string; mac: string; role: string }
   | { type: "command-debug"; requestId: string; mac: string; pxeBoot?: boolean }
   | { type: "command-register"; requestId: string; mac: string; hostname: string; role: string; ip: string }
-  | { type: "command-discover"; requestId: string; mac: string; product?: string; board?: string; serial?: string; manufacturer?: string; cpu_model?: string; cpu_cores?: number; memory_gb?: number; arch?: string; disks?: Array<{ name: string; size_gb: number; model: string }>; nics?: Array<{ name: string; mac: string; state: string }> }
+  | { type: "command-discover"; requestId: string; mac: string; product?: string; board?: string; serial?: string; manufacturer?: string; cpu_model?: string; cpu_cores?: number; memory_gb?: number; arch?: string; disks?: Array<{ name: string; size_gb: number; model: string }>; nics?: Array<{ name: string; mac: string; state: string }>; root_device?: string; root_args?: string }
   | { type: "server-shutdown"; reconnectAfter: number };
 
 export type BastionMessageType = BastionMessage["type"];

bastion/src/shared/src/types/index.ts

@@ -1,6 +1,8 @@
 export type {
   OsId,
   Arch,
+  OnboardMethod,
+  RootCandidate,
   Role,
   HardwareInfo,
   InstallConfig,

bastion/src/shared/src/types/state.ts

@@ -11,6 +11,16 @@ export function isValidOsId(value: string): value is OsId {
   return (SUPPORTED_OS as readonly string[]).includes(value);
 }
 
+/**
+ * How a machine joins the lab.
+ *
+ * "pxe"  -- bare metal we install over the network (the default).
+ * "ssh"  -- the machine already runs a vendor OS we cannot reproduce, so we onboard
+ *           over SSH and manage userspace only. Installing would destroy that OS.
+ *           See classifyOnboard() and os-install-research.md.
+ */
+export type OnboardMethod = "pxe" | "ssh";
+
 export interface HardwareInfo {
   mac: string;
   product: string;
@@ -26,6 +36,23 @@ export interface HardwareInfo {
   first_seen: string;
   last_seen: string;
   bastionId?: string;  // set when aggregated through labd
+  // Onboarding classification -- absent means "pxe" (see classifyOnboard)
+  onboard?: OnboardMethod;
+  vendor_os?: string;   // e.g. "dgx-os": the OS this machine must keep running
+  // Root filesystem, for booting the installed system over PXE (--pxe-boot).
+  // Observed from the machine, never assumed.
+  root_device?: string;                 // e.g. "/dev/mapper/labvg-root"
+  root_args?: string;                   // e.g. "rd.lvm.lv=labvg/root rd.lvm.lv=labvg/swap"
+  root_candidates?: RootCandidate[];    // reported from a rescue shell when unknown
+}
+
+/** A possible root filesystem found while probing an unreachable machine. */
+export interface RootCandidate {
+  device: string;       // e.g. "/dev/mapper/labvg-root"
+  args?: string;        // extra dracut args needed to assemble it
+  fstype?: string;
+  size_gb?: number;
+  os_release?: string;  // PRETTY_NAME from /etc/os-release, if mountable
 }
 
 export type Role = "vanilla" | "worker" | "infra" | "labcontroller";
@@ -103,6 +130,11 @@ export interface InstalledInfo {
   cpu_cores?: number;
   memory_gb?: number;
   arch?: string;
+  onboard?: OnboardMethod;
+  vendor_os?: string;
+  root_device?: string;
+  root_args?: string;
+  root_candidates?: RootCandidate[];
 }
 
 export interface DebugConfig {

bastion/tests/integration/arm-pxe-provision.test.ts

@@ -0,0 +1,537 @@
+// Integration test: aarch64 network PXE boot.
+//
+// The boot-ISO path already covered ARM (arm-iso-provision.test.ts). This covers the
+// network path: DHCP option 93 handing an arm64 client an arm64 iPXE binary, dispatch
+// serving an aarch64 kernel, and `provision debug` reaching a rescue shell -- which is
+// what the DGX Sparks actually need and could not do.
+//
+// Two suites, because they cost very different amounts of time:
+//
+//   "ARM PXE rescue"   NBP handoff -> rescue with SSH.                    ~25-30 min
+//   "ARM PXE install"  discover -> install -> installed.                  ~75-95 min
+//
+// The rescue suite seeds the machine into state as an already-known aarch64 box rather
+// than discovering it first. That is the DGX Spark situation exactly -- SSH-onboarded,
+// never PXE-discovered, architecture known only from its record -- and it holds the test
+// to one emulated boot. Each boot spends ~15 of its ~18 minutes downloading Anaconda's
+// stage2 under TCG, so discovering first would double the runtime without touching any
+// code path the rescue boot does not already exercise.
+//
+// The install suite only runs with ARM_PXE_FULL=1. No ARM machine in the lab is ever
+// PXE-installed except the MS-R1, and an hour-plus test that runs by default is a test
+// nobody runs.
+//
+// IMPORTANT: aarch64 has no KVM on an x86_64 host, so all of this is emulated and
+// roughly 10x slower than native.
+//
+// A note for whoever debugs a failure here: if the VM panics with
+//   VFS: Unable to mount root fs on unknown-block(0,0)
+// that is very likely iPXE silently dropping the initrd because the build lacks
+// EFI_LOAD_FILE2_PROTOCOL -- on arm64 the kernel EFI stub fetches the initrd over
+// LoadFile2, and an iPXE without it accepts the `initrd` line and does nothing. It is
+// NOT a reproduction of the DGX Spark kernel bug that motivated this work, despite
+// being the identical message. assertIpxeSupportsLoadFile2() below checks the build up
+// front so that failure names itself; to check by hand:
+//   node -e 'const b=require("fs").readFileSync("/usr/share/ipxe/arm64-efi/snponly.efi");
+//            console.log(b.indexOf(Buffer.from("c1c00640b3fc3e40996d4a6c8724e06d","hex")))'
+// Fedora's ipxe-bootimgs-aarch64-20240119 has it at 0x3bbf0.
+//
+// Prerequisites:
+//   - qemu-system-aarch64        (sudo dnf install qemu-system-aarch64)
+//   - edk2-aarch64               (sudo dnf install edk2-aarch64)
+//   - ipxe-bootimgs-aarch64      (sudo dnf install ipxe-bootimgs-aarch64)
+//   - libvirtd, sudo, internet access
+//
+// Run: sudo ./scripts/test-provision.sh arm-pxe
+
+import { describe, it, expect, beforeAll, afterAll } from "vitest";
+import { readFileSync, existsSync, mkdirSync, rmSync, copyFileSync, writeFileSync } from "node:fs";
+import { execSync } from "node:child_process";
+import { join } from "node:path";
+import { homedir, tmpdir } from "node:os";
+import { log, waitForSsh } from "./helpers/libvirt.js";
+import { ensurePxeNetwork, destroyPxeNetwork, deleteNftablesRejectRules, PXE_NETWORK_NAME, PXE_GATEWAY, PXE_SUBNET } from "./helpers/pxe-network.js";
+import { createPxeVm, destroyPxeVm, getVmMac, rebootPxeVm, readSerialLog } from "./helpers/pxe-vm.js";
+import { sshExec } from "./helpers/ssh.js";
+
+const IPXE_ARM64 = "/usr/share/ipxe/arm64-efi/snponly.efi";
+const AAVMF = "/usr/share/edk2/aarch64/QEMU_EFI.fd";
+
+const VM_MEMORY = 4096;
+const VM_VCPUS = 2;
+const VM_DISK_GB = 250;
+const SSH_USER = "lab";
+const BASTION_IP = PXE_GATEWAY;
+const DHCP_RANGE_START = `${PXE_SUBNET}.100`;
+const DHCP_RANGE_END = `${PXE_SUBNET}.200`;
+const SERIAL_PORT = 4555;
+
+// Emulated aarch64 -- generous timeouts throughout. Measured on an x86_64 host with no
+// KVM for aarch64: a single PXE boot to a running Anaconda takes ~18 minutes, almost all
+// of it downloading inst.stage2 over the network under TCG. Budget well above that;
+// timing out just short of success wastes a whole run.
+const LEASE_TIMEOUT_MS = 10 * 60_000;
+const DISCOVERY_TIMEOUT_MS = 35 * 60_000;
+const INSTALL_TIMEOUT_MS = 75 * 60_000;
+const SSH_TIMEOUT_MS = 35 * 60_000;
+
+const RUN_FULL_INSTALL = process.env["ARM_PXE_FULL"] === "1";
+
+function sleep(ms: number): Promise<void> {
+  return new Promise((r) => setTimeout(r, ms));
+}
+
+function findSshKey(): { pubKey: string; keyPath: string } {
+  const candidates: string[] = [];
+  if (process.env["SSH_KEY_PATH"]) candidates.push(process.env["SSH_KEY_PATH"]);
+  const homes = [homedir()];
+  const sudoUser = process.env["SUDO_USER"];
+  if (sudoUser) homes.push(join("/home", sudoUser));
+  for (const home of homes) {
+    for (const name of ["id_ed25519", "id_ecdsa", "id_rsa"]) {
+      candidates.push(join(home, ".ssh", name));
+    }
+  }
+  for (const keyPath of candidates) {
+    if (existsSync(keyPath) && existsSync(`${keyPath}.pub`)) {
+      return { pubKey: readFileSync(`${keyPath}.pub`, "utf-8").trim(), keyPath };
+    }
+  }
+  throw new Error("No SSH key found — set SSH_KEY_PATH or ensure keys exist in ~/.ssh/");
+}
+
+async function pollApi<T>(
+  url: string,
+  check: (data: T) => boolean,
+  timeoutMs: number,
+  intervalMs = 10_000,
+): Promise<T> {
+  const start = Date.now();
+  while (Date.now() - start < timeoutMs) {
+    try {
+      const res = await fetch(url);
+      if (res.ok) {
+        const data = (await res.json()) as T;
+        if (check(data)) return data;
+      }
+    } catch { /* bastion not up yet, or a network hiccup */ }
+    await sleep(intervalMs);
+  }
+  throw new Error(`Timeout after ${timeoutMs}ms polling ${url}`);
+}
+
+function requirePrerequisites(): void {
+  if (!existsSync("/usr/bin/qemu-system-aarch64")) {
+    throw new Error("qemu-system-aarch64 not installed. Run: sudo dnf install qemu-system-aarch64");
+  }
+  if (!existsSync(AAVMF)) {
+    throw new Error(`AAVMF firmware not found at ${AAVMF}. Run: sudo dnf install edk2-aarch64`);
+  }
+  if (!existsSync(IPXE_ARM64)) {
+    throw new Error(`arm64 iPXE not found at ${IPXE_ARM64}. Run: sudo dnf install ipxe-bootimgs-aarch64`);
+  }
+}
+
+/**
+ * Confirm the arm64 iPXE binary implements EFI_LOAD_FILE2_PROTOCOL.
+ *
+ * Without it the `initrd` line is accepted and silently ignored, and the kernel panics
+ * with unknown-block(0,0). Checking here turns a confusing 30-minute boot failure into
+ * an immediate, explanatory one.
+ *
+ * GUID 4006c0c1-fcb3-403e-996d-4a6c8724e06d, little-endian in the binary's GUID table.
+ */
+function assertIpxeSupportsLoadFile2(): void {
+  const LOAD_FILE2_GUID = Buffer.from("c1c00640b3fc3e40996d4a6c8724e06d", "hex");
+  const binary = readFileSync(IPXE_ARM64);
+  if (binary.indexOf(LOAD_FILE2_GUID) < 0) {
+    throw new Error(
+      `${IPXE_ARM64} does not reference EFI_LOAD_FILE2_PROTOCOL. On arm64 the kernel ` +
+      `EFI stub fetches the initrd over LoadFile2; without it iPXE drops the initrd ` +
+      `silently and the kernel panics with "unknown-block(0,0)". Rebuild iPXE with ` +
+      `LoadFile2, or chainload grubaa64.efi for aarch64 instead.`,
+    );
+  }
+  log(`iPXE arm64 implements LoadFile2 — initrd will be delivered to the EFI stub`);
+}
+
+interface Harness {
+  testDir: string;
+  app: { close: () => Promise<void> };
+  stopDnsmasq: () => void;
+  state: { update: (fn: (s: BastionStateLike) => void) => void };
+  vmMac: string;
+  httpPort: number;
+}
+
+/** Just the parts of BastionState this test seeds. */
+interface BastionStateLike {
+  discovered: Record<string, Record<string, unknown>>;
+  installed: Record<string, Record<string, unknown>>;
+  install_queue: Record<string, Record<string, unknown>>;
+  debug: Record<string, Record<string, unknown>>;
+}
+
+/** Bring up an isolated network, a bastion with both arch payloads, and an arm64 VM. */
+async function startHarness(vmName: string, httpPort: number, pubKey: string): Promise<Harness> {
+  requirePrerequisites();
+  assertIpxeSupportsLoadFile2();
+
+  log("Setting up PXE test network...");
+  ensurePxeNetwork();
+
+  const testDir = join(tmpdir(), `lab-arm-pxe-test-${Date.now()}`);
+  for (const sub of ["tftp", "http", "logs"]) {
+    mkdirSync(join(testDir, sub), { recursive: true });
+  }
+
+  const { createApp } = await import("../../src/bastion/src/server.js");
+  const { loadConfig } = await import("../../src/bastion/src/config.js");
+  const { generateDnsmasqConf, startDnsmasq, stopDnsmasq } = await import("../../src/bastion/src/services/dnsmasq.js");
+  const { generateDiscoverKickstart } = await import("../../src/bastion/src/services/kickstart-generator.js");
+  const { renderBootIpxe, kernelPath, initrdPath } = await import("../../src/bastion/src/templates/boot.ipxe.js");
+  // Relative, not "@lab/shared": these tests run from the repo root against sources,
+  // where the workspace package alias is not resolvable.
+  const { SUPPORTED_ARCHES, fedoraMirrorFor } = await import("../../src/shared/src/hardware/index.js");
+
+  const config = loadConfig({
+    bastionDir: testDir,
+    httpPort,
+    iface: "virbr-pxe",
+    serverIp: BASTION_IP,
+    network: `${PXE_SUBNET}.0`,
+    gateway: BASTION_IP,
+    dhcpMode: "full",
+    dhcpRangeStart: DHCP_RANGE_START,
+    dhcpRangeEnd: DHCP_RANGE_END,
+    domain: "arm-pxe-test.local",
+    sshKeys: [pubKey],
+    adminUser: SSH_USER,
+  });
+
+  // iPXE binaries. The arm64 one is the whole point: dnsmasq hands it out on DHCP
+  // option 93 -- 11 for UEFI PXE (TFTP) and 19 for UEFI HTTP Boot.
+  //
+  // They go in BOTH directories, exactly as main.ts stages them. AAVMF prefers HTTP
+  // Boot, so it is served an http:// URL and fetches from httpDir; a firmware that
+  // takes the TFTP path reads the same file from tftpDir. Staging only tftpDir gives a
+  // 404 and "No bootable option or device was found" on the console.
+  log("Staging iPXE binaries...");
+  const ipxeX86 = "/usr/share/ipxe/ipxe-snponly-x86_64.efi";
+  copyFileSync(IPXE_ARM64, join(config.tftpDir, "ipxe-arm64.efi"));
+  copyFileSync(IPXE_ARM64, join(config.httpDir, "ipxe-arm64.efi"));
+  if (existsSync(ipxeX86)) {
+    copyFileSync(ipxeX86, join(config.tftpDir, "ipxe.efi"));
+    copyFileSync(ipxeX86, join(config.httpDir, "ipxe.efi"));
+  }
+
+  // Fedora kernel + initrd for both architectures, cached across runs.
+  const cacheDir = "/var/lib/libvirt/images/lab-pxe-cache";
+  execSync(`mkdir -p "${cacheDir}"`, { stdio: "pipe" });
+
+  for (const arch of SUPPORTED_ARCHES) {
+    const mirror = fedoraMirrorFor(config.fedoraVersion, arch);
+    const kernelCache = join(cacheDir, `vmlinuz-${arch}`);
+    const initrdCache = join(cacheDir, `initrd-${arch}.img`);
+
+    if (!existsSync(kernelCache)) {
+      log(`Downloading Fedora ${config.fedoraVersion} ${arch} kernel...`);
+      execSync(`curl -# -L -f -o "${kernelCache}" "${mirror}/images/pxeboot/vmlinuz"`, { stdio: "inherit", timeout: 600_000 });
+    }
+    if (!existsSync(initrdCache)) {
+      log(`Downloading Fedora ${config.fedoraVersion} ${arch} initrd...`);
+      execSync(`curl -# -L -f -o "${initrdCache}" "${mirror}/images/pxeboot/initrd.img"`, { stdio: "inherit", timeout: 600_000 });
+    }
+
+    // Staged under the exact names the iPXE templates will ask for.
+    copyFileSync(kernelCache, join(config.httpDir, kernelPath(arch)));
+    copyFileSync(initrdCache, join(config.httpDir, initrdPath(arch)));
+    log(`Staged ${arch}: ${kernelPath(arch)} + ${initrdPath(arch)}`);
+  }
+
+  writeFileSync(join(config.httpDir, "discover.ks"), generateDiscoverKickstart(config));
+  writeFileSync(
+    join(config.httpDir, "boot.ipxe"),
+    renderBootIpxe({ serverIp: config.serverIp, httpPort: config.httpPort }),
+  );
+  generateDnsmasqConf(config);
+
+  const { app, state, syslog } = createApp(config);
+  await app.listen({ port: config.httpPort, host: "0.0.0.0" });
+  syslog.start();
+  log(`Bastion HTTP listening on :${config.httpPort}`);
+
+  log("Starting dnsmasq (full DHCP)...");
+  startDnsmasq(config).catch((err) => {
+    log(`dnsmasq failed: ${err instanceof Error ? err.message : String(err)}`);
+  });
+  await sleep(1500);
+
+  log("Creating aarch64 PXE VM (emulated — this is slow)...");
+  createPxeVm({
+    name: vmName,
+    memory: VM_MEMORY,
+    vcpus: VM_VCPUS,
+    diskSize: VM_DISK_GB,
+    network: PXE_NETWORK_NAME,
+    arch: "aarch64",
+  });
+
+  const vmMac = getVmMac(vmName);
+  if (!vmMac) throw new Error("Could not determine VM MAC address");
+  log(`ARM VM MAC: ${vmMac}`);
+
+  return {
+    testDir,
+    app,
+    stopDnsmasq,
+    state: state as unknown as Harness["state"],
+    vmMac,
+    httpPort: config.httpPort,
+  };
+}
+
+async function stopHarness(vmName: string, harness: Harness | undefined): Promise<void> {
+  // KEEP_VM=1 leaves the VM, network and bastion up so a failure can be inspected on
+  // the console. Emulated aarch64 runs cost half an hour; tearing the evidence down
+  // automatically means paying that again to see what happened.
+  if (process.env["KEEP_VM"] === "1") {
+    log(`KEEP_VM=1 — leaving ${vmName} running for inspection.`);
+    log(`  console:  sudo virsh screenshot ${vmName} /tmp/vm.ppm`);
+    log(`  serial:   socat - TCP:127.0.0.1:${SERIAL_PORT}`);
+    if (harness) log(`  bastion:  ${harness.testDir} (still serving on :${harness.httpPort})`);
+    log(`  cleanup:  sudo virsh destroy ${vmName}; sudo virsh undefine ${vmName} --remove-all-storage --nvram`);
+    return;
+  }
+
+  log("Cleaning up...");
+  if (harness) {
+    await harness.app.close().catch(() => {});
+    harness.stopDnsmasq();
+  }
+  destroyPxeVm(vmName);
+  destroyPxeNetwork();
+  if (harness) rmSync(harness.testDir, { recursive: true, force: true });
+}
+
+/** Read the DHCP lease the bastion handed a MAC. Rescue mode reports no IP itself. */
+function leaseIpFor(testDir: string, mac: string): string | null {
+  const leaseFile = join(testDir, "dnsmasq.leases");
+  if (!existsSync(leaseFile)) return null;
+  for (const line of readFileSync(leaseFile, "utf-8").split("\n")) {
+    // <expiry> <mac> <ip> <hostname> <clientid>
+    const parts = line.trim().split(/\s+/);
+    if (parts.length >= 3 && parts[1]?.toLowerCase() === mac.toLowerCase()) {
+      return parts[2] ?? null;
+    }
+  }
+  return null;
+}
+
+async function waitForLease(testDir: string, mac: string, timeoutMs: number): Promise<string> {
+  const start = Date.now();
+  while (Date.now() - start < timeoutMs) {
+    const ip = leaseIpFor(testDir, mac);
+    if (ip !== null) return ip;
+    await sleep(5000);
+  }
+  throw new Error(`No DHCP lease for ${mac} within ${timeoutMs}ms`);
+}
+
+// ---------------------------------------------------------------------------
+// Rescue path -- what the DGX Sparks need.
+// ---------------------------------------------------------------------------
+
+describe("ARM PXE rescue", () => {
+  const VM_NAME = "lab-arm-pxe-rescue";
+  const HTTP_PORT = 8096;
+  let harness: Harness | undefined;
+  let sshKeyPath: string;
+  let rescueIp: string;
+
+  beforeAll(async () => {
+    const { pubKey, keyPath } = findSshKey();
+    sshKeyPath = keyPath;
+    harness = await startHarness(VM_NAME, HTTP_PORT, pubKey);
+    const { testDir, vmMac, state } = harness;
+
+    // Seed the machine as an already-known aarch64 box queued for rescue. This is the
+    // DGX Spark situation exactly: SSH-onboarded, never PXE-discovered, architecture
+    // known only from its record -- and it also keeps the test to a SINGLE emulated
+    // boot. Each boot spends ~15 minutes pulling Anaconda's stage2 over the network
+    // under TCG, so discovering first and rescuing second doubles the runtime for no
+    // extra coverage of the path being tested. Discovery is covered by the full suite.
+    log(`Seeding ${vmMac} as a known aarch64 machine queued for rescue...`);
+    state.update((s) => {
+      s.discovered[vmMac] = {
+        mac: vmMac,
+        product: "Test ARM64 Machine",
+        board: "virt",
+        serial: "SN-ARM64",
+        manufacturer: "QEMU",
+        cpu_model: "cortex-a57",
+        cpu_cores: VM_VCPUS,
+        memory_gb: 4,
+        arch: "aarch64",
+        disks: [],
+        nics: [],
+        first_seen: new Date().toISOString(),
+        last_seen: new Date().toISOString(),
+      };
+      s.debug[vmMac] = { hostname: "arm-rescue-test", queued_at: new Date().toISOString() };
+    });
+
+    // Restart so the VM boots against the seeded state. createPxeVm already started it.
+    rebootPxeVm(VM_NAME);
+    await sleep(5_000);
+    deleteNftablesRejectRules();
+
+    // The whole chain now runs once: DHCP option 93 -> arm64 iPXE -> /boot.ipxe ->
+    // /dispatch (architecture from the record, not the query) -> aarch64 kernel +
+    // initrd -> Anaconda rescue -> sshd. Reaching a shell at all proves iPXE handed
+    // the initrd to the EFI stub over LoadFile2; without it the kernel panics first.
+    log("Waiting for the rescue environment's DHCP lease...");
+    rescueIp = await waitForLease(testDir, vmMac, LEASE_TIMEOUT_MS);
+    log(`Rescue IP: ${rescueIp}`);
+
+    log("Waiting for SSH into the rescue shell (started by inst.sshd)...");
+    log("(emulated aarch64 — Anaconda's stage2 download dominates; be patient)");
+    await waitForSsh(rescueIp, "root", SSH_TIMEOUT_MS, sshKeyPath).catch(async (err) => {
+      log("Rescue SSH timed out. Serial console:");
+      try {
+        log(await readSerialLog(SERIAL_PORT, { lastLines: 100, timeoutMs: 15_000 }));
+      } catch { /* console unavailable */ }
+      throw err;
+    });
+    log("ARM PXE rescue reached.");
+  }, LEASE_TIMEOUT_MS + SSH_TIMEOUT_MS + 300_000);
+
+  afterAll(async () => { await stopHarness(VM_NAME, harness); });
+
+  it("resolved the architecture from the machine record", async () => {
+    const res = await fetch(`http://${BASTION_IP}:${HTTP_PORT}/api/machines`);
+    const data = (await res.json()) as { discovered: Record<string, { arch: string }> };
+    expect(data.discovered[harness!.vmMac]?.arch).toBe("aarch64");
+  });
+
+  it("rescue shell is reachable over SSH and is aarch64", () => {
+    const result = sshExec(rescueIp, "root", "uname -m", { keyPath: sshKeyPath, timeout: 60_000 });
+    expect(result.exitCode).toBe(0);
+    expect(result.stdout.trim()).toBe("aarch64");
+  });
+
+  it("booted an initramfs — the LoadFile2 path worked", () => {
+    // If iPXE had dropped the initrd the kernel would never have reached userspace at
+    // all, but assert it explicitly so a regression names itself.
+    const result = sshExec(rescueIp, "root", "cat /proc/cmdline; ls /run/install", {
+      keyPath: sshKeyPath, timeout: 60_000,
+    });
+    expect(result.exitCode).toBe(0);
+    expect(result.stdout).toContain("inst.rescue");
+  });
+
+  it("rescue kernel came from the bastion over HTTP", () => {
+    const result = sshExec(rescueIp, "root", "cat /proc/cmdline", { keyPath: sshKeyPath, timeout: 60_000 });
+    expect(result.stdout).toContain(`${BASTION_IP}:${HTTP_PORT}`);
+    // arm64 gets serial console arguments, never nomodeset.
+    expect(result.stdout).toContain("console=ttyAMA0");
+    expect(result.stdout).not.toContain("nomodeset");
+  });
+
+  it("has LVM tools available for inspecting an installed system", () => {
+    const result = sshExec(rescueIp, "root", "command -v vgchange && command -v lsblk", {
+      keyPath: sshKeyPath, timeout: 60_000,
+    });
+    expect(result.exitCode).toBe(0);
+  });
+});
+
+// ---------------------------------------------------------------------------
+// Full install -- opt-in, ~60-90 minutes emulated.
+// ---------------------------------------------------------------------------
+
+describe.runIf(RUN_FULL_INSTALL)("ARM PXE install", () => {
+  const VM_NAME = "lab-arm-pxe-install";
+  const HTTP_PORT = 8095;
+  let harness: Harness | undefined;
+  let sshKeyPath: string;
+  let vmIp: string;
+
+  beforeAll(async () => {
+    const { pubKey, keyPath } = findSshKey();
+    sshKeyPath = keyPath;
+    harness = await startHarness(VM_NAME, HTTP_PORT, pubKey);
+    const { vmMac } = harness;
+
+    log("Waiting for aarch64 discovery...");
+    await pollApi<{ discovered: Record<string, unknown> }>(
+      `http://${BASTION_IP}:${HTTP_PORT}/api/machines`,
+      (data) => vmMac in data.discovered,
+      DISCOVERY_TIMEOUT_MS,
+    );
+    log("Discovered. Queueing install...");
+
+    const installRes = await fetch(`http://${BASTION_IP}:${HTTP_PORT}/api/install`, {
+      method: "POST",
+      headers: { "Content-Type": "application/json" },
+      body: JSON.stringify({ mac: vmMac, hostname: VM_NAME, disk: "", role: "vanilla" }),
+    });
+    expect(installRes.status).toBe(200);
+
+    await sleep(30_000);
+    rebootPxeVm(VM_NAME);
+
+    log("Waiting for the emulated aarch64 install (60-90 min)...");
+    type LogsResponse = { status: string; progress: string; ip?: string };
+    const final = await pollApi<LogsResponse>(
+      `http://${BASTION_IP}:${HTTP_PORT}/api/logs/${encodeURIComponent(vmMac)}`,
+      (d) => d.status === "installed" || d.progress === "error",
+      INSTALL_TIMEOUT_MS,
+      30_000,
+    );
+
+    if (final.progress === "error") {
+      const logs = await (await fetch(`http://${BASTION_IP}:${HTTP_PORT}/api/logs/${encodeURIComponent(vmMac)}`)).json();
+      log(`ARM install FAILED: ${JSON.stringify(logs, null, 2)}`);
+      throw new Error("ARM PXE install failed — see logs above");
+    }
+
+    vmIp = final.ip ?? "";
+    log(`ARM install complete. IP: ${vmIp}`);
+
+    await sleep(30_000);
+    rebootPxeVm(VM_NAME);
+    await sleep(5_000);
+    deleteNftablesRejectRules();
+    await waitForSsh(vmIp, SSH_USER, SSH_TIMEOUT_MS, sshKeyPath);
+  }, DISCOVERY_TIMEOUT_MS + INSTALL_TIMEOUT_MS + SSH_TIMEOUT_MS + 600_000);
+
+  afterAll(async () => { await stopHarness(VM_NAME, harness); });
+
+  it("machine reached installed state", async () => {
+    const res = await fetch(`http://${BASTION_IP}:${HTTP_PORT}/api/machines`);
+    const data = (await res.json()) as { installed: Record<string, { hostname: string }> };
+    expect(data.installed[harness!.vmMac]?.hostname).toBe(VM_NAME);
+  });
+
+  it("installed system is aarch64", () => {
+    const result = sshExec(vmIp, SSH_USER, "uname -m", { keyPath: sshKeyPath, timeout: 60_000 });
+    expect(result.stdout.trim()).toBe("aarch64");
+  });
+
+  it("SSH works with the admin user", () => {
+    const result = sshExec(vmIp, SSH_USER, "whoami", { keyPath: sshKeyPath, timeout: 60_000 });
+    expect(result.stdout.trim()).toBe(SSH_USER);
+  });
+
+  it("LVM layout is correct", () => {
+    const result = sshExec(vmIp, SSH_USER, "sudo lvs labvg --noheadings -o lv_name", {
+      keyPath: sshKeyPath, timeout: 60_000,
+    });
+    expect(result.exitCode).toBe(0);
+    const lvs = result.stdout.trim().split("\n").map((l) => l.trim());
+    for (const expected of ["root", "var", "varlog", "swap", "home", "srv"]) {
+      expect(lvs).toContain(expected);
+    }
+  });
+});

bastion/tests/integration/pxe-rescue.test.ts

@@ -0,0 +1,210 @@
+// Integration test: `labctl provision debug` -> Anaconda rescue with SSH, on x86_64.
+//
+// The rescue path had no test coverage on any architecture, which matters because it is
+// the lab's recovery tool of last resort -- the thing you reach for when a machine will
+// not boot. It runs here on x86_64 with KVM so it completes in minutes; the aarch64
+// equivalent is the same code path with a different kernel, but is emulated and far too
+// slow to iterate on.
+//
+// Run: sudo ./scripts/test-provision.sh rescue
+
+import { describe, it, expect, beforeAll, afterAll } from "vitest";
+import { readFileSync, existsSync, mkdirSync, rmSync, copyFileSync, writeFileSync } from "node:fs";
+import { execSync } from "node:child_process";
+import { join } from "node:path";
+import { homedir, tmpdir } from "node:os";
+import { log, waitForSsh } from "./helpers/libvirt.js";
+import { ensurePxeNetwork, destroyPxeNetwork, deleteNftablesRejectRules, PXE_NETWORK_NAME, PXE_GATEWAY, PXE_SUBNET } from "./helpers/pxe-network.js";
+import { createPxeVm, destroyPxeVm, getVmMac, rebootPxeVm, readSerialLog } from "./helpers/pxe-vm.js";
+import { sshExec } from "./helpers/ssh.js";
+
+const VM_NAME = "lab-pxe-rescue-test";
+const HTTP_PORT = 8094;
+const VM_MEMORY = 4096;
+const VM_VCPUS = 4;
+const VM_DISK_GB = 20;
+const BASTION_IP = PXE_GATEWAY;
+const SERIAL_PORT = 4555;
+
+const LEASE_TIMEOUT_MS = 8 * 60_000;
+const SSH_TIMEOUT_MS = 15 * 60_000;
+
+function sleep(ms: number): Promise<void> {
+  return new Promise((r) => setTimeout(r, ms));
+}
+
+function findSshKey(): { pubKey: string; keyPath: string } {
+  const candidates: string[] = [];
+  if (process.env["SSH_KEY_PATH"]) candidates.push(process.env["SSH_KEY_PATH"]);
+  const homes = [homedir()];
+  const sudoUser = process.env["SUDO_USER"];
+  if (sudoUser) homes.push(join("/home", sudoUser));
+  for (const home of homes) {
+    for (const name of ["id_ed25519", "id_ecdsa", "id_rsa"]) candidates.push(join(home, ".ssh", name));
+  }
+  for (const keyPath of candidates) {
+    if (existsSync(keyPath) && existsSync(`${keyPath}.pub`)) {
+      return { pubKey: readFileSync(`${keyPath}.pub`, "utf-8").trim(), keyPath };
+    }
+  }
+  throw new Error("No SSH key found — set SSH_KEY_PATH or ensure keys exist in ~/.ssh/");
+}
+
+function leaseIpFor(testDir: string, mac: string): string | null {
+  const leaseFile = join(testDir, "dnsmasq.leases");
+  if (!existsSync(leaseFile)) return null;
+  for (const line of readFileSync(leaseFile, "utf-8").split("\n")) {
+    const parts = line.trim().split(/\s+/);
+    if (parts.length >= 3 && parts[1]?.toLowerCase() === mac.toLowerCase()) return parts[2] ?? null;
+  }
+  return null;
+}
+
+async function waitForLease(testDir: string, mac: string, timeoutMs: number): Promise<string> {
+  const start = Date.now();
+  while (Date.now() - start < timeoutMs) {
+    const ip = leaseIpFor(testDir, mac);
+    if (ip !== null) return ip;
+    await sleep(5000);
+  }
+  throw new Error(`No DHCP lease for ${mac} within ${timeoutMs}ms`);
+}
+
+// Suite name must not be a substring of "ARM PXE rescue" -- vitest -t matches
+// substrings, so a looser name here would drag the emulated aarch64 suite in with it.
+describe("x86 rescue boot", () => {
+  let app: { close: () => Promise<void> };
+  let stopDnsmasqFn: () => void;
+  let testDir: string;
+  let vmMac: string;
+  let rescueIp: string;
+  let sshKeyPath: string;
+
+  beforeAll(async () => {
+    const { pubKey, keyPath } = findSshKey();
+    sshKeyPath = keyPath;
+
+    log("Setting up PXE test network...");
+    ensurePxeNetwork();
+
+    testDir = join(tmpdir(), `lab-pxe-rescue-${Date.now()}`);
+    for (const sub of ["tftp", "http", "logs"]) mkdirSync(join(testDir, sub), { recursive: true });
+
+    const { createApp } = await import("../../src/bastion/src/server.js");
+    const { loadConfig } = await import("../../src/bastion/src/config.js");
+    const { generateDnsmasqConf, startDnsmasq, stopDnsmasq } = await import("../../src/bastion/src/services/dnsmasq.js");
+    const { renderBootIpxe, kernelPath, initrdPath } = await import("../../src/bastion/src/templates/boot.ipxe.js");
+    stopDnsmasqFn = stopDnsmasq;
+
+    const config = loadConfig({
+      bastionDir: testDir,
+      httpPort: HTTP_PORT,
+      iface: "virbr-pxe",
+      serverIp: BASTION_IP,
+      network: `${PXE_SUBNET}.0`,
+      gateway: BASTION_IP,
+      dhcpMode: "full",
+      dhcpRangeStart: `${PXE_SUBNET}.100`,
+      dhcpRangeEnd: `${PXE_SUBNET}.200`,
+      domain: "rescue-test.local",
+      sshKeys: [pubKey],
+      adminUser: "lab",
+    });
+
+    // iPXE in both dirs: TFTP PXE and UEFI HTTP Boot are both possible, and OVMF picks.
+    const ipxeX86 = "/usr/share/ipxe/ipxe-snponly-x86_64.efi";
+    if (!existsSync(ipxeX86)) throw new Error(`iPXE not found: ${ipxeX86}`);
+    copyFileSync(ipxeX86, join(config.tftpDir, "ipxe.efi"));
+    copyFileSync(ipxeX86, join(config.httpDir, "ipxe.efi"));
+
+    const cacheDir = "/var/lib/libvirt/images/lab-pxe-cache";
+    execSync(`mkdir -p "${cacheDir}"`, { stdio: "pipe" });
+    const kernelCache = join(cacheDir, "vmlinuz-x86_64");
+    const initrdCache = join(cacheDir, "initrd-x86_64.img");
+    if (!existsSync(kernelCache)) {
+      log("Downloading Fedora x86_64 kernel...");
+      execSync(`curl -# -L -f -o "${kernelCache}" "${config.fedoraMirror}/images/pxeboot/vmlinuz"`, { stdio: "inherit", timeout: 600_000 });
+    }
+    if (!existsSync(initrdCache)) {
+      log("Downloading Fedora x86_64 initrd...");
+      execSync(`curl -# -L -f -o "${initrdCache}" "${config.fedoraMirror}/images/pxeboot/initrd.img"`, { stdio: "inherit", timeout: 600_000 });
+    }
+    copyFileSync(kernelCache, join(config.httpDir, kernelPath("x86_64")));
+    copyFileSync(initrdCache, join(config.httpDir, initrdPath("x86_64")));
+
+    writeFileSync(join(config.httpDir, "boot.ipxe"), renderBootIpxe({ serverIp: config.serverIp, httpPort: config.httpPort }));
+    generateDnsmasqConf(config);
+
+    const { app: fastify, state, syslog } = createApp(config);
+    app = fastify;
+    await fastify.listen({ port: config.httpPort, host: "0.0.0.0" });
+    syslog.start();
+    log(`Bastion HTTP listening on :${HTTP_PORT}`);
+
+    startDnsmasq(config).catch((err) => log(`dnsmasq failed: ${err instanceof Error ? err.message : String(err)}`));
+    await sleep(1500);
+
+    log("Creating x86_64 PXE VM (KVM)...");
+    createPxeVm({ name: VM_NAME, memory: VM_MEMORY, vcpus: VM_VCPUS, diskSize: VM_DISK_GB, network: PXE_NETWORK_NAME });
+    const mac = getVmMac(VM_NAME);
+    if (!mac) throw new Error("Could not determine VM MAC");
+    vmMac = mac;
+    log(`VM MAC: ${vmMac}`);
+
+    // Queue rescue directly, as `labctl provision debug` does.
+    log("Queueing debug/rescue mode...");
+    state.update((s) => {
+      s.debug[vmMac] = { hostname: "rescue-test", queued_at: new Date().toISOString() };
+    });
+
+    rebootPxeVm(VM_NAME);
+    await sleep(5_000);
+    deleteNftablesRejectRules();
+
+    rescueIp = await waitForLease(testDir, vmMac, LEASE_TIMEOUT_MS);
+    log(`Rescue IP: ${rescueIp}`);
+
+    log("Waiting for SSH into the rescue shell (inst.sshd)...");
+    await waitForSsh(rescueIp, "root", SSH_TIMEOUT_MS, sshKeyPath).catch(async (err) => {
+      log("Rescue SSH timed out. Serial console:");
+      try { log(await readSerialLog(SERIAL_PORT, { lastLines: 120, timeoutMs: 20_000 })); } catch { /* none */ }
+      throw err;
+    });
+    log("Rescue shell reachable.");
+  }, LEASE_TIMEOUT_MS + SSH_TIMEOUT_MS + 300_000);
+
+  afterAll(async () => {
+    if (process.env["KEEP_VM"] === "1") {
+      log(`KEEP_VM=1 — leaving ${VM_NAME} up (serial: socat - TCP:127.0.0.1:${SERIAL_PORT})`);
+      return;
+    }
+    log("Cleaning up...");
+    if (app) await app.close().catch(() => {});
+    if (stopDnsmasqFn) stopDnsmasqFn();
+    destroyPxeVm(VM_NAME);
+    destroyPxeNetwork();
+    if (testDir) rmSync(testDir, { recursive: true, force: true });
+  });
+
+  it("rescue shell is reachable over SSH as root", () => {
+    const result = sshExec(rescueIp, "root", "whoami", { keyPath: sshKeyPath, timeout: 60_000 });
+    expect(result.exitCode).toBe(0);
+    expect(result.stdout.trim()).toBe("root");
+  });
+
+  it("is the Anaconda rescue environment", () => {
+    const result = sshExec(rescueIp, "root", "cat /proc/cmdline", { keyPath: sshKeyPath, timeout: 60_000 });
+    expect(result.stdout).toContain("inst.rescue");
+    expect(result.stdout).toContain("inst.sshd");
+  });
+
+  it("kernel and initrd came from the bastion", () => {
+    const result = sshExec(rescueIp, "root", "cat /proc/cmdline", { keyPath: sshKeyPath, timeout: 60_000 });
+    expect(result.stdout).toContain(`${BASTION_IP}:${HTTP_PORT}`);
+  });
+
+  it("has LVM tools for inspecting an installed system", () => {
+    const result = sshExec(rescueIp, "root", "command -v vgchange && command -v lsblk", { keyPath: sshKeyPath, timeout: 60_000 });
+    expect(result.exitCode).toBe(0);
+  });
+});