docs(bastion): state the rescue-SSH evidence at its actual strength

The previous wording claimed the failure was "reproduced on x86_64", which
overstates it. The aarch64 observation is direct -- port 22 probed for 30
minutes with the Anaconda environment demonstrably up. The x86_64 run only
shows SSH not becoming available inside a 15-minute budget; that VM was
never observed reaching the rescue environment, because vitest's reporter
discards the streamed log.

The distinction decides where the next person looks, so it should not rest
on an inference presented as an observation.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015nRFZXpKwUVE4SRSHw6GjF

bastion/Dockerfile.labd

@@ -11,6 +11,7 @@ WORKDIR /app
 # Copy workspace config and package manifests first (layer cache)
 COPY pnpm-workspace.yaml pnpm-lock.yaml package.json tsconfig.base.json tsconfig.json ./
 COPY src/shared/package.json src/shared/tsconfig.json src/shared/
+COPY src/core/package.json   src/core/tsconfig.json   src/core/
 COPY src/labd/package.json   src/labd/tsconfig.json   src/labd/
 
 # Install all dependencies (dev included -- needed for build)
@@ -22,10 +23,13 @@ RUN pnpm --filter @lab/labd exec prisma generate
 
 # Copy source code
 COPY src/shared/src/ src/shared/src/
+COPY src/core/src/   src/core/src/
 COPY src/labd/src/   src/labd/src/
 
-# Build TypeScript (shared first via project references)
-RUN pnpm --filter @lab/shared build && pnpm --filter @lab/labd build
+# Build TypeScript (shared + core before labd via project references)
+RUN pnpm --filter @lab/shared build \
+ && pnpm --filter @lab/core build \
+ && pnpm --filter @lab/labd build
 
 # Hoist the generated Prisma client so stage 2 can COPY it from a stable path
 RUN mkdir -p /app/_prisma && \
@@ -41,6 +45,7 @@ WORKDIR /app
 # Copy workspace config and package manifests
 COPY pnpm-workspace.yaml pnpm-lock.yaml package.json ./
 COPY src/shared/package.json src/shared/
+COPY src/core/package.json   src/core/
 COPY src/labd/package.json   src/labd/
 
 # Install production dependencies only
@@ -48,6 +53,7 @@ RUN pnpm install --frozen-lockfile --prod 2>/dev/null || pnpm install --prod
 
 # Copy built output from builder
 COPY --from=builder /app/src/shared/dist/ src/shared/dist/
+COPY --from=builder /app/src/core/dist/   src/core/dist/
 COPY --from=builder /app/src/labd/dist/   src/labd/dist/
 
 # Copy Prisma schema + generated client into pnpm store location

bastion/docs/ARCHITECTURE.md

@@ -89,6 +89,83 @@ Side paths:
 
 ---
 
+## Multi-architecture PXE
+
+The bastion serves both `x86_64` and `aarch64` over the network. Nothing about this is
+operator-configured -- there is no `--arch` flag, by design.
+
+### How a client's architecture is decided
+
+1. **DHCP option 93** (Client System Architecture) picks the *bootloader*. dnsmasq matches
+   it and hands out a matching iPXE binary:
+
+   | Option 93 | Client | Served |
+   |---|---|---|
+   | `0` | x86 BIOS | `undionly.kpxe` (TFTP) |
+   | `7`, `9` | x64 UEFI | `ipxe.efi` (TFTP) |
+   | `11` | **ARM64 UEFI** | `ipxe-arm64.efi` (TFTP) |
+   | `16` | x64 UEFI HTTP Boot | `http://…/ipxe.efi` |
+   | `19` | **ARM64 UEFI HTTP Boot** | `http://…/ipxe-arm64.efi` |
+
+   Values come from the IANA Processor Architecture Types registry. Note `19`, not `20` --
+   `20` is *pc/at bios boot from http*. EDK2/AAVMF prefers HTTP Boot over TFTP PXE, so the
+   iPXE binaries are staged in **both** `tftpDir` and `httpDir` (symlinked by `main.ts`).
+
+2. **`/dispatch` picks the kernel.** Option 93 never reaches the HTTP endpoint, so
+   `boot.ipxe` passes iPXE's own `${buildarch}` as `?arch=`. `resolveArch()` prefers, in
+   order: the tracked machine record → the reported `?arch=` → the configured default.
+   The record wins because it is what we observed on the machine itself.
+
+### Artifact naming
+
+`x86_64` keeps the original unsuffixed paths so its rendered iPXE scripts are unchanged;
+everything else is suffixed. `kernelPath()` / `initrdPath()` in `templates/boot.ipxe.ts`
+are the single source of truth, used by both the templates and `main.ts` staging.
+
+| arch | kernel | initrd |
+|---|---|---|
+| `x86_64` | `/vmlinuz` | `/initrd.img` |
+| `aarch64` | `/vmlinuz-aarch64` | `/initrd-aarch64.img` |
+
+`tests/ipxe-x86-regression.test.ts` pins the x86_64 output against a golden fixture.
+
+### arm64 gotchas
+
+- **LoadFile2 is mandatory.** arm64 has no `HdrS` boot protocol; the kernel's EFI stub
+  fetches the initrd over the UEFI `EFI_LOAD_FILE2_PROTOCOL`. An iPXE build without it
+  accepts the `initrd` line, silently drops it, and the kernel panics with
+  `VFS: Unable to mount root fs on unknown-block(0,0)`. Fedora's
+  `ipxe-bootimgs-aarch64` implements it; the integration test asserts this up front so
+  the failure names itself instead of looking like a disk problem.
+- **`nomodeset` is x86-only.** On arm64 there is no VGA path to fall back to. aarch64 gets
+  `console=tty0 console=ttyAMA0,115200` instead — the last `console=` wins for
+  `/dev/console`, so serial is the interactive one.
+- **Ubuntu is x86_64-only.** `releases.ubuntu.com` publishes no arm64 netboot artifacts.
+  `osSupportsArch()` encodes this, and both the install guard and `/dispatch` refuse the
+  combination rather than serving an x86 kernel to an ARM machine.
+
+---
+
+## Onboarding classification (vendor OS)
+
+Machines carry an `onboard` field: `"pxe"` (default) or `"ssh"`, plus `vendor_os` naming
+what they run. `classifyOnboard()` in `@lab/shared` sets it from DMI identity, with known
+hardware also matched by MAC — a machine can sit in state for a long time with no DMI, and
+a DMI-only rule would fail open exactly where it matters.
+
+`onboard: "ssh"` means *we cannot rebuild this machine's OS*. Installs are refused at both
+entry points (`/api/install` and the labd `command-install` handler) with an error naming
+the machine and pointing at `provision debug`. **Rescue is never guarded** — being unable
+to reinstall a machine is precisely when a rescue shell is needed.
+
+This is a fact about the machine, not a blocklist. The refusal follows from "no image in
+our pipeline restores `vendor_os`", so adding a DGX OS image to the pipeline is what
+unblocks the DGX Sparks — no entry needs deleting.
+
+Current classifications: NVIDIA DGX Spark (`spark-2935`, `spark-3a1c`) → `dgx-os`.
+
+---
+
 ## Packages
 
 ### Monorepo Structure
@@ -404,6 +481,36 @@ Hardcoded `/dev/sda` default broke NVMe-only machines. Fix: default to empty str
 ### Anaconda Rescue Mode Limitations
 `%pre` and `%post` sections do not execute in `inst.rescue` mode. SSH in rescue mode is provided by Anaconda's `inst.sshd` kernel parameter + `sshpw` kickstart directive. Manual setup via `curl bastion:8080/debug-setup.sh | bash` for nc listener.
 
+**Unresolved (2026-08-11): rescue SSH has never been observed working.** Adding the first
+integration coverage for `provision debug` (`tests/integration/pxe-rescue.test.ts`) showed the
+rescue environment coming up correctly — the bastion serves the kernel and initrd, Anaconda
+boots, fetches `debug.ks`, and reaches its installer environment — but **nothing ever listens on
+port 22**.
+
+Strength of the evidence, stated precisely because it decides where to look next:
+- **aarch64 — direct.** Port 22 probed every 20s for 30 minutes while the Anaconda installer
+  environment was demonstrably running (NetworkManager, polkitd, rsyslog on the console). Never
+  opened.
+- **x86_64 — corroborating, not conclusive.** One clean KVM run (943s) where SSH never became
+  available inside a 15-minute budget. That VM's progress into the rescue environment was *not*
+  observed — vitest's final reporter discards the streamed log — so it is consistent with the
+  aarch64 result but does not independently prove it. Re-run with `KEEP_VM=1` and probe port 22
+  directly to settle it.
+
+If the x86_64 result holds up, this is orthogonal to the multi-architecture work, since x86_64 is
+untouched by it. Leads worth checking, in order:
+- Does `inst.sshd` actually start `sshd` in `inst.rescue` mode, or only in install mode? The
+  port never opens, so this is the prime suspect — an auth problem would still show an open port.
+- `sshkey` may apply only to the *installed* system, leaving the installer environment
+  password-only via `sshpw`. That would matter once sshd does listen: the test authenticates
+  key-only (`BatchMode=yes`).
+- The `%anaconda`-context directives in `debug.ks` may be skipped entirely when a kickstart is
+  supplied alongside `inst.rescue`.
+
+Until this is resolved, `provision debug` gets you a booted rescue environment on the console
+(including on arm64), but not an SSH shell. The `debug-setup.sh` nc-listener path is the
+documented workaround and is unaffected.
+
 ---
 
 ## Planned Work (Taskmaster)

bastion/package.json

@@ -23,6 +23,12 @@
     "test:integration:iso:host": "sudo -E $(which npx) vitest run -c tests/integration/vitest.config.ts -t 'ISO boot'",
     "test:integration:arm-iso": "vitest run -c tests/integration/vitest.config.ts -t 'ARM ISO'",
     "test:integration:arm-iso:host": "sudo -E $(which npx) vitest run -c tests/integration/vitest.config.ts -t 'ARM ISO'",
+    "test:integration:rescue": "vitest run -c tests/integration/vitest.config.ts -t 'x86 rescue boot'",
+    "test:integration:rescue:host": "sudo -E $(which npx) vitest run -c tests/integration/vitest.config.ts -t 'x86 rescue boot'",
+    "test:integration:arm-pxe": "vitest run -c tests/integration/vitest.config.ts -t 'ARM PXE rescue'",
+    "test:integration:arm-pxe:host": "sudo -E $(which npx) vitest run -c tests/integration/vitest.config.ts -t 'ARM PXE rescue'",
+    "test:integration:arm-pxe-full": "ARM_PXE_FULL=1 vitest run -c tests/integration/vitest.config.ts -t 'ARM PXE'",
+    "test:integration:arm-pxe-full:host": "sudo -E ARM_PXE_FULL=1 $(which npx) vitest run -c tests/integration/vitest.config.ts -t 'ARM PXE'",
     "test:integration:asahi": "vitest run -c tests/integration/vitest.config.ts -t 'asahi firstboot'",
     "test:integration:asahi:host": "sudo -E $(which npx) vitest run -c tests/integration/vitest.config.ts -t 'asahi firstboot'",
     "test:integration:asahi-validate": "vitest run -c tests/integration/vitest.config.ts -t 'asahi.*validation'",

bastion/scripts/test-provision.sh

@@ -2,16 +2,19 @@
 # Run PXE and/or ISO boot integration tests.
 #
 # Usage:
-#   sudo ./scripts/test-provision.sh          # run PXE + ISO (x86_64)
-#   sudo ./scripts/test-provision.sh pxe      # PXE only
-#   sudo ./scripts/test-provision.sh iso      # ISO only (x86_64)
-#   sudo ./scripts/test-provision.sh arm      # ARM ISO boot (emulated, SLOW ~60min)
-#   sudo ./scripts/test-provision.sh all      # all tests including ARM
+#   sudo ./scripts/test-provision.sh              # run PXE + ISO (x86_64)
+#   sudo ./scripts/test-provision.sh pxe          # PXE only
+#   sudo ./scripts/test-provision.sh iso          # ISO only (x86_64)
+#   sudo ./scripts/test-provision.sh rescue       # x86_64 Anaconda rescue boot + SSH (~15min)
+#   sudo ./scripts/test-provision.sh arm          # ARM ISO boot (emulated, SLOW ~60min)
+#   sudo ./scripts/test-provision.sh arm-pxe      # ARM network PXE rescue: NBP + rescue over SSH (~25-30min)
+#   sudo ./scripts/test-provision.sh arm-pxe-full # ARM network PXE incl. discover + full install (~75-95min)
+#   sudo ./scripts/test-provision.sh all          # all tests including ARM
 #
 # Prerequisites:
 #   libvirtd, OVMF (edk2-ovmf), iPXE (ipxe-bootimgs-x86),
 #   dnsmasq, xorriso, mtools, virt-install, qemu-img
-#   ARM: qemu-system-aarch64, edk2-aarch64
+#   ARM: qemu-system-aarch64, edk2-aarch64, ipxe-bootimgs-aarch64
 set -e
 
 SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
@@ -58,6 +61,10 @@ if [ ! -f /usr/share/edk2/ovmf/OVMF_CODE.fd ]; then
     exit 1
 fi
 
+MODE="${1:-both}"
+
+# iPXE binaries are per-architecture. x86_64 is always required (the dnsmasq config
+# references it); arm64 only for the ARM network-PXE modes.
 IPXE_EFI=""
 for f in /usr/share/ipxe/ipxe-snponly-x86_64.efi /usr/share/ipxe/ipxe-snp-x86_64.efi /usr/share/ipxe/ipxe-x86_64.efi; do
     [ -f "$f" ] && IPXE_EFI="$f" && break
@@ -67,6 +74,20 @@ if [ -z "$IPXE_EFI" ]; then
     exit 1
 fi
 
+IPXE_EFI_ARM64=""
+for f in /usr/share/ipxe/arm64-efi/snponly.efi /usr/share/ipxe/arm64-efi/ipxe.efi; do
+    [ -f "$f" ] && IPXE_EFI_ARM64="$f" && break
+done
+
+case "$MODE" in
+    arm-pxe|arm-pxe-full|all)
+        if [ -z "$IPXE_EFI_ARM64" ] && [ "$MODE" != "all" ]; then
+            echo -e "${RED}arm64 iPXE binary not found.${RESET} Install: sudo dnf install ipxe-bootimgs-aarch64"
+            exit 1
+        fi
+        ;;
+esac
+
 # Find SSH key
 SSH_KEY=""
 for name in id_ed25519 id_ecdsa id_rsa; do
@@ -83,10 +104,19 @@ fi
 echo -e "  User:    ${BOLD}$REAL_USER${RESET}"
 echo -e "  SSH key: ${BOLD}$SSH_KEY${RESET}"
 echo -e "  iPXE:    ${BOLD}$IPXE_EFI${RESET}"
+echo -e "  iPXE a64:${BOLD} ${IPXE_EFI_ARM64:-not installed}${RESET}"
 echo ""
 
-# --- Determine which tests to run ---
-MODE="${1:-both}"
+require_arm_emulation() {
+    if ! command -v qemu-system-aarch64 &>/dev/null; then
+        echo -e "${RED}qemu-system-aarch64 not found.${RESET} Install: sudo dnf install qemu-system-aarch64 edk2-aarch64"
+        exit 1
+    fi
+    if [ ! -f /usr/share/edk2/aarch64/QEMU_EFI.fd ]; then
+        echo -e "${RED}AAVMF firmware not found.${RESET} Install: sudo dnf install edk2-aarch64"
+        exit 1
+    fi
+}
 
 run_test() {
     local name="$1" pattern="$2"
@@ -116,13 +146,26 @@ case "$MODE" in
         run_test "ISO boot" "ISO boot" || FAILED=1
         ;;
     arm|arm-iso)
-        if ! command -v qemu-system-aarch64 &>/dev/null; then
-            echo -e "${RED}qemu-system-aarch64 not found.${RESET} Install: sudo dnf install qemu-system-aarch64 edk2-aarch64"
-            exit 1
-        fi
+        require_arm_emulation
         echo -e "${YELLOW}ARM emulation is ~10x slower than native. Expect 30-60 minutes.${RESET}"
         run_test "ARM ISO boot" "ARM ISO" || FAILED=1
         ;;
+    rescue)
+        echo -e "${YELLOW}x86_64 rescue boot (KVM). Expect ~15 minutes.${RESET}"
+        run_test "x86 rescue boot" "x86 rescue boot" || FAILED=1
+        ;;
+    arm-pxe)
+        require_arm_emulation
+        echo -e "${YELLOW}ARM emulation is ~10x slower than native. Expect 25-30 minutes.${RESET}"
+        echo -e "${YELLOW}Covers option 93 -> arm64 NBP, arch resolution, and rescue over SSH.${RESET}"
+        echo -e "${YELLOW}For the full install too, use: $0 arm-pxe-full${RESET}"
+        run_test "ARM PXE rescue" "ARM PXE rescue" || FAILED=1
+        ;;
+    arm-pxe-full)
+        require_arm_emulation
+        echo -e "${YELLOW}ARM emulation is ~10x slower than native. Expect 75-95 minutes.${RESET}"
+        ARM_PXE_FULL=1 run_test "ARM PXE (rescue + install)" "ARM PXE" || FAILED=1
+        ;;
     both)
         run_test "PXE boot" "PXE boot" || FAILED=1
         run_test "ISO boot" "ISO boot" || FAILED=1
@@ -133,12 +176,17 @@ case "$MODE" in
         if command -v qemu-system-aarch64 &>/dev/null; then
             echo -e "${YELLOW}ARM emulation is ~10x slower than native.${RESET}"
             run_test "ARM ISO boot" "ARM ISO" || FAILED=1
+            if [ -n "$IPXE_EFI_ARM64" ]; then
+                run_test "ARM PXE rescue" "ARM PXE rescue" || FAILED=1
+            else
+                echo -e "${YELLOW}Skipping ARM PXE test (ipxe-bootimgs-aarch64 not installed)${RESET}"
+            fi
         else
-            echo -e "${YELLOW}Skipping ARM test (qemu-system-aarch64 not installed)${RESET}"
+            echo -e "${YELLOW}Skipping ARM tests (qemu-system-aarch64 not installed)${RESET}"
         fi
         ;;
     *)
-        echo "Usage: $0 [pxe|iso|arm|both|all]"
+        echo "Usage: $0 [pxe|iso|rescue|arm|arm-pxe|arm-pxe-full|both|all]"
         exit 1
         ;;
 esac

bastion/src/bastion/src/main.ts

@@ -3,7 +3,9 @@
 
 import { mkdirSync, writeFileSync, readFileSync, existsSync, copyFileSync, symlinkSync, unlinkSync } from "node:fs";
 import { execSync } from "node:child_process";
-import type { BastionConfig } from "@lab/shared";
+import type { Arch, BastionConfig } from "@lab/shared";
+import { SUPPORTED_ARCHES, fedoraMirrorFor, classifyOnboard } from "@lab/shared";
+import { kernelPath, initrdPath } from "./templates/boot.ipxe.js";
 import { loadConfig } from "./config.js";
 import { populateNetworkConfig } from "./services/network.js";
 import { createApp } from "./server.js";
@@ -13,6 +15,7 @@ import { renderBootIpxe } from "./templates/boot.ipxe.js";
 import { logger } from "./services/logger.js";
 import { BastionConnection } from "./services/labd-connection.js";
 import { progressBus } from "./services/progress-events.js";
+import { checkInstallAllowed } from "./services/install-guard.js";
 import { ensureBootIso } from "./routes/boot-iso.js";
 
 function copyIfMissing(src: string, dest: string, label: string): void {
@@ -130,9 +133,14 @@ export async function startBastion(overrides: Partial<BastionConfig> = {}): Prom
   mkdirSync(config.tftpDir, { recursive: true });
   mkdirSync(config.httpDir, { recursive: true });
 
+  // Architectures we can actually network boot, reported in the banner so a missing
+  // arm64 payload is visible at startup instead of at 2am when a rescue is needed.
+  const bootArches: Arch[] = [];
+  let ipxeArm64Ready = false;
+
   // Prepare boot artifacts
   if (config.skipArtifacts !== true) {
-    logger.info(`Preparing boot artifacts (Fedora ${config.fedoraVersion} ${config.arch})...`);
+    logger.info(`Preparing boot artifacts (Fedora ${config.fedoraVersion}, ${SUPPORTED_ARCHES.join(" + ")})...`);
 
     copyIfMissing(
       "/usr/share/ipxe/undionly.kpxe",
@@ -150,20 +158,41 @@ export async function startBastion(overrides: Partial<BastionConfig> = {}): Prom
         `${config.tftpDir}/ipxe-arm64.efi`,
         "iPXE UEFI arm64",
       );
+      ipxeArm64Ready = true;
     } catch {
-      logger.warn("arm64 iPXE not available -- skipping");
+      logger.warn("arm64 iPXE not available -- arm64 machines cannot network boot.");
+      logger.warn("  Install with: sudo dnf install ipxe-bootimgs-aarch64");
     }
 
-    download(
-      `${config.fedoraMirror}/images/pxeboot/vmlinuz`,
-      `${config.httpDir}/vmlinuz`,
-      "Fedora kernel",
-    );
-    download(
-      `${config.fedoraMirror}/images/pxeboot/initrd.img`,
-      `${config.httpDir}/initrd.img`,
-      "Fedora initrd",
-    );
+    // Fedora pxeboot kernel + initrd per architecture. x86_64 keeps the unsuffixed
+    // names it has always used; other architectures are suffixed. The iPXE templates
+    // resolve the same paths via kernelPath()/initrdPath().
+    for (const arch of SUPPORTED_ARCHES) {
+      const mirror = fedoraMirrorFor(config.fedoraVersion, arch);
+      try {
+        download(
+          `${mirror}/images/pxeboot/vmlinuz`,
+          `${config.httpDir}${kernelPath(arch)}`,
+          `Fedora ${arch} kernel`,
+        );
+        download(
+          `${mirror}/images/pxeboot/initrd.img`,
+          `${config.httpDir}${initrdPath(arch)}`,
+          `Fedora ${arch} initrd`,
+        );
+        bootArches.push(arch);
+      } catch (err) {
+        // Non-fatal: a bastion with no arm64 artifacts still serves x86_64 fine.
+        // Failing startup over an unreachable mirror for an architecture that may not
+        // even be present on this network would be worse.
+        logger.warn(`Fedora ${arch} kernel/initrd unavailable -- ${arch} PXE disabled`);
+        logger.warn(`  ${err instanceof Error ? err.message : String(err)}`);
+      }
+    }
+
+    if (!bootArches.includes("x86_64")) {
+      throw new Error("Fedora x86_64 kernel/initrd could not be staged -- cannot serve PXE");
+    }
 
     // Ubuntu netboot artifacts (non-fatal — Ubuntu version may not be released yet)
     try {
@@ -254,6 +283,13 @@ export async function startBastion(overrides: Partial<BastionConfig> = {}): Prom
     // Wire up command handlers so labd can send install/forget/role commands
     labdConn.onCommand("command-install", async (msg) => {
       if (msg.type !== "command-install") throw new Error("unexpected");
+      const installMac = msg.mac.toLowerCase().replace(/-/g, ":");
+      const osId = (msg.os as import("@lab/shared").OsId | undefined) ?? "fedora-43";
+      const check = checkInstallAllowed(state.load(), installMac, osId);
+      if (check.allowed === false) {
+        logger.warn(`INSTALL REFUSED: ${installMac} -- ${check.error}`);
+        return { status: "error", error: check.error };
+      }
       state.update((s) => {
         s.install_queue[msg.mac] = {
           hostname: msg.hostname,
@@ -314,13 +350,24 @@ export async function startBastion(overrides: Partial<BastionConfig> = {}): Prom
       const mac = (msg.mac as string).toLowerCase();
       const now = new Date().toISOString();
       const existing = state.load().discovered[mac];
+      const identity = {
+        mac,
+        manufacturer: (msg.manufacturer as string) ?? "unknown",
+        product: (msg.product as string) ?? "unknown",
+        board: (msg.board as string) ?? "unknown",
+        ...(existing?.onboard !== undefined ? { onboard: existing.onboard } : {}),
+        ...(existing?.vendor_os !== undefined ? { vendor_os: existing.vendor_os } : {}),
+      };
+      const onboarding = classifyOnboard(identity);
+      const rootDevice = msg.root_device ?? existing?.root_device;
+      const rootArgs = msg.root_args ?? existing?.root_args;
       state.update((s) => {
         s.discovered[mac] = {
           mac,
-          product: (msg.product as string) ?? "unknown",
-          board: (msg.board as string) ?? "unknown",
+          product: identity.product,
+          board: identity.board,
           serial: (msg.serial as string) ?? "unknown",
-          manufacturer: (msg.manufacturer as string) ?? "unknown",
+          manufacturer: identity.manufacturer,
           cpu_model: (msg.cpu_model as string) ?? "unknown",
           cpu_cores: (msg.cpu_cores as number) ?? 0,
           memory_gb: (msg.memory_gb as number) ?? 0,
@@ -329,7 +376,20 @@ export async function startBastion(overrides: Partial<BastionConfig> = {}): Prom
           nics: (msg.nics as Array<{ name: string; mac: string; state: string }>) ?? [],
           first_seen: existing?.first_seen ?? now,
           last_seen: now,
+          onboard: onboarding.onboard,
+          ...(onboarding.vendor_os !== undefined ? { vendor_os: onboarding.vendor_os } : {}),
+          ...(rootDevice !== undefined ? { root_device: rootDevice } : {}),
+          ...(rootArgs !== undefined ? { root_args: rootArgs } : {}),
         };
+        // Keep the installed record in step -- the guard and --pxe-boot both read it.
+        const inst = s.installed[mac];
+        if (inst) {
+          inst.arch = (msg.arch as string) ?? inst.arch;
+          inst.onboard = onboarding.onboard;
+          if (onboarding.vendor_os !== undefined) inst.vendor_os = onboarding.vendor_os;
+          if (rootDevice !== undefined) inst.root_device = rootDevice;
+          if (rootArgs !== undefined) inst.root_args = rootArgs;
+        }
       });
       logger.info(`HARDWARE UPDATED: ${mac} -- ${msg.manufacturer ?? "?"} ${msg.product ?? "?"} (${msg.cpu_model ?? "?"}, ${msg.cpu_cores ?? "?"} cores, ${msg.memory_gb ?? "?"}GB RAM)`);
       return { status: "ok", data: { mac } };
@@ -364,7 +424,7 @@ export async function startBastion(overrides: Partial<BastionConfig> = {}): Prom
   }
 
   // Print banner
-  printBanner(config);
+  printBanner(config, bootArches, ipxeArm64Ready);
 
   // Graceful shutdown
   const shutdown = async (): Promise<void> => {
@@ -386,11 +446,22 @@ export async function startBastion(overrides: Partial<BastionConfig> = {}): Prom
   await new Promise(() => {});
 }
 
-function printBanner(config: BastionConfig): void {
+function printBanner(config: BastionConfig, bootArches: Arch[], ipxeArm64Ready: boolean): void {
   const dhcpInfo = config.dhcpMode === "full"
     ? `full (${config.dhcpRangeStart}-${config.dhcpRangeEnd})`
     : "proxy (alongside existing DHCP)";
 
+  // arm64 needs both an iPXE binary (DHCP hands it out on option 93 = 0x0b) and a
+  // kernel/initrd pair. Report the combination, since either missing breaks it.
+  const archInfo = config.skipArtifacts === true
+    ? "(artifacts skipped)"
+    : SUPPORTED_ARCHES
+        .map((a) => {
+          const ready = bootArches.includes(a) && (a !== "aarch64" || ipxeArm64Ready);
+          return ready ? a : `${a} (unavailable)`;
+        })
+        .join(", ");
+
   console.log("");
   console.log("\x1b[36m\x1b[1m" + "=".repeat(60) + "\x1b[0m");
   console.log("\x1b[36m\x1b[1m  Lab PXE Bastion -- Discovery Mode\x1b[0m");
@@ -399,7 +470,8 @@ function printBanner(config: BastionConfig): void {
   console.log(`  Network:   \x1b[1m${config.network}/24\x1b[0m via \x1b[1m${config.iface}\x1b[0m`);
   console.log(`  DHCP:      \x1b[1m${dhcpInfo}\x1b[0m`);
   console.log(`  HTTP:      \x1b[1mhttp://${config.serverIp}:${config.httpPort}/\x1b[0m`);
-  console.log(`  OS:        \x1b[1mFedora ${config.fedoraVersion} (${config.arch})\x1b[0m`);
+  console.log(`  OS:        \x1b[1mFedora ${config.fedoraVersion}\x1b[0m`);
+  console.log(`  Net boot:  \x1b[1m${archInfo}\x1b[0m`);
   console.log(`  Domain:    \x1b[1m${config.domain}\x1b[0m`);
   console.log(`  State:     \x1b[1m${config.stateFile}\x1b[0m`);
   console.log("");

bastion/src/bastion/src/routes/api.ts

@@ -6,10 +6,11 @@
 
 import type { FastifyInstance } from "fastify";
 import type { HardwareInfo, InstalledInfo, Role } from "@lab/shared";
-import { isValidOsId, SUPPORTED_ROLES } from "@lab/shared";
+import { isValidOsId, SUPPORTED_ROLES, classifyOnboard } from "@lab/shared";
 import type { StateManager } from "../services/state.js";
 import { logger } from "../services/logger.js";
 import { triggerPostProvisionK3s } from "../services/post-provision.js";
+import { checkInstallAllowed } from "../services/install-guard.js";
 import { progressBus } from "../services/progress-events.js";
 import type { ProgressEvent } from "../services/progress-events.js";
 import type { InstallLogBuffer } from "../services/install-log.js";
@@ -53,6 +54,12 @@ export function registerApiRoutes(
       return reply.status(400).send({ error: `invalid os: '${osId}'. Supported: fedora-43, ubuntu-26.04` });
     }
 
+    const check = checkInstallAllowed(state.load(), mac, osId);
+    if (check.allowed === false) {
+      logger.warn(`INSTALL REFUSED: ${mac} -- ${check.error}`);
+      return reply.status(409).send({ error: check.error });
+    }
+
     state.update((s) => {
       s.install_queue[mac] = {
         hostname: hostname ?? "lab-node",
@@ -284,6 +291,10 @@ export function registerApiRoutes(
       arch?: string;
       disks?: Array<{ name: string; size_gb: number; model: string }>;
       nics?: Array<{ name: string; mac: string; state: string }>;
+      // Root filesystem, when the reporter could observe it (recheck over SSH, or the
+      // probe script run from a rescue shell). Used by --pxe-boot.
+      root_device?: string;
+      root_args?: string;
     };
   }>("/api/discover", async (request, reply) => {
     const data = request.body;
@@ -298,22 +309,53 @@ export function registerApiRoutes(
 
     state.update((s) => {
       const existing = s.discovered[mac];
+      // Classify onboarding from the DMI identity we just received. An explicit
+      // classification already on the record wins (see classifyOnboard).
+      const onboarding = classifyOnboard({
+        mac,
+        manufacturer: data.manufacturer ?? existing?.manufacturer ?? "unknown",
+        product: data.product ?? existing?.product ?? "unknown",
+        board: data.board ?? existing?.board ?? "unknown",
+        ...(existing?.onboard !== undefined ? { onboard: existing.onboard } : {}),
+        ...(existing?.vendor_os !== undefined ? { vendor_os: existing.vendor_os } : {}),
+      });
+      const rootDevice = data.root_device ?? existing?.root_device;
+      const rootArgs = data.root_args ?? existing?.root_args;
+
+      // Absent fields keep whatever we already knew. Reporters are not all the full
+      // discovery kickstart: the rescue-shell probe posts only a root device, and
+      // blanking a machine's hardware inventory as a side effect of that would be
+      // silent data loss.
       const hwInfo: HardwareInfo = {
         mac,
-        product: data.product ?? "unknown",
-        board: data.board ?? "unknown",
-        serial: data.serial ?? "unknown",
-        manufacturer: data.manufacturer ?? "unknown",
-        cpu_model: data.cpu_model ?? "unknown",
-        cpu_cores: data.cpu_cores ?? 0,
-        memory_gb: data.memory_gb ?? 0,
-        arch: data.arch ?? "unknown",
-        disks: data.disks ?? [],
-        nics: data.nics ?? [],
+        product: data.product ?? existing?.product ?? "unknown",
+        board: data.board ?? existing?.board ?? "unknown",
+        serial: data.serial ?? existing?.serial ?? "unknown",
+        manufacturer: data.manufacturer ?? existing?.manufacturer ?? "unknown",
+        cpu_model: data.cpu_model ?? existing?.cpu_model ?? "unknown",
+        cpu_cores: data.cpu_cores ?? existing?.cpu_cores ?? 0,
+        memory_gb: data.memory_gb ?? existing?.memory_gb ?? 0,
+        arch: data.arch ?? existing?.arch ?? "unknown",
+        disks: data.disks ?? existing?.disks ?? [],
+        nics: data.nics ?? existing?.nics ?? [],
         first_seen: existing?.first_seen ?? now,
         last_seen: now,
+        onboard: onboarding.onboard,
+        ...(onboarding.vendor_os !== undefined ? { vendor_os: onboarding.vendor_os } : {}),
+        ...(rootDevice !== undefined ? { root_device: rootDevice } : {}),
+        ...(rootArgs !== undefined ? { root_args: rootArgs } : {}),
       };
       s.discovered[mac] = hwInfo;
+
+      // Keep the installed record in step -- the install guard and --pxe-boot read it.
+      const inst = s.installed[mac];
+      if (inst) {
+        if (data.arch !== undefined) inst.arch = data.arch;
+        inst.onboard = onboarding.onboard;
+        if (onboarding.vendor_os !== undefined) inst.vendor_os = onboarding.vendor_os;
+        if (rootDevice !== undefined) inst.root_device = rootDevice;
+        if (rootArgs !== undefined) inst.root_args = rootArgs;
+      }
     });
 
     const label = isNew ? "NEW MACHINE DISCOVERED" : "MACHINE RE-DISCOVERED";

bastion/src/bastion/src/routes/dispatch.ts

@@ -5,7 +5,8 @@
 //   - unknown       -> discovery mode (collect hardware, POST to bastion)
 
 import type { FastifyInstance } from "fastify";
-import type { BastionConfig } from "@lab/shared";
+import type { Arch, BastionConfig, BastionState, OsId } from "@lab/shared";
+import { normalizeArch, fedoraMirrorFor, osSupportsArch } from "@lab/shared";
 import type { StateManager } from "../services/state.js";
 import {
   renderDiscoverIpxe,
@@ -13,11 +14,51 @@ import {
   renderDebugIpxe,
   renderPxeBootDebugIpxe,
   renderLocalBootIpxe,
+  renderUnsupportedIpxe,
 } from "../templates/boot.ipxe.js";
 import { renderUbuntuInstallIpxe } from "../templates/ubuntu-boot.ipxe.js";
 import { renderDebugKickstart } from "../templates/debug.ks.js";
 import { logger } from "../services/logger.js";
 
+/**
+ * Resolve a booting machine's architecture.
+ *
+ * Order matters. The tracked record is what we actually observed on the machine, so it
+ * wins. `reported` is iPXE's ${buildarch}, which is only as good as the binary DHCP
+ * handed the client -- correct in practice, but a misconfigured option 93 mapping would
+ * make it lie. The configured default is the last resort.
+ *
+ * There is deliberately no operator-supplied architecture anywhere in this path.
+ */
+export function resolveArch(
+  state: BastionState,
+  mac: string,
+  reported: string | undefined,
+  config: BastionConfig,
+): Arch {
+  return normalizeArch(state.installed[mac]?.arch)
+    ?? normalizeArch(state.install_queue[mac]?.arch)
+    ?? normalizeArch(state.discovered[mac]?.arch)
+    ?? normalizeArch(reported)
+    ?? normalizeArch(config.arch)
+    ?? "x86_64";
+}
+
+/** The root filesystem to boot for --pxe-boot, if the machine's record carries one. */
+function resolveRoot(
+  state: BastionState,
+  mac: string,
+): { rootDevice: string; rootArgs?: string } | null {
+  const installed = state.installed[mac];
+  const discovered = state.discovered[mac];
+  const rootDevice = installed?.root_device ?? discovered?.root_device;
+  if (rootDevice === undefined || rootDevice === "") return null;
+  const rootArgs = installed?.root_args ?? discovered?.root_args;
+  return rootArgs !== undefined && rootArgs !== ""
+    ? { rootDevice, rootArgs }
+    : { rootDevice };
+}
+
 export function registerDispatchRoutes(
   app: FastifyInstance,
   config: BastionConfig,
@@ -52,18 +93,68 @@ curl -sf -X POST "http://${config.serverIp}:${config.httpPort}/api/progress" \\
     -H "Content-Type: application/json" \\
     -d "{\\"mac\\":\\"$MAC_ADDR\\",\\"stage\\":\\"debug-ready\\",\\"detail\\":\\"nc $IP_ADDR 2323\\"}" 2>/dev/null || true
 
+# --- Find the installed root filesystem and report it ---
+# This is what 'labctl provision debug --pxe-boot' needs. The rescue image cannot
+# report it by itself: %pre/%post do not run in rescue mode, so it happens here.
+vgchange -ay >/dev/null 2>&1 || true
+
+ROOT_DEVICE=""
+ROOT_ARGS=""
+PROBE_MNT=/tmp/lab-rootprobe
+mkdir -p "$PROBE_MNT"
+
+# Candidates: every LVM logical volume plus every non-LVM partition with a filesystem.
+for CAND in $(lvs --noheadings -o lv_path 2>/dev/null) \\
+            $(blkid -o device 2>/dev/null | grep -v '^/dev/mapper/'); do
+    [ -b "$CAND" ] || continue
+    mount -o ro "$CAND" "$PROBE_MNT" >/dev/null 2>&1 || continue
+    # A root filesystem has both of these; /boot and /home do not.
+    if [ -f "$PROBE_MNT/etc/fstab" ] && [ -d "$PROBE_MNT/usr" ]; then
+        ROOT_DEVICE="$CAND"
+        PRETTY=$(. "$PROBE_MNT/etc/os-release" 2>/dev/null && echo "$PRETTY_NAME")
+        echo "  found root: $CAND \${PRETTY:+($PRETTY)}"
+        if [ "$(lsblk -no TYPE "$CAND" 2>/dev/null | head -1)" = "lvm" ]; then
+            VGLV=$(lvs --noheadings -o vg_name,lv_name "$CAND" 2>/dev/null | awk '{print $1"/"$2}')
+            [ -n "$VGLV" ] && ROOT_ARGS="rd.lvm.lv=$VGLV"
+            # Swap comes from fstab here — /proc/swaps is the rescue image's, not the host's.
+            SWLV=$(awk '$3=="swap" && $1 ~ /^\\/dev\\// {print $1; exit}' "$PROBE_MNT/etc/fstab" 2>/dev/null)
+            if [ -n "$SWLV" ]; then
+                SWVGLV=$(lvs --noheadings -o vg_name,lv_name "$SWLV" 2>/dev/null | awk '{print $1"/"$2}')
+                [ -n "$SWVGLV" ] && [ "$SWVGLV" != "$VGLV" ] && ROOT_ARGS="$ROOT_ARGS rd.lvm.lv=$SWVGLV"
+            fi
+        fi
+        umount "$PROBE_MNT" >/dev/null 2>&1 || true
+        break
+    fi
+    umount "$PROBE_MNT" >/dev/null 2>&1 || true
+done
+
+if [ -n "$ROOT_DEVICE" ]; then
+    curl -sf -X POST "http://${config.serverIp}:${config.httpPort}/api/discover" \\
+        -H "Content-Type: application/json" \\
+        -d "{\\"mac\\":\\"$MAC_ADDR\\",\\"root_device\\":\\"$ROOT_DEVICE\\",\\"root_args\\":\\"$ROOT_ARGS\\"}" 2>/dev/null \\
+        && echo "  reported to bastion — 'labctl provision debug --pxe-boot' will work now"
+else
+    echo "  no root filesystem found — --pxe-boot cannot be used on this machine"
+fi
+
 echo ""
 echo "=== Debug environment ready ==="
 echo "  nc $IP_ADDR 2323    (remote shell)"
 echo "  ssh root@$IP_ADDR   (password: debug)"
+if [ -n "$ROOT_DEVICE" ]; then
+    echo "  root: $ROOT_DEVICE $ROOT_ARGS"
+fi
 echo "==============================="
 `;
     return reply.type("text/plain").send(script);
   });
 
-  app.get<{ Querystring: { mac?: string } }>("/dispatch", async (request, reply) => {
+  app.get<{ Querystring: { mac?: string; arch?: string } }>("/dispatch", async (request, reply) => {
     const mac = (request.query.mac ?? "").toLowerCase().replace(/-/g, ":");
     const currentState = state.load();
+    const arch = resolveArch(currentState, mac, request.query.arch, config);
+    const fedoraMirror = fedoraMirrorFor(config.fedoraVersion, arch);
 
     // Debug mode takes highest priority — auto-clear after serving once
     const debugEntry = currentState.debug[mac];
@@ -72,22 +163,48 @@ echo "==============================="
       state.update((s) => { delete s.debug[mac]; });
 
       let script: string;
-      if (debugEntry.pxeBoot) {
-        logger.info(`PXE BOOT DEBUG: ${mac} -> ${hostname} (kernel+initrd from PXE, root from NVMe)`);
+      const wantsPxeBoot = debugEntry.pxeBoot === true;
+      const root = wantsPxeBoot ? resolveRoot(currentState, mac) : null;
+
+      if (root !== null) {
+        logger.info(`PXE BOOT DEBUG: ${mac} -> ${hostname} (${arch}, root=${root.rootDevice})`);
         script = renderPxeBootDebugIpxe({
           mac,
           hostname,
           serverIp: config.serverIp,
           httpPort: config.httpPort,
+          arch,
+          ...root,
         });
       } else {
-        logger.info(`DEBUG BOOT: ${mac} -> ${hostname} (rescue mode)`);
+        // --pxe-boot without a known root device falls back to rescue rather than
+        // guessing. A wrong root= leaves the machine unbootable, and rescue is where
+        // the operator can find the real one (curl /debug-setup.sh reports it back).
+        const notice = wantsPxeBoot
+          ? [
+              "",
+              "NOTE: --pxe-boot requested, but no root device is recorded",
+              "      for this machine. Booting rescue instead.",
+              "      From the rescue shell, run:",
+              // No pipe or && here: iPXE treats || and && as command separators, so keep
+              // the printed command free of anything its parser might claim.
+              `        curl -s http://${config.serverIp}:${config.httpPort}/debug-setup.sh -o /tmp/s.sh ; sh /tmp/s.sh`,
+              "      then retry --pxe-boot.",
+            ]
+          : undefined;
+        if (wantsPxeBoot) {
+          logger.warn(`PXE BOOT DEBUG: ${mac} -> ${hostname} has no recorded root device -- serving rescue instead`);
+        } else {
+          logger.info(`DEBUG BOOT: ${mac} -> ${hostname} (${arch}, rescue mode)`);
+        }
         script = renderDebugIpxe({
           mac,
           hostname,
           serverIp: config.serverIp,
           httpPort: config.httpPort,
-          fedoraMirror: config.fedoraMirror,
+          fedoraMirror,
+          arch,
+          ...(notice ? { notice } : {}),
         });
       }
       return reply.type("text/plain").send(script);
@@ -97,10 +214,24 @@ echo "==============================="
     if (queueEntry) {
       const hostname = queueEntry.hostname ?? "lab-node";
       const os = queueEntry.os ?? "fedora-43";
-      logger.info(`INSTALL STARTED: ${mac} -> ${hostname} (${os})`);
+      logger.info(`INSTALL STARTED: ${mac} -> ${hostname} (${os}, ${arch})`);
 
       let script: string;
       if (os.startsWith("ubuntu")) {
+        // Last line of defence. The install guard refuses this combination when the
+        // machine's architecture is already known, but a machine queued before it was
+        // discovered can reach here. Serving the x86-only Ubuntu kernel to an arm64
+        // client is precisely the bug this work exists to fix, so stop instead.
+        if (!osSupportsArch(os as OsId, arch)) {
+          logger.error(`INSTALL BLOCKED: ${mac} -> ${hostname} -- ${os} has no ${arch} artifacts`);
+          script = renderUnsupportedIpxe({
+            hostname,
+            mac,
+            reason: `${os} publishes no ${arch} netboot artifacts`,
+            action: `labctl provision install ${mac} ${hostname} --os fedora-43`,
+          });
+          return reply.type("text/plain").send(script);
+        }
         script = renderUbuntuInstallIpxe({
           mac,
           hostname,
@@ -115,7 +246,8 @@ echo "==============================="
           serverIp: config.serverIp,
           httpPort: config.httpPort,
           fedoraVersion: config.fedoraVersion,
-          fedoraMirror: config.fedoraMirror,
+          fedoraMirror,
+          arch,
         });
       }
 
@@ -132,13 +264,14 @@ echo "==============================="
     }
 
     // Unknown MAC -> discovery mode
-    logger.info(`PXE request from ${mac} -> discovery mode`);
+    logger.info(`PXE request from ${mac} (${arch}) -> discovery mode`);
 
     const script = renderDiscoverIpxe({
       mac,
       serverIp: config.serverIp,
       httpPort: config.httpPort,
-      fedoraMirror: config.fedoraMirror,
+      fedoraMirror,
+      arch,
     });
 
     return reply.type("text/plain").send(script);

bastion/src/bastion/src/services/install-guard.ts

@@ -0,0 +1,94 @@
+// Pre-flight checks for queuing an OS install.
+//
+// Both entry points -- the HTTP /api/install route and the labd command-install handler
+// -- run this, so `labctl provision install` and `provision reprovision` are covered
+// whichever way the request arrives.
+//
+// Rescue/debug is deliberately NOT guarded. Being unable to reinstall a machine is
+// exactly when you most need to boot it into a rescue shell.
+
+import type { Arch, BastionState, OsId } from "@lab/shared";
+import { classifyOnboard, normalizeArch, osSupportsArch, vendorOsDescription, archesForOs } from "@lab/shared";
+
+export type InstallCheck =
+  | { allowed: true }
+  | { allowed: false; error: string };
+
+interface MachineIdentity {
+  hostname: string;
+  arch: Arch | undefined;
+  identity: Parameters<typeof classifyOnboard>[0];
+}
+
+/** Best-known identity for a MAC, merged across the three state maps. */
+function identify(state: BastionState, mac: string): MachineIdentity {
+  const discovered = state.discovered[mac];
+  const installed = state.installed[mac];
+  const queued = state.install_queue[mac];
+
+  const manufacturer = discovered?.manufacturer ?? installed?.manufacturer;
+  const product = discovered?.product ?? installed?.product;
+  const board = discovered?.board;
+  const onboard = installed?.onboard ?? discovered?.onboard;
+  const vendorOs = installed?.vendor_os ?? discovered?.vendor_os;
+
+  return {
+    hostname: installed?.hostname ?? queued?.hostname ?? discovered?.product ?? mac,
+    arch: normalizeArch(installed?.arch ?? queued?.arch ?? discovered?.arch),
+    identity: {
+      mac,
+      ...(manufacturer !== undefined ? { manufacturer } : {}),
+      ...(product !== undefined ? { product } : {}),
+      ...(board !== undefined ? { board } : {}),
+      ...(onboard !== undefined ? { onboard } : {}),
+      ...(vendorOs !== undefined ? { vendor_os: vendorOs } : {}),
+    },
+  };
+}
+
+/**
+ * Decide whether `mac` may be queued for an install of `os`.
+ *
+ * Refusals name the machine and the reason, and point at the action that is available
+ * instead. An operator hitting this at 2am should not have to read the source to work
+ * out what happened.
+ */
+export function checkInstallAllowed(
+  state: BastionState,
+  mac: string,
+  os: OsId,
+): InstallCheck {
+  const machine = identify(state, mac);
+  const { onboard, vendor_os } = classifyOnboard(machine.identity);
+
+  // 1. Machines running a vendor OS we cannot rebuild.
+  if (onboard === "ssh") {
+    const what = vendorOsDescription(vendor_os);
+    return {
+      allowed: false,
+      error:
+        `Refusing to install ${machine.hostname} (${mac}): it runs ${what}. ` +
+        `No image in our pipeline can restore it, so installing ${os} would destroy that ` +
+        `driver and firmware stack permanently. This machine is SSH-onboard: we manage its ` +
+        `userspace, not its OS. ` +
+        `To boot it into a rescue shell instead, run: labctl provision debug ${machine.hostname}`,
+      // TODO: when a DGX OS / SparkOS image joins the pipeline, an install targeting a
+      // machine whose vendor_os matches that image should be allowed through here.
+    };
+  }
+
+  // 2. Architecture the OS has no netboot artifacts for.
+  if (machine.arch !== undefined && !osSupportsArch(os, machine.arch)) {
+    const supported = archesForOs(os);
+    return {
+      allowed: false,
+      error:
+        `Refusing to install ${os} on ${machine.hostname} (${mac}): ` +
+        `${os} has no ${machine.arch} netboot artifacts` +
+        (supported.length > 0 ? ` (only ${supported.join(", ")})` : "") +
+        `. Use an OS that supports ${machine.arch}.`,
+    };
+  }
+
+  return { allowed: true };
+}

bastion/src/bastion/src/templates/boot.ipxe.ts

@@ -1,4 +1,55 @@
 // iPXE boot script templates for dispatch routing.
+//
+// Architecture handling: the bastion serves one kernel/initrd pair per architecture.
+// x86_64 keeps the original unsuffixed paths so its output is unchanged; every other
+// architecture gets an arch-suffixed pair. See stageBootArtifacts() in main.ts for the
+// matching staging side, and boot-iso.ts for the same scheme on the ISO path.
+
+import type { Arch } from "@lab/shared";
+
+/** Kernel/initrd URL paths, keyed by architecture. */
+export function kernelPath(arch: Arch): string {
+  return arch === "x86_64" ? "/vmlinuz" : `/vmlinuz-${arch}`;
+}
+
+export function initrdPath(arch: Arch): string {
+  return arch === "x86_64" ? "/initrd.img" : `/initrd-${arch}.img`;
+}
+
+/**
+ * Console arguments per architecture.
+ *
+ * arm64 has no VGA text console: a headless machine only talks over the SoC UART, so
+ * ttyAMA0 must be listed as well. The last console= wins for /dev/console, so serial
+ * is the interactive one while tty0 still receives boot output on machines with a
+ * display attached.
+ */
+const CONSOLE_ARGS: Record<Arch, string> = {
+  x86_64: "console=tty0",
+  aarch64: "console=tty0 console=ttyAMA0,115200",
+};
+
+/**
+ * Anaconda arguments for the graphical-suppression / console setup.
+ *
+ * `nomodeset` disables kernel mode setting, which on x86 forces the generic VGA path
+ * and makes flaky GPU drivers survive the installer. On arm64 it does not mean the
+ * same thing -- there is no VGA fallback to drop back to, and it can leave the machine
+ * with no usable console at all -- so arm64 gets explicit console arguments instead.
+ */
+function installerArgs(arch: Arch): string {
+  return arch === "x86_64" ? "inst.text nomodeset" : `inst.text ${CONSOLE_ARGS[arch]}`;
+}
+
+/** Extra console arguments appended to templates that don't already set them. */
+function extraConsoleArgs(arch: Arch): string {
+  return arch === "x86_64" ? "" : ` ${CONSOLE_ARGS[arch]}`;
+}
+
+/** Join kernel arguments, dropping empties so callers can pass optional groups. */
+function joinArgs(...parts: Array<string | undefined>): string {
+  return parts.filter((p) => p !== undefined && p !== "").join(" ");
+}
 
 export interface BootIpxeParams {
   serverIp: string;
@@ -8,6 +59,11 @@ export interface BootIpxeParams {
 /**
  * Initial iPXE boot script that chains to the dispatch endpoint.
  * This is what dnsmasq serves to iPXE clients via HTTP.
+ *
+ * `${buildarch}` is iPXE's own build architecture ("x86_64" or "arm64"), which is the
+ * one architecture signal available on every path -- network PXE, UEFI HTTP boot and
+ * the boot ISO alike. DHCP option 93 only reaches dnsmasq, never this HTTP endpoint.
+ * dispatch prefers the tracked machine record and falls back to this.
  */
 export function renderBootIpxe(params: BootIpxeParams): string {
   return `#!ipxe
@@ -19,7 +75,7 @@ echo   Contacting server for instructions...
 echo ============================================
 echo
 
-chain http://${params.serverIp}:${params.httpPort}/dispatch?mac=\${net0/mac}
+chain http://${params.serverIp}:${params.httpPort}/dispatch?mac=\${net0/mac}&arch=\${buildarch}
 `;
 }
 
@@ -31,7 +87,9 @@ export function renderDiscoverIpxe(params: {
   serverIp: string;
   httpPort: number;
   fedoraMirror: string;
+  arch: Arch;
 }): string {
+  const base = `http://${params.serverIp}:${params.httpPort}`;
   return `#!ipxe
 
 echo
@@ -42,8 +100,8 @@ echo   Collecting hardware info...
 echo =============================================
 echo
 
-kernel http://${params.serverIp}:${params.httpPort}/vmlinuz inst.ks=http://${params.serverIp}:${params.httpPort}/discover.ks inst.stage2=${params.fedoraMirror} inst.text nomodeset
-initrd http://${params.serverIp}:${params.httpPort}/initrd.img
+kernel ${base}${kernelPath(params.arch)} inst.ks=${base}/discover.ks inst.stage2=${params.fedoraMirror} ${installerArgs(params.arch)}
+initrd ${base}${initrdPath(params.arch)}
 boot
 `;
 }
@@ -58,7 +116,9 @@ export function renderInstallIpxe(params: {
   httpPort: number;
   fedoraVersion: string;
   fedoraMirror: string;
+  arch: Arch;
 }): string {
+  const base = `http://${params.serverIp}:${params.httpPort}`;
   return `#!ipxe
 
 echo
@@ -69,8 +129,8 @@ echo   MAC:    ${params.mac}
 echo =============================================
 echo
 
-kernel http://${params.serverIp}:${params.httpPort}/vmlinuz inst.ks=http://${params.serverIp}:${params.httpPort}/ks?mac=${params.mac} inst.repo=${params.fedoraMirror} inst.text nomodeset
-initrd http://${params.serverIp}:${params.httpPort}/initrd.img
+kernel ${base}${kernelPath(params.arch)} inst.ks=${base}/ks?mac=${params.mac} inst.repo=${params.fedoraMirror} ${installerArgs(params.arch)}
+initrd ${base}${initrdPath(params.arch)}
 boot
 `;
 }
@@ -78,6 +138,9 @@ boot
 /**
  * iPXE script for debug/rescue mode -- boots Fedora installer in rescue mode.
  * Provides a shell with LVM tools, network, and SSH for inspecting installed systems.
+ *
+ * `notice` is shown before the boot line. dispatch uses it to explain why a requested
+ * --pxe-boot fell back to rescue.
  */
 export function renderDebugIpxe(params: {
   mac: string;
@@ -85,7 +148,11 @@ export function renderDebugIpxe(params: {
   serverIp: string;
   httpPort: number;
   fedoraMirror: string;
+  arch: Arch;
+  notice?: string[];
 }): string {
+  const base = `http://${params.serverIp}:${params.httpPort}`;
+  const notice = (params.notice ?? []).map((line) => `echo   ${line}\n`).join("");
   return `#!ipxe
 
 echo
@@ -93,11 +160,11 @@ echo =============================================
 echo   Lab PXE Bastion - DEBUG/RESCUE MODE
 echo   Target: ${params.hostname}
 echo   MAC:    ${params.mac}
-echo =============================================
+${notice}echo =============================================
 echo
 
-kernel http://${params.serverIp}:${params.httpPort}/vmlinuz inst.rescue inst.text inst.sshd inst.ks=http://${params.serverIp}:${params.httpPort}/debug.ks?mac=${params.mac} inst.stage2=${params.fedoraMirror}
-initrd http://${params.serverIp}:${params.httpPort}/initrd.img
+kernel ${base}${kernelPath(params.arch)} inst.rescue inst.text inst.sshd inst.ks=${base}/debug.ks?mac=${params.mac} inst.stage2=${params.fedoraMirror}${extraConsoleArgs(params.arch)}
+initrd ${base}${initrdPath(params.arch)}
 boot
 `;
 }
@@ -106,13 +173,28 @@ boot
  * iPXE script for PXE-boot debug mode -- boots the installed system's root
  * filesystem using the bastion's PXE kernel+initrd instead of local GRUB.
  * Workaround for UEFI firmware bugs that make local disk boot slow.
+ *
+ * rootDevice/rootArgs come from the machine's record -- they are not assumed. Our
+ * Fedora installs use an LVM layout, but nothing guarantees any given machine does,
+ * and a wrong root= here means an unbootable machine. dispatch refuses to render this
+ * script without them.
  */
 export function renderPxeBootDebugIpxe(params: {
   mac: string;
   hostname: string;
   serverIp: string;
   httpPort: number;
+  arch: Arch;
+  rootDevice: string;
+  rootArgs?: string;
 }): string {
+  const base = `http://${params.serverIp}:${params.httpPort}`;
+  const cmdline = joinArgs(
+    `root=${params.rootDevice}`,
+    "ro",
+    params.rootArgs,
+    CONSOLE_ARGS[params.arch],
+  );
   return `#!ipxe
 
 echo
@@ -124,12 +206,40 @@ echo   Kernel+initrd from PXE, root from NVMe
 echo =============================================
 echo
 
-kernel http://${params.serverIp}:${params.httpPort}/vmlinuz root=/dev/mapper/labvg-root ro rd.lvm.lv=labvg/root rd.lvm.lv=labvg/swap console=tty0
-initrd http://${params.serverIp}:${params.httpPort}/initrd.img
+kernel ${base}${kernelPath(params.arch)} ${cmdline}
+initrd ${base}${initrdPath(params.arch)}
 boot
 `;
 }
 
+/**
+ * iPXE script for a request we refuse to serve.
+ *
+ * Better a machine that stops with a legible reason on its console than one handed a
+ * kernel it cannot execute, which fails much later and much less clearly.
+ */
+export function renderUnsupportedIpxe(params: {
+  mac: string;
+  hostname: string;
+  reason: string;
+  action?: string;
+}): string {
+  return `#!ipxe
+
+echo
+echo =============================================
+echo   Lab PXE Bastion - CANNOT BOOT THIS MACHINE
+echo   Target: ${params.hostname}
+echo   MAC:    ${params.mac}
+echo
+echo   ${params.reason}
+${params.action !== undefined ? `echo\necho   Try: ${params.action}\n` : ""}echo =============================================
+echo
+sleep 10
+exit 1
+`;
+}
+
 /**
  * iPXE script for already-installed machines -- exits to boot from local disk.
  */

bastion/src/bastion/src/templates/dnsmasq.conf.ts

@@ -48,15 +48,20 @@ enable-tftp
 tftp-root=${tftpDir}
 tftp-no-blocksize
 
-# Detect client architecture -- PXE (TFTP) clients
+# Detect client architecture -- PXE (TFTP) clients.
+# Values are DHCP option 93 (Client System Architecture), IANA "Processor Architecture
+# Types". Getting these wrong means the machine is handed a bootloader its firmware
+# cannot execute, and it loops or hangs with no console output.
 dhcp-match=set:bios,option:client-arch,0
 dhcp-match=set:efi-x86_64,option:client-arch,7
 dhcp-match=set:efi-x86_64,option:client-arch,9
 dhcp-match=set:efi-arm64,option:client-arch,11
 
-# Detect client architecture -- UEFI HTTP Boot clients (no TFTP size limit)
+# Detect client architecture -- UEFI HTTP Boot clients (no TFTP size limit).
+# 16 = x64 uefi boot from http, 19 = arm uefi 64 boot from http.
+# (20 is pc/at bios boot from http -- not arm64.)
 dhcp-match=set:httpboot-x86_64,option:client-arch,16
-dhcp-match=set:httpboot-arm64,option:client-arch,20
+dhcp-match=set:httpboot-arm64,option:client-arch,19
 
 # Detect iPXE clients (already chainloaded)
 dhcp-userclass=set:ipxe,iPXE

bastion/src/bastion/tests/arch-dispatch.test.ts

@@ -0,0 +1,291 @@
+// aarch64 support in the PXE dispatch path.
+//
+// The x86_64 side is pinned separately by ipxe-x86-regression.test.ts.
+
+import { describe, it, expect, beforeEach, afterEach } from "vitest";
+import { mkdirSync, rmSync } from "node:fs";
+import { join } from "node:path";
+import { tmpdir } from "node:os";
+import type { BastionConfig, BastionState, HardwareInfo } from "@lab/shared";
+import { createApp } from "../src/server.js";
+import { resolveArch } from "../src/routes/dispatch.js";
+import { renderDnsmasqConf } from "../src/templates/dnsmasq.conf.js";
+import type { FastifyInstance } from "fastify";
+import type { StateManager } from "../src/services/state.js";
+
+function createTestConfig(testDir: string): BastionConfig {
+  return {
+    fedoraVersion: "43",
+    arch: "x86_64",
+    httpPort: 0,
+    timezone: "Europe/London",
+    locale: "en_GB.UTF-8",
+    bastionDir: testDir,
+    domain: "test.local",
+    dhcpMode: "proxy",
+    dhcpRangeStart: "",
+    dhcpRangeEnd: "",
+    ubuntuVersion: "26.04",
+    ubuntuMirror: "https://releases.ubuntu.com/26.04",
+    iface: "eth0",
+    serverIp: "10.0.0.1",
+    network: "10.0.0.0",
+    gateway: "10.0.0.1",
+    sshKeys: ["ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAITEST test@test"],
+    adminUser: "testadmin",
+    syslogPort: 15514,
+    skipDnsmasq: true,
+    skipArtifacts: true,
+    fedoraMirror: "https://download.fedoraproject.org/pub/fedora/linux/releases/43/Everything/x86_64/os",
+    tftpDir: join(testDir, "tftp"),
+    httpDir: join(testDir, "http"),
+    stateFile: join(testDir, "state.json"),
+  };
+}
+
+function hardware(mac: string, over: Partial<HardwareInfo> = {}): HardwareInfo {
+  return {
+    mac,
+    product: "TestBox",
+    board: "TestBoard",
+    serial: "SN123",
+    manufacturer: "TestCorp",
+    cpu_model: "Test CPU",
+    cpu_cores: 4,
+    memory_gb: 16,
+    arch: "x86_64",
+    disks: [],
+    nics: [],
+    first_seen: new Date().toISOString(),
+    last_seen: new Date().toISOString(),
+    ...over,
+  };
+}
+
+const emptyState = (): BastionState => ({
+  discovered: {}, install_queue: {}, installed: {}, debug: {},
+});
+
+describe("architecture resolution", () => {
+  const config = createTestConfig("/tmp/unused");
+  const mac = "aa:bb:cc:dd:ee:ff";
+
+  it("prefers the tracked record over what the client reports", () => {
+    const state = emptyState();
+    state.discovered[mac] = hardware(mac, { arch: "aarch64" });
+    // Client claims x86_64; the machine record says otherwise and wins.
+    expect(resolveArch(state, mac, "x86_64", config)).toBe("aarch64");
+  });
+
+  it("falls back to the architecture reported at boot", () => {
+    expect(resolveArch(emptyState(), mac, "arm64", config)).toBe("aarch64");
+  });
+
+  it("normalises iPXE's arm64 spelling to aarch64", () => {
+    expect(resolveArch(emptyState(), mac, "arm64", config)).toBe("aarch64");
+    expect(resolveArch(emptyState(), mac, "x86_64", config)).toBe("x86_64");
+  });
+
+  it("falls back to the configured default for unknown architectures", () => {
+    expect(resolveArch(emptyState(), mac, "riscv64", config)).toBe("x86_64");
+    expect(resolveArch(emptyState(), mac, undefined, config)).toBe("x86_64");
+  });
+
+  it("reads arch from the installed record for already-provisioned machines", () => {
+    const state = emptyState();
+    state.installed[mac] = {
+      hostname: "spark", role: "worker", ip: "10.0.0.5",
+      installed_at: new Date().toISOString(), arch: "aarch64",
+    };
+    expect(resolveArch(state, mac, undefined, config)).toBe("aarch64");
+  });
+});
+
+describe("aarch64 dispatch", () => {
+  let testDir: string;
+  let app: FastifyInstance;
+  let state: StateManager;
+  const mac = "aa:bb:cc:dd:ee:ff";
+
+  beforeEach(() => {
+    testDir = join(tmpdir(), `bastion-arch-test-${Date.now()}-${Math.random().toString(36).slice(2)}`);
+    mkdirSync(join(testDir, "http"), { recursive: true });
+    mkdirSync(join(testDir, "tftp"), { recursive: true });
+    const result = createApp(createTestConfig(testDir));
+    app = result.app;
+    state = result.state;
+  });
+
+  afterEach(async () => {
+    await app.close();
+    rmSync(testDir, { recursive: true, force: true });
+  });
+
+  it("serves the aarch64 kernel and initrd to an arm64 client", async () => {
+    const res = await app.inject({ method: "GET", url: `/dispatch?mac=${mac}&arch=arm64` });
+    expect(res.statusCode).toBe(200);
+    expect(res.body).toContain("/vmlinuz-aarch64");
+    expect(res.body).toContain("/initrd-aarch64.img");
+    expect(res.body).not.toContain("/vmlinuz ");
+  });
+
+  it("points an arm64 client at the aarch64 Fedora mirror", async () => {
+    const res = await app.inject({ method: "GET", url: `/dispatch?mac=${mac}&arch=arm64` });
+    expect(res.body).toContain("Everything/aarch64/os");
+    expect(res.body).not.toContain("Everything/x86_64/os");
+  });
+
+  it("uses serial console arguments and not nomodeset on arm64", async () => {
+    const res = await app.inject({ method: "GET", url: `/dispatch?mac=${mac}&arch=arm64` });
+    expect(res.body).toContain("console=ttyAMA0,115200");
+    expect(res.body).not.toContain("nomodeset");
+  });
+
+  it("refuses to serve the x86-only Ubuntu kernel to an arm64 client", async () => {
+    // A machine queued for Ubuntu before it was discovered as aarch64 reaches dispatch
+    // with no guard having run. Serving it /ubuntu-vmlinuz is the original bug.
+    state.update((s) => {
+      s.install_queue[mac] = {
+        hostname: "arm-node", disk: "", role: "worker",
+        os: "ubuntu-26.04", queued_at: new Date().toISOString(),
+      };
+    });
+
+    const res = await app.inject({ method: "GET", url: `/dispatch?mac=${mac}&arch=arm64` });
+    expect(res.statusCode).toBe(200);
+    expect(res.body).toContain("CANNOT BOOT THIS MACHINE");
+    expect(res.body).toContain("no aarch64 netboot artifacts");
+    expect(res.body).not.toContain("ubuntu-vmlinuz");
+  });
+
+  it("still serves Ubuntu to an x86_64 client", async () => {
+    state.update((s) => {
+      s.install_queue[mac] = {
+        hostname: "x86-node", disk: "", role: "worker",
+        os: "ubuntu-26.04", queued_at: new Date().toISOString(),
+      };
+    });
+
+    const res = await app.inject({ method: "GET", url: `/dispatch?mac=${mac}&arch=x86_64` });
+    expect(res.body).toContain("ubuntu-vmlinuz");
+    expect(res.body).not.toContain("CANNOT BOOT");
+  });
+
+  it("serves a rescue kernel for the recorded architecture, not the requester's", async () => {
+    // The Spark case: machine known to be aarch64, queued for rescue.
+    state.update((s) => {
+      s.discovered[mac] = hardware(mac, { arch: "aarch64" });
+      s.debug[mac] = { hostname: "spark-2935", queued_at: new Date().toISOString() };
+    });
+
+    const res = await app.inject({ method: "GET", url: `/dispatch?mac=${mac}` });
+    expect(res.body).toContain("DEBUG/RESCUE MODE");
+    expect(res.body).toContain("/vmlinuz-aarch64");
+    expect(res.body).toContain("inst.rescue");
+    expect(res.body).toContain("inst.sshd");
+  });
+});
+
+describe("--pxe-boot root device", () => {
+  let testDir: string;
+  let app: FastifyInstance;
+  let state: StateManager;
+  const mac = "aa:bb:cc:dd:ee:ff";
+
+  beforeEach(() => {
+    testDir = join(tmpdir(), `bastion-root-test-${Date.now()}-${Math.random().toString(36).slice(2)}`);
+    mkdirSync(join(testDir, "http"), { recursive: true });
+    mkdirSync(join(testDir, "tftp"), { recursive: true });
+    const result = createApp(createTestConfig(testDir));
+    app = result.app;
+    state = result.state;
+  });
+
+  afterEach(async () => {
+    await app.close();
+    rmSync(testDir, { recursive: true, force: true });
+  });
+
+  it("uses the root device recorded on the machine", async () => {
+    state.update((s) => {
+      s.installed[mac] = {
+        hostname: "worker-1", role: "worker", ip: "10.0.0.50",
+        installed_at: new Date().toISOString(),
+        root_device: "/dev/mapper/otherVG-root",
+        root_args: "rd.lvm.lv=otherVG/root",
+      };
+      s.debug[mac] = { hostname: "worker-1", queued_at: new Date().toISOString(), pxeBoot: true };
+    });
+
+    const res = await app.inject({ method: "GET", url: `/dispatch?mac=${mac}` });
+    expect(res.body).toContain("PXE BOOT (debug)");
+    expect(res.body).toContain("root=/dev/mapper/otherVG-root");
+    expect(res.body).toContain("rd.lvm.lv=otherVG/root");
+    // The old hardcoded layout must not leak back in.
+    expect(res.body).not.toContain("labvg");
+  });
+
+  it("falls back to rescue rather than guessing when no root device is known", async () => {
+    state.update((s) => {
+      s.installed[mac] = {
+        hostname: "spark-2935", role: "worker", ip: "192.168.8.12",
+        installed_at: new Date().toISOString(), arch: "aarch64",
+      };
+      s.debug[mac] = { hostname: "spark-2935", queued_at: new Date().toISOString(), pxeBoot: true };
+    });
+
+    const res = await app.inject({ method: "GET", url: `/dispatch?mac=${mac}` });
+    expect(res.body).toContain("DEBUG/RESCUE MODE");
+    expect(res.body).toContain("no root device is recorded");
+    expect(res.body).toContain("debug-setup.sh");
+    expect(res.body).not.toContain("root=");
+    // And it is still the right architecture.
+    expect(res.body).toContain("/vmlinuz-aarch64");
+  });
+
+  it("records a root device reported from a rescue shell without erasing hardware info", async () => {
+    state.update((s) => {
+      s.discovered[mac] = hardware(mac, { product: "DGX Spark", manufacturer: "NVIDIA", arch: "aarch64" });
+    });
+
+    const res = await app.inject({
+      method: "POST",
+      url: "/api/discover",
+      headers: { "Content-Type": "application/json" },
+      body: JSON.stringify({ mac, root_device: "/dev/nvme0n1p2" }),
+    });
+    expect(res.statusCode).toBe(200);
+
+    const hw = state.load().discovered[mac];
+    expect(hw?.root_device).toBe("/dev/nvme0n1p2");
+    // The partial report must not blank what we already knew.
+    expect(hw?.product).toBe("DGX Spark");
+    expect(hw?.cpu_cores).toBe(4);
+    expect(hw?.arch).toBe("aarch64");
+  });
+});
+
+describe("dnsmasq architecture detection", () => {
+  const conf = renderDnsmasqConf(createTestConfig("/tmp/unused"));
+
+  it("maps DHCP option 93 values to per-architecture bootloaders", () => {
+    // 11 = ARM 64-bit UEFI
+    expect(conf).toContain("dhcp-match=set:efi-arm64,option:client-arch,11");
+    expect(conf).toContain("dhcp-boot=tag:efi-arm64,tag:!ipxe,ipxe-arm64.efi");
+    // 7 / 9 = x64 UEFI, 0 = x86 BIOS
+    expect(conf).toContain("dhcp-match=set:efi-x86_64,option:client-arch,7");
+    expect(conf).toContain("dhcp-match=set:efi-x86_64,option:client-arch,9");
+    expect(conf).toContain("dhcp-match=set:bios,option:client-arch,0");
+  });
+
+  it("matches arm64 UEFI HTTP boot on 19, not 20", () => {
+    // IANA: 19 = arm uefi 64 boot from http, 20 = pc/at bios boot from http.
+    expect(conf).toContain("dhcp-match=set:httpboot-arm64,option:client-arch,19");
+    expect(conf).not.toContain("dhcp-match=set:httpboot-arm64,option:client-arch,20");
+    expect(conf).toContain("dhcp-match=set:httpboot-x86_64,option:client-arch,16");
+  });
+
+  it("offers an arm64 PXE service directive in proxy mode", () => {
+    expect(conf).toContain('pxe-service=tag:!ipxe,ARM64_EFI,"PXE Boot",ipxe-arm64.efi');
+  });
+});

bastion/src/bastion/tests/fixtures/ipxe-x86_64-golden.json

@@ -0,0 +1,8 @@
+{
+  "boot": "#!ipxe\n\necho\necho ============================================\necho   Lab PXE Bastion\necho   Contacting server for instructions...\necho ============================================\necho\n\nchain http://10.0.0.1:8080/dispatch?mac=${net0/mac}\n",
+  "discover": "#!ipxe\n\necho\necho =============================================\necho   Lab PXE Bastion - DISCOVERY MODE\necho   MAC: aa:bb:cc:dd:ee:ff\necho   Collecting hardware info...\necho =============================================\necho\n\nkernel http://10.0.0.1:8080/vmlinuz inst.ks=http://10.0.0.1:8080/discover.ks inst.stage2=https://download.fedoraproject.org/pub/fedora/linux/releases/43/Everything/x86_64/os inst.text nomodeset\ninitrd http://10.0.0.1:8080/initrd.img\nboot\n",
+  "install": "#!ipxe\n\necho\necho =============================================\necho   Lab PXE Bastion - INSTALLING Fedora 43\necho   Target: worker-1\necho   MAC:    aa:bb:cc:dd:ee:ff\necho =============================================\necho\n\nkernel http://10.0.0.1:8080/vmlinuz inst.ks=http://10.0.0.1:8080/ks?mac=aa:bb:cc:dd:ee:ff inst.repo=https://download.fedoraproject.org/pub/fedora/linux/releases/43/Everything/x86_64/os inst.text nomodeset\ninitrd http://10.0.0.1:8080/initrd.img\nboot\n",
+  "debug": "#!ipxe\n\necho\necho =============================================\necho   Lab PXE Bastion - DEBUG/RESCUE MODE\necho   Target: worker-1\necho   MAC:    aa:bb:cc:dd:ee:ff\necho =============================================\necho\n\nkernel http://10.0.0.1:8080/vmlinuz inst.rescue inst.text inst.sshd inst.ks=http://10.0.0.1:8080/debug.ks?mac=aa:bb:cc:dd:ee:ff inst.stage2=https://download.fedoraproject.org/pub/fedora/linux/releases/43/Everything/x86_64/os\ninitrd http://10.0.0.1:8080/initrd.img\nboot\n",
+  "pxeBoot": "#!ipxe\n\necho\necho =============================================\necho   Lab PXE Bastion - PXE BOOT (debug)\necho   Target: worker-1\necho   MAC:    aa:bb:cc:dd:ee:ff\necho   Kernel+initrd from PXE, root from NVMe\necho =============================================\necho\n\nkernel http://10.0.0.1:8080/vmlinuz root=/dev/mapper/labvg-root ro rd.lvm.lv=labvg/root rd.lvm.lv=labvg/swap console=tty0\ninitrd http://10.0.0.1:8080/initrd.img\nboot\n",
+  "localBoot": "#!ipxe\n\necho\necho =============================================\necho   Lab PXE Bastion - worker-1\necho   Already installed, booting from local disk\necho =============================================\necho\nsleep 3\nexit 1\n"
+}

bastion/src/bastion/tests/install-guard.test.ts

@@ -0,0 +1,194 @@
+// Installs must never reach a machine running a vendor OS we cannot restore.
+//
+// This is the guardrail that stops someone reinstalling a DGX Spark at 2am. Rescue is
+// deliberately still allowed for the same machines -- that is the whole point.
+
+import { describe, it, expect, beforeEach, afterEach } from "vitest";
+import { mkdirSync, rmSync } from "node:fs";
+import { join } from "node:path";
+import { tmpdir } from "node:os";
+import type { BastionConfig, BastionState, HardwareInfo } from "@lab/shared";
+import { classifyOnboard } from "@lab/shared";
+import { createApp } from "../src/server.js";
+import { checkInstallAllowed } from "../src/services/install-guard.js";
+import type { FastifyInstance } from "fastify";
+import type { StateManager } from "../src/services/state.js";
+
+// The real machines this exists to protect.
+const SPARK_2935 = "4c:bb:47:7f:29:35";
+const SPARK_3A1C = "48:21:0b:96:3a:1c";
+const ORDINARY = "aa:bb:cc:dd:ee:ff";
+
+function createTestConfig(testDir: string): BastionConfig {
+  return {
+    fedoraVersion: "43", arch: "x86_64", httpPort: 0,
+    timezone: "Europe/London", locale: "en_GB.UTF-8", bastionDir: testDir,
+    domain: "test.local", dhcpMode: "proxy", dhcpRangeStart: "", dhcpRangeEnd: "",
+    ubuntuVersion: "26.04", ubuntuMirror: "https://releases.ubuntu.com/26.04",
+    iface: "eth0", serverIp: "10.0.0.1", network: "10.0.0.0", gateway: "10.0.0.1",
+    sshKeys: [], adminUser: "testadmin", syslogPort: 15514,
+    skipDnsmasq: true, skipArtifacts: true,
+    fedoraMirror: "https://download.fedoraproject.org/pub/fedora/linux/releases/43/Everything/x86_64/os",
+    tftpDir: join(testDir, "tftp"), httpDir: join(testDir, "http"),
+    stateFile: join(testDir, "state.json"),
+  };
+}
+
+function hardware(mac: string, over: Partial<HardwareInfo> = {}): HardwareInfo {
+  return {
+    mac, product: "TestBox", board: "TestBoard", serial: "SN1",
+    manufacturer: "TestCorp", cpu_model: "Test CPU", cpu_cores: 4, memory_gb: 16,
+    arch: "x86_64", disks: [], nics: [],
+    first_seen: new Date().toISOString(), last_seen: new Date().toISOString(),
+    ...over,
+  };
+}
+
+const emptyState = (): BastionState => ({
+  discovered: {}, install_queue: {}, installed: {}, debug: {},
+});
+
+describe("classifyOnboard", () => {
+  it("recognises a DGX Spark from its DMI identity", () => {
+    expect(classifyOnboard({
+      mac: ORDINARY, manufacturer: "NVIDIA", product: "NVIDIA DGX Spark", board: "GB10",
+    })).toEqual({ onboard: "ssh", vendor_os: "dgx-os" });
+  });
+
+  it("recognises the known Sparks even with no DMI recorded", () => {
+    // Neither Spark has hardware info in bastion state today. A DMI-only rule would
+    // fail open on exactly the machines this protects.
+    expect(classifyOnboard({ mac: SPARK_2935 }).onboard).toBe("ssh");
+    expect(classifyOnboard({ mac: SPARK_3A1C }).onboard).toBe("ssh");
+  });
+
+  it("treats ordinary hardware as PXE-installable", () => {
+    expect(classifyOnboard({
+      mac: ORDINARY, manufacturer: "Beelink", product: "SER9", board: "SER9",
+    })).toEqual({ onboard: "pxe" });
+  });
+
+  it("does not override an explicit classification already on the record", () => {
+    expect(classifyOnboard({
+      mac: SPARK_2935, onboard: "pxe",
+    })).toEqual({ onboard: "pxe" });
+  });
+});
+
+describe("checkInstallAllowed", () => {
+  it("refuses a DGX Spark and explains why", () => {
+    const state = emptyState();
+    state.installed[SPARK_2935] = {
+      hostname: "spark-2935", role: "worker", ip: "192.168.8.12",
+      installed_at: new Date().toISOString(), arch: "aarch64",
+    };
+
+    const result = checkInstallAllowed(state, SPARK_2935, "fedora-43");
+    expect(result.allowed).toBe(false);
+    if (result.allowed === false) {
+      expect(result.error).toContain("spark-2935");
+      expect(result.error).toContain("DGX OS");
+      expect(result.error).toContain("provision debug");
+    }
+  });
+
+  it("refuses a Spark that is only known by MAC", () => {
+    expect(checkInstallAllowed(emptyState(), SPARK_3A1C, "fedora-43").allowed).toBe(false);
+  });
+
+  it("allows an ordinary discovered machine", () => {
+    const state = emptyState();
+    state.discovered[ORDINARY] = hardware(ORDINARY);
+    expect(checkInstallAllowed(state, ORDINARY, "fedora-43").allowed).toBe(true);
+  });
+
+  it("allows Fedora on aarch64", () => {
+    const state = emptyState();
+    state.discovered[ORDINARY] = hardware(ORDINARY, { arch: "aarch64" });
+    expect(checkInstallAllowed(state, ORDINARY, "fedora-43").allowed).toBe(true);
+  });
+
+  it("refuses Ubuntu on aarch64 -- no netboot artifacts are published", () => {
+    const state = emptyState();
+    state.discovered[ORDINARY] = hardware(ORDINARY, { arch: "aarch64" });
+    const result = checkInstallAllowed(state, ORDINARY, "ubuntu-26.04");
+    expect(result.allowed).toBe(false);
+    if (result.allowed === false) {
+      expect(result.error).toContain("aarch64");
+    }
+  });
+
+  it("allows Ubuntu on x86_64", () => {
+    const state = emptyState();
+    state.discovered[ORDINARY] = hardware(ORDINARY, { arch: "x86_64" });
+    expect(checkInstallAllowed(state, ORDINARY, "ubuntu-26.04").allowed).toBe(true);
+  });
+});
+
+describe("install route enforces the guard", () => {
+  let testDir: string;
+  let app: FastifyInstance;
+  let state: StateManager;
+
+  beforeEach(() => {
+    testDir = join(tmpdir(), `bastion-guard-test-${Date.now()}-${Math.random().toString(36).slice(2)}`);
+    mkdirSync(join(testDir, "http"), { recursive: true });
+    mkdirSync(join(testDir, "tftp"), { recursive: true });
+    const result = createApp(createTestConfig(testDir));
+    app = result.app;
+    state = result.state;
+  });
+
+  afterEach(async () => {
+    await app.close();
+    rmSync(testDir, { recursive: true, force: true });
+  });
+
+  it("rejects POST /api/install for a Spark and queues nothing", async () => {
+    const res = await app.inject({
+      method: "POST",
+      url: "/api/install",
+      headers: { "Content-Type": "application/json" },
+      body: JSON.stringify({ mac: SPARK_2935, hostname: "spark-2935", role: "worker" }),
+    });
+
+    expect(res.statusCode).toBe(409);
+    expect(JSON.parse(res.body).error).toContain("Refusing to install");
+    expect(state.load().install_queue[SPARK_2935]).toBeUndefined();
+  });
+
+  it("still serves rescue to a Spark -- debug is never guarded", async () => {
+    state.update((s) => {
+      s.installed[SPARK_2935] = {
+        hostname: "spark-2935", role: "worker", ip: "192.168.8.12",
+        installed_at: new Date().toISOString(), arch: "aarch64",
+      };
+      s.debug[SPARK_2935] = { hostname: "spark-2935", queued_at: new Date().toISOString() };
+    });
+
+    const res = await app.inject({ method: "GET", url: `/dispatch?mac=${SPARK_2935}` });
+    expect(res.statusCode).toBe(200);
+    expect(res.body).toContain("DEBUG/RESCUE MODE");
+    expect(res.body).toContain("/vmlinuz-aarch64");
+  });
+
+  it("a Spark that PXE boots unqueued gets discovery, never an install", async () => {
+    const res = await app.inject({ method: "GET", url: `/dispatch?mac=${SPARK_2935}&arch=arm64` });
+    expect(res.body).toContain("DISCOVERY MODE");
+    expect(res.body).not.toContain("INSTALLING");
+  });
+
+  it("still accepts an ordinary machine", async () => {
+    state.update((s) => { s.discovered[ORDINARY] = hardware(ORDINARY); });
+
+    const res = await app.inject({
+      method: "POST",
+      url: "/api/install",
+      headers: { "Content-Type": "application/json" },
+      body: JSON.stringify({ mac: ORDINARY, hostname: "worker-1", role: "worker" }),
+    });
+
+    expect(res.statusCode).toBe(200);
+    expect(state.load().install_queue[ORDINARY]).toBeDefined();
+  });
+});

bastion/src/bastion/tests/ipxe-x86-regression.test.ts

@@ -0,0 +1,89 @@
+// x86_64 iPXE output regression gate.
+//
+// The aarch64 PXE work must not change what an x86_64 machine is served. The golden
+// fixture was dumped from the templates as they stood before that work started, so
+// any diff here is a regression, not an improvement.
+//
+// The one deliberate exception is renderBootIpxe: its chain URL gained
+// `&arch=${buildarch}` so the dispatch endpoint can observe the client's
+// architecture at boot time. That single change is asserted explicitly below
+// rather than being allowed to slip through the byte-for-byte comparison.
+
+import { describe, it, expect } from "vitest";
+import { readFileSync } from "node:fs";
+import { join } from "node:path";
+import { fileURLToPath } from "node:url";
+import { dirname } from "node:path";
+import {
+  renderBootIpxe,
+  renderDiscoverIpxe,
+  renderInstallIpxe,
+  renderDebugIpxe,
+  renderPxeBootDebugIpxe,
+  renderLocalBootIpxe,
+} from "../src/templates/boot.ipxe.js";
+
+const here = dirname(fileURLToPath(import.meta.url));
+const golden = JSON.parse(
+  readFileSync(join(here, "fixtures", "ipxe-x86_64-golden.json"), "utf-8"),
+) as Record<string, string>;
+
+// Exactly the parameters used to dump the fixture.
+const serverIp = "10.0.0.1";
+const httpPort = 8080;
+const mac = "aa:bb:cc:dd:ee:ff";
+const hostname = "worker-1";
+const fedoraVersion = "43";
+const fedoraMirror =
+  "https://download.fedoraproject.org/pub/fedora/linux/releases/43/Everything/x86_64/os";
+
+// The x86_64 LVM layout the fixture was captured with. Before this work the values
+// were hardcoded in the template; they are now supplied by the caller from machine
+// state, so the fixture pins the rendering, not the defaults.
+const x86Root = {
+  rootDevice: "/dev/mapper/labvg-root",
+  rootArgs: "rd.lvm.lv=labvg/root rd.lvm.lv=labvg/swap",
+};
+
+describe("x86_64 iPXE output is unchanged", () => {
+  it("discover script is byte-identical", () => {
+    const rendered = renderDiscoverIpxe({
+      mac, serverIp, httpPort, fedoraMirror, arch: "x86_64",
+    });
+    expect(rendered).toBe(golden["discover"]);
+  });
+
+  it("install script is byte-identical", () => {
+    const rendered = renderInstallIpxe({
+      mac, hostname, serverIp, httpPort, fedoraVersion, fedoraMirror, arch: "x86_64",
+    });
+    expect(rendered).toBe(golden["install"]);
+  });
+
+  it("debug/rescue script is byte-identical", () => {
+    const rendered = renderDebugIpxe({
+      mac, hostname, serverIp, httpPort, fedoraMirror, arch: "x86_64",
+    });
+    expect(rendered).toBe(golden["debug"]);
+  });
+
+  it("--pxe-boot script is byte-identical when state carries the Fedora LVM layout", () => {
+    const rendered = renderPxeBootDebugIpxe({
+      mac, hostname, serverIp, httpPort, arch: "x86_64", ...x86Root,
+    });
+    expect(rendered).toBe(golden["pxeBoot"]);
+  });
+
+  it("local boot script is byte-identical", () => {
+    expect(renderLocalBootIpxe(hostname)).toBe(golden["localBoot"]);
+  });
+
+  it("boot.ipxe differs only by the &arch= chain parameter", () => {
+    const rendered = renderBootIpxe({ serverIp, httpPort });
+    // The sole intended difference.
+    expect(rendered).toBe(golden["boot"].replace(
+      "/dispatch?mac=${net0/mac}",
+      "/dispatch?mac=${net0/mac}&arch=${buildarch}",
+    ));
+  });
+});

bastion/src/cli/src/api/client.ts

@@ -110,6 +110,7 @@ export class LabdClient {
     memory_gb?: number; arch?: string;
     disks?: Array<{ name: string; size_gb: number; model: string }>;
     nics?: Array<{ name: string; mac: string; state: string }>;
+    root_device?: string; root_args?: string;
   }): Promise<{ status: string; error?: string }> {
     return this.request("POST", "/api/machines/discover", { body: data });
   }

bastion/src/cli/src/commands/debug.ts

@@ -8,6 +8,7 @@ import { join } from "node:path";
 import { Command } from "commander";
 import type { BastionState } from "@lab/shared";
 import { getLabdClient } from "../api/config.js";
+import { ROOT_DEVICE_PROBE, parseRootProbe } from "../utils/hardware-probe.js";
 
 /** Resolve a target (hostname, MAC, or IP) to {mac, hostname, ip} from state. */
 function resolveTarget(
@@ -44,6 +45,54 @@ function resolveTarget(
   return null;
 }
 
+/** The local admin account to SSH as (root is not usable — it has no key here). */
+function sshUser(): string {
+  const adminUser = process.env["SUDO_USER"] ?? process.env["USER"] ?? "";
+  return adminUser === "root" ? "" : adminUser;
+}
+
+/** Common ssh arguments, ending with user@host. Null when there is no usable user. */
+function sshBaseArgs(ip: string): string[] | null {
+  const user = sshUser();
+  if (user === "") return null;
+
+  const sudoUser = process.env["SUDO_USER"];
+  const realHome = sudoUser !== undefined ? join("/home", sudoUser) : homedir();
+  const sshKey = ["id_ed25519", "id_rsa", "id_ecdsa"]
+    .map((name) => join(realHome, ".ssh", name))
+    .find((k) => existsSync(k));
+
+  return [
+    "-o", "StrictHostKeyChecking=no",
+    "-o", "UserKnownHostsFile=/dev/null",
+    "-o", "ConnectTimeout=10",
+    ...(sshKey !== undefined ? ["-i", sshKey] : []),
+    `${user}@${ip}`,
+  ];
+}
+
+/**
+ * Run a shell script on the target as root and return its stdout, or null.
+ *
+ * The script goes over stdin rather than the command line so it can contain quotes
+ * without a second round of shell escaping. `sudo -n` fails fast instead of hanging on
+ * a password prompt that would then eat the script.
+ */
+function sshCapture(ip: string, script: string): string | null {
+  const base = sshBaseArgs(ip);
+  if (base === null) return null;
+  try {
+    return execFileSync("ssh", [...base, "sudo", "-n", "sh", "-s"], {
+      input: script,
+      encoding: "utf-8",
+      stdio: ["pipe", "pipe", "pipe"],
+      timeout: 30_000,
+    });
+  } catch {
+    return null;
+  }
+}
+
 export function registerDebugCommand(parent: Command): void {
   parent
     .command("debug <target>")
@@ -71,6 +120,31 @@ export function registerDebugCommand(parent: Command): void {
       }
 
       const { mac, hostname, ip } = resolved;
+
+      // --pxe-boot needs a root= for the installed system. If the machine is still
+      // reachable, observe it now rather than assuming a disk layout: a wrong root=
+      // leaves the machine unbootable. If it isn't reachable, dispatch falls back to
+      // rescue and the operator reports the real one from there.
+      if (opts.pxeBoot === true && ip !== "") {
+        const known = state.installed[mac]?.root_device ?? state.discovered[mac]?.root_device;
+        if (known === undefined || known === "") {
+          console.log(`No root device recorded for ${hostname}. Probing over SSH...`);
+          const probe = sshCapture(ip, ROOT_DEVICE_PROBE);
+          const root = probe === null ? {} : parseRootProbe(probe);
+          if (root.root_device !== undefined) {
+            console.log(`  root=${root.root_device}${root.root_args !== undefined ? ` ${root.root_args}` : ""}`);
+            try {
+              await client.discoverMachine({ mac, ...root });
+            } catch (err) {
+              console.error(`  Could not record it: ${err instanceof Error ? err.message : String(err)}`);
+            }
+          } else {
+            console.log("  Probe failed. Booting rescue instead; report the root device with:");
+            console.log("    curl http://<bastion>:8080/debug-setup.sh | bash");
+          }
+        }
+      }
+
       console.log(`Queuing debug mode for ${hostname} (${mac})...`);
 
       try {
@@ -86,32 +160,15 @@ export function registerDebugCommand(parent: Command): void {
 
       // Try SSH reboot into PXE
       if (ip !== "") {
-        const adminUser = process.env["SUDO_USER"] ?? process.env["USER"] ?? "";
-        const effectiveUser = adminUser === "root" ? "" : adminUser;
-
-        if (effectiveUser !== "") {
-          console.log(`\nAttempting SSH reboot into PXE (${effectiveUser}@${ip})...`);
-
-          const sudoUser = process.env["SUDO_USER"];
-          const realHome = sudoUser !== undefined ? join("/home", sudoUser) : homedir();
-          const keyPaths = [
-            join(realHome, ".ssh", "id_ed25519"),
-            join(realHome, ".ssh", "id_rsa"),
-            join(realHome, ".ssh", "id_ecdsa"),
-          ];
-          const sshKey = keyPaths.find(k => existsSync(k));
-
-          const sshArgs = [
-            "-o", "StrictHostKeyChecking=no",
-            "-o", "UserKnownHostsFile=/dev/null",
-            "-o", "ConnectTimeout=10",
-            ...(sshKey !== undefined ? ["-i", sshKey] : []),
-            `${effectiveUser}@${ip}`,
-            'PXE_ENTRY=$(sudo efibootmgr | grep -iE "pxe|network|ipv4" | head -1 | grep -oP "Boot\\K[0-9A-F]+"); if [ -n "$PXE_ENTRY" ]; then sudo efibootmgr --bootnext "$PXE_ENTRY" && echo "PXE set as next boot" && sudo reboot; else echo "No PXE boot entry found, rebooting anyway..." && sudo reboot; fi',
-          ];
+        const base = sshBaseArgs(ip);
+        if (base !== null) {
+          console.log(`\nAttempting SSH reboot into PXE (${sshUser()}@${ip})...`);
 
           try {
-            execFileSync("ssh", sshArgs, { stdio: "inherit" });
+            execFileSync("ssh", [
+              ...base,
+              'PXE_ENTRY=$(sudo efibootmgr | grep -iE "pxe|network|ipv4" | head -1 | grep -oP "Boot\\K[0-9A-F]+"); if [ -n "$PXE_ENTRY" ]; then sudo efibootmgr --bootnext "$PXE_ENTRY" && echo "PXE set as next boot" && sudo reboot; else echo "No PXE boot entry found, rebooting anyway..." && sudo reboot; fi',
+            ], { stdio: "inherit" });
           } catch {
             // SSH connection closing during reboot is expected
           }

bastion/src/cli/src/commands/recheck.ts

@@ -4,6 +4,7 @@
 import type { Command } from "commander";
 import { sshExec } from "@lab/modules";
 import { getLabdClient } from "../api/config.js";
+import { ROOT_DEVICE_PROBE } from "../utils/hardware-probe.js";
 
 const BOLD = "\x1b[1m";
 const GREEN = "\x1b[0;32m";
@@ -24,7 +25,9 @@ const HW_COLLECT_SCRIPT = [
   'N=$(grep -c "^processor" /proc/cpuinfo 2>/dev/null || echo 0)',
   'R=$(awk "/MemTotal/ {printf \\"%d\\", \\$2/1024/1024}" /proc/meminfo 2>/dev/null || echo 0)',
   'A=$(uname -m)',
-  'printf \'{"product":"%s","board":"%s","serial":"%s","manufacturer":"%s","cpu_model":"%s","cpu_cores":%s,"memory_gb":%s,"arch":"%s"}\\n\' "$P" "$B" "$S" "$M" "$C" "$N" "$R" "$A"',
+  // Root filesystem, so --pxe-boot has a root= to use instead of assuming our layout.
+  ROOT_DEVICE_PROBE,
+  'printf \'{"product":"%s","board":"%s","serial":"%s","manufacturer":"%s","cpu_model":"%s","cpu_cores":%s,"memory_gb":%s,"arch":"%s","root_device":"%s","root_args":"%s"}\\n\' "$P" "$B" "$S" "$M" "$C" "$N" "$R" "$A" "$RD" "$RA"',
 ].join("; ");
 
 export function registerRecheckCommand(parent: Command): void {
@@ -81,7 +84,10 @@ export function registerRecheckCommand(parent: Command): void {
           const cpu = hwData.cpu_model || "?";
           const cores = hwData.cpu_cores || "?";
           const mem = hwData.memory_gb || "?";
-          console.log(`${GREEN}OK${RESET} ${DIM}${cpu}, ${cores} cores, ${mem}GB${RESET}`);
+          const root = typeof hwData.root_device === "string" && hwData.root_device !== ""
+            ? `, root=${hwData.root_device}`
+            : "";
+          console.log(`${GREEN}OK${RESET} ${DIM}${cpu}, ${cores} cores, ${mem}GB${root}${RESET}`);
           updated++;
         } catch (err) {
           console.log(`${RED}FAIL${RESET} ${DIM}${err instanceof Error ? err.message : String(err)}${RESET}`);

bastion/src/cli/src/commands/serve.ts

@@ -11,7 +11,7 @@ export function registerStartCommand(parent: Command): void {
     .command("start")
     .description("Start the bastion server (HTTP + dnsmasq PXE)")
     .option("--port <port>", "HTTP port", "8080")
-    .option("--dir <dir>", "Bastion data directory", "/tmp/lab-bastion")
+    .option("--dir <dir>", "Bastion data directory", process.env["BASTION_DIR"] ?? "/tmp/lab-bastion")
     .option("--domain <domain>", "Internal domain for hostnames", "ad.itaz.eu")
     .option("--dhcp-mode <mode>", "DHCP mode: proxy or full", "proxy")
     .option("--fedora <version>", "Fedora version", "43")

bastion/src/cli/src/commands/stop.ts

@@ -8,7 +8,7 @@ export function registerStopCommand(parent: Command): void {
   parent
     .command("stop")
     .description("Stop a running bastion server")
-    .option("--dir <dir>", "Bastion data directory", "/tmp/lab-bastion")
+    .option("--dir <dir>", "Bastion data directory", process.env["BASTION_DIR"] ?? "/tmp/lab-bastion")
     .action((opts: { dir: string }) => {
       const pidFile = `${opts.dir}/bastion.pid`;
 

bastion/src/cli/src/utils/hardware-probe.ts

@@ -0,0 +1,59 @@
+// Shell snippets for observing a machine's hardware over SSH.
+//
+// Pure shell + awk, no Python: these run on whatever the target happens to be,
+// including a minimal rescue environment.
+
+/**
+ * Report the root filesystem and any dracut arguments needed to assemble it.
+ *
+ * Emits two lines:
+ *   ROOT_DEVICE=<device>
+ *   ROOT_ARGS=<args>
+ *
+ * Used by `--pxe-boot`, which boots the installed system with a kernel and initrd from
+ * the network. Getting root= wrong there leaves the machine unbootable, so this observes
+ * the machine rather than assuming our Fedora LVM layout.
+ *
+ * Device form is chosen for stability across reboots: LVM logical volumes keep their
+ * /dev/mapper path, anything else is reported by UUID, which survives device renumbering.
+ */
+export const ROOT_DEVICE_PROBE = [
+  'RD=$(findmnt -no SOURCE / 2>/dev/null | head -1)',
+  'RA=""',
+  'RT=$(lsblk -no TYPE "$RD" 2>/dev/null | head -1)',
+  'if [ "$RT" = "lvm" ]; then',
+  '  VGLV=$(lvs --noheadings -o vg_name,lv_name "$RD" 2>/dev/null | awk \'{print $1"/"$2}\')',
+  '  [ -n "$VGLV" ] && RA="rd.lvm.lv=$VGLV"',
+  // Swap must be assembled too or resume= stalls the boot waiting for it.
+  '  SW=$(awk \'NR>1 {print $1; exit}\' /proc/swaps 2>/dev/null)',
+  '  if [ -n "$SW" ] && [ "$(lsblk -no TYPE "$SW" 2>/dev/null | head -1)" = "lvm" ]; then',
+  '    SWVGLV=$(lvs --noheadings -o vg_name,lv_name "$SW" 2>/dev/null | awk \'{print $1"/"$2}\')',
+  '    [ -n "$SWVGLV" ] && [ "$SWVGLV" != "$VGLV" ] && RA="$RA rd.lvm.lv=$SWVGLV"',
+  '  fi',
+  'elif [ -n "$RD" ]; then',
+  '  U=$(findmnt -no UUID / 2>/dev/null | head -1)',
+  '  [ -n "$U" ] && RD="UUID=$U"',
+  'fi',
+  'printf \'ROOT_DEVICE=%s\\nROOT_ARGS=%s\\n\' "$RD" "$RA"',
+].join("; ");
+
+export interface RootInfo {
+  root_device?: string;
+  root_args?: string;
+}
+
+/** Parse the ROOT_DEVICE/ROOT_ARGS lines emitted by ROOT_DEVICE_PROBE. */
+export function parseRootProbe(stdout: string): RootInfo {
+  const out: RootInfo = {};
+  for (const line of stdout.split("\n")) {
+    const trimmed = line.trim();
+    if (trimmed.startsWith("ROOT_DEVICE=")) {
+      const v = trimmed.slice("ROOT_DEVICE=".length).trim();
+      if (v !== "") out.root_device = v;
+    } else if (trimmed.startsWith("ROOT_ARGS=")) {
+      const v = trimmed.slice("ROOT_ARGS=".length).trim();
+      if (v !== "") out.root_args = v;
+    }
+  }
+  return out;
+}

bastion/src/core/package.json

@@ -0,0 +1,23 @@
+{
+  "name": "@lab/core",
+  "version": "0.1.0",
+  "private": true,
+  "type": "module",
+  "main": "./dist/index.js",
+  "types": "./dist/index.d.ts",
+  "exports": {
+    ".": {
+      "import": "./dist/index.js",
+      "types": "./dist/index.d.ts"
+    }
+  },
+  "scripts": {
+    "build": "tsc --build",
+    "clean": "rimraf dist",
+    "test": "vitest",
+    "test:run": "vitest run"
+  },
+  "dependencies": {
+    "@pulumi/pulumi": "^3.0.0"
+  }
+}

bastion/src/core/src/audit.ts

@@ -0,0 +1,75 @@
+// Audit event types for the labctl platform.
+// Every mutation is tracked with correlation IDs for causal chains.
+
+export type AuditEventKind =
+  | "resource_created"
+  | "resource_updated"
+  | "resource_deleted"
+  | "resource_state_change"
+  | "plan_generated"
+  | "apply_started"
+  | "apply_step"
+  | "apply_completed"
+  | "driver_translate"
+  | "driver_execute"
+  | "driver_error"
+  | "fleet_discovery"
+  | "fleet_classification"
+  | "fleet_approval"
+  | "fleet_auto_approve"
+  | "pipeline_started"
+  | "pipeline_step_started"
+  | "pipeline_step_completed"
+  | "pipeline_completed"
+  | "deploy_started"
+  | "deploy_completed"
+  | "deploy_failed"
+  | "drift_detected"
+  | "drift_corrected"
+  | "sync_triggered"
+  | "sync_completed"
+  | "auth_login"
+  | "auth_logout"
+  | "auth_bootstrap"
+  | "rbac_decision"
+  | "impersonation"
+  | "server_started"
+  | "controller_started"
+  | "agent_connected"
+  | "agent_disconnected"
+  | "bastion_registered";
+
+export type AuditSource =
+  | "cli"
+  | "labd"
+  | "agent"
+  | "driver"
+  | "fleet-controller"
+  | "sync-controller";
+
+export type AuditResult = "success" | "failure" | "denied" | "skipped";
+
+export interface AuditEvent {
+  id: string;
+  timestamp: Date;
+  eventKind: AuditEventKind;
+  source: AuditSource;
+  verified: boolean;
+
+  userId?: string;
+  userName?: string;
+  sessionId?: string;
+  environmentName?: string;
+  accountName?: string;
+
+  resourceKind?: string;
+  resourceName?: string;
+
+  correlationId: string;
+  parentEventId?: string;
+
+  details: Record<string, unknown>;
+  result: AuditResult;
+  error?: string;
+  durationMs?: number;
+}

bastion/src/core/src/auth.ts

@@ -0,0 +1,50 @@
+// Auth types for the labctl platform.
+// Bearer token auth for CLI/SDK. mTLS stays for agent/bastion.
+
+export type UserRole = "USER" | "ADMIN";
+
+export interface User {
+  id: string;
+  email: string;
+  name?: string;
+  role: UserRole;
+  createdAt: Date;
+}
+
+export interface Session {
+  id: string;
+  userId: string;
+  token: string;
+  expiresAt: Date;
+  createdAt: Date;
+}
+
+export interface Group {
+  id: string;
+  name: string;
+  description?: string;
+}
+
+export type SubjectKind = "User" | "Group" | "ServiceAccount";
+
+export interface RoleBinding {
+  role: "view" | "edit" | "create" | "delete" | "run" | "admin";
+  resource: string;
+  name?: string;
+  environment?: string;
+  action?: string;
+}
+
+export interface RbacSubject {
+  kind: SubjectKind;
+  name: string;
+}
+
+export interface RbacDefinition {
+  id: string;
+  name: string;
+  subjects: RbacSubject[];
+  roleBindings: RoleBinding[];
+  createdAt: Date;
+  updatedAt: Date;
+}

bastion/src/core/src/environment.ts

@@ -0,0 +1,24 @@
+// Environment and Account types.
+// An Environment is a logical boundary (production, staging, dev).
+// An Account is a configured driver instance with credentials.
+
+export interface Environment {
+  id: string;
+  name: string;
+  status: "active" | "archived";
+  createdAt: Date;
+}
+
+export interface Account {
+  id: string;
+  name: string;
+  driver: string;
+  config: Record<string, unknown>;
+  createdAt: Date;
+}
+
+export interface Binding {
+  id: string;
+  environmentId: string;
+  accountId: string;
+}

bastion/src/core/src/index.ts

@@ -0,0 +1,9 @@
+// @lab/core — foundation types for the labctl platform.
+// Phase 1 stub: resource types, auth types, audit types, Output<T>.
+// Phase 5 adds: CompositeResource, evaluator integration, full SDK.
+
+export * from "./resource.js";
+export * from "./environment.js";
+export * from "./audit.js";
+export * from "./auth.js";
+export { Output, output, all, interpolate, secret } from "./output.js";

bastion/src/core/src/output.ts

@@ -0,0 +1,5 @@
+// Re-export Pulumi's Output<T> type for use across the platform.
+// Cloud drivers use this for future values (endpoints, IPs, kubeconfigs).
+// Phase 1: type re-export only. Phase 5 adds full evaluator integration.
+
+export { Output, output, all, interpolate, secret } from "@pulumi/pulumi";

bastion/src/core/src/resource.ts

@@ -0,0 +1,83 @@
+// Core resource types for the labctl platform.
+// Every managed thing (Server, Database, App, Cluster) is a Resource.
+
+export type ResourceOrigin = "file" | "cli" | "fleet" | "imported";
+export type ResourceManagedBy = "gitops" | "manual" | "auto";
+
+export type ResourceStatus =
+  | "pending"
+  | "creating"
+  | "ready"
+  | "updating"
+  | "deleting"
+  | "error"
+  | "unknown";
+
+export interface ResourceMetadata {
+  kind: string;
+  name: string;
+  environmentId: string;
+  accountId: string;
+  origin: ResourceOrigin;
+  managedBy: ResourceManagedBy;
+  sourceRef?: string;
+}
+
+export interface ResourceState {
+  status: ResourceStatus;
+  message?: string;
+  lastReconciled?: Date;
+  platformRef?: string;
+}
+
+export interface Resource<TSpec = Record<string, unknown>> {
+  id: string;
+  metadata: ResourceMetadata;
+  desiredSpec: TSpec;
+  actualSpec?: TSpec;
+  state: ResourceState;
+  createdAt: Date;
+  updatedAt: Date;
+}
+
+// Well-known resource kinds. Drivers register additional kinds.
+export const RESOURCE_KINDS = {
+  SERVER: "server",
+  DATABASE: "database",
+  CACHE: "cache",
+  CLUSTER: "cluster",
+  APP: "app",
+  SERVICE: "service",
+  CRONJOB: "cronjob",
+  NETWORK: "network",
+  LOADBALANCER: "loadbalancer",
+  DNSZONE: "dnszone",
+  CERTIFICATE: "certificate",
+  OBJECTSTORE: "objectstore",
+  QUEUE: "queue",
+  SECRET: "secret",
+  FLEET: "fleet",
+} as const;
+
+export type ResourceKind = (typeof RESOURCE_KINDS)[keyof typeof RESOURCE_KINDS];
+
+// Resource aliases for CLI (kubectl-style shortnames)
+export const RESOURCE_ALIASES: Record<string, string> = {
+  srv: "server",
+  db: "database",
+  cl: "cluster",
+  svc: "service",
+  cj: "cronjob",
+  lb: "loadbalancer",
+  dns: "dnszone",
+  cert: "certificate",
+  os: "objectstore",
+  mq: "queue",
+  sec: "secret",
+  fl: "fleet",
+};
+
+export function resolveResourceKind(input: string): string {
+  const lower = input.toLowerCase();
+  return RESOURCE_ALIASES[lower] ?? lower;
+}

bastion/src/core/tsconfig.json

@@ -0,0 +1,8 @@
+{
+  "extends": "../../tsconfig.base.json",
+  "compilerOptions": {
+    "rootDir": "src",
+    "outDir": "dist"
+  },
+  "include": ["src/**/*.ts"]
+}

bastion/src/labd/package.json

@@ -26,8 +26,10 @@
   "dependencies": {
     "@fastify/rate-limit": "^10.3.0",
     "@fastify/websocket": "^11.0.2",
+    "@lab/core": "workspace:^",
     "@lab/shared": "workspace:*",
     "@prisma/client": "^6.9.0",
+    "bcryptjs": "^3.0.3",
     "fastify": "^5.3.3",
     "winston": "^3.17.0",
     "ws": "^8.19.0",
@@ -37,6 +39,7 @@
     "seed": "tsx prisma/seed.ts"
   },
   "devDependencies": {
+    "@types/bcryptjs": "^3.0.0",
     "@types/node": "^22.14.1",
     "@types/ws": "^8.18.1",
     "prisma": "^6.9.0",

bastion/src/labd/prisma/schema.prisma

@@ -7,23 +7,241 @@ datasource db {
   url      = env("DATABASE_URL")
 }
 
+// ── Auth (mcpctl pattern: email/password + bearer token sessions) ──
+
+model User {
+  id        String   @id @default(cuid())
+  email     String   @unique
+  password  String   // bcrypt
+  name      String?
+  role      UserRole @default(USER)
+  createdAt DateTime @default(now())
+  updatedAt DateTime @updatedAt
+
+  sessions  Session[]
+  auditLogs AuditEvent[]
+  groups    GroupMember[]
+}
+
+enum UserRole {
+  USER
+  ADMIN
+}
+
+model Session {
+  id        String   @id @default(cuid())
+  userId    String
+  user      User     @relation(fields: [userId], references: [id], onDelete: Cascade)
+  token     String   @unique
+  expiresAt DateTime
+  createdAt DateTime @default(now())
+
+  @@index([userId])
+  @@index([token])
+}
+
+model Group {
+  id          String        @id @default(cuid())
+  name        String        @unique
+  description String?
+  createdAt   DateTime      @default(now())
+  members     GroupMember[]
+}
+
+model GroupMember {
+  id      String @id @default(cuid())
+  groupId String
+  group   Group  @relation(fields: [groupId], references: [id], onDelete: Cascade)
+  userId  String
+  user    User   @relation(fields: [userId], references: [id], onDelete: Cascade)
+
+  @@unique([groupId, userId])
+}
+
+model ServiceAccount {
+  id        String   @id @default(cuid())
+  name      String   @unique
+  token     String   @unique
+  createdAt DateTime @default(now())
+}
+
+// ── RBAC (mcpctl pattern: named definitions with JSON subjects/bindings) ──
+
+model RbacDefinition {
+  id           String   @id @default(cuid())
+  name         String   @unique
+  subjects     Json     // [{kind: "User"|"Group"|"ServiceAccount", name: string}]
+  roleBindings Json     // [{role, resource, name?, environment?, action?}]
+  createdAt    DateTime @default(now())
+  updatedAt    DateTime @updatedAt
+}
+
+// ── Audit (mcpctl pattern: fire-and-forget with correlation IDs) ──
+
+model AuditEvent {
+  id              String   @id @default(cuid())
+  timestamp       DateTime @default(now())
+  eventKind       String
+  source          String   // cli | labd | agent | driver | fleet-controller | sync-controller
+  verified        Boolean  @default(false)
+
+  userId          String?
+  user            User?    @relation(fields: [userId], references: [id])
+  userName        String?
+  sessionId       String?
+  environmentName String?
+  accountName     String?
+
+  resourceKind    String?
+  resourceName    String?
+
+  correlationId   String
+  parentEventId   String?
+
+  details         Json     @default("{}")
+  result          String   // success | failure | denied | skipped
+  error           String?
+  durationMs      Int?
+
+  @@index([correlationId])
+  @@index([eventKind, timestamp])
+  @@index([environmentName, timestamp])
+  @@index([resourceKind, resourceName])
+  @@index([userId, timestamp])
+}
+
+// ── Core infrastructure ──
+
+model Environment {
+  id        String   @id @default(cuid())
+  name      String   @unique
+  status    String   @default("active") // active | archived
+  createdAt DateTime @default(now())
+  updatedAt DateTime @updatedAt
+
+  bindings  Binding[]
+  resources Resource[]
+}
+
+model Account {
+  id        String   @id @default(cuid())
+  name      String   @unique
+  driver    String   // baremetal-pxe | aws | gcp | kubernetes | ovh
+  config    Json     @default("{}")
+  // Credentials stored in Infisical, referenced by secretPath
+  secretPath String?
+  createdAt DateTime @default(now())
+  updatedAt DateTime @updatedAt
+
+  bindings  Binding[]
+  resources Resource[]
+}
+
+model Binding {
+  id            String      @id @default(cuid())
+  environmentId String
+  environment   Environment @relation(fields: [environmentId], references: [id], onDelete: Cascade)
+  accountId     String
+  account       Account     @relation(fields: [accountId], references: [id], onDelete: Cascade)
+
+  @@unique([environmentId, accountId])
+}
+
+model Resource {
+  id             String       @id @default(cuid())
+  kind           String
+  name           String
+  environmentId  String
+  environment    Environment  @relation(fields: [environmentId], references: [id])
+  accountId      String
+  account        Account      @relation(fields: [accountId], references: [id])
+  origin         String       @default("cli")    // file | cli | fleet | imported
+  managedBy      String       @default("manual")  // gitops | manual | auto
+  sourceRef      String?
+  desiredSpec    Json         @default("{}")
+  actualSpec     Json?
+  platformRef    String?
+  status         String       @default("pending") // pending | creating | ready | updating | deleting | error
+  statusMessage  String?
+  lastReconciled DateTime?
+  createdAt      DateTime     @default(now())
+  updatedAt      DateTime     @updatedAt
+
+  @@unique([kind, name, environmentId])
+  @@index([environmentId])
+  @@index([accountId])
+  @@index([kind, status])
+}
+
+model Secret {
+  id        String   @id @default(cuid())
+  name      String   @unique
+  // Encrypted data — application-layer encryption as fallback if Infisical unavailable
+  data      Json     @default("{}")
+  version   Int      @default(1)
+  createdAt DateTime @default(now())
+  updatedAt DateTime @updatedAt
+}
+
+// ── Fleet ──
+
+model Fleet {
+  id               String        @id @default(cuid())
+  name             String
+  environmentId    String
+  accountId        String
+  selector         Json          // fact-matching rules
+  onboardPipeline  Json          // step definitions
+  offboardPipeline Json?
+  approvalConfig   Json?
+  status           String        @default("active")
+  createdAt        DateTime      @default(now())
+  updatedAt        DateTime      @updatedAt
+
+  members          FleetMember[]
+}
+
+model FleetMember {
+  id       String   @id @default(cuid())
+  fleetId  String
+  fleet    Fleet    @relation(fields: [fleetId], references: [id], onDelete: Cascade)
+  serverId String
+  status   String   // discovered | pending | onboarding | active | offboarding | removed
+  joinedAt DateTime @default(now())
+
+  @@index([fleetId])
+}
+
+// ── Git sources (for sync controller) ──
+
+model GitSource {
+  id       String    @id @default(cuid())
+  name     String    @unique
+  repo     String
+  branch   String    @default("main")
+  path     String    @default("environments/")
+  lastSync DateTime?
+  createdAt DateTime @default(now())
+}
+
+// ── Existing v1.0 models (kept for bastion/agent compatibility) ──
+
 model Server {
-  id            String   @id @default(uuid())
-  hostname      String   @unique
-  mac           String?  @unique
-  cloud         String   @default("baremetal")
-  environment   String   @default("default")
-  role          String   @default("worker")
-  labels        Json     @default("{}")
+  id            String    @id @default(uuid())
+  hostname      String    @unique
+  mac           String?   @unique
+  cloud         String    @default("baremetal")
+  environment   String    @default("default")
+  role          String    @default("worker")
+  labels        Json      @default("{}")
   ip            String?
   agentVersion  String?
-  status        String   @default("unknown") // unknown, online, offline, provisioning
+  status        String    @default("unknown")
   lastHeartbeat DateTime?
-  createdAt     DateTime @default(now())
-  updatedAt     DateTime @updatedAt
+  createdAt     DateTime  @default(now())
+  updatedAt     DateTime  @updatedAt
 
-  agent         Agent?
-  auditLogs     AuditLog[]
+  agent     Agent?
 }
 
 model Agent {
@@ -33,112 +251,29 @@ model Agent {
   certificatePem String?
   enrolledAt     DateTime @default(now())
   lastSeen       DateTime?
+  facts          Json?    // hardware facts reported by agent
 
   @@index([serverId])
 }
 
-model User {
-  id              String   @id @default(uuid())
-  username        String   @unique
-  displayName     String?
-  certFingerprint String?  @unique
-  createdAt       DateTime @default(now())
-  updatedAt       DateTime @updatedAt
-
-  roleBindings UserRole[]
-  auditLogs    AuditLog[]
-}
-
-model Role {
-  id          String   @id @default(uuid())
-  name        String   @unique
-  description String?
-  createdAt   DateTime @default(now())
-
-  permissions  Permission[]
-  userBindings UserRole[]
-}
-
-model Permission {
-  id          String @id @default(uuid())
-  roleId      String
-  role        Role   @relation(fields: [roleId], references: [id], onDelete: Cascade)
-  type        String @default("allow") // allow or deny
-  action      String // read, exec, apply, destroy, manage, admin, kubectl, *
-  cloud       String @default("*")
-  environment String @default("*")
-  server      String @default("*")
-
-  @@index([roleId])
-}
-
-model UserRole {
-  id     String @id @default(uuid())
-  userId String
-  user   User   @relation(fields: [userId], references: [id], onDelete: Cascade)
-  roleId String
-  role   Role   @relation(fields: [roleId], references: [id], onDelete: Cascade)
-
-  @@unique([userId, roleId])
-  @@index([userId])
-  @@index([roleId])
-}
-
 model JoinToken {
   id        String    @id @default(uuid())
   token     String    @unique
-  type      String    @default("one-time") // one-time or reusable
+  type      String    @default("one-time")
   label     String?
-  usedBy    String? // server hostname that used it
+  usedBy    String?
   usedAt    DateTime?
   revokedAt DateTime?
   createdAt DateTime  @default(now())
   expiresAt DateTime?
 }
 
-model AuditLog {
-  id           String   @id @default(uuid())
-  userId       String?
-  user         User?    @relation(fields: [userId], references: [id])
-  serverId     String?
-  server       Server?  @relation(fields: [serverId], references: [id])
-  sessionId    String?
-  action       String // exec, kubectl, apply, login, rbac-denied, etc.
-  resourceType String?  // server, cluster, role, app, etc.
-  resourceName String?
-  args         String? // sanitized command args
-  result       String   @default("success") // success, denied, error
-  durationMs   Int?
-  sourceIp     String?
-  timestamp    DateTime @default(now())
-
-  @@index([userId])
-  @@index([serverId])
-  @@index([sessionId])
-  @@index([timestamp])
-  @@index([action])
-}
-
-model PulumiRun {
-  id          String    @id @default(uuid())
-  userId      String
-  stackName   String
-  action      String // up, preview, destroy
-  status      String    @default("pending") // pending, running, succeeded, failed
-  output      String?
-  startedAt   DateTime  @default(now())
-  completedAt DateTime?
-
-  @@index([userId])
-  @@index([stackName])
-}
-
 model Bastion {
   id            String    @id @default(uuid())
   hostname      String    @unique
   network       String
   serverIp      String
-  status        String    @default("offline") // online, offline
+  status        String    @default("offline")
   lastHeartbeat DateTime?
   createdAt     DateTime  @default(now())
   updatedAt     DateTime  @updatedAt
@@ -149,7 +284,7 @@ model Cluster {
   name          String   @unique
   cloud         String   @default("baremetal")
   environment   String   @default("default")
-  kubeconfigEnc String? // encrypted kubeconfig
+  kubeconfigEnc String?
   labels        Json     @default("{}")
   createdAt     DateTime @default(now())
   updatedAt     DateTime @updatedAt

bastion/src/labd/src/middleware/bearer-auth.ts

@@ -0,0 +1,65 @@
+// Bearer token auth middleware for Fastify.
+// Validates Authorization header, resolves user identity, attaches to request.
+
+import type { FastifyRequest, FastifyReply } from "fastify";
+import type { AuthService } from "../services/auth.js";
+
+declare module "fastify" {
+  interface FastifyRequest {
+    userId?: string;
+    userEmail?: string;
+    userRole?: string;
+  }
+}
+
+// Paths that don't require authentication
+const PUBLIC_PATHS = new Set([
+  "/health",
+  "/api/auth/login",
+  "/ws/bastion",
+  "/ws/agent",
+  "/api/auth/enroll",
+]);
+
+export function createBearerAuthMiddleware(authService: AuthService) {
+  return async function bearerAuth(
+    request: FastifyRequest,
+    reply: FastifyReply,
+  ): Promise<void> {
+    // Skip auth for public paths
+    if (PUBLIC_PATHS.has(request.url.split("?")[0] ?? "")) {
+      return;
+    }
+
+    // Skip auth for WebSocket upgrade requests (handled by their own auth)
+    if (request.headers.upgrade === "websocket") {
+      return;
+    }
+
+    const authHeader = request.headers.authorization;
+    if (!authHeader) {
+      void reply.code(401).send({ error: "Authorization header required" });
+      return;
+    }
+
+    if (!authHeader.startsWith("Bearer ")) {
+      void reply.code(401).send({ error: "Invalid authorization format, expected: Bearer <token>" });
+      return;
+    }
+
+    const token = authHeader.slice(7);
+    if (token.length === 0) {
+      void reply.code(401).send({ error: "Empty bearer token" });
+      return;
+    }
+
+    try {
+      const identity = await authService.validateToken(token);
+      request.userId = identity.userId;
+      request.userEmail = identity.email;
+      request.userRole = identity.role;
+    } catch {
+      void reply.code(401).send({ error: "Invalid or expired token. Run: labctl login" });
+    }
+  };
+}

bastion/src/labd/src/routes/bastions.ts

@@ -84,7 +84,6 @@ export function registerBastionRoutes(app: FastifyInstance, db: DbClient): void
   app.get("/api/machines", async () => {
     const live = bastionRegistry.getAggregatedState();
 
-    // Merge DB records for machines not currently in any bastion's live state
     try {
       const dbServers = (await db.server.findMany({})) as Array<{
         mac: string | null; hostname: string; role: string; ip: string | null;
@@ -93,9 +92,49 @@ export function registerBastionRoutes(app: FastifyInstance, db: DbClient): void
       for (const s of dbServers) {
         if (!s.mac) continue;
         const mac = s.mac.toLowerCase();
-        // Only add from DB if not already in live state
+
+        // DB knows this machine has been installed at some point if it has a real
+        // hostname+role (not just product-name-as-hostname and role="unknown").
+        // Status alone is unreliable: a rediscovery can re-set it without erasing the
+        // install identity. If the bastion restarted and lost its installed map, the
+        // machine will only show up in live.discovered — promote it here so the CLI
+        // still sees hostname/role/IP.
+        const dbKnowsInstalled =
+          s.role !== "unknown" && s.role !== "" &&
+          s.hostname !== "" && s.hostname !== s.mac;
+
+        if (dbKnowsInstalled && !(mac in live.installed) && !(mac in live.install_queue)) {
+          const hw = live.discovered[mac];
+          live.installed[mac] = {
+            hostname: s.hostname,
+            role: s.role,
+            ip: s.ip ?? "",
+            installed_at: "",
+            bastionId: hw?.bastionId ?? "db",
+            ...(hw ? {
+              product: hw.product,
+              manufacturer: hw.manufacturer,
+              cpu_model: hw.cpu_model,
+              cpu_cores: hw.cpu_cores,
+              memory_gb: hw.memory_gb,
+              arch: hw.arch,
+            } : {}),
+          };
+          delete live.discovered[mac];
+          continue;
+        }
+
+        // Unknown-to-live MAC: fall back to whatever the DB says.
         if (!(mac in live.discovered) && !(mac in live.install_queue) && !(mac in live.installed)) {
-          if (s.status === "discovered") {
+          if (s.status === "online" || s.status === "offline") {
+            live.installed[mac] = {
+              hostname: s.hostname,
+              role: s.role,
+              ip: s.ip ?? "",
+              installed_at: "",
+              bastionId: "db",
+            };
+          } else {
             live.discovered[mac] = {
               mac,
               product: String(s.labels?.product ?? "unknown"),
@@ -112,14 +151,6 @@ export function registerBastionRoutes(app: FastifyInstance, db: DbClient): void
               last_seen: "",
               bastionId: "db",
             };
-          } else if (s.status === "online" || s.status === "offline") {
-            live.installed[mac] = {
-              hostname: s.hostname,
-              role: s.role,
-              ip: s.ip ?? "",
-              installed_at: "",
-              bastionId: "db",
-            };
           }
         }
       }
@@ -268,6 +299,7 @@ export function registerBastionRoutes(app: FastifyInstance, db: DbClient): void
       memory_gb?: number; arch?: string;
       disks?: Array<{ name: string; size_gb: number; model: string }>;
       nics?: Array<{ name: string; mac: string; state: string }>;
+      root_device?: string; root_args?: string;
     };
   }>("/api/machines/discover", async (request, reply) => {
     const data = request.body ?? {};

bastion/src/labd/src/routes/environments.ts

@@ -0,0 +1,191 @@
+// Environment and Account management routes.
+// GET/POST /api/environments       — list/create environments
+// GET/POST /api/accounts           — list/create accounts
+// POST     /api/accounts/bind      — bind account to environment
+// GET      /api/bindings           — list bindings
+
+import type { FastifyInstance } from "fastify";
+import type { PrismaClient, Prisma } from "@prisma/client";
+import type { RbacService } from "../services/rbac.js";
+import type { AuditService } from "../services/audit.js";
+
+export function registerEnvironmentRoutes(
+  app: FastifyInstance,
+  db: PrismaClient,
+  rbacService: RbacService,
+  auditService: AuditService,
+): void {
+  // List environments
+  app.get("/api/environments", async (_request, reply) => {
+    const envs = await db.environment.findMany({ orderBy: { name: "asc" } });
+    return reply.send(envs);
+  });
+
+  // Create environment
+  app.post<{
+    Body: { name?: string };
+  }>("/api/environments", async (request, reply) => {
+    const { name } = request.body ?? {};
+    if (!name) {
+      return reply.code(400).send({ error: "name is required" });
+    }
+
+    const rbac = await rbacService.check({
+      userId: request.userId!,
+      userEmail: request.userEmail!,
+      userRole: request.userRole!,
+      action: "admin",
+      resource: "environments",
+    });
+    if (!rbac.allowed) {
+      return reply.code(403).send({ error: rbac.reason });
+    }
+
+    try {
+      const env = await db.environment.create({ data: { name } });
+      auditService.emit({
+        eventKind: "resource_created",
+        source: "labd",
+        verified: true,
+        userId: request.userId ?? null,
+        resourceKind: "environment",
+        resourceName: name,
+        result: "success",
+      });
+      return reply.code(201).send(env);
+    } catch (err) {
+      if (err instanceof Error && err.message.includes("Unique constraint")) {
+        return reply.code(409).send({ error: `Environment '${name}' already exists` });
+      }
+      throw err;
+    }
+  });
+
+  // List accounts
+  app.get("/api/accounts", async (_request, reply) => {
+    const accounts = await db.account.findMany({
+      orderBy: { name: "asc" },
+      select: { id: true, name: true, driver: true, config: true, createdAt: true, updatedAt: true },
+    });
+    return reply.send(accounts);
+  });
+
+  // Create account
+  app.post<{
+    Body: { name?: string; driver?: string; config?: Record<string, unknown> };
+  }>("/api/accounts", async (request, reply) => {
+    const { name, driver, config } = request.body ?? {};
+    if (!name || !driver) {
+      return reply.code(400).send({ error: "name and driver are required" });
+    }
+
+    const rbac = await rbacService.check({
+      userId: request.userId!,
+      userEmail: request.userEmail!,
+      userRole: request.userRole!,
+      action: "admin",
+      resource: "accounts",
+    });
+    if (!rbac.allowed) {
+      return reply.code(403).send({ error: rbac.reason });
+    }
+
+    try {
+      const account = await db.account.create({
+        data: { name, driver, config: (config ?? {}) as Prisma.InputJsonValue },
+      });
+      auditService.emit({
+        eventKind: "resource_created",
+        source: "labd",
+        verified: true,
+        userId: request.userId ?? null,
+        resourceKind: "account",
+        resourceName: name,
+        result: "success",
+        details: { driver },
+      });
+      return reply.code(201).send(account);
+    } catch (err) {
+      if (err instanceof Error && err.message.includes("Unique constraint")) {
+        return reply.code(409).send({ error: `Account '${name}' already exists` });
+      }
+      throw err;
+    }
+  });
+
+  // Bind account to environment
+  app.post<{
+    Body: { environmentId?: string; accountId?: string };
+  }>("/api/accounts/bind", async (request, reply) => {
+    const { environmentId, accountId } = request.body ?? {};
+    if (!environmentId || !accountId) {
+      return reply.code(400).send({ error: "environmentId and accountId are required" });
+    }
+
+    const rbac = await rbacService.check({
+      userId: request.userId!,
+      userEmail: request.userEmail!,
+      userRole: request.userRole!,
+      action: "admin",
+      resource: "accounts",
+    });
+    if (!rbac.allowed) {
+      return reply.code(403).send({ error: rbac.reason });
+    }
+
+    try {
+      const binding = await db.binding.create({
+        data: { environmentId, accountId },
+      });
+      return reply.code(201).send(binding);
+    } catch (err) {
+      if (err instanceof Error && err.message.includes("Unique constraint")) {
+        return reply.code(409).send({ error: "This account is already bound to this environment" });
+      }
+      throw err;
+    }
+  });
+
+  // List bindings
+  app.get("/api/bindings", async (_request, reply) => {
+    const bindings = await db.binding.findMany({
+      include: { environment: true, account: true },
+    });
+    return reply.send(bindings);
+  });
+
+  // Audit event query
+  app.get<{
+    Querystring: {
+      last?: string;
+      kind?: string;
+      env?: string;
+      correlation?: string;
+      limit?: string;
+    };
+  }>("/api/events", async (request, reply) => {
+    const { last, kind, env, correlation, limit } = request.query as { last?: string; kind?: string; env?: string; correlation?: string; limit?: string };
+
+    const where: Record<string, unknown> = {};
+
+    if (last) {
+      const match = last.match(/^(\d+)(h|d|m)$/);
+      if (match) {
+        const [, num, unit] = match;
+        const ms = { h: 3_600_000, d: 86_400_000, m: 60_000 }[unit!]!;
+        where.timestamp = { gte: new Date(Date.now() - parseInt(num!) * ms) };
+      }
+    }
+    if (kind) where.eventKind = kind;
+    if (env) where.environmentName = env;
+    if (correlation) where.correlationId = correlation;
+
+    const events = await db.auditEvent.findMany({
+      where,
+      orderBy: { timestamp: "desc" },
+      take: Math.min(parseInt(limit ?? "100"), 500),
+    });
+
+    return reply.send(events);
+  });
+}

bastion/src/labd/src/routes/resources.ts

@@ -0,0 +1,196 @@
+// Resource CRUD routes with RBAC enforcement.
+// GET    /api/resources          — list (filtered by RBAC scope)
+// GET    /api/resources/:id      — get
+// POST   /api/resources          — create
+// PUT    /api/resources/:id      — update
+// DELETE /api/resources/:id      — delete (marks as deleting)
+
+import type { FastifyInstance } from "fastify";
+import type { ResourceStore, CreateResourceInput } from "../services/resource-store.js";
+import type { RbacService } from "../services/rbac.js";
+import type { AuditService } from "../services/audit.js";
+import { resolveResourceKind } from "@lab/core";
+
+export function registerResourceRoutes(
+  app: FastifyInstance,
+  resourceStore: ResourceStore,
+  rbacService: RbacService,
+  auditService: AuditService,
+): void {
+  // List resources (filtered by kind, environment, status)
+  app.get<{
+    Querystring: { kind?: string; environment?: string; status?: string };
+  }>("/api/resources", async (request, reply) => {
+    const rbac = await rbacService.check({
+      userId: request.userId!,
+      userEmail: request.userEmail!,
+      userRole: request.userRole!,
+      action: "view",
+      resource: request.query.kind ? resolveResourceKind(request.query.kind) : undefined,
+    });
+
+    if (!rbac.allowed) {
+      return reply.code(403).send({ error: rbac.reason });
+    }
+
+    const resources = await resourceStore.list({
+      kind: request.query.kind ? resolveResourceKind(request.query.kind) : undefined,
+      environmentId: request.query.environment,
+      status: request.query.status,
+    });
+
+    return reply.send(resources);
+  });
+
+  // Get single resource
+  app.get<{
+    Params: { id: string };
+  }>("/api/resources/:id", async (request, reply) => {
+    const resource = await resourceStore.get(request.params.id);
+    if (!resource) {
+      return reply.code(404).send({ error: "Resource not found" });
+    }
+
+    const rbac = await rbacService.check({
+      userId: request.userId!,
+      userEmail: request.userEmail!,
+      userRole: request.userRole!,
+      action: "view",
+      resource: resource.kind,
+      name: resource.name,
+    });
+
+    if (!rbac.allowed) {
+      return reply.code(403).send({ error: rbac.reason });
+    }
+
+    return reply.send(resource);
+  });
+
+  // Create resource
+  app.post<{
+    Body: CreateResourceInput;
+  }>("/api/resources", async (request, reply) => {
+    const input = request.body;
+    if (!input?.kind || !input?.name || !input?.environmentId || !input?.accountId) {
+      return reply.code(400).send({ error: "kind, name, environmentId, and accountId are required" });
+    }
+
+    const kind = resolveResourceKind(input.kind);
+
+    const rbac = await rbacService.check({
+      userId: request.userId!,
+      userEmail: request.userEmail!,
+      userRole: request.userRole!,
+      action: "create",
+      resource: kind,
+    });
+
+    if (!rbac.allowed) {
+      return reply.code(403).send({ error: rbac.reason });
+    }
+
+    const correlationId = auditService.createCorrelation();
+
+    try {
+      const resource = await resourceStore.create({ ...input, kind });
+
+      auditService.emit({
+        eventKind: "resource_created",
+        source: "labd",
+        verified: true,
+        userId: request.userId ?? null,
+        userName: request.userEmail ?? null,
+        resourceKind: kind,
+        resourceName: input.name,
+        correlationId,
+        result: "success",
+      });
+
+      return reply.code(201).send(resource);
+    } catch (err) {
+      // Prisma unique constraint violation
+      if (err instanceof Error && err.message.includes("Unique constraint")) {
+        return reply.code(409).send({ error: `Resource ${kind}/${input.name} already exists in this environment` });
+      }
+      throw err;
+    }
+  });
+
+  // Update resource
+  app.put<{
+    Params: { id: string };
+    Body: { desiredSpec?: Record<string, unknown>; status?: string };
+  }>("/api/resources/:id", async (request, reply) => {
+    const resource = await resourceStore.get(request.params.id);
+    if (!resource) {
+      return reply.code(404).send({ error: "Resource not found" });
+    }
+
+    const rbac = await rbacService.check({
+      userId: request.userId!,
+      userEmail: request.userEmail!,
+      userRole: request.userRole!,
+      action: "edit",
+      resource: resource.kind,
+      name: resource.name,
+    });
+
+    if (!rbac.allowed) {
+      return reply.code(403).send({ error: rbac.reason });
+    }
+
+    const updated = await resourceStore.update(request.params.id, request.body);
+
+    auditService.emit({
+      eventKind: "resource_updated",
+      source: "labd",
+      verified: true,
+      userId: request.userId ?? null,
+      userName: request.userEmail ?? null,
+      resourceKind: resource.kind,
+      resourceName: resource.name,
+      result: "success",
+    });
+
+    return reply.send(updated);
+  });
+
+  // Delete resource (marks as deleting)
+  app.delete<{
+    Params: { id: string };
+  }>("/api/resources/:id", async (request, reply) => {
+    const resource = await resourceStore.get(request.params.id);
+    if (!resource) {
+      return reply.code(404).send({ error: "Resource not found" });
+    }
+
+    const rbac = await rbacService.check({
+      userId: request.userId!,
+      userEmail: request.userEmail!,
+      userRole: request.userRole!,
+      action: "delete",
+      resource: resource.kind,
+      name: resource.name,
+    });
+
+    if (!rbac.allowed) {
+      return reply.code(403).send({ error: rbac.reason });
+    }
+
+    await resourceStore.delete(request.params.id);
+
+    auditService.emit({
+      eventKind: "resource_deleted",
+      source: "labd",
+      verified: true,
+      userId: request.userId ?? null,
+      userName: request.userEmail ?? null,
+      resourceKind: resource.kind,
+      resourceName: resource.name,
+      result: "success",
+    });
+
+    return reply.send({ status: "deleting", id: request.params.id });
+  });
+}

bastion/src/labd/src/routes/v2-auth.ts

@@ -0,0 +1,81 @@
+// v2 Auth routes: bearer token login/logout.
+// POST /api/auth/login  — email + password → session token
+// POST /api/auth/logout — revoke session
+
+import type { FastifyInstance } from "fastify";
+import type { AuthService } from "../services/auth.js";
+import type { AuditService } from "../services/audit.js";
+import { AuthError } from "../services/auth.js";
+
+export function registerV2AuthRoutes(
+  app: FastifyInstance,
+  authService: AuthService,
+  auditService: AuditService,
+): void {
+  app.post<{
+    Body: { email?: string; password?: string };
+  }>("/api/auth/login", async (request, reply) => {
+    const { email, password } = request.body ?? {};
+
+    if (!email || !password) {
+      return reply.code(400).send({ error: "email and password are required" });
+    }
+
+    try {
+      const result = await authService.login(email, password);
+
+      auditService.emit({
+        eventKind: result.isBootstrap ? "auth_bootstrap" : "auth_login",
+        source: "labd",
+        verified: true,
+        userId: result.userId,
+        userName: email,
+        result: "success",
+        details: { isBootstrap: result.isBootstrap },
+      });
+
+      return reply.send({
+        token: result.token,
+        expiresAt: result.expiresAt.toISOString(),
+        isBootstrap: result.isBootstrap,
+      });
+    } catch (err) {
+      if (err instanceof AuthError) {
+        auditService.emit({
+          eventKind: "auth_login",
+          source: "labd",
+          verified: true,
+          userName: email,
+          result: "failure",
+          error: err.message,
+        });
+        return reply.code(401).send({ error: err.message });
+      }
+      return reply.code(500).send({ error: "Login failed" });
+    }
+  });
+
+  app.post("/api/auth/logout", async (request, reply) => {
+    const token = request.headers.authorization?.slice(7);
+    if (!token) {
+      return reply.code(400).send({ error: "Authorization header required" });
+    }
+
+    try {
+      await authService.logout(token);
+      auditService.emit({
+        eventKind: "auth_logout",
+        source: "labd",
+        verified: true,
+        userId: request.userId ?? null,
+        result: "success",
+      });
+      return reply.send({ status: "logged_out" });
+    } catch (err) {
+      if (err instanceof AuthError) {
+        return reply.code(400).send({ error: err.message });
+      }
+      return reply.code(500).send({ error: "Logout failed" });
+    }
+  });
+}

bastion/src/labd/src/server.ts

@@ -2,6 +2,7 @@
 
 import Fastify from "fastify";
 import websocket from "@fastify/websocket";
+import type { PrismaClient } from "@prisma/client";
 import type { LabdConfig } from "./config.js";
 import { logger } from "./services/logger.js";
 import { registerHealthRoutes } from "./routes/health.js";
@@ -9,8 +10,16 @@ import { registerServerRoutes } from "./routes/servers.js";
 import { registerAuthRoutes } from "./routes/auth.js";
 import { registerAgentRoutes } from "./routes/agents.js";
 import { registerBastionRoutes } from "./routes/bastions.js";
+import { registerV2AuthRoutes } from "./routes/v2-auth.js";
+import { registerEnvironmentRoutes } from "./routes/environments.js";
+import { registerResourceRoutes } from "./routes/resources.js";
 import { setupRateLimiting } from "./middleware/rate-limit.js";
+import { createBearerAuthMiddleware } from "./middleware/bearer-auth.js";
 import { bastionRegistry } from "./services/bastion-registry.js";
+import { AuthService } from "./services/auth.js";
+import { RbacService } from "./services/rbac.js";
+import { ResourceStore } from "./services/resource-store.js";
+import { AuditService } from "./services/audit.js";
 import { isBastionMessage } from "@lab/shared";
 
 export interface DbClient {
@@ -37,6 +46,7 @@ export interface DbClient {
 
 export async function createApp(_config: LabdConfig, db: DbClient): Promise<{
   app: ReturnType<typeof Fastify>;
+  auditService: AuditService;
 }> {
   const app = Fastify({
     logger: false, // We use winston instead
@@ -48,13 +58,39 @@ export async function createApp(_config: LabdConfig, db: DbClient): Promise<{
   // Register WebSocket support
   void app.register(websocket);
 
-  // Register route handlers
+  // v2 services. The structural DbClient is a subset of the real PrismaClient;
+  // at runtime db IS the PrismaClient instance, so the cast is safe. Tests that
+  // exercise v2 routes provide a PrismaClient-shaped mock (see auth-bootstrap,
+  // rbac-deny, audit-correlation tests).
+  const prisma = db as unknown as PrismaClient;
+  const authService = new AuthService(prisma);
+  const rbacService = new RbacService(prisma);
+  const resourceStore = new ResourceStore(prisma);
+  const auditService = new AuditService(prisma);
+  auditService.start();
+
+  // Register v1 (legacy) route handlers
   registerHealthRoutes(app, db);
   registerServerRoutes(app, db);
   registerAuthRoutes(app, db);
   registerAgentRoutes(app);
   registerBastionRoutes(app, db);
 
+  // v2 routes live in a scope with bearer-auth as preHandler. Public paths
+  // (login, /health, websockets) are skipped inside the middleware itself.
+  // v1 routes above are unaffected — they're registered on the root scope.
+  await app.register(async (scope) => {
+    scope.addHook("preHandler", createBearerAuthMiddleware(authService));
+    registerV2AuthRoutes(scope, authService, auditService);
+    registerEnvironmentRoutes(scope, prisma, rbacService, auditService);
+    registerResourceRoutes(scope, resourceStore, rbacService, auditService);
+  });
+
+  // Flush pending audit events on shutdown so we never lose the last batch.
+  app.addHook("onClose", async () => {
+    auditService.stop();
+  });
+
   // WebSocket handler for agent connections
   app.register(async (fastify) => {
     fastify.get("/ws/agent", { websocket: true }, (socket, _request) => {
@@ -192,7 +228,9 @@ export async function createApp(_config: LabdConfig, db: DbClient): Promise<{
                           labels: { cpu: hw.cpu_model, cores: hw.cpu_cores, memory_gb: hw.memory_gb, arch: hw.arch, product: hw.product, manufacturer: hw.manufacturer },
                         },
                         update: {
-                          status: "discovered",
+                          // Leave status alone — a previously "online"/"offline" record
+                          // must not be downgraded to "discovered" just because the bastion
+                          // restarted and re-discovered the MAC via DHCP/PXE.
                           lastHeartbeat: new Date(),
                           labels: { cpu: hw.cpu_model, cores: hw.cpu_cores, memory_gb: hw.memory_gb, arch: hw.arch, product: hw.product, manufacturer: hw.manufacturer },
                         },
@@ -265,5 +303,5 @@ export async function createApp(_config: LabdConfig, db: DbClient): Promise<{
     logger.info(`HTTP: ${request.ip} ${request.method} ${request.url}`);
   });
 
-  return { app };
+  return { app, auditService };
 }

bastion/src/labd/src/services/audit.ts

@@ -0,0 +1,106 @@
+// Audit service: fire-and-forget event collection with batching.
+// Batches 50 events or flushes every 5 seconds, whichever comes first.
+// Failures never block the operation being audited.
+
+import { randomBytes } from "node:crypto";
+import type { PrismaClient, Prisma } from "@prisma/client";
+import { logger } from "./logger.js";
+
+const BATCH_SIZE = 50;
+const FLUSH_INTERVAL_MS = 5_000;
+
+export interface AuditEventInput {
+  eventKind: string;
+  source: string;
+  verified?: boolean;
+  userId?: string | null;
+  userName?: string | null;
+  sessionId?: string | null;
+  environmentName?: string | null;
+  accountName?: string | null;
+  resourceKind?: string | null;
+  resourceName?: string | null;
+  correlationId?: string | null;
+  parentEventId?: string | null;
+  details?: Record<string, unknown>;
+  result: string;
+  error?: string | null;
+  durationMs?: number | null;
+}
+
+export class AuditService {
+  private batch: AuditEventInput[] = [];
+  private timer: ReturnType<typeof setInterval> | null = null;
+
+  constructor(private readonly db: PrismaClient) {}
+
+  start(): void {
+    this.timer = setInterval(() => {
+      void this.flush();
+    }, FLUSH_INTERVAL_MS);
+  }
+
+  stop(): void {
+    if (this.timer) {
+      clearInterval(this.timer);
+      this.timer = null;
+    }
+    void this.flush();
+  }
+
+  emit(event: AuditEventInput): void {
+    // Generate correlation ID if not provided
+    if (!event.correlationId) {
+      event.correlationId = `corr_${randomBytes(8).toString("hex")}`;
+    }
+
+    this.batch.push(event);
+
+    if (this.batch.length >= BATCH_SIZE) {
+      void this.flush();
+    }
+  }
+
+  /** Create a correlation context for a chain of related events. */
+  createCorrelation(): string {
+    return `corr_${randomBytes(8).toString("hex")}`;
+  }
+
+  /** Flush all pending events synchronously. Tests await this; production
+   * relies on the interval timer or stop() during shutdown. */
+  async flushPending(): Promise<void> {
+    await this.flush();
+  }
+
+  private async flush(): Promise<void> {
+    if (this.batch.length === 0) return;
+
+    const events = this.batch.splice(0);
+    try {
+      await this.db.auditEvent.createMany({
+        data: events.map((e) => ({
+          eventKind: e.eventKind,
+          source: e.source,
+          verified: e.verified ?? false,
+          userId: e.userId ?? null,
+          userName: e.userName ?? null,
+          sessionId: e.sessionId ?? null,
+          environmentName: e.environmentName ?? null,
+          accountName: e.accountName ?? null,
+          resourceKind: e.resourceKind ?? null,
+          resourceName: e.resourceName ?? null,
+          correlationId: e.correlationId ?? `corr_${randomBytes(8).toString("hex")}`,
+          parentEventId: e.parentEventId ?? null,
+          details: (e.details ?? {}) as Prisma.InputJsonValue,
+          result: e.result,
+          error: e.error ?? null,
+          durationMs: e.durationMs ?? null,
+        })),
+      });
+      logger.info(`AUDIT: flushed ${events.length} events`);
+    } catch (err) {
+      // Fire-and-forget: audit failures never block operations
+      logger.warn(`AUDIT: failed to flush ${events.length} events: ${err instanceof Error ? err.message : String(err)}`);
+    }
+  }
+}

bastion/src/labd/src/services/auth.ts

@@ -0,0 +1,119 @@
+// Auth service: bearer token authentication with bootstrap flow.
+// First login creates the admin user. Subsequent logins return session tokens.
+
+import { randomBytes } from "node:crypto";
+import bcrypt from "bcryptjs";
+import type { PrismaClient } from "@prisma/client";
+import { logger } from "./logger.js";
+
+const SESSION_EXPIRY_DAYS = 30;
+const BCRYPT_ROUNDS = 12;
+
+export interface LoginResult {
+  token: string;
+  expiresAt: Date;
+  userId: string;
+  isBootstrap: boolean;
+}
+
+export class AuthService {
+  constructor(private readonly db: PrismaClient) {}
+
+  async login(email: string, password: string): Promise<LoginResult> {
+    const userCount = await this.db.user.count();
+
+    // Bootstrap: first login creates admin user
+    if (userCount === 0) {
+      return this.bootstrap(email, password);
+    }
+
+    const user = await this.db.user.findUnique({ where: { email } });
+    if (!user) {
+      // Same error for unknown user and wrong password (no enumeration)
+      throw new AuthError("Invalid email or password");
+    }
+
+    const valid = await bcrypt.compare(password, user.password);
+    if (!valid) {
+      throw new AuthError("Invalid email or password");
+    }
+
+    const session = await this.createSession(user.id);
+    logger.info(`AUTH LOGIN: ${email} (${user.id.slice(0, 8)}...)`);
+
+    return {
+      token: session.token,
+      expiresAt: session.expiresAt,
+      userId: user.id,
+      isBootstrap: false,
+    };
+  }
+
+  async logout(token: string): Promise<void> {
+    const session = await this.db.session.findUnique({ where: { token } });
+    if (!session) {
+      throw new AuthError("Invalid session");
+    }
+    await this.db.session.delete({ where: { id: session.id } });
+    logger.info(`AUTH LOGOUT: session ${session.id.slice(0, 8)}...`);
+  }
+
+  async validateToken(token: string): Promise<{ userId: string; email: string; role: string }> {
+    const session = await this.db.session.findUnique({
+      where: { token },
+      include: { user: true },
+    });
+
+    if (!session) {
+      throw new AuthError("Invalid token");
+    }
+    if (session.expiresAt < new Date()) {
+      await this.db.session.delete({ where: { id: session.id } });
+      throw new AuthError("Token expired");
+    }
+
+    return {
+      userId: session.user.id,
+      email: session.user.email,
+      role: session.user.role,
+    };
+  }
+
+  private async bootstrap(email: string, password: string): Promise<LoginResult> {
+    const hashed = await bcrypt.hash(password, BCRYPT_ROUNDS);
+    const user = await this.db.user.create({
+      data: {
+        email,
+        password: hashed,
+        role: "ADMIN",
+        name: email.split("@")[0] ?? null,
+      },
+    });
+
+    const session = await this.createSession(user.id);
+    logger.info(`AUTH BOOTSTRAP: created admin user ${email} (${user.id.slice(0, 8)}...)`);
+
+    return {
+      token: session.token,
+      expiresAt: session.expiresAt,
+      userId: user.id,
+      isBootstrap: true,
+    };
+  }
+
+  private async createSession(userId: string) {
+    const token = randomBytes(32).toString("hex");
+    const expiresAt = new Date(Date.now() + SESSION_EXPIRY_DAYS * 24 * 60 * 60 * 1000);
+
+    return this.db.session.create({
+      data: { userId, token, expiresAt },
+    });
+  }
+}
+
+export class AuthError extends Error {
+  constructor(message: string) {
+    super(message);
+    this.name = "AuthError";
+  }
+}

bastion/src/labd/src/services/rbac.ts

@@ -0,0 +1,123 @@
+// RBAC service: environment-scoped permission checks.
+// Uses named RbacDefinition records with JSON subjects and roleBindings.
+//
+// Resolution flow:
+//   1. Find all RbacDefinitions where subjects match the current user/groups
+//   2. Collect all roleBindings from matching definitions
+//   3. Check if any binding grants the requested action on the requested resource
+
+import type { PrismaClient } from "@prisma/client";
+import { logger } from "./logger.js";
+
+export interface RbacCheck {
+  userId: string;
+  userEmail: string;
+  userRole: string;
+  action: string;       // "view" | "edit" | "create" | "delete" | "run" | "admin"
+  resource?: string | undefined;    // "servers" | "databases" | "clusters" | "*"
+  name?: string | undefined;        // specific resource name
+  environment?: string | undefined; // specific environment name
+}
+
+export interface RbacResult {
+  allowed: boolean;
+  reason: string;
+  matchedDefinition?: string;
+}
+
+interface StoredSubject {
+  kind: string;
+  name: string;
+}
+
+interface StoredBinding {
+  role: string;
+  resource?: string;
+  name?: string;
+  environment?: string;
+  action?: string;
+}
+
+export class RbacService {
+  constructor(private readonly db: PrismaClient) {}
+
+  async check(req: RbacCheck): Promise<RbacResult> {
+    // Admin users bypass RBAC
+    if (req.userRole === "ADMIN") {
+      return { allowed: true, reason: "admin role" };
+    }
+
+    // Collect user's group memberships
+    const memberships = await this.db.groupMember.findMany({
+      where: { userId: req.userId },
+      include: { group: true },
+    });
+    const groupNames = memberships.map((m) => m.group.name);
+
+    // Find all RBAC definitions
+    const definitions = await this.db.rbacDefinition.findMany();
+
+    for (const def of definitions) {
+      const subjects = def.subjects as unknown as StoredSubject[];
+      const bindings = def.roleBindings as unknown as StoredBinding[];
+
+      // Check if this definition's subjects match the user
+      const subjectMatch = subjects.some((s) => {
+        if (s.kind === "User" && s.name === req.userEmail) return true;
+        if (s.kind === "Group" && groupNames.includes(s.name)) return true;
+        return false;
+      });
+
+      if (!subjectMatch) continue;
+
+      // Check if any binding grants the requested permission
+      for (const binding of bindings) {
+        if (this.bindingMatches(binding, req)) {
+          logger.info(`RBAC ALLOW: ${req.userEmail} ${req.action} ${req.resource ?? "*"}${req.name ? `/${req.name}` : ""} via ${def.name}`);
+          return {
+            allowed: true,
+            reason: `granted by ${def.name}`,
+            matchedDefinition: def.name,
+          };
+        }
+      }
+    }
+
+    logger.info(`RBAC DENY: ${req.userEmail} ${req.action} ${req.resource ?? "*"}${req.name ? `/${req.name}` : ""}`);
+    return {
+      allowed: false,
+      reason: `no matching role binding for ${req.action} on ${req.resource ?? "*"}`,
+    };
+  }
+
+  private bindingMatches(binding: StoredBinding, req: RbacCheck): boolean {
+    // Check role grants the action
+    if (!this.roleGrantsAction(binding.role, req.action)) return false;
+
+    // Check resource scope
+    if (binding.resource && binding.resource !== "*" && binding.resource !== req.resource) return false;
+
+    // Check name scope
+    if (binding.name && binding.name !== req.name) return false;
+
+    // Check environment scope
+    if (binding.environment && binding.environment !== req.environment) return false;
+
+    // Check operation scope (for "run" role with specific actions)
+    if (binding.action && binding.action !== "*" && binding.action !== req.action) return false;
+
+    return true;
+  }
+
+  private roleGrantsAction(role: string, action: string): boolean {
+    const grants: Record<string, string[]> = {
+      admin: ["view", "edit", "create", "delete", "run", "admin"],
+      edit: ["view", "edit", "create", "delete"],
+      create: ["create"],
+      delete: ["delete"],
+      view: ["view"],
+      run: ["run"],
+    };
+    return grants[role]?.includes(action) ?? false;
+  }
+}

bastion/src/labd/src/services/resource-store.ts

@@ -0,0 +1,108 @@
+// Resource store: CRUD for generic resources with origin/managedBy tracking.
+// All mutations go through this service so RBAC and audit are applied consistently.
+
+import type { PrismaClient, Resource as PrismaResource, Prisma } from "@prisma/client";
+import { logger } from "./logger.js";
+
+export interface CreateResourceInput {
+  kind: string;
+  name: string;
+  environmentId: string;
+  accountId: string;
+  origin?: string;
+  managedBy?: string;
+  sourceRef?: string;
+  desiredSpec: Record<string, unknown>;
+}
+
+export interface UpdateResourceInput {
+  desiredSpec?: Record<string, unknown>;
+  status?: string;
+  statusMessage?: string;
+  actualSpec?: Record<string, unknown>;
+  platformRef?: string;
+}
+
+export interface ListResourcesFilter {
+  kind?: string | undefined;
+  environmentId?: string | undefined;
+  accountId?: string | undefined;
+  status?: string | undefined;
+}
+
+export class ResourceStore {
+  constructor(private readonly db: PrismaClient) {}
+
+  async create(input: CreateResourceInput): Promise<PrismaResource> {
+    const resource = await this.db.resource.create({
+      data: {
+        kind: input.kind,
+        name: input.name,
+        environmentId: input.environmentId,
+        accountId: input.accountId,
+        origin: input.origin ?? "cli",
+        managedBy: input.managedBy ?? "manual",
+        sourceRef: input.sourceRef ?? null,
+        desiredSpec: input.desiredSpec as Prisma.InputJsonValue,
+        status: "pending",
+      },
+    });
+
+    logger.info(`RESOURCE CREATED: ${input.kind}/${input.name} in env ${input.environmentId.slice(0, 8)}...`);
+    return resource;
+  }
+
+  async get(id: string): Promise<PrismaResource | null> {
+    return this.db.resource.findUnique({ where: { id } });
+  }
+
+  async getByKindNameEnv(kind: string, name: string, environmentId: string): Promise<PrismaResource | null> {
+    return this.db.resource.findUnique({
+      where: { kind_name_environmentId: { kind, name, environmentId } },
+    });
+  }
+
+  async list(filter: ListResourcesFilter = {}): Promise<PrismaResource[]> {
+    return this.db.resource.findMany({
+      where: {
+        ...(filter.kind ? { kind: filter.kind } : {}),
+        ...(filter.environmentId ? { environmentId: filter.environmentId } : {}),
+        ...(filter.accountId ? { accountId: filter.accountId } : {}),
+        ...(filter.status ? { status: filter.status } : {}),
+      },
+      orderBy: { createdAt: "desc" },
+    });
+  }
+
+  async update(id: string, input: UpdateResourceInput): Promise<PrismaResource> {
+    const data: Prisma.ResourceUpdateInput = {};
+    if (input.desiredSpec !== undefined) data.desiredSpec = input.desiredSpec as Prisma.InputJsonValue;
+    if (input.status !== undefined) data.status = input.status;
+    if (input.statusMessage !== undefined) data.statusMessage = input.statusMessage;
+    if (input.actualSpec !== undefined) data.actualSpec = input.actualSpec as Prisma.InputJsonValue;
+    if (input.platformRef !== undefined) data.platformRef = input.platformRef;
+    if (input.status === "ready") data.lastReconciled = new Date();
+
+    const resource = await this.db.resource.update({ where: { id }, data });
+
+    logger.info(`RESOURCE UPDATED: ${resource.kind}/${resource.name} -> ${input.status ?? "spec change"}`);
+    return resource;
+  }
+
+  async delete(id: string): Promise<void> {
+    const resource = await this.db.resource.findUnique({ where: { id } });
+    if (!resource) return;
+
+    // Mark as deleting first (driver handles actual deletion)
+    await this.db.resource.update({
+      where: { id },
+      data: { status: "deleting" },
+    });
+
+    logger.info(`RESOURCE DELETING: ${resource.kind}/${resource.name}`);
+  }
+
+  async hardDelete(id: string): Promise<void> {
+    await this.db.resource.delete({ where: { id } });
+  }
+}

bastion/src/labd/tests/bastions-machines.test.ts

@@ -0,0 +1,144 @@
+import { describe, it, expect, vi, beforeEach } from "vitest";
+import Fastify from "fastify";
+import { registerBastionRoutes } from "../src/routes/bastions.js";
+import { bastionRegistry } from "../src/services/bastion-registry.js";
+import type { DbClient } from "../src/server.js";
+import type { BastionState } from "@lab/shared";
+
+function createMockDb(servers: unknown[] = []): DbClient {
+  return {
+    $queryRaw: vi.fn().mockResolvedValue([{ "?column?": 1 }]),
+    server: {
+      findMany: vi.fn().mockResolvedValue(servers),
+      findUnique: vi.fn().mockResolvedValue(null),
+      upsert: vi.fn().mockResolvedValue({}),
+    },
+    joinToken: {
+      findUnique: vi.fn().mockResolvedValue(null),
+      findMany: vi.fn().mockResolvedValue([]),
+      create: vi.fn().mockResolvedValue({ id: "t" }),
+      update: vi.fn().mockResolvedValue({}),
+    },
+    bastion: {
+      upsert: vi.fn().mockResolvedValue({}),
+      findMany: vi.fn().mockResolvedValue([]),
+      findUnique: vi.fn().mockResolvedValue(null),
+      update: vi.fn().mockResolvedValue({}),
+    },
+  };
+}
+
+function registerFakeBastion(bastionId: string, state: BastionState): void {
+  bastionRegistry.register({
+    bastionId,
+    hostname: "fake",
+    network: "192.168.8.0/24",
+    serverIp: "192.168.8.11",
+    // socket is referenced only on commands, not during aggregation
+    socket: { on: () => undefined, off: () => undefined, send: () => undefined, close: () => undefined } as never,
+    connectedAt: new Date(),
+    lastHeartbeat: new Date(),
+    state,
+  });
+}
+
+describe("GET /api/machines aggregation", () => {
+  beforeEach(() => {
+    for (const b of bastionRegistry.getAll()) bastionRegistry.unregister(b.bastionId);
+  });
+
+  it("promotes a live-discovered MAC to installed when the DB has a real hostname+role for it", async () => {
+    // Simulates the worker0-k8s0 bug: bastion restarted, lost its installed map,
+    // rediscovered the machine via DHCP/PXE. DB still has hostname=worker0-k8s0,
+    // role=infra, ip=192.168.8.23. Without the fix, the CLI sees a "discovered"
+    // row with no hostname/role/IP. With the fix, the row is promoted to
+    // "installed" with full identity preserved.
+    const mac = "78:55:36:08:28:fb";
+    registerFakeBastion("b1", {
+      discovered: {
+        [mac]: {
+          mac, product: "SER", board: "SER", serial: "x", manufacturer: "AZW",
+          cpu_model: "AMD Ryzen 7 255", cpu_cores: 16, memory_gb: 58, arch: "x86_64",
+          disks: [], nics: [], first_seen: "", last_seen: "",
+        },
+      },
+      install_queue: {},
+      installed: {},
+      debug: {},
+    });
+
+    const app = Fastify({ logger: false });
+    const db = createMockDb([
+      { mac, hostname: "worker0-k8s0", role: "infra", ip: "192.168.8.23", status: "discovered", labels: {} },
+    ]);
+    registerBastionRoutes(app, db);
+
+    const res = await app.inject({ method: "GET", url: "/api/machines" });
+    expect(res.statusCode).toBe(200);
+    const body = JSON.parse(res.body);
+
+    expect(body.discovered[mac]).toBeUndefined();
+    expect(body.installed[mac]).toMatchObject({
+      hostname: "worker0-k8s0",
+      role: "infra",
+      ip: "192.168.8.23",
+      cpu_model: "AMD Ryzen 7 255",
+      cpu_cores: 16,
+      memory_gb: 58,
+    });
+
+    await app.close();
+  });
+
+  it("leaves a fresh-discovery MAC in discovered when DB only has a discovery-shaped record", async () => {
+    const mac = "aa:bb:cc:dd:ee:ff";
+    registerFakeBastion("b1", {
+      discovered: {
+        [mac]: {
+          mac, product: "SER", board: "SER", serial: "x", manufacturer: "AZW",
+          cpu_model: "AMD Ryzen 7", cpu_cores: 8, memory_gb: 32, arch: "x86_64",
+          disks: [], nics: [], first_seen: "", last_seen: "",
+        },
+      },
+      install_queue: {},
+      installed: {},
+      debug: {},
+    });
+
+    const app = Fastify({ logger: false });
+    // Matches what labd writes on first discovery: hostname=product, role="unknown"
+    const db = createMockDb([
+      { mac, hostname: "SER", role: "unknown", ip: null, status: "discovered", labels: {} },
+    ]);
+    registerBastionRoutes(app, db);
+
+    const res = await app.inject({ method: "GET", url: "/api/machines" });
+    const body = JSON.parse(res.body);
+
+    expect(body.discovered[mac]).toBeDefined();
+    expect(body.installed[mac]).toBeUndefined();
+
+    await app.close();
+  });
+
+  it("falls back to DB for MACs not in any live bucket", async () => {
+    const mac = "11:22:33:44:55:66";
+    // No bastions connected
+    const app = Fastify({ logger: false });
+    const db = createMockDb([
+      { mac, hostname: "worker1-k8s0", role: "infra", ip: "192.168.8.13", status: "online", labels: {} },
+    ]);
+    registerBastionRoutes(app, db);
+
+    const res = await app.inject({ method: "GET", url: "/api/machines" });
+    const body = JSON.parse(res.body);
+
+    expect(body.installed[mac]).toMatchObject({
+      hostname: "worker1-k8s0",
+      role: "infra",
+      ip: "192.168.8.13",
+    });
+
+    await app.close();
+  });
+});

bastion/src/labd/tests/v2-smoke.test.ts

@@ -0,0 +1,425 @@
+// End-to-end smoke tests for the v2.0 Phase 1 surface (auth bootstrap, RBAC,
+// audit correlation). These exercise the wiring in createApp(): the bearer
+// auth middleware, the v2 routes scope, and the AuditService lifecycle.
+//
+// We don't spin up CockroachDB. Instead we provide a PrismaClient-shaped
+// in-memory mock that matches the surface the v2 services actually touch.
+// Tests follow the project convention of using mock DBs + Fastify.inject().
+
+import { describe, it, expect, beforeEach, afterEach, vi } from "vitest";
+import bcrypt from "bcryptjs";
+import { createApp } from "../src/server.js";
+import type { DbClient } from "../src/server.js";
+import type { AuditService } from "../src/services/audit.js";
+
+const TEST_CONFIG = { port: 0, host: "127.0.0.1", databaseUrl: "", caDir: "/tmp", logLevel: "silent" };
+
+interface UserRow { id: string; email: string; password: string; role: string; name: string | null; }
+interface SessionRow { id: string; userId: string; token: string; expiresAt: Date; user?: UserRow; }
+interface RbacDefRow { id: string; name: string; subjects: unknown; roleBindings: unknown; }
+interface AuditEventRow {
+  id: string;
+  eventKind: string;
+  source: string;
+  verified: boolean;
+  userId: string | null;
+  userName: string | null;
+  environmentName: string | null;
+  resourceKind: string | null;
+  correlationId: string | null;
+  parentEventId: string | null;
+  details: unknown;
+  result: string;
+  error: string | null;
+  durationMs: number | null;
+  timestamp: Date;
+}
+
+interface Stores {
+  users: Map<string, UserRow>;
+  sessions: Map<string, SessionRow>;
+  groupMembers: Array<{ userId: string; group: { name: string } }>;
+  rbacDefs: RbacDefRow[];
+  auditEvents: AuditEventRow[];
+  resources: Array<Record<string, unknown>>;
+}
+
+function makeStores(): Stores {
+  return {
+    users: new Map(),
+    sessions: new Map(),
+    groupMembers: [],
+    rbacDefs: [],
+    auditEvents: [],
+    resources: [],
+  };
+}
+
+function makeMockDb(s: Stores): DbClient {
+  let idCounter = 0;
+  const newId = (prefix: string): string => `${prefix}-${++idCounter}`;
+
+  return {
+    $queryRaw: vi.fn(async () => [{ "?column?": 1 }]),
+    server: { findMany: vi.fn(async () => []), findUnique: vi.fn(), upsert: vi.fn() },
+    joinToken: { findUnique: vi.fn(), findMany: vi.fn(), create: vi.fn(), update: vi.fn() },
+    bastion: { upsert: vi.fn(), findMany: vi.fn(), findUnique: vi.fn(), update: vi.fn() },
+
+    user: {
+      count: vi.fn(async () => s.users.size),
+      findUnique: vi.fn(async (args: { where: { email?: string; id?: string } }) => {
+        if (args.where.email) {
+          for (const u of s.users.values()) if (u.email === args.where.email) return u;
+        }
+        if (args.where.id) return s.users.get(args.where.id) ?? null;
+        return null;
+      }),
+      create: vi.fn(async (args: { data: Omit<UserRow, "id"> }) => {
+        const id = newId("user");
+        const row: UserRow = { id, ...args.data };
+        s.users.set(id, row);
+        return row;
+      }),
+    },
+    session: {
+      findUnique: vi.fn(async (args: { where: { token?: string; id?: string }; include?: { user?: boolean } }) => {
+        let session: SessionRow | undefined;
+        if (args.where.token) {
+          for (const sess of s.sessions.values()) if (sess.token === args.where.token) { session = sess; break; }
+        } else if (args.where.id) {
+          session = s.sessions.get(args.where.id);
+        }
+        if (!session) return null;
+        if (args.include?.user) {
+          return { ...session, user: s.users.get(session.userId)! };
+        }
+        return session;
+      }),
+      create: vi.fn(async (args: { data: { userId: string; token: string; expiresAt: Date } }) => {
+        const id = newId("sess");
+        const row: SessionRow = { id, ...args.data };
+        s.sessions.set(id, row);
+        return row;
+      }),
+      delete: vi.fn(async (args: { where: { id: string } }) => {
+        s.sessions.delete(args.where.id);
+        return null;
+      }),
+    },
+    groupMember: {
+      findMany: vi.fn(async (args: { where: { userId: string } }) =>
+        s.groupMembers.filter((m) => m.userId === args.where.userId),
+      ),
+    },
+    rbacDefinition: {
+      findMany: vi.fn(async () => s.rbacDefs),
+    },
+    auditEvent: {
+      createMany: vi.fn(async (args: { data: Array<Omit<AuditEventRow, "id" | "timestamp">> }) => {
+        const ts = new Date();
+        for (const e of args.data) {
+          s.auditEvents.push({ id: newId("evt"), timestamp: ts, ...e });
+        }
+        return { count: args.data.length };
+      }),
+      findMany: vi.fn(async (args: { where?: Record<string, unknown>; orderBy?: unknown; take?: number }) => {
+        const where = args.where ?? {};
+        const filtered = s.auditEvents.filter((e) => {
+          if (where["eventKind"] && e.eventKind !== where["eventKind"]) return false;
+          if (where["correlationId"] && e.correlationId !== where["correlationId"]) return false;
+          if (where["environmentName"] && e.environmentName !== where["environmentName"]) return false;
+          return true;
+        });
+        return filtered.slice(0, args.take ?? 100);
+      }),
+    },
+    resource: {
+      findMany: vi.fn(async () => s.resources),
+      findUnique: vi.fn(),
+      create: vi.fn(),
+      update: vi.fn(),
+      delete: vi.fn(),
+    },
+    environment: { findMany: vi.fn(async () => []), findUnique: vi.fn(), create: vi.fn() },
+    account: { findMany: vi.fn(async () => []), findUnique: vi.fn(), create: vi.fn() },
+    binding: { findMany: vi.fn(async () => []), create: vi.fn() },
+  } as unknown as DbClient;
+}
+
+async function buildApp(s: Stores) {
+  const db = makeMockDb(s);
+  const result = await createApp(TEST_CONFIG, db);
+  await result.app.ready();
+  return result;
+}
+
+describe("v2 auth: bootstrap flow", () => {
+  let stores: Stores;
+  let app: Awaited<ReturnType<typeof buildApp>>["app"];
+  let auditService: AuditService;
+
+  beforeEach(async () => {
+    stores = makeStores();
+    const built = await buildApp(stores);
+    app = built.app;
+    auditService = built.auditService;
+  });
+
+  afterEach(async () => {
+    await app.close(); // triggers auditService.stop()
+  });
+
+  it("first login with no users seeds the admin and returns a session token", async () => {
+    expect(stores.users.size).toBe(0);
+
+    const resp = await app.inject({
+      method: "POST",
+      url: "/api/auth/login",
+      payload: { email: "admin@itaz.eu", password: "s3cret-pw" },
+    });
+
+    expect(resp.statusCode).toBe(200);
+    const body = resp.json();
+    expect(body.isBootstrap).toBe(true);
+    expect(body.token).toMatch(/^[a-f0-9]{64}$/);
+    expect(typeof body.expiresAt).toBe("string");
+
+    expect(stores.users.size).toBe(1);
+    const created = [...stores.users.values()][0]!;
+    expect(created.email).toBe("admin@itaz.eu");
+    expect(created.role).toBe("ADMIN");
+    // Password is hashed, not stored plaintext.
+    expect(created.password).not.toBe("s3cret-pw");
+    expect(await bcrypt.compare("s3cret-pw", created.password)).toBe(true);
+
+    // Bootstrap emits an audit event.
+    await auditService.flushPending();
+    const bootstrapEvents = stores.auditEvents.filter((e) => e.eventKind === "auth_bootstrap");
+    expect(bootstrapEvents).toHaveLength(1);
+    expect(bootstrapEvents[0]!.result).toBe("success");
+    expect(bootstrapEvents[0]!.userName).toBe("admin@itaz.eu");
+  });
+
+  it("returns 400 for missing credentials", async () => {
+    const resp = await app.inject({ method: "POST", url: "/api/auth/login", payload: {} });
+    expect(resp.statusCode).toBe(400);
+  });
+
+  it("second login uses normal flow (no isBootstrap)", async () => {
+    // Bootstrap once
+    await app.inject({
+      method: "POST",
+      url: "/api/auth/login",
+      payload: { email: "admin@itaz.eu", password: "s3cret-pw" },
+    });
+    expect(stores.users.size).toBe(1);
+
+    // Login again
+    const resp = await app.inject({
+      method: "POST",
+      url: "/api/auth/login",
+      payload: { email: "admin@itaz.eu", password: "s3cret-pw" },
+    });
+
+    expect(resp.statusCode).toBe(200);
+    expect(resp.json().isBootstrap).toBe(false);
+    expect(stores.users.size).toBe(1); // no new user
+  });
+
+  it("rejects wrong password with 401", async () => {
+    // Seed admin
+    await app.inject({
+      method: "POST",
+      url: "/api/auth/login",
+      payload: { email: "admin@itaz.eu", password: "s3cret-pw" },
+    });
+
+    const resp = await app.inject({
+      method: "POST",
+      url: "/api/auth/login",
+      payload: { email: "admin@itaz.eu", password: "wrong" },
+    });
+    expect(resp.statusCode).toBe(401);
+
+    // Failed login is also audited.
+    await auditService.flushPending();
+    const fails = stores.auditEvents.filter((e) => e.eventKind === "auth_login" && e.result === "failure");
+    expect(fails).toHaveLength(1);
+  });
+});
+
+describe("v2 RBAC: env-scoped denial", () => {
+  let stores: Stores;
+  let app: Awaited<ReturnType<typeof buildApp>>["app"];
+
+  async function seedSession(role: string): Promise<string> {
+    stores.users.set("u-1", {
+      id: "u-1",
+      email: `${role.toLowerCase()}@itaz.eu`,
+      password: "x",
+      role,
+      name: null,
+    });
+    const token = "test-token-" + role;
+    stores.sessions.set("s-1", {
+      id: "s-1",
+      userId: "u-1",
+      token,
+      expiresAt: new Date(Date.now() + 86_400_000),
+    });
+    return token;
+  }
+
+  beforeEach(async () => {
+    stores = makeStores();
+    app = (await buildApp(stores)).app;
+  });
+
+  afterEach(async () => {
+    await app.close();
+  });
+
+  it("non-admin user with no role bindings gets 403 on /api/resources", async () => {
+    const token = await seedSession("EDITOR"); // not admin, no bindings
+
+    const resp = await app.inject({
+      method: "GET",
+      url: "/api/resources",
+      headers: { authorization: `Bearer ${token}` },
+    });
+
+    expect(resp.statusCode).toBe(403);
+    expect(resp.json().error).toMatch(/no matching role binding/);
+  });
+
+  it("missing/empty bearer token gets 401 (auth, not RBAC)", async () => {
+    const r1 = await app.inject({ method: "GET", url: "/api/resources" });
+    expect(r1.statusCode).toBe(401);
+
+    const r2 = await app.inject({
+      method: "GET",
+      url: "/api/resources",
+      headers: { authorization: "Bearer " },
+    });
+    expect(r2.statusCode).toBe(401);
+  });
+
+  it("invalid bearer token gets 401", async () => {
+    const resp = await app.inject({
+      method: "GET",
+      url: "/api/resources",
+      headers: { authorization: "Bearer not-a-real-token" },
+    });
+    expect(resp.statusCode).toBe(401);
+  });
+
+  it("admin role bypasses RBAC", async () => {
+    const token = await seedSession("ADMIN");
+
+    const resp = await app.inject({
+      method: "GET",
+      url: "/api/resources",
+      headers: { authorization: `Bearer ${token}` },
+    });
+
+    expect(resp.statusCode).toBe(200);
+    expect(resp.json()).toEqual([]);
+  });
+
+  it("user with binding for env A is denied for resources in env B", async () => {
+    const token = await seedSession("EDITOR");
+    stores.groupMembers.push({ userId: "u-1", group: { name: "team-a" } });
+    stores.rbacDefs.push({
+      id: "rbac-1",
+      name: "team-a-edit-on-env-a",
+      subjects: [{ kind: "Group", name: "team-a" }],
+      roleBindings: [{ role: "edit", environment: "env-a" }],
+    });
+
+    // List in env-a → should pass RBAC (no env query so it's global view, but
+    // the binding scope is environment-specific → for global list the binding
+    // doesn't apply when an environment scope is set on the binding).
+    // Smoke test the targeted denial: trying to create in env-b is rejected.
+    const respB = await app.inject({
+      method: "POST",
+      url: "/api/resources",
+      headers: { authorization: `Bearer ${token}` },
+      payload: { kind: "database", name: "x", environmentId: "env-b", accountId: "acc-1" },
+    });
+
+    expect(respB.statusCode).toBe(403);
+    expect(respB.json().error).toMatch(/no matching role binding/);
+  });
+});
+
+describe("v2 audit: correlation chain visible via /api/events", () => {
+  let stores: Stores;
+  let app: Awaited<ReturnType<typeof buildApp>>["app"];
+  let auditService: AuditService;
+
+  beforeEach(async () => {
+    stores = makeStores();
+    const built = await buildApp(stores);
+    app = built.app;
+    auditService = built.auditService;
+  });
+
+  afterEach(async () => {
+    await app.close();
+  });
+
+  it("emitted audit events are queryable by correlation id", async () => {
+    // Seed admin so /api/events is accessible (it sits behind bearer auth)
+    const loginResp = await app.inject({
+      method: "POST",
+      url: "/api/auth/login",
+      payload: { email: "admin@itaz.eu", password: "pw" },
+    });
+    const token = loginResp.json().token;
+
+    // Force flush so the bootstrap event is in the DB
+    await auditService.flushPending();
+
+    expect(stores.auditEvents.length).toBeGreaterThan(0);
+    const bootstrap = stores.auditEvents.find((e) => e.eventKind === "auth_bootstrap")!;
+    expect(bootstrap.correlationId).toMatch(/^corr_[a-f0-9]{16}$/);
+
+    // Query /api/events filtered by correlation id
+    const queryResp = await app.inject({
+      method: "GET",
+      url: `/api/events?correlation=${bootstrap.correlationId}`,
+      headers: { authorization: `Bearer ${token}` },
+    });
+
+    expect(queryResp.statusCode).toBe(200);
+    const events = queryResp.json() as Array<{ correlationId: string; eventKind: string }>;
+    expect(events.length).toBe(1);
+    expect(events[0]!.eventKind).toBe("auth_bootstrap");
+    expect(events[0]!.correlationId).toBe(bootstrap.correlationId);
+  });
+
+  it("explicit parent/child correlation chain is preserved across emits", async () => {
+    const correlationId = auditService.createCorrelation();
+
+    auditService.emit({
+      eventKind: "test_parent",
+      source: "test",
+      result: "success",
+      correlationId,
+    });
+    auditService.emit({
+      eventKind: "test_child",
+      source: "test",
+      result: "success",
+      correlationId,
+      parentEventId: "evt-1",
+    });
+
+    await auditService.flushPending();
+
+    const chain = stores.auditEvents.filter((e) => e.correlationId === correlationId);
+    expect(chain).toHaveLength(2);
+    expect(chain.map((e) => e.eventKind).sort()).toEqual(["test_child", "test_parent"]);
+    expect(chain.find((e) => e.eventKind === "test_child")!.parentEventId).toBe("evt-1");
+  });
+});

bastion/src/modules/modules/k3s/src/groups/networking.ts

@@ -3,6 +3,8 @@
 import type { OperationContext, OperationResult, OperationGroup } from "../types.js";
 import { runSequential } from "../utils.js";
 import { installCilium } from "../operations/cilium.js";
+import { installMultus } from "../operations/multus.js";
+import { installVlanSetup } from "../operations/vlan-setup.js";
 import { fixCoreDnsUpstream } from "../operations/dns-fix.js";
 import { applyDefaultNetworkPolicies } from "../operations/network-policy.js";
 
@@ -11,6 +13,11 @@ export const networkingGroup: OperationGroup = {
   description: "Install Cilium CNI, fix DNS, apply network policies",
   operations: [
     { name: "Install Cilium CNI", fn: installCilium },
+    // Multus + vlan-setup: give pods a second interface on VLAN 10 (macvlan)
+    // for LAN device discovery (Matter/HomeKit mDNS). Must follow Cilium
+    // (needs cni.exclusive=false + bpf.vlanBypass={10} from installCilium).
+    { name: "Install Multus CNI", fn: installMultus },
+    { name: "Install vlan-setup (lan10 + CNI plugins)", fn: installVlanSetup },
     { name: "Fix CoreDNS upstream", fn: fixCoreDnsUpstream },
     { name: "Apply network policies", fn: applyDefaultNetworkPolicies },
   ],

bastion/src/modules/modules/k3s/src/operations/cilium.ts

@@ -38,12 +38,20 @@ export const installCilium: Operation = async (ctx): Promise<OperationResult> =>
   // Install Cilium
   // - No hardcoded devices: Cilium auto-detects per node (heterogeneous NICs like eno1 vs enP7s7)
   // - k8sServiceHost/Port: k3s agents proxy the API on 127.0.0.1:6444 (not 6443)
+  // - cni.exclusive=false: required so Multus can install its CNI config alongside
+  //   Cilium (Cilium otherwise deletes any non-Cilium CNI conf).
+  // - bpf.vlanBypass={10}: allow VLAN 10 (LoT) tagged traffic through the eBPF
+  //   host VLAN filter, so pods on a macvlan/VLAN-10 interface receive multicast
+  //   (Matter/mDNS ff02::fb + 224.0.0.251). Without this Cilium drops it
+  //   ("VLAN traffic disallowed by VLAN filter", bpf_host.c).
   const installResult = await ctx.ssh.exec(
     `KUBECONFIG=/etc/rancher/k3s/k3s.yaml cilium install \
       --set kubeProxyReplacement=true \
       --set ipam.mode=kubernetes \
       --set k8sServiceHost=127.0.0.1 \
-      --set k8sServicePort=6444`,
+      --set k8sServicePort=6444 \
+      --set cni.exclusive=false \
+      --set bpf.vlanBypass="{10}"`,
     { timeoutMs: 300_000 },
   );
   if (installResult.exitCode !== 0) {

bastion/src/modules/modules/k3s/src/operations/index.ts

@@ -9,6 +9,8 @@ export { writeAuditPolicy } from "./audit-policy.js";
 export { cleanupStaleCni } from "./cni-cleanup.js";
 export { installK3sBinary } from "./k3s-install.js";
 export { installCilium } from "./cilium.js";
+export { installMultus } from "./multus.js";
+export { installVlanSetup } from "./vlan-setup.js";
 export { fixCoreDnsUpstream } from "./dns-fix.js";
 export { configureLogRotation } from "./log-rotation.js";
 export { configureJournaldLimits } from "./journald-limits.js";

bastion/src/modules/modules/k3s/src/operations/multus.ts

@@ -0,0 +1,34 @@
+// Install Multus CNI (thick plugin) — the meta-CNI that lets pods attach an
+// extra interface (macvlan on VLAN 10) alongside Cilium, via a
+// NetworkAttachmentDefinition. Required for Home Assistant's LAN presence
+// (Matter/HomeKit mDNS discovery). Cilium must be installed with
+// cni.exclusive=false first (see cilium.ts) or it deletes Multus's CNI conf.
+
+import type { Operation, OperationResult } from "../types.js";
+import { sshOpts } from "../utils.js";
+
+const MULTUS_VERSION = "v4.1.4";
+const MULTUS_MANIFEST = `https://raw.githubusercontent.com/k8snetworkplumbingwg/multus-cni/${MULTUS_VERSION}/deployments/multus-daemonset-thick.yml`;
+
+export const installMultus: Operation = async (ctx): Promise<OperationResult> => {
+  const K = "KUBECONFIG=/etc/rancher/k3s/k3s.yaml";
+
+  // Idempotent: skip if the Multus DaemonSet is already present.
+  const check = await ctx.ssh.exec(
+    `${K} kubectl -n kube-system get ds kube-multus-ds -o name 2>/dev/null`,
+    sshOpts(ctx),
+  );
+  if (check.exitCode === 0 && check.stdout.includes("kube-multus-ds")) {
+    return { success: true, changed: false, message: `Multus already installed (${MULTUS_VERSION})` };
+  }
+
+  const apply = await ctx.ssh.exec(
+    `${K} kubectl apply -f ${MULTUS_MANIFEST}`,
+    { ...sshOpts(ctx), timeoutMs: 120_000 },
+  );
+  if (apply.exitCode !== 0) {
+    return { success: false, changed: false, message: "Failed to apply Multus manifest", error: apply.stderr };
+  }
+
+  return { success: true, changed: true, message: `Installed Multus ${MULTUS_VERSION} (thick)` };
+};

bastion/src/modules/modules/k3s/src/operations/vlan-setup.ts

@@ -0,0 +1,96 @@
+// vlan-setup DaemonSet — the node-level half of the macvlan/VLAN-10 story.
+// On every node it (1) installs the reference CNI plugins (macvlan/ipvlan/
+// static/host-local/vlan/tuning) into /opt/cni/bin if missing, and (2) creates
+// a `lan10` VLAN-10 sub-interface on the primary NIC that macvlan
+// NetworkAttachmentDefinitions use as their master. Idempotent + self-healing
+// (re-creates lan10 if it disappears). Paired with Multus (multus.ts) + Cilium
+// bpf.vlanBypass={10} (cilium.ts).
+
+import type { Operation, OperationResult } from "../types.js";
+import { sshOpts } from "../utils.js";
+
+const MANIFEST = `apiVersion: v1
+kind: Namespace
+metadata:
+  name: macvlan-sys
+  labels:
+    pod-security.kubernetes.io/enforce: privileged
+    pod-security.kubernetes.io/audit: privileged
+    pod-security.kubernetes.io/warn: privileged
+---
+apiVersion: apps/v1
+kind: DaemonSet
+metadata:
+  name: vlan-setup
+  namespace: macvlan-sys
+spec:
+  selector:
+    matchLabels: { app: vlan-setup }
+  template:
+    metadata:
+      labels: { app: vlan-setup }
+    spec:
+      hostNetwork: true
+      tolerations:
+      - operator: Exists
+      containers:
+      - name: vlan
+        image: nicolaka/netshoot
+        securityContext:
+          privileged: true
+        command:
+        - sh
+        - -c
+        - |
+          set -x
+          # install reference CNI plugins (macvlan/ipvlan/static/host-local) if missing
+          if [ ! -f /host/opt/cni/bin/macvlan ] || [ ! -f /host/opt/cni/bin/ipvlan ]; then
+            case "$(uname -m)" in x86_64) A=amd64;; aarch64) A=arm64;; *) A=amd64;; esac
+            curl -sSL "https://github.com/containernetworking/plugins/releases/download/v1.5.1/cni-plugins-linux-$A-v1.5.1.tgz" -o /tmp/cni.tgz
+            tar -xzf /tmp/cni.tgz -C /host/opt/cni/bin ./macvlan ./ipvlan ./static ./host-local ./vlan ./tuning
+          fi
+          # detect the primary NIC (default route dev, else the one holding 192.168.8.x)
+          NIC="$(ip -o -4 route show default 2>/dev/null | awk '{print $5; exit}')"
+          [ -z "$NIC" ] && NIC="$(ip -o -4 addr show 2>/dev/null | awk '/192\\.168\\.8\\./{print $2; exit}')"
+          echo "primary NIC = $NIC"
+          while true; do
+            if [ -n "$NIC" ]; then
+              ip link show lan10 >/dev/null 2>&1 || ip link add link "$NIC" name lan10 type vlan id 10
+              ip link set lan10 up
+              # NIC-driver workarounds for VLAN multicast RX
+              ip link set "$NIC" allmulticast on 2>/dev/null
+              ethtool -K "$NIC" rxvlan off rx-vlan-filter off 2>/dev/null
+            fi
+            sleep 30
+          done
+        volumeMounts:
+        - name: cnibin
+          mountPath: /host/opt/cni/bin
+      volumes:
+      - name: cnibin
+        hostPath:
+          path: /opt/cni/bin
+`;
+
+export const installVlanSetup: Operation = async (ctx): Promise<OperationResult> => {
+  const K = "KUBECONFIG=/etc/rancher/k3s/k3s.yaml";
+
+  const check = await ctx.ssh.exec(
+    `${K} kubectl -n macvlan-sys get ds vlan-setup -o name 2>/dev/null`,
+    sshOpts(ctx),
+  );
+  if (check.exitCode === 0 && check.stdout.includes("vlan-setup")) {
+    return { success: true, changed: false, message: "vlan-setup DaemonSet already installed" };
+  }
+
+  const b64 = Buffer.from(MANIFEST).toString("base64");
+  const apply = await ctx.ssh.exec(
+    `echo ${b64} | base64 -d | ${K} kubectl apply -f -`,
+    { ...sshOpts(ctx), timeoutMs: 60_000 },
+  );
+  if (apply.exitCode !== 0) {
+    return { success: false, changed: false, message: "Failed to apply vlan-setup DaemonSet", error: apply.stderr };
+  }
+
+  return { success: true, changed: true, message: "Installed vlan-setup DaemonSet (lan10 + CNI plugins)" };
+};

bastion/src/shared/src/hardware/index.ts

@@ -0,0 +1,154 @@
+// Architecture normalisation and machine classification.
+//
+// Both are derived from what the system already observes about a machine -- never from
+// an operator-supplied flag.
+
+import type { Arch, HardwareInfo, OnboardMethod, OsId } from "../types/index.js";
+
+export const SUPPORTED_ARCHES: readonly Arch[] = ["x86_64", "aarch64"] as const;
+
+/**
+ * Normalise an architecture string to one we serve boot artifacts for.
+ *
+ * Sources and their spellings:
+ *   uname -m         -> "x86_64" / "aarch64"
+ *   iPXE ${buildarch}-> "x86_64" / "arm64"
+ *   dpkg/Debian      -> "amd64"  / "arm64"
+ *
+ * Returns undefined for anything we don't serve, so callers fall back rather than
+ * inventing a kernel path that would 404.
+ */
+export function normalizeArch(value: string | undefined | null): Arch | undefined {
+  switch ((value ?? "").trim().toLowerCase()) {
+    case "x86_64":
+    case "x86-64":
+    case "amd64":
+      return "x86_64";
+    case "aarch64":
+    case "arm64":
+      return "aarch64";
+    default:
+      return undefined;
+  }
+}
+
+/** Fedora pxeboot artifact base URL for an architecture. */
+export function fedoraMirrorFor(fedoraVersion: string, arch: Arch): string {
+  return `https://download.fedoraproject.org/pub/fedora/linux/releases/${fedoraVersion}/Everything/${arch}/os`;
+}
+
+/**
+ * Which architectures each OS in the pipeline can actually be installed on.
+ *
+ * Fedora publishes pxeboot vmlinuz/initrd for both. Ubuntu does not: as of 26.04,
+ * releases.ubuntu.com publishes amd64 artifacts only, so there is nothing to netboot an
+ * arm64 machine with. Claiming support would fail at download time with a 404 instead
+ * of a useful message.
+ */
+const OS_ARCH_SUPPORT: Record<OsId, readonly Arch[]> = {
+  "fedora-43": ["x86_64", "aarch64"],
+  "ubuntu-26.04": ["x86_64"],
+};
+
+export function osSupportsArch(os: OsId, arch: Arch): boolean {
+  return (OS_ARCH_SUPPORT[os] ?? []).includes(arch);
+}
+
+export function archesForOs(os: OsId): readonly Arch[] {
+  return OS_ARCH_SUPPORT[os] ?? [];
+}
+
+/**
+ * Machines that run a vendor OS we have no image for.
+ *
+ * These are SSH-onboard: we manage userspace, but reinstalling destroys a driver and
+ * firmware stack our pipeline cannot rebuild. Matched on DMI identity, which is what
+ * discovery and `provision recheck` both collect.
+ *
+ * This is deliberately a property of the machine ("it runs DGX OS"), not a blocklist
+ * ("never install this MAC"). When a DGX OS image joins the pipeline, teaching the
+ * installer about vendor_os "dgx-os" is what unblocks these machines -- no entry here
+ * needs deleting.
+ */
+interface VendorOsRule {
+  vendorOs: string;
+  description: string;
+  matches: (hw: DmiIdentity) => boolean;
+}
+
+interface DmiIdentity {
+  manufacturer: string;
+  product: string;
+  board: string;
+}
+
+const VENDOR_OS_RULES: readonly VendorOsRule[] = [
+  {
+    vendorOs: "dgx-os",
+    description: "NVIDIA DGX OS (proprietary driver + firmware stack, no image in our pipeline)",
+    matches: ({ manufacturer, product, board }) =>
+      (manufacturer.includes("nvidia") || product.includes("nvidia")) &&
+      (product.includes("dgx") || product.includes("spark") ||
+       board.includes("gb10") || product.includes("gb10")),
+  },
+];
+
+/**
+ * Machines known to run a vendor OS, by MAC.
+ *
+ * The DMI rules above only fire once discovery or `provision recheck` has populated a
+ * hardware record. Machines onboarded over SSH may sit in state for a long time with no
+ * DMI at all -- which is exactly the state both DGX Sparks are in today -- so a
+ * DMI-only classifier would fail open on the machines this guard exists to protect.
+ *
+ * This is a statement of fact about known hardware ("this box runs DGX OS"), not an
+ * install policy. Whether that means "refuse" is decided by whether the pipeline has an
+ * image for that vendor OS.
+ */
+const KNOWN_VENDOR_OS_MACS: Record<string, string> = {
+  "4c:bb:47:7f:29:35": "dgx-os",  // spark-2935
+  "48:21:0b:96:3a:1c": "dgx-os",  // spark-3a1c
+};
+
+/**
+ * Classify how a machine should be onboarded, from its hardware record.
+ *
+ * An explicit `onboard` already on the record wins: it may have been set by an operator
+ * or by a rule that has since changed, and silently overriding it would be worse than
+ * leaving it.
+ */
+export function classifyOnboard(
+  hw: Partial<Pick<HardwareInfo, "mac" | "manufacturer" | "product" | "board">>
+    & { onboard?: OnboardMethod; vendor_os?: string },
+): { onboard: OnboardMethod; vendor_os?: string } {
+  if (hw.onboard !== undefined) {
+    return hw.vendor_os !== undefined
+      ? { onboard: hw.onboard, vendor_os: hw.vendor_os }
+      : { onboard: hw.onboard };
+  }
+
+  const knownVendorOs = KNOWN_VENDOR_OS_MACS[(hw.mac ?? "").toLowerCase().replace(/-/g, ":")];
+  if (knownVendorOs !== undefined) {
+    return { onboard: "ssh", vendor_os: knownVendorOs };
+  }
+
+  const identity: DmiIdentity = {
+    manufacturer: (hw.manufacturer ?? "").toLowerCase(),
+    product: (hw.product ?? "").toLowerCase(),
+    board: (hw.board ?? "").toLowerCase(),
+  };
+
+  for (const rule of VENDOR_OS_RULES) {
+    if (rule.matches(identity)) {
+      return { onboard: "ssh", vendor_os: rule.vendorOs };
+    }
+  }
+
+  return { onboard: "pxe" };
+}
+
+/** Human-readable reason a vendor-OS machine must not be reinstalled. */
+export function vendorOsDescription(vendorOs: string | undefined): string {
+  const rule = VENDOR_OS_RULES.find((r) => r.vendorOs === vendorOs);
+  return rule?.description ?? "a vendor OS with no image in our pipeline";
+}

bastion/src/shared/src/index.ts

@@ -1,6 +1,8 @@
 export type {
   OsId,
   Arch,
+  OnboardMethod,
+  RootCandidate,
   Role,
   HardwareInfo,
   InstallConfig,
@@ -10,6 +12,16 @@ export type {
   BastionConfig,
 } from "./types/index.js";
 
+export {
+  SUPPORTED_ARCHES,
+  normalizeArch,
+  fedoraMirrorFor,
+  osSupportsArch,
+  archesForOs,
+  classifyOnboard,
+  vendorOsDescription,
+} from "./hardware/index.js";
+
 export { SUPPORTED_OS, SUPPORTED_ROLES, ROLE_REGISTRY, isValidOsId } from "./types/index.js";
 export type { RoleInfo } from "./types/index.js";
 

bastion/src/shared/src/protocol/index.ts

@@ -113,7 +113,7 @@ export type LabdBastionMessage =
   | { type: "command-role-update"; requestId: string; mac: string; role: string }
   | { type: "command-debug"; requestId: string; mac: string; pxeBoot?: boolean }
   | { type: "command-register"; requestId: string; mac: string; hostname: string; role: string; ip: string }
-  | { type: "command-discover"; requestId: string; mac: string; product?: string; board?: string; serial?: string; manufacturer?: string; cpu_model?: string; cpu_cores?: number; memory_gb?: number; arch?: string; disks?: Array<{ name: string; size_gb: number; model: string }>; nics?: Array<{ name: string; mac: string; state: string }> }
+  | { type: "command-discover"; requestId: string; mac: string; product?: string; board?: string; serial?: string; manufacturer?: string; cpu_model?: string; cpu_cores?: number; memory_gb?: number; arch?: string; disks?: Array<{ name: string; size_gb: number; model: string }>; nics?: Array<{ name: string; mac: string; state: string }>; root_device?: string; root_args?: string }
   | { type: "server-shutdown"; reconnectAfter: number };
 
 export type BastionMessageType = BastionMessage["type"];

bastion/src/shared/src/types/index.ts

@@ -1,6 +1,8 @@
 export type {
   OsId,
   Arch,
+  OnboardMethod,
+  RootCandidate,
   Role,
   HardwareInfo,
   InstallConfig,

bastion/src/shared/src/types/state.ts

@@ -11,6 +11,16 @@ export function isValidOsId(value: string): value is OsId {
   return (SUPPORTED_OS as readonly string[]).includes(value);
 }
 
+/**
+ * How a machine joins the lab.
+ *
+ * "pxe"  -- bare metal we install over the network (the default).
+ * "ssh"  -- the machine already runs a vendor OS we cannot reproduce, so we onboard
+ *           over SSH and manage userspace only. Installing would destroy that OS.
+ *           See classifyOnboard() and os-install-research.md.
+ */
+export type OnboardMethod = "pxe" | "ssh";
+
 export interface HardwareInfo {
   mac: string;
   product: string;
@@ -26,6 +36,23 @@ export interface HardwareInfo {
   first_seen: string;
   last_seen: string;
   bastionId?: string;  // set when aggregated through labd
+  // Onboarding classification -- absent means "pxe" (see classifyOnboard)
+  onboard?: OnboardMethod;
+  vendor_os?: string;   // e.g. "dgx-os": the OS this machine must keep running
+  // Root filesystem, for booting the installed system over PXE (--pxe-boot).
+  // Observed from the machine, never assumed.
+  root_device?: string;                 // e.g. "/dev/mapper/labvg-root"
+  root_args?: string;                   // e.g. "rd.lvm.lv=labvg/root rd.lvm.lv=labvg/swap"
+  root_candidates?: RootCandidate[];    // reported from a rescue shell when unknown
+}
+
+/** A possible root filesystem found while probing an unreachable machine. */
+export interface RootCandidate {
+  device: string;       // e.g. "/dev/mapper/labvg-root"
+  args?: string;        // extra dracut args needed to assemble it
+  fstype?: string;
+  size_gb?: number;
+  os_release?: string;  // PRETTY_NAME from /etc/os-release, if mountable
 }
 
 export type Role = "vanilla" | "worker" | "infra" | "labcontroller";
@@ -103,6 +130,11 @@ export interface InstalledInfo {
   cpu_cores?: number;
   memory_gb?: number;
   arch?: string;
+  onboard?: OnboardMethod;
+  vendor_os?: string;
+  root_device?: string;
+  root_args?: string;
+  root_candidates?: RootCandidate[];
 }
 
 export interface DebugConfig {

bastion/tests/integration/arm-pxe-provision.test.ts

@@ -0,0 +1,537 @@
+// Integration test: aarch64 network PXE boot.
+//
+// The boot-ISO path already covered ARM (arm-iso-provision.test.ts). This covers the
+// network path: DHCP option 93 handing an arm64 client an arm64 iPXE binary, dispatch
+// serving an aarch64 kernel, and `provision debug` reaching a rescue shell -- which is
+// what the DGX Sparks actually need and could not do.
+//
+// Two suites, because they cost very different amounts of time:
+//
+//   "ARM PXE rescue"   NBP handoff -> rescue with SSH.                    ~25-30 min
+//   "ARM PXE install"  discover -> install -> installed.                  ~75-95 min
+//
+// The rescue suite seeds the machine into state as an already-known aarch64 box rather
+// than discovering it first. That is the DGX Spark situation exactly -- SSH-onboarded,
+// never PXE-discovered, architecture known only from its record -- and it holds the test
+// to one emulated boot. Each boot spends ~15 of its ~18 minutes downloading Anaconda's
+// stage2 under TCG, so discovering first would double the runtime without touching any
+// code path the rescue boot does not already exercise.
+//
+// The install suite only runs with ARM_PXE_FULL=1. No ARM machine in the lab is ever
+// PXE-installed except the MS-R1, and an hour-plus test that runs by default is a test
+// nobody runs.
+//
+// IMPORTANT: aarch64 has no KVM on an x86_64 host, so all of this is emulated and
+// roughly 10x slower than native.
+//
+// A note for whoever debugs a failure here: if the VM panics with
+//   VFS: Unable to mount root fs on unknown-block(0,0)
+// that is very likely iPXE silently dropping the initrd because the build lacks
+// EFI_LOAD_FILE2_PROTOCOL -- on arm64 the kernel EFI stub fetches the initrd over
+// LoadFile2, and an iPXE without it accepts the `initrd` line and does nothing. It is
+// NOT a reproduction of the DGX Spark kernel bug that motivated this work, despite
+// being the identical message. assertIpxeSupportsLoadFile2() below checks the build up
+// front so that failure names itself; to check by hand:
+//   node -e 'const b=require("fs").readFileSync("/usr/share/ipxe/arm64-efi/snponly.efi");
+//            console.log(b.indexOf(Buffer.from("c1c00640b3fc3e40996d4a6c8724e06d","hex")))'
+// Fedora's ipxe-bootimgs-aarch64-20240119 has it at 0x3bbf0.
+//
+// Prerequisites:
+//   - qemu-system-aarch64        (sudo dnf install qemu-system-aarch64)
+//   - edk2-aarch64               (sudo dnf install edk2-aarch64)
+//   - ipxe-bootimgs-aarch64      (sudo dnf install ipxe-bootimgs-aarch64)
+//   - libvirtd, sudo, internet access
+//
+// Run: sudo ./scripts/test-provision.sh arm-pxe
+
+import { describe, it, expect, beforeAll, afterAll } from "vitest";
+import { readFileSync, existsSync, mkdirSync, rmSync, copyFileSync, writeFileSync } from "node:fs";
+import { execSync } from "node:child_process";
+import { join } from "node:path";
+import { homedir, tmpdir } from "node:os";
+import { log, waitForSsh } from "./helpers/libvirt.js";
+import { ensurePxeNetwork, destroyPxeNetwork, deleteNftablesRejectRules, PXE_NETWORK_NAME, PXE_GATEWAY, PXE_SUBNET } from "./helpers/pxe-network.js";
+import { createPxeVm, destroyPxeVm, getVmMac, rebootPxeVm, readSerialLog } from "./helpers/pxe-vm.js";
+import { sshExec } from "./helpers/ssh.js";
+
+const IPXE_ARM64 = "/usr/share/ipxe/arm64-efi/snponly.efi";
+const AAVMF = "/usr/share/edk2/aarch64/QEMU_EFI.fd";
+
+const VM_MEMORY = 4096;
+const VM_VCPUS = 2;
+const VM_DISK_GB = 250;
+const SSH_USER = "lab";
+const BASTION_IP = PXE_GATEWAY;
+const DHCP_RANGE_START = `${PXE_SUBNET}.100`;
+const DHCP_RANGE_END = `${PXE_SUBNET}.200`;
+const SERIAL_PORT = 4555;
+
+// Emulated aarch64 -- generous timeouts throughout. Measured on an x86_64 host with no
+// KVM for aarch64: a single PXE boot to a running Anaconda takes ~18 minutes, almost all
+// of it downloading inst.stage2 over the network under TCG. Budget well above that;
+// timing out just short of success wastes a whole run.
+const LEASE_TIMEOUT_MS = 10 * 60_000;
+const DISCOVERY_TIMEOUT_MS = 35 * 60_000;
+const INSTALL_TIMEOUT_MS = 75 * 60_000;
+const SSH_TIMEOUT_MS = 35 * 60_000;
+
+const RUN_FULL_INSTALL = process.env["ARM_PXE_FULL"] === "1";
+
+function sleep(ms: number): Promise<void> {
+  return new Promise((r) => setTimeout(r, ms));
+}
+
+function findSshKey(): { pubKey: string; keyPath: string } {
+  const candidates: string[] = [];
+  if (process.env["SSH_KEY_PATH"]) candidates.push(process.env["SSH_KEY_PATH"]);
+  const homes = [homedir()];
+  const sudoUser = process.env["SUDO_USER"];
+  if (sudoUser) homes.push(join("/home", sudoUser));
+  for (const home of homes) {
+    for (const name of ["id_ed25519", "id_ecdsa", "id_rsa"]) {
+      candidates.push(join(home, ".ssh", name));
+    }
+  }
+  for (const keyPath of candidates) {
+    if (existsSync(keyPath) && existsSync(`${keyPath}.pub`)) {
+      return { pubKey: readFileSync(`${keyPath}.pub`, "utf-8").trim(), keyPath };
+    }
+  }
+  throw new Error("No SSH key found — set SSH_KEY_PATH or ensure keys exist in ~/.ssh/");
+}
+
+async function pollApi<T>(
+  url: string,
+  check: (data: T) => boolean,
+  timeoutMs: number,
+  intervalMs = 10_000,
+): Promise<T> {
+  const start = Date.now();
+  while (Date.now() - start < timeoutMs) {
+    try {
+      const res = await fetch(url);
+      if (res.ok) {
+        const data = (await res.json()) as T;
+        if (check(data)) return data;
+      }
+    } catch { /* bastion not up yet, or a network hiccup */ }
+    await sleep(intervalMs);
+  }
+  throw new Error(`Timeout after ${timeoutMs}ms polling ${url}`);
+}
+
+function requirePrerequisites(): void {
+  if (!existsSync("/usr/bin/qemu-system-aarch64")) {
+    throw new Error("qemu-system-aarch64 not installed. Run: sudo dnf install qemu-system-aarch64");
+  }
+  if (!existsSync(AAVMF)) {
+    throw new Error(`AAVMF firmware not found at ${AAVMF}. Run: sudo dnf install edk2-aarch64`);
+  }
+  if (!existsSync(IPXE_ARM64)) {
+    throw new Error(`arm64 iPXE not found at ${IPXE_ARM64}. Run: sudo dnf install ipxe-bootimgs-aarch64`);
+  }
+}
+
+/**
+ * Confirm the arm64 iPXE binary implements EFI_LOAD_FILE2_PROTOCOL.
+ *
+ * Without it the `initrd` line is accepted and silently ignored, and the kernel panics
+ * with unknown-block(0,0). Checking here turns a confusing 30-minute boot failure into
+ * an immediate, explanatory one.
+ *
+ * GUID 4006c0c1-fcb3-403e-996d-4a6c8724e06d, little-endian in the binary's GUID table.
+ */
+function assertIpxeSupportsLoadFile2(): void {
+  const LOAD_FILE2_GUID = Buffer.from("c1c00640b3fc3e40996d4a6c8724e06d", "hex");
+  const binary = readFileSync(IPXE_ARM64);
+  if (binary.indexOf(LOAD_FILE2_GUID) < 0) {
+    throw new Error(
+      `${IPXE_ARM64} does not reference EFI_LOAD_FILE2_PROTOCOL. On arm64 the kernel ` +
+      `EFI stub fetches the initrd over LoadFile2; without it iPXE drops the initrd ` +
+      `silently and the kernel panics with "unknown-block(0,0)". Rebuild iPXE with ` +
+      `LoadFile2, or chainload grubaa64.efi for aarch64 instead.`,
+    );
+  }
+  log(`iPXE arm64 implements LoadFile2 — initrd will be delivered to the EFI stub`);
+}
+
+interface Harness {
+  testDir: string;
+  app: { close: () => Promise<void> };
+  stopDnsmasq: () => void;
+  state: { update: (fn: (s: BastionStateLike) => void) => void };
+  vmMac: string;
+  httpPort: number;
+}
+
+/** Just the parts of BastionState this test seeds. */
+interface BastionStateLike {
+  discovered: Record<string, Record<string, unknown>>;
+  installed: Record<string, Record<string, unknown>>;
+  install_queue: Record<string, Record<string, unknown>>;
+  debug: Record<string, Record<string, unknown>>;
+}
+
+/** Bring up an isolated network, a bastion with both arch payloads, and an arm64 VM. */
+async function startHarness(vmName: string, httpPort: number, pubKey: string): Promise<Harness> {
+  requirePrerequisites();
+  assertIpxeSupportsLoadFile2();
+
+  log("Setting up PXE test network...");
+  ensurePxeNetwork();
+
+  const testDir = join(tmpdir(), `lab-arm-pxe-test-${Date.now()}`);
+  for (const sub of ["tftp", "http", "logs"]) {
+    mkdirSync(join(testDir, sub), { recursive: true });
+  }
+
+  const { createApp } = await import("../../src/bastion/src/server.js");
+  const { loadConfig } = await import("../../src/bastion/src/config.js");
+  const { generateDnsmasqConf, startDnsmasq, stopDnsmasq } = await import("../../src/bastion/src/services/dnsmasq.js");
+  const { generateDiscoverKickstart } = await import("../../src/bastion/src/services/kickstart-generator.js");
+  const { renderBootIpxe, kernelPath, initrdPath } = await import("../../src/bastion/src/templates/boot.ipxe.js");
+  // Relative, not "@lab/shared": these tests run from the repo root against sources,
+  // where the workspace package alias is not resolvable.
+  const { SUPPORTED_ARCHES, fedoraMirrorFor } = await import("../../src/shared/src/hardware/index.js");
+
+  const config = loadConfig({
+    bastionDir: testDir,
+    httpPort,
+    iface: "virbr-pxe",
+    serverIp: BASTION_IP,
+    network: `${PXE_SUBNET}.0`,
+    gateway: BASTION_IP,
+    dhcpMode: "full",
+    dhcpRangeStart: DHCP_RANGE_START,
+    dhcpRangeEnd: DHCP_RANGE_END,
+    domain: "arm-pxe-test.local",
+    sshKeys: [pubKey],
+    adminUser: SSH_USER,
+  });
+
+  // iPXE binaries. The arm64 one is the whole point: dnsmasq hands it out on DHCP
+  // option 93 -- 11 for UEFI PXE (TFTP) and 19 for UEFI HTTP Boot.
+  //
+  // They go in BOTH directories, exactly as main.ts stages them. AAVMF prefers HTTP
+  // Boot, so it is served an http:// URL and fetches from httpDir; a firmware that
+  // takes the TFTP path reads the same file from tftpDir. Staging only tftpDir gives a
+  // 404 and "No bootable option or device was found" on the console.
+  log("Staging iPXE binaries...");
+  const ipxeX86 = "/usr/share/ipxe/ipxe-snponly-x86_64.efi";
+  copyFileSync(IPXE_ARM64, join(config.tftpDir, "ipxe-arm64.efi"));
+  copyFileSync(IPXE_ARM64, join(config.httpDir, "ipxe-arm64.efi"));
+  if (existsSync(ipxeX86)) {
+    copyFileSync(ipxeX86, join(config.tftpDir, "ipxe.efi"));
+    copyFileSync(ipxeX86, join(config.httpDir, "ipxe.efi"));
+  }
+
+  // Fedora kernel + initrd for both architectures, cached across runs.
+  const cacheDir = "/var/lib/libvirt/images/lab-pxe-cache";
+  execSync(`mkdir -p "${cacheDir}"`, { stdio: "pipe" });
+
+  for (const arch of SUPPORTED_ARCHES) {
+    const mirror = fedoraMirrorFor(config.fedoraVersion, arch);
+    const kernelCache = join(cacheDir, `vmlinuz-${arch}`);
+    const initrdCache = join(cacheDir, `initrd-${arch}.img`);
+
+    if (!existsSync(kernelCache)) {
+      log(`Downloading Fedora ${config.fedoraVersion} ${arch} kernel...`);
+      execSync(`curl -# -L -f -o "${kernelCache}" "${mirror}/images/pxeboot/vmlinuz"`, { stdio: "inherit", timeout: 600_000 });
+    }
+    if (!existsSync(initrdCache)) {
+      log(`Downloading Fedora ${config.fedoraVersion} ${arch} initrd...`);
+      execSync(`curl -# -L -f -o "${initrdCache}" "${mirror}/images/pxeboot/initrd.img"`, { stdio: "inherit", timeout: 600_000 });
+    }
+
+    // Staged under the exact names the iPXE templates will ask for.
+    copyFileSync(kernelCache, join(config.httpDir, kernelPath(arch)));
+    copyFileSync(initrdCache, join(config.httpDir, initrdPath(arch)));
+    log(`Staged ${arch}: ${kernelPath(arch)} + ${initrdPath(arch)}`);
+  }
+
+  writeFileSync(join(config.httpDir, "discover.ks"), generateDiscoverKickstart(config));
+  writeFileSync(
+    join(config.httpDir, "boot.ipxe"),
+    renderBootIpxe({ serverIp: config.serverIp, httpPort: config.httpPort }),
+  );
+  generateDnsmasqConf(config);
+
+  const { app, state, syslog } = createApp(config);
+  await app.listen({ port: config.httpPort, host: "0.0.0.0" });
+  syslog.start();
+  log(`Bastion HTTP listening on :${config.httpPort}`);
+
+  log("Starting dnsmasq (full DHCP)...");
+  startDnsmasq(config).catch((err) => {
+    log(`dnsmasq failed: ${err instanceof Error ? err.message : String(err)}`);
+  });
+  await sleep(1500);
+
+  log("Creating aarch64 PXE VM (emulated — this is slow)...");
+  createPxeVm({
+    name: vmName,
+    memory: VM_MEMORY,
+    vcpus: VM_VCPUS,
+    diskSize: VM_DISK_GB,
+    network: PXE_NETWORK_NAME,
+    arch: "aarch64",
+  });
+
+  const vmMac = getVmMac(vmName);
+  if (!vmMac) throw new Error("Could not determine VM MAC address");
+  log(`ARM VM MAC: ${vmMac}`);
+
+  return {
+    testDir,
+    app,
+    stopDnsmasq,
+    state: state as unknown as Harness["state"],
+    vmMac,
+    httpPort: config.httpPort,
+  };
+}
+
+async function stopHarness(vmName: string, harness: Harness | undefined): Promise<void> {
+  // KEEP_VM=1 leaves the VM, network and bastion up so a failure can be inspected on
+  // the console. Emulated aarch64 runs cost half an hour; tearing the evidence down
+  // automatically means paying that again to see what happened.
+  if (process.env["KEEP_VM"] === "1") {
+    log(`KEEP_VM=1 — leaving ${vmName} running for inspection.`);
+    log(`  console:  sudo virsh screenshot ${vmName} /tmp/vm.ppm`);
+    log(`  serial:   socat - TCP:127.0.0.1:${SERIAL_PORT}`);
+    if (harness) log(`  bastion:  ${harness.testDir} (still serving on :${harness.httpPort})`);
+    log(`  cleanup:  sudo virsh destroy ${vmName}; sudo virsh undefine ${vmName} --remove-all-storage --nvram`);
+    return;
+  }
+
+  log("Cleaning up...");
+  if (harness) {
+    await harness.app.close().catch(() => {});
+    harness.stopDnsmasq();
+  }
+  destroyPxeVm(vmName);
+  destroyPxeNetwork();
+  if (harness) rmSync(harness.testDir, { recursive: true, force: true });
+}
+
+/** Read the DHCP lease the bastion handed a MAC. Rescue mode reports no IP itself. */
+function leaseIpFor(testDir: string, mac: string): string | null {
+  const leaseFile = join(testDir, "dnsmasq.leases");
+  if (!existsSync(leaseFile)) return null;
+  for (const line of readFileSync(leaseFile, "utf-8").split("\n")) {
+    // <expiry> <mac> <ip> <hostname> <clientid>
+    const parts = line.trim().split(/\s+/);
+    if (parts.length >= 3 && parts[1]?.toLowerCase() === mac.toLowerCase()) {
+      return parts[2] ?? null;
+    }
+  }
+  return null;
+}
+
+async function waitForLease(testDir: string, mac: string, timeoutMs: number): Promise<string> {
+  const start = Date.now();
+  while (Date.now() - start < timeoutMs) {
+    const ip = leaseIpFor(testDir, mac);
+    if (ip !== null) return ip;
+    await sleep(5000);
+  }
+  throw new Error(`No DHCP lease for ${mac} within ${timeoutMs}ms`);
+}
+
+// ---------------------------------------------------------------------------
+// Rescue path -- what the DGX Sparks need.
+// ---------------------------------------------------------------------------
+
+describe("ARM PXE rescue", () => {
+  const VM_NAME = "lab-arm-pxe-rescue";
+  const HTTP_PORT = 8096;
+  let harness: Harness | undefined;
+  let sshKeyPath: string;
+  let rescueIp: string;
+
+  beforeAll(async () => {
+    const { pubKey, keyPath } = findSshKey();
+    sshKeyPath = keyPath;
+    harness = await startHarness(VM_NAME, HTTP_PORT, pubKey);
+    const { testDir, vmMac, state } = harness;
+
+    // Seed the machine as an already-known aarch64 box queued for rescue. This is the
+    // DGX Spark situation exactly: SSH-onboarded, never PXE-discovered, architecture
+    // known only from its record -- and it also keeps the test to a SINGLE emulated
+    // boot. Each boot spends ~15 minutes pulling Anaconda's stage2 over the network
+    // under TCG, so discovering first and rescuing second doubles the runtime for no
+    // extra coverage of the path being tested. Discovery is covered by the full suite.
+    log(`Seeding ${vmMac} as a known aarch64 machine queued for rescue...`);
+    state.update((s) => {
+      s.discovered[vmMac] = {
+        mac: vmMac,
+        product: "Test ARM64 Machine",
+        board: "virt",
+        serial: "SN-ARM64",
+        manufacturer: "QEMU",
+        cpu_model: "cortex-a57",
+        cpu_cores: VM_VCPUS,
+        memory_gb: 4,
+        arch: "aarch64",
+        disks: [],
+        nics: [],
+        first_seen: new Date().toISOString(),
+        last_seen: new Date().toISOString(),
+      };
+      s.debug[vmMac] = { hostname: "arm-rescue-test", queued_at: new Date().toISOString() };
+    });
+
+    // Restart so the VM boots against the seeded state. createPxeVm already started it.
+    rebootPxeVm(VM_NAME);
+    await sleep(5_000);
+    deleteNftablesRejectRules();
+
+    // The whole chain now runs once: DHCP option 93 -> arm64 iPXE -> /boot.ipxe ->
+    // /dispatch (architecture from the record, not the query) -> aarch64 kernel +
+    // initrd -> Anaconda rescue -> sshd. Reaching a shell at all proves iPXE handed
+    // the initrd to the EFI stub over LoadFile2; without it the kernel panics first.
+    log("Waiting for the rescue environment's DHCP lease...");
+    rescueIp = await waitForLease(testDir, vmMac, LEASE_TIMEOUT_MS);
+    log(`Rescue IP: ${rescueIp}`);
+
+    log("Waiting for SSH into the rescue shell (started by inst.sshd)...");
+    log("(emulated aarch64 — Anaconda's stage2 download dominates; be patient)");
+    await waitForSsh(rescueIp, "root", SSH_TIMEOUT_MS, sshKeyPath).catch(async (err) => {
+      log("Rescue SSH timed out. Serial console:");
+      try {
+        log(await readSerialLog(SERIAL_PORT, { lastLines: 100, timeoutMs: 15_000 }));
+      } catch { /* console unavailable */ }
+      throw err;
+    });
+    log("ARM PXE rescue reached.");
+  }, LEASE_TIMEOUT_MS + SSH_TIMEOUT_MS + 300_000);
+
+  afterAll(async () => { await stopHarness(VM_NAME, harness); });
+
+  it("resolved the architecture from the machine record", async () => {
+    const res = await fetch(`http://${BASTION_IP}:${HTTP_PORT}/api/machines`);
+    const data = (await res.json()) as { discovered: Record<string, { arch: string }> };
+    expect(data.discovered[harness!.vmMac]?.arch).toBe("aarch64");
+  });
+
+  it("rescue shell is reachable over SSH and is aarch64", () => {
+    const result = sshExec(rescueIp, "root", "uname -m", { keyPath: sshKeyPath, timeout: 60_000 });
+    expect(result.exitCode).toBe(0);
+    expect(result.stdout.trim()).toBe("aarch64");
+  });
+
+  it("booted an initramfs — the LoadFile2 path worked", () => {
+    // If iPXE had dropped the initrd the kernel would never have reached userspace at
+    // all, but assert it explicitly so a regression names itself.
+    const result = sshExec(rescueIp, "root", "cat /proc/cmdline; ls /run/install", {
+      keyPath: sshKeyPath, timeout: 60_000,
+    });
+    expect(result.exitCode).toBe(0);
+    expect(result.stdout).toContain("inst.rescue");
+  });
+
+  it("rescue kernel came from the bastion over HTTP", () => {
+    const result = sshExec(rescueIp, "root", "cat /proc/cmdline", { keyPath: sshKeyPath, timeout: 60_000 });
+    expect(result.stdout).toContain(`${BASTION_IP}:${HTTP_PORT}`);
+    // arm64 gets serial console arguments, never nomodeset.
+    expect(result.stdout).toContain("console=ttyAMA0");
+    expect(result.stdout).not.toContain("nomodeset");
+  });
+
+  it("has LVM tools available for inspecting an installed system", () => {
+    const result = sshExec(rescueIp, "root", "command -v vgchange && command -v lsblk", {
+      keyPath: sshKeyPath, timeout: 60_000,
+    });
+    expect(result.exitCode).toBe(0);
+  });
+});
+
+// ---------------------------------------------------------------------------
+// Full install -- opt-in, ~60-90 minutes emulated.
+// ---------------------------------------------------------------------------
+
+describe.runIf(RUN_FULL_INSTALL)("ARM PXE install", () => {
+  const VM_NAME = "lab-arm-pxe-install";
+  const HTTP_PORT = 8095;
+  let harness: Harness | undefined;
+  let sshKeyPath: string;
+  let vmIp: string;
+
+  beforeAll(async () => {
+    const { pubKey, keyPath } = findSshKey();
+    sshKeyPath = keyPath;
+    harness = await startHarness(VM_NAME, HTTP_PORT, pubKey);
+    const { vmMac } = harness;
+
+    log("Waiting for aarch64 discovery...");
+    await pollApi<{ discovered: Record<string, unknown> }>(
+      `http://${BASTION_IP}:${HTTP_PORT}/api/machines`,
+      (data) => vmMac in data.discovered,
+      DISCOVERY_TIMEOUT_MS,
+    );
+    log("Discovered. Queueing install...");
+
+    const installRes = await fetch(`http://${BASTION_IP}:${HTTP_PORT}/api/install`, {
+      method: "POST",
+      headers: { "Content-Type": "application/json" },
+      body: JSON.stringify({ mac: vmMac, hostname: VM_NAME, disk: "", role: "vanilla" }),
+    });
+    expect(installRes.status).toBe(200);
+
+    await sleep(30_000);
+    rebootPxeVm(VM_NAME);
+
+    log("Waiting for the emulated aarch64 install (60-90 min)...");
+    type LogsResponse = { status: string; progress: string; ip?: string };
+    const final = await pollApi<LogsResponse>(
+      `http://${BASTION_IP}:${HTTP_PORT}/api/logs/${encodeURIComponent(vmMac)}`,
+      (d) => d.status === "installed" || d.progress === "error",
+      INSTALL_TIMEOUT_MS,
+      30_000,
+    );
+
+    if (final.progress === "error") {
+      const logs = await (await fetch(`http://${BASTION_IP}:${HTTP_PORT}/api/logs/${encodeURIComponent(vmMac)}`)).json();
+      log(`ARM install FAILED: ${JSON.stringify(logs, null, 2)}`);
+      throw new Error("ARM PXE install failed — see logs above");
+    }
+
+    vmIp = final.ip ?? "";
+    log(`ARM install complete. IP: ${vmIp}`);
+
+    await sleep(30_000);
+    rebootPxeVm(VM_NAME);
+    await sleep(5_000);
+    deleteNftablesRejectRules();
+    await waitForSsh(vmIp, SSH_USER, SSH_TIMEOUT_MS, sshKeyPath);
+  }, DISCOVERY_TIMEOUT_MS + INSTALL_TIMEOUT_MS + SSH_TIMEOUT_MS + 600_000);
+
+  afterAll(async () => { await stopHarness(VM_NAME, harness); });
+
+  it("machine reached installed state", async () => {
+    const res = await fetch(`http://${BASTION_IP}:${HTTP_PORT}/api/machines`);
+    const data = (await res.json()) as { installed: Record<string, { hostname: string }> };
+    expect(data.installed[harness!.vmMac]?.hostname).toBe(VM_NAME);
+  });
+
+  it("installed system is aarch64", () => {
+    const result = sshExec(vmIp, SSH_USER, "uname -m", { keyPath: sshKeyPath, timeout: 60_000 });
+    expect(result.stdout.trim()).toBe("aarch64");
+  });
+
+  it("SSH works with the admin user", () => {
+    const result = sshExec(vmIp, SSH_USER, "whoami", { keyPath: sshKeyPath, timeout: 60_000 });
+    expect(result.stdout.trim()).toBe(SSH_USER);
+  });
+
+  it("LVM layout is correct", () => {
+    const result = sshExec(vmIp, SSH_USER, "sudo lvs labvg --noheadings -o lv_name", {
+      keyPath: sshKeyPath, timeout: 60_000,
+    });
+    expect(result.exitCode).toBe(0);
+    const lvs = result.stdout.trim().split("\n").map((l) => l.trim());
+    for (const expected of ["root", "var", "varlog", "swap", "home", "srv"]) {
+      expect(lvs).toContain(expected);
+    }
+  });
+});

bastion/tests/integration/pxe-rescue.test.ts

@@ -0,0 +1,210 @@
+// Integration test: `labctl provision debug` -> Anaconda rescue with SSH, on x86_64.
+//
+// The rescue path had no test coverage on any architecture, which matters because it is
+// the lab's recovery tool of last resort -- the thing you reach for when a machine will
+// not boot. It runs here on x86_64 with KVM so it completes in minutes; the aarch64
+// equivalent is the same code path with a different kernel, but is emulated and far too
+// slow to iterate on.
+//
+// Run: sudo ./scripts/test-provision.sh rescue
+
+import { describe, it, expect, beforeAll, afterAll } from "vitest";
+import { readFileSync, existsSync, mkdirSync, rmSync, copyFileSync, writeFileSync } from "node:fs";
+import { execSync } from "node:child_process";
+import { join } from "node:path";
+import { homedir, tmpdir } from "node:os";
+import { log, waitForSsh } from "./helpers/libvirt.js";
+import { ensurePxeNetwork, destroyPxeNetwork, deleteNftablesRejectRules, PXE_NETWORK_NAME, PXE_GATEWAY, PXE_SUBNET } from "./helpers/pxe-network.js";
+import { createPxeVm, destroyPxeVm, getVmMac, rebootPxeVm, readSerialLog } from "./helpers/pxe-vm.js";
+import { sshExec } from "./helpers/ssh.js";
+
+const VM_NAME = "lab-pxe-rescue-test";
+const HTTP_PORT = 8094;
+const VM_MEMORY = 4096;
+const VM_VCPUS = 4;
+const VM_DISK_GB = 20;
+const BASTION_IP = PXE_GATEWAY;
+const SERIAL_PORT = 4555;
+
+const LEASE_TIMEOUT_MS = 8 * 60_000;
+const SSH_TIMEOUT_MS = 15 * 60_000;
+
+function sleep(ms: number): Promise<void> {
+  return new Promise((r) => setTimeout(r, ms));
+}
+
+function findSshKey(): { pubKey: string; keyPath: string } {
+  const candidates: string[] = [];
+  if (process.env["SSH_KEY_PATH"]) candidates.push(process.env["SSH_KEY_PATH"]);
+  const homes = [homedir()];
+  const sudoUser = process.env["SUDO_USER"];
+  if (sudoUser) homes.push(join("/home", sudoUser));
+  for (const home of homes) {
+    for (const name of ["id_ed25519", "id_ecdsa", "id_rsa"]) candidates.push(join(home, ".ssh", name));
+  }
+  for (const keyPath of candidates) {
+    if (existsSync(keyPath) && existsSync(`${keyPath}.pub`)) {
+      return { pubKey: readFileSync(`${keyPath}.pub`, "utf-8").trim(), keyPath };
+    }
+  }
+  throw new Error("No SSH key found — set SSH_KEY_PATH or ensure keys exist in ~/.ssh/");
+}
+
+function leaseIpFor(testDir: string, mac: string): string | null {
+  const leaseFile = join(testDir, "dnsmasq.leases");
+  if (!existsSync(leaseFile)) return null;
+  for (const line of readFileSync(leaseFile, "utf-8").split("\n")) {
+    const parts = line.trim().split(/\s+/);
+    if (parts.length >= 3 && parts[1]?.toLowerCase() === mac.toLowerCase()) return parts[2] ?? null;
+  }
+  return null;
+}
+
+async function waitForLease(testDir: string, mac: string, timeoutMs: number): Promise<string> {
+  const start = Date.now();
+  while (Date.now() - start < timeoutMs) {
+    const ip = leaseIpFor(testDir, mac);
+    if (ip !== null) return ip;
+    await sleep(5000);
+  }
+  throw new Error(`No DHCP lease for ${mac} within ${timeoutMs}ms`);
+}
+
+// Suite name must not be a substring of "ARM PXE rescue" -- vitest -t matches
+// substrings, so a looser name here would drag the emulated aarch64 suite in with it.
+describe("x86 rescue boot", () => {
+  let app: { close: () => Promise<void> };
+  let stopDnsmasqFn: () => void;
+  let testDir: string;
+  let vmMac: string;
+  let rescueIp: string;
+  let sshKeyPath: string;
+
+  beforeAll(async () => {
+    const { pubKey, keyPath } = findSshKey();
+    sshKeyPath = keyPath;
+
+    log("Setting up PXE test network...");
+    ensurePxeNetwork();
+
+    testDir = join(tmpdir(), `lab-pxe-rescue-${Date.now()}`);
+    for (const sub of ["tftp", "http", "logs"]) mkdirSync(join(testDir, sub), { recursive: true });
+
+    const { createApp } = await import("../../src/bastion/src/server.js");
+    const { loadConfig } = await import("../../src/bastion/src/config.js");
+    const { generateDnsmasqConf, startDnsmasq, stopDnsmasq } = await import("../../src/bastion/src/services/dnsmasq.js");
+    const { renderBootIpxe, kernelPath, initrdPath } = await import("../../src/bastion/src/templates/boot.ipxe.js");
+    stopDnsmasqFn = stopDnsmasq;
+
+    const config = loadConfig({
+      bastionDir: testDir,
+      httpPort: HTTP_PORT,
+      iface: "virbr-pxe",
+      serverIp: BASTION_IP,
+      network: `${PXE_SUBNET}.0`,
+      gateway: BASTION_IP,
+      dhcpMode: "full",
+      dhcpRangeStart: `${PXE_SUBNET}.100`,
+      dhcpRangeEnd: `${PXE_SUBNET}.200`,
+      domain: "rescue-test.local",
+      sshKeys: [pubKey],
+      adminUser: "lab",
+    });
+
+    // iPXE in both dirs: TFTP PXE and UEFI HTTP Boot are both possible, and OVMF picks.
+    const ipxeX86 = "/usr/share/ipxe/ipxe-snponly-x86_64.efi";
+    if (!existsSync(ipxeX86)) throw new Error(`iPXE not found: ${ipxeX86}`);
+    copyFileSync(ipxeX86, join(config.tftpDir, "ipxe.efi"));
+    copyFileSync(ipxeX86, join(config.httpDir, "ipxe.efi"));
+
+    const cacheDir = "/var/lib/libvirt/images/lab-pxe-cache";
+    execSync(`mkdir -p "${cacheDir}"`, { stdio: "pipe" });
+    const kernelCache = join(cacheDir, "vmlinuz-x86_64");
+    const initrdCache = join(cacheDir, "initrd-x86_64.img");
+    if (!existsSync(kernelCache)) {
+      log("Downloading Fedora x86_64 kernel...");
+      execSync(`curl -# -L -f -o "${kernelCache}" "${config.fedoraMirror}/images/pxeboot/vmlinuz"`, { stdio: "inherit", timeout: 600_000 });
+    }
+    if (!existsSync(initrdCache)) {
+      log("Downloading Fedora x86_64 initrd...");
+      execSync(`curl -# -L -f -o "${initrdCache}" "${config.fedoraMirror}/images/pxeboot/initrd.img"`, { stdio: "inherit", timeout: 600_000 });
+    }
+    copyFileSync(kernelCache, join(config.httpDir, kernelPath("x86_64")));
+    copyFileSync(initrdCache, join(config.httpDir, initrdPath("x86_64")));
+
+    writeFileSync(join(config.httpDir, "boot.ipxe"), renderBootIpxe({ serverIp: config.serverIp, httpPort: config.httpPort }));
+    generateDnsmasqConf(config);
+
+    const { app: fastify, state, syslog } = createApp(config);
+    app = fastify;
+    await fastify.listen({ port: config.httpPort, host: "0.0.0.0" });
+    syslog.start();
+    log(`Bastion HTTP listening on :${HTTP_PORT}`);
+
+    startDnsmasq(config).catch((err) => log(`dnsmasq failed: ${err instanceof Error ? err.message : String(err)}`));
+    await sleep(1500);
+
+    log("Creating x86_64 PXE VM (KVM)...");
+    createPxeVm({ name: VM_NAME, memory: VM_MEMORY, vcpus: VM_VCPUS, diskSize: VM_DISK_GB, network: PXE_NETWORK_NAME });
+    const mac = getVmMac(VM_NAME);
+    if (!mac) throw new Error("Could not determine VM MAC");
+    vmMac = mac;
+    log(`VM MAC: ${vmMac}`);
+
+    // Queue rescue directly, as `labctl provision debug` does.
+    log("Queueing debug/rescue mode...");
+    state.update((s) => {
+      s.debug[vmMac] = { hostname: "rescue-test", queued_at: new Date().toISOString() };
+    });
+
+    rebootPxeVm(VM_NAME);
+    await sleep(5_000);
+    deleteNftablesRejectRules();
+
+    rescueIp = await waitForLease(testDir, vmMac, LEASE_TIMEOUT_MS);
+    log(`Rescue IP: ${rescueIp}`);
+
+    log("Waiting for SSH into the rescue shell (inst.sshd)...");
+    await waitForSsh(rescueIp, "root", SSH_TIMEOUT_MS, sshKeyPath).catch(async (err) => {
+      log("Rescue SSH timed out. Serial console:");
+      try { log(await readSerialLog(SERIAL_PORT, { lastLines: 120, timeoutMs: 20_000 })); } catch { /* none */ }
+      throw err;
+    });
+    log("Rescue shell reachable.");
+  }, LEASE_TIMEOUT_MS + SSH_TIMEOUT_MS + 300_000);
+
+  afterAll(async () => {
+    if (process.env["KEEP_VM"] === "1") {
+      log(`KEEP_VM=1 — leaving ${VM_NAME} up (serial: socat - TCP:127.0.0.1:${SERIAL_PORT})`);
+      return;
+    }
+    log("Cleaning up...");
+    if (app) await app.close().catch(() => {});
+    if (stopDnsmasqFn) stopDnsmasqFn();
+    destroyPxeVm(VM_NAME);
+    destroyPxeNetwork();
+    if (testDir) rmSync(testDir, { recursive: true, force: true });
+  });
+
+  it("rescue shell is reachable over SSH as root", () => {
+    const result = sshExec(rescueIp, "root", "whoami", { keyPath: sshKeyPath, timeout: 60_000 });
+    expect(result.exitCode).toBe(0);
+    expect(result.stdout.trim()).toBe("root");
+  });
+
+  it("is the Anaconda rescue environment", () => {
+    const result = sshExec(rescueIp, "root", "cat /proc/cmdline", { keyPath: sshKeyPath, timeout: 60_000 });
+    expect(result.stdout).toContain("inst.rescue");
+    expect(result.stdout).toContain("inst.sshd");
+  });
+
+  it("kernel and initrd came from the bastion", () => {
+    const result = sshExec(rescueIp, "root", "cat /proc/cmdline", { keyPath: sshKeyPath, timeout: 60_000 });
+    expect(result.stdout).toContain(`${BASTION_IP}:${HTTP_PORT}`);
+  });
+
+  it("has LVM tools for inspecting an installed system", () => {
+    const result = sshExec(rescueIp, "root", "command -v vgchange && command -v lsblk", { keyPath: sshKeyPath, timeout: 60_000 });
+    expect(result.exitCode).toBe(0);
+  });
+});

bastion/tsconfig.json

@@ -1,6 +1,7 @@
 {
   "files": [],
   "references": [
+    { "path": "src/core" },
     { "path": "src/shared" },
     { "path": "src/bastion" },
     { "path": "src/cli" },