michal/lab
1
0
Fork 0
Code Issues Pull Requests 3 Actions Packages Projects Releases Wiki Activity

feat(bastion): first-class aarch64 support in the network-PXE path #17

Open
michal wants to merge 11 commits from feat/arm64-pxe-support into main
pull from: feat/arm64-pxe-support
merge into: michal:main
Conversation 0 Commits 11 Files Changed 25 +2364 -94
25 changed files with 2364 additions and 94 deletions
Download Patch File Download Diff File Expand all files Collapse all files

107
bastion/docs/ARCHITECTURE.md
View File

@@ -89,6 +89,83 @@ Side paths:
--- ---
## Multi-architecture PXE
The bastion serves both `x86_64` and `aarch64` over the network. Nothing about this is
operator-configured -- there is no `--arch` flag, by design.
### How a client's architecture is decided
1. **DHCP option 93** (Client System Architecture) picks the *bootloader*. dnsmasq matches
it and hands out a matching iPXE binary:
| Option 93 | Client | Served |
|---|---|---|
| `0` | x86 BIOS | `undionly.kpxe` (TFTP) |
| `7`, `9` | x64 UEFI | `ipxe.efi` (TFTP) |
| `11` | **ARM64 UEFI** | `ipxe-arm64.efi` (TFTP) |
| `16` | x64 UEFI HTTP Boot | `http://…/ipxe.efi` |
| `19` | **ARM64 UEFI HTTP Boot** | `http://…/ipxe-arm64.efi` |
Values come from the IANA Processor Architecture Types registry. Note `19`, not `20` --
`20` is *pc/at bios boot from http*. EDK2/AAVMF prefers HTTP Boot over TFTP PXE, so the
iPXE binaries are staged in **both** `tftpDir` and `httpDir` (symlinked by `main.ts`).
2. **`/dispatch` picks the kernel.** Option 93 never reaches the HTTP endpoint, so
`boot.ipxe` passes iPXE's own `${buildarch}` as `?arch=`. `resolveArch()` prefers, in
order: the tracked machine record → the reported `?arch=` → the configured default.
The record wins because it is what we observed on the machine itself.
### Artifact naming
`x86_64` keeps the original unsuffixed paths so its rendered iPXE scripts are unchanged;
everything else is suffixed. `kernelPath()` / `initrdPath()` in `templates/boot.ipxe.ts`
are the single source of truth, used by both the templates and `main.ts` staging.
| arch | kernel | initrd |
|---|---|---|
| `x86_64` | `/vmlinuz` | `/initrd.img` |
| `aarch64` | `/vmlinuz-aarch64` | `/initrd-aarch64.img` |
`tests/ipxe-x86-regression.test.ts` pins the x86_64 output against a golden fixture.
### arm64 gotchas
- **LoadFile2 is mandatory.** arm64 has no `HdrS` boot protocol; the kernel's EFI stub
fetches the initrd over the UEFI `EFI_LOAD_FILE2_PROTOCOL`. An iPXE build without it
accepts the `initrd` line, silently drops it, and the kernel panics with
`VFS: Unable to mount root fs on unknown-block(0,0)`. Fedora's
`ipxe-bootimgs-aarch64` implements it; the integration test asserts this up front so
the failure names itself instead of looking like a disk problem.
- **`nomodeset` is x86-only.** On arm64 there is no VGA path to fall back to. aarch64 gets
`console=tty0 console=ttyAMA0,115200` instead — the last `console=` wins for
`/dev/console`, so serial is the interactive one.
- **Ubuntu is x86_64-only.** `releases.ubuntu.com` publishes no arm64 netboot artifacts.
`osSupportsArch()` encodes this, and both the install guard and `/dispatch` refuse the
combination rather than serving an x86 kernel to an ARM machine.
---
## Onboarding classification (vendor OS)
Machines carry an `onboard` field: `"pxe"` (default) or `"ssh"`, plus `vendor_os` naming
what they run. `classifyOnboard()` in `@lab/shared` sets it from DMI identity, with known
hardware also matched by MAC — a machine can sit in state for a long time with no DMI, and
a DMI-only rule would fail open exactly where it matters.
`onboard: "ssh"` means *we cannot rebuild this machine's OS*. Installs are refused at both
entry points (`/api/install` and the labd `command-install` handler) with an error naming
the machine and pointing at `provision debug`. **Rescue is never guarded** — being unable
to reinstall a machine is precisely when a rescue shell is needed.
This is a fact about the machine, not a blocklist. The refusal follows from "no image in
our pipeline restores `vendor_os`", so adding a DGX OS image to the pipeline is what
unblocks the DGX Sparks — no entry needs deleting.
Current classifications: NVIDIA DGX Spark (`spark-2935`, `spark-3a1c`) → `dgx-os`.
---
## Packages ## Packages
### Monorepo Structure ### Monorepo Structure
@@ -404,6 +481,36 @@ Hardcoded `/dev/sda` default broke NVMe-only machines. Fix: default to empty str
### Anaconda Rescue Mode Limitations ### Anaconda Rescue Mode Limitations
`%pre` and `%post` sections do not execute in `inst.rescue` mode. SSH in rescue mode is provided by Anaconda's `inst.sshd` kernel parameter + `sshpw` kickstart directive. Manual setup via `curl bastion:8080/debug-setup.sh | bash` for nc listener. `%pre` and `%post` sections do not execute in `inst.rescue` mode. SSH in rescue mode is provided by Anaconda's `inst.sshd` kernel parameter + `sshpw` kickstart directive. Manual setup via `curl bastion:8080/debug-setup.sh | bash` for nc listener.
**Unresolved (2026-08-11): rescue SSH has never been observed working.** Adding the first
integration coverage for `provision debug` (`tests/integration/pxe-rescue.test.ts`) showed the
rescue environment coming up correctly — the bastion serves the kernel and initrd, Anaconda
boots, fetches `debug.ks`, and reaches its installer environment — but **nothing ever listens on
port 22**.
Strength of the evidence, stated precisely because it decides where to look next:
- **aarch64 — direct.** Port 22 probed every 20s for 30 minutes while the Anaconda installer
environment was demonstrably running (NetworkManager, polkitd, rsyslog on the console). Never
opened.
- **x86_64 — corroborating, not conclusive.** One clean KVM run (943s) where SSH never became
available inside a 15-minute budget. That VM's progress into the rescue environment was *not*
observed — vitest's final reporter discards the streamed log — so it is consistent with the
aarch64 result but does not independently prove it. Re-run with `KEEP_VM=1` and probe port 22
directly to settle it.
If the x86_64 result holds up, this is orthogonal to the multi-architecture work, since x86_64 is
untouched by it. Leads worth checking, in order:
- Does `inst.sshd` actually start `sshd` in `inst.rescue` mode, or only in install mode? The
port never opens, so this is the prime suspect — an auth problem would still show an open port.
- `sshkey` may apply only to the *installed* system, leaving the installer environment
password-only via `sshpw`. That would matter once sshd does listen: the test authenticates
key-only (`BatchMode=yes`).
- The `%anaconda`-context directives in `debug.ks` may be skipped entirely when a kickstart is
supplied alongside `inst.rescue`.
Until this is resolved, `provision debug` gets you a booted rescue environment on the console
(including on arm64), but not an SSH shell. The `debug-setup.sh` nc-listener path is the
documented workaround and is unaffected.
--- ---
## Planned Work (Taskmaster) ## Planned Work (Taskmaster)

6
bastion/package.json
View File

@@ -23,6 +23,12 @@
"test:integration:iso:host": "sudo -E $(which npx) vitest run -c tests/integration/vitest.config.ts -t 'ISO boot'", "test:integration:iso:host": "sudo -E $(which npx) vitest run -c tests/integration/vitest.config.ts -t 'ISO boot'",
"test:integration:arm-iso": "vitest run -c tests/integration/vitest.config.ts -t 'ARM ISO'", "test:integration:arm-iso": "vitest run -c tests/integration/vitest.config.ts -t 'ARM ISO'",
"test:integration:arm-iso:host": "sudo -E $(which npx) vitest run -c tests/integration/vitest.config.ts -t 'ARM ISO'", "test:integration:arm-iso:host": "sudo -E $(which npx) vitest run -c tests/integration/vitest.config.ts -t 'ARM ISO'",
"test:integration:rescue": "vitest run -c tests/integration/vitest.config.ts -t 'x86 rescue boot'",
"test:integration:rescue:host": "sudo -E $(which npx) vitest run -c tests/integration/vitest.config.ts -t 'x86 rescue boot'",
"test:integration:arm-pxe": "vitest run -c tests/integration/vitest.config.ts -t 'ARM PXE rescue'",
"test:integration:arm-pxe:host": "sudo -E $(which npx) vitest run -c tests/integration/vitest.config.ts -t 'ARM PXE rescue'",
"test:integration:arm-pxe-full": "ARM_PXE_FULL=1 vitest run -c tests/integration/vitest.config.ts -t 'ARM PXE'",
"test:integration:arm-pxe-full:host": "sudo -E ARM_PXE_FULL=1 $(which npx) vitest run -c tests/integration/vitest.config.ts -t 'ARM PXE'",
"test:integration:asahi": "vitest run -c tests/integration/vitest.config.ts -t 'asahi firstboot'", "test:integration:asahi": "vitest run -c tests/integration/vitest.config.ts -t 'asahi firstboot'",
"test:integration:asahi:host": "sudo -E $(which npx) vitest run -c tests/integration/vitest.config.ts -t 'asahi firstboot'", "test:integration:asahi:host": "sudo -E $(which npx) vitest run -c tests/integration/vitest.config.ts -t 'asahi firstboot'",
"test:integration:asahi-validate": "vitest run -c tests/integration/vitest.config.ts -t 'asahi.*validation'", "test:integration:asahi-validate": "vitest run -c tests/integration/vitest.config.ts -t 'asahi.*validation'",

76
bastion/scripts/test-provision.sh
View File

@@ -2,16 +2,19 @@
# Run PXE and/or ISO boot integration tests. # Run PXE and/or ISO boot integration tests.
# #
# Usage: # Usage:
# sudo ./scripts/test-provision.sh # run PXE + ISO (x86_64) # sudo ./scripts/test-provision.sh # run PXE + ISO (x86_64)
# sudo ./scripts/test-provision.sh pxe # PXE only # sudo ./scripts/test-provision.sh pxe # PXE only
# sudo ./scripts/test-provision.sh iso # ISO only (x86_64) # sudo ./scripts/test-provision.sh iso # ISO only (x86_64)
# sudo ./scripts/test-provision.sh arm # ARM ISO boot (emulated, SLOW ~60min) # sudo ./scripts/test-provision.sh rescue # x86_64 Anaconda rescue boot + SSH (~15min)
# sudo ./scripts/test-provision.sh all # all tests including ARM # sudo ./scripts/test-provision.sh arm # ARM ISO boot (emulated, SLOW ~60min)
# sudo ./scripts/test-provision.sh arm-pxe # ARM network PXE rescue: NBP + rescue over SSH (~25-30min)
# sudo ./scripts/test-provision.sh arm-pxe-full # ARM network PXE incl. discover + full install (~75-95min)
# sudo ./scripts/test-provision.sh all # all tests including ARM
# #
# Prerequisites: # Prerequisites:
# libvirtd, OVMF (edk2-ovmf), iPXE (ipxe-bootimgs-x86), # libvirtd, OVMF (edk2-ovmf), iPXE (ipxe-bootimgs-x86),
# dnsmasq, xorriso, mtools, virt-install, qemu-img # dnsmasq, xorriso, mtools, virt-install, qemu-img
# ARM: qemu-system-aarch64, edk2-aarch64 # ARM: qemu-system-aarch64, edk2-aarch64, ipxe-bootimgs-aarch64
set -e set -e
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
@@ -58,6 +61,10 @@ if [ ! -f /usr/share/edk2/ovmf/OVMF_CODE.fd ]; then
exit 1 exit 1
fi fi
MODE="${1:-both}"
# iPXE binaries are per-architecture. x86_64 is always required (the dnsmasq config
# references it); arm64 only for the ARM network-PXE modes.
IPXE_EFI="" IPXE_EFI=""
for f in /usr/share/ipxe/ipxe-snponly-x86_64.efi /usr/share/ipxe/ipxe-snp-x86_64.efi /usr/share/ipxe/ipxe-x86_64.efi; do for f in /usr/share/ipxe/ipxe-snponly-x86_64.efi /usr/share/ipxe/ipxe-snp-x86_64.efi /usr/share/ipxe/ipxe-x86_64.efi; do
[ -f "$f" ] && IPXE_EFI="$f" && break [ -f "$f" ] && IPXE_EFI="$f" && break
@@ -67,6 +74,20 @@ if [ -z "$IPXE_EFI" ]; then
exit 1 exit 1
fi fi
IPXE_EFI_ARM64=""
for f in /usr/share/ipxe/arm64-efi/snponly.efi /usr/share/ipxe/arm64-efi/ipxe.efi; do
[ -f "$f" ] && IPXE_EFI_ARM64="$f" && break
done
case "$MODE" in
arm-pxe|arm-pxe-full|all)
if [ -z "$IPXE_EFI_ARM64" ] && [ "$MODE" != "all" ]; then
echo -e "${RED}arm64 iPXE binary not found.${RESET} Install: sudo dnf install ipxe-bootimgs-aarch64"
exit 1
fi
;;
esac
# Find SSH key # Find SSH key
SSH_KEY="" SSH_KEY=""
for name in id_ed25519 id_ecdsa id_rsa; do for name in id_ed25519 id_ecdsa id_rsa; do
@@ -83,10 +104,19 @@ fi
echo -e " User: ${BOLD}$REAL_USER${RESET}" echo -e " User: ${BOLD}$REAL_USER${RESET}"
echo -e " SSH key: ${BOLD}$SSH_KEY${RESET}" echo -e " SSH key: ${BOLD}$SSH_KEY${RESET}"
echo -e " iPXE: ${BOLD}$IPXE_EFI${RESET}" echo -e " iPXE: ${BOLD}$IPXE_EFI${RESET}"
echo -e " iPXE a64:${BOLD} ${IPXE_EFI_ARM64:-not installed}${RESET}"
echo "" echo ""
# --- Determine which tests to run --- require_arm_emulation() {
MODE="${1:-both}" if ! command -v qemu-system-aarch64 &>/dev/null; then
echo -e "${RED}qemu-system-aarch64 not found.${RESET} Install: sudo dnf install qemu-system-aarch64 edk2-aarch64"
exit 1
fi
if [ ! -f /usr/share/edk2/aarch64/QEMU_EFI.fd ]; then
echo -e "${RED}AAVMF firmware not found.${RESET} Install: sudo dnf install edk2-aarch64"
exit 1
fi
}
run_test() { run_test() {
local name="$1" pattern="$2" local name="$1" pattern="$2"
@@ -116,13 +146,26 @@ case "$MODE" in
run_test "ISO boot" "ISO boot" || FAILED=1 run_test "ISO boot" "ISO boot" || FAILED=1
;; ;;
arm|arm-iso) arm|arm-iso)
if ! command -v qemu-system-aarch64 &>/dev/null; then require_arm_emulation
echo -e "${RED}qemu-system-aarch64 not found.${RESET} Install: sudo dnf install qemu-system-aarch64 edk2-aarch64"
exit 1
fi
echo -e "${YELLOW}ARM emulation is ~10x slower than native. Expect 30-60 minutes.${RESET}" echo -e "${YELLOW}ARM emulation is ~10x slower than native. Expect 30-60 minutes.${RESET}"
run_test "ARM ISO boot" "ARM ISO" || FAILED=1 run_test "ARM ISO boot" "ARM ISO" || FAILED=1
;; ;;
rescue)
echo -e "${YELLOW}x86_64 rescue boot (KVM). Expect ~15 minutes.${RESET}"
run_test "x86 rescue boot" "x86 rescue boot" || FAILED=1
;;
arm-pxe)
require_arm_emulation
echo -e "${YELLOW}ARM emulation is ~10x slower than native. Expect 25-30 minutes.${RESET}"
echo -e "${YELLOW}Covers option 93 -> arm64 NBP, arch resolution, and rescue over SSH.${RESET}"
echo -e "${YELLOW}For the full install too, use: $0 arm-pxe-full${RESET}"
run_test "ARM PXE rescue" "ARM PXE rescue" || FAILED=1
;;
arm-pxe-full)
require_arm_emulation
echo -e "${YELLOW}ARM emulation is ~10x slower than native. Expect 75-95 minutes.${RESET}"
ARM_PXE_FULL=1 run_test "ARM PXE (rescue + install)" "ARM PXE" || FAILED=1
;;
both) both)
run_test "PXE boot" "PXE boot" || FAILED=1 run_test "PXE boot" "PXE boot" || FAILED=1
run_test "ISO boot" "ISO boot" || FAILED=1 run_test "ISO boot" "ISO boot" || FAILED=1
@@ -133,12 +176,17 @@ case "$MODE" in
if command -v qemu-system-aarch64 &>/dev/null; then if command -v qemu-system-aarch64 &>/dev/null; then
echo -e "${YELLOW}ARM emulation is ~10x slower than native.${RESET}" echo -e "${YELLOW}ARM emulation is ~10x slower than native.${RESET}"
run_test "ARM ISO boot" "ARM ISO" || FAILED=1 run_test "ARM ISO boot" "ARM ISO" || FAILED=1
if [ -n "$IPXE_EFI_ARM64" ]; then
run_test "ARM PXE rescue" "ARM PXE rescue" || FAILED=1
else
echo -e "${YELLOW}Skipping ARM PXE test (ipxe-bootimgs-aarch64 not installed)${RESET}"
fi
else else
echo -e "${YELLOW}Skipping ARM test (qemu-system-aarch64 not installed)${RESET}" echo -e "${YELLOW}Skipping ARM tests (qemu-system-aarch64 not installed)${RESET}"
fi fi
;; ;;
*) *)
echo "Usage: $0 [pxe|iso|arm|both|all]" echo "Usage: $0 [pxe|iso|rescue|arm|arm-pxe|arm-pxe-full|both|all]"
exit 1 exit 1
;; ;;
esac esac

110
bastion/src/bastion/src/main.ts
View File

@@ -3,7 +3,9 @@
import { mkdirSync, writeFileSync, readFileSync, existsSync, copyFileSync, symlinkSync, unlinkSync } from "node:fs"; import { mkdirSync, writeFileSync, readFileSync, existsSync, copyFileSync, symlinkSync, unlinkSync } from "node:fs";
import { execSync } from "node:child_process"; import { execSync } from "node:child_process";
import type { BastionConfig } from "@lab/shared"; import type { Arch, BastionConfig } from "@lab/shared";
import { SUPPORTED_ARCHES, fedoraMirrorFor, classifyOnboard } from "@lab/shared";
import { kernelPath, initrdPath } from "./templates/boot.ipxe.js";
import { loadConfig } from "./config.js"; import { loadConfig } from "./config.js";
import { populateNetworkConfig } from "./services/network.js"; import { populateNetworkConfig } from "./services/network.js";
import { createApp } from "./server.js"; import { createApp } from "./server.js";
@@ -13,6 +15,7 @@ import { renderBootIpxe } from "./templates/boot.ipxe.js";
import { logger } from "./services/logger.js"; import { logger } from "./services/logger.js";
import { BastionConnection } from "./services/labd-connection.js"; import { BastionConnection } from "./services/labd-connection.js";
import { progressBus } from "./services/progress-events.js"; import { progressBus } from "./services/progress-events.js";
import { checkInstallAllowed } from "./services/install-guard.js";
import { ensureBootIso } from "./routes/boot-iso.js"; import { ensureBootIso } from "./routes/boot-iso.js";
function copyIfMissing(src: string, dest: string, label: string): void { function copyIfMissing(src: string, dest: string, label: string): void {
@@ -130,9 +133,14 @@ export async function startBastion(overrides: Partial<BastionConfig> = {}): Prom
mkdirSync(config.tftpDir, { recursive: true }); mkdirSync(config.tftpDir, { recursive: true });
mkdirSync(config.httpDir, { recursive: true }); mkdirSync(config.httpDir, { recursive: true });
// Architectures we can actually network boot, reported in the banner so a missing
// arm64 payload is visible at startup instead of at 2am when a rescue is needed.
const bootArches: Arch[] = [];
let ipxeArm64Ready = false;
// Prepare boot artifacts // Prepare boot artifacts
if (config.skipArtifacts !== true) { if (config.skipArtifacts !== true) {
logger.info(`Preparing boot artifacts (Fedora ${config.fedoraVersion} ${config.arch})...`); logger.info(`Preparing boot artifacts (Fedora ${config.fedoraVersion}, ${SUPPORTED_ARCHES.join(" + ")})...`);
copyIfMissing( copyIfMissing(
"/usr/share/ipxe/undionly.kpxe", "/usr/share/ipxe/undionly.kpxe",
@@ -150,20 +158,41 @@ export async function startBastion(overrides: Partial<BastionConfig> = {}): Prom
`${config.tftpDir}/ipxe-arm64.efi`, `${config.tftpDir}/ipxe-arm64.efi`,
"iPXE UEFI arm64", "iPXE UEFI arm64",
); );
ipxeArm64Ready = true;
} catch { } catch {
logger.warn("arm64 iPXE not available -- skipping"); logger.warn("arm64 iPXE not available -- arm64 machines cannot network boot.");
logger.warn(" Install with: sudo dnf install ipxe-bootimgs-aarch64");
} }
download( // Fedora pxeboot kernel + initrd per architecture. x86_64 keeps the unsuffixed
`${config.fedoraMirror}/images/pxeboot/vmlinuz`, // names it has always used; other architectures are suffixed. The iPXE templates
`${config.httpDir}/vmlinuz`, // resolve the same paths via kernelPath()/initrdPath().
"Fedora kernel", for (const arch of SUPPORTED_ARCHES) {
); const mirror = fedoraMirrorFor(config.fedoraVersion, arch);
download( try {
`${config.fedoraMirror}/images/pxeboot/initrd.img`, download(
`${config.httpDir}/initrd.img`, `${mirror}/images/pxeboot/vmlinuz`,
"Fedora initrd", `${config.httpDir}${kernelPath(arch)}`,
); `Fedora ${arch} kernel`,
);
download(
`${mirror}/images/pxeboot/initrd.img`,
`${config.httpDir}${initrdPath(arch)}`,
`Fedora ${arch} initrd`,
);
bootArches.push(arch);
} catch (err) {
// Non-fatal: a bastion with no arm64 artifacts still serves x86_64 fine.
// Failing startup over an unreachable mirror for an architecture that may not
// even be present on this network would be worse.
logger.warn(`Fedora ${arch} kernel/initrd unavailable -- ${arch} PXE disabled`);
logger.warn(` ${err instanceof Error ? err.message : String(err)}`);
}
}
if (!bootArches.includes("x86_64")) {
throw new Error("Fedora x86_64 kernel/initrd could not be staged -- cannot serve PXE");
}
// Ubuntu netboot artifacts (non-fatal — Ubuntu version may not be released yet) // Ubuntu netboot artifacts (non-fatal — Ubuntu version may not be released yet)
try { try {
@@ -254,6 +283,13 @@ export async function startBastion(overrides: Partial<BastionConfig> = {}): Prom
// Wire up command handlers so labd can send install/forget/role commands // Wire up command handlers so labd can send install/forget/role commands
labdConn.onCommand("command-install", async (msg) => { labdConn.onCommand("command-install", async (msg) => {
if (msg.type !== "command-install") throw new Error("unexpected"); if (msg.type !== "command-install") throw new Error("unexpected");
const installMac = msg.mac.toLowerCase().replace(/-/g, ":");
const osId = (msg.os as import("@lab/shared").OsId | undefined) ?? "fedora-43";
const check = checkInstallAllowed(state.load(), installMac, osId);
if (check.allowed === false) {
logger.warn(`INSTALL REFUSED: ${installMac} -- ${check.error}`);
return { status: "error", error: check.error };
}
state.update((s) => { state.update((s) => {
s.install_queue[msg.mac] = { s.install_queue[msg.mac] = {
hostname: msg.hostname, hostname: msg.hostname,
@@ -314,13 +350,24 @@ export async function startBastion(overrides: Partial<BastionConfig> = {}): Prom
const mac = (msg.mac as string).toLowerCase(); const mac = (msg.mac as string).toLowerCase();
const now = new Date().toISOString(); const now = new Date().toISOString();
const existing = state.load().discovered[mac]; const existing = state.load().discovered[mac];
const identity = {
mac,
manufacturer: (msg.manufacturer as string) ?? "unknown",
product: (msg.product as string) ?? "unknown",
board: (msg.board as string) ?? "unknown",
...(existing?.onboard !== undefined ? { onboard: existing.onboard } : {}),
...(existing?.vendor_os !== undefined ? { vendor_os: existing.vendor_os } : {}),
};
const onboarding = classifyOnboard(identity);
const rootDevice = msg.root_device ?? existing?.root_device;
const rootArgs = msg.root_args ?? existing?.root_args;
state.update((s) => { state.update((s) => {
s.discovered[mac] = { s.discovered[mac] = {
mac, mac,
product: (msg.product as string) ?? "unknown", product: identity.product,
board: (msg.board as string) ?? "unknown", board: identity.board,
serial: (msg.serial as string) ?? "unknown", serial: (msg.serial as string) ?? "unknown",
manufacturer: (msg.manufacturer as string) ?? "unknown", manufacturer: identity.manufacturer,
cpu_model: (msg.cpu_model as string) ?? "unknown", cpu_model: (msg.cpu_model as string) ?? "unknown",
cpu_cores: (msg.cpu_cores as number) ?? 0, cpu_cores: (msg.cpu_cores as number) ?? 0,
memory_gb: (msg.memory_gb as number) ?? 0, memory_gb: (msg.memory_gb as number) ?? 0,
@@ -329,7 +376,20 @@ export async function startBastion(overrides: Partial<BastionConfig> = {}): Prom
nics: (msg.nics as Array<{ name: string; mac: string; state: string }>) ?? [], nics: (msg.nics as Array<{ name: string; mac: string; state: string }>) ?? [],
first_seen: existing?.first_seen ?? now, first_seen: existing?.first_seen ?? now,
last_seen: now, last_seen: now,
onboard: onboarding.onboard,
...(onboarding.vendor_os !== undefined ? { vendor_os: onboarding.vendor_os } : {}),
...(rootDevice !== undefined ? { root_device: rootDevice } : {}),
...(rootArgs !== undefined ? { root_args: rootArgs } : {}),
}; };
// Keep the installed record in step -- the guard and --pxe-boot both read it.
const inst = s.installed[mac];
if (inst) {
inst.arch = (msg.arch as string) ?? inst.arch;
inst.onboard = onboarding.onboard;
if (onboarding.vendor_os !== undefined) inst.vendor_os = onboarding.vendor_os;
if (rootDevice !== undefined) inst.root_device = rootDevice;
if (rootArgs !== undefined) inst.root_args = rootArgs;
}
}); });
logger.info(`HARDWARE UPDATED: ${mac} -- ${msg.manufacturer ?? "?"} ${msg.product ?? "?"} (${msg.cpu_model ?? "?"}, ${msg.cpu_cores ?? "?"} cores, ${msg.memory_gb ?? "?"}GB RAM)`); logger.info(`HARDWARE UPDATED: ${mac} -- ${msg.manufacturer ?? "?"} ${msg.product ?? "?"} (${msg.cpu_model ?? "?"}, ${msg.cpu_cores ?? "?"} cores, ${msg.memory_gb ?? "?"}GB RAM)`);
return { status: "ok", data: { mac } }; return { status: "ok", data: { mac } };
@@ -364,7 +424,7 @@ export async function startBastion(overrides: Partial<BastionConfig> = {}): Prom
} }
// Print banner // Print banner
printBanner(config); printBanner(config, bootArches, ipxeArm64Ready);
// Graceful shutdown // Graceful shutdown
const shutdown = async (): Promise<void> => { const shutdown = async (): Promise<void> => {
@@ -386,11 +446,22 @@ export async function startBastion(overrides: Partial<BastionConfig> = {}): Prom
await new Promise(() => {}); await new Promise(() => {});
} }
function printBanner(config: BastionConfig): void { function printBanner(config: BastionConfig, bootArches: Arch[], ipxeArm64Ready: boolean): void {
const dhcpInfo = config.dhcpMode === "full" const dhcpInfo = config.dhcpMode === "full"
? `full (${config.dhcpRangeStart}-${config.dhcpRangeEnd})` ? `full (${config.dhcpRangeStart}-${config.dhcpRangeEnd})`
: "proxy (alongside existing DHCP)"; : "proxy (alongside existing DHCP)";
// arm64 needs both an iPXE binary (DHCP hands it out on option 93 = 0x0b) and a
// kernel/initrd pair. Report the combination, since either missing breaks it.
const archInfo = config.skipArtifacts === true
? "(artifacts skipped)"
: SUPPORTED_ARCHES
.map((a) => {
const ready = bootArches.includes(a) && (a !== "aarch64" || ipxeArm64Ready);
return ready ? a : `${a} (unavailable)`;
})
.join(", ");
console.log(""); console.log("");
console.log("\x1b[36m\x1b[1m" + "=".repeat(60) + "\x1b[0m"); console.log("\x1b[36m\x1b[1m" + "=".repeat(60) + "\x1b[0m");
console.log("\x1b[36m\x1b[1m Lab PXE Bastion -- Discovery Mode\x1b[0m"); console.log("\x1b[36m\x1b[1m Lab PXE Bastion -- Discovery Mode\x1b[0m");
@@ -399,7 +470,8 @@ function printBanner(config: BastionConfig): void {
console.log(` Network: \x1b[1m${config.network}/24\x1b[0m via \x1b[1m${config.iface}\x1b[0m`); console.log(` Network: \x1b[1m${config.network}/24\x1b[0m via \x1b[1m${config.iface}\x1b[0m`);
console.log(` DHCP: \x1b[1m${dhcpInfo}\x1b[0m`); console.log(` DHCP: \x1b[1m${dhcpInfo}\x1b[0m`);
console.log(` HTTP: \x1b[1mhttp://${config.serverIp}:${config.httpPort}/\x1b[0m`); console.log(` HTTP: \x1b[1mhttp://${config.serverIp}:${config.httpPort}/\x1b[0m`);
console.log(` OS: \x1b[1mFedora ${config.fedoraVersion} (${config.arch})\x1b[0m`); console.log(` OS: \x1b[1mFedora ${config.fedoraVersion}\x1b[0m`);
console.log(` Net boot: \x1b[1m${archInfo}\x1b[0m`);
console.log(` Domain: \x1b[1m${config.domain}\x1b[0m`); console.log(` Domain: \x1b[1m${config.domain}\x1b[0m`);
console.log(` State: \x1b[1m${config.stateFile}\x1b[0m`); console.log(` State: \x1b[1m${config.stateFile}\x1b[0m`);
console.log(""); console.log("");

64
bastion/src/bastion/src/routes/api.ts
View File

@@ -6,10 +6,11 @@
import type { FastifyInstance } from "fastify"; import type { FastifyInstance } from "fastify";
import type { HardwareInfo, InstalledInfo, Role } from "@lab/shared"; import type { HardwareInfo, InstalledInfo, Role } from "@lab/shared";
import { isValidOsId, SUPPORTED_ROLES } from "@lab/shared"; import { isValidOsId, SUPPORTED_ROLES, classifyOnboard } from "@lab/shared";
import type { StateManager } from "../services/state.js"; import type { StateManager } from "../services/state.js";
import { logger } from "../services/logger.js"; import { logger } from "../services/logger.js";
import { triggerPostProvisionK3s } from "../services/post-provision.js"; import { triggerPostProvisionK3s } from "../services/post-provision.js";
import { checkInstallAllowed } from "../services/install-guard.js";
import { progressBus } from "../services/progress-events.js"; import { progressBus } from "../services/progress-events.js";
import type { ProgressEvent } from "../services/progress-events.js"; import type { ProgressEvent } from "../services/progress-events.js";
import type { InstallLogBuffer } from "../services/install-log.js"; import type { InstallLogBuffer } from "../services/install-log.js";
@@ -53,6 +54,12 @@ export function registerApiRoutes(
return reply.status(400).send({ error: `invalid os: '${osId}'. Supported: fedora-43, ubuntu-26.04` }); return reply.status(400).send({ error: `invalid os: '${osId}'. Supported: fedora-43, ubuntu-26.04` });
} }
const check = checkInstallAllowed(state.load(), mac, osId);
if (check.allowed === false) {
logger.warn(`INSTALL REFUSED: ${mac} -- ${check.error}`);
return reply.status(409).send({ error: check.error });
}
state.update((s) => { state.update((s) => {
s.install_queue[mac] = { s.install_queue[mac] = {
hostname: hostname ?? "lab-node", hostname: hostname ?? "lab-node",
@@ -284,6 +291,10 @@ export function registerApiRoutes(
arch?: string; arch?: string;
disks?: Array<{ name: string; size_gb: number; model: string }>; disks?: Array<{ name: string; size_gb: number; model: string }>;
nics?: Array<{ name: string; mac: string; state: string }>; nics?: Array<{ name: string; mac: string; state: string }>;
// Root filesystem, when the reporter could observe it (recheck over SSH, or the
// probe script run from a rescue shell). Used by --pxe-boot.
root_device?: string;
root_args?: string;
}; };
}>("/api/discover", async (request, reply) => { }>("/api/discover", async (request, reply) => {
const data = request.body; const data = request.body;
@@ -298,22 +309,53 @@ export function registerApiRoutes(
state.update((s) => { state.update((s) => {
const existing = s.discovered[mac]; const existing = s.discovered[mac];
// Classify onboarding from the DMI identity we just received. An explicit
// classification already on the record wins (see classifyOnboard).
const onboarding = classifyOnboard({
mac,
manufacturer: data.manufacturer ?? existing?.manufacturer ?? "unknown",
product: data.product ?? existing?.product ?? "unknown",
board: data.board ?? existing?.board ?? "unknown",
...(existing?.onboard !== undefined ? { onboard: existing.onboard } : {}),
...(existing?.vendor_os !== undefined ? { vendor_os: existing.vendor_os } : {}),
});
const rootDevice = data.root_device ?? existing?.root_device;
const rootArgs = data.root_args ?? existing?.root_args;
// Absent fields keep whatever we already knew. Reporters are not all the full
// discovery kickstart: the rescue-shell probe posts only a root device, and
// blanking a machine's hardware inventory as a side effect of that would be
// silent data loss.
const hwInfo: HardwareInfo = { const hwInfo: HardwareInfo = {
mac, mac,
product: data.product ?? "unknown", product: data.product ?? existing?.product ?? "unknown",
board: data.board ?? "unknown", board: data.board ?? existing?.board ?? "unknown",
serial: data.serial ?? "unknown", serial: data.serial ?? existing?.serial ?? "unknown",
manufacturer: data.manufacturer ?? "unknown", manufacturer: data.manufacturer ?? existing?.manufacturer ?? "unknown",
cpu_model: data.cpu_model ?? "unknown", cpu_model: data.cpu_model ?? existing?.cpu_model ?? "unknown",
cpu_cores: data.cpu_cores ?? 0, cpu_cores: data.cpu_cores ?? existing?.cpu_cores ?? 0,
memory_gb: data.memory_gb ?? 0, memory_gb: data.memory_gb ?? existing?.memory_gb ?? 0,
arch: data.arch ?? "unknown", arch: data.arch ?? existing?.arch ?? "unknown",
disks: data.disks ?? [], disks: data.disks ?? existing?.disks ?? [],
nics: data.nics ?? [], nics: data.nics ?? existing?.nics ?? [],
first_seen: existing?.first_seen ?? now, first_seen: existing?.first_seen ?? now,
last_seen: now, last_seen: now,
onboard: onboarding.onboard,
...(onboarding.vendor_os !== undefined ? { vendor_os: onboarding.vendor_os } : {}),
...(rootDevice !== undefined ? { root_device: rootDevice } : {}),
...(rootArgs !== undefined ? { root_args: rootArgs } : {}),
}; };
s.discovered[mac] = hwInfo; s.discovered[mac] = hwInfo;
// Keep the installed record in step -- the install guard and --pxe-boot read it.
const inst = s.installed[mac];
if (inst) {
if (data.arch !== undefined) inst.arch = data.arch;
inst.onboard = onboarding.onboard;
if (onboarding.vendor_os !== undefined) inst.vendor_os = onboarding.vendor_os;
if (rootDevice !== undefined) inst.root_device = rootDevice;
if (rootArgs !== undefined) inst.root_args = rootArgs;
}
}); });
const label = isNew ? "NEW MACHINE DISCOVERED" : "MACHINE RE-DISCOVERED"; const label = isNew ? "NEW MACHINE DISCOVERED" : "MACHINE RE-DISCOVERED";

153
bastion/src/bastion/src/routes/dispatch.ts
View File

@@ -5,7 +5,8 @@
// - unknown -> discovery mode (collect hardware, POST to bastion) // - unknown -> discovery mode (collect hardware, POST to bastion)
import type { FastifyInstance } from "fastify"; import type { FastifyInstance } from "fastify";
import type { BastionConfig } from "@lab/shared"; import type { Arch, BastionConfig, BastionState, OsId } from "@lab/shared";
import { normalizeArch, fedoraMirrorFor, osSupportsArch } from "@lab/shared";
import type { StateManager } from "../services/state.js"; import type { StateManager } from "../services/state.js";
import { import {
renderDiscoverIpxe, renderDiscoverIpxe,
@@ -13,11 +14,51 @@ import {
renderDebugIpxe, renderDebugIpxe,
renderPxeBootDebugIpxe, renderPxeBootDebugIpxe,
renderLocalBootIpxe, renderLocalBootIpxe,
renderUnsupportedIpxe,
} from "../templates/boot.ipxe.js"; } from "../templates/boot.ipxe.js";
import { renderUbuntuInstallIpxe } from "../templates/ubuntu-boot.ipxe.js"; import { renderUbuntuInstallIpxe } from "../templates/ubuntu-boot.ipxe.js";
import { renderDebugKickstart } from "../templates/debug.ks.js"; import { renderDebugKickstart } from "../templates/debug.ks.js";
import { logger } from "../services/logger.js"; import { logger } from "../services/logger.js";
/**
* Resolve a booting machine's architecture.
*
* Order matters. The tracked record is what we actually observed on the machine, so it
* wins. `reported` is iPXE's ${buildarch}, which is only as good as the binary DHCP
* handed the client -- correct in practice, but a misconfigured option 93 mapping would
* make it lie. The configured default is the last resort.
*
* There is deliberately no operator-supplied architecture anywhere in this path.
*/
export function resolveArch(
state: BastionState,
mac: string,
reported: string | undefined,
config: BastionConfig,
): Arch {
return normalizeArch(state.installed[mac]?.arch)
?? normalizeArch(state.install_queue[mac]?.arch)
?? normalizeArch(state.discovered[mac]?.arch)
?? normalizeArch(reported)
?? normalizeArch(config.arch)
?? "x86_64";
}
/** The root filesystem to boot for --pxe-boot, if the machine's record carries one. */
function resolveRoot(
state: BastionState,
mac: string,
): { rootDevice: string; rootArgs?: string } | null {
const installed = state.installed[mac];
const discovered = state.discovered[mac];
const rootDevice = installed?.root_device ?? discovered?.root_device;
if (rootDevice === undefined || rootDevice === "") return null;
const rootArgs = installed?.root_args ?? discovered?.root_args;
return rootArgs !== undefined && rootArgs !== ""
? { rootDevice, rootArgs }
: { rootDevice };
}
export function registerDispatchRoutes( export function registerDispatchRoutes(
app: FastifyInstance, app: FastifyInstance,
config: BastionConfig, config: BastionConfig,
@@ -52,18 +93,68 @@ curl -sf -X POST "http://${config.serverIp}:${config.httpPort}/api/progress" \\
-H "Content-Type: application/json" \\ -H "Content-Type: application/json" \\
-d "{\\"mac\\":\\"$MAC_ADDR\\",\\"stage\\":\\"debug-ready\\",\\"detail\\":\\"nc $IP_ADDR 2323\\"}" 2>/dev/null || true -d "{\\"mac\\":\\"$MAC_ADDR\\",\\"stage\\":\\"debug-ready\\",\\"detail\\":\\"nc $IP_ADDR 2323\\"}" 2>/dev/null || true
# --- Find the installed root filesystem and report it ---
# This is what 'labctl provision debug --pxe-boot' needs. The rescue image cannot
# report it by itself: %pre/%post do not run in rescue mode, so it happens here.
vgchange -ay >/dev/null 2>&1 || true
ROOT_DEVICE=""
ROOT_ARGS=""
PROBE_MNT=/tmp/lab-rootprobe
mkdir -p "$PROBE_MNT"
# Candidates: every LVM logical volume plus every non-LVM partition with a filesystem.
for CAND in $(lvs --noheadings -o lv_path 2>/dev/null) \\
$(blkid -o device 2>/dev/null | grep -v '^/dev/mapper/'); do
[ -b "$CAND" ] || continue
mount -o ro "$CAND" "$PROBE_MNT" >/dev/null 2>&1 || continue
# A root filesystem has both of these; /boot and /home do not.
if [ -f "$PROBE_MNT/etc/fstab" ] && [ -d "$PROBE_MNT/usr" ]; then
ROOT_DEVICE="$CAND"
PRETTY=$(. "$PROBE_MNT/etc/os-release" 2>/dev/null && echo "$PRETTY_NAME")
echo " found root: $CAND \${PRETTY:+($PRETTY)}"
if [ "$(lsblk -no TYPE "$CAND" 2>/dev/null | head -1)" = "lvm" ]; then
VGLV=$(lvs --noheadings -o vg_name,lv_name "$CAND" 2>/dev/null | awk '{print $1"/"$2}')
[ -n "$VGLV" ] && ROOT_ARGS="rd.lvm.lv=$VGLV"
# Swap comes from fstab here — /proc/swaps is the rescue image's, not the host's.
SWLV=$(awk '$3=="swap" && $1 ~ /^\\/dev\\// {print $1; exit}' "$PROBE_MNT/etc/fstab" 2>/dev/null)
if [ -n "$SWLV" ]; then
SWVGLV=$(lvs --noheadings -o vg_name,lv_name "$SWLV" 2>/dev/null | awk '{print $1"/"$2}')
[ -n "$SWVGLV" ] && [ "$SWVGLV" != "$VGLV" ] && ROOT_ARGS="$ROOT_ARGS rd.lvm.lv=$SWVGLV"
fi
fi
umount "$PROBE_MNT" >/dev/null 2>&1 || true
break
fi
umount "$PROBE_MNT" >/dev/null 2>&1 || true
done
if [ -n "$ROOT_DEVICE" ]; then
curl -sf -X POST "http://${config.serverIp}:${config.httpPort}/api/discover" \\
-H "Content-Type: application/json" \\
-d "{\\"mac\\":\\"$MAC_ADDR\\",\\"root_device\\":\\"$ROOT_DEVICE\\",\\"root_args\\":\\"$ROOT_ARGS\\"}" 2>/dev/null \\
&& echo " reported to bastion — 'labctl provision debug --pxe-boot' will work now"
else
echo " no root filesystem found — --pxe-boot cannot be used on this machine"
fi
echo "" echo ""
echo "=== Debug environment ready ===" echo "=== Debug environment ready ==="
echo " nc $IP_ADDR 2323 (remote shell)" echo " nc $IP_ADDR 2323 (remote shell)"
echo " ssh root@$IP_ADDR (password: debug)" echo " ssh root@$IP_ADDR (password: debug)"
if [ -n "$ROOT_DEVICE" ]; then
echo " root: $ROOT_DEVICE $ROOT_ARGS"
fi
echo "===============================" echo "==============================="
`; `;
return reply.type("text/plain").send(script); return reply.type("text/plain").send(script);
}); });
app.get<{ Querystring: { mac?: string } }>("/dispatch", async (request, reply) => { app.get<{ Querystring: { mac?: string; arch?: string } }>("/dispatch", async (request, reply) => {
const mac = (request.query.mac ?? "").toLowerCase().replace(/-/g, ":"); const mac = (request.query.mac ?? "").toLowerCase().replace(/-/g, ":");
const currentState = state.load(); const currentState = state.load();
const arch = resolveArch(currentState, mac, request.query.arch, config);
const fedoraMirror = fedoraMirrorFor(config.fedoraVersion, arch);
// Debug mode takes highest priority — auto-clear after serving once // Debug mode takes highest priority — auto-clear after serving once
const debugEntry = currentState.debug[mac]; const debugEntry = currentState.debug[mac];
@@ -72,22 +163,48 @@ echo "==============================="
state.update((s) => { delete s.debug[mac]; }); state.update((s) => { delete s.debug[mac]; });
let script: string; let script: string;
if (debugEntry.pxeBoot) { const wantsPxeBoot = debugEntry.pxeBoot === true;
logger.info(`PXE BOOT DEBUG: ${mac} -> ${hostname} (kernel+initrd from PXE, root from NVMe)`); const root = wantsPxeBoot ? resolveRoot(currentState, mac) : null;
if (root !== null) {
logger.info(`PXE BOOT DEBUG: ${mac} -> ${hostname} (${arch}, root=${root.rootDevice})`);
script = renderPxeBootDebugIpxe({ script = renderPxeBootDebugIpxe({
mac, mac,
hostname, hostname,
serverIp: config.serverIp, serverIp: config.serverIp,
httpPort: config.httpPort, httpPort: config.httpPort,
arch,
...root,
}); });
} else { } else {
logger.info(`DEBUG BOOT: ${mac} -> ${hostname} (rescue mode)`); // --pxe-boot without a known root device falls back to rescue rather than
// guessing. A wrong root= leaves the machine unbootable, and rescue is where
// the operator can find the real one (curl /debug-setup.sh reports it back).
const notice = wantsPxeBoot
? [
"",
"NOTE: --pxe-boot requested, but no root device is recorded",
" for this machine. Booting rescue instead.",
" From the rescue shell, run:",
// No pipe or && here: iPXE treats || and && as command separators, so keep
// the printed command free of anything its parser might claim.
` curl -s http://${config.serverIp}:${config.httpPort}/debug-setup.sh -o /tmp/s.sh ; sh /tmp/s.sh`,
" then retry --pxe-boot.",
]
: undefined;
if (wantsPxeBoot) {
logger.warn(`PXE BOOT DEBUG: ${mac} -> ${hostname} has no recorded root device -- serving rescue instead`);
} else {
logger.info(`DEBUG BOOT: ${mac} -> ${hostname} (${arch}, rescue mode)`);
}
script = renderDebugIpxe({ script = renderDebugIpxe({
mac, mac,
hostname, hostname,
serverIp: config.serverIp, serverIp: config.serverIp,
httpPort: config.httpPort, httpPort: config.httpPort,
fedoraMirror: config.fedoraMirror, fedoraMirror,
arch,
...(notice ? { notice } : {}),
}); });
} }
return reply.type("text/plain").send(script); return reply.type("text/plain").send(script);
@@ -97,10 +214,24 @@ echo "==============================="
if (queueEntry) { if (queueEntry) {
const hostname = queueEntry.hostname ?? "lab-node"; const hostname = queueEntry.hostname ?? "lab-node";
const os = queueEntry.os ?? "fedora-43"; const os = queueEntry.os ?? "fedora-43";
logger.info(`INSTALL STARTED: ${mac} -> ${hostname} (${os})`); logger.info(`INSTALL STARTED: ${mac} -> ${hostname} (${os}, ${arch})`);
let script: string; let script: string;
if (os.startsWith("ubuntu")) { if (os.startsWith("ubuntu")) {
// Last line of defence. The install guard refuses this combination when the
// machine's architecture is already known, but a machine queued before it was
// discovered can reach here. Serving the x86-only Ubuntu kernel to an arm64
// client is precisely the bug this work exists to fix, so stop instead.
if (!osSupportsArch(os as OsId, arch)) {
logger.error(`INSTALL BLOCKED: ${mac} -> ${hostname} -- ${os} has no ${arch} artifacts`);
script = renderUnsupportedIpxe({
hostname,
mac,
reason: `${os} publishes no ${arch} netboot artifacts`,
action: `labctl provision install ${mac} ${hostname} --os fedora-43`,
});
return reply.type("text/plain").send(script);
}
script = renderUbuntuInstallIpxe({ script = renderUbuntuInstallIpxe({
mac, mac,
hostname, hostname,
@@ -115,7 +246,8 @@ echo "==============================="
serverIp: config.serverIp, serverIp: config.serverIp,
httpPort: config.httpPort, httpPort: config.httpPort,
fedoraVersion: config.fedoraVersion, fedoraVersion: config.fedoraVersion,
fedoraMirror: config.fedoraMirror, fedoraMirror,
arch,
}); });
} }
@@ -132,13 +264,14 @@ echo "==============================="
} }
// Unknown MAC -> discovery mode // Unknown MAC -> discovery mode
logger.info(`PXE request from ${mac} -> discovery mode`); logger.info(`PXE request from ${mac} (${arch}) -> discovery mode`);
const script = renderDiscoverIpxe({ const script = renderDiscoverIpxe({
mac, mac,
serverIp: config.serverIp, serverIp: config.serverIp,
httpPort: config.httpPort, httpPort: config.httpPort,
fedoraMirror: config.fedoraMirror, fedoraMirror,
arch,
}); });
return reply.type("text/plain").send(script); return reply.type("text/plain").send(script);

94
bastion/src/bastion/src/services/install-guard.ts Normal file
View File

@@ -0,0 +1,94 @@
// Pre-flight checks for queuing an OS install.
//
// Both entry points -- the HTTP /api/install route and the labd command-install handler
// -- run this, so `labctl provision install` and `provision reprovision` are covered
// whichever way the request arrives.
//
// Rescue/debug is deliberately NOT guarded. Being unable to reinstall a machine is
// exactly when you most need to boot it into a rescue shell.
import type { Arch, BastionState, OsId } from "@lab/shared";
import { classifyOnboard, normalizeArch, osSupportsArch, vendorOsDescription, archesForOs } from "@lab/shared";
export type InstallCheck =
| { allowed: true }
| { allowed: false; error: string };
interface MachineIdentity {
hostname: string;
arch: Arch | undefined;
identity: Parameters<typeof classifyOnboard>[0];
}
/** Best-known identity for a MAC, merged across the three state maps. */
function identify(state: BastionState, mac: string): MachineIdentity {
const discovered = state.discovered[mac];
const installed = state.installed[mac];
const queued = state.install_queue[mac];
const manufacturer = discovered?.manufacturer ?? installed?.manufacturer;
const product = discovered?.product ?? installed?.product;
const board = discovered?.board;
const onboard = installed?.onboard ?? discovered?.onboard;
const vendorOs = installed?.vendor_os ?? discovered?.vendor_os;
return {
hostname: installed?.hostname ?? queued?.hostname ?? discovered?.product ?? mac,
arch: normalizeArch(installed?.arch ?? queued?.arch ?? discovered?.arch),
identity: {
mac,
...(manufacturer !== undefined ? { manufacturer } : {}),
...(product !== undefined ? { product } : {}),
...(board !== undefined ? { board } : {}),
...(onboard !== undefined ? { onboard } : {}),
...(vendorOs !== undefined ? { vendor_os: vendorOs } : {}),
},
};
}
/**
* Decide whether `mac` may be queued for an install of `os`.
*
* Refusals name the machine and the reason, and point at the action that is available
* instead. An operator hitting this at 2am should not have to read the source to work
* out what happened.
*/
export function checkInstallAllowed(
state: BastionState,
mac: string,
os: OsId,
): InstallCheck {
const machine = identify(state, mac);
const { onboard, vendor_os } = classifyOnboard(machine.identity);
// 1. Machines running a vendor OS we cannot rebuild.
if (onboard === "ssh") {
const what = vendorOsDescription(vendor_os);
return {
allowed: false,
error:
`Refusing to install ${machine.hostname} (${mac}): it runs ${what}. ` +
`No image in our pipeline can restore it, so installing ${os} would destroy that ` +
`driver and firmware stack permanently. This machine is SSH-onboard: we manage its ` +
`userspace, not its OS. ` +
`To boot it into a rescue shell instead, run: labctl provision debug ${machine.hostname}`,
// TODO: when a DGX OS / SparkOS image joins the pipeline, an install targeting a
// machine whose vendor_os matches that image should be allowed through here.
};
}
// 2. Architecture the OS has no netboot artifacts for.
if (machine.arch !== undefined && !osSupportsArch(os, machine.arch)) {
const supported = archesForOs(os);
return {
allowed: false,
error:
`Refusing to install ${os} on ${machine.hostname} (${mac}): ` +
`${os} has no ${machine.arch} netboot artifacts` +
(supported.length > 0 ? ` (only ${supported.join(", ")})` : "") +
`. Use an OS that supports ${machine.arch}.`,
};
}
return { allowed: true };
}

130
bastion/src/bastion/src/templates/boot.ipxe.ts
View File

@@ -1,4 +1,55 @@
// iPXE boot script templates for dispatch routing. // iPXE boot script templates for dispatch routing.
//
// Architecture handling: the bastion serves one kernel/initrd pair per architecture.
// x86_64 keeps the original unsuffixed paths so its output is unchanged; every other
// architecture gets an arch-suffixed pair. See stageBootArtifacts() in main.ts for the
// matching staging side, and boot-iso.ts for the same scheme on the ISO path.
import type { Arch } from "@lab/shared";
/** Kernel/initrd URL paths, keyed by architecture. */
export function kernelPath(arch: Arch): string {
return arch === "x86_64" ? "/vmlinuz" : `/vmlinuz-${arch}`;
}
export function initrdPath(arch: Arch): string {
return arch === "x86_64" ? "/initrd.img" : `/initrd-${arch}.img`;
}
/**
* Console arguments per architecture.
*
* arm64 has no VGA text console: a headless machine only talks over the SoC UART, so
* ttyAMA0 must be listed as well. The last console= wins for /dev/console, so serial
* is the interactive one while tty0 still receives boot output on machines with a
* display attached.
*/
const CONSOLE_ARGS: Record<Arch, string> = {
x86_64: "console=tty0",
aarch64: "console=tty0 console=ttyAMA0,115200",
};
/**
* Anaconda arguments for the graphical-suppression / console setup.
*
* `nomodeset` disables kernel mode setting, which on x86 forces the generic VGA path
* and makes flaky GPU drivers survive the installer. On arm64 it does not mean the
* same thing -- there is no VGA fallback to drop back to, and it can leave the machine
* with no usable console at all -- so arm64 gets explicit console arguments instead.
*/
function installerArgs(arch: Arch): string {
return arch === "x86_64" ? "inst.text nomodeset" : `inst.text ${CONSOLE_ARGS[arch]}`;
}
/** Extra console arguments appended to templates that don't already set them. */
function extraConsoleArgs(arch: Arch): string {
return arch === "x86_64" ? "" : ` ${CONSOLE_ARGS[arch]}`;
}
/** Join kernel arguments, dropping empties so callers can pass optional groups. */
function joinArgs(...parts: Array<string | undefined>): string {
return parts.filter((p) => p !== undefined && p !== "").join(" ");
}
export interface BootIpxeParams { export interface BootIpxeParams {
serverIp: string; serverIp: string;
@@ -8,6 +59,11 @@ export interface BootIpxeParams {
/** /**
* Initial iPXE boot script that chains to the dispatch endpoint. * Initial iPXE boot script that chains to the dispatch endpoint.
* This is what dnsmasq serves to iPXE clients via HTTP. * This is what dnsmasq serves to iPXE clients via HTTP.
*
* `${buildarch}` is iPXE's own build architecture ("x86_64" or "arm64"), which is the
* one architecture signal available on every path -- network PXE, UEFI HTTP boot and
* the boot ISO alike. DHCP option 93 only reaches dnsmasq, never this HTTP endpoint.
* dispatch prefers the tracked machine record and falls back to this.
*/ */
export function renderBootIpxe(params: BootIpxeParams): string { export function renderBootIpxe(params: BootIpxeParams): string {
return `#!ipxe return `#!ipxe
@@ -19,7 +75,7 @@ echo Contacting server for instructions...
echo ============================================ echo ============================================
echo echo
chain http://${params.serverIp}:${params.httpPort}/dispatch?mac=\${net0/mac} chain http://${params.serverIp}:${params.httpPort}/dispatch?mac=\${net0/mac}&arch=\${buildarch}
`; `;
} }
@@ -31,7 +87,9 @@ export function renderDiscoverIpxe(params: {
serverIp: string; serverIp: string;
httpPort: number; httpPort: number;
fedoraMirror: string; fedoraMirror: string;
arch: Arch;
}): string { }): string {
const base = `http://${params.serverIp}:${params.httpPort}`;
return `#!ipxe return `#!ipxe
echo echo
@@ -42,8 +100,8 @@ echo Collecting hardware info...
echo ============================================= echo =============================================
echo echo
kernel http://${params.serverIp}:${params.httpPort}/vmlinuz inst.ks=http://${params.serverIp}:${params.httpPort}/discover.ks inst.stage2=${params.fedoraMirror} inst.text nomodeset kernel ${base}${kernelPath(params.arch)} inst.ks=${base}/discover.ks inst.stage2=${params.fedoraMirror} ${installerArgs(params.arch)}
initrd http://${params.serverIp}:${params.httpPort}/initrd.img initrd ${base}${initrdPath(params.arch)}
boot boot
`; `;
} }
@@ -58,7 +116,9 @@ export function renderInstallIpxe(params: {
httpPort: number; httpPort: number;
fedoraVersion: string; fedoraVersion: string;
fedoraMirror: string; fedoraMirror: string;
arch: Arch;
}): string { }): string {
const base = `http://${params.serverIp}:${params.httpPort}`;
return `#!ipxe return `#!ipxe
echo echo
@@ -69,8 +129,8 @@ echo MAC: ${params.mac}
echo ============================================= echo =============================================
echo echo
kernel http://${params.serverIp}:${params.httpPort}/vmlinuz inst.ks=http://${params.serverIp}:${params.httpPort}/ks?mac=${params.mac} inst.repo=${params.fedoraMirror} inst.text nomodeset kernel ${base}${kernelPath(params.arch)} inst.ks=${base}/ks?mac=${params.mac} inst.repo=${params.fedoraMirror} ${installerArgs(params.arch)}
initrd http://${params.serverIp}:${params.httpPort}/initrd.img initrd ${base}${initrdPath(params.arch)}
boot boot
`; `;
} }
@@ -78,6 +138,9 @@ boot
/** /**
* iPXE script for debug/rescue mode -- boots Fedora installer in rescue mode. * iPXE script for debug/rescue mode -- boots Fedora installer in rescue mode.
* Provides a shell with LVM tools, network, and SSH for inspecting installed systems. * Provides a shell with LVM tools, network, and SSH for inspecting installed systems.
*
* `notice` is shown before the boot line. dispatch uses it to explain why a requested
* --pxe-boot fell back to rescue.
*/ */
export function renderDebugIpxe(params: { export function renderDebugIpxe(params: {
mac: string; mac: string;
@@ -85,7 +148,11 @@ export function renderDebugIpxe(params: {
serverIp: string; serverIp: string;
httpPort: number; httpPort: number;
fedoraMirror: string; fedoraMirror: string;
arch: Arch;
notice?: string[];
}): string { }): string {
const base = `http://${params.serverIp}:${params.httpPort}`;
const notice = (params.notice ?? []).map((line) => `echo ${line}\n`).join("");
return `#!ipxe return `#!ipxe
echo echo
@@ -93,11 +160,11 @@ echo =============================================
echo Lab PXE Bastion - DEBUG/RESCUE MODE echo Lab PXE Bastion - DEBUG/RESCUE MODE
echo Target: ${params.hostname} echo Target: ${params.hostname}
echo MAC: ${params.mac} echo MAC: ${params.mac}
echo ============================================= ${notice}echo =============================================
echo echo
kernel http://${params.serverIp}:${params.httpPort}/vmlinuz inst.rescue inst.text inst.sshd inst.ks=http://${params.serverIp}:${params.httpPort}/debug.ks?mac=${params.mac} inst.stage2=${params.fedoraMirror} kernel ${base}${kernelPath(params.arch)} inst.rescue inst.text inst.sshd inst.ks=${base}/debug.ks?mac=${params.mac} inst.stage2=${params.fedoraMirror}${extraConsoleArgs(params.arch)}
initrd http://${params.serverIp}:${params.httpPort}/initrd.img initrd ${base}${initrdPath(params.arch)}
boot boot
`; `;
} }
@@ -106,13 +173,28 @@ boot
* iPXE script for PXE-boot debug mode -- boots the installed system's root * iPXE script for PXE-boot debug mode -- boots the installed system's root
* filesystem using the bastion's PXE kernel+initrd instead of local GRUB. * filesystem using the bastion's PXE kernel+initrd instead of local GRUB.
* Workaround for UEFI firmware bugs that make local disk boot slow. * Workaround for UEFI firmware bugs that make local disk boot slow.
*
* rootDevice/rootArgs come from the machine's record -- they are not assumed. Our
* Fedora installs use an LVM layout, but nothing guarantees any given machine does,
* and a wrong root= here means an unbootable machine. dispatch refuses to render this
* script without them.
*/ */
export function renderPxeBootDebugIpxe(params: { export function renderPxeBootDebugIpxe(params: {
mac: string; mac: string;
hostname: string; hostname: string;
serverIp: string; serverIp: string;
httpPort: number; httpPort: number;
arch: Arch;
rootDevice: string;
rootArgs?: string;
}): string { }): string {
const base = `http://${params.serverIp}:${params.httpPort}`;
const cmdline = joinArgs(
`root=${params.rootDevice}`,
"ro",
params.rootArgs,
CONSOLE_ARGS[params.arch],
);
return `#!ipxe return `#!ipxe
echo echo
@@ -124,12 +206,40 @@ echo Kernel+initrd from PXE, root from NVMe
echo ============================================= echo =============================================
echo echo
kernel http://${params.serverIp}:${params.httpPort}/vmlinuz root=/dev/mapper/labvg-root ro rd.lvm.lv=labvg/root rd.lvm.lv=labvg/swap console=tty0 kernel ${base}${kernelPath(params.arch)} ${cmdline}
initrd http://${params.serverIp}:${params.httpPort}/initrd.img initrd ${base}${initrdPath(params.arch)}
boot boot
`; `;
} }
/**
* iPXE script for a request we refuse to serve.
*
* Better a machine that stops with a legible reason on its console than one handed a
* kernel it cannot execute, which fails much later and much less clearly.
*/
export function renderUnsupportedIpxe(params: {
mac: string;
hostname: string;
reason: string;
action?: string;
}): string {
return `#!ipxe
echo
echo =============================================
echo Lab PXE Bastion - CANNOT BOOT THIS MACHINE
echo Target: ${params.hostname}
echo MAC: ${params.mac}
echo
echo ${params.reason}
${params.action !== undefined ? `echo\necho Try: ${params.action}\n` : ""}echo =============================================
echo
sleep 10
exit 1
`;
}
/** /**
* iPXE script for already-installed machines -- exits to boot from local disk. * iPXE script for already-installed machines -- exits to boot from local disk.
*/ */

11
bastion/src/bastion/src/templates/dnsmasq.conf.ts
View File

@@ -48,15 +48,20 @@ enable-tftp
tftp-root=${tftpDir} tftp-root=${tftpDir}
tftp-no-blocksize tftp-no-blocksize
# Detect client architecture -- PXE (TFTP) clients # Detect client architecture -- PXE (TFTP) clients.
# Values are DHCP option 93 (Client System Architecture), IANA "Processor Architecture
# Types". Getting these wrong means the machine is handed a bootloader its firmware
# cannot execute, and it loops or hangs with no console output.
dhcp-match=set:bios,option:client-arch,0 dhcp-match=set:bios,option:client-arch,0
dhcp-match=set:efi-x86_64,option:client-arch,7 dhcp-match=set:efi-x86_64,option:client-arch,7
dhcp-match=set:efi-x86_64,option:client-arch,9 dhcp-match=set:efi-x86_64,option:client-arch,9
dhcp-match=set:efi-arm64,option:client-arch,11 dhcp-match=set:efi-arm64,option:client-arch,11
# Detect client architecture -- UEFI HTTP Boot clients (no TFTP size limit) # Detect client architecture -- UEFI HTTP Boot clients (no TFTP size limit).
# 16 = x64 uefi boot from http, 19 = arm uefi 64 boot from http.
# (20 is pc/at bios boot from http -- not arm64.)
dhcp-match=set:httpboot-x86_64,option:client-arch,16 dhcp-match=set:httpboot-x86_64,option:client-arch,16
dhcp-match=set:httpboot-arm64,option:client-arch,20 dhcp-match=set:httpboot-arm64,option:client-arch,19
# Detect iPXE clients (already chainloaded) # Detect iPXE clients (already chainloaded)
dhcp-userclass=set:ipxe,iPXE dhcp-userclass=set:ipxe,iPXE

291
bastion/src/bastion/tests/arch-dispatch.test.ts Normal file
View File

@@ -0,0 +1,291 @@
// aarch64 support in the PXE dispatch path.
//
// The x86_64 side is pinned separately by ipxe-x86-regression.test.ts.
import { describe, it, expect, beforeEach, afterEach } from "vitest";
import { mkdirSync, rmSync } from "node:fs";
import { join } from "node:path";
import { tmpdir } from "node:os";
import type { BastionConfig, BastionState, HardwareInfo } from "@lab/shared";
import { createApp } from "../src/server.js";
import { resolveArch } from "../src/routes/dispatch.js";
import { renderDnsmasqConf } from "../src/templates/dnsmasq.conf.js";
import type { FastifyInstance } from "fastify";
import type { StateManager } from "../src/services/state.js";
function createTestConfig(testDir: string): BastionConfig {
return {
fedoraVersion: "43",
arch: "x86_64",
httpPort: 0,
timezone: "Europe/London",
locale: "en_GB.UTF-8",
bastionDir: testDir,
domain: "test.local",
dhcpMode: "proxy",
dhcpRangeStart: "",
dhcpRangeEnd: "",
ubuntuVersion: "26.04",
ubuntuMirror: "https://releases.ubuntu.com/26.04",
iface: "eth0",
serverIp: "10.0.0.1",
network: "10.0.0.0",
gateway: "10.0.0.1",
sshKeys: ["ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAITEST test@test"],
adminUser: "testadmin",
syslogPort: 15514,
skipDnsmasq: true,
skipArtifacts: true,
fedoraMirror: "https://download.fedoraproject.org/pub/fedora/linux/releases/43/Everything/x86_64/os",
tftpDir: join(testDir, "tftp"),
httpDir: join(testDir, "http"),
stateFile: join(testDir, "state.json"),
};
}
function hardware(mac: string, over: Partial<HardwareInfo> = {}): HardwareInfo {
return {
mac,
product: "TestBox",
board: "TestBoard",
serial: "SN123",
manufacturer: "TestCorp",
cpu_model: "Test CPU",
cpu_cores: 4,
memory_gb: 16,
arch: "x86_64",
disks: [],
nics: [],
first_seen: new Date().toISOString(),
last_seen: new Date().toISOString(),
...over,
};
}
const emptyState = (): BastionState => ({
discovered: {}, install_queue: {}, installed: {}, debug: {},
});
describe("architecture resolution", () => {
const config = createTestConfig("/tmp/unused");
const mac = "aa:bb:cc:dd:ee:ff";
it("prefers the tracked record over what the client reports", () => {
const state = emptyState();
state.discovered[mac] = hardware(mac, { arch: "aarch64" });
// Client claims x86_64; the machine record says otherwise and wins.
expect(resolveArch(state, mac, "x86_64", config)).toBe("aarch64");
});
it("falls back to the architecture reported at boot", () => {
expect(resolveArch(emptyState(), mac, "arm64", config)).toBe("aarch64");
});
it("normalises iPXE's arm64 spelling to aarch64", () => {
expect(resolveArch(emptyState(), mac, "arm64", config)).toBe("aarch64");
expect(resolveArch(emptyState(), mac, "x86_64", config)).toBe("x86_64");
});
it("falls back to the configured default for unknown architectures", () => {
expect(resolveArch(emptyState(), mac, "riscv64", config)).toBe("x86_64");
expect(resolveArch(emptyState(), mac, undefined, config)).toBe("x86_64");
});
it("reads arch from the installed record for already-provisioned machines", () => {
const state = emptyState();
state.installed[mac] = {
hostname: "spark", role: "worker", ip: "10.0.0.5",
installed_at: new Date().toISOString(), arch: "aarch64",
};
expect(resolveArch(state, mac, undefined, config)).toBe("aarch64");
});
});
describe("aarch64 dispatch", () => {
let testDir: string;
let app: FastifyInstance;
let state: StateManager;
const mac = "aa:bb:cc:dd:ee:ff";
beforeEach(() => {
testDir = join(tmpdir(), `bastion-arch-test-${Date.now()}-${Math.random().toString(36).slice(2)}`);
mkdirSync(join(testDir, "http"), { recursive: true });
mkdirSync(join(testDir, "tftp"), { recursive: true });
const result = createApp(createTestConfig(testDir));
app = result.app;
state = result.state;
});
afterEach(async () => {
await app.close();
rmSync(testDir, { recursive: true, force: true });
});
it("serves the aarch64 kernel and initrd to an arm64 client", async () => {
const res = await app.inject({ method: "GET", url: `/dispatch?mac=${mac}&arch=arm64` });
expect(res.statusCode).toBe(200);
expect(res.body).toContain("/vmlinuz-aarch64");
expect(res.body).toContain("/initrd-aarch64.img");
expect(res.body).not.toContain("/vmlinuz ");
});
it("points an arm64 client at the aarch64 Fedora mirror", async () => {
const res = await app.inject({ method: "GET", url: `/dispatch?mac=${mac}&arch=arm64` });
expect(res.body).toContain("Everything/aarch64/os");
expect(res.body).not.toContain("Everything/x86_64/os");
});
it("uses serial console arguments and not nomodeset on arm64", async () => {
const res = await app.inject({ method: "GET", url: `/dispatch?mac=${mac}&arch=arm64` });
expect(res.body).toContain("console=ttyAMA0,115200");
expect(res.body).not.toContain("nomodeset");
});
it("refuses to serve the x86-only Ubuntu kernel to an arm64 client", async () => {
// A machine queued for Ubuntu before it was discovered as aarch64 reaches dispatch
// with no guard having run. Serving it /ubuntu-vmlinuz is the original bug.
state.update((s) => {
s.install_queue[mac] = {
hostname: "arm-node", disk: "", role: "worker",
os: "ubuntu-26.04", queued_at: new Date().toISOString(),
};
});
const res = await app.inject({ method: "GET", url: `/dispatch?mac=${mac}&arch=arm64` });
expect(res.statusCode).toBe(200);
expect(res.body).toContain("CANNOT BOOT THIS MACHINE");
expect(res.body).toContain("no aarch64 netboot artifacts");
expect(res.body).not.toContain("ubuntu-vmlinuz");
});
it("still serves Ubuntu to an x86_64 client", async () => {
state.update((s) => {
s.install_queue[mac] = {
hostname: "x86-node", disk: "", role: "worker",
os: "ubuntu-26.04", queued_at: new Date().toISOString(),
};
});
const res = await app.inject({ method: "GET", url: `/dispatch?mac=${mac}&arch=x86_64` });
expect(res.body).toContain("ubuntu-vmlinuz");
expect(res.body).not.toContain("CANNOT BOOT");
});
it("serves a rescue kernel for the recorded architecture, not the requester's", async () => {
// The Spark case: machine known to be aarch64, queued for rescue.
state.update((s) => {
s.discovered[mac] = hardware(mac, { arch: "aarch64" });
s.debug[mac] = { hostname: "spark-2935", queued_at: new Date().toISOString() };
});
const res = await app.inject({ method: "GET", url: `/dispatch?mac=${mac}` });
expect(res.body).toContain("DEBUG/RESCUE MODE");
expect(res.body).toContain("/vmlinuz-aarch64");
expect(res.body).toContain("inst.rescue");
expect(res.body).toContain("inst.sshd");
});
});
describe("--pxe-boot root device", () => {
let testDir: string;
let app: FastifyInstance;
let state: StateManager;
const mac = "aa:bb:cc:dd:ee:ff";
beforeEach(() => {
testDir = join(tmpdir(), `bastion-root-test-${Date.now()}-${Math.random().toString(36).slice(2)}`);
mkdirSync(join(testDir, "http"), { recursive: true });
mkdirSync(join(testDir, "tftp"), { recursive: true });
const result = createApp(createTestConfig(testDir));
app = result.app;
state = result.state;
});
afterEach(async () => {
await app.close();
rmSync(testDir, { recursive: true, force: true });
});
it("uses the root device recorded on the machine", async () => {
state.update((s) => {
s.installed[mac] = {
hostname: "worker-1", role: "worker", ip: "10.0.0.50",
installed_at: new Date().toISOString(),
root_device: "/dev/mapper/otherVG-root",
root_args: "rd.lvm.lv=otherVG/root",
};
s.debug[mac] = { hostname: "worker-1", queued_at: new Date().toISOString(), pxeBoot: true };
});
const res = await app.inject({ method: "GET", url: `/dispatch?mac=${mac}` });
expect(res.body).toContain("PXE BOOT (debug)");
expect(res.body).toContain("root=/dev/mapper/otherVG-root");
expect(res.body).toContain("rd.lvm.lv=otherVG/root");
// The old hardcoded layout must not leak back in.
expect(res.body).not.toContain("labvg");
});
it("falls back to rescue rather than guessing when no root device is known", async () => {
state.update((s) => {
s.installed[mac] = {
hostname: "spark-2935", role: "worker", ip: "192.168.8.12",
installed_at: new Date().toISOString(), arch: "aarch64",
};
s.debug[mac] = { hostname: "spark-2935", queued_at: new Date().toISOString(), pxeBoot: true };
});
const res = await app.inject({ method: "GET", url: `/dispatch?mac=${mac}` });
expect(res.body).toContain("DEBUG/RESCUE MODE");
expect(res.body).toContain("no root device is recorded");
expect(res.body).toContain("debug-setup.sh");
expect(res.body).not.toContain("root=");
// And it is still the right architecture.
expect(res.body).toContain("/vmlinuz-aarch64");
});
it("records a root device reported from a rescue shell without erasing hardware info", async () => {
state.update((s) => {
s.discovered[mac] = hardware(mac, { product: "DGX Spark", manufacturer: "NVIDIA", arch: "aarch64" });
});
const res = await app.inject({
method: "POST",
url: "/api/discover",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ mac, root_device: "/dev/nvme0n1p2" }),
});
expect(res.statusCode).toBe(200);
const hw = state.load().discovered[mac];
expect(hw?.root_device).toBe("/dev/nvme0n1p2");
// The partial report must not blank what we already knew.
expect(hw?.product).toBe("DGX Spark");
expect(hw?.cpu_cores).toBe(4);
expect(hw?.arch).toBe("aarch64");
});
});
describe("dnsmasq architecture detection", () => {
const conf = renderDnsmasqConf(createTestConfig("/tmp/unused"));
it("maps DHCP option 93 values to per-architecture bootloaders", () => {
// 11 = ARM 64-bit UEFI
expect(conf).toContain("dhcp-match=set:efi-arm64,option:client-arch,11");
expect(conf).toContain("dhcp-boot=tag:efi-arm64,tag:!ipxe,ipxe-arm64.efi");
// 7 / 9 = x64 UEFI, 0 = x86 BIOS
expect(conf).toContain("dhcp-match=set:efi-x86_64,option:client-arch,7");
expect(conf).toContain("dhcp-match=set:efi-x86_64,option:client-arch,9");
expect(conf).toContain("dhcp-match=set:bios,option:client-arch,0");
});
it("matches arm64 UEFI HTTP boot on 19, not 20", () => {
// IANA: 19 = arm uefi 64 boot from http, 20 = pc/at bios boot from http.
expect(conf).toContain("dhcp-match=set:httpboot-arm64,option:client-arch,19");
expect(conf).not.toContain("dhcp-match=set:httpboot-arm64,option:client-arch,20");
expect(conf).toContain("dhcp-match=set:httpboot-x86_64,option:client-arch,16");
});
it("offers an arm64 PXE service directive in proxy mode", () => {
expect(conf).toContain('pxe-service=tag:!ipxe,ARM64_EFI,"PXE Boot",ipxe-arm64.efi');
});
});

8
bastion/src/bastion/tests/fixtures/ipxe-x86_64-golden.json vendored Normal file
View File

@@ -0,0 +1,8 @@
{
"boot": "#!ipxe\n\necho\necho ============================================\necho Lab PXE Bastion\necho Contacting server for instructions...\necho ============================================\necho\n\nchain http://10.0.0.1:8080/dispatch?mac=${net0/mac}\n",
"discover": "#!ipxe\n\necho\necho =============================================\necho Lab PXE Bastion - DISCOVERY MODE\necho MAC: aa:bb:cc:dd:ee:ff\necho Collecting hardware info...\necho =============================================\necho\n\nkernel http://10.0.0.1:8080/vmlinuz inst.ks=http://10.0.0.1:8080/discover.ks inst.stage2=https://download.fedoraproject.org/pub/fedora/linux/releases/43/Everything/x86_64/os inst.text nomodeset\ninitrd http://10.0.0.1:8080/initrd.img\nboot\n",
"install": "#!ipxe\n\necho\necho =============================================\necho Lab PXE Bastion - INSTALLING Fedora 43\necho Target: worker-1\necho MAC: aa:bb:cc:dd:ee:ff\necho =============================================\necho\n\nkernel http://10.0.0.1:8080/vmlinuz inst.ks=http://10.0.0.1:8080/ks?mac=aa:bb:cc:dd:ee:ff inst.repo=https://download.fedoraproject.org/pub/fedora/linux/releases/43/Everything/x86_64/os inst.text nomodeset\ninitrd http://10.0.0.1:8080/initrd.img\nboot\n",
"debug": "#!ipxe\n\necho\necho =============================================\necho Lab PXE Bastion - DEBUG/RESCUE MODE\necho Target: worker-1\necho MAC: aa:bb:cc:dd:ee:ff\necho =============================================\necho\n\nkernel http://10.0.0.1:8080/vmlinuz inst.rescue inst.text inst.sshd inst.ks=http://10.0.0.1:8080/debug.ks?mac=aa:bb:cc:dd:ee:ff inst.stage2=https://download.fedoraproject.org/pub/fedora/linux/releases/43/Everything/x86_64/os\ninitrd http://10.0.0.1:8080/initrd.img\nboot\n",
"pxeBoot": "#!ipxe\n\necho\necho =============================================\necho Lab PXE Bastion - PXE BOOT (debug)\necho Target: worker-1\necho MAC: aa:bb:cc:dd:ee:ff\necho Kernel+initrd from PXE, root from NVMe\necho =============================================\necho\n\nkernel http://10.0.0.1:8080/vmlinuz root=/dev/mapper/labvg-root ro rd.lvm.lv=labvg/root rd.lvm.lv=labvg/swap console=tty0\ninitrd http://10.0.0.1:8080/initrd.img\nboot\n",
"localBoot": "#!ipxe\n\necho\necho =============================================\necho Lab PXE Bastion - worker-1\necho Already installed, booting from local disk\necho =============================================\necho\nsleep 3\nexit 1\n"
}

194
bastion/src/bastion/tests/install-guard.test.ts Normal file
View File

@@ -0,0 +1,194 @@
// Installs must never reach a machine running a vendor OS we cannot restore.
//
// This is the guardrail that stops someone reinstalling a DGX Spark at 2am. Rescue is
// deliberately still allowed for the same machines -- that is the whole point.
import { describe, it, expect, beforeEach, afterEach } from "vitest";
import { mkdirSync, rmSync } from "node:fs";
import { join } from "node:path";
import { tmpdir } from "node:os";
import type { BastionConfig, BastionState, HardwareInfo } from "@lab/shared";
import { classifyOnboard } from "@lab/shared";
import { createApp } from "../src/server.js";
import { checkInstallAllowed } from "../src/services/install-guard.js";
import type { FastifyInstance } from "fastify";
import type { StateManager } from "../src/services/state.js";
// The real machines this exists to protect.
const SPARK_2935 = "4c:bb:47:7f:29:35";
const SPARK_3A1C = "48:21:0b:96:3a:1c";
const ORDINARY = "aa:bb:cc:dd:ee:ff";
function createTestConfig(testDir: string): BastionConfig {
return {
fedoraVersion: "43", arch: "x86_64", httpPort: 0,
timezone: "Europe/London", locale: "en_GB.UTF-8", bastionDir: testDir,
domain: "test.local", dhcpMode: "proxy", dhcpRangeStart: "", dhcpRangeEnd: "",
ubuntuVersion: "26.04", ubuntuMirror: "https://releases.ubuntu.com/26.04",
iface: "eth0", serverIp: "10.0.0.1", network: "10.0.0.0", gateway: "10.0.0.1",
sshKeys: [], adminUser: "testadmin", syslogPort: 15514,
skipDnsmasq: true, skipArtifacts: true,
fedoraMirror: "https://download.fedoraproject.org/pub/fedora/linux/releases/43/Everything/x86_64/os",
tftpDir: join(testDir, "tftp"), httpDir: join(testDir, "http"),
stateFile: join(testDir, "state.json"),
};
}
function hardware(mac: string, over: Partial<HardwareInfo> = {}): HardwareInfo {
return {
mac, product: "TestBox", board: "TestBoard", serial: "SN1",
manufacturer: "TestCorp", cpu_model: "Test CPU", cpu_cores: 4, memory_gb: 16,
arch: "x86_64", disks: [], nics: [],
first_seen: new Date().toISOString(), last_seen: new Date().toISOString(),
...over,
};
}
const emptyState = (): BastionState => ({
discovered: {}, install_queue: {}, installed: {}, debug: {},
});
describe("classifyOnboard", () => {
it("recognises a DGX Spark from its DMI identity", () => {
expect(classifyOnboard({
mac: ORDINARY, manufacturer: "NVIDIA", product: "NVIDIA DGX Spark", board: "GB10",
})).toEqual({ onboard: "ssh", vendor_os: "dgx-os" });
});
it("recognises the known Sparks even with no DMI recorded", () => {
// Neither Spark has hardware info in bastion state today. A DMI-only rule would
// fail open on exactly the machines this protects.
expect(classifyOnboard({ mac: SPARK_2935 }).onboard).toBe("ssh");
expect(classifyOnboard({ mac: SPARK_3A1C }).onboard).toBe("ssh");
});
it("treats ordinary hardware as PXE-installable", () => {
expect(classifyOnboard({
mac: ORDINARY, manufacturer: "Beelink", product: "SER9", board: "SER9",
})).toEqual({ onboard: "pxe" });
});
it("does not override an explicit classification already on the record", () => {
expect(classifyOnboard({
mac: SPARK_2935, onboard: "pxe",
})).toEqual({ onboard: "pxe" });
});
});
describe("checkInstallAllowed", () => {
it("refuses a DGX Spark and explains why", () => {
const state = emptyState();
state.installed[SPARK_2935] = {
hostname: "spark-2935", role: "worker", ip: "192.168.8.12",
installed_at: new Date().toISOString(), arch: "aarch64",
};
const result = checkInstallAllowed(state, SPARK_2935, "fedora-43");
expect(result.allowed).toBe(false);
if (result.allowed === false) {
expect(result.error).toContain("spark-2935");
expect(result.error).toContain("DGX OS");
expect(result.error).toContain("provision debug");
}
});
it("refuses a Spark that is only known by MAC", () => {
expect(checkInstallAllowed(emptyState(), SPARK_3A1C, "fedora-43").allowed).toBe(false);
});
it("allows an ordinary discovered machine", () => {
const state = emptyState();
state.discovered[ORDINARY] = hardware(ORDINARY);
expect(checkInstallAllowed(state, ORDINARY, "fedora-43").allowed).toBe(true);
});
it("allows Fedora on aarch64", () => {
const state = emptyState();
state.discovered[ORDINARY] = hardware(ORDINARY, { arch: "aarch64" });
expect(checkInstallAllowed(state, ORDINARY, "fedora-43").allowed).toBe(true);
});
it("refuses Ubuntu on aarch64 -- no netboot artifacts are published", () => {
const state = emptyState();
state.discovered[ORDINARY] = hardware(ORDINARY, { arch: "aarch64" });
const result = checkInstallAllowed(state, ORDINARY, "ubuntu-26.04");
expect(result.allowed).toBe(false);
if (result.allowed === false) {
expect(result.error).toContain("aarch64");
}
});
it("allows Ubuntu on x86_64", () => {
const state = emptyState();
state.discovered[ORDINARY] = hardware(ORDINARY, { arch: "x86_64" });
expect(checkInstallAllowed(state, ORDINARY, "ubuntu-26.04").allowed).toBe(true);
});
});
describe("install route enforces the guard", () => {
let testDir: string;
let app: FastifyInstance;
let state: StateManager;
beforeEach(() => {
testDir = join(tmpdir(), `bastion-guard-test-${Date.now()}-${Math.random().toString(36).slice(2)}`);
mkdirSync(join(testDir, "http"), { recursive: true });
mkdirSync(join(testDir, "tftp"), { recursive: true });
const result = createApp(createTestConfig(testDir));
app = result.app;
state = result.state;
});
afterEach(async () => {
await app.close();
rmSync(testDir, { recursive: true, force: true });
});
it("rejects POST /api/install for a Spark and queues nothing", async () => {
const res = await app.inject({
method: "POST",
url: "/api/install",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ mac: SPARK_2935, hostname: "spark-2935", role: "worker" }),
});
expect(res.statusCode).toBe(409);
expect(JSON.parse(res.body).error).toContain("Refusing to install");
expect(state.load().install_queue[SPARK_2935]).toBeUndefined();
});
it("still serves rescue to a Spark -- debug is never guarded", async () => {
state.update((s) => {
s.installed[SPARK_2935] = {
hostname: "spark-2935", role: "worker", ip: "192.168.8.12",
installed_at: new Date().toISOString(), arch: "aarch64",
};
s.debug[SPARK_2935] = { hostname: "spark-2935", queued_at: new Date().toISOString() };
});
const res = await app.inject({ method: "GET", url: `/dispatch?mac=${SPARK_2935}` });
expect(res.statusCode).toBe(200);
expect(res.body).toContain("DEBUG/RESCUE MODE");
expect(res.body).toContain("/vmlinuz-aarch64");
});
it("a Spark that PXE boots unqueued gets discovery, never an install", async () => {
const res = await app.inject({ method: "GET", url: `/dispatch?mac=${SPARK_2935}&arch=arm64` });
expect(res.body).toContain("DISCOVERY MODE");
expect(res.body).not.toContain("INSTALLING");
});
it("still accepts an ordinary machine", async () => {
state.update((s) => { s.discovered[ORDINARY] = hardware(ORDINARY); });
const res = await app.inject({
method: "POST",
url: "/api/install",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ mac: ORDINARY, hostname: "worker-1", role: "worker" }),
});
expect(res.statusCode).toBe(200);
expect(state.load().install_queue[ORDINARY]).toBeDefined();
});
});

89
bastion/src/bastion/tests/ipxe-x86-regression.test.ts Normal file
View File

@@ -0,0 +1,89 @@
// x86_64 iPXE output regression gate.
//
// The aarch64 PXE work must not change what an x86_64 machine is served. The golden
// fixture was dumped from the templates as they stood before that work started, so
// any diff here is a regression, not an improvement.
//
// The one deliberate exception is renderBootIpxe: its chain URL gained
// `&arch=${buildarch}` so the dispatch endpoint can observe the client's
// architecture at boot time. That single change is asserted explicitly below
// rather than being allowed to slip through the byte-for-byte comparison.
import { describe, it, expect } from "vitest";
import { readFileSync } from "node:fs";
import { join } from "node:path";
import { fileURLToPath } from "node:url";
import { dirname } from "node:path";
import {
renderBootIpxe,
renderDiscoverIpxe,
renderInstallIpxe,
renderDebugIpxe,
renderPxeBootDebugIpxe,
renderLocalBootIpxe,
} from "../src/templates/boot.ipxe.js";
const here = dirname(fileURLToPath(import.meta.url));
const golden = JSON.parse(
readFileSync(join(here, "fixtures", "ipxe-x86_64-golden.json"), "utf-8"),
) as Record<string, string>;
// Exactly the parameters used to dump the fixture.
const serverIp = "10.0.0.1";
const httpPort = 8080;
const mac = "aa:bb:cc:dd:ee:ff";
const hostname = "worker-1";
const fedoraVersion = "43";
const fedoraMirror =
"https://download.fedoraproject.org/pub/fedora/linux/releases/43/Everything/x86_64/os";
// The x86_64 LVM layout the fixture was captured with. Before this work the values
// were hardcoded in the template; they are now supplied by the caller from machine
// state, so the fixture pins the rendering, not the defaults.
const x86Root = {
rootDevice: "/dev/mapper/labvg-root",
rootArgs: "rd.lvm.lv=labvg/root rd.lvm.lv=labvg/swap",
};
describe("x86_64 iPXE output is unchanged", () => {
it("discover script is byte-identical", () => {
const rendered = renderDiscoverIpxe({
mac, serverIp, httpPort, fedoraMirror, arch: "x86_64",
});
expect(rendered).toBe(golden["discover"]);
});
it("install script is byte-identical", () => {
const rendered = renderInstallIpxe({
mac, hostname, serverIp, httpPort, fedoraVersion, fedoraMirror, arch: "x86_64",
});
expect(rendered).toBe(golden["install"]);
});
it("debug/rescue script is byte-identical", () => {
const rendered = renderDebugIpxe({
mac, hostname, serverIp, httpPort, fedoraMirror, arch: "x86_64",
});
expect(rendered).toBe(golden["debug"]);
});
it("--pxe-boot script is byte-identical when state carries the Fedora LVM layout", () => {
const rendered = renderPxeBootDebugIpxe({
mac, hostname, serverIp, httpPort, arch: "x86_64", ...x86Root,
});
expect(rendered).toBe(golden["pxeBoot"]);
});
it("local boot script is byte-identical", () => {
expect(renderLocalBootIpxe(hostname)).toBe(golden["localBoot"]);
});
it("boot.ipxe differs only by the &arch= chain parameter", () => {
const rendered = renderBootIpxe({ serverIp, httpPort });
// The sole intended difference.
expect(rendered).toBe(golden["boot"].replace(
"/dispatch?mac=${net0/mac}",
"/dispatch?mac=${net0/mac}&arch=${buildarch}",
));
});
});

1
bastion/src/cli/src/api/client.ts
View File

@@ -110,6 +110,7 @@ export class LabdClient {
memory_gb?: number; arch?: string; memory_gb?: number; arch?: string;
disks?: Array<{ name: string; size_gb: number; model: string }>; disks?: Array<{ name: string; size_gb: number; model: string }>;
nics?: Array<{ name: string; mac: string; state: string }>; nics?: Array<{ name: string; mac: string; state: string }>;
root_device?: string; root_args?: string;
}): Promise<{ status: string; error?: string }> { }): Promise<{ status: string; error?: string }> {
return this.request("POST", "/api/machines/discover", { body: data }); return this.request("POST", "/api/machines/discover", { body: data });
} }

105
bastion/src/cli/src/commands/debug.ts
View File

@@ -8,6 +8,7 @@ import { join } from "node:path";
import { Command } from "commander"; import { Command } from "commander";
import type { BastionState } from "@lab/shared"; import type { BastionState } from "@lab/shared";
import { getLabdClient } from "../api/config.js"; import { getLabdClient } from "../api/config.js";
import { ROOT_DEVICE_PROBE, parseRootProbe } from "../utils/hardware-probe.js";
/** Resolve a target (hostname, MAC, or IP) to {mac, hostname, ip} from state. */ /** Resolve a target (hostname, MAC, or IP) to {mac, hostname, ip} from state. */
function resolveTarget( function resolveTarget(
@@ -44,6 +45,54 @@ function resolveTarget(
return null; return null;
} }
/** The local admin account to SSH as (root is not usable — it has no key here). */
function sshUser(): string {
const adminUser = process.env["SUDO_USER"] ?? process.env["USER"] ?? "";
return adminUser === "root" ? "" : adminUser;
}
/** Common ssh arguments, ending with user@host. Null when there is no usable user. */
function sshBaseArgs(ip: string): string[] | null {
const user = sshUser();
if (user === "") return null;
const sudoUser = process.env["SUDO_USER"];
const realHome = sudoUser !== undefined ? join("/home", sudoUser) : homedir();
const sshKey = ["id_ed25519", "id_rsa", "id_ecdsa"]
.map((name) => join(realHome, ".ssh", name))
.find((k) => existsSync(k));
return [
"-o", "StrictHostKeyChecking=no",
"-o", "UserKnownHostsFile=/dev/null",
"-o", "ConnectTimeout=10",
...(sshKey !== undefined ? ["-i", sshKey] : []),
`${user}@${ip}`,
];
}
/**
* Run a shell script on the target as root and return its stdout, or null.
*
* The script goes over stdin rather than the command line so it can contain quotes
* without a second round of shell escaping. `sudo -n` fails fast instead of hanging on
* a password prompt that would then eat the script.
*/
function sshCapture(ip: string, script: string): string | null {
const base = sshBaseArgs(ip);
if (base === null) return null;
try {
return execFileSync("ssh", [...base, "sudo", "-n", "sh", "-s"], {
input: script,
encoding: "utf-8",
stdio: ["pipe", "pipe", "pipe"],
timeout: 30_000,
});
} catch {
return null;
}
}
export function registerDebugCommand(parent: Command): void { export function registerDebugCommand(parent: Command): void {
parent parent
.command("debug <target>") .command("debug <target>")
@@ -71,6 +120,31 @@ export function registerDebugCommand(parent: Command): void {
} }
const { mac, hostname, ip } = resolved; const { mac, hostname, ip } = resolved;
// --pxe-boot needs a root= for the installed system. If the machine is still
// reachable, observe it now rather than assuming a disk layout: a wrong root=
// leaves the machine unbootable. If it isn't reachable, dispatch falls back to
// rescue and the operator reports the real one from there.
if (opts.pxeBoot === true && ip !== "") {
const known = state.installed[mac]?.root_device ?? state.discovered[mac]?.root_device;
if (known === undefined || known === "") {
console.log(`No root device recorded for ${hostname}. Probing over SSH...`);
const probe = sshCapture(ip, ROOT_DEVICE_PROBE);
const root = probe === null ? {} : parseRootProbe(probe);
if (root.root_device !== undefined) {
console.log(` root=${root.root_device}${root.root_args !== undefined ? ` ${root.root_args}` : ""}`);
try {
await client.discoverMachine({ mac, ...root });
} catch (err) {
console.error(` Could not record it: ${err instanceof Error ? err.message : String(err)}`);
}
} else {
console.log(" Probe failed. Booting rescue instead; report the root device with:");
console.log(" curl http://<bastion>:8080/debug-setup.sh | bash");
}
}
}
console.log(`Queuing debug mode for ${hostname} (${mac})...`); console.log(`Queuing debug mode for ${hostname} (${mac})...`);
try { try {
@@ -86,32 +160,15 @@ export function registerDebugCommand(parent: Command): void {
// Try SSH reboot into PXE // Try SSH reboot into PXE
if (ip !== "") { if (ip !== "") {
const adminUser = process.env["SUDO_USER"] ?? process.env["USER"] ?? ""; const base = sshBaseArgs(ip);
const effectiveUser = adminUser === "root" ? "" : adminUser; if (base !== null) {
console.log(`\nAttempting SSH reboot into PXE (${sshUser()}@${ip})...`);
if (effectiveUser !== "") {
console.log(`\nAttempting SSH reboot into PXE (${effectiveUser}@${ip})...`);
const sudoUser = process.env["SUDO_USER"];
const realHome = sudoUser !== undefined ? join("/home", sudoUser) : homedir();
const keyPaths = [
join(realHome, ".ssh", "id_ed25519"),
join(realHome, ".ssh", "id_rsa"),
join(realHome, ".ssh", "id_ecdsa"),
];
const sshKey = keyPaths.find(k => existsSync(k));
const sshArgs = [
"-o", "StrictHostKeyChecking=no",
"-o", "UserKnownHostsFile=/dev/null",
"-o", "ConnectTimeout=10",
...(sshKey !== undefined ? ["-i", sshKey] : []),
`${effectiveUser}@${ip}`,
'PXE_ENTRY=$(sudo efibootmgr | grep -iE "pxe|network|ipv4" | head -1 | grep -oP "Boot\\K[0-9A-F]+"); if [ -n "$PXE_ENTRY" ]; then sudo efibootmgr --bootnext "$PXE_ENTRY" && echo "PXE set as next boot" && sudo reboot; else echo "No PXE boot entry found, rebooting anyway..." && sudo reboot; fi',
];
try { try {
execFileSync("ssh", sshArgs, { stdio: "inherit" }); execFileSync("ssh", [
...base,
'PXE_ENTRY=$(sudo efibootmgr | grep -iE "pxe|network|ipv4" | head -1 | grep -oP "Boot\\K[0-9A-F]+"); if [ -n "$PXE_ENTRY" ]; then sudo efibootmgr --bootnext "$PXE_ENTRY" && echo "PXE set as next boot" && sudo reboot; else echo "No PXE boot entry found, rebooting anyway..." && sudo reboot; fi',
], { stdio: "inherit" });
} catch { } catch {
// SSH connection closing during reboot is expected // SSH connection closing during reboot is expected
} }

10
bastion/src/cli/src/commands/recheck.ts
View File

@@ -4,6 +4,7 @@
import type { Command } from "commander"; import type { Command } from "commander";
import { sshExec } from "@lab/modules"; import { sshExec } from "@lab/modules";
import { getLabdClient } from "../api/config.js"; import { getLabdClient } from "../api/config.js";
import { ROOT_DEVICE_PROBE } from "../utils/hardware-probe.js";
const BOLD = "\x1b[1m"; const BOLD = "\x1b[1m";
const GREEN = "\x1b[0;32m"; const GREEN = "\x1b[0;32m";
@@ -24,7 +25,9 @@ const HW_COLLECT_SCRIPT = [
'N=$(grep -c "^processor" /proc/cpuinfo 2>/dev/null || echo 0)', 'N=$(grep -c "^processor" /proc/cpuinfo 2>/dev/null || echo 0)',
'R=$(awk "/MemTotal/ {printf \\"%d\\", \\$2/1024/1024}" /proc/meminfo 2>/dev/null || echo 0)', 'R=$(awk "/MemTotal/ {printf \\"%d\\", \\$2/1024/1024}" /proc/meminfo 2>/dev/null || echo 0)',
'A=$(uname -m)', 'A=$(uname -m)',
'printf \'{"product":"%s","board":"%s","serial":"%s","manufacturer":"%s","cpu_model":"%s","cpu_cores":%s,"memory_gb":%s,"arch":"%s"}\\n\' "$P" "$B" "$S" "$M" "$C" "$N" "$R" "$A"', // Root filesystem, so --pxe-boot has a root= to use instead of assuming our layout.
ROOT_DEVICE_PROBE,
'printf \'{"product":"%s","board":"%s","serial":"%s","manufacturer":"%s","cpu_model":"%s","cpu_cores":%s,"memory_gb":%s,"arch":"%s","root_device":"%s","root_args":"%s"}\\n\' "$P" "$B" "$S" "$M" "$C" "$N" "$R" "$A" "$RD" "$RA"',
].join("; "); ].join("; ");
export function registerRecheckCommand(parent: Command): void { export function registerRecheckCommand(parent: Command): void {
@@ -81,7 +84,10 @@ export function registerRecheckCommand(parent: Command): void {
const cpu = hwData.cpu_model || "?"; const cpu = hwData.cpu_model || "?";
const cores = hwData.cpu_cores || "?"; const cores = hwData.cpu_cores || "?";
const mem = hwData.memory_gb || "?"; const mem = hwData.memory_gb || "?";
console.log(`${GREEN}OK${RESET} ${DIM}${cpu}, ${cores} cores, ${mem}GB${RESET}`); const root = typeof hwData.root_device === "string" && hwData.root_device !== ""
? `, root=${hwData.root_device}`
: "";
console.log(`${GREEN}OK${RESET} ${DIM}${cpu}, ${cores} cores, ${mem}GB${root}${RESET}`);
updated++; updated++;
} catch (err) { } catch (err) {
console.log(`${RED}FAIL${RESET} ${DIM}${err instanceof Error ? err.message : String(err)}${RESET}`); console.log(`${RED}FAIL${RESET} ${DIM}${err instanceof Error ? err.message : String(err)}${RESET}`);

59
bastion/src/cli/src/utils/hardware-probe.ts Normal file
View File

@@ -0,0 +1,59 @@
// Shell snippets for observing a machine's hardware over SSH.
//
// Pure shell + awk, no Python: these run on whatever the target happens to be,
// including a minimal rescue environment.
/**
* Report the root filesystem and any dracut arguments needed to assemble it.
*
* Emits two lines:
* ROOT_DEVICE=<device>
* ROOT_ARGS=<args>
*
* Used by `--pxe-boot`, which boots the installed system with a kernel and initrd from
* the network. Getting root= wrong there leaves the machine unbootable, so this observes
* the machine rather than assuming our Fedora LVM layout.
*
* Device form is chosen for stability across reboots: LVM logical volumes keep their
* /dev/mapper path, anything else is reported by UUID, which survives device renumbering.
*/
export const ROOT_DEVICE_PROBE = [
'RD=$(findmnt -no SOURCE / 2>/dev/null | head -1)',
'RA=""',
'RT=$(lsblk -no TYPE "$RD" 2>/dev/null | head -1)',
'if [ "$RT" = "lvm" ]; then',
' VGLV=$(lvs --noheadings -o vg_name,lv_name "$RD" 2>/dev/null | awk \'{print $1"/"$2}\')',
' [ -n "$VGLV" ] && RA="rd.lvm.lv=$VGLV"',
// Swap must be assembled too or resume= stalls the boot waiting for it.
' SW=$(awk \'NR>1 {print $1; exit}\' /proc/swaps 2>/dev/null)',
' if [ -n "$SW" ] && [ "$(lsblk -no TYPE "$SW" 2>/dev/null | head -1)" = "lvm" ]; then',
' SWVGLV=$(lvs --noheadings -o vg_name,lv_name "$SW" 2>/dev/null | awk \'{print $1"/"$2}\')',
' [ -n "$SWVGLV" ] && [ "$SWVGLV" != "$VGLV" ] && RA="$RA rd.lvm.lv=$SWVGLV"',
' fi',
'elif [ -n "$RD" ]; then',
' U=$(findmnt -no UUID / 2>/dev/null | head -1)',
' [ -n "$U" ] && RD="UUID=$U"',
'fi',
'printf \'ROOT_DEVICE=%s\\nROOT_ARGS=%s\\n\' "$RD" "$RA"',
].join("; ");
export interface RootInfo {
root_device?: string;
root_args?: string;
}
/** Parse the ROOT_DEVICE/ROOT_ARGS lines emitted by ROOT_DEVICE_PROBE. */
export function parseRootProbe(stdout: string): RootInfo {
const out: RootInfo = {};
for (const line of stdout.split("\n")) {
const trimmed = line.trim();
if (trimmed.startsWith("ROOT_DEVICE=")) {
const v = trimmed.slice("ROOT_DEVICE=".length).trim();
if (v !== "") out.root_device = v;
} else if (trimmed.startsWith("ROOT_ARGS=")) {
const v = trimmed.slice("ROOT_ARGS=".length).trim();
if (v !== "") out.root_args = v;
}
}
return out;
}

1
bastion/src/labd/src/routes/bastions.ts
View File

@@ -299,6 +299,7 @@ export function registerBastionRoutes(app: FastifyInstance, db: DbClient): void
memory_gb?: number; arch?: string; memory_gb?: number; arch?: string;
disks?: Array<{ name: string; size_gb: number; model: string }>; disks?: Array<{ name: string; size_gb: number; model: string }>;
nics?: Array<{ name: string; mac: string; state: string }>; nics?: Array<{ name: string; mac: string; state: string }>;
root_device?: string; root_args?: string;
}; };
}>("/api/machines/discover", async (request, reply) => { }>("/api/machines/discover", async (request, reply) => {
const data = request.body ?? {}; const data = request.body ?? {};

154
bastion/src/shared/src/hardware/index.ts Normal file
View File

@@ -0,0 +1,154 @@
// Architecture normalisation and machine classification.
//
// Both are derived from what the system already observes about a machine -- never from
// an operator-supplied flag.
import type { Arch, HardwareInfo, OnboardMethod, OsId } from "../types/index.js";
export const SUPPORTED_ARCHES: readonly Arch[] = ["x86_64", "aarch64"] as const;
/**
* Normalise an architecture string to one we serve boot artifacts for.
*
* Sources and their spellings:
* uname -m -> "x86_64" / "aarch64"
* iPXE ${buildarch}-> "x86_64" / "arm64"
* dpkg/Debian -> "amd64" / "arm64"
*
* Returns undefined for anything we don't serve, so callers fall back rather than
* inventing a kernel path that would 404.
*/
export function normalizeArch(value: string | undefined | null): Arch | undefined {
switch ((value ?? "").trim().toLowerCase()) {
case "x86_64":
case "x86-64":
case "amd64":
return "x86_64";
case "aarch64":
case "arm64":
return "aarch64";
default:
return undefined;
}
}
/** Fedora pxeboot artifact base URL for an architecture. */
export function fedoraMirrorFor(fedoraVersion: string, arch: Arch): string {
return `https://download.fedoraproject.org/pub/fedora/linux/releases/${fedoraVersion}/Everything/${arch}/os`;
}
/**
* Which architectures each OS in the pipeline can actually be installed on.
*
* Fedora publishes pxeboot vmlinuz/initrd for both. Ubuntu does not: as of 26.04,
* releases.ubuntu.com publishes amd64 artifacts only, so there is nothing to netboot an
* arm64 machine with. Claiming support would fail at download time with a 404 instead
* of a useful message.
*/
const OS_ARCH_SUPPORT: Record<OsId, readonly Arch[]> = {
"fedora-43": ["x86_64", "aarch64"],
"ubuntu-26.04": ["x86_64"],
};
export function osSupportsArch(os: OsId, arch: Arch): boolean {
return (OS_ARCH_SUPPORT[os] ?? []).includes(arch);
}
export function archesForOs(os: OsId): readonly Arch[] {
return OS_ARCH_SUPPORT[os] ?? [];
}
/**
* Machines that run a vendor OS we have no image for.
*
* These are SSH-onboard: we manage userspace, but reinstalling destroys a driver and
* firmware stack our pipeline cannot rebuild. Matched on DMI identity, which is what
* discovery and `provision recheck` both collect.
*
* This is deliberately a property of the machine ("it runs DGX OS"), not a blocklist
* ("never install this MAC"). When a DGX OS image joins the pipeline, teaching the
* installer about vendor_os "dgx-os" is what unblocks these machines -- no entry here
* needs deleting.
*/
interface VendorOsRule {
vendorOs: string;
description: string;
matches: (hw: DmiIdentity) => boolean;
}
interface DmiIdentity {
manufacturer: string;
product: string;
board: string;
}
const VENDOR_OS_RULES: readonly VendorOsRule[] = [
{
vendorOs: "dgx-os",
description: "NVIDIA DGX OS (proprietary driver + firmware stack, no image in our pipeline)",
matches: ({ manufacturer, product, board }) =>
(manufacturer.includes("nvidia") || product.includes("nvidia")) &&
(product.includes("dgx") || product.includes("spark") ||
board.includes("gb10") || product.includes("gb10")),
},
];
/**
* Machines known to run a vendor OS, by MAC.
*
* The DMI rules above only fire once discovery or `provision recheck` has populated a
* hardware record. Machines onboarded over SSH may sit in state for a long time with no
* DMI at all -- which is exactly the state both DGX Sparks are in today -- so a
* DMI-only classifier would fail open on the machines this guard exists to protect.
*
* This is a statement of fact about known hardware ("this box runs DGX OS"), not an
* install policy. Whether that means "refuse" is decided by whether the pipeline has an
* image for that vendor OS.
*/
const KNOWN_VENDOR_OS_MACS: Record<string, string> = {
"4c:bb:47:7f:29:35": "dgx-os", // spark-2935
"48:21:0b:96:3a:1c": "dgx-os", // spark-3a1c
};
/**
* Classify how a machine should be onboarded, from its hardware record.
*
* An explicit `onboard` already on the record wins: it may have been set by an operator
* or by a rule that has since changed, and silently overriding it would be worse than
* leaving it.
*/
export function classifyOnboard(
hw: Partial<Pick<HardwareInfo, "mac" | "manufacturer" | "product" | "board">>
& { onboard?: OnboardMethod; vendor_os?: string },
): { onboard: OnboardMethod; vendor_os?: string } {
if (hw.onboard !== undefined) {
return hw.vendor_os !== undefined
? { onboard: hw.onboard, vendor_os: hw.vendor_os }
: { onboard: hw.onboard };
}
const knownVendorOs = KNOWN_VENDOR_OS_MACS[(hw.mac ?? "").toLowerCase().replace(/-/g, ":")];
if (knownVendorOs !== undefined) {
return { onboard: "ssh", vendor_os: knownVendorOs };
}
const identity: DmiIdentity = {
manufacturer: (hw.manufacturer ?? "").toLowerCase(),
product: (hw.product ?? "").toLowerCase(),
board: (hw.board ?? "").toLowerCase(),
};
for (const rule of VENDOR_OS_RULES) {
if (rule.matches(identity)) {
return { onboard: "ssh", vendor_os: rule.vendorOs };
}
}
return { onboard: "pxe" };
}
/** Human-readable reason a vendor-OS machine must not be reinstalled. */
export function vendorOsDescription(vendorOs: string | undefined): string {
const rule = VENDOR_OS_RULES.find((r) => r.vendorOs === vendorOs);
return rule?.description ?? "a vendor OS with no image in our pipeline";
}

12
bastion/src/shared/src/index.ts
View File

@@ -1,6 +1,8 @@
export type { export type {
OsId, OsId,
Arch, Arch,
OnboardMethod,
RootCandidate,
Role, Role,
HardwareInfo, HardwareInfo,
InstallConfig, InstallConfig,
@@ -10,6 +12,16 @@ export type {
BastionConfig, BastionConfig,
} from "./types/index.js"; } from "./types/index.js";
export {
SUPPORTED_ARCHES,
normalizeArch,
fedoraMirrorFor,
osSupportsArch,
archesForOs,
classifyOnboard,
vendorOsDescription,
} from "./hardware/index.js";
export { SUPPORTED_OS, SUPPORTED_ROLES, ROLE_REGISTRY, isValidOsId } from "./types/index.js"; export { SUPPORTED_OS, SUPPORTED_ROLES, ROLE_REGISTRY, isValidOsId } from "./types/index.js";
export type { RoleInfo } from "./types/index.js"; export type { RoleInfo } from "./types/index.js";

2
bastion/src/shared/src/protocol/index.ts
View File

@@ -113,7 +113,7 @@ export type LabdBastionMessage =
| { type: "command-role-update"; requestId: string; mac: string; role: string } | { type: "command-role-update"; requestId: string; mac: string; role: string }
| { type: "command-debug"; requestId: string; mac: string; pxeBoot?: boolean } | { type: "command-debug"; requestId: string; mac: string; pxeBoot?: boolean }
| { type: "command-register"; requestId: string; mac: string; hostname: string; role: string; ip: string } | { type: "command-register"; requestId: string; mac: string; hostname: string; role: string; ip: string }
| { type: "command-discover"; requestId: string; mac: string; product?: string; board?: string; serial?: string; manufacturer?: string; cpu_model?: string; cpu_cores?: number; memory_gb?: number; arch?: string; disks?: Array<{ name: string; size_gb: number; model: string }>; nics?: Array<{ name: string; mac: string; state: string }> } | { type: "command-discover"; requestId: string; mac: string; product?: string; board?: string; serial?: string; manufacturer?: string; cpu_model?: string; cpu_cores?: number; memory_gb?: number; arch?: string; disks?: Array<{ name: string; size_gb: number; model: string }>; nics?: Array<{ name: string; mac: string; state: string }>; root_device?: string; root_args?: string }
| { type: "server-shutdown"; reconnectAfter: number }; | { type: "server-shutdown"; reconnectAfter: number };
export type BastionMessageType = BastionMessage["type"]; export type BastionMessageType = BastionMessage["type"];

2
bastion/src/shared/src/types/index.ts
View File

@@ -1,6 +1,8 @@
export type { export type {
OsId, OsId,
Arch, Arch,
OnboardMethod,
RootCandidate,
Role, Role,
HardwareInfo, HardwareInfo,
InstallConfig, InstallConfig,

32
bastion/src/shared/src/types/state.ts
View File

@@ -11,6 +11,16 @@ export function isValidOsId(value: string): value is OsId {
return (SUPPORTED_OS as readonly string[]).includes(value); return (SUPPORTED_OS as readonly string[]).includes(value);
} }
/**
* How a machine joins the lab.
*
* "pxe" -- bare metal we install over the network (the default).
* "ssh" -- the machine already runs a vendor OS we cannot reproduce, so we onboard
* over SSH and manage userspace only. Installing would destroy that OS.
* See classifyOnboard() and os-install-research.md.
*/
export type OnboardMethod = "pxe" | "ssh";
export interface HardwareInfo { export interface HardwareInfo {
mac: string; mac: string;
product: string; product: string;
@@ -26,6 +36,23 @@ export interface HardwareInfo {
first_seen: string; first_seen: string;
last_seen: string; last_seen: string;
bastionId?: string; // set when aggregated through labd bastionId?: string; // set when aggregated through labd
// Onboarding classification -- absent means "pxe" (see classifyOnboard)
onboard?: OnboardMethod;
vendor_os?: string; // e.g. "dgx-os": the OS this machine must keep running
// Root filesystem, for booting the installed system over PXE (--pxe-boot).
// Observed from the machine, never assumed.
root_device?: string; // e.g. "/dev/mapper/labvg-root"
root_args?: string; // e.g. "rd.lvm.lv=labvg/root rd.lvm.lv=labvg/swap"
root_candidates?: RootCandidate[]; // reported from a rescue shell when unknown
}
/** A possible root filesystem found while probing an unreachable machine. */
export interface RootCandidate {
device: string; // e.g. "/dev/mapper/labvg-root"
args?: string; // extra dracut args needed to assemble it
fstype?: string;
size_gb?: number;
os_release?: string; // PRETTY_NAME from /etc/os-release, if mountable
} }
export type Role = "vanilla" | "worker" | "infra" | "labcontroller"; export type Role = "vanilla" | "worker" | "infra" | "labcontroller";
@@ -103,6 +130,11 @@ export interface InstalledInfo {
cpu_cores?: number; cpu_cores?: number;
memory_gb?: number; memory_gb?: number;
arch?: string; arch?: string;
onboard?: OnboardMethod;
vendor_os?: string;
root_device?: string;
root_args?: string;
root_candidates?: RootCandidate[];
} }
export interface DebugConfig { export interface DebugConfig {

537
bastion/tests/integration/arm-pxe-provision.test.ts Normal file
View File

@@ -0,0 +1,537 @@
// Integration test: aarch64 network PXE boot.
//
// The boot-ISO path already covered ARM (arm-iso-provision.test.ts). This covers the
// network path: DHCP option 93 handing an arm64 client an arm64 iPXE binary, dispatch
// serving an aarch64 kernel, and `provision debug` reaching a rescue shell -- which is
// what the DGX Sparks actually need and could not do.
//
// Two suites, because they cost very different amounts of time:
//
// "ARM PXE rescue" NBP handoff -> rescue with SSH. ~25-30 min
// "ARM PXE install" discover -> install -> installed. ~75-95 min
//
// The rescue suite seeds the machine into state as an already-known aarch64 box rather
// than discovering it first. That is the DGX Spark situation exactly -- SSH-onboarded,
// never PXE-discovered, architecture known only from its record -- and it holds the test
// to one emulated boot. Each boot spends ~15 of its ~18 minutes downloading Anaconda's
// stage2 under TCG, so discovering first would double the runtime without touching any
// code path the rescue boot does not already exercise.
//
// The install suite only runs with ARM_PXE_FULL=1. No ARM machine in the lab is ever
// PXE-installed except the MS-R1, and an hour-plus test that runs by default is a test
// nobody runs.
//
// IMPORTANT: aarch64 has no KVM on an x86_64 host, so all of this is emulated and
// roughly 10x slower than native.
//
// A note for whoever debugs a failure here: if the VM panics with
// VFS: Unable to mount root fs on unknown-block(0,0)
// that is very likely iPXE silently dropping the initrd because the build lacks
// EFI_LOAD_FILE2_PROTOCOL -- on arm64 the kernel EFI stub fetches the initrd over
// LoadFile2, and an iPXE without it accepts the `initrd` line and does nothing. It is
// NOT a reproduction of the DGX Spark kernel bug that motivated this work, despite
// being the identical message. assertIpxeSupportsLoadFile2() below checks the build up
// front so that failure names itself; to check by hand:
// node -e 'const b=require("fs").readFileSync("/usr/share/ipxe/arm64-efi/snponly.efi");
// console.log(b.indexOf(Buffer.from("c1c00640b3fc3e40996d4a6c8724e06d","hex")))'
// Fedora's ipxe-bootimgs-aarch64-20240119 has it at 0x3bbf0.
//
// Prerequisites:
// - qemu-system-aarch64 (sudo dnf install qemu-system-aarch64)
// - edk2-aarch64 (sudo dnf install edk2-aarch64)
// - ipxe-bootimgs-aarch64 (sudo dnf install ipxe-bootimgs-aarch64)
// - libvirtd, sudo, internet access
//
// Run: sudo ./scripts/test-provision.sh arm-pxe
import { describe, it, expect, beforeAll, afterAll } from "vitest";
import { readFileSync, existsSync, mkdirSync, rmSync, copyFileSync, writeFileSync } from "node:fs";
import { execSync } from "node:child_process";
import { join } from "node:path";
import { homedir, tmpdir } from "node:os";
import { log, waitForSsh } from "./helpers/libvirt.js";
import { ensurePxeNetwork, destroyPxeNetwork, deleteNftablesRejectRules, PXE_NETWORK_NAME, PXE_GATEWAY, PXE_SUBNET } from "./helpers/pxe-network.js";
import { createPxeVm, destroyPxeVm, getVmMac, rebootPxeVm, readSerialLog } from "./helpers/pxe-vm.js";
import { sshExec } from "./helpers/ssh.js";
const IPXE_ARM64 = "/usr/share/ipxe/arm64-efi/snponly.efi";
const AAVMF = "/usr/share/edk2/aarch64/QEMU_EFI.fd";
const VM_MEMORY = 4096;
const VM_VCPUS = 2;
const VM_DISK_GB = 250;
const SSH_USER = "lab";
const BASTION_IP = PXE_GATEWAY;
const DHCP_RANGE_START = `${PXE_SUBNET}.100`;
const DHCP_RANGE_END = `${PXE_SUBNET}.200`;
const SERIAL_PORT = 4555;
// Emulated aarch64 -- generous timeouts throughout. Measured on an x86_64 host with no
// KVM for aarch64: a single PXE boot to a running Anaconda takes ~18 minutes, almost all
// of it downloading inst.stage2 over the network under TCG. Budget well above that;
// timing out just short of success wastes a whole run.
const LEASE_TIMEOUT_MS = 10 * 60_000;
const DISCOVERY_TIMEOUT_MS = 35 * 60_000;
const INSTALL_TIMEOUT_MS = 75 * 60_000;
const SSH_TIMEOUT_MS = 35 * 60_000;
const RUN_FULL_INSTALL = process.env["ARM_PXE_FULL"] === "1";
function sleep(ms: number): Promise<void> {
return new Promise((r) => setTimeout(r, ms));
}
function findSshKey(): { pubKey: string; keyPath: string } {
const candidates: string[] = [];
if (process.env["SSH_KEY_PATH"]) candidates.push(process.env["SSH_KEY_PATH"]);
const homes = [homedir()];
const sudoUser = process.env["SUDO_USER"];
if (sudoUser) homes.push(join("/home", sudoUser));
for (const home of homes) {
for (const name of ["id_ed25519", "id_ecdsa", "id_rsa"]) {
candidates.push(join(home, ".ssh", name));
}
}
for (const keyPath of candidates) {
if (existsSync(keyPath) && existsSync(`${keyPath}.pub`)) {
return { pubKey: readFileSync(`${keyPath}.pub`, "utf-8").trim(), keyPath };
}
}
throw new Error("No SSH key found — set SSH_KEY_PATH or ensure keys exist in ~/.ssh/");
}
async function pollApi<T>(
url: string,
check: (data: T) => boolean,
timeoutMs: number,
intervalMs = 10_000,
): Promise<T> {
const start = Date.now();
while (Date.now() - start < timeoutMs) {
try {
const res = await fetch(url);
if (res.ok) {
const data = (await res.json()) as T;
if (check(data)) return data;
}
} catch { /* bastion not up yet, or a network hiccup */ }
await sleep(intervalMs);
}
throw new Error(`Timeout after ${timeoutMs}ms polling ${url}`);
}
function requirePrerequisites(): void {
if (!existsSync("/usr/bin/qemu-system-aarch64")) {
throw new Error("qemu-system-aarch64 not installed. Run: sudo dnf install qemu-system-aarch64");
}
if (!existsSync(AAVMF)) {
throw new Error(`AAVMF firmware not found at ${AAVMF}. Run: sudo dnf install edk2-aarch64`);
}
if (!existsSync(IPXE_ARM64)) {
throw new Error(`arm64 iPXE not found at ${IPXE_ARM64}. Run: sudo dnf install ipxe-bootimgs-aarch64`);
}
}
/**
* Confirm the arm64 iPXE binary implements EFI_LOAD_FILE2_PROTOCOL.
*
* Without it the `initrd` line is accepted and silently ignored, and the kernel panics
* with unknown-block(0,0). Checking here turns a confusing 30-minute boot failure into
* an immediate, explanatory one.
*
* GUID 4006c0c1-fcb3-403e-996d-4a6c8724e06d, little-endian in the binary's GUID table.
*/
function assertIpxeSupportsLoadFile2(): void {
const LOAD_FILE2_GUID = Buffer.from("c1c00640b3fc3e40996d4a6c8724e06d", "hex");
const binary = readFileSync(IPXE_ARM64);
if (binary.indexOf(LOAD_FILE2_GUID) < 0) {
throw new Error(
`${IPXE_ARM64} does not reference EFI_LOAD_FILE2_PROTOCOL. On arm64 the kernel ` +
`EFI stub fetches the initrd over LoadFile2; without it iPXE drops the initrd ` +
`silently and the kernel panics with "unknown-block(0,0)". Rebuild iPXE with ` +
`LoadFile2, or chainload grubaa64.efi for aarch64 instead.`,
);
}
log(`iPXE arm64 implements LoadFile2 — initrd will be delivered to the EFI stub`);
}
interface Harness {
testDir: string;
app: { close: () => Promise<void> };
stopDnsmasq: () => void;
state: { update: (fn: (s: BastionStateLike) => void) => void };
vmMac: string;
httpPort: number;
}
/** Just the parts of BastionState this test seeds. */
interface BastionStateLike {
discovered: Record<string, Record<string, unknown>>;
installed: Record<string, Record<string, unknown>>;
install_queue: Record<string, Record<string, unknown>>;
debug: Record<string, Record<string, unknown>>;
}
/** Bring up an isolated network, a bastion with both arch payloads, and an arm64 VM. */
async function startHarness(vmName: string, httpPort: number, pubKey: string): Promise<Harness> {
requirePrerequisites();
assertIpxeSupportsLoadFile2();
log("Setting up PXE test network...");
ensurePxeNetwork();
const testDir = join(tmpdir(), `lab-arm-pxe-test-${Date.now()}`);
for (const sub of ["tftp", "http", "logs"]) {
mkdirSync(join(testDir, sub), { recursive: true });
}
const { createApp } = await import("../../src/bastion/src/server.js");
const { loadConfig } = await import("../../src/bastion/src/config.js");
const { generateDnsmasqConf, startDnsmasq, stopDnsmasq } = await import("../../src/bastion/src/services/dnsmasq.js");
const { generateDiscoverKickstart } = await import("../../src/bastion/src/services/kickstart-generator.js");
const { renderBootIpxe, kernelPath, initrdPath } = await import("../../src/bastion/src/templates/boot.ipxe.js");
// Relative, not "@lab/shared": these tests run from the repo root against sources,
// where the workspace package alias is not resolvable.
const { SUPPORTED_ARCHES, fedoraMirrorFor } = await import("../../src/shared/src/hardware/index.js");
const config = loadConfig({
bastionDir: testDir,
httpPort,
iface: "virbr-pxe",
serverIp: BASTION_IP,
network: `${PXE_SUBNET}.0`,
gateway: BASTION_IP,
dhcpMode: "full",
dhcpRangeStart: DHCP_RANGE_START,
dhcpRangeEnd: DHCP_RANGE_END,
domain: "arm-pxe-test.local",
sshKeys: [pubKey],
adminUser: SSH_USER,
});
// iPXE binaries. The arm64 one is the whole point: dnsmasq hands it out on DHCP
// option 93 -- 11 for UEFI PXE (TFTP) and 19 for UEFI HTTP Boot.
//
// They go in BOTH directories, exactly as main.ts stages them. AAVMF prefers HTTP
// Boot, so it is served an http:// URL and fetches from httpDir; a firmware that
// takes the TFTP path reads the same file from tftpDir. Staging only tftpDir gives a
// 404 and "No bootable option or device was found" on the console.
log("Staging iPXE binaries...");
const ipxeX86 = "/usr/share/ipxe/ipxe-snponly-x86_64.efi";
copyFileSync(IPXE_ARM64, join(config.tftpDir, "ipxe-arm64.efi"));
copyFileSync(IPXE_ARM64, join(config.httpDir, "ipxe-arm64.efi"));
if (existsSync(ipxeX86)) {
copyFileSync(ipxeX86, join(config.tftpDir, "ipxe.efi"));
copyFileSync(ipxeX86, join(config.httpDir, "ipxe.efi"));
}
// Fedora kernel + initrd for both architectures, cached across runs.
const cacheDir = "/var/lib/libvirt/images/lab-pxe-cache";
execSync(`mkdir -p "${cacheDir}"`, { stdio: "pipe" });
for (const arch of SUPPORTED_ARCHES) {
const mirror = fedoraMirrorFor(config.fedoraVersion, arch);
const kernelCache = join(cacheDir, `vmlinuz-${arch}`);
const initrdCache = join(cacheDir, `initrd-${arch}.img`);
if (!existsSync(kernelCache)) {
log(`Downloading Fedora ${config.fedoraVersion} ${arch} kernel...`);
execSync(`curl -# -L -f -o "${kernelCache}" "${mirror}/images/pxeboot/vmlinuz"`, { stdio: "inherit", timeout: 600_000 });
}
if (!existsSync(initrdCache)) {
log(`Downloading Fedora ${config.fedoraVersion} ${arch} initrd...`);
execSync(`curl -# -L -f -o "${initrdCache}" "${mirror}/images/pxeboot/initrd.img"`, { stdio: "inherit", timeout: 600_000 });
}
// Staged under the exact names the iPXE templates will ask for.
copyFileSync(kernelCache, join(config.httpDir, kernelPath(arch)));
copyFileSync(initrdCache, join(config.httpDir, initrdPath(arch)));
log(`Staged ${arch}: ${kernelPath(arch)} + ${initrdPath(arch)}`);
}
writeFileSync(join(config.httpDir, "discover.ks"), generateDiscoverKickstart(config));
writeFileSync(
join(config.httpDir, "boot.ipxe"),
renderBootIpxe({ serverIp: config.serverIp, httpPort: config.httpPort }),
);
generateDnsmasqConf(config);
const { app, state, syslog } = createApp(config);
await app.listen({ port: config.httpPort, host: "0.0.0.0" });
syslog.start();
log(`Bastion HTTP listening on :${config.httpPort}`);
log("Starting dnsmasq (full DHCP)...");
startDnsmasq(config).catch((err) => {
log(`dnsmasq failed: ${err instanceof Error ? err.message : String(err)}`);
});
await sleep(1500);
log("Creating aarch64 PXE VM (emulated — this is slow)...");
createPxeVm({
name: vmName,
memory: VM_MEMORY,
vcpus: VM_VCPUS,
diskSize: VM_DISK_GB,
network: PXE_NETWORK_NAME,
arch: "aarch64",
});
const vmMac = getVmMac(vmName);
if (!vmMac) throw new Error("Could not determine VM MAC address");
log(`ARM VM MAC: ${vmMac}`);
return {
testDir,
app,
stopDnsmasq,
state: state as unknown as Harness["state"],
vmMac,
httpPort: config.httpPort,
};
}
async function stopHarness(vmName: string, harness: Harness | undefined): Promise<void> {
// KEEP_VM=1 leaves the VM, network and bastion up so a failure can be inspected on
// the console. Emulated aarch64 runs cost half an hour; tearing the evidence down
// automatically means paying that again to see what happened.
if (process.env["KEEP_VM"] === "1") {
log(`KEEP_VM=1 — leaving ${vmName} running for inspection.`);
log(` console: sudo virsh screenshot ${vmName} /tmp/vm.ppm`);
log(` serial: socat - TCP:127.0.0.1:${SERIAL_PORT}`);
if (harness) log(` bastion: ${harness.testDir} (still serving on :${harness.httpPort})`);
log(` cleanup: sudo virsh destroy ${vmName}; sudo virsh undefine ${vmName} --remove-all-storage --nvram`);
return;
}
log("Cleaning up...");
if (harness) {
await harness.app.close().catch(() => {});
harness.stopDnsmasq();
}
destroyPxeVm(vmName);
destroyPxeNetwork();
if (harness) rmSync(harness.testDir, { recursive: true, force: true });
}
/** Read the DHCP lease the bastion handed a MAC. Rescue mode reports no IP itself. */
function leaseIpFor(testDir: string, mac: string): string | null {
const leaseFile = join(testDir, "dnsmasq.leases");
if (!existsSync(leaseFile)) return null;
for (const line of readFileSync(leaseFile, "utf-8").split("\n")) {
// <expiry> <mac> <ip> <hostname> <clientid>
const parts = line.trim().split(/\s+/);
if (parts.length >= 3 && parts[1]?.toLowerCase() === mac.toLowerCase()) {
return parts[2] ?? null;
}
}
return null;
}
async function waitForLease(testDir: string, mac: string, timeoutMs: number): Promise<string> {
const start = Date.now();
while (Date.now() - start < timeoutMs) {
const ip = leaseIpFor(testDir, mac);
if (ip !== null) return ip;
await sleep(5000);
}
throw new Error(`No DHCP lease for ${mac} within ${timeoutMs}ms`);
}
// ---------------------------------------------------------------------------
// Rescue path -- what the DGX Sparks need.
// ---------------------------------------------------------------------------
describe("ARM PXE rescue", () => {
const VM_NAME = "lab-arm-pxe-rescue";
const HTTP_PORT = 8096;
let harness: Harness | undefined;
let sshKeyPath: string;
let rescueIp: string;
beforeAll(async () => {
const { pubKey, keyPath } = findSshKey();
sshKeyPath = keyPath;
harness = await startHarness(VM_NAME, HTTP_PORT, pubKey);
const { testDir, vmMac, state } = harness;
// Seed the machine as an already-known aarch64 box queued for rescue. This is the
// DGX Spark situation exactly: SSH-onboarded, never PXE-discovered, architecture
// known only from its record -- and it also keeps the test to a SINGLE emulated
// boot. Each boot spends ~15 minutes pulling Anaconda's stage2 over the network
// under TCG, so discovering first and rescuing second doubles the runtime for no
// extra coverage of the path being tested. Discovery is covered by the full suite.
log(`Seeding ${vmMac} as a known aarch64 machine queued for rescue...`);
state.update((s) => {
s.discovered[vmMac] = {
mac: vmMac,
product: "Test ARM64 Machine",
board: "virt",
serial: "SN-ARM64",
manufacturer: "QEMU",
cpu_model: "cortex-a57",
cpu_cores: VM_VCPUS,
memory_gb: 4,
arch: "aarch64",
disks: [],
nics: [],
first_seen: new Date().toISOString(),
last_seen: new Date().toISOString(),
};
s.debug[vmMac] = { hostname: "arm-rescue-test", queued_at: new Date().toISOString() };
});
// Restart so the VM boots against the seeded state. createPxeVm already started it.
rebootPxeVm(VM_NAME);
await sleep(5_000);
deleteNftablesRejectRules();
// The whole chain now runs once: DHCP option 93 -> arm64 iPXE -> /boot.ipxe ->
// /dispatch (architecture from the record, not the query) -> aarch64 kernel +
// initrd -> Anaconda rescue -> sshd. Reaching a shell at all proves iPXE handed
// the initrd to the EFI stub over LoadFile2; without it the kernel panics first.
log("Waiting for the rescue environment's DHCP lease...");
rescueIp = await waitForLease(testDir, vmMac, LEASE_TIMEOUT_MS);
log(`Rescue IP: ${rescueIp}`);
log("Waiting for SSH into the rescue shell (started by inst.sshd)...");
log("(emulated aarch64 — Anaconda's stage2 download dominates; be patient)");
await waitForSsh(rescueIp, "root", SSH_TIMEOUT_MS, sshKeyPath).catch(async (err) => {
log("Rescue SSH timed out. Serial console:");
try {
log(await readSerialLog(SERIAL_PORT, { lastLines: 100, timeoutMs: 15_000 }));
} catch { /* console unavailable */ }
throw err;
});
log("ARM PXE rescue reached.");
}, LEASE_TIMEOUT_MS + SSH_TIMEOUT_MS + 300_000);
afterAll(async () => { await stopHarness(VM_NAME, harness); });
it("resolved the architecture from the machine record", async () => {
const res = await fetch(`http://${BASTION_IP}:${HTTP_PORT}/api/machines`);
const data = (await res.json()) as { discovered: Record<string, { arch: string }> };
expect(data.discovered[harness!.vmMac]?.arch).toBe("aarch64");
});
it("rescue shell is reachable over SSH and is aarch64", () => {
const result = sshExec(rescueIp, "root", "uname -m", { keyPath: sshKeyPath, timeout: 60_000 });
expect(result.exitCode).toBe(0);
expect(result.stdout.trim()).toBe("aarch64");
});
it("booted an initramfs — the LoadFile2 path worked", () => {
// If iPXE had dropped the initrd the kernel would never have reached userspace at
// all, but assert it explicitly so a regression names itself.
const result = sshExec(rescueIp, "root", "cat /proc/cmdline; ls /run/install", {
keyPath: sshKeyPath, timeout: 60_000,
});
expect(result.exitCode).toBe(0);
expect(result.stdout).toContain("inst.rescue");
});
it("rescue kernel came from the bastion over HTTP", () => {
const result = sshExec(rescueIp, "root", "cat /proc/cmdline", { keyPath: sshKeyPath, timeout: 60_000 });
expect(result.stdout).toContain(`${BASTION_IP}:${HTTP_PORT}`);
// arm64 gets serial console arguments, never nomodeset.
expect(result.stdout).toContain("console=ttyAMA0");
expect(result.stdout).not.toContain("nomodeset");
});
it("has LVM tools available for inspecting an installed system", () => {
const result = sshExec(rescueIp, "root", "command -v vgchange && command -v lsblk", {
keyPath: sshKeyPath, timeout: 60_000,
});
expect(result.exitCode).toBe(0);
});
});
// ---------------------------------------------------------------------------
// Full install -- opt-in, ~60-90 minutes emulated.
// ---------------------------------------------------------------------------
describe.runIf(RUN_FULL_INSTALL)("ARM PXE install", () => {
const VM_NAME = "lab-arm-pxe-install";
const HTTP_PORT = 8095;
let harness: Harness | undefined;
let sshKeyPath: string;
let vmIp: string;
beforeAll(async () => {
const { pubKey, keyPath } = findSshKey();
sshKeyPath = keyPath;
harness = await startHarness(VM_NAME, HTTP_PORT, pubKey);
const { vmMac } = harness;
log("Waiting for aarch64 discovery...");
await pollApi<{ discovered: Record<string, unknown> }>(
`http://${BASTION_IP}:${HTTP_PORT}/api/machines`,
(data) => vmMac in data.discovered,
DISCOVERY_TIMEOUT_MS,
);
log("Discovered. Queueing install...");
const installRes = await fetch(`http://${BASTION_IP}:${HTTP_PORT}/api/install`, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ mac: vmMac, hostname: VM_NAME, disk: "", role: "vanilla" }),
});
expect(installRes.status).toBe(200);
await sleep(30_000);
rebootPxeVm(VM_NAME);
log("Waiting for the emulated aarch64 install (60-90 min)...");
type LogsResponse = { status: string; progress: string; ip?: string };
const final = await pollApi<LogsResponse>(
`http://${BASTION_IP}:${HTTP_PORT}/api/logs/${encodeURIComponent(vmMac)}`,
(d) => d.status === "installed" || d.progress === "error",
INSTALL_TIMEOUT_MS,
30_000,
);
if (final.progress === "error") {
const logs = await (await fetch(`http://${BASTION_IP}:${HTTP_PORT}/api/logs/${encodeURIComponent(vmMac)}`)).json();
log(`ARM install FAILED: ${JSON.stringify(logs, null, 2)}`);
throw new Error("ARM PXE install failed — see logs above");
}
vmIp = final.ip ?? "";
log(`ARM install complete. IP: ${vmIp}`);
await sleep(30_000);
rebootPxeVm(VM_NAME);
await sleep(5_000);
deleteNftablesRejectRules();
await waitForSsh(vmIp, SSH_USER, SSH_TIMEOUT_MS, sshKeyPath);
}, DISCOVERY_TIMEOUT_MS + INSTALL_TIMEOUT_MS + SSH_TIMEOUT_MS + 600_000);
afterAll(async () => { await stopHarness(VM_NAME, harness); });
it("machine reached installed state", async () => {
const res = await fetch(`http://${BASTION_IP}:${HTTP_PORT}/api/machines`);
const data = (await res.json()) as { installed: Record<string, { hostname: string }> };
expect(data.installed[harness!.vmMac]?.hostname).toBe(VM_NAME);
});
it("installed system is aarch64", () => {
const result = sshExec(vmIp, SSH_USER, "uname -m", { keyPath: sshKeyPath, timeout: 60_000 });
expect(result.stdout.trim()).toBe("aarch64");
});
it("SSH works with the admin user", () => {
const result = sshExec(vmIp, SSH_USER, "whoami", { keyPath: sshKeyPath, timeout: 60_000 });
expect(result.stdout.trim()).toBe(SSH_USER);
});
it("LVM layout is correct", () => {
const result = sshExec(vmIp, SSH_USER, "sudo lvs labvg --noheadings -o lv_name", {
keyPath: sshKeyPath, timeout: 60_000,
});
expect(result.exitCode).toBe(0);
const lvs = result.stdout.trim().split("\n").map((l) => l.trim());
for (const expected of ["root", "var", "varlog", "swap", "home", "srv"]) {
expect(lvs).toContain(expected);
}
});
});

210
bastion/tests/integration/pxe-rescue.test.ts Normal file
View File

@@ -0,0 +1,210 @@
// Integration test: `labctl provision debug` -> Anaconda rescue with SSH, on x86_64.
//
// The rescue path had no test coverage on any architecture, which matters because it is
// the lab's recovery tool of last resort -- the thing you reach for when a machine will
// not boot. It runs here on x86_64 with KVM so it completes in minutes; the aarch64
// equivalent is the same code path with a different kernel, but is emulated and far too
// slow to iterate on.
//
// Run: sudo ./scripts/test-provision.sh rescue
import { describe, it, expect, beforeAll, afterAll } from "vitest";
import { readFileSync, existsSync, mkdirSync, rmSync, copyFileSync, writeFileSync } from "node:fs";
import { execSync } from "node:child_process";
import { join } from "node:path";
import { homedir, tmpdir } from "node:os";
import { log, waitForSsh } from "./helpers/libvirt.js";
import { ensurePxeNetwork, destroyPxeNetwork, deleteNftablesRejectRules, PXE_NETWORK_NAME, PXE_GATEWAY, PXE_SUBNET } from "./helpers/pxe-network.js";
import { createPxeVm, destroyPxeVm, getVmMac, rebootPxeVm, readSerialLog } from "./helpers/pxe-vm.js";
import { sshExec } from "./helpers/ssh.js";
const VM_NAME = "lab-pxe-rescue-test";
const HTTP_PORT = 8094;
const VM_MEMORY = 4096;
const VM_VCPUS = 4;
const VM_DISK_GB = 20;
const BASTION_IP = PXE_GATEWAY;
const SERIAL_PORT = 4555;
const LEASE_TIMEOUT_MS = 8 * 60_000;
const SSH_TIMEOUT_MS = 15 * 60_000;
function sleep(ms: number): Promise<void> {
return new Promise((r) => setTimeout(r, ms));
}
function findSshKey(): { pubKey: string; keyPath: string } {
const candidates: string[] = [];
if (process.env["SSH_KEY_PATH"]) candidates.push(process.env["SSH_KEY_PATH"]);
const homes = [homedir()];
const sudoUser = process.env["SUDO_USER"];
if (sudoUser) homes.push(join("/home", sudoUser));
for (const home of homes) {
for (const name of ["id_ed25519", "id_ecdsa", "id_rsa"]) candidates.push(join(home, ".ssh", name));
}
for (const keyPath of candidates) {
if (existsSync(keyPath) && existsSync(`${keyPath}.pub`)) {
return { pubKey: readFileSync(`${keyPath}.pub`, "utf-8").trim(), keyPath };
}
}
throw new Error("No SSH key found — set SSH_KEY_PATH or ensure keys exist in ~/.ssh/");
}
function leaseIpFor(testDir: string, mac: string): string | null {
const leaseFile = join(testDir, "dnsmasq.leases");
if (!existsSync(leaseFile)) return null;
for (const line of readFileSync(leaseFile, "utf-8").split("\n")) {
const parts = line.trim().split(/\s+/);
if (parts.length >= 3 && parts[1]?.toLowerCase() === mac.toLowerCase()) return parts[2] ?? null;
}
return null;
}
async function waitForLease(testDir: string, mac: string, timeoutMs: number): Promise<string> {
const start = Date.now();
while (Date.now() - start < timeoutMs) {
const ip = leaseIpFor(testDir, mac);
if (ip !== null) return ip;
await sleep(5000);
}
throw new Error(`No DHCP lease for ${mac} within ${timeoutMs}ms`);
}
// Suite name must not be a substring of "ARM PXE rescue" -- vitest -t matches
// substrings, so a looser name here would drag the emulated aarch64 suite in with it.
describe("x86 rescue boot", () => {
let app: { close: () => Promise<void> };
let stopDnsmasqFn: () => void;
let testDir: string;
let vmMac: string;
let rescueIp: string;
let sshKeyPath: string;
beforeAll(async () => {
const { pubKey, keyPath } = findSshKey();
sshKeyPath = keyPath;
log("Setting up PXE test network...");
ensurePxeNetwork();
testDir = join(tmpdir(), `lab-pxe-rescue-${Date.now()}`);
for (const sub of ["tftp", "http", "logs"]) mkdirSync(join(testDir, sub), { recursive: true });
const { createApp } = await import("../../src/bastion/src/server.js");
const { loadConfig } = await import("../../src/bastion/src/config.js");
const { generateDnsmasqConf, startDnsmasq, stopDnsmasq } = await import("../../src/bastion/src/services/dnsmasq.js");
const { renderBootIpxe, kernelPath, initrdPath } = await import("../../src/bastion/src/templates/boot.ipxe.js");
stopDnsmasqFn = stopDnsmasq;
const config = loadConfig({
bastionDir: testDir,
httpPort: HTTP_PORT,
iface: "virbr-pxe",
serverIp: BASTION_IP,
network: `${PXE_SUBNET}.0`,
gateway: BASTION_IP,
dhcpMode: "full",
dhcpRangeStart: `${PXE_SUBNET}.100`,
dhcpRangeEnd: `${PXE_SUBNET}.200`,
domain: "rescue-test.local",
sshKeys: [pubKey],
adminUser: "lab",
});
// iPXE in both dirs: TFTP PXE and UEFI HTTP Boot are both possible, and OVMF picks.
const ipxeX86 = "/usr/share/ipxe/ipxe-snponly-x86_64.efi";
if (!existsSync(ipxeX86)) throw new Error(`iPXE not found: ${ipxeX86}`);
copyFileSync(ipxeX86, join(config.tftpDir, "ipxe.efi"));
copyFileSync(ipxeX86, join(config.httpDir, "ipxe.efi"));
const cacheDir = "/var/lib/libvirt/images/lab-pxe-cache";
execSync(`mkdir -p "${cacheDir}"`, { stdio: "pipe" });
const kernelCache = join(cacheDir, "vmlinuz-x86_64");
const initrdCache = join(cacheDir, "initrd-x86_64.img");
if (!existsSync(kernelCache)) {
log("Downloading Fedora x86_64 kernel...");
execSync(`curl -# -L -f -o "${kernelCache}" "${config.fedoraMirror}/images/pxeboot/vmlinuz"`, { stdio: "inherit", timeout: 600_000 });
}
if (!existsSync(initrdCache)) {
log("Downloading Fedora x86_64 initrd...");
execSync(`curl -# -L -f -o "${initrdCache}" "${config.fedoraMirror}/images/pxeboot/initrd.img"`, { stdio: "inherit", timeout: 600_000 });
}
copyFileSync(kernelCache, join(config.httpDir, kernelPath("x86_64")));
copyFileSync(initrdCache, join(config.httpDir, initrdPath("x86_64")));
writeFileSync(join(config.httpDir, "boot.ipxe"), renderBootIpxe({ serverIp: config.serverIp, httpPort: config.httpPort }));
generateDnsmasqConf(config);
const { app: fastify, state, syslog } = createApp(config);
app = fastify;
await fastify.listen({ port: config.httpPort, host: "0.0.0.0" });
syslog.start();
log(`Bastion HTTP listening on :${HTTP_PORT}`);
startDnsmasq(config).catch((err) => log(`dnsmasq failed: ${err instanceof Error ? err.message : String(err)}`));
await sleep(1500);
log("Creating x86_64 PXE VM (KVM)...");
createPxeVm({ name: VM_NAME, memory: VM_MEMORY, vcpus: VM_VCPUS, diskSize: VM_DISK_GB, network: PXE_NETWORK_NAME });
const mac = getVmMac(VM_NAME);
if (!mac) throw new Error("Could not determine VM MAC");
vmMac = mac;
log(`VM MAC: ${vmMac}`);
// Queue rescue directly, as `labctl provision debug` does.
log("Queueing debug/rescue mode...");
state.update((s) => {
s.debug[vmMac] = { hostname: "rescue-test", queued_at: new Date().toISOString() };
});
rebootPxeVm(VM_NAME);
await sleep(5_000);
deleteNftablesRejectRules();
rescueIp = await waitForLease(testDir, vmMac, LEASE_TIMEOUT_MS);
log(`Rescue IP: ${rescueIp}`);
log("Waiting for SSH into the rescue shell (inst.sshd)...");
await waitForSsh(rescueIp, "root", SSH_TIMEOUT_MS, sshKeyPath).catch(async (err) => {
log("Rescue SSH timed out. Serial console:");
try { log(await readSerialLog(SERIAL_PORT, { lastLines: 120, timeoutMs: 20_000 })); } catch { /* none */ }
throw err;
});
log("Rescue shell reachable.");
}, LEASE_TIMEOUT_MS + SSH_TIMEOUT_MS + 300_000);
afterAll(async () => {
if (process.env["KEEP_VM"] === "1") {
log(`KEEP_VM=1 — leaving ${VM_NAME} up (serial: socat - TCP:127.0.0.1:${SERIAL_PORT})`);
return;
}
log("Cleaning up...");
if (app) await app.close().catch(() => {});
if (stopDnsmasqFn) stopDnsmasqFn();
destroyPxeVm(VM_NAME);
destroyPxeNetwork();
if (testDir) rmSync(testDir, { recursive: true, force: true });
});
it("rescue shell is reachable over SSH as root", () => {
const result = sshExec(rescueIp, "root", "whoami", { keyPath: sshKeyPath, timeout: 60_000 });
expect(result.exitCode).toBe(0);
expect(result.stdout.trim()).toBe("root");
});
it("is the Anaconda rescue environment", () => {
const result = sshExec(rescueIp, "root", "cat /proc/cmdline", { keyPath: sshKeyPath, timeout: 60_000 });
expect(result.stdout).toContain("inst.rescue");
expect(result.stdout).toContain("inst.sshd");
});
it("kernel and initrd came from the bastion", () => {
const result = sshExec(rescueIp, "root", "cat /proc/cmdline", { keyPath: sshKeyPath, timeout: 60_000 });
expect(result.stdout).toContain(`${BASTION_IP}:${HTTP_PORT}`);
});
it("has LVM tools for inspecting an installed system", () => {
const result = sshExec(rescueIp, "root", "command -v vgchange && command -v lsblk", { keyPath: sshKeyPath, timeout: 60_000 });
expect(result.exitCode).toBe(0);
});
});