diff --git a/bastion/docs/ARCHITECTURE.md b/bastion/docs/ARCHITECTURE.md index 3cd9036..048b244 100644 --- a/bastion/docs/ARCHITECTURE.md +++ b/bastion/docs/ARCHITECTURE.md @@ -89,6 +89,83 @@ Side paths: --- +## Multi-architecture PXE + +The bastion serves both `x86_64` and `aarch64` over the network. Nothing about this is +operator-configured -- there is no `--arch` flag, by design. + +### How a client's architecture is decided + +1. **DHCP option 93** (Client System Architecture) picks the *bootloader*. dnsmasq matches + it and hands out a matching iPXE binary: + + | Option 93 | Client | Served | + |---|---|---| + | `0` | x86 BIOS | `undionly.kpxe` (TFTP) | + | `7`, `9` | x64 UEFI | `ipxe.efi` (TFTP) | + | `11` | **ARM64 UEFI** | `ipxe-arm64.efi` (TFTP) | + | `16` | x64 UEFI HTTP Boot | `http://…/ipxe.efi` | + | `19` | **ARM64 UEFI HTTP Boot** | `http://…/ipxe-arm64.efi` | + + Values come from the IANA Processor Architecture Types registry. Note `19`, not `20` -- + `20` is *pc/at bios boot from http*. EDK2/AAVMF prefers HTTP Boot over TFTP PXE, so the + iPXE binaries are staged in **both** `tftpDir` and `httpDir` (symlinked by `main.ts`). + +2. **`/dispatch` picks the kernel.** Option 93 never reaches the HTTP endpoint, so + `boot.ipxe` passes iPXE's own `${buildarch}` as `?arch=`. `resolveArch()` prefers, in + order: the tracked machine record → the reported `?arch=` → the configured default. + The record wins because it is what we observed on the machine itself. + +### Artifact naming + +`x86_64` keeps the original unsuffixed paths so its rendered iPXE scripts are unchanged; +everything else is suffixed. `kernelPath()` / `initrdPath()` in `templates/boot.ipxe.ts` +are the single source of truth, used by both the templates and `main.ts` staging. + +| arch | kernel | initrd | +|---|---|---| +| `x86_64` | `/vmlinuz` | `/initrd.img` | +| `aarch64` | `/vmlinuz-aarch64` | `/initrd-aarch64.img` | + +`tests/ipxe-x86-regression.test.ts` pins the x86_64 output against a golden fixture. + +### arm64 gotchas + +- **LoadFile2 is mandatory.** arm64 has no `HdrS` boot protocol; the kernel's EFI stub + fetches the initrd over the UEFI `EFI_LOAD_FILE2_PROTOCOL`. An iPXE build without it + accepts the `initrd` line, silently drops it, and the kernel panics with + `VFS: Unable to mount root fs on unknown-block(0,0)`. Fedora's + `ipxe-bootimgs-aarch64` implements it; the integration test asserts this up front so + the failure names itself instead of looking like a disk problem. +- **`nomodeset` is x86-only.** On arm64 there is no VGA path to fall back to. aarch64 gets + `console=tty0 console=ttyAMA0,115200` instead — the last `console=` wins for + `/dev/console`, so serial is the interactive one. +- **Ubuntu is x86_64-only.** `releases.ubuntu.com` publishes no arm64 netboot artifacts. + `osSupportsArch()` encodes this, and both the install guard and `/dispatch` refuse the + combination rather than serving an x86 kernel to an ARM machine. + +--- + +## Onboarding classification (vendor OS) + +Machines carry an `onboard` field: `"pxe"` (default) or `"ssh"`, plus `vendor_os` naming +what they run. `classifyOnboard()` in `@lab/shared` sets it from DMI identity, with known +hardware also matched by MAC — a machine can sit in state for a long time with no DMI, and +a DMI-only rule would fail open exactly where it matters. + +`onboard: "ssh"` means *we cannot rebuild this machine's OS*. Installs are refused at both +entry points (`/api/install` and the labd `command-install` handler) with an error naming +the machine and pointing at `provision debug`. **Rescue is never guarded** — being unable +to reinstall a machine is precisely when a rescue shell is needed. + +This is a fact about the machine, not a blocklist. The refusal follows from "no image in +our pipeline restores `vendor_os`", so adding a DGX OS image to the pipeline is what +unblocks the DGX Sparks — no entry needs deleting. + +Current classifications: NVIDIA DGX Spark (`spark-2935`, `spark-3a1c`) → `dgx-os`. + +--- + ## Packages ### Monorepo Structure @@ -404,6 +481,36 @@ Hardcoded `/dev/sda` default broke NVMe-only machines. Fix: default to empty str ### Anaconda Rescue Mode Limitations `%pre` and `%post` sections do not execute in `inst.rescue` mode. SSH in rescue mode is provided by Anaconda's `inst.sshd` kernel parameter + `sshpw` kickstart directive. Manual setup via `curl bastion:8080/debug-setup.sh | bash` for nc listener. +**Unresolved (2026-08-11): rescue SSH has never been observed working.** Adding the first +integration coverage for `provision debug` (`tests/integration/pxe-rescue.test.ts`) showed the +rescue environment coming up correctly — the bastion serves the kernel and initrd, Anaconda +boots, fetches `debug.ks`, and reaches its installer environment — but **nothing ever listens on +port 22**. + +Strength of the evidence, stated precisely because it decides where to look next: +- **aarch64 — direct.** Port 22 probed every 20s for 30 minutes while the Anaconda installer + environment was demonstrably running (NetworkManager, polkitd, rsyslog on the console). Never + opened. +- **x86_64 — corroborating, not conclusive.** One clean KVM run (943s) where SSH never became + available inside a 15-minute budget. That VM's progress into the rescue environment was *not* + observed — vitest's final reporter discards the streamed log — so it is consistent with the + aarch64 result but does not independently prove it. Re-run with `KEEP_VM=1` and probe port 22 + directly to settle it. + +If the x86_64 result holds up, this is orthogonal to the multi-architecture work, since x86_64 is +untouched by it. Leads worth checking, in order: +- Does `inst.sshd` actually start `sshd` in `inst.rescue` mode, or only in install mode? The + port never opens, so this is the prime suspect — an auth problem would still show an open port. +- `sshkey` may apply only to the *installed* system, leaving the installer environment + password-only via `sshpw`. That would matter once sshd does listen: the test authenticates + key-only (`BatchMode=yes`). +- The `%anaconda`-context directives in `debug.ks` may be skipped entirely when a kickstart is + supplied alongside `inst.rescue`. + +Until this is resolved, `provision debug` gets you a booted rescue environment on the console +(including on arm64), but not an SSH shell. The `debug-setup.sh` nc-listener path is the +documented workaround and is unaffected. + --- ## Planned Work (Taskmaster) diff --git a/bastion/package.json b/bastion/package.json index fc1b5a3..f733572 100644 --- a/bastion/package.json +++ b/bastion/package.json @@ -23,6 +23,12 @@ "test:integration:iso:host": "sudo -E $(which npx) vitest run -c tests/integration/vitest.config.ts -t 'ISO boot'", "test:integration:arm-iso": "vitest run -c tests/integration/vitest.config.ts -t 'ARM ISO'", "test:integration:arm-iso:host": "sudo -E $(which npx) vitest run -c tests/integration/vitest.config.ts -t 'ARM ISO'", + "test:integration:rescue": "vitest run -c tests/integration/vitest.config.ts -t 'x86 rescue boot'", + "test:integration:rescue:host": "sudo -E $(which npx) vitest run -c tests/integration/vitest.config.ts -t 'x86 rescue boot'", + "test:integration:arm-pxe": "vitest run -c tests/integration/vitest.config.ts -t 'ARM PXE rescue'", + "test:integration:arm-pxe:host": "sudo -E $(which npx) vitest run -c tests/integration/vitest.config.ts -t 'ARM PXE rescue'", + "test:integration:arm-pxe-full": "ARM_PXE_FULL=1 vitest run -c tests/integration/vitest.config.ts -t 'ARM PXE'", + "test:integration:arm-pxe-full:host": "sudo -E ARM_PXE_FULL=1 $(which npx) vitest run -c tests/integration/vitest.config.ts -t 'ARM PXE'", "test:integration:asahi": "vitest run -c tests/integration/vitest.config.ts -t 'asahi firstboot'", "test:integration:asahi:host": "sudo -E $(which npx) vitest run -c tests/integration/vitest.config.ts -t 'asahi firstboot'", "test:integration:asahi-validate": "vitest run -c tests/integration/vitest.config.ts -t 'asahi.*validation'", diff --git a/bastion/scripts/test-provision.sh b/bastion/scripts/test-provision.sh index 4bc20c6..7a37e91 100755 --- a/bastion/scripts/test-provision.sh +++ b/bastion/scripts/test-provision.sh @@ -2,16 +2,19 @@ # Run PXE and/or ISO boot integration tests. # # Usage: -# sudo ./scripts/test-provision.sh # run PXE + ISO (x86_64) -# sudo ./scripts/test-provision.sh pxe # PXE only -# sudo ./scripts/test-provision.sh iso # ISO only (x86_64) -# sudo ./scripts/test-provision.sh arm # ARM ISO boot (emulated, SLOW ~60min) -# sudo ./scripts/test-provision.sh all # all tests including ARM +# sudo ./scripts/test-provision.sh # run PXE + ISO (x86_64) +# sudo ./scripts/test-provision.sh pxe # PXE only +# sudo ./scripts/test-provision.sh iso # ISO only (x86_64) +# sudo ./scripts/test-provision.sh rescue # x86_64 Anaconda rescue boot + SSH (~15min) +# sudo ./scripts/test-provision.sh arm # ARM ISO boot (emulated, SLOW ~60min) +# sudo ./scripts/test-provision.sh arm-pxe # ARM network PXE rescue: NBP + rescue over SSH (~25-30min) +# sudo ./scripts/test-provision.sh arm-pxe-full # ARM network PXE incl. discover + full install (~75-95min) +# sudo ./scripts/test-provision.sh all # all tests including ARM # # Prerequisites: # libvirtd, OVMF (edk2-ovmf), iPXE (ipxe-bootimgs-x86), # dnsmasq, xorriso, mtools, virt-install, qemu-img -# ARM: qemu-system-aarch64, edk2-aarch64 +# ARM: qemu-system-aarch64, edk2-aarch64, ipxe-bootimgs-aarch64 set -e SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" @@ -58,6 +61,10 @@ if [ ! -f /usr/share/edk2/ovmf/OVMF_CODE.fd ]; then exit 1 fi +MODE="${1:-both}" + +# iPXE binaries are per-architecture. x86_64 is always required (the dnsmasq config +# references it); arm64 only for the ARM network-PXE modes. IPXE_EFI="" for f in /usr/share/ipxe/ipxe-snponly-x86_64.efi /usr/share/ipxe/ipxe-snp-x86_64.efi /usr/share/ipxe/ipxe-x86_64.efi; do [ -f "$f" ] && IPXE_EFI="$f" && break @@ -67,6 +74,20 @@ if [ -z "$IPXE_EFI" ]; then exit 1 fi +IPXE_EFI_ARM64="" +for f in /usr/share/ipxe/arm64-efi/snponly.efi /usr/share/ipxe/arm64-efi/ipxe.efi; do + [ -f "$f" ] && IPXE_EFI_ARM64="$f" && break +done + +case "$MODE" in + arm-pxe|arm-pxe-full|all) + if [ -z "$IPXE_EFI_ARM64" ] && [ "$MODE" != "all" ]; then + echo -e "${RED}arm64 iPXE binary not found.${RESET} Install: sudo dnf install ipxe-bootimgs-aarch64" + exit 1 + fi + ;; +esac + # Find SSH key SSH_KEY="" for name in id_ed25519 id_ecdsa id_rsa; do @@ -83,10 +104,19 @@ fi echo -e " User: ${BOLD}$REAL_USER${RESET}" echo -e " SSH key: ${BOLD}$SSH_KEY${RESET}" echo -e " iPXE: ${BOLD}$IPXE_EFI${RESET}" +echo -e " iPXE a64:${BOLD} ${IPXE_EFI_ARM64:-not installed}${RESET}" echo "" -# --- Determine which tests to run --- -MODE="${1:-both}" +require_arm_emulation() { + if ! command -v qemu-system-aarch64 &>/dev/null; then + echo -e "${RED}qemu-system-aarch64 not found.${RESET} Install: sudo dnf install qemu-system-aarch64 edk2-aarch64" + exit 1 + fi + if [ ! -f /usr/share/edk2/aarch64/QEMU_EFI.fd ]; then + echo -e "${RED}AAVMF firmware not found.${RESET} Install: sudo dnf install edk2-aarch64" + exit 1 + fi +} run_test() { local name="$1" pattern="$2" @@ -116,13 +146,26 @@ case "$MODE" in run_test "ISO boot" "ISO boot" || FAILED=1 ;; arm|arm-iso) - if ! command -v qemu-system-aarch64 &>/dev/null; then - echo -e "${RED}qemu-system-aarch64 not found.${RESET} Install: sudo dnf install qemu-system-aarch64 edk2-aarch64" - exit 1 - fi + require_arm_emulation echo -e "${YELLOW}ARM emulation is ~10x slower than native. Expect 30-60 minutes.${RESET}" run_test "ARM ISO boot" "ARM ISO" || FAILED=1 ;; + rescue) + echo -e "${YELLOW}x86_64 rescue boot (KVM). Expect ~15 minutes.${RESET}" + run_test "x86 rescue boot" "x86 rescue boot" || FAILED=1 + ;; + arm-pxe) + require_arm_emulation + echo -e "${YELLOW}ARM emulation is ~10x slower than native. Expect 25-30 minutes.${RESET}" + echo -e "${YELLOW}Covers option 93 -> arm64 NBP, arch resolution, and rescue over SSH.${RESET}" + echo -e "${YELLOW}For the full install too, use: $0 arm-pxe-full${RESET}" + run_test "ARM PXE rescue" "ARM PXE rescue" || FAILED=1 + ;; + arm-pxe-full) + require_arm_emulation + echo -e "${YELLOW}ARM emulation is ~10x slower than native. Expect 75-95 minutes.${RESET}" + ARM_PXE_FULL=1 run_test "ARM PXE (rescue + install)" "ARM PXE" || FAILED=1 + ;; both) run_test "PXE boot" "PXE boot" || FAILED=1 run_test "ISO boot" "ISO boot" || FAILED=1 @@ -133,12 +176,17 @@ case "$MODE" in if command -v qemu-system-aarch64 &>/dev/null; then echo -e "${YELLOW}ARM emulation is ~10x slower than native.${RESET}" run_test "ARM ISO boot" "ARM ISO" || FAILED=1 + if [ -n "$IPXE_EFI_ARM64" ]; then + run_test "ARM PXE rescue" "ARM PXE rescue" || FAILED=1 + else + echo -e "${YELLOW}Skipping ARM PXE test (ipxe-bootimgs-aarch64 not installed)${RESET}" + fi else - echo -e "${YELLOW}Skipping ARM test (qemu-system-aarch64 not installed)${RESET}" + echo -e "${YELLOW}Skipping ARM tests (qemu-system-aarch64 not installed)${RESET}" fi ;; *) - echo "Usage: $0 [pxe|iso|arm|both|all]" + echo "Usage: $0 [pxe|iso|rescue|arm|arm-pxe|arm-pxe-full|both|all]" exit 1 ;; esac diff --git a/bastion/src/bastion/src/main.ts b/bastion/src/bastion/src/main.ts index 7468f1f..b0ab842 100644 --- a/bastion/src/bastion/src/main.ts +++ b/bastion/src/bastion/src/main.ts @@ -3,7 +3,9 @@ import { mkdirSync, writeFileSync, readFileSync, existsSync, copyFileSync, symlinkSync, unlinkSync } from "node:fs"; import { execSync } from "node:child_process"; -import type { BastionConfig } from "@lab/shared"; +import type { Arch, BastionConfig } from "@lab/shared"; +import { SUPPORTED_ARCHES, fedoraMirrorFor, classifyOnboard } from "@lab/shared"; +import { kernelPath, initrdPath } from "./templates/boot.ipxe.js"; import { loadConfig } from "./config.js"; import { populateNetworkConfig } from "./services/network.js"; import { createApp } from "./server.js"; @@ -13,6 +15,7 @@ import { renderBootIpxe } from "./templates/boot.ipxe.js"; import { logger } from "./services/logger.js"; import { BastionConnection } from "./services/labd-connection.js"; import { progressBus } from "./services/progress-events.js"; +import { checkInstallAllowed } from "./services/install-guard.js"; import { ensureBootIso } from "./routes/boot-iso.js"; function copyIfMissing(src: string, dest: string, label: string): void { @@ -130,9 +133,14 @@ export async function startBastion(overrides: Partial = {}): Prom mkdirSync(config.tftpDir, { recursive: true }); mkdirSync(config.httpDir, { recursive: true }); + // Architectures we can actually network boot, reported in the banner so a missing + // arm64 payload is visible at startup instead of at 2am when a rescue is needed. + const bootArches: Arch[] = []; + let ipxeArm64Ready = false; + // Prepare boot artifacts if (config.skipArtifacts !== true) { - logger.info(`Preparing boot artifacts (Fedora ${config.fedoraVersion} ${config.arch})...`); + logger.info(`Preparing boot artifacts (Fedora ${config.fedoraVersion}, ${SUPPORTED_ARCHES.join(" + ")})...`); copyIfMissing( "/usr/share/ipxe/undionly.kpxe", @@ -150,20 +158,41 @@ export async function startBastion(overrides: Partial = {}): Prom `${config.tftpDir}/ipxe-arm64.efi`, "iPXE UEFI arm64", ); + ipxeArm64Ready = true; } catch { - logger.warn("arm64 iPXE not available -- skipping"); + logger.warn("arm64 iPXE not available -- arm64 machines cannot network boot."); + logger.warn(" Install with: sudo dnf install ipxe-bootimgs-aarch64"); } - download( - `${config.fedoraMirror}/images/pxeboot/vmlinuz`, - `${config.httpDir}/vmlinuz`, - "Fedora kernel", - ); - download( - `${config.fedoraMirror}/images/pxeboot/initrd.img`, - `${config.httpDir}/initrd.img`, - "Fedora initrd", - ); + // Fedora pxeboot kernel + initrd per architecture. x86_64 keeps the unsuffixed + // names it has always used; other architectures are suffixed. The iPXE templates + // resolve the same paths via kernelPath()/initrdPath(). + for (const arch of SUPPORTED_ARCHES) { + const mirror = fedoraMirrorFor(config.fedoraVersion, arch); + try { + download( + `${mirror}/images/pxeboot/vmlinuz`, + `${config.httpDir}${kernelPath(arch)}`, + `Fedora ${arch} kernel`, + ); + download( + `${mirror}/images/pxeboot/initrd.img`, + `${config.httpDir}${initrdPath(arch)}`, + `Fedora ${arch} initrd`, + ); + bootArches.push(arch); + } catch (err) { + // Non-fatal: a bastion with no arm64 artifacts still serves x86_64 fine. + // Failing startup over an unreachable mirror for an architecture that may not + // even be present on this network would be worse. + logger.warn(`Fedora ${arch} kernel/initrd unavailable -- ${arch} PXE disabled`); + logger.warn(` ${err instanceof Error ? err.message : String(err)}`); + } + } + + if (!bootArches.includes("x86_64")) { + throw new Error("Fedora x86_64 kernel/initrd could not be staged -- cannot serve PXE"); + } // Ubuntu netboot artifacts (non-fatal — Ubuntu version may not be released yet) try { @@ -254,6 +283,13 @@ export async function startBastion(overrides: Partial = {}): Prom // Wire up command handlers so labd can send install/forget/role commands labdConn.onCommand("command-install", async (msg) => { if (msg.type !== "command-install") throw new Error("unexpected"); + const installMac = msg.mac.toLowerCase().replace(/-/g, ":"); + const osId = (msg.os as import("@lab/shared").OsId | undefined) ?? "fedora-43"; + const check = checkInstallAllowed(state.load(), installMac, osId); + if (check.allowed === false) { + logger.warn(`INSTALL REFUSED: ${installMac} -- ${check.error}`); + return { status: "error", error: check.error }; + } state.update((s) => { s.install_queue[msg.mac] = { hostname: msg.hostname, @@ -314,13 +350,24 @@ export async function startBastion(overrides: Partial = {}): Prom const mac = (msg.mac as string).toLowerCase(); const now = new Date().toISOString(); const existing = state.load().discovered[mac]; + const identity = { + mac, + manufacturer: (msg.manufacturer as string) ?? "unknown", + product: (msg.product as string) ?? "unknown", + board: (msg.board as string) ?? "unknown", + ...(existing?.onboard !== undefined ? { onboard: existing.onboard } : {}), + ...(existing?.vendor_os !== undefined ? { vendor_os: existing.vendor_os } : {}), + }; + const onboarding = classifyOnboard(identity); + const rootDevice = msg.root_device ?? existing?.root_device; + const rootArgs = msg.root_args ?? existing?.root_args; state.update((s) => { s.discovered[mac] = { mac, - product: (msg.product as string) ?? "unknown", - board: (msg.board as string) ?? "unknown", + product: identity.product, + board: identity.board, serial: (msg.serial as string) ?? "unknown", - manufacturer: (msg.manufacturer as string) ?? "unknown", + manufacturer: identity.manufacturer, cpu_model: (msg.cpu_model as string) ?? "unknown", cpu_cores: (msg.cpu_cores as number) ?? 0, memory_gb: (msg.memory_gb as number) ?? 0, @@ -329,7 +376,20 @@ export async function startBastion(overrides: Partial = {}): Prom nics: (msg.nics as Array<{ name: string; mac: string; state: string }>) ?? [], first_seen: existing?.first_seen ?? now, last_seen: now, + onboard: onboarding.onboard, + ...(onboarding.vendor_os !== undefined ? { vendor_os: onboarding.vendor_os } : {}), + ...(rootDevice !== undefined ? { root_device: rootDevice } : {}), + ...(rootArgs !== undefined ? { root_args: rootArgs } : {}), }; + // Keep the installed record in step -- the guard and --pxe-boot both read it. + const inst = s.installed[mac]; + if (inst) { + inst.arch = (msg.arch as string) ?? inst.arch; + inst.onboard = onboarding.onboard; + if (onboarding.vendor_os !== undefined) inst.vendor_os = onboarding.vendor_os; + if (rootDevice !== undefined) inst.root_device = rootDevice; + if (rootArgs !== undefined) inst.root_args = rootArgs; + } }); logger.info(`HARDWARE UPDATED: ${mac} -- ${msg.manufacturer ?? "?"} ${msg.product ?? "?"} (${msg.cpu_model ?? "?"}, ${msg.cpu_cores ?? "?"} cores, ${msg.memory_gb ?? "?"}GB RAM)`); return { status: "ok", data: { mac } }; @@ -364,7 +424,7 @@ export async function startBastion(overrides: Partial = {}): Prom } // Print banner - printBanner(config); + printBanner(config, bootArches, ipxeArm64Ready); // Graceful shutdown const shutdown = async (): Promise => { @@ -386,11 +446,22 @@ export async function startBastion(overrides: Partial = {}): Prom await new Promise(() => {}); } -function printBanner(config: BastionConfig): void { +function printBanner(config: BastionConfig, bootArches: Arch[], ipxeArm64Ready: boolean): void { const dhcpInfo = config.dhcpMode === "full" ? `full (${config.dhcpRangeStart}-${config.dhcpRangeEnd})` : "proxy (alongside existing DHCP)"; + // arm64 needs both an iPXE binary (DHCP hands it out on option 93 = 0x0b) and a + // kernel/initrd pair. Report the combination, since either missing breaks it. + const archInfo = config.skipArtifacts === true + ? "(artifacts skipped)" + : SUPPORTED_ARCHES + .map((a) => { + const ready = bootArches.includes(a) && (a !== "aarch64" || ipxeArm64Ready); + return ready ? a : `${a} (unavailable)`; + }) + .join(", "); + console.log(""); console.log("\x1b[36m\x1b[1m" + "=".repeat(60) + "\x1b[0m"); console.log("\x1b[36m\x1b[1m Lab PXE Bastion -- Discovery Mode\x1b[0m"); @@ -399,7 +470,8 @@ function printBanner(config: BastionConfig): void { console.log(` Network: \x1b[1m${config.network}/24\x1b[0m via \x1b[1m${config.iface}\x1b[0m`); console.log(` DHCP: \x1b[1m${dhcpInfo}\x1b[0m`); console.log(` HTTP: \x1b[1mhttp://${config.serverIp}:${config.httpPort}/\x1b[0m`); - console.log(` OS: \x1b[1mFedora ${config.fedoraVersion} (${config.arch})\x1b[0m`); + console.log(` OS: \x1b[1mFedora ${config.fedoraVersion}\x1b[0m`); + console.log(` Net boot: \x1b[1m${archInfo}\x1b[0m`); console.log(` Domain: \x1b[1m${config.domain}\x1b[0m`); console.log(` State: \x1b[1m${config.stateFile}\x1b[0m`); console.log(""); diff --git a/bastion/src/bastion/src/routes/api.ts b/bastion/src/bastion/src/routes/api.ts index 8a75c65..3294390 100644 --- a/bastion/src/bastion/src/routes/api.ts +++ b/bastion/src/bastion/src/routes/api.ts @@ -6,10 +6,11 @@ import type { FastifyInstance } from "fastify"; import type { HardwareInfo, InstalledInfo, Role } from "@lab/shared"; -import { isValidOsId, SUPPORTED_ROLES } from "@lab/shared"; +import { isValidOsId, SUPPORTED_ROLES, classifyOnboard } from "@lab/shared"; import type { StateManager } from "../services/state.js"; import { logger } from "../services/logger.js"; import { triggerPostProvisionK3s } from "../services/post-provision.js"; +import { checkInstallAllowed } from "../services/install-guard.js"; import { progressBus } from "../services/progress-events.js"; import type { ProgressEvent } from "../services/progress-events.js"; import type { InstallLogBuffer } from "../services/install-log.js"; @@ -53,6 +54,12 @@ export function registerApiRoutes( return reply.status(400).send({ error: `invalid os: '${osId}'. Supported: fedora-43, ubuntu-26.04` }); } + const check = checkInstallAllowed(state.load(), mac, osId); + if (check.allowed === false) { + logger.warn(`INSTALL REFUSED: ${mac} -- ${check.error}`); + return reply.status(409).send({ error: check.error }); + } + state.update((s) => { s.install_queue[mac] = { hostname: hostname ?? "lab-node", @@ -284,6 +291,10 @@ export function registerApiRoutes( arch?: string; disks?: Array<{ name: string; size_gb: number; model: string }>; nics?: Array<{ name: string; mac: string; state: string }>; + // Root filesystem, when the reporter could observe it (recheck over SSH, or the + // probe script run from a rescue shell). Used by --pxe-boot. + root_device?: string; + root_args?: string; }; }>("/api/discover", async (request, reply) => { const data = request.body; @@ -298,22 +309,53 @@ export function registerApiRoutes( state.update((s) => { const existing = s.discovered[mac]; + // Classify onboarding from the DMI identity we just received. An explicit + // classification already on the record wins (see classifyOnboard). + const onboarding = classifyOnboard({ + mac, + manufacturer: data.manufacturer ?? existing?.manufacturer ?? "unknown", + product: data.product ?? existing?.product ?? "unknown", + board: data.board ?? existing?.board ?? "unknown", + ...(existing?.onboard !== undefined ? { onboard: existing.onboard } : {}), + ...(existing?.vendor_os !== undefined ? { vendor_os: existing.vendor_os } : {}), + }); + const rootDevice = data.root_device ?? existing?.root_device; + const rootArgs = data.root_args ?? existing?.root_args; + + // Absent fields keep whatever we already knew. Reporters are not all the full + // discovery kickstart: the rescue-shell probe posts only a root device, and + // blanking a machine's hardware inventory as a side effect of that would be + // silent data loss. const hwInfo: HardwareInfo = { mac, - product: data.product ?? "unknown", - board: data.board ?? "unknown", - serial: data.serial ?? "unknown", - manufacturer: data.manufacturer ?? "unknown", - cpu_model: data.cpu_model ?? "unknown", - cpu_cores: data.cpu_cores ?? 0, - memory_gb: data.memory_gb ?? 0, - arch: data.arch ?? "unknown", - disks: data.disks ?? [], - nics: data.nics ?? [], + product: data.product ?? existing?.product ?? "unknown", + board: data.board ?? existing?.board ?? "unknown", + serial: data.serial ?? existing?.serial ?? "unknown", + manufacturer: data.manufacturer ?? existing?.manufacturer ?? "unknown", + cpu_model: data.cpu_model ?? existing?.cpu_model ?? "unknown", + cpu_cores: data.cpu_cores ?? existing?.cpu_cores ?? 0, + memory_gb: data.memory_gb ?? existing?.memory_gb ?? 0, + arch: data.arch ?? existing?.arch ?? "unknown", + disks: data.disks ?? existing?.disks ?? [], + nics: data.nics ?? existing?.nics ?? [], first_seen: existing?.first_seen ?? now, last_seen: now, + onboard: onboarding.onboard, + ...(onboarding.vendor_os !== undefined ? { vendor_os: onboarding.vendor_os } : {}), + ...(rootDevice !== undefined ? { root_device: rootDevice } : {}), + ...(rootArgs !== undefined ? { root_args: rootArgs } : {}), }; s.discovered[mac] = hwInfo; + + // Keep the installed record in step -- the install guard and --pxe-boot read it. + const inst = s.installed[mac]; + if (inst) { + if (data.arch !== undefined) inst.arch = data.arch; + inst.onboard = onboarding.onboard; + if (onboarding.vendor_os !== undefined) inst.vendor_os = onboarding.vendor_os; + if (rootDevice !== undefined) inst.root_device = rootDevice; + if (rootArgs !== undefined) inst.root_args = rootArgs; + } }); const label = isNew ? "NEW MACHINE DISCOVERED" : "MACHINE RE-DISCOVERED"; diff --git a/bastion/src/bastion/src/routes/dispatch.ts b/bastion/src/bastion/src/routes/dispatch.ts index 0ecc1c4..5b1a8a6 100644 --- a/bastion/src/bastion/src/routes/dispatch.ts +++ b/bastion/src/bastion/src/routes/dispatch.ts @@ -5,7 +5,8 @@ // - unknown -> discovery mode (collect hardware, POST to bastion) import type { FastifyInstance } from "fastify"; -import type { BastionConfig } from "@lab/shared"; +import type { Arch, BastionConfig, BastionState, OsId } from "@lab/shared"; +import { normalizeArch, fedoraMirrorFor, osSupportsArch } from "@lab/shared"; import type { StateManager } from "../services/state.js"; import { renderDiscoverIpxe, @@ -13,11 +14,51 @@ import { renderDebugIpxe, renderPxeBootDebugIpxe, renderLocalBootIpxe, + renderUnsupportedIpxe, } from "../templates/boot.ipxe.js"; import { renderUbuntuInstallIpxe } from "../templates/ubuntu-boot.ipxe.js"; import { renderDebugKickstart } from "../templates/debug.ks.js"; import { logger } from "../services/logger.js"; +/** + * Resolve a booting machine's architecture. + * + * Order matters. The tracked record is what we actually observed on the machine, so it + * wins. `reported` is iPXE's ${buildarch}, which is only as good as the binary DHCP + * handed the client -- correct in practice, but a misconfigured option 93 mapping would + * make it lie. The configured default is the last resort. + * + * There is deliberately no operator-supplied architecture anywhere in this path. + */ +export function resolveArch( + state: BastionState, + mac: string, + reported: string | undefined, + config: BastionConfig, +): Arch { + return normalizeArch(state.installed[mac]?.arch) + ?? normalizeArch(state.install_queue[mac]?.arch) + ?? normalizeArch(state.discovered[mac]?.arch) + ?? normalizeArch(reported) + ?? normalizeArch(config.arch) + ?? "x86_64"; +} + +/** The root filesystem to boot for --pxe-boot, if the machine's record carries one. */ +function resolveRoot( + state: BastionState, + mac: string, +): { rootDevice: string; rootArgs?: string } | null { + const installed = state.installed[mac]; + const discovered = state.discovered[mac]; + const rootDevice = installed?.root_device ?? discovered?.root_device; + if (rootDevice === undefined || rootDevice === "") return null; + const rootArgs = installed?.root_args ?? discovered?.root_args; + return rootArgs !== undefined && rootArgs !== "" + ? { rootDevice, rootArgs } + : { rootDevice }; +} + export function registerDispatchRoutes( app: FastifyInstance, config: BastionConfig, @@ -52,18 +93,68 @@ curl -sf -X POST "http://${config.serverIp}:${config.httpPort}/api/progress" \\ -H "Content-Type: application/json" \\ -d "{\\"mac\\":\\"$MAC_ADDR\\",\\"stage\\":\\"debug-ready\\",\\"detail\\":\\"nc $IP_ADDR 2323\\"}" 2>/dev/null || true +# --- Find the installed root filesystem and report it --- +# This is what 'labctl provision debug --pxe-boot' needs. The rescue image cannot +# report it by itself: %pre/%post do not run in rescue mode, so it happens here. +vgchange -ay >/dev/null 2>&1 || true + +ROOT_DEVICE="" +ROOT_ARGS="" +PROBE_MNT=/tmp/lab-rootprobe +mkdir -p "$PROBE_MNT" + +# Candidates: every LVM logical volume plus every non-LVM partition with a filesystem. +for CAND in $(lvs --noheadings -o lv_path 2>/dev/null) \\ + $(blkid -o device 2>/dev/null | grep -v '^/dev/mapper/'); do + [ -b "$CAND" ] || continue + mount -o ro "$CAND" "$PROBE_MNT" >/dev/null 2>&1 || continue + # A root filesystem has both of these; /boot and /home do not. + if [ -f "$PROBE_MNT/etc/fstab" ] && [ -d "$PROBE_MNT/usr" ]; then + ROOT_DEVICE="$CAND" + PRETTY=$(. "$PROBE_MNT/etc/os-release" 2>/dev/null && echo "$PRETTY_NAME") + echo " found root: $CAND \${PRETTY:+($PRETTY)}" + if [ "$(lsblk -no TYPE "$CAND" 2>/dev/null | head -1)" = "lvm" ]; then + VGLV=$(lvs --noheadings -o vg_name,lv_name "$CAND" 2>/dev/null | awk '{print $1"/"$2}') + [ -n "$VGLV" ] && ROOT_ARGS="rd.lvm.lv=$VGLV" + # Swap comes from fstab here — /proc/swaps is the rescue image's, not the host's. + SWLV=$(awk '$3=="swap" && $1 ~ /^\\/dev\\// {print $1; exit}' "$PROBE_MNT/etc/fstab" 2>/dev/null) + if [ -n "$SWLV" ]; then + SWVGLV=$(lvs --noheadings -o vg_name,lv_name "$SWLV" 2>/dev/null | awk '{print $1"/"$2}') + [ -n "$SWVGLV" ] && [ "$SWVGLV" != "$VGLV" ] && ROOT_ARGS="$ROOT_ARGS rd.lvm.lv=$SWVGLV" + fi + fi + umount "$PROBE_MNT" >/dev/null 2>&1 || true + break + fi + umount "$PROBE_MNT" >/dev/null 2>&1 || true +done + +if [ -n "$ROOT_DEVICE" ]; then + curl -sf -X POST "http://${config.serverIp}:${config.httpPort}/api/discover" \\ + -H "Content-Type: application/json" \\ + -d "{\\"mac\\":\\"$MAC_ADDR\\",\\"root_device\\":\\"$ROOT_DEVICE\\",\\"root_args\\":\\"$ROOT_ARGS\\"}" 2>/dev/null \\ + && echo " reported to bastion — 'labctl provision debug --pxe-boot' will work now" +else + echo " no root filesystem found — --pxe-boot cannot be used on this machine" +fi + echo "" echo "=== Debug environment ready ===" echo " nc $IP_ADDR 2323 (remote shell)" echo " ssh root@$IP_ADDR (password: debug)" +if [ -n "$ROOT_DEVICE" ]; then + echo " root: $ROOT_DEVICE $ROOT_ARGS" +fi echo "===============================" `; return reply.type("text/plain").send(script); }); - app.get<{ Querystring: { mac?: string } }>("/dispatch", async (request, reply) => { + app.get<{ Querystring: { mac?: string; arch?: string } }>("/dispatch", async (request, reply) => { const mac = (request.query.mac ?? "").toLowerCase().replace(/-/g, ":"); const currentState = state.load(); + const arch = resolveArch(currentState, mac, request.query.arch, config); + const fedoraMirror = fedoraMirrorFor(config.fedoraVersion, arch); // Debug mode takes highest priority — auto-clear after serving once const debugEntry = currentState.debug[mac]; @@ -72,22 +163,48 @@ echo "===============================" state.update((s) => { delete s.debug[mac]; }); let script: string; - if (debugEntry.pxeBoot) { - logger.info(`PXE BOOT DEBUG: ${mac} -> ${hostname} (kernel+initrd from PXE, root from NVMe)`); + const wantsPxeBoot = debugEntry.pxeBoot === true; + const root = wantsPxeBoot ? resolveRoot(currentState, mac) : null; + + if (root !== null) { + logger.info(`PXE BOOT DEBUG: ${mac} -> ${hostname} (${arch}, root=${root.rootDevice})`); script = renderPxeBootDebugIpxe({ mac, hostname, serverIp: config.serverIp, httpPort: config.httpPort, + arch, + ...root, }); } else { - logger.info(`DEBUG BOOT: ${mac} -> ${hostname} (rescue mode)`); + // --pxe-boot without a known root device falls back to rescue rather than + // guessing. A wrong root= leaves the machine unbootable, and rescue is where + // the operator can find the real one (curl /debug-setup.sh reports it back). + const notice = wantsPxeBoot + ? [ + "", + "NOTE: --pxe-boot requested, but no root device is recorded", + " for this machine. Booting rescue instead.", + " From the rescue shell, run:", + // No pipe or && here: iPXE treats || and && as command separators, so keep + // the printed command free of anything its parser might claim. + ` curl -s http://${config.serverIp}:${config.httpPort}/debug-setup.sh -o /tmp/s.sh ; sh /tmp/s.sh`, + " then retry --pxe-boot.", + ] + : undefined; + if (wantsPxeBoot) { + logger.warn(`PXE BOOT DEBUG: ${mac} -> ${hostname} has no recorded root device -- serving rescue instead`); + } else { + logger.info(`DEBUG BOOT: ${mac} -> ${hostname} (${arch}, rescue mode)`); + } script = renderDebugIpxe({ mac, hostname, serverIp: config.serverIp, httpPort: config.httpPort, - fedoraMirror: config.fedoraMirror, + fedoraMirror, + arch, + ...(notice ? { notice } : {}), }); } return reply.type("text/plain").send(script); @@ -97,10 +214,24 @@ echo "===============================" if (queueEntry) { const hostname = queueEntry.hostname ?? "lab-node"; const os = queueEntry.os ?? "fedora-43"; - logger.info(`INSTALL STARTED: ${mac} -> ${hostname} (${os})`); + logger.info(`INSTALL STARTED: ${mac} -> ${hostname} (${os}, ${arch})`); let script: string; if (os.startsWith("ubuntu")) { + // Last line of defence. The install guard refuses this combination when the + // machine's architecture is already known, but a machine queued before it was + // discovered can reach here. Serving the x86-only Ubuntu kernel to an arm64 + // client is precisely the bug this work exists to fix, so stop instead. + if (!osSupportsArch(os as OsId, arch)) { + logger.error(`INSTALL BLOCKED: ${mac} -> ${hostname} -- ${os} has no ${arch} artifacts`); + script = renderUnsupportedIpxe({ + hostname, + mac, + reason: `${os} publishes no ${arch} netboot artifacts`, + action: `labctl provision install ${mac} ${hostname} --os fedora-43`, + }); + return reply.type("text/plain").send(script); + } script = renderUbuntuInstallIpxe({ mac, hostname, @@ -115,7 +246,8 @@ echo "===============================" serverIp: config.serverIp, httpPort: config.httpPort, fedoraVersion: config.fedoraVersion, - fedoraMirror: config.fedoraMirror, + fedoraMirror, + arch, }); } @@ -132,13 +264,14 @@ echo "===============================" } // Unknown MAC -> discovery mode - logger.info(`PXE request from ${mac} -> discovery mode`); + logger.info(`PXE request from ${mac} (${arch}) -> discovery mode`); const script = renderDiscoverIpxe({ mac, serverIp: config.serverIp, httpPort: config.httpPort, - fedoraMirror: config.fedoraMirror, + fedoraMirror, + arch, }); return reply.type("text/plain").send(script); diff --git a/bastion/src/bastion/src/services/install-guard.ts b/bastion/src/bastion/src/services/install-guard.ts new file mode 100644 index 0000000..9dbb517 --- /dev/null +++ b/bastion/src/bastion/src/services/install-guard.ts @@ -0,0 +1,94 @@ +// Pre-flight checks for queuing an OS install. +// +// Both entry points -- the HTTP /api/install route and the labd command-install handler +// -- run this, so `labctl provision install` and `provision reprovision` are covered +// whichever way the request arrives. +// +// Rescue/debug is deliberately NOT guarded. Being unable to reinstall a machine is +// exactly when you most need to boot it into a rescue shell. + +import type { Arch, BastionState, OsId } from "@lab/shared"; +import { classifyOnboard, normalizeArch, osSupportsArch, vendorOsDescription, archesForOs } from "@lab/shared"; + +export type InstallCheck = + | { allowed: true } + | { allowed: false; error: string }; + +interface MachineIdentity { + hostname: string; + arch: Arch | undefined; + identity: Parameters[0]; +} + +/** Best-known identity for a MAC, merged across the three state maps. */ +function identify(state: BastionState, mac: string): MachineIdentity { + const discovered = state.discovered[mac]; + const installed = state.installed[mac]; + const queued = state.install_queue[mac]; + + const manufacturer = discovered?.manufacturer ?? installed?.manufacturer; + const product = discovered?.product ?? installed?.product; + const board = discovered?.board; + const onboard = installed?.onboard ?? discovered?.onboard; + const vendorOs = installed?.vendor_os ?? discovered?.vendor_os; + + return { + hostname: installed?.hostname ?? queued?.hostname ?? discovered?.product ?? mac, + arch: normalizeArch(installed?.arch ?? queued?.arch ?? discovered?.arch), + identity: { + mac, + ...(manufacturer !== undefined ? { manufacturer } : {}), + ...(product !== undefined ? { product } : {}), + ...(board !== undefined ? { board } : {}), + ...(onboard !== undefined ? { onboard } : {}), + ...(vendorOs !== undefined ? { vendor_os: vendorOs } : {}), + }, + }; +} + +/** + * Decide whether `mac` may be queued for an install of `os`. + * + * Refusals name the machine and the reason, and point at the action that is available + * instead. An operator hitting this at 2am should not have to read the source to work + * out what happened. + */ +export function checkInstallAllowed( + state: BastionState, + mac: string, + os: OsId, +): InstallCheck { + const machine = identify(state, mac); + const { onboard, vendor_os } = classifyOnboard(machine.identity); + + // 1. Machines running a vendor OS we cannot rebuild. + if (onboard === "ssh") { + const what = vendorOsDescription(vendor_os); + return { + allowed: false, + error: + `Refusing to install ${machine.hostname} (${mac}): it runs ${what}. ` + + `No image in our pipeline can restore it, so installing ${os} would destroy that ` + + `driver and firmware stack permanently. This machine is SSH-onboard: we manage its ` + + `userspace, not its OS. ` + + `To boot it into a rescue shell instead, run: labctl provision debug ${machine.hostname}`, + // TODO: when a DGX OS / SparkOS image joins the pipeline, an install targeting a + // machine whose vendor_os matches that image should be allowed through here. + }; + } + + // 2. Architecture the OS has no netboot artifacts for. + if (machine.arch !== undefined && !osSupportsArch(os, machine.arch)) { + const supported = archesForOs(os); + return { + allowed: false, + error: + `Refusing to install ${os} on ${machine.hostname} (${mac}): ` + + `${os} has no ${machine.arch} netboot artifacts` + + (supported.length > 0 ? ` (only ${supported.join(", ")})` : "") + + `. Use an OS that supports ${machine.arch}.`, + }; + } + + return { allowed: true }; +} diff --git a/bastion/src/bastion/src/templates/boot.ipxe.ts b/bastion/src/bastion/src/templates/boot.ipxe.ts index 95f36d2..318bc88 100644 --- a/bastion/src/bastion/src/templates/boot.ipxe.ts +++ b/bastion/src/bastion/src/templates/boot.ipxe.ts @@ -1,4 +1,55 @@ // iPXE boot script templates for dispatch routing. +// +// Architecture handling: the bastion serves one kernel/initrd pair per architecture. +// x86_64 keeps the original unsuffixed paths so its output is unchanged; every other +// architecture gets an arch-suffixed pair. See stageBootArtifacts() in main.ts for the +// matching staging side, and boot-iso.ts for the same scheme on the ISO path. + +import type { Arch } from "@lab/shared"; + +/** Kernel/initrd URL paths, keyed by architecture. */ +export function kernelPath(arch: Arch): string { + return arch === "x86_64" ? "/vmlinuz" : `/vmlinuz-${arch}`; +} + +export function initrdPath(arch: Arch): string { + return arch === "x86_64" ? "/initrd.img" : `/initrd-${arch}.img`; +} + +/** + * Console arguments per architecture. + * + * arm64 has no VGA text console: a headless machine only talks over the SoC UART, so + * ttyAMA0 must be listed as well. The last console= wins for /dev/console, so serial + * is the interactive one while tty0 still receives boot output on machines with a + * display attached. + */ +const CONSOLE_ARGS: Record = { + x86_64: "console=tty0", + aarch64: "console=tty0 console=ttyAMA0,115200", +}; + +/** + * Anaconda arguments for the graphical-suppression / console setup. + * + * `nomodeset` disables kernel mode setting, which on x86 forces the generic VGA path + * and makes flaky GPU drivers survive the installer. On arm64 it does not mean the + * same thing -- there is no VGA fallback to drop back to, and it can leave the machine + * with no usable console at all -- so arm64 gets explicit console arguments instead. + */ +function installerArgs(arch: Arch): string { + return arch === "x86_64" ? "inst.text nomodeset" : `inst.text ${CONSOLE_ARGS[arch]}`; +} + +/** Extra console arguments appended to templates that don't already set them. */ +function extraConsoleArgs(arch: Arch): string { + return arch === "x86_64" ? "" : ` ${CONSOLE_ARGS[arch]}`; +} + +/** Join kernel arguments, dropping empties so callers can pass optional groups. */ +function joinArgs(...parts: Array): string { + return parts.filter((p) => p !== undefined && p !== "").join(" "); +} export interface BootIpxeParams { serverIp: string; @@ -8,6 +59,11 @@ export interface BootIpxeParams { /** * Initial iPXE boot script that chains to the dispatch endpoint. * This is what dnsmasq serves to iPXE clients via HTTP. + * + * `${buildarch}` is iPXE's own build architecture ("x86_64" or "arm64"), which is the + * one architecture signal available on every path -- network PXE, UEFI HTTP boot and + * the boot ISO alike. DHCP option 93 only reaches dnsmasq, never this HTTP endpoint. + * dispatch prefers the tracked machine record and falls back to this. */ export function renderBootIpxe(params: BootIpxeParams): string { return `#!ipxe @@ -19,7 +75,7 @@ echo Contacting server for instructions... echo ============================================ echo -chain http://${params.serverIp}:${params.httpPort}/dispatch?mac=\${net0/mac} +chain http://${params.serverIp}:${params.httpPort}/dispatch?mac=\${net0/mac}&arch=\${buildarch} `; } @@ -31,7 +87,9 @@ export function renderDiscoverIpxe(params: { serverIp: string; httpPort: number; fedoraMirror: string; + arch: Arch; }): string { + const base = `http://${params.serverIp}:${params.httpPort}`; return `#!ipxe echo @@ -42,8 +100,8 @@ echo Collecting hardware info... echo ============================================= echo -kernel http://${params.serverIp}:${params.httpPort}/vmlinuz inst.ks=http://${params.serverIp}:${params.httpPort}/discover.ks inst.stage2=${params.fedoraMirror} inst.text nomodeset -initrd http://${params.serverIp}:${params.httpPort}/initrd.img +kernel ${base}${kernelPath(params.arch)} inst.ks=${base}/discover.ks inst.stage2=${params.fedoraMirror} ${installerArgs(params.arch)} +initrd ${base}${initrdPath(params.arch)} boot `; } @@ -58,7 +116,9 @@ export function renderInstallIpxe(params: { httpPort: number; fedoraVersion: string; fedoraMirror: string; + arch: Arch; }): string { + const base = `http://${params.serverIp}:${params.httpPort}`; return `#!ipxe echo @@ -69,8 +129,8 @@ echo MAC: ${params.mac} echo ============================================= echo -kernel http://${params.serverIp}:${params.httpPort}/vmlinuz inst.ks=http://${params.serverIp}:${params.httpPort}/ks?mac=${params.mac} inst.repo=${params.fedoraMirror} inst.text nomodeset -initrd http://${params.serverIp}:${params.httpPort}/initrd.img +kernel ${base}${kernelPath(params.arch)} inst.ks=${base}/ks?mac=${params.mac} inst.repo=${params.fedoraMirror} ${installerArgs(params.arch)} +initrd ${base}${initrdPath(params.arch)} boot `; } @@ -78,6 +138,9 @@ boot /** * iPXE script for debug/rescue mode -- boots Fedora installer in rescue mode. * Provides a shell with LVM tools, network, and SSH for inspecting installed systems. + * + * `notice` is shown before the boot line. dispatch uses it to explain why a requested + * --pxe-boot fell back to rescue. */ export function renderDebugIpxe(params: { mac: string; @@ -85,7 +148,11 @@ export function renderDebugIpxe(params: { serverIp: string; httpPort: number; fedoraMirror: string; + arch: Arch; + notice?: string[]; }): string { + const base = `http://${params.serverIp}:${params.httpPort}`; + const notice = (params.notice ?? []).map((line) => `echo ${line}\n`).join(""); return `#!ipxe echo @@ -93,11 +160,11 @@ echo ============================================= echo Lab PXE Bastion - DEBUG/RESCUE MODE echo Target: ${params.hostname} echo MAC: ${params.mac} -echo ============================================= +${notice}echo ============================================= echo -kernel http://${params.serverIp}:${params.httpPort}/vmlinuz inst.rescue inst.text inst.sshd inst.ks=http://${params.serverIp}:${params.httpPort}/debug.ks?mac=${params.mac} inst.stage2=${params.fedoraMirror} -initrd http://${params.serverIp}:${params.httpPort}/initrd.img +kernel ${base}${kernelPath(params.arch)} inst.rescue inst.text inst.sshd inst.ks=${base}/debug.ks?mac=${params.mac} inst.stage2=${params.fedoraMirror}${extraConsoleArgs(params.arch)} +initrd ${base}${initrdPath(params.arch)} boot `; } @@ -106,13 +173,28 @@ boot * iPXE script for PXE-boot debug mode -- boots the installed system's root * filesystem using the bastion's PXE kernel+initrd instead of local GRUB. * Workaround for UEFI firmware bugs that make local disk boot slow. + * + * rootDevice/rootArgs come from the machine's record -- they are not assumed. Our + * Fedora installs use an LVM layout, but nothing guarantees any given machine does, + * and a wrong root= here means an unbootable machine. dispatch refuses to render this + * script without them. */ export function renderPxeBootDebugIpxe(params: { mac: string; hostname: string; serverIp: string; httpPort: number; + arch: Arch; + rootDevice: string; + rootArgs?: string; }): string { + const base = `http://${params.serverIp}:${params.httpPort}`; + const cmdline = joinArgs( + `root=${params.rootDevice}`, + "ro", + params.rootArgs, + CONSOLE_ARGS[params.arch], + ); return `#!ipxe echo @@ -124,12 +206,40 @@ echo Kernel+initrd from PXE, root from NVMe echo ============================================= echo -kernel http://${params.serverIp}:${params.httpPort}/vmlinuz root=/dev/mapper/labvg-root ro rd.lvm.lv=labvg/root rd.lvm.lv=labvg/swap console=tty0 -initrd http://${params.serverIp}:${params.httpPort}/initrd.img +kernel ${base}${kernelPath(params.arch)} ${cmdline} +initrd ${base}${initrdPath(params.arch)} boot `; } +/** + * iPXE script for a request we refuse to serve. + * + * Better a machine that stops with a legible reason on its console than one handed a + * kernel it cannot execute, which fails much later and much less clearly. + */ +export function renderUnsupportedIpxe(params: { + mac: string; + hostname: string; + reason: string; + action?: string; +}): string { + return `#!ipxe + +echo +echo ============================================= +echo Lab PXE Bastion - CANNOT BOOT THIS MACHINE +echo Target: ${params.hostname} +echo MAC: ${params.mac} +echo +echo ${params.reason} +${params.action !== undefined ? `echo\necho Try: ${params.action}\n` : ""}echo ============================================= +echo +sleep 10 +exit 1 +`; +} + /** * iPXE script for already-installed machines -- exits to boot from local disk. */ diff --git a/bastion/src/bastion/src/templates/dnsmasq.conf.ts b/bastion/src/bastion/src/templates/dnsmasq.conf.ts index af972da..fc0762c 100644 --- a/bastion/src/bastion/src/templates/dnsmasq.conf.ts +++ b/bastion/src/bastion/src/templates/dnsmasq.conf.ts @@ -48,15 +48,20 @@ enable-tftp tftp-root=${tftpDir} tftp-no-blocksize -# Detect client architecture -- PXE (TFTP) clients +# Detect client architecture -- PXE (TFTP) clients. +# Values are DHCP option 93 (Client System Architecture), IANA "Processor Architecture +# Types". Getting these wrong means the machine is handed a bootloader its firmware +# cannot execute, and it loops or hangs with no console output. dhcp-match=set:bios,option:client-arch,0 dhcp-match=set:efi-x86_64,option:client-arch,7 dhcp-match=set:efi-x86_64,option:client-arch,9 dhcp-match=set:efi-arm64,option:client-arch,11 -# Detect client architecture -- UEFI HTTP Boot clients (no TFTP size limit) +# Detect client architecture -- UEFI HTTP Boot clients (no TFTP size limit). +# 16 = x64 uefi boot from http, 19 = arm uefi 64 boot from http. +# (20 is pc/at bios boot from http -- not arm64.) dhcp-match=set:httpboot-x86_64,option:client-arch,16 -dhcp-match=set:httpboot-arm64,option:client-arch,20 +dhcp-match=set:httpboot-arm64,option:client-arch,19 # Detect iPXE clients (already chainloaded) dhcp-userclass=set:ipxe,iPXE diff --git a/bastion/src/bastion/tests/arch-dispatch.test.ts b/bastion/src/bastion/tests/arch-dispatch.test.ts new file mode 100644 index 0000000..b9c6054 --- /dev/null +++ b/bastion/src/bastion/tests/arch-dispatch.test.ts @@ -0,0 +1,291 @@ +// aarch64 support in the PXE dispatch path. +// +// The x86_64 side is pinned separately by ipxe-x86-regression.test.ts. + +import { describe, it, expect, beforeEach, afterEach } from "vitest"; +import { mkdirSync, rmSync } from "node:fs"; +import { join } from "node:path"; +import { tmpdir } from "node:os"; +import type { BastionConfig, BastionState, HardwareInfo } from "@lab/shared"; +import { createApp } from "../src/server.js"; +import { resolveArch } from "../src/routes/dispatch.js"; +import { renderDnsmasqConf } from "../src/templates/dnsmasq.conf.js"; +import type { FastifyInstance } from "fastify"; +import type { StateManager } from "../src/services/state.js"; + +function createTestConfig(testDir: string): BastionConfig { + return { + fedoraVersion: "43", + arch: "x86_64", + httpPort: 0, + timezone: "Europe/London", + locale: "en_GB.UTF-8", + bastionDir: testDir, + domain: "test.local", + dhcpMode: "proxy", + dhcpRangeStart: "", + dhcpRangeEnd: "", + ubuntuVersion: "26.04", + ubuntuMirror: "https://releases.ubuntu.com/26.04", + iface: "eth0", + serverIp: "10.0.0.1", + network: "10.0.0.0", + gateway: "10.0.0.1", + sshKeys: ["ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAITEST test@test"], + adminUser: "testadmin", + syslogPort: 15514, + skipDnsmasq: true, + skipArtifacts: true, + fedoraMirror: "https://download.fedoraproject.org/pub/fedora/linux/releases/43/Everything/x86_64/os", + tftpDir: join(testDir, "tftp"), + httpDir: join(testDir, "http"), + stateFile: join(testDir, "state.json"), + }; +} + +function hardware(mac: string, over: Partial = {}): HardwareInfo { + return { + mac, + product: "TestBox", + board: "TestBoard", + serial: "SN123", + manufacturer: "TestCorp", + cpu_model: "Test CPU", + cpu_cores: 4, + memory_gb: 16, + arch: "x86_64", + disks: [], + nics: [], + first_seen: new Date().toISOString(), + last_seen: new Date().toISOString(), + ...over, + }; +} + +const emptyState = (): BastionState => ({ + discovered: {}, install_queue: {}, installed: {}, debug: {}, +}); + +describe("architecture resolution", () => { + const config = createTestConfig("/tmp/unused"); + const mac = "aa:bb:cc:dd:ee:ff"; + + it("prefers the tracked record over what the client reports", () => { + const state = emptyState(); + state.discovered[mac] = hardware(mac, { arch: "aarch64" }); + // Client claims x86_64; the machine record says otherwise and wins. + expect(resolveArch(state, mac, "x86_64", config)).toBe("aarch64"); + }); + + it("falls back to the architecture reported at boot", () => { + expect(resolveArch(emptyState(), mac, "arm64", config)).toBe("aarch64"); + }); + + it("normalises iPXE's arm64 spelling to aarch64", () => { + expect(resolveArch(emptyState(), mac, "arm64", config)).toBe("aarch64"); + expect(resolveArch(emptyState(), mac, "x86_64", config)).toBe("x86_64"); + }); + + it("falls back to the configured default for unknown architectures", () => { + expect(resolveArch(emptyState(), mac, "riscv64", config)).toBe("x86_64"); + expect(resolveArch(emptyState(), mac, undefined, config)).toBe("x86_64"); + }); + + it("reads arch from the installed record for already-provisioned machines", () => { + const state = emptyState(); + state.installed[mac] = { + hostname: "spark", role: "worker", ip: "10.0.0.5", + installed_at: new Date().toISOString(), arch: "aarch64", + }; + expect(resolveArch(state, mac, undefined, config)).toBe("aarch64"); + }); +}); + +describe("aarch64 dispatch", () => { + let testDir: string; + let app: FastifyInstance; + let state: StateManager; + const mac = "aa:bb:cc:dd:ee:ff"; + + beforeEach(() => { + testDir = join(tmpdir(), `bastion-arch-test-${Date.now()}-${Math.random().toString(36).slice(2)}`); + mkdirSync(join(testDir, "http"), { recursive: true }); + mkdirSync(join(testDir, "tftp"), { recursive: true }); + const result = createApp(createTestConfig(testDir)); + app = result.app; + state = result.state; + }); + + afterEach(async () => { + await app.close(); + rmSync(testDir, { recursive: true, force: true }); + }); + + it("serves the aarch64 kernel and initrd to an arm64 client", async () => { + const res = await app.inject({ method: "GET", url: `/dispatch?mac=${mac}&arch=arm64` }); + expect(res.statusCode).toBe(200); + expect(res.body).toContain("/vmlinuz-aarch64"); + expect(res.body).toContain("/initrd-aarch64.img"); + expect(res.body).not.toContain("/vmlinuz "); + }); + + it("points an arm64 client at the aarch64 Fedora mirror", async () => { + const res = await app.inject({ method: "GET", url: `/dispatch?mac=${mac}&arch=arm64` }); + expect(res.body).toContain("Everything/aarch64/os"); + expect(res.body).not.toContain("Everything/x86_64/os"); + }); + + it("uses serial console arguments and not nomodeset on arm64", async () => { + const res = await app.inject({ method: "GET", url: `/dispatch?mac=${mac}&arch=arm64` }); + expect(res.body).toContain("console=ttyAMA0,115200"); + expect(res.body).not.toContain("nomodeset"); + }); + + it("refuses to serve the x86-only Ubuntu kernel to an arm64 client", async () => { + // A machine queued for Ubuntu before it was discovered as aarch64 reaches dispatch + // with no guard having run. Serving it /ubuntu-vmlinuz is the original bug. + state.update((s) => { + s.install_queue[mac] = { + hostname: "arm-node", disk: "", role: "worker", + os: "ubuntu-26.04", queued_at: new Date().toISOString(), + }; + }); + + const res = await app.inject({ method: "GET", url: `/dispatch?mac=${mac}&arch=arm64` }); + expect(res.statusCode).toBe(200); + expect(res.body).toContain("CANNOT BOOT THIS MACHINE"); + expect(res.body).toContain("no aarch64 netboot artifacts"); + expect(res.body).not.toContain("ubuntu-vmlinuz"); + }); + + it("still serves Ubuntu to an x86_64 client", async () => { + state.update((s) => { + s.install_queue[mac] = { + hostname: "x86-node", disk: "", role: "worker", + os: "ubuntu-26.04", queued_at: new Date().toISOString(), + }; + }); + + const res = await app.inject({ method: "GET", url: `/dispatch?mac=${mac}&arch=x86_64` }); + expect(res.body).toContain("ubuntu-vmlinuz"); + expect(res.body).not.toContain("CANNOT BOOT"); + }); + + it("serves a rescue kernel for the recorded architecture, not the requester's", async () => { + // The Spark case: machine known to be aarch64, queued for rescue. + state.update((s) => { + s.discovered[mac] = hardware(mac, { arch: "aarch64" }); + s.debug[mac] = { hostname: "spark-2935", queued_at: new Date().toISOString() }; + }); + + const res = await app.inject({ method: "GET", url: `/dispatch?mac=${mac}` }); + expect(res.body).toContain("DEBUG/RESCUE MODE"); + expect(res.body).toContain("/vmlinuz-aarch64"); + expect(res.body).toContain("inst.rescue"); + expect(res.body).toContain("inst.sshd"); + }); +}); + +describe("--pxe-boot root device", () => { + let testDir: string; + let app: FastifyInstance; + let state: StateManager; + const mac = "aa:bb:cc:dd:ee:ff"; + + beforeEach(() => { + testDir = join(tmpdir(), `bastion-root-test-${Date.now()}-${Math.random().toString(36).slice(2)}`); + mkdirSync(join(testDir, "http"), { recursive: true }); + mkdirSync(join(testDir, "tftp"), { recursive: true }); + const result = createApp(createTestConfig(testDir)); + app = result.app; + state = result.state; + }); + + afterEach(async () => { + await app.close(); + rmSync(testDir, { recursive: true, force: true }); + }); + + it("uses the root device recorded on the machine", async () => { + state.update((s) => { + s.installed[mac] = { + hostname: "worker-1", role: "worker", ip: "10.0.0.50", + installed_at: new Date().toISOString(), + root_device: "/dev/mapper/otherVG-root", + root_args: "rd.lvm.lv=otherVG/root", + }; + s.debug[mac] = { hostname: "worker-1", queued_at: new Date().toISOString(), pxeBoot: true }; + }); + + const res = await app.inject({ method: "GET", url: `/dispatch?mac=${mac}` }); + expect(res.body).toContain("PXE BOOT (debug)"); + expect(res.body).toContain("root=/dev/mapper/otherVG-root"); + expect(res.body).toContain("rd.lvm.lv=otherVG/root"); + // The old hardcoded layout must not leak back in. + expect(res.body).not.toContain("labvg"); + }); + + it("falls back to rescue rather than guessing when no root device is known", async () => { + state.update((s) => { + s.installed[mac] = { + hostname: "spark-2935", role: "worker", ip: "192.168.8.12", + installed_at: new Date().toISOString(), arch: "aarch64", + }; + s.debug[mac] = { hostname: "spark-2935", queued_at: new Date().toISOString(), pxeBoot: true }; + }); + + const res = await app.inject({ method: "GET", url: `/dispatch?mac=${mac}` }); + expect(res.body).toContain("DEBUG/RESCUE MODE"); + expect(res.body).toContain("no root device is recorded"); + expect(res.body).toContain("debug-setup.sh"); + expect(res.body).not.toContain("root="); + // And it is still the right architecture. + expect(res.body).toContain("/vmlinuz-aarch64"); + }); + + it("records a root device reported from a rescue shell without erasing hardware info", async () => { + state.update((s) => { + s.discovered[mac] = hardware(mac, { product: "DGX Spark", manufacturer: "NVIDIA", arch: "aarch64" }); + }); + + const res = await app.inject({ + method: "POST", + url: "/api/discover", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ mac, root_device: "/dev/nvme0n1p2" }), + }); + expect(res.statusCode).toBe(200); + + const hw = state.load().discovered[mac]; + expect(hw?.root_device).toBe("/dev/nvme0n1p2"); + // The partial report must not blank what we already knew. + expect(hw?.product).toBe("DGX Spark"); + expect(hw?.cpu_cores).toBe(4); + expect(hw?.arch).toBe("aarch64"); + }); +}); + +describe("dnsmasq architecture detection", () => { + const conf = renderDnsmasqConf(createTestConfig("/tmp/unused")); + + it("maps DHCP option 93 values to per-architecture bootloaders", () => { + // 11 = ARM 64-bit UEFI + expect(conf).toContain("dhcp-match=set:efi-arm64,option:client-arch,11"); + expect(conf).toContain("dhcp-boot=tag:efi-arm64,tag:!ipxe,ipxe-arm64.efi"); + // 7 / 9 = x64 UEFI, 0 = x86 BIOS + expect(conf).toContain("dhcp-match=set:efi-x86_64,option:client-arch,7"); + expect(conf).toContain("dhcp-match=set:efi-x86_64,option:client-arch,9"); + expect(conf).toContain("dhcp-match=set:bios,option:client-arch,0"); + }); + + it("matches arm64 UEFI HTTP boot on 19, not 20", () => { + // IANA: 19 = arm uefi 64 boot from http, 20 = pc/at bios boot from http. + expect(conf).toContain("dhcp-match=set:httpboot-arm64,option:client-arch,19"); + expect(conf).not.toContain("dhcp-match=set:httpboot-arm64,option:client-arch,20"); + expect(conf).toContain("dhcp-match=set:httpboot-x86_64,option:client-arch,16"); + }); + + it("offers an arm64 PXE service directive in proxy mode", () => { + expect(conf).toContain('pxe-service=tag:!ipxe,ARM64_EFI,"PXE Boot",ipxe-arm64.efi'); + }); +}); diff --git a/bastion/src/bastion/tests/fixtures/ipxe-x86_64-golden.json b/bastion/src/bastion/tests/fixtures/ipxe-x86_64-golden.json new file mode 100644 index 0000000..bc77b81 --- /dev/null +++ b/bastion/src/bastion/tests/fixtures/ipxe-x86_64-golden.json @@ -0,0 +1,8 @@ +{ + "boot": "#!ipxe\n\necho\necho ============================================\necho Lab PXE Bastion\necho Contacting server for instructions...\necho ============================================\necho\n\nchain http://10.0.0.1:8080/dispatch?mac=${net0/mac}\n", + "discover": "#!ipxe\n\necho\necho =============================================\necho Lab PXE Bastion - DISCOVERY MODE\necho MAC: aa:bb:cc:dd:ee:ff\necho Collecting hardware info...\necho =============================================\necho\n\nkernel http://10.0.0.1:8080/vmlinuz inst.ks=http://10.0.0.1:8080/discover.ks inst.stage2=https://download.fedoraproject.org/pub/fedora/linux/releases/43/Everything/x86_64/os inst.text nomodeset\ninitrd http://10.0.0.1:8080/initrd.img\nboot\n", + "install": "#!ipxe\n\necho\necho =============================================\necho Lab PXE Bastion - INSTALLING Fedora 43\necho Target: worker-1\necho MAC: aa:bb:cc:dd:ee:ff\necho =============================================\necho\n\nkernel http://10.0.0.1:8080/vmlinuz inst.ks=http://10.0.0.1:8080/ks?mac=aa:bb:cc:dd:ee:ff inst.repo=https://download.fedoraproject.org/pub/fedora/linux/releases/43/Everything/x86_64/os inst.text nomodeset\ninitrd http://10.0.0.1:8080/initrd.img\nboot\n", + "debug": "#!ipxe\n\necho\necho =============================================\necho Lab PXE Bastion - DEBUG/RESCUE MODE\necho Target: worker-1\necho MAC: aa:bb:cc:dd:ee:ff\necho =============================================\necho\n\nkernel http://10.0.0.1:8080/vmlinuz inst.rescue inst.text inst.sshd inst.ks=http://10.0.0.1:8080/debug.ks?mac=aa:bb:cc:dd:ee:ff inst.stage2=https://download.fedoraproject.org/pub/fedora/linux/releases/43/Everything/x86_64/os\ninitrd http://10.0.0.1:8080/initrd.img\nboot\n", + "pxeBoot": "#!ipxe\n\necho\necho =============================================\necho Lab PXE Bastion - PXE BOOT (debug)\necho Target: worker-1\necho MAC: aa:bb:cc:dd:ee:ff\necho Kernel+initrd from PXE, root from NVMe\necho =============================================\necho\n\nkernel http://10.0.0.1:8080/vmlinuz root=/dev/mapper/labvg-root ro rd.lvm.lv=labvg/root rd.lvm.lv=labvg/swap console=tty0\ninitrd http://10.0.0.1:8080/initrd.img\nboot\n", + "localBoot": "#!ipxe\n\necho\necho =============================================\necho Lab PXE Bastion - worker-1\necho Already installed, booting from local disk\necho =============================================\necho\nsleep 3\nexit 1\n" +} diff --git a/bastion/src/bastion/tests/install-guard.test.ts b/bastion/src/bastion/tests/install-guard.test.ts new file mode 100644 index 0000000..5b8d23e --- /dev/null +++ b/bastion/src/bastion/tests/install-guard.test.ts @@ -0,0 +1,194 @@ +// Installs must never reach a machine running a vendor OS we cannot restore. +// +// This is the guardrail that stops someone reinstalling a DGX Spark at 2am. Rescue is +// deliberately still allowed for the same machines -- that is the whole point. + +import { describe, it, expect, beforeEach, afterEach } from "vitest"; +import { mkdirSync, rmSync } from "node:fs"; +import { join } from "node:path"; +import { tmpdir } from "node:os"; +import type { BastionConfig, BastionState, HardwareInfo } from "@lab/shared"; +import { classifyOnboard } from "@lab/shared"; +import { createApp } from "../src/server.js"; +import { checkInstallAllowed } from "../src/services/install-guard.js"; +import type { FastifyInstance } from "fastify"; +import type { StateManager } from "../src/services/state.js"; + +// The real machines this exists to protect. +const SPARK_2935 = "4c:bb:47:7f:29:35"; +const SPARK_3A1C = "48:21:0b:96:3a:1c"; +const ORDINARY = "aa:bb:cc:dd:ee:ff"; + +function createTestConfig(testDir: string): BastionConfig { + return { + fedoraVersion: "43", arch: "x86_64", httpPort: 0, + timezone: "Europe/London", locale: "en_GB.UTF-8", bastionDir: testDir, + domain: "test.local", dhcpMode: "proxy", dhcpRangeStart: "", dhcpRangeEnd: "", + ubuntuVersion: "26.04", ubuntuMirror: "https://releases.ubuntu.com/26.04", + iface: "eth0", serverIp: "10.0.0.1", network: "10.0.0.0", gateway: "10.0.0.1", + sshKeys: [], adminUser: "testadmin", syslogPort: 15514, + skipDnsmasq: true, skipArtifacts: true, + fedoraMirror: "https://download.fedoraproject.org/pub/fedora/linux/releases/43/Everything/x86_64/os", + tftpDir: join(testDir, "tftp"), httpDir: join(testDir, "http"), + stateFile: join(testDir, "state.json"), + }; +} + +function hardware(mac: string, over: Partial = {}): HardwareInfo { + return { + mac, product: "TestBox", board: "TestBoard", serial: "SN1", + manufacturer: "TestCorp", cpu_model: "Test CPU", cpu_cores: 4, memory_gb: 16, + arch: "x86_64", disks: [], nics: [], + first_seen: new Date().toISOString(), last_seen: new Date().toISOString(), + ...over, + }; +} + +const emptyState = (): BastionState => ({ + discovered: {}, install_queue: {}, installed: {}, debug: {}, +}); + +describe("classifyOnboard", () => { + it("recognises a DGX Spark from its DMI identity", () => { + expect(classifyOnboard({ + mac: ORDINARY, manufacturer: "NVIDIA", product: "NVIDIA DGX Spark", board: "GB10", + })).toEqual({ onboard: "ssh", vendor_os: "dgx-os" }); + }); + + it("recognises the known Sparks even with no DMI recorded", () => { + // Neither Spark has hardware info in bastion state today. A DMI-only rule would + // fail open on exactly the machines this protects. + expect(classifyOnboard({ mac: SPARK_2935 }).onboard).toBe("ssh"); + expect(classifyOnboard({ mac: SPARK_3A1C }).onboard).toBe("ssh"); + }); + + it("treats ordinary hardware as PXE-installable", () => { + expect(classifyOnboard({ + mac: ORDINARY, manufacturer: "Beelink", product: "SER9", board: "SER9", + })).toEqual({ onboard: "pxe" }); + }); + + it("does not override an explicit classification already on the record", () => { + expect(classifyOnboard({ + mac: SPARK_2935, onboard: "pxe", + })).toEqual({ onboard: "pxe" }); + }); +}); + +describe("checkInstallAllowed", () => { + it("refuses a DGX Spark and explains why", () => { + const state = emptyState(); + state.installed[SPARK_2935] = { + hostname: "spark-2935", role: "worker", ip: "192.168.8.12", + installed_at: new Date().toISOString(), arch: "aarch64", + }; + + const result = checkInstallAllowed(state, SPARK_2935, "fedora-43"); + expect(result.allowed).toBe(false); + if (result.allowed === false) { + expect(result.error).toContain("spark-2935"); + expect(result.error).toContain("DGX OS"); + expect(result.error).toContain("provision debug"); + } + }); + + it("refuses a Spark that is only known by MAC", () => { + expect(checkInstallAllowed(emptyState(), SPARK_3A1C, "fedora-43").allowed).toBe(false); + }); + + it("allows an ordinary discovered machine", () => { + const state = emptyState(); + state.discovered[ORDINARY] = hardware(ORDINARY); + expect(checkInstallAllowed(state, ORDINARY, "fedora-43").allowed).toBe(true); + }); + + it("allows Fedora on aarch64", () => { + const state = emptyState(); + state.discovered[ORDINARY] = hardware(ORDINARY, { arch: "aarch64" }); + expect(checkInstallAllowed(state, ORDINARY, "fedora-43").allowed).toBe(true); + }); + + it("refuses Ubuntu on aarch64 -- no netboot artifacts are published", () => { + const state = emptyState(); + state.discovered[ORDINARY] = hardware(ORDINARY, { arch: "aarch64" }); + const result = checkInstallAllowed(state, ORDINARY, "ubuntu-26.04"); + expect(result.allowed).toBe(false); + if (result.allowed === false) { + expect(result.error).toContain("aarch64"); + } + }); + + it("allows Ubuntu on x86_64", () => { + const state = emptyState(); + state.discovered[ORDINARY] = hardware(ORDINARY, { arch: "x86_64" }); + expect(checkInstallAllowed(state, ORDINARY, "ubuntu-26.04").allowed).toBe(true); + }); +}); + +describe("install route enforces the guard", () => { + let testDir: string; + let app: FastifyInstance; + let state: StateManager; + + beforeEach(() => { + testDir = join(tmpdir(), `bastion-guard-test-${Date.now()}-${Math.random().toString(36).slice(2)}`); + mkdirSync(join(testDir, "http"), { recursive: true }); + mkdirSync(join(testDir, "tftp"), { recursive: true }); + const result = createApp(createTestConfig(testDir)); + app = result.app; + state = result.state; + }); + + afterEach(async () => { + await app.close(); + rmSync(testDir, { recursive: true, force: true }); + }); + + it("rejects POST /api/install for a Spark and queues nothing", async () => { + const res = await app.inject({ + method: "POST", + url: "/api/install", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ mac: SPARK_2935, hostname: "spark-2935", role: "worker" }), + }); + + expect(res.statusCode).toBe(409); + expect(JSON.parse(res.body).error).toContain("Refusing to install"); + expect(state.load().install_queue[SPARK_2935]).toBeUndefined(); + }); + + it("still serves rescue to a Spark -- debug is never guarded", async () => { + state.update((s) => { + s.installed[SPARK_2935] = { + hostname: "spark-2935", role: "worker", ip: "192.168.8.12", + installed_at: new Date().toISOString(), arch: "aarch64", + }; + s.debug[SPARK_2935] = { hostname: "spark-2935", queued_at: new Date().toISOString() }; + }); + + const res = await app.inject({ method: "GET", url: `/dispatch?mac=${SPARK_2935}` }); + expect(res.statusCode).toBe(200); + expect(res.body).toContain("DEBUG/RESCUE MODE"); + expect(res.body).toContain("/vmlinuz-aarch64"); + }); + + it("a Spark that PXE boots unqueued gets discovery, never an install", async () => { + const res = await app.inject({ method: "GET", url: `/dispatch?mac=${SPARK_2935}&arch=arm64` }); + expect(res.body).toContain("DISCOVERY MODE"); + expect(res.body).not.toContain("INSTALLING"); + }); + + it("still accepts an ordinary machine", async () => { + state.update((s) => { s.discovered[ORDINARY] = hardware(ORDINARY); }); + + const res = await app.inject({ + method: "POST", + url: "/api/install", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ mac: ORDINARY, hostname: "worker-1", role: "worker" }), + }); + + expect(res.statusCode).toBe(200); + expect(state.load().install_queue[ORDINARY]).toBeDefined(); + }); +}); diff --git a/bastion/src/bastion/tests/ipxe-x86-regression.test.ts b/bastion/src/bastion/tests/ipxe-x86-regression.test.ts new file mode 100644 index 0000000..2329dce --- /dev/null +++ b/bastion/src/bastion/tests/ipxe-x86-regression.test.ts @@ -0,0 +1,89 @@ +// x86_64 iPXE output regression gate. +// +// The aarch64 PXE work must not change what an x86_64 machine is served. The golden +// fixture was dumped from the templates as they stood before that work started, so +// any diff here is a regression, not an improvement. +// +// The one deliberate exception is renderBootIpxe: its chain URL gained +// `&arch=${buildarch}` so the dispatch endpoint can observe the client's +// architecture at boot time. That single change is asserted explicitly below +// rather than being allowed to slip through the byte-for-byte comparison. + +import { describe, it, expect } from "vitest"; +import { readFileSync } from "node:fs"; +import { join } from "node:path"; +import { fileURLToPath } from "node:url"; +import { dirname } from "node:path"; +import { + renderBootIpxe, + renderDiscoverIpxe, + renderInstallIpxe, + renderDebugIpxe, + renderPxeBootDebugIpxe, + renderLocalBootIpxe, +} from "../src/templates/boot.ipxe.js"; + +const here = dirname(fileURLToPath(import.meta.url)); +const golden = JSON.parse( + readFileSync(join(here, "fixtures", "ipxe-x86_64-golden.json"), "utf-8"), +) as Record; + +// Exactly the parameters used to dump the fixture. +const serverIp = "10.0.0.1"; +const httpPort = 8080; +const mac = "aa:bb:cc:dd:ee:ff"; +const hostname = "worker-1"; +const fedoraVersion = "43"; +const fedoraMirror = + "https://download.fedoraproject.org/pub/fedora/linux/releases/43/Everything/x86_64/os"; + +// The x86_64 LVM layout the fixture was captured with. Before this work the values +// were hardcoded in the template; they are now supplied by the caller from machine +// state, so the fixture pins the rendering, not the defaults. +const x86Root = { + rootDevice: "/dev/mapper/labvg-root", + rootArgs: "rd.lvm.lv=labvg/root rd.lvm.lv=labvg/swap", +}; + +describe("x86_64 iPXE output is unchanged", () => { + it("discover script is byte-identical", () => { + const rendered = renderDiscoverIpxe({ + mac, serverIp, httpPort, fedoraMirror, arch: "x86_64", + }); + expect(rendered).toBe(golden["discover"]); + }); + + it("install script is byte-identical", () => { + const rendered = renderInstallIpxe({ + mac, hostname, serverIp, httpPort, fedoraVersion, fedoraMirror, arch: "x86_64", + }); + expect(rendered).toBe(golden["install"]); + }); + + it("debug/rescue script is byte-identical", () => { + const rendered = renderDebugIpxe({ + mac, hostname, serverIp, httpPort, fedoraMirror, arch: "x86_64", + }); + expect(rendered).toBe(golden["debug"]); + }); + + it("--pxe-boot script is byte-identical when state carries the Fedora LVM layout", () => { + const rendered = renderPxeBootDebugIpxe({ + mac, hostname, serverIp, httpPort, arch: "x86_64", ...x86Root, + }); + expect(rendered).toBe(golden["pxeBoot"]); + }); + + it("local boot script is byte-identical", () => { + expect(renderLocalBootIpxe(hostname)).toBe(golden["localBoot"]); + }); + + it("boot.ipxe differs only by the &arch= chain parameter", () => { + const rendered = renderBootIpxe({ serverIp, httpPort }); + // The sole intended difference. + expect(rendered).toBe(golden["boot"].replace( + "/dispatch?mac=${net0/mac}", + "/dispatch?mac=${net0/mac}&arch=${buildarch}", + )); + }); +}); diff --git a/bastion/src/cli/src/api/client.ts b/bastion/src/cli/src/api/client.ts index 52add1e..3069dd7 100644 --- a/bastion/src/cli/src/api/client.ts +++ b/bastion/src/cli/src/api/client.ts @@ -110,6 +110,7 @@ export class LabdClient { memory_gb?: number; arch?: string; disks?: Array<{ name: string; size_gb: number; model: string }>; nics?: Array<{ name: string; mac: string; state: string }>; + root_device?: string; root_args?: string; }): Promise<{ status: string; error?: string }> { return this.request("POST", "/api/machines/discover", { body: data }); } diff --git a/bastion/src/cli/src/commands/debug.ts b/bastion/src/cli/src/commands/debug.ts index 903ca8a..6569efb 100644 --- a/bastion/src/cli/src/commands/debug.ts +++ b/bastion/src/cli/src/commands/debug.ts @@ -8,6 +8,7 @@ import { join } from "node:path"; import { Command } from "commander"; import type { BastionState } from "@lab/shared"; import { getLabdClient } from "../api/config.js"; +import { ROOT_DEVICE_PROBE, parseRootProbe } from "../utils/hardware-probe.js"; /** Resolve a target (hostname, MAC, or IP) to {mac, hostname, ip} from state. */ function resolveTarget( @@ -44,6 +45,54 @@ function resolveTarget( return null; } +/** The local admin account to SSH as (root is not usable — it has no key here). */ +function sshUser(): string { + const adminUser = process.env["SUDO_USER"] ?? process.env["USER"] ?? ""; + return adminUser === "root" ? "" : adminUser; +} + +/** Common ssh arguments, ending with user@host. Null when there is no usable user. */ +function sshBaseArgs(ip: string): string[] | null { + const user = sshUser(); + if (user === "") return null; + + const sudoUser = process.env["SUDO_USER"]; + const realHome = sudoUser !== undefined ? join("/home", sudoUser) : homedir(); + const sshKey = ["id_ed25519", "id_rsa", "id_ecdsa"] + .map((name) => join(realHome, ".ssh", name)) + .find((k) => existsSync(k)); + + return [ + "-o", "StrictHostKeyChecking=no", + "-o", "UserKnownHostsFile=/dev/null", + "-o", "ConnectTimeout=10", + ...(sshKey !== undefined ? ["-i", sshKey] : []), + `${user}@${ip}`, + ]; +} + +/** + * Run a shell script on the target as root and return its stdout, or null. + * + * The script goes over stdin rather than the command line so it can contain quotes + * without a second round of shell escaping. `sudo -n` fails fast instead of hanging on + * a password prompt that would then eat the script. + */ +function sshCapture(ip: string, script: string): string | null { + const base = sshBaseArgs(ip); + if (base === null) return null; + try { + return execFileSync("ssh", [...base, "sudo", "-n", "sh", "-s"], { + input: script, + encoding: "utf-8", + stdio: ["pipe", "pipe", "pipe"], + timeout: 30_000, + }); + } catch { + return null; + } +} + export function registerDebugCommand(parent: Command): void { parent .command("debug ") @@ -71,6 +120,31 @@ export function registerDebugCommand(parent: Command): void { } const { mac, hostname, ip } = resolved; + + // --pxe-boot needs a root= for the installed system. If the machine is still + // reachable, observe it now rather than assuming a disk layout: a wrong root= + // leaves the machine unbootable. If it isn't reachable, dispatch falls back to + // rescue and the operator reports the real one from there. + if (opts.pxeBoot === true && ip !== "") { + const known = state.installed[mac]?.root_device ?? state.discovered[mac]?.root_device; + if (known === undefined || known === "") { + console.log(`No root device recorded for ${hostname}. Probing over SSH...`); + const probe = sshCapture(ip, ROOT_DEVICE_PROBE); + const root = probe === null ? {} : parseRootProbe(probe); + if (root.root_device !== undefined) { + console.log(` root=${root.root_device}${root.root_args !== undefined ? ` ${root.root_args}` : ""}`); + try { + await client.discoverMachine({ mac, ...root }); + } catch (err) { + console.error(` Could not record it: ${err instanceof Error ? err.message : String(err)}`); + } + } else { + console.log(" Probe failed. Booting rescue instead; report the root device with:"); + console.log(" curl http://:8080/debug-setup.sh | bash"); + } + } + } + console.log(`Queuing debug mode for ${hostname} (${mac})...`); try { @@ -86,32 +160,15 @@ export function registerDebugCommand(parent: Command): void { // Try SSH reboot into PXE if (ip !== "") { - const adminUser = process.env["SUDO_USER"] ?? process.env["USER"] ?? ""; - const effectiveUser = adminUser === "root" ? "" : adminUser; - - if (effectiveUser !== "") { - console.log(`\nAttempting SSH reboot into PXE (${effectiveUser}@${ip})...`); - - const sudoUser = process.env["SUDO_USER"]; - const realHome = sudoUser !== undefined ? join("/home", sudoUser) : homedir(); - const keyPaths = [ - join(realHome, ".ssh", "id_ed25519"), - join(realHome, ".ssh", "id_rsa"), - join(realHome, ".ssh", "id_ecdsa"), - ]; - const sshKey = keyPaths.find(k => existsSync(k)); - - const sshArgs = [ - "-o", "StrictHostKeyChecking=no", - "-o", "UserKnownHostsFile=/dev/null", - "-o", "ConnectTimeout=10", - ...(sshKey !== undefined ? ["-i", sshKey] : []), - `${effectiveUser}@${ip}`, - 'PXE_ENTRY=$(sudo efibootmgr | grep -iE "pxe|network|ipv4" | head -1 | grep -oP "Boot\\K[0-9A-F]+"); if [ -n "$PXE_ENTRY" ]; then sudo efibootmgr --bootnext "$PXE_ENTRY" && echo "PXE set as next boot" && sudo reboot; else echo "No PXE boot entry found, rebooting anyway..." && sudo reboot; fi', - ]; + const base = sshBaseArgs(ip); + if (base !== null) { + console.log(`\nAttempting SSH reboot into PXE (${sshUser()}@${ip})...`); try { - execFileSync("ssh", sshArgs, { stdio: "inherit" }); + execFileSync("ssh", [ + ...base, + 'PXE_ENTRY=$(sudo efibootmgr | grep -iE "pxe|network|ipv4" | head -1 | grep -oP "Boot\\K[0-9A-F]+"); if [ -n "$PXE_ENTRY" ]; then sudo efibootmgr --bootnext "$PXE_ENTRY" && echo "PXE set as next boot" && sudo reboot; else echo "No PXE boot entry found, rebooting anyway..." && sudo reboot; fi', + ], { stdio: "inherit" }); } catch { // SSH connection closing during reboot is expected } diff --git a/bastion/src/cli/src/commands/recheck.ts b/bastion/src/cli/src/commands/recheck.ts index 2bfaba3..183440e 100644 --- a/bastion/src/cli/src/commands/recheck.ts +++ b/bastion/src/cli/src/commands/recheck.ts @@ -4,6 +4,7 @@ import type { Command } from "commander"; import { sshExec } from "@lab/modules"; import { getLabdClient } from "../api/config.js"; +import { ROOT_DEVICE_PROBE } from "../utils/hardware-probe.js"; const BOLD = "\x1b[1m"; const GREEN = "\x1b[0;32m"; @@ -24,7 +25,9 @@ const HW_COLLECT_SCRIPT = [ 'N=$(grep -c "^processor" /proc/cpuinfo 2>/dev/null || echo 0)', 'R=$(awk "/MemTotal/ {printf \\"%d\\", \\$2/1024/1024}" /proc/meminfo 2>/dev/null || echo 0)', 'A=$(uname -m)', - 'printf \'{"product":"%s","board":"%s","serial":"%s","manufacturer":"%s","cpu_model":"%s","cpu_cores":%s,"memory_gb":%s,"arch":"%s"}\\n\' "$P" "$B" "$S" "$M" "$C" "$N" "$R" "$A"', + // Root filesystem, so --pxe-boot has a root= to use instead of assuming our layout. + ROOT_DEVICE_PROBE, + 'printf \'{"product":"%s","board":"%s","serial":"%s","manufacturer":"%s","cpu_model":"%s","cpu_cores":%s,"memory_gb":%s,"arch":"%s","root_device":"%s","root_args":"%s"}\\n\' "$P" "$B" "$S" "$M" "$C" "$N" "$R" "$A" "$RD" "$RA"', ].join("; "); export function registerRecheckCommand(parent: Command): void { @@ -81,7 +84,10 @@ export function registerRecheckCommand(parent: Command): void { const cpu = hwData.cpu_model || "?"; const cores = hwData.cpu_cores || "?"; const mem = hwData.memory_gb || "?"; - console.log(`${GREEN}OK${RESET} ${DIM}${cpu}, ${cores} cores, ${mem}GB${RESET}`); + const root = typeof hwData.root_device === "string" && hwData.root_device !== "" + ? `, root=${hwData.root_device}` + : ""; + console.log(`${GREEN}OK${RESET} ${DIM}${cpu}, ${cores} cores, ${mem}GB${root}${RESET}`); updated++; } catch (err) { console.log(`${RED}FAIL${RESET} ${DIM}${err instanceof Error ? err.message : String(err)}${RESET}`); diff --git a/bastion/src/cli/src/utils/hardware-probe.ts b/bastion/src/cli/src/utils/hardware-probe.ts new file mode 100644 index 0000000..26e6164 --- /dev/null +++ b/bastion/src/cli/src/utils/hardware-probe.ts @@ -0,0 +1,59 @@ +// Shell snippets for observing a machine's hardware over SSH. +// +// Pure shell + awk, no Python: these run on whatever the target happens to be, +// including a minimal rescue environment. + +/** + * Report the root filesystem and any dracut arguments needed to assemble it. + * + * Emits two lines: + * ROOT_DEVICE= + * ROOT_ARGS= + * + * Used by `--pxe-boot`, which boots the installed system with a kernel and initrd from + * the network. Getting root= wrong there leaves the machine unbootable, so this observes + * the machine rather than assuming our Fedora LVM layout. + * + * Device form is chosen for stability across reboots: LVM logical volumes keep their + * /dev/mapper path, anything else is reported by UUID, which survives device renumbering. + */ +export const ROOT_DEVICE_PROBE = [ + 'RD=$(findmnt -no SOURCE / 2>/dev/null | head -1)', + 'RA=""', + 'RT=$(lsblk -no TYPE "$RD" 2>/dev/null | head -1)', + 'if [ "$RT" = "lvm" ]; then', + ' VGLV=$(lvs --noheadings -o vg_name,lv_name "$RD" 2>/dev/null | awk \'{print $1"/"$2}\')', + ' [ -n "$VGLV" ] && RA="rd.lvm.lv=$VGLV"', + // Swap must be assembled too or resume= stalls the boot waiting for it. + ' SW=$(awk \'NR>1 {print $1; exit}\' /proc/swaps 2>/dev/null)', + ' if [ -n "$SW" ] && [ "$(lsblk -no TYPE "$SW" 2>/dev/null | head -1)" = "lvm" ]; then', + ' SWVGLV=$(lvs --noheadings -o vg_name,lv_name "$SW" 2>/dev/null | awk \'{print $1"/"$2}\')', + ' [ -n "$SWVGLV" ] && [ "$SWVGLV" != "$VGLV" ] && RA="$RA rd.lvm.lv=$SWVGLV"', + ' fi', + 'elif [ -n "$RD" ]; then', + ' U=$(findmnt -no UUID / 2>/dev/null | head -1)', + ' [ -n "$U" ] && RD="UUID=$U"', + 'fi', + 'printf \'ROOT_DEVICE=%s\\nROOT_ARGS=%s\\n\' "$RD" "$RA"', +].join("; "); + +export interface RootInfo { + root_device?: string; + root_args?: string; +} + +/** Parse the ROOT_DEVICE/ROOT_ARGS lines emitted by ROOT_DEVICE_PROBE. */ +export function parseRootProbe(stdout: string): RootInfo { + const out: RootInfo = {}; + for (const line of stdout.split("\n")) { + const trimmed = line.trim(); + if (trimmed.startsWith("ROOT_DEVICE=")) { + const v = trimmed.slice("ROOT_DEVICE=".length).trim(); + if (v !== "") out.root_device = v; + } else if (trimmed.startsWith("ROOT_ARGS=")) { + const v = trimmed.slice("ROOT_ARGS=".length).trim(); + if (v !== "") out.root_args = v; + } + } + return out; +} diff --git a/bastion/src/labd/src/routes/bastions.ts b/bastion/src/labd/src/routes/bastions.ts index d77a417..26347d4 100644 --- a/bastion/src/labd/src/routes/bastions.ts +++ b/bastion/src/labd/src/routes/bastions.ts @@ -299,6 +299,7 @@ export function registerBastionRoutes(app: FastifyInstance, db: DbClient): void memory_gb?: number; arch?: string; disks?: Array<{ name: string; size_gb: number; model: string }>; nics?: Array<{ name: string; mac: string; state: string }>; + root_device?: string; root_args?: string; }; }>("/api/machines/discover", async (request, reply) => { const data = request.body ?? {}; diff --git a/bastion/src/shared/src/hardware/index.ts b/bastion/src/shared/src/hardware/index.ts new file mode 100644 index 0000000..2b58f98 --- /dev/null +++ b/bastion/src/shared/src/hardware/index.ts @@ -0,0 +1,154 @@ +// Architecture normalisation and machine classification. +// +// Both are derived from what the system already observes about a machine -- never from +// an operator-supplied flag. + +import type { Arch, HardwareInfo, OnboardMethod, OsId } from "../types/index.js"; + +export const SUPPORTED_ARCHES: readonly Arch[] = ["x86_64", "aarch64"] as const; + +/** + * Normalise an architecture string to one we serve boot artifacts for. + * + * Sources and their spellings: + * uname -m -> "x86_64" / "aarch64" + * iPXE ${buildarch}-> "x86_64" / "arm64" + * dpkg/Debian -> "amd64" / "arm64" + * + * Returns undefined for anything we don't serve, so callers fall back rather than + * inventing a kernel path that would 404. + */ +export function normalizeArch(value: string | undefined | null): Arch | undefined { + switch ((value ?? "").trim().toLowerCase()) { + case "x86_64": + case "x86-64": + case "amd64": + return "x86_64"; + case "aarch64": + case "arm64": + return "aarch64"; + default: + return undefined; + } +} + +/** Fedora pxeboot artifact base URL for an architecture. */ +export function fedoraMirrorFor(fedoraVersion: string, arch: Arch): string { + return `https://download.fedoraproject.org/pub/fedora/linux/releases/${fedoraVersion}/Everything/${arch}/os`; +} + +/** + * Which architectures each OS in the pipeline can actually be installed on. + * + * Fedora publishes pxeboot vmlinuz/initrd for both. Ubuntu does not: as of 26.04, + * releases.ubuntu.com publishes amd64 artifacts only, so there is nothing to netboot an + * arm64 machine with. Claiming support would fail at download time with a 404 instead + * of a useful message. + */ +const OS_ARCH_SUPPORT: Record = { + "fedora-43": ["x86_64", "aarch64"], + "ubuntu-26.04": ["x86_64"], +}; + +export function osSupportsArch(os: OsId, arch: Arch): boolean { + return (OS_ARCH_SUPPORT[os] ?? []).includes(arch); +} + +export function archesForOs(os: OsId): readonly Arch[] { + return OS_ARCH_SUPPORT[os] ?? []; +} + +/** + * Machines that run a vendor OS we have no image for. + * + * These are SSH-onboard: we manage userspace, but reinstalling destroys a driver and + * firmware stack our pipeline cannot rebuild. Matched on DMI identity, which is what + * discovery and `provision recheck` both collect. + * + * This is deliberately a property of the machine ("it runs DGX OS"), not a blocklist + * ("never install this MAC"). When a DGX OS image joins the pipeline, teaching the + * installer about vendor_os "dgx-os" is what unblocks these machines -- no entry here + * needs deleting. + */ +interface VendorOsRule { + vendorOs: string; + description: string; + matches: (hw: DmiIdentity) => boolean; +} + +interface DmiIdentity { + manufacturer: string; + product: string; + board: string; +} + +const VENDOR_OS_RULES: readonly VendorOsRule[] = [ + { + vendorOs: "dgx-os", + description: "NVIDIA DGX OS (proprietary driver + firmware stack, no image in our pipeline)", + matches: ({ manufacturer, product, board }) => + (manufacturer.includes("nvidia") || product.includes("nvidia")) && + (product.includes("dgx") || product.includes("spark") || + board.includes("gb10") || product.includes("gb10")), + }, +]; + +/** + * Machines known to run a vendor OS, by MAC. + * + * The DMI rules above only fire once discovery or `provision recheck` has populated a + * hardware record. Machines onboarded over SSH may sit in state for a long time with no + * DMI at all -- which is exactly the state both DGX Sparks are in today -- so a + * DMI-only classifier would fail open on the machines this guard exists to protect. + * + * This is a statement of fact about known hardware ("this box runs DGX OS"), not an + * install policy. Whether that means "refuse" is decided by whether the pipeline has an + * image for that vendor OS. + */ +const KNOWN_VENDOR_OS_MACS: Record = { + "4c:bb:47:7f:29:35": "dgx-os", // spark-2935 + "48:21:0b:96:3a:1c": "dgx-os", // spark-3a1c +}; + +/** + * Classify how a machine should be onboarded, from its hardware record. + * + * An explicit `onboard` already on the record wins: it may have been set by an operator + * or by a rule that has since changed, and silently overriding it would be worse than + * leaving it. + */ +export function classifyOnboard( + hw: Partial> + & { onboard?: OnboardMethod; vendor_os?: string }, +): { onboard: OnboardMethod; vendor_os?: string } { + if (hw.onboard !== undefined) { + return hw.vendor_os !== undefined + ? { onboard: hw.onboard, vendor_os: hw.vendor_os } + : { onboard: hw.onboard }; + } + + const knownVendorOs = KNOWN_VENDOR_OS_MACS[(hw.mac ?? "").toLowerCase().replace(/-/g, ":")]; + if (knownVendorOs !== undefined) { + return { onboard: "ssh", vendor_os: knownVendorOs }; + } + + const identity: DmiIdentity = { + manufacturer: (hw.manufacturer ?? "").toLowerCase(), + product: (hw.product ?? "").toLowerCase(), + board: (hw.board ?? "").toLowerCase(), + }; + + for (const rule of VENDOR_OS_RULES) { + if (rule.matches(identity)) { + return { onboard: "ssh", vendor_os: rule.vendorOs }; + } + } + + return { onboard: "pxe" }; +} + +/** Human-readable reason a vendor-OS machine must not be reinstalled. */ +export function vendorOsDescription(vendorOs: string | undefined): string { + const rule = VENDOR_OS_RULES.find((r) => r.vendorOs === vendorOs); + return rule?.description ?? "a vendor OS with no image in our pipeline"; +} diff --git a/bastion/src/shared/src/index.ts b/bastion/src/shared/src/index.ts index 443edbc..7100826 100644 --- a/bastion/src/shared/src/index.ts +++ b/bastion/src/shared/src/index.ts @@ -1,6 +1,8 @@ export type { OsId, Arch, + OnboardMethod, + RootCandidate, Role, HardwareInfo, InstallConfig, @@ -10,6 +12,16 @@ export type { BastionConfig, } from "./types/index.js"; +export { + SUPPORTED_ARCHES, + normalizeArch, + fedoraMirrorFor, + osSupportsArch, + archesForOs, + classifyOnboard, + vendorOsDescription, +} from "./hardware/index.js"; + export { SUPPORTED_OS, SUPPORTED_ROLES, ROLE_REGISTRY, isValidOsId } from "./types/index.js"; export type { RoleInfo } from "./types/index.js"; diff --git a/bastion/src/shared/src/protocol/index.ts b/bastion/src/shared/src/protocol/index.ts index d7e3018..d3fa3f6 100644 --- a/bastion/src/shared/src/protocol/index.ts +++ b/bastion/src/shared/src/protocol/index.ts @@ -113,7 +113,7 @@ export type LabdBastionMessage = | { type: "command-role-update"; requestId: string; mac: string; role: string } | { type: "command-debug"; requestId: string; mac: string; pxeBoot?: boolean } | { type: "command-register"; requestId: string; mac: string; hostname: string; role: string; ip: string } - | { type: "command-discover"; requestId: string; mac: string; product?: string; board?: string; serial?: string; manufacturer?: string; cpu_model?: string; cpu_cores?: number; memory_gb?: number; arch?: string; disks?: Array<{ name: string; size_gb: number; model: string }>; nics?: Array<{ name: string; mac: string; state: string }> } + | { type: "command-discover"; requestId: string; mac: string; product?: string; board?: string; serial?: string; manufacturer?: string; cpu_model?: string; cpu_cores?: number; memory_gb?: number; arch?: string; disks?: Array<{ name: string; size_gb: number; model: string }>; nics?: Array<{ name: string; mac: string; state: string }>; root_device?: string; root_args?: string } | { type: "server-shutdown"; reconnectAfter: number }; export type BastionMessageType = BastionMessage["type"]; diff --git a/bastion/src/shared/src/types/index.ts b/bastion/src/shared/src/types/index.ts index 510ee31..35a19a9 100644 --- a/bastion/src/shared/src/types/index.ts +++ b/bastion/src/shared/src/types/index.ts @@ -1,6 +1,8 @@ export type { OsId, Arch, + OnboardMethod, + RootCandidate, Role, HardwareInfo, InstallConfig, diff --git a/bastion/src/shared/src/types/state.ts b/bastion/src/shared/src/types/state.ts index b8599fd..6e98a87 100644 --- a/bastion/src/shared/src/types/state.ts +++ b/bastion/src/shared/src/types/state.ts @@ -11,6 +11,16 @@ export function isValidOsId(value: string): value is OsId { return (SUPPORTED_OS as readonly string[]).includes(value); } +/** + * How a machine joins the lab. + * + * "pxe" -- bare metal we install over the network (the default). + * "ssh" -- the machine already runs a vendor OS we cannot reproduce, so we onboard + * over SSH and manage userspace only. Installing would destroy that OS. + * See classifyOnboard() and os-install-research.md. + */ +export type OnboardMethod = "pxe" | "ssh"; + export interface HardwareInfo { mac: string; product: string; @@ -26,6 +36,23 @@ export interface HardwareInfo { first_seen: string; last_seen: string; bastionId?: string; // set when aggregated through labd + // Onboarding classification -- absent means "pxe" (see classifyOnboard) + onboard?: OnboardMethod; + vendor_os?: string; // e.g. "dgx-os": the OS this machine must keep running + // Root filesystem, for booting the installed system over PXE (--pxe-boot). + // Observed from the machine, never assumed. + root_device?: string; // e.g. "/dev/mapper/labvg-root" + root_args?: string; // e.g. "rd.lvm.lv=labvg/root rd.lvm.lv=labvg/swap" + root_candidates?: RootCandidate[]; // reported from a rescue shell when unknown +} + +/** A possible root filesystem found while probing an unreachable machine. */ +export interface RootCandidate { + device: string; // e.g. "/dev/mapper/labvg-root" + args?: string; // extra dracut args needed to assemble it + fstype?: string; + size_gb?: number; + os_release?: string; // PRETTY_NAME from /etc/os-release, if mountable } export type Role = "vanilla" | "worker" | "infra" | "labcontroller"; @@ -103,6 +130,11 @@ export interface InstalledInfo { cpu_cores?: number; memory_gb?: number; arch?: string; + onboard?: OnboardMethod; + vendor_os?: string; + root_device?: string; + root_args?: string; + root_candidates?: RootCandidate[]; } export interface DebugConfig { diff --git a/bastion/tests/integration/arm-pxe-provision.test.ts b/bastion/tests/integration/arm-pxe-provision.test.ts new file mode 100644 index 0000000..94a1fe2 --- /dev/null +++ b/bastion/tests/integration/arm-pxe-provision.test.ts @@ -0,0 +1,537 @@ +// Integration test: aarch64 network PXE boot. +// +// The boot-ISO path already covered ARM (arm-iso-provision.test.ts). This covers the +// network path: DHCP option 93 handing an arm64 client an arm64 iPXE binary, dispatch +// serving an aarch64 kernel, and `provision debug` reaching a rescue shell -- which is +// what the DGX Sparks actually need and could not do. +// +// Two suites, because they cost very different amounts of time: +// +// "ARM PXE rescue" NBP handoff -> rescue with SSH. ~25-30 min +// "ARM PXE install" discover -> install -> installed. ~75-95 min +// +// The rescue suite seeds the machine into state as an already-known aarch64 box rather +// than discovering it first. That is the DGX Spark situation exactly -- SSH-onboarded, +// never PXE-discovered, architecture known only from its record -- and it holds the test +// to one emulated boot. Each boot spends ~15 of its ~18 minutes downloading Anaconda's +// stage2 under TCG, so discovering first would double the runtime without touching any +// code path the rescue boot does not already exercise. +// +// The install suite only runs with ARM_PXE_FULL=1. No ARM machine in the lab is ever +// PXE-installed except the MS-R1, and an hour-plus test that runs by default is a test +// nobody runs. +// +// IMPORTANT: aarch64 has no KVM on an x86_64 host, so all of this is emulated and +// roughly 10x slower than native. +// +// A note for whoever debugs a failure here: if the VM panics with +// VFS: Unable to mount root fs on unknown-block(0,0) +// that is very likely iPXE silently dropping the initrd because the build lacks +// EFI_LOAD_FILE2_PROTOCOL -- on arm64 the kernel EFI stub fetches the initrd over +// LoadFile2, and an iPXE without it accepts the `initrd` line and does nothing. It is +// NOT a reproduction of the DGX Spark kernel bug that motivated this work, despite +// being the identical message. assertIpxeSupportsLoadFile2() below checks the build up +// front so that failure names itself; to check by hand: +// node -e 'const b=require("fs").readFileSync("/usr/share/ipxe/arm64-efi/snponly.efi"); +// console.log(b.indexOf(Buffer.from("c1c00640b3fc3e40996d4a6c8724e06d","hex")))' +// Fedora's ipxe-bootimgs-aarch64-20240119 has it at 0x3bbf0. +// +// Prerequisites: +// - qemu-system-aarch64 (sudo dnf install qemu-system-aarch64) +// - edk2-aarch64 (sudo dnf install edk2-aarch64) +// - ipxe-bootimgs-aarch64 (sudo dnf install ipxe-bootimgs-aarch64) +// - libvirtd, sudo, internet access +// +// Run: sudo ./scripts/test-provision.sh arm-pxe + +import { describe, it, expect, beforeAll, afterAll } from "vitest"; +import { readFileSync, existsSync, mkdirSync, rmSync, copyFileSync, writeFileSync } from "node:fs"; +import { execSync } from "node:child_process"; +import { join } from "node:path"; +import { homedir, tmpdir } from "node:os"; +import { log, waitForSsh } from "./helpers/libvirt.js"; +import { ensurePxeNetwork, destroyPxeNetwork, deleteNftablesRejectRules, PXE_NETWORK_NAME, PXE_GATEWAY, PXE_SUBNET } from "./helpers/pxe-network.js"; +import { createPxeVm, destroyPxeVm, getVmMac, rebootPxeVm, readSerialLog } from "./helpers/pxe-vm.js"; +import { sshExec } from "./helpers/ssh.js"; + +const IPXE_ARM64 = "/usr/share/ipxe/arm64-efi/snponly.efi"; +const AAVMF = "/usr/share/edk2/aarch64/QEMU_EFI.fd"; + +const VM_MEMORY = 4096; +const VM_VCPUS = 2; +const VM_DISK_GB = 250; +const SSH_USER = "lab"; +const BASTION_IP = PXE_GATEWAY; +const DHCP_RANGE_START = `${PXE_SUBNET}.100`; +const DHCP_RANGE_END = `${PXE_SUBNET}.200`; +const SERIAL_PORT = 4555; + +// Emulated aarch64 -- generous timeouts throughout. Measured on an x86_64 host with no +// KVM for aarch64: a single PXE boot to a running Anaconda takes ~18 minutes, almost all +// of it downloading inst.stage2 over the network under TCG. Budget well above that; +// timing out just short of success wastes a whole run. +const LEASE_TIMEOUT_MS = 10 * 60_000; +const DISCOVERY_TIMEOUT_MS = 35 * 60_000; +const INSTALL_TIMEOUT_MS = 75 * 60_000; +const SSH_TIMEOUT_MS = 35 * 60_000; + +const RUN_FULL_INSTALL = process.env["ARM_PXE_FULL"] === "1"; + +function sleep(ms: number): Promise { + return new Promise((r) => setTimeout(r, ms)); +} + +function findSshKey(): { pubKey: string; keyPath: string } { + const candidates: string[] = []; + if (process.env["SSH_KEY_PATH"]) candidates.push(process.env["SSH_KEY_PATH"]); + const homes = [homedir()]; + const sudoUser = process.env["SUDO_USER"]; + if (sudoUser) homes.push(join("/home", sudoUser)); + for (const home of homes) { + for (const name of ["id_ed25519", "id_ecdsa", "id_rsa"]) { + candidates.push(join(home, ".ssh", name)); + } + } + for (const keyPath of candidates) { + if (existsSync(keyPath) && existsSync(`${keyPath}.pub`)) { + return { pubKey: readFileSync(`${keyPath}.pub`, "utf-8").trim(), keyPath }; + } + } + throw new Error("No SSH key found — set SSH_KEY_PATH or ensure keys exist in ~/.ssh/"); +} + +async function pollApi( + url: string, + check: (data: T) => boolean, + timeoutMs: number, + intervalMs = 10_000, +): Promise { + const start = Date.now(); + while (Date.now() - start < timeoutMs) { + try { + const res = await fetch(url); + if (res.ok) { + const data = (await res.json()) as T; + if (check(data)) return data; + } + } catch { /* bastion not up yet, or a network hiccup */ } + await sleep(intervalMs); + } + throw new Error(`Timeout after ${timeoutMs}ms polling ${url}`); +} + +function requirePrerequisites(): void { + if (!existsSync("/usr/bin/qemu-system-aarch64")) { + throw new Error("qemu-system-aarch64 not installed. Run: sudo dnf install qemu-system-aarch64"); + } + if (!existsSync(AAVMF)) { + throw new Error(`AAVMF firmware not found at ${AAVMF}. Run: sudo dnf install edk2-aarch64`); + } + if (!existsSync(IPXE_ARM64)) { + throw new Error(`arm64 iPXE not found at ${IPXE_ARM64}. Run: sudo dnf install ipxe-bootimgs-aarch64`); + } +} + +/** + * Confirm the arm64 iPXE binary implements EFI_LOAD_FILE2_PROTOCOL. + * + * Without it the `initrd` line is accepted and silently ignored, and the kernel panics + * with unknown-block(0,0). Checking here turns a confusing 30-minute boot failure into + * an immediate, explanatory one. + * + * GUID 4006c0c1-fcb3-403e-996d-4a6c8724e06d, little-endian in the binary's GUID table. + */ +function assertIpxeSupportsLoadFile2(): void { + const LOAD_FILE2_GUID = Buffer.from("c1c00640b3fc3e40996d4a6c8724e06d", "hex"); + const binary = readFileSync(IPXE_ARM64); + if (binary.indexOf(LOAD_FILE2_GUID) < 0) { + throw new Error( + `${IPXE_ARM64} does not reference EFI_LOAD_FILE2_PROTOCOL. On arm64 the kernel ` + + `EFI stub fetches the initrd over LoadFile2; without it iPXE drops the initrd ` + + `silently and the kernel panics with "unknown-block(0,0)". Rebuild iPXE with ` + + `LoadFile2, or chainload grubaa64.efi for aarch64 instead.`, + ); + } + log(`iPXE arm64 implements LoadFile2 — initrd will be delivered to the EFI stub`); +} + +interface Harness { + testDir: string; + app: { close: () => Promise }; + stopDnsmasq: () => void; + state: { update: (fn: (s: BastionStateLike) => void) => void }; + vmMac: string; + httpPort: number; +} + +/** Just the parts of BastionState this test seeds. */ +interface BastionStateLike { + discovered: Record>; + installed: Record>; + install_queue: Record>; + debug: Record>; +} + +/** Bring up an isolated network, a bastion with both arch payloads, and an arm64 VM. */ +async function startHarness(vmName: string, httpPort: number, pubKey: string): Promise { + requirePrerequisites(); + assertIpxeSupportsLoadFile2(); + + log("Setting up PXE test network..."); + ensurePxeNetwork(); + + const testDir = join(tmpdir(), `lab-arm-pxe-test-${Date.now()}`); + for (const sub of ["tftp", "http", "logs"]) { + mkdirSync(join(testDir, sub), { recursive: true }); + } + + const { createApp } = await import("../../src/bastion/src/server.js"); + const { loadConfig } = await import("../../src/bastion/src/config.js"); + const { generateDnsmasqConf, startDnsmasq, stopDnsmasq } = await import("../../src/bastion/src/services/dnsmasq.js"); + const { generateDiscoverKickstart } = await import("../../src/bastion/src/services/kickstart-generator.js"); + const { renderBootIpxe, kernelPath, initrdPath } = await import("../../src/bastion/src/templates/boot.ipxe.js"); + // Relative, not "@lab/shared": these tests run from the repo root against sources, + // where the workspace package alias is not resolvable. + const { SUPPORTED_ARCHES, fedoraMirrorFor } = await import("../../src/shared/src/hardware/index.js"); + + const config = loadConfig({ + bastionDir: testDir, + httpPort, + iface: "virbr-pxe", + serverIp: BASTION_IP, + network: `${PXE_SUBNET}.0`, + gateway: BASTION_IP, + dhcpMode: "full", + dhcpRangeStart: DHCP_RANGE_START, + dhcpRangeEnd: DHCP_RANGE_END, + domain: "arm-pxe-test.local", + sshKeys: [pubKey], + adminUser: SSH_USER, + }); + + // iPXE binaries. The arm64 one is the whole point: dnsmasq hands it out on DHCP + // option 93 -- 11 for UEFI PXE (TFTP) and 19 for UEFI HTTP Boot. + // + // They go in BOTH directories, exactly as main.ts stages them. AAVMF prefers HTTP + // Boot, so it is served an http:// URL and fetches from httpDir; a firmware that + // takes the TFTP path reads the same file from tftpDir. Staging only tftpDir gives a + // 404 and "No bootable option or device was found" on the console. + log("Staging iPXE binaries..."); + const ipxeX86 = "/usr/share/ipxe/ipxe-snponly-x86_64.efi"; + copyFileSync(IPXE_ARM64, join(config.tftpDir, "ipxe-arm64.efi")); + copyFileSync(IPXE_ARM64, join(config.httpDir, "ipxe-arm64.efi")); + if (existsSync(ipxeX86)) { + copyFileSync(ipxeX86, join(config.tftpDir, "ipxe.efi")); + copyFileSync(ipxeX86, join(config.httpDir, "ipxe.efi")); + } + + // Fedora kernel + initrd for both architectures, cached across runs. + const cacheDir = "/var/lib/libvirt/images/lab-pxe-cache"; + execSync(`mkdir -p "${cacheDir}"`, { stdio: "pipe" }); + + for (const arch of SUPPORTED_ARCHES) { + const mirror = fedoraMirrorFor(config.fedoraVersion, arch); + const kernelCache = join(cacheDir, `vmlinuz-${arch}`); + const initrdCache = join(cacheDir, `initrd-${arch}.img`); + + if (!existsSync(kernelCache)) { + log(`Downloading Fedora ${config.fedoraVersion} ${arch} kernel...`); + execSync(`curl -# -L -f -o "${kernelCache}" "${mirror}/images/pxeboot/vmlinuz"`, { stdio: "inherit", timeout: 600_000 }); + } + if (!existsSync(initrdCache)) { + log(`Downloading Fedora ${config.fedoraVersion} ${arch} initrd...`); + execSync(`curl -# -L -f -o "${initrdCache}" "${mirror}/images/pxeboot/initrd.img"`, { stdio: "inherit", timeout: 600_000 }); + } + + // Staged under the exact names the iPXE templates will ask for. + copyFileSync(kernelCache, join(config.httpDir, kernelPath(arch))); + copyFileSync(initrdCache, join(config.httpDir, initrdPath(arch))); + log(`Staged ${arch}: ${kernelPath(arch)} + ${initrdPath(arch)}`); + } + + writeFileSync(join(config.httpDir, "discover.ks"), generateDiscoverKickstart(config)); + writeFileSync( + join(config.httpDir, "boot.ipxe"), + renderBootIpxe({ serverIp: config.serverIp, httpPort: config.httpPort }), + ); + generateDnsmasqConf(config); + + const { app, state, syslog } = createApp(config); + await app.listen({ port: config.httpPort, host: "0.0.0.0" }); + syslog.start(); + log(`Bastion HTTP listening on :${config.httpPort}`); + + log("Starting dnsmasq (full DHCP)..."); + startDnsmasq(config).catch((err) => { + log(`dnsmasq failed: ${err instanceof Error ? err.message : String(err)}`); + }); + await sleep(1500); + + log("Creating aarch64 PXE VM (emulated — this is slow)..."); + createPxeVm({ + name: vmName, + memory: VM_MEMORY, + vcpus: VM_VCPUS, + diskSize: VM_DISK_GB, + network: PXE_NETWORK_NAME, + arch: "aarch64", + }); + + const vmMac = getVmMac(vmName); + if (!vmMac) throw new Error("Could not determine VM MAC address"); + log(`ARM VM MAC: ${vmMac}`); + + return { + testDir, + app, + stopDnsmasq, + state: state as unknown as Harness["state"], + vmMac, + httpPort: config.httpPort, + }; +} + +async function stopHarness(vmName: string, harness: Harness | undefined): Promise { + // KEEP_VM=1 leaves the VM, network and bastion up so a failure can be inspected on + // the console. Emulated aarch64 runs cost half an hour; tearing the evidence down + // automatically means paying that again to see what happened. + if (process.env["KEEP_VM"] === "1") { + log(`KEEP_VM=1 — leaving ${vmName} running for inspection.`); + log(` console: sudo virsh screenshot ${vmName} /tmp/vm.ppm`); + log(` serial: socat - TCP:127.0.0.1:${SERIAL_PORT}`); + if (harness) log(` bastion: ${harness.testDir} (still serving on :${harness.httpPort})`); + log(` cleanup: sudo virsh destroy ${vmName}; sudo virsh undefine ${vmName} --remove-all-storage --nvram`); + return; + } + + log("Cleaning up..."); + if (harness) { + await harness.app.close().catch(() => {}); + harness.stopDnsmasq(); + } + destroyPxeVm(vmName); + destroyPxeNetwork(); + if (harness) rmSync(harness.testDir, { recursive: true, force: true }); +} + +/** Read the DHCP lease the bastion handed a MAC. Rescue mode reports no IP itself. */ +function leaseIpFor(testDir: string, mac: string): string | null { + const leaseFile = join(testDir, "dnsmasq.leases"); + if (!existsSync(leaseFile)) return null; + for (const line of readFileSync(leaseFile, "utf-8").split("\n")) { + // + const parts = line.trim().split(/\s+/); + if (parts.length >= 3 && parts[1]?.toLowerCase() === mac.toLowerCase()) { + return parts[2] ?? null; + } + } + return null; +} + +async function waitForLease(testDir: string, mac: string, timeoutMs: number): Promise { + const start = Date.now(); + while (Date.now() - start < timeoutMs) { + const ip = leaseIpFor(testDir, mac); + if (ip !== null) return ip; + await sleep(5000); + } + throw new Error(`No DHCP lease for ${mac} within ${timeoutMs}ms`); +} + +// --------------------------------------------------------------------------- +// Rescue path -- what the DGX Sparks need. +// --------------------------------------------------------------------------- + +describe("ARM PXE rescue", () => { + const VM_NAME = "lab-arm-pxe-rescue"; + const HTTP_PORT = 8096; + let harness: Harness | undefined; + let sshKeyPath: string; + let rescueIp: string; + + beforeAll(async () => { + const { pubKey, keyPath } = findSshKey(); + sshKeyPath = keyPath; + harness = await startHarness(VM_NAME, HTTP_PORT, pubKey); + const { testDir, vmMac, state } = harness; + + // Seed the machine as an already-known aarch64 box queued for rescue. This is the + // DGX Spark situation exactly: SSH-onboarded, never PXE-discovered, architecture + // known only from its record -- and it also keeps the test to a SINGLE emulated + // boot. Each boot spends ~15 minutes pulling Anaconda's stage2 over the network + // under TCG, so discovering first and rescuing second doubles the runtime for no + // extra coverage of the path being tested. Discovery is covered by the full suite. + log(`Seeding ${vmMac} as a known aarch64 machine queued for rescue...`); + state.update((s) => { + s.discovered[vmMac] = { + mac: vmMac, + product: "Test ARM64 Machine", + board: "virt", + serial: "SN-ARM64", + manufacturer: "QEMU", + cpu_model: "cortex-a57", + cpu_cores: VM_VCPUS, + memory_gb: 4, + arch: "aarch64", + disks: [], + nics: [], + first_seen: new Date().toISOString(), + last_seen: new Date().toISOString(), + }; + s.debug[vmMac] = { hostname: "arm-rescue-test", queued_at: new Date().toISOString() }; + }); + + // Restart so the VM boots against the seeded state. createPxeVm already started it. + rebootPxeVm(VM_NAME); + await sleep(5_000); + deleteNftablesRejectRules(); + + // The whole chain now runs once: DHCP option 93 -> arm64 iPXE -> /boot.ipxe -> + // /dispatch (architecture from the record, not the query) -> aarch64 kernel + + // initrd -> Anaconda rescue -> sshd. Reaching a shell at all proves iPXE handed + // the initrd to the EFI stub over LoadFile2; without it the kernel panics first. + log("Waiting for the rescue environment's DHCP lease..."); + rescueIp = await waitForLease(testDir, vmMac, LEASE_TIMEOUT_MS); + log(`Rescue IP: ${rescueIp}`); + + log("Waiting for SSH into the rescue shell (started by inst.sshd)..."); + log("(emulated aarch64 — Anaconda's stage2 download dominates; be patient)"); + await waitForSsh(rescueIp, "root", SSH_TIMEOUT_MS, sshKeyPath).catch(async (err) => { + log("Rescue SSH timed out. Serial console:"); + try { + log(await readSerialLog(SERIAL_PORT, { lastLines: 100, timeoutMs: 15_000 })); + } catch { /* console unavailable */ } + throw err; + }); + log("ARM PXE rescue reached."); + }, LEASE_TIMEOUT_MS + SSH_TIMEOUT_MS + 300_000); + + afterAll(async () => { await stopHarness(VM_NAME, harness); }); + + it("resolved the architecture from the machine record", async () => { + const res = await fetch(`http://${BASTION_IP}:${HTTP_PORT}/api/machines`); + const data = (await res.json()) as { discovered: Record }; + expect(data.discovered[harness!.vmMac]?.arch).toBe("aarch64"); + }); + + it("rescue shell is reachable over SSH and is aarch64", () => { + const result = sshExec(rescueIp, "root", "uname -m", { keyPath: sshKeyPath, timeout: 60_000 }); + expect(result.exitCode).toBe(0); + expect(result.stdout.trim()).toBe("aarch64"); + }); + + it("booted an initramfs — the LoadFile2 path worked", () => { + // If iPXE had dropped the initrd the kernel would never have reached userspace at + // all, but assert it explicitly so a regression names itself. + const result = sshExec(rescueIp, "root", "cat /proc/cmdline; ls /run/install", { + keyPath: sshKeyPath, timeout: 60_000, + }); + expect(result.exitCode).toBe(0); + expect(result.stdout).toContain("inst.rescue"); + }); + + it("rescue kernel came from the bastion over HTTP", () => { + const result = sshExec(rescueIp, "root", "cat /proc/cmdline", { keyPath: sshKeyPath, timeout: 60_000 }); + expect(result.stdout).toContain(`${BASTION_IP}:${HTTP_PORT}`); + // arm64 gets serial console arguments, never nomodeset. + expect(result.stdout).toContain("console=ttyAMA0"); + expect(result.stdout).not.toContain("nomodeset"); + }); + + it("has LVM tools available for inspecting an installed system", () => { + const result = sshExec(rescueIp, "root", "command -v vgchange && command -v lsblk", { + keyPath: sshKeyPath, timeout: 60_000, + }); + expect(result.exitCode).toBe(0); + }); +}); + +// --------------------------------------------------------------------------- +// Full install -- opt-in, ~60-90 minutes emulated. +// --------------------------------------------------------------------------- + +describe.runIf(RUN_FULL_INSTALL)("ARM PXE install", () => { + const VM_NAME = "lab-arm-pxe-install"; + const HTTP_PORT = 8095; + let harness: Harness | undefined; + let sshKeyPath: string; + let vmIp: string; + + beforeAll(async () => { + const { pubKey, keyPath } = findSshKey(); + sshKeyPath = keyPath; + harness = await startHarness(VM_NAME, HTTP_PORT, pubKey); + const { vmMac } = harness; + + log("Waiting for aarch64 discovery..."); + await pollApi<{ discovered: Record }>( + `http://${BASTION_IP}:${HTTP_PORT}/api/machines`, + (data) => vmMac in data.discovered, + DISCOVERY_TIMEOUT_MS, + ); + log("Discovered. Queueing install..."); + + const installRes = await fetch(`http://${BASTION_IP}:${HTTP_PORT}/api/install`, { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ mac: vmMac, hostname: VM_NAME, disk: "", role: "vanilla" }), + }); + expect(installRes.status).toBe(200); + + await sleep(30_000); + rebootPxeVm(VM_NAME); + + log("Waiting for the emulated aarch64 install (60-90 min)..."); + type LogsResponse = { status: string; progress: string; ip?: string }; + const final = await pollApi( + `http://${BASTION_IP}:${HTTP_PORT}/api/logs/${encodeURIComponent(vmMac)}`, + (d) => d.status === "installed" || d.progress === "error", + INSTALL_TIMEOUT_MS, + 30_000, + ); + + if (final.progress === "error") { + const logs = await (await fetch(`http://${BASTION_IP}:${HTTP_PORT}/api/logs/${encodeURIComponent(vmMac)}`)).json(); + log(`ARM install FAILED: ${JSON.stringify(logs, null, 2)}`); + throw new Error("ARM PXE install failed — see logs above"); + } + + vmIp = final.ip ?? ""; + log(`ARM install complete. IP: ${vmIp}`); + + await sleep(30_000); + rebootPxeVm(VM_NAME); + await sleep(5_000); + deleteNftablesRejectRules(); + await waitForSsh(vmIp, SSH_USER, SSH_TIMEOUT_MS, sshKeyPath); + }, DISCOVERY_TIMEOUT_MS + INSTALL_TIMEOUT_MS + SSH_TIMEOUT_MS + 600_000); + + afterAll(async () => { await stopHarness(VM_NAME, harness); }); + + it("machine reached installed state", async () => { + const res = await fetch(`http://${BASTION_IP}:${HTTP_PORT}/api/machines`); + const data = (await res.json()) as { installed: Record }; + expect(data.installed[harness!.vmMac]?.hostname).toBe(VM_NAME); + }); + + it("installed system is aarch64", () => { + const result = sshExec(vmIp, SSH_USER, "uname -m", { keyPath: sshKeyPath, timeout: 60_000 }); + expect(result.stdout.trim()).toBe("aarch64"); + }); + + it("SSH works with the admin user", () => { + const result = sshExec(vmIp, SSH_USER, "whoami", { keyPath: sshKeyPath, timeout: 60_000 }); + expect(result.stdout.trim()).toBe(SSH_USER); + }); + + it("LVM layout is correct", () => { + const result = sshExec(vmIp, SSH_USER, "sudo lvs labvg --noheadings -o lv_name", { + keyPath: sshKeyPath, timeout: 60_000, + }); + expect(result.exitCode).toBe(0); + const lvs = result.stdout.trim().split("\n").map((l) => l.trim()); + for (const expected of ["root", "var", "varlog", "swap", "home", "srv"]) { + expect(lvs).toContain(expected); + } + }); +}); diff --git a/bastion/tests/integration/pxe-rescue.test.ts b/bastion/tests/integration/pxe-rescue.test.ts new file mode 100644 index 0000000..768ba26 --- /dev/null +++ b/bastion/tests/integration/pxe-rescue.test.ts @@ -0,0 +1,210 @@ +// Integration test: `labctl provision debug` -> Anaconda rescue with SSH, on x86_64. +// +// The rescue path had no test coverage on any architecture, which matters because it is +// the lab's recovery tool of last resort -- the thing you reach for when a machine will +// not boot. It runs here on x86_64 with KVM so it completes in minutes; the aarch64 +// equivalent is the same code path with a different kernel, but is emulated and far too +// slow to iterate on. +// +// Run: sudo ./scripts/test-provision.sh rescue + +import { describe, it, expect, beforeAll, afterAll } from "vitest"; +import { readFileSync, existsSync, mkdirSync, rmSync, copyFileSync, writeFileSync } from "node:fs"; +import { execSync } from "node:child_process"; +import { join } from "node:path"; +import { homedir, tmpdir } from "node:os"; +import { log, waitForSsh } from "./helpers/libvirt.js"; +import { ensurePxeNetwork, destroyPxeNetwork, deleteNftablesRejectRules, PXE_NETWORK_NAME, PXE_GATEWAY, PXE_SUBNET } from "./helpers/pxe-network.js"; +import { createPxeVm, destroyPxeVm, getVmMac, rebootPxeVm, readSerialLog } from "./helpers/pxe-vm.js"; +import { sshExec } from "./helpers/ssh.js"; + +const VM_NAME = "lab-pxe-rescue-test"; +const HTTP_PORT = 8094; +const VM_MEMORY = 4096; +const VM_VCPUS = 4; +const VM_DISK_GB = 20; +const BASTION_IP = PXE_GATEWAY; +const SERIAL_PORT = 4555; + +const LEASE_TIMEOUT_MS = 8 * 60_000; +const SSH_TIMEOUT_MS = 15 * 60_000; + +function sleep(ms: number): Promise { + return new Promise((r) => setTimeout(r, ms)); +} + +function findSshKey(): { pubKey: string; keyPath: string } { + const candidates: string[] = []; + if (process.env["SSH_KEY_PATH"]) candidates.push(process.env["SSH_KEY_PATH"]); + const homes = [homedir()]; + const sudoUser = process.env["SUDO_USER"]; + if (sudoUser) homes.push(join("/home", sudoUser)); + for (const home of homes) { + for (const name of ["id_ed25519", "id_ecdsa", "id_rsa"]) candidates.push(join(home, ".ssh", name)); + } + for (const keyPath of candidates) { + if (existsSync(keyPath) && existsSync(`${keyPath}.pub`)) { + return { pubKey: readFileSync(`${keyPath}.pub`, "utf-8").trim(), keyPath }; + } + } + throw new Error("No SSH key found — set SSH_KEY_PATH or ensure keys exist in ~/.ssh/"); +} + +function leaseIpFor(testDir: string, mac: string): string | null { + const leaseFile = join(testDir, "dnsmasq.leases"); + if (!existsSync(leaseFile)) return null; + for (const line of readFileSync(leaseFile, "utf-8").split("\n")) { + const parts = line.trim().split(/\s+/); + if (parts.length >= 3 && parts[1]?.toLowerCase() === mac.toLowerCase()) return parts[2] ?? null; + } + return null; +} + +async function waitForLease(testDir: string, mac: string, timeoutMs: number): Promise { + const start = Date.now(); + while (Date.now() - start < timeoutMs) { + const ip = leaseIpFor(testDir, mac); + if (ip !== null) return ip; + await sleep(5000); + } + throw new Error(`No DHCP lease for ${mac} within ${timeoutMs}ms`); +} + +// Suite name must not be a substring of "ARM PXE rescue" -- vitest -t matches +// substrings, so a looser name here would drag the emulated aarch64 suite in with it. +describe("x86 rescue boot", () => { + let app: { close: () => Promise }; + let stopDnsmasqFn: () => void; + let testDir: string; + let vmMac: string; + let rescueIp: string; + let sshKeyPath: string; + + beforeAll(async () => { + const { pubKey, keyPath } = findSshKey(); + sshKeyPath = keyPath; + + log("Setting up PXE test network..."); + ensurePxeNetwork(); + + testDir = join(tmpdir(), `lab-pxe-rescue-${Date.now()}`); + for (const sub of ["tftp", "http", "logs"]) mkdirSync(join(testDir, sub), { recursive: true }); + + const { createApp } = await import("../../src/bastion/src/server.js"); + const { loadConfig } = await import("../../src/bastion/src/config.js"); + const { generateDnsmasqConf, startDnsmasq, stopDnsmasq } = await import("../../src/bastion/src/services/dnsmasq.js"); + const { renderBootIpxe, kernelPath, initrdPath } = await import("../../src/bastion/src/templates/boot.ipxe.js"); + stopDnsmasqFn = stopDnsmasq; + + const config = loadConfig({ + bastionDir: testDir, + httpPort: HTTP_PORT, + iface: "virbr-pxe", + serverIp: BASTION_IP, + network: `${PXE_SUBNET}.0`, + gateway: BASTION_IP, + dhcpMode: "full", + dhcpRangeStart: `${PXE_SUBNET}.100`, + dhcpRangeEnd: `${PXE_SUBNET}.200`, + domain: "rescue-test.local", + sshKeys: [pubKey], + adminUser: "lab", + }); + + // iPXE in both dirs: TFTP PXE and UEFI HTTP Boot are both possible, and OVMF picks. + const ipxeX86 = "/usr/share/ipxe/ipxe-snponly-x86_64.efi"; + if (!existsSync(ipxeX86)) throw new Error(`iPXE not found: ${ipxeX86}`); + copyFileSync(ipxeX86, join(config.tftpDir, "ipxe.efi")); + copyFileSync(ipxeX86, join(config.httpDir, "ipxe.efi")); + + const cacheDir = "/var/lib/libvirt/images/lab-pxe-cache"; + execSync(`mkdir -p "${cacheDir}"`, { stdio: "pipe" }); + const kernelCache = join(cacheDir, "vmlinuz-x86_64"); + const initrdCache = join(cacheDir, "initrd-x86_64.img"); + if (!existsSync(kernelCache)) { + log("Downloading Fedora x86_64 kernel..."); + execSync(`curl -# -L -f -o "${kernelCache}" "${config.fedoraMirror}/images/pxeboot/vmlinuz"`, { stdio: "inherit", timeout: 600_000 }); + } + if (!existsSync(initrdCache)) { + log("Downloading Fedora x86_64 initrd..."); + execSync(`curl -# -L -f -o "${initrdCache}" "${config.fedoraMirror}/images/pxeboot/initrd.img"`, { stdio: "inherit", timeout: 600_000 }); + } + copyFileSync(kernelCache, join(config.httpDir, kernelPath("x86_64"))); + copyFileSync(initrdCache, join(config.httpDir, initrdPath("x86_64"))); + + writeFileSync(join(config.httpDir, "boot.ipxe"), renderBootIpxe({ serverIp: config.serverIp, httpPort: config.httpPort })); + generateDnsmasqConf(config); + + const { app: fastify, state, syslog } = createApp(config); + app = fastify; + await fastify.listen({ port: config.httpPort, host: "0.0.0.0" }); + syslog.start(); + log(`Bastion HTTP listening on :${HTTP_PORT}`); + + startDnsmasq(config).catch((err) => log(`dnsmasq failed: ${err instanceof Error ? err.message : String(err)}`)); + await sleep(1500); + + log("Creating x86_64 PXE VM (KVM)..."); + createPxeVm({ name: VM_NAME, memory: VM_MEMORY, vcpus: VM_VCPUS, diskSize: VM_DISK_GB, network: PXE_NETWORK_NAME }); + const mac = getVmMac(VM_NAME); + if (!mac) throw new Error("Could not determine VM MAC"); + vmMac = mac; + log(`VM MAC: ${vmMac}`); + + // Queue rescue directly, as `labctl provision debug` does. + log("Queueing debug/rescue mode..."); + state.update((s) => { + s.debug[vmMac] = { hostname: "rescue-test", queued_at: new Date().toISOString() }; + }); + + rebootPxeVm(VM_NAME); + await sleep(5_000); + deleteNftablesRejectRules(); + + rescueIp = await waitForLease(testDir, vmMac, LEASE_TIMEOUT_MS); + log(`Rescue IP: ${rescueIp}`); + + log("Waiting for SSH into the rescue shell (inst.sshd)..."); + await waitForSsh(rescueIp, "root", SSH_TIMEOUT_MS, sshKeyPath).catch(async (err) => { + log("Rescue SSH timed out. Serial console:"); + try { log(await readSerialLog(SERIAL_PORT, { lastLines: 120, timeoutMs: 20_000 })); } catch { /* none */ } + throw err; + }); + log("Rescue shell reachable."); + }, LEASE_TIMEOUT_MS + SSH_TIMEOUT_MS + 300_000); + + afterAll(async () => { + if (process.env["KEEP_VM"] === "1") { + log(`KEEP_VM=1 — leaving ${VM_NAME} up (serial: socat - TCP:127.0.0.1:${SERIAL_PORT})`); + return; + } + log("Cleaning up..."); + if (app) await app.close().catch(() => {}); + if (stopDnsmasqFn) stopDnsmasqFn(); + destroyPxeVm(VM_NAME); + destroyPxeNetwork(); + if (testDir) rmSync(testDir, { recursive: true, force: true }); + }); + + it("rescue shell is reachable over SSH as root", () => { + const result = sshExec(rescueIp, "root", "whoami", { keyPath: sshKeyPath, timeout: 60_000 }); + expect(result.exitCode).toBe(0); + expect(result.stdout.trim()).toBe("root"); + }); + + it("is the Anaconda rescue environment", () => { + const result = sshExec(rescueIp, "root", "cat /proc/cmdline", { keyPath: sshKeyPath, timeout: 60_000 }); + expect(result.stdout).toContain("inst.rescue"); + expect(result.stdout).toContain("inst.sshd"); + }); + + it("kernel and initrd came from the bastion", () => { + const result = sshExec(rescueIp, "root", "cat /proc/cmdline", { keyPath: sshKeyPath, timeout: 60_000 }); + expect(result.stdout).toContain(`${BASTION_IP}:${HTTP_PORT}`); + }); + + it("has LVM tools for inspecting an installed system", () => { + const result = sshExec(rescueIp, "root", "command -v vgchange && command -v lsblk", { keyPath: sshKeyPath, timeout: 60_000 }); + expect(result.exitCode).toBe(0); + }); +});