diff --git a/bastion/docs/ARCHITECTURE.md b/bastion/docs/ARCHITECTURE.md index 3cd9036..52963a3 100644 --- a/bastion/docs/ARCHITECTURE.md +++ b/bastion/docs/ARCHITECTURE.md @@ -89,6 +89,83 @@ Side paths: --- +## Multi-architecture PXE + +The bastion serves both `x86_64` and `aarch64` over the network. Nothing about this is +operator-configured -- there is no `--arch` flag, by design. + +### How a client's architecture is decided + +1. **DHCP option 93** (Client System Architecture) picks the *bootloader*. dnsmasq matches + it and hands out a matching iPXE binary: + + | Option 93 | Client | Served | + |---|---|---| + | `0` | x86 BIOS | `undionly.kpxe` (TFTP) | + | `7`, `9` | x64 UEFI | `ipxe.efi` (TFTP) | + | `11` | **ARM64 UEFI** | `ipxe-arm64.efi` (TFTP) | + | `16` | x64 UEFI HTTP Boot | `http://…/ipxe.efi` | + | `19` | **ARM64 UEFI HTTP Boot** | `http://…/ipxe-arm64.efi` | + + Values come from the IANA Processor Architecture Types registry. Note `19`, not `20` -- + `20` is *pc/at bios boot from http*. EDK2/AAVMF prefers HTTP Boot over TFTP PXE, so the + iPXE binaries are staged in **both** `tftpDir` and `httpDir` (symlinked by `main.ts`). + +2. **`/dispatch` picks the kernel.** Option 93 never reaches the HTTP endpoint, so + `boot.ipxe` passes iPXE's own `${buildarch}` as `?arch=`. `resolveArch()` prefers, in + order: the tracked machine record → the reported `?arch=` → the configured default. + The record wins because it is what we observed on the machine itself. + +### Artifact naming + +`x86_64` keeps the original unsuffixed paths so its rendered iPXE scripts are unchanged; +everything else is suffixed. `kernelPath()` / `initrdPath()` in `templates/boot.ipxe.ts` +are the single source of truth, used by both the templates and `main.ts` staging. + +| arch | kernel | initrd | +|---|---|---| +| `x86_64` | `/vmlinuz` | `/initrd.img` | +| `aarch64` | `/vmlinuz-aarch64` | `/initrd-aarch64.img` | + +`tests/ipxe-x86-regression.test.ts` pins the x86_64 output against a golden fixture. + +### arm64 gotchas + +- **LoadFile2 is mandatory.** arm64 has no `HdrS` boot protocol; the kernel's EFI stub + fetches the initrd over the UEFI `EFI_LOAD_FILE2_PROTOCOL`. An iPXE build without it + accepts the `initrd` line, silently drops it, and the kernel panics with + `VFS: Unable to mount root fs on unknown-block(0,0)`. Fedora's + `ipxe-bootimgs-aarch64` implements it; the integration test asserts this up front so + the failure names itself instead of looking like a disk problem. +- **`nomodeset` is x86-only.** On arm64 there is no VGA path to fall back to. aarch64 gets + `console=tty0 console=ttyAMA0,115200` instead — the last `console=` wins for + `/dev/console`, so serial is the interactive one. +- **Ubuntu is x86_64-only.** `releases.ubuntu.com` publishes no arm64 netboot artifacts. + `osSupportsArch()` encodes this, and both the install guard and `/dispatch` refuse the + combination rather than serving an x86 kernel to an ARM machine. + +--- + +## Onboarding classification (vendor OS) + +Machines carry an `onboard` field: `"pxe"` (default) or `"ssh"`, plus `vendor_os` naming +what they run. `classifyOnboard()` in `@lab/shared` sets it from DMI identity, with known +hardware also matched by MAC — a machine can sit in state for a long time with no DMI, and +a DMI-only rule would fail open exactly where it matters. + +`onboard: "ssh"` means *we cannot rebuild this machine's OS*. Installs are refused at both +entry points (`/api/install` and the labd `command-install` handler) with an error naming +the machine and pointing at `provision debug`. **Rescue is never guarded** — being unable +to reinstall a machine is precisely when a rescue shell is needed. + +This is a fact about the machine, not a blocklist. The refusal follows from "no image in +our pipeline restores `vendor_os`", so adding a DGX OS image to the pipeline is what +unblocks the DGX Sparks — no entry needs deleting. + +Current classifications: NVIDIA DGX Spark (`spark-2935`, `spark-3a1c`) → `dgx-os`. + +--- + ## Packages ### Monorepo Structure diff --git a/bastion/src/bastion/src/routes/dispatch.ts b/bastion/src/bastion/src/routes/dispatch.ts index 85e44aa..5b1a8a6 100644 --- a/bastion/src/bastion/src/routes/dispatch.ts +++ b/bastion/src/bastion/src/routes/dispatch.ts @@ -186,7 +186,9 @@ echo "===============================" "NOTE: --pxe-boot requested, but no root device is recorded", " for this machine. Booting rescue instead.", " From the rescue shell, run:", - ` curl http://${config.serverIp}:${config.httpPort}/debug-setup.sh | bash`, + // No pipe or && here: iPXE treats || and && as command separators, so keep + // the printed command free of anything its parser might claim. + ` curl -s http://${config.serverIp}:${config.httpPort}/debug-setup.sh -o /tmp/s.sh ; sh /tmp/s.sh`, " then retry --pxe-boot.", ] : undefined; diff --git a/bastion/tests/integration/arm-pxe-provision.test.ts b/bastion/tests/integration/arm-pxe-provision.test.ts index 44b4d6f..30ef2a5 100644 --- a/bastion/tests/integration/arm-pxe-provision.test.ts +++ b/bastion/tests/integration/arm-pxe-provision.test.ts @@ -191,12 +191,20 @@ async function startHarness(vmName: string, httpPort: number, pubKey: string): P }); // iPXE binaries. The arm64 one is the whole point: dnsmasq hands it out on DHCP - // option 93 = 11, and x86_64 is staged too so the config renders as it does in - // production. + // option 93 -- 11 for UEFI PXE (TFTP) and 19 for UEFI HTTP Boot. + // + // They go in BOTH directories, exactly as main.ts stages them. AAVMF prefers HTTP + // Boot, so it is served an http:// URL and fetches from httpDir; a firmware that + // takes the TFTP path reads the same file from tftpDir. Staging only tftpDir gives a + // 404 and "No bootable option or device was found" on the console. log("Staging iPXE binaries..."); - copyFileSync(IPXE_ARM64, join(config.tftpDir, "ipxe-arm64.efi")); const ipxeX86 = "/usr/share/ipxe/ipxe-snponly-x86_64.efi"; - if (existsSync(ipxeX86)) copyFileSync(ipxeX86, join(config.tftpDir, "ipxe.efi")); + copyFileSync(IPXE_ARM64, join(config.tftpDir, "ipxe-arm64.efi")); + copyFileSync(IPXE_ARM64, join(config.httpDir, "ipxe-arm64.efi")); + if (existsSync(ipxeX86)) { + copyFileSync(ipxeX86, join(config.tftpDir, "ipxe.efi")); + copyFileSync(ipxeX86, join(config.httpDir, "ipxe.efi")); + } // Fedora kernel + initrd for both architectures, cached across runs. const cacheDir = "/var/lib/libvirt/images/lab-pxe-cache";