michal/lab
1
0
Fork 0
Code Issues Pull Requests 3 Actions Packages Projects Releases Wiki Activity

test(bastion): cover the rescue boot path and record what it exposed
Some checks failed
CI/CD / lint (pull_request) Failing after 9s
Details
CI/CD / test (pull_request) Failing after 8s
Details
CI/CD / typecheck (pull_request) Failing after 23s
Details
CI/CD / build (pull_request) Has been skipped
Details
CI/CD / publish-rpm (pull_request) Has been skipped
Details
CI/CD / publish-deb (pull_request) Has been skipped
Details

Browse Source 
`provision debug` is the lab's recovery tool of last resort and had no
integration coverage on any architecture. Adding it -- x86_64 on KVM so
it runs in ~15 minutes, plus the aarch64 equivalent -- showed the rescue
environment coming up correctly but nothing ever listening on port 22.

Reproduced on both architectures, so it is neither ARM-specific nor an
emulation artefact, and it is orthogonal to the multi-arch work: x86_64
is unchanged by that. Documented in ARCHITECTURE.md with the leads worth
checking, rather than left as a silent gap.

Also restructures the ARM rescue suite to seed the machine into state
instead of discovering it first. That mirrors the DGX Spark situation --
SSH-onboarded, never PXE-discovered, architecture known only from its
record -- and holds the test to one emulated boot, since each spends
~15 of its ~18 minutes fetching Anaconda's stage2 under TCG.

KEEP_VM=1 leaves the VM up on failure; half-hour emulated runs are too
expensive to pay twice just to see what happened.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015nRFZXpKwUVE4SRSHw6GjF
This commit is contained in:
Michal
2026-08-11 15:25:57 +01:00
parent b75a4e0118
commit 346bd80c13
5 changed files with 326 additions and 49 deletions
Download Patch File Download Diff File Expand all files Collapse all files

124
bastion/tests/integration/arm-pxe-provision.test.ts
View File

@@ -7,8 +7,15 @@
//
// Two suites, because they cost very different amounts of time:
//
// "ARM PXE rescue" NBP handoff -> discovery -> rescue with SSH. ~20-30 min
// "ARM PXE install" full discover -> install -> installed. ~60-90 min
// "ARM PXE rescue" NBP handoff -> rescue with SSH. ~25-30 min
// "ARM PXE install" discover -> install -> installed. ~75-95 min
//
// The rescue suite seeds the machine into state as an already-known aarch64 box rather
// than discovering it first. That is the DGX Spark situation exactly -- SSH-onboarded,
// never PXE-discovered, architecture known only from its record -- and it holds the test
// to one emulated boot. Each boot spends ~15 of its ~18 minutes downloading Anaconda's
// stage2 under TCG, so discovering first would double the runtime without touching any
// code path the rescue boot does not already exercise.
//
// The install suite only runs with ARM_PXE_FULL=1. No ARM machine in the lab is ever
// PXE-installed except the MS-R1, and an hour-plus test that runs by default is a test
@@ -59,11 +66,14 @@ const DHCP_RANGE_START = `${PXE_SUBNET}.100`;
const DHCP_RANGE_END = `${PXE_SUBNET}.200`;
const SERIAL_PORT = 4555;
// Emulated aarch64 -- generous timeouts throughout.
const DISCOVERY_TIMEOUT_MS = 20 * 60_000;
const RESCUE_TIMEOUT_MS = 20 * 60_000;
// Emulated aarch64 -- generous timeouts throughout. Measured on an x86_64 host with no
// KVM for aarch64: a single PXE boot to a running Anaconda takes ~18 minutes, almost all
// of it downloading inst.stage2 over the network under TCG. Budget well above that;
// timing out just short of success wastes a whole run.
const LEASE_TIMEOUT_MS = 10 * 60_000;
const DISCOVERY_TIMEOUT_MS = 35 * 60_000;
const INSTALL_TIMEOUT_MS = 75 * 60_000;
const SSH_TIMEOUT_MS = 20 * 60_000;
const SSH_TIMEOUT_MS = 35 * 60_000;
const RUN_FULL_INSTALL = process.env["ARM_PXE_FULL"] === "1";
@@ -149,10 +159,19 @@ interface Harness {
testDir: string;
app: { close: () => Promise<void> };
stopDnsmasq: () => void;
state: { update: (fn: (s: BastionStateLike) => void) => void };
vmMac: string;
httpPort: number;
}
/** Just the parts of BastionState this test seeds. */
interface BastionStateLike {
discovered: Record<string, Record<string, unknown>>;
installed: Record<string, Record<string, unknown>>;
install_queue: Record<string, Record<string, unknown>>;
debug: Record<string, Record<string, unknown>>;
}
/** Bring up an isolated network, a bastion with both arch payloads, and an arm64 VM. */
async function startHarness(vmName: string, httpPort: number, pubKey: string): Promise<Harness> {
requirePrerequisites();
@@ -237,7 +256,7 @@ async function startHarness(vmName: string, httpPort: number, pubKey: string): P
);
generateDnsmasqConf(config);
const { app, syslog } = createApp(config);
const { app, state, syslog } = createApp(config);
await app.listen({ port: config.httpPort, host: "0.0.0.0" });
syslog.start();
log(`Bastion HTTP listening on :${config.httpPort}`);
@@ -262,10 +281,29 @@ async function startHarness(vmName: string, httpPort: number, pubKey: string): P
if (!vmMac) throw new Error("Could not determine VM MAC address");
log(`ARM VM MAC: ${vmMac}`);
return { testDir, app, stopDnsmasq, vmMac, httpPort: config.httpPort };
return {
testDir,
app,
stopDnsmasq,
state: state as unknown as Harness["state"],
vmMac,
httpPort: config.httpPort,
};
}
async function stopHarness(vmName: string, harness: Harness | undefined): Promise<void> {
// KEEP_VM=1 leaves the VM, network and bastion up so a failure can be inspected on
// the console. Emulated aarch64 runs cost half an hour; tearing the evidence down
// automatically means paying that again to see what happened.
if (process.env["KEEP_VM"] === "1") {
log(`KEEP_VM=1 — leaving ${vmName} running for inspection.`);
log(` console: sudo virsh screenshot ${vmName} /tmp/vm.ppm`);
log(` serial: socat - TCP:127.0.0.1:${SERIAL_PORT}`);
if (harness) log(` bastion: ${harness.testDir} (still serving on :${harness.httpPort})`);
log(` cleanup: sudo virsh destroy ${vmName}; sudo virsh undefine ${vmName} --remove-all-storage --nvram`);
return;
}
log("Cleaning up...");
if (harness) {
await harness.app.close().catch(() => {});
@@ -315,49 +353,49 @@ describe("ARM PXE rescue", () => {
const { pubKey, keyPath } = findSshKey();
sshKeyPath = keyPath;
harness = await startHarness(VM_NAME, HTTP_PORT, pubKey);
const { testDir, vmMac } = harness;
const { testDir, vmMac, state } = harness;
// 1. The VM PXE boots. This exercises the whole chain: DHCP option 93 -> arm64
// iPXE -> /boot.ipxe -> /dispatch?arch=arm64 -> aarch64 kernel + initrd.
// Reaching discovery at all proves the initrd was delivered over LoadFile2 --
// without it the kernel panics before it can run the kickstart.
log("Waiting for aarch64 VM to PXE boot and report discovery...");
log("(emulated aarch64 — be patient)");
type MachinesResponse = { discovered: Record<string, { arch: string }> };
const discovered = await pollApi<MachinesResponse>(
`http://${BASTION_IP}:${HTTP_PORT}/api/machines`,
(data) => vmMac in data.discovered,
DISCOVERY_TIMEOUT_MS,
).catch(async (err) => {
log("Discovery timed out. Serial console:");
try {
log(await readSerialLog(SERIAL_PORT, { lastLines: 100, timeoutMs: 15_000 }));
} catch { /* console unavailable */ }
throw err;
// Seed the machine as an already-known aarch64 box queued for rescue. This is the
// DGX Spark situation exactly: SSH-onboarded, never PXE-discovered, architecture
// known only from its record -- and it also keeps the test to a SINGLE emulated
// boot. Each boot spends ~15 minutes pulling Anaconda's stage2 over the network
// under TCG, so discovering first and rescuing second doubles the runtime for no
// extra coverage of the path being tested. Discovery is covered by the full suite.
log(`Seeding ${vmMac} as a known aarch64 machine queued for rescue...`);
state.update((s) => {
s.discovered[vmMac] = {
mac: vmMac,
product: "Test ARM64 Machine",
board: "virt",
serial: "SN-ARM64",
manufacturer: "QEMU",
cpu_model: "cortex-a57",
cpu_cores: VM_VCPUS,
memory_gb: 4,
arch: "aarch64",
disks: [],
nics: [],
first_seen: new Date().toISOString(),
last_seen: new Date().toISOString(),
};
s.debug[vmMac] = { hostname: "arm-rescue-test", queued_at: new Date().toISOString() };
});
log(`aarch64 VM discovered as ${discovered.discovered[vmMac]?.arch}`);
// 2. Queue rescue, exactly as `labctl provision debug` does.
log("Queueing debug/rescue mode...");
const res = await fetch(`http://${BASTION_IP}:${HTTP_PORT}/api/debug`, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ mac: vmMac }),
});
expect(res.ok).toBe(true);
// 3. Reboot into rescue. Rescue mode runs no %pre/%post, so the machine never
// reports its own IP — read it from the DHCP lease instead.
await sleep(20_000);
// Restart so the VM boots against the seeded state. createPxeVm already started it.
rebootPxeVm(VM_NAME);
await sleep(5_000);
deleteNftablesRejectRules();
log("Waiting for rescue environment DHCP lease...");
rescueIp = await waitForLease(testDir, vmMac, RESCUE_TIMEOUT_MS);
// The whole chain now runs once: DHCP option 93 -> arm64 iPXE -> /boot.ipxe ->
// /dispatch (architecture from the record, not the query) -> aarch64 kernel +
// initrd -> Anaconda rescue -> sshd. Reaching a shell at all proves iPXE handed
// the initrd to the EFI stub over LoadFile2; without it the kernel panics first.
log("Waiting for the rescue environment's DHCP lease...");
rescueIp = await waitForLease(testDir, vmMac, LEASE_TIMEOUT_MS);
log(`Rescue IP: ${rescueIp}`);
log("Waiting for SSH into the rescue shell (started by inst.sshd)...");
log("(emulated aarch64 — Anaconda's stage2 download dominates; be patient)");
await waitForSsh(rescueIp, "root", SSH_TIMEOUT_MS, sshKeyPath).catch(async (err) => {
log("Rescue SSH timed out. Serial console:");
try {
@@ -366,11 +404,11 @@ describe("ARM PXE rescue", () => {
throw err;
});
log("ARM PXE rescue reached.");
}, DISCOVERY_TIMEOUT_MS + RESCUE_TIMEOUT_MS + SSH_TIMEOUT_MS + 300_000);
}, LEASE_TIMEOUT_MS + SSH_TIMEOUT_MS + 300_000);
afterAll(async () => { await stopHarness(VM_NAME, harness); });
it("discovered the machine as aarch64", async () => {
it("resolved the architecture from the machine record", async () => {
const res = await fetch(`http://${BASTION_IP}:${HTTP_PORT}/api/machines`);
const data = (await res.json()) as { discovered: Record<string, { arch: string }> };
expect(data.discovered[harness!.vmMac]?.arch).toBe("aarch64");