feat: install logging, error trapping, PXE/ISO integration tests
Kickstart installs on real hardware failed silently — no error reporting,
only 3 progress callbacks, zero log streaming. This overhaul makes every
install fully observable.
Kickstart improvements:
- Error trapping in %pre and %post (trap ERR sends failure details to bastion)
- 12+ granular progress stages (was 3): SSH, hostname, k3s prep, EFI boot, metadata
- Background log streamer: tails %post output and batch-sends to /api/log
- bastion_log() function for explicit log lines from kickstart scripts
Bastion API:
- POST /api/log — receives raw log lines from kickstart (single or batch)
- InstallLogBuffer — per-MAC ring buffer (2000 lines) + file persistence
- GET /api/logs/:mac — now returns log_lines + log_total alongside stages
- SSE /api/logs/:mac/follow — uses named events (event: stage vs event: log)
- Progress events forwarded to labd via bastion-progress WebSocket message
- Post-provision k3s logs routed through progressBus (was console-only)
dnsmasq fixes found during VM testing:
- HTTP Boot filename: ipxe-real.efi → ipxe.efi (leftover from old 2-stage approach)
- pxe-service directives: only in proxy mode (breaks OVMF PXE in full mode)
- PXEClient vendor class echo for UEFI firmware compatibility
Integration tests:
- PXE boot test: blank UEFI VM → dnsmasq → HTTP Boot → iPXE → bastion → install
- ISO boot test: blank VM boots from bastion-generated ISO → same flow
- Shared helpers: pxe-network (no DHCP, nftables fix), pxe-vm (UEFI + ISO boot)
- test-provision.sh: runs both PXE + ISO tests with prerequisite checks
- 250GB sparse QCOW2 disk (LVM layout needs ~204GB)
201 unit tests passing (11 new).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-26 22:26:33 +00:00
|
|
|
// Create a blank UEFI VM for PXE boot testing.
|
|
|
|
|
// Unlike cloud image VMs, these have an empty disk and boot from network.
|
2026-03-27 15:22:43 +00:00
|
|
|
// Each VM gets a serial console on a TCP port for debugging without network/SSH.
|
feat: install logging, error trapping, PXE/ISO integration tests
Kickstart installs on real hardware failed silently — no error reporting,
only 3 progress callbacks, zero log streaming. This overhaul makes every
install fully observable.
Kickstart improvements:
- Error trapping in %pre and %post (trap ERR sends failure details to bastion)
- 12+ granular progress stages (was 3): SSH, hostname, k3s prep, EFI boot, metadata
- Background log streamer: tails %post output and batch-sends to /api/log
- bastion_log() function for explicit log lines from kickstart scripts
Bastion API:
- POST /api/log — receives raw log lines from kickstart (single or batch)
- InstallLogBuffer — per-MAC ring buffer (2000 lines) + file persistence
- GET /api/logs/:mac — now returns log_lines + log_total alongside stages
- SSE /api/logs/:mac/follow — uses named events (event: stage vs event: log)
- Progress events forwarded to labd via bastion-progress WebSocket message
- Post-provision k3s logs routed through progressBus (was console-only)
dnsmasq fixes found during VM testing:
- HTTP Boot filename: ipxe-real.efi → ipxe.efi (leftover from old 2-stage approach)
- pxe-service directives: only in proxy mode (breaks OVMF PXE in full mode)
- PXEClient vendor class echo for UEFI firmware compatibility
Integration tests:
- PXE boot test: blank UEFI VM → dnsmasq → HTTP Boot → iPXE → bastion → install
- ISO boot test: blank VM boots from bastion-generated ISO → same flow
- Shared helpers: pxe-network (no DHCP, nftables fix), pxe-vm (UEFI + ISO boot)
- test-provision.sh: runs both PXE + ISO tests with prerequisite checks
- 250GB sparse QCOW2 disk (LVM layout needs ~204GB)
201 unit tests passing (11 new).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-26 22:26:33 +00:00
|
|
|
|
|
|
|
|
import { execSync, spawnSync, type SpawnSyncReturns } from "node:child_process";
|
|
|
|
|
import { existsSync } from "node:fs";
|
|
|
|
|
import { join } from "node:path";
|
2026-03-27 15:22:43 +00:00
|
|
|
import { createConnection } from "node:net";
|
feat: install logging, error trapping, PXE/ISO integration tests
Kickstart installs on real hardware failed silently — no error reporting,
only 3 progress callbacks, zero log streaming. This overhaul makes every
install fully observable.
Kickstart improvements:
- Error trapping in %pre and %post (trap ERR sends failure details to bastion)
- 12+ granular progress stages (was 3): SSH, hostname, k3s prep, EFI boot, metadata
- Background log streamer: tails %post output and batch-sends to /api/log
- bastion_log() function for explicit log lines from kickstart scripts
Bastion API:
- POST /api/log — receives raw log lines from kickstart (single or batch)
- InstallLogBuffer — per-MAC ring buffer (2000 lines) + file persistence
- GET /api/logs/:mac — now returns log_lines + log_total alongside stages
- SSE /api/logs/:mac/follow — uses named events (event: stage vs event: log)
- Progress events forwarded to labd via bastion-progress WebSocket message
- Post-provision k3s logs routed through progressBus (was console-only)
dnsmasq fixes found during VM testing:
- HTTP Boot filename: ipxe-real.efi → ipxe.efi (leftover from old 2-stage approach)
- pxe-service directives: only in proxy mode (breaks OVMF PXE in full mode)
- PXEClient vendor class echo for UEFI firmware compatibility
Integration tests:
- PXE boot test: blank UEFI VM → dnsmasq → HTTP Boot → iPXE → bastion → install
- ISO boot test: blank VM boots from bastion-generated ISO → same flow
- Shared helpers: pxe-network (no DHCP, nftables fix), pxe-vm (UEFI + ISO boot)
- test-provision.sh: runs both PXE + ISO tests with prerequisite checks
- 250GB sparse QCOW2 disk (LVM layout needs ~204GB)
201 unit tests passing (11 new).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-26 22:26:33 +00:00
|
|
|
import { log } from "./libvirt.js";
|
|
|
|
|
|
|
|
|
|
const IMAGE_DIR = "/var/lib/libvirt/images";
|
|
|
|
|
const IS_ROOT = process.getuid?.() === 0;
|
|
|
|
|
|
|
|
|
|
function run(cmd: string, opts?: { timeout?: number }): string {
|
|
|
|
|
const full = IS_ROOT ? cmd : `sudo ${cmd}`;
|
|
|
|
|
return execSync(full, { encoding: "utf-8", stdio: "pipe", timeout: opts?.timeout ?? 60_000 });
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
function virsh(...args: string[]): SpawnSyncReturns<string> {
|
|
|
|
|
const cmd = IS_ROOT ? "virsh" : "sudo";
|
|
|
|
|
const finalArgs = IS_ROOT ? args : ["virsh", ...args];
|
|
|
|
|
return spawnSync(cmd, finalArgs, { encoding: "utf-8", stdio: "pipe", timeout: 30_000 });
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
export interface PxeVmConfig {
|
|
|
|
|
name: string;
|
|
|
|
|
memory: number; // MB
|
|
|
|
|
vcpus: number;
|
|
|
|
|
diskSize: number; // GB
|
|
|
|
|
network: string; // libvirt network name
|
|
|
|
|
arch?: "x86_64" | "aarch64";
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/** Create a blank UEFI VM that PXE boots from the network. */
|
|
|
|
|
export function createPxeVm(config: PxeVmConfig): void {
|
|
|
|
|
destroyPxeVm(config.name);
|
|
|
|
|
|
|
|
|
|
const arch = config.arch ?? "x86_64";
|
|
|
|
|
log(`Creating PXE VM: ${config.name} (${arch}, ${config.memory}MB RAM, ${config.vcpus} vCPU, ${config.diskSize}GB disk)`);
|
|
|
|
|
|
|
|
|
|
// Create blank disk
|
|
|
|
|
const diskPath = join(IMAGE_DIR, `${config.name}.qcow2`);
|
|
|
|
|
run(`qemu-img create -f qcow2 "${diskPath}" ${config.diskSize}G`);
|
|
|
|
|
|
2026-03-27 00:26:12 +00:00
|
|
|
// UEFI firmware paths (Fedora)
|
|
|
|
|
if (arch === "aarch64") {
|
|
|
|
|
const aavmf = "/usr/share/edk2/aarch64/QEMU_EFI.fd";
|
|
|
|
|
if (!existsSync(aavmf)) {
|
|
|
|
|
throw new Error(`AAVMF firmware not found at ${aavmf}. Install: sudo dnf install edk2-aarch64`);
|
|
|
|
|
}
|
|
|
|
|
} else {
|
|
|
|
|
const ovmf = "/usr/share/edk2/ovmf/OVMF_CODE.fd";
|
|
|
|
|
if (!existsSync(ovmf)) {
|
|
|
|
|
throw new Error(`OVMF firmware not found at ${ovmf}. Install: sudo dnf install edk2-ovmf`);
|
|
|
|
|
}
|
feat: install logging, error trapping, PXE/ISO integration tests
Kickstart installs on real hardware failed silently — no error reporting,
only 3 progress callbacks, zero log streaming. This overhaul makes every
install fully observable.
Kickstart improvements:
- Error trapping in %pre and %post (trap ERR sends failure details to bastion)
- 12+ granular progress stages (was 3): SSH, hostname, k3s prep, EFI boot, metadata
- Background log streamer: tails %post output and batch-sends to /api/log
- bastion_log() function for explicit log lines from kickstart scripts
Bastion API:
- POST /api/log — receives raw log lines from kickstart (single or batch)
- InstallLogBuffer — per-MAC ring buffer (2000 lines) + file persistence
- GET /api/logs/:mac — now returns log_lines + log_total alongside stages
- SSE /api/logs/:mac/follow — uses named events (event: stage vs event: log)
- Progress events forwarded to labd via bastion-progress WebSocket message
- Post-provision k3s logs routed through progressBus (was console-only)
dnsmasq fixes found during VM testing:
- HTTP Boot filename: ipxe-real.efi → ipxe.efi (leftover from old 2-stage approach)
- pxe-service directives: only in proxy mode (breaks OVMF PXE in full mode)
- PXEClient vendor class echo for UEFI firmware compatibility
Integration tests:
- PXE boot test: blank UEFI VM → dnsmasq → HTTP Boot → iPXE → bastion → install
- ISO boot test: blank VM boots from bastion-generated ISO → same flow
- Shared helpers: pxe-network (no DHCP, nftables fix), pxe-vm (UEFI + ISO boot)
- test-provision.sh: runs both PXE + ISO tests with prerequisite checks
- 250GB sparse QCOW2 disk (LVM layout needs ~204GB)
201 unit tests passing (11 new).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-26 22:26:33 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
const virtInstallArgs = [
|
|
|
|
|
"virt-install",
|
|
|
|
|
`--name=${config.name}`,
|
|
|
|
|
`--memory=${config.memory}`,
|
|
|
|
|
`--vcpus=${config.vcpus}`,
|
|
|
|
|
`--disk=path=${diskPath},format=qcow2,bus=virtio`,
|
|
|
|
|
`--network=network=${config.network},model=virtio`,
|
|
|
|
|
// UEFI firmware — required for PXE boot in modern mode
|
2026-03-28 20:24:14 +00:00
|
|
|
`--boot=uefi,network,hd`,
|
feat: install logging, error trapping, PXE/ISO integration tests
Kickstart installs on real hardware failed silently — no error reporting,
only 3 progress callbacks, zero log streaming. This overhaul makes every
install fully observable.
Kickstart improvements:
- Error trapping in %pre and %post (trap ERR sends failure details to bastion)
- 12+ granular progress stages (was 3): SSH, hostname, k3s prep, EFI boot, metadata
- Background log streamer: tails %post output and batch-sends to /api/log
- bastion_log() function for explicit log lines from kickstart scripts
Bastion API:
- POST /api/log — receives raw log lines from kickstart (single or batch)
- InstallLogBuffer — per-MAC ring buffer (2000 lines) + file persistence
- GET /api/logs/:mac — now returns log_lines + log_total alongside stages
- SSE /api/logs/:mac/follow — uses named events (event: stage vs event: log)
- Progress events forwarded to labd via bastion-progress WebSocket message
- Post-provision k3s logs routed through progressBus (was console-only)
dnsmasq fixes found during VM testing:
- HTTP Boot filename: ipxe-real.efi → ipxe.efi (leftover from old 2-stage approach)
- pxe-service directives: only in proxy mode (breaks OVMF PXE in full mode)
- PXEClient vendor class echo for UEFI firmware compatibility
Integration tests:
- PXE boot test: blank UEFI VM → dnsmasq → HTTP Boot → iPXE → bastion → install
- ISO boot test: blank VM boots from bastion-generated ISO → same flow
- Shared helpers: pxe-network (no DHCP, nftables fix), pxe-vm (UEFI + ISO boot)
- test-provision.sh: runs both PXE + ISO tests with prerequisite checks
- 250GB sparse QCOW2 disk (LVM layout needs ~204GB)
201 unit tests passing (11 new).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-26 22:26:33 +00:00
|
|
|
// No OS to install — PXE provides everything
|
|
|
|
|
"--os-variant=generic",
|
|
|
|
|
"--noautoconsole",
|
|
|
|
|
"--wait=0",
|
|
|
|
|
// Graphics for debugging (VNC, connect with virt-viewer if needed)
|
|
|
|
|
"--graphics=vnc,listen=127.0.0.1",
|
2026-03-27 15:22:43 +00:00
|
|
|
// Serial console via TCP — allows exec without network/SSH
|
|
|
|
|
// Connect: socat - TCP:127.0.0.1:4555
|
|
|
|
|
"--serial=tcp,host=127.0.0.1:4555,mode=bind,protocol=telnet",
|
feat: install logging, error trapping, PXE/ISO integration tests
Kickstart installs on real hardware failed silently — no error reporting,
only 3 progress callbacks, zero log streaming. This overhaul makes every
install fully observable.
Kickstart improvements:
- Error trapping in %pre and %post (trap ERR sends failure details to bastion)
- 12+ granular progress stages (was 3): SSH, hostname, k3s prep, EFI boot, metadata
- Background log streamer: tails %post output and batch-sends to /api/log
- bastion_log() function for explicit log lines from kickstart scripts
Bastion API:
- POST /api/log — receives raw log lines from kickstart (single or batch)
- InstallLogBuffer — per-MAC ring buffer (2000 lines) + file persistence
- GET /api/logs/:mac — now returns log_lines + log_total alongside stages
- SSE /api/logs/:mac/follow — uses named events (event: stage vs event: log)
- Progress events forwarded to labd via bastion-progress WebSocket message
- Post-provision k3s logs routed through progressBus (was console-only)
dnsmasq fixes found during VM testing:
- HTTP Boot filename: ipxe-real.efi → ipxe.efi (leftover from old 2-stage approach)
- pxe-service directives: only in proxy mode (breaks OVMF PXE in full mode)
- PXEClient vendor class echo for UEFI firmware compatibility
Integration tests:
- PXE boot test: blank UEFI VM → dnsmasq → HTTP Boot → iPXE → bastion → install
- ISO boot test: blank VM boots from bastion-generated ISO → same flow
- Shared helpers: pxe-network (no DHCP, nftables fix), pxe-vm (UEFI + ISO boot)
- test-provision.sh: runs both PXE + ISO tests with prerequisite checks
- 250GB sparse QCOW2 disk (LVM layout needs ~204GB)
201 unit tests passing (11 new).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-26 22:26:33 +00:00
|
|
|
];
|
|
|
|
|
|
|
|
|
|
if (arch === "aarch64") {
|
|
|
|
|
virtInstallArgs.push("--arch=aarch64", "--machine=virt");
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
log(`Running: virt-install --name=${config.name} --boot=uefi,network ...`);
|
|
|
|
|
run(virtInstallArgs.join(" "), { timeout: 30_000 });
|
2026-03-27 15:22:43 +00:00
|
|
|
log(`PXE VM ${config.name} created (serial: telnet 127.0.0.1 4555)`);
|
feat: install logging, error trapping, PXE/ISO integration tests
Kickstart installs on real hardware failed silently — no error reporting,
only 3 progress callbacks, zero log streaming. This overhaul makes every
install fully observable.
Kickstart improvements:
- Error trapping in %pre and %post (trap ERR sends failure details to bastion)
- 12+ granular progress stages (was 3): SSH, hostname, k3s prep, EFI boot, metadata
- Background log streamer: tails %post output and batch-sends to /api/log
- bastion_log() function for explicit log lines from kickstart scripts
Bastion API:
- POST /api/log — receives raw log lines from kickstart (single or batch)
- InstallLogBuffer — per-MAC ring buffer (2000 lines) + file persistence
- GET /api/logs/:mac — now returns log_lines + log_total alongside stages
- SSE /api/logs/:mac/follow — uses named events (event: stage vs event: log)
- Progress events forwarded to labd via bastion-progress WebSocket message
- Post-provision k3s logs routed through progressBus (was console-only)
dnsmasq fixes found during VM testing:
- HTTP Boot filename: ipxe-real.efi → ipxe.efi (leftover from old 2-stage approach)
- pxe-service directives: only in proxy mode (breaks OVMF PXE in full mode)
- PXEClient vendor class echo for UEFI firmware compatibility
Integration tests:
- PXE boot test: blank UEFI VM → dnsmasq → HTTP Boot → iPXE → bastion → install
- ISO boot test: blank VM boots from bastion-generated ISO → same flow
- Shared helpers: pxe-network (no DHCP, nftables fix), pxe-vm (UEFI + ISO boot)
- test-provision.sh: runs both PXE + ISO tests with prerequisite checks
- 250GB sparse QCOW2 disk (LVM layout needs ~204GB)
201 unit tests passing (11 new).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-26 22:26:33 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/** Destroy a PXE VM and clean up its disk. */
|
|
|
|
|
export function destroyPxeVm(name: string): void {
|
|
|
|
|
const result = virsh("dominfo", name);
|
|
|
|
|
if (result.status !== 0) return;
|
|
|
|
|
|
|
|
|
|
log(`Destroying PXE VM: ${name}`);
|
|
|
|
|
virsh("destroy", name);
|
|
|
|
|
virsh("undefine", name, "--remove-all-storage", "--nvram");
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/** Get the MAC address of a VM's first NIC. */
|
|
|
|
|
export function getVmMac(name: string): string | null {
|
|
|
|
|
const result = virsh("domiflist", name);
|
|
|
|
|
if (result.status !== 0) return null;
|
|
|
|
|
// Output format: Interface Type Source Model MAC
|
|
|
|
|
const match = result.stdout.match(/([0-9a-f]{2}:[0-9a-f]{2}:[0-9a-f]{2}:[0-9a-f]{2}:[0-9a-f]{2}:[0-9a-f]{2})/i);
|
|
|
|
|
return match ? match[1].toLowerCase() : null;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/** Reboot a VM (force off + start). */
|
|
|
|
|
export function rebootPxeVm(name: string): void {
|
|
|
|
|
log(`Rebooting PXE VM: ${name}`);
|
|
|
|
|
virsh("destroy", name);
|
|
|
|
|
// Brief pause to let resources release
|
|
|
|
|
spawnSync("sleep", ["2"]);
|
|
|
|
|
virsh("start", name);
|
|
|
|
|
log(`PXE VM ${name} restarted`);
|
|
|
|
|
}
|
|
|
|
|
|
2026-03-28 20:24:14 +00:00
|
|
|
/**
|
|
|
|
|
* Read raw output from the VM's serial console (telnet TCP port).
|
|
|
|
|
* Returns the last N lines. Useful for diagnostics when SSH isn't available.
|
|
|
|
|
*/
|
|
|
|
|
export async function readSerialLog(
|
|
|
|
|
port: number,
|
|
|
|
|
opts: { lastLines?: number; timeoutMs?: number } = {},
|
|
|
|
|
): Promise<string> {
|
|
|
|
|
const { lastLines = 50, timeoutMs = 10_000 } = opts;
|
|
|
|
|
return new Promise((resolve) => {
|
|
|
|
|
const sock = createConnection({ host: "127.0.0.1", port });
|
|
|
|
|
let buf = "";
|
|
|
|
|
const timer = setTimeout(() => { sock.destroy(); resolve(buf); }, timeoutMs);
|
|
|
|
|
sock.on("data", (d: Buffer) => { buf += d.toString(); });
|
|
|
|
|
sock.on("error", () => { clearTimeout(timer); resolve(`(connection error) ${buf}`); });
|
|
|
|
|
sock.on("close", () => { clearTimeout(timer); resolve(buf); });
|
|
|
|
|
// Send a newline to trigger any buffered output / prompt
|
|
|
|
|
setTimeout(() => sock.write("\r\n"), 500);
|
|
|
|
|
}).then((raw: unknown) => {
|
|
|
|
|
const lines = (raw as string).split("\n").map(l => l.trimEnd()).filter(Boolean);
|
|
|
|
|
return lines.slice(-lastLines).join("\n");
|
|
|
|
|
});
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/**
|
|
|
|
|
* Execute a command on the VM's serial console via socat.
|
|
|
|
|
* Requires auto-login root shell on the serial port.
|
|
|
|
|
*/
|
|
|
|
|
export function serialExec(
|
|
|
|
|
port: number,
|
|
|
|
|
command: string,
|
|
|
|
|
timeoutMs = 15_000,
|
|
|
|
|
): string {
|
|
|
|
|
const marker = `__END_${Date.now()}__`;
|
|
|
|
|
// Use socat to handle telnet negotiation properly
|
|
|
|
|
const input = `\r\n${command}; echo '${marker}'\r\n`;
|
|
|
|
|
const result = spawnSync("bash", ["-c",
|
|
|
|
|
`echo -e '${input.replace(/'/g, "\\'")}' | socat -T${Math.ceil(timeoutMs / 1000)} - TCP:127.0.0.1:${port} 2>/dev/null`
|
|
|
|
|
], { encoding: "utf-8", stdio: "pipe", timeout: timeoutMs + 5000 });
|
|
|
|
|
const output = result.stdout ?? "";
|
|
|
|
|
const markerIdx = output.indexOf(marker);
|
|
|
|
|
if (markerIdx < 0) return `(no marker) ${output.slice(-500)}`;
|
|
|
|
|
// Get lines between command echo and marker
|
|
|
|
|
const before = output.substring(0, markerIdx);
|
|
|
|
|
const lines = before.split("\n");
|
|
|
|
|
// Skip everything up to and including the command echo line
|
|
|
|
|
const cmdIdx = lines.findIndex(l => l.includes(command.substring(0, 20)));
|
|
|
|
|
return lines.slice(cmdIdx >= 0 ? cmdIdx + 1 : 1).join("\n").trim();
|
2026-03-27 00:26:12 +00:00
|
|
|
}
|
|
|
|
|
|
feat: install logging, error trapping, PXE/ISO integration tests
Kickstart installs on real hardware failed silently — no error reporting,
only 3 progress callbacks, zero log streaming. This overhaul makes every
install fully observable.
Kickstart improvements:
- Error trapping in %pre and %post (trap ERR sends failure details to bastion)
- 12+ granular progress stages (was 3): SSH, hostname, k3s prep, EFI boot, metadata
- Background log streamer: tails %post output and batch-sends to /api/log
- bastion_log() function for explicit log lines from kickstart scripts
Bastion API:
- POST /api/log — receives raw log lines from kickstart (single or batch)
- InstallLogBuffer — per-MAC ring buffer (2000 lines) + file persistence
- GET /api/logs/:mac — now returns log_lines + log_total alongside stages
- SSE /api/logs/:mac/follow — uses named events (event: stage vs event: log)
- Progress events forwarded to labd via bastion-progress WebSocket message
- Post-provision k3s logs routed through progressBus (was console-only)
dnsmasq fixes found during VM testing:
- HTTP Boot filename: ipxe-real.efi → ipxe.efi (leftover from old 2-stage approach)
- pxe-service directives: only in proxy mode (breaks OVMF PXE in full mode)
- PXEClient vendor class echo for UEFI firmware compatibility
Integration tests:
- PXE boot test: blank UEFI VM → dnsmasq → HTTP Boot → iPXE → bastion → install
- ISO boot test: blank VM boots from bastion-generated ISO → same flow
- Shared helpers: pxe-network (no DHCP, nftables fix), pxe-vm (UEFI + ISO boot)
- test-provision.sh: runs both PXE + ISO tests with prerequisite checks
- 250GB sparse QCOW2 disk (LVM layout needs ~204GB)
201 unit tests passing (11 new).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-26 22:26:33 +00:00
|
|
|
export interface IsoVmConfig {
|
|
|
|
|
name: string;
|
|
|
|
|
memory: number; // MB
|
|
|
|
|
vcpus: number;
|
|
|
|
|
diskSize: number; // GB
|
|
|
|
|
network: string; // libvirt network name
|
|
|
|
|
isoPath: string; // path to boot ISO
|
2026-03-27 00:26:12 +00:00
|
|
|
arch?: "x86_64" | "aarch64";
|
feat: install logging, error trapping, PXE/ISO integration tests
Kickstart installs on real hardware failed silently — no error reporting,
only 3 progress callbacks, zero log streaming. This overhaul makes every
install fully observable.
Kickstart improvements:
- Error trapping in %pre and %post (trap ERR sends failure details to bastion)
- 12+ granular progress stages (was 3): SSH, hostname, k3s prep, EFI boot, metadata
- Background log streamer: tails %post output and batch-sends to /api/log
- bastion_log() function for explicit log lines from kickstart scripts
Bastion API:
- POST /api/log — receives raw log lines from kickstart (single or batch)
- InstallLogBuffer — per-MAC ring buffer (2000 lines) + file persistence
- GET /api/logs/:mac — now returns log_lines + log_total alongside stages
- SSE /api/logs/:mac/follow — uses named events (event: stage vs event: log)
- Progress events forwarded to labd via bastion-progress WebSocket message
- Post-provision k3s logs routed through progressBus (was console-only)
dnsmasq fixes found during VM testing:
- HTTP Boot filename: ipxe-real.efi → ipxe.efi (leftover from old 2-stage approach)
- pxe-service directives: only in proxy mode (breaks OVMF PXE in full mode)
- PXEClient vendor class echo for UEFI firmware compatibility
Integration tests:
- PXE boot test: blank UEFI VM → dnsmasq → HTTP Boot → iPXE → bastion → install
- ISO boot test: blank VM boots from bastion-generated ISO → same flow
- Shared helpers: pxe-network (no DHCP, nftables fix), pxe-vm (UEFI + ISO boot)
- test-provision.sh: runs both PXE + ISO tests with prerequisite checks
- 250GB sparse QCOW2 disk (LVM layout needs ~204GB)
201 unit tests passing (11 new).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-26 22:26:33 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/** Create a UEFI VM that boots from a CD-ROM ISO (not PXE). */
|
|
|
|
|
export function createIsoVm(config: IsoVmConfig): void {
|
|
|
|
|
destroyPxeVm(config.name);
|
|
|
|
|
|
2026-03-27 00:26:12 +00:00
|
|
|
const arch = config.arch ?? "x86_64";
|
|
|
|
|
log(`Creating ISO boot VM: ${config.name} (${arch}, ${config.memory}MB RAM, ${config.vcpus} vCPU, ${config.diskSize}GB disk)`);
|
feat: install logging, error trapping, PXE/ISO integration tests
Kickstart installs on real hardware failed silently — no error reporting,
only 3 progress callbacks, zero log streaming. This overhaul makes every
install fully observable.
Kickstart improvements:
- Error trapping in %pre and %post (trap ERR sends failure details to bastion)
- 12+ granular progress stages (was 3): SSH, hostname, k3s prep, EFI boot, metadata
- Background log streamer: tails %post output and batch-sends to /api/log
- bastion_log() function for explicit log lines from kickstart scripts
Bastion API:
- POST /api/log — receives raw log lines from kickstart (single or batch)
- InstallLogBuffer — per-MAC ring buffer (2000 lines) + file persistence
- GET /api/logs/:mac — now returns log_lines + log_total alongside stages
- SSE /api/logs/:mac/follow — uses named events (event: stage vs event: log)
- Progress events forwarded to labd via bastion-progress WebSocket message
- Post-provision k3s logs routed through progressBus (was console-only)
dnsmasq fixes found during VM testing:
- HTTP Boot filename: ipxe-real.efi → ipxe.efi (leftover from old 2-stage approach)
- pxe-service directives: only in proxy mode (breaks OVMF PXE in full mode)
- PXEClient vendor class echo for UEFI firmware compatibility
Integration tests:
- PXE boot test: blank UEFI VM → dnsmasq → HTTP Boot → iPXE → bastion → install
- ISO boot test: blank VM boots from bastion-generated ISO → same flow
- Shared helpers: pxe-network (no DHCP, nftables fix), pxe-vm (UEFI + ISO boot)
- test-provision.sh: runs both PXE + ISO tests with prerequisite checks
- 250GB sparse QCOW2 disk (LVM layout needs ~204GB)
201 unit tests passing (11 new).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-26 22:26:33 +00:00
|
|
|
|
|
|
|
|
// Create blank disk
|
|
|
|
|
const diskPath = join(IMAGE_DIR, `${config.name}.qcow2`);
|
|
|
|
|
run(`qemu-img create -f qcow2 "${diskPath}" ${config.diskSize}G`);
|
|
|
|
|
|
|
|
|
|
const virtInstallArgs = [
|
|
|
|
|
"virt-install",
|
|
|
|
|
`--name=${config.name}`,
|
|
|
|
|
`--memory=${config.memory}`,
|
|
|
|
|
`--vcpus=${config.vcpus}`,
|
|
|
|
|
`--disk=path=${diskPath},format=qcow2,bus=virtio`,
|
|
|
|
|
// Boot ISO as CD-ROM
|
|
|
|
|
`--disk=path=${config.isoPath},device=cdrom,readonly=on`,
|
|
|
|
|
`--network=network=${config.network},model=virtio`,
|
|
|
|
|
// UEFI firmware, boot from cdrom (not network)
|
|
|
|
|
"--boot=uefi,cdrom",
|
|
|
|
|
"--os-variant=generic",
|
|
|
|
|
"--noautoconsole",
|
|
|
|
|
"--wait=0",
|
|
|
|
|
"--graphics=vnc,listen=127.0.0.1",
|
2026-03-27 15:22:43 +00:00
|
|
|
// Serial console via TCP (port 4556 to avoid conflict with PXE VM)
|
|
|
|
|
"--serial=tcp,host=127.0.0.1:4556,mode=bind,protocol=telnet",
|
feat: install logging, error trapping, PXE/ISO integration tests
Kickstart installs on real hardware failed silently — no error reporting,
only 3 progress callbacks, zero log streaming. This overhaul makes every
install fully observable.
Kickstart improvements:
- Error trapping in %pre and %post (trap ERR sends failure details to bastion)
- 12+ granular progress stages (was 3): SSH, hostname, k3s prep, EFI boot, metadata
- Background log streamer: tails %post output and batch-sends to /api/log
- bastion_log() function for explicit log lines from kickstart scripts
Bastion API:
- POST /api/log — receives raw log lines from kickstart (single or batch)
- InstallLogBuffer — per-MAC ring buffer (2000 lines) + file persistence
- GET /api/logs/:mac — now returns log_lines + log_total alongside stages
- SSE /api/logs/:mac/follow — uses named events (event: stage vs event: log)
- Progress events forwarded to labd via bastion-progress WebSocket message
- Post-provision k3s logs routed through progressBus (was console-only)
dnsmasq fixes found during VM testing:
- HTTP Boot filename: ipxe-real.efi → ipxe.efi (leftover from old 2-stage approach)
- pxe-service directives: only in proxy mode (breaks OVMF PXE in full mode)
- PXEClient vendor class echo for UEFI firmware compatibility
Integration tests:
- PXE boot test: blank UEFI VM → dnsmasq → HTTP Boot → iPXE → bastion → install
- ISO boot test: blank VM boots from bastion-generated ISO → same flow
- Shared helpers: pxe-network (no DHCP, nftables fix), pxe-vm (UEFI + ISO boot)
- test-provision.sh: runs both PXE + ISO tests with prerequisite checks
- 250GB sparse QCOW2 disk (LVM layout needs ~204GB)
201 unit tests passing (11 new).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-26 22:26:33 +00:00
|
|
|
];
|
|
|
|
|
|
2026-03-27 00:26:12 +00:00
|
|
|
if (arch === "aarch64") {
|
|
|
|
|
virtInstallArgs.push("--arch=aarch64", "--machine=virt");
|
|
|
|
|
}
|
|
|
|
|
|
feat: install logging, error trapping, PXE/ISO integration tests
Kickstart installs on real hardware failed silently — no error reporting,
only 3 progress callbacks, zero log streaming. This overhaul makes every
install fully observable.
Kickstart improvements:
- Error trapping in %pre and %post (trap ERR sends failure details to bastion)
- 12+ granular progress stages (was 3): SSH, hostname, k3s prep, EFI boot, metadata
- Background log streamer: tails %post output and batch-sends to /api/log
- bastion_log() function for explicit log lines from kickstart scripts
Bastion API:
- POST /api/log — receives raw log lines from kickstart (single or batch)
- InstallLogBuffer — per-MAC ring buffer (2000 lines) + file persistence
- GET /api/logs/:mac — now returns log_lines + log_total alongside stages
- SSE /api/logs/:mac/follow — uses named events (event: stage vs event: log)
- Progress events forwarded to labd via bastion-progress WebSocket message
- Post-provision k3s logs routed through progressBus (was console-only)
dnsmasq fixes found during VM testing:
- HTTP Boot filename: ipxe-real.efi → ipxe.efi (leftover from old 2-stage approach)
- pxe-service directives: only in proxy mode (breaks OVMF PXE in full mode)
- PXEClient vendor class echo for UEFI firmware compatibility
Integration tests:
- PXE boot test: blank UEFI VM → dnsmasq → HTTP Boot → iPXE → bastion → install
- ISO boot test: blank VM boots from bastion-generated ISO → same flow
- Shared helpers: pxe-network (no DHCP, nftables fix), pxe-vm (UEFI + ISO boot)
- test-provision.sh: runs both PXE + ISO tests with prerequisite checks
- 250GB sparse QCOW2 disk (LVM layout needs ~204GB)
201 unit tests passing (11 new).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-26 22:26:33 +00:00
|
|
|
log(`Running: virt-install --name=${config.name} --boot=uefi,cdrom ...`);
|
2026-03-27 00:26:12 +00:00
|
|
|
run(virtInstallArgs.join(" "), { timeout: 60_000 });
|
2026-03-27 15:22:43 +00:00
|
|
|
log(`ISO boot VM ${config.name} created (serial: telnet 127.0.0.1 4556)`);
|
|
|
|
|
}
|
|
|
|
|
|