Both Protectli VP2440s failed to install on real hardware: they fetched
kernel+initrd and then went silent. The console showed why —
Looking for a connected Ethernet interface ... e2 ? e3 ? e4 ? e5 ?
Connected e4 found
Connected e5 found
[4.595647] igc 0000:02:00.0 e2: NIC Link is Up
IP-Config: e4 ... no response after 15 secs - giving up
Unable to find a live file system on the network
live-boot picks the first *connected* interface. The i40e SFP+ pair links
before the igc copper port (up at 4.6s), so it chose the fiber ports, which
have no DHCP, and never tried the NIC that actually PXE booted.
Fix: pass BOOTIF=01-<mac> on the kernel cmdline. live-boot's
Device_from_bootif() (verified present in this image) matches it against
/sys/class/net and sets DEVICE directly. The MAC comes from the dispatch
key — i.e. exactly the NIC that PXE booted — which is more reliable than
iPXE's ${net0} on a box where the booting NIC may not be net0.
Why the integration test missed it: the VM had ONE NIC, so "first connected
interface" was trivially correct, and virtio links instantly so there was no
negotiation race. createPxeVm now takes decoyNics, attaching extra NICs
ahead of the PXE NIC on a network with no route to the bastion; the VyOS
test uses 2. Without BOOTIF that reproduces the hardware failure. getVmMac
is network-aware so it still returns the booting NIC.
Also: the bastion had every clue and said nothing — it logged INSTALL
STARTED, served kernel+initrd, then nothing for 7 minutes. dispatch now
stamps dispatched_at, and /api/logs/:mac returns stalled_for_s / stalled
(8 min threshold, sized for the ~600MB squashfs fetch), so a machine wedged
before the installer environment comes up is diagnosable without a console.
Verified on hardware: both firewalls installed, bond0 802.3ad + VLANs
2/3/9/10/200 + VRRP (priority 200/100, VIP .254 per VLAN) applied, and
/config/lab-provisioned written.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
All accumulated changes to kickstart template, test infrastructure,
and dnsmasq config. None of these produce a clean boot yet — saving
state before reverting to baseline for bisection.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- VMs get serial console on TCP (PXE: port 4555, ISO: port 4556)
- serialExec() helper: runs commands via telnet when SSH/network is down
- PXE test: on SSH failure, dumps hostname, IP, NetworkManager, sshd,
failed units, and fstab via serial console before failing
- Kickstart enables serial-getty@ttyS0 for auto-login on serial
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
ARM integration test:
- arm-iso-provision.test.ts: aarch64 VM boots from bastion-generated ISO
- Uses QEMU aarch64 emulation (slow but validates the R1 scenario)
- Generous timeouts for emulated boot (15min discovery, 60min install)
- test-provision.sh updated: `sudo ./scripts/test-provision.sh arm`
VM boot fixes:
- setBootDisk() preserves UEFI loader/nvram when switching to disk boot
- /boot/efi mount gets nofail in fstab (prevents emergency mode in VMs)
- chronyd enable uses || true (fails in kickstart chroot)
- createIsoVm supports arch parameter for ARM VMs
Note: SSH-after-reboot in OVMF VMs still fails — OVMF doesn't respect
efibootmgr changes and loops PXE/HTTP Boot. Real hardware works fine.
The install flow itself (discovery → kickstart → complete) is validated.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Kickstart installs on real hardware failed silently — no error reporting,
only 3 progress callbacks, zero log streaming. This overhaul makes every
install fully observable.
Kickstart improvements:
- Error trapping in %pre and %post (trap ERR sends failure details to bastion)
- 12+ granular progress stages (was 3): SSH, hostname, k3s prep, EFI boot, metadata
- Background log streamer: tails %post output and batch-sends to /api/log
- bastion_log() function for explicit log lines from kickstart scripts
Bastion API:
- POST /api/log — receives raw log lines from kickstart (single or batch)
- InstallLogBuffer — per-MAC ring buffer (2000 lines) + file persistence
- GET /api/logs/:mac — now returns log_lines + log_total alongside stages
- SSE /api/logs/:mac/follow — uses named events (event: stage vs event: log)
- Progress events forwarded to labd via bastion-progress WebSocket message
- Post-provision k3s logs routed through progressBus (was console-only)
dnsmasq fixes found during VM testing:
- HTTP Boot filename: ipxe-real.efi → ipxe.efi (leftover from old 2-stage approach)
- pxe-service directives: only in proxy mode (breaks OVMF PXE in full mode)
- PXEClient vendor class echo for UEFI firmware compatibility
Integration tests:
- PXE boot test: blank UEFI VM → dnsmasq → HTTP Boot → iPXE → bastion → install
- ISO boot test: blank VM boots from bastion-generated ISO → same flow
- Shared helpers: pxe-network (no DHCP, nftables fix), pxe-vm (UEFI + ISO boot)
- test-provision.sh: runs both PXE + ISO tests with prerequisite checks
- 250GB sparse QCOW2 disk (LVM layout needs ~204GB)
201 unit tests passing (11 new).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
