fix(bastion): pin the VyOS boot NIC by MAC, and detect pre-installer stalls
Some checks failed
CI/CD / typecheck (pull_request) Failing after 10s
CI/CD / test (pull_request) Failing after 10s
CI/CD / lint (pull_request) Failing after 24s
CI/CD / build (pull_request) Has been skipped
CI/CD / publish-rpm (pull_request) Has been skipped
CI/CD / publish-deb (pull_request) Has been skipped
Some checks failed
CI/CD / typecheck (pull_request) Failing after 10s
CI/CD / test (pull_request) Failing after 10s
CI/CD / lint (pull_request) Failing after 24s
CI/CD / build (pull_request) Has been skipped
CI/CD / publish-rpm (pull_request) Has been skipped
CI/CD / publish-deb (pull_request) Has been skipped
Both Protectli VP2440s failed to install on real hardware: they fetched
kernel+initrd and then went silent. The console showed why —
Looking for a connected Ethernet interface ... e2 ? e3 ? e4 ? e5 ?
Connected e4 found
Connected e5 found
[4.595647] igc 0000:02:00.0 e2: NIC Link is Up
IP-Config: e4 ... no response after 15 secs - giving up
Unable to find a live file system on the network
live-boot picks the first *connected* interface. The i40e SFP+ pair links
before the igc copper port (up at 4.6s), so it chose the fiber ports, which
have no DHCP, and never tried the NIC that actually PXE booted.
Fix: pass BOOTIF=01-<mac> on the kernel cmdline. live-boot's
Device_from_bootif() (verified present in this image) matches it against
/sys/class/net and sets DEVICE directly. The MAC comes from the dispatch
key — i.e. exactly the NIC that PXE booted — which is more reliable than
iPXE's ${net0} on a box where the booting NIC may not be net0.
Why the integration test missed it: the VM had ONE NIC, so "first connected
interface" was trivially correct, and virtio links instantly so there was no
negotiation race. createPxeVm now takes decoyNics, attaching extra NICs
ahead of the PXE NIC on a network with no route to the bastion; the VyOS
test uses 2. Without BOOTIF that reproduces the hardware failure. getVmMac
is network-aware so it still returns the booting NIC.
Also: the bastion had every clue and said nothing — it logged INSTALL
STARTED, served kernel+initrd, then nothing for 7 minutes. dispatch now
stamps dispatched_at, and /api/logs/:mac returns stalled_for_s / stalled
(8 min threshold, sized for the ~600MB squashfs fetch), so a machine wedged
before the installer environment comes up is diagnosable without a console.
Verified on hardware: both firewalls installed, bond0 802.3ad + VLANs
2/3/9/10/200 + VRRP (priority 200/100, VIP .254 per VLAN) applied, and
/config/lab-provisioned written.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
This commit is contained in:
@@ -29,6 +29,17 @@ export interface PxeVmConfig {
|
||||
diskSize: number; // GB
|
||||
network: string; // libvirt network name
|
||||
arch?: "x86_64" | "aarch64";
|
||||
/**
|
||||
* Extra NICs enumerated BEFORE the PXE NIC, on a network with no route to
|
||||
* the bastion (defaults to libvirt's "default").
|
||||
*
|
||||
* Real multi-NIC boxes expose a class of bug a single-NIC VM cannot: an
|
||||
* initramfs that picks "the first connected interface" grabs one of these
|
||||
* instead of the NIC that PXE booted, and then cannot reach the bastion.
|
||||
* Defaults to 0 (single NIC).
|
||||
*/
|
||||
decoyNics?: number;
|
||||
decoyNetwork?: string;
|
||||
}
|
||||
|
||||
/** Create a blank UEFI VM that PXE boots from the network. */
|
||||
@@ -61,6 +72,10 @@ export function createPxeVm(config: PxeVmConfig): void {
|
||||
`--memory=${config.memory}`,
|
||||
`--vcpus=${config.vcpus}`,
|
||||
`--disk=path=${diskPath},format=qcow2,bus=virtio`,
|
||||
// Decoys first so they enumerate ahead of the PXE NIC. They are up and
|
||||
// carry a lease, but have no route to the bastion.
|
||||
...Array.from({ length: config.decoyNics ?? 0 }, () =>
|
||||
`--network=network=${config.decoyNetwork ?? "default"},model=virtio`),
|
||||
`--network=network=${config.network},model=virtio`,
|
||||
// UEFI firmware — required for PXE boot in modern mode
|
||||
`--boot=uefi,network,hd`,
|
||||
@@ -95,12 +110,21 @@ export function destroyPxeVm(name: string): void {
|
||||
}
|
||||
|
||||
/** Get the MAC address of a VM's first NIC. */
|
||||
export function getVmMac(name: string): string | null {
|
||||
export function getVmMac(name: string, network?: string): string | null {
|
||||
const result = virsh("domiflist", name);
|
||||
if (result.status !== 0) return null;
|
||||
// Output format: Interface Type Source Model MAC
|
||||
const match = result.stdout.match(/([0-9a-f]{2}:[0-9a-f]{2}:[0-9a-f]{2}:[0-9a-f]{2}:[0-9a-f]{2}:[0-9a-f]{2})/i);
|
||||
return match ? match[1].toLowerCase() : null;
|
||||
// With decoy NICs present, match the line for the PXE network so we return
|
||||
// the NIC that actually boots rather than whichever is listed first.
|
||||
const lines = result.stdout.split("\n");
|
||||
const candidates = network === undefined
|
||||
? lines
|
||||
: lines.filter((l) => l.split(/\s+/).includes(network));
|
||||
for (const line of candidates.length > 0 ? candidates : lines) {
|
||||
const m = line.match(/([0-9a-f]{2}:[0-9a-f]{2}:[0-9a-f]{2}:[0-9a-f]{2}:[0-9a-f]{2}:[0-9a-f]{2})/i);
|
||||
if (m) return m[1].toLowerCase();
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
/** Reboot a VM (force off + start). */
|
||||
|
||||
Reference in New Issue
Block a user