feat(bastion): unattended VyOS network install with HA (bond + VRRP) #16

Open
michal wants to merge 0 commits from feat/vyos-unattended-install into main
Owner

What

Unattended VyOS network install through the PXE bastion, for the two Protectli VP2440 firewall boxes (HA pair: LACP bond on SFP+, VRRP floating gateways, hw-id-pinned interfaces).

VyOS has no unattended installer — install_image() is unconditionally interactive (--no-prompt is wired only to add). The automation is injected via live-config's hooks component: iPXE boots the live kernel with fetch= + live-config.hooks=, the hook fetches a generated per-MAC Python driver, and the driver builds config.boot, stages the rootfs, and drives the interactive installer over a pty, reporting progress to /api/progress (including the full installer transcript on failure).

Key pieces

  • templates/vyos-boot.ipxe.ts — live-boot iPXE script (no nonetworking, no console=ttyS0)
  • templates/vyos-config-spec.ts — bond0 (802.3ad) + tagged VLANs + VRRP groups (vrid = VLAN id) + sync group + hw-id pinning + SSH keys
  • templates/vyos-install.py.ts — the generated pty driver; config built from the image's own config.boot.default via vyos.configtree, version footer reattached via vyos.component_version
  • routes/vyos.ts/vyos/autoinstall.sh + per-MAC /vyos/install.py
  • main.tsprepareVyosArtifacts() (xorriso extraction; ISO URL latest resolves the newest vyos-nightly-build GitHub release — downloads.vyos.io no longer serves direct ISOs)
  • CLI/labd/protocol — vyos spec threaded end to end (--vyos-*, --vlan, --vlan-vip flags with guards; bond-includes-PXE-NIC rejected)

Verification

  • 268 unit tests pass; no new lint errors in touched files
  • Full E2E in a libvirt VM against the real nightly ISO: PXE-equivalent boot → hook → driver → installer → reboot → installed system boots with bond/VRRP/hw-id config applied and zero migrations
  • The VM run found 5 real bugs, all fixed and encoded as code comments (config.boot.default location at hook time, dropped version footer → migration crash, missing boot medium for the rootfs copy, live-config reboot deadlock → --force, full-path disk answers)

Deployed to the labmaster bastion and both firewalls are queued; boxes install on next PXE boot.

🤖 Generated with Claude Code

https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH

## What Unattended VyOS network install through the PXE bastion, for the two Protectli VP2440 firewall boxes (HA pair: LACP bond on SFP+, VRRP floating gateways, hw-id-pinned interfaces). VyOS has no unattended installer — `install_image()` is unconditionally interactive (`--no-prompt` is wired only to `add`). The automation is injected via live-config's `hooks` component: iPXE boots the live kernel with `fetch=` + `live-config.hooks=`, the hook fetches a generated per-MAC Python driver, and the driver builds `config.boot`, stages the rootfs, and drives the interactive installer over a pty, reporting progress to `/api/progress` (including the full installer transcript on failure). ## Key pieces - `templates/vyos-boot.ipxe.ts` — live-boot iPXE script (no `nonetworking`, no `console=ttyS0`) - `templates/vyos-config-spec.ts` — bond0 (802.3ad) + tagged VLANs + VRRP groups (vrid = VLAN id) + sync group + hw-id pinning + SSH keys - `templates/vyos-install.py.ts` — the generated pty driver; config built from the image's own `config.boot.default` via `vyos.configtree`, version footer reattached via `vyos.component_version` - `routes/vyos.ts` — `/vyos/autoinstall.sh` + per-MAC `/vyos/install.py` - `main.ts` — `prepareVyosArtifacts()` (xorriso extraction; ISO URL `latest` resolves the newest `vyos-nightly-build` GitHub release — downloads.vyos.io no longer serves direct ISOs) - CLI/labd/protocol — `vyos` spec threaded end to end (`--vyos-*`, `--vlan`, `--vlan-vip` flags with guards; bond-includes-PXE-NIC rejected) ## Verification - 268 unit tests pass; no new lint errors in touched files - Full E2E in a libvirt VM against the real nightly ISO: PXE-equivalent boot → hook → driver → installer → reboot → **installed system boots with bond/VRRP/hw-id config applied and zero migrations** - The VM run found 5 real bugs, all fixed and encoded as code comments (config.boot.default location at hook time, dropped version footer → migration crash, missing boot medium for the rootfs copy, live-config reboot deadlock → `--force`, full-path disk answers) Deployed to the labmaster bastion and both firewalls are queued; boxes install on next PXE boot. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
michal added 1 commit 2026-08-10 20:53:32 +00:00
feat(bastion): unattended VyOS network install with HA (bond + VRRP)
Some checks failed
CI/CD / lint (pull_request) Failing after 10s
CI/CD / test (pull_request) Failing after 10s
CI/CD / typecheck (pull_request) Failing after 23s
CI/CD / build (pull_request) Has been skipped
CI/CD / publish-rpm (pull_request) Has been skipped
CI/CD / publish-deb (pull_request) Has been skipped
cb9d99dd69
VyOS ships no unattended installer (install_image() is unconditionally
interactive; --no-prompt is wired only to 'add'), so the automation is
injected through live-config's hooks component: iPXE boots the live
kernel with fetch= and live-config.hooks=, the hook fetches a generated
per-MAC Python driver, and the driver builds config.boot, stages the
rootfs, and drives the interactive installer over a pty.

Bastion:
- vyos-boot.ipxe template (no 'nonetworking' — breaks the hook fetch;
  no console=ttyS0 — 30s/systemd-phase on UART-less boards)
- /vyos/autoinstall.sh + /vyos/install.py routes (per-MAC driver with
  the config spec baked in as base64)
- vyos-config-spec: bond0 (802.3ad) + tagged VLANs + VRRP groups
  (vrid = VLAN id) + sync group + hw-id pinning by MAC + SSH keys;
  config built from the image's own config.boot.default via
  vyos.configtree, version footer reattached via component_version
- prepareVyosArtifacts: extract kernel/initrd/squashfs from the nightly
  ISO with xorriso; initrd picked by size from regular files only;
  ISO URL "latest" resolves the newest vyos-nightly-build GH release
  (downloads.vyos.io no longer serves direct ISOs)

Verified end-to-end in a libvirt VM against the real nightly ISO —
installed system boots with bond/VRRP/hw-id config applied and no
migrations. Fixes found by the VM run, encoded in code comments:
config.boot.default lives at /usr/share/vyos at hook time; fetch= boot
has no medium so the rootfs is symlinked to the installer's expected
path; reboot must be --force (the hook is a child of the still-starting
live-config unit); installer disk answers are full /dev paths; zram0
passes the 2GB min-disk filter so the disk is always pinned.

CLI/labd: vyos spec threaded through provision install (--vyos-* and
--vlan/--vlan-vip flags with guards), labd install route, protocol
command-install, and the bastion's direct /api/install.

268 unit tests pass; no new lint errors in touched files.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
michal added 1 commit 2026-08-11 10:16:28 +00:00
feat(bastion): bring VyOS provisioning to Fedora-grade quality
Some checks failed
CI/CD / typecheck (pull_request) Failing after 10s
CI/CD / test (pull_request) Failing after 9s
CI/CD / lint (pull_request) Failing after 24s
CI/CD / build (pull_request) Has been skipped
CI/CD / publish-rpm (pull_request) Has been skipped
CI/CD / publish-deb (pull_request) Has been skipped
5d00c42f5a
Ports the Fedora provisioning features that matter for a router onto the
VyOS path, and adds the libvirt integration test that proves them.

- Live install logs: the driver streams the installer pty (ANSI-stripped,
  batched, best-effort) to POST /api/log, so `labctl provision logs -f`
  works during a VyOS install the way Anaconda's syslog does for Fedora.
- installed.ip: report "ready at <ip>" -- the exact detail format
  routes/api.ts parses -- using the static mgmt address when known, else
  the live DHCP address. Without it VyOS machines landed with an empty IP,
  breaking provision list, logs-by-IP, recheck and reprovision.
  api.ts also guards the complete handler: VyOS boxes get the "vyos" SSH
  hint and never trigger the k3s post-provision.
- EFI network-first boot order: port of the Fedora %post efibootmgr step,
  run from the live env after install (NVRAM, not disk). Best-effort.
- Reinstall semantics: VyOS's installer already carries the previous
  config and SSH host keys forward -- the analog of Fedora's LV
  preservation -- so that stays the default. New --vyos-fresh-config
  overwrites the installed config.boot with the generated one instead,
  via a post-install target mount that also writes /config/lab-provisioned
  (mirrors Fedora's /etc/lab-provisioned, survives image upgrades).
- reprovision/recheck default to the "vyos" SSH user for VyOS machines.

Two hangs found by the VM test and fixed:
- On reinstall the installer asks "Would you like to copy data to the new
  image?" (search_previous_installation). Unanswered, the driver blocked
  on stdin until its stall timeout -- a silent 15-minute hang.
- The RAID regex missed "Would you like to choose two disks for RAID-1
  mirroring?", which would wedge any multi-disk box. Both prompts default
  to yes, so a miss also risks an unwanted mirror.

Both are now covered by a unit test asserting all 17 installer prompts
match exactly one rule -- verified to fail against the unfixed code, so
this class of bug is caught in a second instead of a 45-minute VM run.

tests/integration/vyos-provision.test.ts: fresh install, reinstall
preserves config + /config data, and freshConfig override. All 8 pass
against the real nightly ISO (EXIT=0). 273 unit tests pass; no new lint
errors in touched files.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
Author
Owner

Follow-up: Fedora-grade quality for the VyOS path

Pushed 5d00c42. Audited what the Fedora pipeline gives us and ported everything relevant to a router.

Feature Before Now
Live install logs transcript on failure only streamed to /api/loglabctl provision logs -f
installed.ip empty (broke list/recheck/reprovision) ready at <ip>
EFI network-first boot order none ported from the Fedora %post
Reinstall without losing data untested default + tested; --vyos-fresh-config to override
/etc/lab-provisioned analog none /config/lab-provisioned
Per-OS SSH user assumed Fedora's vyos for VyOS in reprovision/recheck
Integration test none vyos-provision.test.ts, 3 install rounds

Two hangs the VM test caught

  1. On reinstall the installer asks "Would you like to copy data to the new image?" — the prompt that literally is "reinstall without losing data". Unanswered, the driver blocked on stdin for 15 minutes. Silent: no error, no crash.
  2. Chasing that, an exhaustive prompt sweep found a second: "Would you like to choose two disks for RAID-1 mirroring?" would wedge any multi-disk box. Both prompts default to yes, so a miss also risks an unwanted mirror.

Both are now covered by a unit test asserting all 17 MSG_* prompts match exactly one rule. Verified it fails against the unfixed code:

× answers every installer prompt exactly once
  → expected [ 'raidChooseDisks', 'copyData' ] to deeply equal []

Verification

273 unit tests pass (24 files)
8/8 integration tests pass against the real nightly ISO (EXIT=0)
no new lint errors in touched files

Reinstall preservation proven directly in the VM — round 2 queued as vyos-r2 but the box kept host-name "vyos-r1" and the /config marker file, exactly as intended; round 3 with --vyos-fresh-config flipped to the generated config while keeping the marker.

Not done, deliberately: finer progress stages (the live transcript supersedes them) and a VyOS-native rescue mode (existing Fedora Anaconda rescue inspects a VyOS disk fine — plain ext4, no LVM).

🤖 Generated with Claude Code

## Follow-up: Fedora-grade quality for the VyOS path Pushed `5d00c42`. Audited what the Fedora pipeline gives us and ported everything relevant to a router. | Feature | Before | Now | |---|---|---| | Live install logs | transcript on failure only | streamed to `/api/log` → `labctl provision logs -f` | | `installed.ip` | empty (broke list/recheck/reprovision) | `ready at <ip>` | | EFI network-first boot order | none | ported from the Fedora `%post` | | Reinstall without losing data | untested | default + tested; `--vyos-fresh-config` to override | | `/etc/lab-provisioned` analog | none | `/config/lab-provisioned` | | Per-OS SSH user | assumed Fedora's | `vyos` for VyOS in reprovision/recheck | | Integration test | none | `vyos-provision.test.ts`, 3 install rounds | ### Two hangs the VM test caught 1. On **reinstall** the installer asks *"Would you like to copy data to the new image?"* — the prompt that literally *is* "reinstall without losing data". Unanswered, the driver blocked on stdin for 15 minutes. Silent: no error, no crash. 2. Chasing that, an exhaustive prompt sweep found a second: *"Would you like to choose two disks for RAID-1 mirroring?"* would wedge any multi-disk box. Both prompts default to **yes**, so a miss also risks an unwanted mirror. Both are now covered by a unit test asserting all 17 `MSG_*` prompts match exactly one rule. Verified it fails against the unfixed code: ``` × answers every installer prompt exactly once → expected [ 'raidChooseDisks', 'copyData' ] to deeply equal [] ``` ### Verification ``` 273 unit tests pass (24 files) 8/8 integration tests pass against the real nightly ISO (EXIT=0) no new lint errors in touched files ``` Reinstall preservation proven directly in the VM — round 2 queued as `vyos-r2` but the box kept `host-name "vyos-r1"` and the `/config` marker file, exactly as intended; round 3 with `--vyos-fresh-config` flipped to the generated config while keeping the marker. **Not done, deliberately:** finer progress stages (the live transcript supersedes them) and a VyOS-native rescue mode (existing Fedora Anaconda rescue inspects a VyOS disk fine — plain ext4, no LVM). 🤖 Generated with [Claude Code](https://claude.com/claude-code)
michal added 1 commit 2026-08-11 10:25:22 +00:00
chore(cli): regenerate shell completions for VyOS flags
Some checks failed
CI/CD / lint (pull_request) Failing after 23s
CI/CD / typecheck (pull_request) Failing after 23s
CI/CD / test (pull_request) Failing after 23s
CI/CD / build (pull_request) Has been skipped
CI/CD / publish-rpm (pull_request) Has been skipped
CI/CD / publish-deb (pull_request) Has been skipped
e36a7a193c
pnpm completions:check was failing: labctl.fish/bash were stale. The
generated --os choices still listed only fedora-43 and ubuntu-26.04
(missing vyos-rolling since the OsId union gained it), and none of the
--vyos-*/--vlan flags were present.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
michal added 1 commit 2026-08-12 11:35:12 +00:00
fix(bastion): pin the VyOS boot NIC by MAC, and detect pre-installer stalls
Some checks failed
CI/CD / typecheck (pull_request) Failing after 10s
CI/CD / test (pull_request) Failing after 10s
CI/CD / lint (pull_request) Failing after 24s
CI/CD / build (pull_request) Has been skipped
CI/CD / publish-rpm (pull_request) Has been skipped
CI/CD / publish-deb (pull_request) Has been skipped
df2dfc5d71
Both Protectli VP2440s failed to install on real hardware: they fetched
kernel+initrd and then went silent. The console showed why —

  Looking for a connected Ethernet interface ... e2 ? e3 ? e4 ? e5 ?
  Connected e4 found
  Connected e5 found
  [4.595647] igc 0000:02:00.0 e2: NIC Link is Up
  IP-Config: e4 ... no response after 15 secs - giving up
  Unable to find a live file system on the network

live-boot picks the first *connected* interface. The i40e SFP+ pair links
before the igc copper port (up at 4.6s), so it chose the fiber ports, which
have no DHCP, and never tried the NIC that actually PXE booted.

Fix: pass BOOTIF=01-<mac> on the kernel cmdline. live-boot's
Device_from_bootif() (verified present in this image) matches it against
/sys/class/net and sets DEVICE directly. The MAC comes from the dispatch
key — i.e. exactly the NIC that PXE booted — which is more reliable than
iPXE's ${net0} on a box where the booting NIC may not be net0.

Why the integration test missed it: the VM had ONE NIC, so "first connected
interface" was trivially correct, and virtio links instantly so there was no
negotiation race. createPxeVm now takes decoyNics, attaching extra NICs
ahead of the PXE NIC on a network with no route to the bastion; the VyOS
test uses 2. Without BOOTIF that reproduces the hardware failure. getVmMac
is network-aware so it still returns the booting NIC.

Also: the bastion had every clue and said nothing — it logged INSTALL
STARTED, served kernel+initrd, then nothing for 7 minutes. dispatch now
stamps dispatched_at, and /api/logs/:mac returns stalled_for_s / stalled
(8 min threshold, sized for the ~600MB squashfs fetch), so a machine wedged
before the installer environment comes up is diagnosable without a console.

Verified on hardware: both firewalls installed, bond0 802.3ad + VLANs
2/3/9/10/200 + VRRP (priority 200/100, VIP .254 per VLAN) applied, and
/config/lab-provisioned written.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
michal added 1 commit 2026-08-12 23:42:41 +00:00
feat(labsim): libvirt replica of the lab network with LACP + VyOS routing
Some checks failed
CI/CD / lint (pull_request) Failing after 11s
CI/CD / typecheck (pull_request) Failing after 10s
CI/CD / test (pull_request) Failing after 9s
CI/CD / build (pull_request) Has been skipped
CI/CD / publish-rpm (pull_request) Has been skipped
CI/CD / publish-deb (pull_request) Has been skipped
c91e44f796
A throwaway copy of the production VLAN topology so routing and firewall
changes can be tested before they touch the real network. Same VLAN IDs and
roles as UniFi, deliberately different ranges (172.31.<vlan>.0/24) so nothing
here can be mistaken for production.

- OVS fabric: real 802.1Q. Access port per micro VM, host leg per VLAN (.2,
  for SSH only — NOT the VMs' default route, so inter-VLAN tests exercise the
  router rather than the host's routing table), and a trunk portgroup with
  VLAN 1 declared nativeMode='untagged'.
- Six Alpine micro VMs (256MB, copy-on-write overlays on one 176MB image),
  SSH + a hello-world HTTP page naming the VLAN.
- VyOS router installed to disk unattended over the console, with the SAME
  config shape as the VP2440s: two NICs in an LACP bond carrying the trunk,
  VLAN 1 native, bond0.<vlan> holding the .1 gateway on each.
- labsim-matrix.py: full-mesh ICMP/TCP22/TCP80 probe, ~0.2s, --watch
  highlights cells that changed since the last sweep. Guest-side probe is
  python3 (already present via cloud-init) so nothing is installed on VMs
  that have no internet.
- Prometheus + Grafana (anonymous auth, no login) with a provisioned
  dashboard: heatmap plus a state timeline showing exactly when a path
  flipped. Verified end to end: one VyOS rule took sum(labsim_reachable)
  from 90 to 84, blocking precisely kvm<->k8s across all three protocols.

Traps found building this, all now encoded in the scripts:
- virtio-net breaks 802.3ad: the guest's bonding driver reports slaves
  "MII Status: down" despite carrier=1 and never sends an LACPDU, so the bond
  sits in AD_STATE_DEFAULTED. e1000e fixes it with no other change. Matches
  the netdev thread "bonding (IEEE 802.3ad) not working with qemu/virtio".
- OVS defaults bonds to active-backup, which does not speak LACP at all —
  bond_mode=balance-tcp is required.
- LACP deadlock: OVS holds members disabled until negotiation while the
  partner needs carrier before it will send LACPDUs. lacp-fallback-ab breaks it.
- LACPDUs are untagged, so a trunk with no native VLAN has nowhere to put them.
- --boot cdrom,hd re-runs the ISO on every restart, so every commit+save went
  to a live system that evaporated. Install now switches the VM to boot hd.
- cloud-init on Alpine: users stay locked without lock_passwd:false, one
  failing runcmd aborts the rest, busybox here has no httpd applet, and
  start-stop-daemon --exec /usr/bin/python3 matches cloud-init's own python3.
- The user-data heredoc is unquoted, so backticks in a COMMENT were executed
  by the host shell and their output corrupted the YAML. build_seed now
  validates with yaml.safe_load before building the ISO.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
michal added 1 commit 2026-08-12 23:56:08 +00:00
feat(labsim): live topology view with per-path latency
Some checks failed
CI/CD / lint (pull_request) Failing after 9s
CI/CD / test (pull_request) Failing after 9s
CI/CD / typecheck (pull_request) Failing after 24s
CI/CD / build (pull_request) Has been skipped
CI/CD / publish-rpm (pull_request) Has been skipped
CI/CD / publish-deb (pull_request) Has been skipped
a5b36678ed
The Grafana heatmap of 1s and 0s said almost nothing, and the state timeline
was an unreadable pile of overlapping series labels. Replaced as the primary
view with a purpose-built page served by the exporter itself.

- Probe now captures ICMP RTT, exposed as labsim_rtt_ms{src,dst}. A path that
  is up but slow is a different problem from one that is down, and a pass/fail
  grid cannot show it.
- Exporter serves / (topology), /api/matrix (JSON) and /metrics.
- topology.html: node per VLAN in a ring, VyOS router in the centre because
  every inter-VLAN packet really does traverse it, one line per pair coloured
  green/red with the RTT on it. Hovering gives per-direction state. A node ring
  goes red if anything to or from it is blocked. Side panels list blocked paths
  and the slowest links. Refreshes every 5s, no dependencies.

Grafana stays for what it is actually good at — history of when a path flipped.

Label placement is deliberate: RTT captions sit ~32% along each edge with a
perpendicular nudge, because every diagonal of a 6-node mesh crosses the centre
and midpoint labels stack on the router node.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
Some checks failed
CI/CD / lint (pull_request) Failing after 10s
CI/CD / test (pull_request) Failing after 10s
CI/CD / typecheck (pull_request) Failing after 22s
CI/CD / build (pull_request) Has been skipped
CI/CD / publish-rpm (pull_request) Has been skipped
CI/CD / publish-deb (pull_request) Has been skipped
Checking for merge conflicts…
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin feat/vyos-unattended-install:feat/vyos-unattended-install
git checkout feat/vyos-unattended-install
Sign in to join this conversation.
No Reviewers
No Label
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: michal/lab#16