vyos: WAN follows VRRP mastership, rehearsed in labsim

The ISP is consumer with one static IP, so "both routers hold WAN" is not
available: the 10 gig lease is anchored to a cloned MAC and the PPPoE line to a
single credential. The WAN therefore has to move with mastership.

Proven end to end in the sim: the secondary was promoted, took the WAN, got a
lease, installed a default route, and a LAN VM reached the internet through it
(3/3, 9ms). Demoting released it -- link down, no address. Rebooting the master
converged correctly too: it came back BACKUP with the WAN disabled while the
peer kept it.

The rehearsal earned its keep four times over, and none of these were visible
from reading the docs:

  - `transition-script` alone is NOT safe to hang internet on. VyOS delivers it
    through keepalived-fifo.py, and on one promotion that helper logged NOTHING
    while Keepalived_vrrp logged all six instances entering MASTER and the
    built-in notify_master for conntrack-sync ran normally. The result was a
    router holding every VIP with no WAN -- the 2026-09-02 outage, recreated by
    the mechanism meant to prevent it. Hence vrrp-wan-reconcile on a 30s timer:
    the scripts give speed, the timer gives correctness.

  - The health check must ask REALITY, not a marker. Keying "am I master" on a
    /run file written by the transition script meant that when the script did
    not run, the router believed it was backup, passed the check, and kept the
    VIPs it could not serve. It now asks whether the VIP is actually on the box.

  - The old address-based check DEADLOCKED this design: may-I-be-master required
    already having WAN, and only the master gets WAN. That is why vyos002 sat in
    FAULT for ever -- the safety check had silently removed the redundancy it
    existed to protect.

  - script-template must be the FIRST thing a script does. Sourced after an if,
    an exec and a mkdir it terminated the script inside the source, rc=0, no
    output: the reconciler reported success having done nothing. Hence the split
    into vrrp-wan-apply, matching the shape /config/vyos-known-good already uses.

Two hazards found and handled rather than discovered in production:

  - A `configure` session whose process dies leaks a unionfs mount under
    /opt/vyatta/config/tmp, and one of those holds the commit lock -- after
    which every commit fails, including the manual one you try to fix it with.
    A 30s job that can leak one per failure wedges the box on its own, so the
    reconciler reaps dead sessions before it starts. It cleared 8 on the sim.

  - Any `save` while a box is master persists the enabled WAN into config.boot,
    so a reboot would claim the shared MAC regardless of VRRP state. Observed:
    an ordinary console-apply did exactly this. config.boot must keep `disable`
    on BOTH routers; the model asserts it and vyos:verify reports it as drift.

NOT yet applied to production, and it should not be until the remaining item is
settled: a clean, deliberately-triggered failover has been seen via reboot, but
`restart vrrp` twice failed to move mastership at all, so the trigger for a
planned failover is still unproven.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH

.gitignore

@@ -31,3 +31,9 @@ node_modules/
 # Asahi build artifacts (large)
 bastion/.asahi-cache/
 bastion/asahi-repo/*.zip
+
+# Regenerated by labsim/dualstack-lab.sh; derived state, not source.
+labsim/dualstack-evidence/
+
+# Runtime snapshots from labsim/cilium-ipam-switch.sh
+labsim/.ipam-switch-state/

bastion/completions/labctl.bash

@@ -59,10 +59,10 @@ _labctl() {
       COMPREPLY=($(compgen -W "-h --help" -- "$cur"))
       return ;;
     "provision install")
-      COMPREPLY=($(compgen -W "--role --os --disk -h --help" -- "$cur"))
+      COMPREPLY=($(compgen -W "--role --os --disk --vyos-mgmt --vyos-mgmt-address --vyos-bond --vyos-bond-address --vyos-bond-vrrp --vlan-vip --vyos-vrrp-priority --vyos-mgmt-vlan --vlan --vyos-password --vyos-hwid --vyos-fresh-config -h --help" -- "$cur"))
       return ;;
     "provision reprovision")
-      COMPREPLY=($(compgen -W "--role --os --disk -h --help" -- "$cur"))
+      COMPREPLY=($(compgen -W "--role --os --disk --user -h --help" -- "$cur"))
       return ;;
     "provision debug")
       COMPREPLY=($(compgen -W "--pxe-boot -h --help" -- "$cur"))

bastion/completions/labctl.fish

@@ -132,13 +132,26 @@ complete -c labctl -n "__labctl_using_cmd provision" -a recheck -d 'Refresh hard
 
 # provision install options
 complete -c labctl -n "__labctl_in_cmd provision install" -l role -d 'Machine role (see below)' -xa 'vanilla worker infra labcontroller'
-complete -c labctl -n "__labctl_in_cmd provision install" -l os -d 'Operating system' -xa 'fedora-43 ubuntu-26.04'
+complete -c labctl -n "__labctl_in_cmd provision install" -l os -d 'Operating system' -xa 'fedora-43 ubuntu-26.04 vyos-rolling'
 complete -c labctl -n "__labctl_in_cmd provision install" -l disk -d 'Target disk device (auto-detect if omitted)' -x
+complete -c labctl -n "__labctl_in_cmd provision install" -l vyos-mgmt -d 'VyOS: untagged interface the machine PXE boots from (default eth0)' -x
+complete -c labctl -n "__labctl_in_cmd provision install" -l vyos-mgmt-address -d 'VyOS: CIDR for the management interface, or \'dhcp\' (default dhcp)' -x
+complete -c labctl -n "__labctl_in_cmd provision install" -l vyos-bond -d 'VyOS: comma-separated LACP bond members (must exclude the PXE NIC)' -x
+complete -c labctl -n "__labctl_in_cmd provision install" -l vyos-bond-address -d 'VyOS: address on the untagged bond (trunk native VLAN)' -x
+complete -c labctl -n "__labctl_in_cmd provision install" -l vyos-bond-vrrp -d 'VyOS: VRRP VIP floated on the untagged bond' -x
+complete -c labctl -n "__labctl_in_cmd provision install" -l vlan-vip -d 'VyOS: VRRP VIP for a --vlan entry (repeatable)' -x
+complete -c labctl -n "__labctl_in_cmd provision install" -l vyos-vrrp-priority -d 'VyOS: VRRP priority for all groups on this box (higher = master)' -x
+complete -c labctl -n "__labctl_in_cmd provision install" -l vyos-mgmt-vlan -d 'VyOS: tagged management VLAN on the PXE port' -x
+complete -c labctl -n "__labctl_in_cmd provision install" -l vlan -d 'VyOS: tagged VLAN sub-interface on the bond (repeatable)' -x
+complete -c labctl -n "__labctl_in_cmd provision install" -l vyos-password -d 'VyOS: password for the \'vyos\' user' -x
+complete -c labctl -n "__labctl_in_cmd provision install" -l vyos-hwid -d 'VyOS: pin an interface name to a MAC via hw-id (repeatable)' -x
+complete -c labctl -n "__labctl_in_cmd provision install" -l vyos-fresh-config -d 'VyOS: on reinstall, overwrite the preserved config with the generated one'
 
 # provision reprovision options
 complete -c labctl -n "__labctl_in_cmd provision reprovision" -l role -d 'Machine role (see below)' -xa 'vanilla worker infra labcontroller'
-complete -c labctl -n "__labctl_in_cmd provision reprovision" -l os -d 'Operating system' -xa 'fedora-43 ubuntu-26.04'
+complete -c labctl -n "__labctl_in_cmd provision reprovision" -l os -d 'Operating system' -xa 'fedora-43 ubuntu-26.04 vyos-rolling'
 complete -c labctl -n "__labctl_in_cmd provision reprovision" -l disk -d 'Target disk device (auto-detect if omitted)' -x
+complete -c labctl -n "__labctl_in_cmd provision reprovision" -l user -d 'SSH user for the reboot (default: vyos for VyOS machines, else current user)' -x
 
 # provision debug options
 complete -c labctl -n "__labctl_in_cmd provision debug" -l pxe-boot -d 'Boot installed system via PXE (kernel+initrd from network, root from NVMe)'

bastion/docs/ARCHITECTURE.md

@@ -89,83 +89,6 @@ Side paths:
 
 ---
 
-## Multi-architecture PXE
-
-The bastion serves both `x86_64` and `aarch64` over the network. Nothing about this is
-operator-configured -- there is no `--arch` flag, by design.
-
-### How a client's architecture is decided
-
-1. **DHCP option 93** (Client System Architecture) picks the *bootloader*. dnsmasq matches
-   it and hands out a matching iPXE binary:
-
-   | Option 93 | Client | Served |
-   |---|---|---|
-   | `0` | x86 BIOS | `undionly.kpxe` (TFTP) |
-   | `7`, `9` | x64 UEFI | `ipxe.efi` (TFTP) |
-   | `11` | **ARM64 UEFI** | `ipxe-arm64.efi` (TFTP) |
-   | `16` | x64 UEFI HTTP Boot | `http://…/ipxe.efi` |
-   | `19` | **ARM64 UEFI HTTP Boot** | `http://…/ipxe-arm64.efi` |
-
-   Values come from the IANA Processor Architecture Types registry. Note `19`, not `20` --
-   `20` is *pc/at bios boot from http*. EDK2/AAVMF prefers HTTP Boot over TFTP PXE, so the
-   iPXE binaries are staged in **both** `tftpDir` and `httpDir` (symlinked by `main.ts`).
-
-2. **`/dispatch` picks the kernel.** Option 93 never reaches the HTTP endpoint, so
-   `boot.ipxe` passes iPXE's own `${buildarch}` as `?arch=`. `resolveArch()` prefers, in
-   order: the tracked machine record → the reported `?arch=` → the configured default.
-   The record wins because it is what we observed on the machine itself.
-
-### Artifact naming
-
-`x86_64` keeps the original unsuffixed paths so its rendered iPXE scripts are unchanged;
-everything else is suffixed. `kernelPath()` / `initrdPath()` in `templates/boot.ipxe.ts`
-are the single source of truth, used by both the templates and `main.ts` staging.
-
-| arch | kernel | initrd |
-|---|---|---|
-| `x86_64` | `/vmlinuz` | `/initrd.img` |
-| `aarch64` | `/vmlinuz-aarch64` | `/initrd-aarch64.img` |
-
-`tests/ipxe-x86-regression.test.ts` pins the x86_64 output against a golden fixture.
-
-### arm64 gotchas
-
-- **LoadFile2 is mandatory.** arm64 has no `HdrS` boot protocol; the kernel's EFI stub
-  fetches the initrd over the UEFI `EFI_LOAD_FILE2_PROTOCOL`. An iPXE build without it
-  accepts the `initrd` line, silently drops it, and the kernel panics with
-  `VFS: Unable to mount root fs on unknown-block(0,0)`. Fedora's
-  `ipxe-bootimgs-aarch64` implements it; the integration test asserts this up front so
-  the failure names itself instead of looking like a disk problem.
-- **`nomodeset` is x86-only.** On arm64 there is no VGA path to fall back to. aarch64 gets
-  `console=tty0 console=ttyAMA0,115200` instead — the last `console=` wins for
-  `/dev/console`, so serial is the interactive one.
-- **Ubuntu is x86_64-only.** `releases.ubuntu.com` publishes no arm64 netboot artifacts.
-  `osSupportsArch()` encodes this, and both the install guard and `/dispatch` refuse the
-  combination rather than serving an x86 kernel to an ARM machine.
-
----
-
-## Onboarding classification (vendor OS)
-
-Machines carry an `onboard` field: `"pxe"` (default) or `"ssh"`, plus `vendor_os` naming
-what they run. `classifyOnboard()` in `@lab/shared` sets it from DMI identity, with known
-hardware also matched by MAC — a machine can sit in state for a long time with no DMI, and
-a DMI-only rule would fail open exactly where it matters.
-
-`onboard: "ssh"` means *we cannot rebuild this machine's OS*. Installs are refused at both
-entry points (`/api/install` and the labd `command-install` handler) with an error naming
-the machine and pointing at `provision debug`. **Rescue is never guarded** — being unable
-to reinstall a machine is precisely when a rescue shell is needed.
-
-This is a fact about the machine, not a blocklist. The refusal follows from "no image in
-our pipeline restores `vendor_os`", so adding a DGX OS image to the pipeline is what
-unblocks the DGX Sparks — no entry needs deleting.
-
-Current classifications: NVIDIA DGX Spark (`spark-2935`, `spark-3a1c`) → `dgx-os`.
-
----
-
 ## Packages
 
 ### Monorepo Structure
@@ -481,36 +404,6 @@ Hardcoded `/dev/sda` default broke NVMe-only machines. Fix: default to empty str
 ### Anaconda Rescue Mode Limitations
 `%pre` and `%post` sections do not execute in `inst.rescue` mode. SSH in rescue mode is provided by Anaconda's `inst.sshd` kernel parameter + `sshpw` kickstart directive. Manual setup via `curl bastion:8080/debug-setup.sh | bash` for nc listener.
 
-**Unresolved (2026-08-11): rescue SSH has never been observed working.** Adding the first
-integration coverage for `provision debug` (`tests/integration/pxe-rescue.test.ts`) showed the
-rescue environment coming up correctly — the bastion serves the kernel and initrd, Anaconda
-boots, fetches `debug.ks`, and reaches its installer environment — but **nothing ever listens on
-port 22**.
-
-Strength of the evidence, stated precisely because it decides where to look next:
-- **aarch64 — direct.** Port 22 probed every 20s for 30 minutes while the Anaconda installer
-  environment was demonstrably running (NetworkManager, polkitd, rsyslog on the console). Never
-  opened.
-- **x86_64 — corroborating, not conclusive.** One clean KVM run (943s) where SSH never became
-  available inside a 15-minute budget. That VM's progress into the rescue environment was *not*
-  observed — vitest's final reporter discards the streamed log — so it is consistent with the
-  aarch64 result but does not independently prove it. Re-run with `KEEP_VM=1` and probe port 22
-  directly to settle it.
-
-If the x86_64 result holds up, this is orthogonal to the multi-architecture work, since x86_64 is
-untouched by it. Leads worth checking, in order:
-- Does `inst.sshd` actually start `sshd` in `inst.rescue` mode, or only in install mode? The
-  port never opens, so this is the prime suspect — an auth problem would still show an open port.
-- `sshkey` may apply only to the *installed* system, leaving the installer environment
-  password-only via `sshpw`. That would matter once sshd does listen: the test authenticates
-  key-only (`BatchMode=yes`).
-- The `%anaconda`-context directives in `debug.ks` may be skipped entirely when a kickstart is
-  supplied alongside `inst.rescue`.
-
-Until this is resolved, `provision debug` gets you a booted rescue environment on the console
-(including on arm64), but not an SSH shell. The `debug-setup.sh` nc-listener path is the
-documented workaround and is unaffected.
-
 ---
 
 ## Planned Work (Taskmaster)

bastion/package.json

@@ -21,14 +21,10 @@
     "test:integration:pxe:host": "sudo -E $(which npx) vitest run -c tests/integration/vitest.config.ts -t 'PXE boot'",
     "test:integration:iso": "vitest run -c tests/integration/vitest.config.ts -t 'ISO boot'",
     "test:integration:iso:host": "sudo -E $(which npx) vitest run -c tests/integration/vitest.config.ts -t 'ISO boot'",
+    "test:integration:vyos": "vitest run -c tests/integration/vitest.config.ts -t 'VyOS provisioning'",
+    "test:integration:vyos:host": "sudo -E $(which npx) vitest run -c tests/integration/vitest.config.ts -t 'VyOS provisioning'",
     "test:integration:arm-iso": "vitest run -c tests/integration/vitest.config.ts -t 'ARM ISO'",
     "test:integration:arm-iso:host": "sudo -E $(which npx) vitest run -c tests/integration/vitest.config.ts -t 'ARM ISO'",
-    "test:integration:rescue": "vitest run -c tests/integration/vitest.config.ts -t 'x86 rescue boot'",
-    "test:integration:rescue:host": "sudo -E $(which npx) vitest run -c tests/integration/vitest.config.ts -t 'x86 rescue boot'",
-    "test:integration:arm-pxe": "vitest run -c tests/integration/vitest.config.ts -t 'ARM PXE rescue'",
-    "test:integration:arm-pxe:host": "sudo -E $(which npx) vitest run -c tests/integration/vitest.config.ts -t 'ARM PXE rescue'",
-    "test:integration:arm-pxe-full": "ARM_PXE_FULL=1 vitest run -c tests/integration/vitest.config.ts -t 'ARM PXE'",
-    "test:integration:arm-pxe-full:host": "sudo -E ARM_PXE_FULL=1 $(which npx) vitest run -c tests/integration/vitest.config.ts -t 'ARM PXE'",
     "test:integration:asahi": "vitest run -c tests/integration/vitest.config.ts -t 'asahi firstboot'",
     "test:integration:asahi:host": "sudo -E $(which npx) vitest run -c tests/integration/vitest.config.ts -t 'asahi firstboot'",
     "test:integration:asahi-validate": "vitest run -c tests/integration/vitest.config.ts -t 'asahi.*validation'",

bastion/scripts/test-provision.sh

@@ -2,19 +2,16 @@
 # Run PXE and/or ISO boot integration tests.
 #
 # Usage:
-#   sudo ./scripts/test-provision.sh              # run PXE + ISO (x86_64)
-#   sudo ./scripts/test-provision.sh pxe          # PXE only
-#   sudo ./scripts/test-provision.sh iso          # ISO only (x86_64)
-#   sudo ./scripts/test-provision.sh rescue       # x86_64 Anaconda rescue boot + SSH (~15min)
-#   sudo ./scripts/test-provision.sh arm          # ARM ISO boot (emulated, SLOW ~60min)
-#   sudo ./scripts/test-provision.sh arm-pxe      # ARM network PXE rescue: NBP + rescue over SSH (~25-30min)
-#   sudo ./scripts/test-provision.sh arm-pxe-full # ARM network PXE incl. discover + full install (~75-95min)
-#   sudo ./scripts/test-provision.sh all          # all tests including ARM
+#   sudo ./scripts/test-provision.sh          # run PXE + ISO (x86_64)
+#   sudo ./scripts/test-provision.sh pxe      # PXE only
+#   sudo ./scripts/test-provision.sh iso      # ISO only (x86_64)
+#   sudo ./scripts/test-provision.sh arm      # ARM ISO boot (emulated, SLOW ~60min)
+#   sudo ./scripts/test-provision.sh all      # all tests including ARM
 #
 # Prerequisites:
 #   libvirtd, OVMF (edk2-ovmf), iPXE (ipxe-bootimgs-x86),
 #   dnsmasq, xorriso, mtools, virt-install, qemu-img
-#   ARM: qemu-system-aarch64, edk2-aarch64, ipxe-bootimgs-aarch64
+#   ARM: qemu-system-aarch64, edk2-aarch64
 set -e
 
 SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
@@ -61,10 +58,6 @@ if [ ! -f /usr/share/edk2/ovmf/OVMF_CODE.fd ]; then
     exit 1
 fi
 
-MODE="${1:-both}"
-
-# iPXE binaries are per-architecture. x86_64 is always required (the dnsmasq config
-# references it); arm64 only for the ARM network-PXE modes.
 IPXE_EFI=""
 for f in /usr/share/ipxe/ipxe-snponly-x86_64.efi /usr/share/ipxe/ipxe-snp-x86_64.efi /usr/share/ipxe/ipxe-x86_64.efi; do
     [ -f "$f" ] && IPXE_EFI="$f" && break
@@ -74,20 +67,6 @@ if [ -z "$IPXE_EFI" ]; then
     exit 1
 fi
 
-IPXE_EFI_ARM64=""
-for f in /usr/share/ipxe/arm64-efi/snponly.efi /usr/share/ipxe/arm64-efi/ipxe.efi; do
-    [ -f "$f" ] && IPXE_EFI_ARM64="$f" && break
-done
-
-case "$MODE" in
-    arm-pxe|arm-pxe-full|all)
-        if [ -z "$IPXE_EFI_ARM64" ] && [ "$MODE" != "all" ]; then
-            echo -e "${RED}arm64 iPXE binary not found.${RESET} Install: sudo dnf install ipxe-bootimgs-aarch64"
-            exit 1
-        fi
-        ;;
-esac
-
 # Find SSH key
 SSH_KEY=""
 for name in id_ed25519 id_ecdsa id_rsa; do
@@ -104,19 +83,10 @@ fi
 echo -e "  User:    ${BOLD}$REAL_USER${RESET}"
 echo -e "  SSH key: ${BOLD}$SSH_KEY${RESET}"
 echo -e "  iPXE:    ${BOLD}$IPXE_EFI${RESET}"
-echo -e "  iPXE a64:${BOLD} ${IPXE_EFI_ARM64:-not installed}${RESET}"
 echo ""
 
-require_arm_emulation() {
-    if ! command -v qemu-system-aarch64 &>/dev/null; then
-        echo -e "${RED}qemu-system-aarch64 not found.${RESET} Install: sudo dnf install qemu-system-aarch64 edk2-aarch64"
-        exit 1
-    fi
-    if [ ! -f /usr/share/edk2/aarch64/QEMU_EFI.fd ]; then
-        echo -e "${RED}AAVMF firmware not found.${RESET} Install: sudo dnf install edk2-aarch64"
-        exit 1
-    fi
-}
+# --- Determine which tests to run ---
+MODE="${1:-both}"
 
 run_test() {
     local name="$1" pattern="$2"
@@ -146,26 +116,13 @@ case "$MODE" in
         run_test "ISO boot" "ISO boot" || FAILED=1
         ;;
     arm|arm-iso)
-        require_arm_emulation
+        if ! command -v qemu-system-aarch64 &>/dev/null; then
+            echo -e "${RED}qemu-system-aarch64 not found.${RESET} Install: sudo dnf install qemu-system-aarch64 edk2-aarch64"
+            exit 1
+        fi
         echo -e "${YELLOW}ARM emulation is ~10x slower than native. Expect 30-60 minutes.${RESET}"
         run_test "ARM ISO boot" "ARM ISO" || FAILED=1
         ;;
-    rescue)
-        echo -e "${YELLOW}x86_64 rescue boot (KVM). Expect ~15 minutes.${RESET}"
-        run_test "x86 rescue boot" "x86 rescue boot" || FAILED=1
-        ;;
-    arm-pxe)
-        require_arm_emulation
-        echo -e "${YELLOW}ARM emulation is ~10x slower than native. Expect 25-30 minutes.${RESET}"
-        echo -e "${YELLOW}Covers option 93 -> arm64 NBP, arch resolution, and rescue over SSH.${RESET}"
-        echo -e "${YELLOW}For the full install too, use: $0 arm-pxe-full${RESET}"
-        run_test "ARM PXE rescue" "ARM PXE rescue" || FAILED=1
-        ;;
-    arm-pxe-full)
-        require_arm_emulation
-        echo -e "${YELLOW}ARM emulation is ~10x slower than native. Expect 75-95 minutes.${RESET}"
-        ARM_PXE_FULL=1 run_test "ARM PXE (rescue + install)" "ARM PXE" || FAILED=1
-        ;;
     both)
         run_test "PXE boot" "PXE boot" || FAILED=1
         run_test "ISO boot" "ISO boot" || FAILED=1
@@ -176,17 +133,12 @@ case "$MODE" in
         if command -v qemu-system-aarch64 &>/dev/null; then
             echo -e "${YELLOW}ARM emulation is ~10x slower than native.${RESET}"
             run_test "ARM ISO boot" "ARM ISO" || FAILED=1
-            if [ -n "$IPXE_EFI_ARM64" ]; then
-                run_test "ARM PXE rescue" "ARM PXE rescue" || FAILED=1
-            else
-                echo -e "${YELLOW}Skipping ARM PXE test (ipxe-bootimgs-aarch64 not installed)${RESET}"
-            fi
         else
-            echo -e "${YELLOW}Skipping ARM tests (qemu-system-aarch64 not installed)${RESET}"
+            echo -e "${YELLOW}Skipping ARM test (qemu-system-aarch64 not installed)${RESET}"
         fi
         ;;
     *)
-        echo "Usage: $0 [pxe|iso|rescue|arm|arm-pxe|arm-pxe-full|both|all]"
+        echo "Usage: $0 [pxe|iso|arm|both|all]"
         exit 1
         ;;
 esac

bastion/src/bastion/src/config.ts

@@ -20,6 +20,15 @@ export function loadConfig(overrides: Partial<BastionConfig> = {}): BastionConfi
   const ubuntuMirror = overrides.ubuntuMirror ?? process.env["UBUNTU_MIRROR"]
     ?? `https://releases.ubuntu.com/${ubuntuVersion}`;
 
+  // "latest" resolves the newest nightly ISO from the vyos-nightly-build GitHub
+  // releases at startup. downloads.vyos.io no longer serves direct rolling ISOs
+  // (it returns the vyos.io site, and nightly builds sit behind a signup form);
+  // GitHub releases are the remaining free, unauthenticated direct source.
+  // LTS ISOs are subscription-only. Set VYOS_ISO_URL to pin a specific build.
+  const vyosIsoUrl = overrides.vyosIsoUrl ?? process.env["VYOS_ISO_URL"] ?? "latest";
+  const vyosDefaultPassword = overrides.vyosDefaultPassword
+    ?? process.env["VYOS_DEFAULT_PASSWORD"] ?? "vyos";
+
   const fedoraMirror = `https://download.fedoraproject.org/pub/fedora/linux/releases/${fedoraVersion}/Everything/${arch}/os`;
   const tftpDir = `${bastionDir}/tftp`;
   const httpDir = `${bastionDir}/http`;
@@ -38,6 +47,8 @@ export function loadConfig(overrides: Partial<BastionConfig> = {}): BastionConfi
     dhcpRangeEnd,
     ubuntuVersion,
     ubuntuMirror,
+    vyosIsoUrl,
+    vyosDefaultPassword,
     // These are populated at runtime by the network service
     iface: overrides.iface ?? "",
     serverIp: overrides.serverIp ?? "",

bastion/src/bastion/src/main.ts

@@ -3,9 +3,7 @@
 
 import { mkdirSync, writeFileSync, readFileSync, existsSync, copyFileSync, symlinkSync, unlinkSync } from "node:fs";
 import { execSync } from "node:child_process";
-import type { Arch, BastionConfig } from "@lab/shared";
-import { SUPPORTED_ARCHES, fedoraMirrorFor, classifyOnboard } from "@lab/shared";
-import { kernelPath, initrdPath } from "./templates/boot.ipxe.js";
+import type { BastionConfig } from "@lab/shared";
 import { loadConfig } from "./config.js";
 import { populateNetworkConfig } from "./services/network.js";
 import { createApp } from "./server.js";
@@ -15,7 +13,6 @@ import { renderBootIpxe } from "./templates/boot.ipxe.js";
 import { logger } from "./services/logger.js";
 import { BastionConnection } from "./services/labd-connection.js";
 import { progressBus } from "./services/progress-events.js";
-import { checkInstallAllowed } from "./services/install-guard.js";
 import { ensureBootIso } from "./routes/boot-iso.js";
 
 function copyIfMissing(src: string, dest: string, label: string): void {
@@ -43,6 +40,125 @@ function download(url: string, dest: string, label: string): void {
   }
 }
 
+/**
+ * Pick the largest regular-file initrd from an `xorriso -lsl` listing.
+ *
+ * /live carries decoys: a 0-byte initrd.img placeholder on some images, or an
+ * initrd.img SYMLINK to the real version-suffixed file on others. Parsing is
+ * field-based (ls -l layout: perms links uid gid size month day time 'name')
+ * and considers only lines whose mode string marks a regular file — symlinks
+ * report their link size, not the target's, and must not win.
+ */
+export function pickLargestInitrd(
+  listing: string,
+): { name: string; size: number } | undefined {
+  let best: { name: string; size: number } | undefined;
+  for (const line of listing.split("\n")) {
+    if (!line.startsWith("-")) continue;          // regular files only
+    const quoted = /'([^']+)'/.exec(line);
+    const fields = line.trim().split(/\s+/);
+    const size = parseInt(fields[4] ?? "", 10);
+    const name = quoted?.[1] ?? "";
+    if (!name.startsWith("initrd")) continue;
+    if (!Number.isFinite(size) || size <= 0) continue;
+    if (best === undefined || size > best.size) {
+      best = { name, size };
+    }
+  }
+  return best;
+}
+
+const VYOS_NIGHTLY_RELEASES =
+  "https://api.github.com/repos/vyos/vyos-nightly-build/releases/latest";
+
+/**
+ * Resolve the configured VyOS ISO URL, expanding the "latest" sentinel.
+ *
+ * The nightly asset filename embeds a build date, so there is no stable
+ * "latest.iso" path to hardcode — the newest release has to be looked up.
+ * Any other value is used verbatim, which is how VYOS_ISO_URL pins a build
+ * or points at a locally mirrored copy.
+ */
+function resolveVyosIsoUrl(configured: string): string {
+  if (configured !== "latest") return configured;
+
+  const body = execSync(`curl -sSfL "${VYOS_NIGHTLY_RELEASES}"`, {
+    encoding: "utf-8",
+    stdio: ["pipe", "pipe", "pipe"],
+  });
+  const release = JSON.parse(body) as {
+    tag_name?: string;
+    assets?: Array<{ name: string; browser_download_url: string }>;
+  };
+
+  const asset = (release.assets ?? []).find((a) =>
+    /generic-amd64\.iso$/.test(a.name),
+  );
+  if (!asset) {
+    throw new Error(
+      `No generic-amd64 ISO asset in VyOS nightly release ${release.tag_name ?? "?"}`,
+    );
+  }
+
+  logger.info(`  VyOS ISO resolved to ${asset.name} (${release.tag_name ?? "?"})`);
+  return asset.browser_download_url;
+}
+
+/**
+ * Extract VyOS netboot artifacts from the release ISO.
+ *
+ * VyOS publishes no netboot bundle, so kernel/initrd/squashfs have to come out
+ * of the ISO. xorriso is already in the bastion image (used for boot.iso) and
+ * extracts without root or a loop mount.
+ *
+ * The initrd needs care: /live contains an empty initrd.img placeholder
+ * alongside the real one, which carries a version-suffixed name. Booting the
+ * 0-byte file fails with no useful diagnostic, so pick the largest initrd*.
+ */
+export function prepareVyosArtifacts(config: BastionConfig): void {
+  const kernel = `${config.httpDir}/vyos-vmlinuz`;
+  const initrd = `${config.httpDir}/vyos-initrd`;
+  const squashfs = `${config.httpDir}/vyos-filesystem.squashfs`;
+
+  if (existsSync(kernel) && existsSync(initrd) && existsSync(squashfs)) {
+    logger.info("  VyOS netboot artifacts -- cached");
+    return;
+  }
+
+  const iso = `${config.bastionDir}/vyos.iso`;
+  download(resolveVyosIsoUrl(config.vyosIsoUrl), iso, "VyOS ISO");
+
+  const extract = (isoPath: string, dest: string, label: string): void => {
+    execSync(
+      `xorriso -osirrox on -indev "${iso}" -extract "${isoPath}" "${dest}"`,
+      { stdio: "pipe" },
+    );
+    logger.info(`  ${label} -- extracted from ${isoPath}`);
+  };
+
+  extract("/live/vmlinuz", kernel, "VyOS kernel");
+  extract("/live/filesystem.squashfs", squashfs, "VyOS squashfs");
+
+  // Pick the real initrd by size from the ISO's own directory listing.
+  const listing = execSync(`xorriso -indev "${iso}" -lsl /live/ --`, {
+    encoding: "utf-8",
+    stdio: ["pipe", "pipe", "pipe"],
+  });
+
+  const best = pickLargestInitrd(listing);
+  if (best === undefined) {
+    throw new Error("No non-empty initrd found in /live on the VyOS ISO");
+  }
+  extract(`/live/${best.name}`, initrd, `VyOS initrd (${best.name}, ${best.size} bytes)`);
+
+  // The ISO is only needed to produce the three artifacts above.
+  try {
+    unlinkSync(iso);
+  } catch {
+    // Non-fatal: leaving it costs disk but nothing else.
+  }
+}
+
 function symlinkSafe(target: string, linkPath: string): void {
   try {
     symlinkSync(target, linkPath);
@@ -133,14 +249,9 @@ export async function startBastion(overrides: Partial<BastionConfig> = {}): Prom
   mkdirSync(config.tftpDir, { recursive: true });
   mkdirSync(config.httpDir, { recursive: true });
 
-  // Architectures we can actually network boot, reported in the banner so a missing
-  // arm64 payload is visible at startup instead of at 2am when a rescue is needed.
-  const bootArches: Arch[] = [];
-  let ipxeArm64Ready = false;
-
   // Prepare boot artifacts
   if (config.skipArtifacts !== true) {
-    logger.info(`Preparing boot artifacts (Fedora ${config.fedoraVersion}, ${SUPPORTED_ARCHES.join(" + ")})...`);
+    logger.info(`Preparing boot artifacts (Fedora ${config.fedoraVersion} ${config.arch})...`);
 
     copyIfMissing(
       "/usr/share/ipxe/undionly.kpxe",
@@ -158,41 +269,20 @@ export async function startBastion(overrides: Partial<BastionConfig> = {}): Prom
         `${config.tftpDir}/ipxe-arm64.efi`,
         "iPXE UEFI arm64",
       );
-      ipxeArm64Ready = true;
     } catch {
-      logger.warn("arm64 iPXE not available -- arm64 machines cannot network boot.");
-      logger.warn("  Install with: sudo dnf install ipxe-bootimgs-aarch64");
+      logger.warn("arm64 iPXE not available -- skipping");
     }
 
-    // Fedora pxeboot kernel + initrd per architecture. x86_64 keeps the unsuffixed
-    // names it has always used; other architectures are suffixed. The iPXE templates
-    // resolve the same paths via kernelPath()/initrdPath().
-    for (const arch of SUPPORTED_ARCHES) {
-      const mirror = fedoraMirrorFor(config.fedoraVersion, arch);
-      try {
-        download(
-          `${mirror}/images/pxeboot/vmlinuz`,
-          `${config.httpDir}${kernelPath(arch)}`,
-          `Fedora ${arch} kernel`,
-        );
-        download(
-          `${mirror}/images/pxeboot/initrd.img`,
-          `${config.httpDir}${initrdPath(arch)}`,
-          `Fedora ${arch} initrd`,
-        );
-        bootArches.push(arch);
-      } catch (err) {
-        // Non-fatal: a bastion with no arm64 artifacts still serves x86_64 fine.
-        // Failing startup over an unreachable mirror for an architecture that may not
-        // even be present on this network would be worse.
-        logger.warn(`Fedora ${arch} kernel/initrd unavailable -- ${arch} PXE disabled`);
-        logger.warn(`  ${err instanceof Error ? err.message : String(err)}`);
-      }
-    }
-
-    if (!bootArches.includes("x86_64")) {
-      throw new Error("Fedora x86_64 kernel/initrd could not be staged -- cannot serve PXE");
-    }
+    download(
+      `${config.fedoraMirror}/images/pxeboot/vmlinuz`,
+      `${config.httpDir}/vmlinuz`,
+      "Fedora kernel",
+    );
+    download(
+      `${config.fedoraMirror}/images/pxeboot/initrd.img`,
+      `${config.httpDir}/initrd.img`,
+      "Fedora initrd",
+    );
 
     // Ubuntu netboot artifacts (non-fatal — Ubuntu version may not be released yet)
     try {
@@ -211,6 +301,17 @@ export async function startBastion(overrides: Partial<BastionConfig> = {}): Prom
       logger.warn(`Ubuntu ${config.ubuntuVersion} artifacts not available -- Ubuntu provisioning disabled`);
     }
 
+    // VyOS netboot artifacts (non-fatal — same policy as Ubuntu)
+    try {
+      logger.info("Preparing VyOS netboot artifacts...");
+      prepareVyosArtifacts(config);
+    } catch (err) {
+      logger.warn(
+        `VyOS artifacts not available -- VyOS provisioning disabled ` +
+        `(${err instanceof Error ? err.message : String(err)})`,
+      );
+    }
+
     // Symlink iPXE binaries into HTTP dir for UEFI HTTP Boot
     for (const name of ["ipxe.efi", "ipxe-arm64.efi"]) {
       const src = `${config.tftpDir}/${name}`;
@@ -283,13 +384,6 @@ export async function startBastion(overrides: Partial<BastionConfig> = {}): Prom
     // Wire up command handlers so labd can send install/forget/role commands
     labdConn.onCommand("command-install", async (msg) => {
       if (msg.type !== "command-install") throw new Error("unexpected");
-      const installMac = msg.mac.toLowerCase().replace(/-/g, ":");
-      const osId = (msg.os as import("@lab/shared").OsId | undefined) ?? "fedora-43";
-      const check = checkInstallAllowed(state.load(), installMac, osId);
-      if (check.allowed === false) {
-        logger.warn(`INSTALL REFUSED: ${installMac} -- ${check.error}`);
-        return { status: "error", error: check.error };
-      }
       state.update((s) => {
         s.install_queue[msg.mac] = {
           hostname: msg.hostname,
@@ -297,6 +391,7 @@ export async function startBastion(overrides: Partial<BastionConfig> = {}): Prom
           role: msg.role as import("@lab/shared").Role,
           os: msg.os as import("@lab/shared").OsId,
           queued_at: new Date().toISOString(),
+          ...(msg.vyos ? { vyos: msg.vyos } : {}),
         };
       });
       return { status: "ok", data: { mac: msg.mac, hostname: msg.hostname } };
@@ -350,24 +445,13 @@ export async function startBastion(overrides: Partial<BastionConfig> = {}): Prom
       const mac = (msg.mac as string).toLowerCase();
       const now = new Date().toISOString();
       const existing = state.load().discovered[mac];
-      const identity = {
-        mac,
-        manufacturer: (msg.manufacturer as string) ?? "unknown",
-        product: (msg.product as string) ?? "unknown",
-        board: (msg.board as string) ?? "unknown",
-        ...(existing?.onboard !== undefined ? { onboard: existing.onboard } : {}),
-        ...(existing?.vendor_os !== undefined ? { vendor_os: existing.vendor_os } : {}),
-      };
-      const onboarding = classifyOnboard(identity);
-      const rootDevice = msg.root_device ?? existing?.root_device;
-      const rootArgs = msg.root_args ?? existing?.root_args;
       state.update((s) => {
         s.discovered[mac] = {
           mac,
-          product: identity.product,
-          board: identity.board,
+          product: (msg.product as string) ?? "unknown",
+          board: (msg.board as string) ?? "unknown",
           serial: (msg.serial as string) ?? "unknown",
-          manufacturer: identity.manufacturer,
+          manufacturer: (msg.manufacturer as string) ?? "unknown",
           cpu_model: (msg.cpu_model as string) ?? "unknown",
           cpu_cores: (msg.cpu_cores as number) ?? 0,
           memory_gb: (msg.memory_gb as number) ?? 0,
@@ -376,20 +460,7 @@ export async function startBastion(overrides: Partial<BastionConfig> = {}): Prom
           nics: (msg.nics as Array<{ name: string; mac: string; state: string }>) ?? [],
           first_seen: existing?.first_seen ?? now,
           last_seen: now,
-          onboard: onboarding.onboard,
-          ...(onboarding.vendor_os !== undefined ? { vendor_os: onboarding.vendor_os } : {}),
-          ...(rootDevice !== undefined ? { root_device: rootDevice } : {}),
-          ...(rootArgs !== undefined ? { root_args: rootArgs } : {}),
         };
-        // Keep the installed record in step -- the guard and --pxe-boot both read it.
-        const inst = s.installed[mac];
-        if (inst) {
-          inst.arch = (msg.arch as string) ?? inst.arch;
-          inst.onboard = onboarding.onboard;
-          if (onboarding.vendor_os !== undefined) inst.vendor_os = onboarding.vendor_os;
-          if (rootDevice !== undefined) inst.root_device = rootDevice;
-          if (rootArgs !== undefined) inst.root_args = rootArgs;
-        }
       });
       logger.info(`HARDWARE UPDATED: ${mac} -- ${msg.manufacturer ?? "?"} ${msg.product ?? "?"} (${msg.cpu_model ?? "?"}, ${msg.cpu_cores ?? "?"} cores, ${msg.memory_gb ?? "?"}GB RAM)`);
       return { status: "ok", data: { mac } };
@@ -424,7 +495,7 @@ export async function startBastion(overrides: Partial<BastionConfig> = {}): Prom
   }
 
   // Print banner
-  printBanner(config, bootArches, ipxeArm64Ready);
+  printBanner(config);
 
   // Graceful shutdown
   const shutdown = async (): Promise<void> => {
@@ -446,22 +517,11 @@ export async function startBastion(overrides: Partial<BastionConfig> = {}): Prom
   await new Promise(() => {});
 }
 
-function printBanner(config: BastionConfig, bootArches: Arch[], ipxeArm64Ready: boolean): void {
+function printBanner(config: BastionConfig): void {
   const dhcpInfo = config.dhcpMode === "full"
     ? `full (${config.dhcpRangeStart}-${config.dhcpRangeEnd})`
     : "proxy (alongside existing DHCP)";
 
-  // arm64 needs both an iPXE binary (DHCP hands it out on option 93 = 0x0b) and a
-  // kernel/initrd pair. Report the combination, since either missing breaks it.
-  const archInfo = config.skipArtifacts === true
-    ? "(artifacts skipped)"
-    : SUPPORTED_ARCHES
-        .map((a) => {
-          const ready = bootArches.includes(a) && (a !== "aarch64" || ipxeArm64Ready);
-          return ready ? a : `${a} (unavailable)`;
-        })
-        .join(", ");
-
   console.log("");
   console.log("\x1b[36m\x1b[1m" + "=".repeat(60) + "\x1b[0m");
   console.log("\x1b[36m\x1b[1m  Lab PXE Bastion -- Discovery Mode\x1b[0m");
@@ -470,8 +530,7 @@ function printBanner(config: BastionConfig, bootArches: Arch[], ipxeArm64Ready:
   console.log(`  Network:   \x1b[1m${config.network}/24\x1b[0m via \x1b[1m${config.iface}\x1b[0m`);
   console.log(`  DHCP:      \x1b[1m${dhcpInfo}\x1b[0m`);
   console.log(`  HTTP:      \x1b[1mhttp://${config.serverIp}:${config.httpPort}/\x1b[0m`);
-  console.log(`  OS:        \x1b[1mFedora ${config.fedoraVersion}\x1b[0m`);
-  console.log(`  Net boot:  \x1b[1m${archInfo}\x1b[0m`);
+  console.log(`  OS:        \x1b[1mFedora ${config.fedoraVersion} (${config.arch})\x1b[0m`);
   console.log(`  Domain:    \x1b[1m${config.domain}\x1b[0m`);
   console.log(`  State:     \x1b[1m${config.stateFile}\x1b[0m`);
   console.log("");

bastion/src/bastion/src/routes/api.ts

@@ -5,17 +5,23 @@
 // /api/discover  - receive hardware discovery reports from PXE-booted machines
 
 import type { FastifyInstance } from "fastify";
-import type { HardwareInfo, InstalledInfo, Role } from "@lab/shared";
-import { isValidOsId, SUPPORTED_ROLES, classifyOnboard } from "@lab/shared";
+import type { HardwareInfo, InstalledInfo, Role, VyosInstallSpec } from "@lab/shared";
+import { isValidOsId, SUPPORTED_ROLES, SUPPORTED_OS } from "@lab/shared";
 import type { StateManager } from "../services/state.js";
 import { logger } from "../services/logger.js";
 import { triggerPostProvisionK3s } from "../services/post-provision.js";
-import { checkInstallAllowed } from "../services/install-guard.js";
 import { progressBus } from "../services/progress-events.js";
 import type { ProgressEvent } from "../services/progress-events.js";
 import type { InstallLogBuffer } from "../services/install-log.js";
 import type { SyslogListener } from "../services/syslog-listener.js";
 
+/**
+ * Seconds after dispatch with zero progress before a machine is called stalled.
+ * Generous: the slowest legitimate gap is fetching a ~600MB VyOS squashfs over
+ * HTTP before the hook can report anything.
+ */
+const STALL_THRESHOLD_S = 8 * 60;
+
 export function registerApiRoutes(
   app: FastifyInstance,
   state: StateManager,
@@ -35,9 +41,10 @@ export function registerApiRoutes(
       disk?: string;
       role?: string;
       os?: string;
+      vyos?: VyosInstallSpec;
     };
   }>("/api/install", async (request, reply) => {
-    const { mac: rawMac, hostname, disk, role, os } = request.body ?? {};
+    const { mac: rawMac, hostname, disk, role, os, vyos } = request.body ?? {};
     const mac = (rawMac ?? "").toLowerCase().replace(/-/g, ":");
 
     if (mac === "") {
@@ -51,13 +58,7 @@ export function registerApiRoutes(
 
     const osId = os ?? "fedora-43";
     if (!isValidOsId(osId)) {
-      return reply.status(400).send({ error: `invalid os: '${osId}'. Supported: fedora-43, ubuntu-26.04` });
-    }
-
-    const check = checkInstallAllowed(state.load(), mac, osId);
-    if (check.allowed === false) {
-      logger.warn(`INSTALL REFUSED: ${mac} -- ${check.error}`);
-      return reply.status(409).send({ error: check.error });
+      return reply.status(400).send({ error: `invalid os: '${osId}'. Supported: ${SUPPORTED_OS.join(", ")}` });
     }
 
     state.update((s) => {
@@ -67,6 +68,7 @@ export function registerApiRoutes(
         role: validRole as Role,
         os: osId,
         queued_at: new Date().toISOString(),
+        ...(vyos ? { vyos } : {}),
       };
     });
 
@@ -165,11 +167,17 @@ export function registerApiRoutes(
           };
           s.installed[mac] = installedInfo;
 
-          const admin = installedInfo.role !== "vanilla" && installedInfo.role !== "" ? "lab" : "root";
+          // VyOS: the only login user is "vyos", and a router never runs k3s —
+          // without this guard a non-vanilla role + recorded IP would trigger
+          // the k3s post-provision against a VyOS box.
+          const isVyos = (installedInfo.os ?? "").startsWith("vyos");
+          const admin = isVyos
+            ? "vyos"
+            : installedInfo.role !== "vanilla" && installedInfo.role !== "" ? "lab" : "root";
           console.log(`\n  \x1b[0;32m\x1b[1m  ssh ${admin}@${ip}\x1b[0m\n`);  // eslint-disable-line no-console
 
           // Auto-install k3s for non-vanilla roles
-          if (installedInfo.role !== "vanilla" && ip !== "") {
+          if (!isVyos && installedInfo.role !== "vanilla" && ip !== "") {
             void triggerPostProvisionK3s(installedInfo.hostname, ip, installedInfo.role, admin, mac);
           }
         }
@@ -291,10 +299,6 @@ export function registerApiRoutes(
       arch?: string;
       disks?: Array<{ name: string; size_gb: number; model: string }>;
       nics?: Array<{ name: string; mac: string; state: string }>;
-      // Root filesystem, when the reporter could observe it (recheck over SSH, or the
-      // probe script run from a rescue shell). Used by --pxe-boot.
-      root_device?: string;
-      root_args?: string;
     };
   }>("/api/discover", async (request, reply) => {
     const data = request.body;
@@ -309,53 +313,22 @@ export function registerApiRoutes(
 
     state.update((s) => {
       const existing = s.discovered[mac];
-      // Classify onboarding from the DMI identity we just received. An explicit
-      // classification already on the record wins (see classifyOnboard).
-      const onboarding = classifyOnboard({
-        mac,
-        manufacturer: data.manufacturer ?? existing?.manufacturer ?? "unknown",
-        product: data.product ?? existing?.product ?? "unknown",
-        board: data.board ?? existing?.board ?? "unknown",
-        ...(existing?.onboard !== undefined ? { onboard: existing.onboard } : {}),
-        ...(existing?.vendor_os !== undefined ? { vendor_os: existing.vendor_os } : {}),
-      });
-      const rootDevice = data.root_device ?? existing?.root_device;
-      const rootArgs = data.root_args ?? existing?.root_args;
-
-      // Absent fields keep whatever we already knew. Reporters are not all the full
-      // discovery kickstart: the rescue-shell probe posts only a root device, and
-      // blanking a machine's hardware inventory as a side effect of that would be
-      // silent data loss.
       const hwInfo: HardwareInfo = {
         mac,
-        product: data.product ?? existing?.product ?? "unknown",
-        board: data.board ?? existing?.board ?? "unknown",
-        serial: data.serial ?? existing?.serial ?? "unknown",
-        manufacturer: data.manufacturer ?? existing?.manufacturer ?? "unknown",
-        cpu_model: data.cpu_model ?? existing?.cpu_model ?? "unknown",
-        cpu_cores: data.cpu_cores ?? existing?.cpu_cores ?? 0,
-        memory_gb: data.memory_gb ?? existing?.memory_gb ?? 0,
-        arch: data.arch ?? existing?.arch ?? "unknown",
-        disks: data.disks ?? existing?.disks ?? [],
-        nics: data.nics ?? existing?.nics ?? [],
+        product: data.product ?? "unknown",
+        board: data.board ?? "unknown",
+        serial: data.serial ?? "unknown",
+        manufacturer: data.manufacturer ?? "unknown",
+        cpu_model: data.cpu_model ?? "unknown",
+        cpu_cores: data.cpu_cores ?? 0,
+        memory_gb: data.memory_gb ?? 0,
+        arch: data.arch ?? "unknown",
+        disks: data.disks ?? [],
+        nics: data.nics ?? [],
         first_seen: existing?.first_seen ?? now,
         last_seen: now,
-        onboard: onboarding.onboard,
-        ...(onboarding.vendor_os !== undefined ? { vendor_os: onboarding.vendor_os } : {}),
-        ...(rootDevice !== undefined ? { root_device: rootDevice } : {}),
-        ...(rootArgs !== undefined ? { root_args: rootArgs } : {}),
       };
       s.discovered[mac] = hwInfo;
-
-      // Keep the installed record in step -- the install guard and --pxe-boot read it.
-      const inst = s.installed[mac];
-      if (inst) {
-        if (data.arch !== undefined) inst.arch = data.arch;
-        inst.onboard = onboarding.onboard;
-        if (onboarding.vendor_os !== undefined) inst.vendor_os = onboarding.vendor_os;
-        if (rootDevice !== undefined) inst.root_device = rootDevice;
-        if (rootArgs !== undefined) inst.root_args = rootArgs;
-      }
     });
 
     const label = isNew ? "NEW MACHINE DISCOVERED" : "MACHINE RE-DISCOVERED";
@@ -476,6 +449,15 @@ export function registerApiRoutes(
     const installedEntry = currentState.installed[mac];
 
     if (queueEntry) {
+      // A machine that was handed an install script but has reported nothing
+      // since is wedged BEFORE the installer environment came up — a bad
+      // kernel/initrd, no network in the initramfs, or the wrong NIC picked.
+      // Surfacing it here is what makes that diagnosable without a console.
+      const since = queueEntry.progress_at ?? queueEntry.dispatched_at;
+      const stalledForS = since !== undefined && queueEntry.progress === undefined
+        ? Math.floor((Date.now() - new Date(since).getTime()) / 1000)
+        : 0;
+
       return reply.send({
         mac,
         hostname: queueEntry.hostname,
@@ -483,6 +465,9 @@ export function registerApiRoutes(
         progress: queueEntry.progress ?? "queued",
         progress_detail: queueEntry.progress_detail ?? "",
         progress_at: queueEntry.progress_at ?? queueEntry.queued_at,
+        dispatched_at: queueEntry.dispatched_at,
+        stalled_for_s: stalledForS,
+        stalled: stalledForS > STALL_THRESHOLD_S,
         role: queueEntry.role,
         os: queueEntry.os,
         stages: queueEntry.log ?? [],

bastion/src/bastion/src/routes/dispatch.ts

@@ -5,8 +5,7 @@
 //   - unknown       -> discovery mode (collect hardware, POST to bastion)
 
 import type { FastifyInstance } from "fastify";
-import type { Arch, BastionConfig, BastionState, OsId } from "@lab/shared";
-import { normalizeArch, fedoraMirrorFor, osSupportsArch } from "@lab/shared";
+import type { BastionConfig } from "@lab/shared";
 import type { StateManager } from "../services/state.js";
 import {
   renderDiscoverIpxe,
@@ -14,51 +13,12 @@ import {
   renderDebugIpxe,
   renderPxeBootDebugIpxe,
   renderLocalBootIpxe,
-  renderUnsupportedIpxe,
 } from "../templates/boot.ipxe.js";
 import { renderUbuntuInstallIpxe } from "../templates/ubuntu-boot.ipxe.js";
+import { renderVyosInstallIpxe } from "../templates/vyos-boot.ipxe.js";
 import { renderDebugKickstart } from "../templates/debug.ks.js";
 import { logger } from "../services/logger.js";
 
-/**
- * Resolve a booting machine's architecture.
- *
- * Order matters. The tracked record is what we actually observed on the machine, so it
- * wins. `reported` is iPXE's ${buildarch}, which is only as good as the binary DHCP
- * handed the client -- correct in practice, but a misconfigured option 93 mapping would
- * make it lie. The configured default is the last resort.
- *
- * There is deliberately no operator-supplied architecture anywhere in this path.
- */
-export function resolveArch(
-  state: BastionState,
-  mac: string,
-  reported: string | undefined,
-  config: BastionConfig,
-): Arch {
-  return normalizeArch(state.installed[mac]?.arch)
-    ?? normalizeArch(state.install_queue[mac]?.arch)
-    ?? normalizeArch(state.discovered[mac]?.arch)
-    ?? normalizeArch(reported)
-    ?? normalizeArch(config.arch)
-    ?? "x86_64";
-}
-
-/** The root filesystem to boot for --pxe-boot, if the machine's record carries one. */
-function resolveRoot(
-  state: BastionState,
-  mac: string,
-): { rootDevice: string; rootArgs?: string } | null {
-  const installed = state.installed[mac];
-  const discovered = state.discovered[mac];
-  const rootDevice = installed?.root_device ?? discovered?.root_device;
-  if (rootDevice === undefined || rootDevice === "") return null;
-  const rootArgs = installed?.root_args ?? discovered?.root_args;
-  return rootArgs !== undefined && rootArgs !== ""
-    ? { rootDevice, rootArgs }
-    : { rootDevice };
-}
-
 export function registerDispatchRoutes(
   app: FastifyInstance,
   config: BastionConfig,
@@ -93,68 +53,18 @@ curl -sf -X POST "http://${config.serverIp}:${config.httpPort}/api/progress" \\
     -H "Content-Type: application/json" \\
     -d "{\\"mac\\":\\"$MAC_ADDR\\",\\"stage\\":\\"debug-ready\\",\\"detail\\":\\"nc $IP_ADDR 2323\\"}" 2>/dev/null || true
 
-# --- Find the installed root filesystem and report it ---
-# This is what 'labctl provision debug --pxe-boot' needs. The rescue image cannot
-# report it by itself: %pre/%post do not run in rescue mode, so it happens here.
-vgchange -ay >/dev/null 2>&1 || true
-
-ROOT_DEVICE=""
-ROOT_ARGS=""
-PROBE_MNT=/tmp/lab-rootprobe
-mkdir -p "$PROBE_MNT"
-
-# Candidates: every LVM logical volume plus every non-LVM partition with a filesystem.
-for CAND in $(lvs --noheadings -o lv_path 2>/dev/null) \\
-            $(blkid -o device 2>/dev/null | grep -v '^/dev/mapper/'); do
-    [ -b "$CAND" ] || continue
-    mount -o ro "$CAND" "$PROBE_MNT" >/dev/null 2>&1 || continue
-    # A root filesystem has both of these; /boot and /home do not.
-    if [ -f "$PROBE_MNT/etc/fstab" ] && [ -d "$PROBE_MNT/usr" ]; then
-        ROOT_DEVICE="$CAND"
-        PRETTY=$(. "$PROBE_MNT/etc/os-release" 2>/dev/null && echo "$PRETTY_NAME")
-        echo "  found root: $CAND \${PRETTY:+($PRETTY)}"
-        if [ "$(lsblk -no TYPE "$CAND" 2>/dev/null | head -1)" = "lvm" ]; then
-            VGLV=$(lvs --noheadings -o vg_name,lv_name "$CAND" 2>/dev/null | awk '{print $1"/"$2}')
-            [ -n "$VGLV" ] && ROOT_ARGS="rd.lvm.lv=$VGLV"
-            # Swap comes from fstab here — /proc/swaps is the rescue image's, not the host's.
-            SWLV=$(awk '$3=="swap" && $1 ~ /^\\/dev\\// {print $1; exit}' "$PROBE_MNT/etc/fstab" 2>/dev/null)
-            if [ -n "$SWLV" ]; then
-                SWVGLV=$(lvs --noheadings -o vg_name,lv_name "$SWLV" 2>/dev/null | awk '{print $1"/"$2}')
-                [ -n "$SWVGLV" ] && [ "$SWVGLV" != "$VGLV" ] && ROOT_ARGS="$ROOT_ARGS rd.lvm.lv=$SWVGLV"
-            fi
-        fi
-        umount "$PROBE_MNT" >/dev/null 2>&1 || true
-        break
-    fi
-    umount "$PROBE_MNT" >/dev/null 2>&1 || true
-done
-
-if [ -n "$ROOT_DEVICE" ]; then
-    curl -sf -X POST "http://${config.serverIp}:${config.httpPort}/api/discover" \\
-        -H "Content-Type: application/json" \\
-        -d "{\\"mac\\":\\"$MAC_ADDR\\",\\"root_device\\":\\"$ROOT_DEVICE\\",\\"root_args\\":\\"$ROOT_ARGS\\"}" 2>/dev/null \\
-        && echo "  reported to bastion — 'labctl provision debug --pxe-boot' will work now"
-else
-    echo "  no root filesystem found — --pxe-boot cannot be used on this machine"
-fi
-
 echo ""
 echo "=== Debug environment ready ==="
 echo "  nc $IP_ADDR 2323    (remote shell)"
 echo "  ssh root@$IP_ADDR   (password: debug)"
-if [ -n "$ROOT_DEVICE" ]; then
-    echo "  root: $ROOT_DEVICE $ROOT_ARGS"
-fi
 echo "==============================="
 `;
     return reply.type("text/plain").send(script);
   });
 
-  app.get<{ Querystring: { mac?: string; arch?: string } }>("/dispatch", async (request, reply) => {
+  app.get<{ Querystring: { mac?: string } }>("/dispatch", async (request, reply) => {
     const mac = (request.query.mac ?? "").toLowerCase().replace(/-/g, ":");
     const currentState = state.load();
-    const arch = resolveArch(currentState, mac, request.query.arch, config);
-    const fedoraMirror = fedoraMirrorFor(config.fedoraVersion, arch);
 
     // Debug mode takes highest priority — auto-clear after serving once
     const debugEntry = currentState.debug[mac];
@@ -163,48 +73,22 @@ echo "==============================="
       state.update((s) => { delete s.debug[mac]; });
 
       let script: string;
-      const wantsPxeBoot = debugEntry.pxeBoot === true;
-      const root = wantsPxeBoot ? resolveRoot(currentState, mac) : null;
-
-      if (root !== null) {
-        logger.info(`PXE BOOT DEBUG: ${mac} -> ${hostname} (${arch}, root=${root.rootDevice})`);
+      if (debugEntry.pxeBoot) {
+        logger.info(`PXE BOOT DEBUG: ${mac} -> ${hostname} (kernel+initrd from PXE, root from NVMe)`);
         script = renderPxeBootDebugIpxe({
           mac,
           hostname,
           serverIp: config.serverIp,
           httpPort: config.httpPort,
-          arch,
-          ...root,
         });
       } else {
-        // --pxe-boot without a known root device falls back to rescue rather than
-        // guessing. A wrong root= leaves the machine unbootable, and rescue is where
-        // the operator can find the real one (curl /debug-setup.sh reports it back).
-        const notice = wantsPxeBoot
-          ? [
-              "",
-              "NOTE: --pxe-boot requested, but no root device is recorded",
-              "      for this machine. Booting rescue instead.",
-              "      From the rescue shell, run:",
-              // No pipe or && here: iPXE treats || and && as command separators, so keep
-              // the printed command free of anything its parser might claim.
-              `        curl -s http://${config.serverIp}:${config.httpPort}/debug-setup.sh -o /tmp/s.sh ; sh /tmp/s.sh`,
-              "      then retry --pxe-boot.",
-            ]
-          : undefined;
-        if (wantsPxeBoot) {
-          logger.warn(`PXE BOOT DEBUG: ${mac} -> ${hostname} has no recorded root device -- serving rescue instead`);
-        } else {
-          logger.info(`DEBUG BOOT: ${mac} -> ${hostname} (${arch}, rescue mode)`);
-        }
+        logger.info(`DEBUG BOOT: ${mac} -> ${hostname} (rescue mode)`);
         script = renderDebugIpxe({
           mac,
           hostname,
           serverIp: config.serverIp,
           httpPort: config.httpPort,
-          fedoraMirror,
-          arch,
-          ...(notice ? { notice } : {}),
+          fedoraMirror: config.fedoraMirror,
         });
       }
       return reply.type("text/plain").send(script);
@@ -214,24 +98,24 @@ echo "==============================="
     if (queueEntry) {
       const hostname = queueEntry.hostname ?? "lab-node";
       const os = queueEntry.os ?? "fedora-43";
-      logger.info(`INSTALL STARTED: ${mac} -> ${hostname} (${os}, ${arch})`);
+      logger.info(`INSTALL STARTED: ${mac} -> ${hostname} (${os})`);
+
+      // Stamp the handoff so a machine that boots the installer but never
+      // reports can be spotted without a console.
+      state.update((s) => {
+        const entry = s.install_queue[mac];
+        if (entry) entry.dispatched_at = new Date().toISOString();
+      });
 
       let script: string;
-      if (os.startsWith("ubuntu")) {
-        // Last line of defence. The install guard refuses this combination when the
-        // machine's architecture is already known, but a machine queued before it was
-        // discovered can reach here. Serving the x86-only Ubuntu kernel to an arm64
-        // client is precisely the bug this work exists to fix, so stop instead.
-        if (!osSupportsArch(os as OsId, arch)) {
-          logger.error(`INSTALL BLOCKED: ${mac} -> ${hostname} -- ${os} has no ${arch} artifacts`);
-          script = renderUnsupportedIpxe({
-            hostname,
-            mac,
-            reason: `${os} publishes no ${arch} netboot artifacts`,
-            action: `labctl provision install ${mac} ${hostname} --os fedora-43`,
-          });
-          return reply.type("text/plain").send(script);
-        }
+      if (os.startsWith("vyos")) {
+        script = renderVyosInstallIpxe({
+          mac,
+          hostname,
+          serverIp: config.serverIp,
+          httpPort: config.httpPort,
+        });
+      } else if (os.startsWith("ubuntu")) {
         script = renderUbuntuInstallIpxe({
           mac,
           hostname,
@@ -246,8 +130,7 @@ echo "==============================="
           serverIp: config.serverIp,
           httpPort: config.httpPort,
           fedoraVersion: config.fedoraVersion,
-          fedoraMirror,
-          arch,
+          fedoraMirror: config.fedoraMirror,
         });
       }
 
@@ -264,14 +147,13 @@ echo "==============================="
     }
 
     // Unknown MAC -> discovery mode
-    logger.info(`PXE request from ${mac} (${arch}) -> discovery mode`);
+    logger.info(`PXE request from ${mac} -> discovery mode`);
 
     const script = renderDiscoverIpxe({
       mac,
       serverIp: config.serverIp,
       httpPort: config.httpPort,
-      fedoraMirror,
-      arch,
+      fedoraMirror: config.fedoraMirror,
     });
 
     return reply.type("text/plain").send(script);

bastion/src/bastion/src/routes/vyos.ts

@@ -0,0 +1,71 @@
+// VyOS network install routes.
+//
+// VyOS has no unattended installer, so the automation is injected via
+// live-config's `hooks` component: the iPXE script passes
+// live-config.hooks=<.../vyos/autoinstall.sh>, live-config wgets it and runs it
+// as root, and that script fetches and executes the generated install driver.
+
+import type { FastifyInstance } from "fastify";
+import type { BastionConfig } from "@lab/shared";
+import type { StateManager } from "../services/state.js";
+import { buildVyosConfigSpec } from "../templates/vyos-config-spec.js";
+import { renderVyosInstallPy } from "../templates/vyos-install.py.js";
+import { logger } from "../services/logger.js";
+
+function normalizeMac(value: string | undefined): string {
+  return (value ?? "").toLowerCase().replace(/-/g, ":");
+}
+
+export function registerVyosRoutes(
+  app: FastifyInstance,
+  config: BastionConfig,
+  state: StateManager,
+): void {
+  // live-config hook. Kept minimal: everything version-specific lives in the
+  // generated Python. wget is guaranteed present -- live-config used it to
+  // fetch this very script.
+  app.get<{ Querystring: { mac?: string } }>("/vyos/autoinstall.sh", async (request, reply) => {
+    const mac = normalizeMac(request.query.mac);
+    const base = `http://${config.serverIp}:${config.httpPort}`;
+
+    logger.info(`VYOS AUTOINSTALL HOOK served to ${mac || "unknown MAC"}`);
+
+    const script = `#!/bin/sh
+# Lab PXE Bastion -- VyOS unattended install hook (run by live-config as root)
+set -eu
+
+wget -q "${base}/vyos/install.py?mac=${mac}" -O /tmp/vyos-install.py
+exec python3 /tmp/vyos-install.py
+`;
+    return reply.type("text/plain").send(script);
+  });
+
+  // Per-MAC install driver, with the machine's config spec baked in.
+  app.get<{ Querystring: { mac?: string } }>("/vyos/install.py", async (request, reply) => {
+    const mac = normalizeMac(request.query.mac);
+    const queueEntry = state.load().install_queue[mac];
+
+    const spec = buildVyosConfigSpec({
+      hostname: queueEntry?.hostname ?? "vyos",
+      spec: queueEntry?.vyos,
+      defaultPassword: config.vyosDefaultPassword,
+      sshKeys: config.sshKeys,
+      disk: queueEntry?.disk,
+    });
+
+    logger.info(
+      `VYOS INSTALL DRIVER served to ${mac} (${spec.hostname}, ` +
+      `${spec.sets.length} config ops, disk="${spec.disk || "auto"}")`,
+    );
+
+    const script = renderVyosInstallPy({
+      spec,
+      mac,
+      serverIp: config.serverIp,
+      httpPort: config.httpPort,
+      role: queueEntry?.role ?? "vanilla",
+    });
+
+    return reply.type("text/plain").send(script);
+  });
+}

bastion/src/bastion/src/server.ts

@@ -12,6 +12,7 @@ import { registerDispatchRoutes } from "./routes/dispatch.js";
 import { registerKickstartRoutes } from "./routes/kickstart.js";
 import { registerApiRoutes } from "./routes/api.js";
 import { registerAsahiRoutes } from "./routes/asahi.js";
+import { registerVyosRoutes } from "./routes/vyos.js";
 
 
 export function createApp(config: BastionConfig): { app: ReturnType<typeof Fastify>; state: StateManager; installLog: InstallLogBuffer; syslog: SyslogListener } {
@@ -47,6 +48,7 @@ export function createApp(config: BastionConfig): { app: ReturnType<typeof Fasti
   registerKickstartRoutes(app, config, state, syslog);
   registerApiRoutes(app, state, installLog, syslog);
   registerAsahiRoutes(app, config);
+  registerVyosRoutes(app, config, state);
   // boot.iso is generated at startup and served as a static file from httpDir
   // (static serving supports HTTP Range requests, required by JetKVM streaming)
 

bastion/src/bastion/src/services/install-guard.ts

@@ -1,94 +0,0 @@
-// Pre-flight checks for queuing an OS install.
-//
-// Both entry points -- the HTTP /api/install route and the labd command-install handler
-// -- run this, so `labctl provision install` and `provision reprovision` are covered
-// whichever way the request arrives.
-//
-// Rescue/debug is deliberately NOT guarded. Being unable to reinstall a machine is
-// exactly when you most need to boot it into a rescue shell.
-
-import type { Arch, BastionState, OsId } from "@lab/shared";
-import { classifyOnboard, normalizeArch, osSupportsArch, vendorOsDescription, archesForOs } from "@lab/shared";
-
-export type InstallCheck =
-  | { allowed: true }
-  | { allowed: false; error: string };
-
-interface MachineIdentity {
-  hostname: string;
-  arch: Arch | undefined;
-  identity: Parameters<typeof classifyOnboard>[0];
-}
-
-/** Best-known identity for a MAC, merged across the three state maps. */
-function identify(state: BastionState, mac: string): MachineIdentity {
-  const discovered = state.discovered[mac];
-  const installed = state.installed[mac];
-  const queued = state.install_queue[mac];
-
-  const manufacturer = discovered?.manufacturer ?? installed?.manufacturer;
-  const product = discovered?.product ?? installed?.product;
-  const board = discovered?.board;
-  const onboard = installed?.onboard ?? discovered?.onboard;
-  const vendorOs = installed?.vendor_os ?? discovered?.vendor_os;
-
-  return {
-    hostname: installed?.hostname ?? queued?.hostname ?? discovered?.product ?? mac,
-    arch: normalizeArch(installed?.arch ?? queued?.arch ?? discovered?.arch),
-    identity: {
-      mac,
-      ...(manufacturer !== undefined ? { manufacturer } : {}),
-      ...(product !== undefined ? { product } : {}),
-      ...(board !== undefined ? { board } : {}),
-      ...(onboard !== undefined ? { onboard } : {}),
-      ...(vendorOs !== undefined ? { vendor_os: vendorOs } : {}),
-    },
-  };
-}
-
-/**
- * Decide whether `mac` may be queued for an install of `os`.
- *
- * Refusals name the machine and the reason, and point at the action that is available
- * instead. An operator hitting this at 2am should not have to read the source to work
- * out what happened.
- */
-export function checkInstallAllowed(
-  state: BastionState,
-  mac: string,
-  os: OsId,
-): InstallCheck {
-  const machine = identify(state, mac);
-  const { onboard, vendor_os } = classifyOnboard(machine.identity);
-
-  // 1. Machines running a vendor OS we cannot rebuild.
-  if (onboard === "ssh") {
-    const what = vendorOsDescription(vendor_os);
-    return {
-      allowed: false,
-      error:
-        `Refusing to install ${machine.hostname} (${mac}): it runs ${what}. ` +
-        `No image in our pipeline can restore it, so installing ${os} would destroy that ` +
-        `driver and firmware stack permanently. This machine is SSH-onboard: we manage its ` +
-        `userspace, not its OS. ` +
-        `To boot it into a rescue shell instead, run: labctl provision debug ${machine.hostname}`,
-      // TODO: when a DGX OS / SparkOS image joins the pipeline, an install targeting a
-      // machine whose vendor_os matches that image should be allowed through here.
-    };
-  }
-
-  // 2. Architecture the OS has no netboot artifacts for.
-  if (machine.arch !== undefined && !osSupportsArch(os, machine.arch)) {
-    const supported = archesForOs(os);
-    return {
-      allowed: false,
-      error:
-        `Refusing to install ${os} on ${machine.hostname} (${mac}): ` +
-        `${os} has no ${machine.arch} netboot artifacts` +
-        (supported.length > 0 ? ` (only ${supported.join(", ")})` : "") +
-        `. Use an OS that supports ${machine.arch}.`,
-    };
-  }
-
-  return { allowed: true };
-}

bastion/src/bastion/src/templates/boot.ipxe.ts

@@ -1,55 +1,4 @@
 // iPXE boot script templates for dispatch routing.
-//
-// Architecture handling: the bastion serves one kernel/initrd pair per architecture.
-// x86_64 keeps the original unsuffixed paths so its output is unchanged; every other
-// architecture gets an arch-suffixed pair. See stageBootArtifacts() in main.ts for the
-// matching staging side, and boot-iso.ts for the same scheme on the ISO path.
-
-import type { Arch } from "@lab/shared";
-
-/** Kernel/initrd URL paths, keyed by architecture. */
-export function kernelPath(arch: Arch): string {
-  return arch === "x86_64" ? "/vmlinuz" : `/vmlinuz-${arch}`;
-}
-
-export function initrdPath(arch: Arch): string {
-  return arch === "x86_64" ? "/initrd.img" : `/initrd-${arch}.img`;
-}
-
-/**
- * Console arguments per architecture.
- *
- * arm64 has no VGA text console: a headless machine only talks over the SoC UART, so
- * ttyAMA0 must be listed as well. The last console= wins for /dev/console, so serial
- * is the interactive one while tty0 still receives boot output on machines with a
- * display attached.
- */
-const CONSOLE_ARGS: Record<Arch, string> = {
-  x86_64: "console=tty0",
-  aarch64: "console=tty0 console=ttyAMA0,115200",
-};
-
-/**
- * Anaconda arguments for the graphical-suppression / console setup.
- *
- * `nomodeset` disables kernel mode setting, which on x86 forces the generic VGA path
- * and makes flaky GPU drivers survive the installer. On arm64 it does not mean the
- * same thing -- there is no VGA fallback to drop back to, and it can leave the machine
- * with no usable console at all -- so arm64 gets explicit console arguments instead.
- */
-function installerArgs(arch: Arch): string {
-  return arch === "x86_64" ? "inst.text nomodeset" : `inst.text ${CONSOLE_ARGS[arch]}`;
-}
-
-/** Extra console arguments appended to templates that don't already set them. */
-function extraConsoleArgs(arch: Arch): string {
-  return arch === "x86_64" ? "" : ` ${CONSOLE_ARGS[arch]}`;
-}
-
-/** Join kernel arguments, dropping empties so callers can pass optional groups. */
-function joinArgs(...parts: Array<string | undefined>): string {
-  return parts.filter((p) => p !== undefined && p !== "").join(" ");
-}
 
 export interface BootIpxeParams {
   serverIp: string;
@@ -59,11 +8,6 @@ export interface BootIpxeParams {
 /**
  * Initial iPXE boot script that chains to the dispatch endpoint.
  * This is what dnsmasq serves to iPXE clients via HTTP.
- *
- * `${buildarch}` is iPXE's own build architecture ("x86_64" or "arm64"), which is the
- * one architecture signal available on every path -- network PXE, UEFI HTTP boot and
- * the boot ISO alike. DHCP option 93 only reaches dnsmasq, never this HTTP endpoint.
- * dispatch prefers the tracked machine record and falls back to this.
  */
 export function renderBootIpxe(params: BootIpxeParams): string {
   return `#!ipxe
@@ -75,7 +19,7 @@ echo   Contacting server for instructions...
 echo ============================================
 echo
 
-chain http://${params.serverIp}:${params.httpPort}/dispatch?mac=\${net0/mac}&arch=\${buildarch}
+chain http://${params.serverIp}:${params.httpPort}/dispatch?mac=\${net0/mac}
 `;
 }
 
@@ -87,9 +31,7 @@ export function renderDiscoverIpxe(params: {
   serverIp: string;
   httpPort: number;
   fedoraMirror: string;
-  arch: Arch;
 }): string {
-  const base = `http://${params.serverIp}:${params.httpPort}`;
   return `#!ipxe
 
 echo
@@ -100,8 +42,8 @@ echo   Collecting hardware info...
 echo =============================================
 echo
 
-kernel ${base}${kernelPath(params.arch)} inst.ks=${base}/discover.ks inst.stage2=${params.fedoraMirror} ${installerArgs(params.arch)}
-initrd ${base}${initrdPath(params.arch)}
+kernel http://${params.serverIp}:${params.httpPort}/vmlinuz inst.ks=http://${params.serverIp}:${params.httpPort}/discover.ks inst.stage2=${params.fedoraMirror} inst.text nomodeset
+initrd http://${params.serverIp}:${params.httpPort}/initrd.img
 boot
 `;
 }
@@ -116,9 +58,7 @@ export function renderInstallIpxe(params: {
   httpPort: number;
   fedoraVersion: string;
   fedoraMirror: string;
-  arch: Arch;
 }): string {
-  const base = `http://${params.serverIp}:${params.httpPort}`;
   return `#!ipxe
 
 echo
@@ -129,8 +69,8 @@ echo   MAC:    ${params.mac}
 echo =============================================
 echo
 
-kernel ${base}${kernelPath(params.arch)} inst.ks=${base}/ks?mac=${params.mac} inst.repo=${params.fedoraMirror} ${installerArgs(params.arch)}
-initrd ${base}${initrdPath(params.arch)}
+kernel http://${params.serverIp}:${params.httpPort}/vmlinuz inst.ks=http://${params.serverIp}:${params.httpPort}/ks?mac=${params.mac} inst.repo=${params.fedoraMirror} inst.text nomodeset
+initrd http://${params.serverIp}:${params.httpPort}/initrd.img
 boot
 `;
 }
@@ -138,9 +78,6 @@ boot
 /**
  * iPXE script for debug/rescue mode -- boots Fedora installer in rescue mode.
  * Provides a shell with LVM tools, network, and SSH for inspecting installed systems.
- *
- * `notice` is shown before the boot line. dispatch uses it to explain why a requested
- * --pxe-boot fell back to rescue.
  */
 export function renderDebugIpxe(params: {
   mac: string;
@@ -148,11 +85,7 @@ export function renderDebugIpxe(params: {
   serverIp: string;
   httpPort: number;
   fedoraMirror: string;
-  arch: Arch;
-  notice?: string[];
 }): string {
-  const base = `http://${params.serverIp}:${params.httpPort}`;
-  const notice = (params.notice ?? []).map((line) => `echo   ${line}\n`).join("");
   return `#!ipxe
 
 echo
@@ -160,11 +93,11 @@ echo =============================================
 echo   Lab PXE Bastion - DEBUG/RESCUE MODE
 echo   Target: ${params.hostname}
 echo   MAC:    ${params.mac}
-${notice}echo =============================================
+echo =============================================
 echo
 
-kernel ${base}${kernelPath(params.arch)} inst.rescue inst.text inst.sshd inst.ks=${base}/debug.ks?mac=${params.mac} inst.stage2=${params.fedoraMirror}${extraConsoleArgs(params.arch)}
-initrd ${base}${initrdPath(params.arch)}
+kernel http://${params.serverIp}:${params.httpPort}/vmlinuz inst.rescue inst.text inst.sshd inst.ks=http://${params.serverIp}:${params.httpPort}/debug.ks?mac=${params.mac} inst.stage2=${params.fedoraMirror}
+initrd http://${params.serverIp}:${params.httpPort}/initrd.img
 boot
 `;
 }
@@ -173,28 +106,13 @@ boot
  * iPXE script for PXE-boot debug mode -- boots the installed system's root
  * filesystem using the bastion's PXE kernel+initrd instead of local GRUB.
  * Workaround for UEFI firmware bugs that make local disk boot slow.
- *
- * rootDevice/rootArgs come from the machine's record -- they are not assumed. Our
- * Fedora installs use an LVM layout, but nothing guarantees any given machine does,
- * and a wrong root= here means an unbootable machine. dispatch refuses to render this
- * script without them.
  */
 export function renderPxeBootDebugIpxe(params: {
   mac: string;
   hostname: string;
   serverIp: string;
   httpPort: number;
-  arch: Arch;
-  rootDevice: string;
-  rootArgs?: string;
 }): string {
-  const base = `http://${params.serverIp}:${params.httpPort}`;
-  const cmdline = joinArgs(
-    `root=${params.rootDevice}`,
-    "ro",
-    params.rootArgs,
-    CONSOLE_ARGS[params.arch],
-  );
   return `#!ipxe
 
 echo
@@ -206,40 +124,12 @@ echo   Kernel+initrd from PXE, root from NVMe
 echo =============================================
 echo
 
-kernel ${base}${kernelPath(params.arch)} ${cmdline}
-initrd ${base}${initrdPath(params.arch)}
+kernel http://${params.serverIp}:${params.httpPort}/vmlinuz root=/dev/mapper/labvg-root ro rd.lvm.lv=labvg/root rd.lvm.lv=labvg/swap console=tty0
+initrd http://${params.serverIp}:${params.httpPort}/initrd.img
 boot
 `;
 }
 
-/**
- * iPXE script for a request we refuse to serve.
- *
- * Better a machine that stops with a legible reason on its console than one handed a
- * kernel it cannot execute, which fails much later and much less clearly.
- */
-export function renderUnsupportedIpxe(params: {
-  mac: string;
-  hostname: string;
-  reason: string;
-  action?: string;
-}): string {
-  return `#!ipxe
-
-echo
-echo =============================================
-echo   Lab PXE Bastion - CANNOT BOOT THIS MACHINE
-echo   Target: ${params.hostname}
-echo   MAC:    ${params.mac}
-echo
-echo   ${params.reason}
-${params.action !== undefined ? `echo\necho   Try: ${params.action}\n` : ""}echo =============================================
-echo
-sleep 10
-exit 1
-`;
-}
-
 /**
  * iPXE script for already-installed machines -- exits to boot from local disk.
  */

bastion/src/bastion/src/templates/dnsmasq.conf.ts

@@ -48,20 +48,15 @@ enable-tftp
 tftp-root=${tftpDir}
 tftp-no-blocksize
 
-# Detect client architecture -- PXE (TFTP) clients.
-# Values are DHCP option 93 (Client System Architecture), IANA "Processor Architecture
-# Types". Getting these wrong means the machine is handed a bootloader its firmware
-# cannot execute, and it loops or hangs with no console output.
+# Detect client architecture -- PXE (TFTP) clients
 dhcp-match=set:bios,option:client-arch,0
 dhcp-match=set:efi-x86_64,option:client-arch,7
 dhcp-match=set:efi-x86_64,option:client-arch,9
 dhcp-match=set:efi-arm64,option:client-arch,11
 
-# Detect client architecture -- UEFI HTTP Boot clients (no TFTP size limit).
-# 16 = x64 uefi boot from http, 19 = arm uefi 64 boot from http.
-# (20 is pc/at bios boot from http -- not arm64.)
+# Detect client architecture -- UEFI HTTP Boot clients (no TFTP size limit)
 dhcp-match=set:httpboot-x86_64,option:client-arch,16
-dhcp-match=set:httpboot-arm64,option:client-arch,19
+dhcp-match=set:httpboot-arm64,option:client-arch,20
 
 # Detect iPXE clients (already chainloaded)
 dhcp-userclass=set:ipxe,iPXE

bastion/src/bastion/src/templates/install.ks.ts

@@ -40,6 +40,11 @@ export function renderInstallKickstart(params: InstallKickstartParams): string {
   const now = new Date().toISOString();
   const hasLonghorn = role === "worker";
   const hasRancher = role === "infra";
+  // k8s roles get a dedicated 120G image-store LV. 2026-08 incident: the old
+  // 20G LV idled at 85% used, so a single ~5G image pull tripped imagefs
+  // eviction. Must be sized here — longhorn's --grow consumes all remaining
+  // VG space, making post-install lvextend impossible on worker nodes.
+  const hasRancherLv = role === "infra" || role === "worker";
   const isVanilla = role === "vanilla";
 
   // -- Auth section --
@@ -113,9 +118,9 @@ done
     ? `logvol /var/lib/longhorn --vgname=${vg} --name=longhorn --fstype=xfs --grow --size=1`
     : "";
 
-  // -- Rancher LV for fresh install (infra role) --
-  const rancherFreshLine = hasRancher
-    ? `logvol /var/lib/rancher --vgname=${vg} --name=rancher --fstype=xfs --size=20480`
+  // -- Rancher LV for fresh install (k8s roles: worker + infra) --
+  const rancherFreshLine = hasRancherLv
+    ? `logvol /var/lib/rancher --vgname=${vg} --name=rancher --fstype=xfs --size=122880`
     : "";
 
   return `# Lab Bastion -- Fedora ${fedoraVersion} server install

bastion/src/bastion/src/templates/vyos-boot.ipxe.ts

@@ -0,0 +1,53 @@
+// iPXE boot script template for VyOS network install.
+//
+// VyOS ships no unattended installer: `install image` is unconditionally
+// interactive (image_installer.py's install action takes no arguments, and
+// --no-prompt is wired only to `add`). So PXE boots the *live* system and the
+// automation is injected through live-config's `hooks` component, which fetches
+// a script over HTTP and runs it as root late in live boot.
+//
+// Unlike the Fedora/Ubuntu paths this boots a live image rather than an
+// installer, so there is no kickstart/autoinstall equivalent — see
+// routes/vyos.ts for the hook that actually drives the install.
+
+export function renderVyosInstallIpxe(params: {
+  mac: string;
+  hostname: string;
+  serverIp: string;
+  httpPort: number;
+}): string {
+  const base = `http://${params.serverIp}:${params.httpPort}`;
+
+  // Pin the boot NIC by MAC. live-boot otherwise scans for the first
+  // *connected* interface, and on a multi-NIC box that race is lost by
+  // whichever port negotiates slowest: on the Protectli VP2440 the SFP+
+  // pair links first, so live-boot picked the fiber ports (which have no
+  // DHCP), burned 15s per port, and gave up with "Unable to find a live
+  // file system on the network" -- while the copper port that actually PXE
+  // booted came up at 4.6s and was never tried.
+  //
+  // live-boot's Device_from_bootif() strips the "01-" and matches the MAC
+  // against /sys/class/net/*. params.mac is the dispatch key, i.e. exactly
+  // the NIC that PXE booted -- more reliable than iPXE's ${net0} on a box
+  // where the booting NIC may not be net0.
+  const bootif = `01-${params.mac.toLowerCase().replace(/:/g, "-")}`;
+
+  // Deliberately NOT passing `nonetworking` (present in VyOS's own PXE docs):
+  // live-config's hook component needs networking up to fetch the hook over
+  // HTTP. Also no `console=ttyS0` — on hardware without a physical UART that
+  // costs 30s at every systemd boot phase.
+  return `#!ipxe
+
+echo
+echo =============================================
+echo   Lab PXE Bastion - INSTALLING VyOS
+echo   Target: ${params.hostname}
+echo   MAC:    ${params.mac}
+echo =============================================
+echo
+
+kernel ${base}/vyos-vmlinuz boot=live nopersistence noautologin BOOTIF=${bootif} fetch=${base}/vyos-filesystem.squashfs live-config.hooks=${base}/vyos/autoinstall.sh?mac=${params.mac}
+initrd ${base}/vyos-initrd
+boot
+`;
+}

bastion/src/bastion/src/templates/vyos-config-spec.ts

@@ -0,0 +1,352 @@
+// Builds the VyOS configuration spec applied by the autoinstall hook.
+//
+// We deliberately do NOT emit a config.boot file as text. A config.boot carries a
+// `vyos-config-version` trailer; without a trailer matching the running image,
+// VyOS runs its migration scripts from version 0 on first boot. Instead the hook
+// loads the image's own /opt/vyatta/etc/config.boot.default through vyos.configtree
+// and applies these set operations on top, so syntax and version trailer always
+// match the exact image being installed.
+
+import type { VyosInstallSpec } from "@lab/shared";
+
+export interface VyosSetOp {
+  path: string[];
+  value?: string;
+  /** false appends to a multi-value node (e.g. bond members) instead of replacing. */
+  replace?: boolean;
+}
+
+export interface VyosConfigSpec {
+  hostname: string;
+  /** "" means accept the installer default (the running image's version string). */
+  imageName: string;
+  password: string;
+  console: "K" | "S";
+  /** Target disk name (e.g. "nvme0n1"); "" accepts the installer's first-disk default. */
+  disk: string;
+  /**
+   * IP the driver should report in the "complete" callback ("ready at <ip>" —
+   * the exact format routes/api.ts parses installed.ip from). The mgmt
+   * address when static; "" means detect the live DHCP address at runtime.
+   */
+  reportAddress: string;
+  /** Whether to accept RAID-1 when the installer finds more than one disk. */
+  raid: boolean;
+  /** Overwrite the installed config.boot with the generated one on reinstall. */
+  freshConfig: boolean;
+  sets: VyosSetOp[];
+  /** Paths that are VyOS tag nodes — must be marked as such in the ConfigTree. */
+  tags: string[][];
+}
+
+/**
+ * Normalise a target disk to the form the installer expects.
+ *
+ * find_disks() enumerates via `lsblk -Jbp` (-p = full paths), so its valid
+ * responses are "/dev/mmcblk0"-style. A bare "mmcblk0" is rejected by
+ * ask_input()'s valid_responses check and re-prompts forever.
+ */
+function normalizeDiskPath(value: string | undefined): string {
+  const raw = (value ?? "").trim();
+  if (raw === "") return "";
+  return raw.startsWith("/dev/") ? raw : `/dev/${raw}`;
+}
+
+/** Sentinel marking a value that lives in Pulumi config, not in the bundle. */
+const SECRET_PREFIX = "@secret:";
+
+/**
+ * Enable the VyOS HTTP API so the router is manageable the moment it boots.
+ *
+ * This belongs at install time rather than in the Pulumi model: the model is
+ * applied THROUGH this API, so a router that lacks it cannot be brought under
+ * management without a hand-run change on a live firewall. It is also why the
+ * model excludes `service https` outright -- a provider able to rewrite its own
+ * transport can lock itself out permanently.
+ *
+ * `listen-address` is always set. Leaving it unbound would expose a
+ * config-write endpoint on every segment the router touches, the WAN included.
+ */
+function apiSets(apiKey: string, listenAddress: string): VyosSetOp[] {
+  const sets: VyosSetOp[] = [
+    { path: ["service", "https", "api", "keys", "id", "pulumi", "key"], value: apiKey },
+    { path: ["service", "https", "api", "rest"] },
+  ];
+  if (listenAddress !== "") {
+    sets.push({ path: ["service", "https", "listen-address"], value: listenAddress });
+  }
+  return sets;
+}
+
+/** Tag nodes introduced by the API config, needed by the installer's ConfigTree. */
+const API_TAGS: string[][] = [["service", "https", "api", "keys", "id"]];
+
+/**
+ * The address to bind the API to: an explicit choice, else the management
+ * address with its prefix length stripped. Under DHCP there is no address to
+ * bind at build time, so the caller must pass one or the listener stays unbound
+ * and the API is not enabled at all.
+ */
+function apiListenAddress(spec: VyosInstallSpec, mgmtAddress: string): string {
+  if (spec.apiListenAddress !== undefined && spec.apiListenAddress !== "") {
+    return spec.apiListenAddress;
+  }
+  return mgmtAddress.includes("/") ? (mgmtAddress.split("/")[0] ?? "") : "";
+}
+
+/**
+ * Use a Pulumi-rendered bundle as the router's config verbatim.
+ *
+ * Secret-valued nodes are dropped rather than installed with their sentinel
+ * text: writing `@secret:pppoePassword` into config.boot would look configured
+ * while being wrong, which is worse than being absent. The router comes up
+ * without those values and the first `pulumi up` fills them in.
+ *
+ * `system host-name` is forced to the hostname the install was asked for. The
+ * bundle carries the name of whichever router it was exported from, and
+ * installing vyos001's hostname onto vyos002 would collide on the network.
+ */
+function buildFromBundle(
+  params: { hostname: string; defaultPassword: string; disk?: string | undefined },
+  spec: VyosInstallSpec,
+  bundle: NonNullable<VyosInstallSpec["bundle"]>,
+  mgmtAddress: string,
+): VyosConfigSpec {
+  const sets: VyosSetOp[] = [];
+  const dropped: string[] = [];
+  for (const op of bundle.sets) {
+    if (op.value !== undefined && op.value.startsWith(SECRET_PREFIX)) {
+      dropped.push(op.path.join(" "));
+      continue;
+    }
+    if (op.path.length === 2 && op.path[0] === "system" && op.path[1] === "host-name") {
+      continue;
+    }
+    sets.push({
+      path: op.path,
+      ...(op.value === undefined ? {} : { value: op.value }),
+      ...(op.replace === undefined ? {} : { replace: op.replace }),
+    });
+  }
+  sets.unshift({ path: ["system", "host-name"], value: params.hostname });
+
+  if (dropped.length > 0) {
+    console.warn(
+      `vyos ${params.hostname}: ${dropped.length} secret-valued node(s) left unset by the ` +
+        `bundle; run \`pulumi up\` to supply them: ${dropped.join(", ")}`,
+    );
+  }
+
+  const tags = [...bundle.tags];
+  const api = enableApi(spec, params.hostname, mgmtAddress);
+  if (api.length > 0) {
+    sets.push(...api);
+    tags.push(...API_TAGS);
+  }
+
+  return {
+    hostname: params.hostname,
+    imageName: "",
+    password: spec.password ?? params.defaultPassword,
+    console: "K",
+    disk: normalizeDiskPath(params.disk),
+    reportAddress: mgmtAddress.includes("/") ? (mgmtAddress.split("/")[0] ?? "") : "",
+    raid: false,
+    freshConfig: spec.freshConfig ?? false,
+    sets,
+    tags,
+  };
+}
+
+/**
+ * The API config for this install, or nothing when it cannot be enabled safely.
+ *
+ * Refusing to enable it unbound is deliberate. Under DHCP there is no address
+ * known at build time, and the alternative -- binding to every interface --
+ * would publish a config-write endpoint on the WAN. Better to leave the router
+ * SSH-only and say so than to open it everywhere.
+ */
+function enableApi(spec: VyosInstallSpec, hostname: string, mgmtAddress: string): VyosSetOp[] {
+  if (spec.apiKey === undefined || spec.apiKey === "") return [];
+  const listen = apiListenAddress(spec, mgmtAddress);
+  if (listen === "") {
+    console.warn(
+      `vyos ${hostname}: --vyos-api-key given but no address to bind to ` +
+        `(management is "${mgmtAddress}"). Pass --vyos-api-listen <addr>; the HTTP API ` +
+        `has NOT been enabled, so Pulumi cannot manage this router yet.`,
+    );
+    return [];
+  }
+  return apiSets(spec.apiKey, listen);
+}
+
+export function buildVyosConfigSpec(params: {
+  hostname: string;
+  spec?: VyosInstallSpec | undefined;
+  defaultPassword: string;
+  sshKeys?: string[] | undefined;
+  disk?: string | undefined;
+}): VyosConfigSpec {
+  const spec = params.spec ?? {};
+  const mgmt = spec.mgmtInterface ?? "eth0";
+  const mgmtAddress = spec.mgmtAddress ?? "dhcp";
+
+  // A rendered bundle replaces the derived config entirely. Deriving a second
+  // opinion alongside it is the drift the bundle exists to prevent: Pulumi and
+  // labctl would each believe they knew the router's config, and the box would
+  // end up with whichever ran last.
+  if (spec.bundle !== undefined) {
+    return buildFromBundle(params, spec, spec.bundle, mgmtAddress);
+  }
+  const bondMembers = spec.bondMembers ?? [];
+  const vlans = spec.vlans ?? [];
+
+  const sets: VyosSetOp[] = [];
+  const tags: string[][] = [
+    ["interfaces", "ethernet"],
+    ["system", "login", "user"],
+  ];
+
+  const hwIds = spec.hwIds ?? {};
+  const pinHwId = (iface: string): void => {
+    const mac = hwIds[iface];
+    if (mac !== undefined && mac !== "") {
+      sets.push({ path: ["interfaces", "ethernet", iface, "hw-id"], value: mac });
+    }
+  };
+
+  sets.push({ path: ["system", "host-name"], value: params.hostname });
+
+  // Management interface — the NIC that PXE booted, left untagged and unbonded.
+  sets.push({ path: ["interfaces", "ethernet", mgmt, "address"], value: mgmtAddress });
+  pinHwId(mgmt);
+
+  // Tagged management VLAN on the PXE port. Emitted regardless of bonding, so
+  // the box stays reachable on the management VLAN while still booting untagged
+  // on whichever VLAN the bastion's proxy DHCP serves.
+  const mgmtVlan = spec.mgmtVlan;
+  if (mgmtVlan !== undefined) {
+    tags.push(["interfaces", "ethernet", mgmt, "vif"]);
+    const vif = ["interfaces", "ethernet", mgmt, "vif", String(mgmtVlan.id)];
+    sets.push({ path: [...vif, "address"], value: mgmtVlan.address });
+    if (mgmtVlan.description !== undefined && mgmtVlan.description !== "") {
+      sets.push({ path: [...vif, "description"], value: mgmtVlan.description });
+    }
+  }
+
+  // LACP bond. Members must exclude the PXE NIC; firmware PXE cannot run over LACP.
+  const bonded = bondMembers.length > 0;
+  if (bonded) {
+    tags.push(["interfaces", "bonding"]);
+    sets.push({ path: ["interfaces", "bonding", "bond0", "mode"], value: "802.3ad" });
+    sets.push({ path: ["interfaces", "bonding", "bond0", "hash-policy"], value: "layer2+3" });
+    for (const member of bondMembers) {
+      sets.push({
+        path: ["interfaces", "bonding", "bond0", "member", "interface"],
+        value: member,
+        replace: false,
+      });
+      pinHwId(member);
+    }
+    // Address on the trunk's native/untagged VLAN.
+    if (spec.bondAddress !== undefined && spec.bondAddress !== "") {
+      sets.push({ path: ["interfaces", "bonding", "bond0", "address"], value: spec.bondAddress });
+    }
+  }
+
+  // VRRP groups accumulate here; emitted (plus a sync group) after the VLANs.
+  // interface accepts dotted vifs (constraint regex `[0-9]+(.\d+)?`), address
+  // is a tag node (the VIP is the tag value itself), vrid range is 1-255.
+  const vrrpGroups: Array<{ name: string; iface: string; vrid: number; vip: string }> = [];
+  if (bonded && spec.bondVrrp !== undefined && spec.bondVrrp !== "") {
+    // vrid 1 for the untagged group: the native VLAN is never a vif, so this
+    // cannot collide with a vlan-id-derived vrid.
+    vrrpGroups.push({ name: "native", iface: "bond0", vrid: 1, vip: spec.bondVrrp });
+  }
+
+  // Tagged VLAN sub-interfaces hang off the bond when there is one, else off mgmt.
+  const parent = bonded
+    ? ["interfaces", "bonding", "bond0"]
+    : ["interfaces", "ethernet", mgmt];
+  if (vlans.length > 0) {
+    tags.push([...parent, "vif"]);
+    const parentName = bonded ? "bond0" : mgmt;
+    for (const vlan of vlans) {
+      const vif = [...parent, "vif", String(vlan.id)];
+      sets.push({ path: [...vif, "address"], value: vlan.address });
+      if (vlan.description !== undefined && vlan.description !== "") {
+        sets.push({ path: [...vif, "description"], value: vlan.description });
+      }
+      if (vlan.vrrp !== undefined && vlan.vrrp !== "") {
+        vrrpGroups.push({
+          name: `vlan${vlan.id}`,
+          iface: `${parentName}.${vlan.id}`,
+          vrid: vlan.id,
+          vip: vlan.vrrp,
+        });
+      }
+    }
+  }
+
+  // Emit VRRP groups plus one sync group so all VLANs fail over together —
+  // without it a single-link event could split mastership across the pair.
+  if (vrrpGroups.length > 0) {
+    tags.push(["high-availability", "vrrp", "group"]);
+    tags.push(["high-availability", "vrrp", "sync-group"]);
+    const priority = String(spec.vrrpPriority ?? 100);
+    for (const g of vrrpGroups) {
+      const base = ["high-availability", "vrrp", "group", g.name];
+      sets.push({ path: [...base, "interface"], value: g.iface });
+      sets.push({ path: [...base, "vrid"], value: String(g.vrid) });
+      sets.push({ path: [...base, "priority"], value: priority });
+      // address is a tag node: the VIP is the path's final segment, no value.
+      sets.push({ path: [...base, "address", g.vip] });
+      tags.push([...base, "address"]);
+      sets.push({
+        path: ["high-availability", "vrrp", "sync-group", "MAIN", "member"],
+        value: g.name,
+        replace: false,
+      });
+    }
+  }
+
+  sets.push({ path: ["service", "ssh", "port"], value: "22" });
+
+  const sshKeys = params.sshKeys ?? [];
+  if (sshKeys.length > 0) {
+    tags.push(["system", "login", "user", "vyos", "authentication", "public-keys"]);
+    sshKeys.forEach((entry, index) => {
+      const parts = entry.trim().split(/\s+/);
+      const type = parts[0] ?? "";
+      const key = parts[1] ?? "";
+      if (!type.startsWith("ssh-") && !type.startsWith("ecdsa-")) return;
+      if (!key) return;
+      const name = parts[2] ?? `lab-key-${index}`;
+      const base = ["system", "login", "user", "vyos", "authentication", "public-keys", name];
+      sets.push({ path: [...base, "type"], value: type });
+      sets.push({ path: [...base, "key"], value: key });
+    });
+  }
+
+  // Enabled here too, not just for bundle installs: every VyOS this bastion
+  // provisions should be manageable from first boot.
+  const api = enableApi(spec, params.hostname, mgmtAddress);
+  if (api.length > 0) {
+    sets.push(...api);
+    tags.push(...API_TAGS);
+  }
+
+  return {
+    hostname: params.hostname,
+    imageName: "",
+    password: spec.password ?? params.defaultPassword,
+    console: "K",
+    disk: normalizeDiskPath(params.disk),
+    // Static mgmt address wins; under DHCP the driver detects the live IP.
+    reportAddress: mgmtAddress.includes("/") ? (mgmtAddress.split("/")[0] ?? "") : "",
+    raid: false,
+    freshConfig: spec.freshConfig ?? false,
+    sets,
+    tags,
+  };
+}

bastion/src/bastion/src/templates/vyos-install.py.ts

@@ -0,0 +1,514 @@
+// Renders the Python program that performs the unattended VyOS install.
+//
+// It runs as root inside the live system, fetched and executed by live-config's
+// `hooks` component (see vyos-boot.ipxe.ts). It does three things:
+//   1. builds config.boot from the image's own default via vyos.configtree
+//   2. drives the interactive `install image` through a pty
+//   3. reports progress back to the bastion, then reboots
+//
+// A pty is used rather than piping stdin because the installer reads the
+// password through getpass(), which opens /dev/tty directly and would ignore a
+// pipe. Prompts are matched by text rather than replayed positionally: the
+// installer skips the boot-config question when it finds a previous
+// installation, so a fixed answer sequence desyncs on reinstall.
+
+import type { VyosConfigSpec } from "./vyos-config-spec.js";
+
+export function renderVyosInstallPy(params: {
+  spec: VyosConfigSpec;
+  mac: string;
+  serverIp: string;
+  httpPort: number;
+  role: string;
+}): string {
+  // Base64 so arbitrary values (passwords, descriptions, SSH keys) can never
+  // terminate the Python string literal that carries them.
+  const specB64 = Buffer.from(JSON.stringify(params.spec), "utf-8").toString("base64");
+
+  return `#!/usr/bin/env python3
+"""Unattended VyOS install driver -- generated by the lab PXE bastion."""
+
+import base64
+import json
+import os
+import pty
+import re
+import select
+import subprocess
+import sys
+import time
+import urllib.request
+
+SPEC = json.loads(base64.b64decode("${specB64}").decode("utf-8"))
+BASTION = "http://${params.serverIp}:${params.httpPort}"
+MAC = "${params.mac}"
+ROLE = ${JSON.stringify(params.role ?? "vanilla")}
+
+INSTALLER = "/usr/libexec/vyos/op_mode/image_installer.py"
+CONFIG_DIR = "/opt/vyatta/etc/config"
+# The installer copies the rootfs from the boot MEDIUM path -- which only a
+# CD/USB boot provides. With fetch= (HTTP netboot) nothing is mounted there
+# (verified in VM: Errno 2), so the squashfs must be linked or re-fetched into
+# place before 'install image' runs.
+ROOTFS_EXPECTED = "/usr/lib/live/mount/medium/live/filesystem.squashfs"
+SQUASHFS_URL = "http://${params.serverIp}:${params.httpPort}/vyos-filesystem.squashfs"
+# The live-config hook runs BEFORE vyos-router creates the /opt/vyatta compat
+# path, so the squashfs's own location must be tried too (verified in VM: only
+# /usr/share/vyos/config.boot.default exists at hook time).
+DEFAULT_CONFIG_CANDIDATES = [
+    "/opt/vyatta/etc/config.boot.default",
+    "/usr/share/vyos/config.boot.default",
+]
+STALL_TIMEOUT = 900  # seconds without installer output before giving up
+
+
+def detect_ip():
+    """Best-effort local IP as seen on the route toward the bastion.
+
+    Matches Fedora's semantics (IP captured during install): under DHCP the
+    installed system will renew on the same NIC/subnet the live env used.
+    """
+    import socket
+    try:
+        s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
+        s.connect(("${params.serverIp}", ${params.httpPort}))
+        ip = s.getsockname()[0]
+        s.close()
+        return ip
+    except Exception:
+        return ""
+
+
+class LogStreamer:
+    """Stream install output to the bastion's /api/log so 'labctl provision
+    logs -f' works live for VyOS, like Anaconda's syslog does for Fedora.
+
+    Strictly best-effort: a failed POST drops the batch and must never stall
+    the pty read loop or fail the install.
+    """
+
+    ANSI = re.compile(rb"\\x1b\\[[0-9;?]*[a-zA-Z]|\\x1b[=>]|\\r")
+
+    def __init__(self):
+        self.partial = b""
+        self.pending = []
+        self.last_flush = time.time()
+
+    def feed(self, chunk):
+        """Raw pty bytes: split into lines, strip ANSI noise, queue."""
+        self.partial += chunk
+        while b"\\n" in self.partial:
+            raw, self.partial = self.partial.split(b"\\n", 1)
+            text = self.ANSI.sub(b"", raw).decode("utf-8", "replace").rstrip()
+            if text:
+                self.pending.append(text)
+        self.maybe_flush()
+
+    def line(self, text):
+        """A driver-originated message (already a clean string)."""
+        self.pending.append(text)
+        self.maybe_flush()
+
+    def maybe_flush(self):
+        if len(self.pending) >= 20 or (self.pending and time.time() - self.last_flush >= 2):
+            self.flush()
+
+    def flush(self):
+        if not self.pending:
+            return
+        batch, self.pending = self.pending[:200], self.pending[200:]
+        self.last_flush = time.time()
+        try:
+            body = json.dumps({"mac": MAC, "lines": batch}).encode()
+            req = urllib.request.Request(
+                BASTION + "/api/log",
+                data=body,
+                headers={"Content-Type": "application/json"},
+            )
+            urllib.request.urlopen(req, timeout=5).read()
+        except Exception:
+            pass
+
+
+STREAM = LogStreamer()
+
+
+def say(msg):
+    """Print locally and stream to the bastion log buffer."""
+    print(msg)
+    STREAM.line(str(msg))
+
+
+def report(stage, detail=""):
+    """Best-effort progress callback; never fatal."""
+    STREAM.flush()
+    try:
+        body = json.dumps({"mac": MAC, "stage": stage, "detail": detail}).encode()
+        req = urllib.request.Request(
+            BASTION + "/api/progress",
+            data=body,
+            headers={"Content-Type": "application/json"},
+        )
+        urllib.request.urlopen(req, timeout=5).read()
+    except Exception:
+        pass
+
+
+def build_config():
+    """Apply our set operations onto the image's own default config.
+
+    Using config.boot.default as the base keeps the vyos-config-version trailer
+    consistent with the running image, so first boot does not run migrations.
+    """
+    from vyos.configtree import ConfigTree
+
+    default_config = next(
+        (p for p in DEFAULT_CONFIG_CANDIDATES if os.path.exists(p)), None)
+    if default_config is None:
+        raise FileNotFoundError(
+            "no config.boot.default found (tried %s)" % ", ".join(DEFAULT_CONFIG_CANDIDATES))
+    say("base config: %s" % default_config)
+
+    with open(default_config) as handle:
+        config = ConfigTree(handle.read())
+
+    for op in SPEC["sets"]:
+        replace = op.get("replace", True)
+        if "value" in op and op["value"] is not None:
+            config.set(op["path"], value=op["value"], replace=replace)
+        else:
+            config.set(op["path"])
+
+    # Tag nodes must be marked after the nodes exist, as the installer itself does.
+    for tag in SPEC["tags"]:
+        try:
+            config.set_tag(tag)
+        except Exception as err:
+            say("warning: set_tag %s failed: %s" % (tag, err))
+
+    os.makedirs(CONFIG_DIR, exist_ok=True)
+    target = os.path.join(CONFIG_DIR, "config.boot")
+
+    # Re-attach the vyos-config-version footer: ConfigTree.to_string() emits
+    # only the config body, and a config without the footer is treated as
+    # ancient -- the boot migrator then runs every migration over it and (as
+    # observed in the VM test) crashes in system/31-to-32. Building the footer
+    # from the running system pins it to the exact image being installed.
+    body = config.to_string()
+    try:
+        from vyos.component_version import version_info_from_system
+        info = version_info_from_system()
+        info.update_config_body(body)
+        info.write(target)
+        say("wrote %s (footer: %s)" % (target, info.release))
+    except Exception as err:
+        say("warning: version footer failed (%s); writing bare config" % err)
+        with open(target, "w") as handle:
+            handle.write(body)
+    return target
+
+
+def find_live_squashfs():
+    """Locate the squashfs live-boot fetched, without walking into the mounted
+    rootfs or overlay (each would mean traversing the entire OS tree)."""
+    explicit = [
+        "/run/live/medium/live/filesystem.squashfs",
+        "/lib/live/mount/medium/live/filesystem.squashfs",
+    ]
+    for path in explicit:
+        if os.path.isfile(path) and os.path.getsize(path) > 0:
+            return path
+    for root in ("/run/live", "/lib/live/mount", "/usr/lib/live/mount"):
+        for dirpath, dirs, files in os.walk(root):
+            depth = dirpath.count(os.sep) - root.count(os.sep)
+            dirs[:] = [d for d in dirs
+                       if d not in ("rootfs", "overlay")
+                       and not d.endswith(".squashfs")
+                       and depth < 3]
+            if "filesystem.squashfs" in files:
+                path = os.path.join(dirpath, "filesystem.squashfs")
+                if os.path.isfile(path) and os.path.getsize(path) > 0:
+                    return path
+    return None
+
+
+def ensure_rootfs():
+    """Make FILE_ROOTFS_SRC exist so the installer can copy the system image."""
+    if os.path.isfile(ROOTFS_EXPECTED) and os.path.getsize(ROOTFS_EXPECTED) > 0:
+        return
+    src = find_live_squashfs()
+    if src is None:
+        say("squashfs not in live mounts; re-fetching %s" % SQUASHFS_URL)
+        src = "/tmp/filesystem.squashfs"
+        urllib.request.urlretrieve(SQUASHFS_URL, src)
+    os.makedirs(os.path.dirname(ROOTFS_EXPECTED), exist_ok=True)
+    if os.path.lexists(ROOTFS_EXPECTED):
+        os.remove(ROOTFS_EXPECTED)
+    os.symlink(src, ROOTFS_EXPECTED)
+    say("rootfs source: %s -> %s" % (ROOTFS_EXPECTED, src))
+
+
+def build_rules():
+    """Prompt -> response table for the interactive installer."""
+    password = SPEC["password"].encode() + b"\\n"
+    image_name = SPEC["imageName"].encode() + b"\\n"
+    disk = SPEC["disk"].encode() + b"\\n"
+    console = SPEC["console"].encode() + b"\\n"
+    raid = (b"yes\\n" if SPEC["raid"] else b"no\\n")
+
+    return [
+        (re.compile(rb"Would you like to continue\\?"), b"yes\\n"),
+        (re.compile(rb"What would you like to name this image\\?"), image_name),
+        (re.compile(rb"Please confirm password for the .vyos. user:"), password),
+        (re.compile(rb"Please enter a password for the .vyos. user:"), password),
+        (re.compile(rb"What console should be used by default"), console),
+        # Three RAID variants: "configure RAID-1 mirroring?", "...on them?",
+        # and "choose two disks for RAID-1 mirroring?" -- all default to YES,
+        # so a missed one both hangs the install and risks an unwanted mirror.
+        (re.compile(rb"Would you like to [^?]*RAID-1 mirroring"), raid),
+        (re.compile(rb"Installation will delete all data on (?:the drive|both drives)\\. Continue\\?"), b"yes\\n"),
+        (re.compile(rb"Which one should be used for installation\\?"), disk),
+        (re.compile(rb"Would you like to use all the free space on the drive\\?"), b"yes\\n"),
+        (re.compile(rb"Which file would you like as boot config\\?"), b"1\\n"),
+        # Reinstall path only (search_previous_installation): carrying the old
+        # /config and SSH host keys forward is VyOS's "reinstall without losing
+        # data". Always yes -- freshConfig replaces config.boot afterwards, so
+        # answering no here would also discard non-config data under /config.
+        (re.compile(rb"Would you like to copy data to the new image\\?"), b"yes\\n"),
+        (re.compile(rb"Would you like to copy the encrypted config to the new image\\?"), b"yes\\n"),
+        # More than one previous image found -- take the first offered.
+        (re.compile(rb"From which image would you like to save config information\\?"), b"1\\n"),
+        (re.compile(rb"From which image would you like to copy the encrypted config\\?"), b"1\\n"),
+    ]
+
+
+def run_installer():
+    """Drive image_installer.py over a pty, answering prompts as they appear."""
+    rules = build_rules()
+    master, slave = pty.openpty()
+
+    proc = subprocess.Popen(
+        [INSTALLER, "--action", "install"],
+        stdin=slave,
+        stdout=slave,
+        stderr=slave,
+        close_fds=True,
+        preexec_fn=os.setsid,
+    )
+    os.close(slave)
+
+    buf = b""
+    transcript = b""   # rolling tail of everything the installer printed
+    last_output = time.time()
+
+    while True:
+        ready, _, _ = select.select([master], [], [], 1.0)
+
+        if ready:
+            try:
+                chunk = os.read(master, 4096)
+            except OSError:
+                break
+            if not chunk:
+                break
+
+            sys.stdout.buffer.write(chunk)
+            sys.stdout.buffer.flush()
+            buf += chunk
+            transcript = (transcript + chunk)[-8000:]
+            STREAM.feed(chunk)
+            last_output = time.time()
+
+            # Answer every prompt currently in the buffer, earliest first, so
+            # ordering is preserved even when the installer skips questions --
+            # and so a single chunk carrying two prompts gets both answers.
+            while True:
+                best = None
+                for pattern, response in rules:
+                    found = pattern.search(buf)
+                    if found and (best is None or found.start() < best[0].start()):
+                        best = (found, response)
+                if best is None:
+                    break
+                found, response = best
+                os.write(master, response)
+                transcript = (transcript + b"\\n>>> answered: " + response)[-8000:]
+                STREAM.line(">>> answered: " + response.decode("utf-8", "replace").strip())
+                buf = buf[found.end():]
+
+            # Bound memory if the installer emits a lot without prompting.
+            if len(buf) > 65536:
+                buf = buf[-8192:]
+
+        elif proc.poll() is not None:
+            break
+
+        STREAM.maybe_flush()
+
+        if time.time() - last_output > STALL_TIMEOUT:
+            proc.kill()
+            raise SystemExit("installer produced no output for %ds" % STALL_TIMEOUT)
+
+    os.close(master)
+    return proc.wait(), transcript.decode("utf-8", "replace")
+
+
+def ensure_network_boot_first():
+    """Keep network boot first so the bastion intercepts every reboot.
+
+    Port of the Fedora kickstart's %post efibootmgr step (install.ks.ts) --
+    what makes reprovision-by-reboot work. Best-effort: skipped on BIOS boots
+    or when efibootmgr is absent. Runs from the live env after the installer;
+    efibootmgr edits NVRAM, not the disk, so installer cleanup is irrelevant.
+    """
+    import shutil
+    if not os.path.isdir("/sys/firmware/efi") or shutil.which("efibootmgr") is None:
+        say("boot order: skipped (BIOS boot or efibootmgr missing)")
+        return
+    try:
+        out = subprocess.run(["efibootmgr"], capture_output=True, text=True, timeout=30).stdout
+        order = []
+        network_entry = None
+        for line in out.splitlines():
+            m = re.match(r"^BootOrder:\\s*(.*)$", line)
+            if m:
+                order = [x.strip() for x in m.group(1).split(",") if x.strip()]
+                continue
+            m = re.match(r"^Boot([0-9A-Fa-f]{4})\\*?\\s+(.*)$", line)
+            if m and network_entry is None:
+                if re.search(r"network|pxe|ipv4|ipv6|http", m.group(2), re.IGNORECASE):
+                    network_entry = m.group(1).upper()
+        if network_entry is None or not order:
+            say("boot order: no network boot entry found; leaving as is")
+            return
+        new_order = [network_entry] + [x for x in order if x.upper() != network_entry]
+        if [x.upper() for x in order] == [x.upper() for x in new_order]:
+            say("boot order: network entry Boot%s already first" % network_entry)
+            return
+        subprocess.run(["efibootmgr", "-o", ",".join(new_order)],
+                       capture_output=True, timeout=30)
+        say("boot order: moved network entry Boot%s first" % network_entry)
+    except Exception as err:
+        say("warning: boot order adjustment failed: %s" % err)
+
+
+def with_target_mounted(fn):
+    """Mount the installed root partition, call fn(rw_dir), always unmount.
+
+    The installer has unmounted and cleaned the target by the time this runs,
+    so the block device is free. The partition holding boot/<image>/rw is the
+    VyOS root; the glob also yields the installed image's rw dir directly.
+    """
+    import glob
+    disk = SPEC["disk"]
+    if not disk:
+        # No pinned disk (installer picked the default) -- enumerate all disks.
+        candidates = ["/dev/" + b for b in os.listdir("/sys/block")
+                      if not b.startswith(("loop", "ram", "zram", "sr"))]
+    else:
+        candidates = [disk]
+
+    mnt = "/mnt/lab-target"
+    os.makedirs(mnt, exist_ok=True)
+    for dev in candidates:
+        name = os.path.basename(dev)
+        parts = sorted(p for p in os.listdir("/sys/block/%s" % name)
+                       if p.startswith(name)) if os.path.isdir("/sys/block/%s" % name) else []
+        for part in parts:
+            pdev = "/dev/" + part
+            if subprocess.run(["mount", pdev, mnt], capture_output=True).returncode != 0:
+                continue
+            try:
+                rw_dirs = glob.glob(os.path.join(mnt, "boot", "*", "rw"))
+                if rw_dirs:
+                    fn(rw_dirs[0])
+                    return True
+            finally:
+                subprocess.run(["umount", mnt], capture_output=True)
+    return False
+
+
+def post_install_target_steps():
+    """Metadata + optional fresh-config overwrite inside the installed image."""
+    def apply(rw_dir):
+        config_dir = os.path.join(rw_dir, "opt/vyatta/etc/config")
+        os.makedirs(config_dir, exist_ok=True)
+
+        # /config/lab-provisioned -- survives VyOS image upgrades. Mirrors the
+        # Fedora kickstart's /etc/lab-provisioned.
+        try:
+            with open(os.path.join(config_dir, "lab-provisioned"), "w") as handle:
+                handle.write("hostname=%s\\n" % SPEC["hostname"])
+                handle.write("role=%s\\n" % ROLE)
+                handle.write("provisioned=%s\\n" % time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()))
+                handle.write("bastion=%s\\n" % BASTION)
+            say("wrote /config/lab-provisioned")
+        except Exception as err:
+            say("warning: lab-provisioned metadata failed: %s" % err)
+
+        # freshConfig: make the bastion-generated config win over the previous
+        # installation's carried-forward config. Explicit intent -- failure is
+        # fatal (raised out of with_target_mounted).
+        if SPEC.get("freshConfig"):
+            import shutil
+            shutil.copyfile(os.path.join(CONFIG_DIR, "config.boot"),
+                            os.path.join(config_dir, "config.boot"))
+            say("freshConfig: replaced installed config.boot with generated config")
+
+    mounted = with_target_mounted(apply)
+    if not mounted:
+        if SPEC.get("freshConfig"):
+            raise RuntimeError("freshConfig requested but installed root partition not found")
+        say("warning: installed root partition not found; skipping metadata")
+
+
+def main():
+    report("vyos-install", "building config.boot")
+    try:
+        build_config()
+    except Exception as err:
+        report("error", "config generation failed: %s" % err)
+        raise
+
+    report("vyos-install", "staging rootfs for installer")
+    try:
+        ensure_rootfs()
+    except Exception as err:
+        report("error", "rootfs staging failed: %s" % err)
+        raise
+
+    report("vyos-install", "running install image")
+    code, transcript = run_installer()
+
+    if code != 0:
+        # Surface the installer's last words in bastion progress -- the console
+        # they were printed on is usually invisible during unattended installs.
+        report("error", "install image exited %d | tail: %s" % (code, transcript[-4000:]))
+        raise SystemExit(code)
+
+    report("post-install", "boot order + metadata")
+    ensure_network_boot_first()
+    try:
+        post_install_target_steps()
+    except Exception as err:
+        report("error", "post-install target steps failed: %s" % err)
+        raise
+
+    # "complete" is the stage the bastion uses to move a machine out of the
+    # install queue into installed state, and "ready at <ip>" is the exact
+    # detail format it parses installed.ip from -- see routes/api.ts.
+    ip = SPEC.get("reportAddress") or detect_ip()
+    report("complete", "ready at %s" % ip if ip else "VyOS installed, rebooting")
+    os.system("sync")
+    # --force: this driver is a child of live-config.service, whose start job is
+    # still running -- a normal reboot deadlocks waiting for it (verified in VM:
+    # shutdown blocked >1min on "start job is running for live-config"). The
+    # installer has already unmounted and cleaned the target, so an immediate
+    # reboot is safe.
+    os.system("systemctl reboot --force")
+
+
+if __name__ == "__main__":
+    main()
+`;
+}

bastion/src/bastion/tests/arch-dispatch.test.ts

@@ -1,291 +0,0 @@
-// aarch64 support in the PXE dispatch path.
-//
-// The x86_64 side is pinned separately by ipxe-x86-regression.test.ts.
-
-import { describe, it, expect, beforeEach, afterEach } from "vitest";
-import { mkdirSync, rmSync } from "node:fs";
-import { join } from "node:path";
-import { tmpdir } from "node:os";
-import type { BastionConfig, BastionState, HardwareInfo } from "@lab/shared";
-import { createApp } from "../src/server.js";
-import { resolveArch } from "../src/routes/dispatch.js";
-import { renderDnsmasqConf } from "../src/templates/dnsmasq.conf.js";
-import type { FastifyInstance } from "fastify";
-import type { StateManager } from "../src/services/state.js";
-
-function createTestConfig(testDir: string): BastionConfig {
-  return {
-    fedoraVersion: "43",
-    arch: "x86_64",
-    httpPort: 0,
-    timezone: "Europe/London",
-    locale: "en_GB.UTF-8",
-    bastionDir: testDir,
-    domain: "test.local",
-    dhcpMode: "proxy",
-    dhcpRangeStart: "",
-    dhcpRangeEnd: "",
-    ubuntuVersion: "26.04",
-    ubuntuMirror: "https://releases.ubuntu.com/26.04",
-    iface: "eth0",
-    serverIp: "10.0.0.1",
-    network: "10.0.0.0",
-    gateway: "10.0.0.1",
-    sshKeys: ["ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAITEST test@test"],
-    adminUser: "testadmin",
-    syslogPort: 15514,
-    skipDnsmasq: true,
-    skipArtifacts: true,
-    fedoraMirror: "https://download.fedoraproject.org/pub/fedora/linux/releases/43/Everything/x86_64/os",
-    tftpDir: join(testDir, "tftp"),
-    httpDir: join(testDir, "http"),
-    stateFile: join(testDir, "state.json"),
-  };
-}
-
-function hardware(mac: string, over: Partial<HardwareInfo> = {}): HardwareInfo {
-  return {
-    mac,
-    product: "TestBox",
-    board: "TestBoard",
-    serial: "SN123",
-    manufacturer: "TestCorp",
-    cpu_model: "Test CPU",
-    cpu_cores: 4,
-    memory_gb: 16,
-    arch: "x86_64",
-    disks: [],
-    nics: [],
-    first_seen: new Date().toISOString(),
-    last_seen: new Date().toISOString(),
-    ...over,
-  };
-}
-
-const emptyState = (): BastionState => ({
-  discovered: {}, install_queue: {}, installed: {}, debug: {},
-});
-
-describe("architecture resolution", () => {
-  const config = createTestConfig("/tmp/unused");
-  const mac = "aa:bb:cc:dd:ee:ff";
-
-  it("prefers the tracked record over what the client reports", () => {
-    const state = emptyState();
-    state.discovered[mac] = hardware(mac, { arch: "aarch64" });
-    // Client claims x86_64; the machine record says otherwise and wins.
-    expect(resolveArch(state, mac, "x86_64", config)).toBe("aarch64");
-  });
-
-  it("falls back to the architecture reported at boot", () => {
-    expect(resolveArch(emptyState(), mac, "arm64", config)).toBe("aarch64");
-  });
-
-  it("normalises iPXE's arm64 spelling to aarch64", () => {
-    expect(resolveArch(emptyState(), mac, "arm64", config)).toBe("aarch64");
-    expect(resolveArch(emptyState(), mac, "x86_64", config)).toBe("x86_64");
-  });
-
-  it("falls back to the configured default for unknown architectures", () => {
-    expect(resolveArch(emptyState(), mac, "riscv64", config)).toBe("x86_64");
-    expect(resolveArch(emptyState(), mac, undefined, config)).toBe("x86_64");
-  });
-
-  it("reads arch from the installed record for already-provisioned machines", () => {
-    const state = emptyState();
-    state.installed[mac] = {
-      hostname: "spark", role: "worker", ip: "10.0.0.5",
-      installed_at: new Date().toISOString(), arch: "aarch64",
-    };
-    expect(resolveArch(state, mac, undefined, config)).toBe("aarch64");
-  });
-});
-
-describe("aarch64 dispatch", () => {
-  let testDir: string;
-  let app: FastifyInstance;
-  let state: StateManager;
-  const mac = "aa:bb:cc:dd:ee:ff";
-
-  beforeEach(() => {
-    testDir = join(tmpdir(), `bastion-arch-test-${Date.now()}-${Math.random().toString(36).slice(2)}`);
-    mkdirSync(join(testDir, "http"), { recursive: true });
-    mkdirSync(join(testDir, "tftp"), { recursive: true });
-    const result = createApp(createTestConfig(testDir));
-    app = result.app;
-    state = result.state;
-  });
-
-  afterEach(async () => {
-    await app.close();
-    rmSync(testDir, { recursive: true, force: true });
-  });
-
-  it("serves the aarch64 kernel and initrd to an arm64 client", async () => {
-    const res = await app.inject({ method: "GET", url: `/dispatch?mac=${mac}&arch=arm64` });
-    expect(res.statusCode).toBe(200);
-    expect(res.body).toContain("/vmlinuz-aarch64");
-    expect(res.body).toContain("/initrd-aarch64.img");
-    expect(res.body).not.toContain("/vmlinuz ");
-  });
-
-  it("points an arm64 client at the aarch64 Fedora mirror", async () => {
-    const res = await app.inject({ method: "GET", url: `/dispatch?mac=${mac}&arch=arm64` });
-    expect(res.body).toContain("Everything/aarch64/os");
-    expect(res.body).not.toContain("Everything/x86_64/os");
-  });
-
-  it("uses serial console arguments and not nomodeset on arm64", async () => {
-    const res = await app.inject({ method: "GET", url: `/dispatch?mac=${mac}&arch=arm64` });
-    expect(res.body).toContain("console=ttyAMA0,115200");
-    expect(res.body).not.toContain("nomodeset");
-  });
-
-  it("refuses to serve the x86-only Ubuntu kernel to an arm64 client", async () => {
-    // A machine queued for Ubuntu before it was discovered as aarch64 reaches dispatch
-    // with no guard having run. Serving it /ubuntu-vmlinuz is the original bug.
-    state.update((s) => {
-      s.install_queue[mac] = {
-        hostname: "arm-node", disk: "", role: "worker",
-        os: "ubuntu-26.04", queued_at: new Date().toISOString(),
-      };
-    });
-
-    const res = await app.inject({ method: "GET", url: `/dispatch?mac=${mac}&arch=arm64` });
-    expect(res.statusCode).toBe(200);
-    expect(res.body).toContain("CANNOT BOOT THIS MACHINE");
-    expect(res.body).toContain("no aarch64 netboot artifacts");
-    expect(res.body).not.toContain("ubuntu-vmlinuz");
-  });
-
-  it("still serves Ubuntu to an x86_64 client", async () => {
-    state.update((s) => {
-      s.install_queue[mac] = {
-        hostname: "x86-node", disk: "", role: "worker",
-        os: "ubuntu-26.04", queued_at: new Date().toISOString(),
-      };
-    });
-
-    const res = await app.inject({ method: "GET", url: `/dispatch?mac=${mac}&arch=x86_64` });
-    expect(res.body).toContain("ubuntu-vmlinuz");
-    expect(res.body).not.toContain("CANNOT BOOT");
-  });
-
-  it("serves a rescue kernel for the recorded architecture, not the requester's", async () => {
-    // The Spark case: machine known to be aarch64, queued for rescue.
-    state.update((s) => {
-      s.discovered[mac] = hardware(mac, { arch: "aarch64" });
-      s.debug[mac] = { hostname: "spark-2935", queued_at: new Date().toISOString() };
-    });
-
-    const res = await app.inject({ method: "GET", url: `/dispatch?mac=${mac}` });
-    expect(res.body).toContain("DEBUG/RESCUE MODE");
-    expect(res.body).toContain("/vmlinuz-aarch64");
-    expect(res.body).toContain("inst.rescue");
-    expect(res.body).toContain("inst.sshd");
-  });
-});
-
-describe("--pxe-boot root device", () => {
-  let testDir: string;
-  let app: FastifyInstance;
-  let state: StateManager;
-  const mac = "aa:bb:cc:dd:ee:ff";
-
-  beforeEach(() => {
-    testDir = join(tmpdir(), `bastion-root-test-${Date.now()}-${Math.random().toString(36).slice(2)}`);
-    mkdirSync(join(testDir, "http"), { recursive: true });
-    mkdirSync(join(testDir, "tftp"), { recursive: true });
-    const result = createApp(createTestConfig(testDir));
-    app = result.app;
-    state = result.state;
-  });
-
-  afterEach(async () => {
-    await app.close();
-    rmSync(testDir, { recursive: true, force: true });
-  });
-
-  it("uses the root device recorded on the machine", async () => {
-    state.update((s) => {
-      s.installed[mac] = {
-        hostname: "worker-1", role: "worker", ip: "10.0.0.50",
-        installed_at: new Date().toISOString(),
-        root_device: "/dev/mapper/otherVG-root",
-        root_args: "rd.lvm.lv=otherVG/root",
-      };
-      s.debug[mac] = { hostname: "worker-1", queued_at: new Date().toISOString(), pxeBoot: true };
-    });
-
-    const res = await app.inject({ method: "GET", url: `/dispatch?mac=${mac}` });
-    expect(res.body).toContain("PXE BOOT (debug)");
-    expect(res.body).toContain("root=/dev/mapper/otherVG-root");
-    expect(res.body).toContain("rd.lvm.lv=otherVG/root");
-    // The old hardcoded layout must not leak back in.
-    expect(res.body).not.toContain("labvg");
-  });
-
-  it("falls back to rescue rather than guessing when no root device is known", async () => {
-    state.update((s) => {
-      s.installed[mac] = {
-        hostname: "spark-2935", role: "worker", ip: "192.168.8.12",
-        installed_at: new Date().toISOString(), arch: "aarch64",
-      };
-      s.debug[mac] = { hostname: "spark-2935", queued_at: new Date().toISOString(), pxeBoot: true };
-    });
-
-    const res = await app.inject({ method: "GET", url: `/dispatch?mac=${mac}` });
-    expect(res.body).toContain("DEBUG/RESCUE MODE");
-    expect(res.body).toContain("no root device is recorded");
-    expect(res.body).toContain("debug-setup.sh");
-    expect(res.body).not.toContain("root=");
-    // And it is still the right architecture.
-    expect(res.body).toContain("/vmlinuz-aarch64");
-  });
-
-  it("records a root device reported from a rescue shell without erasing hardware info", async () => {
-    state.update((s) => {
-      s.discovered[mac] = hardware(mac, { product: "DGX Spark", manufacturer: "NVIDIA", arch: "aarch64" });
-    });
-
-    const res = await app.inject({
-      method: "POST",
-      url: "/api/discover",
-      headers: { "Content-Type": "application/json" },
-      body: JSON.stringify({ mac, root_device: "/dev/nvme0n1p2" }),
-    });
-    expect(res.statusCode).toBe(200);
-
-    const hw = state.load().discovered[mac];
-    expect(hw?.root_device).toBe("/dev/nvme0n1p2");
-    // The partial report must not blank what we already knew.
-    expect(hw?.product).toBe("DGX Spark");
-    expect(hw?.cpu_cores).toBe(4);
-    expect(hw?.arch).toBe("aarch64");
-  });
-});
-
-describe("dnsmasq architecture detection", () => {
-  const conf = renderDnsmasqConf(createTestConfig("/tmp/unused"));
-
-  it("maps DHCP option 93 values to per-architecture bootloaders", () => {
-    // 11 = ARM 64-bit UEFI
-    expect(conf).toContain("dhcp-match=set:efi-arm64,option:client-arch,11");
-    expect(conf).toContain("dhcp-boot=tag:efi-arm64,tag:!ipxe,ipxe-arm64.efi");
-    // 7 / 9 = x64 UEFI, 0 = x86 BIOS
-    expect(conf).toContain("dhcp-match=set:efi-x86_64,option:client-arch,7");
-    expect(conf).toContain("dhcp-match=set:efi-x86_64,option:client-arch,9");
-    expect(conf).toContain("dhcp-match=set:bios,option:client-arch,0");
-  });
-
-  it("matches arm64 UEFI HTTP boot on 19, not 20", () => {
-    // IANA: 19 = arm uefi 64 boot from http, 20 = pc/at bios boot from http.
-    expect(conf).toContain("dhcp-match=set:httpboot-arm64,option:client-arch,19");
-    expect(conf).not.toContain("dhcp-match=set:httpboot-arm64,option:client-arch,20");
-    expect(conf).toContain("dhcp-match=set:httpboot-x86_64,option:client-arch,16");
-  });
-
-  it("offers an arm64 PXE service directive in proxy mode", () => {
-    expect(conf).toContain('pxe-service=tag:!ipxe,ARM64_EFI,"PXE Boot",ipxe-arm64.efi');
-  });
-});

bastion/src/bastion/tests/dispatch.test.ts

@@ -22,6 +22,8 @@ function createTestConfig(testDir: string): BastionConfig {
     dhcpRangeEnd: "",
     ubuntuVersion: "26.04",
     ubuntuMirror: "https://releases.ubuntu.com/26.04",
+    vyosIsoUrl: "https://downloads.vyos.io/rolling/current/generic/vyos-rolling-latest.iso",
+    vyosDefaultPassword: "vyos",
     iface: "eth0",
     serverIp: "10.0.0.1",
     network: "10.0.0.0",

bastion/src/bastion/tests/fixtures/ipxe-x86_64-golden.json

@@ -1,8 +0,0 @@
-{
-  "boot": "#!ipxe\n\necho\necho ============================================\necho   Lab PXE Bastion\necho   Contacting server for instructions...\necho ============================================\necho\n\nchain http://10.0.0.1:8080/dispatch?mac=${net0/mac}\n",
-  "discover": "#!ipxe\n\necho\necho =============================================\necho   Lab PXE Bastion - DISCOVERY MODE\necho   MAC: aa:bb:cc:dd:ee:ff\necho   Collecting hardware info...\necho =============================================\necho\n\nkernel http://10.0.0.1:8080/vmlinuz inst.ks=http://10.0.0.1:8080/discover.ks inst.stage2=https://download.fedoraproject.org/pub/fedora/linux/releases/43/Everything/x86_64/os inst.text nomodeset\ninitrd http://10.0.0.1:8080/initrd.img\nboot\n",
-  "install": "#!ipxe\n\necho\necho =============================================\necho   Lab PXE Bastion - INSTALLING Fedora 43\necho   Target: worker-1\necho   MAC:    aa:bb:cc:dd:ee:ff\necho =============================================\necho\n\nkernel http://10.0.0.1:8080/vmlinuz inst.ks=http://10.0.0.1:8080/ks?mac=aa:bb:cc:dd:ee:ff inst.repo=https://download.fedoraproject.org/pub/fedora/linux/releases/43/Everything/x86_64/os inst.text nomodeset\ninitrd http://10.0.0.1:8080/initrd.img\nboot\n",
-  "debug": "#!ipxe\n\necho\necho =============================================\necho   Lab PXE Bastion - DEBUG/RESCUE MODE\necho   Target: worker-1\necho   MAC:    aa:bb:cc:dd:ee:ff\necho =============================================\necho\n\nkernel http://10.0.0.1:8080/vmlinuz inst.rescue inst.text inst.sshd inst.ks=http://10.0.0.1:8080/debug.ks?mac=aa:bb:cc:dd:ee:ff inst.stage2=https://download.fedoraproject.org/pub/fedora/linux/releases/43/Everything/x86_64/os\ninitrd http://10.0.0.1:8080/initrd.img\nboot\n",
-  "pxeBoot": "#!ipxe\n\necho\necho =============================================\necho   Lab PXE Bastion - PXE BOOT (debug)\necho   Target: worker-1\necho   MAC:    aa:bb:cc:dd:ee:ff\necho   Kernel+initrd from PXE, root from NVMe\necho =============================================\necho\n\nkernel http://10.0.0.1:8080/vmlinuz root=/dev/mapper/labvg-root ro rd.lvm.lv=labvg/root rd.lvm.lv=labvg/swap console=tty0\ninitrd http://10.0.0.1:8080/initrd.img\nboot\n",
-  "localBoot": "#!ipxe\n\necho\necho =============================================\necho   Lab PXE Bastion - worker-1\necho   Already installed, booting from local disk\necho =============================================\necho\nsleep 3\nexit 1\n"
-}

bastion/src/bastion/tests/install-guard.test.ts

@@ -1,194 +0,0 @@
-// Installs must never reach a machine running a vendor OS we cannot restore.
-//
-// This is the guardrail that stops someone reinstalling a DGX Spark at 2am. Rescue is
-// deliberately still allowed for the same machines -- that is the whole point.
-
-import { describe, it, expect, beforeEach, afterEach } from "vitest";
-import { mkdirSync, rmSync } from "node:fs";
-import { join } from "node:path";
-import { tmpdir } from "node:os";
-import type { BastionConfig, BastionState, HardwareInfo } from "@lab/shared";
-import { classifyOnboard } from "@lab/shared";
-import { createApp } from "../src/server.js";
-import { checkInstallAllowed } from "../src/services/install-guard.js";
-import type { FastifyInstance } from "fastify";
-import type { StateManager } from "../src/services/state.js";
-
-// The real machines this exists to protect.
-const SPARK_2935 = "4c:bb:47:7f:29:35";
-const SPARK_3A1C = "48:21:0b:96:3a:1c";
-const ORDINARY = "aa:bb:cc:dd:ee:ff";
-
-function createTestConfig(testDir: string): BastionConfig {
-  return {
-    fedoraVersion: "43", arch: "x86_64", httpPort: 0,
-    timezone: "Europe/London", locale: "en_GB.UTF-8", bastionDir: testDir,
-    domain: "test.local", dhcpMode: "proxy", dhcpRangeStart: "", dhcpRangeEnd: "",
-    ubuntuVersion: "26.04", ubuntuMirror: "https://releases.ubuntu.com/26.04",
-    iface: "eth0", serverIp: "10.0.0.1", network: "10.0.0.0", gateway: "10.0.0.1",
-    sshKeys: [], adminUser: "testadmin", syslogPort: 15514,
-    skipDnsmasq: true, skipArtifacts: true,
-    fedoraMirror: "https://download.fedoraproject.org/pub/fedora/linux/releases/43/Everything/x86_64/os",
-    tftpDir: join(testDir, "tftp"), httpDir: join(testDir, "http"),
-    stateFile: join(testDir, "state.json"),
-  };
-}
-
-function hardware(mac: string, over: Partial<HardwareInfo> = {}): HardwareInfo {
-  return {
-    mac, product: "TestBox", board: "TestBoard", serial: "SN1",
-    manufacturer: "TestCorp", cpu_model: "Test CPU", cpu_cores: 4, memory_gb: 16,
-    arch: "x86_64", disks: [], nics: [],
-    first_seen: new Date().toISOString(), last_seen: new Date().toISOString(),
-    ...over,
-  };
-}
-
-const emptyState = (): BastionState => ({
-  discovered: {}, install_queue: {}, installed: {}, debug: {},
-});
-
-describe("classifyOnboard", () => {
-  it("recognises a DGX Spark from its DMI identity", () => {
-    expect(classifyOnboard({
-      mac: ORDINARY, manufacturer: "NVIDIA", product: "NVIDIA DGX Spark", board: "GB10",
-    })).toEqual({ onboard: "ssh", vendor_os: "dgx-os" });
-  });
-
-  it("recognises the known Sparks even with no DMI recorded", () => {
-    // Neither Spark has hardware info in bastion state today. A DMI-only rule would
-    // fail open on exactly the machines this protects.
-    expect(classifyOnboard({ mac: SPARK_2935 }).onboard).toBe("ssh");
-    expect(classifyOnboard({ mac: SPARK_3A1C }).onboard).toBe("ssh");
-  });
-
-  it("treats ordinary hardware as PXE-installable", () => {
-    expect(classifyOnboard({
-      mac: ORDINARY, manufacturer: "Beelink", product: "SER9", board: "SER9",
-    })).toEqual({ onboard: "pxe" });
-  });
-
-  it("does not override an explicit classification already on the record", () => {
-    expect(classifyOnboard({
-      mac: SPARK_2935, onboard: "pxe",
-    })).toEqual({ onboard: "pxe" });
-  });
-});
-
-describe("checkInstallAllowed", () => {
-  it("refuses a DGX Spark and explains why", () => {
-    const state = emptyState();
-    state.installed[SPARK_2935] = {
-      hostname: "spark-2935", role: "worker", ip: "192.168.8.12",
-      installed_at: new Date().toISOString(), arch: "aarch64",
-    };
-
-    const result = checkInstallAllowed(state, SPARK_2935, "fedora-43");
-    expect(result.allowed).toBe(false);
-    if (result.allowed === false) {
-      expect(result.error).toContain("spark-2935");
-      expect(result.error).toContain("DGX OS");
-      expect(result.error).toContain("provision debug");
-    }
-  });
-
-  it("refuses a Spark that is only known by MAC", () => {
-    expect(checkInstallAllowed(emptyState(), SPARK_3A1C, "fedora-43").allowed).toBe(false);
-  });
-
-  it("allows an ordinary discovered machine", () => {
-    const state = emptyState();
-    state.discovered[ORDINARY] = hardware(ORDINARY);
-    expect(checkInstallAllowed(state, ORDINARY, "fedora-43").allowed).toBe(true);
-  });
-
-  it("allows Fedora on aarch64", () => {
-    const state = emptyState();
-    state.discovered[ORDINARY] = hardware(ORDINARY, { arch: "aarch64" });
-    expect(checkInstallAllowed(state, ORDINARY, "fedora-43").allowed).toBe(true);
-  });
-
-  it("refuses Ubuntu on aarch64 -- no netboot artifacts are published", () => {
-    const state = emptyState();
-    state.discovered[ORDINARY] = hardware(ORDINARY, { arch: "aarch64" });
-    const result = checkInstallAllowed(state, ORDINARY, "ubuntu-26.04");
-    expect(result.allowed).toBe(false);
-    if (result.allowed === false) {
-      expect(result.error).toContain("aarch64");
-    }
-  });
-
-  it("allows Ubuntu on x86_64", () => {
-    const state = emptyState();
-    state.discovered[ORDINARY] = hardware(ORDINARY, { arch: "x86_64" });
-    expect(checkInstallAllowed(state, ORDINARY, "ubuntu-26.04").allowed).toBe(true);
-  });
-});
-
-describe("install route enforces the guard", () => {
-  let testDir: string;
-  let app: FastifyInstance;
-  let state: StateManager;
-
-  beforeEach(() => {
-    testDir = join(tmpdir(), `bastion-guard-test-${Date.now()}-${Math.random().toString(36).slice(2)}`);
-    mkdirSync(join(testDir, "http"), { recursive: true });
-    mkdirSync(join(testDir, "tftp"), { recursive: true });
-    const result = createApp(createTestConfig(testDir));
-    app = result.app;
-    state = result.state;
-  });
-
-  afterEach(async () => {
-    await app.close();
-    rmSync(testDir, { recursive: true, force: true });
-  });
-
-  it("rejects POST /api/install for a Spark and queues nothing", async () => {
-    const res = await app.inject({
-      method: "POST",
-      url: "/api/install",
-      headers: { "Content-Type": "application/json" },
-      body: JSON.stringify({ mac: SPARK_2935, hostname: "spark-2935", role: "worker" }),
-    });
-
-    expect(res.statusCode).toBe(409);
-    expect(JSON.parse(res.body).error).toContain("Refusing to install");
-    expect(state.load().install_queue[SPARK_2935]).toBeUndefined();
-  });
-
-  it("still serves rescue to a Spark -- debug is never guarded", async () => {
-    state.update((s) => {
-      s.installed[SPARK_2935] = {
-        hostname: "spark-2935", role: "worker", ip: "192.168.8.12",
-        installed_at: new Date().toISOString(), arch: "aarch64",
-      };
-      s.debug[SPARK_2935] = { hostname: "spark-2935", queued_at: new Date().toISOString() };
-    });
-
-    const res = await app.inject({ method: "GET", url: `/dispatch?mac=${SPARK_2935}` });
-    expect(res.statusCode).toBe(200);
-    expect(res.body).toContain("DEBUG/RESCUE MODE");
-    expect(res.body).toContain("/vmlinuz-aarch64");
-  });
-
-  it("a Spark that PXE boots unqueued gets discovery, never an install", async () => {
-    const res = await app.inject({ method: "GET", url: `/dispatch?mac=${SPARK_2935}&arch=arm64` });
-    expect(res.body).toContain("DISCOVERY MODE");
-    expect(res.body).not.toContain("INSTALLING");
-  });
-
-  it("still accepts an ordinary machine", async () => {
-    state.update((s) => { s.discovered[ORDINARY] = hardware(ORDINARY); });
-
-    const res = await app.inject({
-      method: "POST",
-      url: "/api/install",
-      headers: { "Content-Type": "application/json" },
-      body: JSON.stringify({ mac: ORDINARY, hostname: "worker-1", role: "worker" }),
-    });
-
-    expect(res.statusCode).toBe(200);
-    expect(state.load().install_queue[ORDINARY]).toBeDefined();
-  });
-});

bastion/src/bastion/tests/ipxe-x86-regression.test.ts

@@ -1,89 +0,0 @@
-// x86_64 iPXE output regression gate.
-//
-// The aarch64 PXE work must not change what an x86_64 machine is served. The golden
-// fixture was dumped from the templates as they stood before that work started, so
-// any diff here is a regression, not an improvement.
-//
-// The one deliberate exception is renderBootIpxe: its chain URL gained
-// `&arch=${buildarch}` so the dispatch endpoint can observe the client's
-// architecture at boot time. That single change is asserted explicitly below
-// rather than being allowed to slip through the byte-for-byte comparison.
-
-import { describe, it, expect } from "vitest";
-import { readFileSync } from "node:fs";
-import { join } from "node:path";
-import { fileURLToPath } from "node:url";
-import { dirname } from "node:path";
-import {
-  renderBootIpxe,
-  renderDiscoverIpxe,
-  renderInstallIpxe,
-  renderDebugIpxe,
-  renderPxeBootDebugIpxe,
-  renderLocalBootIpxe,
-} from "../src/templates/boot.ipxe.js";
-
-const here = dirname(fileURLToPath(import.meta.url));
-const golden = JSON.parse(
-  readFileSync(join(here, "fixtures", "ipxe-x86_64-golden.json"), "utf-8"),
-) as Record<string, string>;
-
-// Exactly the parameters used to dump the fixture.
-const serverIp = "10.0.0.1";
-const httpPort = 8080;
-const mac = "aa:bb:cc:dd:ee:ff";
-const hostname = "worker-1";
-const fedoraVersion = "43";
-const fedoraMirror =
-  "https://download.fedoraproject.org/pub/fedora/linux/releases/43/Everything/x86_64/os";
-
-// The x86_64 LVM layout the fixture was captured with. Before this work the values
-// were hardcoded in the template; they are now supplied by the caller from machine
-// state, so the fixture pins the rendering, not the defaults.
-const x86Root = {
-  rootDevice: "/dev/mapper/labvg-root",
-  rootArgs: "rd.lvm.lv=labvg/root rd.lvm.lv=labvg/swap",
-};
-
-describe("x86_64 iPXE output is unchanged", () => {
-  it("discover script is byte-identical", () => {
-    const rendered = renderDiscoverIpxe({
-      mac, serverIp, httpPort, fedoraMirror, arch: "x86_64",
-    });
-    expect(rendered).toBe(golden["discover"]);
-  });
-
-  it("install script is byte-identical", () => {
-    const rendered = renderInstallIpxe({
-      mac, hostname, serverIp, httpPort, fedoraVersion, fedoraMirror, arch: "x86_64",
-    });
-    expect(rendered).toBe(golden["install"]);
-  });
-
-  it("debug/rescue script is byte-identical", () => {
-    const rendered = renderDebugIpxe({
-      mac, hostname, serverIp, httpPort, fedoraMirror, arch: "x86_64",
-    });
-    expect(rendered).toBe(golden["debug"]);
-  });
-
-  it("--pxe-boot script is byte-identical when state carries the Fedora LVM layout", () => {
-    const rendered = renderPxeBootDebugIpxe({
-      mac, hostname, serverIp, httpPort, arch: "x86_64", ...x86Root,
-    });
-    expect(rendered).toBe(golden["pxeBoot"]);
-  });
-
-  it("local boot script is byte-identical", () => {
-    expect(renderLocalBootIpxe(hostname)).toBe(golden["localBoot"]);
-  });
-
-  it("boot.ipxe differs only by the &arch= chain parameter", () => {
-    const rendered = renderBootIpxe({ serverIp, httpPort });
-    // The sole intended difference.
-    expect(rendered).toBe(golden["boot"].replace(
-      "/dispatch?mac=${net0/mac}",
-      "/dispatch?mac=${net0/mac}&arch=${buildarch}",
-    ));
-  });
-});

bastion/src/bastion/tests/kickstart.test.ts

@@ -96,9 +96,9 @@ describe("renderInstallKickstart", () => {
     expect(ks).toContain("/api/progress");
   });
 
-  it("infra role has /var/lib/rancher partition", () => {
+  it("infra role has 120G /var/lib/rancher partition", () => {
     const ks = renderInstallKickstart(baseParams({ role: "infra" }));
-    expect(ks).toContain("logvol /var/lib/rancher --vgname=labvg --name=rancher --fstype=xfs --size=20480");
+    expect(ks).toContain("logvol /var/lib/rancher --vgname=labvg --name=rancher --fstype=xfs --size=122880");
   });
 
   it("infra role has k3s install", () => {
@@ -106,10 +106,14 @@ describe("renderInstallKickstart", () => {
     expect(ks).toContain("curl -sfL https://get.k3s.io | INSTALL_K3S_SKIP_START=true sh -");
   });
 
-  it("worker role does NOT have /var/lib/rancher partition in fresh install", () => {
+  it("worker role has 120G /var/lib/rancher partition (imageFs must be sized before longhorn --grow)", () => {
     const ks = renderInstallKickstart(baseParams({ role: "worker" }));
-    // Worker should not have the fresh-install rancher partition line
-    expect(ks).not.toContain("logvol /var/lib/rancher --vgname=labvg --name=rancher --fstype=xfs --size=20480");
+    expect(ks).toContain("logvol /var/lib/rancher --vgname=labvg --name=rancher --fstype=xfs --size=122880");
+  });
+
+  it("vanilla role does NOT have /var/lib/rancher partition in fresh install", () => {
+    const ks = renderInstallKickstart(baseParams({ role: "vanilla" }));
+    expect(ks).not.toContain("--name=rancher --fstype=xfs");
   });
 
   it("worker role does NOT have k3s install", () => {

bastion/src/bastion/tests/vyos-bundle.test.ts

@@ -0,0 +1,155 @@
+import { describe, it, expect, vi } from "vitest";
+import type { VyosBundle } from "@lab/shared";
+import { buildVyosConfigSpec } from "../src/templates/vyos-config-spec.js";
+
+/**
+ * A bundle is what makes "one config, two apply paths" true rather than
+ * aspirational: `pulumi up` POSTs the subtree model to a running router, labctl
+ * writes the same model into config.boot during a PXE install. These tests pin
+ * the properties that keep the two honest.
+ */
+const bundle: VyosBundle = {
+  sets: [
+    { path: ["system", "host-name"], value: "vyos001" },
+    { path: ["interfaces", "bonding", "bond0", "address"], value: "192.168.1.252/24" },
+    { path: ["interfaces", "bonding", "bond0", "member", "interface"], value: "eth1", replace: false },
+    { path: ["interfaces", "bonding", "bond0", "vif", "53", "disable"] },
+    { path: ["interfaces", "pppoe", "pppoe0", "authentication", "password"], value: "@secret:pppoePassword" },
+    { path: ["interfaces", "pppoe", "pppoe0", "mtu"], value: "1492" },
+  ],
+  tags: [["interfaces", "bonding", "bond0"], ["interfaces", "ethernet"]],
+};
+
+const build = (hostname: string, extra: Record<string, unknown> = {}) =>
+  buildVyosConfigSpec({
+    hostname,
+    spec: { bundle, ...extra },
+    defaultPassword: "changeme",
+  });
+
+describe("vyos config spec from a Pulumi bundle", () => {
+  it("applies non-secret nodes verbatim, preserving valuelessness and replace:false", () => {
+    const spec = build("vyos001");
+
+    expect(spec.sets).toContainEqual({
+      path: ["interfaces", "bonding", "bond0", "address"],
+      value: "192.168.1.252/24",
+    });
+    // A multi-value node must keep replace:false or the second bond member
+    // overwrites the first.
+    expect(spec.sets).toContainEqual({
+      path: ["interfaces", "bonding", "bond0", "member", "interface"],
+      value: "eth1",
+      replace: false,
+    });
+    // A valueless node must not acquire a value on the way through.
+    expect(spec.sets).toContainEqual({
+      path: ["interfaces", "bonding", "bond0", "vif", "53", "disable"],
+    });
+    expect(spec.tags).toEqual(bundle.tags);
+  });
+
+  it("drops secret-valued nodes instead of installing the sentinel text", () => {
+    const warn = vi.spyOn(console, "warn").mockImplementation(() => {});
+    const spec = build("vyos001");
+
+    const values = spec.sets.map((s) => s.value ?? "");
+    expect(values.some((v) => v.startsWith("@secret:"))).toBe(false);
+    expect(spec.sets.some((s) => s.path.includes("authentication"))).toBe(false);
+    // Silently dropping the WAN credential would leave someone debugging a dead
+    // PPPoE link, so it has to be said out loud.
+    expect(warn).toHaveBeenCalledWith(expect.stringContaining("pulumi up"));
+    warn.mockRestore();
+  });
+
+  it("forces the hostname the install was asked for, not the bundle's", () => {
+    // The bundle is exported from one router and reused for its peer; taking the
+    // hostname from it would put two vyos001s on the network.
+    const spec = build("vyos002");
+    const hostnames = spec.sets.filter(
+      (s) => s.path.length === 2 && s.path[0] === "system" && s.path[1] === "host-name",
+    );
+    expect(hostnames).toEqual([{ path: ["system", "host-name"], value: "vyos002" }]);
+  });
+
+  it("still honours installer inputs, which are not router config", () => {
+    const spec = buildVyosConfigSpec({
+      hostname: "vyos001",
+      spec: { bundle, password: "s3cret", freshConfig: true },
+      defaultPassword: "changeme",
+      disk: "nvme0n1",
+    });
+    expect(spec.password).toBe("s3cret");
+    expect(spec.freshConfig).toBe(true);
+    expect(spec.disk).toBe("/dev/nvme0n1");
+  });
+
+  it("enables the HTTP API at install so Pulumi can manage the router from first boot", () => {
+    const warn = vi.spyOn(console, "warn").mockImplementation(() => {});
+    const spec = buildVyosConfigSpec({
+      hostname: "vyos001",
+      spec: { bundle, apiKey: "k3y", apiListenAddress: "10.0.1.252" },
+      defaultPassword: "changeme",
+    });
+
+    expect(spec.sets).toContainEqual({
+      path: ["service", "https", "api", "keys", "id", "pulumi", "key"],
+      value: "k3y",
+    });
+    expect(spec.sets).toContainEqual({ path: ["service", "https", "api", "rest"] });
+    expect(spec.sets).toContainEqual({
+      path: ["service", "https", "listen-address"],
+      value: "10.0.1.252",
+    });
+    // The key id is a tag node; without this the installer's ConfigTree rejects it.
+    expect(spec.tags).toContainEqual(["service", "https", "api", "keys", "id"]);
+    warn.mockRestore();
+  });
+
+  it("binds the API to the static management address when none is given", () => {
+    const spec = buildVyosConfigSpec({
+      hostname: "vyos001",
+      spec: { apiKey: "k3y", mgmtAddress: "192.168.1.252/24" },
+      defaultPassword: "changeme",
+    });
+    expect(spec.sets).toContainEqual({
+      path: ["service", "https", "listen-address"],
+      value: "192.168.1.252",
+    });
+  });
+
+  it("refuses to enable the API unbound rather than exposing it on the WAN", () => {
+    const warn = vi.spyOn(console, "warn").mockImplementation(() => {});
+    // Management is DHCP, so there is no address to bind at build time. Binding
+    // to everything would put a config-write endpoint on the WAN.
+    const spec = buildVyosConfigSpec({
+      hostname: "vyos001",
+      spec: { apiKey: "k3y", mgmtAddress: "dhcp" },
+      defaultPassword: "changeme",
+    });
+    expect(spec.sets.some((s) => s.path[0] === "service" && s.path[1] === "https")).toBe(false);
+    expect(warn).toHaveBeenCalledWith(expect.stringContaining("has NOT been enabled"));
+    warn.mockRestore();
+  });
+
+  it("does not enable the API when no key is supplied", () => {
+    const spec = buildVyosConfigSpec({
+      hostname: "vyos001",
+      spec: { mgmtAddress: "192.168.1.252/24" },
+      defaultPassword: "changeme",
+    });
+    expect(spec.sets.some((s) => s.path[0] === "service" && s.path[1] === "https")).toBe(false);
+  });
+
+  it("ignores the derived path entirely when a bundle is present", () => {
+    // Belt and braces: even if topology flags reach this far (the CLI rejects
+    // them), the bundle must win rather than merge.
+    const spec = buildVyosConfigSpec({
+      hostname: "vyos001",
+      spec: { bundle, bondMembers: ["eth2", "eth3"], vlans: [{ id: 99, address: "10.9.9.1/24" }] },
+      defaultPassword: "changeme",
+    });
+    expect(spec.sets.some((s) => s.path.includes("99"))).toBe(false);
+    expect(spec.sets.filter((s) => s.value === "eth2" || s.value === "eth3")).toEqual([]);
+  });
+});

bastion/src/bastion/tests/vyos.test.ts

@@ -0,0 +1,548 @@
+import { describe, it, expect, beforeEach, afterEach } from "vitest";
+import { mkdirSync, rmSync } from "node:fs";
+import { join } from "node:path";
+import { tmpdir } from "node:os";
+import type { BastionConfig } from "@lab/shared";
+import type { FastifyInstance } from "fastify";
+import { createApp } from "../src/server.js";
+import type { StateManager } from "../src/services/state.js";
+import { buildVyosConfigSpec } from "../src/templates/vyos-config-spec.js";
+import { renderVyosInstallPy } from "../src/templates/vyos-install.py.js";
+
+function createTestConfig(testDir: string): BastionConfig {
+  return {
+    fedoraVersion: "43",
+    arch: "x86_64",
+    httpPort: 0,
+    timezone: "Europe/London",
+    locale: "en_GB.UTF-8",
+    bastionDir: testDir,
+    domain: "test.local",
+    dhcpMode: "proxy",
+    dhcpRangeStart: "",
+    dhcpRangeEnd: "",
+    ubuntuVersion: "26.04",
+    ubuntuMirror: "https://releases.ubuntu.com/26.04",
+    vyosIsoUrl: "https://example.invalid/vyos.iso",
+    vyosDefaultPassword: "test-pw",
+    iface: "eth0",
+    serverIp: "10.0.0.1",
+    network: "10.0.0.0",
+    gateway: "10.0.0.1",
+    sshKeys: ["ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAITEST lab@test"],
+    adminUser: "testadmin",
+    syslogPort: 15515,
+    skipDnsmasq: true,
+    skipArtifacts: true,
+    fedoraMirror: "https://example.invalid/fedora",
+    tftpDir: join(testDir, "tftp"),
+    httpDir: join(testDir, "http"),
+    stateFile: join(testDir, "state.json"),
+  };
+}
+
+/** Pull the base64 spec back out of the generated Python driver. */
+function decodeSpecFrom(python: string): Record<string, unknown> {
+  const match = /base64\.b64decode\("([^"]+)"\)/.exec(python);
+  if (!match?.[1]) throw new Error("no base64 spec found in generated driver");
+  return JSON.parse(Buffer.from(match[1], "base64").toString("utf-8"));
+}
+
+describe("vyos config spec", () => {
+  it("puts VLANs on the bond when members are given", () => {
+    const spec = buildVyosConfigSpec({
+      hostname: "fw1",
+      defaultPassword: "pw",
+      spec: {
+        mgmtInterface: "eth0",
+        mgmtAddress: "10.0.8.2/24",
+        bondMembers: ["eth2", "eth3"],
+        vlans: [{ id: 10, address: "10.0.10.1/24", description: "k8s" }],
+      },
+    });
+
+    const paths = spec.sets.map((s) => s.path.join(" "));
+    expect(paths).toContain("interfaces bonding bond0 mode");
+    expect(paths).toContain("interfaces bonding bond0 vif 10 address");
+    // VLANs must hang off the bond, not the management NIC.
+    expect(paths).not.toContain("interfaces ethernet eth0 vif 10 address");
+
+    // Bond members are a multi-value node — appending, not replacing, is what
+    // keeps the second member from overwriting the first.
+    const members = spec.sets.filter(
+      (s) => s.path.join(" ") === "interfaces bonding bond0 member interface",
+    );
+    expect(members.map((m) => m.value)).toEqual(["eth2", "eth3"]);
+    expect(members.every((m) => m.replace === false)).toBe(true);
+  });
+
+  it("falls back to VLANs on the management NIC when unbonded", () => {
+    const spec = buildVyosConfigSpec({
+      hostname: "fw2",
+      defaultPassword: "pw",
+      spec: { mgmtInterface: "eth1", vlans: [{ id: 20, address: "10.0.20.1/24" }] },
+    });
+
+    const paths = spec.sets.map((s) => s.path.join(" "));
+    expect(paths).toContain("interfaces ethernet eth1 vif 20 address");
+  });
+
+  it("normalises the target disk to a full /dev path", () => {
+    // find_disks() enumerates with `lsblk -Jbp`, so valid responses are full
+    // paths; a bare name fails valid_responses and re-prompts forever.
+    expect(buildVyosConfigSpec({ hostname: "fw3", defaultPassword: "pw", disk: "/dev/mmcblk0" }).disk)
+      .toBe("/dev/mmcblk0");
+    expect(buildVyosConfigSpec({ hostname: "fw3", defaultPassword: "pw", disk: "mmcblk0" }).disk)
+      .toBe("/dev/mmcblk0");
+    expect(buildVyosConfigSpec({ hostname: "fw3", defaultPassword: "pw" }).disk).toBe("");
+  });
+
+  it("defaults to dhcp on eth0 and never opts into RAID", () => {
+    const spec = buildVyosConfigSpec({ hostname: "fw4", defaultPassword: "pw" });
+    const address = spec.sets.find(
+      (s) => s.path.join(" ") === "interfaces ethernet eth0 address",
+    );
+    expect(address?.value).toBe("dhcp");
+    // The installer's RAID prompt defaults to yes; a second disk must not
+    // silently produce a mirror.
+    expect(spec.raid).toBe(false);
+  });
+});
+
+describe("vyos routes", () => {
+  let testDir: string;
+  let app: FastifyInstance;
+  let state: StateManager;
+  const mac = "aa:bb:cc:11:22:33";
+
+  beforeEach(() => {
+    testDir = join(tmpdir(), `bastion-vyos-test-${Date.now()}-${Math.random().toString(36).slice(2)}`);
+    mkdirSync(join(testDir, "http"), { recursive: true });
+    mkdirSync(join(testDir, "tftp"), { recursive: true });
+
+    const result = createApp(createTestConfig(testDir));
+    app = result.app;
+    state = result.state;
+  });
+
+  afterEach(async () => {
+    await app.close();
+    rmSync(testDir, { recursive: true, force: true });
+  });
+
+  it("dispatches a queued vyos machine to the live-boot script", async () => {
+    state.update((s) => {
+      s.install_queue[mac] = {
+        hostname: "fw1",
+        disk: "/dev/nvme0n1",
+        role: "worker",
+        os: "vyos-rolling",
+        queued_at: new Date().toISOString(),
+      };
+    });
+
+    const response = await app.inject({ method: "GET", url: `/dispatch?mac=${mac}` });
+    expect(response.statusCode).toBe(200);
+
+    expect(response.body).toContain("/vyos-vmlinuz");
+    expect(response.body).toContain("fetch=http://10.0.0.1:0/vyos-filesystem.squashfs");
+    expect(response.body).toContain(`live-config.hooks=http://10.0.0.1:0/vyos/autoinstall.sh?mac=${mac}`);
+
+    // `nonetworking` appears in VyOS's own PXE docs but breaks the hook fetch,
+    // and console=ttyS0 costs 30s per systemd phase on boards with no UART.
+    expect(response.body).not.toContain("nonetworking");
+    expect(response.body).not.toContain("console=ttyS0");
+  });
+
+  it("serves a hook that fetches and executes the install driver", async () => {
+    const response = await app.inject({ method: "GET", url: `/vyos/autoinstall.sh?mac=${mac}` });
+    expect(response.statusCode).toBe(200);
+    expect(response.body).toContain(`/vyos/install.py?mac=${mac}`);
+    expect(response.body).toContain("python3 /tmp/vyos-install.py");
+  });
+
+  it("bakes the machine's config into the generated install driver", async () => {
+    state.update((s) => {
+      s.install_queue[mac] = {
+        hostname: "fw1",
+        disk: "/dev/nvme0n1",
+        role: "worker",
+        os: "vyos-rolling",
+        queued_at: new Date().toISOString(),
+        vyos: {
+          mgmtInterface: "eth0",
+          mgmtAddress: "10.0.8.2/24",
+          bondMembers: ["eth2", "eth3"],
+          vlans: [{ id: 10, address: "10.0.10.1/24" }],
+          password: "s3cret",
+        },
+      };
+    });
+
+    const response = await app.inject({ method: "GET", url: `/vyos/install.py?mac=${mac}` });
+    expect(response.statusCode).toBe(200);
+
+    // Builds config from the image's own default so the vyos-config-version
+    // trailer matches and first boot skips migrations.
+    expect(response.body).toContain("/opt/vyatta/etc/config.boot.default");
+    expect(response.body).toContain("/usr/libexec/vyos/op_mode/image_installer.py");
+    // "complete" is what moves the machine out of the install queue.
+    expect(response.body).toContain('report("complete"');
+
+    const spec = decodeSpecFrom(response.body);
+    expect(spec["hostname"]).toBe("fw1");
+    expect(spec["password"]).toBe("s3cret");
+    expect(spec["disk"]).toBe("/dev/nvme0n1");
+
+    const paths = (spec["sets"] as Array<{ path: string[] }>).map((s) => s.path.join(" "));
+    expect(paths).toContain("interfaces bonding bond0 vif 10 address");
+    expect(paths).toContain("system host-name");
+  });
+
+  it("falls back to the bastion default password when none is set", async () => {
+    state.update((s) => {
+      s.install_queue[mac] = {
+        hostname: "fw9",
+        disk: "",
+        role: "worker",
+        os: "vyos-rolling",
+        queued_at: new Date().toISOString(),
+      };
+    });
+
+    const response = await app.inject({ method: "GET", url: `/vyos/install.py?mac=${mac}` });
+    const spec = decodeSpecFrom(response.body);
+    expect(spec["password"]).toBe("test-pw");
+    // Empty disk means "accept the installer's first-disk default".
+    expect(spec["disk"]).toBe("");
+  });
+});
+
+describe("vyos hw-id pinning", () => {
+  it("emits hw-id for the mgmt interface and each bond member", () => {
+    // Discovery sees enp2s0/enp1s0f0np0 under Fedora, but VyOS enumerates its
+    // own eth<N>. Pinning by MAC is what makes the mapping deterministic.
+    const spec = buildVyosConfigSpec({
+      hostname: "fw1",
+      defaultPassword: "pw",
+      spec: {
+        mgmtInterface: "eth2",
+        bondMembers: ["eth0", "eth1"],
+        hwIds: {
+          eth2: "64:62:66:25:96:47",
+          eth0: "64:62:66:25:96:45",
+          eth1: "64:62:66:25:96:46",
+        },
+      },
+    });
+
+    const hw = spec.sets.filter((s) => s.path[s.path.length - 1] === "hw-id");
+    expect(hw.map((s) => [s.path[2], s.value])).toEqual([
+      ["eth2", "64:62:66:25:96:47"],
+      ["eth0", "64:62:66:25:96:45"],
+      ["eth1", "64:62:66:25:96:46"],
+    ]);
+  });
+
+  it("omits hw-id entirely when no mapping is given", () => {
+    const spec = buildVyosConfigSpec({ hostname: "fw1", defaultPassword: "pw" });
+    expect(spec.sets.some((s) => s.path.includes("hw-id"))).toBe(false);
+  });
+});
+
+describe("vyos management VLAN", () => {
+  it("puts the mgmt VLAN on the PXE port while the bond carries routed VLANs", () => {
+    // Trunked PXE port: boots untagged on the VLAN the bastion serves, stays
+    // reachable on the tagged management VLAN.
+    const spec = buildVyosConfigSpec({
+      hostname: "vyos001",
+      defaultPassword: "pw",
+      spec: {
+        mgmtInterface: "eth2",
+        mgmtAddress: "dhcp",
+        mgmtVlan: { id: 3, address: "192.168.3.4/24", description: "kvm" },
+        bondMembers: ["eth0", "eth1"],
+        vlans: [{ id: 2, address: "192.168.8.2/23" }],
+      },
+    });
+
+    const paths = spec.sets.map((s) => s.path.join(" "));
+    expect(paths).toContain("interfaces ethernet eth2 vif 3 address");
+    expect(paths).toContain("interfaces bonding bond0 vif 2 address");
+    // The mgmt VLAN must not land on the bond.
+    expect(paths).not.toContain("interfaces bonding bond0 vif 3 address");
+    expect(spec.tags.map((t) => t.join(" "))).toContain("interfaces ethernet eth2 vif");
+  });
+});
+
+describe("vyos VRRP HA", () => {
+  const haSpec = {
+    mgmtInterface: "eth2",
+    mgmtAddress: "dhcp",
+    bondMembers: ["eth0", "eth1"],
+    bondAddress: "192.168.1.252/24",
+    bondVrrp: "192.168.1.254/24",
+    vrrpPriority: 200,
+    vlans: [
+      { id: 3, address: "192.168.3.4/24", vrrp: "192.168.3.254/24" },
+      { id: 200, address: "192.168.2.252/24" },   // no VIP on this one
+    ],
+  };
+
+  it("emits a vrrp group per VIP with vrid = VLAN id and dotted vif interface", () => {
+    const spec = buildVyosConfigSpec({ hostname: "fw1", defaultPassword: "pw", spec: haSpec });
+    const paths = spec.sets.map((s) => `${s.path.join(" ")}${s.value !== undefined ? "=" + s.value : ""}`);
+
+    expect(paths).toContain("interfaces bonding bond0 address=192.168.1.252/24");
+    // untagged bond group: vrid 1, interface bond0 itself
+    expect(paths).toContain("high-availability vrrp group native interface=bond0");
+    expect(paths).toContain("high-availability vrrp group native vrid=1");
+    // address is a tag node -- VIP is the final path segment, no value
+    expect(paths).toContain("high-availability vrrp group native address 192.168.1.254/24");
+    // VLAN group: vrid = VLAN id, dotted vif
+    expect(paths).toContain("high-availability vrrp group vlan3 interface=bond0.3");
+    expect(paths).toContain("high-availability vrrp group vlan3 vrid=3");
+    expect(paths).toContain("high-availability vrrp group vlan3 address 192.168.3.254/24");
+    // VLAN without a VIP gets no group
+    expect(paths.some((p) => p.includes("group vlan200"))).toBe(false);
+  });
+
+  it("applies the box-wide priority and one sync group over all groups", () => {
+    const spec = buildVyosConfigSpec({ hostname: "fw1", defaultPassword: "pw", spec: haSpec });
+    const prio = spec.sets.filter((s) => s.path[s.path.length - 1] === "priority"
+      && s.path[0] === "high-availability");
+    expect(prio).toHaveLength(2);
+    expect(prio.every((s) => s.value === "200")).toBe(true);
+
+    // sync group binds the pair: all groups fail over together
+    const members = spec.sets.filter(
+      (s) => s.path.join(" ") === "high-availability vrrp sync-group MAIN member",
+    );
+    expect(members.map((m) => m.value)).toEqual(["native", "vlan3"]);
+    expect(members.every((m) => m.replace === false)).toBe(true);
+  });
+
+  it("emits no high-availability nodes when no VIPs are given", () => {
+    const spec = buildVyosConfigSpec({
+      hostname: "fw1",
+      defaultPassword: "pw",
+      spec: { bondMembers: ["eth0", "eth1"], vlans: [{ id: 3, address: "192.168.3.4/24" }] },
+    });
+    expect(spec.sets.some((s) => s.path[0] === "high-availability")).toBe(false);
+  });
+});
+
+describe("pickLargestInitrd", async () => {
+  const { pickLargestInitrd } = await import("../src/main.js");
+
+  // Verbatim from `xorriso -lsl /live/` on vyos-2026.08.05-0033-rolling.
+  const realListing = `total 8
+-r--r--r--    1 0        0           22255 Aug  5 01:33 'filesystem.packages'
+-r--r--r--    1 0        0               6 Aug  5 01:33 'filesystem.packages-remove'
+-r--r--r--    1 0        0        541192192 Aug  5 01:33 'filesystem.squashfs'
+-r--r--r--    1 0        0        50352547 Aug  5 01:33 'initrd.img'
+-r--r--r--    1 0        0        50352547 Aug  5 01:33 'initrd.img-6.18.41-vyos'
+-r--r--r--    1 0        0              20 Aug  5 01:33 'packages.txt'
+-r--r--r--    1 0        0         9135104 Aug  2 19:54 'vmlinuz'
+-r--r--r--    1 0        0         9135104 Aug  2 19:54 'vmlinuz-6.18.41-vyos'
+`;
+
+  it("picks a full-size initrd from a real nightly listing", () => {
+    expect(pickLargestInitrd(realListing)).toEqual({ name: "initrd.img", size: 50352547 });
+  });
+
+  it("ignores 0-byte decoys and symlinks (which report link size, not target size)", () => {
+    const listing = `total 8
+-r--r--r--    1 0        0               0 Aug  5 01:33 'initrd.img'
+lrwxrwxrwx    1 0        0              24 Aug  5 01:33 'initrd.img-link' -> 'initrd.img-6.18.41-vyos'
+-r--r--r--    1 0        0        50352547 Aug  5 01:33 'initrd.img-6.18.41-vyos'
+`;
+    expect(pickLargestInitrd(listing)).toEqual({ name: "initrd.img-6.18.41-vyos", size: 50352547 });
+  });
+
+  it("returns undefined when only decoys exist", () => {
+    expect(pickLargestInitrd("-r--r--r-- 1 0 0 0 Aug 5 01:33 'initrd.img'\n")).toBeUndefined();
+  });
+});
+
+describe("vyos fedora-parity features", () => {
+  it("computes reportAddress from a static mgmt address, empty for dhcp", () => {
+    const staticSpec = buildVyosConfigSpec({
+      hostname: "fw1", defaultPassword: "pw",
+      spec: { mgmtAddress: "192.168.8.2/23" },
+    });
+    expect(staticSpec.reportAddress).toBe("192.168.8.2");
+
+    const dhcpSpec = buildVyosConfigSpec({ hostname: "fw1", defaultPassword: "pw" });
+    expect(dhcpSpec.reportAddress).toBe("");
+  });
+
+  it("defaults freshConfig off (reinstall preserves the on-disk config)", () => {
+    expect(buildVyosConfigSpec({ hostname: "fw1", defaultPassword: "pw" }).freshConfig).toBe(false);
+    expect(buildVyosConfigSpec({
+      hostname: "fw1", defaultPassword: "pw", spec: { freshConfig: true },
+    }).freshConfig).toBe(true);
+  });
+
+  it("driver streams logs to /api/log and reports 'ready at' on completion", async () => {
+    const testDir = join(tmpdir(), `bastion-vyos-parity-${Date.now()}`);
+    mkdirSync(join(testDir, "http"), { recursive: true });
+    mkdirSync(join(testDir, "tftp"), { recursive: true });
+    const { app: parityApp, state: parityState } = createApp(createTestConfig(testDir));
+    try {
+      parityState.update((s) => {
+        s.install_queue["aa:bb:cc:44:55:66"] = {
+          hostname: "fw9", disk: "/dev/vda", role: "vanilla",
+          os: "vyos-rolling", queued_at: new Date().toISOString(),
+        };
+      });
+      const response = await parityApp.inject({
+        method: "GET", url: "/vyos/install.py?mac=aa:bb:cc:44:55:66",
+      });
+      expect(response.body).toContain("/api/log");
+      expect(response.body).toContain('"lines": batch');
+      expect(response.body).toContain('report("complete", "ready at %s"');
+      expect(response.body).toContain("ensure_network_boot_first");
+      expect(response.body).toContain("lab-provisioned");
+      expect(response.body).toContain('ROLE = "vanilla"');
+    } finally {
+      await parityApp.close();
+      rmSync(testDir, { recursive: true, force: true });
+    }
+  });
+
+  it("complete with 'ready at' records installed.ip for a vyos machine", async () => {
+    const testDir = join(tmpdir(), `bastion-vyos-complete-${Date.now()}`);
+    mkdirSync(join(testDir, "http"), { recursive: true });
+    mkdirSync(join(testDir, "tftp"), { recursive: true });
+    const { app: cApp, state: cState } = createApp(createTestConfig(testDir));
+    try {
+      const mac2 = "aa:bb:cc:77:88:99";
+      cState.update((s) => {
+        s.install_queue[mac2] = {
+          hostname: "fw1", disk: "/dev/vda", role: "vanilla",
+          os: "vyos-rolling", queued_at: new Date().toISOString(),
+        };
+      });
+      const response = await cApp.inject({
+        method: "POST", url: "/api/progress",
+        headers: { "Content-Type": "application/json" },
+        body: JSON.stringify({ mac: mac2, stage: "complete", detail: "ready at 192.168.8.2" }),
+      });
+      expect(response.statusCode).toBe(200);
+      const installed = cState.load().installed[mac2];
+      expect(installed?.ip).toBe("192.168.8.2");
+      expect(installed?.os).toBe("vyos-rolling");
+    } finally {
+      await cApp.close();
+      rmSync(testDir, { recursive: true, force: true });
+    }
+  });
+});
+
+describe("vyos installer prompt coverage", () => {
+  // Every interactive prompt image_installer.py can emit, copied verbatim from
+  // the MSG_* constants (including the reinstall-only search_previous_installation
+  // ones). An unanswered prompt does not fail loudly -- the installer simply
+  // blocks on stdin until the driver's stall timeout, which is how the reinstall
+  // path silently hung for 15 minutes in the VM test.
+  const PROMPTS: Record<string, string> = {
+    continue: "Would you like to continue? [y/N] ",
+    imageName: "What would you like to name this image? (Default: 1.5-rolling) ",
+    password: 'Please enter a password for the "vyos" user: ',
+    passwordConfirm: 'Please confirm password for the "vyos" user: ',
+    console: "What console should be used by default? (K: KVM, S: Serial)? (Default: K) ",
+    raidConfigure: "Would you like to configure RAID-1 mirroring? [Y/n] ",
+    raidFoundDisks: "Would you like to configure RAID-1 mirroring on them? [Y/n] ",
+    raidChooseDisks: "Would you like to choose two disks for RAID-1 mirroring? [Y/n] ",
+    diskSelect: "Which one should be used for installation? (Default: /dev/vda) ",
+    diskConfirm: "Installation will delete all data on the drive. Continue? [y/N] ",
+    raidConfirm: "Installation will delete all data on both drives. Continue? [y/N] ",
+    rootSizeAll: "Would you like to use all the free space on the drive? [Y/n] ",
+    bootConfig: "Which file would you like as boot config? ",
+    copyData: "Would you like to copy data to the new image? [Y/n] ",
+    chooseCopyData: "From which image would you like to save config information? ",
+    copyEncData: "Would you like to copy the encrypted config to the new image? [Y/n] ",
+    chooseCopyEncData: "From which image would you like to copy the encrypted config? ",
+  };
+
+  it("answers every installer prompt exactly once", () => {
+    const { execFileSync } = require("node:child_process") as typeof import("node:child_process");
+    const { writeFileSync, unlinkSync, mkdtempSync } = require("node:fs") as typeof import("node:fs");
+
+    // Skip cleanly where python3 is unavailable (same spirit as the
+    // ksvalidator-backed kickstart test).
+    try {
+      execFileSync("python3", ["--version"], { stdio: "pipe" });
+    } catch {
+      return;
+    }
+
+    const spec = buildVyosConfigSpec({
+      hostname: "fw1", defaultPassword: "pw", disk: "/dev/vda",
+    });
+    const driver = renderVyosInstallPy({
+      spec, mac: "aa:bb:cc:11:22:33", serverIp: "10.0.0.1", httpPort: 8080, role: "vanilla",
+    });
+
+    const dir = mkdtempSync(join(tmpdir(), "vyos-rules-"));
+    const driverPath = join(dir, "driver.py");
+    const checkPath = join(dir, "check.py");
+    writeFileSync(driverPath, driver);
+    writeFileSync(checkPath, `
+import importlib.util, json, sys
+spec = importlib.util.spec_from_file_location("drv", ${JSON.stringify(driverPath)})
+drv = importlib.util.module_from_spec(spec); spec.loader.exec_module(drv)
+rules = drv.build_rules()
+prompts = json.loads(sys.argv[1])
+out = {}
+for label, text in prompts.items():
+    out[label] = len([r for p, r in rules if p.search(text.encode())])
+print(json.dumps(out))
+`);
+
+    try {
+      const stdout = execFileSync("python3", [checkPath, JSON.stringify(PROMPTS)], {
+        encoding: "utf-8", stdio: ["pipe", "pipe", "pipe"],
+      });
+      const counts = JSON.parse(stdout) as Record<string, number>;
+      const unanswered = Object.entries(counts).filter(([, n]) => n !== 1);
+      expect(unanswered).toEqual([]);
+    } finally {
+      try { unlinkSync(driverPath); unlinkSync(checkPath); } catch { /* best effort */ }
+      try { rmSync(dir, { recursive: true, force: true }); } catch { /* best effort */ }
+    }
+  });
+});
+
+describe("vyos boot NIC pinning", () => {
+  it("pins the boot interface by MAC via BOOTIF", async () => {
+    // Without this, live-boot picks the first *connected* NIC. On the VP2440
+    // the SFP+ pair links before the copper PXE port, so live-boot tried the
+    // fiber ports (no DHCP), timed out 15s each, and failed with "Unable to
+    // find a live file system on the network".
+    const testDir = join(tmpdir(), `bastion-vyos-bootif-${Date.now()}`);
+    mkdirSync(join(testDir, "http"), { recursive: true });
+    mkdirSync(join(testDir, "tftp"), { recursive: true });
+    const { app: a, state: st } = createApp(createTestConfig(testDir));
+    try {
+      const m = "64:62:66:25:96:47";
+      st.update((s) => {
+        s.install_queue[m] = {
+          hostname: "vyos001", disk: "/dev/mmcblk0", role: "vanilla",
+          os: "vyos-rolling", queued_at: new Date().toISOString(),
+        };
+      });
+      const res = await a.inject({ method: "GET", url: `/dispatch?mac=${m}` });
+      // live-boot's Device_from_bootif() expects 01-<mac with dashes>
+      expect(res.body).toContain("BOOTIF=01-64-62-66-25-96-47");
+      // and it must be on the kernel line, before fetch= is attempted
+      const kernelLine = res.body.split("\n").find((l) => l.startsWith("kernel "));
+      expect(kernelLine).toContain("BOOTIF=01-64-62-66-25-96-47");
+      expect(kernelLine).toContain("fetch=");
+    } finally {
+      await a.close();
+      rmSync(testDir, { recursive: true, force: true });
+    }
+  });
+});

bastion/src/cli/src/api/client.ts

@@ -90,6 +90,7 @@ export class LabdClient {
 
   async installMachine(opts: {
     mac: string; hostname: string; disk?: string; role?: string; os?: string;
+    vyos?: import("@lab/shared").VyosInstallSpec;
   }): Promise<{ status: string; data?: unknown; error?: string }> {
     return this.request("POST", "/api/machines/install", { body: opts });
   }
@@ -110,7 +111,6 @@ export class LabdClient {
     memory_gb?: number; arch?: string;
     disks?: Array<{ name: string; size_gb: number; model: string }>;
     nics?: Array<{ name: string; mac: string; state: string }>;
-    root_device?: string; root_args?: string;
   }): Promise<{ status: string; error?: string }> {
     return this.request("POST", "/api/machines/discover", { body: data });
   }

bastion/src/cli/src/commands/debug.ts

@@ -8,7 +8,6 @@ import { join } from "node:path";
 import { Command } from "commander";
 import type { BastionState } from "@lab/shared";
 import { getLabdClient } from "../api/config.js";
-import { ROOT_DEVICE_PROBE, parseRootProbe } from "../utils/hardware-probe.js";
 
 /** Resolve a target (hostname, MAC, or IP) to {mac, hostname, ip} from state. */
 function resolveTarget(
@@ -45,54 +44,6 @@ function resolveTarget(
   return null;
 }
 
-/** The local admin account to SSH as (root is not usable — it has no key here). */
-function sshUser(): string {
-  const adminUser = process.env["SUDO_USER"] ?? process.env["USER"] ?? "";
-  return adminUser === "root" ? "" : adminUser;
-}
-
-/** Common ssh arguments, ending with user@host. Null when there is no usable user. */
-function sshBaseArgs(ip: string): string[] | null {
-  const user = sshUser();
-  if (user === "") return null;
-
-  const sudoUser = process.env["SUDO_USER"];
-  const realHome = sudoUser !== undefined ? join("/home", sudoUser) : homedir();
-  const sshKey = ["id_ed25519", "id_rsa", "id_ecdsa"]
-    .map((name) => join(realHome, ".ssh", name))
-    .find((k) => existsSync(k));
-
-  return [
-    "-o", "StrictHostKeyChecking=no",
-    "-o", "UserKnownHostsFile=/dev/null",
-    "-o", "ConnectTimeout=10",
-    ...(sshKey !== undefined ? ["-i", sshKey] : []),
-    `${user}@${ip}`,
-  ];
-}
-
-/**
- * Run a shell script on the target as root and return its stdout, or null.
- *
- * The script goes over stdin rather than the command line so it can contain quotes
- * without a second round of shell escaping. `sudo -n` fails fast instead of hanging on
- * a password prompt that would then eat the script.
- */
-function sshCapture(ip: string, script: string): string | null {
-  const base = sshBaseArgs(ip);
-  if (base === null) return null;
-  try {
-    return execFileSync("ssh", [...base, "sudo", "-n", "sh", "-s"], {
-      input: script,
-      encoding: "utf-8",
-      stdio: ["pipe", "pipe", "pipe"],
-      timeout: 30_000,
-    });
-  } catch {
-    return null;
-  }
-}
-
 export function registerDebugCommand(parent: Command): void {
   parent
     .command("debug <target>")
@@ -120,31 +71,6 @@ export function registerDebugCommand(parent: Command): void {
       }
 
       const { mac, hostname, ip } = resolved;
-
-      // --pxe-boot needs a root= for the installed system. If the machine is still
-      // reachable, observe it now rather than assuming a disk layout: a wrong root=
-      // leaves the machine unbootable. If it isn't reachable, dispatch falls back to
-      // rescue and the operator reports the real one from there.
-      if (opts.pxeBoot === true && ip !== "") {
-        const known = state.installed[mac]?.root_device ?? state.discovered[mac]?.root_device;
-        if (known === undefined || known === "") {
-          console.log(`No root device recorded for ${hostname}. Probing over SSH...`);
-          const probe = sshCapture(ip, ROOT_DEVICE_PROBE);
-          const root = probe === null ? {} : parseRootProbe(probe);
-          if (root.root_device !== undefined) {
-            console.log(`  root=${root.root_device}${root.root_args !== undefined ? ` ${root.root_args}` : ""}`);
-            try {
-              await client.discoverMachine({ mac, ...root });
-            } catch (err) {
-              console.error(`  Could not record it: ${err instanceof Error ? err.message : String(err)}`);
-            }
-          } else {
-            console.log("  Probe failed. Booting rescue instead; report the root device with:");
-            console.log("    curl http://<bastion>:8080/debug-setup.sh | bash");
-          }
-        }
-      }
-
       console.log(`Queuing debug mode for ${hostname} (${mac})...`);
 
       try {
@@ -160,15 +86,32 @@ export function registerDebugCommand(parent: Command): void {
 
       // Try SSH reboot into PXE
       if (ip !== "") {
-        const base = sshBaseArgs(ip);
-        if (base !== null) {
-          console.log(`\nAttempting SSH reboot into PXE (${sshUser()}@${ip})...`);
+        const adminUser = process.env["SUDO_USER"] ?? process.env["USER"] ?? "";
+        const effectiveUser = adminUser === "root" ? "" : adminUser;
+
+        if (effectiveUser !== "") {
+          console.log(`\nAttempting SSH reboot into PXE (${effectiveUser}@${ip})...`);
+
+          const sudoUser = process.env["SUDO_USER"];
+          const realHome = sudoUser !== undefined ? join("/home", sudoUser) : homedir();
+          const keyPaths = [
+            join(realHome, ".ssh", "id_ed25519"),
+            join(realHome, ".ssh", "id_rsa"),
+            join(realHome, ".ssh", "id_ecdsa"),
+          ];
+          const sshKey = keyPaths.find(k => existsSync(k));
+
+          const sshArgs = [
+            "-o", "StrictHostKeyChecking=no",
+            "-o", "UserKnownHostsFile=/dev/null",
+            "-o", "ConnectTimeout=10",
+            ...(sshKey !== undefined ? ["-i", sshKey] : []),
+            `${effectiveUser}@${ip}`,
+            'PXE_ENTRY=$(sudo efibootmgr | grep -iE "pxe|network|ipv4" | head -1 | grep -oP "Boot\\K[0-9A-F]+"); if [ -n "$PXE_ENTRY" ]; then sudo efibootmgr --bootnext "$PXE_ENTRY" && echo "PXE set as next boot" && sudo reboot; else echo "No PXE boot entry found, rebooting anyway..." && sudo reboot; fi',
+          ];
 
           try {
-            execFileSync("ssh", [
-              ...base,
-              'PXE_ENTRY=$(sudo efibootmgr | grep -iE "pxe|network|ipv4" | head -1 | grep -oP "Boot\\K[0-9A-F]+"); if [ -n "$PXE_ENTRY" ]; then sudo efibootmgr --bootnext "$PXE_ENTRY" && echo "PXE set as next boot" && sudo reboot; else echo "No PXE boot entry found, rebooting anyway..." && sudo reboot; fi',
-            ], { stdio: "inherit" });
+            execFileSync("ssh", sshArgs, { stdio: "inherit" });
           } catch {
             // SSH connection closing during reboot is expected
           }

bastion/src/cli/src/commands/install.ts

@@ -1,10 +1,61 @@
 // CLI command: provision install
 // Queue a discovered machine for OS installation via labd.
 
-import { Command, Option } from "commander";
+import { readFileSync } from "node:fs";
+import { Command, Option, InvalidArgumentError } from "commander";
 import { isValidOsId, SUPPORTED_OS, SUPPORTED_ROLES, ROLE_REGISTRY } from "@lab/shared";
+import type { VyosBundle, VyosInstallSpec, VyosVlanSpec } from "@lab/shared";
 import { getLabdClient } from "../api/config.js";
 
+/**
+ * Load one router's config out of a Pulumi-rendered bundle.
+ *
+ * The bundle is produced by `kubernetes-deployment` (npm run vyos:bundle) and
+ * holds every router it manages, keyed by name. Selecting by hostname here is
+ * what keeps bring-up and `pulumi up` describing the same box: labctl replays
+ * the declared config rather than deriving its own.
+ */
+export function loadVyosBundle(path: string, hostname: string): VyosBundle {
+  let parsed: { version?: number; routers?: Record<string, VyosBundle> };
+  try {
+    parsed = JSON.parse(readFileSync(path, "utf8"));
+  } catch (e) {
+    throw new InvalidArgumentError(`Cannot read VyOS bundle ${path}: ${(e as Error).message}`);
+  }
+  if (parsed.version !== 1) {
+    throw new InvalidArgumentError(
+      `VyOS bundle ${path} has version ${parsed.version ?? "<none>"}; this labctl understands 1`,
+    );
+  }
+  const router = parsed.routers?.[hostname];
+  if (router === undefined) {
+    const known = Object.keys(parsed.routers ?? {}).join(", ") || "<none>";
+    throw new InvalidArgumentError(
+      `VyOS bundle ${path} has no entry for "${hostname}" (has: ${known})`,
+    );
+  }
+  return router;
+}
+
+/** Parse a repeated --vlan flag: "<id>:<cidr>[:<description>]". */
+export function parseVlan(value: string, previous: VyosVlanSpec[] = []): VyosVlanSpec[] {
+  const parts = value.split(":");
+  const id = Number(parts[0]);
+  const address = parts[1] ?? "";
+  // InvalidArgumentError makes commander print a clean message instead of
+  // dumping a stack trace at the operator.
+  if (!Number.isInteger(id) || id < 1 || id > 4094) {
+    throw new InvalidArgumentError(`Invalid VLAN id in "${value}" (expected 1-4094)`);
+  }
+  if (!address.includes("/")) {
+    throw new InvalidArgumentError(
+      `Invalid VLAN address in "${value}" (expected CIDR, e.g. 10.0.10.1/24)`,
+    );
+  }
+  const description = parts.slice(2).join(":");
+  return [...previous, { id, address, ...(description ? { description } : {}) }];
+}
+
 function roleTable(): string {
   const lines: string[] = ["", "Available roles:"];
   for (const r of ROLE_REGISTRY) {
@@ -15,6 +66,38 @@ function roleTable(): string {
   return lines.join("\n");
 }
 
+/** Parse a repeated --vlan-vip flag: "<id>:<cidr>" — VRRP VIP for a --vlan entry. */
+export function parseVlanVip(
+  value: string,
+  previous: Record<number, string> = {},
+): Record<number, string> {
+  const index = value.indexOf(":");
+  const id = Number(index === -1 ? Number.NaN : value.slice(0, index));
+  const cidr = index === -1 ? "" : value.slice(index + 1).trim();
+  if (!Number.isInteger(id) || id < 1 || id > 4094 || !cidr.includes("/")) {
+    throw new InvalidArgumentError(
+      `Invalid VLAN VIP "${value}" (expected <id>:<cidr>, e.g. 3:192.168.3.254/24)`,
+    );
+  }
+  return { ...previous, [id]: cidr };
+}
+
+/** Parse a repeated --vyos-hwid flag: "<iface>=<mac>". */
+export function parseHwId(
+  value: string,
+  previous: Record<string, string> = {},
+): Record<string, string> {
+  const index = value.indexOf("=");
+  const iface = index === -1 ? "" : value.slice(0, index).trim();
+  const mac = index === -1 ? "" : value.slice(index + 1).trim().toLowerCase();
+  if (iface === "" || !/^([0-9a-f]{2}:){5}[0-9a-f]{2}$/.test(mac)) {
+    throw new InvalidArgumentError(
+      `Invalid hw-id "${value}" (expected <iface>=<mac>, e.g. eth2=64:62:66:25:96:47)`,
+    );
+  }
+  return { ...previous, [iface]: mac };
+}
+
 export function registerInstallCommand(parent: Command): void {
   parent
     .command("install <mac> <hostname>")
@@ -24,10 +107,51 @@ export function registerInstallCommand(parent: Command): void {
     .addOption(new Option("--role <role>", "Machine role (see below)").choices([...SUPPORTED_ROLES]).default("worker"))
     .addOption(new Option("--os <os>", "Operating system").choices([...SUPPORTED_OS]).default("fedora-43"))
     .option("--disk <device>", "Target disk device (auto-detect if omitted)")
+    .option("--vyos-mgmt <iface>", "VyOS: untagged interface the machine PXE boots from (default eth0)")
+    .option("--vyos-mgmt-address <addr>", "VyOS: CIDR for the management interface, or 'dhcp' (default dhcp)")
+    .option("--vyos-bond <ifaces>", "VyOS: comma-separated LACP bond members (must exclude the PXE NIC)")
+    .option("--vyos-bond-address <cidr>", "VyOS: address on the untagged bond (trunk native VLAN)")
+    .option("--vyos-bond-vrrp <cidr>", "VyOS: VRRP VIP floated on the untagged bond")
+    .option("--vlan-vip <id:cidr>", "VyOS: VRRP VIP for a --vlan entry (repeatable)", parseVlanVip)
+    .option("--vyos-vrrp-priority <n>", "VyOS: VRRP priority for all groups on this box (higher = master)")
+    .option("--vyos-mgmt-vlan <id:cidr[:desc]>", "VyOS: tagged management VLAN on the PXE port")
+    .option("--vlan <id:cidr[:desc]>", "VyOS: tagged VLAN sub-interface on the bond (repeatable)", parseVlan)
+    .option("--vyos-password <password>", "VyOS: password for the 'vyos' user")
+    .option("--vyos-hwid <iface=mac>", "VyOS: pin an interface name to a MAC via hw-id (repeatable)", parseHwId)
+    .option("--vyos-fresh-config", "VyOS: on reinstall, overwrite the preserved config with the generated one")
+    .option(
+      "--vyos-bundle <path>",
+      "VyOS: apply a Pulumi-rendered bundle verbatim (kubernetes-deployment/infra/vyos/vyos-bundle.json). " +
+        "Replaces the derived --vyos-bond/--vlan/... config; secret values are left unset for `pulumi up`.",
+    )
+    .option(
+      "--vyos-api-key <key>",
+      "VyOS: enable the HTTP API with this key so Pulumi can manage the router from first boot",
+    )
+    .option(
+      "--vyos-api-listen <addr>",
+      "VyOS: address the HTTP API binds to (default: the static management address). " +
+        "Required when management is DHCP; the API is never bound to all interfaces.",
+    )
     .action(async (mac: string, hostname: string, opts: {
       role: string;
       os: string;
       disk?: string;
+      vyosMgmt?: string;
+      vyosMgmtAddress?: string;
+      vyosBond?: string;
+      vyosBondAddress?: string;
+      vyosBondVrrp?: string;
+      vlan?: VyosVlanSpec[];
+      vlanVip?: Record<number, string>;
+      vyosVrrpPriority?: string;
+      vyosMgmtVlan?: string;
+      vyosPassword?: string;
+      vyosHwid?: Record<string, string>;
+      vyosFreshConfig?: boolean;
+      vyosBundle?: string;
+      vyosApiKey?: string;
+      vyosApiListen?: string;
     }) => {
       if (!isValidOsId(opts.os)) {
         console.error(`Unknown OS: ${opts.os}. Supported: ${SUPPORTED_OS.join(", ")}`);
@@ -39,6 +163,89 @@ export function registerInstallCommand(parent: Command): void {
         process.exit(1);
       }
 
+      const bondMembers = opts.vyosBond !== undefined && opts.vyosBond !== ""
+        ? opts.vyosBond.split(",").map((s) => s.trim()).filter((s) => s.length > 0)
+        : [];
+
+      // Attach --vlan-vip entries to their --vlan definitions. A VIP for a VLAN
+      // that was never defined is a typo that would otherwise vanish silently.
+      const vips = opts.vlanVip ?? {};
+      const vlans = (opts.vlan ?? []).map((v) =>
+        vips[v.id] !== undefined ? { ...v, vrrp: vips[v.id] as string } : v,
+      );
+      for (const id of Object.keys(vips)) {
+        if (!vlans.some((v) => String(v.id) === id)) {
+          console.error(`--vlan-vip ${id}:... has no matching --vlan ${id}:... entry`);
+          process.exit(1);
+        }
+      }
+
+      const vrrpPriority = opts.vyosVrrpPriority !== undefined && opts.vyosVrrpPriority !== ""
+        ? Number(opts.vyosVrrpPriority)
+        : undefined;
+      if (vrrpPriority !== undefined
+          && (!Number.isInteger(vrrpPriority) || vrrpPriority < 1 || vrrpPriority > 255)) {
+        console.error(`--vyos-vrrp-priority must be an integer 1-255 (got ${opts.vyosVrrpPriority})`);
+        process.exit(1);
+      }
+      const vyos: VyosInstallSpec = {
+        ...(opts.vyosMgmt !== undefined && opts.vyosMgmt !== ""
+          ? { mgmtInterface: opts.vyosMgmt } : {}),
+        ...(opts.vyosMgmtAddress !== undefined && opts.vyosMgmtAddress !== ""
+          ? { mgmtAddress: opts.vyosMgmtAddress } : {}),
+        ...(bondMembers.length > 0 ? { bondMembers } : {}),
+        ...(opts.vyosBondAddress !== undefined && opts.vyosBondAddress !== ""
+          ? { bondAddress: opts.vyosBondAddress } : {}),
+        ...(opts.vyosBondVrrp !== undefined && opts.vyosBondVrrp !== ""
+          ? { bondVrrp: opts.vyosBondVrrp } : {}),
+        ...(vrrpPriority !== undefined ? { vrrpPriority } : {}),
+        ...(vlans.length > 0 ? { vlans } : {}),
+        ...(opts.vyosPassword !== undefined && opts.vyosPassword !== ""
+          ? { password: opts.vyosPassword } : {}),
+        ...(opts.vyosHwid !== undefined && Object.keys(opts.vyosHwid).length > 0
+          ? { hwIds: opts.vyosHwid } : {}),
+        ...(opts.vyosMgmtVlan !== undefined && opts.vyosMgmtVlan !== ""
+          ? { mgmtVlan: parseVlan(opts.vyosMgmtVlan)[0] as VyosVlanSpec } : {}),
+        ...(opts.vyosFreshConfig === true ? { freshConfig: true } : {}),
+        ...(opts.vyosBundle !== undefined && opts.vyosBundle !== ""
+          ? { bundle: loadVyosBundle(opts.vyosBundle, hostname) } : {}),
+        ...(opts.vyosApiKey !== undefined && opts.vyosApiKey !== ""
+          ? { apiKey: opts.vyosApiKey } : {}),
+        ...(opts.vyosApiListen !== undefined && opts.vyosApiListen !== ""
+          ? { apiListenAddress: opts.vyosApiListen } : {}),
+      };
+      const hasVyosOptions = Object.keys(vyos).length > 0;
+
+      // A bundle already describes the whole router. Accepting derived topology
+      // flags alongside it would silently discard them (the bundle wins in
+      // buildVyosConfigSpec), so say so rather than appear to honour both.
+      if (vyos.bundle !== undefined) {
+        const derived = ["mgmtInterface", "mgmtAddress", "bondMembers", "bondAddress",
+                         "bondVrrp", "vrrpPriority", "vlans", "mgmtVlan"] as const;
+        const conflicting = derived.filter((k) => vyos[k] !== undefined);
+        if (conflicting.length > 0) {
+          console.error(
+            `--vyos-bundle describes the whole router; these would be ignored: ${conflicting.join(", ")}`,
+          );
+          console.error("Remove them, or change the bundle in kubernetes-deployment and re-render.");
+          process.exit(1);
+        }
+      }
+
+      if (hasVyosOptions && !opts.os.startsWith("vyos")) {
+        console.error(`VyOS options require --os vyos-rolling (got --os ${opts.os})`);
+        process.exit(1);
+      }
+
+      // Firmware PXE cannot run over LACP, so the NIC that boots the installer
+      // must stay out of the bond — otherwise the next reinstall has no path in.
+      const mgmt = vyos.mgmtInterface ?? "eth0";
+      if (bondMembers.includes(mgmt)) {
+        console.error(`--vyos-bond must not include the PXE/management interface "${mgmt}"`);
+        console.error("PXE cannot boot over an LACP bond; keep that NIC unbonded.");
+        process.exit(1);
+      }
+
       try {
         const result = await getLabdClient().installMachine({
           mac,
@@ -46,11 +253,14 @@ export function registerInstallCommand(parent: Command): void {
           role: opts.role,
           os: opts.os,
           ...(opts.disk ? { disk: opts.disk } : {}),
+          ...(hasVyosOptions ? { vyos } : {}),
         });
 
         console.log(JSON.stringify(result, null, 2));
         console.log("");
-        const osLabel = opts.os.startsWith("ubuntu") ? "Ubuntu" : "Fedora";
+        const osLabel = opts.os.startsWith("ubuntu")
+          ? "Ubuntu"
+          : opts.os.startsWith("vyos") ? "VyOS" : "Fedora";
         console.log(`Power on the machine to start ${osLabel} installation.`);
 
         const roleInfo = ROLE_REGISTRY.find(r => r.name === opts.role);

bastion/src/cli/src/commands/recheck.ts

@@ -4,7 +4,6 @@
 import type { Command } from "commander";
 import { sshExec } from "@lab/modules";
 import { getLabdClient } from "../api/config.js";
-import { ROOT_DEVICE_PROBE } from "../utils/hardware-probe.js";
 
 const BOLD = "\x1b[1m";
 const GREEN = "\x1b[0;32m";
@@ -25,9 +24,7 @@ const HW_COLLECT_SCRIPT = [
   'N=$(grep -c "^processor" /proc/cpuinfo 2>/dev/null || echo 0)',
   'R=$(awk "/MemTotal/ {printf \\"%d\\", \\$2/1024/1024}" /proc/meminfo 2>/dev/null || echo 0)',
   'A=$(uname -m)',
-  // Root filesystem, so --pxe-boot has a root= to use instead of assuming our layout.
-  ROOT_DEVICE_PROBE,
-  'printf \'{"product":"%s","board":"%s","serial":"%s","manufacturer":"%s","cpu_model":"%s","cpu_cores":%s,"memory_gb":%s,"arch":"%s","root_device":"%s","root_args":"%s"}\\n\' "$P" "$B" "$S" "$M" "$C" "$N" "$R" "$A" "$RD" "$RA"',
+  'printf \'{"product":"%s","board":"%s","serial":"%s","manufacturer":"%s","cpu_model":"%s","cpu_cores":%s,"memory_gb":%s,"arch":"%s"}\\n\' "$P" "$B" "$S" "$M" "$C" "$N" "$R" "$A"',
 ].join("; ");
 
 export function registerRecheckCommand(parent: Command): void {
@@ -47,11 +44,14 @@ export function registerRecheckCommand(parent: Command): void {
       }
 
       // Build list of machines to check
-      const targets: Array<{ mac: string; hostname: string; ip: string }> = [];
+      const targets: Array<{ mac: string; hostname: string; ip: string; sshUser: string }> = [];
+      const userIsDefault = opts.user === "root";
       for (const [mac, info] of Object.entries(state.installed)) {
         if (!info.ip) continue;
         if (opts.target && info.hostname !== opts.target && mac !== opts.target) continue;
-        targets.push({ mac, hostname: info.hostname, ip: info.ip });
+        // VyOS boxes only have the "vyos" login; honor an explicit --user.
+        const sshUser = userIsDefault && (info.os ?? "").startsWith("vyos") ? "vyos" : opts.user;
+        targets.push({ mac, hostname: info.hostname, ip: info.ip, sshUser });
       }
 
       if (targets.length === 0) {
@@ -64,12 +64,12 @@ export function registerRecheckCommand(parent: Command): void {
       let updated = 0;
       let failed = 0;
 
-      for (const { mac, hostname, ip } of targets) {
+      for (const { mac, hostname, ip, sshUser } of targets) {
         process.stdout.write(`  ${hostname.padEnd(24)} ${DIM}(${ip})${RESET}  `);
 
         try {
           const t0 = Date.now();
-          const result = await sshExec(ip, opts.user, HW_COLLECT_SCRIPT, SSH_OPTS);
+          const result = await sshExec(ip, sshUser, HW_COLLECT_SCRIPT, SSH_OPTS);
           const elapsed = Date.now() - t0;
           if (result.exitCode !== 0) {
             console.log(`${RED}SSH failed (exit ${result.exitCode}, ${elapsed}ms)${RESET}`);
@@ -84,10 +84,7 @@ export function registerRecheckCommand(parent: Command): void {
           const cpu = hwData.cpu_model || "?";
           const cores = hwData.cpu_cores || "?";
           const mem = hwData.memory_gb || "?";
-          const root = typeof hwData.root_device === "string" && hwData.root_device !== ""
-            ? `, root=${hwData.root_device}`
-            : "";
-          console.log(`${GREEN}OK${RESET} ${DIM}${cpu}, ${cores} cores, ${mem}GB${root}${RESET}`);
+          console.log(`${GREEN}OK${RESET} ${DIM}${cpu}, ${cores} cores, ${mem}GB${RESET}`);
           updated++;
         } catch (err) {
           console.log(`${RED}FAIL${RESET} ${DIM}${err instanceof Error ? err.message : String(err)}${RESET}`);

bastion/src/cli/src/commands/reprovision.ts

@@ -24,12 +24,12 @@ function roleTable(): string {
 function resolveTarget(
   target: string,
   state: BastionState,
-): { mac: string; hostname: string; ip: string } | null {
+): { mac: string; hostname: string; ip: string; os?: string } | null {
   const normalized = target.toLowerCase().replace(/-/g, ":");
 
   if (state.installed[normalized]) {
     const info = state.installed[normalized];
-    return { mac: normalized, hostname: info.hostname, ip: info.ip };
+    return { mac: normalized, hostname: info.hostname, ip: info.ip, ...(info.os !== undefined ? { os: info.os } : {}) };
   }
 
   if (state.discovered[normalized]) {
@@ -38,13 +38,13 @@ function resolveTarget(
 
   for (const [mac, info] of Object.entries(state.installed)) {
     if (info.hostname === target || info.hostname.startsWith(target + ".")) {
-      return { mac, hostname: info.hostname, ip: info.ip };
+      return { mac, hostname: info.hostname, ip: info.ip, ...(info.os !== undefined ? { os: info.os } : {}) };
     }
   }
 
   for (const [mac, info] of Object.entries(state.installed)) {
     if (info.ip === target) {
-      return { mac, hostname: info.hostname, ip: info.ip };
+      return { mac, hostname: info.hostname, ip: info.ip, ...(info.os !== undefined ? { os: info.os } : {}) };
     }
   }
 
@@ -60,10 +60,12 @@ export function registerReprovisionCommand(parent: Command): void {
     .addOption(new Option("--role <role>", "Machine role (see below)").choices([...SUPPORTED_ROLES]).default("worker"))
     .addOption(new Option("--os <os>", "Operating system").choices([...SUPPORTED_OS]).default("fedora-43"))
     .option("--disk <device>", "Target disk device (auto-detect if omitted)")
+    .option("--user <user>", "SSH user for the reboot (default: vyos for VyOS machines, else current user)")
     .action(async (target: string, hostnameOverride: string | undefined, opts: {
       role: string;
       os: string;
       disk?: string;
+      user?: string;
     }) => {
       if (!isValidOsId(opts.os)) {
         console.error(`Unknown OS: ${opts.os}. Supported: ${SUPPORTED_OS.join(", ")}`);
@@ -123,7 +125,11 @@ export function registerReprovisionCommand(parent: Command): void {
         return;
       }
 
-      const adminUser = process.env["SUDO_USER"] ?? process.env["USER"] ?? "";
+      // SSH user: explicit flag > the machine's current OS (VyOS boxes only
+      // have the "vyos" login) > the invoking user.
+      const currentOsIsVyos = (resolved.os ?? "").startsWith("vyos");
+      const adminUser = opts.user
+        ?? (currentOsIsVyos ? "vyos" : (process.env["SUDO_USER"] ?? process.env["USER"] ?? ""));
       const effectiveUser = adminUser === "root" ? "" : adminUser;
 
       if (effectiveUser === "") {

bastion/src/cli/src/utils/hardware-probe.ts

@@ -1,59 +0,0 @@
-// Shell snippets for observing a machine's hardware over SSH.
-//
-// Pure shell + awk, no Python: these run on whatever the target happens to be,
-// including a minimal rescue environment.
-
-/**
- * Report the root filesystem and any dracut arguments needed to assemble it.
- *
- * Emits two lines:
- *   ROOT_DEVICE=<device>
- *   ROOT_ARGS=<args>
- *
- * Used by `--pxe-boot`, which boots the installed system with a kernel and initrd from
- * the network. Getting root= wrong there leaves the machine unbootable, so this observes
- * the machine rather than assuming our Fedora LVM layout.
- *
- * Device form is chosen for stability across reboots: LVM logical volumes keep their
- * /dev/mapper path, anything else is reported by UUID, which survives device renumbering.
- */
-export const ROOT_DEVICE_PROBE = [
-  'RD=$(findmnt -no SOURCE / 2>/dev/null | head -1)',
-  'RA=""',
-  'RT=$(lsblk -no TYPE "$RD" 2>/dev/null | head -1)',
-  'if [ "$RT" = "lvm" ]; then',
-  '  VGLV=$(lvs --noheadings -o vg_name,lv_name "$RD" 2>/dev/null | awk \'{print $1"/"$2}\')',
-  '  [ -n "$VGLV" ] && RA="rd.lvm.lv=$VGLV"',
-  // Swap must be assembled too or resume= stalls the boot waiting for it.
-  '  SW=$(awk \'NR>1 {print $1; exit}\' /proc/swaps 2>/dev/null)',
-  '  if [ -n "$SW" ] && [ "$(lsblk -no TYPE "$SW" 2>/dev/null | head -1)" = "lvm" ]; then',
-  '    SWVGLV=$(lvs --noheadings -o vg_name,lv_name "$SW" 2>/dev/null | awk \'{print $1"/"$2}\')',
-  '    [ -n "$SWVGLV" ] && [ "$SWVGLV" != "$VGLV" ] && RA="$RA rd.lvm.lv=$SWVGLV"',
-  '  fi',
-  'elif [ -n "$RD" ]; then',
-  '  U=$(findmnt -no UUID / 2>/dev/null | head -1)',
-  '  [ -n "$U" ] && RD="UUID=$U"',
-  'fi',
-  'printf \'ROOT_DEVICE=%s\\nROOT_ARGS=%s\\n\' "$RD" "$RA"',
-].join("; ");
-
-export interface RootInfo {
-  root_device?: string;
-  root_args?: string;
-}
-
-/** Parse the ROOT_DEVICE/ROOT_ARGS lines emitted by ROOT_DEVICE_PROBE. */
-export function parseRootProbe(stdout: string): RootInfo {
-  const out: RootInfo = {};
-  for (const line of stdout.split("\n")) {
-    const trimmed = line.trim();
-    if (trimmed.startsWith("ROOT_DEVICE=")) {
-      const v = trimmed.slice("ROOT_DEVICE=".length).trim();
-      if (v !== "") out.root_device = v;
-    } else if (trimmed.startsWith("ROOT_ARGS=")) {
-      const v = trimmed.slice("ROOT_ARGS=".length).trim();
-      if (v !== "") out.root_args = v;
-    }
-  }
-  return out;
-}

bastion/src/cli/tests/install-vyos.test.ts

@@ -0,0 +1,35 @@
+// Tests for VyOS install option parsing.
+
+import { describe, it, expect } from "vitest";
+import { parseVlan } from "../src/commands/install.js";
+
+describe("parseVlan", () => {
+  it("parses id and CIDR", () => {
+    expect(parseVlan("10:10.0.10.1/24")).toEqual([{ id: 10, address: "10.0.10.1/24" }]);
+  });
+
+  it("accumulates across repeated flags", () => {
+    const first = parseVlan("10:10.0.10.1/24");
+    const both = parseVlan("20:10.0.20.1/24", first);
+    expect(both).toHaveLength(2);
+    expect(both[1]).toEqual({ id: 20, address: "10.0.20.1/24" });
+  });
+
+  it("keeps a description, including one containing colons", () => {
+    expect(parseVlan("30:10.0.30.1/24:mgmt:secondary")).toEqual([
+      { id: 30, address: "10.0.30.1/24", description: "mgmt:secondary" },
+    ]);
+  });
+
+  it("rejects an address that is not CIDR", () => {
+    // A bare address would produce a VyOS config that fails to commit on first
+    // boot, long after the operator has stopped watching.
+    expect(() => parseVlan("10:10.0.10.1")).toThrow(/CIDR/);
+  });
+
+  it("rejects out-of-range and non-numeric VLAN ids", () => {
+    expect(() => parseVlan("0:10.0.10.1/24")).toThrow(/1-4094/);
+    expect(() => parseVlan("4095:10.0.10.1/24")).toThrow(/1-4094/);
+    expect(() => parseVlan("abc:10.0.10.1/24")).toThrow(/1-4094/);
+  });
+});

bastion/src/labd/src/routes/bastions.ts

@@ -10,6 +10,7 @@ import type { FastifyInstance } from "fastify";
 import type { DbClient } from "../server.js";
 import { bastionRegistry } from "../services/bastion-registry.js";
 import { generateRequestId } from "@lab/shared";
+import type { VyosInstallSpec } from "@lab/shared";
 
 const COMMAND_TIMEOUT_MS = 15_000;
 
@@ -163,9 +164,9 @@ export function registerBastionRoutes(app: FastifyInstance, db: DbClient): void
 
   // Queue install — route to correct bastion by MAC
   app.post<{
-    Body: { mac?: string; hostname?: string; disk?: string; role?: string; os?: string };
+    Body: { mac?: string; hostname?: string; disk?: string; role?: string; os?: string; vyos?: VyosInstallSpec };
   }>("/api/machines/install", async (request, reply) => {
-    const { mac, hostname, disk, role, os } = request.body ?? {};
+    const { mac, hostname, disk, role, os, vyos } = request.body ?? {};
     if (!mac || !hostname) {
       return reply.code(400).send({ error: "mac and hostname are required" });
     }
@@ -183,6 +184,7 @@ export function registerBastionRoutes(app: FastifyInstance, db: DbClient): void
           const result = await sendCommand(all[0]!.bastionId, {
             type: "command-install",
             mac, hostname, disk: disk ?? "", role: role ?? "infra", os: os ?? "fedora-43",
+            ...(vyos ? { vyos } : {}),
           });
           return reply.code(result.status === "ok" ? 200 : 500).send(result);
         } catch (err) {
@@ -196,6 +198,7 @@ export function registerBastionRoutes(app: FastifyInstance, db: DbClient): void
       const result = await sendCommand(bastion.bastionId, {
         type: "command-install",
         mac, hostname, disk: disk ?? "", role: role ?? "infra", os: os ?? "fedora-43",
+        ...(vyos ? { vyos } : {}),
       });
       return reply.code(result.status === "ok" ? 200 : 500).send(result);
     } catch (err) {
@@ -299,7 +302,6 @@ export function registerBastionRoutes(app: FastifyInstance, db: DbClient): void
       memory_gb?: number; arch?: string;
       disks?: Array<{ name: string; size_gb: number; model: string }>;
       nics?: Array<{ name: string; mac: string; state: string }>;
-      root_device?: string; root_args?: string;
     };
   }>("/api/machines/discover", async (request, reply) => {
     const data = request.body ?? {};

bastion/src/modules/modules/k3s/src/groups/host-prep.ts

@@ -1,21 +1,23 @@
-// Host preparation: kernel modules, sysctl, swap, firewall, SELinux.
+// Host preparation: kernel modules, sysctl, swap, storage, firewall, SELinux.
 
 import type { OperationContext, OperationResult, OperationGroup } from "../types.js";
 import { runSequential } from "../utils.js";
 import { loadKernelModules } from "../operations/kernel-modules.js";
 import { applyCisHardening } from "../operations/sysctl.js";
-import { disableSwap } from "../operations/swap.js";
+import { enableSwap } from "../operations/swap.js";
+import { growRancherLv } from "../operations/rancher-storage.js";
 import { disableFirewall } from "../operations/firewall.js";
 import { setSelinuxPermissive } from "../operations/selinux.js";
 import { enableIscsi } from "../operations/iscsi.js";
 
 export const hostPrepGroup: OperationGroup = {
   name: "host-prep",
-  description: "Prepare host for k3s: kernel modules, sysctl, swap, firewall, SELinux, iSCSI",
+  description: "Prepare host for k3s: kernel modules, sysctl, swap, imageFs sizing, firewall, SELinux, iSCSI",
   operations: [
     { name: "Load kernel modules", fn: loadKernelModules },
     { name: "Apply CIS sysctl", fn: applyCisHardening },
-    { name: "Disable swap", fn: disableSwap },
+    { name: "Enable swap", fn: enableSwap },
+    { name: "Grow rancher LV", fn: growRancherLv },
     { name: "Disable firewall", fn: disableFirewall },
     { name: "Set SELinux permissive", fn: setSelinuxPermissive },
     { name: "Enable iSCSI", fn: enableIscsi },

bastion/src/modules/modules/k3s/src/install.ts

@@ -215,7 +215,9 @@ echo "  Using network device: \$DEFAULT_DEV"
 
 KUBECONFIG=/etc/rancher/k3s/k3s.yaml cilium install \\
   --set kubeProxyReplacement=true \\
-  --set ipam.mode=kubernetes \\
+  --set ipam.mode=cluster-pool \\
+  --set ipam.operator.clusterPoolIPv4PodCIDRList='{10.42.0.0/16}' \\
+  --set ipam.operator.clusterPoolIPv4MaskSize=24 \\
   --set devices="\$DEFAULT_DEV" \\
   --set nodePort.directRoutingDevice="\$DEFAULT_DEV"
 

bastion/src/modules/modules/k3s/src/operations/cilium.ts

@@ -47,7 +47,9 @@ export const installCilium: Operation = async (ctx): Promise<OperationResult> =>
   const installResult = await ctx.ssh.exec(
     `KUBECONFIG=/etc/rancher/k3s/k3s.yaml cilium install \
       --set kubeProxyReplacement=true \
-      --set ipam.mode=kubernetes \
+      --set ipam.mode=cluster-pool \
+      --set ipam.operator.clusterPoolIPv4PodCIDRList='{10.42.0.0/16}' \
+      --set ipam.operator.clusterPoolIPv4MaskSize=24 \
       --set k8sServiceHost=127.0.0.1 \
       --set k8sServicePort=6444 \
       --set cni.exclusive=false \

bastion/src/modules/modules/k3s/src/operations/index.ts

@@ -1,6 +1,7 @@
 export { loadKernelModules } from "./kernel-modules.js";
 export { applyCisHardening } from "./sysctl.js";
-export { disableSwap } from "./swap.js";
+export { enableSwap } from "./swap.js";
+export { growRancherLv } from "./rancher-storage.js";
 export { enableIscsi } from "./iscsi.js";
 export { disableFirewall } from "./firewall.js";
 export { setSelinuxPermissive } from "./selinux.js";

bastion/src/modules/modules/k3s/src/operations/rancher-storage.ts

@@ -0,0 +1,48 @@
+// Grow the labvg/rancher LV (k3s image store / imageFs) to 120G.
+// 2026-08 incident: the original 20G LV sat at 85% used from steady-state
+// images alone, so one ~5G image pull tripped imagefs eviction and evicted
+// unrelated pods. Fresh installs are sized at 120G by the kickstart; this op
+// covers nodes installed before that change and vanilla nodes converted to
+// k8s later. Never removes or shrinks anything — if the VG lacks free space
+// (e.g. a longhorn --grow LV consumed it), it reports and moves on.
+
+import type { Operation, OperationResult } from "../types.js";
+import { sshOpts } from "../utils.js";
+
+const RANCHER_LV = "labvg/rancher";
+const TARGET_MIB = 122880; // 120G
+
+export const growRancherLv: Operation = async (ctx): Promise<OperationResult> => {
+  const lv = await ctx.ssh.exec(
+    `lvs --noheadings --units m --nosuffix -o lv_size ${RANCHER_LV} 2>/dev/null || true`,
+    sshOpts(ctx),
+  );
+  const sizeMib = Number.parseFloat(lv.stdout.trim());
+  if (Number.isNaN(sizeMib)) {
+    return { success: true, changed: false, message: "No labvg/rancher LV — imageFs shares /var, skipping" };
+  }
+  if (sizeMib >= TARGET_MIB) {
+    return { success: true, changed: false, message: `rancher LV already ${Math.round(sizeMib / 1024)}G` };
+  }
+
+  const vg = await ctx.ssh.exec(`vgs --noheadings --units m --nosuffix -o vg_free labvg`, sshOpts(ctx));
+  const freeMib = Number.parseFloat(vg.stdout.trim());
+  const neededMib = TARGET_MIB - sizeMib;
+  if (Number.isNaN(freeMib) || freeMib < neededMib) {
+    return {
+      success: true,
+      changed: false,
+      message: `VG labvg has ${Math.floor((Number.isNaN(freeMib) ? 0 : freeMib) / 1024)}G free — ` +
+        `need ${Math.ceil(neededMib / 1024)}G to grow rancher LV to 120G (manual LV rebuild required)`,
+    };
+  }
+
+  await ctx.ssh.exec(`lvextend -L ${TARGET_MIB}m /dev/${RANCHER_LV}`, sshOpts(ctx));
+  await ctx.ssh.exec(`xfs_growfs /var/lib/rancher`, sshOpts(ctx));
+
+  return {
+    success: true,
+    changed: true,
+    message: `rancher LV grown ${Math.round(sizeMib / 1024)}G → 120G`,
+  };
+};

bastion/src/modules/modules/k3s/src/operations/swap.ts

@@ -1,22 +1,40 @@
-// Disable swap (CIS requirement for k3s).
+// Enable swap so memory pressure spills to disk instead of OOM-killing.
+// kubelet runs with failSwapOn=false (k3s default); zram stays the fast tier,
+// the labvg-swap LV is the overflow tier. Replaces the old CIS-style
+// disableSwap op — a kernel OOM kill of a node daemon is worse than slow swap.
 
 import type { Operation, OperationResult } from "../types.js";
 import { sshOpts } from "../utils.js";
 
-export const disableSwap: Operation = async (ctx): Promise<OperationResult> => {
-  const check = await ctx.ssh.exec("swapon --show --noheadings", sshOpts(ctx));
-  const active = check.stdout.trim().length > 0;
+const SWAP_DEV = "/dev/mapper/labvg-swap";
 
-  if (active) {
-    await ctx.ssh.exec("swapoff -a", sshOpts(ctx));
+export const enableSwap: Operation = async (ctx): Promise<OperationResult> => {
+  const lv = await ctx.ssh.exec(`test -b ${SWAP_DEV} && echo yes || echo no`, sshOpts(ctx));
+  if (lv.stdout.trim() !== "yes") {
+    return { success: true, changed: false, message: "No labvg-swap LV — skipping swap enable" };
   }
 
-  // Remove swap entries from fstab permanently
-  await ctx.ssh.exec("sed -i '/\\sswap\\s/d' /etc/fstab", sshOpts(ctx));
+  const active = await ctx.ssh.exec(
+    `grep -q "^$(readlink -f ${SWAP_DEV}) " /proc/swaps && echo on || echo off`,
+    sshOpts(ctx),
+  );
+  const wasOff = active.stdout.trim() !== "on";
+
+  if (wasOff) {
+    // Format if the LV was never (or wrongly) initialised, then activate
+    await ctx.ssh.exec(`blkid ${SWAP_DEV} | grep -q 'TYPE="swap"' || mkswap ${SWAP_DEV}`, sshOpts(ctx));
+    await ctx.ssh.exec(`swapon ${SWAP_DEV}`, sshOpts(ctx));
+  }
+
+  // Persist across reboots (idempotent)
+  await ctx.ssh.exec(
+    `grep -q "labvg-swap" /etc/fstab || echo "${SWAP_DEV} none swap defaults 0 0" >> /etc/fstab`,
+    sshOpts(ctx),
+  );
 
   return {
     success: true,
-    changed: active,
-    message: active ? "Swap disabled" : "Swap already disabled",
+    changed: wasOff,
+    message: wasOff ? "LV swap enabled" : "LV swap already active",
   };
 };

bastion/src/modules/modules/k3s/tests/operations.test.ts

@@ -72,31 +72,97 @@ describe("applyCisHardening", () => {
 
 // --- Swap ---
 
-import { disableSwap } from "../src/operations/swap.js";
+import { enableSwap } from "../src/operations/swap.js";
 
-describe("disableSwap", () => {
-  it("disables active swap", async () => {
+describe("enableSwap", () => {
+  it("activates LV swap when present but off", async () => {
     const ctx = mockCtx();
     ctx.ssh.exec
-      .mockResolvedValueOnce(stdout("/dev/sda2 partition 2G"))  // swap active
-      .mockResolvedValueOnce(OK)   // swapoff
-      .mockResolvedValueOnce(OK);  // sed fstab
+      .mockResolvedValueOnce(stdout("yes"))  // LV exists
+      .mockResolvedValueOnce(stdout("off"))  // not in /proc/swaps
+      .mockResolvedValueOnce(OK)             // blkid || mkswap
+      .mockResolvedValueOnce(OK)             // swapon
+      .mockResolvedValueOnce(OK);            // fstab entry
 
-    const result = await disableSwap(ctx);
+    const result = await enableSwap(ctx);
     expect(result.success).toBe(true);
     expect(result.changed).toBe(true);
-    expectCommand(ctx.ssh, "swapoff -a");
+    expectCommand(ctx.ssh, "swapon /dev/mapper/labvg-swap");
   });
 
-  it("is idempotent when swap already off", async () => {
+  it("is idempotent when LV swap already active", async () => {
     const ctx = mockCtx();
     ctx.ssh.exec
-      .mockResolvedValueOnce(stdout(""))  // no swap
-      .mockResolvedValueOnce(OK);          // sed fstab (always runs)
+      .mockResolvedValueOnce(stdout("yes"))  // LV exists
+      .mockResolvedValueOnce(stdout("on"))   // already in /proc/swaps
+      .mockResolvedValueOnce(OK);            // fstab entry (always ensured)
 
-    const result = await disableSwap(ctx);
+    const result = await enableSwap(ctx);
     expect(result.changed).toBe(false);
-    expectNoCommand(ctx.ssh, "swapoff");
+    expectNoCommand(ctx.ssh, "swapon /dev/mapper/labvg-swap");
+  });
+
+  it("skips when no labvg-swap LV exists", async () => {
+    const ctx = mockCtx();
+    ctx.ssh.exec.mockResolvedValueOnce(stdout("no")); // LV missing
+
+    const result = await enableSwap(ctx);
+    expect(result.success).toBe(true);
+    expect(result.changed).toBe(false);
+    expectNoCommand(ctx.ssh, "swapon");
+  });
+});
+
+// --- Rancher LV (imageFs sizing) ---
+
+import { growRancherLv } from "../src/operations/rancher-storage.js";
+
+describe("growRancherLv", () => {
+  it("grows a 20G LV to 120G when the VG has space", async () => {
+    const ctx = mockCtx();
+    ctx.ssh.exec
+      .mockResolvedValueOnce(stdout("  20480.00"))   // lv_size
+      .mockResolvedValueOnce(stdout("  747807.00"))  // vg_free
+      .mockResolvedValueOnce(OK)                     // lvextend
+      .mockResolvedValueOnce(OK);                    // xfs_growfs
+
+    const result = await growRancherLv(ctx);
+    expect(result.success).toBe(true);
+    expect(result.changed).toBe(true);
+    expectCommand(ctx.ssh, "lvextend -L 122880m /dev/labvg/rancher");
+    expectCommand(ctx.ssh, "xfs_growfs /var/lib/rancher");
+  });
+
+  it("is idempotent when the LV is already 120G", async () => {
+    const ctx = mockCtx();
+    ctx.ssh.exec.mockResolvedValueOnce(stdout("  122880.00")); // lv_size
+
+    const result = await growRancherLv(ctx);
+    expect(result.changed).toBe(false);
+    expectNoCommand(ctx.ssh, "lvextend");
+  });
+
+  it("reports without failing when the VG has no free space", async () => {
+    const ctx = mockCtx();
+    ctx.ssh.exec
+      .mockResolvedValueOnce(stdout("  20480.00"))  // lv_size
+      .mockResolvedValueOnce(stdout("  0.00"));     // vg_free
+
+    const result = await growRancherLv(ctx);
+    expect(result.success).toBe(true);
+    expect(result.changed).toBe(false);
+    expect(result.message).toContain("free");
+    expectNoCommand(ctx.ssh, "lvextend");
+  });
+
+  it("skips when there is no rancher LV", async () => {
+    const ctx = mockCtx();
+    ctx.ssh.exec.mockResolvedValueOnce(stdout("")); // lvs empty
+
+    const result = await growRancherLv(ctx);
+    expect(result.success).toBe(true);
+    expect(result.changed).toBe(false);
+    expectNoCommand(ctx.ssh, "lvextend");
   });
 });
 

bastion/src/modules/modules/k3s/tests/smoke.test.ts

@@ -16,7 +16,7 @@ describe("smoke: full server install pipeline", () => {
     const pipeline: NamedOperation[] = [
       { name: "Kernel modules", fn: ops.loadKernelModules },
       { name: "Sysctl hardening", fn: ops.applyCisHardening },
-      { name: "Disable swap", fn: ops.disableSwap },
+      { name: "Enable swap", fn: ops.enableSwap },
       { name: "Disable firewall", fn: ops.disableFirewall },
       { name: "SELinux permissive", fn: ops.setSelinuxPermissive },
       { name: "Write k3s config", fn: ops.writeK3sConfig },
@@ -73,7 +73,7 @@ describe("smoke: pipeline stops on failure", () => {
     };
 
     const results = await runSequential(ctx, [
-      { name: "OK op", fn: ops.disableSwap },
+      { name: "OK op", fn: ops.enableSwap },
       { name: "Failing op", fn: failingOp },
       { name: "Never called", fn: neverCalled },
     ]);
@@ -98,11 +98,12 @@ describe("smoke: agent install rejects missing config", () => {
 });
 
 describe("smoke: all operations are exported", () => {
-  it("exports all 15 operations", () => {
+  it("exports all 16 operations", () => {
     const exported = [
       ops.loadKernelModules,
       ops.applyCisHardening,
-      ops.disableSwap,
+      ops.enableSwap,
+      ops.growRancherLv,
       ops.disableFirewall,
       ops.setSelinuxPermissive,
       ops.writeK3sConfig,
@@ -117,7 +118,7 @@ describe("smoke: all operations are exported", () => {
       ops.checkCertExpiry,
     ];
 
-    expect(exported).toHaveLength(15);
+    expect(exported).toHaveLength(16);
     for (const op of exported) {
       expect(typeof op).toBe("function");
     }

bastion/src/shared/src/hardware/index.ts

@@ -1,154 +0,0 @@
-// Architecture normalisation and machine classification.
-//
-// Both are derived from what the system already observes about a machine -- never from
-// an operator-supplied flag.
-
-import type { Arch, HardwareInfo, OnboardMethod, OsId } from "../types/index.js";
-
-export const SUPPORTED_ARCHES: readonly Arch[] = ["x86_64", "aarch64"] as const;
-
-/**
- * Normalise an architecture string to one we serve boot artifacts for.
- *
- * Sources and their spellings:
- *   uname -m         -> "x86_64" / "aarch64"
- *   iPXE ${buildarch}-> "x86_64" / "arm64"
- *   dpkg/Debian      -> "amd64"  / "arm64"
- *
- * Returns undefined for anything we don't serve, so callers fall back rather than
- * inventing a kernel path that would 404.
- */
-export function normalizeArch(value: string | undefined | null): Arch | undefined {
-  switch ((value ?? "").trim().toLowerCase()) {
-    case "x86_64":
-    case "x86-64":
-    case "amd64":
-      return "x86_64";
-    case "aarch64":
-    case "arm64":
-      return "aarch64";
-    default:
-      return undefined;
-  }
-}
-
-/** Fedora pxeboot artifact base URL for an architecture. */
-export function fedoraMirrorFor(fedoraVersion: string, arch: Arch): string {
-  return `https://download.fedoraproject.org/pub/fedora/linux/releases/${fedoraVersion}/Everything/${arch}/os`;
-}
-
-/**
- * Which architectures each OS in the pipeline can actually be installed on.
- *
- * Fedora publishes pxeboot vmlinuz/initrd for both. Ubuntu does not: as of 26.04,
- * releases.ubuntu.com publishes amd64 artifacts only, so there is nothing to netboot an
- * arm64 machine with. Claiming support would fail at download time with a 404 instead
- * of a useful message.
- */
-const OS_ARCH_SUPPORT: Record<OsId, readonly Arch[]> = {
-  "fedora-43": ["x86_64", "aarch64"],
-  "ubuntu-26.04": ["x86_64"],
-};
-
-export function osSupportsArch(os: OsId, arch: Arch): boolean {
-  return (OS_ARCH_SUPPORT[os] ?? []).includes(arch);
-}
-
-export function archesForOs(os: OsId): readonly Arch[] {
-  return OS_ARCH_SUPPORT[os] ?? [];
-}
-
-/**
- * Machines that run a vendor OS we have no image for.
- *
- * These are SSH-onboard: we manage userspace, but reinstalling destroys a driver and
- * firmware stack our pipeline cannot rebuild. Matched on DMI identity, which is what
- * discovery and `provision recheck` both collect.
- *
- * This is deliberately a property of the machine ("it runs DGX OS"), not a blocklist
- * ("never install this MAC"). When a DGX OS image joins the pipeline, teaching the
- * installer about vendor_os "dgx-os" is what unblocks these machines -- no entry here
- * needs deleting.
- */
-interface VendorOsRule {
-  vendorOs: string;
-  description: string;
-  matches: (hw: DmiIdentity) => boolean;
-}
-
-interface DmiIdentity {
-  manufacturer: string;
-  product: string;
-  board: string;
-}
-
-const VENDOR_OS_RULES: readonly VendorOsRule[] = [
-  {
-    vendorOs: "dgx-os",
-    description: "NVIDIA DGX OS (proprietary driver + firmware stack, no image in our pipeline)",
-    matches: ({ manufacturer, product, board }) =>
-      (manufacturer.includes("nvidia") || product.includes("nvidia")) &&
-      (product.includes("dgx") || product.includes("spark") ||
-       board.includes("gb10") || product.includes("gb10")),
-  },
-];
-
-/**
- * Machines known to run a vendor OS, by MAC.
- *
- * The DMI rules above only fire once discovery or `provision recheck` has populated a
- * hardware record. Machines onboarded over SSH may sit in state for a long time with no
- * DMI at all -- which is exactly the state both DGX Sparks are in today -- so a
- * DMI-only classifier would fail open on the machines this guard exists to protect.
- *
- * This is a statement of fact about known hardware ("this box runs DGX OS"), not an
- * install policy. Whether that means "refuse" is decided by whether the pipeline has an
- * image for that vendor OS.
- */
-const KNOWN_VENDOR_OS_MACS: Record<string, string> = {
-  "4c:bb:47:7f:29:35": "dgx-os",  // spark-2935
-  "48:21:0b:96:3a:1c": "dgx-os",  // spark-3a1c
-};
-
-/**
- * Classify how a machine should be onboarded, from its hardware record.
- *
- * An explicit `onboard` already on the record wins: it may have been set by an operator
- * or by a rule that has since changed, and silently overriding it would be worse than
- * leaving it.
- */
-export function classifyOnboard(
-  hw: Partial<Pick<HardwareInfo, "mac" | "manufacturer" | "product" | "board">>
-    & { onboard?: OnboardMethod; vendor_os?: string },
-): { onboard: OnboardMethod; vendor_os?: string } {
-  if (hw.onboard !== undefined) {
-    return hw.vendor_os !== undefined
-      ? { onboard: hw.onboard, vendor_os: hw.vendor_os }
-      : { onboard: hw.onboard };
-  }
-
-  const knownVendorOs = KNOWN_VENDOR_OS_MACS[(hw.mac ?? "").toLowerCase().replace(/-/g, ":")];
-  if (knownVendorOs !== undefined) {
-    return { onboard: "ssh", vendor_os: knownVendorOs };
-  }
-
-  const identity: DmiIdentity = {
-    manufacturer: (hw.manufacturer ?? "").toLowerCase(),
-    product: (hw.product ?? "").toLowerCase(),
-    board: (hw.board ?? "").toLowerCase(),
-  };
-
-  for (const rule of VENDOR_OS_RULES) {
-    if (rule.matches(identity)) {
-      return { onboard: "ssh", vendor_os: rule.vendorOs };
-    }
-  }
-
-  return { onboard: "pxe" };
-}
-
-/** Human-readable reason a vendor-OS machine must not be reinstalled. */
-export function vendorOsDescription(vendorOs: string | undefined): string {
-  const rule = VENDOR_OS_RULES.find((r) => r.vendorOs === vendorOs);
-  return rule?.description ?? "a vendor OS with no image in our pipeline";
-}

bastion/src/shared/src/index.ts

@@ -1,8 +1,6 @@
 export type {
   OsId,
   Arch,
-  OnboardMethod,
-  RootCandidate,
   Role,
   HardwareInfo,
   InstallConfig,
@@ -10,18 +8,12 @@ export type {
   DebugConfig,
   BastionState,
   BastionConfig,
+  VyosVlanSpec,
+  VyosInstallSpec,
+  VyosBundle,
+  VyosBundleSetOp,
 } from "./types/index.js";
 
-export {
-  SUPPORTED_ARCHES,
-  normalizeArch,
-  fedoraMirrorFor,
-  osSupportsArch,
-  archesForOs,
-  classifyOnboard,
-  vendorOsDescription,
-} from "./hardware/index.js";
-
 export { SUPPORTED_OS, SUPPORTED_ROLES, ROLE_REGISTRY, isValidOsId } from "./types/index.js";
 export type { RoleInfo } from "./types/index.js";
 

bastion/src/shared/src/protocol/index.ts

@@ -1,6 +1,7 @@
 // Protocol types for agent-labd WebSocket communication.
 
 import { randomUUID } from "node:crypto";
+import type { VyosInstallSpec } from "../types/state.js";
 
 // --- Agent -> labd messages ---
 
@@ -108,12 +109,12 @@ export type BastionMessage =
 export type LabdBastionMessage =
   | { type: "bastion-enrolled"; bastionId: string }
   | { type: "bastion-heartbeat-ack"; serverTime: string }
-  | { type: "command-install"; requestId: string; mac: string; hostname: string; disk?: string; role: string; os: string }
+  | { type: "command-install"; requestId: string; mac: string; hostname: string; disk?: string; role: string; os: string; vyos?: VyosInstallSpec }
   | { type: "command-forget"; requestId: string; mac: string }
   | { type: "command-role-update"; requestId: string; mac: string; role: string }
   | { type: "command-debug"; requestId: string; mac: string; pxeBoot?: boolean }
   | { type: "command-register"; requestId: string; mac: string; hostname: string; role: string; ip: string }
-  | { type: "command-discover"; requestId: string; mac: string; product?: string; board?: string; serial?: string; manufacturer?: string; cpu_model?: string; cpu_cores?: number; memory_gb?: number; arch?: string; disks?: Array<{ name: string; size_gb: number; model: string }>; nics?: Array<{ name: string; mac: string; state: string }>; root_device?: string; root_args?: string }
+  | { type: "command-discover"; requestId: string; mac: string; product?: string; board?: string; serial?: string; manufacturer?: string; cpu_model?: string; cpu_cores?: number; memory_gb?: number; arch?: string; disks?: Array<{ name: string; size_gb: number; model: string }>; nics?: Array<{ name: string; mac: string; state: string }> }
   | { type: "server-shutdown"; reconnectAfter: number };
 
 export type BastionMessageType = BastionMessage["type"];

bastion/src/shared/src/types/config.ts

@@ -14,6 +14,10 @@ export interface BastionConfig {
   // Ubuntu support
   ubuntuVersion: string;
   ubuntuMirror: string;
+  // VyOS support — netboot artifacts are extracted from the ISO at startup.
+  // LTS ISOs are subscription-only, so this defaults to a rolling release.
+  vyosIsoUrl: string;
+  vyosDefaultPassword: string;
   // Syslog listener for install logs (Anaconda logging --host)
   syslogPort: number;
   // Flags

bastion/src/shared/src/types/index.ts

@@ -1,14 +1,16 @@
 export type {
   OsId,
   Arch,
-  OnboardMethod,
-  RootCandidate,
   Role,
   HardwareInfo,
   InstallConfig,
   InstalledInfo,
   DebugConfig,
   BastionState,
+  VyosVlanSpec,
+  VyosInstallSpec,
+  VyosBundle,
+  VyosBundleSetOp,
 } from "./state.js";
 
 export { SUPPORTED_OS, SUPPORTED_ROLES, ROLE_REGISTRY, isValidOsId } from "./state.js";

bastion/src/shared/src/types/state.ts

@@ -2,25 +2,15 @@
 
 export type ProvisionStackType = "dhcpproxy" | "iso" | "cloud-init";
 
-export type OsId = "fedora-43" | "ubuntu-26.04";
+export type OsId = "fedora-43" | "ubuntu-26.04" | "vyos-rolling";
 export type Arch = "x86_64" | "aarch64";
 
-export const SUPPORTED_OS: readonly OsId[] = ["fedora-43", "ubuntu-26.04"] as const;
+export const SUPPORTED_OS: readonly OsId[] = ["fedora-43", "ubuntu-26.04", "vyos-rolling"] as const;
 
 export function isValidOsId(value: string): value is OsId {
   return (SUPPORTED_OS as readonly string[]).includes(value);
 }
 
-/**
- * How a machine joins the lab.
- *
- * "pxe"  -- bare metal we install over the network (the default).
- * "ssh"  -- the machine already runs a vendor OS we cannot reproduce, so we onboard
- *           over SSH and manage userspace only. Installing would destroy that OS.
- *           See classifyOnboard() and os-install-research.md.
- */
-export type OnboardMethod = "pxe" | "ssh";
-
 export interface HardwareInfo {
   mac: string;
   product: string;
@@ -36,23 +26,6 @@ export interface HardwareInfo {
   first_seen: string;
   last_seen: string;
   bastionId?: string;  // set when aggregated through labd
-  // Onboarding classification -- absent means "pxe" (see classifyOnboard)
-  onboard?: OnboardMethod;
-  vendor_os?: string;   // e.g. "dgx-os": the OS this machine must keep running
-  // Root filesystem, for booting the installed system over PXE (--pxe-boot).
-  // Observed from the machine, never assumed.
-  root_device?: string;                 // e.g. "/dev/mapper/labvg-root"
-  root_args?: string;                   // e.g. "rd.lvm.lv=labvg/root rd.lvm.lv=labvg/swap"
-  root_candidates?: RootCandidate[];    // reported from a rescue shell when unknown
-}
-
-/** A possible root filesystem found while probing an unreachable machine. */
-export interface RootCandidate {
-  device: string;       // e.g. "/dev/mapper/labvg-root"
-  args?: string;        // extra dracut args needed to assemble it
-  fstype?: string;
-  size_gb?: number;
-  os_release?: string;  // PRETTY_NAME from /etc/os-release, if mountable
 }
 
 export type Role = "vanilla" | "worker" | "infra" | "labcontroller";
@@ -102,13 +75,141 @@ export interface ProgressLogEntry {
   timestamp: string;
 }
 
+/** A tagged VLAN sub-interface on the bond (or on the mgmt NIC when unbonded). */
+export interface VyosVlanSpec {
+  id: number;
+  address: string;         // CIDR, e.g. "10.0.10.1/24"
+  description?: string;
+  /**
+   * VRRP virtual address (CIDR) floated on this VLAN. Emitted as a
+   * high-availability vrrp group with vrid = VLAN id, so the same spec on both
+   * HA peers (with different priorities) produces a matching group pair.
+   */
+  vrrp?: string;
+}
+
+/** One config node in a rendered bundle. Mirrors VyosSetOp on the bastion side. */
+export interface VyosBundleSetOp {
+  path: string[];
+  value?: string;
+  /** false appends to a multi-value node (e.g. bond members) instead of replacing. */
+  replace?: boolean;
+}
+
+/**
+ * A router's complete desired config, rendered from the Pulumi model.
+ *
+ * Produced by `kubernetes-deployment/scripts/vyos-render-bundle.ts` from the
+ * same subtree model `pulumi up` applies. The point is that labctl never
+ * authors VyOS config: bring-up replays what Pulumi already declares, so a
+ * freshly installed router and a `pulumi up` cannot disagree.
+ *
+ * Secret values arrive as `@secret:<key>` sentinels and are DROPPED at install
+ * time -- the bundle is committed to git and must stay safe to read. The router
+ * comes up on the LAN without its PPPoE credential; the first `pulumi up`
+ * supplies it. That handoff is deliberate.
+ */
+export interface VyosBundle {
+  sets: VyosBundleSetOp[];
+  /** Paths that are VyOS tag nodes — the installer's ConfigTree needs them marked. */
+  tags: string[][];
+}
+
+/**
+ * VyOS-specific install parameters. Rendered into the config.boot that the
+ * installer adopts, so the router comes up already configured.
+ *
+ * NOTE: bondMembers must NOT include the interface PXE booted from. Firmware
+ * PXE cannot run over LACP, so the install-time NIC has to stay unbonded.
+ */
+export interface VyosInstallSpec {
+  /**
+   * A complete rendered config for this router. When present it REPLACES the
+   * derived interface/VLAN/VRRP config below -- the bundle already describes
+   * all of it, and deriving a second opinion is exactly the drift this exists
+   * to prevent. The remaining install parameters (password, disk, console) are
+   * still honoured because they are installer inputs, not router config.
+   */
+  bundle?: VyosBundle;
+  /**
+   * Key for the VyOS HTTP API, enabled at install so the router is manageable
+   * from the moment it boots.
+   *
+   * Without this the box comes up reachable only over SSH, and enabling the API
+   * later is a hand-run config change on a live firewall -- which is exactly the
+   * gap that left vyos001/vyos002 unmanageable by Pulumi after their cutover.
+   * The API is deliberately NOT part of the Pulumi model: a provider that
+   * manages its own transport can revoke its own access.
+   */
+  apiKey?: string;
+  /**
+   * Address the API listens on. Defaults to the management address when static.
+   * Never left unbound: an unrestricted listener puts a config-write endpoint on
+   * every segment the router touches, including the WAN.
+   */
+  apiListenAddress?: string;
+  /** Interfaces aggregated into bond0 with LACP (802.3ad). Omit for no bond. */
+  bondMembers?: string[];
+  /** CIDR address on bond0 itself — the switch trunk's native/untagged VLAN. */
+  bondAddress?: string;
+  /** VRRP virtual address (CIDR) floated on the untagged bond (vrid 1). */
+  bondVrrp?: string;
+  /**
+   * VRRP priority for every group on this box. Higher wins mastership.
+   * The HA pair differs ONLY here (e.g. 200 on the primary, 100 on the
+   * standby) — addresses differ per box, VIPs and vrids match.
+   */
+  vrrpPriority?: number;
+  /** Tagged VLAN sub-interfaces, created on bond0 when bonded, else on mgmtInterface. */
+  vlans?: VyosVlanSpec[];
+  /** Untagged interface the machine PXE booted from. Defaults to "eth0". */
+  mgmtInterface?: string;
+  /** CIDR address for mgmtInterface, or "dhcp". Defaults to "dhcp". */
+  mgmtAddress?: string;
+  /**
+   * Tagged management VLAN on mgmtInterface, separate from the routed VLANs
+   * carried by the bond.
+   *
+   * Needed when the PXE port is a trunk: it boots untagged on the VLAN the
+   * bastion's proxy DHCP serves, and carries the management VLAN tagged so the
+   * router stays reachable there without giving up reinstallability.
+   */
+  mgmtVlan?: VyosVlanSpec;
+  /** Password for the "vyos" user. Falls back to the bastion default. */
+  password?: string;
+  /**
+   * On reinstall the VyOS installer carries the previous on-disk config (and
+   * SSH host keys) forward -- the "reinstall without losing data" default.
+   * Set true to make the bastion-generated config win instead: after install
+   * the driver overwrites the installed image's config.boot.
+   */
+  freshConfig?: boolean;
+  /**
+   * VyOS interface name -> MAC, emitted as `hw-id` so names bind deterministically.
+   *
+   * Discovery runs under Fedora and reports predictable names (enp2s0,
+   * enp1s0f0np0), but VyOS enumerates its own eth<N> names, so a name observed
+   * during discovery cannot be used directly. Pinning by MAC removes the guess
+   * about which physical port a given eth<N> is.
+   */
+  hwIds?: Record<string, string>;
+}
+
 export interface InstallConfig {
   hostname: string;
   disk: string;
   role: Role;
   os?: OsId;          // defaults to "fedora-43" for backward compat
+  vyos?: VyosInstallSpec;  // only consulted when os is "vyos-rolling"
   arch?: Arch;         // detected from HardwareInfo or overridden
   queued_at: string;
+  /**
+   * When dispatch last served this machine an install boot script. Progress
+   * callbacks only start once the installer environment is up, so a machine
+   * dispatched long ago with no progress is wedged before that point (bad
+   * kernel/initrd, no network in the initramfs, wrong NIC picked...).
+   */
+  dispatched_at?: string;
   progress?: string;
   progress_at?: string;
   progress_detail?: string;
@@ -130,11 +231,6 @@ export interface InstalledInfo {
   cpu_cores?: number;
   memory_gb?: number;
   arch?: string;
-  onboard?: OnboardMethod;
-  vendor_os?: string;
-  root_device?: string;
-  root_args?: string;
-  root_candidates?: RootCandidate[];
 }
 
 export interface DebugConfig {

bastion/tests/integration/arm-pxe-provision.test.ts

@@ -1,537 +0,0 @@
-// Integration test: aarch64 network PXE boot.
-//
-// The boot-ISO path already covered ARM (arm-iso-provision.test.ts). This covers the
-// network path: DHCP option 93 handing an arm64 client an arm64 iPXE binary, dispatch
-// serving an aarch64 kernel, and `provision debug` reaching a rescue shell -- which is
-// what the DGX Sparks actually need and could not do.
-//
-// Two suites, because they cost very different amounts of time:
-//
-//   "ARM PXE rescue"   NBP handoff -> rescue with SSH.                    ~25-30 min
-//   "ARM PXE install"  discover -> install -> installed.                  ~75-95 min
-//
-// The rescue suite seeds the machine into state as an already-known aarch64 box rather
-// than discovering it first. That is the DGX Spark situation exactly -- SSH-onboarded,
-// never PXE-discovered, architecture known only from its record -- and it holds the test
-// to one emulated boot. Each boot spends ~15 of its ~18 minutes downloading Anaconda's
-// stage2 under TCG, so discovering first would double the runtime without touching any
-// code path the rescue boot does not already exercise.
-//
-// The install suite only runs with ARM_PXE_FULL=1. No ARM machine in the lab is ever
-// PXE-installed except the MS-R1, and an hour-plus test that runs by default is a test
-// nobody runs.
-//
-// IMPORTANT: aarch64 has no KVM on an x86_64 host, so all of this is emulated and
-// roughly 10x slower than native.
-//
-// A note for whoever debugs a failure here: if the VM panics with
-//   VFS: Unable to mount root fs on unknown-block(0,0)
-// that is very likely iPXE silently dropping the initrd because the build lacks
-// EFI_LOAD_FILE2_PROTOCOL -- on arm64 the kernel EFI stub fetches the initrd over
-// LoadFile2, and an iPXE without it accepts the `initrd` line and does nothing. It is
-// NOT a reproduction of the DGX Spark kernel bug that motivated this work, despite
-// being the identical message. assertIpxeSupportsLoadFile2() below checks the build up
-// front so that failure names itself; to check by hand:
-//   node -e 'const b=require("fs").readFileSync("/usr/share/ipxe/arm64-efi/snponly.efi");
-//            console.log(b.indexOf(Buffer.from("c1c00640b3fc3e40996d4a6c8724e06d","hex")))'
-// Fedora's ipxe-bootimgs-aarch64-20240119 has it at 0x3bbf0.
-//
-// Prerequisites:
-//   - qemu-system-aarch64        (sudo dnf install qemu-system-aarch64)
-//   - edk2-aarch64               (sudo dnf install edk2-aarch64)
-//   - ipxe-bootimgs-aarch64      (sudo dnf install ipxe-bootimgs-aarch64)
-//   - libvirtd, sudo, internet access
-//
-// Run: sudo ./scripts/test-provision.sh arm-pxe
-
-import { describe, it, expect, beforeAll, afterAll } from "vitest";
-import { readFileSync, existsSync, mkdirSync, rmSync, copyFileSync, writeFileSync } from "node:fs";
-import { execSync } from "node:child_process";
-import { join } from "node:path";
-import { homedir, tmpdir } from "node:os";
-import { log, waitForSsh } from "./helpers/libvirt.js";
-import { ensurePxeNetwork, destroyPxeNetwork, deleteNftablesRejectRules, PXE_NETWORK_NAME, PXE_GATEWAY, PXE_SUBNET } from "./helpers/pxe-network.js";
-import { createPxeVm, destroyPxeVm, getVmMac, rebootPxeVm, readSerialLog } from "./helpers/pxe-vm.js";
-import { sshExec } from "./helpers/ssh.js";
-
-const IPXE_ARM64 = "/usr/share/ipxe/arm64-efi/snponly.efi";
-const AAVMF = "/usr/share/edk2/aarch64/QEMU_EFI.fd";
-
-const VM_MEMORY = 4096;
-const VM_VCPUS = 2;
-const VM_DISK_GB = 250;
-const SSH_USER = "lab";
-const BASTION_IP = PXE_GATEWAY;
-const DHCP_RANGE_START = `${PXE_SUBNET}.100`;
-const DHCP_RANGE_END = `${PXE_SUBNET}.200`;
-const SERIAL_PORT = 4555;
-
-// Emulated aarch64 -- generous timeouts throughout. Measured on an x86_64 host with no
-// KVM for aarch64: a single PXE boot to a running Anaconda takes ~18 minutes, almost all
-// of it downloading inst.stage2 over the network under TCG. Budget well above that;
-// timing out just short of success wastes a whole run.
-const LEASE_TIMEOUT_MS = 10 * 60_000;
-const DISCOVERY_TIMEOUT_MS = 35 * 60_000;
-const INSTALL_TIMEOUT_MS = 75 * 60_000;
-const SSH_TIMEOUT_MS = 35 * 60_000;
-
-const RUN_FULL_INSTALL = process.env["ARM_PXE_FULL"] === "1";
-
-function sleep(ms: number): Promise<void> {
-  return new Promise((r) => setTimeout(r, ms));
-}
-
-function findSshKey(): { pubKey: string; keyPath: string } {
-  const candidates: string[] = [];
-  if (process.env["SSH_KEY_PATH"]) candidates.push(process.env["SSH_KEY_PATH"]);
-  const homes = [homedir()];
-  const sudoUser = process.env["SUDO_USER"];
-  if (sudoUser) homes.push(join("/home", sudoUser));
-  for (const home of homes) {
-    for (const name of ["id_ed25519", "id_ecdsa", "id_rsa"]) {
-      candidates.push(join(home, ".ssh", name));
-    }
-  }
-  for (const keyPath of candidates) {
-    if (existsSync(keyPath) && existsSync(`${keyPath}.pub`)) {
-      return { pubKey: readFileSync(`${keyPath}.pub`, "utf-8").trim(), keyPath };
-    }
-  }
-  throw new Error("No SSH key found — set SSH_KEY_PATH or ensure keys exist in ~/.ssh/");
-}
-
-async function pollApi<T>(
-  url: string,
-  check: (data: T) => boolean,
-  timeoutMs: number,
-  intervalMs = 10_000,
-): Promise<T> {
-  const start = Date.now();
-  while (Date.now() - start < timeoutMs) {
-    try {
-      const res = await fetch(url);
-      if (res.ok) {
-        const data = (await res.json()) as T;
-        if (check(data)) return data;
-      }
-    } catch { /* bastion not up yet, or a network hiccup */ }
-    await sleep(intervalMs);
-  }
-  throw new Error(`Timeout after ${timeoutMs}ms polling ${url}`);
-}
-
-function requirePrerequisites(): void {
-  if (!existsSync("/usr/bin/qemu-system-aarch64")) {
-    throw new Error("qemu-system-aarch64 not installed. Run: sudo dnf install qemu-system-aarch64");
-  }
-  if (!existsSync(AAVMF)) {
-    throw new Error(`AAVMF firmware not found at ${AAVMF}. Run: sudo dnf install edk2-aarch64`);
-  }
-  if (!existsSync(IPXE_ARM64)) {
-    throw new Error(`arm64 iPXE not found at ${IPXE_ARM64}. Run: sudo dnf install ipxe-bootimgs-aarch64`);
-  }
-}
-
-/**
- * Confirm the arm64 iPXE binary implements EFI_LOAD_FILE2_PROTOCOL.
- *
- * Without it the `initrd` line is accepted and silently ignored, and the kernel panics
- * with unknown-block(0,0). Checking here turns a confusing 30-minute boot failure into
- * an immediate, explanatory one.
- *
- * GUID 4006c0c1-fcb3-403e-996d-4a6c8724e06d, little-endian in the binary's GUID table.
- */
-function assertIpxeSupportsLoadFile2(): void {
-  const LOAD_FILE2_GUID = Buffer.from("c1c00640b3fc3e40996d4a6c8724e06d", "hex");
-  const binary = readFileSync(IPXE_ARM64);
-  if (binary.indexOf(LOAD_FILE2_GUID) < 0) {
-    throw new Error(
-      `${IPXE_ARM64} does not reference EFI_LOAD_FILE2_PROTOCOL. On arm64 the kernel ` +
-      `EFI stub fetches the initrd over LoadFile2; without it iPXE drops the initrd ` +
-      `silently and the kernel panics with "unknown-block(0,0)". Rebuild iPXE with ` +
-      `LoadFile2, or chainload grubaa64.efi for aarch64 instead.`,
-    );
-  }
-  log(`iPXE arm64 implements LoadFile2 — initrd will be delivered to the EFI stub`);
-}
-
-interface Harness {
-  testDir: string;
-  app: { close: () => Promise<void> };
-  stopDnsmasq: () => void;
-  state: { update: (fn: (s: BastionStateLike) => void) => void };
-  vmMac: string;
-  httpPort: number;
-}
-
-/** Just the parts of BastionState this test seeds. */
-interface BastionStateLike {
-  discovered: Record<string, Record<string, unknown>>;
-  installed: Record<string, Record<string, unknown>>;
-  install_queue: Record<string, Record<string, unknown>>;
-  debug: Record<string, Record<string, unknown>>;
-}
-
-/** Bring up an isolated network, a bastion with both arch payloads, and an arm64 VM. */
-async function startHarness(vmName: string, httpPort: number, pubKey: string): Promise<Harness> {
-  requirePrerequisites();
-  assertIpxeSupportsLoadFile2();
-
-  log("Setting up PXE test network...");
-  ensurePxeNetwork();
-
-  const testDir = join(tmpdir(), `lab-arm-pxe-test-${Date.now()}`);
-  for (const sub of ["tftp", "http", "logs"]) {
-    mkdirSync(join(testDir, sub), { recursive: true });
-  }
-
-  const { createApp } = await import("../../src/bastion/src/server.js");
-  const { loadConfig } = await import("../../src/bastion/src/config.js");
-  const { generateDnsmasqConf, startDnsmasq, stopDnsmasq } = await import("../../src/bastion/src/services/dnsmasq.js");
-  const { generateDiscoverKickstart } = await import("../../src/bastion/src/services/kickstart-generator.js");
-  const { renderBootIpxe, kernelPath, initrdPath } = await import("../../src/bastion/src/templates/boot.ipxe.js");
-  // Relative, not "@lab/shared": these tests run from the repo root against sources,
-  // where the workspace package alias is not resolvable.
-  const { SUPPORTED_ARCHES, fedoraMirrorFor } = await import("../../src/shared/src/hardware/index.js");
-
-  const config = loadConfig({
-    bastionDir: testDir,
-    httpPort,
-    iface: "virbr-pxe",
-    serverIp: BASTION_IP,
-    network: `${PXE_SUBNET}.0`,
-    gateway: BASTION_IP,
-    dhcpMode: "full",
-    dhcpRangeStart: DHCP_RANGE_START,
-    dhcpRangeEnd: DHCP_RANGE_END,
-    domain: "arm-pxe-test.local",
-    sshKeys: [pubKey],
-    adminUser: SSH_USER,
-  });
-
-  // iPXE binaries. The arm64 one is the whole point: dnsmasq hands it out on DHCP
-  // option 93 -- 11 for UEFI PXE (TFTP) and 19 for UEFI HTTP Boot.
-  //
-  // They go in BOTH directories, exactly as main.ts stages them. AAVMF prefers HTTP
-  // Boot, so it is served an http:// URL and fetches from httpDir; a firmware that
-  // takes the TFTP path reads the same file from tftpDir. Staging only tftpDir gives a
-  // 404 and "No bootable option or device was found" on the console.
-  log("Staging iPXE binaries...");
-  const ipxeX86 = "/usr/share/ipxe/ipxe-snponly-x86_64.efi";
-  copyFileSync(IPXE_ARM64, join(config.tftpDir, "ipxe-arm64.efi"));
-  copyFileSync(IPXE_ARM64, join(config.httpDir, "ipxe-arm64.efi"));
-  if (existsSync(ipxeX86)) {
-    copyFileSync(ipxeX86, join(config.tftpDir, "ipxe.efi"));
-    copyFileSync(ipxeX86, join(config.httpDir, "ipxe.efi"));
-  }
-
-  // Fedora kernel + initrd for both architectures, cached across runs.
-  const cacheDir = "/var/lib/libvirt/images/lab-pxe-cache";
-  execSync(`mkdir -p "${cacheDir}"`, { stdio: "pipe" });
-
-  for (const arch of SUPPORTED_ARCHES) {
-    const mirror = fedoraMirrorFor(config.fedoraVersion, arch);
-    const kernelCache = join(cacheDir, `vmlinuz-${arch}`);
-    const initrdCache = join(cacheDir, `initrd-${arch}.img`);
-
-    if (!existsSync(kernelCache)) {
-      log(`Downloading Fedora ${config.fedoraVersion} ${arch} kernel...`);
-      execSync(`curl -# -L -f -o "${kernelCache}" "${mirror}/images/pxeboot/vmlinuz"`, { stdio: "inherit", timeout: 600_000 });
-    }
-    if (!existsSync(initrdCache)) {
-      log(`Downloading Fedora ${config.fedoraVersion} ${arch} initrd...`);
-      execSync(`curl -# -L -f -o "${initrdCache}" "${mirror}/images/pxeboot/initrd.img"`, { stdio: "inherit", timeout: 600_000 });
-    }
-
-    // Staged under the exact names the iPXE templates will ask for.
-    copyFileSync(kernelCache, join(config.httpDir, kernelPath(arch)));
-    copyFileSync(initrdCache, join(config.httpDir, initrdPath(arch)));
-    log(`Staged ${arch}: ${kernelPath(arch)} + ${initrdPath(arch)}`);
-  }
-
-  writeFileSync(join(config.httpDir, "discover.ks"), generateDiscoverKickstart(config));
-  writeFileSync(
-    join(config.httpDir, "boot.ipxe"),
-    renderBootIpxe({ serverIp: config.serverIp, httpPort: config.httpPort }),
-  );
-  generateDnsmasqConf(config);
-
-  const { app, state, syslog } = createApp(config);
-  await app.listen({ port: config.httpPort, host: "0.0.0.0" });
-  syslog.start();
-  log(`Bastion HTTP listening on :${config.httpPort}`);
-
-  log("Starting dnsmasq (full DHCP)...");
-  startDnsmasq(config).catch((err) => {
-    log(`dnsmasq failed: ${err instanceof Error ? err.message : String(err)}`);
-  });
-  await sleep(1500);
-
-  log("Creating aarch64 PXE VM (emulated — this is slow)...");
-  createPxeVm({
-    name: vmName,
-    memory: VM_MEMORY,
-    vcpus: VM_VCPUS,
-    diskSize: VM_DISK_GB,
-    network: PXE_NETWORK_NAME,
-    arch: "aarch64",
-  });
-
-  const vmMac = getVmMac(vmName);
-  if (!vmMac) throw new Error("Could not determine VM MAC address");
-  log(`ARM VM MAC: ${vmMac}`);
-
-  return {
-    testDir,
-    app,
-    stopDnsmasq,
-    state: state as unknown as Harness["state"],
-    vmMac,
-    httpPort: config.httpPort,
-  };
-}
-
-async function stopHarness(vmName: string, harness: Harness | undefined): Promise<void> {
-  // KEEP_VM=1 leaves the VM, network and bastion up so a failure can be inspected on
-  // the console. Emulated aarch64 runs cost half an hour; tearing the evidence down
-  // automatically means paying that again to see what happened.
-  if (process.env["KEEP_VM"] === "1") {
-    log(`KEEP_VM=1 — leaving ${vmName} running for inspection.`);
-    log(`  console:  sudo virsh screenshot ${vmName} /tmp/vm.ppm`);
-    log(`  serial:   socat - TCP:127.0.0.1:${SERIAL_PORT}`);
-    if (harness) log(`  bastion:  ${harness.testDir} (still serving on :${harness.httpPort})`);
-    log(`  cleanup:  sudo virsh destroy ${vmName}; sudo virsh undefine ${vmName} --remove-all-storage --nvram`);
-    return;
-  }
-
-  log("Cleaning up...");
-  if (harness) {
-    await harness.app.close().catch(() => {});
-    harness.stopDnsmasq();
-  }
-  destroyPxeVm(vmName);
-  destroyPxeNetwork();
-  if (harness) rmSync(harness.testDir, { recursive: true, force: true });
-}
-
-/** Read the DHCP lease the bastion handed a MAC. Rescue mode reports no IP itself. */
-function leaseIpFor(testDir: string, mac: string): string | null {
-  const leaseFile = join(testDir, "dnsmasq.leases");
-  if (!existsSync(leaseFile)) return null;
-  for (const line of readFileSync(leaseFile, "utf-8").split("\n")) {
-    // <expiry> <mac> <ip> <hostname> <clientid>
-    const parts = line.trim().split(/\s+/);
-    if (parts.length >= 3 && parts[1]?.toLowerCase() === mac.toLowerCase()) {
-      return parts[2] ?? null;
-    }
-  }
-  return null;
-}
-
-async function waitForLease(testDir: string, mac: string, timeoutMs: number): Promise<string> {
-  const start = Date.now();
-  while (Date.now() - start < timeoutMs) {
-    const ip = leaseIpFor(testDir, mac);
-    if (ip !== null) return ip;
-    await sleep(5000);
-  }
-  throw new Error(`No DHCP lease for ${mac} within ${timeoutMs}ms`);
-}
-
-// ---------------------------------------------------------------------------
-// Rescue path -- what the DGX Sparks need.
-// ---------------------------------------------------------------------------
-
-describe("ARM PXE rescue", () => {
-  const VM_NAME = "lab-arm-pxe-rescue";
-  const HTTP_PORT = 8096;
-  let harness: Harness | undefined;
-  let sshKeyPath: string;
-  let rescueIp: string;
-
-  beforeAll(async () => {
-    const { pubKey, keyPath } = findSshKey();
-    sshKeyPath = keyPath;
-    harness = await startHarness(VM_NAME, HTTP_PORT, pubKey);
-    const { testDir, vmMac, state } = harness;
-
-    // Seed the machine as an already-known aarch64 box queued for rescue. This is the
-    // DGX Spark situation exactly: SSH-onboarded, never PXE-discovered, architecture
-    // known only from its record -- and it also keeps the test to a SINGLE emulated
-    // boot. Each boot spends ~15 minutes pulling Anaconda's stage2 over the network
-    // under TCG, so discovering first and rescuing second doubles the runtime for no
-    // extra coverage of the path being tested. Discovery is covered by the full suite.
-    log(`Seeding ${vmMac} as a known aarch64 machine queued for rescue...`);
-    state.update((s) => {
-      s.discovered[vmMac] = {
-        mac: vmMac,
-        product: "Test ARM64 Machine",
-        board: "virt",
-        serial: "SN-ARM64",
-        manufacturer: "QEMU",
-        cpu_model: "cortex-a57",
-        cpu_cores: VM_VCPUS,
-        memory_gb: 4,
-        arch: "aarch64",
-        disks: [],
-        nics: [],
-        first_seen: new Date().toISOString(),
-        last_seen: new Date().toISOString(),
-      };
-      s.debug[vmMac] = { hostname: "arm-rescue-test", queued_at: new Date().toISOString() };
-    });
-
-    // Restart so the VM boots against the seeded state. createPxeVm already started it.
-    rebootPxeVm(VM_NAME);
-    await sleep(5_000);
-    deleteNftablesRejectRules();
-
-    // The whole chain now runs once: DHCP option 93 -> arm64 iPXE -> /boot.ipxe ->
-    // /dispatch (architecture from the record, not the query) -> aarch64 kernel +
-    // initrd -> Anaconda rescue -> sshd. Reaching a shell at all proves iPXE handed
-    // the initrd to the EFI stub over LoadFile2; without it the kernel panics first.
-    log("Waiting for the rescue environment's DHCP lease...");
-    rescueIp = await waitForLease(testDir, vmMac, LEASE_TIMEOUT_MS);
-    log(`Rescue IP: ${rescueIp}`);
-
-    log("Waiting for SSH into the rescue shell (started by inst.sshd)...");
-    log("(emulated aarch64 — Anaconda's stage2 download dominates; be patient)");
-    await waitForSsh(rescueIp, "root", SSH_TIMEOUT_MS, sshKeyPath).catch(async (err) => {
-      log("Rescue SSH timed out. Serial console:");
-      try {
-        log(await readSerialLog(SERIAL_PORT, { lastLines: 100, timeoutMs: 15_000 }));
-      } catch { /* console unavailable */ }
-      throw err;
-    });
-    log("ARM PXE rescue reached.");
-  }, LEASE_TIMEOUT_MS + SSH_TIMEOUT_MS + 300_000);
-
-  afterAll(async () => { await stopHarness(VM_NAME, harness); });
-
-  it("resolved the architecture from the machine record", async () => {
-    const res = await fetch(`http://${BASTION_IP}:${HTTP_PORT}/api/machines`);
-    const data = (await res.json()) as { discovered: Record<string, { arch: string }> };
-    expect(data.discovered[harness!.vmMac]?.arch).toBe("aarch64");
-  });
-
-  it("rescue shell is reachable over SSH and is aarch64", () => {
-    const result = sshExec(rescueIp, "root", "uname -m", { keyPath: sshKeyPath, timeout: 60_000 });
-    expect(result.exitCode).toBe(0);
-    expect(result.stdout.trim()).toBe("aarch64");
-  });
-
-  it("booted an initramfs — the LoadFile2 path worked", () => {
-    // If iPXE had dropped the initrd the kernel would never have reached userspace at
-    // all, but assert it explicitly so a regression names itself.
-    const result = sshExec(rescueIp, "root", "cat /proc/cmdline; ls /run/install", {
-      keyPath: sshKeyPath, timeout: 60_000,
-    });
-    expect(result.exitCode).toBe(0);
-    expect(result.stdout).toContain("inst.rescue");
-  });
-
-  it("rescue kernel came from the bastion over HTTP", () => {
-    const result = sshExec(rescueIp, "root", "cat /proc/cmdline", { keyPath: sshKeyPath, timeout: 60_000 });
-    expect(result.stdout).toContain(`${BASTION_IP}:${HTTP_PORT}`);
-    // arm64 gets serial console arguments, never nomodeset.
-    expect(result.stdout).toContain("console=ttyAMA0");
-    expect(result.stdout).not.toContain("nomodeset");
-  });
-
-  it("has LVM tools available for inspecting an installed system", () => {
-    const result = sshExec(rescueIp, "root", "command -v vgchange && command -v lsblk", {
-      keyPath: sshKeyPath, timeout: 60_000,
-    });
-    expect(result.exitCode).toBe(0);
-  });
-});
-
-// ---------------------------------------------------------------------------
-// Full install -- opt-in, ~60-90 minutes emulated.
-// ---------------------------------------------------------------------------
-
-describe.runIf(RUN_FULL_INSTALL)("ARM PXE install", () => {
-  const VM_NAME = "lab-arm-pxe-install";
-  const HTTP_PORT = 8095;
-  let harness: Harness | undefined;
-  let sshKeyPath: string;
-  let vmIp: string;
-
-  beforeAll(async () => {
-    const { pubKey, keyPath } = findSshKey();
-    sshKeyPath = keyPath;
-    harness = await startHarness(VM_NAME, HTTP_PORT, pubKey);
-    const { vmMac } = harness;
-
-    log("Waiting for aarch64 discovery...");
-    await pollApi<{ discovered: Record<string, unknown> }>(
-      `http://${BASTION_IP}:${HTTP_PORT}/api/machines`,
-      (data) => vmMac in data.discovered,
-      DISCOVERY_TIMEOUT_MS,
-    );
-    log("Discovered. Queueing install...");
-
-    const installRes = await fetch(`http://${BASTION_IP}:${HTTP_PORT}/api/install`, {
-      method: "POST",
-      headers: { "Content-Type": "application/json" },
-      body: JSON.stringify({ mac: vmMac, hostname: VM_NAME, disk: "", role: "vanilla" }),
-    });
-    expect(installRes.status).toBe(200);
-
-    await sleep(30_000);
-    rebootPxeVm(VM_NAME);
-
-    log("Waiting for the emulated aarch64 install (60-90 min)...");
-    type LogsResponse = { status: string; progress: string; ip?: string };
-    const final = await pollApi<LogsResponse>(
-      `http://${BASTION_IP}:${HTTP_PORT}/api/logs/${encodeURIComponent(vmMac)}`,
-      (d) => d.status === "installed" || d.progress === "error",
-      INSTALL_TIMEOUT_MS,
-      30_000,
-    );
-
-    if (final.progress === "error") {
-      const logs = await (await fetch(`http://${BASTION_IP}:${HTTP_PORT}/api/logs/${encodeURIComponent(vmMac)}`)).json();
-      log(`ARM install FAILED: ${JSON.stringify(logs, null, 2)}`);
-      throw new Error("ARM PXE install failed — see logs above");
-    }
-
-    vmIp = final.ip ?? "";
-    log(`ARM install complete. IP: ${vmIp}`);
-
-    await sleep(30_000);
-    rebootPxeVm(VM_NAME);
-    await sleep(5_000);
-    deleteNftablesRejectRules();
-    await waitForSsh(vmIp, SSH_USER, SSH_TIMEOUT_MS, sshKeyPath);
-  }, DISCOVERY_TIMEOUT_MS + INSTALL_TIMEOUT_MS + SSH_TIMEOUT_MS + 600_000);
-
-  afterAll(async () => { await stopHarness(VM_NAME, harness); });
-
-  it("machine reached installed state", async () => {
-    const res = await fetch(`http://${BASTION_IP}:${HTTP_PORT}/api/machines`);
-    const data = (await res.json()) as { installed: Record<string, { hostname: string }> };
-    expect(data.installed[harness!.vmMac]?.hostname).toBe(VM_NAME);
-  });
-
-  it("installed system is aarch64", () => {
-    const result = sshExec(vmIp, SSH_USER, "uname -m", { keyPath: sshKeyPath, timeout: 60_000 });
-    expect(result.stdout.trim()).toBe("aarch64");
-  });
-
-  it("SSH works with the admin user", () => {
-    const result = sshExec(vmIp, SSH_USER, "whoami", { keyPath: sshKeyPath, timeout: 60_000 });
-    expect(result.stdout.trim()).toBe(SSH_USER);
-  });
-
-  it("LVM layout is correct", () => {
-    const result = sshExec(vmIp, SSH_USER, "sudo lvs labvg --noheadings -o lv_name", {
-      keyPath: sshKeyPath, timeout: 60_000,
-    });
-    expect(result.exitCode).toBe(0);
-    const lvs = result.stdout.trim().split("\n").map((l) => l.trim());
-    for (const expected of ["root", "var", "varlog", "swap", "home", "srv"]) {
-      expect(lvs).toContain(expected);
-    }
-  });
-});

bastion/tests/integration/helpers/pxe-vm.ts

@@ -29,6 +29,17 @@ export interface PxeVmConfig {
   diskSize: number;     // GB
   network: string;      // libvirt network name
   arch?: "x86_64" | "aarch64";
+  /**
+   * Extra NICs enumerated BEFORE the PXE NIC, on a network with no route to
+   * the bastion (defaults to libvirt's "default").
+   *
+   * Real multi-NIC boxes expose a class of bug a single-NIC VM cannot: an
+   * initramfs that picks "the first connected interface" grabs one of these
+   * instead of the NIC that PXE booted, and then cannot reach the bastion.
+   * Defaults to 0 (single NIC).
+   */
+  decoyNics?: number;
+  decoyNetwork?: string;
 }
 
 /** Create a blank UEFI VM that PXE boots from the network. */
@@ -61,6 +72,10 @@ export function createPxeVm(config: PxeVmConfig): void {
     `--memory=${config.memory}`,
     `--vcpus=${config.vcpus}`,
     `--disk=path=${diskPath},format=qcow2,bus=virtio`,
+    // Decoys first so they enumerate ahead of the PXE NIC. They are up and
+    // carry a lease, but have no route to the bastion.
+    ...Array.from({ length: config.decoyNics ?? 0 }, () =>
+      `--network=network=${config.decoyNetwork ?? "default"},model=virtio`),
     `--network=network=${config.network},model=virtio`,
     // UEFI firmware — required for PXE boot in modern mode
     `--boot=uefi,network,hd`,
@@ -95,12 +110,21 @@ export function destroyPxeVm(name: string): void {
 }
 
 /** Get the MAC address of a VM's first NIC. */
-export function getVmMac(name: string): string | null {
+export function getVmMac(name: string, network?: string): string | null {
   const result = virsh("domiflist", name);
   if (result.status !== 0) return null;
   // Output format: Interface  Type  Source  Model  MAC
-  const match = result.stdout.match(/([0-9a-f]{2}:[0-9a-f]{2}:[0-9a-f]{2}:[0-9a-f]{2}:[0-9a-f]{2}:[0-9a-f]{2})/i);
-  return match ? match[1].toLowerCase() : null;
+  // With decoy NICs present, match the line for the PXE network so we return
+  // the NIC that actually boots rather than whichever is listed first.
+  const lines = result.stdout.split("\n");
+  const candidates = network === undefined
+    ? lines
+    : lines.filter((l) => l.split(/\s+/).includes(network));
+  for (const line of candidates.length > 0 ? candidates : lines) {
+    const m = line.match(/([0-9a-f]{2}:[0-9a-f]{2}:[0-9a-f]{2}:[0-9a-f]{2}:[0-9a-f]{2}:[0-9a-f]{2})/i);
+    if (m) return m[1].toLowerCase();
+  }
+  return null;
 }
 
 /** Reboot a VM (force off + start). */

bastion/tests/integration/k3s-single-node.test.ts

@@ -161,7 +161,7 @@ EOF'
       CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
       curl -L --fail --silent "https://github.com/cilium/cilium-cli/releases/download/\${CILIUM_CLI_VERSION}/cilium-linux-amd64.tar.gz" | sudo tar xz -C /usr/local/bin
       DEFAULT_DEV=$(ip -4 route show default | awk '{print $5}' | head -1)
-      sudo KUBECONFIG=/etc/rancher/k3s/k3s.yaml cilium install --set kubeProxyReplacement=true --set ipam.mode=kubernetes --set devices=$DEFAULT_DEV --set nodePort.directRoutingDevice=$DEFAULT_DEV
+      sudo KUBECONFIG=/etc/rancher/k3s/k3s.yaml cilium install --set kubeProxyReplacement=true --set ipam.mode=cluster-pool --set ipam.operator.clusterPoolIPv4PodCIDRList='{10.42.0.0/16}' --set ipam.operator.clusterPoolIPv4MaskSize=24 --set devices=$DEFAULT_DEV --set nodePort.directRoutingDevice=$DEFAULT_DEV
     `.trim(), "cilium install", { keyPath: sshKeyPath, timeout: 120_000 });
 
     log("Waiting for Cilium to be ready...");

bastion/tests/integration/pxe-rescue.test.ts

@@ -1,210 +0,0 @@
-// Integration test: `labctl provision debug` -> Anaconda rescue with SSH, on x86_64.
-//
-// The rescue path had no test coverage on any architecture, which matters because it is
-// the lab's recovery tool of last resort -- the thing you reach for when a machine will
-// not boot. It runs here on x86_64 with KVM so it completes in minutes; the aarch64
-// equivalent is the same code path with a different kernel, but is emulated and far too
-// slow to iterate on.
-//
-// Run: sudo ./scripts/test-provision.sh rescue
-
-import { describe, it, expect, beforeAll, afterAll } from "vitest";
-import { readFileSync, existsSync, mkdirSync, rmSync, copyFileSync, writeFileSync } from "node:fs";
-import { execSync } from "node:child_process";
-import { join } from "node:path";
-import { homedir, tmpdir } from "node:os";
-import { log, waitForSsh } from "./helpers/libvirt.js";
-import { ensurePxeNetwork, destroyPxeNetwork, deleteNftablesRejectRules, PXE_NETWORK_NAME, PXE_GATEWAY, PXE_SUBNET } from "./helpers/pxe-network.js";
-import { createPxeVm, destroyPxeVm, getVmMac, rebootPxeVm, readSerialLog } from "./helpers/pxe-vm.js";
-import { sshExec } from "./helpers/ssh.js";
-
-const VM_NAME = "lab-pxe-rescue-test";
-const HTTP_PORT = 8094;
-const VM_MEMORY = 4096;
-const VM_VCPUS = 4;
-const VM_DISK_GB = 20;
-const BASTION_IP = PXE_GATEWAY;
-const SERIAL_PORT = 4555;
-
-const LEASE_TIMEOUT_MS = 8 * 60_000;
-const SSH_TIMEOUT_MS = 15 * 60_000;
-
-function sleep(ms: number): Promise<void> {
-  return new Promise((r) => setTimeout(r, ms));
-}
-
-function findSshKey(): { pubKey: string; keyPath: string } {
-  const candidates: string[] = [];
-  if (process.env["SSH_KEY_PATH"]) candidates.push(process.env["SSH_KEY_PATH"]);
-  const homes = [homedir()];
-  const sudoUser = process.env["SUDO_USER"];
-  if (sudoUser) homes.push(join("/home", sudoUser));
-  for (const home of homes) {
-    for (const name of ["id_ed25519", "id_ecdsa", "id_rsa"]) candidates.push(join(home, ".ssh", name));
-  }
-  for (const keyPath of candidates) {
-    if (existsSync(keyPath) && existsSync(`${keyPath}.pub`)) {
-      return { pubKey: readFileSync(`${keyPath}.pub`, "utf-8").trim(), keyPath };
-    }
-  }
-  throw new Error("No SSH key found — set SSH_KEY_PATH or ensure keys exist in ~/.ssh/");
-}
-
-function leaseIpFor(testDir: string, mac: string): string | null {
-  const leaseFile = join(testDir, "dnsmasq.leases");
-  if (!existsSync(leaseFile)) return null;
-  for (const line of readFileSync(leaseFile, "utf-8").split("\n")) {
-    const parts = line.trim().split(/\s+/);
-    if (parts.length >= 3 && parts[1]?.toLowerCase() === mac.toLowerCase()) return parts[2] ?? null;
-  }
-  return null;
-}
-
-async function waitForLease(testDir: string, mac: string, timeoutMs: number): Promise<string> {
-  const start = Date.now();
-  while (Date.now() - start < timeoutMs) {
-    const ip = leaseIpFor(testDir, mac);
-    if (ip !== null) return ip;
-    await sleep(5000);
-  }
-  throw new Error(`No DHCP lease for ${mac} within ${timeoutMs}ms`);
-}
-
-// Suite name must not be a substring of "ARM PXE rescue" -- vitest -t matches
-// substrings, so a looser name here would drag the emulated aarch64 suite in with it.
-describe("x86 rescue boot", () => {
-  let app: { close: () => Promise<void> };
-  let stopDnsmasqFn: () => void;
-  let testDir: string;
-  let vmMac: string;
-  let rescueIp: string;
-  let sshKeyPath: string;
-
-  beforeAll(async () => {
-    const { pubKey, keyPath } = findSshKey();
-    sshKeyPath = keyPath;
-
-    log("Setting up PXE test network...");
-    ensurePxeNetwork();
-
-    testDir = join(tmpdir(), `lab-pxe-rescue-${Date.now()}`);
-    for (const sub of ["tftp", "http", "logs"]) mkdirSync(join(testDir, sub), { recursive: true });
-
-    const { createApp } = await import("../../src/bastion/src/server.js");
-    const { loadConfig } = await import("../../src/bastion/src/config.js");
-    const { generateDnsmasqConf, startDnsmasq, stopDnsmasq } = await import("../../src/bastion/src/services/dnsmasq.js");
-    const { renderBootIpxe, kernelPath, initrdPath } = await import("../../src/bastion/src/templates/boot.ipxe.js");
-    stopDnsmasqFn = stopDnsmasq;
-
-    const config = loadConfig({
-      bastionDir: testDir,
-      httpPort: HTTP_PORT,
-      iface: "virbr-pxe",
-      serverIp: BASTION_IP,
-      network: `${PXE_SUBNET}.0`,
-      gateway: BASTION_IP,
-      dhcpMode: "full",
-      dhcpRangeStart: `${PXE_SUBNET}.100`,
-      dhcpRangeEnd: `${PXE_SUBNET}.200`,
-      domain: "rescue-test.local",
-      sshKeys: [pubKey],
-      adminUser: "lab",
-    });
-
-    // iPXE in both dirs: TFTP PXE and UEFI HTTP Boot are both possible, and OVMF picks.
-    const ipxeX86 = "/usr/share/ipxe/ipxe-snponly-x86_64.efi";
-    if (!existsSync(ipxeX86)) throw new Error(`iPXE not found: ${ipxeX86}`);
-    copyFileSync(ipxeX86, join(config.tftpDir, "ipxe.efi"));
-    copyFileSync(ipxeX86, join(config.httpDir, "ipxe.efi"));
-
-    const cacheDir = "/var/lib/libvirt/images/lab-pxe-cache";
-    execSync(`mkdir -p "${cacheDir}"`, { stdio: "pipe" });
-    const kernelCache = join(cacheDir, "vmlinuz-x86_64");
-    const initrdCache = join(cacheDir, "initrd-x86_64.img");
-    if (!existsSync(kernelCache)) {
-      log("Downloading Fedora x86_64 kernel...");
-      execSync(`curl -# -L -f -o "${kernelCache}" "${config.fedoraMirror}/images/pxeboot/vmlinuz"`, { stdio: "inherit", timeout: 600_000 });
-    }
-    if (!existsSync(initrdCache)) {
-      log("Downloading Fedora x86_64 initrd...");
-      execSync(`curl -# -L -f -o "${initrdCache}" "${config.fedoraMirror}/images/pxeboot/initrd.img"`, { stdio: "inherit", timeout: 600_000 });
-    }
-    copyFileSync(kernelCache, join(config.httpDir, kernelPath("x86_64")));
-    copyFileSync(initrdCache, join(config.httpDir, initrdPath("x86_64")));
-
-    writeFileSync(join(config.httpDir, "boot.ipxe"), renderBootIpxe({ serverIp: config.serverIp, httpPort: config.httpPort }));
-    generateDnsmasqConf(config);
-
-    const { app: fastify, state, syslog } = createApp(config);
-    app = fastify;
-    await fastify.listen({ port: config.httpPort, host: "0.0.0.0" });
-    syslog.start();
-    log(`Bastion HTTP listening on :${HTTP_PORT}`);
-
-    startDnsmasq(config).catch((err) => log(`dnsmasq failed: ${err instanceof Error ? err.message : String(err)}`));
-    await sleep(1500);
-
-    log("Creating x86_64 PXE VM (KVM)...");
-    createPxeVm({ name: VM_NAME, memory: VM_MEMORY, vcpus: VM_VCPUS, diskSize: VM_DISK_GB, network: PXE_NETWORK_NAME });
-    const mac = getVmMac(VM_NAME);
-    if (!mac) throw new Error("Could not determine VM MAC");
-    vmMac = mac;
-    log(`VM MAC: ${vmMac}`);
-
-    // Queue rescue directly, as `labctl provision debug` does.
-    log("Queueing debug/rescue mode...");
-    state.update((s) => {
-      s.debug[vmMac] = { hostname: "rescue-test", queued_at: new Date().toISOString() };
-    });
-
-    rebootPxeVm(VM_NAME);
-    await sleep(5_000);
-    deleteNftablesRejectRules();
-
-    rescueIp = await waitForLease(testDir, vmMac, LEASE_TIMEOUT_MS);
-    log(`Rescue IP: ${rescueIp}`);
-
-    log("Waiting for SSH into the rescue shell (inst.sshd)...");
-    await waitForSsh(rescueIp, "root", SSH_TIMEOUT_MS, sshKeyPath).catch(async (err) => {
-      log("Rescue SSH timed out. Serial console:");
-      try { log(await readSerialLog(SERIAL_PORT, { lastLines: 120, timeoutMs: 20_000 })); } catch { /* none */ }
-      throw err;
-    });
-    log("Rescue shell reachable.");
-  }, LEASE_TIMEOUT_MS + SSH_TIMEOUT_MS + 300_000);
-
-  afterAll(async () => {
-    if (process.env["KEEP_VM"] === "1") {
-      log(`KEEP_VM=1 — leaving ${VM_NAME} up (serial: socat - TCP:127.0.0.1:${SERIAL_PORT})`);
-      return;
-    }
-    log("Cleaning up...");
-    if (app) await app.close().catch(() => {});
-    if (stopDnsmasqFn) stopDnsmasqFn();
-    destroyPxeVm(VM_NAME);
-    destroyPxeNetwork();
-    if (testDir) rmSync(testDir, { recursive: true, force: true });
-  });
-
-  it("rescue shell is reachable over SSH as root", () => {
-    const result = sshExec(rescueIp, "root", "whoami", { keyPath: sshKeyPath, timeout: 60_000 });
-    expect(result.exitCode).toBe(0);
-    expect(result.stdout.trim()).toBe("root");
-  });
-
-  it("is the Anaconda rescue environment", () => {
-    const result = sshExec(rescueIp, "root", "cat /proc/cmdline", { keyPath: sshKeyPath, timeout: 60_000 });
-    expect(result.stdout).toContain("inst.rescue");
-    expect(result.stdout).toContain("inst.sshd");
-  });
-
-  it("kernel and initrd came from the bastion", () => {
-    const result = sshExec(rescueIp, "root", "cat /proc/cmdline", { keyPath: sshKeyPath, timeout: 60_000 });
-    expect(result.stdout).toContain(`${BASTION_IP}:${HTTP_PORT}`);
-  });
-
-  it("has LVM tools for inspecting an installed system", () => {
-    const result = sshExec(rescueIp, "root", "command -v vgchange && command -v lsblk", { keyPath: sshKeyPath, timeout: 60_000 });
-    expect(result.exitCode).toBe(0);
-  });
-});

bastion/tests/integration/vyos-provision.test.ts

@@ -0,0 +1,387 @@
+// Integration test: full VyOS unattended provisioning flow.
+//
+// Validates the VyOS install path end-to-end, at the same depth as the Fedora
+// pxe-provision test:
+//   1. Bastion (HTTP + dnsmasq) on the isolated libvirt PXE network
+//   2. Blank UEFI VM PXE boots -> Fedora-based discovery (OS-neutral)
+//   3. Queue os=vyos-rolling -> live boot + live-config hook + pty driver
+//   4. Fresh-install asserts: installed.ip, streamed logs, applied config,
+//      /config/lab-provisioned, boot-order handling
+//   5. REINSTALL round: previous config + /config data carried forward
+//      ("reinstall without losing data", VyOS-flavored)
+//   6. freshConfig round: bastion-generated config wins, /config data kept
+//
+// Prerequisites: libvirtd, OVMF, ipxe-bootimgs-x86, sudo, internet
+// (first run downloads the ~600MB VyOS nightly ISO; artifacts are cached).
+// Run: sudo pnpm run test:integration:vyos
+
+import { describe, it, expect, beforeAll, afterAll } from "vitest";
+import { readFileSync, existsSync, mkdirSync, rmSync, copyFileSync, symlinkSync, writeFileSync } from "node:fs";
+import { execSync } from "node:child_process";
+import { join } from "node:path";
+import { homedir, tmpdir } from "node:os";
+import { log, waitForSsh } from "./helpers/libvirt.js";
+import { ensurePxeNetwork, destroyPxeNetwork, deleteNftablesRejectRules, PXE_NETWORK_NAME, PXE_GATEWAY, PXE_SUBNET } from "./helpers/pxe-network.js";
+import { createPxeVm, destroyPxeVm, getVmMac, rebootPxeVm } from "./helpers/pxe-vm.js";
+import { sshExec } from "./helpers/ssh.js";
+
+const VM_NAME = "lab-vyos-test";
+const VM_MEMORY = 4096;
+const VM_VCPUS = 4;
+const VM_DISK_GB = 10;       // VyOS image install needs ~2GB minimum
+const HTTP_PORT = 8099;
+const SSH_USER = "vyos";     // the only VyOS login user
+const BASTION_IP = PXE_GATEWAY;
+const DHCP_RANGE_START = `${PXE_SUBNET}.100`;
+const DHCP_RANGE_END = `${PXE_SUBNET}.200`;
+
+const DISCOVERY_TIMEOUT_MS = 5 * 60_000;
+const INSTALL_TIMEOUT_MS = 15 * 60_000;  // squashfs fetch + copy; much faster than Anaconda
+const SSH_TIMEOUT_MS = 8 * 60_000;
+
+const HOSTNAME_R1 = "vyos-r1";
+const HOSTNAME_R2 = "vyos-r2";
+const HOSTNAME_R3 = "vyos-r3";
+
+function findSshKey(): { pubKey: string; keyPath: string } {
+  const homes = [homedir()];
+  const sudoUser = process.env["SUDO_USER"];
+  if (sudoUser) homes.push(join("/home", sudoUser));
+  if (process.env["SSH_KEY_PATH"]) {
+    const keyPath = process.env["SSH_KEY_PATH"];
+    const pubPath = `${keyPath}.pub`;
+    if (existsSync(keyPath) && existsSync(pubPath)) {
+      return { pubKey: readFileSync(pubPath, "utf-8").trim(), keyPath };
+    }
+  }
+  for (const home of homes) {
+    for (const name of ["id_ed25519", "id_ecdsa", "id_rsa"]) {
+      const keyPath = join(home, ".ssh", name);
+      const pubPath = `${keyPath}.pub`;
+      if (existsSync(keyPath) && existsSync(pubPath)) {
+        return { pubKey: readFileSync(pubPath, "utf-8").trim(), keyPath };
+      }
+    }
+  }
+  throw new Error("No SSH key found — set SSH_KEY_PATH or ensure keys exist in ~/.ssh/");
+}
+
+function sleep(ms: number): Promise<void> {
+  return new Promise((r) => setTimeout(r, ms));
+}
+
+async function pollApi<T>(
+  url: string,
+  check: (data: T) => boolean,
+  timeoutMs: number,
+  intervalMs = 5000,
+): Promise<T> {
+  const start = Date.now();
+  while (Date.now() - start < timeoutMs) {
+    try {
+      const res = await fetch(url);
+      if (res.ok) {
+        const data = (await res.json()) as T;
+        if (check(data)) return data;
+      }
+    } catch { /* not ready yet */ }
+    await sleep(intervalMs);
+  }
+  throw new Error(`Timeout after ${timeoutMs}ms polling ${url}`);
+}
+
+type LogsResponse = {
+  status: string;
+  progress: string;
+  progress_detail?: string;
+  ip?: string;
+  log_total?: number;
+  log_lines?: Array<{ line: string }>;
+};
+
+/** Queue a VyOS install, reboot the VM into PXE, wait for completion + SSH. */
+async function installRound(opts: {
+  mac: string;
+  hostname: string;
+  freshConfig?: boolean;
+}): Promise<string> {
+  const body = {
+    mac: opts.mac,
+    hostname: opts.hostname,
+    disk: "/dev/vda",
+    role: "vanilla",
+    os: "vyos-rolling",
+    vyos: {
+      mgmtInterface: "eth0",
+      mgmtAddress: "dhcp",
+      hwIds: { eth0: opts.mac },
+      ...(opts.freshConfig ? { freshConfig: true } : {}),
+    },
+  };
+  const res = await fetch(`http://${BASTION_IP}:${HTTP_PORT}/api/install`, {
+    method: "POST",
+    headers: { "Content-Type": "application/json" },
+    body: JSON.stringify(body),
+  });
+  log(`Install queued (${opts.hostname}): ${JSON.stringify(await res.json())}`);
+
+  await sleep(5_000);
+  rebootPxeVm(VM_NAME);
+  await sleep(3_000);
+  deleteNftablesRejectRules();
+
+  const finalState = await pollApi<LogsResponse>(
+    `http://${BASTION_IP}:${HTTP_PORT}/api/logs/${encodeURIComponent(opts.mac)}`,
+    (data) => data.status === "installed" || data.progress === "error",
+    INSTALL_TIMEOUT_MS,
+    10_000,
+  );
+  if (finalState.progress === "error") {
+    log(`INSTALL FAILED: ${JSON.stringify(finalState.progress_detail ?? finalState, null, 2)}`);
+    throw new Error(`VyOS install failed for ${opts.hostname}`);
+  }
+  const ip = finalState.ip ?? "";
+  log(`Install complete (${opts.hostname}). IP: ${ip}`);
+
+  // The driver force-reboots; the VM PXE boots, dispatch says installed ->
+  // localboot exit -> GRUB -> VyOS. nftables reject rules do not reappear
+  // (guest reboot, not a libvirt restart), but clearing is harmless.
+  deleteNftablesRejectRules();
+  await waitForSsh(ip, SSH_USER, SSH_TIMEOUT_MS, sshKeyPathGlobal);
+  return ip;
+}
+
+let sshKeyPathGlobal = "";
+
+describe("VyOS provisioning", () => {
+  let bastionApp: { close: () => Promise<void> };
+  let testDir: string;
+  let vmMac: string;
+  let vmIp: string;
+
+  beforeAll(async () => {
+    const { pubKey, keyPath } = findSshKey();
+    sshKeyPathGlobal = keyPath;
+
+    log("Setting up PXE test network...");
+    ensurePxeNetwork();
+
+    testDir = join(tmpdir(), `lab-vyos-test-${Date.now()}`);
+    mkdirSync(join(testDir, "tftp"), { recursive: true });
+    mkdirSync(join(testDir, "http"), { recursive: true });
+    mkdirSync(join(testDir, "logs"), { recursive: true });
+
+    log("Starting bastion...");
+    const { createApp } = await import("../../src/bastion/src/server.js");
+    const { loadConfig } = await import("../../src/bastion/src/config.js");
+    const { generateDnsmasqConf, startDnsmasq } = await import("../../src/bastion/src/services/dnsmasq.js");
+    const { generateDiscoverKickstart } = await import("../../src/bastion/src/services/kickstart-generator.js");
+    const { renderBootIpxe } = await import("../../src/bastion/src/templates/boot.ipxe.js");
+    const { prepareVyosArtifacts } = await import("../../src/bastion/src/main.js");
+
+    const config = loadConfig({
+      bastionDir: testDir,
+      httpPort: HTTP_PORT,
+      iface: "virbr-pxe",
+      serverIp: BASTION_IP,
+      network: `${PXE_SUBNET}.0`,
+      gateway: BASTION_IP,
+      dhcpMode: "full",
+      dhcpRangeStart: DHCP_RANGE_START,
+      dhcpRangeEnd: DHCP_RANGE_END,
+      domain: "pxe-test.local",
+      sshKeys: [pubKey],
+      adminUser: "lab",
+    });
+
+    // iPXE binary
+    const ipxeSrc = "/usr/share/ipxe/ipxe-snponly-x86_64.efi";
+    if (!existsSync(ipxeSrc)) {
+      throw new Error(`iPXE not found: ${ipxeSrc}. Install: sudo dnf install ipxe-bootimgs-x86`);
+    }
+    copyFileSync(ipxeSrc, join(config.tftpDir, "ipxe.efi"));
+    try { symlinkSync(join(config.tftpDir, "ipxe.efi"), join(config.httpDir, "ipxe.efi")); } catch { /* exists */ }
+
+    const cacheDir = "/var/lib/libvirt/images/lab-pxe-cache";
+    execSync(`mkdir -p "${cacheDir}"`, { stdio: "pipe" });
+
+    // Fedora kernel+initrd for DISCOVERY (OS-neutral, same as pxe test)
+    const kernel = join(cacheDir, `vmlinuz-${config.fedoraVersion}`);
+    const initrd = join(cacheDir, `initrd-${config.fedoraVersion}.img`);
+    if (!existsSync(kernel)) {
+      log(`Downloading Fedora ${config.fedoraVersion} kernel (discovery)...`);
+      execSync(`curl -# -L -f -o "${kernel}" "${config.fedoraMirror}/images/pxeboot/vmlinuz"`, { stdio: "inherit", timeout: 300_000 });
+    }
+    if (!existsSync(initrd)) {
+      log(`Downloading Fedora ${config.fedoraVersion} initrd (discovery)...`);
+      execSync(`curl -# -L -f -o "${initrd}" "${config.fedoraMirror}/images/pxeboot/initrd.img"`, { stdio: "inherit", timeout: 300_000 });
+    }
+    copyFileSync(kernel, join(config.httpDir, "vmlinuz"));
+    copyFileSync(initrd, join(config.httpDir, "initrd.img"));
+
+    // VyOS netboot artifacts — cache the three extracted files across runs
+    const vyosCache = {
+      kernel: join(cacheDir, "vyos-vmlinuz"),
+      initrd: join(cacheDir, "vyos-initrd"),
+      squashfs: join(cacheDir, "vyos-filesystem.squashfs"),
+    };
+    if (Object.values(vyosCache).every((p) => existsSync(p))) {
+      log("VyOS netboot artifacts cached");
+      copyFileSync(vyosCache.kernel, join(config.httpDir, "vyos-vmlinuz"));
+      copyFileSync(vyosCache.initrd, join(config.httpDir, "vyos-initrd"));
+      copyFileSync(vyosCache.squashfs, join(config.httpDir, "vyos-filesystem.squashfs"));
+    } else {
+      log("Extracting VyOS artifacts from ISO (downloads ~600MB on first run)...");
+      prepareVyosArtifacts(config);
+      copyFileSync(join(config.httpDir, "vyos-vmlinuz"), vyosCache.kernel);
+      copyFileSync(join(config.httpDir, "vyos-initrd"), vyosCache.initrd);
+      copyFileSync(join(config.httpDir, "vyos-filesystem.squashfs"), vyosCache.squashfs);
+    }
+
+    writeFileSync(join(config.httpDir, "discover.ks"), generateDiscoverKickstart(config));
+    writeFileSync(join(config.httpDir, "boot.ipxe"), renderBootIpxe({ serverIp: config.serverIp, httpPort: config.httpPort }));
+    generateDnsmasqConf(config);
+
+    const { app, syslog } = createApp(config);
+    bastionApp = app;
+    await app.listen({ port: config.httpPort, host: "0.0.0.0" });
+    syslog.start();
+    log(`Bastion listening on :${HTTP_PORT}`);
+
+    log("Starting dnsmasq...");
+    startDnsmasq(config).catch((err) => {
+      log(`dnsmasq failed (expected without root): ${err instanceof Error ? err.message : String(err)}`);
+    });
+    await sleep(1000);
+
+    log("Creating PXE VM...");
+    // Two decoy NICs ahead of the PXE NIC, on a network with no route to the
+    // bastion. This reproduces the real VP2440 topology: live-boot scans for
+    // "the first connected interface", and without BOOTIF it picks a decoy,
+    // times out on DHCP/fetch, and dies with "Unable to find a live file
+    // system on the network". A single-NIC VM cannot catch that.
+    createPxeVm({
+      name: VM_NAME,
+      memory: VM_MEMORY,
+      vcpus: VM_VCPUS,
+      diskSize: VM_DISK_GB,
+      network: PXE_NETWORK_NAME,
+      decoyNics: 2,
+    });
+    const mac = getVmMac(VM_NAME, PXE_NETWORK_NAME);
+    if (!mac) throw new Error("Could not determine VM MAC address");
+    vmMac = mac;
+    log(`VM MAC: ${vmMac}`);
+
+    log("Waiting for discovery...");
+    type MachinesResponse = { discovered: Record<string, unknown> };
+    await pollApi<MachinesResponse>(
+      `http://${BASTION_IP}:${HTTP_PORT}/api/machines`,
+      (data) => vmMac in data.discovered,
+      DISCOVERY_TIMEOUT_MS,
+    );
+    log("VM discovered. Running fresh VyOS install (round 1)...");
+
+    await sleep(15_000); // discovery reboot cycle
+    vmIp = await installRound({ mac: vmMac, hostname: HOSTNAME_R1 });
+    log("Round 1 (fresh install) complete.");
+  }, DISCOVERY_TIMEOUT_MS + INSTALL_TIMEOUT_MS + SSH_TIMEOUT_MS + 300_000);
+
+  afterAll(async () => {
+    log("Cleaning up...");
+    if (bastionApp) await bastionApp.close().catch(() => {});
+    const { stopDnsmasq } = await import("../../src/bastion/src/services/dnsmasq.js");
+    stopDnsmasq();
+    destroyPxeVm(VM_NAME);
+    destroyPxeNetwork();
+    if (testDir) rmSync(testDir, { recursive: true, force: true });
+  });
+
+  it("machine is installed with a real IP (WI-1: ready-at parsing)", async () => {
+    const res = await fetch(`http://${BASTION_IP}:${HTTP_PORT}/api/machines`);
+    const data = (await res.json()) as { installed: Record<string, { ip: string; os?: string }> };
+    const machine = data.installed[vmMac];
+    expect(machine).toBeDefined();
+    expect(machine.ip).toMatch(/^\d+\.\d+\.\d+\.\d+$/);
+    expect(machine.os).toBe("vyos-rolling");
+  });
+
+  it("install logs were streamed live (WI-2)", async () => {
+    const res = await fetch(`http://${BASTION_IP}:${HTTP_PORT}/api/logs/${encodeURIComponent(vmMac)}`);
+    const data = (await res.json()) as LogsResponse;
+    expect(data.log_total).toBeGreaterThan(0);
+    const lines = (data.log_lines ?? []).map((l) => l.line).join("\n");
+    // Installer transcript lines and driver messages both flow through /api/log
+    expect(lines).toMatch(/Welcome to VyOS installation|>>> answered|base config:/);
+  });
+
+  it("SSH works as the vyos user with the injected key", () => {
+    const result = sshExec(vmIp, SSH_USER, "whoami", { keyPath: sshKeyPathGlobal });
+    expect(result.exitCode).toBe(0);
+    expect(result.stdout.trim()).toBe("vyos");
+  });
+
+  it("generated config was adopted (hostname + ssh key)", () => {
+    const result = sshExec(vmIp, SSH_USER, "cat /opt/vyatta/etc/config/config.boot", { keyPath: sshKeyPathGlobal });
+    expect(result.exitCode).toBe(0);
+    expect(result.stdout).toContain(`host-name "${HOSTNAME_R1}"`);
+    expect(result.stdout).toContain("public-keys");
+  });
+
+  it("boot-order step ran and reported (WI-3)", async () => {
+    const res = await fetch(`http://${BASTION_IP}:${HTTP_PORT}/api/logs/${encodeURIComponent(vmMac)}`);
+    const data = (await res.json()) as LogsResponse;
+    const lines = (data.log_lines ?? []).map((l) => l.line).join("\n");
+    expect(lines).toContain("boot order:");
+  });
+
+  it("provisioning metadata persisted to /config (WI-4)", () => {
+    const result = sshExec(vmIp, SSH_USER, "cat /config/lab-provisioned 2>/dev/null || cat /opt/vyatta/etc/config/lab-provisioned", { keyPath: sshKeyPathGlobal });
+    expect(result.exitCode).toBe(0);
+    expect(result.stdout).toContain(`hostname=${HOSTNAME_R1}`);
+    expect(result.stdout).toContain("role=vanilla");
+    expect(result.stdout).toContain(`bastion=http://${BASTION_IP}:${HTTP_PORT}`);
+  });
+
+  it("reinstall preserves config and /config data (round 2)", async () => {
+    // Drop a marker in /config — the installer's previous-installation copy
+    // must carry it (and the whole old config) into the new image.
+    // `sync` is REQUIRED: rebootPxeVm uses `virsh destroy` (a hard power-cut),
+    // so an unsynced write never reaches the disk and the marker vanishes for
+    // reasons that have nothing to do with the installer.
+    const marker = sshExec(vmIp, SSH_USER, "echo LAB-MARKER-R2 > /config/lab-marker && sync && cat /config/lab-marker", { keyPath: sshKeyPathGlobal });
+    expect(marker.exitCode).toBe(0);
+    expect(marker.stdout).toContain("LAB-MARKER-R2");
+
+    // Queue with a DIFFERENT hostname: with preserve semantics the previous
+    // config must win, so the hostname must NOT change.
+    vmIp = await installRound({ mac: vmMac, hostname: HOSTNAME_R2 });
+
+    // Assert the config carry-forward first — it is the primary preservation
+    // signal and does not depend on the marker mechanism above.
+    const cfg = sshExec(vmIp, SSH_USER, "cat /opt/vyatta/etc/config/config.boot", { keyPath: sshKeyPathGlobal });
+    expect(cfg.stdout).toContain(`host-name "${HOSTNAME_R1}"`);   // old config carried
+    expect(cfg.stdout).not.toContain(`host-name "${HOSTNAME_R2}"`);
+
+    const markerAfter = sshExec(vmIp, SSH_USER, "cat /config/lab-marker", { keyPath: sshKeyPathGlobal });
+    expect(markerAfter.exitCode).toBe(0);
+    expect(markerAfter.stdout).toContain("LAB-MARKER-R2");
+  }, INSTALL_TIMEOUT_MS + SSH_TIMEOUT_MS + 60_000);
+
+  it("freshConfig makes the generated config win, data still kept (round 3)", async () => {
+    // Re-assert the marker is on disk and synced before the next power-cut.
+    const pre = sshExec(vmIp, SSH_USER, "sync && cat /config/lab-marker", { keyPath: sshKeyPathGlobal });
+    expect(pre.stdout).toContain("LAB-MARKER-R2");
+
+    vmIp = await installRound({ mac: vmMac, hostname: HOSTNAME_R3, freshConfig: true });
+
+    const cfg = sshExec(vmIp, SSH_USER, "cat /opt/vyatta/etc/config/config.boot", { keyPath: sshKeyPathGlobal });
+    expect(cfg.stdout).toContain(`host-name "${HOSTNAME_R3}"`);   // generated config won
+
+    // The marker file (non-config data under /config) still survives —
+    // freshConfig replaces only config.boot, not the carried data.
+    const markerAfter = sshExec(vmIp, SSH_USER, "cat /config/lab-marker", { keyPath: sshKeyPathGlobal });
+    expect(markerAfter.exitCode).toBe(0);
+    expect(markerAfter.stdout).toContain("LAB-MARKER-R2");
+  }, INSTALL_TIMEOUT_MS + SSH_TIMEOUT_MS + 60_000);
+});

labsim/.gitignore

@@ -0,0 +1,8 @@
+# runtime artifacts, not source
+*.log
+labsim_matrix_lib.py
+__pycache__/
+
+# Cluster-admin credentials for the rehearsal cluster, written by
+# `k8s-up.sh --kubeconfig`. Regenerate it rather than commit it.
+*.kubeconfig

labsim/README.md

@@ -0,0 +1,276 @@
+# labsim — libvirt replica of the lab network
+
+A throwaway copy of the production VLAN topology for testing routing, firewall
+rules and failover **without touching the real network**. Same VLAN IDs and
+roles as UniFi, deliberately different IP ranges so nothing can be confused for
+production.
+
+## Topology
+
+Each VLAN is its own isolated libvirt network with one tiny Alpine VM on it.
+
+| VLAN | Name | Sim subnet | VM address | Mirrors production |
+|-----:|------|------------|-----------|--------------------|
+| 1 | management | 172.31.1.0/24 | 172.31.1.10 | 192.168.1.0/24 |
+| 2 | k8s | 172.31.2.0/24 | 172.31.2.10 | 192.168.8.0/23 |
+| 3 | kvm | 172.31.3.0/24 | 172.31.3.10 | 192.168.3.0/24 |
+| 9 | private | 172.31.9.0/24 | 172.31.9.10 | 10.0.9.0/23 |
+| 10 | lot | **172.31.10.0/23** | 172.31.10.10 | 10.0.0.0/23 |
+| 200 | roomates | 172.31.200.0/24 | 172.31.200.10 | 192.168.2.0/24 |
+
+The sim subnet encodes the VLAN id: `172.31.<vlan>.0/24`, with one exception.
+**VLAN 10 is a `/23`** because every UniFi DHCP reservation lives in LoT and LoT
+spans `10.0.0.x` *and* `10.0.1.x`, which a `/24` cannot hold. The mapping stays
+readable — `10.0.0.46 → 172.31.10.46`, `10.0.1.67 → 172.31.11.67`.
+
+LoT's host leg is `.3`, not `.2`, because `10.0.0.2` is a real reservation
+(Hubitat) that maps onto `172.31.10.2`. `.3` is unreserved and sits below the
+DHCP pool, so it can never be handed out.
+
+`vlans.conf` therefore takes two optional trailing fields:
+
+```
+vlan_id:name:sim_prefix:real_subnet[:masklen][:host_octet]
+```
+
+defaulting to `24` and `2`. k8s and Private are also `/23` in production but
+hold no reservations, so they keep their `/24` and their DHCP range is clamped
+— reported at generation time, never silently.
+
+Address plan, identical on every VLAN:
+
+| Address | Role |
+|---------|------|
+| `.1` | gateway under test — a router VM you add (not created by default) |
+| `.2` | host bridge — how you reach the VMs from this workstation |
+| `.10` | the VLAN's micro VM |
+| `.254` | reserved for a VRRP VIP, mirroring production |
+
+The host sits at `.2` purely so you can SSH in. It is deliberately **not** the
+VMs' default route — that is `.1` — so inter-VLAN tests fail loudly when no
+router is present instead of being silently served by the host's own routing
+table. libvirt also installs reject rules that stop these networks forwarding
+to each other, so traffic between VLANs only works once a router VM bridges
+them.
+
+## Usage
+
+```bash
+./labsim-up.sh              # bring up every VLAN (idempotent)
+./labsim-up.sh 2 3          # only VLANs 2 and 3
+./labsim-down.sh            # destroy VMs + networks, keep the base image
+./labsim-down.sh --purge    # also delete the downloaded Alpine image
+```
+
+Each VM: 256 MB, 1 vCPU, a copy-on-write overlay on one shared 176 MB Alpine
+image (so six VMs cost a few MB of disk, not 1 GB).
+
+## Access
+
+```bash
+ssh alpine@172.31.2.10       # normal user (password: labsim)
+ssh root@172.31.2.10         # privileged — this image has no sudo
+curl http://172.31.2.10/     # hello-world page naming the VLAN
+```
+
+Console, when the network is the thing that is broken:
+
+```bash
+sudo virsh console labsim-2-k8s     # root / labsim
+```
+
+## Watching it
+
+```bash
+./labsim-matrix.py --watch 2     # terminal grid, changed cells highlighted
+./monitoring-up.sh               # topology page + Prometheus + Grafana
+```
+
+## Testing the DHCP migration
+
+`./labsim-dhcp-test.sh` boots throwaway VMs whose MACs are **real production
+MACs** and checks each gets the address UniFi reserved for it. MACs are the one
+piece of production config that transplants verbatim, which is what makes this a
+test rather than a rehearsal. It is safe because `ovs-labsim` has no physical
+NIC — verified with `ovs-vsctl show` — so a production MAC cannot reach the real
+LAN.
+
+Apply the config first, from `../migration`:
+
+```bash
+python3 unifi-to-vyos.py --mode sim -o /tmp/sim.conf   # 6 subnets, 31 mappings
+# load onto labsim-vyos, then:
+./labsim-dhcp-test.sh
+```
+
+**Result on VyOS 2026.08 (kea): all four cases pass.** The one that mattered:
+most UniFi reservations sit *inside* the DHCP pool, and **kea honours in-pool
+host reservations** — `printer1` received `172.31.10.46` from within the
+`.10.11–.11.254` pool. That was the open question blocking the cutover.
+
+### The lease database will lie to you
+
+The script wipes `/config/dhcp/dhcp4-leases.csv*` before every run, and both
+halves of that matter:
+
+- **Stale leases defeat reservations.** Re-running against yesterday's leases,
+  kea handed dynamic addresses to three devices that have reservations. The
+  reservation was present and correct in `/run/kea/kea-dhcp4.conf` the whole
+  time. Kea saw the reserved address as already leased to "another client" —
+  same MAC, but a different client-id from the earlier boot — and allocated
+  elsewhere. The cutover itself starts with an empty lease database, so this is
+  a *testing* artifact, but it is worth knowing that a reservation is not an
+  unconditional guarantee once leases exist.
+- **The `*` is load-bearing.** Kea's memfile backend keeps lease-file-cleanup
+  rotations (`dhcp4-leases.csv.2`) and restores from them on start, so
+  truncating only the primary file changes nothing.
+
+Both of those first appeared as a *passing* test. The verdict logic now refuses
+to score a MAC with more than one lease, because taking the first match had
+reported an hours-old lease as the current answer and turned three failures
+into apparent passes.
+
+Still open: whether kea will hand a *reserved* address to a *different* client
+while the reserved device is offline. The negative case here only proves an
+unreserved MAC gets an unreserved address.
+
+- **http://localhost:9101/** — live mesh: a node per VLAN, the router in the
+  middle, one line per pair coloured green/red with the ICMP RTT on it. Hover a
+  line for per-direction detail. Refreshes every 5s. This is the one to watch
+  while changing firewall rules.
+- **http://localhost:3000/d/labsim-matrix** — Grafana (anonymous, no login) for
+  *history*: when did a path flip, and how has latency moved.
+- **http://localhost:9101/metrics** — `labsim_reachable{src,dst,proto}` and
+  `labsim_rtt_ms{src,dst}`.
+
+## Routing: BGP, dual WAN, and the ISP VMs
+
+`sim-ha-config.py` covers the LAN side of the routers. `sim-net-config.py`
+covers everything that makes this a rehearsal for production *routing*:
+
+| role | VM | what it generates |
+|---|---|---|
+| `primary` | `labsim-vyos` | BGP + dual WAN + health-checked failover |
+| `secondary` | `labsim-vyos2` | BGP only |
+| `isp-dhcp` | `labsim-isp-dhcp` | 10gig-equivalent ISP on VLAN 53 |
+| `isp-pppoe` | `labsim-isp-pppoe` | Vodafone-equivalent PPPoE ISP on VLAN 51 |
+
+Both ISP VMs are VyOS with two NICs: one on the OVS trunk facing the sim
+router, one on libvirt's `default` network, NATing customers to the real
+internet. They use RFC 5737 documentation ranges (`203.0.113.0/24`,
+`198.51.100.0/24`) so a leaked sim route cannot blackhole anything real.
+
+```sh
+./sim-net-apply.sh check     # VM state vs what the code says — run this first
+./sim-net-apply.sh apply     # push generated config over the serial console
+```
+
+`check` is the important one. All of this previously existed only as running
+state, applied by hand over SSH; rebuilding a VM lost it, and nothing recorded
+why any of it was shaped the way it was.
+
+### Known gaps vs production
+
+- **WAN is on the primary router only.** Production has WAN on both. Two PPPoE
+  clients sharing one credential against a single access concentrator is a
+  failure mode production does not have, so the sim does not model it. VRRP and
+  conntrack failover are still exercised.
+- **ISP VM interface names are not stable across a rebuild** — `isp-dhcp` came
+  up as `eth0`/`eth1` and `isp-pppoe` as `eth2`/`eth3` from identical XML.
+  Check `show interfaces` and pass `--wan-if` / `--uplink-if` rather than
+  trusting the defaults.
+- **`eth2` on the primary router** is a libvirt-NAT uplink predating the ISP
+  VMs: a third default route with no production equivalent that masks real WAN
+  failures during a failover test. `--drop-scaffold` removes it.
+- **Committing on `isp-pppoe` drops the router's PPPoE session**, and the
+  client does not redial promptly. After any change there, check `pppoe0` on
+  the router and `sudo systemctl restart ppp@pppoe0` if it is missing.
+
+## The trunk carries every VLAN tagged, including Management
+
+There is deliberately **no native/untagged VLAN** on the trunks to the routers,
+and Management lives on `bond0.1`, not on the bare `bond0`.
+
+A native VLAN is what puts a subnet on the bond **parent** while every other
+VLAN sits on a sub-interface of it. With `dhcp-socket-type: raw`, kea receives
+each tagged frame *twice* — once on the sub-interface and once on the parent —
+and answers from the parent's pool as well (ISC Kea
+[#1117](https://gitlab.isc.org/isc-projects/kea/-/issues/1117)). A client on
+VLAN 3 gets two OFFERs and keeps whichever arrives first:
+
+```
+bond0.3 :  172.31.3.252 → 172.31.3.11   correct
+bond0   :  172.31.1.252 → 172.31.1.8    UNTAGGED, Management pool, wrong
+```
+
+`./labsim-vlan-leak-test.sh` makes one client on a tagged VLAN send a DISCOVER
+and captures on the parent and the sub-interface at once. The verdict is how
+many OFFERs the **server** emitted and from which subnets — deliberately not
+"did the client get the right address", because a client picking correctly is
+exactly how this hid. Both orderings were observed across runs, so a passing
+client proves nothing.
+
+```sh
+./labsim-vlan-leak-test.sh --vlan 3           # PASS on the current shape
+LABSIM_NATIVE_VLAN=1 ./router-up.sh           # restore the old shape...
+./labsim-vlan-leak-test.sh --vlan 3           # ...and it FAILs again
+```
+
+Three things this cost, all of which apply to production:
+
+- **Kea must be restarted after the address moves.** VyOS does not restart it
+  for an interface address change, so it keeps a raw socket bound with the old
+  address and the bug survives the fix. In the sim kea had been running since
+  16 Aug; the first post-fix test failed for this reason alone and looked like
+  the fix simply not working.
+- **The firewall interface-group must move too.** `interface-group LAN` named
+  the bare `bond0`; with a default-deny ruleset, moving the address without
+  moving the group drops every management session and all VLAN 1 routing.
+- **Duplicate delivery does not stop.** #1117 says only that there is no longer
+  a subnet on the parent to match, and that is exactly what happens: two replies
+  per DISCOVER, both now from the correct pool. Harmless, but do not read a
+  duplicate as a failure.
+
+### Tagged and untagged Management coexist
+
+Verified directly, and it is what makes the production cutover a rolling change
+rather than an outage: with the primary still untagged on `bond0` and the
+secondary already tagged on `bond0.1`, both routers were reachable, the VIP
+stayed up and a VLAN 1 client kept its gateway. One VLAN is one broadcast
+domain regardless of how each port tags it, so the two firewalls can be
+converted one at a time. See `migration/MANAGEMENT-VLAN-TAGGED.md`.
+
+`./vlan1-move-monitor.sh` logs VIP/router liveness once a second during the
+change, because VRRP reconverges and leaves no trace of who held the VIP.
+
+## Notes for whoever extends this
+
+Things that cost time the first time round, all verified on this image:
+
+- **No `sudo`.** Alpine ships `doas`; cloud-init's `sudo:` directive is inert
+  here. Use `root@` for privileged work.
+- **cloud-init leaves users locked** (`!*` in `/etc/shadow`) unless
+  `lock_passwd: false`, and sshd then refuses key auth for that user.
+- **One failing `runcmd` aborts every command after it.** Each entry is
+  `|| true` for that reason.
+- **busybox here has no `httpd` applet**, and the VMs have no internet to
+  `apk add` one — so the hello-world server is `python3 -m http.server`
+  (python3 is already present because cloud-init depends on it).
+- **`start-stop-daemon --exec /usr/bin/python3` matches cloud-init's own
+  python3** at boot and refuses to start anything.
+- **busybox `pgrep -f PATTERN` matches its own argv**, so a "skip if already
+  running" guard always fires. Verified: `guard_exit=0` with nothing listening.
+
+## Not modelled (yet)
+
+- **The secondary's bond was fiction until 2026-09-02.** `ovs_bond_router`'s
+  "already bonded, nothing to do" check compared only the trunk VLAN list, not
+  the membership. Restarting a VM recreates its taps under new names, so the
+  bond sat there holding two interfaces that no longer existed while the router's
+  real taps ran in the bridge as two *independent* ports — no LACP, and carrying
+  libvirt's own portgroup VLAN config rather than the bond's. It reconciles
+  membership now, but the lesson generalises: a sim that reports success is not
+  the same as a sim that models the thing.
+- **`labsim-vyos` has a third NIC** on libvirt's `default` network (the scaffold
+  uplink, see `--drop-scaffold`). The tap count is filtered to `$OVS_NET` for
+  that reason; an unfiltered count is 3 and silently skipped the primary's bond.

labsim/cilium-ipam-switch.sh

@@ -0,0 +1,144 @@
+#!/usr/bin/env bash
+# Procedure around a Cilium IPAM mode change. Works against any cluster, so the
+# rehearsal in labsim and the real thing in production run the SAME steps.
+#
+# It deliberately does NOT change the mode itself. In labsim that is `helm
+# upgrade`; in production Pulumi owns the release and a script racing it would
+# just reintroduce drift. What this owns is everything around the apply -- the
+# evidence, the deadlock, and the verdict.
+#
+#   ./cilium-ipam-switch.sh preflight   record what the cluster looks like now
+#   ./cilium-ipam-switch.sh unstick     break the agent-not-ready taint deadlock
+#   ./cilium-ipam-switch.sh verify      compare against preflight, report renumbering
+#
+#   KUBECONFIG=... ./cilium-ipam-switch.sh preflight
+#
+# Whether a recycle is needed is CONDITIONAL, and `verify` is what decides it.
+#
+# The operator does not preserve which node held which /24 -- it adopts whatever
+# CiliumNode.spec.ipam.podCIDRs already says. So:
+#
+#   * If CiliumNode already agrees with node.spec.podCIDRs on every node -- which
+#     is the case for any cluster that has only ever run ipam=kubernetes, because
+#     the operator syncs one from the other -- the pool adopts the existing
+#     allocation, no node is renumbered, and NO pod recycle is needed. Verified
+#     on the 3-node labsim cluster: CIDRs unchanged, nothing stranded, the only
+#     blip was the cilium DaemonSet restarting itself.
+#
+#   * If the two sources DISAGREE, nodes can swap /24s. Their running pods keep
+#     addresses that no longer fall inside the node's range, every other node
+#     routes that prefix to the wrong node, and those pods go unreachable
+#     cross-node while still showing Running. Then a full recycle is mandatory.
+#
+# Do not skip `verify` on the assumption of the good case. Run it and read it.
+set -uo pipefail
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+STATE="${STATE:-$SCRIPT_DIR/.ipam-switch-state}"
+K="kubectl"
+
+say()  { printf '\033[0;36m[ipam]\033[0m %s\n' "$*"; }
+warn() { printf '\033[1;33m[ipam]\033[0m %s\n' "$*" >&2; }
+
+snapshot() {
+  echo "## nodes"
+  $K get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.podCIDRs}{"\n"}{end}' 2>/dev/null
+  echo "## ciliumnodes"
+  $K get ciliumnode -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.ipam.podCIDRs}{"\n"}{end}' 2>/dev/null
+  echo "## pods"
+  $K get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}{"\t"}{.status.podIP}{"\n"}{end}' 2>/dev/null \
+    | grep -vP '\t$' | sort
+  echo "## ipam"
+  $K -n kube-system get cm cilium-config -o jsonpath='{.data.ipam}' 2>/dev/null; echo
+}
+
+cmd_preflight() {
+  mkdir -p "$STATE"
+  snapshot > "$STATE/before.txt"
+  say "recorded $(grep -c . "$STATE/before.txt") lines -> $STATE/before.txt"
+  say "mode now: $(sed -n '/^## ipam/,$p' "$STATE/before.txt" | tail -1)"
+  # The pod inventory is the rollback reference: if the switch renumbers, this
+  # is the only record of what an address USED to be.
+  say "pods on the pod network: $(sed -n '/^## pods/,/^## ipam/p' "$STATE/before.txt" | grep -c '10\.')"
+}
+
+# The deadlock, in one place because it WILL happen and doing it by hand under
+# time pressure is how the wrong node gets untainted:
+#   agent has no pod CIDR -> agent not ready -> node keeps
+#   node.cilium.io/agent-not-ready:NoSchedule -> the operator that would assign
+#   the CIDR cannot schedule -> agent still has no pod CIDR.
+# Removing the taint is safe: it exists to keep normal workloads off a node
+# without working networking, and the operator is precisely the thing that fixes
+# that. Kubernetes re-adds it on the next agent restart.
+cmd_unstick() {
+  local stuck=0
+  for n in $($K get nodes -o name 2>/dev/null); do
+    $K get "$n" -o jsonpath='{.spec.taints[*].key}' 2>/dev/null | grep -q 'agent-not-ready' || continue
+    warn "${n#node/} carries agent-not-ready; removing so the operator can schedule"
+    $K taint "$n" node.cilium.io/agent-not-ready- >/dev/null 2>&1 && stuck=$((stuck+1))
+  done
+  [ "$stuck" -eq 0 ] && say "no node was stuck" || say "cleared $stuck node(s)"
+  local pend
+  pend="$($K -n kube-system get pods -l io.cilium/app=operator --no-headers 2>/dev/null | grep -c Pending)"
+  [ "${pend:-0}" -gt 0 ] && warn "$pend operator pod(s) still Pending — check tolerations, not just taints"
+  return 0
+}
+
+cmd_verify() {
+  [ -f "$STATE/before.txt" ] || { warn "no preflight snapshot; nothing to compare"; return 1; }
+  snapshot > "$STATE/after.txt"
+  echo
+  say "mode: $(sed -n '/^## ipam/,$p' "$STATE/before.txt" | tail -1) -> $(sed -n '/^## ipam/,$p' "$STATE/after.txt" | tail -1)"
+
+  # The question that decides the size of the maintenance window: did per-node
+  # CIDRs survive, or was every node renumbered (and every pod with it)?
+  local moved=0
+  while IFS=$'\t' read -r node cidr; do
+    [ -z "${node:-}" ] && continue
+    local now; now="$(sed -n '/^## ciliumnodes/,/^## pods/p' "$STATE/after.txt" | awk -F'\t' -v n="$node" '$1==n{print $2}')"
+    if [ -n "$now" ] && [ "$now" != "$cidr" ]; then
+      printf '    %-16s %s -> %s\n' "$node" "$cidr" "$now"; moved=$((moved+1))
+    fi
+  done < <(sed -n '/^## ciliumnodes/,/^## pods/p' "$STATE/before.txt" | grep -P '\t')
+  if [ "$moved" -eq 0 ]; then
+    say "per-node CIDRs UNCHANGED — the pool adopted the existing allocation"
+  else
+    warn "$moved node(s) renumbered — every pod on them must be recycled"
+  fi
+
+  local before after same
+  before="$(sed -n '/^## pods/,/^## ipam/p' "$STATE/before.txt" | grep -P '\t10\.' | wc -l)"
+  after="$(sed -n '/^## pods/,/^## ipam/p' "$STATE/after.txt"  | grep -P '\t10\.' | wc -l)"
+  same="$(comm -12 <(sed -n '/^## pods/,/^## ipam/p' "$STATE/before.txt" | grep -P '\t10\.' | sort) \
+                   <(sed -n '/^## pods/,/^## ipam/p' "$STATE/after.txt"  | grep -P '\t10\.' | sort) | wc -l)"
+  say "pods: $before before, $after after, $same kept the SAME address"
+  # Keeping the address is NOT the good outcome. If a node's CIDR moved, its
+  # existing pods keep IPs that no longer fall inside it, every other node routes
+  # that prefix to the WRONG node, and those pods go unreachable cross-node while
+  # looking perfectly healthy. Observed in labsim: two nodes swapped CIDRs and
+  # cross-node ping to their pods dropped 100%, with every pod still Running.
+  # This is the check that decides whether a recycle is optional or mandatory.
+  local stranded=0
+  while read -r ns name ip node; do
+    [ -z "${node:-}" ] && continue
+    local cidr; cidr="$($K get ciliumnode "$node" -o jsonpath='{.spec.ipam.podCIDRs[0]}' 2>/dev/null)"
+    [ -z "$cidr" ] && continue
+    case "$ip" in
+      "${cidr%.*/*}".*) ;;
+      *) printf '    STRANDED %-40s %-15s on %s (now %s)\n' "$ns/$name" "$ip" "$node" "$cidr"; stranded=$((stranded+1)) ;;
+    esac
+  done < <($K get pods -A -o jsonpath='{range .items[?(@.status.podIP)]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.status.podIP}{" "}{.spec.nodeName}{"\n"}{end}' 2>/dev/null | grep -E ' 10\.')
+  if [ "$stranded" -gt 0 ]; then
+    warn "$stranded pod(s) sit OUTSIDE their node CIDR — unreachable cross-node until recycled"
+    warn "recycle: for ns in $(kubectl get ns -o name | cut -d/ -f2); do kubectl -n $ns rollout restart deploy,ds,sts 2>/dev/null; done"
+  else
+    say "every pod is inside its node CIDR — no recycle needed"
+  fi
+  say "not-Running pods: $($K get pods -A --no-headers 2>/dev/null | grep -vcE 'Running|Completed')"
+}
+
+case "${1:-}" in
+  preflight) cmd_preflight ;;
+  unstick)   cmd_unstick ;;
+  verify)    cmd_verify ;;
+  *) sed -n '2,16p' "$0"; exit 1 ;;
+esac

labsim/console-apply.py

@@ -0,0 +1,109 @@
+#!/usr/bin/env python3
+"""Apply VyOS config to a labsim VM over its serial console.
+
+Needed because a freshly installed VyOS comes up holding the same addresses as
+its peer, so there is a window where it cannot safely be reached over the
+network at all. The console does not care.
+
+  ./console-apply.py --vm labsim-vyos2 --config r2.conf
+"""
+from __future__ import annotations
+
+import argparse
+import sys
+import time
+
+import pexpect
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--vm", required=True)
+    ap.add_argument("--config", required=True)
+    ap.add_argument("--user", default="vyos")
+    ap.add_argument("--password", default="vyos")
+    args = ap.parse_args()
+
+    cmds = [l.rstrip() for l in open(args.config)
+            if l.strip() and not l.lstrip().startswith("#")]
+    print(f"{len(cmds)} commands to apply to {args.vm}", file=sys.stderr)
+
+    c = pexpect.spawn(f"virsh --connect qemu:///system console {args.vm}",
+                      timeout=90, encoding="utf-8")
+    c.logfile_read = None
+    c.sendline("")
+    time.sleep(2)
+    c.sendline("")
+
+    # Log in. A freshly booted box may still be starting services, so allow a
+    # generous window and re-prod the console rather than failing on the first
+    # miss.
+    #
+    # `# ` matters as much as `$ `: a previous run that died mid-config leaves
+    # the console sitting in configuration mode, and waiting only for the
+    # operational prompt then hangs forever against a perfectly healthy VM.
+    in_config = False
+    for _ in range(40):
+        i = c.expect([r"login:", r"\$ ", r"# ", pexpect.TIMEOUT], timeout=15)
+        if i == 0:
+            c.sendline(args.user)
+            c.expect("Password:", timeout=30)
+            c.sendline(args.password)
+            c.expect([r"\$ ", r"# "], timeout=60)
+            break
+        if i == 1:
+            break
+        if i == 2:
+            in_config = True
+            break
+        c.sendline("")
+    else:
+        print("never reached a prompt", file=sys.stderr)
+        return 1
+
+    if in_config:
+        # Drop whatever the previous run left half-built rather than committing
+        # a candidate nobody has seen.
+        print("console was left in config mode; discarding stale candidate",
+              file=sys.stderr)
+        c.sendline("discard")
+        c.expect(r"# ", timeout=60)
+    else:
+        c.sendline("configure")
+        c.expect(r"# ", timeout=60)
+
+    for cmd in cmds:
+        c.sendline(cmd)
+        c.expect(r"# ", timeout=60)
+        out = c.before or ""
+        if "Set failed" in out or "not valid" in out or "Invalid" in out:
+            print(f"FAILED: {cmd}\n  {out.strip()[:200]}", file=sys.stderr)
+
+    print("committing...", file=sys.stderr)
+    c.sendline("commit")
+    c.expect(r"# ", timeout=300)
+    commit_out = c.before or ""
+    c.sendline("save")
+    c.expect(r"# ", timeout=120)
+    # Accept either prompt on the way out. Insisting on `$ ` here hangs against
+    # a healthy box -- and worse, leaves the console parked in config mode, so
+    # the NEXT run finds a `# ` it was not expecting either. One strict expect
+    # turned into two failures.
+    c.sendline("exit")
+    c.expect([r"\$ ", r"# ", pexpect.TIMEOUT], timeout=60)
+    c.sendline("exit")
+    c.close(force=True)
+
+    bad = [l for l in commit_out.splitlines()
+           if "failed" in l.lower() or "error" in l.lower()]
+    if bad:
+        print("commit reported:", file=sys.stderr)
+        for l in bad[:10]:
+            print(f"  {l.strip()}", file=sys.stderr)
+        return 1
+    print("committed and saved", file=sys.stderr)
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

labsim/dualstack-lab.sh

@@ -0,0 +1,318 @@
+#!/usr/bin/env bash
+# Differential study: what ACTUALLY differs between a k3s cluster born
+# dual-stack and one converted in place?
+#
+# k3s says dual-stack "cannot be enabled on an existing cluster". The stated
+# reason is narrow -- nodes get Pod CIDRs only at join and the Kubernetes IPAM
+# controller will not hand out a new IPv6 CIDR later -- and it does not obviously
+# apply to a cluster where Cilium owns IPAM. Rather than argue from docs, build
+# both shapes and diff them.
+#
+#   ./dualstack-lab.sh up v4         single-node k3s, IPv4 only     (.21)
+#   ./dualstack-lab.sh up dual       single-node k3s, dual-stack    (.22)
+#   ./dualstack-lab.sh pristine v4   reflink copy of v4's disk, so the upgrade
+#                                    attempt can be rolled back and retried
+#   ./dualstack-lab.sh restore v4    put that copy back
+#   ./dualstack-lab.sh collect <n>   normalized state dump -> evidence/<n>/
+#   ./dualstack-lab.sh compare a b   semantic diff of two collections
+#   ./dualstack-lab.sh virtdiff a b  whole-filesystem diff, offline (libguestfs)
+#   ./dualstack-lab.sh down [name]
+#
+# The comparison that matters is `compare dual upgraded`: everything it prints
+# is a way the converted cluster failed to reach the shape of a native one.
+#
+# Single node on purpose. Dual-stack is decided by server flags and CNI config,
+# both of which a one-node cluster exercises fully, and it rebuilds in minutes.
+# Node-rejoin behaviour needs the 3-node cluster and is a separate question.
+set -euo pipefail
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+source "$SCRIPT_DIR/lib.sh"
+source "$SCRIPT_DIR/ovs.sh"
+
+K8S_VLAN="${K8S_VLAN:-2}"
+DS_PREFIX="${DS_PREFIX:-172.31.2}"
+MEM="${MEM:-4096}"; CPUS="${CPUS:-2}"; DISK_GB="${DISK_GB:-12}"
+TOKEN="${TOKEN:-labsim-ds-token}"
+CILIUM_VERSION="${CILIUM_VERSION:-1.19.1}"   # same as production
+DEB_BASE="${DEB_BASE:-$IMG_DIR/debian-13-genericcloud-amd64.qcow2}"
+EVIDENCE="$SCRIPT_DIR/dualstack-evidence"
+
+# Pod/Service ranges. IPv4 halves are k3s's own defaults, so the v4-only build is
+# a stock cluster and the diff is not polluted by gratuitous differences.
+# IPv6 halves are ULA: this cluster never routes off-box, and using the real /48
+# here would put lab addresses into a prefix that production also announces.
+V4_CLUSTER="10.42.0.0/16"; V4_SERVICE="10.43.0.0/16"
+V6_CLUSTER="${V6_CLUSTER:-fd00:42::/56}"
+V6_SERVICE="${V6_SERVICE:-fd00:43::/112}"   # /112 -- apiserver caps v6 service ranges
+V6_PREFIX="${V6_PREFIX:-fd00:2}"            # node addresses: fd00:2::<octet>
+
+vm_name() { echo "labsim-ds-$1"; }
+vm_ip()   { case "$1" in v4) echo "$DS_PREFIX.21";; dual) echo "$DS_PREFIX.22";; *) die "unknown build '$1'";; esac; }
+vm_ip6()  { case "$1" in v4) echo "$V6_PREFIX::21";; dual) echo "$V6_PREFIX::22";; *) die "unknown build '$1'";; esac; }
+disk_of() { echo "$IMG_DIR/$(vm_name "$1").qcow2"; }
+
+ssh_vm() { local ip="$1"; shift; ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
+             -o LogLevel=ERROR -o ConnectTimeout=8 -o BatchMode=yes "debian@$ip" "$@"; }
+
+# --- seed -----------------------------------------------------------------
+build_seed() {
+  local iso="$1" vm="$2" mode="$3" pubkey="$4"
+  local ip ip6 tmp; ip="$(vm_ip "$mode")"; ip6="$(vm_ip6 "$mode")"; tmp="$(mktemp -d)"
+
+  echo "instance-id: $vm" > "$tmp/meta-data"
+  # Static v6 on both builds. The v4-only cluster still gets an IPv6 ADDRESS --
+  # only its Kubernetes config is v4-only. Otherwise the diff would be dominated
+  # by host addressing rather than by what Kubernetes did differently.
+  cat > "$tmp/network-config" <<EOF
+version: 2
+ethernets:
+  enp1s0:
+    addresses: [${ip}/24, ${ip6}/64]
+    routes:
+      - to: default
+        via: ${DS_PREFIX}.1
+    nameservers:
+      addresses: [8.8.8.8, 1.1.1.1]
+EOF
+
+  local exec_args="server --flannel-backend=none --disable-network-policy --disable=servicelb --disable=traefik --tls-san=$ip --cluster-init"
+  if [ "$mode" = dual ]; then
+    exec_args="$exec_args --cluster-cidr=${V4_CLUSTER},${V6_CLUSTER} --service-cidr=${V4_SERVICE},${V6_SERVICE} --node-ip=${ip},${ip6}"
+  else
+    exec_args="$exec_args --node-ip=${ip}"
+  fi
+
+  cat > "$tmp/user-data" <<EOF
+#cloud-config
+hostname: $vm
+fqdn: $vm
+users:
+  - name: debian
+    groups: [sudo]
+    shell: /bin/bash
+    sudo: ["ALL=(ALL) NOPASSWD:ALL"]
+    lock_passwd: false
+    plain_text_passwd: labsim
+    ssh_authorized_keys: [$pubkey]
+ssh_pwauth: true
+disable_root: false
+package_update: true
+packages: [curl, jq, iproute2, nftables]
+write_files:
+  - path: /etc/modules-load.d/cilium.conf
+    content: |
+      br_netfilter
+  - path: /etc/dualstack-lab-mode
+    content: |
+      $mode
+runcmd:
+  - modprobe br_netfilter || true
+  - |
+    curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="$exec_args" K3S_TOKEN="$TOKEN" sh -
+  - |
+    # Cilium via helm, matching the production version. IPAM stays 'kubernetes'
+    # in BOTH builds on purpose: that is what production runs, and it is the
+    # mode the k3s objection is actually about. If the converted cluster needs
+    # cluster-pool to work, the diff should be what tells us so.
+    curl -sfL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash || true
+    export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
+    helm repo add cilium https://helm.cilium.io >/dev/null 2>&1 || true
+    helm repo update >/dev/null 2>&1 || true
+    for i in \$(seq 1 60); do kubectl get nodes >/dev/null 2>&1 && break; sleep 5; done
+    if [ "$mode" = dual ]; then
+      helm install cilium cilium/cilium --version $CILIUM_VERSION -n kube-system \\
+        --set kubeProxyReplacement=false --set ipam.mode=kubernetes \\
+        --set ipv4.enabled=true --set ipv6.enabled=true \\
+        --set k8sServiceHost=$ip --set k8sServicePort=6443 || true
+    else
+      helm install cilium cilium/cilium --version $CILIUM_VERSION -n kube-system \\
+        --set kubeProxyReplacement=false --set ipam.mode=kubernetes \\
+        --set ipv4.enabled=true --set ipv6.enabled=false \\
+        --set k8sServiceHost=$ip --set k8sServicePort=6443 || true
+    fi
+    touch /etc/dualstack-lab-ready
+EOF
+  sudo mkdir -p "$(dirname "$iso")"
+  sudo genisoimage -quiet -output "$iso" -volid cidata -joliet -rock \
+    "$tmp/user-data" "$tmp/meta-data" "$tmp/network-config"
+  rm -rf "$tmp"
+}
+
+cmd_up() {
+  local mode="${1:?usage: up <v4|dual>}"
+  local vm ip disk seed pubkey
+  vm="$(vm_name "$mode")"; ip="$(vm_ip "$mode")"; disk="$(disk_of "$mode")"
+  seed="$IMG_DIR/${vm}-seed.iso"; pubkey="$(find_ssh_pubkey)"
+
+  [ -f "$DEB_BASE" ] || die "base image missing: $DEB_BASE (run ./k8s-up.sh once)"
+  if virsh_q dominfo "$vm" >/dev/null 2>&1; then
+    log "$vm exists — starting if stopped"
+    [ "$(virsh_q domstate "$vm" | head -1)" = "running" ] || virsh_q start "$vm" >/dev/null
+    return
+  fi
+  selected_vlans; ovs_up
+  log "creating $vm ($mode) at $ip / $(vm_ip6 "$mode")"
+  sudo qemu-img create -q -f qcow2 -F qcow2 -b "$DEB_BASE" "$disk" "${DISK_GB}G" >/dev/null
+  build_seed "$seed" "$vm" "$mode" "$pubkey"
+  sudo virt-install --connect "$LIBVIRT_URI" --name "$vm" \
+    --memory "$MEM" --vcpus "$CPUS" \
+    --disk "path=$disk,format=qcow2,bus=virtio" \
+    --disk "path=$seed,device=cdrom" \
+    --network "network=$OVS_NET,portgroup=vlan${K8S_VLAN},model=virtio" \
+    --os-variant debian12 --graphics none --noautoconsole --import >/dev/null
+  log "installing in background; watch: ssh debian@$ip 'ls /etc/dualstack-lab-ready'"
+}
+
+# --- pristine copy / restore ---------------------------------------------
+# reflink so the copy is instant and independent on btrfs/xfs. A qcow2 backing
+# chain would be cheaper still but makes the parent read-only in practice: boot
+# the parent again and every child silently corrupts.
+cmd_pristine() {
+  local mode="${1:?usage: pristine <v4|dual>}" vm disk
+  vm="$(vm_name "$mode")"; disk="$(disk_of "$mode")"
+  [ "$(virsh_q domstate "$vm" 2>/dev/null | head -1)" = "running" ] && \
+    die "$vm is running — shut it down first (virsh shutdown $vm), a copy of a live disk is not consistent"
+  sudo cp --reflink=auto "$disk" "${disk}.pristine"
+  log "pristine copy: ${disk}.pristine"
+}
+cmd_restore() {
+  local mode="${1:?usage: restore <v4|dual>}" vm disk
+  vm="$(vm_name "$mode")"; disk="$(disk_of "$mode")"
+  [ -f "${disk}.pristine" ] || die "no pristine copy for $mode"
+  [ "$(virsh_q domstate "$vm" 2>/dev/null | head -1)" = "running" ] && \
+    die "$vm is running — shut it down first"
+  sudo cp --reflink=auto "${disk}.pristine" "$disk"
+  log "restored $mode from pristine"
+}
+
+
+# --- the experiment ------------------------------------------------------
+# Convert the IPv4-only cluster in place, mirroring the flags the native build
+# was BORN with. Each step prints what the cluster did, because the interesting
+# output is which step refuses rather than whether the end state is pretty.
+cmd_upgrade() {
+  local ip; ip="$(vm_ip v4)"; local ip6; ip6="$(vm_ip6 v4)"
+  log "step 1/4: add dual CIDRs + dual node-ip to the k3s unit"
+  # Done with python on the box, not nested sed: quoting a multi-line systemd
+  # continuation through ssh -> sh -> sed produced a literal \\n in the unit, and
+  # k3s then saw a dual cluster-cidr with a still-IPv4 service-cidr and refused
+  # to start. All three flags go on one line -- systemd does not care, and there
+  # is nothing left to escape.
+  ssh_vm "$ip" "sudo python3 - <<'PYEOF'
+import re
+u = '/etc/systemd/system/k3s.service'
+s = open(u).read()
+old = \"'--node-ip=${ip}'\"
+new = \"'--cluster-cidr=${V4_CLUSTER},${V6_CLUSTER}' '--service-cidr=${V4_SERVICE},${V6_SERVICE}' '--node-ip=${ip},${ip6}'\"
+assert old in s, 'node-ip flag not found in unit'
+open(u,'w').write(s.replace(old, new))
+print('    unit rewritten')
+PYEOF
+sudo systemctl daemon-reload" || die "unit edit failed"
+  ssh_vm "$ip" "grep -oE \"'--(cluster|service)-cidr=[^']*'|'--node-ip=[^']*'\" /etc/systemd/system/k3s.service | sed 's/^/    /'"
+
+  log "step 2/4: restart k3s and see whether it accepts the changed ranges"
+  ssh_vm "$ip" "sudo systemctl restart k3s" || true
+  for i in $(seq 1 40); do
+    ssh_vm "$ip" "sudo k3s kubectl get --raw /readyz >/dev/null 2>&1" && break
+    sleep 5
+  done
+  ssh_vm "$ip" "sudo journalctl -u k3s --since '2 min ago' --no-pager 2>/dev/null | grep -iE 'cidr|dual|ipv6|invalid|cannot|fail' | tail -12 | sed 's/^/    /'" || true
+
+  log "step 3/4: what the API says now"
+  ssh_vm "$ip" "echo -n '    servicecidr: '; sudo k3s kubectl get servicecidr -o jsonpath='{.items[*].spec.cidrs}'; echo; \
+                echo -n '    node podCIDRs: '; sudo k3s kubectl get node -o jsonpath='{.items[0].spec.podCIDRs}'; echo; \
+                echo -n '    node addresses: '; sudo k3s kubectl get node -o jsonpath='{.items[0].status.addresses[*].address}'; echo" || true
+
+  log "step 4/4: turn on IPv6 in Cilium"
+  ssh_vm "$ip" "export KUBECONFIG=/etc/rancher/k3s/k3s.yaml; sudo -E helm upgrade cilium cilium/cilium --version ${CILIUM_VERSION} -n kube-system --reuse-values --set ipv6.enabled=true >/dev/null 2>&1 && echo '    cilium upgraded' || echo '    cilium upgrade FAILED'" || true
+  ssh_vm "$ip" "sudo k3s kubectl -n kube-system rollout restart ds/cilium >/dev/null 2>&1; sleep 20; sudo k3s kubectl -n kube-system get pods -l k8s-app=cilium --no-headers | sed 's/^/    /'" || true
+  log "now: ./dualstack-lab.sh collect upgraded ${ip} && ./dualstack-lab.sh compare dual upgraded"
+}
+
+# --- evidence collection --------------------------------------------------
+# Normalized on purpose. Two independently built clusters differ in certs,
+# tokens, UUIDs, timestamps and log lines; left raw, that noise buries the
+# handful of differences that actually mean something.
+cmd_collect() {
+  local name="${1:?usage: collect <name> [ip]}"
+  local ip="${2:-}"
+  [ -n "$ip" ] || ip="$(vm_ip "$name" 2>/dev/null || true)"
+  [ -n "$ip" ] || die "collect: give an ip for a non-standard name"
+  local out="$EVIDENCE/$name"; mkdir -p "$out"
+  log "collecting from $name ($ip) -> $out"
+
+  ssh_vm "$ip" 'sudo cat /etc/rancher/k3s/config.yaml 2>/dev/null; sudo systemctl cat k3s 2>/dev/null | grep -A30 ExecStart' \
+    > "$out/k3s-config.txt" 2>/dev/null || true
+  ssh_vm "$ip" 'sudo tr "\0" "\n" < /proc/$(pgrep -f "k3s server" | head -1)/cmdline | grep -v "^$"' \
+    > "$out/k3s-cmdline.txt" 2>/dev/null || true
+  ssh_vm "$ip" 'ip -o addr show | awk "{print \$2, \$3, \$4}"; echo ---; ip -4 route show; echo ---; ip -6 route show' \
+    > "$out/host-net.txt" 2>/dev/null || true
+  ssh_vm "$ip" 'sudo sysctl -a 2>/dev/null | grep -E "net\.ipv6\.conf\.(all|default)\.(forwarding|disable_ipv6)|net\.ipv4\.ip_forward"' \
+    > "$out/sysctl.txt" 2>/dev/null || true
+
+  local K='sudo k3s kubectl'
+  ssh_vm "$ip" "$K get servicecidr -o yaml"       > "$out/servicecidr.yaml" 2>/dev/null || true
+  ssh_vm "$ip" "$K get nodes -o yaml"             > "$out/nodes.yaml.raw"   2>/dev/null || true
+  ssh_vm "$ip" "$K get ciliumnodes -o yaml"       > "$out/ciliumnodes.yaml.raw" 2>/dev/null || true
+  ssh_vm "$ip" "$K -n kube-system get cm cilium-config -o yaml" > "$out/cilium-config.yaml.raw" 2>/dev/null || true
+  ssh_vm "$ip" "$K get svc -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,FAMILYPOLICY:.spec.ipFamilyPolicy,FAMILIES:.spec.ipFamilies,IPS:.spec.clusterIPs" \
+    > "$out/services.txt" 2>/dev/null || true
+  ssh_vm "$ip" "$K get pods -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,IPS:.status.podIPs" \
+    > "$out/podips.txt" 2>/dev/null || true
+
+  # Strip the things that differ every build regardless of configuration.
+  for f in "$out"/*.raw; do
+    [ -e "$f" ] || continue
+    sed -E \
+      -e 's/[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9:]+Z?/<TIME>/g' \
+      -e 's/(uid|resourceVersion|creationTimestamp|generation|observedGeneration): .*/\1: <X>/' \
+      -e 's/[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/<UUID>/g' \
+      -e 's/(LS0tLS1|[A-Za-z0-9+\/]{60,}=*)/<B64>/g' \
+      "$f" > "${f%.raw}"
+    rm -f "$f"
+  done
+  log "collected $(ls "$out" | wc -l) artefacts"
+}
+
+cmd_compare() {
+  local a="${1:?usage: compare <a> <b>}" b="${2:?}"
+  [ -d "$EVIDENCE/$a" ] && [ -d "$EVIDENCE/$b" ] || die "collect both first"
+  echo "### semantic diff: $a (<) vs $b (>)"
+  diff -ru "$EVIDENCE/$a" "$EVIDENCE/$b" || true
+}
+
+cmd_virtdiff() {
+  local a="${1:?usage: virtdiff <a> <b>}" b="${2:?}"
+  for m in "$a" "$b"; do
+    [ "$(virsh_q domstate "$(vm_name "$m")" 2>/dev/null | head -1)" = "running" ] && \
+      die "$(vm_name "$m") is running — virt-diff needs the disks quiescent"
+  done
+  log "whole-filesystem diff (slow); noise is expected — use it to find what the collector missed"
+  sudo virt-diff -a "$(disk_of "$a")" -A "$(disk_of "$b")" \
+    | grep -vE '/(var/log|tmp|run|proc|sys)/|\.log$|/var/lib/rancher/k3s/(server/(tls|cred|db)|agent)' || true
+}
+
+cmd_down() {
+  local only="${1:-}"
+  for m in v4 dual; do
+    [ -n "$only" ] && [ "$only" != "$m" ] && continue
+    local vm; vm="$(vm_name "$m")"
+    virsh_q dominfo "$vm" >/dev/null 2>&1 || continue
+    log "removing $vm"
+    virsh_q destroy "$vm" >/dev/null 2>&1 || true
+    virsh_q undefine "$vm" --remove-all-storage >/dev/null 2>&1 || true
+  done
+}
+
+case "${1:-}" in
+  up)       shift; cmd_up "$@" ;;
+  upgrade)  shift; cmd_upgrade "$@" ;;
+  pristine) shift; cmd_pristine "$@" ;;
+  restore)  shift; cmd_restore "$@" ;;
+  collect)  shift; cmd_collect "$@" ;;
+  compare)  shift; cmd_compare "$@" ;;
+  virtdiff) shift; cmd_virtdiff "$@" ;;
+  down)     shift; cmd_down "$@" ;;
+  *) sed -n '2,30p' "$0"; exit 1 ;;
+esac

labsim/k8s-up.sh

@@ -0,0 +1,252 @@
+#!/bin/bash
+# A real Kubernetes cluster inside labsim, on the OVS fabric, for rehearsing
+# Cilium <-> VyOS BGP before it goes near the production routers.
+#
+# Why VMs and not k3d: the thing under test is eBGP between Cilium and VyOS
+# across the switch fabric — nodes on VLAN 2, peering with the router's bond0.2
+# leg, directly connected. k3d would put the nodes on a container bridge, which
+# is a different L2 path and would prove something else. (It also needs Docker;
+# this host has podman.)
+#
+# Why not the existing micro VMs: they are Alpine with 256 MB and 1 vCPU. k3s
+# plus Cilium needs an order of magnitude more, and a glibc distro with a stock
+# kernel that Cilium's eBPF probes are actually tested against.
+#
+# Three nodes, not two: ECMP is only meaningfully tested if a node can be
+# drained and MORE THAN ONE path survives.
+#
+# Layout (mirrors production's shape, not its addresses):
+#   labsim-k8s1   172.31.2.11   k3s server
+#   labsim-k8s2   172.31.2.12   agent
+#   labsim-k8s3   172.31.2.13   agent
+#   gateway       172.31.2.1    the VRRP VIP of the router pair under test
+#   BGP peers     172.31.2.252 / .253   the routers' real per-box addresses
+#
+# Idempotent: re-running only creates what is missing.
+#
+# Usage:
+#   ./k8s-up.sh              create/start the cluster
+#   ./k8s-up.sh --kubeconfig fetch kubeconfig to ./labsim-k8s.kubeconfig
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+source "$SCRIPT_DIR/lib.sh"
+source "$SCRIPT_DIR/ovs.sh"
+
+# --- knobs ----------------------------------------------------------------
+K8S_VLAN="${K8S_VLAN:-2}"
+K8S_PREFIX="${K8S_PREFIX:-172.31.2}"
+K8S_NODES="${K8S_NODES:-3}"
+K8S_FIRST_OCTET="${K8S_FIRST_OCTET:-11}"
+K8S_MEM="${K8S_MEM:-4096}"          # MB — k3s + cilium + a workload
+K8S_CPUS="${K8S_CPUS:-2}"
+K8S_DISK_GB="${K8S_DISK_GB:-12}"
+K8S_TOKEN="${K8S_TOKEN:-labsim-k3s-token}"
+
+# Debian rather than Alpine: glibc, a stock kernel, and cloud-init that applies
+# network-config properly (the Alpine base in this sim notably does not).
+DEB_URL="${DEB_URL:-https://cloud.debian.org/images/cloud/trixie/latest/debian-13-genericcloud-amd64.qcow2}"
+DEB_BASE="${DEB_BASE:-$IMG_DIR/debian-13-genericcloud-amd64.qcow2}"
+
+# Same version production runs, so CRD shapes and chart flags transfer exactly.
+CILIUM_VERSION="${CILIUM_VERSION:-1.19.1}"
+
+node_name() { echo "labsim-k8s$1"; }
+node_ip()   { echo "${K8S_PREFIX}.$((K8S_FIRST_OCTET + $1 - 1))"; }
+
+# --- base image -----------------------------------------------------------
+ensure_base_image() {
+  if [ -f "$DEB_BASE" ]; then
+    log "base image present: $(basename "$DEB_BASE")"
+    return
+  fi
+  log "fetching Debian cloud image (~330 MB) -> $DEB_BASE"
+  sudo mkdir -p "$IMG_DIR"
+  # .tmp + mv so an interrupted download never leaves a half image that later
+  # runs treat as valid.
+  sudo curl -fsSL --retry 3 -o "${DEB_BASE}.tmp" "$DEB_URL" \
+    || die "could not fetch $DEB_URL"
+  sudo mv "${DEB_BASE}.tmp" "$DEB_BASE"
+  log "base image ready"
+}
+
+# --- cloud-init -----------------------------------------------------------
+# The server node writes the join token; agents wait for the API to answer
+# before joining, because cloud-init ordering across VMs is not guaranteed and
+# a failed join leaves an agent that never retries.
+build_k8s_seed() {
+  local iso="$1" vm="$2" ip="$3" role="$4" server_ip="$5" pubkey="$6"
+  local tmp; tmp="$(mktemp -d)"
+
+  cat > "$tmp/meta-data" <<EOF
+instance-id: $vm
+local-hostname: $vm
+EOF
+
+  cat > "$tmp/network-config" <<EOF
+version: 2
+ethernets:
+  enp1s0:
+    match:
+      name: "en*"
+    addresses: [$ip/24]
+    routes:
+      - to: default
+        via: ${K8S_PREFIX}.1
+    nameservers:
+      addresses: [8.8.8.8, 1.1.1.1]
+EOF
+
+  local k3s_exec
+  if [ "$role" = "server" ]; then
+    # flannel/servicelb/traefik off: Cilium is the CNI under test, and k3s's
+    # own ServiceLB would fight Cilium for LoadBalancer addresses.
+    k3s_exec="server --flannel-backend=none --disable-network-policy --disable=servicelb --disable=traefik --node-ip=$ip --tls-san=$ip --cluster-init"
+  else
+    k3s_exec="agent --server https://${server_ip}:6443 --node-ip=$ip"
+  fi
+
+  cat > "$tmp/user-data" <<EOF
+#cloud-config
+hostname: $vm
+fqdn: $vm
+users:
+  - name: debian
+    groups: [sudo]
+    shell: /bin/bash
+    sudo: ["ALL=(ALL) NOPASSWD:ALL"]
+    lock_passwd: false
+    plain_text_passwd: labsim
+    ssh_authorized_keys:
+      - $pubkey
+ssh_pwauth: true
+disable_root: false
+ssh_authorized_keys:
+  - $pubkey
+
+package_update: true
+packages: [curl, jq, iproute2, tcpdump, bird2]
+
+write_files:
+  # Cilium replaces kube-proxy and needs these; Debian cloud images ship
+  # neither loaded nor persisted.
+  - path: /etc/modules-load.d/cilium.conf
+    content: |
+      br_netfilter
+      overlay
+  - path: /etc/sysctl.d/99-k8s.conf
+    content: |
+      net.ipv4.ip_forward = 1
+      net.bridge.bridge-nf-call-iptables = 1
+
+runcmd:
+  - [ modprobe, br_netfilter ]
+  - [ modprobe, overlay ]
+  - [ sysctl, --system ]
+  - |
+    # Wait for the server's API before an agent tries to join. Without this the
+    # agent fails once and the unit backs off for minutes.
+    if [ "$role" != "server" ]; then
+      for i in \$(seq 1 60); do
+        curl -sk --max-time 3 https://${server_ip}:6443/ping >/dev/null 2>&1 && break
+        sleep 5
+      done
+    fi
+  - |
+    curl -sfL https://get.k3s.io | \
+      INSTALL_K3S_EXEC="$k3s_exec" \
+      K3S_TOKEN="$K8S_TOKEN" \
+      sh -
+EOF
+
+  sudo mkdir -p "$(dirname "$iso")"
+  sudo genisoimage -quiet -output "$iso" -volid cidata -joliet -rock \
+    "$tmp/user-data" "$tmp/meta-data" "$tmp/network-config"
+  rm -rf "$tmp"
+}
+
+# --- VM creation ----------------------------------------------------------
+create_node() {
+  local n="$1" pubkey="$2"
+  local vm; vm="$(node_name "$n")"
+  local ip; ip="$(node_ip "$n")"
+  local role="agent"; [ "$n" -eq 1 ] && role="server"
+  local server_ip; server_ip="$(node_ip 1)"
+
+  if virsh_q dominfo "$vm" >/dev/null 2>&1; then
+    local state; state="$(virsh_q domstate "$vm" 2>/dev/null | head -1 | tr -d '\n')"
+    if [ "$state" = "running" ]; then
+      log "$vm already running ($ip, $role)"
+    else
+      log "$vm exists but is $state — starting"
+      virsh_q start "$vm" >/dev/null
+    fi
+    return
+  fi
+
+  local disk="$IMG_DIR/${vm}.qcow2"
+  local seed="$IMG_DIR/${vm}-seed.iso"
+
+  log "creating $vm ($ip, $role, ${K8S_MEM}MB/${K8S_CPUS}cpu)"
+  sudo qemu-img create -q -f qcow2 -F qcow2 -b "$DEB_BASE" "$disk" "${K8S_DISK_GB}G" >/dev/null
+  build_k8s_seed "$seed" "$vm" "$ip" "$role" "$server_ip" "$pubkey"
+
+  # Access port on the k8s VLAN — same broadcast domain as the routers'
+  # bond0.2 leg, so BGP peering is directly connected exactly as in production.
+  sudo virt-install --connect "$LIBVIRT_URI" --name "$vm" \
+    --memory "$K8S_MEM" --vcpus "$K8S_CPUS" \
+    --disk "path=$disk,format=qcow2,bus=virtio" \
+    --disk "path=$seed,device=cdrom" \
+    --network "network=$OVS_NET,portgroup=vlan${K8S_VLAN},model=virtio" \
+    --os-variant debian12 \
+    --graphics none --noautoconsole --import >/dev/null
+}
+
+fetch_kubeconfig() {
+  local server_ip; server_ip="$(node_ip 1)"
+  local out="$SCRIPT_DIR/labsim-k8s.kubeconfig"
+  log "fetching kubeconfig from $server_ip"
+  ssh -o StrictHostKeyChecking=no -o ConnectTimeout=10 \
+      "debian@${server_ip}" "sudo cat /etc/rancher/k3s/k3s.yaml" \
+    | sed "s|127.0.0.1|${server_ip}|" > "$out"
+  chmod 600 "$out"
+  log "wrote $out"
+  log "use: KUBECONFIG=$out kubectl get nodes"
+}
+
+main() {
+  if [ "${1:-}" = "--kubeconfig" ]; then
+    fetch_kubeconfig
+    return
+  fi
+
+  require_tools
+  command -v genisoimage >/dev/null || die "genisoimage missing (dnf install genisoimage)"
+
+  local pubkey; pubkey="$(find_ssh_pubkey)"
+  log "using SSH key: ${pubkey%% *} ...${pubkey##* }"
+
+  ensure_base_image
+
+  # Select EVERY VLAN, not just the k8s one. ovs_up re-defines the libvirt
+  # network from SELECTED, so narrowing it here silently drops the portgroups
+  # for every other VLAN -- running VMs keep working (their taps are already
+  # attached) and nothing complains until the next VM cannot be attached.
+  # Observed: this deleted vlan1/3/9/10/200/51/53 and only surfaced when the
+  # ISP VMs needed vlan51 and vlan53.
+  selected_vlans
+  log "ensuring OVS fabric (all VLANs, so no portgroup is dropped)"
+  ovs_up
+
+  for n in $(seq 1 "$K8S_NODES"); do
+    create_node "$n" "$pubkey"
+  done
+
+  echo
+  log "nodes created. k3s installs on first boot (a few minutes)."
+  log "watch:   ssh debian@$(node_ip 1) 'sudo systemctl status k3s'"
+  log "then:    $0 --kubeconfig"
+  log "then install Cilium $CILIUM_VERSION and the BGP resources (see README)."
+}
+
+main "$@"

labsim/labsim-dhcp-test.sh

@@ -0,0 +1,219 @@
+#!/bin/bash
+# Prove that VyOS hands each device the address UniFi reserved for it.
+#
+# The question this answers is narrow and important: 30 of the 31 UniFi
+# reservations sit INSIDE the DHCP pool (LoT's pool is 10.0.0.11-10.0.1.254 and
+# only 10.0.0.2 falls outside it). UniFi's dhcpd tolerates that. VyOS uses kea,
+# and whether kea honours in-pool host reservations decides whether the cutover
+# silently renumbers 30 devices. That is not something to predict.
+#
+# Method: boot throwaway VMs whose MAC is a REAL production MAC, on the sim
+# VLAN, and check the address they are given. MACs are the one piece of
+# production config that transplants verbatim -- the subnet is rewritten, the
+# MAC is not -- which is what makes this a real test rather than a rehearsal.
+#
+# Safe: the ovs-labsim bridge contains only internal ports and VM taps, with no
+# physical NIC, so a production MAC here cannot reach or confuse the real LAN.
+# Verified with `ovs-vsctl show` before this script was written.
+#
+#   ./labsim-dhcp-test.sh              run the standard cases
+#   ./labsim-dhcp-test.sh --keep       leave the VMs up for inspection
+#   ./labsim-dhcp-test.sh --clean      just remove any leftover test VMs
+set -uo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+source "$SCRIPT_DIR/lib.sh"
+
+ROUTER_IP="${ROUTER_IP:-172.31.1.1}"
+ROUTER_PW="${ROUTER_PW:-vyos}"
+TEST_VLAN="${TEST_VLAN:-10}"
+BOOT_WAIT="${BOOT_WAIT:-150}"
+TAG="labsim-dhcptest"
+
+# mac|expected|why. "POOL" means: must get an address from the pool and must
+# NOT get any reserved address -- the negative case that stops a pass from
+# meaning merely "DHCP works".
+CASES=(
+  "f8:0d:ac:90:65:c6|172.31.10.46|printer1 - reservation inside the pool"
+  "1c:69:20:7f:bc:77|172.31.11.67|sonoff-matter - in-pool AND across the /23 boundary"
+  "34:e1:d1:80:29:ce|172.31.10.2|Hubitat - the one reservation OUTSIDE the pool"
+  "52:54:00:ab:cd:ef|POOL|unreserved MAC - must get a pool address, not a reserved one"
+)
+
+vm_of() { echo "${TAG}-$(echo "$1" | tr -d ':')"; }
+
+cleanup_vms() {
+  local n=0
+  while read -r vm; do
+    [ -z "$vm" ] && continue
+    virsh_q destroy "$vm" >/dev/null 2>&1
+    virsh_q undefine "$vm" --remove-all-storage >/dev/null 2>&1
+    n=$((n + 1))
+  done < <(virsh_q list --all --name 2>/dev/null | grep "^${TAG}-" || true)
+  [ "$n" -gt 0 ] && log "removed $n test VM(s)"
+  sudo rm -f "$IMG_DIR/${TAG}-"*.qcow2 "$IMG_DIR/${TAG}-"*-seed.iso 2>/dev/null
+  return 0
+}
+
+# A seed that asks for DHCP instead of taking a static address. Alpine's
+# cloud-init ignores network-config here (verified previously and documented in
+# README), so /etc/network/interfaces is what actually takes effect.
+build_dhcp_seed() {
+  local iso="$1" vm="$2" pubkey="$3"
+  local tmp; tmp="$(mktemp -d)"
+  cat > "$tmp/meta-data" <<EOF
+instance-id: $vm
+local-hostname: $vm
+EOF
+  cat > "$tmp/user-data" <<EOF
+#cloud-config
+hostname: $vm
+users:
+  - name: alpine
+    shell: /bin/ash
+    lock_passwd: false
+    plain_text_passwd: labsim
+    ssh_authorized_keys:
+      - $pubkey
+ssh_authorized_keys:
+  - $pubkey
+disable_root: false
+chpasswd:
+  list: |
+    root:labsim
+  expire: false
+write_files:
+  - path: /etc/network/interfaces
+    content: |
+      auto lo
+      iface lo inet loopback
+      auto eth0
+      iface eth0 inet dhcp
+runcmd:
+  - [ sh, -c, "ifdown eth0 2>/dev/null; ifup eth0 || udhcpc -i eth0 -q || true" ]
+EOF
+  python3 - "$tmp/user-data" <<'PY' || die "generated user-data is not valid YAML"
+import sys, yaml
+yaml.safe_load(open(sys.argv[1]).read().split("#cloud-config",1)[1])
+PY
+  sudo genisoimage -quiet -output "$iso" -volid cidata -joliet -rock \
+    "$tmp/user-data" "$tmp/meta-data" >/dev/null 2>&1 || die "seed build failed"
+  rm -rf "$tmp"
+}
+
+router() {
+  timeout 30 sshpass -p "$ROUTER_PW" ssh -o StrictHostKeyChecking=no \
+    -o BatchMode=no -o ConnectTimeout=8 "vyos@$ROUTER_IP" "$@" 2>/dev/null
+}
+
+# --- argument handling ----------------------------------------------------
+KEEP=0
+case "${1:-}" in
+  --clean) cleanup_vms; exit 0 ;;
+  --keep)  KEEP=1 ;;
+  "")      ;;
+  *)       die "usage: $0 [--keep|--clean]" ;;
+esac
+
+command -v sshpass >/dev/null || die "sshpass required"
+require_tools
+[ -f "$BASE_IMAGE" ] || die "base image missing: $BASE_IMAGE (run labsim-up.sh first)"
+
+log "checking the router is serving DHCP..."
+subnets=$(router '/opt/vyatta/bin/vyatta-op-cmd-wrapper show configuration commands | grep -c subnet-id')
+maps=$(router '/opt/vyatta/bin/vyatta-op-cmd-wrapper show configuration commands | grep -c "static-mapping .* mac"')
+log "  router has ${subnets:-0} subnets and ${maps:-0} static-mappings"
+[ "${maps:-0}" -gt 0 ] || die "router has no static-mappings -- apply the generated config first"
+
+cleanup_vms
+
+# Flush the lease database first. This is not tidiness -- it is the condition
+# the cutover actually runs under, because kea does not inherit UniFi's leases
+# and starts empty. It also makes the test deterministic: with stale leases
+# present, kea saw the reserved address as held by "another client" (the same
+# MAC but a different client-id from a previous boot) and allocated a dynamic
+# address instead, which produced three misleading results before this existed.
+log "flushing the router's lease database (cutover starts with an empty one)"
+# Every dhcp4-leases.csv* must go, not just the main file: kea's memfile
+# backend keeps lease-file-cleanup rotations (.1/.2) and restores from them on
+# start, so truncating only the primary leaves the old leases intact.
+router 'sudo systemctl stop isc-kea-dhcp4-server;
+        sudo sh -c "rm -f /config/dhcp/dhcp4-leases.csv*";
+        sudo systemctl start isc-kea-dhcp4-server' >/dev/null
+sleep 5
+remaining="$(router '/opt/vyatta/bin/vyatta-op-cmd-wrapper show dhcp server leases' | sed -n '3,$p' | grep -c .)"
+[ "${remaining:-0}" -eq 0 ] || warn "lease table still has ${remaining} row(s) after flush"
+
+SSH_PUB="$(find_ssh_pubkey)"
+sudo mkdir -p "$IMG_DIR"
+
+# --- boot one VM per case -------------------------------------------------
+for c in "${CASES[@]}"; do
+  IFS='|' read -r mac expected why <<<"$c"
+  vm="$(vm_of "$mac")"
+  disk="$IMG_DIR/${vm}.qcow2"; seed="$IMG_DIR/${vm}-seed.iso"
+  log "booting $vm  mac=$mac  ($why)"
+  sudo qemu-img create -q -f qcow2 -F qcow2 -b "$BASE_IMAGE" "$disk" "$VM_DISK" >/dev/null
+  build_dhcp_seed "$seed" "$vm" "$SSH_PUB"
+  sudo virt-install --connect "$LIBVIRT_URI" --name "$vm" \
+    --memory "$VM_MEM" --vcpus "$VM_CPUS" \
+    --disk "path=$disk,format=qcow2,bus=virtio" \
+    --disk "path=$seed,device=cdrom,readonly=on" \
+    --network "network=labsim-ovs,portgroup=vlan${TEST_VLAN},model=virtio,mac=$mac" \
+    --os-variant alpinelinux3.18 --graphics none --noautoconsole --import >/dev/null \
+    || die "virt-install failed for $vm"
+done
+
+log "waiting ${BOOT_WAIT}s for boot + DHCP..."
+sleep "$BOOT_WAIT"
+
+# --- verdict --------------------------------------------------------------
+# The lease table on the router is the authority: it says what the server
+# decided, independent of whether the guest brought the interface up cleanly.
+leases="$(router '/opt/vyatta/bin/vyatta-op-cmd-wrapper show dhcp server leases')"
+echo
+echo "=== router lease table ==="
+echo "$leases"
+echo
+
+reserved_ips="$(cd "$SCRIPT_DIR/../migration" && python3 unifi-to-vyos.py --mode sim 2>/dev/null \
+  | awk '/static-mapping .* ip-address/ {print $NF}')"
+
+pass=0; fail=0
+printf '%-19s %-16s %-16s %s\n' "MAC" "EXPECTED" "GOT" "RESULT"
+for c in "${CASES[@]}"; do
+  IFS='|' read -r mac expected why <<<"$c"
+  # Never guess which lease is "the" lease. Taking the first match is how an
+  # hours-old lease was once reported as the current answer, turning three
+  # failures into apparent passes.
+  matches="$(echo "$leases" | awk -v m="$mac" 'tolower($2) == tolower(m) {print $1}')"
+  n_match="$(echo "$matches" | grep -c . )"
+  if [ "$n_match" -gt 1 ]; then
+    got="AMBIGUOUS($(echo "$matches" | tr '\n' ',' | sed 's/,$//'))"
+  else
+    got="${matches:-<none>}"
+  fi
+  if [ "${got#AMBIGUOUS}" != "$got" ]; then
+    # More than one lease for this MAC means the flush did not take. Any
+    # verdict from here is a guess, so refuse to give one.
+    result="FAIL (multiple leases -- flush did not take)"
+  elif [ "$expected" = "POOL" ]; then
+    if [ "$got" = "<none>" ]; then
+      result="FAIL (no lease at all)"
+    elif echo "$reserved_ips" | grep -qx "$got"; then
+      result="FAIL (got a RESERVED address)"
+    else
+      result="pass"
+    fi
+  else
+    [ "$got" = "$expected" ] && result="pass" || result="FAIL"
+  fi
+  [ "$result" = "pass" ] && pass=$((pass + 1)) || fail=$((fail + 1))
+  printf '%-19s %-16s %-16s %s\n' "$mac" "$expected" "$got" "$result"
+  printf '  %s\n' "$why"
+done
+
+echo
+log "$pass passed, $fail failed"
+[ "$KEEP" -eq 1 ] && log "VMs left running (--keep). Remove with: $0 --clean" || cleanup_vms
+[ "$fail" -eq 0 ] || exit 1

labsim/labsim-down.sh

@@ -0,0 +1,61 @@
+#!/bin/bash
+# Tear down the lab network simulation.
+#
+# By default this destroys VMs and networks but KEEPS the downloaded base
+# image, so the next bring-up is fast. Pass --purge to remove that too.
+#
+# Usage: ./labsim-down.sh [--purge] [vlan-id ...]
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+source "$SCRIPT_DIR/lib.sh"
+source "$SCRIPT_DIR/ovs.sh"
+
+PURGE=false
+ARGS=()
+for a in "$@"; do
+  case "$a" in
+    --purge) PURGE=true ;;
+    *) ARGS+=("$a") ;;
+  esac
+done
+
+selected_vlans "${ARGS[@]+"${ARGS[@]}"}"
+
+for entry in "${SELECTED[@]}"; do
+  IFS=: read -r vid name prefix _r <<<"$entry"
+  vm="$(vm_name "$vid" "$name")"
+
+  if virsh_q dominfo "$vm" >/dev/null 2>&1; then
+    log "destroying VM $vm"
+    virsh_q destroy "$vm" >/dev/null 2>&1 || true
+    virsh_q undefine "$vm" --nvram >/dev/null 2>&1 || virsh_q undefine "$vm" >/dev/null 2>&1 || true
+  fi
+  sudo rm -f "$IMG_DIR/${vm}.qcow2" "$IMG_DIR/${vm}-seed.iso"
+
+done
+
+# Legacy per-VLAN Linux-bridge networks from before the OVS migration. If
+# these survive they keep a duplicate <prefix>.2/24 on a dead bridge, and the
+# kernel may prefer that route over the OVS host leg — which looks exactly
+# like "the VM is unreachable" while ping -I hostvN works fine.
+for entry in "${SELECTED[@]}"; do
+  IFS=: read -r vid _n _p _r <<<"$entry"
+  legacy="labsim-vlan${vid}"
+  if virsh_q net-info "$legacy" >/dev/null 2>&1; then
+    log "removing legacy network $legacy"
+    virsh_q net-destroy "$legacy" >/dev/null 2>&1 || true
+    virsh_q net-undefine "$legacy" >/dev/null 2>&1 || true
+  fi
+done
+
+log "removing OVS fabric"
+ovs_down
+
+if [ "$PURGE" = true ]; then
+  log "purging base image $BASE_IMAGE"
+  sudo rm -f "$BASE_IMAGE"
+  sudo rmdir "$IMG_DIR" 2>/dev/null || true
+fi
+
+log "environment is DOWN"

labsim/labsim-exporter.py

@@ -0,0 +1,157 @@
+#!/usr/bin/env python3
+"""Prometheus exporter for the labsim connectivity matrix.
+
+Runs the same sweep as labsim-matrix.py on an interval and exposes it as
+metrics, so Grafana can show the mesh as a heatmap and — more usefully — a
+history of exactly when a cell flipped after a firewall change.
+
+  labsim_reachable{src,dst,proto}   1 = reachable, 0 = blocked
+  labsim_sweep_seconds              how long the last sweep took
+  labsim_sweep_total                sweeps completed since start
+  labsim_up                         1 while the exporter is alive
+
+Deliberately stdlib-only (http.server + threads): this runs on the workstation
+next to libvirt, and adding a dependency to watch a lab network is silly.
+
+  ./labsim-exporter.py --port 9101 --interval 15
+"""
+from __future__ import annotations
+
+import argparse
+import http.server
+import json
+import os
+import threading
+import time
+
+import labsim_matrix_lib as m  # thin import shim, see below
+
+
+class Collector:
+    def __init__(self, interval: int, timeout: int) -> None:
+        self.interval = interval
+        self.timeout = timeout
+        self.vlans = m.load_vlans()
+        self.lock = threading.Lock()
+        self.results: dict = {}
+        self.duration = 0.0
+        self.sweeps = 0
+
+    def loop(self) -> None:
+        while True:
+            started = time.time()
+            try:
+                results = m.sweep(self.vlans, self.timeout)
+                with self.lock:
+                    self.results = results
+                    self.duration = time.time() - started
+                    self.sweeps += 1
+            except Exception:  # noqa: BLE001 - never let the loop die
+                pass
+            time.sleep(max(1.0, self.interval - (time.time() - started)))
+
+    def snapshot(self) -> dict:
+        """Everything the topology page needs, in one JSON payload."""
+        with self.lock:
+            results, duration = dict(self.results), self.duration
+        reach = total = 0
+        for data in results.values():
+            if "__error__" in data:
+                continue
+            for protos in data.values():
+                for proto, ok in protos.items():
+                    if proto == "rtt_ms":
+                        continue
+                    total += 1
+                    if ok:
+                        reach += 1
+        return {"vlans": self.vlans, "results": results, "reachable": reach,
+                "total": total, "sweep_seconds": duration}
+
+    def render(self) -> str:
+        with self.lock:
+            results, duration, sweeps = dict(self.results), self.duration, self.sweeps
+
+        out = [
+            "# HELP labsim_reachable 1 if dst is reachable from src over proto",
+            "# TYPE labsim_reachable gauge",
+        ]
+        rtts = []
+        for src, data in results.items():
+            if "__error__" in data:
+                continue
+            for dst, protos in data.items():
+                for proto, ok in protos.items():
+                    if proto == "rtt_ms":
+                        if isinstance(ok, (int, float)):
+                            rtts.append((src, dst, ok))
+                        continue
+                    out.append(
+                        f'labsim_reachable{{src="{src}",dst="{dst}",proto="{proto}"}} {1 if ok else 0}')
+        out += ["# HELP labsim_rtt_ms ICMP round-trip time",
+                "# TYPE labsim_rtt_ms gauge"]
+        for src, dst, val in rtts:
+            out.append(f'labsim_rtt_ms{{src="{src}",dst="{dst}"}} {val}')
+        out += [
+            "# HELP labsim_sweep_seconds duration of the last sweep",
+            "# TYPE labsim_sweep_seconds gauge",
+            f"labsim_sweep_seconds {duration:.3f}",
+            "# HELP labsim_sweep_total sweeps completed",
+            "# TYPE labsim_sweep_total counter",
+            f"labsim_sweep_total {sweeps}",
+            "# HELP labsim_up exporter liveness",
+            "# TYPE labsim_up gauge",
+            "labsim_up 1",
+        ]
+        return "\n".join(out) + "\n"
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--port", type=int, default=9101)
+    ap.add_argument("--interval", type=int, default=15)
+    ap.add_argument("--timeout", type=int, default=30)
+    args = ap.parse_args()
+
+    collector = Collector(args.interval, args.timeout)
+    threading.Thread(target=collector.loop, daemon=True).start()
+
+    here = os.path.dirname(os.path.abspath(__file__))
+
+    class Handler(http.server.BaseHTTPRequestHandler):
+        def _send(self, body: bytes, ctype: str) -> None:
+            self.send_response(200)
+            self.send_header("Content-Type", ctype)
+            self.send_header("Content-Length", str(len(body)))
+            self.send_header("Cache-Control", "no-store")
+            self.end_headers()
+            self.wfile.write(body)
+
+        def do_GET(self) -> None:  # noqa: N802 - stdlib API
+            path = self.path.split("?")[0].rstrip("/")
+            if path in ("", "/topology"):
+                # Live topology view — the thing you actually watch.
+                try:
+                    with open(os.path.join(here, "topology.html"), "rb") as fh:
+                        self._send(fh.read(), "text/html; charset=utf-8")
+                except OSError:
+                    self.send_error(500, "topology.html missing")
+            elif path == "/api/matrix":
+                self._send(json.dumps(collector.snapshot()).encode(), "application/json")
+            elif path == "/metrics":
+                self._send(collector.render().encode(), "text/plain; version=0.0.4")
+            else:
+                self.send_error(404)
+
+        def log_message(self, *_args) -> None:  # keep the console quiet
+            return
+
+    srv = http.server.ThreadingHTTPServer(("0.0.0.0", args.port), Handler)
+    print(f"labsim topology  http://localhost:{args.port}/")
+    print(f"labsim metrics   http://localhost:{args.port}/metrics  (sweep every {args.interval}s)")
+    srv.serve_forever()
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

labsim/labsim-matrix.py

@@ -0,0 +1,208 @@
+#!/usr/bin/env python3
+"""Full-mesh connectivity matrix for the labsim VLANs.
+
+Probes every VLAN VM from every other VLAN VM (ICMP + TCP/22 + TCP/80) and
+prints a grid. Use --watch to keep it live: cells that changed since the last
+sweep are highlighted, so adding or removing a VyOS firewall rule shows up
+within one refresh.
+
+Deliberately dependency-free on the guests: the probe runs with python3, which
+is already installed there (cloud-init needs it), so nothing has to be
+installed on VMs that have no internet.
+
+  ./labsim-matrix.py                 # one sweep
+  ./labsim-matrix.py --watch         # live, refresh every 5s
+  ./labsim-matrix.py --watch 2       # live, every 2s
+  ./labsim-matrix.py --proto icmp    # single protocol
+  ./labsim-matrix.py --json          # machine-readable
+"""
+from __future__ import annotations
+
+import argparse
+import concurrent.futures
+import json
+import os
+import subprocess
+import sys
+import time
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+CONF = os.path.join(HERE, "vlans.conf")
+
+GREEN, RED, GREY, YELLOW, BOLD, RESET = (
+    "\033[0;32m", "\033[0;31m", "\033[0;90m", "\033[1;33m", "\033[1m", "\033[0m")
+
+PROTOS = ("icmp", "tcp22", "tcp80")
+
+# Runs ON the guest. Keep it stdlib-only and quick — a hung probe delays the
+# whole sweep, so every check is hard-bounded by a timeout.
+PROBE = r'''
+import json, re, socket, subprocess, sys
+targets = json.load(sys.stdin)
+out = {}
+for name, ip in targets.items():
+    res = {}
+    try:
+        p = subprocess.run(["ping", "-c", "1", "-W", "1", ip],
+                           stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, timeout=4)
+        res["icmp"] = p.returncode == 0
+        # RTT as well as pass/fail: a path that is up but slow is a different
+        # problem from one that is down, and the grid alone cannot show it.
+        res["rtt_ms"] = None
+        if res["icmp"]:
+            m = re.search(r"time[=<]\s*([0-9.]+)\s*ms", p.stdout.decode("utf-8", "replace"))
+            if m:
+                res["rtt_ms"] = float(m.group(1))
+    except Exception:
+        res["icmp"] = False
+        res["rtt_ms"] = None
+    for port in (22, 80):
+        s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+        s.settimeout(1.5)
+        try:
+            s.connect((ip, port)); res["tcp%d" % port] = True
+        except Exception:
+            res["tcp%d" % port] = False
+        finally:
+            try: s.close()
+            except Exception: pass
+    out[name] = res
+print(json.dumps(out))
+'''
+
+
+def load_vlans() -> list[dict]:
+    vlans = []
+    with open(CONF) as fh:
+        for line in fh:
+            line = line.strip()
+            if not line or line.startswith("#"):
+                continue
+            # masklen and host_octet are optional trailing fields; VLAN 10 sets
+            # both because it must be a /23 (see vlans.conf).
+            parts = line.split(":")
+            vid, name, prefix, real = parts[0], parts[1], parts[2], parts[3]
+            masklen = int(parts[4]) if len(parts) > 4 and parts[4] else 24
+            host = parts[5] if len(parts) > 5 and parts[5] else "2"
+            vlans.append({"vid": vid, "name": name, "ip": f"{prefix}.10",
+                          "label": f"{vid}:{name}", "real": real,
+                          "masklen": masklen, "host_ip": f"{prefix}.{host}"})
+    return vlans
+
+
+def probe_from(src: dict, targets: list[dict], timeout: int) -> tuple[str, dict]:
+    """SSH once into src and probe every target from there."""
+    payload = json.dumps({t["label"]: t["ip"] for t in targets if t["label"] != src["label"]})
+    cmd = [
+        "ssh", "-o", "StrictHostKeyChecking=no", "-o", "UserKnownHostsFile=/dev/null",
+        "-o", "BatchMode=yes", "-o", "ConnectTimeout=5", "-o", "LogLevel=ERROR",
+        f"alpine@{src['ip']}", "python3", "-",
+    ]
+    try:
+        # The probe script goes on stdin, the target list follows it — the guest
+        # reads the script from argv-less stdin, so send both in one stream.
+        proc = subprocess.run(
+            cmd, input=PROBE.replace("json.load(sys.stdin)", f"json.loads({payload!r})"),
+            capture_output=True, text=True, timeout=timeout)
+        if proc.returncode != 0:
+            return src["label"], {"__error__": (proc.stderr or "ssh failed").strip()[:60]}
+        return src["label"], json.loads(proc.stdout)
+    except subprocess.TimeoutExpired:
+        return src["label"], {"__error__": "probe timed out"}
+    except Exception as exc:  # noqa: BLE001 - report, never crash the sweep
+        return src["label"], {"__error__": f"{type(exc).__name__}: {exc}"[:60]}
+
+
+def sweep(vlans: list[dict], timeout: int) -> dict:
+    results: dict = {}
+    with concurrent.futures.ThreadPoolExecutor(max_workers=len(vlans)) as pool:
+        futures = [pool.submit(probe_from, v, vlans, timeout) for v in vlans]
+        for fut in concurrent.futures.as_completed(futures):
+            label, data = fut.result()
+            results[label] = data
+    return results
+
+
+def cell(ok: bool | None, changed: bool) -> str:
+    if ok is None:
+        return f"{GREY} ·  {RESET}"
+    mark = "ok  " if ok else "--  "
+    colour = GREEN if ok else RED
+    if changed:
+        return f"{YELLOW}{BOLD}{'OK*' if ok else 'XX*':<4}{RESET}"
+    return f"{colour}{mark}{RESET}"
+
+
+def render(vlans: list[dict], results: dict, prev: dict | None, protos: tuple[str, ...]) -> None:
+    labels = [v["label"] for v in vlans]
+    width = max(len(x) for x in labels) + 2
+
+    for proto in protos:
+        print(f"\n{BOLD}{proto.upper()}{RESET}   (rows = source, columns = destination)")
+        header = " " * width + "".join(f"{lbl:<{width}}" for lbl in labels)
+        print(f"{GREY}{header}{RESET}")
+
+        for src in vlans:
+            row = f"{src['label']:<{width}}"
+            data = results.get(src["label"], {})
+            if "__error__" in data:
+                print(row + f"{RED}{data['__error__']}{RESET}")
+                continue
+            for dst in vlans:
+                if dst["label"] == src["label"]:
+                    row += f"{GREY}{'·':<{width}}{RESET}"
+                    continue
+                ok = data.get(dst["label"], {}).get(proto)
+                was = (prev or {}).get(src["label"], {}).get(dst["label"], {}).get(proto)
+                changed = prev is not None and was is not None and was != ok
+                txt = cell(ok, changed)
+                row += txt + " " * (width - 4)
+            print(row)
+
+    reach = sum(1 for s in results.values() if "__error__" not in s
+                for d in s.values() for p in protos if d.get(p) is True)
+    total = sum(1 for s in results.values() if "__error__" not in s
+                for _d in s.values() for _p in protos)
+    print(f"\n  reachable: {reach}/{total}   "
+          f"{GREEN}ok{RESET}=allowed  {RED}--{RESET}=blocked/no route  "
+          f"{YELLOW}*{RESET}=changed since last sweep")
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser(description=__doc__,
+                                 formatter_class=argparse.RawDescriptionHelpFormatter)
+    ap.add_argument("--watch", nargs="?", const=5, type=int, metavar="SECONDS",
+                    help="refresh continuously (default every 5s)")
+    ap.add_argument("--proto", choices=PROTOS, help="only this protocol")
+    ap.add_argument("--json", action="store_true", help="emit raw JSON and exit")
+    ap.add_argument("--timeout", type=int, default=30, help="per-host probe timeout")
+    args = ap.parse_args()
+
+    vlans = load_vlans()
+    protos = (args.proto,) if args.proto else PROTOS
+
+    if args.json:
+        print(json.dumps(sweep(vlans, args.timeout), indent=2))
+        return 0
+
+    prev = None
+    while True:
+        started = time.time()
+        results = sweep(vlans, args.timeout)
+        if args.watch:
+            os.system("clear")
+            print(f"{BOLD}labsim connectivity matrix{RESET}   "
+                  f"{time.strftime('%H:%M:%S')}   (refresh {args.watch}s, Ctrl-C to stop)")
+        render(vlans, results, prev, protos)
+        if not args.watch:
+            return 0
+        prev = results
+        time.sleep(max(0.0, args.watch - (time.time() - started)))
+
+
+if __name__ == "__main__":
+    try:
+        sys.exit(main())
+    except KeyboardInterrupt:
+        print()
+        sys.exit(130)

labsim/labsim-up.sh

@@ -0,0 +1,71 @@
+#!/bin/bash
+# Bring up the lab network simulation: one isolated libvirt network per VLAN,
+# each with a single tiny Alpine VM offering SSH + a hello-world HTTP page.
+#
+# Idempotent: re-running only creates what is missing. Safe to run repeatedly.
+#
+# Usage: ./labsim-up.sh [vlan-id ...]      (default: every VLAN in vlans.conf)
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+source "$SCRIPT_DIR/lib.sh"
+source "$SCRIPT_DIR/ovs.sh"
+
+require_tools
+[ -f "$BASE_IMAGE" ] || die "base image missing: $BASE_IMAGE (see README)"
+
+SSH_PUB="$(find_ssh_pubkey)"
+log "Using SSH key: ${SSH_PUB%% *} ...${SSH_PUB##* }"
+
+selected_vlans "$@"
+
+# --- switch fabric ------------------------------------------------------
+log "bringing up OVS fabric ($OVS_BR) with host legs per VLAN"
+ovs_up
+
+# --- VMs ------------------------------------------------------------------
+for entry in "${SELECTED[@]}"; do
+  parse_vlan_entry "$entry"
+  vid="$V_VID"; name="$V_NAME"; prefix="$V_PREFIX"; real="$V_REAL"
+  vm="$(vm_name "$vid" "$name")"
+  ip="${prefix}.10"
+
+  if virsh_q dominfo "$vm" >/dev/null 2>&1; then
+    state="$(virsh_q domstate "$vm" 2>/dev/null | head -1 | tr -d '\n')"
+    if [ "$state" = "running" ]; then
+      log "VM $vm already running ($ip)"
+      continue
+    fi
+    log "VM $vm exists but is $state — starting"
+    virsh_q start "$vm" >/dev/null
+    continue
+  fi
+
+  log "creating VM $vm  ($ip on vlan $vid/$name)"
+
+  disk="$IMG_DIR/${vm}.qcow2"
+  seed="$IMG_DIR/${vm}-seed.iso"
+
+  # Copy-on-write overlay: each VM costs a few MB, not 176.
+  sudo qemu-img create -q -f qcow2 -F qcow2 -b "$BASE_IMAGE" "$disk" "$VM_DISK" >/dev/null
+
+  build_seed "$seed" "$vm" "$vid" "$name" "$prefix" "$ip" "$real" "$SSH_PUB" "$V_MASK"
+
+  sudo virt-install \
+    --connect "$LIBVIRT_URI" \
+    --name "$vm" \
+    --memory "$VM_MEM" --vcpus "$VM_CPUS" \
+    --disk "path=$disk,format=qcow2,bus=virtio" \
+    --disk "path=$seed,device=cdrom,readonly=on" \
+    --network "network=$OVS_NET,portgroup=vlan${vid},model=virtio" \
+    --os-variant alpinelinux3.18 \
+    --graphics none --noautoconsole --import >/dev/null
+done
+
+echo
+log "waiting for VMs to answer on SSH + HTTP..."
+wait_ready
+echo
+status_table
+echo
+log "environment is UP. Tear down with: $SCRIPT_DIR/labsim-down.sh"

labsim/labsim-vlan-leak-test.sh

@@ -0,0 +1,178 @@
+#!/bin/bash
+# Does the router offer an address from the WRONG VLAN's pool?
+#
+# The fault (ISC Kea #1117, "Mix of physical and virtual interfaces (VLAN) does
+# not work"): with `dhcp-socket-type: raw`, a frame tagged for a sub-interface is
+# ALSO delivered to the PARENT's AF_PACKET socket. Kea then selects a subnet from
+# the parent's own address and answers a second time from the wrong pool. Both
+# offers race to the client and the CLIENT decides which one wins -- which is why
+# the symptom looks device-dependent and unreproducible.
+#
+# Production and this sim have the identical shape that triggers it: Management
+# is the NATIVE/untagged VLAN on `bond0` and therefore has a subnet on the
+# parent, while every other VLAN is a `bond0.<vif>` sub-interface of that same
+# bond.
+#
+# Method: make one DHCP client on a TAGGED VLAN send a DISCOVER, and capture
+# simultaneously on the parent and on the sub-interface. The verdict is not
+# "did the client get the right address" -- the client picking correctly is
+# exactly how this hid for weeks. The verdict is how many OFFERs the SERVER
+# emitted and which source addresses they carried.
+#
+#   ./labsim-vlan-leak-test.sh                 test VLAN 3
+#   ./labsim-vlan-leak-test.sh --vlan 9        test another VLAN
+#   ./labsim-vlan-leak-test.sh --save before   also write the raw captures to
+#                                              vlan-leak-evidence/before/
+set -uo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+
+ROUTER_IP="${ROUTER_IP:-172.31.1.1}"
+ROUTER_PW="${ROUTER_PW:-vyos}"
+CLIENT_PW="${CLIENT_PW:-labsim}"
+VLAN=3
+CLIENT=""
+SAVE=""
+
+while [ $# -gt 0 ]; do
+  case "$1" in
+    --vlan)   VLAN="$2"; shift 2 ;;
+    --client) CLIENT="$2"; shift 2 ;;
+    --save)   SAVE="$2"; shift 2 ;;
+    *) echo "usage: $0 [--vlan N] [--client IP] [--save LABEL]" >&2; exit 2 ;;
+  esac
+done
+: "${CLIENT:=172.31.${VLAN}.10}"
+
+log() { printf '\033[36m==>\033[0m %s\n' "$*"; }
+die() { printf '\033[31merror:\033[0m %s\n' "$*" >&2; exit 1; }
+
+command -v sshpass >/dev/null || die "sshpass required"
+
+# A silent router is the one verdict worth double-checking before reporting.
+#
+# Kea can be `is-active` and answering nothing -- it reopens sockets on a retry
+# loop, and some configurations (`listen-interface`, notably) leave individual
+# VLANs dead while the rest work. Both look identical to a one-shot test: "no
+# reply at all". Two opposite and equally wrong conclusions about
+# `listen-interface` came out of believing a single negative run, in both
+# directions, before a retry made the real pattern obvious.
+#
+# Kea's fallback UDP socket appearing is NOT a readiness signal -- it is bound
+# well before the server actually answers. Checked, and it does not work.
+RETRIED="${RETRIED:-0}"
+
+router() {
+  timeout 40 sshpass -p "$ROUTER_PW" ssh -o StrictHostKeyChecking=no \
+    -o ConnectTimeout=8 "vyos@$ROUTER_IP" "$@" 2>/dev/null
+}
+# VyOS's login shell is vbash, which returns 255 on anything it does not like --
+# in particular a backgrounded job. Feeding the script to `bash -s` on stdin
+# sidesteps vbash entirely and is the only reliable way to leave a daemon behind.
+router_sh() {
+  timeout 40 sshpass -p "$ROUTER_PW" ssh -o StrictHostKeyChecking=no \
+    -o ConnectTimeout=8 "vyos@$ROUTER_IP" 'bash -s' 2>/dev/null
+}
+client() {
+  timeout 60 sshpass -p "$CLIENT_PW" ssh -o StrictHostKeyChecking=no \
+    -o ConnectTimeout=8 "root@$CLIENT" "$@" 2>/dev/null
+}
+
+# Which interfaces to watch. The parent is the whole point: after the fix it
+# should carry no DHCP traffic of its own at all.
+PARENT="bond0"
+VIF="bond0.${VLAN}"
+
+log "router $ROUTER_IP -- capturing on $PARENT and $VIF"
+# Kill EVERY tcpdump first, not just ones matching this run's pattern, and count
+# only afterwards. Counting `pgrep -f 'tcpdump -i bond0'` while a stray tcpdump
+# from an earlier session was still running satisfied the >=2 guard with zero of
+# THIS run's captures alive -- and a capture that records nothing reports
+# "the router sent no reply at all", which reads as a DHCP outage. That sent me
+# chasing a fault in the router that was entirely in the test harness.
+started="$(router_sh <<EOF
+sudo pkill -x tcpdump >/dev/null 2>&1
+sleep 1
+sudo rm -f /tmp/leak-*.txt
+sudo nohup tcpdump -i $PARENT -e -nn -l 'udp port 67 or udp port 68' > /tmp/leak-parent.txt 2>/dev/null &
+sudo nohup tcpdump -i $VIF -e -nn -l 'udp port 67 or udp port 68' > /tmp/leak-vif.txt 2>/dev/null &
+sleep 3
+pgrep -c -x tcpdump
+EOF
+)"
+[ "${started:-0}" -eq 2 ] || die "capture did not start on the router (got ${started:-0}, expected exactly 2)"
+
+# -s /bin/true: ask, observe the answer, apply nothing. The client's existing
+# static address is left alone, so this is safe to run against a live sim VM.
+log "client $CLIENT -- sending DISCOVER on VLAN $VLAN"
+client_out="$(client "udhcpc -n -q -f -i eth0 -s /bin/true -t 3 -T 3 2>&1")"
+[ -n "$client_out" ] || die "no response from client $CLIENT"
+
+sleep 2
+router "sudo pkill -x tcpdump" >/dev/null
+parent="$(router 'sudo cat /tmp/leak-parent.txt')"
+vif="$(router 'sudo cat /tmp/leak-vif.txt')"
+
+echo
+echo "--- client ---"
+echo "$client_out" | sed 's/^/  /'
+echo
+echo "--- $PARENT (parent) ---"
+echo "${parent:-  (nothing)}" | sed 's/^/  /'
+echo
+echo "--- $VIF (sub-interface) ---"
+echo "${vif:-  (nothing)}" | sed 's/^/  /'
+echo
+
+if [ -n "$SAVE" ]; then
+  d="$SCRIPT_DIR/vlan-leak-evidence/$SAVE"
+  mkdir -p "$d"
+  printf '%s\n' "$client_out" > "$d/client.txt"
+  printf '%s\n' "$parent"     > "$d/capture-parent.txt"
+  printf '%s\n' "$vif"        > "$d/capture-vif.txt"
+  router '/opt/vyatta/bin/vyatta-op-cmd-wrapper show configuration commands' \
+    | grep -E 'interfaces bonding|vrrp group' > "$d/router-config.txt"
+  log "evidence saved to vlan-leak-evidence/$SAVE/"
+fi
+
+# --- verdict ---------------------------------------------------------------
+# Every BOOTP Reply seen anywhere, reduced to its source address. A reply whose
+# source is not this VLAN's router leg is an offer from the wrong subnet.
+replies="$(printf '%s\n%s\n' "$parent" "$vif" \
+  | grep -o '[0-9.]*\.67 > [0-9.]*\.68' | awk '{print $1}' | sed 's/\.67$//' \
+  | sort -u)"
+want_prefix="172.31.${VLAN}."
+
+echo "=== verdict ==="
+if [ -z "$replies" ]; then
+  if [ "$RETRIED" -eq 0 ]; then
+    log "no reply -- retrying once in 20s before calling DHCP down"
+    sleep 20; RETRIED=1 exec "$0" --vlan "$VLAN" --client "$CLIENT" ${SAVE:+--save "$SAVE"}
+  fi
+  echo "INCONCLUSIVE: the router sent no reply at all, twice -- DHCP is down on VLAN $VLAN"
+  exit 2
+fi
+
+bad=0
+while read -r src; do
+  [ -z "$src" ] && continue
+  case "$src" in
+    "$want_prefix"*) printf '  ok    offer from %s (this VLAN)\n' "$src" ;;
+    *)               printf '  LEAK  offer from %s (WRONG subnet)\n' "$src"; bad=1 ;;
+  esac
+done <<<"$replies"
+
+# The parent carrying any DHCP of its own is the mechanism, not just a symptom:
+# it means the parent still has a subnet kea can match a tagged frame against.
+if printf '%s' "$parent" | grep -q 'ethertype IPv4' \
+   && printf '%s' "$parent" | grep -v 'vlan ' | grep -q '\.67 > '; then
+  echo "  note  $PARENT emitted an UNTAGGED reply -- the parent still serves a subnet"
+fi
+
+echo
+if [ "$bad" -eq 0 ]; then
+  echo "PASS: only this VLAN's pool answered."
+  exit 0
+fi
+echo "FAIL: the router answered from another VLAN's pool (kea #1117)."
+exit 1

labsim/lib.sh

@@ -0,0 +1,239 @@
+#!/bin/bash
+# Shared helpers for the lab network simulation.
+# shellcheck disable=SC2034
+
+LIBVIRT_URI="${LIBVIRT_URI:-qemu:///system}"
+IMG_DIR="${IMG_DIR:-/var/lib/libvirt/images/labsim}"
+BASE_IMAGE="${BASE_IMAGE:-$IMG_DIR/alpine-base.qcow2}"
+ALPINE_URL="${ALPINE_URL:-https://dl-cdn.alpinelinux.org/alpine/latest-stable/releases/cloud/generic_alpine-3.24.1-x86_64-bios-cloudinit-r0.qcow2}"
+
+VM_MEM="${VM_MEM:-256}"      # MB — Alpine is happy here
+VM_CPUS="${VM_CPUS:-1}"
+VM_DISK="${VM_DISK:-1G}"
+PREFIX="labsim"
+
+CONF="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/vlans.conf"
+
+log()  { printf '\033[0;36m[labsim]\033[0m %s\n' "$*"; }
+warn() { printf '\033[1;33m[labsim]\033[0m %s\n' "$*" >&2; }
+die()  { printf '\033[0;31m[labsim]\033[0m %s\n' "$*" >&2; exit 1; }
+
+virsh_q() { sudo virsh --connect "$LIBVIRT_URI" "$@"; }
+
+net_name() { echo "${PREFIX}-vlan$1"; }
+vm_name()  { echo "${PREFIX}-$1-$2"; }          # labsim-2-k8s
+# Linux bridge names are capped at 15 chars — keep it short and unique.
+br_name()  { echo "vbr-ls$1"; }
+
+require_tools() {
+  for t in virsh virt-install qemu-img genisoimage; do
+    command -v "$t" >/dev/null 2>&1 || die "missing required tool: $t"
+  done
+  sudo -n true 2>/dev/null || warn "sudo may prompt for a password"
+}
+
+find_ssh_pubkey() {
+  local home="${SUDO_USER:+/home/$SUDO_USER}"
+  home="${home:-$HOME}"
+  for n in id_ed25519 id_ecdsa id_rsa; do
+    [ -f "$home/.ssh/$n.pub" ] && { cat "$home/.ssh/$n.pub"; return; }
+  done
+  die "no SSH public key found in $home/.ssh"
+}
+
+# Populate SELECTED[] from argv (VLAN ids) or the whole config.
+selected_vlans() {
+  SELECTED=()
+  local want=("$@")
+  while IFS= read -r line; do
+    [[ "$line" =~ ^[[:space:]]*# ]] && continue
+    [[ -z "${line// }" ]] && continue
+    local vid="${line%%:*}"
+    if [ ${#want[@]} -eq 0 ]; then
+      SELECTED+=("$line")
+    else
+      for w in "${want[@]}"; do [ "$w" = "$vid" ] && SELECTED+=("$line"); done
+    fi
+  done < "$CONF"
+  [ ${#SELECTED[@]} -gt 0 ] || die "no VLANs selected (checked $CONF)"
+}
+
+# Split one vlans.conf line, applying defaults for the two optional trailing
+# fields. Sets V_VID V_NAME V_PREFIX V_REAL V_MASK V_HOST.
+parse_vlan_entry() {
+  IFS=: read -r V_VID V_NAME V_PREFIX V_REAL V_MASK V_HOST <<<"$1"
+  V_MASK="${V_MASK:-24}"
+  V_HOST="${V_HOST:-2}"
+}
+
+# Dotted netmask for a prefix length — cloud-init's network-config v1 wants the
+# dotted form, not a /len. /24 -> 255.255.255.0, /23 -> 255.255.254.0.
+netmask_for() {
+  local len="$1" i bits out=()
+  for i in 0 1 2 3; do
+    bits=$(( len - i * 8 ))
+    (( bits > 8 )) && bits=8
+    (( bits < 0 )) && bits=0
+    out+=( $(( 256 - 2 ** (8 - bits) )) )
+  done
+  local IFS=.; echo "${out[*]}"
+}
+
+# cloud-init NoCloud seed: static addressing + SSH key + hello-world HTTP.
+build_seed() {
+  local iso="$1" vm="$2" vid="$3" name="$4" prefix="$5" ip="$6" real="$7" pubkey="$8"
+  local masklen="${9:-24}"
+  local netmask; netmask="$(netmask_for "$masklen")"
+  local tmp; tmp="$(mktemp -d)"
+
+  cat > "$tmp/meta-data" <<EOF
+instance-id: $vm
+local-hostname: $vm
+EOF
+
+  # Alpine's cloud-init does not reliably apply netplan-style network-config,
+  # and these networks have no DHCP server on purpose — so configure the
+  # interface the Alpine-native way instead (verified: hostname applied but no
+  # address, i.e. the seed was read and network-config was ignored).
+  #
+  # The default route deliberately points at the router under test (.1), not
+  # the host (.2), so a broken/absent router shows up as a failed test rather
+  # than being silently papered over by host routing. post-up ... || true keeps
+  # the interface up even while no router exists yet.
+  cat > "$tmp/network-config" <<EOF
+version: 1
+config:
+  - type: physical
+    name: eth0
+    subnets:
+      - type: static
+        address: $ip
+        netmask: $netmask
+        # Default route via the router under test. Without this the VMs can
+        # reach their own /24 and their gateway, but nothing beyond it — which
+        # looks exactly like "the router is broken" in the matrix.
+        gateway: ${prefix}.1
+EOF
+
+  cat > "$tmp/user-data" <<EOF
+#cloud-config
+hostname: $vm
+users:
+  - name: alpine
+    # NOTE: this Alpine image ships no sudo (and cloud-init's sudo: directive
+    # is therefore inert). For privileged work in these VMs, ssh as root —
+    # the key is installed there too.
+    shell: /bin/ash
+    # Without this cloud-init leaves the account locked ("!*" in /etc/shadow)
+    # and sshd refuses key auth for it — verified on the first build.
+    lock_passwd: false
+    plain_text_passwd: labsim
+    ssh_authorized_keys:
+      - $pubkey
+ssh_authorized_keys:
+  - $pubkey
+disable_root: false
+chpasswd:
+  list: |
+    root:labsim
+  expire: false
+write_files:
+  - path: /etc/network/interfaces
+    content: |
+      auto lo
+      iface lo inet loopback
+      auto eth0
+      iface eth0 inet static
+        address $ip
+        netmask $netmask
+        post-up ip route add default via ${prefix}.1 || true
+  - path: /var/www/index.html
+    content: |
+      <html><body>
+      <h1>labsim vlan $vid — $name</h1>
+      <p>host: $vm</p>
+      <p>address: $ip/$masklen</p>
+      <p>gateway under test: ${prefix}.1</p>
+      <p>mirrors production: $real</p>
+      </body></html>
+  - path: /etc/local.d/labsim-http.start
+    permissions: '0755'
+    content: |
+      #!/bin/sh
+      # This image's busybox has no httpd applet ("applet not found"), and the
+      # VMs are isolated so apk cannot fetch one. python3 is already present
+      # (cloud-init depends on it), so serve with http.server — no packages,
+      # no internet.
+      #
+      # Two traps already hit here, both silent:
+      #  - start-stop-daemon --exec /usr/bin/python3 matches cloud-init's OWN
+      #    python3 at boot, says "already running", starts nothing.
+      #  - busybox pgrep -f PATTERN matches its own argv, so a
+      #    "skip if running" guard always fires (verified: guard_exit=0 with
+      #    nothing listening).
+      # So: no guard, no start-stop-daemon. Binding twice is harmless — the
+      # second just fails to bind.
+      nohup /usr/bin/python3 -m http.server 80 --directory /var/www \\
+        >/var/log/labsim-http.log 2>&1 &
+runcmd:
+  # cloud-init's network-config (v1, above) already applies the address, so do
+  # NOT restart networking here — it fails, and one failing runcmd aborts every
+  # command after it, which is what silently left httpd unstarted. Each command
+  # is || true for the same reason.
+  - [ sh, -c, "rc-update add sshd default || true" ]
+  - [ sh, -c, "rc-update add local default || true" ]
+  - [ sh, -c, "/etc/local.d/labsim-http.start || true" ]
+EOF
+
+  # Validate before building the ISO. The heredoc above is intentionally
+  # unquoted (it interpolates $ip/$prefix), which means backticks or $( ) in
+  # ANY line — including comments — get executed by the host shell and their
+  # output silently corrupts the YAML. Cheap check, expensive bug.
+  python3 -c "import yaml,sys; yaml.safe_load(open(sys.argv[1]))" "$tmp/user-data" \
+    || die "generated user-data is not valid YAML (backticks or \$( ) in build_seed?): $tmp/user-data"
+
+  sudo genisoimage -quiet -output "$iso" -volid cidata -joliet -rock \
+    "$tmp/user-data" "$tmp/meta-data" "$tmp/network-config"
+  rm -rf "$tmp"
+}
+
+ssh_to() {
+  local ip="$1"; shift
+  timeout 12 ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
+    -o ConnectTimeout=5 -o BatchMode=yes -o LogLevel=ERROR \
+    "alpine@$ip" "$@" 2>/dev/null
+}
+
+wait_ready() {
+  local deadline=$((SECONDS + 240)) pending=1
+  while [ $SECONDS -lt $deadline ]; do
+    pending=0
+    for entry in "${SELECTED[@]}"; do
+      IFS=: read -r vid _n prefix _r <<<"$entry"
+      # Wait for BOTH: sshd is up well before cloud-init's runcmd starts the
+      # web server, so checking SSH alone reports "ready" then shows HTTP FAIL.
+      ssh_to "${prefix}.10" true >/dev/null 2>&1 \
+        && curl -sS -o /dev/null --max-time 4 "http://${prefix}.10/" 2>/dev/null \
+        || pending=$((pending + 1))
+    done
+    [ $pending -eq 0 ] && { log "all ${#SELECTED[@]} VMs reachable"; return 0; }
+    sleep 5
+  done
+  warn "$pending VM(s) still not answering SSH after 240s — see status below"
+  return 0
+}
+
+status_table() {
+  printf '  %-18s %-6s %-16s %-9s %-7s %s\n' VM VLAN ADDRESS STATE SSH HTTP
+  printf '  %-18s %-6s %-16s %-9s %-7s %s\n' ------------------ ------ ---------------- --------- ------- ----
+  for entry in "${SELECTED[@]}"; do
+    IFS=: read -r vid name prefix _r <<<"$entry"
+    local vm ip state ssh http
+    vm="$(vm_name "$vid" "$name")"; ip="${prefix}.10"
+    state="$(virsh_q domstate "$vm" 2>/dev/null | head -1 | tr -d '\n')"
+    [ -z "$state" ] && state="absent"
+    ssh_to "$ip" true >/dev/null 2>&1 && ssh=ok || ssh=FAIL
+    if curl -sS -o /dev/null --max-time 5 "http://$ip/" 2>/dev/null; then http=ok; else http=FAIL; fi
+    printf '  %-18s %-6s %-16s %-9s %-7s %s\n' "$vm" "$vid" "$ip" "$state" "$ssh" "$http"
+  done
+}

labsim/monitoring-up.sh

@@ -0,0 +1,82 @@
+#!/bin/bash
+# Prometheus + Grafana for the labsim connectivity matrix.
+#
+# Grafana runs with anonymous auth as Admin — NO LOGIN. That is deliberate for
+# a throwaway lab on localhost; do not copy this into anything reachable.
+#
+#   ./monitoring-up.sh          start exporter + prometheus + grafana
+#   ./monitoring-up.sh --down   stop and remove them
+#
+# Grafana:    http://localhost:3000   (dashboard "labsim — VLAN connectivity matrix")
+# Prometheus: http://localhost:9090
+# Exporter:   http://localhost:9101/metrics
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+source "$SCRIPT_DIR/lib.sh"
+
+GRAFANA_PORT="${GRAFANA_PORT:-3000}"
+PROM_PORT="${PROM_PORT:-9090}"
+EXPORTER_PORT="${EXPORTER_PORT:-9101}"
+NET="labsim-mon"
+
+if [ "${1:-}" = "--down" ]; then
+  pkill -f "labsim-exporter.py" 2>/dev/null || true
+  podman rm -f labsim-grafana labsim-prometheus >/dev/null 2>&1 || true
+  podman network rm -f "$NET" >/dev/null 2>&1 || true
+  log "monitoring stopped"
+  exit 0
+fi
+
+command -v podman >/dev/null 2>&1 || die "podman not installed"
+
+# --- exporter (on the host: it needs SSH access to the VMs) ----------------
+if pgrep -f "labsim-exporter.py" >/dev/null 2>&1; then
+  log "exporter already running on :$EXPORTER_PORT"
+else
+  log "starting exporter on :$EXPORTER_PORT"
+  nohup "$SCRIPT_DIR/labsim-exporter.py" --port "$EXPORTER_PORT" --interval 15 \
+    > /tmp/labsim-exporter.log 2>&1 &
+  sleep 3
+fi
+curl -sS --max-time 5 "http://127.0.0.1:${EXPORTER_PORT}/metrics" >/dev/null \
+  || die "exporter not answering on :$EXPORTER_PORT (see /tmp/labsim-exporter.log)"
+
+podman network exists "$NET" 2>/dev/null || podman network create "$NET" >/dev/null
+
+# --- prometheus -----------------------------------------------------------
+podman rm -f labsim-prometheus >/dev/null 2>&1 || true
+log "starting prometheus on :$PROM_PORT"
+podman run -d --name labsim-prometheus --network "$NET" \
+  -p "${PROM_PORT}:9090" \
+  -v "$SCRIPT_DIR/monitoring/prometheus.yml:/etc/prometheus/prometheus.yml:ro,Z" \
+  --add-host "host.containers.internal:host-gateway" \
+  docker.io/prom/prometheus:latest >/dev/null
+
+# --- grafana (anonymous, no login) ----------------------------------------
+podman rm -f labsim-grafana >/dev/null 2>&1 || true
+log "starting grafana on :$GRAFANA_PORT (anonymous auth — no password)"
+podman run -d --name labsim-grafana --network "$NET" \
+  -p "${GRAFANA_PORT}:3000" \
+  -e GF_AUTH_ANONYMOUS_ENABLED=true \
+  -e GF_AUTH_ANONYMOUS_ORG_ROLE=Admin \
+  -e GF_AUTH_DISABLE_LOGIN_FORM=true \
+  -e GF_AUTH_BASIC_ENABLED=false \
+  -e GF_SECURITY_ALLOW_EMBEDDING=true \
+  -e GF_USERS_DEFAULT_THEME=dark \
+  -v "$SCRIPT_DIR/monitoring/grafana/provisioning:/etc/grafana/provisioning:ro,Z" \
+  docker.io/grafana/grafana:latest >/dev/null
+
+log "waiting for grafana..."
+for _ in $(seq 1 40); do
+  if curl -sS --max-time 3 "http://127.0.0.1:${GRAFANA_PORT}/api/health" >/dev/null 2>&1; then
+    break
+  fi
+  sleep 3
+done
+
+echo
+log "Topology:   http://localhost:${EXPORTER_PORT}/   <- live mesh, red/green + RTT"
+log "Grafana:    http://localhost:${GRAFANA_PORT}/d/labsim-matrix  (no login, history)"
+log "Prometheus: http://localhost:${PROM_PORT}"
+log "Exporter:   http://localhost:${EXPORTER_PORT}/metrics"

labsim/monitoring/grafana/provisioning/dashboards/dashboards.yml

@@ -0,0 +1,9 @@
+apiVersion: 1
+providers:
+  - name: labsim
+    folder: ''
+    type: file
+    disableDeletion: false
+    updateIntervalSeconds: 10
+    options:
+      path: /etc/grafana/provisioning/dashboards

labsim/monitoring/grafana/provisioning/dashboards/labsim.json

@@ -0,0 +1,59 @@
+{
+  "uid": "labsim-matrix",
+  "title": "labsim — VLAN connectivity matrix",
+  "tags": ["labsim"],
+  "timezone": "browser",
+  "refresh": "10s",
+  "time": { "from": "now-30m", "to": "now" },
+  "panels": [
+    {
+      "type": "stat",
+      "title": "Reachable paths",
+      "gridPos": { "h": 4, "w": 6, "x": 0, "y": 0 },
+      "targets": [ { "expr": "sum(labsim_reachable)", "refId": "A" } ],
+      "fieldConfig": { "defaults": { "thresholds": { "mode": "absolute",
+        "steps": [ { "color": "red", "value": null }, { "color": "green", "value": 90 } ] } } }
+    },
+    {
+      "type": "stat",
+      "title": "Blocked paths",
+      "gridPos": { "h": 4, "w": 6, "x": 6, "y": 0 },
+      "targets": [ { "expr": "count(labsim_reachable == 0) or vector(0)", "refId": "A" } ],
+      "fieldConfig": { "defaults": { "thresholds": { "mode": "absolute",
+        "steps": [ { "color": "green", "value": null }, { "color": "orange", "value": 1 } ] } } }
+    },
+    {
+      "type": "stat",
+      "title": "Sweep duration (s)",
+      "gridPos": { "h": 4, "w": 6, "x": 12, "y": 0 },
+      "targets": [ { "expr": "labsim_sweep_seconds", "refId": "A" } ]
+    },
+    {
+      "type": "stat",
+      "title": "Sweeps",
+      "gridPos": { "h": 4, "w": 6, "x": 18, "y": 0 },
+      "targets": [ { "expr": "labsim_sweep_total", "refId": "A" } ]
+    },
+    {
+      "type": "heatmap",
+      "title": "ICMP matrix (src → dst) — green = reachable",
+      "gridPos": { "h": 10, "w": 24, "x": 0, "y": 4 },
+      "targets": [ { "expr": "labsim_reachable{proto=\"icmp\"}",
+                     "legendFormat": "{{src}} → {{dst}}", "refId": "A" } ]
+    },
+    {
+      "type": "state-timeline",
+      "title": "Every path over time — a firewall change shows up here immediately",
+      "gridPos": { "h": 12, "w": 24, "x": 0, "y": 14 },
+      "targets": [ { "expr": "labsim_reachable",
+                     "legendFormat": "{{proto}} {{src}} → {{dst}}", "refId": "A" } ],
+      "fieldConfig": { "defaults": {
+        "mappings": [ { "type": "value", "options": {
+          "0": { "text": "blocked", "color": "red", "index": 0 },
+          "1": { "text": "ok", "color": "green", "index": 1 } } } ] } },
+      "options": { "mergeValues": true, "showValue": "never" }
+    }
+  ],
+  "schemaVersion": 39,
+  "version": 1
+}

labsim/monitoring/grafana/provisioning/datasources/prometheus.yml

@@ -0,0 +1,7 @@
+apiVersion: 1
+datasources:
+  - name: Prometheus
+    type: prometheus
+    access: proxy
+    url: http://labsim-prometheus:9090
+    isDefault: true

labsim/monitoring/prometheus.yml

@@ -0,0 +1,9 @@
+# Scrapes the labsim connectivity exporter running on the host.
+global:
+  scrape_interval: 15s
+  evaluation_interval: 15s
+
+scrape_configs:
+  - job_name: labsim
+    static_configs:
+      - targets: ['host.containers.internal:9101']

labsim/ovs.sh

@@ -0,0 +1,254 @@
+#!/bin/bash
+# Open vSwitch fabric for labsim — the "switch" the whole sim hangs off.
+#
+# Why OVS and not a Linux bridge: a Linux bridge cannot do LACP at all, and its
+# VLAN support is awkward to drive from libvirt. OVS gives real 802.1Q access
+# and trunk ports plus real LACP bonds, so a router VM can run the SAME bond0 +
+# vif config as the production VP2440s instead of an approximation.
+#
+# Layout:
+#   ovs-labsim            the switch
+#     ├─ vm ports         access ports, tag=<vlan>       (micro VM per VLAN)
+#     ├─ hostv<vlan>      internal ports, tag=<vlan>     (host leg, for SSH)
+#     └─ lag-vyos         LACP bond, trunk of all VLANs  (router under test)
+# shellcheck disable=SC2034
+
+OVS_BR="${OVS_BR:-ovs-labsim}"
+OVS_NET="${OVS_NET:-labsim-ovs}"          # libvirt network wrapping the bridge
+LAG_NAME="${LAG_NAME:-lag-vyos}"
+
+# Native (untagged) VLAN on the trunks to the routers. Empty means NONE: every
+# VLAN, Management included, is tagged.
+#
+# This is not a style choice. A native VLAN is what puts a subnet on the bond
+# PARENT (`bond0`) while every other VLAN lives on a sub-interface of it. With
+# `dhcp-socket-type: raw`, kea then receives each tagged frame TWICE -- once on
+# the sub-interface and once on the parent -- and answers from the parent's pool
+# as well, so a client on VLAN 3 is offered a Management address and picks
+# whichever reply arrives first (ISC Kea #1117).
+#
+# Set LABSIM_NATIVE_VLAN=1 to restore the old shape and reproduce the bug:
+#   LABSIM_NATIVE_VLAN=1 ./router-up.sh && ./labsim-vlan-leak-test.sh   # FAIL
+#                        ./router-up.sh && ./labsim-vlan-leak-test.sh   # PASS
+NATIVE_VLAN="${LABSIM_NATIVE_VLAN:-}"
+
+ovs() { sudo ovs-vsctl "$@"; }
+
+ovs_require() {
+  command -v ovs-vsctl >/dev/null 2>&1 || die "openvswitch not installed (dnf install openvswitch)"
+  systemctl is-active --quiet openvswitch || sudo systemctl start openvswitch \
+    || die "could not start openvswitch"
+}
+
+# A comma-separated VLAN list, numerically sorted, for comparing two lists that
+# came from different places and need not agree on order.
+vlan_sorted() { echo "$1" | tr ',' '\n' | grep -v '^$' | sort -n | paste -sd, -; }
+
+# All VLAN ids from the config, comma separated — used for trunk ports.
+vlan_id_list() {
+  local ids=()
+  for entry in "${SELECTED[@]}"; do ids+=("${entry%%:*}"); done
+  (IFS=,; echo "${ids[*]}")
+}
+
+ovs_up() {
+  ovs_require
+  ovs --may-exist add-br "$OVS_BR"
+
+  # Host leg per VLAN: an OVS internal port carrying that VLAN's tag, given the
+  # .2 address. This is how you SSH to the VMs. It is deliberately NOT their
+  # default route (.1 is), so inter-VLAN tests exercise the router, not the
+  # host's routing table.
+  for entry in "${SELECTED[@]}"; do
+    parse_vlan_entry "$entry"
+    local port="hostv${V_VID}"
+    ovs --may-exist add-port "$OVS_BR" "$port" tag="$V_VID" \
+      -- set interface "$port" type=internal
+    sudo ip link set "$port" up 2>/dev/null || true
+    # Drop any address from a previous mask/octet so a changed vlans.conf does
+    # not leave a stale second address on the port.
+    sudo ip -4 addr flush dev "$port" 2>/dev/null || true
+    # host_octet 0 means "no host leg": the WAN transport VLANs belong to the
+    # fake ISPs, and giving the host an address there would misrepresent the
+    # segment -- the whole point is that VyOS reaches an ISP, not the host.
+    if [ "$V_HOST" != "0" ]; then
+      sudo ip addr replace "${V_PREFIX}.${V_HOST}/${V_MASK}" dev "$port"
+    fi
+  done
+
+  ovs_define_libvirt_net
+}
+
+# A libvirt network that hands out OVS ports: one portgroup per VLAN (access)
+# plus a trunk portgroup for the router.
+ovs_define_libvirt_net() {
+  local pg="" ids
+  for entry in "${SELECTED[@]}"; do
+    IFS=: read -r vid name _p _r <<<"$entry"
+    pg+="  <portgroup name='vlan${vid}'>
+    <vlan><tag id='${vid}'/></vlan>
+  </portgroup>
+"
+  done
+
+  # Trunk: every VLAN tagged, and by default NO native VLAN (see NATIVE_VLAN at
+  # the top of this file for why -- it is the kea #1117 fix, not tidiness).
+  # libvirt expresses a native VLAN declaratively via nativeMode='untagged'
+  # (see libvirt formatnetwork.html), so it needs no fixing up by hand.
+  #
+  # The worry that a trunk with no native VLAN has nowhere to put LACPDUs is
+  # unfounded, and was tested rather than reasoned about: with vlan_mode=trunk
+  # and no tag, `ovs-appctl bond/show` still reports lacp_status: negotiated
+  # with both members enabled. LACPDUs are slow-protocol frames handled per
+  # member, below the VLAN layer.
+  local trunk="  <portgroup name='trunk'>
+    <vlan trunk='yes'>
+"
+  for entry in "${SELECTED[@]}"; do
+    IFS=: read -r vid _n _p _r <<<"$entry"
+    if [ -n "$NATIVE_VLAN" ] && [ "$vid" = "$NATIVE_VLAN" ]; then
+      trunk+="      <tag id='${vid}' nativeMode='untagged'/>
+"
+    else
+      trunk+="      <tag id='${vid}'/>
+"
+    fi
+  done
+  trunk+="    </vlan>
+  </portgroup>
+"
+
+  local xml="<network>
+  <name>${OVS_NET}</name>
+  <forward mode='bridge'/>
+  <bridge name='${OVS_BR}'/>
+  <virtualport type='openvswitch'/>
+${pg}${trunk}</network>"
+
+  if virsh_q net-info "$OVS_NET" >/dev/null 2>&1; then
+    virsh_q net-destroy "$OVS_NET" >/dev/null 2>&1 || true
+    virsh_q net-undefine "$OVS_NET" >/dev/null 2>&1 || true
+  fi
+  echo "$xml" | virsh_q net-define /dev/stdin >/dev/null
+  virsh_q net-start "$OVS_NET" >/dev/null
+  log "libvirt network $OVS_NET bound to $OVS_BR (access portgroups + trunk)"
+}
+
+# Replace the router VM's two individual OVS ports with a single LACP bond.
+# libvirt attaches each NIC separately; only ovs-vsctl can bond them, and the
+# taps only exist once the VM is running — so this runs post-start.
+ovs_bond_router() {
+  local vm="$1"
+  local taps
+  # Two filters, both load-bearing:
+  #
+  #   $1 ~ /^vnet/ -- domiflist indents its rows, so anchor on the FIELD, not
+  #   the line. /^vnet/ silently matches nothing and the bond never gets built.
+  #
+  #   $3 == OVS_NET -- count only the taps on the sim fabric. The primary also
+  #   carries a libvirt-NAT scaffold NIC (see --drop-scaffold in the README), so
+  #   an unfiltered count is 3, and this function's "expected 2" guard then
+  #   skipped the primary's bond entirely while reporting only a warning.
+  taps="$(virsh_q domiflist "$vm" 2>/dev/null \
+          | awk -v net="$OVS_NET" '$1 ~ /^vnet/ && $3 == net {print $1}')"
+  local count; count="$(echo "$taps" | grep -c .)"
+  [ "$count" -eq 2 ] \
+    || { warn "router $vm has $count tap(s) on $OVS_NET, expected 2 — skipping bond"; return 1; }
+
+  local t1 t2; t1="$(echo "$taps" | sed -n 1p)"; t2="$(echo "$taps" | sed -n 2p)"
+  local want; want="$(vlan_id_list)"
+
+  # Already bonded? Re-runs must still reconcile BOTH the VLAN list and the
+  # membership, and each has drawn blood:
+  #
+  #   VLANs -- adding a VLAN to vlans.conf and finding the bond unchanged is how
+  #   a VLAN silently fails to reach a router: interface present, tag missing,
+  #   frames dropped by the switch.
+  #
+  #   MEMBERS -- restarting the VM recreates its taps with NEW names, leaving the
+  #   bond holding two interfaces that no longer exist. `list-ports` still shows
+  #   the bond, so this early return declared success while the router's real
+  #   taps sat in the bridge as two INDEPENDENT ports, each carrying libvirt's
+  #   own portgroup VLAN config. That is how labsim-vyos2 ran for weeks with no
+  #   LACP at all and a native VLAN nobody had asked for -- and it is invisible
+  #   until you change the trunk and only one router follows.
+  if ovs list-ports "$OVS_BR" 2>/dev/null | grep -qx "$LAG_NAME"; then
+    local members; members="$(ovs-appctl-members)"
+    if [ "$members" != "$(printf '%s\n%s' "$t1" "$t2" | sort | paste -sd, -)" ]; then
+      warn "bond $LAG_NAME holds stale members [$members], VM has [$t1,$t2] — rebuilding"
+      ovs --if-exists del-port "$OVS_BR" "$LAG_NAME"
+    else
+      # Compare as SETS. vlan_id_list yields config order (1,2,3,9,10,200,51,53)
+      # while OVS returns its own sorted order, so a raw string compare reports
+      # drift on every run and rewrites a trunk that was already correct.
+      local have; have="$(ovs get port "$LAG_NAME" trunks 2>/dev/null | tr -d '[] ')"
+      if [ "$(vlan_sorted "$want")" != "$(vlan_sorted "$have")" ]; then
+        log "bond $LAG_NAME trunk drift: [$have] -> [$want]; updating"
+        ovs set port "$LAG_NAME" trunks="$want"
+      else
+        log "LACP bond $LAG_NAME already present, trunk correct"
+      fi
+      ovs_set_native "$LAG_NAME"
+      return 0
+    fi
+  fi
+
+  log "bonding $t1 + $t2 into $LAG_NAME (LACP active, balance-tcp)"
+  ovs del-port "$OVS_BR" "$t1" 2>/dev/null || true
+  ovs del-port "$OVS_BR" "$t2" 2>/dev/null || true
+
+  # bond_mode=balance-tcp is REQUIRED: OVS defaults a bond to active-backup,
+  # which does not speak LACP at all (confirmed on ovs-discuss). It is also the
+  # equivalent of VyOS's 802.3ad + layer2+3 hashing.
+  #
+  # lacp-fallback-ab breaks a genuine deadlock: OVS keeps members disabled
+  # until LACP negotiates, while the partner needs carrier before it will send
+  # LACPDUs. Falling back to active-backup brings the links up so negotiation
+  # can start.
+  #
+  # The VLAN mode is set inline: libvirt's portgroup config does NOT apply here,
+  # because the bond is a port libvirt never created.
+  ovs add-bond "$OVS_BR" "$LAG_NAME" "$t1" "$t2" \
+      lacp=active bond_mode=balance-tcp trunks="$want" \
+    -- set port "$LAG_NAME" other_config:lacp-time=fast \
+    -- set port "$LAG_NAME" other_config:lacp-fallback-ab=true
+  ovs_set_native "$LAG_NAME"
+}
+
+# The bond's current members, sorted and comma-joined, or empty if the bond does
+# not resolve at all (which is itself the stale case worth rebuilding for).
+ovs-appctl-members() {
+  sudo ovs-appctl bond/show "$LAG_NAME" 2>/dev/null \
+    | awk '/^member /{gsub(/:/,"",$2); print $2}' | sort | paste -sd, -
+}
+
+# Apply NATIVE_VLAN to a trunk port.
+#
+# `tag` MUST be removed, not merely left alone, when there is no native VLAN.
+# Setting vlan_mode=trunk while a stale `tag` remains looks correct in
+# `ovs-vsctl list port` -- it prints vlan_mode: trunk right next to tag: 1 --
+# but the port keeps egressing that VLAN untagged. Half an hour went into
+# "the router is ignoring the trunk change" before the tag was the answer.
+ovs_set_native() {
+  local port="$1"
+  if [ -n "$NATIVE_VLAN" ]; then
+    ovs set port "$port" vlan_mode=native-untagged tag="$NATIVE_VLAN"
+  else
+    ovs set port "$port" vlan_mode=trunk -- clear port "$port" tag
+  fi
+}
+
+ovs_bond_status() {
+  echo "--- ovs bond ---"
+  sudo ovs-appctl bond/show "$LAG_NAME" 2>/dev/null | grep -E "bond_mode|lacp_status|^member|may_enable" || echo "(no bond)"
+  echo "--- lacp ---"
+  sudo ovs-appctl lacp/show "$LAG_NAME" 2>/dev/null | grep -E "status|aggregation key|^member|attached" || true
+}
+
+ovs_down() {
+  virsh_q net-destroy "$OVS_NET" >/dev/null 2>&1 || true
+  virsh_q net-undefine "$OVS_NET" >/dev/null 2>&1 || true
+  if command -v ovs-vsctl >/dev/null 2>&1; then
+    ovs --if-exists del-br "$OVS_BR" 2>/dev/null || true
+  fi
+}

labsim/router-install.py

@@ -0,0 +1,162 @@
+#!/usr/bin/env python3
+"""Drive the labsim VyOS router over its serial console.
+
+Three phases:
+  --phase live       wait for the live system and log in
+  --phase install    run `install image` unattended
+  --phase configure  apply bond0 (LACP) + per-VLAN gateway addresses
+
+The installer prompt list is the same one the bastion's install driver answers
+(src/bastion/src/templates/vyos-install.py.ts). Two of them are easy to miss and
+both hang forever rather than failing: the reinstall-only "copy data to the new
+image?", and "choose two disks for RAID-1 mirroring?" — every RAID prompt
+defaults to YES.
+"""
+from __future__ import annotations
+
+import argparse
+import sys
+import time
+
+import pexpect
+
+PASSWORD = "vyos"
+PROMPT = r"[\$#] $"
+
+
+def console(vm: str, timeout: int = 60) -> pexpect.spawn:
+    c = pexpect.spawn(f"sudo virsh console {vm} --force", encoding="utf-8", timeout=timeout)
+    c.expect("Connected to domain", timeout=30)
+    return c
+
+
+def login(c: pexpect.spawn, timeout: int = 300) -> None:
+    """Get to a shell prompt, whether we land at a login or an open session."""
+    deadline = time.time() + timeout
+    while time.time() < deadline:
+        c.sendline("")
+        i = c.expect(["login:", PROMPT, pexpect.TIMEOUT], timeout=20)
+        if i == 0:
+            c.sendline("vyos")
+            c.expect("assword:", timeout=20)
+            c.sendline(PASSWORD)
+            j = c.expect([PROMPT, "incorrect", pexpect.TIMEOUT], timeout=30)
+            if j == 0:
+                return
+        elif i == 1:
+            return
+    raise SystemExit("timed out waiting for a VyOS shell")
+
+
+def run(c: pexpect.spawn, cmd: str, timeout: int = 60) -> str:
+    c.sendline(cmd)
+    c.expect(PROMPT, timeout=timeout)
+    return c.before or ""
+
+
+def phase_install(c: pexpect.spawn) -> None:
+    """Answer `install image` end to end."""
+    rules: list[tuple[str, str]] = [
+        (r"Would you like to continue\?", "yes"),
+        (r"What would you like to name this image\?", ""),
+        (r"Please confirm password for the .vyos. user:", PASSWORD),
+        (r"Please enter a password for the .vyos. user:", PASSWORD),
+        (r"What console should be used by default", "K"),
+        # every RAID variant defaults to YES — decline them all
+        (r"Would you like to [^?]*RAID-1 mirroring", "no"),
+        (r"Installation will delete all data on (?:the drive|both drives)\. Continue\?", "yes"),
+        (r"Which one should be used for installation\?", "/dev/vda"),
+        (r"Would you like to use all the free space on the drive\?", "yes"),
+        (r"Which file would you like as boot config\?", "1"),
+        # reinstall-only; unanswered it blocks on stdin until the world ends
+        (r"Would you like to copy data to the new image\?", "yes"),
+        (r"From which image would you like to save config information\?", "1"),
+    ]
+    patterns = [r for r, _ in rules] + [r"The image installed successfully",
+                                        r"Unable to install VyOS", pexpect.TIMEOUT]
+
+    c.sendline("install image")
+    for _ in range(60):
+        i = c.expect(patterns, timeout=180)
+        if i < len(rules):
+            c.sendline(rules[i][1])
+            continue
+        if i == len(rules):
+            print("  installer: success")
+            return
+        if i == len(rules) + 1:
+            raise SystemExit("installer reported failure")
+        raise SystemExit("installer went quiet (unanswered prompt?)")
+    raise SystemExit("installer exceeded expected prompt count")
+
+
+def phase_configure(c: pexpect.spawn, vlans: list[tuple[str, str, str]]) -> None:
+    """bond0 over eth0+eth1 with LACP, then a gateway address per VLAN."""
+    # Production shape: VLAN 1 (management) is the NATIVE/untagged VLAN on the
+    # bond, everything else is a tagged vif. This matters beyond fidelity —
+    # LACPDUs are untagged, so a trunk with no native VLAN has nowhere to put
+    # them and the bond never negotiates.
+    native = [v for v in vlans if v[0] == "1"]
+    tagged = [v for v in vlans if v[0] != "1"]
+
+    cmds = [
+        "configure",
+        "set interfaces bonding bond0 mode '802.3ad'",
+        "set interfaces bonding bond0 hash-policy 'layer2+3'",
+        "set interfaces bonding bond0 lacp-rate 'fast'",
+        "set interfaces bonding bond0 member interface 'eth0'",
+        "set interfaces bonding bond0 member interface 'eth1'",
+        "set service ssh port '22'",
+        "set system login user vyos authentication plaintext-password 'vyos'",
+    ]
+    for vid, name, prefix in native:
+        cmds.append(f"set interfaces bonding bond0 address '{prefix}.1/24'")
+        cmds.append(f"set interfaces bonding bond0 description '{name} (native)'")
+    for vid, name, prefix in tagged:
+        cmds.append(f"set interfaces bonding bond0 vif {vid} address '{prefix}.1/24'")
+        cmds.append(f"set interfaces bonding bond0 vif {vid} description '{name}'")
+    cmds += ["commit", "save", "exit"]
+
+    for cmd in cmds:
+        out = run(c, cmd, timeout=180)
+        low = out.lower()
+        if "invalid" in low or "syntax error" in low or "commit failed" in low:
+            print(f"  !! {cmd}\n{out.strip()[-300:]}")
+            raise SystemExit(f"config command rejected: {cmd}")
+    print("  config committed and saved")
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--vm", required=True)
+    ap.add_argument("--phase", required=True, choices=["live", "install", "configure"])
+    ap.add_argument("--vlans", default="", help="space separated vid:name:prefix:real entries")
+    args = ap.parse_args()
+
+    c = console(args.vm)
+    try:
+        login(c)
+        if args.phase == "live":
+            print("  live system reachable")
+        elif args.phase == "install":
+            phase_install(c)
+        else:
+            vlans = []
+            for entry in args.vlans.split():
+                parts = entry.split(":")
+                if len(parts) >= 3:
+                    vlans.append((parts[0], parts[1], parts[2]))
+            if not vlans:
+                raise SystemExit("no VLANs passed to configure")
+            phase_configure(c, vlans)
+        return 0
+    finally:
+        try:
+            c.sendline("")
+            c.close(force=True)
+        except Exception:
+            pass
+
+
+if __name__ == "__main__":
+    sys.exit(main())

labsim/router-up.sh

@@ -0,0 +1,124 @@
+#!/bin/bash
+# Add the VyOS router under test to labsim.
+#
+# Mirrors the production VP2440 pair: TWO NICs bonded with LACP carrying a
+# trunk of every VLAN, then bond0.<vlan> sub-interfaces holding the .1 gateway
+# address on each. That is the same config shape the real firewalls run, so a
+# rule tested here means something.
+#
+# NIC model is e1000e, NOT virtio, and that is load-bearing: with virtio the
+# guest's bonding driver reports its slaves "MII Status: down" despite
+# carrier=1 and never emits a single LACPDU, so the bond sits in
+# AD_STATE_DEFAULTED forever. Known issue — see the netdev thread "bonding
+# (IEEE 802.3ad) not working with qemu/virtio"; e1000e fixes it with no other
+# change. 802.3ad also requires the MII link monitor, which virtio cannot back.
+#
+#   host                       OVS "switch"                    VyOS VM
+#   hostv<vlan> (.2) ──────── ovs-labsim ──── lag-vyos ═════ eth0 + eth1
+#                              (tagged)       (LACP, trunk)   └─ bond0.<vlan> = .1
+#
+# Usage: ./router-up.sh            build + install + configure
+#        ./router-up.sh --status   show bond/LACP + interface state
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+source "$SCRIPT_DIR/lib.sh"
+source "$SCRIPT_DIR/ovs.sh"
+
+ROUTER_VM="${ROUTER_VM:-labsim-vyos}"
+ROUTER_MEM="${ROUTER_MEM:-2048}"
+ROUTER_CPUS="${ROUTER_CPUS:-2}"
+ROUTER_DISK_GB="${ROUTER_DISK_GB:-8}"
+VYOS_ISO="${VYOS_ISO:-$IMG_DIR/vyos.iso}"
+VYOS_CACHE="/var/lib/libvirt/images/lab-pxe-cache"
+
+selected_vlans
+
+if [ "${1:-}" = "--status" ]; then
+  ovs_bond_status
+  echo "--- vyos gateway addresses (probed from each host leg) ---"
+  for entry in "${SELECTED[@]}"; do
+    IFS=: read -r vid _n prefix _r <<<"$entry"
+    printf '  vlan %-5s %-16s ' "$vid" "${prefix}.1"
+    ping -c1 -W2 "${prefix}.1" >/dev/null 2>&1 && echo up || echo down
+  done
+  exit 0
+fi
+
+ovs_require
+
+# --- ISO ------------------------------------------------------------------
+if [ ! -f "$VYOS_ISO" ]; then
+  # Reuse the bastion's cached nightly if it is already on this box.
+  if [ -f "$VYOS_CACHE/vyos.iso" ]; then
+    log "reusing cached VyOS ISO"
+    sudo cp "$VYOS_CACHE/vyos.iso" "$VYOS_ISO"
+  else
+    log "resolving latest VyOS nightly ISO..."
+    url="$(curl -sSL https://api.github.com/repos/vyos/vyos-nightly-build/releases/latest \
+      | python3 -c "import json,sys;print(next(a['browser_download_url'] for a in json.load(sys.stdin)['assets'] if a['name'].endswith('generic-amd64.iso')))")"
+    log "downloading $url"
+    sudo curl -sSL --max-time 1800 -o "$VYOS_ISO" "$url"
+  fi
+fi
+[ -f "$VYOS_ISO" ] || die "no VyOS ISO at $VYOS_ISO"
+
+# --- VM -------------------------------------------------------------------
+if virsh_q dominfo "$ROUTER_VM" >/dev/null 2>&1; then
+  log "router VM $ROUTER_VM exists"
+  virsh_q start "$ROUTER_VM" >/dev/null 2>&1 || true
+else
+  log "creating router VM $ROUTER_VM (2 NICs on the trunk, for LACP)"
+  sudo qemu-img create -q -f qcow2 "$IMG_DIR/${ROUTER_VM}.qcow2" "${ROUTER_DISK_GB}G" >/dev/null
+
+  # Two trunk NICs — OVS bonds them after boot (libvirt cannot create bonds).
+  sudo virt-install \
+    --connect "$LIBVIRT_URI" \
+    --name "$ROUTER_VM" \
+    --memory "$ROUTER_MEM" --vcpus "$ROUTER_CPUS" \
+    --disk "path=$IMG_DIR/${ROUTER_VM}.qcow2,format=qcow2,bus=virtio" \
+    --disk "path=$VYOS_ISO,device=cdrom,readonly=on" \
+    --network "network=$OVS_NET,portgroup=trunk,model=e1000e,trustGuestRxFilters=yes" \
+    --network "network=$OVS_NET,portgroup=trunk,model=e1000e,trustGuestRxFilters=yes" \
+    --boot cdrom,hd \
+    --os-variant debian12 \
+    --graphics none --noautoconsole --import >/dev/null
+fi
+
+log "waiting for the live system to boot (VyOS live login)..."
+python3 "$SCRIPT_DIR/router-install.py" --vm "$ROUTER_VM" --phase live || die "live boot failed"
+
+log "installing VyOS to disk (unattended over the console)..."
+python3 "$SCRIPT_DIR/router-install.py" --vm "$ROUTER_VM" --phase install || die "install failed"
+
+# Boot the INSTALLED system from here on. Without this the VM was created with
+# --boot cdrom,hd and every restart re-runs the ISO, so the live system comes
+# back with no config and every `commit; save` silently evaporates.
+log "switching boot to disk and ejecting the install media..."
+virsh_q destroy "$ROUTER_VM" >/dev/null 2>&1 || true
+sleep 2
+sudo virt-xml "$ROUTER_VM" --edit --boot hd >/dev/null
+sudo virt-xml "$ROUTER_VM" --remove-device --disk device=cdrom >/dev/null 2>&1 || true
+virsh_q start "$ROUTER_VM" >/dev/null
+sleep 10
+
+# Bond the taps only now: they are recreated by the restart above, so bonding
+# before this would bond stale interfaces.
+ovs_bond_router "$ROUTER_VM"
+
+log "applying router config (bond0 LACP + VLAN gateways)..."
+python3 "$SCRIPT_DIR/router-install.py" --vm "$ROUTER_VM" --phase configure \
+  --vlans "$(printf '%s\n' "${SELECTED[@]}" | tr '\n' ' ')" || die "configure failed"
+
+log "waiting for LACP to negotiate..."
+for _ in $(seq 1 30); do
+  if sudo ovs-appctl lacp/show "$LAG_NAME" 2>/dev/null | grep -q "current attached"; then
+    log "LACP negotiated"; break
+  fi
+  sleep 5
+done
+
+echo
+ovs_bond_status
+echo
+log "router is up. Check reachability with: $SCRIPT_DIR/labsim-matrix.py --watch 2"

labsim/sim-ha-config.py

@@ -0,0 +1,115 @@
+#!/usr/bin/env python3
+"""Generate the HA config for the labsim VyOS pair.
+
+Exists to answer one question that cannot be answered on a single router, and
+that would otherwise only be discovered at cutover: with kea HA active-passive,
+does exactly ONE box answer a DHCP request?
+
+Mirrors the production shape so the answer transfers:
+
+    router1  172.31.<v>.252   priority 200   DHCP HA primary
+    router2  172.31.<v>.253   priority 100   DHCP HA secondary
+    VIP      172.31.<v>.1     (what clients use as their gateway)
+
+Note the sim's LoT VLAN is a /23 like production, so the VIP prefix differs
+there -- getting that wrong produces a config that commits and then behaves
+subtly wrongly, which is worse than a failure.
+
+  ./sim-ha-config.py --role primary   > r1.conf
+  ./sim-ha-config.py --role secondary > r2.conf
+"""
+from __future__ import annotations
+
+import argparse
+import importlib.util
+import json
+import os
+import sys
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+MIG = os.path.join(HERE, "..", "migration")
+
+# Reuse the DHCP/DNS generator rather than hand-writing subnets: the whole
+# point is that what is proven here and what production gets share a code path.
+_spec = importlib.util.spec_from_file_location(
+    "unifi_to_vyos", os.path.join(MIG, "unifi-to-vyos.py"))
+unifi_to_vyos = importlib.util.module_from_spec(_spec)
+_spec.loader.exec_module(unifi_to_vyos)
+
+# vlan -> (prefix, cidr). LoT is a /23 in the sim, matching production.
+VLANS = {
+    1:   ("172.31.1",   24),
+    2:   ("172.31.2",   24),
+    3:   ("172.31.3",   24),
+    9:   ("172.31.9",   24),
+    10:  ("172.31.10",  23),
+    200: ("172.31.200", 24),
+}
+DHCP_HA_NAME = "labsim-dhcp-pair"     # must not equal either host-name
+
+
+def group(vlan: int) -> str:
+    return "native" if vlan == 1 else f"vlan{vlan}"
+
+
+def build(role: str) -> list[str]:
+    primary = role == "primary"
+    self_o, peer_o = (252, 253) if primary else (253, 252)
+    prio = 200 if primary else 100
+    out = [f"# labsim VyOS HA -- {role}", ""]
+
+    for vlan, (pfx, cidr) in VLANS.items():
+        g = group(vlan)
+        # EVERY VLAN is a sub-interface, Management (VLAN 1) included. Putting
+        # Management on the bare `bond0` is what gives the parent a subnet, and
+        # kea then answers tagged frames from it as well as from the correct
+        # sub-interface -- clients on other VLANs get offered a Management
+        # address (ISC Kea #1117). See NATIVE_VLAN in ovs.sh; proven by
+        # labsim-vlan-leak-test.sh.
+        iface = f"bond0 vif {vlan}"
+        out += [
+            f"# VLAN {vlan}",
+            # The node's own address replaces the .1 it used to hold directly;
+            # .1 becomes the floating VIP, exactly as production will be.
+            f"delete interfaces bonding {iface} address",
+            f"set interfaces bonding {iface} address '{pfx}.{self_o}/{cidr}'",
+            f"set high-availability vrrp group {g} interface bond0.{vlan}",
+            f"set high-availability vrrp group {g} vrid {vlan}",
+            f"set high-availability vrrp group {g} address {pfx}.1/{cidr}",
+            f"set high-availability vrrp group {g} priority {prio}",
+            f"set high-availability vrrp group {g} hello-source-address {pfx}.{self_o}",
+            f"set high-availability vrrp group {g} peer-address {pfx}.{peer_o}",
+            f"set high-availability vrrp group {g} no-preempt",
+            f"set high-availability vrrp sync-group MAIN member {g}",
+            "",
+        ]
+
+    out += [
+        "# --- DHCP high-availability ---",
+        "# The thing under test: active-passive should mean exactly one OFFER.",
+        "set service dhcp-server high-availability mode active-passive",
+        f"set service dhcp-server high-availability status {role}",
+        f"set service dhcp-server high-availability name {DHCP_HA_NAME}",
+        f"set service dhcp-server high-availability source-address 172.31.10.{self_o}",
+        f"set service dhcp-server high-availability remote 172.31.10.{peer_o}",
+        "",
+    ]
+
+    inv = json.load(open(os.path.join(MIG, "export", "inventory.json")))
+    dhcp, stats = unifi_to_vyos.build(inv, "sim")
+    out += [l for l in dhcp if l.strip() and not l.startswith("#")]
+    print(f"{role}: {stats['subnets']} subnets, {stats['mappings']} mappings",
+          file=sys.stderr)
+    return out
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--role", choices=("primary", "secondary"), required=True)
+    args = ap.parse_args()
+    sys.stdout.write("\n".join(build(args.role)) + "\n")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

labsim/sim-net-apply.sh

@@ -0,0 +1,59 @@
+#!/usr/bin/env bash
+# Apply -- or drift-check -- the labsim routing config on all four VMs.
+#
+#   ./sim-net-apply.sh check    what the VMs run vs what sim-net-config.py says
+#   ./sim-net-apply.sh apply    push the generated config over the serial console
+#
+# `check` is the one you want most of the time. The whole failure mode this
+# guards against is somebody (including me) fixing something on a VM over SSH
+# and never writing it down, so the next rebuild silently loses it.
+#
+# Applied over the serial console rather than SSH because a freshly installed
+# sim router holds the same addresses as its peer -- there is a window where it
+# is not safely reachable over the network at all. See console-apply.py.
+set -uo pipefail
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ACTION="${1:-check}"
+WORK="$(mktemp -d)"; trap 'rm -rf "$WORK"' EXIT
+
+# role : vm : address : regex selecting the subtrees this generator owns
+TARGETS=(
+  "primary:labsim-vyos:172.31.1.252:^set (protocols (bgp|failover|static)|policy (prefix-list|route-map)|nat source rule 1[12]0|interfaces (pppoe|bonding bond0 vif 5[13])|firewall (group interface-group LAN|ipv4|ipv6))"
+  "secondary:labsim-vyos2:172.31.1.253:^set (protocols bgp|policy (prefix-list|route-map)|firewall (group interface-group LAN|ipv4|ipv6))"
+  "isp-dhcp:labsim-isp-dhcp:192.168.122.136:^set (interfaces ethernet|nat source|service dhcp-server|firewall ipv4 forward|system host-name)"
+  "isp-pppoe:labsim-isp-pppoe:192.168.122.63:^set (interfaces ethernet|nat source|service pppoe-server|firewall ipv4 forward|system host-name)"
+)
+# Sim-only credential; these VMs hold nothing real and are not reachable from
+# outside the hypervisor.
+SSH_OPTS=(-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null
+          -o LogLevel=ERROR -o PreferredAuthentications=password -o ConnectTimeout=5)
+live() { timeout 30 sshpass -p vyos ssh "${SSH_OPTS[@]}" "vyos@$1" \
+           "/opt/vyatta/bin/vyatta-op-cmd-wrapper show configuration commands" 2>/dev/null; }
+norm() { sed "s/'//g" | grep -v 'hw-id\|offload' | sort -u; }
+
+rc=0
+for t in "${TARGETS[@]}"; do
+  IFS=: read -r role vm addr rx <<<"$t"
+  "$HERE/sim-net-config.py" --role "$role" >"$WORK/$role.conf" 2>/dev/null || {
+    printf '  %-11s GENERATE FAILED\n' "$role"; rc=1; continue; }
+
+  if [ "$ACTION" = apply ]; then
+    printf '  %-11s applying to %s over console...\n' "$role" "$vm"
+    "$HERE/console-apply.py" --vm "$vm" --config "$WORK/$role.conf" || rc=1
+    continue
+  fi
+
+  if ! live "$addr" >"$WORK/$role.live" || [ ! -s "$WORK/$role.live" ]; then
+    printf '  %-11s UNREACHABLE (%s)\n' "$role" "$addr"; rc=1; continue
+  fi
+  grep -E '^set ' "$WORK/$role.conf" | norm >"$WORK/$role.g"
+  grep -E "$rx" "$WORK/$role.live" | norm >"$WORK/$role.l"
+  if d="$(diff "$WORK/$role.g" "$WORK/$role.l")" && [ -z "$d" ]; then
+    printf '  %-11s in sync (%s commands)\n' "$role" "$(wc -l <"$WORK/$role.g")"
+  else
+    printf '  %-11s DRIFT — "<" only in code, ">" only on the VM:\n' "$role"
+    printf '%s\n' "$d" | sed 's/^/      /'
+    rc=1
+  fi
+done
+exit $rc

labsim/sim-net-config.py

@@ -0,0 +1,417 @@
+#!/usr/bin/env python3
+"""Generate the labsim routing + WAN config: BGP, dual WAN, and the two ISP VMs.
+
+`sim-ha-config.py` covers the LAN side of the sim routers (addresses, VRRP,
+conntrack-sync, DHCP). This covers everything that makes the sim a rehearsal for
+production routing rather than just a LAN:
+
+  * eBGP between the sim routers and the k3s nodes, carrying the service range
+  * dual WAN -- DHCP on VLAN 53, PPPoE on VLAN 51 -- with health-checked failover
+  * the two ISP VMs that terminate those WANs and NAT to the real internet
+
+All of it previously existed only as running state on the VMs, applied by hand
+over SSH. Rebuilding a VM lost the rehearsal, and nothing recorded *why* any of
+it was shaped the way it is. That is the entire reason this file exists.
+
+  ./sim-net-config.py --role primary   > r1-net.conf
+  ./console-apply.py --vm labsim-vyos  --config r1-net.conf
+
+or apply all four at once with ./sim-net-apply.sh.
+"""
+from __future__ import annotations
+
+import argparse
+import sys
+
+# ---------------------------------------------------------------------------
+# BGP. Numbers match production so what is proven here ports over unchanged.
+# ---------------------------------------------------------------------------
+ROUTER_AS = 65000
+CLUSTER_AS = 65001
+# The service range Cilium advertises. Chosen against a survey of third-party
+# RFC1918 defaults (docker-desktop, tailscale, k3s, EKS...) so it cannot collide
+# with something we adopt later. Production uses the same /22 -- keep them equal.
+SERVICE_CIDR = "10.61.0.0/22"
+K8S_VLAN = 2
+K8S_NODES = ["172.31.2.11", "172.31.2.12", "172.31.2.13"]
+PEER_GROUP = "K8S"
+PFX_LIST = "K8S-SERVICE-IPS"
+RM_IN, RM_OUT = "K8S-IN", "K8S-OUT"
+# One route per node; ECMP across all three. 4 leaves headroom for a fourth node
+# without a config change.
+MAX_PATHS = 4
+# A safety valve, not a capacity plan: a misconfigured Cilium that starts
+# advertising pod CIDRs should tear the session down, not quietly fill the FIB.
+MAX_PREFIX = 100
+
+# ---------------------------------------------------------------------------
+# Dual WAN. The sim ISPs deliberately use TEST-NET-3 (203.0.113.0/24) and
+# TEST-NET-2 (198.51.100.0/24) from RFC 5737: documentation ranges that are
+# guaranteed never to be real destinations, so a leaked sim route cannot
+# blackhole something that matters.
+# ---------------------------------------------------------------------------
+WAN_DHCP_VLAN = 53          # "10gig-equivalent" -- the primary in production
+WAN_PPPOE_VLAN = 51         # "Vodafone-equivalent" -- the backup
+ISP_DHCP_NET = "203.0.113.0/24"
+ISP_DHCP_GW = "203.0.113.1"
+ISP_DHCP_POOL = ("203.0.113.100", "203.0.113.150")
+ISP_PPPOE_NET = "198.51.100.0/24"
+ISP_PPPOE_GW = "198.51.100.1"
+ISP_PPPOE_POOL = ("198.51.100.100", "198.51.100.150")
+# Sim-only fake credentials. Both ends are in this file on purpose: they
+# authenticate nothing real, and splitting them across a secret store would make
+# the sim unreproducible for no security gain. The PRODUCTION PPPoE password
+# lives in /config/wan-secrets on the router and is never in git.
+PPPOE_USER, PPPOE_PASS = "simdsl", "simpass"
+PPPOE_MTU = 1492            # 1500 - 8 bytes of PPPoE header
+PPPOE_AC = "sim-isp"
+
+# Failover probe targets. NOT 8.8.8.8/8.8.4.4: those are `system name-server`,
+# so a probe failure and a DNS failure would be the same event and the router
+# would flap the WAN every time DNS hiccuped.
+PROBE_TARGETS = ["9.9.9.9", "208.67.222.222"]
+# The bug this shape fixes (WI-8, found here, fixed in production): `ping -I
+# bond0.53` binds the SOURCE address but does not make the kernel use that
+# interface's gateway. On a cold boot where PPPoE won the default route, probes
+# for the 10 gig egressed via PPPoE, succeeded, and the 10 gig was still never
+# selected -- the house ran on the backup line silently. Pinning each target as
+# a /32 via `dhcp-interface` forces the probe onto the line being tested.
+WAN_DHCP_DISTANCE = 210     # NOT `no-default-route`, which blanks new_routers
+PPPOE_DISTANCE = 10         #   in the lease file, leaving failover no gateway
+                            #   to install and silently handing the default
+                            #   route to the backup line.
+SIM_LAN = "172.31.0.0/16"
+
+# The sim routers' own libvirt-NAT uplink, from before the ISP VMs existed. It
+# is a third default route that does not exist in production and quietly masks
+# WAN failures during a failover test. `--drop-scaffold` removes it.
+SCAFFOLD_IF = "eth2"
+SCAFFOLD_NAT_RULE = 100
+
+
+def bgp(role: str) -> list[str]:
+    """eBGP toward the k3s nodes. Identical on both routers except router-id."""
+    octet = 252 if role == "primary" else 253
+    out = [
+        f"# --- BGP: AS{ROUTER_AS} <-> AS{CLUSTER_AS} (k3s/Cilium) ---",
+        # FRR enforces RFC 8212: an eBGP session with no policy establishes but
+        # exchanges ZERO prefixes, silently. Both directions need a policy or
+        # the session looks perfectly healthy and carries nothing.
+        f"set policy prefix-list {PFX_LIST} rule 10 action permit",
+        f"set policy prefix-list {PFX_LIST} rule 10 prefix {SERVICE_CIDR}",
+        # `le 32` because Cilium advertises individual /32 service addresses out
+        # of the pool, not the aggregate.
+        f"set policy prefix-list {PFX_LIST} rule 10 le 32",
+        f"set policy route-map {RM_IN} rule 10 action permit",
+        f"set policy route-map {RM_IN} rule 10 match ip address prefix-list {PFX_LIST}",
+        # Deny everything outbound. The cluster must never learn a default route
+        # from us -- Cilium would install it and blackhole pod egress.
+        f"set policy route-map {RM_OUT} rule 10 action deny",
+        f"set protocols bgp system-as {ROUTER_AS}",
+        f"set protocols bgp parameters router-id 172.31.{K8S_VLAN}.{octet}",
+        f"set protocols bgp address-family ipv4-unicast maximum-paths ebgp {MAX_PATHS}",
+        f"set protocols bgp peer-group {PEER_GROUP} remote-as {CLUSTER_AS}",
+        f"set protocols bgp peer-group {PEER_GROUP} address-family ipv4-unicast route-map import {RM_IN}",
+        f"set protocols bgp peer-group {PEER_GROUP} address-family ipv4-unicast route-map export {RM_OUT}",
+        f"set protocols bgp peer-group {PEER_GROUP} address-family ipv4-unicast maximum-prefix {MAX_PREFIX}",
+    ]
+    out += [f"set protocols bgp neighbor {n} peer-group {PEER_GROUP}" for n in K8S_NODES]
+    out.append("")
+    return out
+
+
+def wan(drop_scaffold: bool) -> list[str]:
+    """Dual WAN + health-checked failover. Primary router only -- see README."""
+    out = [
+        "# --- WAN: DHCP (primary) + PPPoE (backup), health-checked ---",
+        f"set interfaces bonding bond0 vif {WAN_PPPOE_VLAN} description "
+        f"'WAN1 Vodafone-equivalent (sim ISP PPPoE)'",
+        f"set interfaces bonding bond0 vif {WAN_DHCP_VLAN} address dhcp",
+        f"set interfaces bonding bond0 vif {WAN_DHCP_VLAN} description "
+        f"'WAN3 10gig-equivalent (sim ISP DHCP)'",
+        f"set interfaces bonding bond0 vif {WAN_DHCP_VLAN} dhcp-options "
+        f"default-route-distance {WAN_DHCP_DISTANCE}",
+        f"set interfaces pppoe pppoe0 source-interface bond0.{WAN_PPPOE_VLAN}",
+        f"set interfaces pppoe pppoe0 authentication username {PPPOE_USER}",
+        f"set interfaces pppoe pppoe0 authentication password {PPPOE_PASS}",
+        f"set interfaces pppoe pppoe0 default-route-distance {PPPOE_DISTANCE}",
+        f"set interfaces pppoe pppoe0 mtu {PPPOE_MTU}",
+        # The ISP's resolvers would otherwise overwrite ours in resolv.conf every
+        # time the session comes up.
+        "set interfaces pppoe pppoe0 no-peer-dns",
+        "",
+        "# Failover: prefer the DHCP WAN, fall back to PPPoE when probes fail.",
+        f"set protocols failover route 0.0.0.0/0 dhcp-interface bond0.{WAN_DHCP_VLAN} metric 1",
+        f"set protocols failover route 0.0.0.0/0 dhcp-interface bond0.{WAN_DHCP_VLAN} check type icmp",
+        f"set protocols failover route 0.0.0.0/0 dhcp-interface bond0.{WAN_DHCP_VLAN} check timeout 5",
+        # any-available, not all: one unreachable public resolver is a normal
+        # internet event, not a reason to abandon a working 10 gig line.
+        f"set protocols failover route 0.0.0.0/0 dhcp-interface bond0.{WAN_DHCP_VLAN} check policy any-available",
+    ]
+    for t in PROBE_TARGETS:
+        out.append(f"set protocols failover route 0.0.0.0/0 dhcp-interface "
+                   f"bond0.{WAN_DHCP_VLAN} check target {t}")
+    out.append("")
+    out.append("# Pin the probe targets to the line under test (WI-8 -- see above).")
+    for t in PROBE_TARGETS:
+        out.append(f"set protocols static route {t}/32 dhcp-interface bond0.{WAN_DHCP_VLAN}")
+    out += [
+        "",
+        "# Masquerade out of whichever WAN currently holds the default route.",
+        f"set nat source rule 110 outbound-interface name bond0.{WAN_DHCP_VLAN}",
+        f"set nat source rule 110 source address {SIM_LAN}",
+        "set nat source rule 110 translation address masquerade",
+        "set nat source rule 120 outbound-interface name pppoe0",
+        f"set nat source rule 120 source address {SIM_LAN}",
+        "set nat source rule 120 translation address masquerade",
+        "",
+    ]
+    if drop_scaffold:
+        out += [
+            "# Remove the pre-ISP-VM libvirt-NAT uplink: a third default route",
+            "# that has no production equivalent and hides real WAN failures.",
+            f"delete interfaces ethernet {SCAFFOLD_IF} address",
+            f"delete nat source rule {SCAFFOLD_NAT_RULE}",
+            "",
+        ]
+    return out
+
+
+# ---------------------------------------------------------------------------
+# Firewall. The policy is: internal VLANs talk to each other and to the
+# internet; the internet initiates nothing inward.
+#
+# That was already the *effect* of the previous IPv4 ruleset, but it was built
+# as a blacklist -- `default-action accept` plus explicit drops on each WAN
+# interface. The result is identical right up until someone adds a WAN, at
+# which point it is wide open and nothing looks wrong. This is the same policy
+# expressed as a whitelist, so a new interface is closed until it is named.
+# ---------------------------------------------------------------------------
+# Management is `bond0.1`, NOT the bare `bond0`. Every VLAN is tagged and the
+# bond parent carries no subnet at all -- see NATIVE_VLAN in ovs.sh for why.
+#
+# This line is the trap in that change. The address move is the visible part and
+# the part you remember; leaving `bond0` here instead of `bond0.1` means the
+# whole Management VLAN falls outside the LAN group, and with a default-deny
+# ruleset that is every management session and all inter-VLAN routing for VLAN 1
+# dropped the instant the commit lands -- on a router you reach through itself.
+LAN_IFACES = ["bond0.1", "bond0.2", "bond0.3", "bond0.9", "bond0.10", "bond0.200"]
+LAN_GROUP = "LAN"
+
+
+def firewall(wan_dhcp_if: str | None = f"bond0.{WAN_DHCP_VLAN}") -> list[str]:
+    """wan_dhcp_if=None on a router with no DHCP WAN -- a firewall rule naming
+    an interface that does not exist is rejected at commit."""
+    out = [f"# --- firewall: LAN-to-anywhere, internet-to-nothing ---"]
+    # Delete each filter before rebuilding it. `set` on a rule number is
+    # ADDITIVE: if a rule 10 already exists carrying an inbound-interface
+    # constraint, `set ... rule 10 state established` silently ANDs onto it,
+    # and you get a stateful-accept rule that only applies to one interface
+    # pair. Observed in labsim: return traffic from the internet matched
+    # neither that rule nor the LAN rule and hit the default drop, so LAN
+    # hosts could reach nothing outbound. Everything here is one commit, so
+    # nftables is rebuilt atomically -- there is no window with no firewall.
+    out += [f"delete firewall {fam} {hook} filter"
+            for fam in ("ipv4", "ipv6") for hook in ("forward", "input")]
+    out += [f"set firewall group interface-group {LAN_GROUP} interface {i}"
+            for i in LAN_IFACES]
+    out += [
+        "",
+        # INPUT -- traffic terminating ON the router.
+        # Loopback first. Under a default-drop input policy, services talking to
+        # 127.0.0.1 are filtered like anything else, and the failures are
+        # bizarre and hard to attribute. Nothing off-box can forge iif lo.
+        "set firewall ipv4 input filter rule 5 action accept",
+        "set firewall ipv4 input filter rule 5 description 'loopback'",
+        "set firewall ipv4 input filter rule 5 inbound-interface name lo",
+        "set firewall ipv4 input filter rule 10 action accept",
+        "set firewall ipv4 input filter rule 10 description 'established/related'",
+        "set firewall ipv4 input filter rule 10 state established",
+        "set firewall ipv4 input filter rule 10 state related",
+        # One rule covers VRRP, conntrack-sync, kea HA, SSH, DNS and BGP,
+        # because every one of them arrives on a LAN interface. Enumerating the
+        # protocols instead would mean a new firewall rule every time the pair
+        # gains a feature -- and a lockout the day someone forgets.
+        f"set firewall ipv4 input filter rule 20 action accept",
+        f"set firewall ipv4 input filter rule 20 description 'trusted LAN to the router'",
+        f"set firewall ipv4 input filter rule 20 inbound-interface group {LAN_GROUP}",
+        # DHCP client. Lease RENEWAL is unicast UDP to port 68 and conntrack
+        # does not reliably cover it, so without this the WAN keeps working
+        # until the lease expires and then dies -- a delayed failure that looks
+        # nothing like a firewall change.
+        "set firewall ipv4 input filter default-action drop",
+        "",
+        # FORWARD -- traffic passing THROUGH the router.
+        "set firewall ipv4 forward filter rule 10 action accept",
+        "set firewall ipv4 forward filter rule 10 description 'established/related'",
+        "set firewall ipv4 forward filter rule 10 state established",
+        "set firewall ipv4 forward filter rule 10 state related",
+        # Inter-VLAN *and* LAN-to-internet in one rule: both are "came in on a
+        # LAN interface". Deliberately no restriction between internal VLANs --
+        # segmenting them is a separate decision, not a side effect of this one.
+        f"set firewall ipv4 forward filter rule 20 action accept",
+        f"set firewall ipv4 forward filter rule 20 description 'LAN to anywhere (inter-VLAN + internet)'",
+        f"set firewall ipv4 forward filter rule 20 inbound-interface group {LAN_GROUP}",
+        "set firewall ipv4 forward filter default-action drop",
+        "",
+        # IPv6 already runs default-deny. It only lacks the loopback rule.
+        "set firewall ipv6 input filter rule 5 action accept",
+        "set firewall ipv6 input filter rule 5 description 'loopback'",
+        "set firewall ipv6 input filter rule 5 inbound-interface name lo",
+        "set firewall ipv6 input filter rule 10 action accept",
+        "set firewall ipv6 input filter rule 10 description 'replies to our own traffic'",
+        "set firewall ipv6 input filter rule 10 state established",
+        "set firewall ipv6 input filter rule 10 state related",
+        # RFC 4890: filtering ICMPv6 wholesale breaks ND and PMTUD, which
+        # presents as "IPv6 works until something large", not as a block.
+        "set firewall ipv6 input filter rule 20 action accept",
+        "set firewall ipv6 input filter rule 20 description 'ICMPv6 - ND/RA/PMTUD'",
+        "set firewall ipv6 input filter rule 20 protocol icmpv6",
+        f"set firewall ipv6 input filter rule 30 action accept",
+        f"set firewall ipv6 input filter rule 30 description 'trusted LAN to the router'",
+        f"set firewall ipv6 input filter rule 30 inbound-interface group {LAN_GROUP}",
+        "set firewall ipv6 input filter default-action drop",
+        "set firewall ipv6 forward filter rule 10 action accept",
+        "set firewall ipv6 forward filter rule 10 description 'replies to our own traffic'",
+        "set firewall ipv6 forward filter rule 10 state established",
+        "set firewall ipv6 forward filter rule 10 state related",
+        "set firewall ipv6 forward filter rule 20 action accept",
+        "set firewall ipv6 forward filter rule 20 description 'ICMPv6 - ND/RA/PMTUD'",
+        "set firewall ipv6 forward filter rule 20 protocol icmpv6",
+        f"set firewall ipv6 forward filter rule 30 action accept",
+        f"set firewall ipv6 forward filter rule 30 description 'trusted LAN interfaces only'",
+        f"set firewall ipv6 forward filter rule 30 inbound-interface group {LAN_GROUP}",
+        "set firewall ipv6 forward filter default-action drop",
+        "",
+    ]
+    if wan_dhcp_if:
+        dhcp = [
+            "set firewall ipv4 input filter rule 140 action accept",
+            "set firewall ipv4 input filter rule 140 description 'DHCP client lease renewal'",
+            "set firewall ipv4 input filter rule 140 protocol udp",
+            "set firewall ipv4 input filter rule 140 destination port 68",
+            f"set firewall ipv4 input filter rule 140 inbound-interface name {wan_dhcp_if}",
+        ]
+        i = out.index("set firewall ipv4 input filter default-action drop")
+        out[i:i] = dhcp
+    return out
+
+
+def isp_dhcp(wan_if: str, uplink_if: str) -> list[str]:
+    """The 10gig-equivalent ISP: hands out a lease, NATs to the real internet."""
+    return [
+        f"# --- sim ISP: DHCP WAN on VLAN {WAN_DHCP_VLAN} ---",
+        "set system host-name isp-dhcp",
+        f"set interfaces ethernet {wan_if} address {ISP_DHCP_GW}/24",
+        f"set interfaces ethernet {wan_if} description "
+        f"'sim ISP - 10gig-equivalent WAN on VLAN{WAN_DHCP_VLAN}'",
+        f"set interfaces ethernet {uplink_if} address dhcp",
+        f"set interfaces ethernet {uplink_if} description 'uplink to the real internet'",
+        f"set service dhcp-server shared-network-name WAN{WAN_DHCP_VLAN} "
+        f"subnet {ISP_DHCP_NET} subnet-id 1",
+        f"set service dhcp-server shared-network-name WAN{WAN_DHCP_VLAN} "
+        f"subnet {ISP_DHCP_NET} option default-router {ISP_DHCP_GW}",
+        f"set service dhcp-server shared-network-name WAN{WAN_DHCP_VLAN} "
+        f"subnet {ISP_DHCP_NET} option name-server 8.8.8.8",
+        f"set service dhcp-server shared-network-name WAN{WAN_DHCP_VLAN} "
+        f"subnet {ISP_DHCP_NET} range CUST start {ISP_DHCP_POOL[0]}",
+        f"set service dhcp-server shared-network-name WAN{WAN_DHCP_VLAN} "
+        f"subnet {ISP_DHCP_NET} range CUST stop {ISP_DHCP_POOL[1]}",
+        "",
+    ] + _isp_common(uplink_if, ISP_DHCP_NET, "sim ISP: NAT customers to the real internet")
+
+
+def isp_pppoe(wan_if: str, uplink_if: str) -> list[str]:
+    """The Vodafone-equivalent ISP: terminates PPPoE, NATs to the real internet."""
+    return [
+        f"# --- sim ISP: PPPoE WAN on VLAN {WAN_PPPOE_VLAN} ---",
+        "set system host-name isp-pppoe",
+        f"set interfaces ethernet {wan_if} description "
+        f"'sim ISP - Vodafone-equivalent WAN on VLAN{WAN_PPPOE_VLAN} (PPPoE)'",
+        f"set interfaces ethernet {uplink_if} address dhcp",
+        f"set interfaces ethernet {uplink_if} description 'uplink to the real internet'",
+        f"set service pppoe-server access-concentrator {PPPOE_AC}",
+        f"set service pppoe-server interface {wan_if}",
+        f"set service pppoe-server gateway-address {ISP_PPPOE_GW}",
+        "set service pppoe-server authentication mode local",
+        f"set service pppoe-server authentication local-users username {PPPOE_USER} "
+        f"password {PPPOE_PASS}",
+        f"set service pppoe-server client-ip-pool CUST range "
+        f"{ISP_PPPOE_POOL[0]}-{ISP_PPPOE_POOL[1]}",
+        "set service pppoe-server default-pool CUST",
+        "set service pppoe-server name-server 8.8.8.8",
+        "",
+    ] + _isp_common(uplink_if, ISP_PPPOE_NET, "sim ISP: NAT PPPoE customers to the real internet")
+
+
+def _isp_common(uplink_if: str, customer_net: str, desc: str) -> list[str]:
+    return [
+        "set nat source rule 100 description " + f"'{desc}'",
+        f"set nat source rule 100 outbound-interface name {uplink_if}",
+        f"set nat source rule 100 source address {customer_net}",
+        "set nat source rule 100 translation address masquerade",
+        "",
+        # An ISP that drops return traffic is not simulating an ISP. The forward
+        # chain defaults to accept here on purpose -- these VMs model the
+        # internet, and the thing under test is the router's firewall, not this.
+        "set firewall ipv4 forward filter default-action accept",
+        "set firewall ipv4 forward filter rule 10 action accept",
+        "set firewall ipv4 forward filter rule 10 state established",
+        "set firewall ipv4 forward filter rule 10 state related",
+        "set firewall ipv4 forward filter rule 10 description conntrack-engage",
+        "",
+    ]
+
+
+def build(role: str, drop_scaffold: bool, wan_if: str, uplink_if: str) -> list[str]:
+    if role == "primary":
+        # WAN lives on the primary only. Production has WAN on both routers;
+        # the sim does not, because two PPPoE clients sharing one credential
+        # against a single access concentrator is a different failure mode than
+        # anything production has. VRRP/conntrack failover is still exercised --
+        # see README, "known gaps".
+        return ([f"# labsim routing -- {role}", ""]
+                + bgp(role) + wan(drop_scaffold) + firewall())
+    if role == "secondary":
+        # The backup has no WAN in the sim, so it has no DHCP client to
+        # exempt -- but it gets the same policy otherwise, because after a VRRP
+        # failover it IS the router and a divergent ruleset would only be
+        # discovered during the failover.
+        return ([f"# labsim routing -- {role}", ""]
+                + bgp(role) + firewall(wan_dhcp_if=None))
+    if role == "isp-dhcp":
+        return isp_dhcp(wan_if, uplink_if)
+    return isp_pppoe(wan_if, uplink_if)
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--role", required=True,
+                    choices=("primary", "secondary", "isp-dhcp", "isp-pppoe"))
+    ap.add_argument("--drop-scaffold", action="store_true",
+                    help="also remove the pre-ISP-VM libvirt-NAT uplink (primary only)")
+    # The ISP VMs' interface names depend on PCI enumeration order, which is not
+    # stable across a rebuild: isp-dhcp came up as eth0/eth1 and isp-pppoe as
+    # eth2/eth3 from identical XML. Check with `show interfaces` before applying
+    # rather than trusting these defaults.
+    ap.add_argument("--wan-if", default=None, help="ISP VM: customer-facing NIC")
+    ap.add_argument("--uplink-if", default=None, help="ISP VM: internet-facing NIC")
+    args = ap.parse_args()
+
+    defaults = {"isp-dhcp": ("eth0", "eth1"), "isp-pppoe": ("eth2", "eth3")}
+    w, u = defaults.get(args.role, ("", ""))
+    w, u = args.wan_if or w, args.uplink_if or u
+
+    if args.drop_scaffold and args.role != "primary":
+        print("--drop-scaffold only applies to --role primary", file=sys.stderr)
+        return 2
+
+    lines = build(args.role, args.drop_scaffold, w, u)
+    sys.stdout.write("\n".join(lines) + "\n")
+    n = len([l for l in lines if l.startswith(("set ", "delete "))])
+    print(f"{args.role}: {n} commands", file=sys.stderr)
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

labsim/topology.html

@@ -0,0 +1,181 @@
+<!doctype html>
+<html lang="en">
+<head>
+<meta charset="utf-8">
+<title>labsim — live VLAN topology</title>
+<style>
+  :root {
+    --bg:#0e1116; --panel:#161b22; --line:#30363d; --text:#e6edf3; --dim:#8b949e;
+    --ok:#3fb950; --bad:#f85149; --warn:#d29922; --router:#58a6ff;
+  }
+  * { box-sizing:border-box; }
+  body { margin:0; background:var(--bg); color:var(--text);
+         font:14px/1.5 ui-sans-serif,system-ui,-apple-system,"Segoe UI",sans-serif; }
+  header { display:flex; align-items:baseline; gap:16px; flex-wrap:wrap;
+           padding:14px 20px; border-bottom:1px solid var(--line); }
+  h1 { font-size:16px; margin:0; font-weight:650; letter-spacing:.2px; }
+  .meta { color:var(--dim); font-size:12px; }
+  .pill { padding:2px 8px; border-radius:999px; font-size:12px; font-weight:600; }
+  .pill.ok { background:rgba(63,185,80,.15); color:var(--ok); }
+  .pill.bad { background:rgba(248,81,73,.15); color:var(--bad); }
+  main { display:grid; grid-template-columns:minmax(0,1.35fr) minmax(320px,.65fr);
+         gap:16px; padding:16px 20px; align-items:start; }
+  @media (max-width:1000px){ main { grid-template-columns:1fr; } }
+  .card { background:var(--panel); border:1px solid var(--line); border-radius:10px; padding:14px; }
+  .card h2 { margin:0 0 10px; font-size:13px; font-weight:600; color:var(--dim);
+             text-transform:uppercase; letter-spacing:.6px; }
+  svg { width:100%; height:auto; display:block; }
+  .edge { stroke-width:2.5; transition:stroke .25s, opacity .25s; }
+  .edge.ok { stroke:var(--ok); opacity:.55; }
+  .edge.bad { stroke:var(--bad); opacity:.95; stroke-dasharray:7 5; }
+  .edge:hover { opacity:1; stroke-width:4; }
+  .node circle { fill:#0d1117; stroke-width:2.5; }
+  .node text { text-anchor:middle; font-size:11px; font-weight:600; fill:var(--text); }
+  .node .sub { font-size:9.5px; font-weight:400; fill:var(--dim); }
+  .rtt { font-size:9px; fill:var(--dim); text-anchor:middle; }
+  table { width:100%; border-collapse:collapse; font-size:12.5px; }
+  th,td { text-align:left; padding:5px 8px; border-bottom:1px solid var(--line); }
+  th { color:var(--dim); font-weight:600; font-size:11px; text-transform:uppercase; }
+  td.n { text-align:right; font-variant-numeric:tabular-nums; }
+  .b-ok { color:var(--ok); } .b-bad { color:var(--bad); }
+  .empty { color:var(--dim); padding:10px 4px; }
+  .legend { display:flex; gap:14px; align-items:center; color:var(--dim);
+            font-size:11.5px; margin-top:10px; flex-wrap:wrap; }
+  .swatch { display:inline-block; width:22px; height:0; border-top:2.5px solid; margin-right:5px;
+            vertical-align:middle; }
+</style>
+</head>
+<body>
+<header>
+  <h1>labsim — live VLAN topology</h1>
+  <span id="summary" class="pill ok">…</span>
+  <span class="meta">every path is probed <em>from</em> a VM <em>to</em> every other VM, through the VyOS router</span>
+  <span class="meta" id="clock" style="margin-left:auto"></span>
+</header>
+
+<main>
+  <section class="card">
+    <h2>Mesh — line colour is reachability, label is ICMP RTT</h2>
+    <svg id="topo" viewBox="0 0 720 560" role="img" aria-label="VLAN topology"></svg>
+    <div class="legend">
+      <span><i class="swatch" style="border-color:var(--ok)"></i>reachable</span>
+      <span><i class="swatch" style="border-color:var(--bad); border-top-style:dashed"></i>blocked</span>
+      <span>hover a line for detail · node ring turns red if anything to/from it is blocked</span>
+    </div>
+  </section>
+
+  <aside style="display:grid; gap:16px">
+    <section class="card">
+      <h2>Blocked paths</h2>
+      <div id="blocked"></div>
+    </section>
+    <section class="card">
+      <h2>Latency (ICMP, ms)</h2>
+      <table><thead><tr><th>path</th><th class="n">rtt</th></tr></thead>
+      <tbody id="lat"></tbody></table>
+    </section>
+  </aside>
+</main>
+
+<script>
+const REFRESH_MS = 5000;
+const CX = 360, CY = 250, R = 185;
+
+function polar(i, n) {
+  const a = (i / n) * Math.PI * 2 - Math.PI / 2;
+  return { x: CX + R * Math.cos(a), y: CY + R * Math.sin(a) };
+}
+
+function render(data) {
+  const vlans = data.vlans, res = data.results;
+  const svg = document.getElementById('topo');
+  const n = vlans.length;
+  const pos = vlans.map((_, i) => polar(i, n));
+  let out = '';
+
+  // Router in the middle — every inter-VLAN packet really does traverse it.
+  out += `<circle cx="${CX}" cy="${CY}" r="40" fill="#0d1117" stroke="var(--router)" stroke-width="2.5"/>`;
+  out += `<text x="${CX}" y="${CY-6}" text-anchor="middle" font-size="12" font-weight="700" fill="var(--router)">VyOS</text>`;
+  out += `<text x="${CX}" y="${CY+9}" text-anchor="middle" font-size="8.5" fill="var(--dim)">bond0</text>`;
+  out += `<text x="${CX}" y="${CY+20}" text-anchor="middle" font-size="8.5" fill="var(--dim)">LACP</text>`;
+
+  const bad = new Set();
+  // One line per unordered pair; a pair is bad if EITHER direction fails.
+  for (let i = 0; i < n; i++) {
+    for (let j = i + 1; j < n; j++) {
+      const a = vlans[i].label, b = vlans[j].label;
+      const ab = (res[a] || {})[b] || {}, ba = (res[b] || {})[a] || {};
+      const okAB = ab.icmp === true, okBA = ba.icmp === true;
+      const ok = okAB && okBA;
+      if (!ok) { bad.add(a); bad.add(b); }
+      const rtts = [ab.rtt_ms, ba.rtt_ms].filter(v => typeof v === 'number');
+      const rtt = rtts.length ? (rtts.reduce((s,v)=>s+v,0)/rtts.length) : null;
+      // Place the label ~32% along the edge, not at the midpoint: diagonals of
+      // a 6-node mesh all cross the centre, so midpoint labels stack on top of
+      // the router node. Plus a small perpendicular nudge off the line itself.
+      const dx = pos[j].x - pos[i].x, dy = pos[j].y - pos[i].y;
+      const len = Math.hypot(dx, dy) || 1;
+      const t = 0.32;
+      const mx = pos[i].x + dx * t + (-dy / len) * 8;
+      const my = pos[i].y + dy * t + ( dx / len) * 8;
+      const tip = `${a} ↔ ${b}\n→ ${okAB ? 'ok' : 'BLOCKED'}   ← ${okBA ? 'ok' : 'BLOCKED'}` +
+                  (rtt !== null ? `\nrtt ${rtt.toFixed(2)} ms` : '');
+      out += `<line class="edge ${ok?'ok':'bad'}" x1="${pos[i].x}" y1="${pos[i].y}" x2="${pos[j].x}" y2="${pos[j].y}"><title>${tip}</title></line>`;
+      if (ok && rtt !== null)
+        out += `<text class="rtt" x="${mx}" y="${my}">${rtt.toFixed(2)}</text>`;
+    }
+  }
+
+  vlans.forEach((v, i) => {
+    const p = pos[i], isBad = bad.has(v.label);
+    out += `<g class="node"><circle cx="${p.x}" cy="${p.y}" r="30" stroke="${isBad?'var(--bad)':'var(--ok)'}"/>` +
+           `<text x="${p.x}" y="${p.y-2}">${v.name}</text>` +
+           `<text class="sub" x="${p.x}" y="${p.y+11}">vlan ${v.vid}</text>` +
+           `<text class="sub" x="${p.x}" y="${p.y+47}">${v.ip}</text></g>`;
+  });
+  svg.innerHTML = out;
+
+  // Blocked list — the thing you actually act on.
+  const rows = [];
+  for (const src of vlans) for (const dst of vlans) {
+    if (src.label === dst.label) continue;
+    const d = (res[src.label] || {})[dst.label] || {};
+    for (const proto of ['icmp','tcp22','tcp80'])
+      if (d[proto] === false) rows.push(`${src.label} → ${dst.label} <span style="color:var(--dim)">(${proto})</span>`);
+  }
+  document.getElementById('blocked').innerHTML = rows.length
+    ? `<table><tbody>${rows.map(r=>`<tr><td class="b-bad">${r}</td></tr>`).join('')}</tbody></table>`
+    : `<div class="empty">none — all ${vlans.length*(vlans.length-1)*3} paths open</div>`;
+
+  // Latency table, slowest first.
+  const lat = [];
+  for (const src of vlans) for (const dst of vlans) {
+    if (src.label === dst.label) continue;
+    const d = (res[src.label] || {})[dst.label] || {};
+    if (typeof d.rtt_ms === 'number') lat.push([`${src.label} → ${dst.label}`, d.rtt_ms]);
+  }
+  lat.sort((a,b) => b[1]-a[1]);
+  document.getElementById('lat').innerHTML = lat.slice(0,12)
+    .map(([k,v]) => `<tr><td>${k}</td><td class="n">${v.toFixed(2)}</td></tr>`).join('')
+    || `<tr><td class="empty" colspan="2">no RTT data</td></tr>`;
+
+  const total = data.total, reach = data.reachable;
+  const pill = document.getElementById('summary');
+  pill.textContent = `${reach}/${total} paths open`;
+  pill.className = 'pill ' + (reach === total ? 'ok' : 'bad');
+  document.getElementById('clock').textContent =
+    `updated ${new Date().toLocaleTimeString()} · sweep ${data.sweep_seconds.toFixed(2)}s · refresh ${REFRESH_MS/1000}s`;
+}
+
+async function tick() {
+  try {
+    const r = await fetch('/api/matrix', {cache:'no-store'});
+    render(await r.json());
+  } catch (e) {
+    document.getElementById('clock').textContent = 'exporter unreachable — ' + e;
+  }
+}
+tick(); setInterval(tick, REFRESH_MS);
+</script>
+</body>
+</html>

labsim/vlan-leak-evidence/after-vlan1/capture-parent.txt

@@ -0,0 +1,6 @@
+12:52:35.919490 52:54:00:6d:71:e7 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 346: vlan 1, p 0, ethertype IPv4 (0x0800), 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:6d:71:e7, length 300
+12:52:35.920089 52:54:00:e5:95:a2 > 52:54:00:6d:71:e7, ethertype 802.1Q (0x8100), length 329: vlan 1, p 0, ethertype IPv4 (0x0800), 172.31.1.252.67 > 172.31.1.6.68: BOOTP/DHCP, Reply, length 283
+12:52:35.920307 52:54:00:e5:95:a2 > 52:54:00:6d:71:e7, ethertype 802.1Q (0x8100), length 329: vlan 1, p 0, ethertype IPv4 (0x0800), 172.31.1.252.67 > 172.31.1.7.68: BOOTP/DHCP, Reply, length 283
+12:52:35.922052 52:54:00:6d:71:e7 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 346: vlan 1, p 0, ethertype IPv4 (0x0800), 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:6d:71:e7, length 300
+12:52:35.922509 52:54:00:e5:95:a2 > 52:54:00:6d:71:e7, ethertype 802.1Q (0x8100), length 329: vlan 1, p 0, ethertype IPv4 (0x0800), 172.31.1.252.67 > 172.31.1.6.68: BOOTP/DHCP, Reply, length 283
+12:52:35.923327 52:54:00:e5:95:a2 > 52:54:00:6d:71:e7, ethertype 802.1Q (0x8100), length 329: vlan 1, p 0, ethertype IPv4 (0x0800), 172.31.1.252.67 > 172.31.1.6.68: BOOTP/DHCP, Reply, length 283

labsim/vlan-leak-evidence/after-vlan1/capture-vif.txt

@@ -0,0 +1,6 @@
+12:52:35.919490 52:54:00:6d:71:e7 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:6d:71:e7, length 300
+12:52:35.920081 52:54:00:e5:95:a2 > 52:54:00:6d:71:e7, ethertype IPv4 (0x0800), length 325: 172.31.1.252.67 > 172.31.1.6.68: BOOTP/DHCP, Reply, length 283
+12:52:35.920305 52:54:00:e5:95:a2 > 52:54:00:6d:71:e7, ethertype IPv4 (0x0800), length 325: 172.31.1.252.67 > 172.31.1.7.68: BOOTP/DHCP, Reply, length 283
+12:52:35.922052 52:54:00:6d:71:e7 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:6d:71:e7, length 300
+12:52:35.922507 52:54:00:e5:95:a2 > 52:54:00:6d:71:e7, ethertype IPv4 (0x0800), length 325: 172.31.1.252.67 > 172.31.1.6.68: BOOTP/DHCP, Reply, length 283
+12:52:35.923326 52:54:00:e5:95:a2 > 52:54:00:6d:71:e7, ethertype IPv4 (0x0800), length 325: 172.31.1.252.67 > 172.31.1.6.68: BOOTP/DHCP, Reply, length 283

labsim/vlan-leak-evidence/after-vlan1/client.txt

@@ -0,0 +1,4 @@
+udhcpc: started, v1.37.0
+udhcpc: broadcasting discover
+udhcpc: broadcasting select for 172.31.1.6, server 172.31.1.252
+udhcpc: lease of 172.31.1.6 obtained from 172.31.1.252, lease time 86400

labsim/vlan-leak-evidence/after-vlan1/router-config.txt

@@ -0,0 +1,64 @@
+set high-availability vrrp group native address 172.31.1.1/24
+set high-availability vrrp group native hello-source-address '172.31.1.252'
+set high-availability vrrp group native interface 'bond0.1'
+set high-availability vrrp group native no-preempt
+set high-availability vrrp group native peer-address '172.31.1.253'
+set high-availability vrrp group native priority '200'
+set high-availability vrrp group native vrid '1'
+set high-availability vrrp group vlan2 address 172.31.2.1/24
+set high-availability vrrp group vlan2 hello-source-address '172.31.2.252'
+set high-availability vrrp group vlan2 interface 'bond0.2'
+set high-availability vrrp group vlan2 no-preempt
+set high-availability vrrp group vlan2 peer-address '172.31.2.253'
+set high-availability vrrp group vlan2 priority '200'
+set high-availability vrrp group vlan2 vrid '2'
+set high-availability vrrp group vlan3 address 172.31.3.1/24
+set high-availability vrrp group vlan3 hello-source-address '172.31.3.252'
+set high-availability vrrp group vlan3 interface 'bond0.3'
+set high-availability vrrp group vlan3 no-preempt
+set high-availability vrrp group vlan3 peer-address '172.31.3.253'
+set high-availability vrrp group vlan3 priority '200'
+set high-availability vrrp group vlan3 vrid '3'
+set high-availability vrrp group vlan9 address 172.31.9.1/24
+set high-availability vrrp group vlan9 hello-source-address '172.31.9.252'
+set high-availability vrrp group vlan9 interface 'bond0.9'
+set high-availability vrrp group vlan9 no-preempt
+set high-availability vrrp group vlan9 peer-address '172.31.9.253'
+set high-availability vrrp group vlan9 priority '200'
+set high-availability vrrp group vlan9 vrid '9'
+set high-availability vrrp group vlan10 address 172.31.10.1/23
+set high-availability vrrp group vlan10 hello-source-address '172.31.10.252'
+set high-availability vrrp group vlan10 interface 'bond0.10'
+set high-availability vrrp group vlan10 no-preempt
+set high-availability vrrp group vlan10 peer-address '172.31.10.253'
+set high-availability vrrp group vlan10 priority '200'
+set high-availability vrrp group vlan10 vrid '10'
+set high-availability vrrp group vlan200 address 172.31.200.1/24
+set high-availability vrrp group vlan200 hello-source-address '172.31.200.252'
+set high-availability vrrp group vlan200 interface 'bond0.200'
+set high-availability vrrp group vlan200 no-preempt
+set high-availability vrrp group vlan200 peer-address '172.31.200.253'
+set high-availability vrrp group vlan200 priority '200'
+set high-availability vrrp group vlan200 vrid '200'
+set interfaces bonding bond0 description 'api-batch-test'
+set interfaces bonding bond0 hash-policy 'layer2+3'
+set interfaces bonding bond0 lacp-rate 'fast'
+set interfaces bonding bond0 member interface 'eth0'
+set interfaces bonding bond0 member interface 'eth1'
+set interfaces bonding bond0 mode '802.3ad'
+set interfaces bonding bond0 vif 1 address '172.31.1.252/24'
+set interfaces bonding bond0 vif 1 description 'management'
+set interfaces bonding bond0 vif 2 address '172.31.2.252/24'
+set interfaces bonding bond0 vif 2 description 'k8s'
+set interfaces bonding bond0 vif 3 address '172.31.3.252/24'
+set interfaces bonding bond0 vif 3 description 'kvm'
+set interfaces bonding bond0 vif 9 address '172.31.9.252/24'
+set interfaces bonding bond0 vif 9 description 'private'
+set interfaces bonding bond0 vif 10 address '172.31.10.252/23'
+set interfaces bonding bond0 vif 10 description 'lot'
+set interfaces bonding bond0 vif 51 description 'WAN1 Vodafone-equivalent (sim ISP PPPoE)'
+set interfaces bonding bond0 vif 53 address 'dhcp'
+set interfaces bonding bond0 vif 53 description 'WAN3 10gig-equivalent (sim ISP DHCP)'
+set interfaces bonding bond0 vif 53 dhcp-options default-route-distance '210'
+set interfaces bonding bond0 vif 200 address '172.31.200.252/24'
+set interfaces bonding bond0 vif 200 description 'roomates'

labsim/vlan-leak-evidence/after/capture-parent.txt

@@ -0,0 +1,6 @@
+12:52:14.639170 52:54:00:02:2e:b1 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 346: vlan 3, p 0, ethertype IPv4 (0x0800), 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:02:2e:b1, length 300
+12:52:14.640467 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype 802.1Q (0x8100), length 329: vlan 3, p 0, ethertype IPv4 (0x0800), 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283
+12:52:14.640846 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype 802.1Q (0x8100), length 329: vlan 3, p 0, ethertype IPv4 (0x0800), 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283
+12:52:14.642554 52:54:00:02:2e:b1 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 346: vlan 3, p 0, ethertype IPv4 (0x0800), 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:02:2e:b1, length 300
+12:52:14.642766 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype 802.1Q (0x8100), length 329: vlan 3, p 0, ethertype IPv4 (0x0800), 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283
+12:52:14.643056 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype 802.1Q (0x8100), length 329: vlan 3, p 0, ethertype IPv4 (0x0800), 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283

labsim/vlan-leak-evidence/after/capture-vif.txt

@@ -0,0 +1,6 @@
+12:52:14.639170 52:54:00:02:2e:b1 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:02:2e:b1, length 300
+12:52:14.640465 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype IPv4 (0x0800), length 325: 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283
+12:52:14.640845 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype IPv4 (0x0800), length 325: 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283
+12:52:14.642554 52:54:00:02:2e:b1 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:02:2e:b1, length 300
+12:52:14.642764 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype IPv4 (0x0800), length 325: 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283
+12:52:14.643055 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype IPv4 (0x0800), length 325: 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283

labsim/vlan-leak-evidence/after/client.txt

@@ -0,0 +1,4 @@
+udhcpc: started, v1.37.0
+udhcpc: broadcasting discover
+udhcpc: broadcasting select for 172.31.3.11, server 172.31.3.252
+udhcpc: lease of 172.31.3.11 obtained from 172.31.3.252, lease time 85374

labsim/vlan-leak-evidence/after/router-config.txt

@@ -0,0 +1,64 @@
+set high-availability vrrp group native address 172.31.1.1/24
+set high-availability vrrp group native hello-source-address '172.31.1.252'
+set high-availability vrrp group native interface 'bond0.1'
+set high-availability vrrp group native no-preempt
+set high-availability vrrp group native peer-address '172.31.1.253'
+set high-availability vrrp group native priority '200'
+set high-availability vrrp group native vrid '1'
+set high-availability vrrp group vlan2 address 172.31.2.1/24
+set high-availability vrrp group vlan2 hello-source-address '172.31.2.252'
+set high-availability vrrp group vlan2 interface 'bond0.2'
+set high-availability vrrp group vlan2 no-preempt
+set high-availability vrrp group vlan2 peer-address '172.31.2.253'
+set high-availability vrrp group vlan2 priority '200'
+set high-availability vrrp group vlan2 vrid '2'
+set high-availability vrrp group vlan3 address 172.31.3.1/24
+set high-availability vrrp group vlan3 hello-source-address '172.31.3.252'
+set high-availability vrrp group vlan3 interface 'bond0.3'
+set high-availability vrrp group vlan3 no-preempt
+set high-availability vrrp group vlan3 peer-address '172.31.3.253'
+set high-availability vrrp group vlan3 priority '200'
+set high-availability vrrp group vlan3 vrid '3'
+set high-availability vrrp group vlan9 address 172.31.9.1/24
+set high-availability vrrp group vlan9 hello-source-address '172.31.9.252'
+set high-availability vrrp group vlan9 interface 'bond0.9'
+set high-availability vrrp group vlan9 no-preempt
+set high-availability vrrp group vlan9 peer-address '172.31.9.253'
+set high-availability vrrp group vlan9 priority '200'
+set high-availability vrrp group vlan9 vrid '9'
+set high-availability vrrp group vlan10 address 172.31.10.1/23
+set high-availability vrrp group vlan10 hello-source-address '172.31.10.252'
+set high-availability vrrp group vlan10 interface 'bond0.10'
+set high-availability vrrp group vlan10 no-preempt
+set high-availability vrrp group vlan10 peer-address '172.31.10.253'
+set high-availability vrrp group vlan10 priority '200'
+set high-availability vrrp group vlan10 vrid '10'
+set high-availability vrrp group vlan200 address 172.31.200.1/24
+set high-availability vrrp group vlan200 hello-source-address '172.31.200.252'
+set high-availability vrrp group vlan200 interface 'bond0.200'
+set high-availability vrrp group vlan200 no-preempt
+set high-availability vrrp group vlan200 peer-address '172.31.200.253'
+set high-availability vrrp group vlan200 priority '200'
+set high-availability vrrp group vlan200 vrid '200'
+set interfaces bonding bond0 description 'api-batch-test'
+set interfaces bonding bond0 hash-policy 'layer2+3'
+set interfaces bonding bond0 lacp-rate 'fast'
+set interfaces bonding bond0 member interface 'eth0'
+set interfaces bonding bond0 member interface 'eth1'
+set interfaces bonding bond0 mode '802.3ad'
+set interfaces bonding bond0 vif 1 address '172.31.1.252/24'
+set interfaces bonding bond0 vif 1 description 'management'
+set interfaces bonding bond0 vif 2 address '172.31.2.252/24'
+set interfaces bonding bond0 vif 2 description 'k8s'
+set interfaces bonding bond0 vif 3 address '172.31.3.252/24'
+set interfaces bonding bond0 vif 3 description 'kvm'
+set interfaces bonding bond0 vif 9 address '172.31.9.252/24'
+set interfaces bonding bond0 vif 9 description 'private'
+set interfaces bonding bond0 vif 10 address '172.31.10.252/23'
+set interfaces bonding bond0 vif 10 description 'lot'
+set interfaces bonding bond0 vif 51 description 'WAN1 Vodafone-equivalent (sim ISP PPPoE)'
+set interfaces bonding bond0 vif 53 address 'dhcp'
+set interfaces bonding bond0 vif 53 description 'WAN3 10gig-equivalent (sim ISP DHCP)'
+set interfaces bonding bond0 vif 53 dhcp-options default-route-distance '210'
+set interfaces bonding bond0 vif 200 address '172.31.200.252/24'
+set interfaces bonding bond0 vif 200 description 'roomates'

labsim/vlan-leak-evidence/before/capture-parent.txt

@@ -0,0 +1,5 @@
+12:37:08.491910 52:54:00:02:2e:b1 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 346: vlan 3, p 0, ethertype IPv4 (0x0800), 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:02:2e:b1, length 300
+12:37:08.492629 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype IPv4 (0x0800), length 325: 172.31.1.252.67 > 172.31.1.9.68: BOOTP/DHCP, Reply, length 283
+12:37:08.493628 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype 802.1Q (0x8100), length 329: vlan 3, p 0, ethertype IPv4 (0x0800), 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283
+12:37:08.495587 52:54:00:02:2e:b1 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 346: vlan 3, p 0, ethertype IPv4 (0x0800), 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:02:2e:b1, length 300
+12:37:08.495946 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype 802.1Q (0x8100), length 329: vlan 3, p 0, ethertype IPv4 (0x0800), 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283

labsim/vlan-leak-evidence/before/capture-vif.txt

@@ -0,0 +1,4 @@
+12:37:08.491910 52:54:00:02:2e:b1 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:02:2e:b1, length 300
+12:37:08.493625 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype IPv4 (0x0800), length 325: 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283
+12:37:08.495587 52:54:00:02:2e:b1 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:02:2e:b1, length 300
+12:37:08.495944 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype IPv4 (0x0800), length 325: 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283

labsim/vlan-leak-evidence/before/client.txt

@@ -0,0 +1,4 @@
+udhcpc: started, v1.37.0
+udhcpc: broadcasting discover
+udhcpc: broadcasting select for 172.31.3.11, server 172.31.3.252
+udhcpc: lease of 172.31.3.11 obtained from 172.31.3.252, lease time 86280

labsim/vlan-leak-evidence/before/router-config.txt

@@ -0,0 +1,63 @@
+set high-availability vrrp group native address 172.31.1.1/24
+set high-availability vrrp group native hello-source-address '172.31.1.252'
+set high-availability vrrp group native interface 'bond0'
+set high-availability vrrp group native no-preempt
+set high-availability vrrp group native peer-address '172.31.1.253'
+set high-availability vrrp group native priority '200'
+set high-availability vrrp group native vrid '1'
+set high-availability vrrp group vlan2 address 172.31.2.1/24
+set high-availability vrrp group vlan2 hello-source-address '172.31.2.252'
+set high-availability vrrp group vlan2 interface 'bond0.2'
+set high-availability vrrp group vlan2 no-preempt
+set high-availability vrrp group vlan2 peer-address '172.31.2.253'
+set high-availability vrrp group vlan2 priority '200'
+set high-availability vrrp group vlan2 vrid '2'
+set high-availability vrrp group vlan3 address 172.31.3.1/24
+set high-availability vrrp group vlan3 hello-source-address '172.31.3.252'
+set high-availability vrrp group vlan3 interface 'bond0.3'
+set high-availability vrrp group vlan3 no-preempt
+set high-availability vrrp group vlan3 peer-address '172.31.3.253'
+set high-availability vrrp group vlan3 priority '200'
+set high-availability vrrp group vlan3 vrid '3'
+set high-availability vrrp group vlan9 address 172.31.9.1/24
+set high-availability vrrp group vlan9 hello-source-address '172.31.9.252'
+set high-availability vrrp group vlan9 interface 'bond0.9'
+set high-availability vrrp group vlan9 no-preempt
+set high-availability vrrp group vlan9 peer-address '172.31.9.253'
+set high-availability vrrp group vlan9 priority '200'
+set high-availability vrrp group vlan9 vrid '9'
+set high-availability vrrp group vlan10 address 172.31.10.1/23
+set high-availability vrrp group vlan10 hello-source-address '172.31.10.252'
+set high-availability vrrp group vlan10 interface 'bond0.10'
+set high-availability vrrp group vlan10 no-preempt
+set high-availability vrrp group vlan10 peer-address '172.31.10.253'
+set high-availability vrrp group vlan10 priority '200'
+set high-availability vrrp group vlan10 vrid '10'
+set high-availability vrrp group vlan200 address 172.31.200.1/24
+set high-availability vrrp group vlan200 hello-source-address '172.31.200.252'
+set high-availability vrrp group vlan200 interface 'bond0.200'
+set high-availability vrrp group vlan200 no-preempt
+set high-availability vrrp group vlan200 peer-address '172.31.200.253'
+set high-availability vrrp group vlan200 priority '200'
+set high-availability vrrp group vlan200 vrid '200'
+set interfaces bonding bond0 address '172.31.1.252/24'
+set interfaces bonding bond0 description 'api-batch-test'
+set interfaces bonding bond0 hash-policy 'layer2+3'
+set interfaces bonding bond0 lacp-rate 'fast'
+set interfaces bonding bond0 member interface 'eth0'
+set interfaces bonding bond0 member interface 'eth1'
+set interfaces bonding bond0 mode '802.3ad'
+set interfaces bonding bond0 vif 2 address '172.31.2.252/24'
+set interfaces bonding bond0 vif 2 description 'k8s'
+set interfaces bonding bond0 vif 3 address '172.31.3.252/24'
+set interfaces bonding bond0 vif 3 description 'kvm'
+set interfaces bonding bond0 vif 9 address '172.31.9.252/24'
+set interfaces bonding bond0 vif 9 description 'private'
+set interfaces bonding bond0 vif 10 address '172.31.10.252/23'
+set interfaces bonding bond0 vif 10 description 'lot'
+set interfaces bonding bond0 vif 51 description 'WAN1 Vodafone-equivalent (sim ISP PPPoE)'
+set interfaces bonding bond0 vif 53 address 'dhcp'
+set interfaces bonding bond0 vif 53 description 'WAN3 10gig-equivalent (sim ISP DHCP)'
+set interfaces bonding bond0 vif 53 dhcp-options default-route-distance '210'
+set interfaces bonding bond0 vif 200 address '172.31.200.252/24'
+set interfaces bonding bond0 vif 200 description 'roomates'

labsim/vlan1-move-monitor.sh

@@ -0,0 +1,17 @@
+#!/bin/bash
+# Timestamped liveness log for the Management VLAN during the bond0 -> bond0.1 move.
+#
+# The question this answers is not "did it work" but "for how long was it not
+# working, and what held the VIP while it was not". Both are invisible after the
+# fact: VRRP reconverges and leaves no trace of who was master during the gap.
+#
+#   ./vlan1-move-monitor.sh > /tmp/move.log &
+# Columns: time  VIP-ping  R1-ping  R2-ping  VIP-mac
+VIP="${VIP:-172.31.1.1}"; R1="${R1:-172.31.1.252}"; R2="${R2:-172.31.1.253}"
+p() { ping -c1 -W1 -n "$1" >/dev/null 2>&1 && echo up || echo DOWN; }
+while :; do
+  mac="$(ip neigh show "$VIP" 2>/dev/null | awk '{for(i=1;i<=NF;i++) if($i=="lladdr") print $(i+1)}')"
+  printf '%s  vip=%-4s r1=%-4s r2=%-4s vipmac=%s\n' \
+    "$(date +%H:%M:%S)" "$(p "$VIP")" "$(p "$R1")" "$(p "$R2")" "${mac:-none}"
+  sleep 1
+done

labsim/vlans.conf

@@ -0,0 +1,43 @@
+# Lab network simulation — VLAN map.
+#
+# Mirrors the real UniFi topology (same VLAN IDs, same roles) but with
+# deliberately DIFFERENT IP ranges so nothing here can collide with, or be
+# confused for, production. The sim subnet always encodes the VLAN id:
+#
+#     172.31.<vlan-id>.0/24
+#
+# Per-subnet address plan (same shape on every VLAN):
+#   .1    gateway under test   (VyOS/router VM — not created by default)
+#   .2    host bridge          (how you SSH in from this workstation)
+#   .10   the micro VM for this VLAN
+#   .254  VRRP VIP             (reserved, mirrors production)
+#
+# Format: vlan_id:name:sim_subnet_prefix:real_subnet:[masklen]:[host_octet]
+#
+# masklen defaults to 24 and host_octet to 2. Both exist for VLAN 10, which is
+# the one VLAN that has to be a /23 here: every UniFi DHCP reservation lives in
+# LoT, and LoT spans 10.0.0.x AND 10.0.1.x, which a /24 cannot represent. With
+# /23 the mapping stays readable — 10.0.0.46 -> 172.31.10.46 and
+# 10.0.1.67 -> 172.31.11.67.
+#
+# LoT's host leg is .3 rather than .2 because 10.0.0.2 is a real reservation
+# (Hubitat) and would map straight onto the host's own address. .3 is free in
+# production and sits below the DHCP pool (which starts at .11), so it can
+# never be handed out.
+1:management:172.31.1:192.168.1.0/24
+2:k8s:172.31.2:192.168.8.0/23
+3:kvm:172.31.3:192.168.3.0/24
+9:private:172.31.9:10.0.9.0/23
+10:lot:172.31.10:10.0.0.0/23:23:3
+200:roomates:172.31.200:192.168.2.0/24
+
+# WAN transport VLANs, mirroring production. These exist so the sim can run a
+# fake ISP on each and the switch script's WAN health checks actually execute
+# instead of printing "this delta configures no WAN -- skipping". A cutover
+# attempt failed on the WAN with nothing having tested it, because the sim
+# modelled every LAN VLAN faithfully and omitted the WAN entirely.
+#
+# No host leg is wanted here (host_octet 0 means "skip"): the ISP VMs own these
+# segments, and a host address on a WAN transport VLAN would be a lie.
+51:wan1:172.31.51:vodafone-pppoe(VLAN 51):24:0
+53:wan3:172.31.53:10gig-dhcp(VLAN 53):24:0

migration/.gitignore

@@ -0,0 +1,4 @@
+# Raw UniFi export: contains WiFi passphrases (wlanconf) and controller auth
+# material (setting). The inventory is regenerable — never commit it.
+export/
+__pycache__/

migration/CUTOVER.md

@@ -0,0 +1,209 @@
+# Cutover runbook — USG to VyOS
+
+**Print this.** During the cutover there is no internet, so there is no
+assistant and no web search. Everything you need is on this page and on the
+boxes themselves.
+
+## Use these addresses. Not the other ones.
+
+| | use this | do NOT use |
+|---|---|---|
+| vyos001 (MASTER) | **`10.0.1.252`** | ~~192.168.8.143~~ |
+| vyos002 (BACKUP) | **`10.0.1.253`** | ~~192.168.8.144~~ |
+
+`ssh vyos@10.0.1.252` — by IP, not by name.
+
+**The `192.168.8.x` addresses stop working the instant the USG is unplugged.**
+That is not a maybe. Your workstation is on LoT (`10.0.0.210/23`) and reaching
+`192.168.8.x` requires routing *through the USG*:
+
+```
+ip route get 192.168.8.143  ->  via 10.0.0.1     <- the USG. Gone.
+ip route get 10.0.1.252     ->  dev lanbr0       <- same L2. Survives.
+```
+
+`10.0.1.252` and `.253` are on the LoT VLAN, the same broadcast domain as your
+workstation, so they need no gateway at all. They are the only remote path that
+survives the cutover.
+
+**Between unplugging the USG and finishing the switch there is no inter-VLAN
+routing.** In that window:
+
+- the **JetKVMs are unreachable** from your workstation (they are on Management
+  and kvm) — they are *not* a fallback during the gap
+- **Tailscale is down** with the internet
+- your workstation keeps `10.0.0.210` (86400s lease) and can still resolve via
+  `10.0.0.194`, which is also link-scope
+
+If LoT SSH fails, the next step is physical console, not the network.
+
+| | |
+|---|---|
+| JetKVMs (after routing is restored) | `192.168.1.28`, `192.168.1.29`, `192.168.3.6` |
+| Switch script | `/config/vyos-unifi-switch` on each box |
+| Peer link | `eth3` ↔ `eth3` direct cable, 2.5 GbE, `10.255.255.0/30` — conntrack state sync |
+| Login | user `vyos` |
+
+---
+
+## If something is wrong, do this
+
+```
+sudo /config/vyos-unifi-switch unifi
+```
+
+Then reconnect the USG. That command runs no health checks, asks nothing and
+cannot refuse. It restores a byte-exact copy of the configuration the box had
+before the cutover — verified by diff, not by assumption.
+
+**You do not have to be quick.** If you do nothing at all after
+`vyos-unifi-switch vyos`, the box reverts by itself within 10 minutes. Verified:
+config returns to the previous state and the box does **not** reboot
+(`uptime` and boot-id unchanged across an auto-revert).
+
+---
+
+## What has actually been tested
+
+Proven on the labsim router (same VyOS version, isolated OVS bridge with no
+physical NIC), by loading **vyos001's real running config** and applying the
+**real production delta**:
+
+- All 317 commands accepted, and the whole delta **commits** (`COMMIT OK`).
+- `unifi` mode restores the previous config **byte-exact** (diff clean).
+- Auto-revert fires when the commit is not confirmed: config returns to the
+  saved state and the box does **not** reboot — `uptime` and boot-id unchanged
+  across the revert.
+- Failed health checks trigger an immediate revert rather than waiting out the
+  timer.
+
+Two bugs were found this way and would each have failed the entire switch,
+since the delta commits as one unit: `bond0.51` did not exist for PPPoE to
+reference, and `translation port` rejects a port list.
+
+**Not tested, and untestable in advance:**
+
+- **PPPoE.** The line permits one session and the USG holds it. The first real
+  attempt is during the cutover.
+- **The commit on the real boxes.** The rehearsal ran with vyos001's `eth2` and
+  `eth3` stanzas stripped, because the sim VM has only two NICs. Those are
+  plain interface configs that already work on the real hardware, but they were
+  not part of what committed.
+
+## Before you unplug anything
+
+1. Tether your workstation to your phone if you want the assistant available.
+   Cutting the USG cuts your internet, not your LAN.
+2. On **both** boxes, confirm the machinery is present:
+   ```
+   sudo /config/vyos-unifi-switch status
+   ls -la /config/modes/          # unifi.boot + to-vyos.commands
+   ls -la /config/wan-secrets     # must be 0600
+   ```
+   `status` must report `mode: unifi`. If `unifi.boot` is missing, **stop** —
+   there is no way back without it.
+3. Confirm the revert action is `reload`, not `reboot`:
+   ```
+   show configuration commands | match commit-confirm
+   ```
+   Must show `action 'reload'`. Without it a failed switch **reboots** the
+   firewall instead of reverting it. The switch script refuses to run if this
+   is missing, but check anyway.
+
+## The cutover
+
+0. **Open both SSH sessions BEFORE you unplug anything**, and leave them open:
+   ```
+   ssh vyos@10.0.1.253      # vyos002, BACKUP
+   ssh vyos@10.0.1.252      # vyos001, MASTER
+   ```
+   If either will not connect, stop. Do not unplug the USG.
+
+1. **Physically disconnect the USG.** Not just powered off — disconnected. The
+   switch script refuses to run while anything still answers on a gateway
+   address, because two devices on `.1` is the worst available outcome.
+   You cannot switch first and unplug after, for exactly that reason.
+
+2. In the **vyos002 (BACKUP)** session, first:
+   ```
+   sudo /config/vyos-unifi-switch vyos
+   ```
+3. Watch the health checks. They cover PPPoE, the default route, kea, the DNS
+   forwarder and reachability. On failure the script reverts immediately and
+   tells you so.
+4. If vyos002 came up clean, repeat on **vyos001 (MASTER)**.
+5. Check a real client: does it get an address, and is it the *same* address as
+   before? Every active client has a reservation, so it should be.
+
+## What will probably go wrong first
+
+**The WAN.** There are two, and they behave differently:
+
+| | line | VLAN | transport | notes |
+|---|---|---|---|---|
+| WAN2 | 10 gig ISP | **53** | DHCP, public `87.192.101.48/21` | primary, distance 1 |
+| WAN1 | Vodafone | **51** | PPPoE, ~900/700 Mbit | failover, distance 10 |
+
+VyOS clones the USG's WAN2 MAC (`f0:9f:c2:12:9b:4f`) on `bond0.53`, which is how
+it keeps the existing public lease rather than asking for a new one.
+
+**Both boxes carry the identical WAN and NAT config.** vyos002's WAN interfaces
+are simply held administratively down, so the cloned MAC is never live on two
+boxes at once. To move the internet path to vyos002:
+
+```
+configure
+delete interfaces bonding bond0 vif 53 disable
+delete interfaces pppoe pppoe0 disable
+commit; save
+```
+
+Two lines. Do it only when vyos001 is genuinely down or disconnected — two boxes
+holding that MAC at once is exactly what the disable prevents.
+
+PPPoE is no longer an unknown: it was proven on the USG before cutover
+(`pppoe0` came up with `90.241.226.213`, MTU 1492). What remains untested is
+VyOS dialling it, and whether the ISP hands the same lease to the cloned MAC.
+
+If the WAN check fails:
+
+```
+show interfaces pppoe pppoe0
+sudo journalctl -u ppp@pppoe0 -n 50 --no-pager
+```
+
+Check the credential in `/config/wan-secrets` and that VLAN 51 actually reaches
+the box. If it will not come up, run `vyos-unifi-switch unifi`, reconnect the
+USG, and debug with the internet back on.
+
+## Things that are true and easy to forget
+
+- **WiFi keeps working, but through VyOS.** The SSIDs stay in UniFi and the APs
+  are untouched, but 37 of 83 active clients are wireless and every one is on
+  LoT — they get their addresses from VyOS now.
+- **DHCP leases last 24h (86400s).** A device that does not renew promptly keeps
+  its old address for a while. That is fine, not a symptom.
+- **The firewalls resolve via `8.8.8.8` / `8.8.4.4`** — matching the DNS the USG
+  used on its WAN. This means their own name resolution now depends on the
+  *internet* being up, so between unplugging the USG and PPPoE establishing,
+  the boxes have no DNS at all. That is expected and harmless: they only need
+  DNS for NTP hostnames, and the switch's own health checks use it precisely to
+  prove the WAN came up. Nothing in the switch itself resolves a name.
+- Internal `ad.itaz.eu` names still resolve through Google, because that zone is
+  published publicly with private addresses in it (`nas001` → `10.0.0.194`,
+  `kvm-macstudio1` → `192.168.3.8`). Convenient here; worth knowing it is public.
+- **The USG was a DNS resolver** for every VLAN except LoT. VyOS now runs
+  `dns forwarding` in its place. If names stop resolving but IPs still work,
+  that is where to look.
+- **`eth2` and `bond0.2` are both in `192.168.8.0/23`.** It works, but if you
+  see odd source-address behaviour on the management NIC, that is why.
+
+## Afterwards
+
+Once it has been stable for a day:
+
+- Re-run `migration/unifi-export.py` — the UniFi controller is no longer the
+  source of truth for DHCP, and the export will drift.
+- The VPN rules (ESP, UDP 500/4500) are carried over but the VPN itself still
+  terminated on the USG. Decide whether it moves.
+- `labsim` still holds a deliberate `kvm→k8s` drop rule from earlier testing.

migration/MANAGEMENT-VLAN-TAGGED.md

@@ -0,0 +1,137 @@
+# Moving Management onto a tagged VLAN
+
+Rehearsed end to end in labsim on 2026-09-02. This is the fix for kea serving
+addresses from the wrong VLAN's pool.
+
+## Why
+
+ISC Kea [#1117](https://gitlab.isc.org/isc-projects/kea/-/issues/1117): with
+`dhcp-socket-type: raw`, a frame tagged for a sub-interface is **also** delivered
+to the parent's `AF_PACKET` socket. If the parent serves a subnet, kea answers
+from it too. Ours does — Management is the native/untagged VLAN on `bond0` while
+VLANs 2/3/9/10/200 are sub-interfaces of that same bond — so one DISCOVER on
+VLAN 3 produces two OFFERs and the *client* decides which to keep:
+
+```
+bond0.3 :  192.168.3.14  correct
+bond0   :  192.168.1.28  UNTAGGED, Management pool, wrong
+```
+
+The fix is to leave **no subnet on the parent**: every VLAN tagged, Management
+included, moved from `bond0` to `bond0.1`.
+
+Confirmed in labsim across all six LAN VLANs: fails before, passes after.
+`labsim/labsim-vlan-leak-test.sh` is the test; evidence in
+`labsim/vlan-leak-evidence/`.
+
+## What must change together
+
+Per router:
+
+| | from | to |
+|---|---|---|
+| address | `interfaces bonding bond0 address` | `interfaces bonding bond0 vif 1 address` |
+| firewall | `interface-group LAN interface bond0` | `... interface bond0.1` |
+| VRRP | `vrrp group native interface bond0` | `... interface bond0.1` |
+| kea | — | **restart it** (see traps) |
+
+On the switch: Native VLAN = **None** on the trunk to that firewall, with VLAN 1
+added to the tagged set.
+
+## The ordering constraint
+
+**There is no overlap state.** An 802.1Q port always egresses its native VLAN
+untagged, so while VLAN 1 is native the router can *send* tagged VLAN 1 but can
+never *receive* it. Verified: a tagged VLAN 1 ARP sent from the switch arrived on
+`bond0` untagged and never on `bond0.1`. Configuring "native VLAN 1 **and** VLAN 1
+tagged" as a make-before-break does not work; the switch and router changes for a
+given firewall are strictly simultaneous, and that router loses Management in
+between.
+
+What makes this safe anyway: **tagged and untagged Management coexist on the
+same VLAN.** One VLAN is one broadcast domain no matter how each port tags it, so
+the firewalls can be converted one at a time — verified with the primary untagged
+and the secondary already tagged, both reachable, VIP up, VLAN 1 clients fine.
+
+Access ports are untouched throughout. The UniFi controller at 192.168.1.5 and
+your workstation are on access ports and never traverse the firewall trunks, so
+you keep the controller you are making the change from. Only the router being
+converted goes dark, and only until its own config lands.
+
+## Procedure
+
+Do the **backup** router first, then fail the VIPs over and do the other. You
+need console (JetKVM) on the router being converted — its Management SSH dies the
+moment the switch port changes.
+
+For each router in turn:
+
+1. Confirm the *other* router is MASTER and healthy:
+   `show vrrp` and `sudo /config/vrrp-wan-health; echo $?` (must be 0).
+2. Start the monitor from a workstation on an access port:
+   `labsim/vlan1-move-monitor.sh` (edit the three addresses for production).
+3. UniFi: on this firewall's trunk ports, Native VLAN → None, VLAN 1 → tagged.
+   This router's Management drops now.
+4. Over the console, in one commit:
+   ```
+   set interfaces bonding bond0 vif 1 address '192.168.1.252/24'   # .253 on vyos002
+   set interfaces bonding bond0 vif 1 description 'management'
+   delete interfaces bonding bond0 address
+   set firewall group interface-group LAN interface 'bond0.1'
+   delete firewall group interface-group LAN interface 'bond0'
+   set high-availability vrrp group native interface 'bond0.1'
+   commit
+   save
+   ```
+5. `sudo systemctl restart isc-kea-dhcp4-server` — see traps.
+6. Verify: Management SSH back, `show vrrp` shows `native` on `bond0.1`, and the
+   leak test passes.
+
+Then fail back if the VIPs moved (below), and repeat for the other router.
+
+### Measured windows (labsim)
+
+| | |
+|---|---|
+| this router's own Management unreachable | ~27 s (the console apply) |
+| VIP `.1` unreachable, peer already converted | **0 s** |
+| VIP `.1` unreachable, converting the current MASTER | ~6 s (VRRP failover) |
+| VIP unreachable if you convert both routers before the switch | **5 min 30 s** |
+
+That last row is the failure mode to avoid: with both routers untagged and the
+trunks already changed, the VIP is a black hole and **the healthy BACKUP does not
+take over**. Its `native` group stays BACKUP because the *other* VLANs still hear
+the master, and the sync group holds them together. Redundancy does not help you
+here; only ordering does.
+
+## Traps
+
+- **Restart kea.** VyOS does not restart it for an interface address change, so
+  it keeps a raw socket bound with the old address and keeps emitting the wrong
+  offers. The first post-fix test in the sim failed for this reason alone and
+  looked exactly like the fix not working.
+- **`interface-group LAN`.** Moving the address without moving the group means
+  Management falls outside the group, and with default-deny that is every
+  management session and all VLAN 1 inter-VLAN routing, gone on commit — on a
+  router you reach through itself. Use `commit-confirm` if you are not on console.
+- **The VIPs may move, and `no-preempt` keeps them moved.** Converting a router
+  restarts keepalived and re-initialises *every* group, not just `native`. In one
+  rehearsal the priority-100 secondary took all six VIPs and held them while the
+  priority-200 primary sat at BACKUP; in another the restart was quick enough that
+  nothing moved. It is non-deterministic — check afterwards, every time.
+  Fail back with `restart vrrp` **on the router currently holding them**.
+- **Duplicate delivery does not stop**, and should not be read as failure. #1117
+  only promises there is no longer a subnet on the parent to match. Expect two
+  identical replies per DISCOVER, both from the correct pool.
+- **Both firewalls' trunks must end up the same.** If UniFi shares one port
+  profile between them, changing it converts both at once and you get the 5m30s
+  row above. Check before you start; use per-port overrides if it does.
+
+## Not covered by the rehearsal
+
+- Whether UniFi's port profile can express "no native VLAN" the way OVS can, and
+  whether the two firewalls share a profile. Unverified — check on the controller.
+- Why the JetKVM consoles specifically accepted the wrong OFFER when a VLAN 3
+  access port should not receive an untagged VLAN 1 frame at all. Their port
+  profile likely passes VLAN 1 untagged. Worth confirming, though it does not
+  change the fix.

migration/RECOVERY-CARD-vlan1-move.md

@@ -0,0 +1,194 @@
+# Recovery card — moving Management to tagged VLAN 1
+
+Print or keep open. **During this change there is no internet, so no Claude.**
+Everything you need is on this page.
+
+---
+
+## The one thing that matters
+
+```
+ssh vyos@10.0.1.252
+```
+
+Your workstation is `10.0.0.210/23`; vyos001's LoT leg is `10.0.1.252/23`. Same
+subnet, same VLAN, **direct L2** — verified: `ip route get` returns
+`dev lanbr0 src 10.0.0.210` with no `via`, MAC `64:62:66:25:96:45`.
+
+It therefore does **not** depend on: the Management VLAN, VRRP, the VIPs,
+inter-VLAN routing, DNS, or the switch trunk config. If the router is up and its
+bond has link, this works. `bond0.10` is untouched by the change and stays in the
+firewall `LAN` group throughout.
+
+vyos002, once it is up, is `10.0.1.253` the same way.
+
+Other legs that also survive: `192.168.3.4` (kvm), `192.168.2.252` (Roomates).
+
+---
+
+## Before you touch anything
+
+```
+ssh vyos@10.0.1.252
+sudo /config/vyos-known-good save
+```
+
+The existing snapshot is from **2026-08-24** and predates today's fixes
+(eth2 removal, VRRP health-check) — restoring that one would undo them. Take a
+fresh one first. Check with `sudo /config/vyos-known-good status`.
+
+---
+
+## Order: switch FIRST, router SECOND
+
+This matters and is easy to get backwards.
+
+The UniFi controller is `192.168.1.5`, on the **Management** VLAN. Your
+workstation is on LoT and reaches it *through vyos001*. The moment the router
+has Management on `bond0.1` while the switch is still sending it untagged, that
+routing is dead — **and you lose the controller**, which is the thing you still
+need in order to change the switch.
+
+So:
+
+1. **UniFi first**, while everything still works:
+   USW Aggregation → port 1 `firewall001` (LAG, members 1+2) →
+   Native VLAN: Management → **None**, and make sure VLAN 1 is tagged/allowed.
+   *vyos001 loses Management the instant this lands. That is expected.*
+   Do **not** touch port 3 `firewall002` — that is vyos002, and it is down.
+2. **Router second**, over `ssh vyos@10.0.1.252` (still works — L2 direct).
+
+If UniFi will not offer "no native VLAN", stop and read *"If UniFi cannot do it"*
+below rather than improvising.
+
+---
+
+## The router change
+
+```
+ssh vyos@10.0.1.252
+configure
+set interfaces bonding bond0 vif 1 address '192.168.1.252/24'
+set interfaces bonding bond0 vif 1 description 'management'
+delete interfaces bonding bond0 address
+set firewall group interface-group LAN interface 'bond0.1'
+delete firewall group interface-group LAN interface 'bond0'
+set high-availability vrrp group native interface 'bond0.1'
+commit-confirm 10
+save
+exit
+```
+
+**Use `commit-confirm 10`, not `commit`.** If it goes wrong and you cannot get
+back in, the router reverts itself after 10 minutes and comes back on its own.
+That is your safety net with no internet and no help.
+
+Once you have confirmed it works (below), run:
+
+```
+configure
+confirm
+save
+exit
+```
+
+`save` after `confirm`, or a reboot loses it.
+
+### Then, and this is the step that gets forgotten
+
+```
+sudo systemctl restart isc-kea-dhcp4-server
+```
+
+VyOS does **not** restart kea for an interface address change. Without this it
+keeps a raw socket bound to the old address and keeps handing out wrong-VLAN
+addresses — the fix looks like it did nothing. Give it ~60s before judging;
+kea reopens sockets on a retry loop and answers nothing for a while after a
+restart (measured: still silent at 55s in the sim, then fine).
+
+Also check DNS came back, since the forwarder binds the VIP `192.168.1.1`:
+
+```
+sudo systemctl status pdns-recursor --no-pager | head -3
+dig @192.168.1.1 google.com +short
+```
+
+---
+
+## Verify
+
+```
+ssh vyos@192.168.1.252          # Management back, now tagged
+show vrrp                        # native should be on bond0.1
+show dhcp server leases | head
+```
+
+Then from a machine on VLAN 3, force a DHCP renew and confirm it gets a
+`192.168.3.x` address and not a `192.168.1.x` one.
+
+---
+
+## If you are locked out
+
+In order:
+
+1. **Wait 10 minutes.** `commit-confirm` reverts by itself. This is the answer
+   most of the time. Do not power-cycle during this — you will lose the revert.
+2. `ssh vyos@10.0.1.252` — the LoT leg. Then `configure` / `rollback 1` / `commit`.
+3. Other legs: `ssh vyos@192.168.3.4`, `ssh vyos@192.168.2.252`.
+4. `sudo /config/vyos-known-good restore` — back to the snapshot you took at the
+   start. It is itself commit-confirmed, so even this cannot strand you.
+5. Put the UniFi port back: Native VLAN → Management on USW Aggregation port 1.
+   That alone restores the old shape and Management comes back untagged.
+
+**Do not** power-cycle vyos001 as a first move. Everything above is faster and
+non-destructive, and a reboot loses an unsaved `commit-confirm` revert.
+
+---
+
+## Do NOT power on vyos002 yet
+
+It still has `interfaces ethernet eth2 address 192.168.8.144/23` on the box — the
+same subnet as `bond0.2`. That is what ARP-poisoned `192.168.8.1` and took the
+cluster down. It also has no `/config/vrrp-wan-health`, so it can take the
+floating IPs with no WAN.
+
+Its console (`kvm - vyos002`, US24 port 9) is currently **unreachable** — it sits
+on a VLAN 3 port holding a Management lease `192.168.1.28`, which is the very bug
+being fixed here. Fixing DHCP first is what gets that console back.
+
+---
+
+## If UniFi cannot do it
+
+Classic UniFi (this is a classic controller, 10.4.57) may not offer
+"Native VLAN = None" — every switch port has a PVID. Two things make this awkward
+here: Management is UniFi's *default* network with **no VLAN ID at all**
+(`vlan: null`), so there may be nothing to "tag VLAN 1" with.
+
+If so, **stop and change nothing.** The workaround is to point the trunk's native
+VLAN at a VLAN the router does not serve (so `bond0` still ends up with no
+subnet), which needs a throwaway VLAN-only network created first. That is a
+design decision, not something to improvise at 1am with no internet. Put the port
+back to Native = Management and everything returns to today's working state.
+
+---
+
+## Facts worth having on paper
+
+| | |
+|---|---|
+| vyos001 Management | `192.168.1.252` → becomes `bond0.1` |
+| vyos001 LoT (recovery) | `10.0.1.252`, L2-direct from your workstation |
+| vyos002 Management | `192.168.1.253` (down) |
+| VIP Management | `192.168.1.1` |
+| UniFi controller | `192.168.1.5` (on Management — you lose it mid-change) |
+| firewall001 trunk | USW Aggregation port 1, LAG members 1+2 |
+| firewall002 trunk | USW Aggregation port 3, LAG members 3+4 |
+| SSH user / pass | `vyos` / `vyos` |
+| vyos001 bond MAC | `64:62:66:25:96:45` |
+
+Measured in labsim: converting the router while the peer is already converted
+costs **0s** of VIP downtime; converting it while it holds the VIPs costs about
+**6s**. vyos002 is down, so vyos001 holds everything — expect the ~6s, and expect
+Management to be gone from the UniFi change until the router change lands.

migration/_unifi.py

@@ -0,0 +1,84 @@
+"""Shared UniFi API client for the migration tooling.
+
+The controller is a CLASSIC self-hosted UniFi Network app (server_version
+10.4.x), not UniFi OS: login is /api/login and data lives under
+/api/s/<site>/... . UniFi OS would use /api/auth/login + /proxy/network/api.
+Credentials come from the mcpctl server definition so they are not duplicated
+here.
+"""
+from __future__ import annotations
+
+import json, re, ssl, subprocess, urllib.request, http.cookiejar
+
+
+def client():
+    raw = subprocess.run(["mcpctl", "describe", "server", "unifi-network"],
+                         capture_output=True, text=True).stdout
+    m = re.search(r"UNIFI_TARGETS\s+(\[.*)", raw)
+    if not m:
+        raise SystemExit("could not read UNIFI_TARGETS from mcpctl")
+    blob = m.group(1).strip()
+    try:
+        targets = json.loads(blob)
+    except json.JSONDecodeError:
+        targets = json.loads(blob + "}" * (blob.count("{") - blob.count("}")))
+    t = targets[0]
+    base = t["base_url"].rstrip("/")
+    auth = t.get("auth", {})
+    site = t.get("default_site", "default")
+
+    ctx = ssl.create_default_context()
+    ctx.check_hostname = False
+    ctx.verify_mode = ssl.CERT_NONE
+    opener = urllib.request.build_opener(
+        urllib.request.HTTPCookieProcessor(http.cookiejar.CookieJar()),
+        urllib.request.HTTPSHandler(context=ctx))
+    req = urllib.request.Request(
+        f"{base}/api/login",
+        data=json.dumps({"username": auth.get("username"),
+                         "password": auth.get("password")}).encode(),
+        headers={"Content-Type": "application/json"})
+    opener.open(req, timeout=20).read()
+    return opener, base, site
+
+
+def get(opener, base, site, path):
+    """GET /api/s/<site>/<path>, returning the `data` list (never raising)."""
+    try:
+        body = opener.open(f"{base}/api/s/{site}/{path}", timeout=30).read()
+        return json.loads(body).get("data", [])
+    except Exception as exc:
+        return {"__error__": f"{type(exc).__name__}: {exc}"}
+
+
+def post(opener, base, site, path, payload):
+    """POST to /api/s/<site>/<path> -- used for device commands (cmd/devmgr)."""
+    req = urllib.request.Request(
+        f"{base}/api/s/{site}/{path}",
+        data=json.dumps(payload).encode(),
+        headers={"Content-Type": "application/json"}, method="POST")
+    try:
+        return json.loads(opener.open(req, timeout=30).read()).get("data", [])
+    except urllib.error.HTTPError as exc:
+        return {"__error__": f"HTTP {exc.code}: {exc.read()[:300].decode(errors='replace')}"}
+    except Exception as exc:
+        return {"__error__": f"{type(exc).__name__}: {exc}"}
+
+
+def put(opener, base, site, path, payload):
+    """PUT to /api/s/<site>/<path>. Returns the `data` list or an __error__ dict.
+
+    Classic controllers accept the session cookie alone -- no CSRF token, which
+    UniFi OS would require. Errors are returned rather than raised so a caller
+    changing production config can report and stop rather than traceback.
+    """
+    req = urllib.request.Request(
+        f"{base}/api/s/{site}/{path}",
+        data=json.dumps(payload).encode(),
+        headers={"Content-Type": "application/json"}, method="PUT")
+    try:
+        return json.loads(opener.open(req, timeout=30).read()).get("data", [])
+    except urllib.error.HTTPError as exc:
+        return {"__error__": f"HTTP {exc.code}: {exc.read()[:300].decode(errors='replace')}"}
+    except Exception as exc:
+        return {"__error__": f"{type(exc).__name__}: {exc}"}

migration/he-tunnel-follow

@@ -0,0 +1,121 @@
+#!/bin/bash
+# Keep the Hurricane Electric 6in4 tunnel pointed at whichever WAN is live.
+#
+# The tunnel is anchored to a source IPv4. When failover moves the default route
+# from the 10 gig to PPPoE, 6in4 packets keep leaving with the old source, HE
+# drops them, and IPv6 goes dark while IPv4 keeps working -- a partial outage
+# that presents as "some sites are broken", which is far worse to diagnose than
+# a clean one.
+#
+# Changes are made at KERNEL level (`ip tunnel change`), not in VyOS config, on
+# purpose:
+#   - no commit per WAN flip, so a flapping line cannot churn the config;
+#   - no drift against the Pulumi model, so `vyos-verify` stays meaningful;
+#   - a reboot restores config.boot, which pins the 10 gig -- the correct
+#     default -- so the wrong state cannot survive a restart.
+#
+#   he-tunnel-follow status     what is live vs what should be (read-only)
+#   he-tunnel-follow run        reconcile, updating HE if the source changed
+#   he-tunnel-follow run --dry  say what it would do, change nothing
+#
+# Credentials in /config/he-secrets (0600), NOT in git:
+#   HE_USER=<tunnelbroker username>
+#   HE_UPDATE_KEY=<from the tunnel's Advanced tab -- replaces the account password>
+#   HE_TUNNEL_ID=<numeric tunnel id>
+set -uo pipefail
+
+TUNNEL="${TUNNEL:-tun0}"
+SECRETS="${SECRETS:-/config/he-secrets}"
+STATE="${STATE:-/run/he-tunnel-follow.state}"
+# 6in4 costs 20 bytes. The 10 gig path is 1500 -> 1480; PPPoE is 1492 -> 1472.
+# Getting this wrong is the classic "IPv6 works until something large" failure.
+declare -A WAN_MTU=( ["bond0.53"]=1480 ["pppoe0"]=1472 )
+# Require the same answer twice before acting. HE rate-limits updates, and a
+# flapping WAN would otherwise hammer the API exactly when it is needed most.
+HYSTERESIS="${HYSTERESIS:-2}"
+
+log() { logger -t he-tunnel-follow -- "$*"; printf '  %s\n' "$*"; }
+die() { logger -t he-tunnel-follow -p user.err -- "$*"; printf '  ERROR: %s\n' "$*" >&2; exit 1; }
+
+active_wan() { ip -4 route show default 2>/dev/null | awk '/^default/{for(i=1;i<=NF;i++) if($i=="dev") print $(i+1); exit}'; }
+addr_of()    { ip -4 -br addr show "$1" 2>/dev/null | awk '{print $3}' | cut -d/ -f1; }
+tunnel_src() { ip tunnel show "$TUNNEL" 2>/dev/null | sed -nE 's/.* local ([0-9.]+).*/\1/p'; }
+tunnel_mtu() { cat "/sys/class/net/$TUNNEL/mtu" 2>/dev/null; }
+
+# HE's dyndns-style endpoint. `myip` is passed EXPLICITLY rather than letting HE
+# infer it from the request source: mid-failover the request itself may egress
+# either line, and inferring would happily point the tunnel at the WAN we just
+# left.
+he_update() {
+  local ip="$1"
+  [ -r "$SECRETS" ] || die "no $SECRETS -- create it with HE_USER / HE_UPDATE_KEY / HE_TUNNEL_ID (0600)"
+  # shellcheck disable=SC1090
+  . "$SECRETS"
+  [ -n "${HE_USER:-}" ] && [ -n "${HE_UPDATE_KEY:-}" ] && [ -n "${HE_TUNNEL_ID:-}" ] \
+    || die "$SECRETS is missing HE_USER, HE_UPDATE_KEY or HE_TUNNEL_ID"
+
+  local out
+  out="$(curl -sS --max-time 25 \
+      --data-urlencode "username=$HE_USER" \
+      --data-urlencode "password=$HE_UPDATE_KEY" \
+      --data-urlencode "hostname=$HE_TUNNEL_ID" \
+      --data-urlencode "myip=$ip" \
+      "https://ipv4.tunnelbroker.net/nic/update" 2>&1)"
+  # dyndns protocol: "good <ip>" or "nochg <ip>" are both success.
+  case "$out" in
+    good*|nochg*) log "HE endpoint set to $ip ($out)"; return 0 ;;
+    *)            die "HE update refused: $out" ;;
+  esac
+}
+
+reconcile() {
+  local dry="${1:-}"
+  local wan src want_mtu cur_src cur_mtu
+  wan="$(active_wan)"; [ -n "$wan" ] || die "no default route; refusing to guess"
+  src="$(addr_of "$wan")"; [ -n "$src" ] || die "no IPv4 address on $wan"
+  want_mtu="${WAN_MTU[$wan]:-}"
+  [ -n "$want_mtu" ] || die "unknown WAN '$wan' -- add it to WAN_MTU rather than guessing an MTU"
+  cur_src="$(tunnel_src)"; cur_mtu="$(tunnel_mtu)"
+
+  if [ "$cur_src" = "$src" ] && [ "$cur_mtu" = "$want_mtu" ]; then
+    rm -f "$STATE"
+    log "in sync: $TUNNEL via $wan src $src mtu $cur_mtu"
+    return 0
+  fi
+
+  # Hysteresis: count consecutive runs agreeing on the same target.
+  local seen=0 last=""
+  [ -r "$STATE" ] && { read -r last seen < "$STATE"; }
+  if [ "$last" = "$src" ]; then seen=$((seen + 1)); else seen=1; fi
+  echo "$src $seen" > "$STATE"
+  if [ "$seen" -lt "$HYSTERESIS" ]; then
+    log "change seen ($cur_src -> $src) but waiting for stability ($seen/$HYSTERESIS)"
+    return 0
+  fi
+
+  if [ "$dry" = "--dry" ]; then
+    log "DRY RUN: would set HE endpoint to $src, then $TUNNEL local $src mtu $want_mtu"
+    return 0
+  fi
+
+  # HE first, then local. Either order costs a brief drop, but changing locally
+  # first guarantees HE discards our packets for the whole window.
+  he_update "$src" || return 1
+  sudo ip tunnel change "$TUNNEL" mode sit local "$src" || die "failed to set tunnel local address"
+  sudo ip link set "$TUNNEL" mtu "$want_mtu"            || die "failed to set tunnel MTU"
+  rm -f "$STATE"
+  log "moved $TUNNEL to $wan: src $cur_src -> $src, mtu $cur_mtu -> $want_mtu"
+}
+
+case "${1:-status}" in
+  status)
+    wan="$(active_wan)"
+    printf '  active WAN : %s\n' "${wan:-<none>}"
+    printf '  wan addr   : %s\n' "$(addr_of "${wan:-lo}")"
+    printf '  tunnel src : %s\n' "$(tunnel_src)"
+    printf '  tunnel mtu : %s (want %s)\n' "$(tunnel_mtu)" "${WAN_MTU[${wan:-}]:-?}"
+    [ -r "$SECRETS" ] && printf '  credentials: present\n' || printf '  credentials: MISSING (%s)\n' "$SECRETS"
+    ;;
+  run) reconcile "${2:-}" ;;
+  *)   die "usage: he-tunnel-follow {status|run [--dry]}" ;;
+esac

migration/unifi-export.py

@@ -0,0 +1,335 @@
+#!/usr/bin/env python3
+"""Export everything from the UniFi controller that the VyOS cutover must preserve.
+
+The point is that nothing quietly stops working after the switch. That means
+capturing not just the networks but every DHCP reservation, every port forward
+and every firewall rule — the things nobody remembers configuring until they
+break.
+
+Writes one JSON file per endpoint into ./export/ (raw, unmodified — the source
+of truth) plus inventory.json, a normalised view used by the VyOS generator.
+
+  ./unifi-export.py                 # export to ./export/
+  ./unifi-export.py --out /tmp/x    # elsewhere
+  ./unifi-export.py --summary       # print a human summary of what was found
+
+WARNING: the raw export contains secrets (wlanconf holds WiFi passphrases,
+setting holds RADIUS/auth material). ./export/ is gitignored — keep it that way.
+"""
+from __future__ import annotations
+
+import argparse
+import ipaddress
+import json
+import os
+import sys
+
+import _unifi
+
+# endpoint -> why it matters for the cutover
+ENDPOINTS = {
+    "rest/networkconf": "networks: VLANs, subnets, DHCP ranges, DNS, lease time",
+    "rest/user":        "known clients — this is where fixed DHCP reservations live",
+    "stat/sta":         "currently active clients and their live IPs",
+    "rest/firewallrule": "firewall rules",
+    "rest/firewallgroup": "address/port groups referenced by rules",
+    "rest/portforward": "port forwards (inbound NAT)",
+    "rest/routing":     "static routes",
+    "rest/dhcpoption":  "custom DHCP options",
+    "rest/wlanconf":    "wireless networks (VLAN bindings)",
+    "stat/device":      "switches/APs incl. per-port VLAN config",
+    "rest/setting":     "controller settings (incl. USG/gateway config)",
+    "rest/usergroup":   "bandwidth groups referenced by clients",
+    "rest/dynamicdns":  "dynamic DNS",
+}
+
+
+def build_inventory(raw: dict) -> dict:
+    """Normalise the parts a migration actually has to reproduce."""
+    nets_by_id = {n["_id"]: n for n in raw.get("rest/networkconf", []) if isinstance(n, dict)}
+
+    networks = []
+    for n in raw.get("rest/networkconf", []):
+        if not isinstance(n, dict):
+            continue
+        networks.append({
+            "id": n.get("_id"),
+            "name": n.get("name"),
+            "purpose": n.get("purpose"),
+            "vlan": n.get("vlan"),
+            "vlan_enabled": n.get("vlan_enabled"),
+            "subnet": n.get("ip_subnet"),
+            "domain_name": n.get("domain_name"),
+            "dhcp_enabled": n.get("dhcpd_enabled"),
+            "dhcp_start": n.get("dhcpd_start"),
+            "dhcp_stop": n.get("dhcpd_stop"),
+            "dhcp_lease": n.get("dhcpd_leasetime"),
+            "dhcp_dns": [n.get(f"dhcpd_dns_{i}") for i in (1, 2, 3, 4) if n.get(f"dhcpd_dns_{i}")],
+            "dhcp_gateway": n.get("dhcpd_gateway") or n.get("dhcpd_gateway_enabled"),
+            "dhcp_ntp": [n.get(f"dhcpd_ntp_{i}") for i in (1, 2) if n.get(f"dhcpd_ntp_{i}")],
+            "igmp_snooping": n.get("igmp_snooping"),
+            "enabled": n.get("enabled", True),
+        })
+
+    # ip_subnet is the GATEWAY address with a prefix ("10.0.0.1/23"), not the
+    # network address — so derive the real subnet before matching against it.
+    subnets = []
+    for n in networks:
+        if not n["subnet"]:
+            continue
+        try:
+            iface = ipaddress.ip_interface(n["subnet"])
+        except ValueError:
+            continue
+        subnets.append((iface.network, n))
+
+    def resolve_net(ip: str | None, network_id: str | None) -> dict:
+        """Which network does this reservation belong to?
+
+        Most reservations here (23 of 31 as of the first export) carry no
+        network_id at all — UniFi simply does not bind them. VyOS needs the
+        subnet to place a static-mapping, so fall back to containment.
+        """
+        if network_id and network_id in nets_by_id:
+            nid = nets_by_id[network_id]
+            return {"name": nid.get("name"), "vlan": nid.get("vlan"), "by": "network_id"}
+        if ip:
+            try:
+                addr = ipaddress.ip_address(ip)
+            except ValueError:
+                return {"name": None, "vlan": None, "by": "unresolved"}
+            for net, meta in subnets:
+                if addr in net:
+                    return {"name": meta["name"], "vlan": meta["vlan"], "by": "subnet"}
+        return {"name": None, "vlan": None, "by": "unresolved"}
+
+    # Fixed reservations: the single most important thing to carry over, and
+    # the easiest to lose — nobody has these written down anywhere else.
+    reservations = []
+    for u in raw.get("rest/user", []):
+        if not isinstance(u, dict) or not u.get("use_fixedip"):
+            continue
+        net = resolve_net(u.get("fixed_ip"), u.get("network_id"))
+        reservations.append({
+            "mac": (u.get("mac") or "").lower(),
+            "ip": u.get("fixed_ip"),
+            "name": u.get("name") or u.get("hostname") or "",
+            "hostname": u.get("hostname") or "",
+            "network_id": u.get("network_id"),
+            "network_name": net["name"],
+            "network_vlan": net["vlan"],
+            "resolved_by": net["by"],
+            "note": (u.get("note") or "").strip(),
+        })
+    reservations.sort(key=lambda r: tuple(int(p) for p in r["ip"].split(".")) if r["ip"] else (0,))
+
+    # Active leases without a reservation: these devices work today by luck of
+    # the lease database. After a DHCP server swap they get a NEW address.
+    reserved_macs = {r["mac"] for r in reservations}
+    dynamic = []
+    for c in raw.get("stat/sta", []):
+        if not isinstance(c, dict):
+            continue
+        mac = (c.get("mac") or "").lower()
+        if mac in reserved_macs or not c.get("ip"):
+            continue
+        dynamic.append({
+            "mac": mac,
+            "ip": c.get("ip"),
+            "name": c.get("name") or c.get("hostname") or "",
+            "network": c.get("network"),
+        })
+    dynamic.sort(key=lambda r: tuple(int(p) for p in r["ip"].split(".")) if r["ip"] else (0,))
+
+    port_forwards = [{
+        "name": p.get("name"), "enabled": p.get("enabled"),
+        "proto": p.get("proto"), "src": p.get("src"),
+        "dst_port": p.get("dst_port"), "fwd": p.get("fwd"),
+        "fwd_port": p.get("fwd_port"), "log": p.get("log"),
+    } for p in raw.get("rest/portforward", []) if isinstance(p, dict)]
+
+    firewall_rules = [{
+        "name": r.get("name"), "enabled": r.get("enabled"), "action": r.get("action"),
+        "ruleset": r.get("ruleset"), "rule_index": r.get("rule_index"),
+        "protocol": r.get("protocol"),
+        "src_address": r.get("src_address"), "dst_address": r.get("dst_address"),
+        "src_firewallgroup_ids": r.get("src_firewallgroup_ids"),
+        "dst_firewallgroup_ids": r.get("dst_firewallgroup_ids"),
+        "src_networkconf_id": r.get("src_networkconf_id"),
+        "dst_networkconf_id": r.get("dst_networkconf_id"),
+    } for r in raw.get("rest/firewallrule", []) if isinstance(r, dict)]
+
+    firewall_groups = [{
+        "id": g.get("_id"), "name": g.get("name"),
+        "type": g.get("group_type"), "members": g.get("group_members"),
+    } for g in raw.get("rest/firewallgroup", []) if isinstance(g, dict)]
+
+    static_routes = [{
+        "name": r.get("name"), "enabled": r.get("enabled"),
+        "network": r.get("static-route_network"),
+        "nexthop": r.get("static-route_nexthop"),
+        "distance": r.get("static-route_distance"),
+        "type": r.get("static-route_type"),
+    } for r in raw.get("rest/routing", []) if isinstance(r, dict)]
+
+    return {
+        "networks": networks,
+        "reservations": reservations,
+        "dynamic_clients": dynamic,
+        "port_forwards": port_forwards,
+        "firewall_rules": firewall_rules,
+        "firewall_groups": firewall_groups,
+        "static_routes": static_routes,
+        "warnings": find_warnings(networks, reservations),
+    }
+
+
+def find_warnings(networks: list, reservations: list) -> list:
+    """Things that are fine under UniFi but bite when rebuilt on VyOS."""
+    warns = []
+
+    for n in networks:
+        if not n["subnet"]:
+            continue
+        try:
+            iface = ipaddress.ip_interface(n["subnet"])
+        except ValueError:
+            warns.append({"kind": "bad_subnet", "network": n["name"], "detail": n["subnet"]})
+            continue
+        # UniFi stores the gateway in ip_subnet. A gateway equal to the network
+        # address is legal in a /23 but plenty of tooling rejects it, so it must
+        # not be discovered during the cutover window.
+        if iface.ip == iface.network.network_address:
+            warns.append({
+                "kind": "gateway_is_network_address", "network": n["name"],
+                "detail": f"gateway {iface.ip} is the network address of {iface.network}",
+            })
+
+    # Reservations that sit inside the dynamic pool. UniFi's dhcpd tolerates
+    # this; whether VyOS does depends on its DHCP backend, so every one of these
+    # is a config that must be proven on the sim before cutover.
+    ranges = []
+    for n in networks:
+        if n["dhcp_enabled"] and n["dhcp_start"] and n["dhcp_stop"]:
+            try:
+                ranges.append((n["name"], ipaddress.ip_address(n["dhcp_start"]),
+                               ipaddress.ip_address(n["dhcp_stop"])))
+            except ValueError:
+                pass
+    inside = []
+    for r in reservations:
+        if not r["ip"]:
+            continue
+        try:
+            addr = ipaddress.ip_address(r["ip"])
+        except ValueError:
+            continue
+        for name, lo, hi in ranges:
+            if lo <= addr <= hi:
+                inside.append(f"{r['ip']} ({r['name'] or r['mac']}) in {name} pool")
+                break
+    if inside:
+        warns.append({"kind": "reservation_inside_dhcp_pool",
+                      "count": len(inside), "detail": inside})
+
+    unresolved = [f"{r['ip']} {r['mac']} {r['name']}"
+                  for r in reservations if r["resolved_by"] == "unresolved"]
+    if unresolved:
+        warns.append({"kind": "reservation_matches_no_subnet",
+                      "count": len(unresolved), "detail": unresolved})
+    return warns
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--out", default=os.path.join(os.path.dirname(os.path.abspath(__file__)), "export"))
+    ap.add_argument("--summary", action="store_true")
+    args = ap.parse_args()
+
+    os.makedirs(args.out, exist_ok=True)
+    opener, base, site = _unifi.client()
+    print(f"controller {base}  site {site}")
+
+    raw: dict = {}
+    errors = []
+    for path, why in ENDPOINTS.items():
+        data = _unifi.get(opener, base, site, path)
+        if isinstance(data, dict) and "__error__" in data:
+            errors.append((path, data["__error__"]))
+            print(f"  {path:22} FAILED  {data['__error__'][:50]}")
+            continue
+        raw[path] = data
+        fname = path.replace("/", "_") + ".json"
+        with open(os.path.join(args.out, fname), "w") as fh:
+            json.dump(data, fh, indent=2, sort_keys=True)
+        print(f"  {path:22} {len(data):4d}  -> {fname}")
+
+    inventory = build_inventory(raw)
+    with open(os.path.join(args.out, "inventory.json"), "w") as fh:
+        json.dump(inventory, fh, indent=2, sort_keys=True)
+
+    print(f"\nwrote {args.out}/inventory.json")
+    print(f"  networks           {len(inventory['networks'])}")
+    print(f"  DHCP reservations  {len(inventory['reservations'])}")
+    print(f"  dynamic clients    {len(inventory['dynamic_clients'])}  (no reservation — see summary)")
+    print(f"  port forwards      {len(inventory['port_forwards'])}")
+    print(f"  firewall rules     {len(inventory['firewall_rules'])}")
+    print(f"  static routes      {len(inventory['static_routes'])}")
+    if errors:
+        print(f"  endpoints failed   {len(errors)}: {[e[0] for e in errors]}")
+
+    if args.summary:
+        print_summary(inventory)
+    return 0
+
+
+def print_summary(inv: dict) -> None:
+    print("\n=== networks ===")
+    print(f"{'name':22} {'vlan':>5}  {'subnet':20} {'dhcp range':32} lease")
+    for n in sorted(inv["networks"], key=lambda x: (x["vlan"] or 0)):
+        rng = f"{n['dhcp_start']} - {n['dhcp_stop']}" if n["dhcp_enabled"] else "(dhcp off)"
+        print(f"{(n['name'] or '')[:22]:22} {str(n['vlan'] or '-'):>5}  "
+              f"{(n['subnet'] or '-'):20} {rng:32} {n['dhcp_lease'] or '-'}")
+
+    print(f"\n=== DHCP reservations ({len(inv['reservations'])}) ===")
+    for r in inv["reservations"]:
+        print(f"  {r['ip']:16} {r['mac']:18} {(r['network_name'] or '?')[:14]:14} {r['name'][:30]}")
+
+    if inv["port_forwards"]:
+        print(f"\n=== port forwards ({len(inv['port_forwards'])}) ===")
+        for p in inv["port_forwards"]:
+            state = "" if p["enabled"] else "  [DISABLED]"
+            print(f"  {p['proto']:6} {str(p['src']):16}:{str(p['dst_port']):11} -> "
+                  f"{p['fwd']}:{p['fwd_port']}   {p['name']}{state}")
+
+    if inv["firewall_rules"]:
+        print(f"\n=== firewall rules ({len(inv['firewall_rules'])}) ===")
+        for r in inv["firewall_rules"]:
+            state = "" if r["enabled"] else "  [DISABLED]"
+            print(f"  {str(r['ruleset']):22} {str(r['action']):8} {r['name']}{state}")
+
+    if inv["warnings"]:
+        print(f"\n=== {len(inv['warnings'])} things to settle before cutover ===")
+        for w in inv["warnings"]:
+            n = w.get("count")
+            print(f"  [{w['kind']}]" + (f"  x{n}" if n else ""))
+            det = w["detail"]
+            for line in (det if isinstance(det, list) else [det])[:6]:
+                print(f"      {line}")
+            if isinstance(det, list) and len(det) > 6:
+                print(f"      ... and {len(det) - 6} more (see inventory.json)")
+
+    n_dyn = len(inv["dynamic_clients"])
+    if n_dyn:
+        print(f"\n=== {n_dyn} active clients WITHOUT a reservation ===")
+        print("  These hold their address only via the current lease database. A DHCP")
+        print("  server swap hands them a different one — fine for phones, not fine for")
+        print("  anything another host reaches by IP. Review before cutover:")
+        for c in inv["dynamic_clients"][:40]:
+            print(f"  {c['ip']:16} {c['mac']:18} {(c['network'] or '')[:14]:14} {c['name'][:30]}")
+        if n_dyn > 40:
+            print(f"  ... and {n_dyn - 40} more (see inventory.json)")
+
+
+if __name__ == "__main__":
+    sys.exit(main())