labsim: add IPv6 BGP config layer for the Gateway-API public-v6 rehearsal

The sim's eBGP was IPv4-only; extend it to the v6 family so labsim can rehearse
Cilium BGP advertising a public-style v6 LoadBalancer /64 to the VyOS pair before
it touches production (approved plan: public IPv6 via Cilium BGP + Gateway API).

- sim-net-config.py: bgp6 inside bgp() -- ipv6-unicast peer-group K8S6 over
  fd00:2::11/12/13, prefix-list6 (le 128, /128 host routes), K8S-IN6/OUT6 route-maps
  (export deny -- never hand the cluster a default). ULA fd61:1e00::/64 as the sim
  LB range so nothing leaks into the real HE /48. All v6 lines pass the
  sim-net-apply.sh whitelist.
- sim-ha-config.py: VLANS6 = {2: fd00:2/64} -> routers hold fd00:2::252/253 on
  bond0.2 so they can peer the nodes over v6 (no v6 VRRP VIP; BGP peers the per-box
  address, as production).
- k8s-up.sh: nodes get fd00:2::11/12/13 + dual-stack k3s (dual node-ip, dual
  cluster/service CIDR). Peers line up with sim-net-config K8S_NODES_V6.

Config-gen verified for all three. Live chunk (Cilium v6+BGP+Gateway install,
bring-up, datapath/#26847 proof) is next.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH

.gitignore

@@ -31,3 +31,9 @@ node_modules/
 # Asahi build artifacts (large)
 bastion/.asahi-cache/
 bastion/asahi-repo/*.zip
+
+# Regenerated by labsim/dualstack-lab.sh; derived state, not source.
+labsim/dualstack-evidence/
+
+# Runtime snapshots from labsim/cilium-ipam-switch.sh
+labsim/.ipam-switch-state/

bastion/completions/labctl.bash

@@ -59,10 +59,10 @@ _labctl() {
       COMPREPLY=($(compgen -W "-h --help" -- "$cur"))
       return ;;
     "provision install")
-      COMPREPLY=($(compgen -W "--role --os --disk -h --help" -- "$cur"))
+      COMPREPLY=($(compgen -W "--role --os --disk --vyos-mgmt --vyos-mgmt-address --vyos-bond --vyos-bond-address --vyos-bond-vrrp --vlan-vip --vyos-vrrp-priority --vyos-mgmt-vlan --vlan --vyos-password --vyos-hwid --vyos-fresh-config -h --help" -- "$cur"))
       return ;;
     "provision reprovision")
-      COMPREPLY=($(compgen -W "--role --os --disk -h --help" -- "$cur"))
+      COMPREPLY=($(compgen -W "--role --os --disk --user -h --help" -- "$cur"))
       return ;;
     "provision debug")
       COMPREPLY=($(compgen -W "--pxe-boot -h --help" -- "$cur"))

bastion/completions/labctl.fish

@@ -132,13 +132,26 @@ complete -c labctl -n "__labctl_using_cmd provision" -a recheck -d 'Refresh hard
 
 # provision install options
 complete -c labctl -n "__labctl_in_cmd provision install" -l role -d 'Machine role (see below)' -xa 'vanilla worker infra labcontroller'
-complete -c labctl -n "__labctl_in_cmd provision install" -l os -d 'Operating system' -xa 'fedora-43 ubuntu-26.04'
+complete -c labctl -n "__labctl_in_cmd provision install" -l os -d 'Operating system' -xa 'fedora-43 ubuntu-26.04 vyos-rolling'
 complete -c labctl -n "__labctl_in_cmd provision install" -l disk -d 'Target disk device (auto-detect if omitted)' -x
+complete -c labctl -n "__labctl_in_cmd provision install" -l vyos-mgmt -d 'VyOS: untagged interface the machine PXE boots from (default eth0)' -x
+complete -c labctl -n "__labctl_in_cmd provision install" -l vyos-mgmt-address -d 'VyOS: CIDR for the management interface, or \'dhcp\' (default dhcp)' -x
+complete -c labctl -n "__labctl_in_cmd provision install" -l vyos-bond -d 'VyOS: comma-separated LACP bond members (must exclude the PXE NIC)' -x
+complete -c labctl -n "__labctl_in_cmd provision install" -l vyos-bond-address -d 'VyOS: address on the untagged bond (trunk native VLAN)' -x
+complete -c labctl -n "__labctl_in_cmd provision install" -l vyos-bond-vrrp -d 'VyOS: VRRP VIP floated on the untagged bond' -x
+complete -c labctl -n "__labctl_in_cmd provision install" -l vlan-vip -d 'VyOS: VRRP VIP for a --vlan entry (repeatable)' -x
+complete -c labctl -n "__labctl_in_cmd provision install" -l vyos-vrrp-priority -d 'VyOS: VRRP priority for all groups on this box (higher = master)' -x
+complete -c labctl -n "__labctl_in_cmd provision install" -l vyos-mgmt-vlan -d 'VyOS: tagged management VLAN on the PXE port' -x
+complete -c labctl -n "__labctl_in_cmd provision install" -l vlan -d 'VyOS: tagged VLAN sub-interface on the bond (repeatable)' -x
+complete -c labctl -n "__labctl_in_cmd provision install" -l vyos-password -d 'VyOS: password for the \'vyos\' user' -x
+complete -c labctl -n "__labctl_in_cmd provision install" -l vyos-hwid -d 'VyOS: pin an interface name to a MAC via hw-id (repeatable)' -x
+complete -c labctl -n "__labctl_in_cmd provision install" -l vyos-fresh-config -d 'VyOS: on reinstall, overwrite the preserved config with the generated one'
 
 # provision reprovision options
 complete -c labctl -n "__labctl_in_cmd provision reprovision" -l role -d 'Machine role (see below)' -xa 'vanilla worker infra labcontroller'
-complete -c labctl -n "__labctl_in_cmd provision reprovision" -l os -d 'Operating system' -xa 'fedora-43 ubuntu-26.04'
+complete -c labctl -n "__labctl_in_cmd provision reprovision" -l os -d 'Operating system' -xa 'fedora-43 ubuntu-26.04 vyos-rolling'
 complete -c labctl -n "__labctl_in_cmd provision reprovision" -l disk -d 'Target disk device (auto-detect if omitted)' -x
+complete -c labctl -n "__labctl_in_cmd provision reprovision" -l user -d 'SSH user for the reboot (default: vyos for VyOS machines, else current user)' -x
 
 # provision debug options
 complete -c labctl -n "__labctl_in_cmd provision debug" -l pxe-boot -d 'Boot installed system via PXE (kernel+initrd from network, root from NVMe)'

bastion/docs/ARCHITECTURE.md

@@ -89,83 +89,6 @@ Side paths:
 
 ---
 
-## Multi-architecture PXE
-
-The bastion serves both `x86_64` and `aarch64` over the network. Nothing about this is
-operator-configured -- there is no `--arch` flag, by design.
-
-### How a client's architecture is decided
-
-1. **DHCP option 93** (Client System Architecture) picks the *bootloader*. dnsmasq matches
-   it and hands out a matching iPXE binary:
-
-   | Option 93 | Client | Served |
-   |---|---|---|
-   | `0` | x86 BIOS | `undionly.kpxe` (TFTP) |
-   | `7`, `9` | x64 UEFI | `ipxe.efi` (TFTP) |
-   | `11` | **ARM64 UEFI** | `ipxe-arm64.efi` (TFTP) |
-   | `16` | x64 UEFI HTTP Boot | `http://…/ipxe.efi` |
-   | `19` | **ARM64 UEFI HTTP Boot** | `http://…/ipxe-arm64.efi` |
-
-   Values come from the IANA Processor Architecture Types registry. Note `19`, not `20` --
-   `20` is *pc/at bios boot from http*. EDK2/AAVMF prefers HTTP Boot over TFTP PXE, so the
-   iPXE binaries are staged in **both** `tftpDir` and `httpDir` (symlinked by `main.ts`).
-
-2. **`/dispatch` picks the kernel.** Option 93 never reaches the HTTP endpoint, so
-   `boot.ipxe` passes iPXE's own `${buildarch}` as `?arch=`. `resolveArch()` prefers, in
-   order: the tracked machine record → the reported `?arch=` → the configured default.
-   The record wins because it is what we observed on the machine itself.
-
-### Artifact naming
-
-`x86_64` keeps the original unsuffixed paths so its rendered iPXE scripts are unchanged;
-everything else is suffixed. `kernelPath()` / `initrdPath()` in `templates/boot.ipxe.ts`
-are the single source of truth, used by both the templates and `main.ts` staging.
-
-| arch | kernel | initrd |
-|---|---|---|
-| `x86_64` | `/vmlinuz` | `/initrd.img` |
-| `aarch64` | `/vmlinuz-aarch64` | `/initrd-aarch64.img` |
-
-`tests/ipxe-x86-regression.test.ts` pins the x86_64 output against a golden fixture.
-
-### arm64 gotchas
-
-- **LoadFile2 is mandatory.** arm64 has no `HdrS` boot protocol; the kernel's EFI stub
-  fetches the initrd over the UEFI `EFI_LOAD_FILE2_PROTOCOL`. An iPXE build without it
-  accepts the `initrd` line, silently drops it, and the kernel panics with
-  `VFS: Unable to mount root fs on unknown-block(0,0)`. Fedora's
-  `ipxe-bootimgs-aarch64` implements it; the integration test asserts this up front so
-  the failure names itself instead of looking like a disk problem.
-- **`nomodeset` is x86-only.** On arm64 there is no VGA path to fall back to. aarch64 gets
-  `console=tty0 console=ttyAMA0,115200` instead — the last `console=` wins for
-  `/dev/console`, so serial is the interactive one.
-- **Ubuntu is x86_64-only.** `releases.ubuntu.com` publishes no arm64 netboot artifacts.
-  `osSupportsArch()` encodes this, and both the install guard and `/dispatch` refuse the
-  combination rather than serving an x86 kernel to an ARM machine.
-
----
-
-## Onboarding classification (vendor OS)
-
-Machines carry an `onboard` field: `"pxe"` (default) or `"ssh"`, plus `vendor_os` naming
-what they run. `classifyOnboard()` in `@lab/shared` sets it from DMI identity, with known
-hardware also matched by MAC — a machine can sit in state for a long time with no DMI, and
-a DMI-only rule would fail open exactly where it matters.
-
-`onboard: "ssh"` means *we cannot rebuild this machine's OS*. Installs are refused at both
-entry points (`/api/install` and the labd `command-install` handler) with an error naming
-the machine and pointing at `provision debug`. **Rescue is never guarded** — being unable
-to reinstall a machine is precisely when a rescue shell is needed.
-
-This is a fact about the machine, not a blocklist. The refusal follows from "no image in
-our pipeline restores `vendor_os`", so adding a DGX OS image to the pipeline is what
-unblocks the DGX Sparks — no entry needs deleting.
-
-Current classifications: NVIDIA DGX Spark (`spark-2935`, `spark-3a1c`) → `dgx-os`.
-
----
-
 ## Packages
 
 ### Monorepo Structure
@@ -481,28 +404,6 @@ Hardcoded `/dev/sda` default broke NVMe-only machines. Fix: default to empty str
 ### Anaconda Rescue Mode Limitations
 `%pre` and `%post` sections do not execute in `inst.rescue` mode. SSH in rescue mode is provided by Anaconda's `inst.sshd` kernel parameter + `sshpw` kickstart directive. Manual setup via `curl bastion:8080/debug-setup.sh | bash` for nc listener.
 
-**Unresolved (2026-08-11): rescue SSH has never been observed working.** Adding the first
-integration coverage for `provision debug` (`tests/integration/pxe-rescue.test.ts`) showed the
-rescue environment coming up correctly — the bastion serves the kernel and initrd, Anaconda
-boots, fetches `debug.ks`, and reaches its installer environment — but **nothing ever listens on
-port 22**. Confirmed on aarch64 by probing the port for 30 minutes while the Anaconda environment
-was demonstrably running (NetworkManager, polkitd, rsyslog all up), and reproduced on x86_64 with
-KVM, so it is not architecture-specific and not an emulation artefact.
-
-This is orthogonal to the multi-architecture work: the same failure occurs on x86_64, which that
-work does not touch. Leads worth checking, in order:
-- Does `inst.sshd` actually start `sshd` in `inst.rescue` mode, or only in install mode? The
-  port never opens, so this is the prime suspect — an auth problem would still show an open port.
-- `sshkey` may apply only to the *installed* system, leaving the installer environment
-  password-only via `sshpw`. That would matter once sshd does listen: the test authenticates
-  key-only (`BatchMode=yes`).
-- The `%anaconda`-context directives in `debug.ks` may be skipped entirely when a kickstart is
-  supplied alongside `inst.rescue`.
-
-Until this is resolved, `provision debug` gets you a booted rescue environment on the console
-(including on arm64), but not an SSH shell. The `debug-setup.sh` nc-listener path is the
-documented workaround and is unaffected.
-
 ---
 
 ## Planned Work (Taskmaster)

bastion/package.json

@@ -21,14 +21,10 @@
     "test:integration:pxe:host": "sudo -E $(which npx) vitest run -c tests/integration/vitest.config.ts -t 'PXE boot'",
     "test:integration:iso": "vitest run -c tests/integration/vitest.config.ts -t 'ISO boot'",
     "test:integration:iso:host": "sudo -E $(which npx) vitest run -c tests/integration/vitest.config.ts -t 'ISO boot'",
+    "test:integration:vyos": "vitest run -c tests/integration/vitest.config.ts -t 'VyOS provisioning'",
+    "test:integration:vyos:host": "sudo -E $(which npx) vitest run -c tests/integration/vitest.config.ts -t 'VyOS provisioning'",
     "test:integration:arm-iso": "vitest run -c tests/integration/vitest.config.ts -t 'ARM ISO'",
     "test:integration:arm-iso:host": "sudo -E $(which npx) vitest run -c tests/integration/vitest.config.ts -t 'ARM ISO'",
-    "test:integration:rescue": "vitest run -c tests/integration/vitest.config.ts -t 'x86 rescue boot'",
-    "test:integration:rescue:host": "sudo -E $(which npx) vitest run -c tests/integration/vitest.config.ts -t 'x86 rescue boot'",
-    "test:integration:arm-pxe": "vitest run -c tests/integration/vitest.config.ts -t 'ARM PXE rescue'",
-    "test:integration:arm-pxe:host": "sudo -E $(which npx) vitest run -c tests/integration/vitest.config.ts -t 'ARM PXE rescue'",
-    "test:integration:arm-pxe-full": "ARM_PXE_FULL=1 vitest run -c tests/integration/vitest.config.ts -t 'ARM PXE'",
-    "test:integration:arm-pxe-full:host": "sudo -E ARM_PXE_FULL=1 $(which npx) vitest run -c tests/integration/vitest.config.ts -t 'ARM PXE'",
     "test:integration:asahi": "vitest run -c tests/integration/vitest.config.ts -t 'asahi firstboot'",
     "test:integration:asahi:host": "sudo -E $(which npx) vitest run -c tests/integration/vitest.config.ts -t 'asahi firstboot'",
     "test:integration:asahi-validate": "vitest run -c tests/integration/vitest.config.ts -t 'asahi.*validation'",

bastion/scripts/test-provision.sh

@@ -2,19 +2,16 @@
 # Run PXE and/or ISO boot integration tests.
 #
 # Usage:
-#   sudo ./scripts/test-provision.sh              # run PXE + ISO (x86_64)
-#   sudo ./scripts/test-provision.sh pxe          # PXE only
-#   sudo ./scripts/test-provision.sh iso          # ISO only (x86_64)
-#   sudo ./scripts/test-provision.sh rescue       # x86_64 Anaconda rescue boot + SSH (~15min)
-#   sudo ./scripts/test-provision.sh arm          # ARM ISO boot (emulated, SLOW ~60min)
-#   sudo ./scripts/test-provision.sh arm-pxe      # ARM network PXE rescue: NBP + rescue over SSH (~25-30min)
-#   sudo ./scripts/test-provision.sh arm-pxe-full # ARM network PXE incl. discover + full install (~75-95min)
-#   sudo ./scripts/test-provision.sh all          # all tests including ARM
+#   sudo ./scripts/test-provision.sh          # run PXE + ISO (x86_64)
+#   sudo ./scripts/test-provision.sh pxe      # PXE only
+#   sudo ./scripts/test-provision.sh iso      # ISO only (x86_64)
+#   sudo ./scripts/test-provision.sh arm      # ARM ISO boot (emulated, SLOW ~60min)
+#   sudo ./scripts/test-provision.sh all      # all tests including ARM
 #
 # Prerequisites:
 #   libvirtd, OVMF (edk2-ovmf), iPXE (ipxe-bootimgs-x86),
 #   dnsmasq, xorriso, mtools, virt-install, qemu-img
-#   ARM: qemu-system-aarch64, edk2-aarch64, ipxe-bootimgs-aarch64
+#   ARM: qemu-system-aarch64, edk2-aarch64
 set -e
 
 SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
@@ -61,10 +58,6 @@ if [ ! -f /usr/share/edk2/ovmf/OVMF_CODE.fd ]; then
     exit 1
 fi
 
-MODE="${1:-both}"
-
-# iPXE binaries are per-architecture. x86_64 is always required (the dnsmasq config
-# references it); arm64 only for the ARM network-PXE modes.
 IPXE_EFI=""
 for f in /usr/share/ipxe/ipxe-snponly-x86_64.efi /usr/share/ipxe/ipxe-snp-x86_64.efi /usr/share/ipxe/ipxe-x86_64.efi; do
     [ -f "$f" ] && IPXE_EFI="$f" && break
@@ -74,20 +67,6 @@ if [ -z "$IPXE_EFI" ]; then
     exit 1
 fi
 
-IPXE_EFI_ARM64=""
-for f in /usr/share/ipxe/arm64-efi/snponly.efi /usr/share/ipxe/arm64-efi/ipxe.efi; do
-    [ -f "$f" ] && IPXE_EFI_ARM64="$f" && break
-done
-
-case "$MODE" in
-    arm-pxe|arm-pxe-full|all)
-        if [ -z "$IPXE_EFI_ARM64" ] && [ "$MODE" != "all" ]; then
-            echo -e "${RED}arm64 iPXE binary not found.${RESET} Install: sudo dnf install ipxe-bootimgs-aarch64"
-            exit 1
-        fi
-        ;;
-esac
-
 # Find SSH key
 SSH_KEY=""
 for name in id_ed25519 id_ecdsa id_rsa; do
@@ -104,19 +83,10 @@ fi
 echo -e "  User:    ${BOLD}$REAL_USER${RESET}"
 echo -e "  SSH key: ${BOLD}$SSH_KEY${RESET}"
 echo -e "  iPXE:    ${BOLD}$IPXE_EFI${RESET}"
-echo -e "  iPXE a64:${BOLD} ${IPXE_EFI_ARM64:-not installed}${RESET}"
 echo ""
 
-require_arm_emulation() {
-    if ! command -v qemu-system-aarch64 &>/dev/null; then
-        echo -e "${RED}qemu-system-aarch64 not found.${RESET} Install: sudo dnf install qemu-system-aarch64 edk2-aarch64"
-        exit 1
-    fi
-    if [ ! -f /usr/share/edk2/aarch64/QEMU_EFI.fd ]; then
-        echo -e "${RED}AAVMF firmware not found.${RESET} Install: sudo dnf install edk2-aarch64"
-        exit 1
-    fi
-}
+# --- Determine which tests to run ---
+MODE="${1:-both}"
 
 run_test() {
     local name="$1" pattern="$2"
@@ -146,26 +116,13 @@ case "$MODE" in
         run_test "ISO boot" "ISO boot" || FAILED=1
         ;;
     arm|arm-iso)
-        require_arm_emulation
+        if ! command -v qemu-system-aarch64 &>/dev/null; then
+            echo -e "${RED}qemu-system-aarch64 not found.${RESET} Install: sudo dnf install qemu-system-aarch64 edk2-aarch64"
+            exit 1
+        fi
         echo -e "${YELLOW}ARM emulation is ~10x slower than native. Expect 30-60 minutes.${RESET}"
         run_test "ARM ISO boot" "ARM ISO" || FAILED=1
         ;;
-    rescue)
-        echo -e "${YELLOW}x86_64 rescue boot (KVM). Expect ~15 minutes.${RESET}"
-        run_test "x86 rescue boot" "x86 rescue boot" || FAILED=1
-        ;;
-    arm-pxe)
-        require_arm_emulation
-        echo -e "${YELLOW}ARM emulation is ~10x slower than native. Expect 25-30 minutes.${RESET}"
-        echo -e "${YELLOW}Covers option 93 -> arm64 NBP, arch resolution, and rescue over SSH.${RESET}"
-        echo -e "${YELLOW}For the full install too, use: $0 arm-pxe-full${RESET}"
-        run_test "ARM PXE rescue" "ARM PXE rescue" || FAILED=1
-        ;;
-    arm-pxe-full)
-        require_arm_emulation
-        echo -e "${YELLOW}ARM emulation is ~10x slower than native. Expect 75-95 minutes.${RESET}"
-        ARM_PXE_FULL=1 run_test "ARM PXE (rescue + install)" "ARM PXE" || FAILED=1
-        ;;
     both)
         run_test "PXE boot" "PXE boot" || FAILED=1
         run_test "ISO boot" "ISO boot" || FAILED=1
@@ -176,17 +133,12 @@ case "$MODE" in
         if command -v qemu-system-aarch64 &>/dev/null; then
             echo -e "${YELLOW}ARM emulation is ~10x slower than native.${RESET}"
             run_test "ARM ISO boot" "ARM ISO" || FAILED=1
-            if [ -n "$IPXE_EFI_ARM64" ]; then
-                run_test "ARM PXE rescue" "ARM PXE rescue" || FAILED=1
-            else
-                echo -e "${YELLOW}Skipping ARM PXE test (ipxe-bootimgs-aarch64 not installed)${RESET}"
-            fi
         else
-            echo -e "${YELLOW}Skipping ARM tests (qemu-system-aarch64 not installed)${RESET}"
+            echo -e "${YELLOW}Skipping ARM test (qemu-system-aarch64 not installed)${RESET}"
         fi
         ;;
     *)
-        echo "Usage: $0 [pxe|iso|rescue|arm|arm-pxe|arm-pxe-full|both|all]"
+        echo "Usage: $0 [pxe|iso|arm|both|all]"
         exit 1
         ;;
 esac

bastion/src/bastion/src/config.ts

@@ -20,6 +20,15 @@ export function loadConfig(overrides: Partial<BastionConfig> = {}): BastionConfi
   const ubuntuMirror = overrides.ubuntuMirror ?? process.env["UBUNTU_MIRROR"]
     ?? `https://releases.ubuntu.com/${ubuntuVersion}`;
 
+  // "latest" resolves the newest nightly ISO from the vyos-nightly-build GitHub
+  // releases at startup. downloads.vyos.io no longer serves direct rolling ISOs
+  // (it returns the vyos.io site, and nightly builds sit behind a signup form);
+  // GitHub releases are the remaining free, unauthenticated direct source.
+  // LTS ISOs are subscription-only. Set VYOS_ISO_URL to pin a specific build.
+  const vyosIsoUrl = overrides.vyosIsoUrl ?? process.env["VYOS_ISO_URL"] ?? "latest";
+  const vyosDefaultPassword = overrides.vyosDefaultPassword
+    ?? process.env["VYOS_DEFAULT_PASSWORD"] ?? "vyos";
+
   const fedoraMirror = `https://download.fedoraproject.org/pub/fedora/linux/releases/${fedoraVersion}/Everything/${arch}/os`;
   const tftpDir = `${bastionDir}/tftp`;
   const httpDir = `${bastionDir}/http`;
@@ -38,6 +47,8 @@ export function loadConfig(overrides: Partial<BastionConfig> = {}): BastionConfi
     dhcpRangeEnd,
     ubuntuVersion,
     ubuntuMirror,
+    vyosIsoUrl,
+    vyosDefaultPassword,
     // These are populated at runtime by the network service
     iface: overrides.iface ?? "",
     serverIp: overrides.serverIp ?? "",

bastion/src/bastion/src/main.ts

@@ -3,9 +3,7 @@
 
 import { mkdirSync, writeFileSync, readFileSync, existsSync, copyFileSync, symlinkSync, unlinkSync } from "node:fs";
 import { execSync } from "node:child_process";
-import type { Arch, BastionConfig } from "@lab/shared";
-import { SUPPORTED_ARCHES, fedoraMirrorFor, classifyOnboard } from "@lab/shared";
-import { kernelPath, initrdPath } from "./templates/boot.ipxe.js";
+import type { BastionConfig } from "@lab/shared";
 import { loadConfig } from "./config.js";
 import { populateNetworkConfig } from "./services/network.js";
 import { createApp } from "./server.js";
@@ -15,7 +13,6 @@ import { renderBootIpxe } from "./templates/boot.ipxe.js";
 import { logger } from "./services/logger.js";
 import { BastionConnection } from "./services/labd-connection.js";
 import { progressBus } from "./services/progress-events.js";
-import { checkInstallAllowed } from "./services/install-guard.js";
 import { ensureBootIso } from "./routes/boot-iso.js";
 
 function copyIfMissing(src: string, dest: string, label: string): void {
@@ -43,6 +40,125 @@ function download(url: string, dest: string, label: string): void {
   }
 }
 
+/**
+ * Pick the largest regular-file initrd from an `xorriso -lsl` listing.
+ *
+ * /live carries decoys: a 0-byte initrd.img placeholder on some images, or an
+ * initrd.img SYMLINK to the real version-suffixed file on others. Parsing is
+ * field-based (ls -l layout: perms links uid gid size month day time 'name')
+ * and considers only lines whose mode string marks a regular file — symlinks
+ * report their link size, not the target's, and must not win.
+ */
+export function pickLargestInitrd(
+  listing: string,
+): { name: string; size: number } | undefined {
+  let best: { name: string; size: number } | undefined;
+  for (const line of listing.split("\n")) {
+    if (!line.startsWith("-")) continue;          // regular files only
+    const quoted = /'([^']+)'/.exec(line);
+    const fields = line.trim().split(/\s+/);
+    const size = parseInt(fields[4] ?? "", 10);
+    const name = quoted?.[1] ?? "";
+    if (!name.startsWith("initrd")) continue;
+    if (!Number.isFinite(size) || size <= 0) continue;
+    if (best === undefined || size > best.size) {
+      best = { name, size };
+    }
+  }
+  return best;
+}
+
+const VYOS_NIGHTLY_RELEASES =
+  "https://api.github.com/repos/vyos/vyos-nightly-build/releases/latest";
+
+/**
+ * Resolve the configured VyOS ISO URL, expanding the "latest" sentinel.
+ *
+ * The nightly asset filename embeds a build date, so there is no stable
+ * "latest.iso" path to hardcode — the newest release has to be looked up.
+ * Any other value is used verbatim, which is how VYOS_ISO_URL pins a build
+ * or points at a locally mirrored copy.
+ */
+function resolveVyosIsoUrl(configured: string): string {
+  if (configured !== "latest") return configured;
+
+  const body = execSync(`curl -sSfL "${VYOS_NIGHTLY_RELEASES}"`, {
+    encoding: "utf-8",
+    stdio: ["pipe", "pipe", "pipe"],
+  });
+  const release = JSON.parse(body) as {
+    tag_name?: string;
+    assets?: Array<{ name: string; browser_download_url: string }>;
+  };
+
+  const asset = (release.assets ?? []).find((a) =>
+    /generic-amd64\.iso$/.test(a.name),
+  );
+  if (!asset) {
+    throw new Error(
+      `No generic-amd64 ISO asset in VyOS nightly release ${release.tag_name ?? "?"}`,
+    );
+  }
+
+  logger.info(`  VyOS ISO resolved to ${asset.name} (${release.tag_name ?? "?"})`);
+  return asset.browser_download_url;
+}
+
+/**
+ * Extract VyOS netboot artifacts from the release ISO.
+ *
+ * VyOS publishes no netboot bundle, so kernel/initrd/squashfs have to come out
+ * of the ISO. xorriso is already in the bastion image (used for boot.iso) and
+ * extracts without root or a loop mount.
+ *
+ * The initrd needs care: /live contains an empty initrd.img placeholder
+ * alongside the real one, which carries a version-suffixed name. Booting the
+ * 0-byte file fails with no useful diagnostic, so pick the largest initrd*.
+ */
+export function prepareVyosArtifacts(config: BastionConfig): void {
+  const kernel = `${config.httpDir}/vyos-vmlinuz`;
+  const initrd = `${config.httpDir}/vyos-initrd`;
+  const squashfs = `${config.httpDir}/vyos-filesystem.squashfs`;
+
+  if (existsSync(kernel) && existsSync(initrd) && existsSync(squashfs)) {
+    logger.info("  VyOS netboot artifacts -- cached");
+    return;
+  }
+
+  const iso = `${config.bastionDir}/vyos.iso`;
+  download(resolveVyosIsoUrl(config.vyosIsoUrl), iso, "VyOS ISO");
+
+  const extract = (isoPath: string, dest: string, label: string): void => {
+    execSync(
+      `xorriso -osirrox on -indev "${iso}" -extract "${isoPath}" "${dest}"`,
+      { stdio: "pipe" },
+    );
+    logger.info(`  ${label} -- extracted from ${isoPath}`);
+  };
+
+  extract("/live/vmlinuz", kernel, "VyOS kernel");
+  extract("/live/filesystem.squashfs", squashfs, "VyOS squashfs");
+
+  // Pick the real initrd by size from the ISO's own directory listing.
+  const listing = execSync(`xorriso -indev "${iso}" -lsl /live/ --`, {
+    encoding: "utf-8",
+    stdio: ["pipe", "pipe", "pipe"],
+  });
+
+  const best = pickLargestInitrd(listing);
+  if (best === undefined) {
+    throw new Error("No non-empty initrd found in /live on the VyOS ISO");
+  }
+  extract(`/live/${best.name}`, initrd, `VyOS initrd (${best.name}, ${best.size} bytes)`);
+
+  // The ISO is only needed to produce the three artifacts above.
+  try {
+    unlinkSync(iso);
+  } catch {
+    // Non-fatal: leaving it costs disk but nothing else.
+  }
+}
+
 function symlinkSafe(target: string, linkPath: string): void {
   try {
     symlinkSync(target, linkPath);
@@ -133,14 +249,9 @@ export async function startBastion(overrides: Partial<BastionConfig> = {}): Prom
   mkdirSync(config.tftpDir, { recursive: true });
   mkdirSync(config.httpDir, { recursive: true });
 
-  // Architectures we can actually network boot, reported in the banner so a missing
-  // arm64 payload is visible at startup instead of at 2am when a rescue is needed.
-  const bootArches: Arch[] = [];
-  let ipxeArm64Ready = false;
-
   // Prepare boot artifacts
   if (config.skipArtifacts !== true) {
-    logger.info(`Preparing boot artifacts (Fedora ${config.fedoraVersion}, ${SUPPORTED_ARCHES.join(" + ")})...`);
+    logger.info(`Preparing boot artifacts (Fedora ${config.fedoraVersion} ${config.arch})...`);
 
     copyIfMissing(
       "/usr/share/ipxe/undionly.kpxe",
@@ -158,41 +269,20 @@ export async function startBastion(overrides: Partial<BastionConfig> = {}): Prom
         `${config.tftpDir}/ipxe-arm64.efi`,
         "iPXE UEFI arm64",
       );
-      ipxeArm64Ready = true;
     } catch {
-      logger.warn("arm64 iPXE not available -- arm64 machines cannot network boot.");
-      logger.warn("  Install with: sudo dnf install ipxe-bootimgs-aarch64");
+      logger.warn("arm64 iPXE not available -- skipping");
     }
 
-    // Fedora pxeboot kernel + initrd per architecture. x86_64 keeps the unsuffixed
-    // names it has always used; other architectures are suffixed. The iPXE templates
-    // resolve the same paths via kernelPath()/initrdPath().
-    for (const arch of SUPPORTED_ARCHES) {
-      const mirror = fedoraMirrorFor(config.fedoraVersion, arch);
-      try {
-        download(
-          `${mirror}/images/pxeboot/vmlinuz`,
-          `${config.httpDir}${kernelPath(arch)}`,
-          `Fedora ${arch} kernel`,
-        );
-        download(
-          `${mirror}/images/pxeboot/initrd.img`,
-          `${config.httpDir}${initrdPath(arch)}`,
-          `Fedora ${arch} initrd`,
-        );
-        bootArches.push(arch);
-      } catch (err) {
-        // Non-fatal: a bastion with no arm64 artifacts still serves x86_64 fine.
-        // Failing startup over an unreachable mirror for an architecture that may not
-        // even be present on this network would be worse.
-        logger.warn(`Fedora ${arch} kernel/initrd unavailable -- ${arch} PXE disabled`);
-        logger.warn(`  ${err instanceof Error ? err.message : String(err)}`);
-      }
-    }
-
-    if (!bootArches.includes("x86_64")) {
-      throw new Error("Fedora x86_64 kernel/initrd could not be staged -- cannot serve PXE");
-    }
+    download(
+      `${config.fedoraMirror}/images/pxeboot/vmlinuz`,
+      `${config.httpDir}/vmlinuz`,
+      "Fedora kernel",
+    );
+    download(
+      `${config.fedoraMirror}/images/pxeboot/initrd.img`,
+      `${config.httpDir}/initrd.img`,
+      "Fedora initrd",
+    );
 
     // Ubuntu netboot artifacts (non-fatal — Ubuntu version may not be released yet)
     try {
@@ -211,6 +301,17 @@ export async function startBastion(overrides: Partial<BastionConfig> = {}): Prom
       logger.warn(`Ubuntu ${config.ubuntuVersion} artifacts not available -- Ubuntu provisioning disabled`);
     }
 
+    // VyOS netboot artifacts (non-fatal — same policy as Ubuntu)
+    try {
+      logger.info("Preparing VyOS netboot artifacts...");
+      prepareVyosArtifacts(config);
+    } catch (err) {
+      logger.warn(
+        `VyOS artifacts not available -- VyOS provisioning disabled ` +
+        `(${err instanceof Error ? err.message : String(err)})`,
+      );
+    }
+
     // Symlink iPXE binaries into HTTP dir for UEFI HTTP Boot
     for (const name of ["ipxe.efi", "ipxe-arm64.efi"]) {
       const src = `${config.tftpDir}/${name}`;
@@ -283,13 +384,6 @@ export async function startBastion(overrides: Partial<BastionConfig> = {}): Prom
     // Wire up command handlers so labd can send install/forget/role commands
     labdConn.onCommand("command-install", async (msg) => {
       if (msg.type !== "command-install") throw new Error("unexpected");
-      const installMac = msg.mac.toLowerCase().replace(/-/g, ":");
-      const osId = (msg.os as import("@lab/shared").OsId | undefined) ?? "fedora-43";
-      const check = checkInstallAllowed(state.load(), installMac, osId);
-      if (check.allowed === false) {
-        logger.warn(`INSTALL REFUSED: ${installMac} -- ${check.error}`);
-        return { status: "error", error: check.error };
-      }
       state.update((s) => {
         s.install_queue[msg.mac] = {
           hostname: msg.hostname,
@@ -297,6 +391,7 @@ export async function startBastion(overrides: Partial<BastionConfig> = {}): Prom
           role: msg.role as import("@lab/shared").Role,
           os: msg.os as import("@lab/shared").OsId,
           queued_at: new Date().toISOString(),
+          ...(msg.vyos ? { vyos: msg.vyos } : {}),
         };
       });
       return { status: "ok", data: { mac: msg.mac, hostname: msg.hostname } };
@@ -350,24 +445,13 @@ export async function startBastion(overrides: Partial<BastionConfig> = {}): Prom
       const mac = (msg.mac as string).toLowerCase();
       const now = new Date().toISOString();
       const existing = state.load().discovered[mac];
-      const identity = {
-        mac,
-        manufacturer: (msg.manufacturer as string) ?? "unknown",
-        product: (msg.product as string) ?? "unknown",
-        board: (msg.board as string) ?? "unknown",
-        ...(existing?.onboard !== undefined ? { onboard: existing.onboard } : {}),
-        ...(existing?.vendor_os !== undefined ? { vendor_os: existing.vendor_os } : {}),
-      };
-      const onboarding = classifyOnboard(identity);
-      const rootDevice = msg.root_device ?? existing?.root_device;
-      const rootArgs = msg.root_args ?? existing?.root_args;
       state.update((s) => {
         s.discovered[mac] = {
           mac,
-          product: identity.product,
-          board: identity.board,
+          product: (msg.product as string) ?? "unknown",
+          board: (msg.board as string) ?? "unknown",
           serial: (msg.serial as string) ?? "unknown",
-          manufacturer: identity.manufacturer,
+          manufacturer: (msg.manufacturer as string) ?? "unknown",
           cpu_model: (msg.cpu_model as string) ?? "unknown",
           cpu_cores: (msg.cpu_cores as number) ?? 0,
           memory_gb: (msg.memory_gb as number) ?? 0,
@@ -376,20 +460,7 @@ export async function startBastion(overrides: Partial<BastionConfig> = {}): Prom
           nics: (msg.nics as Array<{ name: string; mac: string; state: string }>) ?? [],
           first_seen: existing?.first_seen ?? now,
           last_seen: now,
-          onboard: onboarding.onboard,
-          ...(onboarding.vendor_os !== undefined ? { vendor_os: onboarding.vendor_os } : {}),
-          ...(rootDevice !== undefined ? { root_device: rootDevice } : {}),
-          ...(rootArgs !== undefined ? { root_args: rootArgs } : {}),
         };
-        // Keep the installed record in step -- the guard and --pxe-boot both read it.
-        const inst = s.installed[mac];
-        if (inst) {
-          inst.arch = (msg.arch as string) ?? inst.arch;
-          inst.onboard = onboarding.onboard;
-          if (onboarding.vendor_os !== undefined) inst.vendor_os = onboarding.vendor_os;
-          if (rootDevice !== undefined) inst.root_device = rootDevice;
-          if (rootArgs !== undefined) inst.root_args = rootArgs;
-        }
       });
       logger.info(`HARDWARE UPDATED: ${mac} -- ${msg.manufacturer ?? "?"} ${msg.product ?? "?"} (${msg.cpu_model ?? "?"}, ${msg.cpu_cores ?? "?"} cores, ${msg.memory_gb ?? "?"}GB RAM)`);
       return { status: "ok", data: { mac } };
@@ -424,7 +495,7 @@ export async function startBastion(overrides: Partial<BastionConfig> = {}): Prom
   }
 
   // Print banner
-  printBanner(config, bootArches, ipxeArm64Ready);
+  printBanner(config);
 
   // Graceful shutdown
   const shutdown = async (): Promise<void> => {
@@ -446,22 +517,11 @@ export async function startBastion(overrides: Partial<BastionConfig> = {}): Prom
   await new Promise(() => {});
 }
 
-function printBanner(config: BastionConfig, bootArches: Arch[], ipxeArm64Ready: boolean): void {
+function printBanner(config: BastionConfig): void {
   const dhcpInfo = config.dhcpMode === "full"
     ? `full (${config.dhcpRangeStart}-${config.dhcpRangeEnd})`
     : "proxy (alongside existing DHCP)";
 
-  // arm64 needs both an iPXE binary (DHCP hands it out on option 93 = 0x0b) and a
-  // kernel/initrd pair. Report the combination, since either missing breaks it.
-  const archInfo = config.skipArtifacts === true
-    ? "(artifacts skipped)"
-    : SUPPORTED_ARCHES
-        .map((a) => {
-          const ready = bootArches.includes(a) && (a !== "aarch64" || ipxeArm64Ready);
-          return ready ? a : `${a} (unavailable)`;
-        })
-        .join(", ");
-
   console.log("");
   console.log("\x1b[36m\x1b[1m" + "=".repeat(60) + "\x1b[0m");
   console.log("\x1b[36m\x1b[1m  Lab PXE Bastion -- Discovery Mode\x1b[0m");
@@ -470,8 +530,7 @@ function printBanner(config: BastionConfig, bootArches: Arch[], ipxeArm64Ready:
   console.log(`  Network:   \x1b[1m${config.network}/24\x1b[0m via \x1b[1m${config.iface}\x1b[0m`);
   console.log(`  DHCP:      \x1b[1m${dhcpInfo}\x1b[0m`);
   console.log(`  HTTP:      \x1b[1mhttp://${config.serverIp}:${config.httpPort}/\x1b[0m`);
-  console.log(`  OS:        \x1b[1mFedora ${config.fedoraVersion}\x1b[0m`);
-  console.log(`  Net boot:  \x1b[1m${archInfo}\x1b[0m`);
+  console.log(`  OS:        \x1b[1mFedora ${config.fedoraVersion} (${config.arch})\x1b[0m`);
   console.log(`  Domain:    \x1b[1m${config.domain}\x1b[0m`);
   console.log(`  State:     \x1b[1m${config.stateFile}\x1b[0m`);
   console.log("");

bastion/src/bastion/src/routes/api.ts

@@ -5,17 +5,23 @@
 // /api/discover  - receive hardware discovery reports from PXE-booted machines
 
 import type { FastifyInstance } from "fastify";
-import type { HardwareInfo, InstalledInfo, Role } from "@lab/shared";
-import { isValidOsId, SUPPORTED_ROLES, classifyOnboard } from "@lab/shared";
+import type { HardwareInfo, InstalledInfo, Role, VyosInstallSpec } from "@lab/shared";
+import { isValidOsId, SUPPORTED_ROLES, SUPPORTED_OS } from "@lab/shared";
 import type { StateManager } from "../services/state.js";
 import { logger } from "../services/logger.js";
 import { triggerPostProvisionK3s } from "../services/post-provision.js";
-import { checkInstallAllowed } from "../services/install-guard.js";
 import { progressBus } from "../services/progress-events.js";
 import type { ProgressEvent } from "../services/progress-events.js";
 import type { InstallLogBuffer } from "../services/install-log.js";
 import type { SyslogListener } from "../services/syslog-listener.js";
 
+/**
+ * Seconds after dispatch with zero progress before a machine is called stalled.
+ * Generous: the slowest legitimate gap is fetching a ~600MB VyOS squashfs over
+ * HTTP before the hook can report anything.
+ */
+const STALL_THRESHOLD_S = 8 * 60;
+
 export function registerApiRoutes(
   app: FastifyInstance,
   state: StateManager,
@@ -35,9 +41,10 @@ export function registerApiRoutes(
       disk?: string;
       role?: string;
       os?: string;
+      vyos?: VyosInstallSpec;
     };
   }>("/api/install", async (request, reply) => {
-    const { mac: rawMac, hostname, disk, role, os } = request.body ?? {};
+    const { mac: rawMac, hostname, disk, role, os, vyos } = request.body ?? {};
     const mac = (rawMac ?? "").toLowerCase().replace(/-/g, ":");
 
     if (mac === "") {
@@ -51,13 +58,7 @@ export function registerApiRoutes(
 
     const osId = os ?? "fedora-43";
     if (!isValidOsId(osId)) {
-      return reply.status(400).send({ error: `invalid os: '${osId}'. Supported: fedora-43, ubuntu-26.04` });
-    }
-
-    const check = checkInstallAllowed(state.load(), mac, osId);
-    if (check.allowed === false) {
-      logger.warn(`INSTALL REFUSED: ${mac} -- ${check.error}`);
-      return reply.status(409).send({ error: check.error });
+      return reply.status(400).send({ error: `invalid os: '${osId}'. Supported: ${SUPPORTED_OS.join(", ")}` });
     }
 
     state.update((s) => {
@@ -67,6 +68,7 @@ export function registerApiRoutes(
         role: validRole as Role,
         os: osId,
         queued_at: new Date().toISOString(),
+        ...(vyos ? { vyos } : {}),
       };
     });
 
@@ -165,11 +167,17 @@ export function registerApiRoutes(
           };
           s.installed[mac] = installedInfo;
 
-          const admin = installedInfo.role !== "vanilla" && installedInfo.role !== "" ? "lab" : "root";
+          // VyOS: the only login user is "vyos", and a router never runs k3s —
+          // without this guard a non-vanilla role + recorded IP would trigger
+          // the k3s post-provision against a VyOS box.
+          const isVyos = (installedInfo.os ?? "").startsWith("vyos");
+          const admin = isVyos
+            ? "vyos"
+            : installedInfo.role !== "vanilla" && installedInfo.role !== "" ? "lab" : "root";
           console.log(`\n  \x1b[0;32m\x1b[1m  ssh ${admin}@${ip}\x1b[0m\n`);  // eslint-disable-line no-console
 
           // Auto-install k3s for non-vanilla roles
-          if (installedInfo.role !== "vanilla" && ip !== "") {
+          if (!isVyos && installedInfo.role !== "vanilla" && ip !== "") {
             void triggerPostProvisionK3s(installedInfo.hostname, ip, installedInfo.role, admin, mac);
           }
         }
@@ -291,10 +299,6 @@ export function registerApiRoutes(
       arch?: string;
       disks?: Array<{ name: string; size_gb: number; model: string }>;
       nics?: Array<{ name: string; mac: string; state: string }>;
-      // Root filesystem, when the reporter could observe it (recheck over SSH, or the
-      // probe script run from a rescue shell). Used by --pxe-boot.
-      root_device?: string;
-      root_args?: string;
     };
   }>("/api/discover", async (request, reply) => {
     const data = request.body;
@@ -309,53 +313,22 @@ export function registerApiRoutes(
 
     state.update((s) => {
       const existing = s.discovered[mac];
-      // Classify onboarding from the DMI identity we just received. An explicit
-      // classification already on the record wins (see classifyOnboard).
-      const onboarding = classifyOnboard({
-        mac,
-        manufacturer: data.manufacturer ?? existing?.manufacturer ?? "unknown",
-        product: data.product ?? existing?.product ?? "unknown",
-        board: data.board ?? existing?.board ?? "unknown",
-        ...(existing?.onboard !== undefined ? { onboard: existing.onboard } : {}),
-        ...(existing?.vendor_os !== undefined ? { vendor_os: existing.vendor_os } : {}),
-      });
-      const rootDevice = data.root_device ?? existing?.root_device;
-      const rootArgs = data.root_args ?? existing?.root_args;
-
-      // Absent fields keep whatever we already knew. Reporters are not all the full
-      // discovery kickstart: the rescue-shell probe posts only a root device, and
-      // blanking a machine's hardware inventory as a side effect of that would be
-      // silent data loss.
       const hwInfo: HardwareInfo = {
         mac,
-        product: data.product ?? existing?.product ?? "unknown",
-        board: data.board ?? existing?.board ?? "unknown",
-        serial: data.serial ?? existing?.serial ?? "unknown",
-        manufacturer: data.manufacturer ?? existing?.manufacturer ?? "unknown",
-        cpu_model: data.cpu_model ?? existing?.cpu_model ?? "unknown",
-        cpu_cores: data.cpu_cores ?? existing?.cpu_cores ?? 0,
-        memory_gb: data.memory_gb ?? existing?.memory_gb ?? 0,
-        arch: data.arch ?? existing?.arch ?? "unknown",
-        disks: data.disks ?? existing?.disks ?? [],
-        nics: data.nics ?? existing?.nics ?? [],
+        product: data.product ?? "unknown",
+        board: data.board ?? "unknown",
+        serial: data.serial ?? "unknown",
+        manufacturer: data.manufacturer ?? "unknown",
+        cpu_model: data.cpu_model ?? "unknown",
+        cpu_cores: data.cpu_cores ?? 0,
+        memory_gb: data.memory_gb ?? 0,
+        arch: data.arch ?? "unknown",
+        disks: data.disks ?? [],
+        nics: data.nics ?? [],
         first_seen: existing?.first_seen ?? now,
         last_seen: now,
-        onboard: onboarding.onboard,
-        ...(onboarding.vendor_os !== undefined ? { vendor_os: onboarding.vendor_os } : {}),
-        ...(rootDevice !== undefined ? { root_device: rootDevice } : {}),
-        ...(rootArgs !== undefined ? { root_args: rootArgs } : {}),
       };
       s.discovered[mac] = hwInfo;
-
-      // Keep the installed record in step -- the install guard and --pxe-boot read it.
-      const inst = s.installed[mac];
-      if (inst) {
-        if (data.arch !== undefined) inst.arch = data.arch;
-        inst.onboard = onboarding.onboard;
-        if (onboarding.vendor_os !== undefined) inst.vendor_os = onboarding.vendor_os;
-        if (rootDevice !== undefined) inst.root_device = rootDevice;
-        if (rootArgs !== undefined) inst.root_args = rootArgs;
-      }
     });
 
     const label = isNew ? "NEW MACHINE DISCOVERED" : "MACHINE RE-DISCOVERED";
@@ -476,6 +449,15 @@ export function registerApiRoutes(
     const installedEntry = currentState.installed[mac];
 
     if (queueEntry) {
+      // A machine that was handed an install script but has reported nothing
+      // since is wedged BEFORE the installer environment came up — a bad
+      // kernel/initrd, no network in the initramfs, or the wrong NIC picked.
+      // Surfacing it here is what makes that diagnosable without a console.
+      const since = queueEntry.progress_at ?? queueEntry.dispatched_at;
+      const stalledForS = since !== undefined && queueEntry.progress === undefined
+        ? Math.floor((Date.now() - new Date(since).getTime()) / 1000)
+        : 0;
+
       return reply.send({
         mac,
         hostname: queueEntry.hostname,
@@ -483,6 +465,9 @@ export function registerApiRoutes(
         progress: queueEntry.progress ?? "queued",
         progress_detail: queueEntry.progress_detail ?? "",
         progress_at: queueEntry.progress_at ?? queueEntry.queued_at,
+        dispatched_at: queueEntry.dispatched_at,
+        stalled_for_s: stalledForS,
+        stalled: stalledForS > STALL_THRESHOLD_S,
         role: queueEntry.role,
         os: queueEntry.os,
         stages: queueEntry.log ?? [],

bastion/src/bastion/src/routes/dispatch.ts

@@ -5,8 +5,7 @@
 //   - unknown       -> discovery mode (collect hardware, POST to bastion)
 
 import type { FastifyInstance } from "fastify";
-import type { Arch, BastionConfig, BastionState, OsId } from "@lab/shared";
-import { normalizeArch, fedoraMirrorFor, osSupportsArch } from "@lab/shared";
+import type { BastionConfig } from "@lab/shared";
 import type { StateManager } from "../services/state.js";
 import {
   renderDiscoverIpxe,
@@ -14,51 +13,12 @@ import {
   renderDebugIpxe,
   renderPxeBootDebugIpxe,
   renderLocalBootIpxe,
-  renderUnsupportedIpxe,
 } from "../templates/boot.ipxe.js";
 import { renderUbuntuInstallIpxe } from "../templates/ubuntu-boot.ipxe.js";
+import { renderVyosInstallIpxe } from "../templates/vyos-boot.ipxe.js";
 import { renderDebugKickstart } from "../templates/debug.ks.js";
 import { logger } from "../services/logger.js";
 
-/**
- * Resolve a booting machine's architecture.
- *
- * Order matters. The tracked record is what we actually observed on the machine, so it
- * wins. `reported` is iPXE's ${buildarch}, which is only as good as the binary DHCP
- * handed the client -- correct in practice, but a misconfigured option 93 mapping would
- * make it lie. The configured default is the last resort.
- *
- * There is deliberately no operator-supplied architecture anywhere in this path.
- */
-export function resolveArch(
-  state: BastionState,
-  mac: string,
-  reported: string | undefined,
-  config: BastionConfig,
-): Arch {
-  return normalizeArch(state.installed[mac]?.arch)
-    ?? normalizeArch(state.install_queue[mac]?.arch)
-    ?? normalizeArch(state.discovered[mac]?.arch)
-    ?? normalizeArch(reported)
-    ?? normalizeArch(config.arch)
-    ?? "x86_64";
-}
-
-/** The root filesystem to boot for --pxe-boot, if the machine's record carries one. */
-function resolveRoot(
-  state: BastionState,
-  mac: string,
-): { rootDevice: string; rootArgs?: string } | null {
-  const installed = state.installed[mac];
-  const discovered = state.discovered[mac];
-  const rootDevice = installed?.root_device ?? discovered?.root_device;
-  if (rootDevice === undefined || rootDevice === "") return null;
-  const rootArgs = installed?.root_args ?? discovered?.root_args;
-  return rootArgs !== undefined && rootArgs !== ""
-    ? { rootDevice, rootArgs }
-    : { rootDevice };
-}
-
 export function registerDispatchRoutes(
   app: FastifyInstance,
   config: BastionConfig,
@@ -93,68 +53,18 @@ curl -sf -X POST "http://${config.serverIp}:${config.httpPort}/api/progress" \\
     -H "Content-Type: application/json" \\
     -d "{\\"mac\\":\\"$MAC_ADDR\\",\\"stage\\":\\"debug-ready\\",\\"detail\\":\\"nc $IP_ADDR 2323\\"}" 2>/dev/null || true
 
-# --- Find the installed root filesystem and report it ---
-# This is what 'labctl provision debug --pxe-boot' needs. The rescue image cannot
-# report it by itself: %pre/%post do not run in rescue mode, so it happens here.
-vgchange -ay >/dev/null 2>&1 || true
-
-ROOT_DEVICE=""
-ROOT_ARGS=""
-PROBE_MNT=/tmp/lab-rootprobe
-mkdir -p "$PROBE_MNT"
-
-# Candidates: every LVM logical volume plus every non-LVM partition with a filesystem.
-for CAND in $(lvs --noheadings -o lv_path 2>/dev/null) \\
-            $(blkid -o device 2>/dev/null | grep -v '^/dev/mapper/'); do
-    [ -b "$CAND" ] || continue
-    mount -o ro "$CAND" "$PROBE_MNT" >/dev/null 2>&1 || continue
-    # A root filesystem has both of these; /boot and /home do not.
-    if [ -f "$PROBE_MNT/etc/fstab" ] && [ -d "$PROBE_MNT/usr" ]; then
-        ROOT_DEVICE="$CAND"
-        PRETTY=$(. "$PROBE_MNT/etc/os-release" 2>/dev/null && echo "$PRETTY_NAME")
-        echo "  found root: $CAND \${PRETTY:+($PRETTY)}"
-        if [ "$(lsblk -no TYPE "$CAND" 2>/dev/null | head -1)" = "lvm" ]; then
-            VGLV=$(lvs --noheadings -o vg_name,lv_name "$CAND" 2>/dev/null | awk '{print $1"/"$2}')
-            [ -n "$VGLV" ] && ROOT_ARGS="rd.lvm.lv=$VGLV"
-            # Swap comes from fstab here — /proc/swaps is the rescue image's, not the host's.
-            SWLV=$(awk '$3=="swap" && $1 ~ /^\\/dev\\// {print $1; exit}' "$PROBE_MNT/etc/fstab" 2>/dev/null)
-            if [ -n "$SWLV" ]; then
-                SWVGLV=$(lvs --noheadings -o vg_name,lv_name "$SWLV" 2>/dev/null | awk '{print $1"/"$2}')
-                [ -n "$SWVGLV" ] && [ "$SWVGLV" != "$VGLV" ] && ROOT_ARGS="$ROOT_ARGS rd.lvm.lv=$SWVGLV"
-            fi
-        fi
-        umount "$PROBE_MNT" >/dev/null 2>&1 || true
-        break
-    fi
-    umount "$PROBE_MNT" >/dev/null 2>&1 || true
-done
-
-if [ -n "$ROOT_DEVICE" ]; then
-    curl -sf -X POST "http://${config.serverIp}:${config.httpPort}/api/discover" \\
-        -H "Content-Type: application/json" \\
-        -d "{\\"mac\\":\\"$MAC_ADDR\\",\\"root_device\\":\\"$ROOT_DEVICE\\",\\"root_args\\":\\"$ROOT_ARGS\\"}" 2>/dev/null \\
-        && echo "  reported to bastion — 'labctl provision debug --pxe-boot' will work now"
-else
-    echo "  no root filesystem found — --pxe-boot cannot be used on this machine"
-fi
-
 echo ""
 echo "=== Debug environment ready ==="
 echo "  nc $IP_ADDR 2323    (remote shell)"
 echo "  ssh root@$IP_ADDR   (password: debug)"
-if [ -n "$ROOT_DEVICE" ]; then
-    echo "  root: $ROOT_DEVICE $ROOT_ARGS"
-fi
 echo "==============================="
 `;
     return reply.type("text/plain").send(script);
   });
 
-  app.get<{ Querystring: { mac?: string; arch?: string } }>("/dispatch", async (request, reply) => {
+  app.get<{ Querystring: { mac?: string } }>("/dispatch", async (request, reply) => {
     const mac = (request.query.mac ?? "").toLowerCase().replace(/-/g, ":");
     const currentState = state.load();
-    const arch = resolveArch(currentState, mac, request.query.arch, config);
-    const fedoraMirror = fedoraMirrorFor(config.fedoraVersion, arch);
 
     // Debug mode takes highest priority — auto-clear after serving once
     const debugEntry = currentState.debug[mac];
@@ -163,48 +73,22 @@ echo "==============================="
       state.update((s) => { delete s.debug[mac]; });
 
       let script: string;
-      const wantsPxeBoot = debugEntry.pxeBoot === true;
-      const root = wantsPxeBoot ? resolveRoot(currentState, mac) : null;
-
-      if (root !== null) {
-        logger.info(`PXE BOOT DEBUG: ${mac} -> ${hostname} (${arch}, root=${root.rootDevice})`);
+      if (debugEntry.pxeBoot) {
+        logger.info(`PXE BOOT DEBUG: ${mac} -> ${hostname} (kernel+initrd from PXE, root from NVMe)`);
         script = renderPxeBootDebugIpxe({
           mac,
           hostname,
           serverIp: config.serverIp,
           httpPort: config.httpPort,
-          arch,
-          ...root,
         });
       } else {
-        // --pxe-boot without a known root device falls back to rescue rather than
-        // guessing. A wrong root= leaves the machine unbootable, and rescue is where
-        // the operator can find the real one (curl /debug-setup.sh reports it back).
-        const notice = wantsPxeBoot
-          ? [
-              "",
-              "NOTE: --pxe-boot requested, but no root device is recorded",
-              "      for this machine. Booting rescue instead.",
-              "      From the rescue shell, run:",
-              // No pipe or && here: iPXE treats || and && as command separators, so keep
-              // the printed command free of anything its parser might claim.
-              `        curl -s http://${config.serverIp}:${config.httpPort}/debug-setup.sh -o /tmp/s.sh ; sh /tmp/s.sh`,
-              "      then retry --pxe-boot.",
-            ]
-          : undefined;
-        if (wantsPxeBoot) {
-          logger.warn(`PXE BOOT DEBUG: ${mac} -> ${hostname} has no recorded root device -- serving rescue instead`);
-        } else {
-          logger.info(`DEBUG BOOT: ${mac} -> ${hostname} (${arch}, rescue mode)`);
-        }
+        logger.info(`DEBUG BOOT: ${mac} -> ${hostname} (rescue mode)`);
         script = renderDebugIpxe({
           mac,
           hostname,
           serverIp: config.serverIp,
           httpPort: config.httpPort,
-          fedoraMirror,
-          arch,
-          ...(notice ? { notice } : {}),
+          fedoraMirror: config.fedoraMirror,
         });
       }
       return reply.type("text/plain").send(script);
@@ -214,24 +98,24 @@ echo "==============================="
     if (queueEntry) {
       const hostname = queueEntry.hostname ?? "lab-node";
       const os = queueEntry.os ?? "fedora-43";
-      logger.info(`INSTALL STARTED: ${mac} -> ${hostname} (${os}, ${arch})`);
+      logger.info(`INSTALL STARTED: ${mac} -> ${hostname} (${os})`);
+
+      // Stamp the handoff so a machine that boots the installer but never
+      // reports can be spotted without a console.
+      state.update((s) => {
+        const entry = s.install_queue[mac];
+        if (entry) entry.dispatched_at = new Date().toISOString();
+      });
 
       let script: string;
-      if (os.startsWith("ubuntu")) {
-        // Last line of defence. The install guard refuses this combination when the
-        // machine's architecture is already known, but a machine queued before it was
-        // discovered can reach here. Serving the x86-only Ubuntu kernel to an arm64
-        // client is precisely the bug this work exists to fix, so stop instead.
-        if (!osSupportsArch(os as OsId, arch)) {
-          logger.error(`INSTALL BLOCKED: ${mac} -> ${hostname} -- ${os} has no ${arch} artifacts`);
-          script = renderUnsupportedIpxe({
-            hostname,
-            mac,
-            reason: `${os} publishes no ${arch} netboot artifacts`,
-            action: `labctl provision install ${mac} ${hostname} --os fedora-43`,
-          });
-          return reply.type("text/plain").send(script);
-        }
+      if (os.startsWith("vyos")) {
+        script = renderVyosInstallIpxe({
+          mac,
+          hostname,
+          serverIp: config.serverIp,
+          httpPort: config.httpPort,
+        });
+      } else if (os.startsWith("ubuntu")) {
         script = renderUbuntuInstallIpxe({
           mac,
           hostname,
@@ -246,8 +130,7 @@ echo "==============================="
           serverIp: config.serverIp,
           httpPort: config.httpPort,
           fedoraVersion: config.fedoraVersion,
-          fedoraMirror,
-          arch,
+          fedoraMirror: config.fedoraMirror,
         });
       }
 
@@ -264,14 +147,13 @@ echo "==============================="
     }
 
     // Unknown MAC -> discovery mode
-    logger.info(`PXE request from ${mac} (${arch}) -> discovery mode`);
+    logger.info(`PXE request from ${mac} -> discovery mode`);
 
     const script = renderDiscoverIpxe({
       mac,
       serverIp: config.serverIp,
       httpPort: config.httpPort,
-      fedoraMirror,
-      arch,
+      fedoraMirror: config.fedoraMirror,
     });
 
     return reply.type("text/plain").send(script);

bastion/src/bastion/src/routes/vyos.ts

@@ -0,0 +1,71 @@
+// VyOS network install routes.
+//
+// VyOS has no unattended installer, so the automation is injected via
+// live-config's `hooks` component: the iPXE script passes
+// live-config.hooks=<.../vyos/autoinstall.sh>, live-config wgets it and runs it
+// as root, and that script fetches and executes the generated install driver.
+
+import type { FastifyInstance } from "fastify";
+import type { BastionConfig } from "@lab/shared";
+import type { StateManager } from "../services/state.js";
+import { buildVyosConfigSpec } from "../templates/vyos-config-spec.js";
+import { renderVyosInstallPy } from "../templates/vyos-install.py.js";
+import { logger } from "../services/logger.js";
+
+function normalizeMac(value: string | undefined): string {
+  return (value ?? "").toLowerCase().replace(/-/g, ":");
+}
+
+export function registerVyosRoutes(
+  app: FastifyInstance,
+  config: BastionConfig,
+  state: StateManager,
+): void {
+  // live-config hook. Kept minimal: everything version-specific lives in the
+  // generated Python. wget is guaranteed present -- live-config used it to
+  // fetch this very script.
+  app.get<{ Querystring: { mac?: string } }>("/vyos/autoinstall.sh", async (request, reply) => {
+    const mac = normalizeMac(request.query.mac);
+    const base = `http://${config.serverIp}:${config.httpPort}`;
+
+    logger.info(`VYOS AUTOINSTALL HOOK served to ${mac || "unknown MAC"}`);
+
+    const script = `#!/bin/sh
+# Lab PXE Bastion -- VyOS unattended install hook (run by live-config as root)
+set -eu
+
+wget -q "${base}/vyos/install.py?mac=${mac}" -O /tmp/vyos-install.py
+exec python3 /tmp/vyos-install.py
+`;
+    return reply.type("text/plain").send(script);
+  });
+
+  // Per-MAC install driver, with the machine's config spec baked in.
+  app.get<{ Querystring: { mac?: string } }>("/vyos/install.py", async (request, reply) => {
+    const mac = normalizeMac(request.query.mac);
+    const queueEntry = state.load().install_queue[mac];
+
+    const spec = buildVyosConfigSpec({
+      hostname: queueEntry?.hostname ?? "vyos",
+      spec: queueEntry?.vyos,
+      defaultPassword: config.vyosDefaultPassword,
+      sshKeys: config.sshKeys,
+      disk: queueEntry?.disk,
+    });
+
+    logger.info(
+      `VYOS INSTALL DRIVER served to ${mac} (${spec.hostname}, ` +
+      `${spec.sets.length} config ops, disk="${spec.disk || "auto"}")`,
+    );
+
+    const script = renderVyosInstallPy({
+      spec,
+      mac,
+      serverIp: config.serverIp,
+      httpPort: config.httpPort,
+      role: queueEntry?.role ?? "vanilla",
+    });
+
+    return reply.type("text/plain").send(script);
+  });
+}

bastion/src/bastion/src/server.ts

@@ -12,6 +12,7 @@ import { registerDispatchRoutes } from "./routes/dispatch.js";
 import { registerKickstartRoutes } from "./routes/kickstart.js";
 import { registerApiRoutes } from "./routes/api.js";
 import { registerAsahiRoutes } from "./routes/asahi.js";
+import { registerVyosRoutes } from "./routes/vyos.js";
 
 
 export function createApp(config: BastionConfig): { app: ReturnType<typeof Fastify>; state: StateManager; installLog: InstallLogBuffer; syslog: SyslogListener } {
@@ -47,6 +48,7 @@ export function createApp(config: BastionConfig): { app: ReturnType<typeof Fasti
   registerKickstartRoutes(app, config, state, syslog);
   registerApiRoutes(app, state, installLog, syslog);
   registerAsahiRoutes(app, config);
+  registerVyosRoutes(app, config, state);
   // boot.iso is generated at startup and served as a static file from httpDir
   // (static serving supports HTTP Range requests, required by JetKVM streaming)
 

bastion/src/bastion/src/services/install-guard.ts

@@ -1,94 +0,0 @@
-// Pre-flight checks for queuing an OS install.
-//
-// Both entry points -- the HTTP /api/install route and the labd command-install handler
-// -- run this, so `labctl provision install` and `provision reprovision` are covered
-// whichever way the request arrives.
-//
-// Rescue/debug is deliberately NOT guarded. Being unable to reinstall a machine is
-// exactly when you most need to boot it into a rescue shell.
-
-import type { Arch, BastionState, OsId } from "@lab/shared";
-import { classifyOnboard, normalizeArch, osSupportsArch, vendorOsDescription, archesForOs } from "@lab/shared";
-
-export type InstallCheck =
-  | { allowed: true }
-  | { allowed: false; error: string };
-
-interface MachineIdentity {
-  hostname: string;
-  arch: Arch | undefined;
-  identity: Parameters<typeof classifyOnboard>[0];
-}
-
-/** Best-known identity for a MAC, merged across the three state maps. */
-function identify(state: BastionState, mac: string): MachineIdentity {
-  const discovered = state.discovered[mac];
-  const installed = state.installed[mac];
-  const queued = state.install_queue[mac];
-
-  const manufacturer = discovered?.manufacturer ?? installed?.manufacturer;
-  const product = discovered?.product ?? installed?.product;
-  const board = discovered?.board;
-  const onboard = installed?.onboard ?? discovered?.onboard;
-  const vendorOs = installed?.vendor_os ?? discovered?.vendor_os;
-
-  return {
-    hostname: installed?.hostname ?? queued?.hostname ?? discovered?.product ?? mac,
-    arch: normalizeArch(installed?.arch ?? queued?.arch ?? discovered?.arch),
-    identity: {
-      mac,
-      ...(manufacturer !== undefined ? { manufacturer } : {}),
-      ...(product !== undefined ? { product } : {}),
-      ...(board !== undefined ? { board } : {}),
-      ...(onboard !== undefined ? { onboard } : {}),
-      ...(vendorOs !== undefined ? { vendor_os: vendorOs } : {}),
-    },
-  };
-}
-
-/**
- * Decide whether `mac` may be queued for an install of `os`.
- *
- * Refusals name the machine and the reason, and point at the action that is available
- * instead. An operator hitting this at 2am should not have to read the source to work
- * out what happened.
- */
-export function checkInstallAllowed(
-  state: BastionState,
-  mac: string,
-  os: OsId,
-): InstallCheck {
-  const machine = identify(state, mac);
-  const { onboard, vendor_os } = classifyOnboard(machine.identity);
-
-  // 1. Machines running a vendor OS we cannot rebuild.
-  if (onboard === "ssh") {
-    const what = vendorOsDescription(vendor_os);
-    return {
-      allowed: false,
-      error:
-        `Refusing to install ${machine.hostname} (${mac}): it runs ${what}. ` +
-        `No image in our pipeline can restore it, so installing ${os} would destroy that ` +
-        `driver and firmware stack permanently. This machine is SSH-onboard: we manage its ` +
-        `userspace, not its OS. ` +
-        `To boot it into a rescue shell instead, run: labctl provision debug ${machine.hostname}`,
-      // TODO: when a DGX OS / SparkOS image joins the pipeline, an install targeting a
-      // machine whose vendor_os matches that image should be allowed through here.
-    };
-  }
-
-  // 2. Architecture the OS has no netboot artifacts for.
-  if (machine.arch !== undefined && !osSupportsArch(os, machine.arch)) {
-    const supported = archesForOs(os);
-    return {
-      allowed: false,
-      error:
-        `Refusing to install ${os} on ${machine.hostname} (${mac}): ` +
-        `${os} has no ${machine.arch} netboot artifacts` +
-        (supported.length > 0 ? ` (only ${supported.join(", ")})` : "") +
-        `. Use an OS that supports ${machine.arch}.`,
-    };
-  }
-
-  return { allowed: true };
-}

bastion/src/bastion/src/templates/boot.ipxe.ts

@@ -1,55 +1,4 @@
 // iPXE boot script templates for dispatch routing.
-//
-// Architecture handling: the bastion serves one kernel/initrd pair per architecture.
-// x86_64 keeps the original unsuffixed paths so its output is unchanged; every other
-// architecture gets an arch-suffixed pair. See stageBootArtifacts() in main.ts for the
-// matching staging side, and boot-iso.ts for the same scheme on the ISO path.
-
-import type { Arch } from "@lab/shared";
-
-/** Kernel/initrd URL paths, keyed by architecture. */
-export function kernelPath(arch: Arch): string {
-  return arch === "x86_64" ? "/vmlinuz" : `/vmlinuz-${arch}`;
-}
-
-export function initrdPath(arch: Arch): string {
-  return arch === "x86_64" ? "/initrd.img" : `/initrd-${arch}.img`;
-}
-
-/**
- * Console arguments per architecture.
- *
- * arm64 has no VGA text console: a headless machine only talks over the SoC UART, so
- * ttyAMA0 must be listed as well. The last console= wins for /dev/console, so serial
- * is the interactive one while tty0 still receives boot output on machines with a
- * display attached.
- */
-const CONSOLE_ARGS: Record<Arch, string> = {
-  x86_64: "console=tty0",
-  aarch64: "console=tty0 console=ttyAMA0,115200",
-};
-
-/**
- * Anaconda arguments for the graphical-suppression / console setup.
- *
- * `nomodeset` disables kernel mode setting, which on x86 forces the generic VGA path
- * and makes flaky GPU drivers survive the installer. On arm64 it does not mean the
- * same thing -- there is no VGA fallback to drop back to, and it can leave the machine
- * with no usable console at all -- so arm64 gets explicit console arguments instead.
- */
-function installerArgs(arch: Arch): string {
-  return arch === "x86_64" ? "inst.text nomodeset" : `inst.text ${CONSOLE_ARGS[arch]}`;
-}
-
-/** Extra console arguments appended to templates that don't already set them. */
-function extraConsoleArgs(arch: Arch): string {
-  return arch === "x86_64" ? "" : ` ${CONSOLE_ARGS[arch]}`;
-}
-
-/** Join kernel arguments, dropping empties so callers can pass optional groups. */
-function joinArgs(...parts: Array<string | undefined>): string {
-  return parts.filter((p) => p !== undefined && p !== "").join(" ");
-}
 
 export interface BootIpxeParams {
   serverIp: string;
@@ -59,11 +8,6 @@ export interface BootIpxeParams {
 /**
  * Initial iPXE boot script that chains to the dispatch endpoint.
  * This is what dnsmasq serves to iPXE clients via HTTP.
- *
- * `${buildarch}` is iPXE's own build architecture ("x86_64" or "arm64"), which is the
- * one architecture signal available on every path -- network PXE, UEFI HTTP boot and
- * the boot ISO alike. DHCP option 93 only reaches dnsmasq, never this HTTP endpoint.
- * dispatch prefers the tracked machine record and falls back to this.
  */
 export function renderBootIpxe(params: BootIpxeParams): string {
   return `#!ipxe
@@ -75,7 +19,7 @@ echo   Contacting server for instructions...
 echo ============================================
 echo
 
-chain http://${params.serverIp}:${params.httpPort}/dispatch?mac=\${net0/mac}&arch=\${buildarch}
+chain http://${params.serverIp}:${params.httpPort}/dispatch?mac=\${net0/mac}
 `;
 }
 
@@ -87,9 +31,7 @@ export function renderDiscoverIpxe(params: {
   serverIp: string;
   httpPort: number;
   fedoraMirror: string;
-  arch: Arch;
 }): string {
-  const base = `http://${params.serverIp}:${params.httpPort}`;
   return `#!ipxe
 
 echo
@@ -100,8 +42,8 @@ echo   Collecting hardware info...
 echo =============================================
 echo
 
-kernel ${base}${kernelPath(params.arch)} inst.ks=${base}/discover.ks inst.stage2=${params.fedoraMirror} ${installerArgs(params.arch)}
-initrd ${base}${initrdPath(params.arch)}
+kernel http://${params.serverIp}:${params.httpPort}/vmlinuz inst.ks=http://${params.serverIp}:${params.httpPort}/discover.ks inst.stage2=${params.fedoraMirror} inst.text nomodeset
+initrd http://${params.serverIp}:${params.httpPort}/initrd.img
 boot
 `;
 }
@@ -116,9 +58,7 @@ export function renderInstallIpxe(params: {
   httpPort: number;
   fedoraVersion: string;
   fedoraMirror: string;
-  arch: Arch;
 }): string {
-  const base = `http://${params.serverIp}:${params.httpPort}`;
   return `#!ipxe
 
 echo
@@ -129,8 +69,8 @@ echo   MAC:    ${params.mac}
 echo =============================================
 echo
 
-kernel ${base}${kernelPath(params.arch)} inst.ks=${base}/ks?mac=${params.mac} inst.repo=${params.fedoraMirror} ${installerArgs(params.arch)}
-initrd ${base}${initrdPath(params.arch)}
+kernel http://${params.serverIp}:${params.httpPort}/vmlinuz inst.ks=http://${params.serverIp}:${params.httpPort}/ks?mac=${params.mac} inst.repo=${params.fedoraMirror} inst.text nomodeset
+initrd http://${params.serverIp}:${params.httpPort}/initrd.img
 boot
 `;
 }
@@ -138,9 +78,6 @@ boot
 /**
  * iPXE script for debug/rescue mode -- boots Fedora installer in rescue mode.
  * Provides a shell with LVM tools, network, and SSH for inspecting installed systems.
- *
- * `notice` is shown before the boot line. dispatch uses it to explain why a requested
- * --pxe-boot fell back to rescue.
  */
 export function renderDebugIpxe(params: {
   mac: string;
@@ -148,11 +85,7 @@ export function renderDebugIpxe(params: {
   serverIp: string;
   httpPort: number;
   fedoraMirror: string;
-  arch: Arch;
-  notice?: string[];
 }): string {
-  const base = `http://${params.serverIp}:${params.httpPort}`;
-  const notice = (params.notice ?? []).map((line) => `echo   ${line}\n`).join("");
   return `#!ipxe
 
 echo
@@ -160,11 +93,11 @@ echo =============================================
 echo   Lab PXE Bastion - DEBUG/RESCUE MODE
 echo   Target: ${params.hostname}
 echo   MAC:    ${params.mac}
-${notice}echo =============================================
+echo =============================================
 echo
 
-kernel ${base}${kernelPath(params.arch)} inst.rescue inst.text inst.sshd inst.ks=${base}/debug.ks?mac=${params.mac} inst.stage2=${params.fedoraMirror}${extraConsoleArgs(params.arch)}
-initrd ${base}${initrdPath(params.arch)}
+kernel http://${params.serverIp}:${params.httpPort}/vmlinuz inst.rescue inst.text inst.sshd inst.ks=http://${params.serverIp}:${params.httpPort}/debug.ks?mac=${params.mac} inst.stage2=${params.fedoraMirror}
+initrd http://${params.serverIp}:${params.httpPort}/initrd.img
 boot
 `;
 }
@@ -173,28 +106,13 @@ boot
  * iPXE script for PXE-boot debug mode -- boots the installed system's root
  * filesystem using the bastion's PXE kernel+initrd instead of local GRUB.
  * Workaround for UEFI firmware bugs that make local disk boot slow.
- *
- * rootDevice/rootArgs come from the machine's record -- they are not assumed. Our
- * Fedora installs use an LVM layout, but nothing guarantees any given machine does,
- * and a wrong root= here means an unbootable machine. dispatch refuses to render this
- * script without them.
  */
 export function renderPxeBootDebugIpxe(params: {
   mac: string;
   hostname: string;
   serverIp: string;
   httpPort: number;
-  arch: Arch;
-  rootDevice: string;
-  rootArgs?: string;
 }): string {
-  const base = `http://${params.serverIp}:${params.httpPort}`;
-  const cmdline = joinArgs(
-    `root=${params.rootDevice}`,
-    "ro",
-    params.rootArgs,
-    CONSOLE_ARGS[params.arch],
-  );
   return `#!ipxe
 
 echo
@@ -206,40 +124,12 @@ echo   Kernel+initrd from PXE, root from NVMe
 echo =============================================
 echo
 
-kernel ${base}${kernelPath(params.arch)} ${cmdline}
-initrd ${base}${initrdPath(params.arch)}
+kernel http://${params.serverIp}:${params.httpPort}/vmlinuz root=/dev/mapper/labvg-root ro rd.lvm.lv=labvg/root rd.lvm.lv=labvg/swap console=tty0
+initrd http://${params.serverIp}:${params.httpPort}/initrd.img
 boot
 `;
 }
 
-/**
- * iPXE script for a request we refuse to serve.
- *
- * Better a machine that stops with a legible reason on its console than one handed a
- * kernel it cannot execute, which fails much later and much less clearly.
- */
-export function renderUnsupportedIpxe(params: {
-  mac: string;
-  hostname: string;
-  reason: string;
-  action?: string;
-}): string {
-  return `#!ipxe
-
-echo
-echo =============================================
-echo   Lab PXE Bastion - CANNOT BOOT THIS MACHINE
-echo   Target: ${params.hostname}
-echo   MAC:    ${params.mac}
-echo
-echo   ${params.reason}
-${params.action !== undefined ? `echo\necho   Try: ${params.action}\n` : ""}echo =============================================
-echo
-sleep 10
-exit 1
-`;
-}
-
 /**
  * iPXE script for already-installed machines -- exits to boot from local disk.
  */

bastion/src/bastion/src/templates/dnsmasq.conf.ts

@@ -48,20 +48,15 @@ enable-tftp
 tftp-root=${tftpDir}
 tftp-no-blocksize
 
-# Detect client architecture -- PXE (TFTP) clients.
-# Values are DHCP option 93 (Client System Architecture), IANA "Processor Architecture
-# Types". Getting these wrong means the machine is handed a bootloader its firmware
-# cannot execute, and it loops or hangs with no console output.
+# Detect client architecture -- PXE (TFTP) clients
 dhcp-match=set:bios,option:client-arch,0
 dhcp-match=set:efi-x86_64,option:client-arch,7
 dhcp-match=set:efi-x86_64,option:client-arch,9
 dhcp-match=set:efi-arm64,option:client-arch,11
 
-# Detect client architecture -- UEFI HTTP Boot clients (no TFTP size limit).
-# 16 = x64 uefi boot from http, 19 = arm uefi 64 boot from http.
-# (20 is pc/at bios boot from http -- not arm64.)
+# Detect client architecture -- UEFI HTTP Boot clients (no TFTP size limit)
 dhcp-match=set:httpboot-x86_64,option:client-arch,16
-dhcp-match=set:httpboot-arm64,option:client-arch,19
+dhcp-match=set:httpboot-arm64,option:client-arch,20
 
 # Detect iPXE clients (already chainloaded)
 dhcp-userclass=set:ipxe,iPXE

bastion/src/bastion/src/templates/install.ks.ts

@@ -40,6 +40,11 @@ export function renderInstallKickstart(params: InstallKickstartParams): string {
   const now = new Date().toISOString();
   const hasLonghorn = role === "worker";
   const hasRancher = role === "infra";
+  // k8s roles get a dedicated 120G image-store LV. 2026-08 incident: the old
+  // 20G LV idled at 85% used, so a single ~5G image pull tripped imagefs
+  // eviction. Must be sized here — longhorn's --grow consumes all remaining
+  // VG space, making post-install lvextend impossible on worker nodes.
+  const hasRancherLv = role === "infra" || role === "worker";
   const isVanilla = role === "vanilla";
 
   // -- Auth section --
@@ -113,9 +118,9 @@ done
     ? `logvol /var/lib/longhorn --vgname=${vg} --name=longhorn --fstype=xfs --grow --size=1`
     : "";
 
-  // -- Rancher LV for fresh install (infra role) --
-  const rancherFreshLine = hasRancher
-    ? `logvol /var/lib/rancher --vgname=${vg} --name=rancher --fstype=xfs --size=20480`
+  // -- Rancher LV for fresh install (k8s roles: worker + infra) --
+  const rancherFreshLine = hasRancherLv
+    ? `logvol /var/lib/rancher --vgname=${vg} --name=rancher --fstype=xfs --size=122880`
     : "";
 
   return `# Lab Bastion -- Fedora ${fedoraVersion} server install
@@ -129,7 +134,13 @@ lang ${locale}
 keyboard uk
 timezone ${timezone} --utc
 
-network --bootproto=dhcp --activate --hostname=${fqdn}
+# --ipv6=auto, not =dhcp: "auto" follows the router advertisement, so the RA's
+# managed-flag is what steers the node to DHCPv6, and a VLAN with no DHCPv6 yet
+# still installs instead of blocking on a lease that will never come. Failing an
+# OS install because IPv6 was not ready would be a worse trade than a node that
+# briefly has no v6. The address itself comes from a kea DHCPv6 reservation
+# keyed on MAC -- the same source of truth as the v4 address.
+network --bootproto=dhcp --ipv6=auto --activate --hostname=${fqdn}
 
 ${auth}
 ${userDirective}
@@ -361,6 +372,21 @@ fs.inotify.max_user_watches         = 1048576
 SYSCTL
 sysctl --system || true
 
+# -- IPv6 link-local address generation: EUI-64, fleet-wide --
+# A cluster node takes its IPv6 from a MAC-keyed DHCPv6 reservation. kea can only
+# match that reservation when it can recover the node's MAC, and for a modern
+# client that sends a DUID-UUID (no MAC in it) the only place kea can find one is
+# an EUI-64 link-local. NetworkManager's default is stable-privacy (RFC 7217),
+# whose link-local hides the MAC -- so a node on the default silently never gets
+# its reserved address, and with a reservations-only subnet it gets nothing at
+# all. Proven on 2026-09-06: worker0/worker2 (eui64) bound; worker1/spark
+# (default) did not, until flipped. Setting it here means a new node is correct
+# from first boot, before its connection is ever activated.
+cat > /etc/NetworkManager/conf.d/10-ipv6-eui64.conf << 'NMEUI64'
+[connection]
+ipv6.addr-gen-mode=eui64
+NMEUI64
+
 # -- Disable firewalld permanently (k3s/Cilium manage iptables directly) --
 # Note: no '--now' — systemd is not running in the Anaconda chroot
 systemctl disable firewalld || true

bastion/src/bastion/src/templates/ubuntu-autoinstall.ts

@@ -40,34 +40,39 @@ export function renderUbuntuAutoinstall(params: UbuntuAutoinstallParams): string
   // Build the LVM layout to match Fedora kickstart sizes
   const extraLvs: string[] = [];
   if (hasLonghorn) {
-    extraLvs.push(`        - id: lv-longhorn
-          name: longhorn
-          type: lvm_partition
-          volgroup: vg0
-          size: -1
-        - id: fs-longhorn
-          type: format
-          volume: lv-longhorn
-          fstype: xfs
-        - id: mount-longhorn
-          type: mount
-          device: fs-longhorn
-          path: /var/lib/longhorn`);
+    // 6 spaces for the list item, 8 for its keys -- these are siblings of the
+    // lv-home/lv-srv entries in storage.config, which sit at 6. At 8 the
+    // rendered document is not valid YAML at all ("expected <block end>, but
+    // found '-'"), so an Ubuntu node with the longhorn role could never have
+    // installed. Ubuntu + longhorn is exactly the worker shape.
+    extraLvs.push(`      - id: lv-longhorn
+        name: longhorn
+        type: lvm_partition
+        volgroup: vg0
+        size: -1
+      - id: fs-longhorn
+        type: format
+        volume: lv-longhorn
+        fstype: xfs
+      - id: mount-longhorn
+        type: mount
+        device: fs-longhorn
+        path: /var/lib/longhorn`);
   }
   if (hasRancher) {
-    extraLvs.push(`        - id: lv-rancher
-          name: rancher
-          type: lvm_partition
-          volgroup: vg0
-          size: 20G
-        - id: fs-rancher
-          type: format
-          volume: lv-rancher
-          fstype: xfs
-        - id: mount-rancher
-          type: mount
-          device: fs-rancher
-          path: /var/lib/rancher`);
+    extraLvs.push(`      - id: lv-rancher
+        name: rancher
+        type: lvm_partition
+        volgroup: vg0
+        size: 20G
+      - id: fs-rancher
+        type: format
+        volume: lv-rancher
+        fstype: xfs
+      - id: mount-rancher
+        type: mount
+        device: fs-rancher
+        path: /var/lib/rancher`);
   }
 
   const extraLvsBlock = extraLvs.length > 0 ? "\n" + extraLvs.join("\n") : "";
@@ -81,6 +86,11 @@ export function renderUbuntuAutoinstall(params: UbuntuAutoinstallParams): string
     `curtin in-target -- bash -c 'cat > /etc/modules-load.d/k3s.conf << EOF\nbr_netfilter\noverlay\nip_conntrack\nEOF'`,
     // Sysctl for k3s networking
     `curtin in-target -- bash -c 'cat > /etc/sysctl.d/90-k3s.conf << EOF\nnet.bridge.bridge-nf-call-iptables  = 1\nnet.bridge.bridge-nf-call-ip6tables = 1\nnet.ipv4.ip_forward                 = 1\nnet.ipv6.conf.all.forwarding        = 1\nfs.inotify.max_user_instances       = 524288\nfs.inotify.max_user_watches         = 1048576\nEOF'`,
+    // IPv6 link-local = EUI-64, so a MAC-keyed DHCPv6 reservation can match: kea
+    // recovers the node's MAC from an EUI-64 link-local when the client sends a
+    // DUID-UUID (no MAC in it). NM's stable-privacy default hides the MAC and the
+    // node silently never gets its reserved address. Proven 2026-09-06.
+    `curtin in-target -- bash -c 'cat > /etc/NetworkManager/conf.d/10-ipv6-eui64.conf << EOF\n[connection]\nipv6.addr-gen-mode=eui64\nEOF'`,
     // Disable ufw firewall
     `curtin in-target -- systemctl disable ufw || true`,
     // Enable chrony/ntp
@@ -121,7 +131,20 @@ export function renderUbuntuAutoinstall(params: UbuntuAutoinstallParams): string
     `curtin in-target -- bash -c 'IP_ADDR=$(ip -4 addr show | awk "/inet / && !/127.0.0/ {split(\\$2,a,\\"/\\"); print a[1]; exit}"); curl -sf -X POST "http://${serverIp}:${httpPort}/api/progress" -H "Content-Type: application/json" -d "{\\"mac\\":\\"$(ip link show | awk "/ether/ && !/00:00:00:00/ {print \\$2; exit}")\\",\\"stage\\":\\"complete\\",\\"detail\\":\\"ready at $IP_ADDR\\"}" || true'`,
   );
 
-  const lateCommandsYaml = lateCommands.map((c) => `        - "${c}"`).join("\n");
+  // JSON.stringify, not `"${c}"`. JSON is a subset of YAML, so this produces a
+  // correctly escaped double-quoted scalar for free -- and the naive version
+  // was broken in two ways at once, for every role:
+  //
+  //   * embedded double quotes ended the scalar early
+  //     (`echo "tmpfs /tmp ..." >> /etc/fstab` -> "expected <block end>")
+  //   * the heredocs contain REAL newlines, and YAML folds newlines inside a
+  //     double-quoted scalar into spaces -- so even where it parsed, the
+  //     heredoc arrived at the target as one long line and wrote a file with
+  //     no line breaks.
+  //
+  // JSON escaping turns the newlines into \n, which YAML unescapes back to
+  // real newlines on parse, so the heredoc survives intact.
+  const lateCommandsYaml = lateCommands.map((c) => `        - ${JSON.stringify(c)}`).join("\n");
 
   return `#cloud-config
 autoinstall:
@@ -139,6 +162,30 @@ autoinstall:
     allow-pw: false
     authorized-keys:
 ${sshKeysYaml}
+  # Both address families. Without dhcp6 the installer's default is IPv4-only,
+  # so a node provisioned into a dual-stack cluster comes up with no IPv6, k3s
+  # has no v6 node-ip to bind, and it joins as an IPv4-only member of a
+  # dual-stack cluster -- which surfaces later as pods on that node being
+  # unreachable over v6 while the node itself reads Ready.
+  #
+  # The address itself comes from a kea DHCPv6 reservation keyed on MAC, the
+  # same source of truth as the v4 address, so nothing here needs to know it.
+  #
+  # optional: true matters -- it lets the install proceed if the v6 lease is
+  # slow or the VLAN has no DHCPv6 yet, rather than blocking on a timeout. The
+  # node still needs the address before k3s starts, but that is a later step's
+  # problem and failing the OS install over it would be worse.
+  # (No backticks in this comment: it lives inside a TS template literal, and a
+  # backtick here ends the literal and breaks the build.)
+  network:
+    version: 2
+    ethernets:
+      primary:
+        match:
+          name: "en*"
+        dhcp4: true
+        dhcp6: true
+        optional: true
   storage:
     config:
       - id: disk0

bastion/src/bastion/src/templates/vyos-boot.ipxe.ts

@@ -0,0 +1,53 @@
+// iPXE boot script template for VyOS network install.
+//
+// VyOS ships no unattended installer: `install image` is unconditionally
+// interactive (image_installer.py's install action takes no arguments, and
+// --no-prompt is wired only to `add`). So PXE boots the *live* system and the
+// automation is injected through live-config's `hooks` component, which fetches
+// a script over HTTP and runs it as root late in live boot.
+//
+// Unlike the Fedora/Ubuntu paths this boots a live image rather than an
+// installer, so there is no kickstart/autoinstall equivalent — see
+// routes/vyos.ts for the hook that actually drives the install.
+
+export function renderVyosInstallIpxe(params: {
+  mac: string;
+  hostname: string;
+  serverIp: string;
+  httpPort: number;
+}): string {
+  const base = `http://${params.serverIp}:${params.httpPort}`;
+
+  // Pin the boot NIC by MAC. live-boot otherwise scans for the first
+  // *connected* interface, and on a multi-NIC box that race is lost by
+  // whichever port negotiates slowest: on the Protectli VP2440 the SFP+
+  // pair links first, so live-boot picked the fiber ports (which have no
+  // DHCP), burned 15s per port, and gave up with "Unable to find a live
+  // file system on the network" -- while the copper port that actually PXE
+  // booted came up at 4.6s and was never tried.
+  //
+  // live-boot's Device_from_bootif() strips the "01-" and matches the MAC
+  // against /sys/class/net/*. params.mac is the dispatch key, i.e. exactly
+  // the NIC that PXE booted -- more reliable than iPXE's ${net0} on a box
+  // where the booting NIC may not be net0.
+  const bootif = `01-${params.mac.toLowerCase().replace(/:/g, "-")}`;
+
+  // Deliberately NOT passing `nonetworking` (present in VyOS's own PXE docs):
+  // live-config's hook component needs networking up to fetch the hook over
+  // HTTP. Also no `console=ttyS0` — on hardware without a physical UART that
+  // costs 30s at every systemd boot phase.
+  return `#!ipxe
+
+echo
+echo =============================================
+echo   Lab PXE Bastion - INSTALLING VyOS
+echo   Target: ${params.hostname}
+echo   MAC:    ${params.mac}
+echo =============================================
+echo
+
+kernel ${base}/vyos-vmlinuz boot=live nopersistence noautologin BOOTIF=${bootif} fetch=${base}/vyos-filesystem.squashfs live-config.hooks=${base}/vyos/autoinstall.sh?mac=${params.mac}
+initrd ${base}/vyos-initrd
+boot
+`;
+}

bastion/src/bastion/src/templates/vyos-config-spec.ts

@@ -0,0 +1,352 @@
+// Builds the VyOS configuration spec applied by the autoinstall hook.
+//
+// We deliberately do NOT emit a config.boot file as text. A config.boot carries a
+// `vyos-config-version` trailer; without a trailer matching the running image,
+// VyOS runs its migration scripts from version 0 on first boot. Instead the hook
+// loads the image's own /opt/vyatta/etc/config.boot.default through vyos.configtree
+// and applies these set operations on top, so syntax and version trailer always
+// match the exact image being installed.
+
+import type { VyosInstallSpec } from "@lab/shared";
+
+export interface VyosSetOp {
+  path: string[];
+  value?: string;
+  /** false appends to a multi-value node (e.g. bond members) instead of replacing. */
+  replace?: boolean;
+}
+
+export interface VyosConfigSpec {
+  hostname: string;
+  /** "" means accept the installer default (the running image's version string). */
+  imageName: string;
+  password: string;
+  console: "K" | "S";
+  /** Target disk name (e.g. "nvme0n1"); "" accepts the installer's first-disk default. */
+  disk: string;
+  /**
+   * IP the driver should report in the "complete" callback ("ready at <ip>" —
+   * the exact format routes/api.ts parses installed.ip from). The mgmt
+   * address when static; "" means detect the live DHCP address at runtime.
+   */
+  reportAddress: string;
+  /** Whether to accept RAID-1 when the installer finds more than one disk. */
+  raid: boolean;
+  /** Overwrite the installed config.boot with the generated one on reinstall. */
+  freshConfig: boolean;
+  sets: VyosSetOp[];
+  /** Paths that are VyOS tag nodes — must be marked as such in the ConfigTree. */
+  tags: string[][];
+}
+
+/**
+ * Normalise a target disk to the form the installer expects.
+ *
+ * find_disks() enumerates via `lsblk -Jbp` (-p = full paths), so its valid
+ * responses are "/dev/mmcblk0"-style. A bare "mmcblk0" is rejected by
+ * ask_input()'s valid_responses check and re-prompts forever.
+ */
+function normalizeDiskPath(value: string | undefined): string {
+  const raw = (value ?? "").trim();
+  if (raw === "") return "";
+  return raw.startsWith("/dev/") ? raw : `/dev/${raw}`;
+}
+
+/** Sentinel marking a value that lives in Pulumi config, not in the bundle. */
+const SECRET_PREFIX = "@secret:";
+
+/**
+ * Enable the VyOS HTTP API so the router is manageable the moment it boots.
+ *
+ * This belongs at install time rather than in the Pulumi model: the model is
+ * applied THROUGH this API, so a router that lacks it cannot be brought under
+ * management without a hand-run change on a live firewall. It is also why the
+ * model excludes `service https` outright -- a provider able to rewrite its own
+ * transport can lock itself out permanently.
+ *
+ * `listen-address` is always set. Leaving it unbound would expose a
+ * config-write endpoint on every segment the router touches, the WAN included.
+ */
+function apiSets(apiKey: string, listenAddress: string): VyosSetOp[] {
+  const sets: VyosSetOp[] = [
+    { path: ["service", "https", "api", "keys", "id", "pulumi", "key"], value: apiKey },
+    { path: ["service", "https", "api", "rest"] },
+  ];
+  if (listenAddress !== "") {
+    sets.push({ path: ["service", "https", "listen-address"], value: listenAddress });
+  }
+  return sets;
+}
+
+/** Tag nodes introduced by the API config, needed by the installer's ConfigTree. */
+const API_TAGS: string[][] = [["service", "https", "api", "keys", "id"]];
+
+/**
+ * The address to bind the API to: an explicit choice, else the management
+ * address with its prefix length stripped. Under DHCP there is no address to
+ * bind at build time, so the caller must pass one or the listener stays unbound
+ * and the API is not enabled at all.
+ */
+function apiListenAddress(spec: VyosInstallSpec, mgmtAddress: string): string {
+  if (spec.apiListenAddress !== undefined && spec.apiListenAddress !== "") {
+    return spec.apiListenAddress;
+  }
+  return mgmtAddress.includes("/") ? (mgmtAddress.split("/")[0] ?? "") : "";
+}
+
+/**
+ * Use a Pulumi-rendered bundle as the router's config verbatim.
+ *
+ * Secret-valued nodes are dropped rather than installed with their sentinel
+ * text: writing `@secret:pppoePassword` into config.boot would look configured
+ * while being wrong, which is worse than being absent. The router comes up
+ * without those values and the first `pulumi up` fills them in.
+ *
+ * `system host-name` is forced to the hostname the install was asked for. The
+ * bundle carries the name of whichever router it was exported from, and
+ * installing vyos001's hostname onto vyos002 would collide on the network.
+ */
+function buildFromBundle(
+  params: { hostname: string; defaultPassword: string; disk?: string | undefined },
+  spec: VyosInstallSpec,
+  bundle: NonNullable<VyosInstallSpec["bundle"]>,
+  mgmtAddress: string,
+): VyosConfigSpec {
+  const sets: VyosSetOp[] = [];
+  const dropped: string[] = [];
+  for (const op of bundle.sets) {
+    if (op.value !== undefined && op.value.startsWith(SECRET_PREFIX)) {
+      dropped.push(op.path.join(" "));
+      continue;
+    }
+    if (op.path.length === 2 && op.path[0] === "system" && op.path[1] === "host-name") {
+      continue;
+    }
+    sets.push({
+      path: op.path,
+      ...(op.value === undefined ? {} : { value: op.value }),
+      ...(op.replace === undefined ? {} : { replace: op.replace }),
+    });
+  }
+  sets.unshift({ path: ["system", "host-name"], value: params.hostname });
+
+  if (dropped.length > 0) {
+    console.warn(
+      `vyos ${params.hostname}: ${dropped.length} secret-valued node(s) left unset by the ` +
+        `bundle; run \`pulumi up\` to supply them: ${dropped.join(", ")}`,
+    );
+  }
+
+  const tags = [...bundle.tags];
+  const api = enableApi(spec, params.hostname, mgmtAddress);
+  if (api.length > 0) {
+    sets.push(...api);
+    tags.push(...API_TAGS);
+  }
+
+  return {
+    hostname: params.hostname,
+    imageName: "",
+    password: spec.password ?? params.defaultPassword,
+    console: "K",
+    disk: normalizeDiskPath(params.disk),
+    reportAddress: mgmtAddress.includes("/") ? (mgmtAddress.split("/")[0] ?? "") : "",
+    raid: false,
+    freshConfig: spec.freshConfig ?? false,
+    sets,
+    tags,
+  };
+}
+
+/**
+ * The API config for this install, or nothing when it cannot be enabled safely.
+ *
+ * Refusing to enable it unbound is deliberate. Under DHCP there is no address
+ * known at build time, and the alternative -- binding to every interface --
+ * would publish a config-write endpoint on the WAN. Better to leave the router
+ * SSH-only and say so than to open it everywhere.
+ */
+function enableApi(spec: VyosInstallSpec, hostname: string, mgmtAddress: string): VyosSetOp[] {
+  if (spec.apiKey === undefined || spec.apiKey === "") return [];
+  const listen = apiListenAddress(spec, mgmtAddress);
+  if (listen === "") {
+    console.warn(
+      `vyos ${hostname}: --vyos-api-key given but no address to bind to ` +
+        `(management is "${mgmtAddress}"). Pass --vyos-api-listen <addr>; the HTTP API ` +
+        `has NOT been enabled, so Pulumi cannot manage this router yet.`,
+    );
+    return [];
+  }
+  return apiSets(spec.apiKey, listen);
+}
+
+export function buildVyosConfigSpec(params: {
+  hostname: string;
+  spec?: VyosInstallSpec | undefined;
+  defaultPassword: string;
+  sshKeys?: string[] | undefined;
+  disk?: string | undefined;
+}): VyosConfigSpec {
+  const spec = params.spec ?? {};
+  const mgmt = spec.mgmtInterface ?? "eth0";
+  const mgmtAddress = spec.mgmtAddress ?? "dhcp";
+
+  // A rendered bundle replaces the derived config entirely. Deriving a second
+  // opinion alongside it is the drift the bundle exists to prevent: Pulumi and
+  // labctl would each believe they knew the router's config, and the box would
+  // end up with whichever ran last.
+  if (spec.bundle !== undefined) {
+    return buildFromBundle(params, spec, spec.bundle, mgmtAddress);
+  }
+  const bondMembers = spec.bondMembers ?? [];
+  const vlans = spec.vlans ?? [];
+
+  const sets: VyosSetOp[] = [];
+  const tags: string[][] = [
+    ["interfaces", "ethernet"],
+    ["system", "login", "user"],
+  ];
+
+  const hwIds = spec.hwIds ?? {};
+  const pinHwId = (iface: string): void => {
+    const mac = hwIds[iface];
+    if (mac !== undefined && mac !== "") {
+      sets.push({ path: ["interfaces", "ethernet", iface, "hw-id"], value: mac });
+    }
+  };
+
+  sets.push({ path: ["system", "host-name"], value: params.hostname });
+
+  // Management interface — the NIC that PXE booted, left untagged and unbonded.
+  sets.push({ path: ["interfaces", "ethernet", mgmt, "address"], value: mgmtAddress });
+  pinHwId(mgmt);
+
+  // Tagged management VLAN on the PXE port. Emitted regardless of bonding, so
+  // the box stays reachable on the management VLAN while still booting untagged
+  // on whichever VLAN the bastion's proxy DHCP serves.
+  const mgmtVlan = spec.mgmtVlan;
+  if (mgmtVlan !== undefined) {
+    tags.push(["interfaces", "ethernet", mgmt, "vif"]);
+    const vif = ["interfaces", "ethernet", mgmt, "vif", String(mgmtVlan.id)];
+    sets.push({ path: [...vif, "address"], value: mgmtVlan.address });
+    if (mgmtVlan.description !== undefined && mgmtVlan.description !== "") {
+      sets.push({ path: [...vif, "description"], value: mgmtVlan.description });
+    }
+  }
+
+  // LACP bond. Members must exclude the PXE NIC; firmware PXE cannot run over LACP.
+  const bonded = bondMembers.length > 0;
+  if (bonded) {
+    tags.push(["interfaces", "bonding"]);
+    sets.push({ path: ["interfaces", "bonding", "bond0", "mode"], value: "802.3ad" });
+    sets.push({ path: ["interfaces", "bonding", "bond0", "hash-policy"], value: "layer2+3" });
+    for (const member of bondMembers) {
+      sets.push({
+        path: ["interfaces", "bonding", "bond0", "member", "interface"],
+        value: member,
+        replace: false,
+      });
+      pinHwId(member);
+    }
+    // Address on the trunk's native/untagged VLAN.
+    if (spec.bondAddress !== undefined && spec.bondAddress !== "") {
+      sets.push({ path: ["interfaces", "bonding", "bond0", "address"], value: spec.bondAddress });
+    }
+  }
+
+  // VRRP groups accumulate here; emitted (plus a sync group) after the VLANs.
+  // interface accepts dotted vifs (constraint regex `[0-9]+(.\d+)?`), address
+  // is a tag node (the VIP is the tag value itself), vrid range is 1-255.
+  const vrrpGroups: Array<{ name: string; iface: string; vrid: number; vip: string }> = [];
+  if (bonded && spec.bondVrrp !== undefined && spec.bondVrrp !== "") {
+    // vrid 1 for the untagged group: the native VLAN is never a vif, so this
+    // cannot collide with a vlan-id-derived vrid.
+    vrrpGroups.push({ name: "native", iface: "bond0", vrid: 1, vip: spec.bondVrrp });
+  }
+
+  // Tagged VLAN sub-interfaces hang off the bond when there is one, else off mgmt.
+  const parent = bonded
+    ? ["interfaces", "bonding", "bond0"]
+    : ["interfaces", "ethernet", mgmt];
+  if (vlans.length > 0) {
+    tags.push([...parent, "vif"]);
+    const parentName = bonded ? "bond0" : mgmt;
+    for (const vlan of vlans) {
+      const vif = [...parent, "vif", String(vlan.id)];
+      sets.push({ path: [...vif, "address"], value: vlan.address });
+      if (vlan.description !== undefined && vlan.description !== "") {
+        sets.push({ path: [...vif, "description"], value: vlan.description });
+      }
+      if (vlan.vrrp !== undefined && vlan.vrrp !== "") {
+        vrrpGroups.push({
+          name: `vlan${vlan.id}`,
+          iface: `${parentName}.${vlan.id}`,
+          vrid: vlan.id,
+          vip: vlan.vrrp,
+        });
+      }
+    }
+  }
+
+  // Emit VRRP groups plus one sync group so all VLANs fail over together —
+  // without it a single-link event could split mastership across the pair.
+  if (vrrpGroups.length > 0) {
+    tags.push(["high-availability", "vrrp", "group"]);
+    tags.push(["high-availability", "vrrp", "sync-group"]);
+    const priority = String(spec.vrrpPriority ?? 100);
+    for (const g of vrrpGroups) {
+      const base = ["high-availability", "vrrp", "group", g.name];
+      sets.push({ path: [...base, "interface"], value: g.iface });
+      sets.push({ path: [...base, "vrid"], value: String(g.vrid) });
+      sets.push({ path: [...base, "priority"], value: priority });
+      // address is a tag node: the VIP is the path's final segment, no value.
+      sets.push({ path: [...base, "address", g.vip] });
+      tags.push([...base, "address"]);
+      sets.push({
+        path: ["high-availability", "vrrp", "sync-group", "MAIN", "member"],
+        value: g.name,
+        replace: false,
+      });
+    }
+  }
+
+  sets.push({ path: ["service", "ssh", "port"], value: "22" });
+
+  const sshKeys = params.sshKeys ?? [];
+  if (sshKeys.length > 0) {
+    tags.push(["system", "login", "user", "vyos", "authentication", "public-keys"]);
+    sshKeys.forEach((entry, index) => {
+      const parts = entry.trim().split(/\s+/);
+      const type = parts[0] ?? "";
+      const key = parts[1] ?? "";
+      if (!type.startsWith("ssh-") && !type.startsWith("ecdsa-")) return;
+      if (!key) return;
+      const name = parts[2] ?? `lab-key-${index}`;
+      const base = ["system", "login", "user", "vyos", "authentication", "public-keys", name];
+      sets.push({ path: [...base, "type"], value: type });
+      sets.push({ path: [...base, "key"], value: key });
+    });
+  }
+
+  // Enabled here too, not just for bundle installs: every VyOS this bastion
+  // provisions should be manageable from first boot.
+  const api = enableApi(spec, params.hostname, mgmtAddress);
+  if (api.length > 0) {
+    sets.push(...api);
+    tags.push(...API_TAGS);
+  }
+
+  return {
+    hostname: params.hostname,
+    imageName: "",
+    password: spec.password ?? params.defaultPassword,
+    console: "K",
+    disk: normalizeDiskPath(params.disk),
+    // Static mgmt address wins; under DHCP the driver detects the live IP.
+    reportAddress: mgmtAddress.includes("/") ? (mgmtAddress.split("/")[0] ?? "") : "",
+    raid: false,
+    freshConfig: spec.freshConfig ?? false,
+    sets,
+    tags,
+  };
+}

bastion/src/bastion/src/templates/vyos-install.py.ts

@@ -0,0 +1,514 @@
+// Renders the Python program that performs the unattended VyOS install.
+//
+// It runs as root inside the live system, fetched and executed by live-config's
+// `hooks` component (see vyos-boot.ipxe.ts). It does three things:
+//   1. builds config.boot from the image's own default via vyos.configtree
+//   2. drives the interactive `install image` through a pty
+//   3. reports progress back to the bastion, then reboots
+//
+// A pty is used rather than piping stdin because the installer reads the
+// password through getpass(), which opens /dev/tty directly and would ignore a
+// pipe. Prompts are matched by text rather than replayed positionally: the
+// installer skips the boot-config question when it finds a previous
+// installation, so a fixed answer sequence desyncs on reinstall.
+
+import type { VyosConfigSpec } from "./vyos-config-spec.js";
+
+export function renderVyosInstallPy(params: {
+  spec: VyosConfigSpec;
+  mac: string;
+  serverIp: string;
+  httpPort: number;
+  role: string;
+}): string {
+  // Base64 so arbitrary values (passwords, descriptions, SSH keys) can never
+  // terminate the Python string literal that carries them.
+  const specB64 = Buffer.from(JSON.stringify(params.spec), "utf-8").toString("base64");
+
+  return `#!/usr/bin/env python3
+"""Unattended VyOS install driver -- generated by the lab PXE bastion."""
+
+import base64
+import json
+import os
+import pty
+import re
+import select
+import subprocess
+import sys
+import time
+import urllib.request
+
+SPEC = json.loads(base64.b64decode("${specB64}").decode("utf-8"))
+BASTION = "http://${params.serverIp}:${params.httpPort}"
+MAC = "${params.mac}"
+ROLE = ${JSON.stringify(params.role ?? "vanilla")}
+
+INSTALLER = "/usr/libexec/vyos/op_mode/image_installer.py"
+CONFIG_DIR = "/opt/vyatta/etc/config"
+# The installer copies the rootfs from the boot MEDIUM path -- which only a
+# CD/USB boot provides. With fetch= (HTTP netboot) nothing is mounted there
+# (verified in VM: Errno 2), so the squashfs must be linked or re-fetched into
+# place before 'install image' runs.
+ROOTFS_EXPECTED = "/usr/lib/live/mount/medium/live/filesystem.squashfs"
+SQUASHFS_URL = "http://${params.serverIp}:${params.httpPort}/vyos-filesystem.squashfs"
+# The live-config hook runs BEFORE vyos-router creates the /opt/vyatta compat
+# path, so the squashfs's own location must be tried too (verified in VM: only
+# /usr/share/vyos/config.boot.default exists at hook time).
+DEFAULT_CONFIG_CANDIDATES = [
+    "/opt/vyatta/etc/config.boot.default",
+    "/usr/share/vyos/config.boot.default",
+]
+STALL_TIMEOUT = 900  # seconds without installer output before giving up
+
+
+def detect_ip():
+    """Best-effort local IP as seen on the route toward the bastion.
+
+    Matches Fedora's semantics (IP captured during install): under DHCP the
+    installed system will renew on the same NIC/subnet the live env used.
+    """
+    import socket
+    try:
+        s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
+        s.connect(("${params.serverIp}", ${params.httpPort}))
+        ip = s.getsockname()[0]
+        s.close()
+        return ip
+    except Exception:
+        return ""
+
+
+class LogStreamer:
+    """Stream install output to the bastion's /api/log so 'labctl provision
+    logs -f' works live for VyOS, like Anaconda's syslog does for Fedora.
+
+    Strictly best-effort: a failed POST drops the batch and must never stall
+    the pty read loop or fail the install.
+    """
+
+    ANSI = re.compile(rb"\\x1b\\[[0-9;?]*[a-zA-Z]|\\x1b[=>]|\\r")
+
+    def __init__(self):
+        self.partial = b""
+        self.pending = []
+        self.last_flush = time.time()
+
+    def feed(self, chunk):
+        """Raw pty bytes: split into lines, strip ANSI noise, queue."""
+        self.partial += chunk
+        while b"\\n" in self.partial:
+            raw, self.partial = self.partial.split(b"\\n", 1)
+            text = self.ANSI.sub(b"", raw).decode("utf-8", "replace").rstrip()
+            if text:
+                self.pending.append(text)
+        self.maybe_flush()
+
+    def line(self, text):
+        """A driver-originated message (already a clean string)."""
+        self.pending.append(text)
+        self.maybe_flush()
+
+    def maybe_flush(self):
+        if len(self.pending) >= 20 or (self.pending and time.time() - self.last_flush >= 2):
+            self.flush()
+
+    def flush(self):
+        if not self.pending:
+            return
+        batch, self.pending = self.pending[:200], self.pending[200:]
+        self.last_flush = time.time()
+        try:
+            body = json.dumps({"mac": MAC, "lines": batch}).encode()
+            req = urllib.request.Request(
+                BASTION + "/api/log",
+                data=body,
+                headers={"Content-Type": "application/json"},
+            )
+            urllib.request.urlopen(req, timeout=5).read()
+        except Exception:
+            pass
+
+
+STREAM = LogStreamer()
+
+
+def say(msg):
+    """Print locally and stream to the bastion log buffer."""
+    print(msg)
+    STREAM.line(str(msg))
+
+
+def report(stage, detail=""):
+    """Best-effort progress callback; never fatal."""
+    STREAM.flush()
+    try:
+        body = json.dumps({"mac": MAC, "stage": stage, "detail": detail}).encode()
+        req = urllib.request.Request(
+            BASTION + "/api/progress",
+            data=body,
+            headers={"Content-Type": "application/json"},
+        )
+        urllib.request.urlopen(req, timeout=5).read()
+    except Exception:
+        pass
+
+
+def build_config():
+    """Apply our set operations onto the image's own default config.
+
+    Using config.boot.default as the base keeps the vyos-config-version trailer
+    consistent with the running image, so first boot does not run migrations.
+    """
+    from vyos.configtree import ConfigTree
+
+    default_config = next(
+        (p for p in DEFAULT_CONFIG_CANDIDATES if os.path.exists(p)), None)
+    if default_config is None:
+        raise FileNotFoundError(
+            "no config.boot.default found (tried %s)" % ", ".join(DEFAULT_CONFIG_CANDIDATES))
+    say("base config: %s" % default_config)
+
+    with open(default_config) as handle:
+        config = ConfigTree(handle.read())
+
+    for op in SPEC["sets"]:
+        replace = op.get("replace", True)
+        if "value" in op and op["value"] is not None:
+            config.set(op["path"], value=op["value"], replace=replace)
+        else:
+            config.set(op["path"])
+
+    # Tag nodes must be marked after the nodes exist, as the installer itself does.
+    for tag in SPEC["tags"]:
+        try:
+            config.set_tag(tag)
+        except Exception as err:
+            say("warning: set_tag %s failed: %s" % (tag, err))
+
+    os.makedirs(CONFIG_DIR, exist_ok=True)
+    target = os.path.join(CONFIG_DIR, "config.boot")
+
+    # Re-attach the vyos-config-version footer: ConfigTree.to_string() emits
+    # only the config body, and a config without the footer is treated as
+    # ancient -- the boot migrator then runs every migration over it and (as
+    # observed in the VM test) crashes in system/31-to-32. Building the footer
+    # from the running system pins it to the exact image being installed.
+    body = config.to_string()
+    try:
+        from vyos.component_version import version_info_from_system
+        info = version_info_from_system()
+        info.update_config_body(body)
+        info.write(target)
+        say("wrote %s (footer: %s)" % (target, info.release))
+    except Exception as err:
+        say("warning: version footer failed (%s); writing bare config" % err)
+        with open(target, "w") as handle:
+            handle.write(body)
+    return target
+
+
+def find_live_squashfs():
+    """Locate the squashfs live-boot fetched, without walking into the mounted
+    rootfs or overlay (each would mean traversing the entire OS tree)."""
+    explicit = [
+        "/run/live/medium/live/filesystem.squashfs",
+        "/lib/live/mount/medium/live/filesystem.squashfs",
+    ]
+    for path in explicit:
+        if os.path.isfile(path) and os.path.getsize(path) > 0:
+            return path
+    for root in ("/run/live", "/lib/live/mount", "/usr/lib/live/mount"):
+        for dirpath, dirs, files in os.walk(root):
+            depth = dirpath.count(os.sep) - root.count(os.sep)
+            dirs[:] = [d for d in dirs
+                       if d not in ("rootfs", "overlay")
+                       and not d.endswith(".squashfs")
+                       and depth < 3]
+            if "filesystem.squashfs" in files:
+                path = os.path.join(dirpath, "filesystem.squashfs")
+                if os.path.isfile(path) and os.path.getsize(path) > 0:
+                    return path
+    return None
+
+
+def ensure_rootfs():
+    """Make FILE_ROOTFS_SRC exist so the installer can copy the system image."""
+    if os.path.isfile(ROOTFS_EXPECTED) and os.path.getsize(ROOTFS_EXPECTED) > 0:
+        return
+    src = find_live_squashfs()
+    if src is None:
+        say("squashfs not in live mounts; re-fetching %s" % SQUASHFS_URL)
+        src = "/tmp/filesystem.squashfs"
+        urllib.request.urlretrieve(SQUASHFS_URL, src)
+    os.makedirs(os.path.dirname(ROOTFS_EXPECTED), exist_ok=True)
+    if os.path.lexists(ROOTFS_EXPECTED):
+        os.remove(ROOTFS_EXPECTED)
+    os.symlink(src, ROOTFS_EXPECTED)
+    say("rootfs source: %s -> %s" % (ROOTFS_EXPECTED, src))
+
+
+def build_rules():
+    """Prompt -> response table for the interactive installer."""
+    password = SPEC["password"].encode() + b"\\n"
+    image_name = SPEC["imageName"].encode() + b"\\n"
+    disk = SPEC["disk"].encode() + b"\\n"
+    console = SPEC["console"].encode() + b"\\n"
+    raid = (b"yes\\n" if SPEC["raid"] else b"no\\n")
+
+    return [
+        (re.compile(rb"Would you like to continue\\?"), b"yes\\n"),
+        (re.compile(rb"What would you like to name this image\\?"), image_name),
+        (re.compile(rb"Please confirm password for the .vyos. user:"), password),
+        (re.compile(rb"Please enter a password for the .vyos. user:"), password),
+        (re.compile(rb"What console should be used by default"), console),
+        # Three RAID variants: "configure RAID-1 mirroring?", "...on them?",
+        # and "choose two disks for RAID-1 mirroring?" -- all default to YES,
+        # so a missed one both hangs the install and risks an unwanted mirror.
+        (re.compile(rb"Would you like to [^?]*RAID-1 mirroring"), raid),
+        (re.compile(rb"Installation will delete all data on (?:the drive|both drives)\\. Continue\\?"), b"yes\\n"),
+        (re.compile(rb"Which one should be used for installation\\?"), disk),
+        (re.compile(rb"Would you like to use all the free space on the drive\\?"), b"yes\\n"),
+        (re.compile(rb"Which file would you like as boot config\\?"), b"1\\n"),
+        # Reinstall path only (search_previous_installation): carrying the old
+        # /config and SSH host keys forward is VyOS's "reinstall without losing
+        # data". Always yes -- freshConfig replaces config.boot afterwards, so
+        # answering no here would also discard non-config data under /config.
+        (re.compile(rb"Would you like to copy data to the new image\\?"), b"yes\\n"),
+        (re.compile(rb"Would you like to copy the encrypted config to the new image\\?"), b"yes\\n"),
+        # More than one previous image found -- take the first offered.
+        (re.compile(rb"From which image would you like to save config information\\?"), b"1\\n"),
+        (re.compile(rb"From which image would you like to copy the encrypted config\\?"), b"1\\n"),
+    ]
+
+
+def run_installer():
+    """Drive image_installer.py over a pty, answering prompts as they appear."""
+    rules = build_rules()
+    master, slave = pty.openpty()
+
+    proc = subprocess.Popen(
+        [INSTALLER, "--action", "install"],
+        stdin=slave,
+        stdout=slave,
+        stderr=slave,
+        close_fds=True,
+        preexec_fn=os.setsid,
+    )
+    os.close(slave)
+
+    buf = b""
+    transcript = b""   # rolling tail of everything the installer printed
+    last_output = time.time()
+
+    while True:
+        ready, _, _ = select.select([master], [], [], 1.0)
+
+        if ready:
+            try:
+                chunk = os.read(master, 4096)
+            except OSError:
+                break
+            if not chunk:
+                break
+
+            sys.stdout.buffer.write(chunk)
+            sys.stdout.buffer.flush()
+            buf += chunk
+            transcript = (transcript + chunk)[-8000:]
+            STREAM.feed(chunk)
+            last_output = time.time()
+
+            # Answer every prompt currently in the buffer, earliest first, so
+            # ordering is preserved even when the installer skips questions --
+            # and so a single chunk carrying two prompts gets both answers.
+            while True:
+                best = None
+                for pattern, response in rules:
+                    found = pattern.search(buf)
+                    if found and (best is None or found.start() < best[0].start()):
+                        best = (found, response)
+                if best is None:
+                    break
+                found, response = best
+                os.write(master, response)
+                transcript = (transcript + b"\\n>>> answered: " + response)[-8000:]
+                STREAM.line(">>> answered: " + response.decode("utf-8", "replace").strip())
+                buf = buf[found.end():]
+
+            # Bound memory if the installer emits a lot without prompting.
+            if len(buf) > 65536:
+                buf = buf[-8192:]
+
+        elif proc.poll() is not None:
+            break
+
+        STREAM.maybe_flush()
+
+        if time.time() - last_output > STALL_TIMEOUT:
+            proc.kill()
+            raise SystemExit("installer produced no output for %ds" % STALL_TIMEOUT)
+
+    os.close(master)
+    return proc.wait(), transcript.decode("utf-8", "replace")
+
+
+def ensure_network_boot_first():
+    """Keep network boot first so the bastion intercepts every reboot.
+
+    Port of the Fedora kickstart's %post efibootmgr step (install.ks.ts) --
+    what makes reprovision-by-reboot work. Best-effort: skipped on BIOS boots
+    or when efibootmgr is absent. Runs from the live env after the installer;
+    efibootmgr edits NVRAM, not the disk, so installer cleanup is irrelevant.
+    """
+    import shutil
+    if not os.path.isdir("/sys/firmware/efi") or shutil.which("efibootmgr") is None:
+        say("boot order: skipped (BIOS boot or efibootmgr missing)")
+        return
+    try:
+        out = subprocess.run(["efibootmgr"], capture_output=True, text=True, timeout=30).stdout
+        order = []
+        network_entry = None
+        for line in out.splitlines():
+            m = re.match(r"^BootOrder:\\s*(.*)$", line)
+            if m:
+                order = [x.strip() for x in m.group(1).split(",") if x.strip()]
+                continue
+            m = re.match(r"^Boot([0-9A-Fa-f]{4})\\*?\\s+(.*)$", line)
+            if m and network_entry is None:
+                if re.search(r"network|pxe|ipv4|ipv6|http", m.group(2), re.IGNORECASE):
+                    network_entry = m.group(1).upper()
+        if network_entry is None or not order:
+            say("boot order: no network boot entry found; leaving as is")
+            return
+        new_order = [network_entry] + [x for x in order if x.upper() != network_entry]
+        if [x.upper() for x in order] == [x.upper() for x in new_order]:
+            say("boot order: network entry Boot%s already first" % network_entry)
+            return
+        subprocess.run(["efibootmgr", "-o", ",".join(new_order)],
+                       capture_output=True, timeout=30)
+        say("boot order: moved network entry Boot%s first" % network_entry)
+    except Exception as err:
+        say("warning: boot order adjustment failed: %s" % err)
+
+
+def with_target_mounted(fn):
+    """Mount the installed root partition, call fn(rw_dir), always unmount.
+
+    The installer has unmounted and cleaned the target by the time this runs,
+    so the block device is free. The partition holding boot/<image>/rw is the
+    VyOS root; the glob also yields the installed image's rw dir directly.
+    """
+    import glob
+    disk = SPEC["disk"]
+    if not disk:
+        # No pinned disk (installer picked the default) -- enumerate all disks.
+        candidates = ["/dev/" + b for b in os.listdir("/sys/block")
+                      if not b.startswith(("loop", "ram", "zram", "sr"))]
+    else:
+        candidates = [disk]
+
+    mnt = "/mnt/lab-target"
+    os.makedirs(mnt, exist_ok=True)
+    for dev in candidates:
+        name = os.path.basename(dev)
+        parts = sorted(p for p in os.listdir("/sys/block/%s" % name)
+                       if p.startswith(name)) if os.path.isdir("/sys/block/%s" % name) else []
+        for part in parts:
+            pdev = "/dev/" + part
+            if subprocess.run(["mount", pdev, mnt], capture_output=True).returncode != 0:
+                continue
+            try:
+                rw_dirs = glob.glob(os.path.join(mnt, "boot", "*", "rw"))
+                if rw_dirs:
+                    fn(rw_dirs[0])
+                    return True
+            finally:
+                subprocess.run(["umount", mnt], capture_output=True)
+    return False
+
+
+def post_install_target_steps():
+    """Metadata + optional fresh-config overwrite inside the installed image."""
+    def apply(rw_dir):
+        config_dir = os.path.join(rw_dir, "opt/vyatta/etc/config")
+        os.makedirs(config_dir, exist_ok=True)
+
+        # /config/lab-provisioned -- survives VyOS image upgrades. Mirrors the
+        # Fedora kickstart's /etc/lab-provisioned.
+        try:
+            with open(os.path.join(config_dir, "lab-provisioned"), "w") as handle:
+                handle.write("hostname=%s\\n" % SPEC["hostname"])
+                handle.write("role=%s\\n" % ROLE)
+                handle.write("provisioned=%s\\n" % time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()))
+                handle.write("bastion=%s\\n" % BASTION)
+            say("wrote /config/lab-provisioned")
+        except Exception as err:
+            say("warning: lab-provisioned metadata failed: %s" % err)
+
+        # freshConfig: make the bastion-generated config win over the previous
+        # installation's carried-forward config. Explicit intent -- failure is
+        # fatal (raised out of with_target_mounted).
+        if SPEC.get("freshConfig"):
+            import shutil
+            shutil.copyfile(os.path.join(CONFIG_DIR, "config.boot"),
+                            os.path.join(config_dir, "config.boot"))
+            say("freshConfig: replaced installed config.boot with generated config")
+
+    mounted = with_target_mounted(apply)
+    if not mounted:
+        if SPEC.get("freshConfig"):
+            raise RuntimeError("freshConfig requested but installed root partition not found")
+        say("warning: installed root partition not found; skipping metadata")
+
+
+def main():
+    report("vyos-install", "building config.boot")
+    try:
+        build_config()
+    except Exception as err:
+        report("error", "config generation failed: %s" % err)
+        raise
+
+    report("vyos-install", "staging rootfs for installer")
+    try:
+        ensure_rootfs()
+    except Exception as err:
+        report("error", "rootfs staging failed: %s" % err)
+        raise
+
+    report("vyos-install", "running install image")
+    code, transcript = run_installer()
+
+    if code != 0:
+        # Surface the installer's last words in bastion progress -- the console
+        # they were printed on is usually invisible during unattended installs.
+        report("error", "install image exited %d | tail: %s" % (code, transcript[-4000:]))
+        raise SystemExit(code)
+
+    report("post-install", "boot order + metadata")
+    ensure_network_boot_first()
+    try:
+        post_install_target_steps()
+    except Exception as err:
+        report("error", "post-install target steps failed: %s" % err)
+        raise
+
+    # "complete" is the stage the bastion uses to move a machine out of the
+    # install queue into installed state, and "ready at <ip>" is the exact
+    # detail format it parses installed.ip from -- see routes/api.ts.
+    ip = SPEC.get("reportAddress") or detect_ip()
+    report("complete", "ready at %s" % ip if ip else "VyOS installed, rebooting")
+    os.system("sync")
+    # --force: this driver is a child of live-config.service, whose start job is
+    # still running -- a normal reboot deadlocks waiting for it (verified in VM:
+    # shutdown blocked >1min on "start job is running for live-config"). The
+    # installer has already unmounted and cleaned the target, so an immediate
+    # reboot is safe.
+    os.system("systemctl reboot --force")
+
+
+if __name__ == "__main__":
+    main()
+`;
+}

bastion/src/bastion/tests/arch-dispatch.test.ts

@@ -1,291 +0,0 @@
-// aarch64 support in the PXE dispatch path.
-//
-// The x86_64 side is pinned separately by ipxe-x86-regression.test.ts.
-
-import { describe, it, expect, beforeEach, afterEach } from "vitest";
-import { mkdirSync, rmSync } from "node:fs";
-import { join } from "node:path";
-import { tmpdir } from "node:os";
-import type { BastionConfig, BastionState, HardwareInfo } from "@lab/shared";
-import { createApp } from "../src/server.js";
-import { resolveArch } from "../src/routes/dispatch.js";
-import { renderDnsmasqConf } from "../src/templates/dnsmasq.conf.js";
-import type { FastifyInstance } from "fastify";
-import type { StateManager } from "../src/services/state.js";
-
-function createTestConfig(testDir: string): BastionConfig {
-  return {
-    fedoraVersion: "43",
-    arch: "x86_64",
-    httpPort: 0,
-    timezone: "Europe/London",
-    locale: "en_GB.UTF-8",
-    bastionDir: testDir,
-    domain: "test.local",
-    dhcpMode: "proxy",
-    dhcpRangeStart: "",
-    dhcpRangeEnd: "",
-    ubuntuVersion: "26.04",
-    ubuntuMirror: "https://releases.ubuntu.com/26.04",
-    iface: "eth0",
-    serverIp: "10.0.0.1",
-    network: "10.0.0.0",
-    gateway: "10.0.0.1",
-    sshKeys: ["ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAITEST test@test"],
-    adminUser: "testadmin",
-    syslogPort: 15514,
-    skipDnsmasq: true,
-    skipArtifacts: true,
-    fedoraMirror: "https://download.fedoraproject.org/pub/fedora/linux/releases/43/Everything/x86_64/os",
-    tftpDir: join(testDir, "tftp"),
-    httpDir: join(testDir, "http"),
-    stateFile: join(testDir, "state.json"),
-  };
-}
-
-function hardware(mac: string, over: Partial<HardwareInfo> = {}): HardwareInfo {
-  return {
-    mac,
-    product: "TestBox",
-    board: "TestBoard",
-    serial: "SN123",
-    manufacturer: "TestCorp",
-    cpu_model: "Test CPU",
-    cpu_cores: 4,
-    memory_gb: 16,
-    arch: "x86_64",
-    disks: [],
-    nics: [],
-    first_seen: new Date().toISOString(),
-    last_seen: new Date().toISOString(),
-    ...over,
-  };
-}
-
-const emptyState = (): BastionState => ({
-  discovered: {}, install_queue: {}, installed: {}, debug: {},
-});
-
-describe("architecture resolution", () => {
-  const config = createTestConfig("/tmp/unused");
-  const mac = "aa:bb:cc:dd:ee:ff";
-
-  it("prefers the tracked record over what the client reports", () => {
-    const state = emptyState();
-    state.discovered[mac] = hardware(mac, { arch: "aarch64" });
-    // Client claims x86_64; the machine record says otherwise and wins.
-    expect(resolveArch(state, mac, "x86_64", config)).toBe("aarch64");
-  });
-
-  it("falls back to the architecture reported at boot", () => {
-    expect(resolveArch(emptyState(), mac, "arm64", config)).toBe("aarch64");
-  });
-
-  it("normalises iPXE's arm64 spelling to aarch64", () => {
-    expect(resolveArch(emptyState(), mac, "arm64", config)).toBe("aarch64");
-    expect(resolveArch(emptyState(), mac, "x86_64", config)).toBe("x86_64");
-  });
-
-  it("falls back to the configured default for unknown architectures", () => {
-    expect(resolveArch(emptyState(), mac, "riscv64", config)).toBe("x86_64");
-    expect(resolveArch(emptyState(), mac, undefined, config)).toBe("x86_64");
-  });
-
-  it("reads arch from the installed record for already-provisioned machines", () => {
-    const state = emptyState();
-    state.installed[mac] = {
-      hostname: "spark", role: "worker", ip: "10.0.0.5",
-      installed_at: new Date().toISOString(), arch: "aarch64",
-    };
-    expect(resolveArch(state, mac, undefined, config)).toBe("aarch64");
-  });
-});
-
-describe("aarch64 dispatch", () => {
-  let testDir: string;
-  let app: FastifyInstance;
-  let state: StateManager;
-  const mac = "aa:bb:cc:dd:ee:ff";
-
-  beforeEach(() => {
-    testDir = join(tmpdir(), `bastion-arch-test-${Date.now()}-${Math.random().toString(36).slice(2)}`);
-    mkdirSync(join(testDir, "http"), { recursive: true });
-    mkdirSync(join(testDir, "tftp"), { recursive: true });
-    const result = createApp(createTestConfig(testDir));
-    app = result.app;
-    state = result.state;
-  });
-
-  afterEach(async () => {
-    await app.close();
-    rmSync(testDir, { recursive: true, force: true });
-  });
-
-  it("serves the aarch64 kernel and initrd to an arm64 client", async () => {
-    const res = await app.inject({ method: "GET", url: `/dispatch?mac=${mac}&arch=arm64` });
-    expect(res.statusCode).toBe(200);
-    expect(res.body).toContain("/vmlinuz-aarch64");
-    expect(res.body).toContain("/initrd-aarch64.img");
-    expect(res.body).not.toContain("/vmlinuz ");
-  });
-
-  it("points an arm64 client at the aarch64 Fedora mirror", async () => {
-    const res = await app.inject({ method: "GET", url: `/dispatch?mac=${mac}&arch=arm64` });
-    expect(res.body).toContain("Everything/aarch64/os");
-    expect(res.body).not.toContain("Everything/x86_64/os");
-  });
-
-  it("uses serial console arguments and not nomodeset on arm64", async () => {
-    const res = await app.inject({ method: "GET", url: `/dispatch?mac=${mac}&arch=arm64` });
-    expect(res.body).toContain("console=ttyAMA0,115200");
-    expect(res.body).not.toContain("nomodeset");
-  });
-
-  it("refuses to serve the x86-only Ubuntu kernel to an arm64 client", async () => {
-    // A machine queued for Ubuntu before it was discovered as aarch64 reaches dispatch
-    // with no guard having run. Serving it /ubuntu-vmlinuz is the original bug.
-    state.update((s) => {
-      s.install_queue[mac] = {
-        hostname: "arm-node", disk: "", role: "worker",
-        os: "ubuntu-26.04", queued_at: new Date().toISOString(),
-      };
-    });
-
-    const res = await app.inject({ method: "GET", url: `/dispatch?mac=${mac}&arch=arm64` });
-    expect(res.statusCode).toBe(200);
-    expect(res.body).toContain("CANNOT BOOT THIS MACHINE");
-    expect(res.body).toContain("no aarch64 netboot artifacts");
-    expect(res.body).not.toContain("ubuntu-vmlinuz");
-  });
-
-  it("still serves Ubuntu to an x86_64 client", async () => {
-    state.update((s) => {
-      s.install_queue[mac] = {
-        hostname: "x86-node", disk: "", role: "worker",
-        os: "ubuntu-26.04", queued_at: new Date().toISOString(),
-      };
-    });
-
-    const res = await app.inject({ method: "GET", url: `/dispatch?mac=${mac}&arch=x86_64` });
-    expect(res.body).toContain("ubuntu-vmlinuz");
-    expect(res.body).not.toContain("CANNOT BOOT");
-  });
-
-  it("serves a rescue kernel for the recorded architecture, not the requester's", async () => {
-    // The Spark case: machine known to be aarch64, queued for rescue.
-    state.update((s) => {
-      s.discovered[mac] = hardware(mac, { arch: "aarch64" });
-      s.debug[mac] = { hostname: "spark-2935", queued_at: new Date().toISOString() };
-    });
-
-    const res = await app.inject({ method: "GET", url: `/dispatch?mac=${mac}` });
-    expect(res.body).toContain("DEBUG/RESCUE MODE");
-    expect(res.body).toContain("/vmlinuz-aarch64");
-    expect(res.body).toContain("inst.rescue");
-    expect(res.body).toContain("inst.sshd");
-  });
-});
-
-describe("--pxe-boot root device", () => {
-  let testDir: string;
-  let app: FastifyInstance;
-  let state: StateManager;
-  const mac = "aa:bb:cc:dd:ee:ff";
-
-  beforeEach(() => {
-    testDir = join(tmpdir(), `bastion-root-test-${Date.now()}-${Math.random().toString(36).slice(2)}`);
-    mkdirSync(join(testDir, "http"), { recursive: true });
-    mkdirSync(join(testDir, "tftp"), { recursive: true });
-    const result = createApp(createTestConfig(testDir));
-    app = result.app;
-    state = result.state;
-  });
-
-  afterEach(async () => {
-    await app.close();
-    rmSync(testDir, { recursive: true, force: true });
-  });
-
-  it("uses the root device recorded on the machine", async () => {
-    state.update((s) => {
-      s.installed[mac] = {
-        hostname: "worker-1", role: "worker", ip: "10.0.0.50",
-        installed_at: new Date().toISOString(),
-        root_device: "/dev/mapper/otherVG-root",
-        root_args: "rd.lvm.lv=otherVG/root",
-      };
-      s.debug[mac] = { hostname: "worker-1", queued_at: new Date().toISOString(), pxeBoot: true };
-    });
-
-    const res = await app.inject({ method: "GET", url: `/dispatch?mac=${mac}` });
-    expect(res.body).toContain("PXE BOOT (debug)");
-    expect(res.body).toContain("root=/dev/mapper/otherVG-root");
-    expect(res.body).toContain("rd.lvm.lv=otherVG/root");
-    // The old hardcoded layout must not leak back in.
-    expect(res.body).not.toContain("labvg");
-  });
-
-  it("falls back to rescue rather than guessing when no root device is known", async () => {
-    state.update((s) => {
-      s.installed[mac] = {
-        hostname: "spark-2935", role: "worker", ip: "192.168.8.12",
-        installed_at: new Date().toISOString(), arch: "aarch64",
-      };
-      s.debug[mac] = { hostname: "spark-2935", queued_at: new Date().toISOString(), pxeBoot: true };
-    });
-
-    const res = await app.inject({ method: "GET", url: `/dispatch?mac=${mac}` });
-    expect(res.body).toContain("DEBUG/RESCUE MODE");
-    expect(res.body).toContain("no root device is recorded");
-    expect(res.body).toContain("debug-setup.sh");
-    expect(res.body).not.toContain("root=");
-    // And it is still the right architecture.
-    expect(res.body).toContain("/vmlinuz-aarch64");
-  });
-
-  it("records a root device reported from a rescue shell without erasing hardware info", async () => {
-    state.update((s) => {
-      s.discovered[mac] = hardware(mac, { product: "DGX Spark", manufacturer: "NVIDIA", arch: "aarch64" });
-    });
-
-    const res = await app.inject({
-      method: "POST",
-      url: "/api/discover",
-      headers: { "Content-Type": "application/json" },
-      body: JSON.stringify({ mac, root_device: "/dev/nvme0n1p2" }),
-    });
-    expect(res.statusCode).toBe(200);
-
-    const hw = state.load().discovered[mac];
-    expect(hw?.root_device).toBe("/dev/nvme0n1p2");
-    // The partial report must not blank what we already knew.
-    expect(hw?.product).toBe("DGX Spark");
-    expect(hw?.cpu_cores).toBe(4);
-    expect(hw?.arch).toBe("aarch64");
-  });
-});
-
-describe("dnsmasq architecture detection", () => {
-  const conf = renderDnsmasqConf(createTestConfig("/tmp/unused"));
-
-  it("maps DHCP option 93 values to per-architecture bootloaders", () => {
-    // 11 = ARM 64-bit UEFI
-    expect(conf).toContain("dhcp-match=set:efi-arm64,option:client-arch,11");
-    expect(conf).toContain("dhcp-boot=tag:efi-arm64,tag:!ipxe,ipxe-arm64.efi");
-    // 7 / 9 = x64 UEFI, 0 = x86 BIOS
-    expect(conf).toContain("dhcp-match=set:efi-x86_64,option:client-arch,7");
-    expect(conf).toContain("dhcp-match=set:efi-x86_64,option:client-arch,9");
-    expect(conf).toContain("dhcp-match=set:bios,option:client-arch,0");
-  });
-
-  it("matches arm64 UEFI HTTP boot on 19, not 20", () => {
-    // IANA: 19 = arm uefi 64 boot from http, 20 = pc/at bios boot from http.
-    expect(conf).toContain("dhcp-match=set:httpboot-arm64,option:client-arch,19");
-    expect(conf).not.toContain("dhcp-match=set:httpboot-arm64,option:client-arch,20");
-    expect(conf).toContain("dhcp-match=set:httpboot-x86_64,option:client-arch,16");
-  });
-
-  it("offers an arm64 PXE service directive in proxy mode", () => {
-    expect(conf).toContain('pxe-service=tag:!ipxe,ARM64_EFI,"PXE Boot",ipxe-arm64.efi');
-  });
-});

bastion/src/bastion/tests/dispatch.test.ts

@@ -22,6 +22,8 @@ function createTestConfig(testDir: string): BastionConfig {
     dhcpRangeEnd: "",
     ubuntuVersion: "26.04",
     ubuntuMirror: "https://releases.ubuntu.com/26.04",
+    vyosIsoUrl: "https://downloads.vyos.io/rolling/current/generic/vyos-rolling-latest.iso",
+    vyosDefaultPassword: "vyos",
     iface: "eth0",
     serverIp: "10.0.0.1",
     network: "10.0.0.0",

bastion/src/bastion/tests/fixtures/ipxe-x86_64-golden.json

@@ -1,8 +0,0 @@
-{
-  "boot": "#!ipxe\n\necho\necho ============================================\necho   Lab PXE Bastion\necho   Contacting server for instructions...\necho ============================================\necho\n\nchain http://10.0.0.1:8080/dispatch?mac=${net0/mac}\n",
-  "discover": "#!ipxe\n\necho\necho =============================================\necho   Lab PXE Bastion - DISCOVERY MODE\necho   MAC: aa:bb:cc:dd:ee:ff\necho   Collecting hardware info...\necho =============================================\necho\n\nkernel http://10.0.0.1:8080/vmlinuz inst.ks=http://10.0.0.1:8080/discover.ks inst.stage2=https://download.fedoraproject.org/pub/fedora/linux/releases/43/Everything/x86_64/os inst.text nomodeset\ninitrd http://10.0.0.1:8080/initrd.img\nboot\n",
-  "install": "#!ipxe\n\necho\necho =============================================\necho   Lab PXE Bastion - INSTALLING Fedora 43\necho   Target: worker-1\necho   MAC:    aa:bb:cc:dd:ee:ff\necho =============================================\necho\n\nkernel http://10.0.0.1:8080/vmlinuz inst.ks=http://10.0.0.1:8080/ks?mac=aa:bb:cc:dd:ee:ff inst.repo=https://download.fedoraproject.org/pub/fedora/linux/releases/43/Everything/x86_64/os inst.text nomodeset\ninitrd http://10.0.0.1:8080/initrd.img\nboot\n",
-  "debug": "#!ipxe\n\necho\necho =============================================\necho   Lab PXE Bastion - DEBUG/RESCUE MODE\necho   Target: worker-1\necho   MAC:    aa:bb:cc:dd:ee:ff\necho =============================================\necho\n\nkernel http://10.0.0.1:8080/vmlinuz inst.rescue inst.text inst.sshd inst.ks=http://10.0.0.1:8080/debug.ks?mac=aa:bb:cc:dd:ee:ff inst.stage2=https://download.fedoraproject.org/pub/fedora/linux/releases/43/Everything/x86_64/os\ninitrd http://10.0.0.1:8080/initrd.img\nboot\n",
-  "pxeBoot": "#!ipxe\n\necho\necho =============================================\necho   Lab PXE Bastion - PXE BOOT (debug)\necho   Target: worker-1\necho   MAC:    aa:bb:cc:dd:ee:ff\necho   Kernel+initrd from PXE, root from NVMe\necho =============================================\necho\n\nkernel http://10.0.0.1:8080/vmlinuz root=/dev/mapper/labvg-root ro rd.lvm.lv=labvg/root rd.lvm.lv=labvg/swap console=tty0\ninitrd http://10.0.0.1:8080/initrd.img\nboot\n",
-  "localBoot": "#!ipxe\n\necho\necho =============================================\necho   Lab PXE Bastion - worker-1\necho   Already installed, booting from local disk\necho =============================================\necho\nsleep 3\nexit 1\n"
-}

bastion/src/bastion/tests/install-guard.test.ts

@@ -1,194 +0,0 @@
-// Installs must never reach a machine running a vendor OS we cannot restore.
-//
-// This is the guardrail that stops someone reinstalling a DGX Spark at 2am. Rescue is
-// deliberately still allowed for the same machines -- that is the whole point.
-
-import { describe, it, expect, beforeEach, afterEach } from "vitest";
-import { mkdirSync, rmSync } from "node:fs";
-import { join } from "node:path";
-import { tmpdir } from "node:os";
-import type { BastionConfig, BastionState, HardwareInfo } from "@lab/shared";
-import { classifyOnboard } from "@lab/shared";
-import { createApp } from "../src/server.js";
-import { checkInstallAllowed } from "../src/services/install-guard.js";
-import type { FastifyInstance } from "fastify";
-import type { StateManager } from "../src/services/state.js";
-
-// The real machines this exists to protect.
-const SPARK_2935 = "4c:bb:47:7f:29:35";
-const SPARK_3A1C = "48:21:0b:96:3a:1c";
-const ORDINARY = "aa:bb:cc:dd:ee:ff";
-
-function createTestConfig(testDir: string): BastionConfig {
-  return {
-    fedoraVersion: "43", arch: "x86_64", httpPort: 0,
-    timezone: "Europe/London", locale: "en_GB.UTF-8", bastionDir: testDir,
-    domain: "test.local", dhcpMode: "proxy", dhcpRangeStart: "", dhcpRangeEnd: "",
-    ubuntuVersion: "26.04", ubuntuMirror: "https://releases.ubuntu.com/26.04",
-    iface: "eth0", serverIp: "10.0.0.1", network: "10.0.0.0", gateway: "10.0.0.1",
-    sshKeys: [], adminUser: "testadmin", syslogPort: 15514,
-    skipDnsmasq: true, skipArtifacts: true,
-    fedoraMirror: "https://download.fedoraproject.org/pub/fedora/linux/releases/43/Everything/x86_64/os",
-    tftpDir: join(testDir, "tftp"), httpDir: join(testDir, "http"),
-    stateFile: join(testDir, "state.json"),
-  };
-}
-
-function hardware(mac: string, over: Partial<HardwareInfo> = {}): HardwareInfo {
-  return {
-    mac, product: "TestBox", board: "TestBoard", serial: "SN1",
-    manufacturer: "TestCorp", cpu_model: "Test CPU", cpu_cores: 4, memory_gb: 16,
-    arch: "x86_64", disks: [], nics: [],
-    first_seen: new Date().toISOString(), last_seen: new Date().toISOString(),
-    ...over,
-  };
-}
-
-const emptyState = (): BastionState => ({
-  discovered: {}, install_queue: {}, installed: {}, debug: {},
-});
-
-describe("classifyOnboard", () => {
-  it("recognises a DGX Spark from its DMI identity", () => {
-    expect(classifyOnboard({
-      mac: ORDINARY, manufacturer: "NVIDIA", product: "NVIDIA DGX Spark", board: "GB10",
-    })).toEqual({ onboard: "ssh", vendor_os: "dgx-os" });
-  });
-
-  it("recognises the known Sparks even with no DMI recorded", () => {
-    // Neither Spark has hardware info in bastion state today. A DMI-only rule would
-    // fail open on exactly the machines this protects.
-    expect(classifyOnboard({ mac: SPARK_2935 }).onboard).toBe("ssh");
-    expect(classifyOnboard({ mac: SPARK_3A1C }).onboard).toBe("ssh");
-  });
-
-  it("treats ordinary hardware as PXE-installable", () => {
-    expect(classifyOnboard({
-      mac: ORDINARY, manufacturer: "Beelink", product: "SER9", board: "SER9",
-    })).toEqual({ onboard: "pxe" });
-  });
-
-  it("does not override an explicit classification already on the record", () => {
-    expect(classifyOnboard({
-      mac: SPARK_2935, onboard: "pxe",
-    })).toEqual({ onboard: "pxe" });
-  });
-});
-
-describe("checkInstallAllowed", () => {
-  it("refuses a DGX Spark and explains why", () => {
-    const state = emptyState();
-    state.installed[SPARK_2935] = {
-      hostname: "spark-2935", role: "worker", ip: "192.168.8.12",
-      installed_at: new Date().toISOString(), arch: "aarch64",
-    };
-
-    const result = checkInstallAllowed(state, SPARK_2935, "fedora-43");
-    expect(result.allowed).toBe(false);
-    if (result.allowed === false) {
-      expect(result.error).toContain("spark-2935");
-      expect(result.error).toContain("DGX OS");
-      expect(result.error).toContain("provision debug");
-    }
-  });
-
-  it("refuses a Spark that is only known by MAC", () => {
-    expect(checkInstallAllowed(emptyState(), SPARK_3A1C, "fedora-43").allowed).toBe(false);
-  });
-
-  it("allows an ordinary discovered machine", () => {
-    const state = emptyState();
-    state.discovered[ORDINARY] = hardware(ORDINARY);
-    expect(checkInstallAllowed(state, ORDINARY, "fedora-43").allowed).toBe(true);
-  });
-
-  it("allows Fedora on aarch64", () => {
-    const state = emptyState();
-    state.discovered[ORDINARY] = hardware(ORDINARY, { arch: "aarch64" });
-    expect(checkInstallAllowed(state, ORDINARY, "fedora-43").allowed).toBe(true);
-  });
-
-  it("refuses Ubuntu on aarch64 -- no netboot artifacts are published", () => {
-    const state = emptyState();
-    state.discovered[ORDINARY] = hardware(ORDINARY, { arch: "aarch64" });
-    const result = checkInstallAllowed(state, ORDINARY, "ubuntu-26.04");
-    expect(result.allowed).toBe(false);
-    if (result.allowed === false) {
-      expect(result.error).toContain("aarch64");
-    }
-  });
-
-  it("allows Ubuntu on x86_64", () => {
-    const state = emptyState();
-    state.discovered[ORDINARY] = hardware(ORDINARY, { arch: "x86_64" });
-    expect(checkInstallAllowed(state, ORDINARY, "ubuntu-26.04").allowed).toBe(true);
-  });
-});
-
-describe("install route enforces the guard", () => {
-  let testDir: string;
-  let app: FastifyInstance;
-  let state: StateManager;
-
-  beforeEach(() => {
-    testDir = join(tmpdir(), `bastion-guard-test-${Date.now()}-${Math.random().toString(36).slice(2)}`);
-    mkdirSync(join(testDir, "http"), { recursive: true });
-    mkdirSync(join(testDir, "tftp"), { recursive: true });
-    const result = createApp(createTestConfig(testDir));
-    app = result.app;
-    state = result.state;
-  });
-
-  afterEach(async () => {
-    await app.close();
-    rmSync(testDir, { recursive: true, force: true });
-  });
-
-  it("rejects POST /api/install for a Spark and queues nothing", async () => {
-    const res = await app.inject({
-      method: "POST",
-      url: "/api/install",
-      headers: { "Content-Type": "application/json" },
-      body: JSON.stringify({ mac: SPARK_2935, hostname: "spark-2935", role: "worker" }),
-    });
-
-    expect(res.statusCode).toBe(409);
-    expect(JSON.parse(res.body).error).toContain("Refusing to install");
-    expect(state.load().install_queue[SPARK_2935]).toBeUndefined();
-  });
-
-  it("still serves rescue to a Spark -- debug is never guarded", async () => {
-    state.update((s) => {
-      s.installed[SPARK_2935] = {
-        hostname: "spark-2935", role: "worker", ip: "192.168.8.12",
-        installed_at: new Date().toISOString(), arch: "aarch64",
-      };
-      s.debug[SPARK_2935] = { hostname: "spark-2935", queued_at: new Date().toISOString() };
-    });
-
-    const res = await app.inject({ method: "GET", url: `/dispatch?mac=${SPARK_2935}` });
-    expect(res.statusCode).toBe(200);
-    expect(res.body).toContain("DEBUG/RESCUE MODE");
-    expect(res.body).toContain("/vmlinuz-aarch64");
-  });
-
-  it("a Spark that PXE boots unqueued gets discovery, never an install", async () => {
-    const res = await app.inject({ method: "GET", url: `/dispatch?mac=${SPARK_2935}&arch=arm64` });
-    expect(res.body).toContain("DISCOVERY MODE");
-    expect(res.body).not.toContain("INSTALLING");
-  });
-
-  it("still accepts an ordinary machine", async () => {
-    state.update((s) => { s.discovered[ORDINARY] = hardware(ORDINARY); });
-
-    const res = await app.inject({
-      method: "POST",
-      url: "/api/install",
-      headers: { "Content-Type": "application/json" },
-      body: JSON.stringify({ mac: ORDINARY, hostname: "worker-1", role: "worker" }),
-    });
-
-    expect(res.statusCode).toBe(200);
-    expect(state.load().install_queue[ORDINARY]).toBeDefined();
-  });
-});

bastion/src/bastion/tests/ipxe-x86-regression.test.ts

@@ -1,89 +0,0 @@
-// x86_64 iPXE output regression gate.
-//
-// The aarch64 PXE work must not change what an x86_64 machine is served. The golden
-// fixture was dumped from the templates as they stood before that work started, so
-// any diff here is a regression, not an improvement.
-//
-// The one deliberate exception is renderBootIpxe: its chain URL gained
-// `&arch=${buildarch}` so the dispatch endpoint can observe the client's
-// architecture at boot time. That single change is asserted explicitly below
-// rather than being allowed to slip through the byte-for-byte comparison.
-
-import { describe, it, expect } from "vitest";
-import { readFileSync } from "node:fs";
-import { join } from "node:path";
-import { fileURLToPath } from "node:url";
-import { dirname } from "node:path";
-import {
-  renderBootIpxe,
-  renderDiscoverIpxe,
-  renderInstallIpxe,
-  renderDebugIpxe,
-  renderPxeBootDebugIpxe,
-  renderLocalBootIpxe,
-} from "../src/templates/boot.ipxe.js";
-
-const here = dirname(fileURLToPath(import.meta.url));
-const golden = JSON.parse(
-  readFileSync(join(here, "fixtures", "ipxe-x86_64-golden.json"), "utf-8"),
-) as Record<string, string>;
-
-// Exactly the parameters used to dump the fixture.
-const serverIp = "10.0.0.1";
-const httpPort = 8080;
-const mac = "aa:bb:cc:dd:ee:ff";
-const hostname = "worker-1";
-const fedoraVersion = "43";
-const fedoraMirror =
-  "https://download.fedoraproject.org/pub/fedora/linux/releases/43/Everything/x86_64/os";
-
-// The x86_64 LVM layout the fixture was captured with. Before this work the values
-// were hardcoded in the template; they are now supplied by the caller from machine
-// state, so the fixture pins the rendering, not the defaults.
-const x86Root = {
-  rootDevice: "/dev/mapper/labvg-root",
-  rootArgs: "rd.lvm.lv=labvg/root rd.lvm.lv=labvg/swap",
-};
-
-describe("x86_64 iPXE output is unchanged", () => {
-  it("discover script is byte-identical", () => {
-    const rendered = renderDiscoverIpxe({
-      mac, serverIp, httpPort, fedoraMirror, arch: "x86_64",
-    });
-    expect(rendered).toBe(golden["discover"]);
-  });
-
-  it("install script is byte-identical", () => {
-    const rendered = renderInstallIpxe({
-      mac, hostname, serverIp, httpPort, fedoraVersion, fedoraMirror, arch: "x86_64",
-    });
-    expect(rendered).toBe(golden["install"]);
-  });
-
-  it("debug/rescue script is byte-identical", () => {
-    const rendered = renderDebugIpxe({
-      mac, hostname, serverIp, httpPort, fedoraMirror, arch: "x86_64",
-    });
-    expect(rendered).toBe(golden["debug"]);
-  });
-
-  it("--pxe-boot script is byte-identical when state carries the Fedora LVM layout", () => {
-    const rendered = renderPxeBootDebugIpxe({
-      mac, hostname, serverIp, httpPort, arch: "x86_64", ...x86Root,
-    });
-    expect(rendered).toBe(golden["pxeBoot"]);
-  });
-
-  it("local boot script is byte-identical", () => {
-    expect(renderLocalBootIpxe(hostname)).toBe(golden["localBoot"]);
-  });
-
-  it("boot.ipxe differs only by the &arch= chain parameter", () => {
-    const rendered = renderBootIpxe({ serverIp, httpPort });
-    // The sole intended difference.
-    expect(rendered).toBe(golden["boot"].replace(
-      "/dispatch?mac=${net0/mac}",
-      "/dispatch?mac=${net0/mac}&arch=${buildarch}",
-    ));
-  });
-});

bastion/src/bastion/tests/kickstart.test.ts

@@ -96,9 +96,9 @@ describe("renderInstallKickstart", () => {
     expect(ks).toContain("/api/progress");
   });
 
-  it("infra role has /var/lib/rancher partition", () => {
+  it("infra role has 120G /var/lib/rancher partition", () => {
     const ks = renderInstallKickstart(baseParams({ role: "infra" }));
-    expect(ks).toContain("logvol /var/lib/rancher --vgname=labvg --name=rancher --fstype=xfs --size=20480");
+    expect(ks).toContain("logvol /var/lib/rancher --vgname=labvg --name=rancher --fstype=xfs --size=122880");
   });
 
   it("infra role has k3s install", () => {
@@ -106,10 +106,14 @@ describe("renderInstallKickstart", () => {
     expect(ks).toContain("curl -sfL https://get.k3s.io | INSTALL_K3S_SKIP_START=true sh -");
   });
 
-  it("worker role does NOT have /var/lib/rancher partition in fresh install", () => {
+  it("worker role has 120G /var/lib/rancher partition (imageFs must be sized before longhorn --grow)", () => {
     const ks = renderInstallKickstart(baseParams({ role: "worker" }));
-    // Worker should not have the fresh-install rancher partition line
-    expect(ks).not.toContain("logvol /var/lib/rancher --vgname=labvg --name=rancher --fstype=xfs --size=20480");
+    expect(ks).toContain("logvol /var/lib/rancher --vgname=labvg --name=rancher --fstype=xfs --size=122880");
+  });
+
+  it("vanilla role does NOT have /var/lib/rancher partition in fresh install", () => {
+    const ks = renderInstallKickstart(baseParams({ role: "vanilla" }));
+    expect(ks).not.toContain("--name=rancher --fstype=xfs");
   });
 
   it("worker role does NOT have k3s install", () => {

bastion/src/bastion/tests/ubuntu-autoinstall.test.ts

@@ -0,0 +1,87 @@
+// The Ubuntu autoinstall document must be valid YAML for EVERY role.
+//
+// There was no test here, and the template shipped a document that did not
+// parse: the longhorn/rancher LVM entries were indented 8 spaces while their
+// siblings in storage.config sit at 6, giving "expected <block end>, but found
+// '-'". Every role that gets a longhorn volume -- which is the worker shape --
+// rendered an uninstallable document. A `toContain` assertion would not have
+// caught that; only parsing does.
+//
+// Parsed with python3's yaml rather than a new npm dependency, mirroring how
+// kickstart.test.ts shells out to `ksvalidator`: the point is to check the
+// artefact with a real parser, not to grow the dependency tree.
+
+import { describe, it, expect } from "vitest";
+import { execFileSync } from "node:child_process";
+import { writeFileSync, unlinkSync } from "node:fs";
+import { renderUbuntuAutoinstall } from "../src/templates/ubuntu-autoinstall.js";
+
+const base = {
+  hostname: "n6",
+  disk: "/dev/sda",
+  domain: "ad.itaz.eu",
+  ubuntuVersion: "24.04",
+  timezone: "Europe/London",
+  locale: "en_GB.UTF-8",
+  serverIp: "10.0.0.1",
+  httpPort: 8080,
+  sshKeys: ["ssh-ed25519 AAAAtest test@lab"],
+  adminUser: "root",
+};
+
+/** Parse with python3's yaml and return the document as JSON. */
+function parseYaml(text: string, label: string): Record<string, any> {
+  const tmp = `/tmp/autoinstall-test-${label}.yaml`;
+  writeFileSync(tmp, text);
+  try {
+    const out = execFileSync(
+      "python3",
+      ["-c", "import sys,yaml,json; json.dump(yaml.safe_load(open(sys.argv[1])), sys.stdout)", tmp],
+      { encoding: "utf-8" },
+    );
+    return JSON.parse(out);
+  } catch (err: unknown) {
+    const msg = err instanceof Error ? (err as { stderr?: string }).stderr ?? err.message : String(err);
+    throw new Error(`autoinstall YAML did not parse for ${label}: ${msg}`);
+  } finally {
+    try { unlinkSync(tmp); } catch { /* ignore */ }
+  }
+}
+
+describe("renderUbuntuAutoinstall", () => {
+  for (const role of ["vanilla", "worker", "infra"]) {
+    it(`renders parseable YAML for role=${role}`, () => {
+      const doc = parseYaml(renderUbuntuAutoinstall({ ...base, role }), role);
+      expect(doc.autoinstall).toBeDefined();
+      expect(doc.autoinstall.version).toBe(1);
+      // storage.config must be a flat list; the indentation bug produced a
+      // nested map here, which is how it went unnoticed.
+      expect(Array.isArray(doc.autoinstall.storage.config)).toBe(true);
+    });
+  }
+
+  it("gives the longhorn role its volume as a sibling entry, not a nested map", () => {
+    const doc = parseYaml(renderUbuntuAutoinstall({ ...base, role: "worker" }), "longhorn");
+    const ids = doc.autoinstall.storage.config.map((e: { id: string }) => e.id);
+    expect(ids).toContain("lv-longhorn");
+    expect(ids).toContain("mount-longhorn");
+  });
+
+  it("sets EUI-64 link-local so MAC-keyed DHCPv6 reservations can match", () => {
+    const doc = parseYaml(renderUbuntuAutoinstall({ ...base, role: "worker" }), "eui64");
+    const late = (doc.autoinstall["late-commands"] as string[]).join("\n");
+    expect(late).toContain("10-ipv6-eui64.conf");
+    expect(late).toContain("ipv6.addr-gen-mode=eui64");
+  });
+
+  it("requests both address families on the primary NIC", () => {
+    const doc = parseYaml(renderUbuntuAutoinstall({ ...base, role: "worker" }), "net");
+    const eth = doc.autoinstall.network.ethernets.primary;
+    expect(eth.dhcp4).toBe(true);
+    // Without this a node provisioned into a dual-stack cluster comes up with
+    // no IPv6 and joins as an IPv4-only member.
+    expect(eth.dhcp6).toBe(true);
+    // The install must not block waiting for a v6 lease that may never come.
+    expect(eth.optional).toBe(true);
+  });
+});

bastion/src/bastion/tests/vyos-bundle.test.ts

@@ -0,0 +1,155 @@
+import { describe, it, expect, vi } from "vitest";
+import type { VyosBundle } from "@lab/shared";
+import { buildVyosConfigSpec } from "../src/templates/vyos-config-spec.js";
+
+/**
+ * A bundle is what makes "one config, two apply paths" true rather than
+ * aspirational: `pulumi up` POSTs the subtree model to a running router, labctl
+ * writes the same model into config.boot during a PXE install. These tests pin
+ * the properties that keep the two honest.
+ */
+const bundle: VyosBundle = {
+  sets: [
+    { path: ["system", "host-name"], value: "vyos001" },
+    { path: ["interfaces", "bonding", "bond0", "address"], value: "192.168.1.252/24" },
+    { path: ["interfaces", "bonding", "bond0", "member", "interface"], value: "eth1", replace: false },
+    { path: ["interfaces", "bonding", "bond0", "vif", "53", "disable"] },
+    { path: ["interfaces", "pppoe", "pppoe0", "authentication", "password"], value: "@secret:pppoePassword" },
+    { path: ["interfaces", "pppoe", "pppoe0", "mtu"], value: "1492" },
+  ],
+  tags: [["interfaces", "bonding", "bond0"], ["interfaces", "ethernet"]],
+};
+
+const build = (hostname: string, extra: Record<string, unknown> = {}) =>
+  buildVyosConfigSpec({
+    hostname,
+    spec: { bundle, ...extra },
+    defaultPassword: "changeme",
+  });
+
+describe("vyos config spec from a Pulumi bundle", () => {
+  it("applies non-secret nodes verbatim, preserving valuelessness and replace:false", () => {
+    const spec = build("vyos001");
+
+    expect(spec.sets).toContainEqual({
+      path: ["interfaces", "bonding", "bond0", "address"],
+      value: "192.168.1.252/24",
+    });
+    // A multi-value node must keep replace:false or the second bond member
+    // overwrites the first.
+    expect(spec.sets).toContainEqual({
+      path: ["interfaces", "bonding", "bond0", "member", "interface"],
+      value: "eth1",
+      replace: false,
+    });
+    // A valueless node must not acquire a value on the way through.
+    expect(spec.sets).toContainEqual({
+      path: ["interfaces", "bonding", "bond0", "vif", "53", "disable"],
+    });
+    expect(spec.tags).toEqual(bundle.tags);
+  });
+
+  it("drops secret-valued nodes instead of installing the sentinel text", () => {
+    const warn = vi.spyOn(console, "warn").mockImplementation(() => {});
+    const spec = build("vyos001");
+
+    const values = spec.sets.map((s) => s.value ?? "");
+    expect(values.some((v) => v.startsWith("@secret:"))).toBe(false);
+    expect(spec.sets.some((s) => s.path.includes("authentication"))).toBe(false);
+    // Silently dropping the WAN credential would leave someone debugging a dead
+    // PPPoE link, so it has to be said out loud.
+    expect(warn).toHaveBeenCalledWith(expect.stringContaining("pulumi up"));
+    warn.mockRestore();
+  });
+
+  it("forces the hostname the install was asked for, not the bundle's", () => {
+    // The bundle is exported from one router and reused for its peer; taking the
+    // hostname from it would put two vyos001s on the network.
+    const spec = build("vyos002");
+    const hostnames = spec.sets.filter(
+      (s) => s.path.length === 2 && s.path[0] === "system" && s.path[1] === "host-name",
+    );
+    expect(hostnames).toEqual([{ path: ["system", "host-name"], value: "vyos002" }]);
+  });
+
+  it("still honours installer inputs, which are not router config", () => {
+    const spec = buildVyosConfigSpec({
+      hostname: "vyos001",
+      spec: { bundle, password: "s3cret", freshConfig: true },
+      defaultPassword: "changeme",
+      disk: "nvme0n1",
+    });
+    expect(spec.password).toBe("s3cret");
+    expect(spec.freshConfig).toBe(true);
+    expect(spec.disk).toBe("/dev/nvme0n1");
+  });
+
+  it("enables the HTTP API at install so Pulumi can manage the router from first boot", () => {
+    const warn = vi.spyOn(console, "warn").mockImplementation(() => {});
+    const spec = buildVyosConfigSpec({
+      hostname: "vyos001",
+      spec: { bundle, apiKey: "k3y", apiListenAddress: "10.0.1.252" },
+      defaultPassword: "changeme",
+    });
+
+    expect(spec.sets).toContainEqual({
+      path: ["service", "https", "api", "keys", "id", "pulumi", "key"],
+      value: "k3y",
+    });
+    expect(spec.sets).toContainEqual({ path: ["service", "https", "api", "rest"] });
+    expect(spec.sets).toContainEqual({
+      path: ["service", "https", "listen-address"],
+      value: "10.0.1.252",
+    });
+    // The key id is a tag node; without this the installer's ConfigTree rejects it.
+    expect(spec.tags).toContainEqual(["service", "https", "api", "keys", "id"]);
+    warn.mockRestore();
+  });
+
+  it("binds the API to the static management address when none is given", () => {
+    const spec = buildVyosConfigSpec({
+      hostname: "vyos001",
+      spec: { apiKey: "k3y", mgmtAddress: "192.168.1.252/24" },
+      defaultPassword: "changeme",
+    });
+    expect(spec.sets).toContainEqual({
+      path: ["service", "https", "listen-address"],
+      value: "192.168.1.252",
+    });
+  });
+
+  it("refuses to enable the API unbound rather than exposing it on the WAN", () => {
+    const warn = vi.spyOn(console, "warn").mockImplementation(() => {});
+    // Management is DHCP, so there is no address to bind at build time. Binding
+    // to everything would put a config-write endpoint on the WAN.
+    const spec = buildVyosConfigSpec({
+      hostname: "vyos001",
+      spec: { apiKey: "k3y", mgmtAddress: "dhcp" },
+      defaultPassword: "changeme",
+    });
+    expect(spec.sets.some((s) => s.path[0] === "service" && s.path[1] === "https")).toBe(false);
+    expect(warn).toHaveBeenCalledWith(expect.stringContaining("has NOT been enabled"));
+    warn.mockRestore();
+  });
+
+  it("does not enable the API when no key is supplied", () => {
+    const spec = buildVyosConfigSpec({
+      hostname: "vyos001",
+      spec: { mgmtAddress: "192.168.1.252/24" },
+      defaultPassword: "changeme",
+    });
+    expect(spec.sets.some((s) => s.path[0] === "service" && s.path[1] === "https")).toBe(false);
+  });
+
+  it("ignores the derived path entirely when a bundle is present", () => {
+    // Belt and braces: even if topology flags reach this far (the CLI rejects
+    // them), the bundle must win rather than merge.
+    const spec = buildVyosConfigSpec({
+      hostname: "vyos001",
+      spec: { bundle, bondMembers: ["eth2", "eth3"], vlans: [{ id: 99, address: "10.9.9.1/24" }] },
+      defaultPassword: "changeme",
+    });
+    expect(spec.sets.some((s) => s.path.includes("99"))).toBe(false);
+    expect(spec.sets.filter((s) => s.value === "eth2" || s.value === "eth3")).toEqual([]);
+  });
+});

bastion/src/bastion/tests/vyos.test.ts

@@ -0,0 +1,548 @@
+import { describe, it, expect, beforeEach, afterEach } from "vitest";
+import { mkdirSync, rmSync } from "node:fs";
+import { join } from "node:path";
+import { tmpdir } from "node:os";
+import type { BastionConfig } from "@lab/shared";
+import type { FastifyInstance } from "fastify";
+import { createApp } from "../src/server.js";
+import type { StateManager } from "../src/services/state.js";
+import { buildVyosConfigSpec } from "../src/templates/vyos-config-spec.js";
+import { renderVyosInstallPy } from "../src/templates/vyos-install.py.js";
+
+function createTestConfig(testDir: string): BastionConfig {
+  return {
+    fedoraVersion: "43",
+    arch: "x86_64",
+    httpPort: 0,
+    timezone: "Europe/London",
+    locale: "en_GB.UTF-8",
+    bastionDir: testDir,
+    domain: "test.local",
+    dhcpMode: "proxy",
+    dhcpRangeStart: "",
+    dhcpRangeEnd: "",
+    ubuntuVersion: "26.04",
+    ubuntuMirror: "https://releases.ubuntu.com/26.04",
+    vyosIsoUrl: "https://example.invalid/vyos.iso",
+    vyosDefaultPassword: "test-pw",
+    iface: "eth0",
+    serverIp: "10.0.0.1",
+    network: "10.0.0.0",
+    gateway: "10.0.0.1",
+    sshKeys: ["ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAITEST lab@test"],
+    adminUser: "testadmin",
+    syslogPort: 15515,
+    skipDnsmasq: true,
+    skipArtifacts: true,
+    fedoraMirror: "https://example.invalid/fedora",
+    tftpDir: join(testDir, "tftp"),
+    httpDir: join(testDir, "http"),
+    stateFile: join(testDir, "state.json"),
+  };
+}
+
+/** Pull the base64 spec back out of the generated Python driver. */
+function decodeSpecFrom(python: string): Record<string, unknown> {
+  const match = /base64\.b64decode\("([^"]+)"\)/.exec(python);
+  if (!match?.[1]) throw new Error("no base64 spec found in generated driver");
+  return JSON.parse(Buffer.from(match[1], "base64").toString("utf-8"));
+}
+
+describe("vyos config spec", () => {
+  it("puts VLANs on the bond when members are given", () => {
+    const spec = buildVyosConfigSpec({
+      hostname: "fw1",
+      defaultPassword: "pw",
+      spec: {
+        mgmtInterface: "eth0",
+        mgmtAddress: "10.0.8.2/24",
+        bondMembers: ["eth2", "eth3"],
+        vlans: [{ id: 10, address: "10.0.10.1/24", description: "k8s" }],
+      },
+    });
+
+    const paths = spec.sets.map((s) => s.path.join(" "));
+    expect(paths).toContain("interfaces bonding bond0 mode");
+    expect(paths).toContain("interfaces bonding bond0 vif 10 address");
+    // VLANs must hang off the bond, not the management NIC.
+    expect(paths).not.toContain("interfaces ethernet eth0 vif 10 address");
+
+    // Bond members are a multi-value node — appending, not replacing, is what
+    // keeps the second member from overwriting the first.
+    const members = spec.sets.filter(
+      (s) => s.path.join(" ") === "interfaces bonding bond0 member interface",
+    );
+    expect(members.map((m) => m.value)).toEqual(["eth2", "eth3"]);
+    expect(members.every((m) => m.replace === false)).toBe(true);
+  });
+
+  it("falls back to VLANs on the management NIC when unbonded", () => {
+    const spec = buildVyosConfigSpec({
+      hostname: "fw2",
+      defaultPassword: "pw",
+      spec: { mgmtInterface: "eth1", vlans: [{ id: 20, address: "10.0.20.1/24" }] },
+    });
+
+    const paths = spec.sets.map((s) => s.path.join(" "));
+    expect(paths).toContain("interfaces ethernet eth1 vif 20 address");
+  });
+
+  it("normalises the target disk to a full /dev path", () => {
+    // find_disks() enumerates with `lsblk -Jbp`, so valid responses are full
+    // paths; a bare name fails valid_responses and re-prompts forever.
+    expect(buildVyosConfigSpec({ hostname: "fw3", defaultPassword: "pw", disk: "/dev/mmcblk0" }).disk)
+      .toBe("/dev/mmcblk0");
+    expect(buildVyosConfigSpec({ hostname: "fw3", defaultPassword: "pw", disk: "mmcblk0" }).disk)
+      .toBe("/dev/mmcblk0");
+    expect(buildVyosConfigSpec({ hostname: "fw3", defaultPassword: "pw" }).disk).toBe("");
+  });
+
+  it("defaults to dhcp on eth0 and never opts into RAID", () => {
+    const spec = buildVyosConfigSpec({ hostname: "fw4", defaultPassword: "pw" });
+    const address = spec.sets.find(
+      (s) => s.path.join(" ") === "interfaces ethernet eth0 address",
+    );
+    expect(address?.value).toBe("dhcp");
+    // The installer's RAID prompt defaults to yes; a second disk must not
+    // silently produce a mirror.
+    expect(spec.raid).toBe(false);
+  });
+});
+
+describe("vyos routes", () => {
+  let testDir: string;
+  let app: FastifyInstance;
+  let state: StateManager;
+  const mac = "aa:bb:cc:11:22:33";
+
+  beforeEach(() => {
+    testDir = join(tmpdir(), `bastion-vyos-test-${Date.now()}-${Math.random().toString(36).slice(2)}`);
+    mkdirSync(join(testDir, "http"), { recursive: true });
+    mkdirSync(join(testDir, "tftp"), { recursive: true });
+
+    const result = createApp(createTestConfig(testDir));
+    app = result.app;
+    state = result.state;
+  });
+
+  afterEach(async () => {
+    await app.close();
+    rmSync(testDir, { recursive: true, force: true });
+  });
+
+  it("dispatches a queued vyos machine to the live-boot script", async () => {
+    state.update((s) => {
+      s.install_queue[mac] = {
+        hostname: "fw1",
+        disk: "/dev/nvme0n1",
+        role: "worker",
+        os: "vyos-rolling",
+        queued_at: new Date().toISOString(),
+      };
+    });
+
+    const response = await app.inject({ method: "GET", url: `/dispatch?mac=${mac}` });
+    expect(response.statusCode).toBe(200);
+
+    expect(response.body).toContain("/vyos-vmlinuz");
+    expect(response.body).toContain("fetch=http://10.0.0.1:0/vyos-filesystem.squashfs");
+    expect(response.body).toContain(`live-config.hooks=http://10.0.0.1:0/vyos/autoinstall.sh?mac=${mac}`);
+
+    // `nonetworking` appears in VyOS's own PXE docs but breaks the hook fetch,
+    // and console=ttyS0 costs 30s per systemd phase on boards with no UART.
+    expect(response.body).not.toContain("nonetworking");
+    expect(response.body).not.toContain("console=ttyS0");
+  });
+
+  it("serves a hook that fetches and executes the install driver", async () => {
+    const response = await app.inject({ method: "GET", url: `/vyos/autoinstall.sh?mac=${mac}` });
+    expect(response.statusCode).toBe(200);
+    expect(response.body).toContain(`/vyos/install.py?mac=${mac}`);
+    expect(response.body).toContain("python3 /tmp/vyos-install.py");
+  });
+
+  it("bakes the machine's config into the generated install driver", async () => {
+    state.update((s) => {
+      s.install_queue[mac] = {
+        hostname: "fw1",
+        disk: "/dev/nvme0n1",
+        role: "worker",
+        os: "vyos-rolling",
+        queued_at: new Date().toISOString(),
+        vyos: {
+          mgmtInterface: "eth0",
+          mgmtAddress: "10.0.8.2/24",
+          bondMembers: ["eth2", "eth3"],
+          vlans: [{ id: 10, address: "10.0.10.1/24" }],
+          password: "s3cret",
+        },
+      };
+    });
+
+    const response = await app.inject({ method: "GET", url: `/vyos/install.py?mac=${mac}` });
+    expect(response.statusCode).toBe(200);
+
+    // Builds config from the image's own default so the vyos-config-version
+    // trailer matches and first boot skips migrations.
+    expect(response.body).toContain("/opt/vyatta/etc/config.boot.default");
+    expect(response.body).toContain("/usr/libexec/vyos/op_mode/image_installer.py");
+    // "complete" is what moves the machine out of the install queue.
+    expect(response.body).toContain('report("complete"');
+
+    const spec = decodeSpecFrom(response.body);
+    expect(spec["hostname"]).toBe("fw1");
+    expect(spec["password"]).toBe("s3cret");
+    expect(spec["disk"]).toBe("/dev/nvme0n1");
+
+    const paths = (spec["sets"] as Array<{ path: string[] }>).map((s) => s.path.join(" "));
+    expect(paths).toContain("interfaces bonding bond0 vif 10 address");
+    expect(paths).toContain("system host-name");
+  });
+
+  it("falls back to the bastion default password when none is set", async () => {
+    state.update((s) => {
+      s.install_queue[mac] = {
+        hostname: "fw9",
+        disk: "",
+        role: "worker",
+        os: "vyos-rolling",
+        queued_at: new Date().toISOString(),
+      };
+    });
+
+    const response = await app.inject({ method: "GET", url: `/vyos/install.py?mac=${mac}` });
+    const spec = decodeSpecFrom(response.body);
+    expect(spec["password"]).toBe("test-pw");
+    // Empty disk means "accept the installer's first-disk default".
+    expect(spec["disk"]).toBe("");
+  });
+});
+
+describe("vyos hw-id pinning", () => {
+  it("emits hw-id for the mgmt interface and each bond member", () => {
+    // Discovery sees enp2s0/enp1s0f0np0 under Fedora, but VyOS enumerates its
+    // own eth<N>. Pinning by MAC is what makes the mapping deterministic.
+    const spec = buildVyosConfigSpec({
+      hostname: "fw1",
+      defaultPassword: "pw",
+      spec: {
+        mgmtInterface: "eth2",
+        bondMembers: ["eth0", "eth1"],
+        hwIds: {
+          eth2: "64:62:66:25:96:47",
+          eth0: "64:62:66:25:96:45",
+          eth1: "64:62:66:25:96:46",
+        },
+      },
+    });
+
+    const hw = spec.sets.filter((s) => s.path[s.path.length - 1] === "hw-id");
+    expect(hw.map((s) => [s.path[2], s.value])).toEqual([
+      ["eth2", "64:62:66:25:96:47"],
+      ["eth0", "64:62:66:25:96:45"],
+      ["eth1", "64:62:66:25:96:46"],
+    ]);
+  });
+
+  it("omits hw-id entirely when no mapping is given", () => {
+    const spec = buildVyosConfigSpec({ hostname: "fw1", defaultPassword: "pw" });
+    expect(spec.sets.some((s) => s.path.includes("hw-id"))).toBe(false);
+  });
+});
+
+describe("vyos management VLAN", () => {
+  it("puts the mgmt VLAN on the PXE port while the bond carries routed VLANs", () => {
+    // Trunked PXE port: boots untagged on the VLAN the bastion serves, stays
+    // reachable on the tagged management VLAN.
+    const spec = buildVyosConfigSpec({
+      hostname: "vyos001",
+      defaultPassword: "pw",
+      spec: {
+        mgmtInterface: "eth2",
+        mgmtAddress: "dhcp",
+        mgmtVlan: { id: 3, address: "192.168.3.4/24", description: "kvm" },
+        bondMembers: ["eth0", "eth1"],
+        vlans: [{ id: 2, address: "192.168.8.2/23" }],
+      },
+    });
+
+    const paths = spec.sets.map((s) => s.path.join(" "));
+    expect(paths).toContain("interfaces ethernet eth2 vif 3 address");
+    expect(paths).toContain("interfaces bonding bond0 vif 2 address");
+    // The mgmt VLAN must not land on the bond.
+    expect(paths).not.toContain("interfaces bonding bond0 vif 3 address");
+    expect(spec.tags.map((t) => t.join(" "))).toContain("interfaces ethernet eth2 vif");
+  });
+});
+
+describe("vyos VRRP HA", () => {
+  const haSpec = {
+    mgmtInterface: "eth2",
+    mgmtAddress: "dhcp",
+    bondMembers: ["eth0", "eth1"],
+    bondAddress: "192.168.1.252/24",
+    bondVrrp: "192.168.1.254/24",
+    vrrpPriority: 200,
+    vlans: [
+      { id: 3, address: "192.168.3.4/24", vrrp: "192.168.3.254/24" },
+      { id: 200, address: "192.168.2.252/24" },   // no VIP on this one
+    ],
+  };
+
+  it("emits a vrrp group per VIP with vrid = VLAN id and dotted vif interface", () => {
+    const spec = buildVyosConfigSpec({ hostname: "fw1", defaultPassword: "pw", spec: haSpec });
+    const paths = spec.sets.map((s) => `${s.path.join(" ")}${s.value !== undefined ? "=" + s.value : ""}`);
+
+    expect(paths).toContain("interfaces bonding bond0 address=192.168.1.252/24");
+    // untagged bond group: vrid 1, interface bond0 itself
+    expect(paths).toContain("high-availability vrrp group native interface=bond0");
+    expect(paths).toContain("high-availability vrrp group native vrid=1");
+    // address is a tag node -- VIP is the final path segment, no value
+    expect(paths).toContain("high-availability vrrp group native address 192.168.1.254/24");
+    // VLAN group: vrid = VLAN id, dotted vif
+    expect(paths).toContain("high-availability vrrp group vlan3 interface=bond0.3");
+    expect(paths).toContain("high-availability vrrp group vlan3 vrid=3");
+    expect(paths).toContain("high-availability vrrp group vlan3 address 192.168.3.254/24");
+    // VLAN without a VIP gets no group
+    expect(paths.some((p) => p.includes("group vlan200"))).toBe(false);
+  });
+
+  it("applies the box-wide priority and one sync group over all groups", () => {
+    const spec = buildVyosConfigSpec({ hostname: "fw1", defaultPassword: "pw", spec: haSpec });
+    const prio = spec.sets.filter((s) => s.path[s.path.length - 1] === "priority"
+      && s.path[0] === "high-availability");
+    expect(prio).toHaveLength(2);
+    expect(prio.every((s) => s.value === "200")).toBe(true);
+
+    // sync group binds the pair: all groups fail over together
+    const members = spec.sets.filter(
+      (s) => s.path.join(" ") === "high-availability vrrp sync-group MAIN member",
+    );
+    expect(members.map((m) => m.value)).toEqual(["native", "vlan3"]);
+    expect(members.every((m) => m.replace === false)).toBe(true);
+  });
+
+  it("emits no high-availability nodes when no VIPs are given", () => {
+    const spec = buildVyosConfigSpec({
+      hostname: "fw1",
+      defaultPassword: "pw",
+      spec: { bondMembers: ["eth0", "eth1"], vlans: [{ id: 3, address: "192.168.3.4/24" }] },
+    });
+    expect(spec.sets.some((s) => s.path[0] === "high-availability")).toBe(false);
+  });
+});
+
+describe("pickLargestInitrd", async () => {
+  const { pickLargestInitrd } = await import("../src/main.js");
+
+  // Verbatim from `xorriso -lsl /live/` on vyos-2026.08.05-0033-rolling.
+  const realListing = `total 8
+-r--r--r--    1 0        0           22255 Aug  5 01:33 'filesystem.packages'
+-r--r--r--    1 0        0               6 Aug  5 01:33 'filesystem.packages-remove'
+-r--r--r--    1 0        0        541192192 Aug  5 01:33 'filesystem.squashfs'
+-r--r--r--    1 0        0        50352547 Aug  5 01:33 'initrd.img'
+-r--r--r--    1 0        0        50352547 Aug  5 01:33 'initrd.img-6.18.41-vyos'
+-r--r--r--    1 0        0              20 Aug  5 01:33 'packages.txt'
+-r--r--r--    1 0        0         9135104 Aug  2 19:54 'vmlinuz'
+-r--r--r--    1 0        0         9135104 Aug  2 19:54 'vmlinuz-6.18.41-vyos'
+`;
+
+  it("picks a full-size initrd from a real nightly listing", () => {
+    expect(pickLargestInitrd(realListing)).toEqual({ name: "initrd.img", size: 50352547 });
+  });
+
+  it("ignores 0-byte decoys and symlinks (which report link size, not target size)", () => {
+    const listing = `total 8
+-r--r--r--    1 0        0               0 Aug  5 01:33 'initrd.img'
+lrwxrwxrwx    1 0        0              24 Aug  5 01:33 'initrd.img-link' -> 'initrd.img-6.18.41-vyos'
+-r--r--r--    1 0        0        50352547 Aug  5 01:33 'initrd.img-6.18.41-vyos'
+`;
+    expect(pickLargestInitrd(listing)).toEqual({ name: "initrd.img-6.18.41-vyos", size: 50352547 });
+  });
+
+  it("returns undefined when only decoys exist", () => {
+    expect(pickLargestInitrd("-r--r--r-- 1 0 0 0 Aug 5 01:33 'initrd.img'\n")).toBeUndefined();
+  });
+});
+
+describe("vyos fedora-parity features", () => {
+  it("computes reportAddress from a static mgmt address, empty for dhcp", () => {
+    const staticSpec = buildVyosConfigSpec({
+      hostname: "fw1", defaultPassword: "pw",
+      spec: { mgmtAddress: "192.168.8.2/23" },
+    });
+    expect(staticSpec.reportAddress).toBe("192.168.8.2");
+
+    const dhcpSpec = buildVyosConfigSpec({ hostname: "fw1", defaultPassword: "pw" });
+    expect(dhcpSpec.reportAddress).toBe("");
+  });
+
+  it("defaults freshConfig off (reinstall preserves the on-disk config)", () => {
+    expect(buildVyosConfigSpec({ hostname: "fw1", defaultPassword: "pw" }).freshConfig).toBe(false);
+    expect(buildVyosConfigSpec({
+      hostname: "fw1", defaultPassword: "pw", spec: { freshConfig: true },
+    }).freshConfig).toBe(true);
+  });
+
+  it("driver streams logs to /api/log and reports 'ready at' on completion", async () => {
+    const testDir = join(tmpdir(), `bastion-vyos-parity-${Date.now()}`);
+    mkdirSync(join(testDir, "http"), { recursive: true });
+    mkdirSync(join(testDir, "tftp"), { recursive: true });
+    const { app: parityApp, state: parityState } = createApp(createTestConfig(testDir));
+    try {
+      parityState.update((s) => {
+        s.install_queue["aa:bb:cc:44:55:66"] = {
+          hostname: "fw9", disk: "/dev/vda", role: "vanilla",
+          os: "vyos-rolling", queued_at: new Date().toISOString(),
+        };
+      });
+      const response = await parityApp.inject({
+        method: "GET", url: "/vyos/install.py?mac=aa:bb:cc:44:55:66",
+      });
+      expect(response.body).toContain("/api/log");
+      expect(response.body).toContain('"lines": batch');
+      expect(response.body).toContain('report("complete", "ready at %s"');
+      expect(response.body).toContain("ensure_network_boot_first");
+      expect(response.body).toContain("lab-provisioned");
+      expect(response.body).toContain('ROLE = "vanilla"');
+    } finally {
+      await parityApp.close();
+      rmSync(testDir, { recursive: true, force: true });
+    }
+  });
+
+  it("complete with 'ready at' records installed.ip for a vyos machine", async () => {
+    const testDir = join(tmpdir(), `bastion-vyos-complete-${Date.now()}`);
+    mkdirSync(join(testDir, "http"), { recursive: true });
+    mkdirSync(join(testDir, "tftp"), { recursive: true });
+    const { app: cApp, state: cState } = createApp(createTestConfig(testDir));
+    try {
+      const mac2 = "aa:bb:cc:77:88:99";
+      cState.update((s) => {
+        s.install_queue[mac2] = {
+          hostname: "fw1", disk: "/dev/vda", role: "vanilla",
+          os: "vyos-rolling", queued_at: new Date().toISOString(),
+        };
+      });
+      const response = await cApp.inject({
+        method: "POST", url: "/api/progress",
+        headers: { "Content-Type": "application/json" },
+        body: JSON.stringify({ mac: mac2, stage: "complete", detail: "ready at 192.168.8.2" }),
+      });
+      expect(response.statusCode).toBe(200);
+      const installed = cState.load().installed[mac2];
+      expect(installed?.ip).toBe("192.168.8.2");
+      expect(installed?.os).toBe("vyos-rolling");
+    } finally {
+      await cApp.close();
+      rmSync(testDir, { recursive: true, force: true });
+    }
+  });
+});
+
+describe("vyos installer prompt coverage", () => {
+  // Every interactive prompt image_installer.py can emit, copied verbatim from
+  // the MSG_* constants (including the reinstall-only search_previous_installation
+  // ones). An unanswered prompt does not fail loudly -- the installer simply
+  // blocks on stdin until the driver's stall timeout, which is how the reinstall
+  // path silently hung for 15 minutes in the VM test.
+  const PROMPTS: Record<string, string> = {
+    continue: "Would you like to continue? [y/N] ",
+    imageName: "What would you like to name this image? (Default: 1.5-rolling) ",
+    password: 'Please enter a password for the "vyos" user: ',
+    passwordConfirm: 'Please confirm password for the "vyos" user: ',
+    console: "What console should be used by default? (K: KVM, S: Serial)? (Default: K) ",
+    raidConfigure: "Would you like to configure RAID-1 mirroring? [Y/n] ",
+    raidFoundDisks: "Would you like to configure RAID-1 mirroring on them? [Y/n] ",
+    raidChooseDisks: "Would you like to choose two disks for RAID-1 mirroring? [Y/n] ",
+    diskSelect: "Which one should be used for installation? (Default: /dev/vda) ",
+    diskConfirm: "Installation will delete all data on the drive. Continue? [y/N] ",
+    raidConfirm: "Installation will delete all data on both drives. Continue? [y/N] ",
+    rootSizeAll: "Would you like to use all the free space on the drive? [Y/n] ",
+    bootConfig: "Which file would you like as boot config? ",
+    copyData: "Would you like to copy data to the new image? [Y/n] ",
+    chooseCopyData: "From which image would you like to save config information? ",
+    copyEncData: "Would you like to copy the encrypted config to the new image? [Y/n] ",
+    chooseCopyEncData: "From which image would you like to copy the encrypted config? ",
+  };
+
+  it("answers every installer prompt exactly once", () => {
+    const { execFileSync } = require("node:child_process") as typeof import("node:child_process");
+    const { writeFileSync, unlinkSync, mkdtempSync } = require("node:fs") as typeof import("node:fs");
+
+    // Skip cleanly where python3 is unavailable (same spirit as the
+    // ksvalidator-backed kickstart test).
+    try {
+      execFileSync("python3", ["--version"], { stdio: "pipe" });
+    } catch {
+      return;
+    }
+
+    const spec = buildVyosConfigSpec({
+      hostname: "fw1", defaultPassword: "pw", disk: "/dev/vda",
+    });
+    const driver = renderVyosInstallPy({
+      spec, mac: "aa:bb:cc:11:22:33", serverIp: "10.0.0.1", httpPort: 8080, role: "vanilla",
+    });
+
+    const dir = mkdtempSync(join(tmpdir(), "vyos-rules-"));
+    const driverPath = join(dir, "driver.py");
+    const checkPath = join(dir, "check.py");
+    writeFileSync(driverPath, driver);
+    writeFileSync(checkPath, `
+import importlib.util, json, sys
+spec = importlib.util.spec_from_file_location("drv", ${JSON.stringify(driverPath)})
+drv = importlib.util.module_from_spec(spec); spec.loader.exec_module(drv)
+rules = drv.build_rules()
+prompts = json.loads(sys.argv[1])
+out = {}
+for label, text in prompts.items():
+    out[label] = len([r for p, r in rules if p.search(text.encode())])
+print(json.dumps(out))
+`);
+
+    try {
+      const stdout = execFileSync("python3", [checkPath, JSON.stringify(PROMPTS)], {
+        encoding: "utf-8", stdio: ["pipe", "pipe", "pipe"],
+      });
+      const counts = JSON.parse(stdout) as Record<string, number>;
+      const unanswered = Object.entries(counts).filter(([, n]) => n !== 1);
+      expect(unanswered).toEqual([]);
+    } finally {
+      try { unlinkSync(driverPath); unlinkSync(checkPath); } catch { /* best effort */ }
+      try { rmSync(dir, { recursive: true, force: true }); } catch { /* best effort */ }
+    }
+  });
+});
+
+describe("vyos boot NIC pinning", () => {
+  it("pins the boot interface by MAC via BOOTIF", async () => {
+    // Without this, live-boot picks the first *connected* NIC. On the VP2440
+    // the SFP+ pair links before the copper PXE port, so live-boot tried the
+    // fiber ports (no DHCP), timed out 15s each, and failed with "Unable to
+    // find a live file system on the network".
+    const testDir = join(tmpdir(), `bastion-vyos-bootif-${Date.now()}`);
+    mkdirSync(join(testDir, "http"), { recursive: true });
+    mkdirSync(join(testDir, "tftp"), { recursive: true });
+    const { app: a, state: st } = createApp(createTestConfig(testDir));
+    try {
+      const m = "64:62:66:25:96:47";
+      st.update((s) => {
+        s.install_queue[m] = {
+          hostname: "vyos001", disk: "/dev/mmcblk0", role: "vanilla",
+          os: "vyos-rolling", queued_at: new Date().toISOString(),
+        };
+      });
+      const res = await a.inject({ method: "GET", url: `/dispatch?mac=${m}` });
+      // live-boot's Device_from_bootif() expects 01-<mac with dashes>
+      expect(res.body).toContain("BOOTIF=01-64-62-66-25-96-47");
+      // and it must be on the kernel line, before fetch= is attempted
+      const kernelLine = res.body.split("\n").find((l) => l.startsWith("kernel "));
+      expect(kernelLine).toContain("BOOTIF=01-64-62-66-25-96-47");
+      expect(kernelLine).toContain("fetch=");
+    } finally {
+      await a.close();
+      rmSync(testDir, { recursive: true, force: true });
+    }
+  });
+});

bastion/src/cli/src/api/client.ts

@@ -90,6 +90,7 @@ export class LabdClient {
 
   async installMachine(opts: {
     mac: string; hostname: string; disk?: string; role?: string; os?: string;
+    vyos?: import("@lab/shared").VyosInstallSpec;
   }): Promise<{ status: string; data?: unknown; error?: string }> {
     return this.request("POST", "/api/machines/install", { body: opts });
   }
@@ -110,7 +111,6 @@ export class LabdClient {
     memory_gb?: number; arch?: string;
     disks?: Array<{ name: string; size_gb: number; model: string }>;
     nics?: Array<{ name: string; mac: string; state: string }>;
-    root_device?: string; root_args?: string;
   }): Promise<{ status: string; error?: string }> {
     return this.request("POST", "/api/machines/discover", { body: data });
   }

bastion/src/cli/src/commands/debug.ts

@@ -8,7 +8,6 @@ import { join } from "node:path";
 import { Command } from "commander";
 import type { BastionState } from "@lab/shared";
 import { getLabdClient } from "../api/config.js";
-import { ROOT_DEVICE_PROBE, parseRootProbe } from "../utils/hardware-probe.js";
 
 /** Resolve a target (hostname, MAC, or IP) to {mac, hostname, ip} from state. */
 function resolveTarget(
@@ -45,54 +44,6 @@ function resolveTarget(
   return null;
 }
 
-/** The local admin account to SSH as (root is not usable — it has no key here). */
-function sshUser(): string {
-  const adminUser = process.env["SUDO_USER"] ?? process.env["USER"] ?? "";
-  return adminUser === "root" ? "" : adminUser;
-}
-
-/** Common ssh arguments, ending with user@host. Null when there is no usable user. */
-function sshBaseArgs(ip: string): string[] | null {
-  const user = sshUser();
-  if (user === "") return null;
-
-  const sudoUser = process.env["SUDO_USER"];
-  const realHome = sudoUser !== undefined ? join("/home", sudoUser) : homedir();
-  const sshKey = ["id_ed25519", "id_rsa", "id_ecdsa"]
-    .map((name) => join(realHome, ".ssh", name))
-    .find((k) => existsSync(k));
-
-  return [
-    "-o", "StrictHostKeyChecking=no",
-    "-o", "UserKnownHostsFile=/dev/null",
-    "-o", "ConnectTimeout=10",
-    ...(sshKey !== undefined ? ["-i", sshKey] : []),
-    `${user}@${ip}`,
-  ];
-}
-
-/**
- * Run a shell script on the target as root and return its stdout, or null.
- *
- * The script goes over stdin rather than the command line so it can contain quotes
- * without a second round of shell escaping. `sudo -n` fails fast instead of hanging on
- * a password prompt that would then eat the script.
- */
-function sshCapture(ip: string, script: string): string | null {
-  const base = sshBaseArgs(ip);
-  if (base === null) return null;
-  try {
-    return execFileSync("ssh", [...base, "sudo", "-n", "sh", "-s"], {
-      input: script,
-      encoding: "utf-8",
-      stdio: ["pipe", "pipe", "pipe"],
-      timeout: 30_000,
-    });
-  } catch {
-    return null;
-  }
-}
-
 export function registerDebugCommand(parent: Command): void {
   parent
     .command("debug <target>")
@@ -120,31 +71,6 @@ export function registerDebugCommand(parent: Command): void {
       }
 
       const { mac, hostname, ip } = resolved;
-
-      // --pxe-boot needs a root= for the installed system. If the machine is still
-      // reachable, observe it now rather than assuming a disk layout: a wrong root=
-      // leaves the machine unbootable. If it isn't reachable, dispatch falls back to
-      // rescue and the operator reports the real one from there.
-      if (opts.pxeBoot === true && ip !== "") {
-        const known = state.installed[mac]?.root_device ?? state.discovered[mac]?.root_device;
-        if (known === undefined || known === "") {
-          console.log(`No root device recorded for ${hostname}. Probing over SSH...`);
-          const probe = sshCapture(ip, ROOT_DEVICE_PROBE);
-          const root = probe === null ? {} : parseRootProbe(probe);
-          if (root.root_device !== undefined) {
-            console.log(`  root=${root.root_device}${root.root_args !== undefined ? ` ${root.root_args}` : ""}`);
-            try {
-              await client.discoverMachine({ mac, ...root });
-            } catch (err) {
-              console.error(`  Could not record it: ${err instanceof Error ? err.message : String(err)}`);
-            }
-          } else {
-            console.log("  Probe failed. Booting rescue instead; report the root device with:");
-            console.log("    curl http://<bastion>:8080/debug-setup.sh | bash");
-          }
-        }
-      }
-
       console.log(`Queuing debug mode for ${hostname} (${mac})...`);
 
       try {
@@ -160,15 +86,32 @@ export function registerDebugCommand(parent: Command): void {
 
       // Try SSH reboot into PXE
       if (ip !== "") {
-        const base = sshBaseArgs(ip);
-        if (base !== null) {
-          console.log(`\nAttempting SSH reboot into PXE (${sshUser()}@${ip})...`);
+        const adminUser = process.env["SUDO_USER"] ?? process.env["USER"] ?? "";
+        const effectiveUser = adminUser === "root" ? "" : adminUser;
+
+        if (effectiveUser !== "") {
+          console.log(`\nAttempting SSH reboot into PXE (${effectiveUser}@${ip})...`);
+
+          const sudoUser = process.env["SUDO_USER"];
+          const realHome = sudoUser !== undefined ? join("/home", sudoUser) : homedir();
+          const keyPaths = [
+            join(realHome, ".ssh", "id_ed25519"),
+            join(realHome, ".ssh", "id_rsa"),
+            join(realHome, ".ssh", "id_ecdsa"),
+          ];
+          const sshKey = keyPaths.find(k => existsSync(k));
+
+          const sshArgs = [
+            "-o", "StrictHostKeyChecking=no",
+            "-o", "UserKnownHostsFile=/dev/null",
+            "-o", "ConnectTimeout=10",
+            ...(sshKey !== undefined ? ["-i", sshKey] : []),
+            `${effectiveUser}@${ip}`,
+            'PXE_ENTRY=$(sudo efibootmgr | grep -iE "pxe|network|ipv4" | head -1 | grep -oP "Boot\\K[0-9A-F]+"); if [ -n "$PXE_ENTRY" ]; then sudo efibootmgr --bootnext "$PXE_ENTRY" && echo "PXE set as next boot" && sudo reboot; else echo "No PXE boot entry found, rebooting anyway..." && sudo reboot; fi',
+          ];
 
           try {
-            execFileSync("ssh", [
-              ...base,
-              'PXE_ENTRY=$(sudo efibootmgr | grep -iE "pxe|network|ipv4" | head -1 | grep -oP "Boot\\K[0-9A-F]+"); if [ -n "$PXE_ENTRY" ]; then sudo efibootmgr --bootnext "$PXE_ENTRY" && echo "PXE set as next boot" && sudo reboot; else echo "No PXE boot entry found, rebooting anyway..." && sudo reboot; fi',
-            ], { stdio: "inherit" });
+            execFileSync("ssh", sshArgs, { stdio: "inherit" });
           } catch {
             // SSH connection closing during reboot is expected
           }

bastion/src/cli/src/commands/install.ts

@@ -1,10 +1,61 @@
 // CLI command: provision install
 // Queue a discovered machine for OS installation via labd.
 
-import { Command, Option } from "commander";
+import { readFileSync } from "node:fs";
+import { Command, Option, InvalidArgumentError } from "commander";
 import { isValidOsId, SUPPORTED_OS, SUPPORTED_ROLES, ROLE_REGISTRY } from "@lab/shared";
+import type { VyosBundle, VyosInstallSpec, VyosVlanSpec } from "@lab/shared";
 import { getLabdClient } from "../api/config.js";
 
+/**
+ * Load one router's config out of a Pulumi-rendered bundle.
+ *
+ * The bundle is produced by `kubernetes-deployment` (npm run vyos:bundle) and
+ * holds every router it manages, keyed by name. Selecting by hostname here is
+ * what keeps bring-up and `pulumi up` describing the same box: labctl replays
+ * the declared config rather than deriving its own.
+ */
+export function loadVyosBundle(path: string, hostname: string): VyosBundle {
+  let parsed: { version?: number; routers?: Record<string, VyosBundle> };
+  try {
+    parsed = JSON.parse(readFileSync(path, "utf8"));
+  } catch (e) {
+    throw new InvalidArgumentError(`Cannot read VyOS bundle ${path}: ${(e as Error).message}`);
+  }
+  if (parsed.version !== 1) {
+    throw new InvalidArgumentError(
+      `VyOS bundle ${path} has version ${parsed.version ?? "<none>"}; this labctl understands 1`,
+    );
+  }
+  const router = parsed.routers?.[hostname];
+  if (router === undefined) {
+    const known = Object.keys(parsed.routers ?? {}).join(", ") || "<none>";
+    throw new InvalidArgumentError(
+      `VyOS bundle ${path} has no entry for "${hostname}" (has: ${known})`,
+    );
+  }
+  return router;
+}
+
+/** Parse a repeated --vlan flag: "<id>:<cidr>[:<description>]". */
+export function parseVlan(value: string, previous: VyosVlanSpec[] = []): VyosVlanSpec[] {
+  const parts = value.split(":");
+  const id = Number(parts[0]);
+  const address = parts[1] ?? "";
+  // InvalidArgumentError makes commander print a clean message instead of
+  // dumping a stack trace at the operator.
+  if (!Number.isInteger(id) || id < 1 || id > 4094) {
+    throw new InvalidArgumentError(`Invalid VLAN id in "${value}" (expected 1-4094)`);
+  }
+  if (!address.includes("/")) {
+    throw new InvalidArgumentError(
+      `Invalid VLAN address in "${value}" (expected CIDR, e.g. 10.0.10.1/24)`,
+    );
+  }
+  const description = parts.slice(2).join(":");
+  return [...previous, { id, address, ...(description ? { description } : {}) }];
+}
+
 function roleTable(): string {
   const lines: string[] = ["", "Available roles:"];
   for (const r of ROLE_REGISTRY) {
@@ -15,6 +66,38 @@ function roleTable(): string {
   return lines.join("\n");
 }
 
+/** Parse a repeated --vlan-vip flag: "<id>:<cidr>" — VRRP VIP for a --vlan entry. */
+export function parseVlanVip(
+  value: string,
+  previous: Record<number, string> = {},
+): Record<number, string> {
+  const index = value.indexOf(":");
+  const id = Number(index === -1 ? Number.NaN : value.slice(0, index));
+  const cidr = index === -1 ? "" : value.slice(index + 1).trim();
+  if (!Number.isInteger(id) || id < 1 || id > 4094 || !cidr.includes("/")) {
+    throw new InvalidArgumentError(
+      `Invalid VLAN VIP "${value}" (expected <id>:<cidr>, e.g. 3:192.168.3.254/24)`,
+    );
+  }
+  return { ...previous, [id]: cidr };
+}
+
+/** Parse a repeated --vyos-hwid flag: "<iface>=<mac>". */
+export function parseHwId(
+  value: string,
+  previous: Record<string, string> = {},
+): Record<string, string> {
+  const index = value.indexOf("=");
+  const iface = index === -1 ? "" : value.slice(0, index).trim();
+  const mac = index === -1 ? "" : value.slice(index + 1).trim().toLowerCase();
+  if (iface === "" || !/^([0-9a-f]{2}:){5}[0-9a-f]{2}$/.test(mac)) {
+    throw new InvalidArgumentError(
+      `Invalid hw-id "${value}" (expected <iface>=<mac>, e.g. eth2=64:62:66:25:96:47)`,
+    );
+  }
+  return { ...previous, [iface]: mac };
+}
+
 export function registerInstallCommand(parent: Command): void {
   parent
     .command("install <mac> <hostname>")
@@ -24,10 +107,51 @@ export function registerInstallCommand(parent: Command): void {
     .addOption(new Option("--role <role>", "Machine role (see below)").choices([...SUPPORTED_ROLES]).default("worker"))
     .addOption(new Option("--os <os>", "Operating system").choices([...SUPPORTED_OS]).default("fedora-43"))
     .option("--disk <device>", "Target disk device (auto-detect if omitted)")
+    .option("--vyos-mgmt <iface>", "VyOS: untagged interface the machine PXE boots from (default eth0)")
+    .option("--vyos-mgmt-address <addr>", "VyOS: CIDR for the management interface, or 'dhcp' (default dhcp)")
+    .option("--vyos-bond <ifaces>", "VyOS: comma-separated LACP bond members (must exclude the PXE NIC)")
+    .option("--vyos-bond-address <cidr>", "VyOS: address on the untagged bond (trunk native VLAN)")
+    .option("--vyos-bond-vrrp <cidr>", "VyOS: VRRP VIP floated on the untagged bond")
+    .option("--vlan-vip <id:cidr>", "VyOS: VRRP VIP for a --vlan entry (repeatable)", parseVlanVip)
+    .option("--vyos-vrrp-priority <n>", "VyOS: VRRP priority for all groups on this box (higher = master)")
+    .option("--vyos-mgmt-vlan <id:cidr[:desc]>", "VyOS: tagged management VLAN on the PXE port")
+    .option("--vlan <id:cidr[:desc]>", "VyOS: tagged VLAN sub-interface on the bond (repeatable)", parseVlan)
+    .option("--vyos-password <password>", "VyOS: password for the 'vyos' user")
+    .option("--vyos-hwid <iface=mac>", "VyOS: pin an interface name to a MAC via hw-id (repeatable)", parseHwId)
+    .option("--vyos-fresh-config", "VyOS: on reinstall, overwrite the preserved config with the generated one")
+    .option(
+      "--vyos-bundle <path>",
+      "VyOS: apply a Pulumi-rendered bundle verbatim (kubernetes-deployment/infra/vyos/vyos-bundle.json). " +
+        "Replaces the derived --vyos-bond/--vlan/... config; secret values are left unset for `pulumi up`.",
+    )
+    .option(
+      "--vyos-api-key <key>",
+      "VyOS: enable the HTTP API with this key so Pulumi can manage the router from first boot",
+    )
+    .option(
+      "--vyos-api-listen <addr>",
+      "VyOS: address the HTTP API binds to (default: the static management address). " +
+        "Required when management is DHCP; the API is never bound to all interfaces.",
+    )
     .action(async (mac: string, hostname: string, opts: {
       role: string;
       os: string;
       disk?: string;
+      vyosMgmt?: string;
+      vyosMgmtAddress?: string;
+      vyosBond?: string;
+      vyosBondAddress?: string;
+      vyosBondVrrp?: string;
+      vlan?: VyosVlanSpec[];
+      vlanVip?: Record<number, string>;
+      vyosVrrpPriority?: string;
+      vyosMgmtVlan?: string;
+      vyosPassword?: string;
+      vyosHwid?: Record<string, string>;
+      vyosFreshConfig?: boolean;
+      vyosBundle?: string;
+      vyosApiKey?: string;
+      vyosApiListen?: string;
     }) => {
       if (!isValidOsId(opts.os)) {
         console.error(`Unknown OS: ${opts.os}. Supported: ${SUPPORTED_OS.join(", ")}`);
@@ -39,6 +163,89 @@ export function registerInstallCommand(parent: Command): void {
         process.exit(1);
       }
 
+      const bondMembers = opts.vyosBond !== undefined && opts.vyosBond !== ""
+        ? opts.vyosBond.split(",").map((s) => s.trim()).filter((s) => s.length > 0)
+        : [];
+
+      // Attach --vlan-vip entries to their --vlan definitions. A VIP for a VLAN
+      // that was never defined is a typo that would otherwise vanish silently.
+      const vips = opts.vlanVip ?? {};
+      const vlans = (opts.vlan ?? []).map((v) =>
+        vips[v.id] !== undefined ? { ...v, vrrp: vips[v.id] as string } : v,
+      );
+      for (const id of Object.keys(vips)) {
+        if (!vlans.some((v) => String(v.id) === id)) {
+          console.error(`--vlan-vip ${id}:... has no matching --vlan ${id}:... entry`);
+          process.exit(1);
+        }
+      }
+
+      const vrrpPriority = opts.vyosVrrpPriority !== undefined && opts.vyosVrrpPriority !== ""
+        ? Number(opts.vyosVrrpPriority)
+        : undefined;
+      if (vrrpPriority !== undefined
+          && (!Number.isInteger(vrrpPriority) || vrrpPriority < 1 || vrrpPriority > 255)) {
+        console.error(`--vyos-vrrp-priority must be an integer 1-255 (got ${opts.vyosVrrpPriority})`);
+        process.exit(1);
+      }
+      const vyos: VyosInstallSpec = {
+        ...(opts.vyosMgmt !== undefined && opts.vyosMgmt !== ""
+          ? { mgmtInterface: opts.vyosMgmt } : {}),
+        ...(opts.vyosMgmtAddress !== undefined && opts.vyosMgmtAddress !== ""
+          ? { mgmtAddress: opts.vyosMgmtAddress } : {}),
+        ...(bondMembers.length > 0 ? { bondMembers } : {}),
+        ...(opts.vyosBondAddress !== undefined && opts.vyosBondAddress !== ""
+          ? { bondAddress: opts.vyosBondAddress } : {}),
+        ...(opts.vyosBondVrrp !== undefined && opts.vyosBondVrrp !== ""
+          ? { bondVrrp: opts.vyosBondVrrp } : {}),
+        ...(vrrpPriority !== undefined ? { vrrpPriority } : {}),
+        ...(vlans.length > 0 ? { vlans } : {}),
+        ...(opts.vyosPassword !== undefined && opts.vyosPassword !== ""
+          ? { password: opts.vyosPassword } : {}),
+        ...(opts.vyosHwid !== undefined && Object.keys(opts.vyosHwid).length > 0
+          ? { hwIds: opts.vyosHwid } : {}),
+        ...(opts.vyosMgmtVlan !== undefined && opts.vyosMgmtVlan !== ""
+          ? { mgmtVlan: parseVlan(opts.vyosMgmtVlan)[0] as VyosVlanSpec } : {}),
+        ...(opts.vyosFreshConfig === true ? { freshConfig: true } : {}),
+        ...(opts.vyosBundle !== undefined && opts.vyosBundle !== ""
+          ? { bundle: loadVyosBundle(opts.vyosBundle, hostname) } : {}),
+        ...(opts.vyosApiKey !== undefined && opts.vyosApiKey !== ""
+          ? { apiKey: opts.vyosApiKey } : {}),
+        ...(opts.vyosApiListen !== undefined && opts.vyosApiListen !== ""
+          ? { apiListenAddress: opts.vyosApiListen } : {}),
+      };
+      const hasVyosOptions = Object.keys(vyos).length > 0;
+
+      // A bundle already describes the whole router. Accepting derived topology
+      // flags alongside it would silently discard them (the bundle wins in
+      // buildVyosConfigSpec), so say so rather than appear to honour both.
+      if (vyos.bundle !== undefined) {
+        const derived = ["mgmtInterface", "mgmtAddress", "bondMembers", "bondAddress",
+                         "bondVrrp", "vrrpPriority", "vlans", "mgmtVlan"] as const;
+        const conflicting = derived.filter((k) => vyos[k] !== undefined);
+        if (conflicting.length > 0) {
+          console.error(
+            `--vyos-bundle describes the whole router; these would be ignored: ${conflicting.join(", ")}`,
+          );
+          console.error("Remove them, or change the bundle in kubernetes-deployment and re-render.");
+          process.exit(1);
+        }
+      }
+
+      if (hasVyosOptions && !opts.os.startsWith("vyos")) {
+        console.error(`VyOS options require --os vyos-rolling (got --os ${opts.os})`);
+        process.exit(1);
+      }
+
+      // Firmware PXE cannot run over LACP, so the NIC that boots the installer
+      // must stay out of the bond — otherwise the next reinstall has no path in.
+      const mgmt = vyos.mgmtInterface ?? "eth0";
+      if (bondMembers.includes(mgmt)) {
+        console.error(`--vyos-bond must not include the PXE/management interface "${mgmt}"`);
+        console.error("PXE cannot boot over an LACP bond; keep that NIC unbonded.");
+        process.exit(1);
+      }
+
       try {
         const result = await getLabdClient().installMachine({
           mac,
@@ -46,11 +253,14 @@ export function registerInstallCommand(parent: Command): void {
           role: opts.role,
           os: opts.os,
           ...(opts.disk ? { disk: opts.disk } : {}),
+          ...(hasVyosOptions ? { vyos } : {}),
         });
 
         console.log(JSON.stringify(result, null, 2));
         console.log("");
-        const osLabel = opts.os.startsWith("ubuntu") ? "Ubuntu" : "Fedora";
+        const osLabel = opts.os.startsWith("ubuntu")
+          ? "Ubuntu"
+          : opts.os.startsWith("vyos") ? "VyOS" : "Fedora";
         console.log(`Power on the machine to start ${osLabel} installation.`);
 
         const roleInfo = ROLE_REGISTRY.find(r => r.name === opts.role);

bastion/src/cli/src/commands/recheck.ts

@@ -4,7 +4,6 @@
 import type { Command } from "commander";
 import { sshExec } from "@lab/modules";
 import { getLabdClient } from "../api/config.js";
-import { ROOT_DEVICE_PROBE } from "../utils/hardware-probe.js";
 
 const BOLD = "\x1b[1m";
 const GREEN = "\x1b[0;32m";
@@ -25,9 +24,7 @@ const HW_COLLECT_SCRIPT = [
   'N=$(grep -c "^processor" /proc/cpuinfo 2>/dev/null || echo 0)',
   'R=$(awk "/MemTotal/ {printf \\"%d\\", \\$2/1024/1024}" /proc/meminfo 2>/dev/null || echo 0)',
   'A=$(uname -m)',
-  // Root filesystem, so --pxe-boot has a root= to use instead of assuming our layout.
-  ROOT_DEVICE_PROBE,
-  'printf \'{"product":"%s","board":"%s","serial":"%s","manufacturer":"%s","cpu_model":"%s","cpu_cores":%s,"memory_gb":%s,"arch":"%s","root_device":"%s","root_args":"%s"}\\n\' "$P" "$B" "$S" "$M" "$C" "$N" "$R" "$A" "$RD" "$RA"',
+  'printf \'{"product":"%s","board":"%s","serial":"%s","manufacturer":"%s","cpu_model":"%s","cpu_cores":%s,"memory_gb":%s,"arch":"%s"}\\n\' "$P" "$B" "$S" "$M" "$C" "$N" "$R" "$A"',
 ].join("; ");
 
 export function registerRecheckCommand(parent: Command): void {
@@ -47,11 +44,14 @@ export function registerRecheckCommand(parent: Command): void {
       }
 
       // Build list of machines to check
-      const targets: Array<{ mac: string; hostname: string; ip: string }> = [];
+      const targets: Array<{ mac: string; hostname: string; ip: string; sshUser: string }> = [];
+      const userIsDefault = opts.user === "root";
       for (const [mac, info] of Object.entries(state.installed)) {
         if (!info.ip) continue;
         if (opts.target && info.hostname !== opts.target && mac !== opts.target) continue;
-        targets.push({ mac, hostname: info.hostname, ip: info.ip });
+        // VyOS boxes only have the "vyos" login; honor an explicit --user.
+        const sshUser = userIsDefault && (info.os ?? "").startsWith("vyos") ? "vyos" : opts.user;
+        targets.push({ mac, hostname: info.hostname, ip: info.ip, sshUser });
       }
 
       if (targets.length === 0) {
@@ -64,12 +64,12 @@ export function registerRecheckCommand(parent: Command): void {
       let updated = 0;
       let failed = 0;
 
-      for (const { mac, hostname, ip } of targets) {
+      for (const { mac, hostname, ip, sshUser } of targets) {
         process.stdout.write(`  ${hostname.padEnd(24)} ${DIM}(${ip})${RESET}  `);
 
         try {
           const t0 = Date.now();
-          const result = await sshExec(ip, opts.user, HW_COLLECT_SCRIPT, SSH_OPTS);
+          const result = await sshExec(ip, sshUser, HW_COLLECT_SCRIPT, SSH_OPTS);
           const elapsed = Date.now() - t0;
           if (result.exitCode !== 0) {
             console.log(`${RED}SSH failed (exit ${result.exitCode}, ${elapsed}ms)${RESET}`);
@@ -84,10 +84,7 @@ export function registerRecheckCommand(parent: Command): void {
           const cpu = hwData.cpu_model || "?";
           const cores = hwData.cpu_cores || "?";
           const mem = hwData.memory_gb || "?";
-          const root = typeof hwData.root_device === "string" && hwData.root_device !== ""
-            ? `, root=${hwData.root_device}`
-            : "";
-          console.log(`${GREEN}OK${RESET} ${DIM}${cpu}, ${cores} cores, ${mem}GB${root}${RESET}`);
+          console.log(`${GREEN}OK${RESET} ${DIM}${cpu}, ${cores} cores, ${mem}GB${RESET}`);
           updated++;
         } catch (err) {
           console.log(`${RED}FAIL${RESET} ${DIM}${err instanceof Error ? err.message : String(err)}${RESET}`);

bastion/src/cli/src/commands/reprovision.ts

@@ -24,12 +24,12 @@ function roleTable(): string {
 function resolveTarget(
   target: string,
   state: BastionState,
-): { mac: string; hostname: string; ip: string } | null {
+): { mac: string; hostname: string; ip: string; os?: string } | null {
   const normalized = target.toLowerCase().replace(/-/g, ":");
 
   if (state.installed[normalized]) {
     const info = state.installed[normalized];
-    return { mac: normalized, hostname: info.hostname, ip: info.ip };
+    return { mac: normalized, hostname: info.hostname, ip: info.ip, ...(info.os !== undefined ? { os: info.os } : {}) };
   }
 
   if (state.discovered[normalized]) {
@@ -38,13 +38,13 @@ function resolveTarget(
 
   for (const [mac, info] of Object.entries(state.installed)) {
     if (info.hostname === target || info.hostname.startsWith(target + ".")) {
-      return { mac, hostname: info.hostname, ip: info.ip };
+      return { mac, hostname: info.hostname, ip: info.ip, ...(info.os !== undefined ? { os: info.os } : {}) };
     }
   }
 
   for (const [mac, info] of Object.entries(state.installed)) {
     if (info.ip === target) {
-      return { mac, hostname: info.hostname, ip: info.ip };
+      return { mac, hostname: info.hostname, ip: info.ip, ...(info.os !== undefined ? { os: info.os } : {}) };
     }
   }
 
@@ -60,10 +60,12 @@ export function registerReprovisionCommand(parent: Command): void {
     .addOption(new Option("--role <role>", "Machine role (see below)").choices([...SUPPORTED_ROLES]).default("worker"))
     .addOption(new Option("--os <os>", "Operating system").choices([...SUPPORTED_OS]).default("fedora-43"))
     .option("--disk <device>", "Target disk device (auto-detect if omitted)")
+    .option("--user <user>", "SSH user for the reboot (default: vyos for VyOS machines, else current user)")
     .action(async (target: string, hostnameOverride: string | undefined, opts: {
       role: string;
       os: string;
       disk?: string;
+      user?: string;
     }) => {
       if (!isValidOsId(opts.os)) {
         console.error(`Unknown OS: ${opts.os}. Supported: ${SUPPORTED_OS.join(", ")}`);
@@ -123,7 +125,11 @@ export function registerReprovisionCommand(parent: Command): void {
         return;
       }
 
-      const adminUser = process.env["SUDO_USER"] ?? process.env["USER"] ?? "";
+      // SSH user: explicit flag > the machine's current OS (VyOS boxes only
+      // have the "vyos" login) > the invoking user.
+      const currentOsIsVyos = (resolved.os ?? "").startsWith("vyos");
+      const adminUser = opts.user
+        ?? (currentOsIsVyos ? "vyos" : (process.env["SUDO_USER"] ?? process.env["USER"] ?? ""));
       const effectiveUser = adminUser === "root" ? "" : adminUser;
 
       if (effectiveUser === "") {

bastion/src/cli/src/utils/hardware-probe.ts

@@ -1,59 +0,0 @@
-// Shell snippets for observing a machine's hardware over SSH.
-//
-// Pure shell + awk, no Python: these run on whatever the target happens to be,
-// including a minimal rescue environment.
-
-/**
- * Report the root filesystem and any dracut arguments needed to assemble it.
- *
- * Emits two lines:
- *   ROOT_DEVICE=<device>
- *   ROOT_ARGS=<args>
- *
- * Used by `--pxe-boot`, which boots the installed system with a kernel and initrd from
- * the network. Getting root= wrong there leaves the machine unbootable, so this observes
- * the machine rather than assuming our Fedora LVM layout.
- *
- * Device form is chosen for stability across reboots: LVM logical volumes keep their
- * /dev/mapper path, anything else is reported by UUID, which survives device renumbering.
- */
-export const ROOT_DEVICE_PROBE = [
-  'RD=$(findmnt -no SOURCE / 2>/dev/null | head -1)',
-  'RA=""',
-  'RT=$(lsblk -no TYPE "$RD" 2>/dev/null | head -1)',
-  'if [ "$RT" = "lvm" ]; then',
-  '  VGLV=$(lvs --noheadings -o vg_name,lv_name "$RD" 2>/dev/null | awk \'{print $1"/"$2}\')',
-  '  [ -n "$VGLV" ] && RA="rd.lvm.lv=$VGLV"',
-  // Swap must be assembled too or resume= stalls the boot waiting for it.
-  '  SW=$(awk \'NR>1 {print $1; exit}\' /proc/swaps 2>/dev/null)',
-  '  if [ -n "$SW" ] && [ "$(lsblk -no TYPE "$SW" 2>/dev/null | head -1)" = "lvm" ]; then',
-  '    SWVGLV=$(lvs --noheadings -o vg_name,lv_name "$SW" 2>/dev/null | awk \'{print $1"/"$2}\')',
-  '    [ -n "$SWVGLV" ] && [ "$SWVGLV" != "$VGLV" ] && RA="$RA rd.lvm.lv=$SWVGLV"',
-  '  fi',
-  'elif [ -n "$RD" ]; then',
-  '  U=$(findmnt -no UUID / 2>/dev/null | head -1)',
-  '  [ -n "$U" ] && RD="UUID=$U"',
-  'fi',
-  'printf \'ROOT_DEVICE=%s\\nROOT_ARGS=%s\\n\' "$RD" "$RA"',
-].join("; ");
-
-export interface RootInfo {
-  root_device?: string;
-  root_args?: string;
-}
-
-/** Parse the ROOT_DEVICE/ROOT_ARGS lines emitted by ROOT_DEVICE_PROBE. */
-export function parseRootProbe(stdout: string): RootInfo {
-  const out: RootInfo = {};
-  for (const line of stdout.split("\n")) {
-    const trimmed = line.trim();
-    if (trimmed.startsWith("ROOT_DEVICE=")) {
-      const v = trimmed.slice("ROOT_DEVICE=".length).trim();
-      if (v !== "") out.root_device = v;
-    } else if (trimmed.startsWith("ROOT_ARGS=")) {
-      const v = trimmed.slice("ROOT_ARGS=".length).trim();
-      if (v !== "") out.root_args = v;
-    }
-  }
-  return out;
-}

bastion/src/cli/tests/install-vyos.test.ts

@@ -0,0 +1,35 @@
+// Tests for VyOS install option parsing.
+
+import { describe, it, expect } from "vitest";
+import { parseVlan } from "../src/commands/install.js";
+
+describe("parseVlan", () => {
+  it("parses id and CIDR", () => {
+    expect(parseVlan("10:10.0.10.1/24")).toEqual([{ id: 10, address: "10.0.10.1/24" }]);
+  });
+
+  it("accumulates across repeated flags", () => {
+    const first = parseVlan("10:10.0.10.1/24");
+    const both = parseVlan("20:10.0.20.1/24", first);
+    expect(both).toHaveLength(2);
+    expect(both[1]).toEqual({ id: 20, address: "10.0.20.1/24" });
+  });
+
+  it("keeps a description, including one containing colons", () => {
+    expect(parseVlan("30:10.0.30.1/24:mgmt:secondary")).toEqual([
+      { id: 30, address: "10.0.30.1/24", description: "mgmt:secondary" },
+    ]);
+  });
+
+  it("rejects an address that is not CIDR", () => {
+    // A bare address would produce a VyOS config that fails to commit on first
+    // boot, long after the operator has stopped watching.
+    expect(() => parseVlan("10:10.0.10.1")).toThrow(/CIDR/);
+  });
+
+  it("rejects out-of-range and non-numeric VLAN ids", () => {
+    expect(() => parseVlan("0:10.0.10.1/24")).toThrow(/1-4094/);
+    expect(() => parseVlan("4095:10.0.10.1/24")).toThrow(/1-4094/);
+    expect(() => parseVlan("abc:10.0.10.1/24")).toThrow(/1-4094/);
+  });
+});

bastion/src/labd/src/routes/bastions.ts

@@ -10,6 +10,7 @@ import type { FastifyInstance } from "fastify";
 import type { DbClient } from "../server.js";
 import { bastionRegistry } from "../services/bastion-registry.js";
 import { generateRequestId } from "@lab/shared";
+import type { VyosInstallSpec } from "@lab/shared";
 
 const COMMAND_TIMEOUT_MS = 15_000;
 
@@ -163,9 +164,9 @@ export function registerBastionRoutes(app: FastifyInstance, db: DbClient): void
 
   // Queue install — route to correct bastion by MAC
   app.post<{
-    Body: { mac?: string; hostname?: string; disk?: string; role?: string; os?: string };
+    Body: { mac?: string; hostname?: string; disk?: string; role?: string; os?: string; vyos?: VyosInstallSpec };
   }>("/api/machines/install", async (request, reply) => {
-    const { mac, hostname, disk, role, os } = request.body ?? {};
+    const { mac, hostname, disk, role, os, vyos } = request.body ?? {};
     if (!mac || !hostname) {
       return reply.code(400).send({ error: "mac and hostname are required" });
     }
@@ -183,6 +184,7 @@ export function registerBastionRoutes(app: FastifyInstance, db: DbClient): void
           const result = await sendCommand(all[0]!.bastionId, {
             type: "command-install",
             mac, hostname, disk: disk ?? "", role: role ?? "infra", os: os ?? "fedora-43",
+            ...(vyos ? { vyos } : {}),
           });
           return reply.code(result.status === "ok" ? 200 : 500).send(result);
         } catch (err) {
@@ -196,6 +198,7 @@ export function registerBastionRoutes(app: FastifyInstance, db: DbClient): void
       const result = await sendCommand(bastion.bastionId, {
         type: "command-install",
         mac, hostname, disk: disk ?? "", role: role ?? "infra", os: os ?? "fedora-43",
+        ...(vyos ? { vyos } : {}),
       });
       return reply.code(result.status === "ok" ? 200 : 500).send(result);
     } catch (err) {
@@ -299,7 +302,6 @@ export function registerBastionRoutes(app: FastifyInstance, db: DbClient): void
       memory_gb?: number; arch?: string;
       disks?: Array<{ name: string; size_gb: number; model: string }>;
       nics?: Array<{ name: string; mac: string; state: string }>;
-      root_device?: string; root_args?: string;
     };
   }>("/api/machines/discover", async (request, reply) => {
     const data = request.body ?? {};

bastion/src/modules/modules/k3s/bin/render-config.ts

@@ -0,0 +1,42 @@
+#!/usr/bin/env node
+// Render /etc/rancher/k3s/config.yaml using the PRODUCTION generator.
+//
+// Exists so labsim (and anything else) can produce the exact config.yaml a node
+// would get from `labctl install`, without an SSH/OperationContext. Driving the
+// rehearsal through this means the sim tests the same code path production runs
+// -- if generateServerConfig ever changes shape, the sim moves with it.
+//
+// All input via env, so a sim's cloud-init or a shell can call it plainly:
+//
+//   ROLE=infra HOSTNAME=k8s1 IP=172.31.2.11 render-config.ts        # cluster-init server
+//   ROLE=infra HOSTNAME=k8s2 IP=172.31.2.12 \
+//     K3S_SERVER_URL=https://172.31.2.11:6443 K3S_TOKEN=... render-config.ts   # joining server
+//   ROLE=worker HOSTNAME=k8s4 IP=172.31.2.14 K3S_SERVER_URL=... K3S_TOKEN=... render-config.ts  # agent
+//
+// Dual-stack is opt-in and matches K3sConfig exactly: set IPV6, CLUSTER_CIDR,
+// SERVICE_CIDR (comma-separated families) and the generator emits the dual
+// node-ip + CIDRs. Omit them and the output is byte-identical to a v4-only node.
+import { generateServerConfig, generateAgentConfig } from "../src/operations/k3s-config.js";
+import type { K3sConfig } from "../src/types.js";
+import type { Role } from "@lab/shared";
+
+const env = process.env;
+const role = (env.ROLE ?? "worker") as Role;
+const isServer = role === "infra" || role === "labcontroller";
+
+const splitCsv = (v: string | undefined): string[] | undefined =>
+  v ? v.split(",").map((s) => s.trim()).filter(Boolean) : undefined;
+
+const cfg: K3sConfig = {
+  hostname: env.HOSTNAME ?? "node",
+  ip: env.IP ?? (() => { throw new Error("IP is required"); })(),
+  role,
+  k3sServerUrl: env.K3S_SERVER_URL,
+  k3sToken: env.K3S_TOKEN,
+  tlsSans: splitCsv(env.TLS_SANS),
+  ipv6: env.IPV6,
+  clusterCidr: splitCsv(env.CLUSTER_CIDR),
+  serviceCidr: splitCsv(env.SERVICE_CIDR),
+};
+
+process.stdout.write(isServer ? generateServerConfig(cfg) : generateAgentConfig(cfg));

bastion/src/modules/modules/k3s/src/groups/host-prep.ts

@@ -1,21 +1,23 @@
-// Host preparation: kernel modules, sysctl, swap, firewall, SELinux.
+// Host preparation: kernel modules, sysctl, swap, storage, firewall, SELinux.
 
 import type { OperationContext, OperationResult, OperationGroup } from "../types.js";
 import { runSequential } from "../utils.js";
 import { loadKernelModules } from "../operations/kernel-modules.js";
 import { applyCisHardening } from "../operations/sysctl.js";
-import { disableSwap } from "../operations/swap.js";
+import { enableSwap } from "../operations/swap.js";
+import { growRancherLv } from "../operations/rancher-storage.js";
 import { disableFirewall } from "../operations/firewall.js";
 import { setSelinuxPermissive } from "../operations/selinux.js";
 import { enableIscsi } from "../operations/iscsi.js";
 
 export const hostPrepGroup: OperationGroup = {
   name: "host-prep",
-  description: "Prepare host for k3s: kernel modules, sysctl, swap, firewall, SELinux, iSCSI",
+  description: "Prepare host for k3s: kernel modules, sysctl, swap, imageFs sizing, firewall, SELinux, iSCSI",
   operations: [
     { name: "Load kernel modules", fn: loadKernelModules },
     { name: "Apply CIS sysctl", fn: applyCisHardening },
-    { name: "Disable swap", fn: disableSwap },
+    { name: "Enable swap", fn: enableSwap },
+    { name: "Grow rancher LV", fn: growRancherLv },
     { name: "Disable firewall", fn: disableFirewall },
     { name: "Set SELinux permissive", fn: setSelinuxPermissive },
     { name: "Enable iSCSI", fn: enableIscsi },

bastion/src/modules/modules/k3s/src/install.ts

@@ -215,7 +215,9 @@ echo "  Using network device: \$DEFAULT_DEV"
 
 KUBECONFIG=/etc/rancher/k3s/k3s.yaml cilium install \\
   --set kubeProxyReplacement=true \\
-  --set ipam.mode=kubernetes \\
+  --set ipam.mode=cluster-pool \\
+  --set ipam.operator.clusterPoolIPv4PodCIDRList='{10.42.0.0/16}' \\
+  --set ipam.operator.clusterPoolIPv4MaskSize=24 \\
   --set devices="\$DEFAULT_DEV" \\
   --set nodePort.directRoutingDevice="\$DEFAULT_DEV"
 

bastion/src/modules/modules/k3s/src/operations/cilium.ts

@@ -47,7 +47,9 @@ export const installCilium: Operation = async (ctx): Promise<OperationResult> =>
   const installResult = await ctx.ssh.exec(
     `KUBECONFIG=/etc/rancher/k3s/k3s.yaml cilium install \
       --set kubeProxyReplacement=true \
-      --set ipam.mode=kubernetes \
+      --set ipam.mode=cluster-pool \
+      --set ipam.operator.clusterPoolIPv4PodCIDRList='{10.42.0.0/16}' \
+      --set ipam.operator.clusterPoolIPv4MaskSize=24 \
       --set k8sServiceHost=127.0.0.1 \
       --set k8sServicePort=6444 \
       --set cni.exclusive=false \

bastion/src/modules/modules/k3s/src/operations/index.ts

@@ -1,10 +1,11 @@
 export { loadKernelModules } from "./kernel-modules.js";
 export { applyCisHardening } from "./sysctl.js";
-export { disableSwap } from "./swap.js";
+export { enableSwap } from "./swap.js";
+export { growRancherLv } from "./rancher-storage.js";
 export { enableIscsi } from "./iscsi.js";
 export { disableFirewall } from "./firewall.js";
 export { setSelinuxPermissive } from "./selinux.js";
-export { writeK3sConfig } from "./k3s-config.js";
+export { writeK3sConfig, generateServerConfig, generateAgentConfig } from "./k3s-config.js";
 export { writeAuditPolicy } from "./audit-policy.js";
 export { cleanupStaleCni } from "./cni-cleanup.js";
 export { installK3sBinary } from "./k3s-install.js";

bastion/src/modules/modules/k3s/src/operations/k3s-config.ts

@@ -7,8 +7,52 @@ function isServerRole(role: string): boolean {
   return role === "infra" || role === "labcontroller";
 }
 
-function generateServerConfig(config: K3sConfig): string {
-  const tlsSans = [config.hostname, config.ip, ...(config.tlsSans ?? [])];
+/**
+ * The address-family block: `cluster-cidr`, `service-cidr` and `node-ip`.
+ *
+ * Emitted ONLY when the corresponding config is supplied, and that is
+ * deliberate. With no `ipv6` and no CIDRs this returns "", so the generated
+ * file is byte-identical to what every existing node already has -- no diff,
+ * so `writeRemoteFile` reports unchanged and nothing restarts k3s. Dual-stack
+ * is therefore opt-in per node rather than a flag day.
+ *
+ * `node-ip` is only written once there is a second family to name. k3s
+ * auto-detects a sensible IPv4 on its own, and writing it out unconditionally
+ * would rewrite the config of five healthy nodes to tell them what they had
+ * already worked out.
+ */
+function addressFamilyLines(config: K3sConfig, opts: { cidrs: boolean }): string {
+  const lines: string[] = [];
+  if (opts.cidrs && config.clusterCidr?.length) {
+    lines.push(`cluster-cidr: "${config.clusterCidr.join(",")}"`);
+  }
+  if (opts.cidrs && config.serviceCidr?.length) {
+    lines.push(`service-cidr: "${config.serviceCidr.join(",")}"`);
+  }
+  if (config.ipv6) {
+    // Order matters to k3s: the FIRST entry is the primary family, and the
+    // supported single-to-dual-stack conversion is the one that preserves it.
+    // IPv4 stays primary so existing Services keep their ClusterIP.
+    lines.push(`node-ip: "${config.ip},${config.ipv6}"`);
+  }
+  return lines.length ? `${lines.join("\n")}\n` : "";
+}
+
+// Exported so the exact production config.yaml can be rendered outside an SSH
+// context -- notably by the labsim 3-server-etcd rehearsal, which must drive its
+// nodes through THIS generator rather than a parallel set of INSTALL_K3S_EXEC
+// flags, or it proves a mechanism production does not run.
+export function generateServerConfig(config: K3sConfig): string {
+  // The IPv6 address goes in the cert too. Without it, anything that reaches
+  // this apiserver over v6 -- a peer server joining, or kubectl against the v6
+  // address -- fails TLS verification, and the error names the certificate
+  // rather than the missing SAN, which is a long way from the cause.
+  const tlsSans = [
+    config.hostname,
+    config.ip,
+    ...(config.ipv6 ? [config.ipv6] : []),
+    ...(config.tlsSans ?? []),
+  ];
   const isJoining = !!config.k3sServerUrl;
   const clusterLines = isJoining
     ? `server: "${config.k3sServerUrl}"\ntoken: "${config.k3sToken}"`
@@ -21,7 +65,7 @@ function generateServerConfig(config: K3sConfig): string {
   // and never expire.
   return `# k3s server configuration — CIS hardened, etcd HA
 ${clusterLines}
-protect-kernel-defaults: true
+${addressFamilyLines(config, { cidrs: true })}protect-kernel-defaults: true
 secrets-encryption: true
 write-kubeconfig-mode: "0640"
 
@@ -51,8 +95,13 @@ ${tlsSans.map((s) => `  - "${s}"`).join("\n")}
 `;
 }
 
-function generateAgentConfig(): string {
-  return `protect-kernel-defaults: true
+// Takes the config now: an agent needs its own dual `node-ip` just as much as a
+// server does. Without one it joins as an IPv4-only node into a dual-stack
+// cluster, gets no IPv6 pod CIDR, and the failure surfaces later as pods on that
+// node being unreachable over v6 while the node itself reads Ready.
+// It takes no cluster/service CIDRs -- those are server-side only.
+export function generateAgentConfig(config: K3sConfig): string {
+  return `${addressFamilyLines(config, { cidrs: false })}protect-kernel-defaults: true
 node-label:
   - "node-role.kubernetes.io/worker=true"
   - "node.longhorn.io/create-default-disk=config"
@@ -64,11 +113,41 @@ kubelet-arg:
 }
 
 export const writeK3sConfig: Operation = async (ctx): Promise<OperationResult> => {
+  // Refuse to name an address the node does not have.
+  //
+  // Most of this estate is SSH-onboard, not PXE-provisioned: Asahi cannot PXE
+  // at all, and the DGX Sparks run NVIDIA's own OS and must never be
+  // reinstalled. For those nodes the install templates govern nothing and this
+  // module is the ONLY labctl touchpoint, so nothing upstream can guarantee the
+  // vendor OS actually took a DHCPv6 lease.
+  //
+  // Writing node-ip for a missing address does not fail here -- it fails later,
+  // when k3s will not start, with an error about binding rather than about
+  // addressing. Checking costs one ssh round trip and turns a confusing
+  // start-up failure into a sentence naming the address and the node.
+  if (ctx.config.ipv6) {
+    const probe = await ctx.ssh.exec(
+      `ip -6 -o addr show 2>/dev/null | grep -qF " ${ctx.config.ipv6}/" && echo present || true`,
+      sshOpts(ctx),
+    );
+    if (!probe.stdout.includes("present")) {
+      return {
+        success: false,
+        changed: false,
+        message: `Node does not have IPv6 ${ctx.config.ipv6}`,
+        error:
+          `k3s config would set node-ip to ${ctx.config.ipv6}, but that address is not on any ` +
+          `interface. k3s resolves node-ip at start-up and would fail to bind. Check the node ` +
+          `took its DHCPv6 lease (a kea reservation keyed on its MAC) before retrying.`,
+      };
+    }
+  }
+
   await ctx.ssh.exec("mkdir -p /etc/rancher/k3s", sshOpts(ctx));
 
   const content = isServerRole(ctx.config.role)
     ? generateServerConfig(ctx.config)
-    : generateAgentConfig();
+    : generateAgentConfig(ctx.config);
 
   const changed = await writeRemoteFile(ctx, "/etc/rancher/k3s/config.yaml", content);
 

bastion/src/modules/modules/k3s/src/operations/rancher-storage.ts

@@ -0,0 +1,48 @@
+// Grow the labvg/rancher LV (k3s image store / imageFs) to 120G.
+// 2026-08 incident: the original 20G LV sat at 85% used from steady-state
+// images alone, so one ~5G image pull tripped imagefs eviction and evicted
+// unrelated pods. Fresh installs are sized at 120G by the kickstart; this op
+// covers nodes installed before that change and vanilla nodes converted to
+// k8s later. Never removes or shrinks anything — if the VG lacks free space
+// (e.g. a longhorn --grow LV consumed it), it reports and moves on.
+
+import type { Operation, OperationResult } from "../types.js";
+import { sshOpts } from "../utils.js";
+
+const RANCHER_LV = "labvg/rancher";
+const TARGET_MIB = 122880; // 120G
+
+export const growRancherLv: Operation = async (ctx): Promise<OperationResult> => {
+  const lv = await ctx.ssh.exec(
+    `lvs --noheadings --units m --nosuffix -o lv_size ${RANCHER_LV} 2>/dev/null || true`,
+    sshOpts(ctx),
+  );
+  const sizeMib = Number.parseFloat(lv.stdout.trim());
+  if (Number.isNaN(sizeMib)) {
+    return { success: true, changed: false, message: "No labvg/rancher LV — imageFs shares /var, skipping" };
+  }
+  if (sizeMib >= TARGET_MIB) {
+    return { success: true, changed: false, message: `rancher LV already ${Math.round(sizeMib / 1024)}G` };
+  }
+
+  const vg = await ctx.ssh.exec(`vgs --noheadings --units m --nosuffix -o vg_free labvg`, sshOpts(ctx));
+  const freeMib = Number.parseFloat(vg.stdout.trim());
+  const neededMib = TARGET_MIB - sizeMib;
+  if (Number.isNaN(freeMib) || freeMib < neededMib) {
+    return {
+      success: true,
+      changed: false,
+      message: `VG labvg has ${Math.floor((Number.isNaN(freeMib) ? 0 : freeMib) / 1024)}G free — ` +
+        `need ${Math.ceil(neededMib / 1024)}G to grow rancher LV to 120G (manual LV rebuild required)`,
+    };
+  }
+
+  await ctx.ssh.exec(`lvextend -L ${TARGET_MIB}m /dev/${RANCHER_LV}`, sshOpts(ctx));
+  await ctx.ssh.exec(`xfs_growfs /var/lib/rancher`, sshOpts(ctx));
+
+  return {
+    success: true,
+    changed: true,
+    message: `rancher LV grown ${Math.round(sizeMib / 1024)}G → 120G`,
+  };
+};

bastion/src/modules/modules/k3s/src/operations/swap.ts

@@ -1,22 +1,40 @@
-// Disable swap (CIS requirement for k3s).
+// Enable swap so memory pressure spills to disk instead of OOM-killing.
+// kubelet runs with failSwapOn=false (k3s default); zram stays the fast tier,
+// the labvg-swap LV is the overflow tier. Replaces the old CIS-style
+// disableSwap op — a kernel OOM kill of a node daemon is worse than slow swap.
 
 import type { Operation, OperationResult } from "../types.js";
 import { sshOpts } from "../utils.js";
 
-export const disableSwap: Operation = async (ctx): Promise<OperationResult> => {
-  const check = await ctx.ssh.exec("swapon --show --noheadings", sshOpts(ctx));
-  const active = check.stdout.trim().length > 0;
+const SWAP_DEV = "/dev/mapper/labvg-swap";
 
-  if (active) {
-    await ctx.ssh.exec("swapoff -a", sshOpts(ctx));
+export const enableSwap: Operation = async (ctx): Promise<OperationResult> => {
+  const lv = await ctx.ssh.exec(`test -b ${SWAP_DEV} && echo yes || echo no`, sshOpts(ctx));
+  if (lv.stdout.trim() !== "yes") {
+    return { success: true, changed: false, message: "No labvg-swap LV — skipping swap enable" };
   }
 
-  // Remove swap entries from fstab permanently
-  await ctx.ssh.exec("sed -i '/\\sswap\\s/d' /etc/fstab", sshOpts(ctx));
+  const active = await ctx.ssh.exec(
+    `grep -q "^$(readlink -f ${SWAP_DEV}) " /proc/swaps && echo on || echo off`,
+    sshOpts(ctx),
+  );
+  const wasOff = active.stdout.trim() !== "on";
+
+  if (wasOff) {
+    // Format if the LV was never (or wrongly) initialised, then activate
+    await ctx.ssh.exec(`blkid ${SWAP_DEV} | grep -q 'TYPE="swap"' || mkswap ${SWAP_DEV}`, sshOpts(ctx));
+    await ctx.ssh.exec(`swapon ${SWAP_DEV}`, sshOpts(ctx));
+  }
+
+  // Persist across reboots (idempotent)
+  await ctx.ssh.exec(
+    `grep -q "labvg-swap" /etc/fstab || echo "${SWAP_DEV} none swap defaults 0 0" >> /etc/fstab`,
+    sshOpts(ctx),
+  );
 
   return {
     success: true,
-    changed: active,
-    message: active ? "Swap disabled" : "Swap already disabled",
+    changed: wasOff,
+    message: wasOff ? "LV swap enabled" : "LV swap already active",
   };
 };

bastion/src/modules/modules/k3s/src/types.ts

@@ -16,6 +16,33 @@ export interface K3sConfig {
 
   // Additional TLS SANs for API server certificate
   tlsSans?: string[] | undefined;
+
+  /**
+   * IPv6 address of this node on the cluster VLAN. Its PRESENCE is what makes a
+   * node dual-stack: supply it and `node-ip` is emitted as `<v4>,<v6>`; omit it
+   * and the generated config is byte-identical to the IPv4-only one.
+   *
+   * It must be a real address on the node before k3s starts. k3s resolves
+   * node-ip at boot, so a SLAAC or DHCPv6 lease that has not landed yet leaves
+   * the node with no usable v6 identity. The estate takes it from a kea DHCPv6
+   * reservation keyed on MAC, the same way it takes its IPv4 address.
+   */
+  ipv6?: string | undefined;
+
+  /**
+   * Pod and Service ranges, one entry per address family. Passed rather than
+   * hardcoded so the ranges stay configuration -- and so labsim can rehearse
+   * with its own addresses through this same generator, instead of a parallel
+   * set of INSTALL_K3S_EXEC flags that would prove a different mechanism.
+   *
+   * k3s validates the two together and crash-loops on a mismatch (a dual
+   * cluster-cidr with a single-family service-cidr, say), which is a safe
+   * failure but an avoidable one: set both or neither.
+   *
+   * Servers only -- agents take neither.
+   */
+  clusterCidr?: string[] | undefined;
+  serviceCidr?: string[] | undefined;
 }
 
 /** SSH execution interface injected into operations. */

bastion/src/modules/modules/k3s/tests/operations.test.ts

@@ -72,31 +72,97 @@ describe("applyCisHardening", () => {
 
 // --- Swap ---
 
-import { disableSwap } from "../src/operations/swap.js";
+import { enableSwap } from "../src/operations/swap.js";
 
-describe("disableSwap", () => {
-  it("disables active swap", async () => {
+describe("enableSwap", () => {
+  it("activates LV swap when present but off", async () => {
     const ctx = mockCtx();
     ctx.ssh.exec
-      .mockResolvedValueOnce(stdout("/dev/sda2 partition 2G"))  // swap active
-      .mockResolvedValueOnce(OK)   // swapoff
-      .mockResolvedValueOnce(OK);  // sed fstab
+      .mockResolvedValueOnce(stdout("yes"))  // LV exists
+      .mockResolvedValueOnce(stdout("off"))  // not in /proc/swaps
+      .mockResolvedValueOnce(OK)             // blkid || mkswap
+      .mockResolvedValueOnce(OK)             // swapon
+      .mockResolvedValueOnce(OK);            // fstab entry
 
-    const result = await disableSwap(ctx);
+    const result = await enableSwap(ctx);
     expect(result.success).toBe(true);
     expect(result.changed).toBe(true);
-    expectCommand(ctx.ssh, "swapoff -a");
+    expectCommand(ctx.ssh, "swapon /dev/mapper/labvg-swap");
   });
 
-  it("is idempotent when swap already off", async () => {
+  it("is idempotent when LV swap already active", async () => {
     const ctx = mockCtx();
     ctx.ssh.exec
-      .mockResolvedValueOnce(stdout(""))  // no swap
-      .mockResolvedValueOnce(OK);          // sed fstab (always runs)
+      .mockResolvedValueOnce(stdout("yes"))  // LV exists
+      .mockResolvedValueOnce(stdout("on"))   // already in /proc/swaps
+      .mockResolvedValueOnce(OK);            // fstab entry (always ensured)
 
-    const result = await disableSwap(ctx);
+    const result = await enableSwap(ctx);
     expect(result.changed).toBe(false);
-    expectNoCommand(ctx.ssh, "swapoff");
+    expectNoCommand(ctx.ssh, "swapon /dev/mapper/labvg-swap");
+  });
+
+  it("skips when no labvg-swap LV exists", async () => {
+    const ctx = mockCtx();
+    ctx.ssh.exec.mockResolvedValueOnce(stdout("no")); // LV missing
+
+    const result = await enableSwap(ctx);
+    expect(result.success).toBe(true);
+    expect(result.changed).toBe(false);
+    expectNoCommand(ctx.ssh, "swapon");
+  });
+});
+
+// --- Rancher LV (imageFs sizing) ---
+
+import { growRancherLv } from "../src/operations/rancher-storage.js";
+
+describe("growRancherLv", () => {
+  it("grows a 20G LV to 120G when the VG has space", async () => {
+    const ctx = mockCtx();
+    ctx.ssh.exec
+      .mockResolvedValueOnce(stdout("  20480.00"))   // lv_size
+      .mockResolvedValueOnce(stdout("  747807.00"))  // vg_free
+      .mockResolvedValueOnce(OK)                     // lvextend
+      .mockResolvedValueOnce(OK);                    // xfs_growfs
+
+    const result = await growRancherLv(ctx);
+    expect(result.success).toBe(true);
+    expect(result.changed).toBe(true);
+    expectCommand(ctx.ssh, "lvextend -L 122880m /dev/labvg/rancher");
+    expectCommand(ctx.ssh, "xfs_growfs /var/lib/rancher");
+  });
+
+  it("is idempotent when the LV is already 120G", async () => {
+    const ctx = mockCtx();
+    ctx.ssh.exec.mockResolvedValueOnce(stdout("  122880.00")); // lv_size
+
+    const result = await growRancherLv(ctx);
+    expect(result.changed).toBe(false);
+    expectNoCommand(ctx.ssh, "lvextend");
+  });
+
+  it("reports without failing when the VG has no free space", async () => {
+    const ctx = mockCtx();
+    ctx.ssh.exec
+      .mockResolvedValueOnce(stdout("  20480.00"))  // lv_size
+      .mockResolvedValueOnce(stdout("  0.00"));     // vg_free
+
+    const result = await growRancherLv(ctx);
+    expect(result.success).toBe(true);
+    expect(result.changed).toBe(false);
+    expect(result.message).toContain("free");
+    expectNoCommand(ctx.ssh, "lvextend");
+  });
+
+  it("skips when there is no rancher LV", async () => {
+    const ctx = mockCtx();
+    ctx.ssh.exec.mockResolvedValueOnce(stdout("")); // lvs empty
+
+    const result = await growRancherLv(ctx);
+    expect(result.success).toBe(true);
+    expect(result.changed).toBe(false);
+    expectNoCommand(ctx.ssh, "lvextend");
   });
 });
 
@@ -202,6 +268,108 @@ describe("writeK3sConfig", () => {
     expect(writeCall).toContain("protect-kernel-defaults: true");
     expect(writeCall).not.toContain("secrets-encryption");
   });
+
+  // --- dual-stack ---
+  //
+  // The property that matters most is the NEGATIVE one: with no ipv6 and no
+  // CIDRs the output must be byte-identical to what the five existing nodes
+  // already have on disk. If it is not, rolling this out rewrites every node's
+  // config.yaml and restarts a healthy cluster to tell it what it already knew.
+
+  // With an ipv6 configured there is an extra ssh round trip up front -- the
+  // probe that checks the address is really on the node -- so the write lands
+  // one call later.
+  const writtenBy = async (config: Parameters<typeof mockCtx>[0]) => {
+    const ctx = mockCtx(config);
+    const hasV6 = !!(config as { ipv6?: string }).ipv6;
+    if (hasV6) ctx.ssh.exec.mockResolvedValueOnce(stdout("present"));
+    ctx.ssh.exec
+      .mockResolvedValueOnce(OK)
+      .mockResolvedValueOnce(stdout("__LABCTL_NOT_FOUND__"))
+      .mockResolvedValueOnce(OK);
+    await writeK3sConfig(ctx);
+    return ctx.ssh.exec.mock.calls[hasV6 ? 3 : 2]![0] as string;
+  };
+
+  it("emits no address-family lines at all when single-stack", async () => {
+    const server = await writtenBy({ hostname: "n1.lab", ip: "10.0.1.1", role: "infra" });
+    expect(server).not.toContain("node-ip");
+    expect(server).not.toContain("cluster-cidr");
+    expect(server).not.toContain("service-cidr");
+
+    const agent = await writtenBy({ role: "worker" });
+    expect(agent).not.toContain("node-ip");
+    // Nothing inserted ahead of, or between, the lines the agent config has
+    // always opened with.
+    expect(agent).toContain("protect-kernel-defaults: true\nnode-label:");
+  });
+
+  it("emits dual node-ip and CIDRs on a server, IPv4 first", async () => {
+    const out = await writtenBy({
+      hostname: "n1.lab",
+      ip: "192.168.8.23",
+      role: "infra",
+      ipv6: "2001:470:187e:2::23",
+      clusterCidr: ["10.42.0.0/16", "2001:470:187e:1000::/56"],
+      serviceCidr: ["10.43.0.0/16", "2001:470:187e:1fff::/112"],
+    });
+    expect(out).toContain('node-ip: "192.168.8.23,2001:470:187e:2::23"');
+    expect(out).toContain('cluster-cidr: "10.42.0.0/16,2001:470:187e:1000::/56"');
+    expect(out).toContain('service-cidr: "10.43.0.0/16,2001:470:187e:1fff::/112"');
+    // IPv4 must stay the primary family -- that is the supported conversion
+    // path and what lets existing Services keep their ClusterIP.
+    expect(out.indexOf("10.42.0.0/16")).toBeLessThan(out.indexOf("2001:470:187e:1000::/56"));
+    // still a valid server config
+    expect(out).toContain("cluster-init: true");
+    expect(out).toContain("secrets-encryption: true");
+    // the v6 address must be a TLS SAN, or a peer joining over v6 fails
+    // verification with an error that names the cert, not the missing SAN
+    expect(out).toContain('  - "2001:470:187e:2::23"');
+  });
+
+  it("gives an AGENT its own dual node-ip but no CIDRs", async () => {
+    const out = await writtenBy({
+      ip: "192.168.8.12",
+      role: "worker",
+      ipv6: "2001:470:187e:2::12",
+      // deliberately supplied: an agent must ignore them
+      clusterCidr: ["10.42.0.0/16", "2001:470:187e:1000::/56"],
+      serviceCidr: ["10.43.0.0/16", "2001:470:187e:1fff::/112"],
+    });
+    expect(out).toContain('node-ip: "192.168.8.12,2001:470:187e:2::12"');
+    expect(out).not.toContain("cluster-cidr");
+    expect(out).not.toContain("service-cidr");
+  });
+
+  it("refuses to write a config naming an IPv6 the node does not have", async () => {
+    // The SSH-onboard case: Asahi and the DGX Sparks run an OS labctl never
+    // installed, so nothing upstream guarantees a DHCPv6 lease was taken.
+    // Writing the config anyway defers the failure to k3s start-up, where it
+    // reads as a bind error rather than a missing address.
+    const ctx = mockCtx({ ip: "192.168.8.12", role: "worker", ipv6: "2001:470:187e:2::12" });
+    ctx.ssh.exec.mockResolvedValueOnce(stdout("")); // probe: address absent
+
+    const result = await writeK3sConfig(ctx);
+    expect(result.success).toBe(false);
+    expect(result.changed).toBe(false);
+    expect(result.error).toContain("2001:470:187e:2::12");
+    // and it must not have written anything
+    expect(ctx.ssh.exec).toHaveBeenCalledTimes(1);
+  });
+
+  it("does not write node-ip for a v4-only node even when CIDRs are given", async () => {
+    // Guards the flag-day risk: supplying ranges alone must not start rewriting
+    // node identity on nodes that have no IPv6 yet.
+    const out = await writtenBy({
+      hostname: "n1.lab",
+      ip: "10.0.1.1",
+      role: "infra",
+      clusterCidr: ["10.42.0.0/16"],
+      serviceCidr: ["10.43.0.0/16"],
+    });
+    expect(out).toContain('cluster-cidr: "10.42.0.0/16"');
+    expect(out).not.toContain("node-ip");
+  });
 });
 
 // --- CNI Cleanup ---

bastion/src/modules/modules/k3s/tests/smoke.test.ts

@@ -16,7 +16,7 @@ describe("smoke: full server install pipeline", () => {
     const pipeline: NamedOperation[] = [
       { name: "Kernel modules", fn: ops.loadKernelModules },
       { name: "Sysctl hardening", fn: ops.applyCisHardening },
-      { name: "Disable swap", fn: ops.disableSwap },
+      { name: "Enable swap", fn: ops.enableSwap },
       { name: "Disable firewall", fn: ops.disableFirewall },
       { name: "SELinux permissive", fn: ops.setSelinuxPermissive },
       { name: "Write k3s config", fn: ops.writeK3sConfig },
@@ -73,7 +73,7 @@ describe("smoke: pipeline stops on failure", () => {
     };
 
     const results = await runSequential(ctx, [
-      { name: "OK op", fn: ops.disableSwap },
+      { name: "OK op", fn: ops.enableSwap },
       { name: "Failing op", fn: failingOp },
       { name: "Never called", fn: neverCalled },
     ]);
@@ -98,11 +98,12 @@ describe("smoke: agent install rejects missing config", () => {
 });
 
 describe("smoke: all operations are exported", () => {
-  it("exports all 15 operations", () => {
+  it("exports all 16 operations", () => {
     const exported = [
       ops.loadKernelModules,
       ops.applyCisHardening,
-      ops.disableSwap,
+      ops.enableSwap,
+      ops.growRancherLv,
       ops.disableFirewall,
       ops.setSelinuxPermissive,
       ops.writeK3sConfig,
@@ -117,7 +118,7 @@ describe("smoke: all operations are exported", () => {
       ops.checkCertExpiry,
     ];
 
-    expect(exported).toHaveLength(15);
+    expect(exported).toHaveLength(16);
     for (const op of exported) {
       expect(typeof op).toBe("function");
     }

bastion/src/shared/src/hardware/index.ts

@@ -1,154 +0,0 @@
-// Architecture normalisation and machine classification.
-//
-// Both are derived from what the system already observes about a machine -- never from
-// an operator-supplied flag.
-
-import type { Arch, HardwareInfo, OnboardMethod, OsId } from "../types/index.js";
-
-export const SUPPORTED_ARCHES: readonly Arch[] = ["x86_64", "aarch64"] as const;
-
-/**
- * Normalise an architecture string to one we serve boot artifacts for.
- *
- * Sources and their spellings:
- *   uname -m         -> "x86_64" / "aarch64"
- *   iPXE ${buildarch}-> "x86_64" / "arm64"
- *   dpkg/Debian      -> "amd64"  / "arm64"
- *
- * Returns undefined for anything we don't serve, so callers fall back rather than
- * inventing a kernel path that would 404.
- */
-export function normalizeArch(value: string | undefined | null): Arch | undefined {
-  switch ((value ?? "").trim().toLowerCase()) {
-    case "x86_64":
-    case "x86-64":
-    case "amd64":
-      return "x86_64";
-    case "aarch64":
-    case "arm64":
-      return "aarch64";
-    default:
-      return undefined;
-  }
-}
-
-/** Fedora pxeboot artifact base URL for an architecture. */
-export function fedoraMirrorFor(fedoraVersion: string, arch: Arch): string {
-  return `https://download.fedoraproject.org/pub/fedora/linux/releases/${fedoraVersion}/Everything/${arch}/os`;
-}
-
-/**
- * Which architectures each OS in the pipeline can actually be installed on.
- *
- * Fedora publishes pxeboot vmlinuz/initrd for both. Ubuntu does not: as of 26.04,
- * releases.ubuntu.com publishes amd64 artifacts only, so there is nothing to netboot an
- * arm64 machine with. Claiming support would fail at download time with a 404 instead
- * of a useful message.
- */
-const OS_ARCH_SUPPORT: Record<OsId, readonly Arch[]> = {
-  "fedora-43": ["x86_64", "aarch64"],
-  "ubuntu-26.04": ["x86_64"],
-};
-
-export function osSupportsArch(os: OsId, arch: Arch): boolean {
-  return (OS_ARCH_SUPPORT[os] ?? []).includes(arch);
-}
-
-export function archesForOs(os: OsId): readonly Arch[] {
-  return OS_ARCH_SUPPORT[os] ?? [];
-}
-
-/**
- * Machines that run a vendor OS we have no image for.
- *
- * These are SSH-onboard: we manage userspace, but reinstalling destroys a driver and
- * firmware stack our pipeline cannot rebuild. Matched on DMI identity, which is what
- * discovery and `provision recheck` both collect.
- *
- * This is deliberately a property of the machine ("it runs DGX OS"), not a blocklist
- * ("never install this MAC"). When a DGX OS image joins the pipeline, teaching the
- * installer about vendor_os "dgx-os" is what unblocks these machines -- no entry here
- * needs deleting.
- */
-interface VendorOsRule {
-  vendorOs: string;
-  description: string;
-  matches: (hw: DmiIdentity) => boolean;
-}
-
-interface DmiIdentity {
-  manufacturer: string;
-  product: string;
-  board: string;
-}
-
-const VENDOR_OS_RULES: readonly VendorOsRule[] = [
-  {
-    vendorOs: "dgx-os",
-    description: "NVIDIA DGX OS (proprietary driver + firmware stack, no image in our pipeline)",
-    matches: ({ manufacturer, product, board }) =>
-      (manufacturer.includes("nvidia") || product.includes("nvidia")) &&
-      (product.includes("dgx") || product.includes("spark") ||
-       board.includes("gb10") || product.includes("gb10")),
-  },
-];
-
-/**
- * Machines known to run a vendor OS, by MAC.
- *
- * The DMI rules above only fire once discovery or `provision recheck` has populated a
- * hardware record. Machines onboarded over SSH may sit in state for a long time with no
- * DMI at all -- which is exactly the state both DGX Sparks are in today -- so a
- * DMI-only classifier would fail open on the machines this guard exists to protect.
- *
- * This is a statement of fact about known hardware ("this box runs DGX OS"), not an
- * install policy. Whether that means "refuse" is decided by whether the pipeline has an
- * image for that vendor OS.
- */
-const KNOWN_VENDOR_OS_MACS: Record<string, string> = {
-  "4c:bb:47:7f:29:35": "dgx-os",  // spark-2935
-  "48:21:0b:96:3a:1c": "dgx-os",  // spark-3a1c
-};
-
-/**
- * Classify how a machine should be onboarded, from its hardware record.
- *
- * An explicit `onboard` already on the record wins: it may have been set by an operator
- * or by a rule that has since changed, and silently overriding it would be worse than
- * leaving it.
- */
-export function classifyOnboard(
-  hw: Partial<Pick<HardwareInfo, "mac" | "manufacturer" | "product" | "board">>
-    & { onboard?: OnboardMethod; vendor_os?: string },
-): { onboard: OnboardMethod; vendor_os?: string } {
-  if (hw.onboard !== undefined) {
-    return hw.vendor_os !== undefined
-      ? { onboard: hw.onboard, vendor_os: hw.vendor_os }
-      : { onboard: hw.onboard };
-  }
-
-  const knownVendorOs = KNOWN_VENDOR_OS_MACS[(hw.mac ?? "").toLowerCase().replace(/-/g, ":")];
-  if (knownVendorOs !== undefined) {
-    return { onboard: "ssh", vendor_os: knownVendorOs };
-  }
-
-  const identity: DmiIdentity = {
-    manufacturer: (hw.manufacturer ?? "").toLowerCase(),
-    product: (hw.product ?? "").toLowerCase(),
-    board: (hw.board ?? "").toLowerCase(),
-  };
-
-  for (const rule of VENDOR_OS_RULES) {
-    if (rule.matches(identity)) {
-      return { onboard: "ssh", vendor_os: rule.vendorOs };
-    }
-  }
-
-  return { onboard: "pxe" };
-}
-
-/** Human-readable reason a vendor-OS machine must not be reinstalled. */
-export function vendorOsDescription(vendorOs: string | undefined): string {
-  const rule = VENDOR_OS_RULES.find((r) => r.vendorOs === vendorOs);
-  return rule?.description ?? "a vendor OS with no image in our pipeline";
-}

bastion/src/shared/src/index.ts

@@ -1,8 +1,6 @@
 export type {
   OsId,
   Arch,
-  OnboardMethod,
-  RootCandidate,
   Role,
   HardwareInfo,
   InstallConfig,
@@ -10,18 +8,12 @@ export type {
   DebugConfig,
   BastionState,
   BastionConfig,
+  VyosVlanSpec,
+  VyosInstallSpec,
+  VyosBundle,
+  VyosBundleSetOp,
 } from "./types/index.js";
 
-export {
-  SUPPORTED_ARCHES,
-  normalizeArch,
-  fedoraMirrorFor,
-  osSupportsArch,
-  archesForOs,
-  classifyOnboard,
-  vendorOsDescription,
-} from "./hardware/index.js";
-
 export { SUPPORTED_OS, SUPPORTED_ROLES, ROLE_REGISTRY, isValidOsId } from "./types/index.js";
 export type { RoleInfo } from "./types/index.js";
 

bastion/src/shared/src/protocol/index.ts

@@ -1,6 +1,7 @@
 // Protocol types for agent-labd WebSocket communication.
 
 import { randomUUID } from "node:crypto";
+import type { VyosInstallSpec } from "../types/state.js";
 
 // --- Agent -> labd messages ---
 
@@ -108,12 +109,12 @@ export type BastionMessage =
 export type LabdBastionMessage =
   | { type: "bastion-enrolled"; bastionId: string }
   | { type: "bastion-heartbeat-ack"; serverTime: string }
-  | { type: "command-install"; requestId: string; mac: string; hostname: string; disk?: string; role: string; os: string }
+  | { type: "command-install"; requestId: string; mac: string; hostname: string; disk?: string; role: string; os: string; vyos?: VyosInstallSpec }
   | { type: "command-forget"; requestId: string; mac: string }
   | { type: "command-role-update"; requestId: string; mac: string; role: string }
   | { type: "command-debug"; requestId: string; mac: string; pxeBoot?: boolean }
   | { type: "command-register"; requestId: string; mac: string; hostname: string; role: string; ip: string }
-  | { type: "command-discover"; requestId: string; mac: string; product?: string; board?: string; serial?: string; manufacturer?: string; cpu_model?: string; cpu_cores?: number; memory_gb?: number; arch?: string; disks?: Array<{ name: string; size_gb: number; model: string }>; nics?: Array<{ name: string; mac: string; state: string }>; root_device?: string; root_args?: string }
+  | { type: "command-discover"; requestId: string; mac: string; product?: string; board?: string; serial?: string; manufacturer?: string; cpu_model?: string; cpu_cores?: number; memory_gb?: number; arch?: string; disks?: Array<{ name: string; size_gb: number; model: string }>; nics?: Array<{ name: string; mac: string; state: string }> }
   | { type: "server-shutdown"; reconnectAfter: number };
 
 export type BastionMessageType = BastionMessage["type"];

bastion/src/shared/src/types/config.ts

@@ -14,6 +14,10 @@ export interface BastionConfig {
   // Ubuntu support
   ubuntuVersion: string;
   ubuntuMirror: string;
+  // VyOS support — netboot artifacts are extracted from the ISO at startup.
+  // LTS ISOs are subscription-only, so this defaults to a rolling release.
+  vyosIsoUrl: string;
+  vyosDefaultPassword: string;
   // Syslog listener for install logs (Anaconda logging --host)
   syslogPort: number;
   // Flags

bastion/src/shared/src/types/index.ts

@@ -1,14 +1,16 @@
 export type {
   OsId,
   Arch,
-  OnboardMethod,
-  RootCandidate,
   Role,
   HardwareInfo,
   InstallConfig,
   InstalledInfo,
   DebugConfig,
   BastionState,
+  VyosVlanSpec,
+  VyosInstallSpec,
+  VyosBundle,
+  VyosBundleSetOp,
 } from "./state.js";
 
 export { SUPPORTED_OS, SUPPORTED_ROLES, ROLE_REGISTRY, isValidOsId } from "./state.js";

bastion/src/shared/src/types/state.ts

@@ -2,25 +2,15 @@
 
 export type ProvisionStackType = "dhcpproxy" | "iso" | "cloud-init";
 
-export type OsId = "fedora-43" | "ubuntu-26.04";
+export type OsId = "fedora-43" | "ubuntu-26.04" | "vyos-rolling";
 export type Arch = "x86_64" | "aarch64";
 
-export const SUPPORTED_OS: readonly OsId[] = ["fedora-43", "ubuntu-26.04"] as const;
+export const SUPPORTED_OS: readonly OsId[] = ["fedora-43", "ubuntu-26.04", "vyos-rolling"] as const;
 
 export function isValidOsId(value: string): value is OsId {
   return (SUPPORTED_OS as readonly string[]).includes(value);
 }
 
-/**
- * How a machine joins the lab.
- *
- * "pxe"  -- bare metal we install over the network (the default).
- * "ssh"  -- the machine already runs a vendor OS we cannot reproduce, so we onboard
- *           over SSH and manage userspace only. Installing would destroy that OS.
- *           See classifyOnboard() and os-install-research.md.
- */
-export type OnboardMethod = "pxe" | "ssh";
-
 export interface HardwareInfo {
   mac: string;
   product: string;
@@ -36,23 +26,6 @@ export interface HardwareInfo {
   first_seen: string;
   last_seen: string;
   bastionId?: string;  // set when aggregated through labd
-  // Onboarding classification -- absent means "pxe" (see classifyOnboard)
-  onboard?: OnboardMethod;
-  vendor_os?: string;   // e.g. "dgx-os": the OS this machine must keep running
-  // Root filesystem, for booting the installed system over PXE (--pxe-boot).
-  // Observed from the machine, never assumed.
-  root_device?: string;                 // e.g. "/dev/mapper/labvg-root"
-  root_args?: string;                   // e.g. "rd.lvm.lv=labvg/root rd.lvm.lv=labvg/swap"
-  root_candidates?: RootCandidate[];    // reported from a rescue shell when unknown
-}
-
-/** A possible root filesystem found while probing an unreachable machine. */
-export interface RootCandidate {
-  device: string;       // e.g. "/dev/mapper/labvg-root"
-  args?: string;        // extra dracut args needed to assemble it
-  fstype?: string;
-  size_gb?: number;
-  os_release?: string;  // PRETTY_NAME from /etc/os-release, if mountable
 }
 
 export type Role = "vanilla" | "worker" | "infra" | "labcontroller";
@@ -102,13 +75,141 @@ export interface ProgressLogEntry {
   timestamp: string;
 }
 
+/** A tagged VLAN sub-interface on the bond (or on the mgmt NIC when unbonded). */
+export interface VyosVlanSpec {
+  id: number;
+  address: string;         // CIDR, e.g. "10.0.10.1/24"
+  description?: string;
+  /**
+   * VRRP virtual address (CIDR) floated on this VLAN. Emitted as a
+   * high-availability vrrp group with vrid = VLAN id, so the same spec on both
+   * HA peers (with different priorities) produces a matching group pair.
+   */
+  vrrp?: string;
+}
+
+/** One config node in a rendered bundle. Mirrors VyosSetOp on the bastion side. */
+export interface VyosBundleSetOp {
+  path: string[];
+  value?: string;
+  /** false appends to a multi-value node (e.g. bond members) instead of replacing. */
+  replace?: boolean;
+}
+
+/**
+ * A router's complete desired config, rendered from the Pulumi model.
+ *
+ * Produced by `kubernetes-deployment/scripts/vyos-render-bundle.ts` from the
+ * same subtree model `pulumi up` applies. The point is that labctl never
+ * authors VyOS config: bring-up replays what Pulumi already declares, so a
+ * freshly installed router and a `pulumi up` cannot disagree.
+ *
+ * Secret values arrive as `@secret:<key>` sentinels and are DROPPED at install
+ * time -- the bundle is committed to git and must stay safe to read. The router
+ * comes up on the LAN without its PPPoE credential; the first `pulumi up`
+ * supplies it. That handoff is deliberate.
+ */
+export interface VyosBundle {
+  sets: VyosBundleSetOp[];
+  /** Paths that are VyOS tag nodes — the installer's ConfigTree needs them marked. */
+  tags: string[][];
+}
+
+/**
+ * VyOS-specific install parameters. Rendered into the config.boot that the
+ * installer adopts, so the router comes up already configured.
+ *
+ * NOTE: bondMembers must NOT include the interface PXE booted from. Firmware
+ * PXE cannot run over LACP, so the install-time NIC has to stay unbonded.
+ */
+export interface VyosInstallSpec {
+  /**
+   * A complete rendered config for this router. When present it REPLACES the
+   * derived interface/VLAN/VRRP config below -- the bundle already describes
+   * all of it, and deriving a second opinion is exactly the drift this exists
+   * to prevent. The remaining install parameters (password, disk, console) are
+   * still honoured because they are installer inputs, not router config.
+   */
+  bundle?: VyosBundle;
+  /**
+   * Key for the VyOS HTTP API, enabled at install so the router is manageable
+   * from the moment it boots.
+   *
+   * Without this the box comes up reachable only over SSH, and enabling the API
+   * later is a hand-run config change on a live firewall -- which is exactly the
+   * gap that left vyos001/vyos002 unmanageable by Pulumi after their cutover.
+   * The API is deliberately NOT part of the Pulumi model: a provider that
+   * manages its own transport can revoke its own access.
+   */
+  apiKey?: string;
+  /**
+   * Address the API listens on. Defaults to the management address when static.
+   * Never left unbound: an unrestricted listener puts a config-write endpoint on
+   * every segment the router touches, including the WAN.
+   */
+  apiListenAddress?: string;
+  /** Interfaces aggregated into bond0 with LACP (802.3ad). Omit for no bond. */
+  bondMembers?: string[];
+  /** CIDR address on bond0 itself — the switch trunk's native/untagged VLAN. */
+  bondAddress?: string;
+  /** VRRP virtual address (CIDR) floated on the untagged bond (vrid 1). */
+  bondVrrp?: string;
+  /**
+   * VRRP priority for every group on this box. Higher wins mastership.
+   * The HA pair differs ONLY here (e.g. 200 on the primary, 100 on the
+   * standby) — addresses differ per box, VIPs and vrids match.
+   */
+  vrrpPriority?: number;
+  /** Tagged VLAN sub-interfaces, created on bond0 when bonded, else on mgmtInterface. */
+  vlans?: VyosVlanSpec[];
+  /** Untagged interface the machine PXE booted from. Defaults to "eth0". */
+  mgmtInterface?: string;
+  /** CIDR address for mgmtInterface, or "dhcp". Defaults to "dhcp". */
+  mgmtAddress?: string;
+  /**
+   * Tagged management VLAN on mgmtInterface, separate from the routed VLANs
+   * carried by the bond.
+   *
+   * Needed when the PXE port is a trunk: it boots untagged on the VLAN the
+   * bastion's proxy DHCP serves, and carries the management VLAN tagged so the
+   * router stays reachable there without giving up reinstallability.
+   */
+  mgmtVlan?: VyosVlanSpec;
+  /** Password for the "vyos" user. Falls back to the bastion default. */
+  password?: string;
+  /**
+   * On reinstall the VyOS installer carries the previous on-disk config (and
+   * SSH host keys) forward -- the "reinstall without losing data" default.
+   * Set true to make the bastion-generated config win instead: after install
+   * the driver overwrites the installed image's config.boot.
+   */
+  freshConfig?: boolean;
+  /**
+   * VyOS interface name -> MAC, emitted as `hw-id` so names bind deterministically.
+   *
+   * Discovery runs under Fedora and reports predictable names (enp2s0,
+   * enp1s0f0np0), but VyOS enumerates its own eth<N> names, so a name observed
+   * during discovery cannot be used directly. Pinning by MAC removes the guess
+   * about which physical port a given eth<N> is.
+   */
+  hwIds?: Record<string, string>;
+}
+
 export interface InstallConfig {
   hostname: string;
   disk: string;
   role: Role;
   os?: OsId;          // defaults to "fedora-43" for backward compat
+  vyos?: VyosInstallSpec;  // only consulted when os is "vyos-rolling"
   arch?: Arch;         // detected from HardwareInfo or overridden
   queued_at: string;
+  /**
+   * When dispatch last served this machine an install boot script. Progress
+   * callbacks only start once the installer environment is up, so a machine
+   * dispatched long ago with no progress is wedged before that point (bad
+   * kernel/initrd, no network in the initramfs, wrong NIC picked...).
+   */
+  dispatched_at?: string;
   progress?: string;
   progress_at?: string;
   progress_detail?: string;
@@ -130,11 +231,6 @@ export interface InstalledInfo {
   cpu_cores?: number;
   memory_gb?: number;
   arch?: string;
-  onboard?: OnboardMethod;
-  vendor_os?: string;
-  root_device?: string;
-  root_args?: string;
-  root_candidates?: RootCandidate[];
 }
 
 export interface DebugConfig {

bastion/tests/integration/arm-pxe-provision.test.ts

@@ -1,537 +0,0 @@
-// Integration test: aarch64 network PXE boot.
-//
-// The boot-ISO path already covered ARM (arm-iso-provision.test.ts). This covers the
-// network path: DHCP option 93 handing an arm64 client an arm64 iPXE binary, dispatch
-// serving an aarch64 kernel, and `provision debug` reaching a rescue shell -- which is
-// what the DGX Sparks actually need and could not do.
-//
-// Two suites, because they cost very different amounts of time:
-//
-//   "ARM PXE rescue"   NBP handoff -> rescue with SSH.                    ~25-30 min
-//   "ARM PXE install"  discover -> install -> installed.                  ~75-95 min
-//
-// The rescue suite seeds the machine into state as an already-known aarch64 box rather
-// than discovering it first. That is the DGX Spark situation exactly -- SSH-onboarded,
-// never PXE-discovered, architecture known only from its record -- and it holds the test
-// to one emulated boot. Each boot spends ~15 of its ~18 minutes downloading Anaconda's
-// stage2 under TCG, so discovering first would double the runtime without touching any
-// code path the rescue boot does not already exercise.
-//
-// The install suite only runs with ARM_PXE_FULL=1. No ARM machine in the lab is ever
-// PXE-installed except the MS-R1, and an hour-plus test that runs by default is a test
-// nobody runs.
-//
-// IMPORTANT: aarch64 has no KVM on an x86_64 host, so all of this is emulated and
-// roughly 10x slower than native.
-//
-// A note for whoever debugs a failure here: if the VM panics with
-//   VFS: Unable to mount root fs on unknown-block(0,0)
-// that is very likely iPXE silently dropping the initrd because the build lacks
-// EFI_LOAD_FILE2_PROTOCOL -- on arm64 the kernel EFI stub fetches the initrd over
-// LoadFile2, and an iPXE without it accepts the `initrd` line and does nothing. It is
-// NOT a reproduction of the DGX Spark kernel bug that motivated this work, despite
-// being the identical message. assertIpxeSupportsLoadFile2() below checks the build up
-// front so that failure names itself; to check by hand:
-//   node -e 'const b=require("fs").readFileSync("/usr/share/ipxe/arm64-efi/snponly.efi");
-//            console.log(b.indexOf(Buffer.from("c1c00640b3fc3e40996d4a6c8724e06d","hex")))'
-// Fedora's ipxe-bootimgs-aarch64-20240119 has it at 0x3bbf0.
-//
-// Prerequisites:
-//   - qemu-system-aarch64        (sudo dnf install qemu-system-aarch64)
-//   - edk2-aarch64               (sudo dnf install edk2-aarch64)
-//   - ipxe-bootimgs-aarch64      (sudo dnf install ipxe-bootimgs-aarch64)
-//   - libvirtd, sudo, internet access
-//
-// Run: sudo ./scripts/test-provision.sh arm-pxe
-
-import { describe, it, expect, beforeAll, afterAll } from "vitest";
-import { readFileSync, existsSync, mkdirSync, rmSync, copyFileSync, writeFileSync } from "node:fs";
-import { execSync } from "node:child_process";
-import { join } from "node:path";
-import { homedir, tmpdir } from "node:os";
-import { log, waitForSsh } from "./helpers/libvirt.js";
-import { ensurePxeNetwork, destroyPxeNetwork, deleteNftablesRejectRules, PXE_NETWORK_NAME, PXE_GATEWAY, PXE_SUBNET } from "./helpers/pxe-network.js";
-import { createPxeVm, destroyPxeVm, getVmMac, rebootPxeVm, readSerialLog } from "./helpers/pxe-vm.js";
-import { sshExec } from "./helpers/ssh.js";
-
-const IPXE_ARM64 = "/usr/share/ipxe/arm64-efi/snponly.efi";
-const AAVMF = "/usr/share/edk2/aarch64/QEMU_EFI.fd";
-
-const VM_MEMORY = 4096;
-const VM_VCPUS = 2;
-const VM_DISK_GB = 250;
-const SSH_USER = "lab";
-const BASTION_IP = PXE_GATEWAY;
-const DHCP_RANGE_START = `${PXE_SUBNET}.100`;
-const DHCP_RANGE_END = `${PXE_SUBNET}.200`;
-const SERIAL_PORT = 4555;
-
-// Emulated aarch64 -- generous timeouts throughout. Measured on an x86_64 host with no
-// KVM for aarch64: a single PXE boot to a running Anaconda takes ~18 minutes, almost all
-// of it downloading inst.stage2 over the network under TCG. Budget well above that;
-// timing out just short of success wastes a whole run.
-const LEASE_TIMEOUT_MS = 10 * 60_000;
-const DISCOVERY_TIMEOUT_MS = 35 * 60_000;
-const INSTALL_TIMEOUT_MS = 75 * 60_000;
-const SSH_TIMEOUT_MS = 35 * 60_000;
-
-const RUN_FULL_INSTALL = process.env["ARM_PXE_FULL"] === "1";
-
-function sleep(ms: number): Promise<void> {
-  return new Promise((r) => setTimeout(r, ms));
-}
-
-function findSshKey(): { pubKey: string; keyPath: string } {
-  const candidates: string[] = [];
-  if (process.env["SSH_KEY_PATH"]) candidates.push(process.env["SSH_KEY_PATH"]);
-  const homes = [homedir()];
-  const sudoUser = process.env["SUDO_USER"];
-  if (sudoUser) homes.push(join("/home", sudoUser));
-  for (const home of homes) {
-    for (const name of ["id_ed25519", "id_ecdsa", "id_rsa"]) {
-      candidates.push(join(home, ".ssh", name));
-    }
-  }
-  for (const keyPath of candidates) {
-    if (existsSync(keyPath) && existsSync(`${keyPath}.pub`)) {
-      return { pubKey: readFileSync(`${keyPath}.pub`, "utf-8").trim(), keyPath };
-    }
-  }
-  throw new Error("No SSH key found — set SSH_KEY_PATH or ensure keys exist in ~/.ssh/");
-}
-
-async function pollApi<T>(
-  url: string,
-  check: (data: T) => boolean,
-  timeoutMs: number,
-  intervalMs = 10_000,
-): Promise<T> {
-  const start = Date.now();
-  while (Date.now() - start < timeoutMs) {
-    try {
-      const res = await fetch(url);
-      if (res.ok) {
-        const data = (await res.json()) as T;
-        if (check(data)) return data;
-      }
-    } catch { /* bastion not up yet, or a network hiccup */ }
-    await sleep(intervalMs);
-  }
-  throw new Error(`Timeout after ${timeoutMs}ms polling ${url}`);
-}
-
-function requirePrerequisites(): void {
-  if (!existsSync("/usr/bin/qemu-system-aarch64")) {
-    throw new Error("qemu-system-aarch64 not installed. Run: sudo dnf install qemu-system-aarch64");
-  }
-  if (!existsSync(AAVMF)) {
-    throw new Error(`AAVMF firmware not found at ${AAVMF}. Run: sudo dnf install edk2-aarch64`);
-  }
-  if (!existsSync(IPXE_ARM64)) {
-    throw new Error(`arm64 iPXE not found at ${IPXE_ARM64}. Run: sudo dnf install ipxe-bootimgs-aarch64`);
-  }
-}
-
-/**
- * Confirm the arm64 iPXE binary implements EFI_LOAD_FILE2_PROTOCOL.
- *
- * Without it the `initrd` line is accepted and silently ignored, and the kernel panics
- * with unknown-block(0,0). Checking here turns a confusing 30-minute boot failure into
- * an immediate, explanatory one.
- *
- * GUID 4006c0c1-fcb3-403e-996d-4a6c8724e06d, little-endian in the binary's GUID table.
- */
-function assertIpxeSupportsLoadFile2(): void {
-  const LOAD_FILE2_GUID = Buffer.from("c1c00640b3fc3e40996d4a6c8724e06d", "hex");
-  const binary = readFileSync(IPXE_ARM64);
-  if (binary.indexOf(LOAD_FILE2_GUID) < 0) {
-    throw new Error(
-      `${IPXE_ARM64} does not reference EFI_LOAD_FILE2_PROTOCOL. On arm64 the kernel ` +
-      `EFI stub fetches the initrd over LoadFile2; without it iPXE drops the initrd ` +
-      `silently and the kernel panics with "unknown-block(0,0)". Rebuild iPXE with ` +
-      `LoadFile2, or chainload grubaa64.efi for aarch64 instead.`,
-    );
-  }
-  log(`iPXE arm64 implements LoadFile2 — initrd will be delivered to the EFI stub`);
-}
-
-interface Harness {
-  testDir: string;
-  app: { close: () => Promise<void> };
-  stopDnsmasq: () => void;
-  state: { update: (fn: (s: BastionStateLike) => void) => void };
-  vmMac: string;
-  httpPort: number;
-}
-
-/** Just the parts of BastionState this test seeds. */
-interface BastionStateLike {
-  discovered: Record<string, Record<string, unknown>>;
-  installed: Record<string, Record<string, unknown>>;
-  install_queue: Record<string, Record<string, unknown>>;
-  debug: Record<string, Record<string, unknown>>;
-}
-
-/** Bring up an isolated network, a bastion with both arch payloads, and an arm64 VM. */
-async function startHarness(vmName: string, httpPort: number, pubKey: string): Promise<Harness> {
-  requirePrerequisites();
-  assertIpxeSupportsLoadFile2();
-
-  log("Setting up PXE test network...");
-  ensurePxeNetwork();
-
-  const testDir = join(tmpdir(), `lab-arm-pxe-test-${Date.now()}`);
-  for (const sub of ["tftp", "http", "logs"]) {
-    mkdirSync(join(testDir, sub), { recursive: true });
-  }
-
-  const { createApp } = await import("../../src/bastion/src/server.js");
-  const { loadConfig } = await import("../../src/bastion/src/config.js");
-  const { generateDnsmasqConf, startDnsmasq, stopDnsmasq } = await import("../../src/bastion/src/services/dnsmasq.js");
-  const { generateDiscoverKickstart } = await import("../../src/bastion/src/services/kickstart-generator.js");
-  const { renderBootIpxe, kernelPath, initrdPath } = await import("../../src/bastion/src/templates/boot.ipxe.js");
-  // Relative, not "@lab/shared": these tests run from the repo root against sources,
-  // where the workspace package alias is not resolvable.
-  const { SUPPORTED_ARCHES, fedoraMirrorFor } = await import("../../src/shared/src/hardware/index.js");
-
-  const config = loadConfig({
-    bastionDir: testDir,
-    httpPort,
-    iface: "virbr-pxe",
-    serverIp: BASTION_IP,
-    network: `${PXE_SUBNET}.0`,
-    gateway: BASTION_IP,
-    dhcpMode: "full",
-    dhcpRangeStart: DHCP_RANGE_START,
-    dhcpRangeEnd: DHCP_RANGE_END,
-    domain: "arm-pxe-test.local",
-    sshKeys: [pubKey],
-    adminUser: SSH_USER,
-  });
-
-  // iPXE binaries. The arm64 one is the whole point: dnsmasq hands it out on DHCP
-  // option 93 -- 11 for UEFI PXE (TFTP) and 19 for UEFI HTTP Boot.
-  //
-  // They go in BOTH directories, exactly as main.ts stages them. AAVMF prefers HTTP
-  // Boot, so it is served an http:// URL and fetches from httpDir; a firmware that
-  // takes the TFTP path reads the same file from tftpDir. Staging only tftpDir gives a
-  // 404 and "No bootable option or device was found" on the console.
-  log("Staging iPXE binaries...");
-  const ipxeX86 = "/usr/share/ipxe/ipxe-snponly-x86_64.efi";
-  copyFileSync(IPXE_ARM64, join(config.tftpDir, "ipxe-arm64.efi"));
-  copyFileSync(IPXE_ARM64, join(config.httpDir, "ipxe-arm64.efi"));
-  if (existsSync(ipxeX86)) {
-    copyFileSync(ipxeX86, join(config.tftpDir, "ipxe.efi"));
-    copyFileSync(ipxeX86, join(config.httpDir, "ipxe.efi"));
-  }
-
-  // Fedora kernel + initrd for both architectures, cached across runs.
-  const cacheDir = "/var/lib/libvirt/images/lab-pxe-cache";
-  execSync(`mkdir -p "${cacheDir}"`, { stdio: "pipe" });
-
-  for (const arch of SUPPORTED_ARCHES) {
-    const mirror = fedoraMirrorFor(config.fedoraVersion, arch);
-    const kernelCache = join(cacheDir, `vmlinuz-${arch}`);
-    const initrdCache = join(cacheDir, `initrd-${arch}.img`);
-
-    if (!existsSync(kernelCache)) {
-      log(`Downloading Fedora ${config.fedoraVersion} ${arch} kernel...`);
-      execSync(`curl -# -L -f -o "${kernelCache}" "${mirror}/images/pxeboot/vmlinuz"`, { stdio: "inherit", timeout: 600_000 });
-    }
-    if (!existsSync(initrdCache)) {
-      log(`Downloading Fedora ${config.fedoraVersion} ${arch} initrd...`);
-      execSync(`curl -# -L -f -o "${initrdCache}" "${mirror}/images/pxeboot/initrd.img"`, { stdio: "inherit", timeout: 600_000 });
-    }
-
-    // Staged under the exact names the iPXE templates will ask for.
-    copyFileSync(kernelCache, join(config.httpDir, kernelPath(arch)));
-    copyFileSync(initrdCache, join(config.httpDir, initrdPath(arch)));
-    log(`Staged ${arch}: ${kernelPath(arch)} + ${initrdPath(arch)}`);
-  }
-
-  writeFileSync(join(config.httpDir, "discover.ks"), generateDiscoverKickstart(config));
-  writeFileSync(
-    join(config.httpDir, "boot.ipxe"),
-    renderBootIpxe({ serverIp: config.serverIp, httpPort: config.httpPort }),
-  );
-  generateDnsmasqConf(config);
-
-  const { app, state, syslog } = createApp(config);
-  await app.listen({ port: config.httpPort, host: "0.0.0.0" });
-  syslog.start();
-  log(`Bastion HTTP listening on :${config.httpPort}`);
-
-  log("Starting dnsmasq (full DHCP)...");
-  startDnsmasq(config).catch((err) => {
-    log(`dnsmasq failed: ${err instanceof Error ? err.message : String(err)}`);
-  });
-  await sleep(1500);
-
-  log("Creating aarch64 PXE VM (emulated — this is slow)...");
-  createPxeVm({
-    name: vmName,
-    memory: VM_MEMORY,
-    vcpus: VM_VCPUS,
-    diskSize: VM_DISK_GB,
-    network: PXE_NETWORK_NAME,
-    arch: "aarch64",
-  });
-
-  const vmMac = getVmMac(vmName);
-  if (!vmMac) throw new Error("Could not determine VM MAC address");
-  log(`ARM VM MAC: ${vmMac}`);
-
-  return {
-    testDir,
-    app,
-    stopDnsmasq,
-    state: state as unknown as Harness["state"],
-    vmMac,
-    httpPort: config.httpPort,
-  };
-}
-
-async function stopHarness(vmName: string, harness: Harness | undefined): Promise<void> {
-  // KEEP_VM=1 leaves the VM, network and bastion up so a failure can be inspected on
-  // the console. Emulated aarch64 runs cost half an hour; tearing the evidence down
-  // automatically means paying that again to see what happened.
-  if (process.env["KEEP_VM"] === "1") {
-    log(`KEEP_VM=1 — leaving ${vmName} running for inspection.`);
-    log(`  console:  sudo virsh screenshot ${vmName} /tmp/vm.ppm`);
-    log(`  serial:   socat - TCP:127.0.0.1:${SERIAL_PORT}`);
-    if (harness) log(`  bastion:  ${harness.testDir} (still serving on :${harness.httpPort})`);
-    log(`  cleanup:  sudo virsh destroy ${vmName}; sudo virsh undefine ${vmName} --remove-all-storage --nvram`);
-    return;
-  }
-
-  log("Cleaning up...");
-  if (harness) {
-    await harness.app.close().catch(() => {});
-    harness.stopDnsmasq();
-  }
-  destroyPxeVm(vmName);
-  destroyPxeNetwork();
-  if (harness) rmSync(harness.testDir, { recursive: true, force: true });
-}
-
-/** Read the DHCP lease the bastion handed a MAC. Rescue mode reports no IP itself. */
-function leaseIpFor(testDir: string, mac: string): string | null {
-  const leaseFile = join(testDir, "dnsmasq.leases");
-  if (!existsSync(leaseFile)) return null;
-  for (const line of readFileSync(leaseFile, "utf-8").split("\n")) {
-    // <expiry> <mac> <ip> <hostname> <clientid>
-    const parts = line.trim().split(/\s+/);
-    if (parts.length >= 3 && parts[1]?.toLowerCase() === mac.toLowerCase()) {
-      return parts[2] ?? null;
-    }
-  }
-  return null;
-}
-
-async function waitForLease(testDir: string, mac: string, timeoutMs: number): Promise<string> {
-  const start = Date.now();
-  while (Date.now() - start < timeoutMs) {
-    const ip = leaseIpFor(testDir, mac);
-    if (ip !== null) return ip;
-    await sleep(5000);
-  }
-  throw new Error(`No DHCP lease for ${mac} within ${timeoutMs}ms`);
-}
-
-// ---------------------------------------------------------------------------
-// Rescue path -- what the DGX Sparks need.
-// ---------------------------------------------------------------------------
-
-describe("ARM PXE rescue", () => {
-  const VM_NAME = "lab-arm-pxe-rescue";
-  const HTTP_PORT = 8096;
-  let harness: Harness | undefined;
-  let sshKeyPath: string;
-  let rescueIp: string;
-
-  beforeAll(async () => {
-    const { pubKey, keyPath } = findSshKey();
-    sshKeyPath = keyPath;
-    harness = await startHarness(VM_NAME, HTTP_PORT, pubKey);
-    const { testDir, vmMac, state } = harness;
-
-    // Seed the machine as an already-known aarch64 box queued for rescue. This is the
-    // DGX Spark situation exactly: SSH-onboarded, never PXE-discovered, architecture
-    // known only from its record -- and it also keeps the test to a SINGLE emulated
-    // boot. Each boot spends ~15 minutes pulling Anaconda's stage2 over the network
-    // under TCG, so discovering first and rescuing second doubles the runtime for no
-    // extra coverage of the path being tested. Discovery is covered by the full suite.
-    log(`Seeding ${vmMac} as a known aarch64 machine queued for rescue...`);
-    state.update((s) => {
-      s.discovered[vmMac] = {
-        mac: vmMac,
-        product: "Test ARM64 Machine",
-        board: "virt",
-        serial: "SN-ARM64",
-        manufacturer: "QEMU",
-        cpu_model: "cortex-a57",
-        cpu_cores: VM_VCPUS,
-        memory_gb: 4,
-        arch: "aarch64",
-        disks: [],
-        nics: [],
-        first_seen: new Date().toISOString(),
-        last_seen: new Date().toISOString(),
-      };
-      s.debug[vmMac] = { hostname: "arm-rescue-test", queued_at: new Date().toISOString() };
-    });
-
-    // Restart so the VM boots against the seeded state. createPxeVm already started it.
-    rebootPxeVm(VM_NAME);
-    await sleep(5_000);
-    deleteNftablesRejectRules();
-
-    // The whole chain now runs once: DHCP option 93 -> arm64 iPXE -> /boot.ipxe ->
-    // /dispatch (architecture from the record, not the query) -> aarch64 kernel +
-    // initrd -> Anaconda rescue -> sshd. Reaching a shell at all proves iPXE handed
-    // the initrd to the EFI stub over LoadFile2; without it the kernel panics first.
-    log("Waiting for the rescue environment's DHCP lease...");
-    rescueIp = await waitForLease(testDir, vmMac, LEASE_TIMEOUT_MS);
-    log(`Rescue IP: ${rescueIp}`);
-
-    log("Waiting for SSH into the rescue shell (started by inst.sshd)...");
-    log("(emulated aarch64 — Anaconda's stage2 download dominates; be patient)");
-    await waitForSsh(rescueIp, "root", SSH_TIMEOUT_MS, sshKeyPath).catch(async (err) => {
-      log("Rescue SSH timed out. Serial console:");
-      try {
-        log(await readSerialLog(SERIAL_PORT, { lastLines: 100, timeoutMs: 15_000 }));
-      } catch { /* console unavailable */ }
-      throw err;
-    });
-    log("ARM PXE rescue reached.");
-  }, LEASE_TIMEOUT_MS + SSH_TIMEOUT_MS + 300_000);
-
-  afterAll(async () => { await stopHarness(VM_NAME, harness); });
-
-  it("resolved the architecture from the machine record", async () => {
-    const res = await fetch(`http://${BASTION_IP}:${HTTP_PORT}/api/machines`);
-    const data = (await res.json()) as { discovered: Record<string, { arch: string }> };
-    expect(data.discovered[harness!.vmMac]?.arch).toBe("aarch64");
-  });
-
-  it("rescue shell is reachable over SSH and is aarch64", () => {
-    const result = sshExec(rescueIp, "root", "uname -m", { keyPath: sshKeyPath, timeout: 60_000 });
-    expect(result.exitCode).toBe(0);
-    expect(result.stdout.trim()).toBe("aarch64");
-  });
-
-  it("booted an initramfs — the LoadFile2 path worked", () => {
-    // If iPXE had dropped the initrd the kernel would never have reached userspace at
-    // all, but assert it explicitly so a regression names itself.
-    const result = sshExec(rescueIp, "root", "cat /proc/cmdline; ls /run/install", {
-      keyPath: sshKeyPath, timeout: 60_000,
-    });
-    expect(result.exitCode).toBe(0);
-    expect(result.stdout).toContain("inst.rescue");
-  });
-
-  it("rescue kernel came from the bastion over HTTP", () => {
-    const result = sshExec(rescueIp, "root", "cat /proc/cmdline", { keyPath: sshKeyPath, timeout: 60_000 });
-    expect(result.stdout).toContain(`${BASTION_IP}:${HTTP_PORT}`);
-    // arm64 gets serial console arguments, never nomodeset.
-    expect(result.stdout).toContain("console=ttyAMA0");
-    expect(result.stdout).not.toContain("nomodeset");
-  });
-
-  it("has LVM tools available for inspecting an installed system", () => {
-    const result = sshExec(rescueIp, "root", "command -v vgchange && command -v lsblk", {
-      keyPath: sshKeyPath, timeout: 60_000,
-    });
-    expect(result.exitCode).toBe(0);
-  });
-});
-
-// ---------------------------------------------------------------------------
-// Full install -- opt-in, ~60-90 minutes emulated.
-// ---------------------------------------------------------------------------
-
-describe.runIf(RUN_FULL_INSTALL)("ARM PXE install", () => {
-  const VM_NAME = "lab-arm-pxe-install";
-  const HTTP_PORT = 8095;
-  let harness: Harness | undefined;
-  let sshKeyPath: string;
-  let vmIp: string;
-
-  beforeAll(async () => {
-    const { pubKey, keyPath } = findSshKey();
-    sshKeyPath = keyPath;
-    harness = await startHarness(VM_NAME, HTTP_PORT, pubKey);
-    const { vmMac } = harness;
-
-    log("Waiting for aarch64 discovery...");
-    await pollApi<{ discovered: Record<string, unknown> }>(
-      `http://${BASTION_IP}:${HTTP_PORT}/api/machines`,
-      (data) => vmMac in data.discovered,
-      DISCOVERY_TIMEOUT_MS,
-    );
-    log("Discovered. Queueing install...");
-
-    const installRes = await fetch(`http://${BASTION_IP}:${HTTP_PORT}/api/install`, {
-      method: "POST",
-      headers: { "Content-Type": "application/json" },
-      body: JSON.stringify({ mac: vmMac, hostname: VM_NAME, disk: "", role: "vanilla" }),
-    });
-    expect(installRes.status).toBe(200);
-
-    await sleep(30_000);
-    rebootPxeVm(VM_NAME);
-
-    log("Waiting for the emulated aarch64 install (60-90 min)...");
-    type LogsResponse = { status: string; progress: string; ip?: string };
-    const final = await pollApi<LogsResponse>(
-      `http://${BASTION_IP}:${HTTP_PORT}/api/logs/${encodeURIComponent(vmMac)}`,
-      (d) => d.status === "installed" || d.progress === "error",
-      INSTALL_TIMEOUT_MS,
-      30_000,
-    );
-
-    if (final.progress === "error") {
-      const logs = await (await fetch(`http://${BASTION_IP}:${HTTP_PORT}/api/logs/${encodeURIComponent(vmMac)}`)).json();
-      log(`ARM install FAILED: ${JSON.stringify(logs, null, 2)}`);
-      throw new Error("ARM PXE install failed — see logs above");
-    }
-
-    vmIp = final.ip ?? "";
-    log(`ARM install complete. IP: ${vmIp}`);
-
-    await sleep(30_000);
-    rebootPxeVm(VM_NAME);
-    await sleep(5_000);
-    deleteNftablesRejectRules();
-    await waitForSsh(vmIp, SSH_USER, SSH_TIMEOUT_MS, sshKeyPath);
-  }, DISCOVERY_TIMEOUT_MS + INSTALL_TIMEOUT_MS + SSH_TIMEOUT_MS + 600_000);
-
-  afterAll(async () => { await stopHarness(VM_NAME, harness); });
-
-  it("machine reached installed state", async () => {
-    const res = await fetch(`http://${BASTION_IP}:${HTTP_PORT}/api/machines`);
-    const data = (await res.json()) as { installed: Record<string, { hostname: string }> };
-    expect(data.installed[harness!.vmMac]?.hostname).toBe(VM_NAME);
-  });
-
-  it("installed system is aarch64", () => {
-    const result = sshExec(vmIp, SSH_USER, "uname -m", { keyPath: sshKeyPath, timeout: 60_000 });
-    expect(result.stdout.trim()).toBe("aarch64");
-  });
-
-  it("SSH works with the admin user", () => {
-    const result = sshExec(vmIp, SSH_USER, "whoami", { keyPath: sshKeyPath, timeout: 60_000 });
-    expect(result.stdout.trim()).toBe(SSH_USER);
-  });
-
-  it("LVM layout is correct", () => {
-    const result = sshExec(vmIp, SSH_USER, "sudo lvs labvg --noheadings -o lv_name", {
-      keyPath: sshKeyPath, timeout: 60_000,
-    });
-    expect(result.exitCode).toBe(0);
-    const lvs = result.stdout.trim().split("\n").map((l) => l.trim());
-    for (const expected of ["root", "var", "varlog", "swap", "home", "srv"]) {
-      expect(lvs).toContain(expected);
-    }
-  });
-});

bastion/tests/integration/helpers/pxe-vm.ts

@@ -29,6 +29,17 @@ export interface PxeVmConfig {
   diskSize: number;     // GB
   network: string;      // libvirt network name
   arch?: "x86_64" | "aarch64";
+  /**
+   * Extra NICs enumerated BEFORE the PXE NIC, on a network with no route to
+   * the bastion (defaults to libvirt's "default").
+   *
+   * Real multi-NIC boxes expose a class of bug a single-NIC VM cannot: an
+   * initramfs that picks "the first connected interface" grabs one of these
+   * instead of the NIC that PXE booted, and then cannot reach the bastion.
+   * Defaults to 0 (single NIC).
+   */
+  decoyNics?: number;
+  decoyNetwork?: string;
 }
 
 /** Create a blank UEFI VM that PXE boots from the network. */
@@ -61,6 +72,10 @@ export function createPxeVm(config: PxeVmConfig): void {
     `--memory=${config.memory}`,
     `--vcpus=${config.vcpus}`,
     `--disk=path=${diskPath},format=qcow2,bus=virtio`,
+    // Decoys first so they enumerate ahead of the PXE NIC. They are up and
+    // carry a lease, but have no route to the bastion.
+    ...Array.from({ length: config.decoyNics ?? 0 }, () =>
+      `--network=network=${config.decoyNetwork ?? "default"},model=virtio`),
     `--network=network=${config.network},model=virtio`,
     // UEFI firmware — required for PXE boot in modern mode
     `--boot=uefi,network,hd`,
@@ -95,12 +110,21 @@ export function destroyPxeVm(name: string): void {
 }
 
 /** Get the MAC address of a VM's first NIC. */
-export function getVmMac(name: string): string | null {
+export function getVmMac(name: string, network?: string): string | null {
   const result = virsh("domiflist", name);
   if (result.status !== 0) return null;
   // Output format: Interface  Type  Source  Model  MAC
-  const match = result.stdout.match(/([0-9a-f]{2}:[0-9a-f]{2}:[0-9a-f]{2}:[0-9a-f]{2}:[0-9a-f]{2}:[0-9a-f]{2})/i);
-  return match ? match[1].toLowerCase() : null;
+  // With decoy NICs present, match the line for the PXE network so we return
+  // the NIC that actually boots rather than whichever is listed first.
+  const lines = result.stdout.split("\n");
+  const candidates = network === undefined
+    ? lines
+    : lines.filter((l) => l.split(/\s+/).includes(network));
+  for (const line of candidates.length > 0 ? candidates : lines) {
+    const m = line.match(/([0-9a-f]{2}:[0-9a-f]{2}:[0-9a-f]{2}:[0-9a-f]{2}:[0-9a-f]{2}:[0-9a-f]{2})/i);
+    if (m) return m[1].toLowerCase();
+  }
+  return null;
 }
 
 /** Reboot a VM (force off + start). */

bastion/tests/integration/k3s-single-node.test.ts

@@ -161,7 +161,7 @@ EOF'
       CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
       curl -L --fail --silent "https://github.com/cilium/cilium-cli/releases/download/\${CILIUM_CLI_VERSION}/cilium-linux-amd64.tar.gz" | sudo tar xz -C /usr/local/bin
       DEFAULT_DEV=$(ip -4 route show default | awk '{print $5}' | head -1)
-      sudo KUBECONFIG=/etc/rancher/k3s/k3s.yaml cilium install --set kubeProxyReplacement=true --set ipam.mode=kubernetes --set devices=$DEFAULT_DEV --set nodePort.directRoutingDevice=$DEFAULT_DEV
+      sudo KUBECONFIG=/etc/rancher/k3s/k3s.yaml cilium install --set kubeProxyReplacement=true --set ipam.mode=cluster-pool --set ipam.operator.clusterPoolIPv4PodCIDRList='{10.42.0.0/16}' --set ipam.operator.clusterPoolIPv4MaskSize=24 --set devices=$DEFAULT_DEV --set nodePort.directRoutingDevice=$DEFAULT_DEV
     `.trim(), "cilium install", { keyPath: sshKeyPath, timeout: 120_000 });
 
     log("Waiting for Cilium to be ready...");

bastion/tests/integration/pxe-rescue.test.ts

@@ -1,210 +0,0 @@
-// Integration test: `labctl provision debug` -> Anaconda rescue with SSH, on x86_64.
-//
-// The rescue path had no test coverage on any architecture, which matters because it is
-// the lab's recovery tool of last resort -- the thing you reach for when a machine will
-// not boot. It runs here on x86_64 with KVM so it completes in minutes; the aarch64
-// equivalent is the same code path with a different kernel, but is emulated and far too
-// slow to iterate on.
-//
-// Run: sudo ./scripts/test-provision.sh rescue
-
-import { describe, it, expect, beforeAll, afterAll } from "vitest";
-import { readFileSync, existsSync, mkdirSync, rmSync, copyFileSync, writeFileSync } from "node:fs";
-import { execSync } from "node:child_process";
-import { join } from "node:path";
-import { homedir, tmpdir } from "node:os";
-import { log, waitForSsh } from "./helpers/libvirt.js";
-import { ensurePxeNetwork, destroyPxeNetwork, deleteNftablesRejectRules, PXE_NETWORK_NAME, PXE_GATEWAY, PXE_SUBNET } from "./helpers/pxe-network.js";
-import { createPxeVm, destroyPxeVm, getVmMac, rebootPxeVm, readSerialLog } from "./helpers/pxe-vm.js";
-import { sshExec } from "./helpers/ssh.js";
-
-const VM_NAME = "lab-pxe-rescue-test";
-const HTTP_PORT = 8094;
-const VM_MEMORY = 4096;
-const VM_VCPUS = 4;
-const VM_DISK_GB = 20;
-const BASTION_IP = PXE_GATEWAY;
-const SERIAL_PORT = 4555;
-
-const LEASE_TIMEOUT_MS = 8 * 60_000;
-const SSH_TIMEOUT_MS = 15 * 60_000;
-
-function sleep(ms: number): Promise<void> {
-  return new Promise((r) => setTimeout(r, ms));
-}
-
-function findSshKey(): { pubKey: string; keyPath: string } {
-  const candidates: string[] = [];
-  if (process.env["SSH_KEY_PATH"]) candidates.push(process.env["SSH_KEY_PATH"]);
-  const homes = [homedir()];
-  const sudoUser = process.env["SUDO_USER"];
-  if (sudoUser) homes.push(join("/home", sudoUser));
-  for (const home of homes) {
-    for (const name of ["id_ed25519", "id_ecdsa", "id_rsa"]) candidates.push(join(home, ".ssh", name));
-  }
-  for (const keyPath of candidates) {
-    if (existsSync(keyPath) && existsSync(`${keyPath}.pub`)) {
-      return { pubKey: readFileSync(`${keyPath}.pub`, "utf-8").trim(), keyPath };
-    }
-  }
-  throw new Error("No SSH key found — set SSH_KEY_PATH or ensure keys exist in ~/.ssh/");
-}
-
-function leaseIpFor(testDir: string, mac: string): string | null {
-  const leaseFile = join(testDir, "dnsmasq.leases");
-  if (!existsSync(leaseFile)) return null;
-  for (const line of readFileSync(leaseFile, "utf-8").split("\n")) {
-    const parts = line.trim().split(/\s+/);
-    if (parts.length >= 3 && parts[1]?.toLowerCase() === mac.toLowerCase()) return parts[2] ?? null;
-  }
-  return null;
-}
-
-async function waitForLease(testDir: string, mac: string, timeoutMs: number): Promise<string> {
-  const start = Date.now();
-  while (Date.now() - start < timeoutMs) {
-    const ip = leaseIpFor(testDir, mac);
-    if (ip !== null) return ip;
-    await sleep(5000);
-  }
-  throw new Error(`No DHCP lease for ${mac} within ${timeoutMs}ms`);
-}
-
-// Suite name must not be a substring of "ARM PXE rescue" -- vitest -t matches
-// substrings, so a looser name here would drag the emulated aarch64 suite in with it.
-describe("x86 rescue boot", () => {
-  let app: { close: () => Promise<void> };
-  let stopDnsmasqFn: () => void;
-  let testDir: string;
-  let vmMac: string;
-  let rescueIp: string;
-  let sshKeyPath: string;
-
-  beforeAll(async () => {
-    const { pubKey, keyPath } = findSshKey();
-    sshKeyPath = keyPath;
-
-    log("Setting up PXE test network...");
-    ensurePxeNetwork();
-
-    testDir = join(tmpdir(), `lab-pxe-rescue-${Date.now()}`);
-    for (const sub of ["tftp", "http", "logs"]) mkdirSync(join(testDir, sub), { recursive: true });
-
-    const { createApp } = await import("../../src/bastion/src/server.js");
-    const { loadConfig } = await import("../../src/bastion/src/config.js");
-    const { generateDnsmasqConf, startDnsmasq, stopDnsmasq } = await import("../../src/bastion/src/services/dnsmasq.js");
-    const { renderBootIpxe, kernelPath, initrdPath } = await import("../../src/bastion/src/templates/boot.ipxe.js");
-    stopDnsmasqFn = stopDnsmasq;
-
-    const config = loadConfig({
-      bastionDir: testDir,
-      httpPort: HTTP_PORT,
-      iface: "virbr-pxe",
-      serverIp: BASTION_IP,
-      network: `${PXE_SUBNET}.0`,
-      gateway: BASTION_IP,
-      dhcpMode: "full",
-      dhcpRangeStart: `${PXE_SUBNET}.100`,
-      dhcpRangeEnd: `${PXE_SUBNET}.200`,
-      domain: "rescue-test.local",
-      sshKeys: [pubKey],
-      adminUser: "lab",
-    });
-
-    // iPXE in both dirs: TFTP PXE and UEFI HTTP Boot are both possible, and OVMF picks.
-    const ipxeX86 = "/usr/share/ipxe/ipxe-snponly-x86_64.efi";
-    if (!existsSync(ipxeX86)) throw new Error(`iPXE not found: ${ipxeX86}`);
-    copyFileSync(ipxeX86, join(config.tftpDir, "ipxe.efi"));
-    copyFileSync(ipxeX86, join(config.httpDir, "ipxe.efi"));
-
-    const cacheDir = "/var/lib/libvirt/images/lab-pxe-cache";
-    execSync(`mkdir -p "${cacheDir}"`, { stdio: "pipe" });
-    const kernelCache = join(cacheDir, "vmlinuz-x86_64");
-    const initrdCache = join(cacheDir, "initrd-x86_64.img");
-    if (!existsSync(kernelCache)) {
-      log("Downloading Fedora x86_64 kernel...");
-      execSync(`curl -# -L -f -o "${kernelCache}" "${config.fedoraMirror}/images/pxeboot/vmlinuz"`, { stdio: "inherit", timeout: 600_000 });
-    }
-    if (!existsSync(initrdCache)) {
-      log("Downloading Fedora x86_64 initrd...");
-      execSync(`curl -# -L -f -o "${initrdCache}" "${config.fedoraMirror}/images/pxeboot/initrd.img"`, { stdio: "inherit", timeout: 600_000 });
-    }
-    copyFileSync(kernelCache, join(config.httpDir, kernelPath("x86_64")));
-    copyFileSync(initrdCache, join(config.httpDir, initrdPath("x86_64")));
-
-    writeFileSync(join(config.httpDir, "boot.ipxe"), renderBootIpxe({ serverIp: config.serverIp, httpPort: config.httpPort }));
-    generateDnsmasqConf(config);
-
-    const { app: fastify, state, syslog } = createApp(config);
-    app = fastify;
-    await fastify.listen({ port: config.httpPort, host: "0.0.0.0" });
-    syslog.start();
-    log(`Bastion HTTP listening on :${HTTP_PORT}`);
-
-    startDnsmasq(config).catch((err) => log(`dnsmasq failed: ${err instanceof Error ? err.message : String(err)}`));
-    await sleep(1500);
-
-    log("Creating x86_64 PXE VM (KVM)...");
-    createPxeVm({ name: VM_NAME, memory: VM_MEMORY, vcpus: VM_VCPUS, diskSize: VM_DISK_GB, network: PXE_NETWORK_NAME });
-    const mac = getVmMac(VM_NAME);
-    if (!mac) throw new Error("Could not determine VM MAC");
-    vmMac = mac;
-    log(`VM MAC: ${vmMac}`);
-
-    // Queue rescue directly, as `labctl provision debug` does.
-    log("Queueing debug/rescue mode...");
-    state.update((s) => {
-      s.debug[vmMac] = { hostname: "rescue-test", queued_at: new Date().toISOString() };
-    });
-
-    rebootPxeVm(VM_NAME);
-    await sleep(5_000);
-    deleteNftablesRejectRules();
-
-    rescueIp = await waitForLease(testDir, vmMac, LEASE_TIMEOUT_MS);
-    log(`Rescue IP: ${rescueIp}`);
-
-    log("Waiting for SSH into the rescue shell (inst.sshd)...");
-    await waitForSsh(rescueIp, "root", SSH_TIMEOUT_MS, sshKeyPath).catch(async (err) => {
-      log("Rescue SSH timed out. Serial console:");
-      try { log(await readSerialLog(SERIAL_PORT, { lastLines: 120, timeoutMs: 20_000 })); } catch { /* none */ }
-      throw err;
-    });
-    log("Rescue shell reachable.");
-  }, LEASE_TIMEOUT_MS + SSH_TIMEOUT_MS + 300_000);
-
-  afterAll(async () => {
-    if (process.env["KEEP_VM"] === "1") {
-      log(`KEEP_VM=1 — leaving ${VM_NAME} up (serial: socat - TCP:127.0.0.1:${SERIAL_PORT})`);
-      return;
-    }
-    log("Cleaning up...");
-    if (app) await app.close().catch(() => {});
-    if (stopDnsmasqFn) stopDnsmasqFn();
-    destroyPxeVm(VM_NAME);
-    destroyPxeNetwork();
-    if (testDir) rmSync(testDir, { recursive: true, force: true });
-  });
-
-  it("rescue shell is reachable over SSH as root", () => {
-    const result = sshExec(rescueIp, "root", "whoami", { keyPath: sshKeyPath, timeout: 60_000 });
-    expect(result.exitCode).toBe(0);
-    expect(result.stdout.trim()).toBe("root");
-  });
-
-  it("is the Anaconda rescue environment", () => {
-    const result = sshExec(rescueIp, "root", "cat /proc/cmdline", { keyPath: sshKeyPath, timeout: 60_000 });
-    expect(result.stdout).toContain("inst.rescue");
-    expect(result.stdout).toContain("inst.sshd");
-  });
-
-  it("kernel and initrd came from the bastion", () => {
-    const result = sshExec(rescueIp, "root", "cat /proc/cmdline", { keyPath: sshKeyPath, timeout: 60_000 });
-    expect(result.stdout).toContain(`${BASTION_IP}:${HTTP_PORT}`);
-  });
-
-  it("has LVM tools for inspecting an installed system", () => {
-    const result = sshExec(rescueIp, "root", "command -v vgchange && command -v lsblk", { keyPath: sshKeyPath, timeout: 60_000 });
-    expect(result.exitCode).toBe(0);
-  });
-});

bastion/tests/integration/vyos-provision.test.ts

@@ -0,0 +1,387 @@
+// Integration test: full VyOS unattended provisioning flow.
+//
+// Validates the VyOS install path end-to-end, at the same depth as the Fedora
+// pxe-provision test:
+//   1. Bastion (HTTP + dnsmasq) on the isolated libvirt PXE network
+//   2. Blank UEFI VM PXE boots -> Fedora-based discovery (OS-neutral)
+//   3. Queue os=vyos-rolling -> live boot + live-config hook + pty driver
+//   4. Fresh-install asserts: installed.ip, streamed logs, applied config,
+//      /config/lab-provisioned, boot-order handling
+//   5. REINSTALL round: previous config + /config data carried forward
+//      ("reinstall without losing data", VyOS-flavored)
+//   6. freshConfig round: bastion-generated config wins, /config data kept
+//
+// Prerequisites: libvirtd, OVMF, ipxe-bootimgs-x86, sudo, internet
+// (first run downloads the ~600MB VyOS nightly ISO; artifacts are cached).
+// Run: sudo pnpm run test:integration:vyos
+
+import { describe, it, expect, beforeAll, afterAll } from "vitest";
+import { readFileSync, existsSync, mkdirSync, rmSync, copyFileSync, symlinkSync, writeFileSync } from "node:fs";
+import { execSync } from "node:child_process";
+import { join } from "node:path";
+import { homedir, tmpdir } from "node:os";
+import { log, waitForSsh } from "./helpers/libvirt.js";
+import { ensurePxeNetwork, destroyPxeNetwork, deleteNftablesRejectRules, PXE_NETWORK_NAME, PXE_GATEWAY, PXE_SUBNET } from "./helpers/pxe-network.js";
+import { createPxeVm, destroyPxeVm, getVmMac, rebootPxeVm } from "./helpers/pxe-vm.js";
+import { sshExec } from "./helpers/ssh.js";
+
+const VM_NAME = "lab-vyos-test";
+const VM_MEMORY = 4096;
+const VM_VCPUS = 4;
+const VM_DISK_GB = 10;       // VyOS image install needs ~2GB minimum
+const HTTP_PORT = 8099;
+const SSH_USER = "vyos";     // the only VyOS login user
+const BASTION_IP = PXE_GATEWAY;
+const DHCP_RANGE_START = `${PXE_SUBNET}.100`;
+const DHCP_RANGE_END = `${PXE_SUBNET}.200`;
+
+const DISCOVERY_TIMEOUT_MS = 5 * 60_000;
+const INSTALL_TIMEOUT_MS = 15 * 60_000;  // squashfs fetch + copy; much faster than Anaconda
+const SSH_TIMEOUT_MS = 8 * 60_000;
+
+const HOSTNAME_R1 = "vyos-r1";
+const HOSTNAME_R2 = "vyos-r2";
+const HOSTNAME_R3 = "vyos-r3";
+
+function findSshKey(): { pubKey: string; keyPath: string } {
+  const homes = [homedir()];
+  const sudoUser = process.env["SUDO_USER"];
+  if (sudoUser) homes.push(join("/home", sudoUser));
+  if (process.env["SSH_KEY_PATH"]) {
+    const keyPath = process.env["SSH_KEY_PATH"];
+    const pubPath = `${keyPath}.pub`;
+    if (existsSync(keyPath) && existsSync(pubPath)) {
+      return { pubKey: readFileSync(pubPath, "utf-8").trim(), keyPath };
+    }
+  }
+  for (const home of homes) {
+    for (const name of ["id_ed25519", "id_ecdsa", "id_rsa"]) {
+      const keyPath = join(home, ".ssh", name);
+      const pubPath = `${keyPath}.pub`;
+      if (existsSync(keyPath) && existsSync(pubPath)) {
+        return { pubKey: readFileSync(pubPath, "utf-8").trim(), keyPath };
+      }
+    }
+  }
+  throw new Error("No SSH key found — set SSH_KEY_PATH or ensure keys exist in ~/.ssh/");
+}
+
+function sleep(ms: number): Promise<void> {
+  return new Promise((r) => setTimeout(r, ms));
+}
+
+async function pollApi<T>(
+  url: string,
+  check: (data: T) => boolean,
+  timeoutMs: number,
+  intervalMs = 5000,
+): Promise<T> {
+  const start = Date.now();
+  while (Date.now() - start < timeoutMs) {
+    try {
+      const res = await fetch(url);
+      if (res.ok) {
+        const data = (await res.json()) as T;
+        if (check(data)) return data;
+      }
+    } catch { /* not ready yet */ }
+    await sleep(intervalMs);
+  }
+  throw new Error(`Timeout after ${timeoutMs}ms polling ${url}`);
+}
+
+type LogsResponse = {
+  status: string;
+  progress: string;
+  progress_detail?: string;
+  ip?: string;
+  log_total?: number;
+  log_lines?: Array<{ line: string }>;
+};
+
+/** Queue a VyOS install, reboot the VM into PXE, wait for completion + SSH. */
+async function installRound(opts: {
+  mac: string;
+  hostname: string;
+  freshConfig?: boolean;
+}): Promise<string> {
+  const body = {
+    mac: opts.mac,
+    hostname: opts.hostname,
+    disk: "/dev/vda",
+    role: "vanilla",
+    os: "vyos-rolling",
+    vyos: {
+      mgmtInterface: "eth0",
+      mgmtAddress: "dhcp",
+      hwIds: { eth0: opts.mac },
+      ...(opts.freshConfig ? { freshConfig: true } : {}),
+    },
+  };
+  const res = await fetch(`http://${BASTION_IP}:${HTTP_PORT}/api/install`, {
+    method: "POST",
+    headers: { "Content-Type": "application/json" },
+    body: JSON.stringify(body),
+  });
+  log(`Install queued (${opts.hostname}): ${JSON.stringify(await res.json())}`);
+
+  await sleep(5_000);
+  rebootPxeVm(VM_NAME);
+  await sleep(3_000);
+  deleteNftablesRejectRules();
+
+  const finalState = await pollApi<LogsResponse>(
+    `http://${BASTION_IP}:${HTTP_PORT}/api/logs/${encodeURIComponent(opts.mac)}`,
+    (data) => data.status === "installed" || data.progress === "error",
+    INSTALL_TIMEOUT_MS,
+    10_000,
+  );
+  if (finalState.progress === "error") {
+    log(`INSTALL FAILED: ${JSON.stringify(finalState.progress_detail ?? finalState, null, 2)}`);
+    throw new Error(`VyOS install failed for ${opts.hostname}`);
+  }
+  const ip = finalState.ip ?? "";
+  log(`Install complete (${opts.hostname}). IP: ${ip}`);
+
+  // The driver force-reboots; the VM PXE boots, dispatch says installed ->
+  // localboot exit -> GRUB -> VyOS. nftables reject rules do not reappear
+  // (guest reboot, not a libvirt restart), but clearing is harmless.
+  deleteNftablesRejectRules();
+  await waitForSsh(ip, SSH_USER, SSH_TIMEOUT_MS, sshKeyPathGlobal);
+  return ip;
+}
+
+let sshKeyPathGlobal = "";
+
+describe("VyOS provisioning", () => {
+  let bastionApp: { close: () => Promise<void> };
+  let testDir: string;
+  let vmMac: string;
+  let vmIp: string;
+
+  beforeAll(async () => {
+    const { pubKey, keyPath } = findSshKey();
+    sshKeyPathGlobal = keyPath;
+
+    log("Setting up PXE test network...");
+    ensurePxeNetwork();
+
+    testDir = join(tmpdir(), `lab-vyos-test-${Date.now()}`);
+    mkdirSync(join(testDir, "tftp"), { recursive: true });
+    mkdirSync(join(testDir, "http"), { recursive: true });
+    mkdirSync(join(testDir, "logs"), { recursive: true });
+
+    log("Starting bastion...");
+    const { createApp } = await import("../../src/bastion/src/server.js");
+    const { loadConfig } = await import("../../src/bastion/src/config.js");
+    const { generateDnsmasqConf, startDnsmasq } = await import("../../src/bastion/src/services/dnsmasq.js");
+    const { generateDiscoverKickstart } = await import("../../src/bastion/src/services/kickstart-generator.js");
+    const { renderBootIpxe } = await import("../../src/bastion/src/templates/boot.ipxe.js");
+    const { prepareVyosArtifacts } = await import("../../src/bastion/src/main.js");
+
+    const config = loadConfig({
+      bastionDir: testDir,
+      httpPort: HTTP_PORT,
+      iface: "virbr-pxe",
+      serverIp: BASTION_IP,
+      network: `${PXE_SUBNET}.0`,
+      gateway: BASTION_IP,
+      dhcpMode: "full",
+      dhcpRangeStart: DHCP_RANGE_START,
+      dhcpRangeEnd: DHCP_RANGE_END,
+      domain: "pxe-test.local",
+      sshKeys: [pubKey],
+      adminUser: "lab",
+    });
+
+    // iPXE binary
+    const ipxeSrc = "/usr/share/ipxe/ipxe-snponly-x86_64.efi";
+    if (!existsSync(ipxeSrc)) {
+      throw new Error(`iPXE not found: ${ipxeSrc}. Install: sudo dnf install ipxe-bootimgs-x86`);
+    }
+    copyFileSync(ipxeSrc, join(config.tftpDir, "ipxe.efi"));
+    try { symlinkSync(join(config.tftpDir, "ipxe.efi"), join(config.httpDir, "ipxe.efi")); } catch { /* exists */ }
+
+    const cacheDir = "/var/lib/libvirt/images/lab-pxe-cache";
+    execSync(`mkdir -p "${cacheDir}"`, { stdio: "pipe" });
+
+    // Fedora kernel+initrd for DISCOVERY (OS-neutral, same as pxe test)
+    const kernel = join(cacheDir, `vmlinuz-${config.fedoraVersion}`);
+    const initrd = join(cacheDir, `initrd-${config.fedoraVersion}.img`);
+    if (!existsSync(kernel)) {
+      log(`Downloading Fedora ${config.fedoraVersion} kernel (discovery)...`);
+      execSync(`curl -# -L -f -o "${kernel}" "${config.fedoraMirror}/images/pxeboot/vmlinuz"`, { stdio: "inherit", timeout: 300_000 });
+    }
+    if (!existsSync(initrd)) {
+      log(`Downloading Fedora ${config.fedoraVersion} initrd (discovery)...`);
+      execSync(`curl -# -L -f -o "${initrd}" "${config.fedoraMirror}/images/pxeboot/initrd.img"`, { stdio: "inherit", timeout: 300_000 });
+    }
+    copyFileSync(kernel, join(config.httpDir, "vmlinuz"));
+    copyFileSync(initrd, join(config.httpDir, "initrd.img"));
+
+    // VyOS netboot artifacts — cache the three extracted files across runs
+    const vyosCache = {
+      kernel: join(cacheDir, "vyos-vmlinuz"),
+      initrd: join(cacheDir, "vyos-initrd"),
+      squashfs: join(cacheDir, "vyos-filesystem.squashfs"),
+    };
+    if (Object.values(vyosCache).every((p) => existsSync(p))) {
+      log("VyOS netboot artifacts cached");
+      copyFileSync(vyosCache.kernel, join(config.httpDir, "vyos-vmlinuz"));
+      copyFileSync(vyosCache.initrd, join(config.httpDir, "vyos-initrd"));
+      copyFileSync(vyosCache.squashfs, join(config.httpDir, "vyos-filesystem.squashfs"));
+    } else {
+      log("Extracting VyOS artifacts from ISO (downloads ~600MB on first run)...");
+      prepareVyosArtifacts(config);
+      copyFileSync(join(config.httpDir, "vyos-vmlinuz"), vyosCache.kernel);
+      copyFileSync(join(config.httpDir, "vyos-initrd"), vyosCache.initrd);
+      copyFileSync(join(config.httpDir, "vyos-filesystem.squashfs"), vyosCache.squashfs);
+    }
+
+    writeFileSync(join(config.httpDir, "discover.ks"), generateDiscoverKickstart(config));
+    writeFileSync(join(config.httpDir, "boot.ipxe"), renderBootIpxe({ serverIp: config.serverIp, httpPort: config.httpPort }));
+    generateDnsmasqConf(config);
+
+    const { app, syslog } = createApp(config);
+    bastionApp = app;
+    await app.listen({ port: config.httpPort, host: "0.0.0.0" });
+    syslog.start();
+    log(`Bastion listening on :${HTTP_PORT}`);
+
+    log("Starting dnsmasq...");
+    startDnsmasq(config).catch((err) => {
+      log(`dnsmasq failed (expected without root): ${err instanceof Error ? err.message : String(err)}`);
+    });
+    await sleep(1000);
+
+    log("Creating PXE VM...");
+    // Two decoy NICs ahead of the PXE NIC, on a network with no route to the
+    // bastion. This reproduces the real VP2440 topology: live-boot scans for
+    // "the first connected interface", and without BOOTIF it picks a decoy,
+    // times out on DHCP/fetch, and dies with "Unable to find a live file
+    // system on the network". A single-NIC VM cannot catch that.
+    createPxeVm({
+      name: VM_NAME,
+      memory: VM_MEMORY,
+      vcpus: VM_VCPUS,
+      diskSize: VM_DISK_GB,
+      network: PXE_NETWORK_NAME,
+      decoyNics: 2,
+    });
+    const mac = getVmMac(VM_NAME, PXE_NETWORK_NAME);
+    if (!mac) throw new Error("Could not determine VM MAC address");
+    vmMac = mac;
+    log(`VM MAC: ${vmMac}`);
+
+    log("Waiting for discovery...");
+    type MachinesResponse = { discovered: Record<string, unknown> };
+    await pollApi<MachinesResponse>(
+      `http://${BASTION_IP}:${HTTP_PORT}/api/machines`,
+      (data) => vmMac in data.discovered,
+      DISCOVERY_TIMEOUT_MS,
+    );
+    log("VM discovered. Running fresh VyOS install (round 1)...");
+
+    await sleep(15_000); // discovery reboot cycle
+    vmIp = await installRound({ mac: vmMac, hostname: HOSTNAME_R1 });
+    log("Round 1 (fresh install) complete.");
+  }, DISCOVERY_TIMEOUT_MS + INSTALL_TIMEOUT_MS + SSH_TIMEOUT_MS + 300_000);
+
+  afterAll(async () => {
+    log("Cleaning up...");
+    if (bastionApp) await bastionApp.close().catch(() => {});
+    const { stopDnsmasq } = await import("../../src/bastion/src/services/dnsmasq.js");
+    stopDnsmasq();
+    destroyPxeVm(VM_NAME);
+    destroyPxeNetwork();
+    if (testDir) rmSync(testDir, { recursive: true, force: true });
+  });
+
+  it("machine is installed with a real IP (WI-1: ready-at parsing)", async () => {
+    const res = await fetch(`http://${BASTION_IP}:${HTTP_PORT}/api/machines`);
+    const data = (await res.json()) as { installed: Record<string, { ip: string; os?: string }> };
+    const machine = data.installed[vmMac];
+    expect(machine).toBeDefined();
+    expect(machine.ip).toMatch(/^\d+\.\d+\.\d+\.\d+$/);
+    expect(machine.os).toBe("vyos-rolling");
+  });
+
+  it("install logs were streamed live (WI-2)", async () => {
+    const res = await fetch(`http://${BASTION_IP}:${HTTP_PORT}/api/logs/${encodeURIComponent(vmMac)}`);
+    const data = (await res.json()) as LogsResponse;
+    expect(data.log_total).toBeGreaterThan(0);
+    const lines = (data.log_lines ?? []).map((l) => l.line).join("\n");
+    // Installer transcript lines and driver messages both flow through /api/log
+    expect(lines).toMatch(/Welcome to VyOS installation|>>> answered|base config:/);
+  });
+
+  it("SSH works as the vyos user with the injected key", () => {
+    const result = sshExec(vmIp, SSH_USER, "whoami", { keyPath: sshKeyPathGlobal });
+    expect(result.exitCode).toBe(0);
+    expect(result.stdout.trim()).toBe("vyos");
+  });
+
+  it("generated config was adopted (hostname + ssh key)", () => {
+    const result = sshExec(vmIp, SSH_USER, "cat /opt/vyatta/etc/config/config.boot", { keyPath: sshKeyPathGlobal });
+    expect(result.exitCode).toBe(0);
+    expect(result.stdout).toContain(`host-name "${HOSTNAME_R1}"`);
+    expect(result.stdout).toContain("public-keys");
+  });
+
+  it("boot-order step ran and reported (WI-3)", async () => {
+    const res = await fetch(`http://${BASTION_IP}:${HTTP_PORT}/api/logs/${encodeURIComponent(vmMac)}`);
+    const data = (await res.json()) as LogsResponse;
+    const lines = (data.log_lines ?? []).map((l) => l.line).join("\n");
+    expect(lines).toContain("boot order:");
+  });
+
+  it("provisioning metadata persisted to /config (WI-4)", () => {
+    const result = sshExec(vmIp, SSH_USER, "cat /config/lab-provisioned 2>/dev/null || cat /opt/vyatta/etc/config/lab-provisioned", { keyPath: sshKeyPathGlobal });
+    expect(result.exitCode).toBe(0);
+    expect(result.stdout).toContain(`hostname=${HOSTNAME_R1}`);
+    expect(result.stdout).toContain("role=vanilla");
+    expect(result.stdout).toContain(`bastion=http://${BASTION_IP}:${HTTP_PORT}`);
+  });
+
+  it("reinstall preserves config and /config data (round 2)", async () => {
+    // Drop a marker in /config — the installer's previous-installation copy
+    // must carry it (and the whole old config) into the new image.
+    // `sync` is REQUIRED: rebootPxeVm uses `virsh destroy` (a hard power-cut),
+    // so an unsynced write never reaches the disk and the marker vanishes for
+    // reasons that have nothing to do with the installer.
+    const marker = sshExec(vmIp, SSH_USER, "echo LAB-MARKER-R2 > /config/lab-marker && sync && cat /config/lab-marker", { keyPath: sshKeyPathGlobal });
+    expect(marker.exitCode).toBe(0);
+    expect(marker.stdout).toContain("LAB-MARKER-R2");
+
+    // Queue with a DIFFERENT hostname: with preserve semantics the previous
+    // config must win, so the hostname must NOT change.
+    vmIp = await installRound({ mac: vmMac, hostname: HOSTNAME_R2 });
+
+    // Assert the config carry-forward first — it is the primary preservation
+    // signal and does not depend on the marker mechanism above.
+    const cfg = sshExec(vmIp, SSH_USER, "cat /opt/vyatta/etc/config/config.boot", { keyPath: sshKeyPathGlobal });
+    expect(cfg.stdout).toContain(`host-name "${HOSTNAME_R1}"`);   // old config carried
+    expect(cfg.stdout).not.toContain(`host-name "${HOSTNAME_R2}"`);
+
+    const markerAfter = sshExec(vmIp, SSH_USER, "cat /config/lab-marker", { keyPath: sshKeyPathGlobal });
+    expect(markerAfter.exitCode).toBe(0);
+    expect(markerAfter.stdout).toContain("LAB-MARKER-R2");
+  }, INSTALL_TIMEOUT_MS + SSH_TIMEOUT_MS + 60_000);
+
+  it("freshConfig makes the generated config win, data still kept (round 3)", async () => {
+    // Re-assert the marker is on disk and synced before the next power-cut.
+    const pre = sshExec(vmIp, SSH_USER, "sync && cat /config/lab-marker", { keyPath: sshKeyPathGlobal });
+    expect(pre.stdout).toContain("LAB-MARKER-R2");
+
+    vmIp = await installRound({ mac: vmMac, hostname: HOSTNAME_R3, freshConfig: true });
+
+    const cfg = sshExec(vmIp, SSH_USER, "cat /opt/vyatta/etc/config/config.boot", { keyPath: sshKeyPathGlobal });
+    expect(cfg.stdout).toContain(`host-name "${HOSTNAME_R3}"`);   // generated config won
+
+    // The marker file (non-config data under /config) still survives —
+    // freshConfig replaces only config.boot, not the carried data.
+    const markerAfter = sshExec(vmIp, SSH_USER, "cat /config/lab-marker", { keyPath: sshKeyPathGlobal });
+    expect(markerAfter.exitCode).toBe(0);
+    expect(markerAfter.stdout).toContain("LAB-MARKER-R2");
+  }, INSTALL_TIMEOUT_MS + SSH_TIMEOUT_MS + 60_000);
+});

labsim/.gitignore

@@ -0,0 +1,8 @@
+# runtime artifacts, not source
+*.log
+labsim_matrix_lib.py
+__pycache__/
+
+# Cluster-admin credentials for the rehearsal cluster, written by
+# `k8s-up.sh --kubeconfig`. Regenerate it rather than commit it.
+*.kubeconfig

labsim/README.md

@@ -0,0 +1,337 @@
+# labsim — libvirt replica of the lab network
+
+A throwaway copy of the production VLAN topology for testing routing, firewall
+rules and failover **without touching the real network**. Same VLAN IDs and
+roles as UniFi, deliberately different IP ranges so nothing can be confused for
+production.
+
+## Topology
+
+Each VLAN is its own isolated libvirt network with one tiny Alpine VM on it.
+
+| VLAN | Name | Sim subnet | VM address | Mirrors production |
+|-----:|------|------------|-----------|--------------------|
+| 1 | management | 172.31.1.0/24 | 172.31.1.10 | 192.168.1.0/24 |
+| 2 | k8s | 172.31.2.0/24 | 172.31.2.10 | 192.168.8.0/23 |
+| 3 | kvm | 172.31.3.0/24 | 172.31.3.10 | 192.168.3.0/24 |
+| 9 | private | 172.31.9.0/24 | 172.31.9.10 | 10.0.9.0/23 |
+| 10 | lot | **172.31.10.0/23** | 172.31.10.10 | 10.0.0.0/23 |
+| 200 | roomates | 172.31.200.0/24 | 172.31.200.10 | 192.168.2.0/24 |
+
+The sim subnet encodes the VLAN id: `172.31.<vlan>.0/24`, with one exception.
+**VLAN 10 is a `/23`** because every UniFi DHCP reservation lives in LoT and LoT
+spans `10.0.0.x` *and* `10.0.1.x`, which a `/24` cannot hold. The mapping stays
+readable — `10.0.0.46 → 172.31.10.46`, `10.0.1.67 → 172.31.11.67`.
+
+LoT's host leg is `.3`, not `.2`, because `10.0.0.2` is a real reservation
+(Hubitat) that maps onto `172.31.10.2`. `.3` is unreserved and sits below the
+DHCP pool, so it can never be handed out.
+
+`vlans.conf` therefore takes two optional trailing fields:
+
+```
+vlan_id:name:sim_prefix:real_subnet[:masklen][:host_octet]
+```
+
+defaulting to `24` and `2`. k8s and Private are also `/23` in production but
+hold no reservations, so they keep their `/24` and their DHCP range is clamped
+— reported at generation time, never silently.
+
+Address plan, identical on every VLAN:
+
+| Address | Role |
+|---------|------|
+| `.1` | gateway under test — a router VM you add (not created by default) |
+| `.2` | host bridge — how you reach the VMs from this workstation |
+| `.10` | the VLAN's micro VM |
+| `.254` | reserved for a VRRP VIP, mirroring production |
+
+The host sits at `.2` purely so you can SSH in. It is deliberately **not** the
+VMs' default route — that is `.1` — so inter-VLAN tests fail loudly when no
+router is present instead of being silently served by the host's own routing
+table. libvirt also installs reject rules that stop these networks forwarding
+to each other, so traffic between VLANs only works once a router VM bridges
+them.
+
+## Usage
+
+```bash
+./labsim-up.sh              # bring up every VLAN (idempotent)
+./labsim-up.sh 2 3          # only VLANs 2 and 3
+./labsim-down.sh            # destroy VMs + networks, keep the base image
+./labsim-down.sh --purge    # also delete the downloaded Alpine image
+```
+
+Each VM: 256 MB, 1 vCPU, a copy-on-write overlay on one shared 176 MB Alpine
+image (so six VMs cost a few MB of disk, not 1 GB).
+
+## Access
+
+```bash
+ssh alpine@172.31.2.10       # normal user (password: labsim)
+ssh root@172.31.2.10         # privileged — this image has no sudo
+curl http://172.31.2.10/     # hello-world page naming the VLAN
+```
+
+Console, when the network is the thing that is broken:
+
+```bash
+sudo virsh console labsim-2-k8s     # root / labsim
+```
+
+## Watching it
+
+```bash
+./labsim-matrix.py --watch 2     # terminal grid, changed cells highlighted
+./monitoring-up.sh               # topology page + Prometheus + Grafana
+```
+
+## Testing the DHCP migration
+
+`./labsim-dhcp-test.sh` boots throwaway VMs whose MACs are **real production
+MACs** and checks each gets the address UniFi reserved for it. MACs are the one
+piece of production config that transplants verbatim, which is what makes this a
+test rather than a rehearsal. It is safe because `ovs-labsim` has no physical
+NIC — verified with `ovs-vsctl show` — so a production MAC cannot reach the real
+LAN.
+
+Apply the config first, from `../migration`:
+
+```bash
+python3 unifi-to-vyos.py --mode sim -o /tmp/sim.conf   # 6 subnets, 31 mappings
+# load onto labsim-vyos, then:
+./labsim-dhcp-test.sh
+```
+
+**Result on VyOS 2026.08 (kea): all four cases pass.** The one that mattered:
+most UniFi reservations sit *inside* the DHCP pool, and **kea honours in-pool
+host reservations** — `printer1` received `172.31.10.46` from within the
+`.10.11–.11.254` pool. That was the open question blocking the cutover.
+
+### The lease database will lie to you
+
+The script wipes `/config/dhcp/dhcp4-leases.csv*` before every run, and both
+halves of that matter:
+
+- **Stale leases defeat reservations.** Re-running against yesterday's leases,
+  kea handed dynamic addresses to three devices that have reservations. The
+  reservation was present and correct in `/run/kea/kea-dhcp4.conf` the whole
+  time. Kea saw the reserved address as already leased to "another client" —
+  same MAC, but a different client-id from the earlier boot — and allocated
+  elsewhere. The cutover itself starts with an empty lease database, so this is
+  a *testing* artifact, but it is worth knowing that a reservation is not an
+  unconditional guarantee once leases exist.
+- **The `*` is load-bearing.** Kea's memfile backend keeps lease-file-cleanup
+  rotations (`dhcp4-leases.csv.2`) and restores from them on start, so
+  truncating only the primary file changes nothing.
+
+Both of those first appeared as a *passing* test. The verdict logic now refuses
+to score a MAC with more than one lease, because taking the first match had
+reported an hours-old lease as the current answer and turned three failures
+into apparent passes.
+
+Still open: whether kea will hand a *reserved* address to a *different* client
+while the reserved device is offline. The negative case here only proves an
+unreserved MAC gets an unreserved address.
+
+- **http://localhost:9101/** — live mesh: a node per VLAN, the router in the
+  middle, one line per pair coloured green/red with the ICMP RTT on it. Hover a
+  line for per-direction detail. Refreshes every 5s. This is the one to watch
+  while changing firewall rules.
+- **http://localhost:3000/d/labsim-matrix** — Grafana (anonymous, no login) for
+  *history*: when did a path flip, and how has latency moved.
+- **http://localhost:9101/metrics** — `labsim_reachable{src,dst,proto}` and
+  `labsim_rtt_ms{src,dst}`.
+
+## WAN follows VRRP, and the PPPoE half of it
+
+One consumer ISP account, two routers. The 10 gig line's lease is bound to a
+cloned MAC and the Vodafone line to a single credential, so neither can be live
+on both boxes: the WAN has to move with mastership.
+
+The two halves use different control planes, and that asymmetry is the design:
+
+| | plane | why |
+|---|---|---|
+| `bond0.53` | VyOS **config** (`disable`) | only config can move a MAC |
+| `pppoe0` | **systemd** unit gate | see below |
+
+`set interfaces pppoe pppoe0 disable` cannot work as a resting state.
+`interfaces_pppoe.py` treats `disable` and `delete` identically and **unlinks
+`/etc/ppp/peers/pppoe0`** — which is pppd's own options file. The resting state
+therefore destroyed what the promotion path needed, and `ppp@pppoe0`
+restart-looped against it (47 restarts, zero sessions at the AC). It also makes
+op-mode `connect interface pppoe0` refuse, and puts every failover behind a
+priority-322 commit where one unrelated bad node fails the lot.
+
+So `pppoe0` is configured identically and **enabled on both**, and dialling is
+gated by a drop-in:
+
+```ini
+ConditionPathExists=/run/vrrp-wan/may-dial
+ConditionPathExists=/etc/ppp/peers/pppoe0
+```
+
+`/run` is tmpfs, so the gate is shut at boot. That matters more than it looks:
+with the node enabled, `interfaces_pppoe.py` restarts ppp on **every** commit
+touching the pppoe subtree when the daemon isn't running — so the backup
+actively tries to dial whenever anything commits. The gate is the only thing
+making that a no-op, which is why the reconciler refuses to bless a box whose
+drop-in is missing (`/etc` is per-image; a VyOS upgrade would silently remove
+the protection).
+
+`may-dial` is a **lease**, not a flag: `ConditionPathExists` is evaluated at
+start only, so it can prevent a dial but never revoke one. `vrrp-wan-reconcile`
+renews it every 30s; `vrrp-wan-guard` runs every 5s and only ever revokes.
+
+### Testing it
+
+```sh
+./labsim-pppoe-ha-test.sh --all
+```
+
+The verdict is what the **routers** and the **access concentrator** did, never
+what a client happened to get — and the harness refuses to run at all while a
+router still has a default route via `eth2`, because the libvirt-NAT scaffold
+answers connectivity checks that the WAN under test would have failed. The
+invariant it enforces throughout: *the AC never reports two `simdsl` sessions,
+and no two routers ever hold `pppoe0`.*
+
+Two failures the harness itself produced, both worth remembering: waiting for
+"exactly one holder" returns instantly during a handover (it was already true),
+and judging connectivity on a single ping 20s after a link drop reports an
+outage that has already healed. Ask **who** holds it, and poll.
+
+Evidence in `wan-failover-evidence/`.
+
+## Routing: BGP, dual WAN, and the ISP VMs
+
+`sim-ha-config.py` covers the LAN side of the routers. `sim-net-config.py`
+covers everything that makes this a rehearsal for production *routing*:
+
+| role | VM | what it generates |
+|---|---|---|
+| `primary` | `labsim-vyos` | BGP + dual WAN + health-checked failover |
+| `secondary` | `labsim-vyos2` | BGP only |
+| `isp-dhcp` | `labsim-isp-dhcp` | 10gig-equivalent ISP on VLAN 53 |
+| `isp-pppoe` | `labsim-isp-pppoe` | Vodafone-equivalent PPPoE ISP on VLAN 51 |
+
+Both ISP VMs are VyOS with two NICs: one on the OVS trunk facing the sim
+router, one on libvirt's `default` network, NATing customers to the real
+internet. They use RFC 5737 documentation ranges (`203.0.113.0/24`,
+`198.51.100.0/24`) so a leaked sim route cannot blackhole anything real.
+
+```sh
+./sim-net-apply.sh check     # VM state vs what the code says — run this first
+./sim-net-apply.sh apply     # push generated config over the serial console
+```
+
+`check` is the important one. All of this previously existed only as running
+state, applied by hand over SSH; rebuilding a VM lost it, and nothing recorded
+why any of it was shaped the way it was.
+
+### Known gaps vs production
+
+- **WAN is on the primary router only.** Production has WAN on both. Two PPPoE
+  clients sharing one credential against a single access concentrator is a
+  failure mode production does not have, so the sim does not model it. VRRP and
+  conntrack failover are still exercised.
+- **ISP VM interface names are not stable across a rebuild** — `isp-dhcp` came
+  up as `eth0`/`eth1` and `isp-pppoe` as `eth2`/`eth3` from identical XML.
+  Check `show interfaces` and pass `--wan-if` / `--uplink-if` rather than
+  trusting the defaults.
+- **`eth2` on the primary router** is a libvirt-NAT uplink predating the ISP
+  VMs: a third default route with no production equivalent that masks real WAN
+  failures during a failover test. `--drop-scaffold` removes it.
+- **Committing on `isp-pppoe` drops the router's PPPoE session**, and the
+  client does not redial promptly. After any change there, check `pppoe0` on
+  the router and `sudo systemctl restart ppp@pppoe0` if it is missing.
+
+## The trunk carries every VLAN tagged, including Management
+
+There is deliberately **no native/untagged VLAN** on the trunks to the routers,
+and Management lives on `bond0.1`, not on the bare `bond0`.
+
+A native VLAN is what puts a subnet on the bond **parent** while every other
+VLAN sits on a sub-interface of it. With `dhcp-socket-type: raw`, kea receives
+each tagged frame *twice* — once on the sub-interface and once on the parent —
+and answers from the parent's pool as well (ISC Kea
+[#1117](https://gitlab.isc.org/isc-projects/kea/-/issues/1117)). A client on
+VLAN 3 gets two OFFERs and keeps whichever arrives first:
+
+```
+bond0.3 :  172.31.3.252 → 172.31.3.11   correct
+bond0   :  172.31.1.252 → 172.31.1.8    UNTAGGED, Management pool, wrong
+```
+
+`./labsim-vlan-leak-test.sh` makes one client on a tagged VLAN send a DISCOVER
+and captures on the parent and the sub-interface at once. The verdict is how
+many OFFERs the **server** emitted and from which subnets — deliberately not
+"did the client get the right address", because a client picking correctly is
+exactly how this hid. Both orderings were observed across runs, so a passing
+client proves nothing.
+
+```sh
+./labsim-vlan-leak-test.sh --vlan 3           # PASS on the current shape
+LABSIM_NATIVE_VLAN=1 ./router-up.sh           # restore the old shape...
+./labsim-vlan-leak-test.sh --vlan 3           # ...and it FAILs again
+```
+
+Three things this cost, all of which apply to production:
+
+- **Kea must be restarted after the address moves.** VyOS does not restart it
+  for an interface address change, so it keeps a raw socket bound with the old
+  address and the bug survives the fix. In the sim kea had been running since
+  16 Aug; the first post-fix test failed for this reason alone and looked like
+  the fix simply not working.
+- **The firewall interface-group must move too.** `interface-group LAN` named
+  the bare `bond0`; with a default-deny ruleset, moving the address without
+  moving the group drops every management session and all VLAN 1 routing.
+- **Duplicate delivery does not stop.** #1117 says only that there is no longer
+  a subnet on the parent to match, and that is exactly what happens: two replies
+  per DISCOVER, both now from the correct pool. Harmless, but do not read a
+  duplicate as a failure.
+
+### Tagged and untagged Management coexist
+
+Verified directly, and it is what makes the production cutover a rolling change
+rather than an outage: with the primary still untagged on `bond0` and the
+secondary already tagged on `bond0.1`, both routers were reachable, the VIP
+stayed up and a VLAN 1 client kept its gateway. One VLAN is one broadcast
+domain regardless of how each port tags it, so the two firewalls can be
+converted one at a time. See `migration/MANAGEMENT-VLAN-TAGGED.md`.
+
+`./vlan1-move-monitor.sh` logs VIP/router liveness once a second during the
+change, because VRRP reconverges and leaves no trace of who held the VIP.
+
+## Notes for whoever extends this
+
+Things that cost time the first time round, all verified on this image:
+
+- **No `sudo`.** Alpine ships `doas`; cloud-init's `sudo:` directive is inert
+  here. Use `root@` for privileged work.
+- **cloud-init leaves users locked** (`!*` in `/etc/shadow`) unless
+  `lock_passwd: false`, and sshd then refuses key auth for that user.
+- **One failing `runcmd` aborts every command after it.** Each entry is
+  `|| true` for that reason.
+- **busybox here has no `httpd` applet**, and the VMs have no internet to
+  `apk add` one — so the hello-world server is `python3 -m http.server`
+  (python3 is already present because cloud-init depends on it).
+- **`start-stop-daemon --exec /usr/bin/python3` matches cloud-init's own
+  python3** at boot and refuses to start anything.
+- **busybox `pgrep -f PATTERN` matches its own argv**, so a "skip if already
+  running" guard always fires. Verified: `guard_exit=0` with nothing listening.
+
+## Not modelled (yet)
+
+- **The secondary's bond was fiction until 2026-09-02.** `ovs_bond_router`'s
+  "already bonded, nothing to do" check compared only the trunk VLAN list, not
+  the membership. Restarting a VM recreates its taps under new names, so the
+  bond sat there holding two interfaces that no longer existed while the router's
+  real taps ran in the bridge as two *independent* ports — no LACP, and carrying
+  libvirt's own portgroup VLAN config rather than the bond's. It reconciles
+  membership now, but the lesson generalises: a sim that reports success is not
+  the same as a sim that models the thing.
+- **`labsim-vyos` has a third NIC** on libvirt's `default` network (the scaffold
+  uplink, see `--drop-scaffold`). The tap count is filtered to `$OVS_NET` for
+  that reason; an unfiltered count is 3 and silently skipped the primary's bond.

labsim/cilium-ipam-switch.sh

@@ -0,0 +1,144 @@
+#!/usr/bin/env bash
+# Procedure around a Cilium IPAM mode change. Works against any cluster, so the
+# rehearsal in labsim and the real thing in production run the SAME steps.
+#
+# It deliberately does NOT change the mode itself. In labsim that is `helm
+# upgrade`; in production Pulumi owns the release and a script racing it would
+# just reintroduce drift. What this owns is everything around the apply -- the
+# evidence, the deadlock, and the verdict.
+#
+#   ./cilium-ipam-switch.sh preflight   record what the cluster looks like now
+#   ./cilium-ipam-switch.sh unstick     break the agent-not-ready taint deadlock
+#   ./cilium-ipam-switch.sh verify      compare against preflight, report renumbering
+#
+#   KUBECONFIG=... ./cilium-ipam-switch.sh preflight
+#
+# Whether a recycle is needed is CONDITIONAL, and `verify` is what decides it.
+#
+# The operator does not preserve which node held which /24 -- it adopts whatever
+# CiliumNode.spec.ipam.podCIDRs already says. So:
+#
+#   * If CiliumNode already agrees with node.spec.podCIDRs on every node -- which
+#     is the case for any cluster that has only ever run ipam=kubernetes, because
+#     the operator syncs one from the other -- the pool adopts the existing
+#     allocation, no node is renumbered, and NO pod recycle is needed. Verified
+#     on the 3-node labsim cluster: CIDRs unchanged, nothing stranded, the only
+#     blip was the cilium DaemonSet restarting itself.
+#
+#   * If the two sources DISAGREE, nodes can swap /24s. Their running pods keep
+#     addresses that no longer fall inside the node's range, every other node
+#     routes that prefix to the wrong node, and those pods go unreachable
+#     cross-node while still showing Running. Then a full recycle is mandatory.
+#
+# Do not skip `verify` on the assumption of the good case. Run it and read it.
+set -uo pipefail
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+STATE="${STATE:-$SCRIPT_DIR/.ipam-switch-state}"
+K="kubectl"
+
+say()  { printf '\033[0;36m[ipam]\033[0m %s\n' "$*"; }
+warn() { printf '\033[1;33m[ipam]\033[0m %s\n' "$*" >&2; }
+
+snapshot() {
+  echo "## nodes"
+  $K get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.podCIDRs}{"\n"}{end}' 2>/dev/null
+  echo "## ciliumnodes"
+  $K get ciliumnode -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.ipam.podCIDRs}{"\n"}{end}' 2>/dev/null
+  echo "## pods"
+  $K get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}{"\t"}{.status.podIP}{"\n"}{end}' 2>/dev/null \
+    | grep -vP '\t$' | sort
+  echo "## ipam"
+  $K -n kube-system get cm cilium-config -o jsonpath='{.data.ipam}' 2>/dev/null; echo
+}
+
+cmd_preflight() {
+  mkdir -p "$STATE"
+  snapshot > "$STATE/before.txt"
+  say "recorded $(grep -c . "$STATE/before.txt") lines -> $STATE/before.txt"
+  say "mode now: $(sed -n '/^## ipam/,$p' "$STATE/before.txt" | tail -1)"
+  # The pod inventory is the rollback reference: if the switch renumbers, this
+  # is the only record of what an address USED to be.
+  say "pods on the pod network: $(sed -n '/^## pods/,/^## ipam/p' "$STATE/before.txt" | grep -c '10\.')"
+}
+
+# The deadlock, in one place because it WILL happen and doing it by hand under
+# time pressure is how the wrong node gets untainted:
+#   agent has no pod CIDR -> agent not ready -> node keeps
+#   node.cilium.io/agent-not-ready:NoSchedule -> the operator that would assign
+#   the CIDR cannot schedule -> agent still has no pod CIDR.
+# Removing the taint is safe: it exists to keep normal workloads off a node
+# without working networking, and the operator is precisely the thing that fixes
+# that. Kubernetes re-adds it on the next agent restart.
+cmd_unstick() {
+  local stuck=0
+  for n in $($K get nodes -o name 2>/dev/null); do
+    $K get "$n" -o jsonpath='{.spec.taints[*].key}' 2>/dev/null | grep -q 'agent-not-ready' || continue
+    warn "${n#node/} carries agent-not-ready; removing so the operator can schedule"
+    $K taint "$n" node.cilium.io/agent-not-ready- >/dev/null 2>&1 && stuck=$((stuck+1))
+  done
+  [ "$stuck" -eq 0 ] && say "no node was stuck" || say "cleared $stuck node(s)"
+  local pend
+  pend="$($K -n kube-system get pods -l io.cilium/app=operator --no-headers 2>/dev/null | grep -c Pending)"
+  [ "${pend:-0}" -gt 0 ] && warn "$pend operator pod(s) still Pending — check tolerations, not just taints"
+  return 0
+}
+
+cmd_verify() {
+  [ -f "$STATE/before.txt" ] || { warn "no preflight snapshot; nothing to compare"; return 1; }
+  snapshot > "$STATE/after.txt"
+  echo
+  say "mode: $(sed -n '/^## ipam/,$p' "$STATE/before.txt" | tail -1) -> $(sed -n '/^## ipam/,$p' "$STATE/after.txt" | tail -1)"
+
+  # The question that decides the size of the maintenance window: did per-node
+  # CIDRs survive, or was every node renumbered (and every pod with it)?
+  local moved=0
+  while IFS=$'\t' read -r node cidr; do
+    [ -z "${node:-}" ] && continue
+    local now; now="$(sed -n '/^## ciliumnodes/,/^## pods/p' "$STATE/after.txt" | awk -F'\t' -v n="$node" '$1==n{print $2}')"
+    if [ -n "$now" ] && [ "$now" != "$cidr" ]; then
+      printf '    %-16s %s -> %s\n' "$node" "$cidr" "$now"; moved=$((moved+1))
+    fi
+  done < <(sed -n '/^## ciliumnodes/,/^## pods/p' "$STATE/before.txt" | grep -P '\t')
+  if [ "$moved" -eq 0 ]; then
+    say "per-node CIDRs UNCHANGED — the pool adopted the existing allocation"
+  else
+    warn "$moved node(s) renumbered — every pod on them must be recycled"
+  fi
+
+  local before after same
+  before="$(sed -n '/^## pods/,/^## ipam/p' "$STATE/before.txt" | grep -P '\t10\.' | wc -l)"
+  after="$(sed -n '/^## pods/,/^## ipam/p' "$STATE/after.txt"  | grep -P '\t10\.' | wc -l)"
+  same="$(comm -12 <(sed -n '/^## pods/,/^## ipam/p' "$STATE/before.txt" | grep -P '\t10\.' | sort) \
+                   <(sed -n '/^## pods/,/^## ipam/p' "$STATE/after.txt"  | grep -P '\t10\.' | sort) | wc -l)"
+  say "pods: $before before, $after after, $same kept the SAME address"
+  # Keeping the address is NOT the good outcome. If a node's CIDR moved, its
+  # existing pods keep IPs that no longer fall inside it, every other node routes
+  # that prefix to the WRONG node, and those pods go unreachable cross-node while
+  # looking perfectly healthy. Observed in labsim: two nodes swapped CIDRs and
+  # cross-node ping to their pods dropped 100%, with every pod still Running.
+  # This is the check that decides whether a recycle is optional or mandatory.
+  local stranded=0
+  while read -r ns name ip node; do
+    [ -z "${node:-}" ] && continue
+    local cidr; cidr="$($K get ciliumnode "$node" -o jsonpath='{.spec.ipam.podCIDRs[0]}' 2>/dev/null)"
+    [ -z "$cidr" ] && continue
+    case "$ip" in
+      "${cidr%.*/*}".*) ;;
+      *) printf '    STRANDED %-40s %-15s on %s (now %s)\n' "$ns/$name" "$ip" "$node" "$cidr"; stranded=$((stranded+1)) ;;
+    esac
+  done < <($K get pods -A -o jsonpath='{range .items[?(@.status.podIP)]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.status.podIP}{" "}{.spec.nodeName}{"\n"}{end}' 2>/dev/null | grep -E ' 10\.')
+  if [ "$stranded" -gt 0 ]; then
+    warn "$stranded pod(s) sit OUTSIDE their node CIDR — unreachable cross-node until recycled"
+    warn "recycle: for ns in $(kubectl get ns -o name | cut -d/ -f2); do kubectl -n $ns rollout restart deploy,ds,sts 2>/dev/null; done"
+  else
+    say "every pod is inside its node CIDR — no recycle needed"
+  fi
+  say "not-Running pods: $($K get pods -A --no-headers 2>/dev/null | grep -vcE 'Running|Completed')"
+}
+
+case "${1:-}" in
+  preflight) cmd_preflight ;;
+  unstick)   cmd_unstick ;;
+  verify)    cmd_verify ;;
+  *) sed -n '2,16p' "$0"; exit 1 ;;
+esac

labsim/console-apply.py

@@ -0,0 +1,116 @@
+#!/usr/bin/env python3
+"""Apply VyOS config to a labsim VM over its serial console.
+
+Needed because a freshly installed VyOS comes up holding the same addresses as
+its peer, so there is a window where it cannot safely be reached over the
+network at all. The console does not care.
+
+  ./console-apply.py --vm labsim-vyos2 --config r2.conf
+"""
+from __future__ import annotations
+
+import argparse
+import sys
+import time
+
+import pexpect
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--vm", required=True)
+    ap.add_argument("--config", required=True)
+    ap.add_argument("--user", default="vyos")
+    ap.add_argument("--password", default="vyos")
+    # `save` writes config.boot. For WAN work that is dangerous: the resting
+    # state must stay `vif 53 disable` on both routers, and saving while a box
+    # is master persists the ENABLED state -- so a reboot would have it claim
+    # the cloned MAC. Observed in labsim on 2026-09-02.
+    ap.add_argument("--no-save", action="store_true",
+                    help="commit without saving (leave config.boot untouched)")
+    args = ap.parse_args()
+
+    cmds = [l.rstrip() for l in open(args.config)
+            if l.strip() and not l.lstrip().startswith("#")]
+    print(f"{len(cmds)} commands to apply to {args.vm}", file=sys.stderr)
+
+    c = pexpect.spawn(f"virsh --connect qemu:///system console {args.vm}",
+                      timeout=90, encoding="utf-8")
+    c.logfile_read = None
+    c.sendline("")
+    time.sleep(2)
+    c.sendline("")
+
+    # Log in. A freshly booted box may still be starting services, so allow a
+    # generous window and re-prod the console rather than failing on the first
+    # miss.
+    #
+    # `# ` matters as much as `$ `: a previous run that died mid-config leaves
+    # the console sitting in configuration mode, and waiting only for the
+    # operational prompt then hangs forever against a perfectly healthy VM.
+    in_config = False
+    for _ in range(40):
+        i = c.expect([r"login:", r"\$ ", r"# ", pexpect.TIMEOUT], timeout=15)
+        if i == 0:
+            c.sendline(args.user)
+            c.expect("Password:", timeout=30)
+            c.sendline(args.password)
+            c.expect([r"\$ ", r"# "], timeout=60)
+            break
+        if i == 1:
+            break
+        if i == 2:
+            in_config = True
+            break
+        c.sendline("")
+    else:
+        print("never reached a prompt", file=sys.stderr)
+        return 1
+
+    if in_config:
+        # Drop whatever the previous run left half-built rather than committing
+        # a candidate nobody has seen.
+        print("console was left in config mode; discarding stale candidate",
+              file=sys.stderr)
+        c.sendline("discard")
+        c.expect(r"# ", timeout=60)
+    else:
+        c.sendline("configure")
+        c.expect(r"# ", timeout=60)
+
+    for cmd in cmds:
+        c.sendline(cmd)
+        c.expect(r"# ", timeout=60)
+        out = c.before or ""
+        if "Set failed" in out or "not valid" in out or "Invalid" in out:
+            print(f"FAILED: {cmd}\n  {out.strip()[:200]}", file=sys.stderr)
+
+    print("committing...", file=sys.stderr)
+    c.sendline("commit")
+    c.expect(r"# ", timeout=300)
+    commit_out = c.before or ""
+    if not args.no_save:
+        c.sendline("save")
+        c.expect(r"# ", timeout=120)
+    # Accept either prompt on the way out. Insisting on `$ ` here hangs against
+    # a healthy box -- and worse, leaves the console parked in config mode, so
+    # the NEXT run finds a `# ` it was not expecting either. One strict expect
+    # turned into two failures.
+    c.sendline("exit")
+    c.expect([r"\$ ", r"# ", pexpect.TIMEOUT], timeout=60)
+    c.sendline("exit")
+    c.close(force=True)
+
+    bad = [l for l in commit_out.splitlines()
+           if "failed" in l.lower() or "error" in l.lower()]
+    if bad:
+        print("commit reported:", file=sys.stderr)
+        for l in bad[:10]:
+            print(f"  {l.strip()}", file=sys.stderr)
+        return 1
+    print("committed and saved", file=sys.stderr)
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

labsim/dualstack-lab.sh

@@ -0,0 +1,318 @@
+#!/usr/bin/env bash
+# Differential study: what ACTUALLY differs between a k3s cluster born
+# dual-stack and one converted in place?
+#
+# k3s says dual-stack "cannot be enabled on an existing cluster". The stated
+# reason is narrow -- nodes get Pod CIDRs only at join and the Kubernetes IPAM
+# controller will not hand out a new IPv6 CIDR later -- and it does not obviously
+# apply to a cluster where Cilium owns IPAM. Rather than argue from docs, build
+# both shapes and diff them.
+#
+#   ./dualstack-lab.sh up v4         single-node k3s, IPv4 only     (.21)
+#   ./dualstack-lab.sh up dual       single-node k3s, dual-stack    (.22)
+#   ./dualstack-lab.sh pristine v4   reflink copy of v4's disk, so the upgrade
+#                                    attempt can be rolled back and retried
+#   ./dualstack-lab.sh restore v4    put that copy back
+#   ./dualstack-lab.sh collect <n>   normalized state dump -> evidence/<n>/
+#   ./dualstack-lab.sh compare a b   semantic diff of two collections
+#   ./dualstack-lab.sh virtdiff a b  whole-filesystem diff, offline (libguestfs)
+#   ./dualstack-lab.sh down [name]
+#
+# The comparison that matters is `compare dual upgraded`: everything it prints
+# is a way the converted cluster failed to reach the shape of a native one.
+#
+# Single node on purpose. Dual-stack is decided by server flags and CNI config,
+# both of which a one-node cluster exercises fully, and it rebuilds in minutes.
+# Node-rejoin behaviour needs the 3-node cluster and is a separate question.
+set -euo pipefail
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+source "$SCRIPT_DIR/lib.sh"
+source "$SCRIPT_DIR/ovs.sh"
+
+K8S_VLAN="${K8S_VLAN:-2}"
+DS_PREFIX="${DS_PREFIX:-172.31.2}"
+MEM="${MEM:-4096}"; CPUS="${CPUS:-2}"; DISK_GB="${DISK_GB:-12}"
+TOKEN="${TOKEN:-labsim-ds-token}"
+CILIUM_VERSION="${CILIUM_VERSION:-1.19.1}"   # same as production
+DEB_BASE="${DEB_BASE:-$IMG_DIR/debian-13-genericcloud-amd64.qcow2}"
+EVIDENCE="$SCRIPT_DIR/dualstack-evidence"
+
+# Pod/Service ranges. IPv4 halves are k3s's own defaults, so the v4-only build is
+# a stock cluster and the diff is not polluted by gratuitous differences.
+# IPv6 halves are ULA: this cluster never routes off-box, and using the real /48
+# here would put lab addresses into a prefix that production also announces.
+V4_CLUSTER="10.42.0.0/16"; V4_SERVICE="10.43.0.0/16"
+V6_CLUSTER="${V6_CLUSTER:-fd00:42::/56}"
+V6_SERVICE="${V6_SERVICE:-fd00:43::/112}"   # /112 -- apiserver caps v6 service ranges
+V6_PREFIX="${V6_PREFIX:-fd00:2}"            # node addresses: fd00:2::<octet>
+
+vm_name() { echo "labsim-ds-$1"; }
+vm_ip()   { case "$1" in v4) echo "$DS_PREFIX.21";; dual) echo "$DS_PREFIX.22";; *) die "unknown build '$1'";; esac; }
+vm_ip6()  { case "$1" in v4) echo "$V6_PREFIX::21";; dual) echo "$V6_PREFIX::22";; *) die "unknown build '$1'";; esac; }
+disk_of() { echo "$IMG_DIR/$(vm_name "$1").qcow2"; }
+
+ssh_vm() { local ip="$1"; shift; ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
+             -o LogLevel=ERROR -o ConnectTimeout=8 -o BatchMode=yes "debian@$ip" "$@"; }
+
+# --- seed -----------------------------------------------------------------
+build_seed() {
+  local iso="$1" vm="$2" mode="$3" pubkey="$4"
+  local ip ip6 tmp; ip="$(vm_ip "$mode")"; ip6="$(vm_ip6 "$mode")"; tmp="$(mktemp -d)"
+
+  echo "instance-id: $vm" > "$tmp/meta-data"
+  # Static v6 on both builds. The v4-only cluster still gets an IPv6 ADDRESS --
+  # only its Kubernetes config is v4-only. Otherwise the diff would be dominated
+  # by host addressing rather than by what Kubernetes did differently.
+  cat > "$tmp/network-config" <<EOF
+version: 2
+ethernets:
+  enp1s0:
+    addresses: [${ip}/24, ${ip6}/64]
+    routes:
+      - to: default
+        via: ${DS_PREFIX}.1
+    nameservers:
+      addresses: [8.8.8.8, 1.1.1.1]
+EOF
+
+  local exec_args="server --flannel-backend=none --disable-network-policy --disable=servicelb --disable=traefik --tls-san=$ip --cluster-init"
+  if [ "$mode" = dual ]; then
+    exec_args="$exec_args --cluster-cidr=${V4_CLUSTER},${V6_CLUSTER} --service-cidr=${V4_SERVICE},${V6_SERVICE} --node-ip=${ip},${ip6}"
+  else
+    exec_args="$exec_args --node-ip=${ip}"
+  fi
+
+  cat > "$tmp/user-data" <<EOF
+#cloud-config
+hostname: $vm
+fqdn: $vm
+users:
+  - name: debian
+    groups: [sudo]
+    shell: /bin/bash
+    sudo: ["ALL=(ALL) NOPASSWD:ALL"]
+    lock_passwd: false
+    plain_text_passwd: labsim
+    ssh_authorized_keys: [$pubkey]
+ssh_pwauth: true
+disable_root: false
+package_update: true
+packages: [curl, jq, iproute2, nftables]
+write_files:
+  - path: /etc/modules-load.d/cilium.conf
+    content: |
+      br_netfilter
+  - path: /etc/dualstack-lab-mode
+    content: |
+      $mode
+runcmd:
+  - modprobe br_netfilter || true
+  - |
+    curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="$exec_args" K3S_TOKEN="$TOKEN" sh -
+  - |
+    # Cilium via helm, matching the production version. IPAM stays 'kubernetes'
+    # in BOTH builds on purpose: that is what production runs, and it is the
+    # mode the k3s objection is actually about. If the converted cluster needs
+    # cluster-pool to work, the diff should be what tells us so.
+    curl -sfL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash || true
+    export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
+    helm repo add cilium https://helm.cilium.io >/dev/null 2>&1 || true
+    helm repo update >/dev/null 2>&1 || true
+    for i in \$(seq 1 60); do kubectl get nodes >/dev/null 2>&1 && break; sleep 5; done
+    if [ "$mode" = dual ]; then
+      helm install cilium cilium/cilium --version $CILIUM_VERSION -n kube-system \\
+        --set kubeProxyReplacement=false --set ipam.mode=kubernetes \\
+        --set ipv4.enabled=true --set ipv6.enabled=true \\
+        --set k8sServiceHost=$ip --set k8sServicePort=6443 || true
+    else
+      helm install cilium cilium/cilium --version $CILIUM_VERSION -n kube-system \\
+        --set kubeProxyReplacement=false --set ipam.mode=kubernetes \\
+        --set ipv4.enabled=true --set ipv6.enabled=false \\
+        --set k8sServiceHost=$ip --set k8sServicePort=6443 || true
+    fi
+    touch /etc/dualstack-lab-ready
+EOF
+  sudo mkdir -p "$(dirname "$iso")"
+  sudo genisoimage -quiet -output "$iso" -volid cidata -joliet -rock \
+    "$tmp/user-data" "$tmp/meta-data" "$tmp/network-config"
+  rm -rf "$tmp"
+}
+
+cmd_up() {
+  local mode="${1:?usage: up <v4|dual>}"
+  local vm ip disk seed pubkey
+  vm="$(vm_name "$mode")"; ip="$(vm_ip "$mode")"; disk="$(disk_of "$mode")"
+  seed="$IMG_DIR/${vm}-seed.iso"; pubkey="$(find_ssh_pubkey)"
+
+  [ -f "$DEB_BASE" ] || die "base image missing: $DEB_BASE (run ./k8s-up.sh once)"
+  if virsh_q dominfo "$vm" >/dev/null 2>&1; then
+    log "$vm exists — starting if stopped"
+    [ "$(virsh_q domstate "$vm" | head -1)" = "running" ] || virsh_q start "$vm" >/dev/null
+    return
+  fi
+  selected_vlans; ovs_up
+  log "creating $vm ($mode) at $ip / $(vm_ip6 "$mode")"
+  sudo qemu-img create -q -f qcow2 -F qcow2 -b "$DEB_BASE" "$disk" "${DISK_GB}G" >/dev/null
+  build_seed "$seed" "$vm" "$mode" "$pubkey"
+  sudo virt-install --connect "$LIBVIRT_URI" --name "$vm" \
+    --memory "$MEM" --vcpus "$CPUS" \
+    --disk "path=$disk,format=qcow2,bus=virtio" \
+    --disk "path=$seed,device=cdrom" \
+    --network "network=$OVS_NET,portgroup=vlan${K8S_VLAN},model=virtio" \
+    --os-variant debian12 --graphics none --noautoconsole --import >/dev/null
+  log "installing in background; watch: ssh debian@$ip 'ls /etc/dualstack-lab-ready'"
+}
+
+# --- pristine copy / restore ---------------------------------------------
+# reflink so the copy is instant and independent on btrfs/xfs. A qcow2 backing
+# chain would be cheaper still but makes the parent read-only in practice: boot
+# the parent again and every child silently corrupts.
+cmd_pristine() {
+  local mode="${1:?usage: pristine <v4|dual>}" vm disk
+  vm="$(vm_name "$mode")"; disk="$(disk_of "$mode")"
+  [ "$(virsh_q domstate "$vm" 2>/dev/null | head -1)" = "running" ] && \
+    die "$vm is running — shut it down first (virsh shutdown $vm), a copy of a live disk is not consistent"
+  sudo cp --reflink=auto "$disk" "${disk}.pristine"
+  log "pristine copy: ${disk}.pristine"
+}
+cmd_restore() {
+  local mode="${1:?usage: restore <v4|dual>}" vm disk
+  vm="$(vm_name "$mode")"; disk="$(disk_of "$mode")"
+  [ -f "${disk}.pristine" ] || die "no pristine copy for $mode"
+  [ "$(virsh_q domstate "$vm" 2>/dev/null | head -1)" = "running" ] && \
+    die "$vm is running — shut it down first"
+  sudo cp --reflink=auto "${disk}.pristine" "$disk"
+  log "restored $mode from pristine"
+}
+
+
+# --- the experiment ------------------------------------------------------
+# Convert the IPv4-only cluster in place, mirroring the flags the native build
+# was BORN with. Each step prints what the cluster did, because the interesting
+# output is which step refuses rather than whether the end state is pretty.
+cmd_upgrade() {
+  local ip; ip="$(vm_ip v4)"; local ip6; ip6="$(vm_ip6 v4)"
+  log "step 1/4: add dual CIDRs + dual node-ip to the k3s unit"
+  # Done with python on the box, not nested sed: quoting a multi-line systemd
+  # continuation through ssh -> sh -> sed produced a literal \\n in the unit, and
+  # k3s then saw a dual cluster-cidr with a still-IPv4 service-cidr and refused
+  # to start. All three flags go on one line -- systemd does not care, and there
+  # is nothing left to escape.
+  ssh_vm "$ip" "sudo python3 - <<'PYEOF'
+import re
+u = '/etc/systemd/system/k3s.service'
+s = open(u).read()
+old = \"'--node-ip=${ip}'\"
+new = \"'--cluster-cidr=${V4_CLUSTER},${V6_CLUSTER}' '--service-cidr=${V4_SERVICE},${V6_SERVICE}' '--node-ip=${ip},${ip6}'\"
+assert old in s, 'node-ip flag not found in unit'
+open(u,'w').write(s.replace(old, new))
+print('    unit rewritten')
+PYEOF
+sudo systemctl daemon-reload" || die "unit edit failed"
+  ssh_vm "$ip" "grep -oE \"'--(cluster|service)-cidr=[^']*'|'--node-ip=[^']*'\" /etc/systemd/system/k3s.service | sed 's/^/    /'"
+
+  log "step 2/4: restart k3s and see whether it accepts the changed ranges"
+  ssh_vm "$ip" "sudo systemctl restart k3s" || true
+  for i in $(seq 1 40); do
+    ssh_vm "$ip" "sudo k3s kubectl get --raw /readyz >/dev/null 2>&1" && break
+    sleep 5
+  done
+  ssh_vm "$ip" "sudo journalctl -u k3s --since '2 min ago' --no-pager 2>/dev/null | grep -iE 'cidr|dual|ipv6|invalid|cannot|fail' | tail -12 | sed 's/^/    /'" || true
+
+  log "step 3/4: what the API says now"
+  ssh_vm "$ip" "echo -n '    servicecidr: '; sudo k3s kubectl get servicecidr -o jsonpath='{.items[*].spec.cidrs}'; echo; \
+                echo -n '    node podCIDRs: '; sudo k3s kubectl get node -o jsonpath='{.items[0].spec.podCIDRs}'; echo; \
+                echo -n '    node addresses: '; sudo k3s kubectl get node -o jsonpath='{.items[0].status.addresses[*].address}'; echo" || true
+
+  log "step 4/4: turn on IPv6 in Cilium"
+  ssh_vm "$ip" "export KUBECONFIG=/etc/rancher/k3s/k3s.yaml; sudo -E helm upgrade cilium cilium/cilium --version ${CILIUM_VERSION} -n kube-system --reuse-values --set ipv6.enabled=true >/dev/null 2>&1 && echo '    cilium upgraded' || echo '    cilium upgrade FAILED'" || true
+  ssh_vm "$ip" "sudo k3s kubectl -n kube-system rollout restart ds/cilium >/dev/null 2>&1; sleep 20; sudo k3s kubectl -n kube-system get pods -l k8s-app=cilium --no-headers | sed 's/^/    /'" || true
+  log "now: ./dualstack-lab.sh collect upgraded ${ip} && ./dualstack-lab.sh compare dual upgraded"
+}
+
+# --- evidence collection --------------------------------------------------
+# Normalized on purpose. Two independently built clusters differ in certs,
+# tokens, UUIDs, timestamps and log lines; left raw, that noise buries the
+# handful of differences that actually mean something.
+cmd_collect() {
+  local name="${1:?usage: collect <name> [ip]}"
+  local ip="${2:-}"
+  [ -n "$ip" ] || ip="$(vm_ip "$name" 2>/dev/null || true)"
+  [ -n "$ip" ] || die "collect: give an ip for a non-standard name"
+  local out="$EVIDENCE/$name"; mkdir -p "$out"
+  log "collecting from $name ($ip) -> $out"
+
+  ssh_vm "$ip" 'sudo cat /etc/rancher/k3s/config.yaml 2>/dev/null; sudo systemctl cat k3s 2>/dev/null | grep -A30 ExecStart' \
+    > "$out/k3s-config.txt" 2>/dev/null || true
+  ssh_vm "$ip" 'sudo tr "\0" "\n" < /proc/$(pgrep -f "k3s server" | head -1)/cmdline | grep -v "^$"' \
+    > "$out/k3s-cmdline.txt" 2>/dev/null || true
+  ssh_vm "$ip" 'ip -o addr show | awk "{print \$2, \$3, \$4}"; echo ---; ip -4 route show; echo ---; ip -6 route show' \
+    > "$out/host-net.txt" 2>/dev/null || true
+  ssh_vm "$ip" 'sudo sysctl -a 2>/dev/null | grep -E "net\.ipv6\.conf\.(all|default)\.(forwarding|disable_ipv6)|net\.ipv4\.ip_forward"' \
+    > "$out/sysctl.txt" 2>/dev/null || true
+
+  local K='sudo k3s kubectl'
+  ssh_vm "$ip" "$K get servicecidr -o yaml"       > "$out/servicecidr.yaml" 2>/dev/null || true
+  ssh_vm "$ip" "$K get nodes -o yaml"             > "$out/nodes.yaml.raw"   2>/dev/null || true
+  ssh_vm "$ip" "$K get ciliumnodes -o yaml"       > "$out/ciliumnodes.yaml.raw" 2>/dev/null || true
+  ssh_vm "$ip" "$K -n kube-system get cm cilium-config -o yaml" > "$out/cilium-config.yaml.raw" 2>/dev/null || true
+  ssh_vm "$ip" "$K get svc -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,FAMILYPOLICY:.spec.ipFamilyPolicy,FAMILIES:.spec.ipFamilies,IPS:.spec.clusterIPs" \
+    > "$out/services.txt" 2>/dev/null || true
+  ssh_vm "$ip" "$K get pods -A -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,IPS:.status.podIPs" \
+    > "$out/podips.txt" 2>/dev/null || true
+
+  # Strip the things that differ every build regardless of configuration.
+  for f in "$out"/*.raw; do
+    [ -e "$f" ] || continue
+    sed -E \
+      -e 's/[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9:]+Z?/<TIME>/g' \
+      -e 's/(uid|resourceVersion|creationTimestamp|generation|observedGeneration): .*/\1: <X>/' \
+      -e 's/[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/<UUID>/g' \
+      -e 's/(LS0tLS1|[A-Za-z0-9+\/]{60,}=*)/<B64>/g' \
+      "$f" > "${f%.raw}"
+    rm -f "$f"
+  done
+  log "collected $(ls "$out" | wc -l) artefacts"
+}
+
+cmd_compare() {
+  local a="${1:?usage: compare <a> <b>}" b="${2:?}"
+  [ -d "$EVIDENCE/$a" ] && [ -d "$EVIDENCE/$b" ] || die "collect both first"
+  echo "### semantic diff: $a (<) vs $b (>)"
+  diff -ru "$EVIDENCE/$a" "$EVIDENCE/$b" || true
+}
+
+cmd_virtdiff() {
+  local a="${1:?usage: virtdiff <a> <b>}" b="${2:?}"
+  for m in "$a" "$b"; do
+    [ "$(virsh_q domstate "$(vm_name "$m")" 2>/dev/null | head -1)" = "running" ] && \
+      die "$(vm_name "$m") is running — virt-diff needs the disks quiescent"
+  done
+  log "whole-filesystem diff (slow); noise is expected — use it to find what the collector missed"
+  sudo virt-diff -a "$(disk_of "$a")" -A "$(disk_of "$b")" \
+    | grep -vE '/(var/log|tmp|run|proc|sys)/|\.log$|/var/lib/rancher/k3s/(server/(tls|cred|db)|agent)' || true
+}
+
+cmd_down() {
+  local only="${1:-}"
+  for m in v4 dual; do
+    [ -n "$only" ] && [ "$only" != "$m" ] && continue
+    local vm; vm="$(vm_name "$m")"
+    virsh_q dominfo "$vm" >/dev/null 2>&1 || continue
+    log "removing $vm"
+    virsh_q destroy "$vm" >/dev/null 2>&1 || true
+    virsh_q undefine "$vm" --remove-all-storage >/dev/null 2>&1 || true
+  done
+}
+
+case "${1:-}" in
+  up)       shift; cmd_up "$@" ;;
+  upgrade)  shift; cmd_upgrade "$@" ;;
+  pristine) shift; cmd_pristine "$@" ;;
+  restore)  shift; cmd_restore "$@" ;;
+  collect)  shift; cmd_collect "$@" ;;
+  compare)  shift; cmd_compare "$@" ;;
+  virtdiff) shift; cmd_virtdiff "$@" ;;
+  down)     shift; cmd_down "$@" ;;
+  *) sed -n '2,30p' "$0"; exit 1 ;;
+esac

labsim/ipv6-ha-evidence/V0-baseline/state.txt

@@ -0,0 +1,43 @@
+=== 2026-09-06T14:25:25+01:00 ===
+--- HE endpoint ---
+  HE address : 192.0.2.10/32
+  he-sim     : he-sim: ipv6/ip remote 198.51.100.137 local 192.0.2.10 ttl 64 6rd-prefix 2002::/16
+  he-sim v6  : 2001:db8:1f1c:f6::1/64
+  API        : running
+  API bound  : nohost
+  API calls  : 4
+  route back : 198.51.100.0/24 via 192.168.122.63 dev eth1 
+--- 172.31.1.252 ---
+vip=172.31.1.1 holds_vip=no wan_disabled=yes wan_up=no ppp_up=no ppp_active=no may_dial=no lease_age=- dropin=yes role=backup tun=DOWN radvd=inactive
+inactive
+tun0@NONE        DOWN           203.0.113.108 <POINTOPOINT,NOARP> 
+tun0: ipv6/ip remote 192.0.2.10 local 203.0.113.108 ttl 64 tos inherit 6rd-prefix 2002::/16
+inactive
+Sep 05 23:27:45 apitest vrrp-wan[13361]: bond0.53 enable commit took 5s
+Sep 05 23:29:01 apitest vrrp-wan[16369]: GUARD: lease stale (200s > 75s; is vrrp-wan-reconcile.timer running?) -- hanging up pppoe0
+Sep 05 23:29:27 apitest vrrp-wan[17398]: MASTER: dialling pppoe0
+Sep 05 23:29:57 apitest vrrp-wan[18610]: MASTER: dialling pppoe0
+-- Boot 6efc3c9ba47f455e9668454ee6c2fc37 --
+Sep 05 23:34:48 apitest vrrp-wan[6266]: MASTER: dialling pppoe0
+Sep 05 23:34:49 apitest vrrp-wan[6422]: MASTER with bond0.53 disabled -> enabling
+Sep 05 23:34:54 apitest vrrp-wan[7130]: bond0.53 enable commit took 5s
+-- Boot 866611afd7c542d4bf8c5117978dcd4a --
+Sep 06 13:21:54 apitest vrrp-wan[544215]: not MASTER: stopping radvd (deprecates the v6 gateway)
+Sep 06 13:21:54 apitest vrrp-wan[544221]: not MASTER: bringing tun0 down
+Sep 06 13:22:01 apitest he-tunnel-follow[544377]: role is now backup
+--- 172.31.1.253 ---
+vip=172.31.1.1 holds_vip=yes wan_disabled=no wan_up=yes ppp_up=yes ppp_active=yes may_dial=yes lease_age=25 dropin=yes role=master tun=UNKNOWN radvd=active
+tun0@NONE        UNKNOWN        198.51.100.137 <POINTOPOINT,NOARP,UP,LOWER_UP> 
+tun0: ipv6/ip remote 192.0.2.10 local 198.51.100.137 ttl 64 tos inherit 6rd-prefix 2002::/16
+default nhid 111 via 2001:db8:1f1c:f6::1 dev tun0 proto static metric 20 pref medium
+active
+Sep 06 09:29:21 vyos vrrp-wan[422441]: MASTER with bond0.53 disabled -> enabling
+Sep 06 09:29:25 vyos vrrp-wan[423516]: bond0.53 enable commit took 4s
+Sep 06 13:22:01 vyos he-tunnel-follow[593096]: role is now master
+Sep 06 13:22:01 vyos he-tunnel-follow[593109]: change seen (203.0.113.108 -> 198.51.100.137) but waiting for stability (1/2)
+Sep 06 13:23:01 vyos he-tunnel-follow[594799]: HE endpoint set to 198.51.100.137 (good 198.51.100.137)
+Sep 06 13:23:01 vyos he-tunnel-follow[594805]: moved tun0 to pppoe0: src 203.0.113.108 -> 198.51.100.137, mtu 1480 -> 1472
+Sep 06 13:24:02 vyos he-tunnel-follow[595700]: in sync: tun0 via pppoe0 src 198.51.100.137 mtu 1472
+Sep 06 13:24:26 vyos he-tunnel-follow[595886]: in sync: tun0 via pppoe0 src 198.51.100.137 mtu 1472
+Sep 06 13:24:26 vyos he-tunnel-follow[595904]: in sync: tun0 via pppoe0 src 198.51.100.137 mtu 1472
+Sep 06 13:25:01 vyos he-tunnel-follow[596311]: in sync: tun0 via pppoe0 src 198.51.100.137 mtu 1472

labsim/ipv6-ha-evidence/mechanism-2026-09-06.txt

@@ -0,0 +1,23 @@
+=== mechanism evidence, labsim, 2026-09-06T14:28:11+01:00 ===
+--- 172.31.1.252 ---
+vip=172.31.1.1 holds_vip=no wan_disabled=yes wan_up=no ppp_up=no ppp_active=no may_dial=no lease_age=- dropin=yes role=backup tun=DOWN radvd=inactive
+inactive
+tun0@NONE        DOWN           203.0.113.108 <POINTOPOINT,NOARP> 
+inactive
+Sep 06 13:22:01 apitest he-tunnel-follow[544377]: role is now backup
+--- 172.31.1.253 ---
+vip=172.31.1.1 holds_vip=yes wan_disabled=no wan_up=yes ppp_up=yes ppp_active=yes may_dial=yes lease_age=26 dropin=yes role=master tun=UNKNOWN radvd=active
+tun0@NONE        UNKNOWN        198.51.100.137 <POINTOPOINT,NOARP,UP,LOWER_UP> 
+active
+Sep 06 13:25:01 vyos he-tunnel-follow[596311]: in sync: tun0 via pppoe0 src 198.51.100.137 mtu 1472
+Sep 06 13:26:01 vyos he-tunnel-follow[598274]: in sync: tun0 via pppoe0 src 198.51.100.137 mtu 1472
+Sep 06 13:27:01 vyos he-tunnel-follow[599393]: in sync: tun0 via pppoe0 src 198.51.100.137 mtu 1472
+Sep 06 13:28:01 vyos he-tunnel-follow[600596]: in sync: tun0 via pppoe0 src 198.51.100.137 mtu 1472
+--- HE endpoint ---
+  HE address : 192.0.2.10/32
+  he-sim     : he-sim: ipv6/ip remote 198.51.100.137 local 192.0.2.10 ttl 64 6rd-prefix 2002::/16
+  he-sim v6  : 2001:db8:1f1c:f6::1/64
+  API        : running
+  API bound  : nohost
+  API calls  : 5
+  route back : 198.51.100.0/24 via 192.168.122.63 dev eth1 

labsim/k8s-up.sh

@@ -0,0 +1,266 @@
+#!/bin/bash
+# A real Kubernetes cluster inside labsim, on the OVS fabric, for rehearsing
+# Cilium <-> VyOS BGP before it goes near the production routers.
+#
+# Why VMs and not k3d: the thing under test is eBGP between Cilium and VyOS
+# across the switch fabric — nodes on VLAN 2, peering with the router's bond0.2
+# leg, directly connected. k3d would put the nodes on a container bridge, which
+# is a different L2 path and would prove something else. (It also needs Docker;
+# this host has podman.)
+#
+# Why not the existing micro VMs: they are Alpine with 256 MB and 1 vCPU. k3s
+# plus Cilium needs an order of magnitude more, and a glibc distro with a stock
+# kernel that Cilium's eBPF probes are actually tested against.
+#
+# Three nodes, not two: ECMP is only meaningfully tested if a node can be
+# drained and MORE THAN ONE path survives.
+#
+# Layout (mirrors production's shape, not its addresses):
+#   labsim-k8s1   172.31.2.11   k3s server
+#   labsim-k8s2   172.31.2.12   agent
+#   labsim-k8s3   172.31.2.13   agent
+#   gateway       172.31.2.1    the VRRP VIP of the router pair under test
+#   BGP peers     172.31.2.252 / .253   the routers' real per-box addresses
+#
+# Idempotent: re-running only creates what is missing.
+#
+# Usage:
+#   ./k8s-up.sh              create/start the cluster
+#   ./k8s-up.sh --kubeconfig fetch kubeconfig to ./labsim-k8s.kubeconfig
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+source "$SCRIPT_DIR/lib.sh"
+source "$SCRIPT_DIR/ovs.sh"
+
+# --- knobs ----------------------------------------------------------------
+K8S_VLAN="${K8S_VLAN:-2}"
+K8S_PREFIX="${K8S_PREFIX:-172.31.2}"
+K8S_NODES="${K8S_NODES:-3}"
+K8S_FIRST_OCTET="${K8S_FIRST_OCTET:-11}"
+# Dual-stack, mirroring production's shape (not its addresses). VLAN 2 v6 is a
+# ULA so nothing here can leak into the real HE /48; node fd00:2::1x matches the
+# BGP peers in sim-net-config.py (K8S_NODES_V6). cluster/service v6 are ULAs too;
+# the LB pool fd61:1e00::/64 is what Cilium advertises (SERVICE_CIDR_V6 there).
+K8S_PREFIX_V6="${K8S_PREFIX_V6:-fd00:2}"        # nodes fd00:2::11/12/13, router ::252/::253
+K8S_ROUTER_V6="${K8S_ROUTER_V6:-fd00:2::252}"   # v6 next-hop (primary router bond0.2)
+CLUSTER_CIDR_V6="${CLUSTER_CIDR_V6:-fd00:42::/56}"
+SERVICE_CIDR_V6="${SERVICE_CIDR_V6:-fd00:43::/112}"
+K8S_MEM="${K8S_MEM:-4096}"          # MB — k3s + cilium + a workload
+K8S_CPUS="${K8S_CPUS:-2}"
+K8S_DISK_GB="${K8S_DISK_GB:-12}"
+K8S_TOKEN="${K8S_TOKEN:-labsim-k3s-token}"
+
+# Debian rather than Alpine: glibc, a stock kernel, and cloud-init that applies
+# network-config properly (the Alpine base in this sim notably does not).
+DEB_URL="${DEB_URL:-https://cloud.debian.org/images/cloud/trixie/latest/debian-13-genericcloud-amd64.qcow2}"
+DEB_BASE="${DEB_BASE:-$IMG_DIR/debian-13-genericcloud-amd64.qcow2}"
+
+# Same version production runs, so CRD shapes and chart flags transfer exactly.
+CILIUM_VERSION="${CILIUM_VERSION:-1.19.1}"
+
+node_name() { echo "labsim-k8s$1"; }
+node_ip()   { echo "${K8S_PREFIX}.$((K8S_FIRST_OCTET + $1 - 1))"; }
+node_ip6()  { echo "${K8S_PREFIX_V6}::$((K8S_FIRST_OCTET + $1 - 1))"; }
+
+# --- base image -----------------------------------------------------------
+ensure_base_image() {
+  if [ -f "$DEB_BASE" ]; then
+    log "base image present: $(basename "$DEB_BASE")"
+    return
+  fi
+  log "fetching Debian cloud image (~330 MB) -> $DEB_BASE"
+  sudo mkdir -p "$IMG_DIR"
+  # .tmp + mv so an interrupted download never leaves a half image that later
+  # runs treat as valid.
+  sudo curl -fsSL --retry 3 -o "${DEB_BASE}.tmp" "$DEB_URL" \
+    || die "could not fetch $DEB_URL"
+  sudo mv "${DEB_BASE}.tmp" "$DEB_BASE"
+  log "base image ready"
+}
+
+# --- cloud-init -----------------------------------------------------------
+# The server node writes the join token; agents wait for the API to answer
+# before joining, because cloud-init ordering across VMs is not guaranteed and
+# a failed join leaves an agent that never retries.
+build_k8s_seed() {
+  local iso="$1" vm="$2" ip="$3" role="$4" server_ip="$5" pubkey="$6" ip6="$7" server_ip6="$8"
+  local tmp; tmp="$(mktemp -d)"
+
+  cat > "$tmp/meta-data" <<EOF
+instance-id: $vm
+local-hostname: $vm
+EOF
+
+  cat > "$tmp/network-config" <<EOF
+version: 2
+ethernets:
+  enp1s0:
+    match:
+      name: "en*"
+    addresses: [$ip/24, $ip6/64]
+    routes:
+      - to: default
+        via: ${K8S_PREFIX}.1
+    nameservers:
+      addresses: [8.8.8.8, 1.1.1.1]
+EOF
+
+  local k3s_exec
+  if [ "$role" = "server" ]; then
+    # flannel/servicelb/traefik off: Cilium is the CNI under test, and k3s's
+    # own ServiceLB would fight Cilium for LoadBalancer addresses. Dual-stack:
+    # both families in cluster-cidr/service-cidr and a dual node-ip, mirroring
+    # production. Pod IPs come from Cilium cluster-pool (node.spec.podCIDRs is
+    # immutable), so cluster-cidr v6 just marks the cluster dual-stack.
+    k3s_exec="server --flannel-backend=none --disable-network-policy --disable=servicelb --disable=traefik --node-ip=$ip,$ip6 --tls-san=$ip --cluster-cidr=10.42.0.0/16,${CLUSTER_CIDR_V6} --service-cidr=10.43.0.0/16,${SERVICE_CIDR_V6} --cluster-init"
+  else
+    k3s_exec="agent --server https://${server_ip}:6443 --node-ip=$ip,$ip6"
+  fi
+
+  cat > "$tmp/user-data" <<EOF
+#cloud-config
+hostname: $vm
+fqdn: $vm
+users:
+  - name: debian
+    groups: [sudo]
+    shell: /bin/bash
+    sudo: ["ALL=(ALL) NOPASSWD:ALL"]
+    lock_passwd: false
+    plain_text_passwd: labsim
+    ssh_authorized_keys:
+      - $pubkey
+ssh_pwauth: true
+disable_root: false
+ssh_authorized_keys:
+  - $pubkey
+
+package_update: true
+packages: [curl, jq, iproute2, tcpdump, bird2]
+
+write_files:
+  # Cilium replaces kube-proxy and needs these; Debian cloud images ship
+  # neither loaded nor persisted.
+  - path: /etc/modules-load.d/cilium.conf
+    content: |
+      br_netfilter
+      overlay
+  - path: /etc/sysctl.d/99-k8s.conf
+    content: |
+      net.ipv4.ip_forward = 1
+      net.bridge.bridge-nf-call-iptables = 1
+
+runcmd:
+  - [ modprobe, br_netfilter ]
+  - [ modprobe, overlay ]
+  - [ sysctl, --system ]
+  - |
+    # Wait for the server's API before an agent tries to join. Without this the
+    # agent fails once and the unit backs off for minutes.
+    if [ "$role" != "server" ]; then
+      for i in \$(seq 1 60); do
+        curl -sk --max-time 3 https://${server_ip}:6443/ping >/dev/null 2>&1 && break
+        sleep 5
+      done
+    fi
+  - |
+    curl -sfL https://get.k3s.io | \
+      INSTALL_K3S_EXEC="$k3s_exec" \
+      K3S_TOKEN="$K8S_TOKEN" \
+      sh -
+EOF
+
+  sudo mkdir -p "$(dirname "$iso")"
+  sudo genisoimage -quiet -output "$iso" -volid cidata -joliet -rock \
+    "$tmp/user-data" "$tmp/meta-data" "$tmp/network-config"
+  rm -rf "$tmp"
+}
+
+# --- VM creation ----------------------------------------------------------
+create_node() {
+  local n="$1" pubkey="$2"
+  local vm; vm="$(node_name "$n")"
+  local ip; ip="$(node_ip "$n")"
+  local ip6; ip6="$(node_ip6 "$n")"
+  local role="agent"; [ "$n" -eq 1 ] && role="server"
+  local server_ip; server_ip="$(node_ip 1)"
+  local server_ip6; server_ip6="$(node_ip6 1)"
+
+  if virsh_q dominfo "$vm" >/dev/null 2>&1; then
+    local state; state="$(virsh_q domstate "$vm" 2>/dev/null | head -1 | tr -d '\n')"
+    if [ "$state" = "running" ]; then
+      log "$vm already running ($ip, $role)"
+    else
+      log "$vm exists but is $state — starting"
+      virsh_q start "$vm" >/dev/null
+    fi
+    return
+  fi
+
+  local disk="$IMG_DIR/${vm}.qcow2"
+  local seed="$IMG_DIR/${vm}-seed.iso"
+
+  log "creating $vm ($ip, $role, ${K8S_MEM}MB/${K8S_CPUS}cpu)"
+  sudo qemu-img create -q -f qcow2 -F qcow2 -b "$DEB_BASE" "$disk" "${K8S_DISK_GB}G" >/dev/null
+  build_k8s_seed "$seed" "$vm" "$ip" "$role" "$server_ip" "$pubkey" "$ip6" "$server_ip6"
+
+  # Access port on the k8s VLAN — same broadcast domain as the routers'
+  # bond0.2 leg, so BGP peering is directly connected exactly as in production.
+  sudo virt-install --connect "$LIBVIRT_URI" --name "$vm" \
+    --memory "$K8S_MEM" --vcpus "$K8S_CPUS" \
+    --disk "path=$disk,format=qcow2,bus=virtio" \
+    --disk "path=$seed,device=cdrom" \
+    --network "network=$OVS_NET,portgroup=vlan${K8S_VLAN},model=virtio" \
+    --os-variant debian12 \
+    --graphics none --noautoconsole --import >/dev/null
+}
+
+fetch_kubeconfig() {
+  local server_ip; server_ip="$(node_ip 1)"
+  local out="$SCRIPT_DIR/labsim-k8s.kubeconfig"
+  log "fetching kubeconfig from $server_ip"
+  ssh -o StrictHostKeyChecking=no -o ConnectTimeout=10 \
+      "debian@${server_ip}" "sudo cat /etc/rancher/k3s/k3s.yaml" \
+    | sed "s|127.0.0.1|${server_ip}|" > "$out"
+  chmod 600 "$out"
+  log "wrote $out"
+  log "use: KUBECONFIG=$out kubectl get nodes"
+}
+
+main() {
+  if [ "${1:-}" = "--kubeconfig" ]; then
+    fetch_kubeconfig
+    return
+  fi
+
+  require_tools
+  command -v genisoimage >/dev/null || die "genisoimage missing (dnf install genisoimage)"
+
+  local pubkey; pubkey="$(find_ssh_pubkey)"
+  log "using SSH key: ${pubkey%% *} ...${pubkey##* }"
+
+  ensure_base_image
+
+  # Select EVERY VLAN, not just the k8s one. ovs_up re-defines the libvirt
+  # network from SELECTED, so narrowing it here silently drops the portgroups
+  # for every other VLAN -- running VMs keep working (their taps are already
+  # attached) and nothing complains until the next VM cannot be attached.
+  # Observed: this deleted vlan1/3/9/10/200/51/53 and only surfaced when the
+  # ISP VMs needed vlan51 and vlan53.
+  selected_vlans
+  log "ensuring OVS fabric (all VLANs, so no portgroup is dropped)"
+  ovs_up
+
+  for n in $(seq 1 "$K8S_NODES"); do
+    create_node "$n" "$pubkey"
+  done
+
+  echo
+  log "nodes created. k3s installs on first boot (a few minutes)."
+  log "watch:   ssh debian@$(node_ip 1) 'sudo systemctl status k3s'"
+  log "then:    $0 --kubeconfig"
+  log "then install Cilium $CILIUM_VERSION and the BGP resources (see README)."
+}
+
+main "$@"

labsim/labsim-dhcp-test.sh

@@ -0,0 +1,219 @@
+#!/bin/bash
+# Prove that VyOS hands each device the address UniFi reserved for it.
+#
+# The question this answers is narrow and important: 30 of the 31 UniFi
+# reservations sit INSIDE the DHCP pool (LoT's pool is 10.0.0.11-10.0.1.254 and
+# only 10.0.0.2 falls outside it). UniFi's dhcpd tolerates that. VyOS uses kea,
+# and whether kea honours in-pool host reservations decides whether the cutover
+# silently renumbers 30 devices. That is not something to predict.
+#
+# Method: boot throwaway VMs whose MAC is a REAL production MAC, on the sim
+# VLAN, and check the address they are given. MACs are the one piece of
+# production config that transplants verbatim -- the subnet is rewritten, the
+# MAC is not -- which is what makes this a real test rather than a rehearsal.
+#
+# Safe: the ovs-labsim bridge contains only internal ports and VM taps, with no
+# physical NIC, so a production MAC here cannot reach or confuse the real LAN.
+# Verified with `ovs-vsctl show` before this script was written.
+#
+#   ./labsim-dhcp-test.sh              run the standard cases
+#   ./labsim-dhcp-test.sh --keep       leave the VMs up for inspection
+#   ./labsim-dhcp-test.sh --clean      just remove any leftover test VMs
+set -uo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+source "$SCRIPT_DIR/lib.sh"
+
+ROUTER_IP="${ROUTER_IP:-172.31.1.1}"
+ROUTER_PW="${ROUTER_PW:-vyos}"
+TEST_VLAN="${TEST_VLAN:-10}"
+BOOT_WAIT="${BOOT_WAIT:-150}"
+TAG="labsim-dhcptest"
+
+# mac|expected|why. "POOL" means: must get an address from the pool and must
+# NOT get any reserved address -- the negative case that stops a pass from
+# meaning merely "DHCP works".
+CASES=(
+  "f8:0d:ac:90:65:c6|172.31.10.46|printer1 - reservation inside the pool"
+  "1c:69:20:7f:bc:77|172.31.11.67|sonoff-matter - in-pool AND across the /23 boundary"
+  "34:e1:d1:80:29:ce|172.31.10.2|Hubitat - the one reservation OUTSIDE the pool"
+  "52:54:00:ab:cd:ef|POOL|unreserved MAC - must get a pool address, not a reserved one"
+)
+
+vm_of() { echo "${TAG}-$(echo "$1" | tr -d ':')"; }
+
+cleanup_vms() {
+  local n=0
+  while read -r vm; do
+    [ -z "$vm" ] && continue
+    virsh_q destroy "$vm" >/dev/null 2>&1
+    virsh_q undefine "$vm" --remove-all-storage >/dev/null 2>&1
+    n=$((n + 1))
+  done < <(virsh_q list --all --name 2>/dev/null | grep "^${TAG}-" || true)
+  [ "$n" -gt 0 ] && log "removed $n test VM(s)"
+  sudo rm -f "$IMG_DIR/${TAG}-"*.qcow2 "$IMG_DIR/${TAG}-"*-seed.iso 2>/dev/null
+  return 0
+}
+
+# A seed that asks for DHCP instead of taking a static address. Alpine's
+# cloud-init ignores network-config here (verified previously and documented in
+# README), so /etc/network/interfaces is what actually takes effect.
+build_dhcp_seed() {
+  local iso="$1" vm="$2" pubkey="$3"
+  local tmp; tmp="$(mktemp -d)"
+  cat > "$tmp/meta-data" <<EOF
+instance-id: $vm
+local-hostname: $vm
+EOF
+  cat > "$tmp/user-data" <<EOF
+#cloud-config
+hostname: $vm
+users:
+  - name: alpine
+    shell: /bin/ash
+    lock_passwd: false
+    plain_text_passwd: labsim
+    ssh_authorized_keys:
+      - $pubkey
+ssh_authorized_keys:
+  - $pubkey
+disable_root: false
+chpasswd:
+  list: |
+    root:labsim
+  expire: false
+write_files:
+  - path: /etc/network/interfaces
+    content: |
+      auto lo
+      iface lo inet loopback
+      auto eth0
+      iface eth0 inet dhcp
+runcmd:
+  - [ sh, -c, "ifdown eth0 2>/dev/null; ifup eth0 || udhcpc -i eth0 -q || true" ]
+EOF
+  python3 - "$tmp/user-data" <<'PY' || die "generated user-data is not valid YAML"
+import sys, yaml
+yaml.safe_load(open(sys.argv[1]).read().split("#cloud-config",1)[1])
+PY
+  sudo genisoimage -quiet -output "$iso" -volid cidata -joliet -rock \
+    "$tmp/user-data" "$tmp/meta-data" >/dev/null 2>&1 || die "seed build failed"
+  rm -rf "$tmp"
+}
+
+router() {
+  timeout 30 sshpass -p "$ROUTER_PW" ssh -o StrictHostKeyChecking=no \
+    -o BatchMode=no -o ConnectTimeout=8 "vyos@$ROUTER_IP" "$@" 2>/dev/null
+}
+
+# --- argument handling ----------------------------------------------------
+KEEP=0
+case "${1:-}" in
+  --clean) cleanup_vms; exit 0 ;;
+  --keep)  KEEP=1 ;;
+  "")      ;;
+  *)       die "usage: $0 [--keep|--clean]" ;;
+esac
+
+command -v sshpass >/dev/null || die "sshpass required"
+require_tools
+[ -f "$BASE_IMAGE" ] || die "base image missing: $BASE_IMAGE (run labsim-up.sh first)"
+
+log "checking the router is serving DHCP..."
+subnets=$(router '/opt/vyatta/bin/vyatta-op-cmd-wrapper show configuration commands | grep -c subnet-id')
+maps=$(router '/opt/vyatta/bin/vyatta-op-cmd-wrapper show configuration commands | grep -c "static-mapping .* mac"')
+log "  router has ${subnets:-0} subnets and ${maps:-0} static-mappings"
+[ "${maps:-0}" -gt 0 ] || die "router has no static-mappings -- apply the generated config first"
+
+cleanup_vms
+
+# Flush the lease database first. This is not tidiness -- it is the condition
+# the cutover actually runs under, because kea does not inherit UniFi's leases
+# and starts empty. It also makes the test deterministic: with stale leases
+# present, kea saw the reserved address as held by "another client" (the same
+# MAC but a different client-id from a previous boot) and allocated a dynamic
+# address instead, which produced three misleading results before this existed.
+log "flushing the router's lease database (cutover starts with an empty one)"
+# Every dhcp4-leases.csv* must go, not just the main file: kea's memfile
+# backend keeps lease-file-cleanup rotations (.1/.2) and restores from them on
+# start, so truncating only the primary leaves the old leases intact.
+router 'sudo systemctl stop isc-kea-dhcp4-server;
+        sudo sh -c "rm -f /config/dhcp/dhcp4-leases.csv*";
+        sudo systemctl start isc-kea-dhcp4-server' >/dev/null
+sleep 5
+remaining="$(router '/opt/vyatta/bin/vyatta-op-cmd-wrapper show dhcp server leases' | sed -n '3,$p' | grep -c .)"
+[ "${remaining:-0}" -eq 0 ] || warn "lease table still has ${remaining} row(s) after flush"
+
+SSH_PUB="$(find_ssh_pubkey)"
+sudo mkdir -p "$IMG_DIR"
+
+# --- boot one VM per case -------------------------------------------------
+for c in "${CASES[@]}"; do
+  IFS='|' read -r mac expected why <<<"$c"
+  vm="$(vm_of "$mac")"
+  disk="$IMG_DIR/${vm}.qcow2"; seed="$IMG_DIR/${vm}-seed.iso"
+  log "booting $vm  mac=$mac  ($why)"
+  sudo qemu-img create -q -f qcow2 -F qcow2 -b "$BASE_IMAGE" "$disk" "$VM_DISK" >/dev/null
+  build_dhcp_seed "$seed" "$vm" "$SSH_PUB"
+  sudo virt-install --connect "$LIBVIRT_URI" --name "$vm" \
+    --memory "$VM_MEM" --vcpus "$VM_CPUS" \
+    --disk "path=$disk,format=qcow2,bus=virtio" \
+    --disk "path=$seed,device=cdrom,readonly=on" \
+    --network "network=labsim-ovs,portgroup=vlan${TEST_VLAN},model=virtio,mac=$mac" \
+    --os-variant alpinelinux3.18 --graphics none --noautoconsole --import >/dev/null \
+    || die "virt-install failed for $vm"
+done
+
+log "waiting ${BOOT_WAIT}s for boot + DHCP..."
+sleep "$BOOT_WAIT"
+
+# --- verdict --------------------------------------------------------------
+# The lease table on the router is the authority: it says what the server
+# decided, independent of whether the guest brought the interface up cleanly.
+leases="$(router '/opt/vyatta/bin/vyatta-op-cmd-wrapper show dhcp server leases')"
+echo
+echo "=== router lease table ==="
+echo "$leases"
+echo
+
+reserved_ips="$(cd "$SCRIPT_DIR/../migration" && python3 unifi-to-vyos.py --mode sim 2>/dev/null \
+  | awk '/static-mapping .* ip-address/ {print $NF}')"
+
+pass=0; fail=0
+printf '%-19s %-16s %-16s %s\n' "MAC" "EXPECTED" "GOT" "RESULT"
+for c in "${CASES[@]}"; do
+  IFS='|' read -r mac expected why <<<"$c"
+  # Never guess which lease is "the" lease. Taking the first match is how an
+  # hours-old lease was once reported as the current answer, turning three
+  # failures into apparent passes.
+  matches="$(echo "$leases" | awk -v m="$mac" 'tolower($2) == tolower(m) {print $1}')"
+  n_match="$(echo "$matches" | grep -c . )"
+  if [ "$n_match" -gt 1 ]; then
+    got="AMBIGUOUS($(echo "$matches" | tr '\n' ',' | sed 's/,$//'))"
+  else
+    got="${matches:-<none>}"
+  fi
+  if [ "${got#AMBIGUOUS}" != "$got" ]; then
+    # More than one lease for this MAC means the flush did not take. Any
+    # verdict from here is a guess, so refuse to give one.
+    result="FAIL (multiple leases -- flush did not take)"
+  elif [ "$expected" = "POOL" ]; then
+    if [ "$got" = "<none>" ]; then
+      result="FAIL (no lease at all)"
+    elif echo "$reserved_ips" | grep -qx "$got"; then
+      result="FAIL (got a RESERVED address)"
+    else
+      result="pass"
+    fi
+  else
+    [ "$got" = "$expected" ] && result="pass" || result="FAIL"
+  fi
+  [ "$result" = "pass" ] && pass=$((pass + 1)) || fail=$((fail + 1))
+  printf '%-19s %-16s %-16s %s\n' "$mac" "$expected" "$got" "$result"
+  printf '  %s\n' "$why"
+done
+
+echo
+log "$pass passed, $fail failed"
+[ "$KEEP" -eq 1 ] && log "VMs left running (--keep). Remove with: $0 --clean" || cleanup_vms
+[ "$fail" -eq 0 ] || exit 1

labsim/labsim-down.sh

@@ -0,0 +1,61 @@
+#!/bin/bash
+# Tear down the lab network simulation.
+#
+# By default this destroys VMs and networks but KEEPS the downloaded base
+# image, so the next bring-up is fast. Pass --purge to remove that too.
+#
+# Usage: ./labsim-down.sh [--purge] [vlan-id ...]
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+source "$SCRIPT_DIR/lib.sh"
+source "$SCRIPT_DIR/ovs.sh"
+
+PURGE=false
+ARGS=()
+for a in "$@"; do
+  case "$a" in
+    --purge) PURGE=true ;;
+    *) ARGS+=("$a") ;;
+  esac
+done
+
+selected_vlans "${ARGS[@]+"${ARGS[@]}"}"
+
+for entry in "${SELECTED[@]}"; do
+  IFS=: read -r vid name prefix _r <<<"$entry"
+  vm="$(vm_name "$vid" "$name")"
+
+  if virsh_q dominfo "$vm" >/dev/null 2>&1; then
+    log "destroying VM $vm"
+    virsh_q destroy "$vm" >/dev/null 2>&1 || true
+    virsh_q undefine "$vm" --nvram >/dev/null 2>&1 || virsh_q undefine "$vm" >/dev/null 2>&1 || true
+  fi
+  sudo rm -f "$IMG_DIR/${vm}.qcow2" "$IMG_DIR/${vm}-seed.iso"
+
+done
+
+# Legacy per-VLAN Linux-bridge networks from before the OVS migration. If
+# these survive they keep a duplicate <prefix>.2/24 on a dead bridge, and the
+# kernel may prefer that route over the OVS host leg — which looks exactly
+# like "the VM is unreachable" while ping -I hostvN works fine.
+for entry in "${SELECTED[@]}"; do
+  IFS=: read -r vid _n _p _r <<<"$entry"
+  legacy="labsim-vlan${vid}"
+  if virsh_q net-info "$legacy" >/dev/null 2>&1; then
+    log "removing legacy network $legacy"
+    virsh_q net-destroy "$legacy" >/dev/null 2>&1 || true
+    virsh_q net-undefine "$legacy" >/dev/null 2>&1 || true
+  fi
+done
+
+log "removing OVS fabric"
+ovs_down
+
+if [ "$PURGE" = true ]; then
+  log "purging base image $BASE_IMAGE"
+  sudo rm -f "$BASE_IMAGE"
+  sudo rmdir "$IMG_DIR" 2>/dev/null || true
+fi
+
+log "environment is DOWN"

labsim/labsim-dualstack-convert.sh

@@ -0,0 +1,116 @@
+#!/bin/bash
+# Rehearse the k3s single -> dual-stack conversion on the 3-server etcd cluster,
+# the way production Phase 4 will do it: edit each server's config.yaml (rendered
+# by the PRODUCTION generator) to add the second address family, restart ONE
+# server at a time, and watch what happens in between.
+#
+# Answers the questions a single-node lab cannot:
+#   - does quorum survive a rolling config.yaml change across 3 etcd servers?
+#   - what does a MIXED control plane do (one server dual, two still v4-only)?
+#   - does ServiceCIDR pick up the v6 range on the FIRST server's restart, or
+#     only once all three agree?
+#
+# The node-ip v6 must exist on the box before k3s reads it, so each server first
+# gets a ULA on its interface (fd00:2::3x), mirroring how production nodes get a
+# DHCPv6 address before k3s starts.
+#
+#   ./labsim-dualstack-convert.sh baseline   snapshot the v4-only starting state
+#   ./labsim-dualstack-convert.sh convert    roll the conversion, snapshotting each step
+#   ./labsim-dualstack-convert.sh snapshot   print current SC / podCIDRs / quorum
+set -uo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+NET="${NET:-172.31.2}"; FIRST_OCTET="${FIRST_OCTET:-31}"; SERVERS="${SERVERS:-3}"
+TOKEN="${TOKEN:-labsim-etcd-token}"
+RENDER="${RENDER:-$SCRIPT_DIR/../bastion/src/modules/dist/modules/k3s/bin/render-config.js}"
+EVID="${EVID:-$SCRIPT_DIR/dualstack-evidence}"
+
+# Dual-stack target ranges. ULA/v4 -- the mechanism is what's under test, not the
+# addresses; using ULA keeps sim traffic out of the real /48.
+V4_CLUSTER="10.42.0.0/16"; V6_CLUSTER="fd00:42::/56"
+V4_SERVICE="10.43.0.0/16"; V6_SERVICE="fd00:43::/112"   # /112: apiserver caps v6 service ranges
+V6_NODE_PREFIX="fd00:2"                                  # node-ip v6: fd00:2::3x
+
+SSH=(-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR -o ConnectTimeout=8 -o BatchMode=yes)
+node_ip()   { echo "${NET}.$((FIRST_OCTET + $1 - 1))"; }
+node_v6()   { echo "${V6_NODE_PREFIX}::$((FIRST_OCTET + $1 - 1))"; }
+node_name() { echo "labsim-etcd$1"; }
+s() { local n="$1"; shift; timeout "${TMO:-25}" ssh "${SSH[@]}" "debian@$(node_ip "$n")" "$@" 2>/dev/null; }
+k() { s 1 "sudo k3s kubectl $*"; }
+log() { printf '\033[36m==>\033[0m %s\n' "$*"; }
+
+snapshot() {
+  # custom-columns, not jsonpath: the jsonpath range/quotes get mangled through
+  # ssh -> sudo -> kubectl and came back empty.
+  echo "  servicecidr :"
+  s 1 'sudo k3s kubectl get servicecidr -o custom-columns=NAME:.metadata.name,CIDRS:.spec.cidrs --no-headers' 2>/dev/null | sed 's/^/    /'
+  echo "  node podCIDRs:"
+  s 1 'sudo k3s kubectl get nodes -o custom-columns=NAME:.metadata.name,PODCIDRS:.spec.podCIDRs --no-headers' 2>/dev/null | sed 's/^/    /'
+  local ready; ready="$(k get nodes --no-headers 2>/dev/null | wc -l)"
+  echo "  nodes registered: $ready ; apiserver: $(k get --raw /readyz >/dev/null 2>&1 && echo ok || echo DOWN)"
+}
+
+# Re-render node n's config.yaml WITH the dual families and write it back.
+convert_node() {
+  local n="$1"
+  local ip; ip="$(node_ip "$n")"; local v6; v6="$(node_v6 "$n")"
+  log "server $n ($ip): add $v6, re-render dual config.yaml, restart k3s"
+
+  # 1. node-ip v6 must exist before k3s reads it.
+  s "$n" "sudo ip -6 addr add ${v6}/64 dev enp1s0 2>/dev/null; ip -6 -br addr show enp1s0 | grep -o '${v6}/64'" | sed 's/^/    addr: /'
+
+  # 2. render the dual config from the PRODUCTION generator, preserving this
+  #    node's role (node 1 cluster-init, else joining server).
+  local extra=""
+  [ "$n" -ne 1 ] && extra="K3S_SERVER_URL=https://$(node_ip 1):6443 K3S_TOKEN=$TOKEN"
+  local cfg
+  cfg="$(env ROLE=infra HOSTNAME="$(node_name "$n")" IP="$ip" TLS_SANS="$ip" \
+      IPV6="$v6" CLUSTER_CIDR="${V4_CLUSTER},${V6_CLUSTER}" SERVICE_CIDR="${V4_SERVICE},${V6_SERVICE}" \
+      $extra node "$RENDER")"
+  # sanity: the render must actually carry both families or the restart is pointless
+  echo "$cfg" | grep -q "$V6_CLUSTER" || { echo "    RENDER MISSING v6 -- aborting"; return 1; }
+
+  # 3. write it back and restart k3s on this one server.
+  printf '%s\n' "$cfg" | s "$n" "sudo tee /etc/rancher/k3s/config.yaml >/dev/null && sudo systemctl restart k3s"
+
+  # 4. wait for THIS server's k3s to come back and the apiserver to answer.
+  local i
+  for i in $(seq 1 30); do
+    [ "$(s "$n" 'sudo systemctl is-active k3s' 2>/dev/null)" = active ] && \
+      s "$n" 'sudo k3s kubectl get --raw /readyz >/dev/null 2>&1' && break
+    sleep 10
+  done
+  echo "    server $n k3s=$(s "$n" 'systemctl is-active k3s')  restarts=$(s "$n" 'systemctl show k3s -p NRestarts --value')"
+}
+
+cmd_baseline() {
+  mkdir -p "$EVID"
+  { echo "=== BASELINE (v4-only) $(date -u +%FT%TZ) ==="; snapshot; } | tee "$EVID/convert-baseline.txt"
+}
+
+cmd_snapshot() { snapshot; }
+
+cmd_convert() {
+  mkdir -p "$EVID"
+  local out="$EVID/convert-run.txt"
+  {
+    echo "=== 3-SERVER DUAL-STACK CONVERSION $(date -u +%FT%TZ) ==="
+    echo "--- before ---"; snapshot
+    local n
+    for n in $(seq 1 "$SERVERS"); do
+      echo; echo "### converting server $n of $SERVERS ###"
+      convert_node "$n" || { echo "convert_node $n failed"; break; }
+      echo "--- state after server $n (MIXED until n=$SERVERS) ---"
+      snapshot
+    done
+    echo; echo "--- FINAL ---"; snapshot
+  } 2>&1 | tee "$out"
+  log "evidence -> $out"
+}
+
+case "${1:-snapshot}" in
+  baseline) cmd_baseline ;;
+  convert)  cmd_convert ;;
+  snapshot) cmd_snapshot ;;
+  *) echo "usage: $0 {baseline|convert|snapshot}" >&2; exit 2 ;;
+esac

labsim/labsim-dualstack-net.sh

@@ -0,0 +1,221 @@
+#!/bin/bash
+# VLAN 2 gets IPv6 in labsim: addresses, router advertisements and a DHCPv6
+# server with per-MAC reservations.
+#
+# This is the rehearsal of the production change (dual-stack plan, phase 2b) and
+# runs the SAME VyOS config, against the sim router pair, so the production
+# apply is a repeat rather than a first attempt.
+#
+# WHY DHCPv6 AND NOT SLAAC. The cluster needs each node's IPv6 to be knowable in
+# advance and stable: k3s resolves node-ip once at start-up, and a node's
+# identity cannot be allowed to change under it. The estate already answers that
+# question for IPv4 with kea reservations keyed on MAC, so IPv6 answers it the
+# same way and stays one source of truth.
+#
+# DHCPv6 normally keys on DUID, not MAC -- a DUID is generated by the client and
+# is not derivable from its MAC, which would have meant a second, client-owned
+# source of truth. VyOS's static-mapping accepts `mac` as well as `duid`
+# (verified on the sim: the node.tag directory offers duid, mac, ipv6-address,
+# ipv6-prefix), so the reservation can key on the same MAC the v4 one does.
+#
+# The RA carries managed-flag WITH no-autonomous-flag. That combination is what
+# makes the node's address unambiguous: managed sends it to DHCPv6, and
+# non-autonomous stops it also forming a SLAAC address from the same prefix.
+# Leave autonomous on and every node has two global addresses, only one of which
+# anybody reserved -- and whichever labctl happens to find is the one that ends
+# up in node-ip.
+#
+# Both are VALUELESS nodes on this VyOS: `managed-flag` not `managed-flag true`,
+# and `prefix <p> no-autonomous-flag` not `autonomous-flag false`. The value
+# forms are rejected with "is not valid".
+#
+# A FAILED COMMIT DOES NOT MEAN NOTHING CHANGED. VyOS commits node groups
+# independently, so an earlier run of this script left the bond0.2 address and
+# most of the router-advert block applied while `[[service dhcpv6-server]]
+# failed` -- the interface and RA groups had already succeeded. Re-read the
+# config after any failure rather than assuming a clean rollback; that matters
+# more in production than here.
+#
+# TWO THINGS THIS REHEARSAL FOUND, both of which would have bitten production:
+#
+# 1. managed-flag is NECESSARY BUT NOT SUFFICIENT. It only tells the host to use
+#    DHCPv6; the kernel's accept_ra implements SLAAC and nothing else, so a node
+#    with no DHCPv6 *client* running takes no lease at all. The sim's Debian
+#    nodes have neither NetworkManager nor a networkd .network file managing the
+#    interface, so they ended up with a kernel SLAAC address
+#    (2001:db8:187e:2:5054:ff:fe53:731 -- EUI-64 from the MAC) and never asked
+#    for the reserved ::11. Production's Fedora nodes use NetworkManager, which
+#    does run a DHCPv6 client on the managed flag, and the DGX Sparks run
+#    NetworkManager too -- but that must be VERIFIED per node, not assumed. The
+#    k3s module's preflight catches the consequence; the fix is node-side.
+#
+# 2. TURNING AUTONOMOUS OFF DOES NOT RETRACT ADDRESSES ALREADY FORMED. An
+#    earlier partial apply advertised the prefix while autonomous was still on;
+#    the nodes autoconfigured, and adding no-autonomous-flag afterwards left
+#    those addresses in place with a 30-day valid lifetime. So in production,
+#    where VLAN 2 has no IPv6 at all yet, no-autonomous-flag MUST be in the same
+#    commit that first advertises the prefix. Advertise first and tighten later
+#    and every node carries an unreserved EUI-64 address that labctl might pick
+#    up as node-ip.
+#
+#   ./labsim-dualstack-net.sh up       apply to both sim routers
+#   ./labsim-dualstack-net.sh down     remove it again
+#   ./labsim-dualstack-net.sh status   what the routers and nodes think
+#   ./labsim-dualstack-net.sh leases   who has taken an address
+set -uo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+R1="${R1:-172.31.1.252}"   # sim vyos001 -> ::1
+R2="${R2:-172.31.1.253}"   # sim vyos002 -> ::2
+PW="${VYOS_PW:-vyos}"
+
+# Mirrors production's scheme (2001:470:187e:<vlan>::/64) using the
+# documentation prefix, so the shape is rehearsed without putting sim addresses
+# inside a range production announces.
+V6_PREFIX="${V6_PREFIX:-2001:db8:187e:2}"
+LINK_MTU="${LINK_MTU:-1472}"
+# VyOS requires a unique subnet-id per DHCPv6 subnet ("Unique subnet ID not
+# specified for subnet"). Using the VLAN id keeps it self-documenting and
+# collision-free across VLANs.
+VLAN_ID="${VLAN_ID:-2}"
+
+# node -> last hextet. Mirrors the v4 host part (172.31.2.11 -> ::11) so a
+# reservation is readable next to its IPv4 twin.
+NODES=(labsim-k8s1:11 labsim-k8s2:12 labsim-k8s3:13)
+
+SSH=(-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null
+     -o LogLevel=ERROR -o ConnectTimeout=6 -o PreferredAuthentications=password)
+
+log()  { printf '\033[0;36m[ds-net]\033[0m %s\n' "$*"; }
+warn() { printf '\033[1;33m[ds-net]\033[0m %s\n' "$*" >&2; }
+die()  { printf '\033[0;31m[ds-net]\033[0m %s\n' "$*" >&2; exit 1; }
+
+# Drive VyOS from a script FILE, never `vbash -c`. The latter never starts a
+# config session: commit fails to stderr, a helper discards it, and the run
+# reports success having changed nothing. labsim-pppoe-ha-test.sh lost a whole
+# policy matrix to exactly that.
+vyos_apply() {  # host, set-lines on stdin
+    local h="$1" out
+    out="$({ printf '#!/bin/vbash\nsource /opt/vyatta/etc/functions/script-template\nconfigure\n'
+             cat
+             printf 'commit\nsave\nexit\n'
+           } | timeout 120 sshpass -p "$PW" ssh "${SSH[@]}" "vyos@$h" \
+                 'cat > /tmp/ds-net.sh && chmod +x /tmp/ds-net.sh && sudo /tmp/ds-net.sh' 2>&1)"
+    # Read the outcome instead of assuming it. The first version of this script
+    # printed "up" after BOTH routers had failed to commit -- the same shape of
+    # lie this repo has been bitten by before (a matrix reporting coverage it did
+    # not have). vbash exits 0 even when the commit fails, so the text is the
+    # only honest signal.
+    printf '%s\n' "$out" | grep -vE '^\s*$' | sed 's/^/      /' | tail -6
+    if printf '%s' "$out" | grep -qiE 'Commit failed|\[\[.*\]\] failed|Set failed'; then
+        return 1
+    fi
+    return 0
+}
+
+r() { timeout 40 sshpass -p "$PW" ssh "${SSH[@]}" "vyos@$1" "${@:2}" 2>/dev/null; }
+
+mac_of() { sudo virsh domiflist "$1" 2>/dev/null | awk '/52:54/{print $5; exit}'; }
+
+up() {
+    log "collecting node MACs (the reservation key, same as IPv4)"
+    local mappings="" name hextet mac
+    for entry in "${NODES[@]}"; do
+        name="${entry%%:*}"; hextet="${entry##*:}"
+        mac="$(mac_of "$name")"
+        [ -n "$mac" ] || die "no MAC for $name -- is the sim cluster up? (./k8s-up.sh)"
+        log "    $name $mac -> ${V6_PREFIX}::${hextet}"
+        # No trailing newline: `${mappings}` sits on its own line in the heredoc
+        # below and supplies its own. A heredoc terminator is matched LITERALLY
+        # in the source before any expansion, so `${mappings}EOF` is not the
+        # delimiter -- the heredoc ran on, swallowed the rest of this function
+        # and part of down(), and the apply failed with "Invalid command: [EOF]".
+        [ -n "$mappings" ] && mappings+=$'\n'
+        mappings+="set service dhcpv6-server shared-network-name K8S subnet ${V6_PREFIX}::/64 static-mapping ${name} mac '${mac}'
+set service dhcpv6-server shared-network-name K8S subnet ${V6_PREFIX}::/64 static-mapping ${name} ipv6-address '${V6_PREFIX}::${hextet}'"
+    done
+
+    local host pref hextet_self
+    for host in "$R1" "$R2"; do
+        if [ "$host" = "$R1" ]; then pref=high; hextet_self=1; else pref=low; hextet_self=2; fi
+        log "applying to $host (router ${V6_PREFIX}::${hextet_self}, RA preference $pref)"
+        vyos_apply "$host" <<EOF || die "commit failed on $host -- nothing applied there"
+set interfaces bonding bond0 vif 2 address '${V6_PREFIX}::${hextet_self}/64'
+set service router-advert interface bond0.2 prefix ${V6_PREFIX}::/64 valid-lifetime '2592000'
+set service router-advert interface bond0.2 prefix ${V6_PREFIX}::/64 preferred-lifetime '604800'
+set service router-advert interface bond0.2 prefix ${V6_PREFIX}::/64 no-autonomous-flag
+set service router-advert interface bond0.2 managed-flag
+set service router-advert interface bond0.2 link-mtu '${LINK_MTU}'
+set service router-advert interface bond0.2 default-preference '${pref}'
+set service dhcpv6-server shared-network-name K8S subnet ${V6_PREFIX}::/64 subnet-id '${VLAN_ID}'
+${mappings}
+EOF
+    done
+    log "up. Nodes need a DHCPv6 client on their VLAN 2 interface -- see 'status'."
+}
+
+down() {
+    local host
+    for host in "$R1" "$R2"; do
+        log "removing from $host"
+        vyos_apply "$host" <<EOF
+delete service dhcpv6-server
+delete service router-advert interface bond0.2
+delete interfaces bonding bond0 vif 2 address '${V6_PREFIX}::$([ "$host" = "$R1" ] && echo 1 || echo 2)/64'
+EOF
+    done
+    log "down"
+}
+
+status() {
+    local host
+    for host in "$R1" "$R2"; do
+        printf '  --- %s ---\n' "$host"
+        printf '    bond0.2 v6 : %s\n' "$(r "$host" 'ip -6 -br addr show bond0.2 | tr -s " "' || echo '<unreachable>')"
+        printf '    radvd      : %s\n' "$(r "$host" 'systemctl is-active radvd')"
+        # ps|grep, not pgrep: the bracket idiom that stops a self-match gets
+        # mangled through ssh -> vbash quoting and reported "NOT running" for a
+        # daemon that was plainly up.
+        printf '    kea-dhcp6  : %s\n' "$(r "$host" 'c=$(ps -ef | grep -c "[k]ea-dhcp6"); [ "$c" -gt 0 ] && echo running || echo "NOT running"')"
+        # radvd is expected INACTIVE on the backup: vrrp-wan-reconcile's IPv6
+        # plane stops it there so only the master advertises on any VLAN. Not a
+        # fault -- see the step-0 IPv6-follows-master work.
+        printf '    role       : %s\n' "$(r "$host" 'sudo /config/vrrp-wan-reconcile --status 2>/dev/null | grep -o "role=[a-z]*"')"
+    done
+    printf '  --- nodes ---\n'
+    local entry name
+    for entry in "${NODES[@]}"; do
+        name="${entry%%:*}"
+        printf '    %-14s %s\n' "$name" "$(node_v6 "$name")"
+    done
+}
+
+# What global IPv6 does the node actually hold? This is the question labctl asks
+# before it will write node-ip, so ask it the same way.
+node_v6() {
+    # Ask the node, not the hypervisor: virsh domifaddr --source agent needs
+    # qemu-guest-agent, which these images do not carry, and returned nothing
+    # while the nodes plainly had addresses.
+    local name="$1" v4 out
+    case "$name" in *k8s1) v4=172.31.2.11 ;; *k8s2) v4=172.31.2.12 ;; *k8s3) v4=172.31.2.13 ;; *) echo "<unknown>"; return ;; esac
+    out="$(timeout 20 ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
+             -o LogLevel=ERROR -o ConnectTimeout=6 -o BatchMode=yes "debian@$v4" \
+             'ip -6 -br addr show scope global 2>/dev/null | tr -s " "' 2>/dev/null)"
+    echo "${out:-<unreachable>}"
+}
+
+leases() {
+    local host
+    for host in "$R1" "$R2"; do
+        printf '  --- %s ---\n' "$host"
+        r "$host" '/opt/vyatta/bin/vyatta-op-cmd-wrapper show dhcpv6-server leases 2>/dev/null || echo "    (no leases command / no leases)"'
+    done
+}
+
+case "${1:-status}" in
+    up)     up ;;
+    down)   down ;;
+    status) status ;;
+    leases) leases ;;
+    *) die "usage: $0 {up|down|status|leases}" ;;
+esac

labsim/labsim-exporter.py

@@ -0,0 +1,157 @@
+#!/usr/bin/env python3
+"""Prometheus exporter for the labsim connectivity matrix.
+
+Runs the same sweep as labsim-matrix.py on an interval and exposes it as
+metrics, so Grafana can show the mesh as a heatmap and — more usefully — a
+history of exactly when a cell flipped after a firewall change.
+
+  labsim_reachable{src,dst,proto}   1 = reachable, 0 = blocked
+  labsim_sweep_seconds              how long the last sweep took
+  labsim_sweep_total                sweeps completed since start
+  labsim_up                         1 while the exporter is alive
+
+Deliberately stdlib-only (http.server + threads): this runs on the workstation
+next to libvirt, and adding a dependency to watch a lab network is silly.
+
+  ./labsim-exporter.py --port 9101 --interval 15
+"""
+from __future__ import annotations
+
+import argparse
+import http.server
+import json
+import os
+import threading
+import time
+
+import labsim_matrix_lib as m  # thin import shim, see below
+
+
+class Collector:
+    def __init__(self, interval: int, timeout: int) -> None:
+        self.interval = interval
+        self.timeout = timeout
+        self.vlans = m.load_vlans()
+        self.lock = threading.Lock()
+        self.results: dict = {}
+        self.duration = 0.0
+        self.sweeps = 0
+
+    def loop(self) -> None:
+        while True:
+            started = time.time()
+            try:
+                results = m.sweep(self.vlans, self.timeout)
+                with self.lock:
+                    self.results = results
+                    self.duration = time.time() - started
+                    self.sweeps += 1
+            except Exception:  # noqa: BLE001 - never let the loop die
+                pass
+            time.sleep(max(1.0, self.interval - (time.time() - started)))
+
+    def snapshot(self) -> dict:
+        """Everything the topology page needs, in one JSON payload."""
+        with self.lock:
+            results, duration = dict(self.results), self.duration
+        reach = total = 0
+        for data in results.values():
+            if "__error__" in data:
+                continue
+            for protos in data.values():
+                for proto, ok in protos.items():
+                    if proto == "rtt_ms":
+                        continue
+                    total += 1
+                    if ok:
+                        reach += 1
+        return {"vlans": self.vlans, "results": results, "reachable": reach,
+                "total": total, "sweep_seconds": duration}
+
+    def render(self) -> str:
+        with self.lock:
+            results, duration, sweeps = dict(self.results), self.duration, self.sweeps
+
+        out = [
+            "# HELP labsim_reachable 1 if dst is reachable from src over proto",
+            "# TYPE labsim_reachable gauge",
+        ]
+        rtts = []
+        for src, data in results.items():
+            if "__error__" in data:
+                continue
+            for dst, protos in data.items():
+                for proto, ok in protos.items():
+                    if proto == "rtt_ms":
+                        if isinstance(ok, (int, float)):
+                            rtts.append((src, dst, ok))
+                        continue
+                    out.append(
+                        f'labsim_reachable{{src="{src}",dst="{dst}",proto="{proto}"}} {1 if ok else 0}')
+        out += ["# HELP labsim_rtt_ms ICMP round-trip time",
+                "# TYPE labsim_rtt_ms gauge"]
+        for src, dst, val in rtts:
+            out.append(f'labsim_rtt_ms{{src="{src}",dst="{dst}"}} {val}')
+        out += [
+            "# HELP labsim_sweep_seconds duration of the last sweep",
+            "# TYPE labsim_sweep_seconds gauge",
+            f"labsim_sweep_seconds {duration:.3f}",
+            "# HELP labsim_sweep_total sweeps completed",
+            "# TYPE labsim_sweep_total counter",
+            f"labsim_sweep_total {sweeps}",
+            "# HELP labsim_up exporter liveness",
+            "# TYPE labsim_up gauge",
+            "labsim_up 1",
+        ]
+        return "\n".join(out) + "\n"
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--port", type=int, default=9101)
+    ap.add_argument("--interval", type=int, default=15)
+    ap.add_argument("--timeout", type=int, default=30)
+    args = ap.parse_args()
+
+    collector = Collector(args.interval, args.timeout)
+    threading.Thread(target=collector.loop, daemon=True).start()
+
+    here = os.path.dirname(os.path.abspath(__file__))
+
+    class Handler(http.server.BaseHTTPRequestHandler):
+        def _send(self, body: bytes, ctype: str) -> None:
+            self.send_response(200)
+            self.send_header("Content-Type", ctype)
+            self.send_header("Content-Length", str(len(body)))
+            self.send_header("Cache-Control", "no-store")
+            self.end_headers()
+            self.wfile.write(body)
+
+        def do_GET(self) -> None:  # noqa: N802 - stdlib API
+            path = self.path.split("?")[0].rstrip("/")
+            if path in ("", "/topology"):
+                # Live topology view — the thing you actually watch.
+                try:
+                    with open(os.path.join(here, "topology.html"), "rb") as fh:
+                        self._send(fh.read(), "text/html; charset=utf-8")
+                except OSError:
+                    self.send_error(500, "topology.html missing")
+            elif path == "/api/matrix":
+                self._send(json.dumps(collector.snapshot()).encode(), "application/json")
+            elif path == "/metrics":
+                self._send(collector.render().encode(), "text/plain; version=0.0.4")
+            else:
+                self.send_error(404)
+
+        def log_message(self, *_args) -> None:  # keep the console quiet
+            return
+
+    srv = http.server.ThreadingHTTPServer(("0.0.0.0", args.port), Handler)
+    print(f"labsim topology  http://localhost:{args.port}/")
+    print(f"labsim metrics   http://localhost:{args.port}/metrics  (sweep every {args.interval}s)")
+    srv.serve_forever()
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

labsim/labsim-he-endpoint.sh

@@ -0,0 +1,280 @@
+#!/bin/bash
+# Build a fake Hurricane Electric 6in4 endpoint inside labsim.
+#
+# WHY THIS EXISTS
+# The production HE tunnel was never rehearsed. The override that introduced it
+# says so in its own reason text -- "the sim has no public IPv4 and no HE
+# endpoint, so there is nothing to tunnel to" -- and that gap is why IPv6 was
+# the one half of the WAN story with no matrix behind it. Then the WAN became
+# HA and IPv6 did not follow, which nobody caught, because nothing tests it.
+#
+# THE PROBLEM THIS HAD TO SOLVE
+# The sim's two WANs are isolated islands. Verified:
+#
+#     203.0.113.1 from 203.0.113.107 : OK          <- 10 gig analogue
+#     203.0.113.1 from 198.51.100.137 : unreachable
+#     198.51.100.1 from 198.51.100.137 : OK        <- PPPoE analogue
+#     198.51.100.1 from 203.0.113.107 : unreachable
+#
+# A 6in4 tunnel has ONE remote address, and production never changes it -- so an
+# endpoint reachable over only one WAN could not rehearse the case that matters
+# most: the WITHIN-box fall back from the 10 gig to PPPoE, where he-tunnel-follow
+# re-points the tunnel and calls the HE API. That is exactly where the 2026-09-06
+# near-miss lived.
+#
+# So the sim needs a minimal "internet": both ISP boxes already sit on the
+# libvirt default network (192.168.122.0/24) and both forward, so that becomes
+# the backbone, and HE lives on a single address behind it, reachable over
+# either WAN. No new VMs, no new networks.
+#
+#     192.0.2.10  "HE"  -- on isp-dhcp, reached from the PPPoE island via
+#                          192.168.122.136, and directly from the 10 gig island
+#
+# EVERYTHING HERE IS KERNEL-LEVEL, not VyOS config. The ISP boxes are scaffold,
+# not the thing under test: `ip` commands leave no config to drift, no commit to
+# fail, and a reboot cleans up. The ROUTER side is deliberately the opposite --
+# it goes through real VyOS config, because "will VyOS commit a tunnel whose
+# source-address does not exist on this box?" is one of the questions.
+#
+#   ./labsim-he-endpoint.sh up        build it
+#   ./labsim-he-endpoint.sh down      tear it down
+#   ./labsim-he-endpoint.sh status    what is live
+#   ./labsim-he-endpoint.sh calls     how many times the HE API was called
+#   ./labsim-he-endpoint.sh point IP  point the endpoint by hand (sim bookkeeping)
+set -uo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+DHCP_ISP="${DHCP_ISP:-192.168.122.136}"   # owns the 10 gig segment, hosts "HE"
+PPPOE_ISP="${PPPOE_ISP:-192.168.122.63}"  # owns the PPPoE segment
+PW="${VYOS_PW:-vyos}"
+
+# TEST-NET-1 for the endpoint, and the documentation prefix for v6. Production
+# uses 2001:470:187e::/48 from HE; the sim mirrors its SHAPE
+# (2001:db8:187e:<vlan>::/64) so the scheme is exercised, not just the tunnel.
+HE_ADDR="${HE_ADDR:-192.0.2.10}"
+HE_LINK6="${HE_LINK6:-2001:db8:1f1c:f6::1}"   # HE side of the tunnel /64
+RT_LINK6="${RT_LINK6:-2001:db8:1f1c:f6::2}"   # router side
+SITE6="${SITE6:-2001:db8:187e::/48}"          # routed to the router side
+TENGIG_NET="${TENGIG_NET:-203.0.113.0/24}"
+# Any valid address that will never be a router WAN -- see the tunnel creation
+# below for why this cannot be 0.0.0.0.
+PLACEHOLDER_REMOTE="${PLACEHOLDER_REMOTE:-203.0.113.1}"
+PPPOE_NET="${PPPOE_NET:-198.51.100.0/24}"
+
+SSH=(-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null
+     -o LogLevel=ERROR -o ConnectTimeout=6 -o PreferredAuthentications=password)
+dhcp_isp()  { timeout 40 sshpass -p "$PW" ssh "${SSH[@]}" "vyos@$DHCP_ISP" "$@" 2>/dev/null; }
+pppoe_isp() { timeout 40 sshpass -p "$PW" ssh "${SSH[@]}" "vyos@$PPPOE_ISP" "$@" 2>/dev/null; }
+
+log()  { printf '\033[0;36m[he-sim]\033[0m %s\n' "$*"; }
+die()  { printf '\033[0;31m[he-sim]\033[0m %s\n' "$*" >&2; exit 1; }
+
+# --- the stub tunnelbroker API ---------------------------------------------
+# HE's real endpoint is a dyndns-style updater that re-points the tunnel's remote
+# address. This is that, in 40 lines, so the sim can exercise the HE-SIDE half of
+# a failover -- the half production can never safely test.
+#
+# It logs every call to /run/he-sim-api.log, which is what lets the matrix assert
+# the invariant that matters: a ROUTER-level failover must call this ZERO times,
+# because the 10 gig address follows the cloned MAC to the other box unchanged.
+API_PY='
+import http.server, subprocess, urllib.parse, datetime, sys, re
+
+TUN  = "he-sim"
+LOCAL    = sys.argv[1] if len(sys.argv) > 1 else "192.0.2.10"
+V6_LOCAL = sys.argv[2] if len(sys.argv) > 2 else "2001:db8:1f1c:f6::1/64"
+V6_PEER  = sys.argv[3] if len(sys.argv) > 3 else "2001:db8:1f1c:f6::2"
+SITE6    = sys.argv[4] if len(sys.argv) > 4 else "2001:db8:187e::/48"
+LOG  = "/run/he-sim-api.log"
+
+def note(msg):
+    with open(LOG, "a") as f:
+        f.write("%s %s\n" % (datetime.datetime.now().isoformat(timespec="seconds"), msg))
+
+def current_remote():
+    out = subprocess.run(["ip", "tunnel", "show", TUN], capture_output=True, text=True).stdout
+    m = re.search(r"remote ([0-9.]+)", out)
+    return m.group(1) if m else None
+
+class H(http.server.BaseHTTPRequestHandler):
+    def reply(self, body):
+        b = body.encode()
+        self.send_response(200)
+        self.send_header("Content-Type", "text/plain")
+        self.send_header("Content-Length", str(len(b)))
+        self.end_headers()
+        self.wfile.write(b)
+
+    def handle_update(self, qs):
+        q = urllib.parse.parse_qs(qs)
+        ip = (q.get("myip") or [""])[0]
+        if not ip:
+            note("REFUSED no myip"); return self.reply("nohost")
+        cur = current_remote()
+        if cur == ip:
+            note("nochg %s" % ip); return self.reply("nochg %s" % ip)
+        rc = subprocess.run(["ip", "tunnel", "change", TUN, "mode", "sit",
+                             "local", LOCAL, "remote", ip],
+                            capture_output=True, text=True)
+        if rc.returncode != 0:
+            # Recreate rather than report a success we did not achieve. This is
+            # the path that a multipoint tunnel takes; keeping it means a stub
+            # that cannot silently no-op.
+            note("change failed (%s) -- recreating" % rc.stderr.strip())
+            subprocess.run(["ip", "tunnel", "del", TUN], check=False)
+            subprocess.run(["ip", "tunnel", "add", TUN, "mode", "sit",
+                            "local", LOCAL, "remote", ip, "ttl", "64"], check=False)
+            subprocess.run(["ip", "link", "set", TUN, "up", "mtu", "1480"], check=False)
+            subprocess.run(["ip", "-6", "addr", "replace", V6_LOCAL, "dev", TUN], check=False)
+            subprocess.run(["ip", "-6", "route", "replace", SITE6, "via", V6_PEER,
+                            "dev", TUN], check=False)
+        # Verify rather than trust: read the remote back.
+        got = current_remote()
+        if got != ip:
+            note("FAILED to point %s at %s (reads %s)" % (TUN, ip, got))
+            return self.reply("dnserr")
+        note("good %s (was %s)" % (ip, cur))
+        self.reply("good %s" % ip)
+
+    def do_GET(self):
+        u = urllib.parse.urlparse(self.path)
+        if u.path == "/nic/update": self.handle_update(u.query)
+        else: self.reply("badauth")
+
+    def do_POST(self):
+        n = int(self.headers.get("Content-Length") or 0)
+        self.handle_update(self.rfile.read(n).decode())
+
+    def log_message(self, *a): pass
+
+http.server.HTTPServer(("0.0.0.0", 80), H).serve_forever()
+'
+
+up() {
+    log "backbone: teaching each ISP box how to reach the other island"
+    # isp-dhcp owns HE and must be able to answer a router that arrived over
+    # PPPoE, so it needs a route back to that island via the backbone.
+    dhcp_isp "sudo ip route replace $PPPOE_NET via $PPPOE_ISP" \
+        || die "could not add the PPPoE-island route on isp-dhcp"
+    # isp-pppoe must forward its clients' traffic for HE across the backbone.
+    pppoe_isp "sudo ip route replace $HE_ADDR/32 via $DHCP_ISP" \
+        || die "could not add the HE route on isp-pppoe"
+
+    log "HE endpoint: $HE_ADDR on isp-dhcp"
+    # A dummy interface, not a loopback alias: `ip tunnel` wants a real local
+    # address and a dummy is the honest way to have one that is not tied to
+    # either WAN segment -- which is the point, HE is neither.
+    dhcp_isp "sudo modprobe dummy 2>/dev/null;
+              sudo ip link add he-lo type dummy 2>/dev/null;
+              sudo ip link set he-lo up;
+              sudo ip addr replace $HE_ADDR/32 dev he-lo"
+
+    log "6in4 tunnel he-sim: local $HE_ADDR, remote set by the API on demand"
+    # A PLACEHOLDER remote, not 0.0.0.0. A sit tunnel created with `remote any`
+    # is multipoint (6rd-shaped), and `ip tunnel change` then refuses to convert
+    # it to point-to-point -- "add tunnel he-sim failed: Invalid argument". The
+    # API's update silently did nothing, so the stub logged "good", production's
+    # he-tunnel-follow logged success, and the tunnel still pointed nowhere.
+    # Created point-to-point from the start, `change` works.
+    dhcp_isp "sudo ip tunnel del he-sim 2>/dev/null;
+              sudo ip tunnel add he-sim mode sit local $HE_ADDR remote $PLACEHOLDER_REMOTE ttl 64;
+              sudo ip link set he-sim up mtu 1480;
+              sudo ip -6 addr replace $HE_LINK6/64 dev he-sim;
+              sudo ip -6 route replace $SITE6 via $RT_LINK6 dev he-sim;
+              sudo sysctl -qw net.ipv6.conf.all.forwarding=1"
+
+    log "stub tunnelbroker API on $HE_ADDR:80"
+    printf '%s' "$API_PY" | dhcp_isp "cat > /tmp/he-sim-api.py"
+    # Launch from a script FILE, not an inline ssh command. The remote login
+    # shell is vbash, and a multi-line inlined `sudo setsid nohup ... &` through
+    # it silently ran nothing at all: no process, no /run/he-sim-api.out, and a
+    # `pgrep -f he-sim-api.py` status check that reported "running" because the
+    # unbracketed pattern matched its OWN ssh command line. Two self-inflicted
+    # illusions stacked on each other.
+    #
+    # This repo already learned this once -- see isp_session_control() in
+    # labsim-pppoe-ha-test.sh, where driving vbash inline made every iteration
+    # of the T4 matrix test the wrong policy while printing the right one.
+    printf '%s\n' \
+        '#!/bin/sh' \
+        '# started detached so it outlives the ssh session that launched it' \
+        'pkill -f "he-sim-api[.]py" 2>/dev/null' \
+        'rm -f /run/he-sim-api.log /run/he-sim-api.out' \
+        "exec setsid python3 /tmp/he-sim-api.py $HE_ADDR '$HE_LINK6/64' $RT_LINK6 $SITE6 >/run/he-sim-api.out 2>&1 </dev/null &" \
+        | dhcp_isp "cat > /tmp/he-sim-start.sh"
+    dhcp_isp "chmod +x /tmp/he-sim-start.sh && sudo /tmp/he-sim-start.sh" >/dev/null
+    # Poll for the bind rather than sleeping a guessed interval.
+    local i probe=""
+    for i in $(seq 1 10); do
+        sleep 1
+        probe="$(dhcp_isp "curl -sS --max-time 3 'http://$HE_ADDR/nic/update' 2>&1")"
+        [ "$probe" = nohost ] && break
+    done
+    case "$probe" in
+        nohost) log "API answering (returned 'nohost' for a call with no myip -- correct)" ;;
+        *)      die "stub API not answering on $HE_ADDR:80 (got: ${probe:-<nothing>})" ;;
+    esac
+
+    log "up. Router side is NOT configured by this script -- that is real VyOS"
+    log "config and belongs to the matrix; see labsim-ipv6-ha-test.sh --setup."
+}
+
+down() {
+    log "tearing down"
+    dhcp_isp "sudo pkill -f 'he-sim-api[.]py' 2>/dev/null;
+              sudo ip tunnel del he-sim 2>/dev/null;
+              sudo ip link del he-lo 2>/dev/null;
+              sudo ip route del $PPPOE_NET via $PPPOE_ISP 2>/dev/null" >/dev/null
+    pppoe_isp "sudo ip route del $HE_ADDR/32 via $DHCP_ISP 2>/dev/null" >/dev/null
+    log "down"
+}
+
+status() {
+    printf '  HE address : %s\n' "$(dhcp_isp "ip -4 -br addr show he-lo 2>/dev/null | awk '{print \$3}'" || echo '<absent>')"
+    printf '  he-sim     : %s\n' "$(dhcp_isp "ip tunnel show he-sim 2>/dev/null" || echo '<absent>')"
+    printf '  he-sim v6  : %s\n' "$(dhcp_isp "ip -6 -br addr show he-sim 2>/dev/null | awk '{print \$3}'" || echo '-')"
+    # Bracketed so the pattern cannot match the ssh command line carrying it --
+    # unbracketed, this reported "running" while nothing was listening at all.
+    printf '  API        : %s\n' "$(dhcp_isp "pgrep -f 'he-sim-api[.]py' >/dev/null && echo running || echo stopped")"
+    printf '  API bound  : %s\n' "$(dhcp_isp "curl -sS --max-time 3 'http://$HE_ADDR/nic/update' 2>/dev/null" || echo 'NOT ANSWERING')"
+    printf '  API calls  : %s\n' "$(dhcp_isp "grep -c . /run/he-sim-api.log 2>/dev/null" || echo 0)"
+    printf '  route back : %s\n' "$(dhcp_isp "ip route show $PPPOE_NET 2>/dev/null" || echo '<none>')"
+}
+
+# Count of endpoint-CHANGING calls. `nochg` does not count: production's
+# he-tunnel-follow re-sends the same address happily and HE treats it as a
+# no-op, so only a real move is evidence that something re-pointed the tunnel.
+calls() { dhcp_isp "grep -c ' good ' /run/he-sim-api.log 2>/dev/null" | tr -d ' \n'; }
+
+# Point the endpoint at an address WITHOUT going through the API, and without
+# counting as an API call.
+#
+# Needed because rebuilding the sim endpoint resets its remote, while the
+# routers' he-tunnel-follow still reads "in sync" and therefore never re-asserts
+# -- it only calls HE when its own LOCAL source changes, and has no way to learn
+# that the far end drifted. The real HE does not forget, so this is sim
+# bookkeeping, not a behaviour production needs. Keeping it out of the call
+# counter is the point: the matrix asserts on that counter.
+point() {
+    local ip="$1"
+    [ -n "$ip" ] || die "usage: $0 point <ipv4>"
+    dhcp_isp "sudo ip tunnel change he-sim mode sit local $HE_ADDR remote $ip 2>/dev/null \
+              || { sudo ip tunnel del he-sim 2>/dev/null;
+                   sudo ip tunnel add he-sim mode sit local $HE_ADDR remote $ip ttl 64;
+                   sudo ip link set he-sim up mtu 1480;
+                   sudo ip -6 addr replace $HE_LINK6/64 dev he-sim;
+                   sudo ip -6 route replace $SITE6 via $RT_LINK6 dev he-sim; }"
+    local got
+    got="$(dhcp_isp "ip tunnel show he-sim 2>/dev/null | sed -nE 's/.* remote ([0-9.]+).*/\\1/p'" | tr -d ' \n')"
+    [ "$got" = "$ip" ] || die "endpoint still points at ${got:-nothing}, wanted $ip"
+    log "endpoint now points at $ip"
+}
+
+case "${1:-status}" in
+    up)     up ;;
+    down)   down ;;
+    status) status ;;
+    calls)  calls; echo ;;
+    point)  point "${2:-}" ;;
+    *) die "usage: $0 {up|down|status|calls}" ;;
+esac

labsim/labsim-ipv6-ha-test.sh

@@ -0,0 +1,430 @@
+#!/bin/bash
+# Does IPv6 follow VRRP mastership, and does it do so WITHOUT touching HE?
+#
+# The WAN became HA on 2026-09-06 and IPv6 did not follow it. Nothing caught
+# that, because nothing tested it: wan-drill measured IPv4 only, and PPPOE-HA.md
+# recorded "IPv6 stayed up at 15.5ms" from a reading taken outside the failover
+# window. This is the matrix that would have caught it.
+#
+# TWO INVARIANTS, checked independently of any individual test:
+#
+#   1. At most ONE router ever has a live tunnel. An UP tunnel on a box that
+#      does not own the source address is not harmless -- see V4, it is a
+#      BLACKHOLE that will happily attract the v6 default route.
+#   2. A ROUTER-level failover calls the HE API ZERO times. The 10 gig address
+#      is bound to a cloned MAC and follows the VIP to the other box unchanged,
+#      so there is nothing to tell HE. A non-zero count means something
+#      re-pointed the tunnel at a PPPoE address -- which the ISP re-issues on
+#      every dial, so it would be wrong within minutes.
+#
+# KNOWN SIM GAP, 2026-09-06 -- read this before believing a v6_online failure.
+# The MECHANISM is proven here: tun0 up on the master and down on the backup,
+# radvd following mastership, he-tunnel-follow's master guard, its hysteresis,
+# its HE call and the 1480->1472 MTU switch, and VLAN 9 hosts autoconfiguring
+# from the RA (observed: real SLAAC traffic from 2001:db8:187e:9::/64 arriving
+# at the endpoint encapsulated).
+#
+# What is NOT yet proven is the end-to-end v6 DATAPATH, because the sim's
+# "internet" is asymmetric: 6in4 packets from the PPPoE island reach the
+# endpoint with an outer source of 192.168.122.1 -- the libvirt host's NAT --
+# so HE's replies go back to the tunnel remote by a path with no NAT state and
+# are lost. Forward works, return does not.
+#
+# That is a topology fault in the scaffold, not in the thing under test. Fixing
+# it means giving the two ISP islands a real transit path that does not traverse
+# libvirt NAT. Until then, treat v6_online failures as UNPROVEN rather than as
+# evidence the design is wrong -- and do not let that ambiguity leak into
+# production sign-off, which is exactly the mistake wan-drill made by asserting
+# "IPv6 stayed up" from a reading taken outside the window.
+#
+# Requires the fake HE endpoint: ./labsim-he-endpoint.sh up
+#
+#   ./labsim-ipv6-ha-test.sh --setup    configure the router side (once)
+#   ./labsim-ipv6-ha-test.sh --list
+#   ./labsim-ipv6-ha-test.sh V1
+#   ./labsim-ipv6-ha-test.sh --all
+set -uo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+R1="${R1:-172.31.1.252}"; R2="${R2:-172.31.1.253}"
+VIP="${VIP:-172.31.1.1}"
+LAN9="${LAN9:-172.31.9.10}"                  # VLAN 9 client, for RA tests
+HE_ADDR="${HE_ADDR:-192.0.2.10}"
+HE_LINK6="${HE_LINK6:-2001:db8:1f1c:f6::1}"
+RT_LINK6="${RT_LINK6:-2001:db8:1f1c:f6::2}"
+V9_PREFIX="${V9_PREFIX:-2001:db8:187e:9}"
+# Mirrors production: one MAC, one lease, whichever router holds the VIP.
+WAN_MAC="${WAN_MAC:-02:9f:c2:12:9b:4f}"
+PW="${VYOS_PW:-vyos}"; LANPW="${LANPW:-labsim}"
+EVID="$SCRIPT_DIR/ipv6-ha-evidence"
+HE="$SCRIPT_DIR/labsim-he-endpoint.sh"
+
+SSH=(-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null
+     -o LogLevel=ERROR -o ConnectTimeout=6 -o PreferredAuthentications=password)
+r() { timeout 45 sshpass -p "$PW" ssh "${SSH[@]}" "vyos@$1" "${@:2}" 2>/dev/null; }
+# The Alpine LAN VMs do not offer `password` auth -- reusing the routers' option
+# set makes ssh exit 255 before running anything, which reads as "the network is
+# broken". Same trap as lan() in labsim-pppoe-ha-test.sh.
+LAN_SSH=(-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null
+         -o LogLevel=ERROR -o ConnectTimeout=6)
+lan9() { timeout 30 sshpass -p "$LANPW" ssh "${LAN_SSH[@]}" "root@$LAN9" "$@" 2>/dev/null; }
+
+log()  { printf '\033[36m==>\033[0m %s\n' "$*"; }
+pass() { printf '  \033[32mPASS\033[0m %s\n' "$*"; }
+fail() { printf '  \033[31mFAIL\033[0m %s\n' "$*"; FAILED=$((FAILED+1)); }
+warn() { printf '  \033[33mWARN\033[0m %s\n' "$*"; }
+FAILED=0
+
+# --- observations ----------------------------------------------------------
+# A destroyed or unreachable router is emphatically NOT holding the tunnel, but
+# ssh returns an EMPTY string, and `[ "" = 0 ]` is false -- the pppoe matrix hung
+# on exactly this waiting for a dead box to report zero. Default everything to 0.
+tun_state() { local v; v="$(r "$1" "ip -br link show tun0 2>/dev/null | awk '{print \$2}'" | tr -d ' \n')"; echo "${v:-absent}"; }
+tun_src()   { r "$1" 'ip tunnel show tun0 2>/dev/null | sed -nE "s/.* local ([0-9.]+).*/\1/p"' | tr -d ' \n'; }
+# Whichever router currently holds the 10 gig lease -- under the cloned MAC only
+# one ever does. Read rather than assumed: the address changes when the MAC does.
+tengig_addr() { local h a; for h in "$R1" "$R2"; do
+                  a="$(r "$h" 'ip -4 addr show bond0.53 2>/dev/null | sed -nE "s/.*inet ([0-9.]+).*/\1/p"' | tr -d ' \n')"
+                  [ -n "$a" ] && { echo "$a"; return; }
+                done; echo ""; }
+tun_mtu()   { local v; v="$(r "$1" 'cat /sys/class/net/tun0/mtu 2>/dev/null' | tr -d ' \n')"; echo "${v:-0}"; }
+holder()    { for h in "$R1" "$R2"; do
+                [ "$(r "$h" "ip -4 -o addr show | grep -c ' ${VIP}/'" | tr -d ' \n')" != 0 ] \
+                  && { echo "$h"; return; }; done; echo none; }
+# Ask the ROUTER, not a client: a client can be answered by the wrong path.
+v6_online() { [ "$(r "$1" "ping -6 -c1 -W3 $HE_LINK6 >/dev/null 2>&1 && echo y" | tr -d ' \n')" = y ]; }
+radvd_on()  { [ "$(r "$1" 'systemctl is-active radvd 2>/dev/null' | tr -d ' \n')" = active ]; }
+he_calls()  { timeout 60 "$HE" calls | tr -d ' \n'; }
+
+# How many routers have a tunnel that is UP. Invariant 1's numerator.
+tun_holders() { local n=0 h; for h in "$R1" "$R2"; do
+                  case "$(tun_state "$h")" in UP|UNKNOWN) n=$((n+1)) ;; esac
+                done; echo "$n"; }
+
+check_invariants() {
+    local t ok=0
+    t="$(tun_holders)"
+    [ "${t:-0}" -le 1 ] || { fail "INVARIANT: $t routers have tun0 up"; ok=1; }
+    return $ok
+}
+
+save_evidence() {
+    local name="$1"; local d="$EVID/$name"; mkdir -p "$d"
+    { echo "=== $(date -Is) ==="
+      echo "--- HE endpoint ---"; timeout 60 "$HE" status
+      for h in "$R1" "$R2"; do echo "--- $h ---"
+        r "$h" 'sudo /config/vrrp-wan-reconcile --status 2>/dev/null
+                ip -br link show tun0 2>/dev/null; ip tunnel show tun0 2>/dev/null
+                ip -6 route show default; systemctl is-active radvd
+                sudo journalctl -t vrrp-wan -t he-tunnel-follow -n 10 --no-pager'
+      done; } > "$d/state.txt" 2>&1
+    log "evidence -> ipv6-ha-evidence/$name/"
+}
+
+# --- setup ------------------------------------------------------------------
+# Applied through REAL VyOS config, unlike the ISP-side scaffold, because "will
+# VyOS accept this?" is one of the questions being asked.
+vyos_apply() {   # host, then set-lines on stdin
+    local h="$1"
+    { printf '#!/bin/vbash\nsource /opt/vyatta/etc/functions/script-template\nconfigure\n'
+      cat
+      printf 'commit\nsave\nexit\n'
+    } | timeout 90 sshpass -p "$PW" ssh "${SSH[@]}" "vyos@$h" \
+          'cat > /tmp/v6-apply.sh && chmod +x /tmp/v6-apply.sh && sudo /tmp/v6-apply.sh' 2>&1 | tail -3
+    # `vbash -c` never starts a config session and commit fails to stderr, which
+    # a helper like this discards -- the T4 matrix in labsim-pppoe-ha-test.sh ran
+    # its whole policy sweep against the default while printing the mode it
+    # thought it was testing. Always read the value back.
+}
+
+setup() {
+    log "--setup: router-side IPv6, both routers"
+
+    # THE CLONED MAC, first and on its own. Production pins f0:9f:c2:12:9b:4f on
+    # vif 53 so the 10 gig lease follows the VIP and the tunnel source is the
+    # SAME address on either box. The sim never had it -- each router took its
+    # own lease -- so the sim could not reproduce the one property the whole
+    # IPv6-HA design leans on, and invariant 2 would have been untestable here.
+    local h pref v9
+    for h in "$R1" "$R2"; do
+        log "    $h: pinning the cloned WAN MAC $WAN_MAC"
+        vyos_apply "$h" <<EOF
+set interfaces bonding bond0 vif 53 mac '$WAN_MAC'
+EOF
+    done
+
+    # A new MAC means a NEW lease, so the tunnel source cannot be hardcoded --
+    # discover it. Hardcoding the pre-change address here would have configured
+    # every tunnel with a source neither router owns, i.e. the V4 blackhole, on
+    # both boxes, while the matrix reported setup success.
+    local src="" i
+    for i in $(seq 1 30); do
+        src="$(tengig_addr)"; [ -n "$src" ] && break
+        sleep 5
+    done
+    [ -n "$src" ] || { fail "no router took a 10 gig lease after the MAC change -- cannot set a tunnel source"; return 1; }
+    log "    10 gig lease under the cloned MAC: $src"
+
+    # Credentials pointing at the stub. HE_UPDATE_URL is read AFTER the secrets
+    # file is sourced, so putting it here overrides the production default
+    # without the script needing a sim-specific branch.
+    for h in "$R1" "$R2"; do
+        printf 'HE_USER=sim\nHE_UPDATE_KEY=sim\nHE_TUNNEL_ID=1\nHE_UPDATE_URL=http://%s/nic/update\n' "$HE_ADDR" \
+          | timeout 30 sshpass -p "$PW" ssh "${SSH[@]}" "vyos@$h" \
+              'cat > /tmp/he-secrets && sudo install -o root -g vyattacfg -m 0640 /tmp/he-secrets /config/he-secrets' >/dev/null
+    done
+
+    for h in "$R1" "$R2"; do
+        [ "$h" = "$R1" ] && { pref=high; v9=1; } || { pref=low; v9=2; }
+        log "    $h (RA preference $pref, bond0.9 ::${v9})"
+        vyos_apply "$h" <<EOF
+set interfaces tunnel tun0 encapsulation 'sit'
+set interfaces tunnel tun0 source-address '$src'
+set interfaces tunnel tun0 remote '$HE_ADDR'
+set interfaces tunnel tun0 address '$RT_LINK6/64'
+set interfaces tunnel tun0 mtu '1480'
+set protocols static route6 ::/0 next-hop '$HE_LINK6'
+set interfaces bonding bond0 vif 9 address '${V9_PREFIX}::${v9}/64'
+set service router-advert interface bond0.9 prefix ${V9_PREFIX}::/64 preferred-lifetime '604800'
+set service router-advert interface bond0.9 link-mtu '1472'
+set service router-advert interface bond0.9 default-preference '$pref'
+set system task-scheduler task he-tunnel-follow executable path '/config/he-tunnel-follow'
+set system task-scheduler task he-tunnel-follow executable arguments 'run'
+set system task-scheduler task he-tunnel-follow interval '1m'
+EOF
+    done
+    # RA link-mtu is 1472, the PPPoE figure, on BOTH -- deliberately not 1480.
+    # It cannot be reconciled at runtime (it needs a commit, and the tunnel plane
+    # is commit-free on purpose), so advertise the lower of the two paths and be
+    # correct on either WAN. Production pinned 1480 and was wrong whenever the
+    # WAN fell back.
+    log "    setup done -- run V0 to check the baseline"
+}
+
+# --- tests ------------------------------------------------------------------
+V0() {  # baseline
+    log "V0  baseline: the VIP holder owns the tunnel, the backup does not"
+    local h o; h="$(holder)"; o=$([ "$h" = "$R1" ] && echo "$R2" || echo "$R1")
+    [ "$h" = none ] && { fail "no VIP holder"; return; }
+    case "$(tun_state "$h")" in UP|UNKNOWN) pass "master $h has tun0 up" ;;
+        *) fail "master $h tun0 is $(tun_state "$h")" ;; esac
+    case "$(tun_state "$o")" in DOWN|absent) pass "backup $o tun0 is $(tun_state "$o")" ;;
+        *) fail "backup $o tun0 is $(tun_state "$o") -- it should be held down" ;; esac
+    v6_online "$h" && pass "master reaches HE over v6" || fail "master has no IPv6"
+    radvd_on "$h" && pass "master is advertising on VLAN 9" || fail "master radvd not running"
+    radvd_on "$o" && fail "backup is ALSO advertising -- two default routers on VLAN 9" \
+                  || pass "backup is not advertising"
+    check_invariants
+    save_evidence V0-baseline
+}
+
+V1() {  # clean failover: v6 follows, and HE is never called
+    log "V1  clean failover: IPv6 follows, HE API untouched"
+    local from to t0 before after i
+    from="$(holder)"; to=$([ "$from" = "$R1" ] && echo "$R2" || echo "$R1")
+    before="$(he_calls)"
+    log "    master=$from -> expecting $to (HE calls so far: ${before:-0})"
+    t0=$(date +%s)
+    r "$from" 'sudo mkdir -p /run/vrrp-wan && sudo touch /run/vrrp-wan/force-fault'
+    local took=""
+    for i in $(seq 1 36); do
+        sleep 5
+        [ "$(holder)" = "$to" ] && v6_online "$to" && { took=$(( $(date +%s) - t0 )); break; }
+    done
+    [ -n "$took" ] && pass "IPv6 reached $to in ${took}s" \
+                   || fail "IPv6 never followed to $to within 180s"
+    case "$(tun_state "$from")" in DOWN|absent) pass "$from released its tunnel" ;;
+        *) fail "$from still has tun0 $(tun_state "$from") -- blackhole risk" ;; esac
+    after="$(he_calls)"
+    # THE invariant this test exists for.
+    [ "${after:-0}" = "${before:-0}" ] \
+        && pass "HE API not called (${after:-0} total) -- the address followed the MAC" \
+        || fail "HE API called $(( ${after:-0} - ${before:-0} )) time(s) during a ROUTER failover"
+    check_invariants
+    save_evidence V1-clean-failover
+    r "$from" 'sudo rm -f /run/vrrp-wan/force-fault'
+    sleep 40
+}
+
+V2() {  # 10 gig down on the master: HE must be told, exactly once
+    log "V2  10 gig down: he-tunnel-follow re-points the tunnel and tells HE"
+    local h before after mtu i ok=no
+    h="$(holder)"; before="$(he_calls)"
+    local tengig; tengig="$(tengig_addr)"
+    r "$h" 'sudo ip link set bond0.53 down'
+    # he-tunnel-follow runs on a 1m task-scheduler with a 2-tick hysteresis, so
+    # allow well past 2 minutes before calling it a failure.
+    for i in $(seq 1 30); do
+        sleep 10
+        [ "$(tun_src "$h")" != "$tengig" ] && { ok=yes; break; }
+    done
+    [ "$ok" = yes ] && pass "tunnel source moved to $(tun_src "$h") after $((i*10))s" \
+                    || fail "tunnel source never left the 10 gig address"
+    mtu="$(tun_mtu "$h")"
+    [ "$mtu" = 1472 ] && pass "MTU dropped to 1472 for the PPPoE path" \
+                      || fail "MTU is $mtu, want 1472 -- large transfers will hang"
+    after="$(he_calls)"
+    [ "$(( ${after:-0} - ${before:-0} ))" -ge 1 ] \
+        && pass "HE API called $(( ${after:-0} - ${before:-0} )) time(s), as it must be here" \
+        || fail "HE was never told -- it still points at an address this box no longer has"
+    v6_online "$h" && pass "IPv6 still up over PPPoE" || fail "IPv6 down on the PPPoE path"
+    save_evidence V2-tengig-down
+    r "$h" 'sudo ip link set bond0.53 up'
+    sleep 60
+}
+
+V3() {  # a cold backup must not advertise, dial, or blackhole
+    log "V3  cold backup: no tunnel, no RA, no HE call"
+    local h o before after
+    h="$(holder)"; o=$([ "$h" = "$R1" ] && echo "$R2" || echo "$R1")
+    before="$(he_calls)"
+    r "$o" 'sudo systemctl restart vrrp-wan-reconcile.service' >/dev/null
+    sleep 20
+    case "$(tun_state "$o")" in DOWN|absent) pass "backup tunnel stays $(tun_state "$o")" ;;
+        *) fail "backup brought tun0 up while not holding the VIP" ;; esac
+    radvd_on "$o" && fail "backup is advertising on VLAN 9" || pass "backup is silent on VLAN 9"
+    after="$(he_calls)"
+    [ "${after:-0}" = "${before:-0}" ] && pass "backup made no HE call" \
+        || fail "the BACKUP called the HE API -- it would point HE at its own idle line"
+    save_evidence V3-cold-backup
+}
+
+V4() {  # the assumption the production override was built on
+    log "V4  a tunnel whose source-address is absent: what does VyOS actually do?"
+    # The production override says such a tunnel "would simply stay down", and
+    # treats that as the reason it was safe to leave IPv6 single-homed. Measured
+    # on the sim backup 2026-09-06: the commit SUCCEEDS and the link comes up
+    # anyway -- it is a blackhole, not an inert node. That is why the runtime
+    # gate is load-bearing rather than a nicety, exactly like the PPPoE gate.
+    local o h; h="$(holder)"; o=$([ "$h" = "$R1" ] && echo "$R2" || echo "$R1")
+    r "$o" 'sudo ip link set tun0 up' >/dev/null; sleep 3
+    case "$(tun_state "$o")" in
+        UP|UNKNOWN) pass "confirmed: VyOS leaves it UP with no source address (blackhole)" ;;
+        *) warn "this VyOS version keeps it $(tun_state "$o") -- the override's assumption holds here; re-check the production version before relying on it" ;;
+    esac
+    v6_online "$o" && fail "the backup somehow reached HE -- two live tunnels" \
+                   || pass "and it carries nothing, as expected"
+    # Hand it straight back to the reconciler rather than leaving it up.
+    r "$o" 'sudo systemctl restart vrrp-wan-reconcile.service' >/dev/null
+    sleep 15
+    case "$(tun_state "$o")" in DOWN|absent) pass "the reconciler put it back down" ;;
+        *) fail "the reconciler did NOT re-close the gate -- this is the load-bearing bit" ;; esac
+    save_evidence V4-absent-source-address
+}
+
+V5() {  # never two live tunnels, even mid-transition
+    log "V5  both routers momentarily master: never two live tunnels"
+    local from to i worst=0 n
+    from="$(holder)"; to=$([ "$from" = "$R1" ] && echo "$R2" || echo "$R1")
+    r "$from" 'sudo mkdir -p /run/vrrp-wan && sudo touch /run/vrrp-wan/force-fault'
+    # Sample THROUGH the transition rather than at the ends. The interesting
+    # window is the one where both boxes briefly think they are in charge.
+    for i in $(seq 1 24); do
+        n="$(tun_holders)"; [ "${n:-0}" -gt "$worst" ] && worst="$n"
+        sleep 5
+    done
+    [ "$worst" -le 1 ] && pass "at most $worst live tunnel throughout the transition" \
+                       || fail "saw $worst live tunnels at once -- HE would receive two claimants"
+    r "$from" 'sudo rm -f /run/vrrp-wan/force-fault'
+    save_evidence V5-transition-invariant
+    sleep 40
+}
+
+V6() {  # RA deprecation: does a VLAN 9 host drop the dead gateway?
+    log "V6  RA deprecation: the client must stop using a demoted router"
+    local h before
+    h="$(holder)"
+    before="$(lan9 'ip -6 route show default 2>/dev/null | head -1')"
+    if [ -z "$before" ]; then
+        warn "VLAN 9 client has no IPv6 default route -- SLAAC may not have run; skipping"
+        return
+    fi
+    log "    client default was: $before"
+    r "$h" 'sudo mkdir -p /run/vrrp-wan && sudo touch /run/vrrp-wan/force-fault'
+    sleep 45
+    local after; after="$(lan9 'ip -6 route show default 2>/dev/null | head -1')"
+    log "    client default now: ${after:-<none>}"
+    # radvd emits a final RA with router-lifetime 0 on a graceful stop. Either
+    # the client moved to the new master or it dropped the route entirely; both
+    # are correct. Still pointing at the demoted box is not.
+    if [ "$after" = "$before" ]; then
+        fail "client still points at the demoted router -- the farewell RA did not land"
+    else
+        pass "client stopped using the demoted router"
+    fi
+    r "$h" 'sudo rm -f /run/vrrp-wan/force-fault'
+    save_evidence V6-ra-deprecation
+    sleep 40
+}
+
+V7() {  # replay the 2026-09-06 near-miss, but slowly
+    log "V7  slow WAN restore: does the hysteresis still hold?"
+    # On 2026-09-06 vif53-pin-boot-disable bounced the 10 gig, he-tunnel-follow
+    # ticked once and saw the PPPoE address, and vyos-failover restored the 10
+    # gig 22 SECONDS before the second tick would have pushed HE at an address
+    # Vodafone reissues on every dial. 22s of margin is not a safety property.
+    # Here the restore is deliberately slower than the hysteresis window.
+    local h before after tengig
+    h="$(holder)"; before="$(he_calls)"; tengig="$(tengig_addr)"
+    r "$h" 'sudo ip link set bond0.53 down'
+    sleep 200                                  # > 2 ticks of a 1m scheduler
+    r "$h" 'sudo ip link set bond0.53 up'
+    sleep 90
+    after="$(he_calls)"
+    if [ "$(( ${after:-0} - ${before:-0} ))" -ge 1 ]; then
+        warn "HE was updated $(( ${after:-0} - ${before:-0} )) time(s) and then had to move back -- this is the 2026-09-06 shape, now reproduced deliberately. Either widen HYSTERESIS or make vif53-pin-boot-disable hold he-tunnel-follow off for the bounce."
+    else
+        pass "no HE churn across a slow WAN bounce"
+    fi
+    # Whatever happened, the tunnel must end up back on the 10 gig.
+    local i
+    for i in $(seq 1 24); do
+        [ "$(tun_src "$h")" = "$tengig" ] && break
+        sleep 10
+    done
+    [ "$(tun_src "$h")" = "$tengig" ] && pass "tunnel returned to the 10 gig address" \
+        || fail "tunnel stuck on $(tun_src "$h") after the 10 gig came back"
+    save_evidence V7-slow-restore
+}
+
+preflight() {
+    log "preflight"
+    local rc=0
+    [ "$(timeout 60 "$HE" status | grep -c 'API bound  : nohost')" = 1 ] \
+        || { fail "the fake HE endpoint is not answering -- run ./labsim-he-endpoint.sh up"; rc=1; }
+    local h
+    for h in "$R1" "$R2"; do
+        [ "$(tun_state "$h")" = absent ] \
+            && { fail "$h has no tun0 -- run --setup first"; rc=1; }
+        [ "$(r "$h" 'systemctl is-active vrrp-wan-reconcile.timer')" = active ] \
+            || { fail "$h vrrp-wan-reconcile.timer not active"; rc=1; }
+        # The reconciler must be the version that knows about the v6 plane, or
+        # every result below measures the OLD behaviour while printing the new
+        # test names -- the failure mode this repo has already been bitten by.
+        r "$h" 'grep -q v6_take /config/vrrp-wan-reconcile' \
+            || { fail "$h has a vrrp-wan-reconcile with no IPv6 plane -- run migration/vrrp-wan-install"; rc=1; }
+    done
+    # Line the sim's endpoint up with whoever actually holds the tunnel. The real
+    # HE remembers where it was pointed; a rebuilt sim endpoint does not, and
+    # he-tunnel-follow will not re-assert because from its side nothing changed.
+    # Does NOT count as an API call, so the invariant-2 assertions stay honest.
+    local m src
+    m="$(holder)"; src="$(tun_src "$m")"
+    if [ -n "$src" ]; then
+        timeout 60 "$HE" point "$src" >/dev/null 2>&1 \
+            || { fail "could not point the sim HE endpoint at $src"; rc=1; }
+    fi
+    [ "$rc" -eq 0 ] && pass "HE endpoint answering and pointed at $src, both routers have tun0 and a v6-aware reconciler"
+    return $rc
+}
+
+case "${1:---all}" in
+  --list)  echo "V0 baseline | V1 clean failover | V2 10gig-down | V3 cold backup | V4 absent source-address | V5 transition invariant | V6 RA deprecation | V7 slow restore"; exit 0 ;;
+  --setup) setup; exit 0 ;;
+  --all)   preflight || exit 1; V0; V1; V2; V3; V4; V5; V6; V7 ;;
+  *)       preflight || exit 1; "$1" ;;
+esac
+
+echo
+[ "$FAILED" -eq 0 ] && { echo "ALL PASS"; exit 0; }
+echo "$FAILED check(s) FAILED"; exit 1

labsim/labsim-k8s-etcd.sh

@@ -0,0 +1,279 @@
+#!/bin/bash
+# A 3-SERVER embedded-etcd k3s cluster in labsim, configured the way PRODUCTION
+# is -- through /etc/rancher/k3s/config.yaml rendered by labctl's own generator.
+#
+# WHY THIS EXISTS, separate from k8s-up.sh. k8s-up.sh is 1 server + 2 agents and
+# drives k3s with inline INSTALL_K3S_EXEC flags. Production is 3 control-plane
+# servers with embedded etcd, configured by config.yaml from k3s-config.ts. The
+# dual-stack conversion touches etcd quorum on all three at once and rolls a
+# config.yaml change one server at a time -- a failure mode a single-server lab
+# structurally cannot show, driven by a mechanism k8s-up.sh does not use. So this
+# harness models the real shape and the real code path, or the rehearsal is
+# theatre.
+#
+#   ./labsim-k8s-etcd.sh up            build the 3-server cluster
+#   ./labsim-k8s-etcd.sh kubeconfig    fetch ./labsim-etcd.kubeconfig
+#   ./labsim-k8s-etcd.sh status        node + etcd health
+#   ./labsim-k8s-etcd.sh down          destroy the three VMs
+#   ./labsim-k8s-etcd.sh render N      print the config.yaml node N would get
+#
+# The config.yaml is produced by:
+#   bastion .../k3s/bin/render-config.js   (the production generator, exported)
+# so if that generator changes shape, this cluster moves with it.
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+source "$SCRIPT_DIR/lib.sh"
+source "$SCRIPT_DIR/ovs.sh"
+
+K8S_VLAN="${K8S_VLAN:-2}"
+NET="${NET:-172.31.2}"
+FIRST_OCTET="${FIRST_OCTET:-31}"          # .31/.32/.33 -- clear of k8s-up.sh's .11-.13
+SERVERS="${SERVERS:-3}"
+MEM="${MEM:-4096}"; CPUS="${CPUS:-2}"; DISK_GB="${DISK_GB:-12}"
+TOKEN="${TOKEN:-labsim-etcd-token}"
+CILIUM_VERSION="${CILIUM_VERSION:-1.19.1}"
+
+DEB_URL="${DEB_URL:-https://cloud.debian.org/images/cloud/trixie/latest/debian-13-genericcloud-amd64.qcow2}"
+DEB_BASE="${DEB_BASE:-$IMG_DIR/debian-13-genericcloud-amd64.qcow2}"
+
+# The production generator, compiled. Built by `npm --prefix bastion/src/modules run build`.
+RENDER="${RENDER:-$SCRIPT_DIR/../bastion/src/modules/dist/modules/k3s/bin/render-config.js}"
+
+node_name() { echo "labsim-etcd$1"; }
+node_ip()   { echo "${NET}.$((FIRST_OCTET + $1 - 1))"; }
+
+# The audit policy the generated config.yaml references. Without the file the
+# apiserver refuses to start (audit-policy-file points at a missing path), which
+# is a silent-looking crash loop. Kept byte-identical to labctl's audit-policy.ts.
+AUDIT_POLICY='apiVersion: audit.k8s.io/v1
+kind: Policy
+rules:
+  - level: Metadata
+    resources:
+    - group: ""
+      resources: ["secrets", "configmaps"]
+  - level: RequestResponse
+    verbs: ["create", "update", "patch", "delete"]
+    resources:
+    - group: ""
+      resources: ["pods", "services", "deployments"]
+  - level: None
+    resources:
+    - group: ""
+      resources: ["endpoints", "events"]
+    users: ["system:kube-proxy", "system:apiserver"]
+  - level: Metadata
+    omitStages:
+    - "RequestReceived"'
+
+# Render node N's config.yaml with the production generator. Node 1 is
+# cluster-init; 2..N join as SERVERS (not agents) -- role=infra with a server
+# URL is exactly a joining etcd member, the same as production worker1/worker2.
+render_config() {
+  local n="$1"
+  local ip; ip="$(node_ip "$n")"
+  local server1; server1="$(node_ip 1)"
+  if [ "$n" -eq 1 ]; then
+    ROLE=infra HOSTNAME="$(node_name 1)" IP="$ip" TLS_SANS="$ip" \
+      node "$RENDER"
+  else
+    ROLE=infra HOSTNAME="$(node_name "$n")" IP="$ip" TLS_SANS="$ip" \
+      K3S_SERVER_URL="https://${server1}:6443" K3S_TOKEN="$TOKEN" \
+      node "$RENDER"
+  fi
+}
+
+ensure_base_image() {
+  [ -f "$DEB_BASE" ] && { log "base image present"; return; }
+  log "fetching Debian cloud image -> $DEB_BASE"
+  sudo mkdir -p "$IMG_DIR"
+  sudo curl -fsSL --retry 3 -o "${DEB_BASE}.tmp" "$DEB_URL" || die "fetch failed"
+  sudo mv "${DEB_BASE}.tmp" "$DEB_BASE"
+}
+
+build_seed() {
+  local iso="$1" n="$2" pubkey="$3"
+  local vm; vm="$(node_name "$n")"; local ip; ip="$(node_ip "$n")"
+  local server1; server1="$(node_ip 1)"
+  local tmp; tmp="$(mktemp -d)"
+  local cfg; cfg="$(render_config "$n")"
+
+  cat > "$tmp/meta-data" <<EOF
+instance-id: $vm
+local-hostname: $vm
+EOF
+  cat > "$tmp/network-config" <<EOF
+version: 2
+ethernets:
+  enp1s0:
+    match: { name: "en*" }
+    addresses: [$ip/24]
+    routes: [{ to: default, via: ${NET}.1 }]
+    nameservers: { addresses: [8.8.8.8, 1.1.1.1] }
+EOF
+
+  # config.yaml and audit policy embedded via write_files, indented for YAML.
+  local cfg_ind audit_ind
+  cfg_ind="$(printf '%s\n' "$cfg" | sed 's/^/      /')"
+  audit_ind="$(printf '%s\n' "$AUDIT_POLICY" | sed 's/^/      /')"
+
+  cat > "$tmp/user-data" <<EOF
+#cloud-config
+hostname: $vm
+users:
+  - name: debian
+    groups: [sudo]
+    shell: /bin/bash
+    sudo: ["ALL=(ALL) NOPASSWD:ALL"]
+    lock_passwd: false
+    plain_text_passwd: labsim
+    ssh_authorized_keys: [ $pubkey ]
+ssh_pwauth: true
+ssh_authorized_keys: [ $pubkey ]
+
+package_update: true
+packages: [curl, jq, iproute2, tcpdump, etcd-client]
+
+write_files:
+  - path: /etc/rancher/k3s/config.yaml
+    content: |
+$cfg_ind
+  - path: /etc/rancher/k3s/audit-policy.yaml
+    content: |
+$audit_ind
+  - path: /etc/modules-load.d/cilium.conf
+    content: |
+      br_netfilter
+      overlay
+  # The CIS sysctls the generated config's `protect-kernel-defaults: true`
+  # REQUIRES -- byte-for-byte from labctl's sysctl.ts (applyCisHardening), plus
+  # v6 forwarding. Without vm.overcommit_memory=1 / kernel.panic=10 /
+  # kernel.panic_on_oops=1 the kubelet REFUSES to start ("invalid kernel flag"),
+  # k3s exits 1 and crash-loops -- which presents downstream as etcd
+  # re-initialising and the apiserver flapping, i.e. it looks like an etcd/CPU
+  # problem when it is not. In production these come from install.ks.ts + this
+  # operation; the sim must set them too.
+  - path: /etc/sysctl.d/90-k3s-cis.conf
+    content: |
+      net.bridge.bridge-nf-call-iptables  = 1
+      net.bridge.bridge-nf-call-ip6tables = 1
+      net.ipv4.ip_forward                 = 1
+      net.ipv6.conf.all.forwarding        = 1
+      vm.panic_on_oom                     = 0
+      vm.overcommit_memory                = 1
+      kernel.panic                        = 10
+      kernel.panic_on_oops                = 1
+      fs.inotify.max_user_instances       = 524288
+      fs.inotify.max_user_watches         = 524288
+
+runcmd:
+  - [ modprobe, br_netfilter ]
+  - [ modprobe, overlay ]
+  - [ sysctl, --system ]
+  - |
+    # Joining servers must wait for the cluster-init server's API, or the join
+    # races etcd bootstrap and the unit backs off for minutes.
+    if [ "$n" -ne 1 ]; then
+      for i in \$(seq 1 90); do
+        curl -sk --max-time 3 https://${server1}:6443/ping >/dev/null 2>&1 && break
+        sleep 5
+      done
+    fi
+  - |
+    # INSTALL_K3S_EXEC=server (bare) -- everything else comes from config.yaml,
+    # exactly as production. The token is passed via env for the join; on node 1
+    # it seeds the cluster token.
+    #
+    # The etcd-arg tuning is LAB-ONLY (not in production's config): relaxed
+    # heartbeat/election timers so etcd tolerates nested-virt scheduling jitter.
+    # NB: this was NOT what fixed the first run's failure -- that was the missing
+    # protect-kernel-defaults sysctls above, which crash-looped the kubelet and
+    # only LOOKED like etcd instability. The tuning is kept as cheap defensive
+    # insurance for a busy host; it changes nothing the conversion test
+    # exercises.
+    curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="server --etcd-arg=heartbeat-interval=500 --etcd-arg=election-timeout=5000" K3S_TOKEN="$TOKEN" sh -
+EOF
+
+  # Guard the generated YAML before building the ISO -- a bad indent in the
+  # embedded config.yaml would fail on the node, minutes later and opaquely.
+  python3 -c "import yaml,sys; yaml.safe_load(open(sys.argv[1]))" "$tmp/user-data" \
+    || die "generated user-data is not valid YAML for node $n"
+
+  sudo mkdir -p "$(dirname "$iso")"
+  sudo genisoimage -quiet -output "$iso" -volid cidata -joliet -rock \
+    "$tmp/user-data" "$tmp/meta-data" "$tmp/network-config"
+  rm -rf "$tmp"
+}
+
+create_node() {
+  local n="$1" pubkey="$2"
+  local vm; vm="$(node_name "$n")"; local ip; ip="$(node_ip "$n")"
+  if virsh_q dominfo "$vm" >/dev/null 2>&1; then
+    local st; st="$(virsh_q domstate "$vm" 2>/dev/null | head -1 | tr -d '\n')"
+    [ "$st" = running ] && { log "$vm already running ($ip)"; return; }
+    log "$vm is $st -- starting"; virsh_q start "$vm" >/dev/null; return
+  fi
+  local disk="$IMG_DIR/${vm}.qcow2" seed="$IMG_DIR/${vm}-seed.iso"
+  log "creating $vm ($ip, server ${n}, ${MEM}MB/${CPUS}cpu)"
+  sudo qemu-img create -q -f qcow2 -F qcow2 -b "$DEB_BASE" "$disk" "${DISK_GB}G" >/dev/null
+  build_seed "$seed" "$n" "$pubkey"
+  sudo virt-install --connect "$LIBVIRT_URI" --name "$vm" \
+    --memory "$MEM" --vcpus "$CPUS" \
+    --disk "path=$disk,format=qcow2,bus=virtio" \
+    --disk "path=$seed,device=cdrom" \
+    --network "network=$OVS_NET,portgroup=vlan${K8S_VLAN},model=virtio" \
+    --os-variant debian12 --graphics none --noautoconsole --import >/dev/null
+}
+
+ssh_node() { ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
+               -o LogLevel=ERROR -o ConnectTimeout=8 "debian@$1" "$2" 2>/dev/null; }
+
+cmd_up() {
+  require_tools
+  command -v genisoimage >/dev/null || die "genisoimage missing"
+  [ -f "$RENDER" ] || die "render CLI not built: $RENDER (run: npm --prefix bastion/src/modules run build)"
+  local pubkey; pubkey="$(find_ssh_pubkey)"
+  ensure_base_image
+  selected_vlans; log "ensuring OVS fabric"; ovs_up
+  local n; for n in $(seq 1 "$SERVERS"); do create_node "$n" "$pubkey"; done
+  echo
+  log "3 servers booting. Node 1 cluster-inits; 2/3 join as etcd members."
+  log "watch:  ssh debian@$(node_ip 1) 'sudo k3s kubectl get nodes'"
+  log "then:   $0 kubeconfig  &&  $0 status"
+}
+
+cmd_kubeconfig() {
+  local s1; s1="$(node_ip 1)"; local out="$SCRIPT_DIR/labsim-etcd.kubeconfig"
+  ssh_node "$s1" "sudo cat /etc/rancher/k3s/k3s.yaml" | sed "s|127.0.0.1|$s1|" > "$out"
+  chmod 600 "$out"; log "wrote $out"
+}
+
+cmd_status() {
+  local s1; s1="$(node_ip 1)"
+  echo "=== nodes ==="
+  ssh_node "$s1" "sudo k3s kubectl get nodes -o wide 2>/dev/null" | sed 's/^/  /'
+  echo "=== etcd members (quorum needs 2 of 3) ==="
+  ssh_node "$s1" 'sudo k3s kubectl get nodes -l node-role.kubernetes.io/etcd=true --no-headers 2>/dev/null | wc -l' | sed 's/^/  etcd nodes: /'
+  echo "=== servicecidr (both families once dual-stack) ==="
+  ssh_node "$s1" "sudo k3s kubectl get servicecidr -o jsonpath='{range .items[*]}{.metadata.name}={.spec.cidrs}{\"\\n\"}{end}' 2>/dev/null" | sed 's/^/  /'
+}
+
+cmd_down() {
+  local n vm
+  for n in $(seq 1 "$SERVERS"); do
+    vm="$(node_name "$n")"
+    virsh_q destroy "$vm" >/dev/null 2>&1 || true
+    virsh_q undefine "$vm" --remove-all-storage >/dev/null 2>&1 || true
+    log "removed $vm"
+  done
+}
+
+case "${1:-up}" in
+  up)         cmd_up ;;
+  kubeconfig) cmd_kubeconfig ;;
+  status)     cmd_status ;;
+  down)       cmd_down ;;
+  render)     render_config "${2:-1}" ;;
+  *) die "usage: $0 {up|kubeconfig|status|down|render N}" ;;
+esac

labsim/labsim-matrix.py

@@ -0,0 +1,208 @@
+#!/usr/bin/env python3
+"""Full-mesh connectivity matrix for the labsim VLANs.
+
+Probes every VLAN VM from every other VLAN VM (ICMP + TCP/22 + TCP/80) and
+prints a grid. Use --watch to keep it live: cells that changed since the last
+sweep are highlighted, so adding or removing a VyOS firewall rule shows up
+within one refresh.
+
+Deliberately dependency-free on the guests: the probe runs with python3, which
+is already installed there (cloud-init needs it), so nothing has to be
+installed on VMs that have no internet.
+
+  ./labsim-matrix.py                 # one sweep
+  ./labsim-matrix.py --watch         # live, refresh every 5s
+  ./labsim-matrix.py --watch 2       # live, every 2s
+  ./labsim-matrix.py --proto icmp    # single protocol
+  ./labsim-matrix.py --json          # machine-readable
+"""
+from __future__ import annotations
+
+import argparse
+import concurrent.futures
+import json
+import os
+import subprocess
+import sys
+import time
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+CONF = os.path.join(HERE, "vlans.conf")
+
+GREEN, RED, GREY, YELLOW, BOLD, RESET = (
+    "\033[0;32m", "\033[0;31m", "\033[0;90m", "\033[1;33m", "\033[1m", "\033[0m")
+
+PROTOS = ("icmp", "tcp22", "tcp80")
+
+# Runs ON the guest. Keep it stdlib-only and quick — a hung probe delays the
+# whole sweep, so every check is hard-bounded by a timeout.
+PROBE = r'''
+import json, re, socket, subprocess, sys
+targets = json.load(sys.stdin)
+out = {}
+for name, ip in targets.items():
+    res = {}
+    try:
+        p = subprocess.run(["ping", "-c", "1", "-W", "1", ip],
+                           stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, timeout=4)
+        res["icmp"] = p.returncode == 0
+        # RTT as well as pass/fail: a path that is up but slow is a different
+        # problem from one that is down, and the grid alone cannot show it.
+        res["rtt_ms"] = None
+        if res["icmp"]:
+            m = re.search(r"time[=<]\s*([0-9.]+)\s*ms", p.stdout.decode("utf-8", "replace"))
+            if m:
+                res["rtt_ms"] = float(m.group(1))
+    except Exception:
+        res["icmp"] = False
+        res["rtt_ms"] = None
+    for port in (22, 80):
+        s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+        s.settimeout(1.5)
+        try:
+            s.connect((ip, port)); res["tcp%d" % port] = True
+        except Exception:
+            res["tcp%d" % port] = False
+        finally:
+            try: s.close()
+            except Exception: pass
+    out[name] = res
+print(json.dumps(out))
+'''
+
+
+def load_vlans() -> list[dict]:
+    vlans = []
+    with open(CONF) as fh:
+        for line in fh:
+            line = line.strip()
+            if not line or line.startswith("#"):
+                continue
+            # masklen and host_octet are optional trailing fields; VLAN 10 sets
+            # both because it must be a /23 (see vlans.conf).
+            parts = line.split(":")
+            vid, name, prefix, real = parts[0], parts[1], parts[2], parts[3]
+            masklen = int(parts[4]) if len(parts) > 4 and parts[4] else 24
+            host = parts[5] if len(parts) > 5 and parts[5] else "2"
+            vlans.append({"vid": vid, "name": name, "ip": f"{prefix}.10",
+                          "label": f"{vid}:{name}", "real": real,
+                          "masklen": masklen, "host_ip": f"{prefix}.{host}"})
+    return vlans
+
+
+def probe_from(src: dict, targets: list[dict], timeout: int) -> tuple[str, dict]:
+    """SSH once into src and probe every target from there."""
+    payload = json.dumps({t["label"]: t["ip"] for t in targets if t["label"] != src["label"]})
+    cmd = [
+        "ssh", "-o", "StrictHostKeyChecking=no", "-o", "UserKnownHostsFile=/dev/null",
+        "-o", "BatchMode=yes", "-o", "ConnectTimeout=5", "-o", "LogLevel=ERROR",
+        f"alpine@{src['ip']}", "python3", "-",
+    ]
+    try:
+        # The probe script goes on stdin, the target list follows it — the guest
+        # reads the script from argv-less stdin, so send both in one stream.
+        proc = subprocess.run(
+            cmd, input=PROBE.replace("json.load(sys.stdin)", f"json.loads({payload!r})"),
+            capture_output=True, text=True, timeout=timeout)
+        if proc.returncode != 0:
+            return src["label"], {"__error__": (proc.stderr or "ssh failed").strip()[:60]}
+        return src["label"], json.loads(proc.stdout)
+    except subprocess.TimeoutExpired:
+        return src["label"], {"__error__": "probe timed out"}
+    except Exception as exc:  # noqa: BLE001 - report, never crash the sweep
+        return src["label"], {"__error__": f"{type(exc).__name__}: {exc}"[:60]}
+
+
+def sweep(vlans: list[dict], timeout: int) -> dict:
+    results: dict = {}
+    with concurrent.futures.ThreadPoolExecutor(max_workers=len(vlans)) as pool:
+        futures = [pool.submit(probe_from, v, vlans, timeout) for v in vlans]
+        for fut in concurrent.futures.as_completed(futures):
+            label, data = fut.result()
+            results[label] = data
+    return results
+
+
+def cell(ok: bool | None, changed: bool) -> str:
+    if ok is None:
+        return f"{GREY} ·  {RESET}"
+    mark = "ok  " if ok else "--  "
+    colour = GREEN if ok else RED
+    if changed:
+        return f"{YELLOW}{BOLD}{'OK*' if ok else 'XX*':<4}{RESET}"
+    return f"{colour}{mark}{RESET}"
+
+
+def render(vlans: list[dict], results: dict, prev: dict | None, protos: tuple[str, ...]) -> None:
+    labels = [v["label"] for v in vlans]
+    width = max(len(x) for x in labels) + 2
+
+    for proto in protos:
+        print(f"\n{BOLD}{proto.upper()}{RESET}   (rows = source, columns = destination)")
+        header = " " * width + "".join(f"{lbl:<{width}}" for lbl in labels)
+        print(f"{GREY}{header}{RESET}")
+
+        for src in vlans:
+            row = f"{src['label']:<{width}}"
+            data = results.get(src["label"], {})
+            if "__error__" in data:
+                print(row + f"{RED}{data['__error__']}{RESET}")
+                continue
+            for dst in vlans:
+                if dst["label"] == src["label"]:
+                    row += f"{GREY}{'·':<{width}}{RESET}"
+                    continue
+                ok = data.get(dst["label"], {}).get(proto)
+                was = (prev or {}).get(src["label"], {}).get(dst["label"], {}).get(proto)
+                changed = prev is not None and was is not None and was != ok
+                txt = cell(ok, changed)
+                row += txt + " " * (width - 4)
+            print(row)
+
+    reach = sum(1 for s in results.values() if "__error__" not in s
+                for d in s.values() for p in protos if d.get(p) is True)
+    total = sum(1 for s in results.values() if "__error__" not in s
+                for _d in s.values() for _p in protos)
+    print(f"\n  reachable: {reach}/{total}   "
+          f"{GREEN}ok{RESET}=allowed  {RED}--{RESET}=blocked/no route  "
+          f"{YELLOW}*{RESET}=changed since last sweep")
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser(description=__doc__,
+                                 formatter_class=argparse.RawDescriptionHelpFormatter)
+    ap.add_argument("--watch", nargs="?", const=5, type=int, metavar="SECONDS",
+                    help="refresh continuously (default every 5s)")
+    ap.add_argument("--proto", choices=PROTOS, help="only this protocol")
+    ap.add_argument("--json", action="store_true", help="emit raw JSON and exit")
+    ap.add_argument("--timeout", type=int, default=30, help="per-host probe timeout")
+    args = ap.parse_args()
+
+    vlans = load_vlans()
+    protos = (args.proto,) if args.proto else PROTOS
+
+    if args.json:
+        print(json.dumps(sweep(vlans, args.timeout), indent=2))
+        return 0
+
+    prev = None
+    while True:
+        started = time.time()
+        results = sweep(vlans, args.timeout)
+        if args.watch:
+            os.system("clear")
+            print(f"{BOLD}labsim connectivity matrix{RESET}   "
+                  f"{time.strftime('%H:%M:%S')}   (refresh {args.watch}s, Ctrl-C to stop)")
+        render(vlans, results, prev, protos)
+        if not args.watch:
+            return 0
+        prev = results
+        time.sleep(max(0.0, args.watch - (time.time() - started)))
+
+
+if __name__ == "__main__":
+    try:
+        sys.exit(main())
+    except KeyboardInterrupt:
+        print()
+        sys.exit(130)

labsim/labsim-pppoe-ha-test.sh

@@ -0,0 +1,381 @@
+#!/bin/bash
+# Does the WAN follow VRRP mastership, and does exactly ONE router ever hold the
+# ISP session?
+#
+# The question is not "did a client get internet". A client can be answered by
+# the wrong path entirely -- for months labsim-vyos's only default route was the
+# libvirt-NAT scaffold on eth2, so every "the LAN still has internet" verdict was
+# answered by eth2 rather than by the WAN under test. This script therefore
+# refuses to run while that is true, and asks its questions of the ROUTERS and
+# the ACCESS CONCENTRATOR, which cannot be answered by accident.
+#
+# The invariant, checked continuously and independently of any individual test:
+#
+#     the AC never reports two `simdsl` sessions, and no two routers ever have a
+#     pppoe0 interface at the same time
+#
+# A run that violates it FAILS regardless of its own verdict, because a single
+# consumer credential is the whole constraint the design exists to satisfy.
+#
+#   ./labsim-pppoe-ha-test.sh --list
+#   ./labsim-pppoe-ha-test.sh T3
+#   ./labsim-pppoe-ha-test.sh --all
+set -uo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+R1="${R1:-172.31.1.252}"; R2="${R2:-172.31.1.253}"
+ISP="${ISP:-192.168.122.63}"          # the fake access concentrator
+LANVM="${LANVM:-172.31.10.10}"
+VIP="${VIP:-172.31.1.1}"
+PW="${VYOS_PW:-vyos}"; LANPW="${LANPW:-labsim}"
+EVID="$SCRIPT_DIR/wan-failover-evidence"
+
+SSH=(-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null
+     -o LogLevel=ERROR -o ConnectTimeout=6 -o PreferredAuthentications=password)
+r()   { timeout 45 sshpass -p "$PW" ssh "${SSH[@]}" "vyos@$1" "${@:2}" 2>/dev/null; }
+isp() { timeout 30 sshpass -p "$PW" ssh "${SSH[@]}" "vyos@$ISP" "$@" 2>/dev/null; }
+
+# Set the AC's session policy, and PROVE it landed.
+#
+# `vbash -c 'source script-template; configure; ...; commit'` does NOT work: the
+# config session never starts and commit dies with "Invalid command: [commit]",
+# on stderr, which the isp() helper discards. The whole T4 matrix therefore ran
+# all three iterations against the accel-ppp DEFAULT while printing
+# "--- session-control=deny ---" -- it reported coverage it did not have, which
+# is worse than reporting a failure. Drive it from a real script FILE, then read
+# the value back and abort the run if it disagrees.
+isp_session_control() {
+    local mode="$1"
+    printf '#!/bin/vbash\nsource /opt/vyatta/etc/functions/script-template\nconfigure\nset service pppoe-server session-control %s\ncommit\nsave\nexit\n' "$mode" \
+        | timeout 30 sshpass -p "$PW" ssh "${SSH[@]}" "vyos@$ISP" 'cat > /tmp/set-sc.sh && chmod +x /tmp/set-sc.sh && sudo /tmp/set-sc.sh' >/dev/null 2>&1
+    local got
+    got="$(isp '/opt/vyatta/bin/vyatta-op-cmd-wrapper show configuration commands' \
+             | sed -n "s/.*session-control '\\(.*\\)'/\\1/p")"
+    if [ "$got" = "$mode" ]; then
+        log "    AC session-control=$mode (verified)"
+        return 0
+    fi
+    fail "could not set AC session-control=$mode (reads '${got:-unset}') -- results would be fiction"
+    return 1
+}
+# The LAN VMs are Alpine and their sshd offers keyboard-interactive, not
+# `password`. Reusing the routers' option set here made ssh exit 255 BEFORE
+# running anything, and T5 read that as "the LAN lost the internet" while a
+# tcpdump on the router showed the pings flowing out pppoe0 and the replies
+# coming back. An exit code that can mean "the network is broken" or "I could
+# not log in" is not a connectivity test.
+LAN_SSH=(-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null
+         -o LogLevel=ERROR -o ConnectTimeout=6)
+lan() { timeout 45 sshpass -p "$LANPW" ssh "${LAN_SSH[@]}" "root@$LANVM" "$@" 2>/dev/null; }
+
+# Assert on what the guest actually reported, not on ssh's exit status.
+lan_online() { [ "$(lan 'ping -c2 -W3 9.9.9.9 >/dev/null 2>&1 && echo ONLINE')" = ONLINE ]; }
+
+log()  { printf '\033[36m==>\033[0m %s\n' "$*"; }
+pass() { printf '  \033[32mPASS\033[0m %s\n' "$*"; }
+fail() { printf '  \033[31mFAIL\033[0m %s\n' "$*"; FAILED=$((FAILED+1)); }
+FAILED=0
+
+# --- observations ----------------------------------------------------------
+ac_sessions() { isp '/opt/vyatta/bin/vyatta-op-cmd-wrapper show pppoe-server sessions' \
+                  | grep -c ' simdsl ' || true; }
+ac_detail()   { isp '/opt/vyatta/bin/vyatta-op-cmd-wrapper show pppoe-server sessions'; }
+# A destroyed or unreachable router is emphatically NOT holding pppoe0, but ssh
+# returns an EMPTY string rather than 0 -- and `[ "" = 0 ]` is false, so a
+# hard-failover test waited for the dead box to "report" zero and hung until its
+# timeout, long after the survivor had taken over correctly. Default to 0.
+ppp_on()      { local v; v="$(r "$1" 'ip -4 addr show pppoe0 2>/dev/null | grep -c inet' | tr -d ' \n')"
+                echo "${v:-0}"; }
+holder()      { for h in "$R1" "$R2"; do
+                  [ "$(r "$h" "ip -4 -o addr show | grep -c ' ${VIP}/'" | tr -d ' \n')" != 0 ] \
+                    && { echo "$h"; return; }; done; echo none; }
+status()      { r "$1" 'sudo /config/vrrp-wan-reconcile --status'; }
+
+# How many routers currently hold a PPPoE interface. The invariant's other half.
+ppp_holders() { n=0; for h in "$R1" "$R2"; do
+                  [ "$(ppp_on "$h")" != 0 ] && n=$((n+1)); done; echo "$n"; }
+
+warn() { printf '  \033[33mWARN\033[0m %s\n' "$*"; }
+
+check_invariant() {
+    local s p ok=0
+    s="$(ac_sessions)"; p="$(ppp_holders)"
+
+    # THE invariant. Two of OUR routers dialled at once is the failure that
+    # matters: one ISP account, and against a real ISP that is how you get
+    # rate-limited or locked out.
+    [ "${p:-0}" -le 1 ] || { fail "INVARIANT: $p routers hold pppoe0"; ok=1; }
+
+    # The AC's session count is a PROXY for the above, and only a valid one
+    # while the AC enforces single-session. Under session-control=disable it
+    # does not, so a destroyed router's session simply stays in the table and
+    # the count reads 2 while exactly one live router is dialled -- which is AC
+    # bookkeeping, not a double dial. Attribute it rather than failing blind:
+    # only call it a violation when more than one router is ACTUALLY dialled.
+    #
+    # Do not silence it either. An orphaned session still occupies the single
+    # slot at a real ISP, and that is exactly what made session-control=deny
+    # take 148s while the survivor's dial attempts were refused.
+    if [ "${s:-0}" -gt 1 ]; then
+        if [ "${p:-0}" -gt 1 ]; then
+            fail "INVARIANT: AC reports $s simdsl sessions AND $p routers are dialled"
+            ok=1
+        else
+            warn "AC reports $s simdsl sessions but only ${p:-0} router is dialled -- stale session from the destroyed peer (expected where the AC does not enforce single-session; it is what a hostile AC holds against the survivor)"
+        fi
+    fi
+    return $ok
+}
+
+# --- preconditions ---------------------------------------------------------
+# The scaffold check is a hard gate, not a warning. A default route via eth2
+# means the box can reach the internet without the WAN working at all, and every
+# connectivity verdict below would be a lie.
+preflight() {
+    log "preflight"
+    local rc=0
+    for h in "$R1" "$R2"; do
+        if r "$h" 'ip route show default' | grep -q 'dev eth2'; then
+            fail "$h still routes via eth2 (libvirt-NAT scaffold) -- run sim-net-config.py --drop-scaffold"
+            rc=1
+        fi
+        if [ "$(r "$h" '[ -f /etc/systemd/system/ppp@pppoe0.service.d/10-vrrp-wan-gate.conf ] && echo y')" != y ]; then
+            fail "$h is missing the ppp gate drop-in -- run migration/vrrp-wan-install"
+            rc=1
+        fi
+        [ "$(r "$h" 'systemctl is-active vrrp-wan-guard.timer')" = active ] \
+            || { fail "$h vrrp-wan-guard.timer not active"; rc=1; }
+        # A router with no peers file CANNOT dial, and says so only once in the
+        # journal. Every failover result in the run would then be a false
+        # negative blamed on the ISP. Check both, and check the config that
+        # renders it -- an unsaved commit reverts on reboot and takes pppoe0
+        # with it, which is how the sim secondary silently stopped dialling.
+        if [ "$(r "$h" '[ -f /etc/ppp/peers/pppoe0 ] && echo y')" != y ]; then
+            fail "$h has no /etc/ppp/peers/pppoe0 -- it cannot dial; re-commit the pppoe subtree"
+            rc=1
+        fi
+        # The op-mode WRAPPER, not a bare `show`: over non-interactive ssh the
+        # bare form is not on PATH, so this silently matched nothing and failed
+        # both routers that were in fact configured correctly.
+        if ! r "$h" '/opt/vyatta/bin/vyatta-op-cmd-wrapper show configuration commands' 2>/dev/null | grep -q 'interfaces pppoe pppoe0 source-interface'; then
+            fail "$h has no pppoe0 in config (unsaved commit lost on reboot?)"
+            rc=1
+        fi
+    done
+    [ "$rc" -eq 0 ] && pass "scaffold dropped, gate present, guard running, both can dial"
+    return $rc
+}
+
+settle() {  # wait until exactly one router holds pppoe0, or give up
+    local i
+    for i in $(seq 1 "${1:-24}"); do
+        [ "$(ppp_holders)" = 1 ] && return 0
+        sleep 5
+    done
+    return 1
+}
+
+# Wait until a SPECIFIC router holds pppoe0 and the other does not.
+#
+# The obvious `settle` is wrong for a failover: "exactly one holder" is already
+# true before the handover starts, so it returns instantly and the test reports
+# that nothing moved while the handover is still in flight. Asking who holds it
+# is the only useful form of the question.
+settle_on() {
+    local want="$1" other i
+    other=$([ "$want" = "$R1" ] && echo "$R2" || echo "$R1")
+    for i in $(seq 1 "${2:-30}"); do
+        [ "$(ppp_on "$want")" != 0 ] && [ "$(ppp_on "$other")" = 0 ] && return 0
+        sleep 5
+    done
+    return 1
+}
+
+save_evidence() {
+    local name="$1"; local d="$EVID/$name"; mkdir -p "$d"
+    { echo "=== $(date -Is) ==="; echo "--- AC sessions ---"; ac_detail
+      for h in "$R1" "$R2"; do echo "--- $h ---"; status "$h"
+        r "$h" 'ip -4 -br addr show pppoe0 2>/dev/null; ip route show default; sudo journalctl -t vrrp-wan -n 8 --no-pager'
+      done; } > "$d/state.txt" 2>&1
+    log "evidence -> wan-failover-evidence/$name/"
+}
+
+# --- tests -----------------------------------------------------------------
+T0() {  # baseline
+    log "T0  baseline: exactly one session, held by the VIP holder"
+    local h s; h="$(holder)"; s="$(ac_sessions)"
+    [ "$s" = 1 ] && pass "AC reports 1 session" || fail "AC reports $s sessions"
+    [ "$(ppp_on "$h")" != 0 ] && pass "the VIP holder ($h) is the one dialled" \
+        || fail "VIP holder $h has no pppoe0"
+    local other; other=$([ "$h" = "$R1" ] && echo "$R2" || echo "$R1")
+    [ "$(ppp_on "$other")" = 0 ] && pass "the backup ($other) is not dialled" \
+        || fail "backup $other also holds pppoe0"
+    save_evidence T0-baseline
+}
+
+T3() {  # clean, deliberate failover
+    log "T3  clean failover via force-fault"
+    local from to t0 t1; from="$(holder)"
+    to=$([ "$from" = "$R1" ] && echo "$R2" || echo "$R1")
+    log "    master=$from -> expecting $to"
+    t0=$(date +%s)
+    r "$from" 'sudo mkdir -p /run/vrrp-wan && sudo touch /run/vrrp-wan/force-fault'
+    if settle_on "$to" 30; then
+        t1=$(date +%s)
+        [ "$(ppp_on "$to")" != 0 ] && pass "pppoe0 moved to $to in $((t1-t0))s" \
+            || fail "pppoe0 did not move to $to"
+        [ "$(ppp_on "$from")" = 0 ] && pass "$from released pppoe0" \
+            || fail "$from still holds pppoe0"
+    else
+        fail "never settled to exactly one pppoe0 holder"
+    fi
+    check_invariant
+    save_evidence T3-clean-failover
+    r "$from" 'sudo rm -f /run/vrrp-wan/force-fault'
+    settle 30 >/dev/null
+}
+
+T5() {  # 10gig down -> PPPoE carries traffic
+    log "T5  10 gig down on the master: traffic must survive on pppoe0"
+    local h; h="$(holder)"
+    r "$h" 'sudo ip link set bond0.53 down'
+    sleep 20
+    local via; via="$(r "$h" 'ip route show default' | head -1)"
+    if echo "$via" | grep -q pppoe0; then
+        pass "default route moved to pppoe0: $via"
+    else
+        fail "default route did not move to pppoe0: ${via:-<none>}"
+    fi
+    # Poll, do not sample. Judging connectivity on one ping 20s after the link
+    # dropped failed while the path was still reconverging, and reported "the LAN
+    # lost the internet" for a path that came back moments later. A single
+    # negative sample is the least trustworthy verdict this harness can produce.
+    local ok=no i
+    for i in $(seq 1 12); do
+        lan_online && { ok=yes; break; }
+        sleep 5
+    done
+    [ "$ok" = yes ] && pass "LAN reaches the internet over pppoe0 (after $((i*5))s)" \
+        || fail "LAN never regained the internet with only pppoe0 up (60s)"
+    save_evidence T5-tengig-down
+    r "$h" 'sudo ip link set bond0.53 up'
+    sleep 20
+}
+
+T11() { # a blessed box with no peers file must not restart-loop
+    log "T11 missing peers file must not restart-loop"
+    local h; h="$(holder)"
+    # COPY then remove, never move: only a commit touching the pppoe subtree
+    # re-renders this file, so losing it strands the box permanently -- the gate
+    # blocks every dial, systemd says "skipped because of an unmet condition
+    # check" exactly once, and nothing else complains. An earlier `mv` pair did
+    # exactly that to the sim secondary and cost a debugging session.
+    r "$h" 'sudo cp -a /etc/ppp/peers/pppoe0 /run/peers.bak && sudo rm -f /etc/ppp/peers/pppoe0; sudo systemctl restart ppp@pppoe0'
+    sleep 12
+    local n; n="$(r "$h" 'systemctl show ppp@pppoe0 -p NRestarts --value')"
+    [ "${n:-99}" -le 1 ] && pass "NRestarts=$n (gate refused the start)" \
+        || fail "NRestarts=$n -- restart loop is back"
+    # Restore on the SAME host we broke, and prove it landed. Do not trust the
+    # copy back: if it silently failed, every later test in the run would be
+    # measuring a router that physically cannot dial.
+    r "$h" 'sudo cp -a /run/peers.bak /etc/ppp/peers/pppoe0'
+    if r "$h" 'test -f /etc/ppp/peers/pppoe0'; then
+        pass "peers file restored on $h"
+    else
+        fail "peers file NOT restored on $h -- that router can no longer dial"
+    fi
+    save_evidence T11-no-peers-file
+    settle 24 >/dev/null
+}
+
+T8() {  # lease expiry: the guard must hang up a demoted-but-unreconciled box
+    log "T8  lease expiry revokes the session"
+    local h; h="$(holder)"
+    r "$h" 'sudo systemctl stop vrrp-wan-reconcile.timer'
+    r "$h" 'sudo touch -d "-200 seconds" /run/vrrp-wan/may-dial'
+    sleep 12
+    [ "$(ppp_on "$h")" = 0 ] && pass "guard hung up on a stale lease" \
+        || fail "stale lease did not revoke the session"
+    r "$h" 'sudo systemctl start vrrp-wan-reconcile.timer'
+    save_evidence T8-lease-expiry
+    settle 24 >/dev/null
+}
+
+
+T4() {  # hard failover across all three AC session-control policies
+    log "T4  hard failover (destroy the master) x session-control"
+    # Vodafone's policy is unknowable from here, so prove the design survives
+    # every one VyOS can express. `replace` is the accel-ppp default and the
+    # friendly case; `deny` is the hostile one, where the AC refuses the second
+    # session until its own dead-peer timer (lcp-echo-interval 30 x failure 3 =
+    # 90s) frees the first -- which is exactly why GRACE is no longer 90.
+    local mode from to vm t0 t1
+    for mode in replace deny disable; do
+        log "    --- session-control=$mode ---"
+        # Skip the iteration rather than measure the wrong policy.
+        isp_session_control "$mode" || continue
+        sleep 5
+        from="$(holder)"; to=$([ "$from" = "$R1" ] && echo "$R2" || echo "$R1")
+        vm=$([ "$from" = "$R1" ] && echo labsim-vyos || echo labsim-vyos2)
+        [ "$from" = none ] && { fail "no master before $mode run"; continue; }
+        log "    destroying $vm (master=$from), expecting $to"
+        t0=$(date +%s)
+        sudo virsh destroy "$vm" >/dev/null 2>&1
+        if settle_on "$to" 48; then
+            t1=$(date +%s)
+            pass "$mode: pppoe0 reached $to in $((t1-t0))s"
+        else
+            fail "$mode: $to never dialled within 240s"
+        fi
+        check_invariant
+        save_evidence "T4-hard-failover-$mode"
+        sudo virsh start "$vm" >/dev/null 2>&1
+        # Re-bond. A VM restart recreates its taps under NEW names, and the OVS
+        # bond keeps the old ones -- lacp dies, VLAN 1 goes with it, and the box
+        # comes back reachable on some VLANs but not others. ovs_bond_router
+        # detects the stale membership and rebuilds, but nothing runs it
+        # automatically, so a destroy/start test must do it or the survivor
+        # looks like a failover failure.
+        ( source "$SCRIPT_DIR/lib.sh"; source "$SCRIPT_DIR/ovs.sh"; selected_vlans
+          LAG_NAME=$([ "$vm" = labsim-vyos ] && echo lag-vyos || echo lag-vyos2)
+          ovs_bond_router "$vm" ) >/dev/null 2>&1
+        # Give the returning box time to boot and settle as BACKUP before the
+        # next iteration; it must NOT dial on the way up.
+        sleep 90
+        [ "$(ppp_on "$from")" = 0 ] && pass "$mode: $from did not dial on reboot" \
+            || fail "$mode: $from dialled on reboot (gate failed)"
+    done
+    isp_session_control replace >/dev/null
+    log "    AC restored to session-control=replace"
+}
+
+T12() { # the flap damper must never tear down an ESTABLISHED session
+    log "T12 flap holdoff must not kill a live session"
+    local h; h="$(holder)"
+    [ "$h" = none ] && { fail "no master to test"; return; }
+    # Forge a holdoff far in the future, as a dial storm would. Before the fix
+    # ppp_dial() returned here BEFORE renewing may-dial, the lease went stale,
+    # and vrrp-wan-guard hung up the master's working WAN ~80s later.
+    r "$h" 'sudo sh -c "echo $(( $(date +%s) + 900 )) > /run/vrrp-wan/holdoff"'
+    # Sleep past LEASE_TTL (75s) so a non-renewed lease would definitely expire.
+    sleep 100
+    if [ "$(ppp_on "$h")" = 1 ]; then
+        pass "session survived a 900s holdoff (lease still renewed)"
+    else
+        fail "holdoff killed the live session -- damper is tearing down the WAN"
+    fi
+    r "$h" 'sudo rm -f /run/vrrp-wan/holdoff /run/vrrp-wan/dials'
+    save_evidence T12-holdoff-keeps-session
+    settle 24 >/dev/null
+}
+
+case "${1:---all}" in
+  --list) echo "T0 baseline | T3 clean failover | T4 hard failover x policy | T5 10gig-down | T8 lease expiry | T11 no-peers-file | T12 holdoff-keeps-session"; exit 0 ;;
+  --all)  preflight || exit 1; T0; T3; T5; T8; T11; T12 ;;
+  --hard) preflight || exit 1; T4 ;;
+  *)      preflight || exit 1; "$1" ;;
+esac
+
+echo
+[ "$FAILED" -eq 0 ] && { echo "ALL PASS"; exit 0; }
+echo "$FAILED check(s) FAILED"; exit 1

labsim/labsim-up.sh

@@ -0,0 +1,71 @@
+#!/bin/bash
+# Bring up the lab network simulation: one isolated libvirt network per VLAN,
+# each with a single tiny Alpine VM offering SSH + a hello-world HTTP page.
+#
+# Idempotent: re-running only creates what is missing. Safe to run repeatedly.
+#
+# Usage: ./labsim-up.sh [vlan-id ...]      (default: every VLAN in vlans.conf)
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+source "$SCRIPT_DIR/lib.sh"
+source "$SCRIPT_DIR/ovs.sh"
+
+require_tools
+[ -f "$BASE_IMAGE" ] || die "base image missing: $BASE_IMAGE (see README)"
+
+SSH_PUB="$(find_ssh_pubkey)"
+log "Using SSH key: ${SSH_PUB%% *} ...${SSH_PUB##* }"
+
+selected_vlans "$@"
+
+# --- switch fabric ------------------------------------------------------
+log "bringing up OVS fabric ($OVS_BR) with host legs per VLAN"
+ovs_up
+
+# --- VMs ------------------------------------------------------------------
+for entry in "${SELECTED[@]}"; do
+  parse_vlan_entry "$entry"
+  vid="$V_VID"; name="$V_NAME"; prefix="$V_PREFIX"; real="$V_REAL"
+  vm="$(vm_name "$vid" "$name")"
+  ip="${prefix}.10"
+
+  if virsh_q dominfo "$vm" >/dev/null 2>&1; then
+    state="$(virsh_q domstate "$vm" 2>/dev/null | head -1 | tr -d '\n')"
+    if [ "$state" = "running" ]; then
+      log "VM $vm already running ($ip)"
+      continue
+    fi
+    log "VM $vm exists but is $state — starting"
+    virsh_q start "$vm" >/dev/null
+    continue
+  fi
+
+  log "creating VM $vm  ($ip on vlan $vid/$name)"
+
+  disk="$IMG_DIR/${vm}.qcow2"
+  seed="$IMG_DIR/${vm}-seed.iso"
+
+  # Copy-on-write overlay: each VM costs a few MB, not 176.
+  sudo qemu-img create -q -f qcow2 -F qcow2 -b "$BASE_IMAGE" "$disk" "$VM_DISK" >/dev/null
+
+  build_seed "$seed" "$vm" "$vid" "$name" "$prefix" "$ip" "$real" "$SSH_PUB" "$V_MASK"
+
+  sudo virt-install \
+    --connect "$LIBVIRT_URI" \
+    --name "$vm" \
+    --memory "$VM_MEM" --vcpus "$VM_CPUS" \
+    --disk "path=$disk,format=qcow2,bus=virtio" \
+    --disk "path=$seed,device=cdrom,readonly=on" \
+    --network "network=$OVS_NET,portgroup=vlan${vid},model=virtio" \
+    --os-variant alpinelinux3.18 \
+    --graphics none --noautoconsole --import >/dev/null
+done
+
+echo
+log "waiting for VMs to answer on SSH + HTTP..."
+wait_ready
+echo
+status_table
+echo
+log "environment is UP. Tear down with: $SCRIPT_DIR/labsim-down.sh"

labsim/labsim-vlan-leak-test.sh

@@ -0,0 +1,178 @@
+#!/bin/bash
+# Does the router offer an address from the WRONG VLAN's pool?
+#
+# The fault (ISC Kea #1117, "Mix of physical and virtual interfaces (VLAN) does
+# not work"): with `dhcp-socket-type: raw`, a frame tagged for a sub-interface is
+# ALSO delivered to the PARENT's AF_PACKET socket. Kea then selects a subnet from
+# the parent's own address and answers a second time from the wrong pool. Both
+# offers race to the client and the CLIENT decides which one wins -- which is why
+# the symptom looks device-dependent and unreproducible.
+#
+# Production and this sim have the identical shape that triggers it: Management
+# is the NATIVE/untagged VLAN on `bond0` and therefore has a subnet on the
+# parent, while every other VLAN is a `bond0.<vif>` sub-interface of that same
+# bond.
+#
+# Method: make one DHCP client on a TAGGED VLAN send a DISCOVER, and capture
+# simultaneously on the parent and on the sub-interface. The verdict is not
+# "did the client get the right address" -- the client picking correctly is
+# exactly how this hid for weeks. The verdict is how many OFFERs the SERVER
+# emitted and which source addresses they carried.
+#
+#   ./labsim-vlan-leak-test.sh                 test VLAN 3
+#   ./labsim-vlan-leak-test.sh --vlan 9        test another VLAN
+#   ./labsim-vlan-leak-test.sh --save before   also write the raw captures to
+#                                              vlan-leak-evidence/before/
+set -uo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+
+ROUTER_IP="${ROUTER_IP:-172.31.1.1}"
+ROUTER_PW="${ROUTER_PW:-vyos}"
+CLIENT_PW="${CLIENT_PW:-labsim}"
+VLAN=3
+CLIENT=""
+SAVE=""
+
+while [ $# -gt 0 ]; do
+  case "$1" in
+    --vlan)   VLAN="$2"; shift 2 ;;
+    --client) CLIENT="$2"; shift 2 ;;
+    --save)   SAVE="$2"; shift 2 ;;
+    *) echo "usage: $0 [--vlan N] [--client IP] [--save LABEL]" >&2; exit 2 ;;
+  esac
+done
+: "${CLIENT:=172.31.${VLAN}.10}"
+
+log() { printf '\033[36m==>\033[0m %s\n' "$*"; }
+die() { printf '\033[31merror:\033[0m %s\n' "$*" >&2; exit 1; }
+
+command -v sshpass >/dev/null || die "sshpass required"
+
+# A silent router is the one verdict worth double-checking before reporting.
+#
+# Kea can be `is-active` and answering nothing -- it reopens sockets on a retry
+# loop, and some configurations (`listen-interface`, notably) leave individual
+# VLANs dead while the rest work. Both look identical to a one-shot test: "no
+# reply at all". Two opposite and equally wrong conclusions about
+# `listen-interface` came out of believing a single negative run, in both
+# directions, before a retry made the real pattern obvious.
+#
+# Kea's fallback UDP socket appearing is NOT a readiness signal -- it is bound
+# well before the server actually answers. Checked, and it does not work.
+RETRIED="${RETRIED:-0}"
+
+router() {
+  timeout 40 sshpass -p "$ROUTER_PW" ssh -o StrictHostKeyChecking=no \
+    -o ConnectTimeout=8 "vyos@$ROUTER_IP" "$@" 2>/dev/null
+}
+# VyOS's login shell is vbash, which returns 255 on anything it does not like --
+# in particular a backgrounded job. Feeding the script to `bash -s` on stdin
+# sidesteps vbash entirely and is the only reliable way to leave a daemon behind.
+router_sh() {
+  timeout 40 sshpass -p "$ROUTER_PW" ssh -o StrictHostKeyChecking=no \
+    -o ConnectTimeout=8 "vyos@$ROUTER_IP" 'bash -s' 2>/dev/null
+}
+client() {
+  timeout 60 sshpass -p "$CLIENT_PW" ssh -o StrictHostKeyChecking=no \
+    -o ConnectTimeout=8 "root@$CLIENT" "$@" 2>/dev/null
+}
+
+# Which interfaces to watch. The parent is the whole point: after the fix it
+# should carry no DHCP traffic of its own at all.
+PARENT="bond0"
+VIF="bond0.${VLAN}"
+
+log "router $ROUTER_IP -- capturing on $PARENT and $VIF"
+# Kill EVERY tcpdump first, not just ones matching this run's pattern, and count
+# only afterwards. Counting `pgrep -f 'tcpdump -i bond0'` while a stray tcpdump
+# from an earlier session was still running satisfied the >=2 guard with zero of
+# THIS run's captures alive -- and a capture that records nothing reports
+# "the router sent no reply at all", which reads as a DHCP outage. That sent me
+# chasing a fault in the router that was entirely in the test harness.
+started="$(router_sh <<EOF
+sudo pkill -x tcpdump >/dev/null 2>&1
+sleep 1
+sudo rm -f /tmp/leak-*.txt
+sudo nohup tcpdump -i $PARENT -e -nn -l 'udp port 67 or udp port 68' > /tmp/leak-parent.txt 2>/dev/null &
+sudo nohup tcpdump -i $VIF -e -nn -l 'udp port 67 or udp port 68' > /tmp/leak-vif.txt 2>/dev/null &
+sleep 3
+pgrep -c -x tcpdump
+EOF
+)"
+[ "${started:-0}" -eq 2 ] || die "capture did not start on the router (got ${started:-0}, expected exactly 2)"
+
+# -s /bin/true: ask, observe the answer, apply nothing. The client's existing
+# static address is left alone, so this is safe to run against a live sim VM.
+log "client $CLIENT -- sending DISCOVER on VLAN $VLAN"
+client_out="$(client "udhcpc -n -q -f -i eth0 -s /bin/true -t 3 -T 3 2>&1")"
+[ -n "$client_out" ] || die "no response from client $CLIENT"
+
+sleep 2
+router "sudo pkill -x tcpdump" >/dev/null
+parent="$(router 'sudo cat /tmp/leak-parent.txt')"
+vif="$(router 'sudo cat /tmp/leak-vif.txt')"
+
+echo
+echo "--- client ---"
+echo "$client_out" | sed 's/^/  /'
+echo
+echo "--- $PARENT (parent) ---"
+echo "${parent:-  (nothing)}" | sed 's/^/  /'
+echo
+echo "--- $VIF (sub-interface) ---"
+echo "${vif:-  (nothing)}" | sed 's/^/  /'
+echo
+
+if [ -n "$SAVE" ]; then
+  d="$SCRIPT_DIR/vlan-leak-evidence/$SAVE"
+  mkdir -p "$d"
+  printf '%s\n' "$client_out" > "$d/client.txt"
+  printf '%s\n' "$parent"     > "$d/capture-parent.txt"
+  printf '%s\n' "$vif"        > "$d/capture-vif.txt"
+  router '/opt/vyatta/bin/vyatta-op-cmd-wrapper show configuration commands' \
+    | grep -E 'interfaces bonding|vrrp group' > "$d/router-config.txt"
+  log "evidence saved to vlan-leak-evidence/$SAVE/"
+fi
+
+# --- verdict ---------------------------------------------------------------
+# Every BOOTP Reply seen anywhere, reduced to its source address. A reply whose
+# source is not this VLAN's router leg is an offer from the wrong subnet.
+replies="$(printf '%s\n%s\n' "$parent" "$vif" \
+  | grep -o '[0-9.]*\.67 > [0-9.]*\.68' | awk '{print $1}' | sed 's/\.67$//' \
+  | sort -u)"
+want_prefix="172.31.${VLAN}."
+
+echo "=== verdict ==="
+if [ -z "$replies" ]; then
+  if [ "$RETRIED" -eq 0 ]; then
+    log "no reply -- retrying once in 20s before calling DHCP down"
+    sleep 20; RETRIED=1 exec "$0" --vlan "$VLAN" --client "$CLIENT" ${SAVE:+--save "$SAVE"}
+  fi
+  echo "INCONCLUSIVE: the router sent no reply at all, twice -- DHCP is down on VLAN $VLAN"
+  exit 2
+fi
+
+bad=0
+while read -r src; do
+  [ -z "$src" ] && continue
+  case "$src" in
+    "$want_prefix"*) printf '  ok    offer from %s (this VLAN)\n' "$src" ;;
+    *)               printf '  LEAK  offer from %s (WRONG subnet)\n' "$src"; bad=1 ;;
+  esac
+done <<<"$replies"
+
+# The parent carrying any DHCP of its own is the mechanism, not just a symptom:
+# it means the parent still has a subnet kea can match a tagged frame against.
+if printf '%s' "$parent" | grep -q 'ethertype IPv4' \
+   && printf '%s' "$parent" | grep -v 'vlan ' | grep -q '\.67 > '; then
+  echo "  note  $PARENT emitted an UNTAGGED reply -- the parent still serves a subnet"
+fi
+
+echo
+if [ "$bad" -eq 0 ]; then
+  echo "PASS: only this VLAN's pool answered."
+  exit 0
+fi
+echo "FAIL: the router answered from another VLAN's pool (kea #1117)."
+exit 1

labsim/lib.sh

@@ -0,0 +1,239 @@
+#!/bin/bash
+# Shared helpers for the lab network simulation.
+# shellcheck disable=SC2034
+
+LIBVIRT_URI="${LIBVIRT_URI:-qemu:///system}"
+IMG_DIR="${IMG_DIR:-/var/lib/libvirt/images/labsim}"
+BASE_IMAGE="${BASE_IMAGE:-$IMG_DIR/alpine-base.qcow2}"
+ALPINE_URL="${ALPINE_URL:-https://dl-cdn.alpinelinux.org/alpine/latest-stable/releases/cloud/generic_alpine-3.24.1-x86_64-bios-cloudinit-r0.qcow2}"
+
+VM_MEM="${VM_MEM:-256}"      # MB — Alpine is happy here
+VM_CPUS="${VM_CPUS:-1}"
+VM_DISK="${VM_DISK:-1G}"
+PREFIX="labsim"
+
+CONF="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/vlans.conf"
+
+log()  { printf '\033[0;36m[labsim]\033[0m %s\n' "$*"; }
+warn() { printf '\033[1;33m[labsim]\033[0m %s\n' "$*" >&2; }
+die()  { printf '\033[0;31m[labsim]\033[0m %s\n' "$*" >&2; exit 1; }
+
+virsh_q() { sudo virsh --connect "$LIBVIRT_URI" "$@"; }
+
+net_name() { echo "${PREFIX}-vlan$1"; }
+vm_name()  { echo "${PREFIX}-$1-$2"; }          # labsim-2-k8s
+# Linux bridge names are capped at 15 chars — keep it short and unique.
+br_name()  { echo "vbr-ls$1"; }
+
+require_tools() {
+  for t in virsh virt-install qemu-img genisoimage; do
+    command -v "$t" >/dev/null 2>&1 || die "missing required tool: $t"
+  done
+  sudo -n true 2>/dev/null || warn "sudo may prompt for a password"
+}
+
+find_ssh_pubkey() {
+  local home="${SUDO_USER:+/home/$SUDO_USER}"
+  home="${home:-$HOME}"
+  for n in id_ed25519 id_ecdsa id_rsa; do
+    [ -f "$home/.ssh/$n.pub" ] && { cat "$home/.ssh/$n.pub"; return; }
+  done
+  die "no SSH public key found in $home/.ssh"
+}
+
+# Populate SELECTED[] from argv (VLAN ids) or the whole config.
+selected_vlans() {
+  SELECTED=()
+  local want=("$@")
+  while IFS= read -r line; do
+    [[ "$line" =~ ^[[:space:]]*# ]] && continue
+    [[ -z "${line// }" ]] && continue
+    local vid="${line%%:*}"
+    if [ ${#want[@]} -eq 0 ]; then
+      SELECTED+=("$line")
+    else
+      for w in "${want[@]}"; do [ "$w" = "$vid" ] && SELECTED+=("$line"); done
+    fi
+  done < "$CONF"
+  [ ${#SELECTED[@]} -gt 0 ] || die "no VLANs selected (checked $CONF)"
+}
+
+# Split one vlans.conf line, applying defaults for the two optional trailing
+# fields. Sets V_VID V_NAME V_PREFIX V_REAL V_MASK V_HOST.
+parse_vlan_entry() {
+  IFS=: read -r V_VID V_NAME V_PREFIX V_REAL V_MASK V_HOST <<<"$1"
+  V_MASK="${V_MASK:-24}"
+  V_HOST="${V_HOST:-2}"
+}
+
+# Dotted netmask for a prefix length — cloud-init's network-config v1 wants the
+# dotted form, not a /len. /24 -> 255.255.255.0, /23 -> 255.255.254.0.
+netmask_for() {
+  local len="$1" i bits out=()
+  for i in 0 1 2 3; do
+    bits=$(( len - i * 8 ))
+    (( bits > 8 )) && bits=8
+    (( bits < 0 )) && bits=0
+    out+=( $(( 256 - 2 ** (8 - bits) )) )
+  done
+  local IFS=.; echo "${out[*]}"
+}
+
+# cloud-init NoCloud seed: static addressing + SSH key + hello-world HTTP.
+build_seed() {
+  local iso="$1" vm="$2" vid="$3" name="$4" prefix="$5" ip="$6" real="$7" pubkey="$8"
+  local masklen="${9:-24}"
+  local netmask; netmask="$(netmask_for "$masklen")"
+  local tmp; tmp="$(mktemp -d)"
+
+  cat > "$tmp/meta-data" <<EOF
+instance-id: $vm
+local-hostname: $vm
+EOF
+
+  # Alpine's cloud-init does not reliably apply netplan-style network-config,
+  # and these networks have no DHCP server on purpose — so configure the
+  # interface the Alpine-native way instead (verified: hostname applied but no
+  # address, i.e. the seed was read and network-config was ignored).
+  #
+  # The default route deliberately points at the router under test (.1), not
+  # the host (.2), so a broken/absent router shows up as a failed test rather
+  # than being silently papered over by host routing. post-up ... || true keeps
+  # the interface up even while no router exists yet.
+  cat > "$tmp/network-config" <<EOF
+version: 1
+config:
+  - type: physical
+    name: eth0
+    subnets:
+      - type: static
+        address: $ip
+        netmask: $netmask
+        # Default route via the router under test. Without this the VMs can
+        # reach their own /24 and their gateway, but nothing beyond it — which
+        # looks exactly like "the router is broken" in the matrix.
+        gateway: ${prefix}.1
+EOF
+
+  cat > "$tmp/user-data" <<EOF
+#cloud-config
+hostname: $vm
+users:
+  - name: alpine
+    # NOTE: this Alpine image ships no sudo (and cloud-init's sudo: directive
+    # is therefore inert). For privileged work in these VMs, ssh as root —
+    # the key is installed there too.
+    shell: /bin/ash
+    # Without this cloud-init leaves the account locked ("!*" in /etc/shadow)
+    # and sshd refuses key auth for it — verified on the first build.
+    lock_passwd: false
+    plain_text_passwd: labsim
+    ssh_authorized_keys:
+      - $pubkey
+ssh_authorized_keys:
+  - $pubkey
+disable_root: false
+chpasswd:
+  list: |
+    root:labsim
+  expire: false
+write_files:
+  - path: /etc/network/interfaces
+    content: |
+      auto lo
+      iface lo inet loopback
+      auto eth0
+      iface eth0 inet static
+        address $ip
+        netmask $netmask
+        post-up ip route add default via ${prefix}.1 || true
+  - path: /var/www/index.html
+    content: |
+      <html><body>
+      <h1>labsim vlan $vid — $name</h1>
+      <p>host: $vm</p>
+      <p>address: $ip/$masklen</p>
+      <p>gateway under test: ${prefix}.1</p>
+      <p>mirrors production: $real</p>
+      </body></html>
+  - path: /etc/local.d/labsim-http.start
+    permissions: '0755'
+    content: |
+      #!/bin/sh
+      # This image's busybox has no httpd applet ("applet not found"), and the
+      # VMs are isolated so apk cannot fetch one. python3 is already present
+      # (cloud-init depends on it), so serve with http.server — no packages,
+      # no internet.
+      #
+      # Two traps already hit here, both silent:
+      #  - start-stop-daemon --exec /usr/bin/python3 matches cloud-init's OWN
+      #    python3 at boot, says "already running", starts nothing.
+      #  - busybox pgrep -f PATTERN matches its own argv, so a
+      #    "skip if running" guard always fires (verified: guard_exit=0 with
+      #    nothing listening).
+      # So: no guard, no start-stop-daemon. Binding twice is harmless — the
+      # second just fails to bind.
+      nohup /usr/bin/python3 -m http.server 80 --directory /var/www \\
+        >/var/log/labsim-http.log 2>&1 &
+runcmd:
+  # cloud-init's network-config (v1, above) already applies the address, so do
+  # NOT restart networking here — it fails, and one failing runcmd aborts every
+  # command after it, which is what silently left httpd unstarted. Each command
+  # is || true for the same reason.
+  - [ sh, -c, "rc-update add sshd default || true" ]
+  - [ sh, -c, "rc-update add local default || true" ]
+  - [ sh, -c, "/etc/local.d/labsim-http.start || true" ]
+EOF
+
+  # Validate before building the ISO. The heredoc above is intentionally
+  # unquoted (it interpolates $ip/$prefix), which means backticks or $( ) in
+  # ANY line — including comments — get executed by the host shell and their
+  # output silently corrupts the YAML. Cheap check, expensive bug.
+  python3 -c "import yaml,sys; yaml.safe_load(open(sys.argv[1]))" "$tmp/user-data" \
+    || die "generated user-data is not valid YAML (backticks or \$( ) in build_seed?): $tmp/user-data"
+
+  sudo genisoimage -quiet -output "$iso" -volid cidata -joliet -rock \
+    "$tmp/user-data" "$tmp/meta-data" "$tmp/network-config"
+  rm -rf "$tmp"
+}
+
+ssh_to() {
+  local ip="$1"; shift
+  timeout 12 ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
+    -o ConnectTimeout=5 -o BatchMode=yes -o LogLevel=ERROR \
+    "alpine@$ip" "$@" 2>/dev/null
+}
+
+wait_ready() {
+  local deadline=$((SECONDS + 240)) pending=1
+  while [ $SECONDS -lt $deadline ]; do
+    pending=0
+    for entry in "${SELECTED[@]}"; do
+      IFS=: read -r vid _n prefix _r <<<"$entry"
+      # Wait for BOTH: sshd is up well before cloud-init's runcmd starts the
+      # web server, so checking SSH alone reports "ready" then shows HTTP FAIL.
+      ssh_to "${prefix}.10" true >/dev/null 2>&1 \
+        && curl -sS -o /dev/null --max-time 4 "http://${prefix}.10/" 2>/dev/null \
+        || pending=$((pending + 1))
+    done
+    [ $pending -eq 0 ] && { log "all ${#SELECTED[@]} VMs reachable"; return 0; }
+    sleep 5
+  done
+  warn "$pending VM(s) still not answering SSH after 240s — see status below"
+  return 0
+}
+
+status_table() {
+  printf '  %-18s %-6s %-16s %-9s %-7s %s\n' VM VLAN ADDRESS STATE SSH HTTP
+  printf '  %-18s %-6s %-16s %-9s %-7s %s\n' ------------------ ------ ---------------- --------- ------- ----
+  for entry in "${SELECTED[@]}"; do
+    IFS=: read -r vid name prefix _r <<<"$entry"
+    local vm ip state ssh http
+    vm="$(vm_name "$vid" "$name")"; ip="${prefix}.10"
+    state="$(virsh_q domstate "$vm" 2>/dev/null | head -1 | tr -d '\n')"
+    [ -z "$state" ] && state="absent"
+    ssh_to "$ip" true >/dev/null 2>&1 && ssh=ok || ssh=FAIL
+    if curl -sS -o /dev/null --max-time 5 "http://$ip/" 2>/dev/null; then http=ok; else http=FAIL; fi
+    printf '  %-18s %-6s %-16s %-9s %-7s %s\n' "$vm" "$vid" "$ip" "$state" "$ssh" "$http"
+  done
+}

labsim/monitoring-up.sh

@@ -0,0 +1,82 @@
+#!/bin/bash
+# Prometheus + Grafana for the labsim connectivity matrix.
+#
+# Grafana runs with anonymous auth as Admin — NO LOGIN. That is deliberate for
+# a throwaway lab on localhost; do not copy this into anything reachable.
+#
+#   ./monitoring-up.sh          start exporter + prometheus + grafana
+#   ./monitoring-up.sh --down   stop and remove them
+#
+# Grafana:    http://localhost:3000   (dashboard "labsim — VLAN connectivity matrix")
+# Prometheus: http://localhost:9090
+# Exporter:   http://localhost:9101/metrics
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+source "$SCRIPT_DIR/lib.sh"
+
+GRAFANA_PORT="${GRAFANA_PORT:-3000}"
+PROM_PORT="${PROM_PORT:-9090}"
+EXPORTER_PORT="${EXPORTER_PORT:-9101}"
+NET="labsim-mon"
+
+if [ "${1:-}" = "--down" ]; then
+  pkill -f "labsim-exporter.py" 2>/dev/null || true
+  podman rm -f labsim-grafana labsim-prometheus >/dev/null 2>&1 || true
+  podman network rm -f "$NET" >/dev/null 2>&1 || true
+  log "monitoring stopped"
+  exit 0
+fi
+
+command -v podman >/dev/null 2>&1 || die "podman not installed"
+
+# --- exporter (on the host: it needs SSH access to the VMs) ----------------
+if pgrep -f "labsim-exporter.py" >/dev/null 2>&1; then
+  log "exporter already running on :$EXPORTER_PORT"
+else
+  log "starting exporter on :$EXPORTER_PORT"
+  nohup "$SCRIPT_DIR/labsim-exporter.py" --port "$EXPORTER_PORT" --interval 15 \
+    > /tmp/labsim-exporter.log 2>&1 &
+  sleep 3
+fi
+curl -sS --max-time 5 "http://127.0.0.1:${EXPORTER_PORT}/metrics" >/dev/null \
+  || die "exporter not answering on :$EXPORTER_PORT (see /tmp/labsim-exporter.log)"
+
+podman network exists "$NET" 2>/dev/null || podman network create "$NET" >/dev/null
+
+# --- prometheus -----------------------------------------------------------
+podman rm -f labsim-prometheus >/dev/null 2>&1 || true
+log "starting prometheus on :$PROM_PORT"
+podman run -d --name labsim-prometheus --network "$NET" \
+  -p "${PROM_PORT}:9090" \
+  -v "$SCRIPT_DIR/monitoring/prometheus.yml:/etc/prometheus/prometheus.yml:ro,Z" \
+  --add-host "host.containers.internal:host-gateway" \
+  docker.io/prom/prometheus:latest >/dev/null
+
+# --- grafana (anonymous, no login) ----------------------------------------
+podman rm -f labsim-grafana >/dev/null 2>&1 || true
+log "starting grafana on :$GRAFANA_PORT (anonymous auth — no password)"
+podman run -d --name labsim-grafana --network "$NET" \
+  -p "${GRAFANA_PORT}:3000" \
+  -e GF_AUTH_ANONYMOUS_ENABLED=true \
+  -e GF_AUTH_ANONYMOUS_ORG_ROLE=Admin \
+  -e GF_AUTH_DISABLE_LOGIN_FORM=true \
+  -e GF_AUTH_BASIC_ENABLED=false \
+  -e GF_SECURITY_ALLOW_EMBEDDING=true \
+  -e GF_USERS_DEFAULT_THEME=dark \
+  -v "$SCRIPT_DIR/monitoring/grafana/provisioning:/etc/grafana/provisioning:ro,Z" \
+  docker.io/grafana/grafana:latest >/dev/null
+
+log "waiting for grafana..."
+for _ in $(seq 1 40); do
+  if curl -sS --max-time 3 "http://127.0.0.1:${GRAFANA_PORT}/api/health" >/dev/null 2>&1; then
+    break
+  fi
+  sleep 3
+done
+
+echo
+log "Topology:   http://localhost:${EXPORTER_PORT}/   <- live mesh, red/green + RTT"
+log "Grafana:    http://localhost:${GRAFANA_PORT}/d/labsim-matrix  (no login, history)"
+log "Prometheus: http://localhost:${PROM_PORT}"
+log "Exporter:   http://localhost:${EXPORTER_PORT}/metrics"

labsim/monitoring/grafana/provisioning/dashboards/dashboards.yml

@@ -0,0 +1,9 @@
+apiVersion: 1
+providers:
+  - name: labsim
+    folder: ''
+    type: file
+    disableDeletion: false
+    updateIntervalSeconds: 10
+    options:
+      path: /etc/grafana/provisioning/dashboards

labsim/monitoring/grafana/provisioning/dashboards/labsim.json

@@ -0,0 +1,59 @@
+{
+  "uid": "labsim-matrix",
+  "title": "labsim — VLAN connectivity matrix",
+  "tags": ["labsim"],
+  "timezone": "browser",
+  "refresh": "10s",
+  "time": { "from": "now-30m", "to": "now" },
+  "panels": [
+    {
+      "type": "stat",
+      "title": "Reachable paths",
+      "gridPos": { "h": 4, "w": 6, "x": 0, "y": 0 },
+      "targets": [ { "expr": "sum(labsim_reachable)", "refId": "A" } ],
+      "fieldConfig": { "defaults": { "thresholds": { "mode": "absolute",
+        "steps": [ { "color": "red", "value": null }, { "color": "green", "value": 90 } ] } } }
+    },
+    {
+      "type": "stat",
+      "title": "Blocked paths",
+      "gridPos": { "h": 4, "w": 6, "x": 6, "y": 0 },
+      "targets": [ { "expr": "count(labsim_reachable == 0) or vector(0)", "refId": "A" } ],
+      "fieldConfig": { "defaults": { "thresholds": { "mode": "absolute",
+        "steps": [ { "color": "green", "value": null }, { "color": "orange", "value": 1 } ] } } }
+    },
+    {
+      "type": "stat",
+      "title": "Sweep duration (s)",
+      "gridPos": { "h": 4, "w": 6, "x": 12, "y": 0 },
+      "targets": [ { "expr": "labsim_sweep_seconds", "refId": "A" } ]
+    },
+    {
+      "type": "stat",
+      "title": "Sweeps",
+      "gridPos": { "h": 4, "w": 6, "x": 18, "y": 0 },
+      "targets": [ { "expr": "labsim_sweep_total", "refId": "A" } ]
+    },
+    {
+      "type": "heatmap",
+      "title": "ICMP matrix (src → dst) — green = reachable",
+      "gridPos": { "h": 10, "w": 24, "x": 0, "y": 4 },
+      "targets": [ { "expr": "labsim_reachable{proto=\"icmp\"}",
+                     "legendFormat": "{{src}} → {{dst}}", "refId": "A" } ]
+    },
+    {
+      "type": "state-timeline",
+      "title": "Every path over time — a firewall change shows up here immediately",
+      "gridPos": { "h": 12, "w": 24, "x": 0, "y": 14 },
+      "targets": [ { "expr": "labsim_reachable",
+                     "legendFormat": "{{proto}} {{src}} → {{dst}}", "refId": "A" } ],
+      "fieldConfig": { "defaults": {
+        "mappings": [ { "type": "value", "options": {
+          "0": { "text": "blocked", "color": "red", "index": 0 },
+          "1": { "text": "ok", "color": "green", "index": 1 } } } ] } },
+      "options": { "mergeValues": true, "showValue": "never" }
+    }
+  ],
+  "schemaVersion": 39,
+  "version": 1
+}

labsim/monitoring/grafana/provisioning/datasources/prometheus.yml

@@ -0,0 +1,7 @@
+apiVersion: 1
+datasources:
+  - name: Prometheus
+    type: prometheus
+    access: proxy
+    url: http://labsim-prometheus:9090
+    isDefault: true

labsim/monitoring/prometheus.yml

@@ -0,0 +1,9 @@
+# Scrapes the labsim connectivity exporter running on the host.
+global:
+  scrape_interval: 15s
+  evaluation_interval: 15s
+
+scrape_configs:
+  - job_name: labsim
+    static_configs:
+      - targets: ['host.containers.internal:9101']

labsim/ovs.sh

@@ -0,0 +1,254 @@
+#!/bin/bash
+# Open vSwitch fabric for labsim — the "switch" the whole sim hangs off.
+#
+# Why OVS and not a Linux bridge: a Linux bridge cannot do LACP at all, and its
+# VLAN support is awkward to drive from libvirt. OVS gives real 802.1Q access
+# and trunk ports plus real LACP bonds, so a router VM can run the SAME bond0 +
+# vif config as the production VP2440s instead of an approximation.
+#
+# Layout:
+#   ovs-labsim            the switch
+#     ├─ vm ports         access ports, tag=<vlan>       (micro VM per VLAN)
+#     ├─ hostv<vlan>      internal ports, tag=<vlan>     (host leg, for SSH)
+#     └─ lag-vyos         LACP bond, trunk of all VLANs  (router under test)
+# shellcheck disable=SC2034
+
+OVS_BR="${OVS_BR:-ovs-labsim}"
+OVS_NET="${OVS_NET:-labsim-ovs}"          # libvirt network wrapping the bridge
+LAG_NAME="${LAG_NAME:-lag-vyos}"
+
+# Native (untagged) VLAN on the trunks to the routers. Empty means NONE: every
+# VLAN, Management included, is tagged.
+#
+# This is not a style choice. A native VLAN is what puts a subnet on the bond
+# PARENT (`bond0`) while every other VLAN lives on a sub-interface of it. With
+# `dhcp-socket-type: raw`, kea then receives each tagged frame TWICE -- once on
+# the sub-interface and once on the parent -- and answers from the parent's pool
+# as well, so a client on VLAN 3 is offered a Management address and picks
+# whichever reply arrives first (ISC Kea #1117).
+#
+# Set LABSIM_NATIVE_VLAN=1 to restore the old shape and reproduce the bug:
+#   LABSIM_NATIVE_VLAN=1 ./router-up.sh && ./labsim-vlan-leak-test.sh   # FAIL
+#                        ./router-up.sh && ./labsim-vlan-leak-test.sh   # PASS
+NATIVE_VLAN="${LABSIM_NATIVE_VLAN:-}"
+
+ovs() { sudo ovs-vsctl "$@"; }
+
+ovs_require() {
+  command -v ovs-vsctl >/dev/null 2>&1 || die "openvswitch not installed (dnf install openvswitch)"
+  systemctl is-active --quiet openvswitch || sudo systemctl start openvswitch \
+    || die "could not start openvswitch"
+}
+
+# A comma-separated VLAN list, numerically sorted, for comparing two lists that
+# came from different places and need not agree on order.
+vlan_sorted() { echo "$1" | tr ',' '\n' | grep -v '^$' | sort -n | paste -sd, -; }
+
+# All VLAN ids from the config, comma separated — used for trunk ports.
+vlan_id_list() {
+  local ids=()
+  for entry in "${SELECTED[@]}"; do ids+=("${entry%%:*}"); done
+  (IFS=,; echo "${ids[*]}")
+}
+
+ovs_up() {
+  ovs_require
+  ovs --may-exist add-br "$OVS_BR"
+
+  # Host leg per VLAN: an OVS internal port carrying that VLAN's tag, given the
+  # .2 address. This is how you SSH to the VMs. It is deliberately NOT their
+  # default route (.1 is), so inter-VLAN tests exercise the router, not the
+  # host's routing table.
+  for entry in "${SELECTED[@]}"; do
+    parse_vlan_entry "$entry"
+    local port="hostv${V_VID}"
+    ovs --may-exist add-port "$OVS_BR" "$port" tag="$V_VID" \
+      -- set interface "$port" type=internal
+    sudo ip link set "$port" up 2>/dev/null || true
+    # Drop any address from a previous mask/octet so a changed vlans.conf does
+    # not leave a stale second address on the port.
+    sudo ip -4 addr flush dev "$port" 2>/dev/null || true
+    # host_octet 0 means "no host leg": the WAN transport VLANs belong to the
+    # fake ISPs, and giving the host an address there would misrepresent the
+    # segment -- the whole point is that VyOS reaches an ISP, not the host.
+    if [ "$V_HOST" != "0" ]; then
+      sudo ip addr replace "${V_PREFIX}.${V_HOST}/${V_MASK}" dev "$port"
+    fi
+  done
+
+  ovs_define_libvirt_net
+}
+
+# A libvirt network that hands out OVS ports: one portgroup per VLAN (access)
+# plus a trunk portgroup for the router.
+ovs_define_libvirt_net() {
+  local pg="" ids
+  for entry in "${SELECTED[@]}"; do
+    IFS=: read -r vid name _p _r <<<"$entry"
+    pg+="  <portgroup name='vlan${vid}'>
+    <vlan><tag id='${vid}'/></vlan>
+  </portgroup>
+"
+  done
+
+  # Trunk: every VLAN tagged, and by default NO native VLAN (see NATIVE_VLAN at
+  # the top of this file for why -- it is the kea #1117 fix, not tidiness).
+  # libvirt expresses a native VLAN declaratively via nativeMode='untagged'
+  # (see libvirt formatnetwork.html), so it needs no fixing up by hand.
+  #
+  # The worry that a trunk with no native VLAN has nowhere to put LACPDUs is
+  # unfounded, and was tested rather than reasoned about: with vlan_mode=trunk
+  # and no tag, `ovs-appctl bond/show` still reports lacp_status: negotiated
+  # with both members enabled. LACPDUs are slow-protocol frames handled per
+  # member, below the VLAN layer.
+  local trunk="  <portgroup name='trunk'>
+    <vlan trunk='yes'>
+"
+  for entry in "${SELECTED[@]}"; do
+    IFS=: read -r vid _n _p _r <<<"$entry"
+    if [ -n "$NATIVE_VLAN" ] && [ "$vid" = "$NATIVE_VLAN" ]; then
+      trunk+="      <tag id='${vid}' nativeMode='untagged'/>
+"
+    else
+      trunk+="      <tag id='${vid}'/>
+"
+    fi
+  done
+  trunk+="    </vlan>
+  </portgroup>
+"
+
+  local xml="<network>
+  <name>${OVS_NET}</name>
+  <forward mode='bridge'/>
+  <bridge name='${OVS_BR}'/>
+  <virtualport type='openvswitch'/>
+${pg}${trunk}</network>"
+
+  if virsh_q net-info "$OVS_NET" >/dev/null 2>&1; then
+    virsh_q net-destroy "$OVS_NET" >/dev/null 2>&1 || true
+    virsh_q net-undefine "$OVS_NET" >/dev/null 2>&1 || true
+  fi
+  echo "$xml" | virsh_q net-define /dev/stdin >/dev/null
+  virsh_q net-start "$OVS_NET" >/dev/null
+  log "libvirt network $OVS_NET bound to $OVS_BR (access portgroups + trunk)"
+}
+
+# Replace the router VM's two individual OVS ports with a single LACP bond.
+# libvirt attaches each NIC separately; only ovs-vsctl can bond them, and the
+# taps only exist once the VM is running — so this runs post-start.
+ovs_bond_router() {
+  local vm="$1"
+  local taps
+  # Two filters, both load-bearing:
+  #
+  #   $1 ~ /^vnet/ -- domiflist indents its rows, so anchor on the FIELD, not
+  #   the line. /^vnet/ silently matches nothing and the bond never gets built.
+  #
+  #   $3 == OVS_NET -- count only the taps on the sim fabric. The primary also
+  #   carries a libvirt-NAT scaffold NIC (see --drop-scaffold in the README), so
+  #   an unfiltered count is 3, and this function's "expected 2" guard then
+  #   skipped the primary's bond entirely while reporting only a warning.
+  taps="$(virsh_q domiflist "$vm" 2>/dev/null \
+          | awk -v net="$OVS_NET" '$1 ~ /^vnet/ && $3 == net {print $1}')"
+  local count; count="$(echo "$taps" | grep -c .)"
+  [ "$count" -eq 2 ] \
+    || { warn "router $vm has $count tap(s) on $OVS_NET, expected 2 — skipping bond"; return 1; }
+
+  local t1 t2; t1="$(echo "$taps" | sed -n 1p)"; t2="$(echo "$taps" | sed -n 2p)"
+  local want; want="$(vlan_id_list)"
+
+  # Already bonded? Re-runs must still reconcile BOTH the VLAN list and the
+  # membership, and each has drawn blood:
+  #
+  #   VLANs -- adding a VLAN to vlans.conf and finding the bond unchanged is how
+  #   a VLAN silently fails to reach a router: interface present, tag missing,
+  #   frames dropped by the switch.
+  #
+  #   MEMBERS -- restarting the VM recreates its taps with NEW names, leaving the
+  #   bond holding two interfaces that no longer exist. `list-ports` still shows
+  #   the bond, so this early return declared success while the router's real
+  #   taps sat in the bridge as two INDEPENDENT ports, each carrying libvirt's
+  #   own portgroup VLAN config. That is how labsim-vyos2 ran for weeks with no
+  #   LACP at all and a native VLAN nobody had asked for -- and it is invisible
+  #   until you change the trunk and only one router follows.
+  if ovs list-ports "$OVS_BR" 2>/dev/null | grep -qx "$LAG_NAME"; then
+    local members; members="$(ovs-appctl-members)"
+    if [ "$members" != "$(printf '%s\n%s' "$t1" "$t2" | sort | paste -sd, -)" ]; then
+      warn "bond $LAG_NAME holds stale members [$members], VM has [$t1,$t2] — rebuilding"
+      ovs --if-exists del-port "$OVS_BR" "$LAG_NAME"
+    else
+      # Compare as SETS. vlan_id_list yields config order (1,2,3,9,10,200,51,53)
+      # while OVS returns its own sorted order, so a raw string compare reports
+      # drift on every run and rewrites a trunk that was already correct.
+      local have; have="$(ovs get port "$LAG_NAME" trunks 2>/dev/null | tr -d '[] ')"
+      if [ "$(vlan_sorted "$want")" != "$(vlan_sorted "$have")" ]; then
+        log "bond $LAG_NAME trunk drift: [$have] -> [$want]; updating"
+        ovs set port "$LAG_NAME" trunks="$want"
+      else
+        log "LACP bond $LAG_NAME already present, trunk correct"
+      fi
+      ovs_set_native "$LAG_NAME"
+      return 0
+    fi
+  fi
+
+  log "bonding $t1 + $t2 into $LAG_NAME (LACP active, balance-tcp)"
+  ovs del-port "$OVS_BR" "$t1" 2>/dev/null || true
+  ovs del-port "$OVS_BR" "$t2" 2>/dev/null || true
+
+  # bond_mode=balance-tcp is REQUIRED: OVS defaults a bond to active-backup,
+  # which does not speak LACP at all (confirmed on ovs-discuss). It is also the
+  # equivalent of VyOS's 802.3ad + layer2+3 hashing.
+  #
+  # lacp-fallback-ab breaks a genuine deadlock: OVS keeps members disabled
+  # until LACP negotiates, while the partner needs carrier before it will send
+  # LACPDUs. Falling back to active-backup brings the links up so negotiation
+  # can start.
+  #
+  # The VLAN mode is set inline: libvirt's portgroup config does NOT apply here,
+  # because the bond is a port libvirt never created.
+  ovs add-bond "$OVS_BR" "$LAG_NAME" "$t1" "$t2" \
+      lacp=active bond_mode=balance-tcp trunks="$want" \
+    -- set port "$LAG_NAME" other_config:lacp-time=fast \
+    -- set port "$LAG_NAME" other_config:lacp-fallback-ab=true
+  ovs_set_native "$LAG_NAME"
+}
+
+# The bond's current members, sorted and comma-joined, or empty if the bond does
+# not resolve at all (which is itself the stale case worth rebuilding for).
+ovs-appctl-members() {
+  sudo ovs-appctl bond/show "$LAG_NAME" 2>/dev/null \
+    | awk '/^member /{gsub(/:/,"",$2); print $2}' | sort | paste -sd, -
+}
+
+# Apply NATIVE_VLAN to a trunk port.
+#
+# `tag` MUST be removed, not merely left alone, when there is no native VLAN.
+# Setting vlan_mode=trunk while a stale `tag` remains looks correct in
+# `ovs-vsctl list port` -- it prints vlan_mode: trunk right next to tag: 1 --
+# but the port keeps egressing that VLAN untagged. Half an hour went into
+# "the router is ignoring the trunk change" before the tag was the answer.
+ovs_set_native() {
+  local port="$1"
+  if [ -n "$NATIVE_VLAN" ]; then
+    ovs set port "$port" vlan_mode=native-untagged tag="$NATIVE_VLAN"
+  else
+    ovs set port "$port" vlan_mode=trunk -- clear port "$port" tag
+  fi
+}
+
+ovs_bond_status() {
+  echo "--- ovs bond ---"
+  sudo ovs-appctl bond/show "$LAG_NAME" 2>/dev/null | grep -E "bond_mode|lacp_status|^member|may_enable" || echo "(no bond)"
+  echo "--- lacp ---"
+  sudo ovs-appctl lacp/show "$LAG_NAME" 2>/dev/null | grep -E "status|aggregation key|^member|attached" || true
+}
+
+ovs_down() {
+  virsh_q net-destroy "$OVS_NET" >/dev/null 2>&1 || true
+  virsh_q net-undefine "$OVS_NET" >/dev/null 2>&1 || true
+  if command -v ovs-vsctl >/dev/null 2>&1; then
+    ovs --if-exists del-br "$OVS_BR" 2>/dev/null || true
+  fi
+}

labsim/router-install.py

@@ -0,0 +1,162 @@
+#!/usr/bin/env python3
+"""Drive the labsim VyOS router over its serial console.
+
+Three phases:
+  --phase live       wait for the live system and log in
+  --phase install    run `install image` unattended
+  --phase configure  apply bond0 (LACP) + per-VLAN gateway addresses
+
+The installer prompt list is the same one the bastion's install driver answers
+(src/bastion/src/templates/vyos-install.py.ts). Two of them are easy to miss and
+both hang forever rather than failing: the reinstall-only "copy data to the new
+image?", and "choose two disks for RAID-1 mirroring?" — every RAID prompt
+defaults to YES.
+"""
+from __future__ import annotations
+
+import argparse
+import sys
+import time
+
+import pexpect
+
+PASSWORD = "vyos"
+PROMPT = r"[\$#] $"
+
+
+def console(vm: str, timeout: int = 60) -> pexpect.spawn:
+    c = pexpect.spawn(f"sudo virsh console {vm} --force", encoding="utf-8", timeout=timeout)
+    c.expect("Connected to domain", timeout=30)
+    return c
+
+
+def login(c: pexpect.spawn, timeout: int = 300) -> None:
+    """Get to a shell prompt, whether we land at a login or an open session."""
+    deadline = time.time() + timeout
+    while time.time() < deadline:
+        c.sendline("")
+        i = c.expect(["login:", PROMPT, pexpect.TIMEOUT], timeout=20)
+        if i == 0:
+            c.sendline("vyos")
+            c.expect("assword:", timeout=20)
+            c.sendline(PASSWORD)
+            j = c.expect([PROMPT, "incorrect", pexpect.TIMEOUT], timeout=30)
+            if j == 0:
+                return
+        elif i == 1:
+            return
+    raise SystemExit("timed out waiting for a VyOS shell")
+
+
+def run(c: pexpect.spawn, cmd: str, timeout: int = 60) -> str:
+    c.sendline(cmd)
+    c.expect(PROMPT, timeout=timeout)
+    return c.before or ""
+
+
+def phase_install(c: pexpect.spawn) -> None:
+    """Answer `install image` end to end."""
+    rules: list[tuple[str, str]] = [
+        (r"Would you like to continue\?", "yes"),
+        (r"What would you like to name this image\?", ""),
+        (r"Please confirm password for the .vyos. user:", PASSWORD),
+        (r"Please enter a password for the .vyos. user:", PASSWORD),
+        (r"What console should be used by default", "K"),
+        # every RAID variant defaults to YES — decline them all
+        (r"Would you like to [^?]*RAID-1 mirroring", "no"),
+        (r"Installation will delete all data on (?:the drive|both drives)\. Continue\?", "yes"),
+        (r"Which one should be used for installation\?", "/dev/vda"),
+        (r"Would you like to use all the free space on the drive\?", "yes"),
+        (r"Which file would you like as boot config\?", "1"),
+        # reinstall-only; unanswered it blocks on stdin until the world ends
+        (r"Would you like to copy data to the new image\?", "yes"),
+        (r"From which image would you like to save config information\?", "1"),
+    ]
+    patterns = [r for r, _ in rules] + [r"The image installed successfully",
+                                        r"Unable to install VyOS", pexpect.TIMEOUT]
+
+    c.sendline("install image")
+    for _ in range(60):
+        i = c.expect(patterns, timeout=180)
+        if i < len(rules):
+            c.sendline(rules[i][1])
+            continue
+        if i == len(rules):
+            print("  installer: success")
+            return
+        if i == len(rules) + 1:
+            raise SystemExit("installer reported failure")
+        raise SystemExit("installer went quiet (unanswered prompt?)")
+    raise SystemExit("installer exceeded expected prompt count")
+
+
+def phase_configure(c: pexpect.spawn, vlans: list[tuple[str, str, str]]) -> None:
+    """bond0 over eth0+eth1 with LACP, then a gateway address per VLAN."""
+    # Production shape: VLAN 1 (management) is the NATIVE/untagged VLAN on the
+    # bond, everything else is a tagged vif. This matters beyond fidelity —
+    # LACPDUs are untagged, so a trunk with no native VLAN has nowhere to put
+    # them and the bond never negotiates.
+    native = [v for v in vlans if v[0] == "1"]
+    tagged = [v for v in vlans if v[0] != "1"]
+
+    cmds = [
+        "configure",
+        "set interfaces bonding bond0 mode '802.3ad'",
+        "set interfaces bonding bond0 hash-policy 'layer2+3'",
+        "set interfaces bonding bond0 lacp-rate 'fast'",
+        "set interfaces bonding bond0 member interface 'eth0'",
+        "set interfaces bonding bond0 member interface 'eth1'",
+        "set service ssh port '22'",
+        "set system login user vyos authentication plaintext-password 'vyos'",
+    ]
+    for vid, name, prefix in native:
+        cmds.append(f"set interfaces bonding bond0 address '{prefix}.1/24'")
+        cmds.append(f"set interfaces bonding bond0 description '{name} (native)'")
+    for vid, name, prefix in tagged:
+        cmds.append(f"set interfaces bonding bond0 vif {vid} address '{prefix}.1/24'")
+        cmds.append(f"set interfaces bonding bond0 vif {vid} description '{name}'")
+    cmds += ["commit", "save", "exit"]
+
+    for cmd in cmds:
+        out = run(c, cmd, timeout=180)
+        low = out.lower()
+        if "invalid" in low or "syntax error" in low or "commit failed" in low:
+            print(f"  !! {cmd}\n{out.strip()[-300:]}")
+            raise SystemExit(f"config command rejected: {cmd}")
+    print("  config committed and saved")
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--vm", required=True)
+    ap.add_argument("--phase", required=True, choices=["live", "install", "configure"])
+    ap.add_argument("--vlans", default="", help="space separated vid:name:prefix:real entries")
+    args = ap.parse_args()
+
+    c = console(args.vm)
+    try:
+        login(c)
+        if args.phase == "live":
+            print("  live system reachable")
+        elif args.phase == "install":
+            phase_install(c)
+        else:
+            vlans = []
+            for entry in args.vlans.split():
+                parts = entry.split(":")
+                if len(parts) >= 3:
+                    vlans.append((parts[0], parts[1], parts[2]))
+            if not vlans:
+                raise SystemExit("no VLANs passed to configure")
+            phase_configure(c, vlans)
+        return 0
+    finally:
+        try:
+            c.sendline("")
+            c.close(force=True)
+        except Exception:
+            pass
+
+
+if __name__ == "__main__":
+    sys.exit(main())

labsim/router-up.sh

@@ -0,0 +1,124 @@
+#!/bin/bash
+# Add the VyOS router under test to labsim.
+#
+# Mirrors the production VP2440 pair: TWO NICs bonded with LACP carrying a
+# trunk of every VLAN, then bond0.<vlan> sub-interfaces holding the .1 gateway
+# address on each. That is the same config shape the real firewalls run, so a
+# rule tested here means something.
+#
+# NIC model is e1000e, NOT virtio, and that is load-bearing: with virtio the
+# guest's bonding driver reports its slaves "MII Status: down" despite
+# carrier=1 and never emits a single LACPDU, so the bond sits in
+# AD_STATE_DEFAULTED forever. Known issue — see the netdev thread "bonding
+# (IEEE 802.3ad) not working with qemu/virtio"; e1000e fixes it with no other
+# change. 802.3ad also requires the MII link monitor, which virtio cannot back.
+#
+#   host                       OVS "switch"                    VyOS VM
+#   hostv<vlan> (.2) ──────── ovs-labsim ──── lag-vyos ═════ eth0 + eth1
+#                              (tagged)       (LACP, trunk)   └─ bond0.<vlan> = .1
+#
+# Usage: ./router-up.sh            build + install + configure
+#        ./router-up.sh --status   show bond/LACP + interface state
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+source "$SCRIPT_DIR/lib.sh"
+source "$SCRIPT_DIR/ovs.sh"
+
+ROUTER_VM="${ROUTER_VM:-labsim-vyos}"
+ROUTER_MEM="${ROUTER_MEM:-2048}"
+ROUTER_CPUS="${ROUTER_CPUS:-2}"
+ROUTER_DISK_GB="${ROUTER_DISK_GB:-8}"
+VYOS_ISO="${VYOS_ISO:-$IMG_DIR/vyos.iso}"
+VYOS_CACHE="/var/lib/libvirt/images/lab-pxe-cache"
+
+selected_vlans
+
+if [ "${1:-}" = "--status" ]; then
+  ovs_bond_status
+  echo "--- vyos gateway addresses (probed from each host leg) ---"
+  for entry in "${SELECTED[@]}"; do
+    IFS=: read -r vid _n prefix _r <<<"$entry"
+    printf '  vlan %-5s %-16s ' "$vid" "${prefix}.1"
+    ping -c1 -W2 "${prefix}.1" >/dev/null 2>&1 && echo up || echo down
+  done
+  exit 0
+fi
+
+ovs_require
+
+# --- ISO ------------------------------------------------------------------
+if [ ! -f "$VYOS_ISO" ]; then
+  # Reuse the bastion's cached nightly if it is already on this box.
+  if [ -f "$VYOS_CACHE/vyos.iso" ]; then
+    log "reusing cached VyOS ISO"
+    sudo cp "$VYOS_CACHE/vyos.iso" "$VYOS_ISO"
+  else
+    log "resolving latest VyOS nightly ISO..."
+    url="$(curl -sSL https://api.github.com/repos/vyos/vyos-nightly-build/releases/latest \
+      | python3 -c "import json,sys;print(next(a['browser_download_url'] for a in json.load(sys.stdin)['assets'] if a['name'].endswith('generic-amd64.iso')))")"
+    log "downloading $url"
+    sudo curl -sSL --max-time 1800 -o "$VYOS_ISO" "$url"
+  fi
+fi
+[ -f "$VYOS_ISO" ] || die "no VyOS ISO at $VYOS_ISO"
+
+# --- VM -------------------------------------------------------------------
+if virsh_q dominfo "$ROUTER_VM" >/dev/null 2>&1; then
+  log "router VM $ROUTER_VM exists"
+  virsh_q start "$ROUTER_VM" >/dev/null 2>&1 || true
+else
+  log "creating router VM $ROUTER_VM (2 NICs on the trunk, for LACP)"
+  sudo qemu-img create -q -f qcow2 "$IMG_DIR/${ROUTER_VM}.qcow2" "${ROUTER_DISK_GB}G" >/dev/null
+
+  # Two trunk NICs — OVS bonds them after boot (libvirt cannot create bonds).
+  sudo virt-install \
+    --connect "$LIBVIRT_URI" \
+    --name "$ROUTER_VM" \
+    --memory "$ROUTER_MEM" --vcpus "$ROUTER_CPUS" \
+    --disk "path=$IMG_DIR/${ROUTER_VM}.qcow2,format=qcow2,bus=virtio" \
+    --disk "path=$VYOS_ISO,device=cdrom,readonly=on" \
+    --network "network=$OVS_NET,portgroup=trunk,model=e1000e,trustGuestRxFilters=yes" \
+    --network "network=$OVS_NET,portgroup=trunk,model=e1000e,trustGuestRxFilters=yes" \
+    --boot cdrom,hd \
+    --os-variant debian12 \
+    --graphics none --noautoconsole --import >/dev/null
+fi
+
+log "waiting for the live system to boot (VyOS live login)..."
+python3 "$SCRIPT_DIR/router-install.py" --vm "$ROUTER_VM" --phase live || die "live boot failed"
+
+log "installing VyOS to disk (unattended over the console)..."
+python3 "$SCRIPT_DIR/router-install.py" --vm "$ROUTER_VM" --phase install || die "install failed"
+
+# Boot the INSTALLED system from here on. Without this the VM was created with
+# --boot cdrom,hd and every restart re-runs the ISO, so the live system comes
+# back with no config and every `commit; save` silently evaporates.
+log "switching boot to disk and ejecting the install media..."
+virsh_q destroy "$ROUTER_VM" >/dev/null 2>&1 || true
+sleep 2
+sudo virt-xml "$ROUTER_VM" --edit --boot hd >/dev/null
+sudo virt-xml "$ROUTER_VM" --remove-device --disk device=cdrom >/dev/null 2>&1 || true
+virsh_q start "$ROUTER_VM" >/dev/null
+sleep 10
+
+# Bond the taps only now: they are recreated by the restart above, so bonding
+# before this would bond stale interfaces.
+ovs_bond_router "$ROUTER_VM"
+
+log "applying router config (bond0 LACP + VLAN gateways)..."
+python3 "$SCRIPT_DIR/router-install.py" --vm "$ROUTER_VM" --phase configure \
+  --vlans "$(printf '%s\n' "${SELECTED[@]}" | tr '\n' ' ')" || die "configure failed"
+
+log "waiting for LACP to negotiate..."
+for _ in $(seq 1 30); do
+  if sudo ovs-appctl lacp/show "$LAG_NAME" 2>/dev/null | grep -q "current attached"; then
+    log "LACP negotiated"; break
+  fi
+  sleep 5
+done
+
+echo
+ovs_bond_status
+echo
+log "router is up. Check reachability with: $SCRIPT_DIR/labsim-matrix.py --watch 2"

labsim/sim-ha-config.py

@@ -0,0 +1,144 @@
+#!/usr/bin/env python3
+"""Generate the HA config for the labsim VyOS pair.
+
+Exists to answer one question that cannot be answered on a single router, and
+that would otherwise only be discovered at cutover: with kea HA active-passive,
+does exactly ONE box answer a DHCP request?
+
+Mirrors the production shape so the answer transfers:
+
+    router1  172.31.<v>.252   priority 200   DHCP HA primary
+    router2  172.31.<v>.253   priority 100   DHCP HA secondary
+    VIP      172.31.<v>.1     (what clients use as their gateway)
+
+Note the sim's LoT VLAN is a /23 like production, so the VIP prefix differs
+there -- getting that wrong produces a config that commits and then behaves
+subtly wrongly, which is worse than a failure.
+
+  ./sim-ha-config.py --role primary   > r1.conf
+  ./sim-ha-config.py --role secondary > r2.conf
+"""
+from __future__ import annotations
+
+import argparse
+import importlib.util
+import json
+import os
+import sys
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+MIG = os.path.join(HERE, "..", "migration")
+
+# Reuse the DHCP/DNS generator rather than hand-writing subnets: the whole
+# point is that what is proven here and what production gets share a code path.
+_spec = importlib.util.spec_from_file_location(
+    "unifi_to_vyos", os.path.join(MIG, "unifi-to-vyos.py"))
+unifi_to_vyos = importlib.util.module_from_spec(_spec)
+_spec.loader.exec_module(unifi_to_vyos)
+
+# vlan -> (prefix, cidr). LoT is a /23 in the sim, matching production.
+VLANS = {
+    1:   ("172.31.1",   24),
+    2:   ("172.31.2",   24),
+    3:   ("172.31.3",   24),
+    9:   ("172.31.9",   24),
+    10:  ("172.31.10",  23),
+    200: ("172.31.200", 24),
+}
+# VLAN 2 (k8s) also gets a ULA IPv6, so the routers can peer eBGP with the nodes
+# over IPv6 (the neighbors in sim-net-config.py K8S_NODES_V6 = fd00:2::1x). Only
+# VLAN 2 needs it for the BGP rehearsal; a ULA keeps sim traffic out of the real
+# HE /48. Routers hold ::252 / ::253 (no v6 VRRP VIP -- BGP peers the real per-box
+# address, exactly as production).
+VLANS6 = {2: ("fd00:2", 64)}
+DHCP_HA_NAME = "labsim-dhcp-pair"     # must not equal either host-name
+
+
+def group(vlan: int) -> str:
+    return "native" if vlan == 1 else f"vlan{vlan}"
+
+
+def build(role: str) -> list[str]:
+    primary = role == "primary"
+    self_o, peer_o = (252, 253) if primary else (253, 252)
+    prio = 200 if primary else 100
+    out = [f"# labsim VyOS HA -- {role}", ""]
+
+    for vlan, (pfx, cidr) in VLANS.items():
+        g = group(vlan)
+        # EVERY VLAN is a sub-interface, Management (VLAN 1) included. Putting
+        # Management on the bare `bond0` is what gives the parent a subnet, and
+        # kea then answers tagged frames from it as well as from the correct
+        # sub-interface -- clients on other VLANs get offered a Management
+        # address (ISC Kea #1117). See NATIVE_VLAN in ovs.sh; proven by
+        # labsim-vlan-leak-test.sh.
+        iface = f"bond0 vif {vlan}"
+        out += [
+            f"# VLAN {vlan}",
+            # The node's own address replaces the .1 it used to hold directly;
+            # .1 becomes the floating VIP, exactly as production will be.
+            f"delete interfaces bonding {iface} address",
+            f"set interfaces bonding {iface} address '{pfx}.{self_o}/{cidr}'",
+            *([f"set interfaces bonding {iface} address '{VLANS6[vlan][0]}::{self_o}/{VLANS6[vlan][1]}'"]
+              if vlan in VLANS6 else []),
+            f"set high-availability vrrp group {g} interface bond0.{vlan}",
+            f"set high-availability vrrp group {g} vrid {vlan}",
+            f"set high-availability vrrp group {g} address {pfx}.1/{cidr}",
+            f"set high-availability vrrp group {g} priority {prio}",
+            f"set high-availability vrrp group {g} hello-source-address {pfx}.{self_o}",
+            f"set high-availability vrrp group {g} peer-address {pfx}.{peer_o}",
+            f"set high-availability vrrp group {g} no-preempt",
+            f"set high-availability vrrp sync-group MAIN member {g}",
+            "",
+        ]
+
+    out += [
+        "# --- WAN follows VRRP mastership ---",
+        # These four hooks and the health check existed on both live sim VMs but
+        # in NEITHER generator, so `sim-net-apply.sh check` reported "in sync"
+        # while the mechanism under test was pure undetected drift -- exactly
+        # the failure mode this file was written to end.
+        #
+        # The check goes on the SYNC GROUP, not per group: VyOS rejects a
+        # per-group check while the group is in a sync group ("Only sync group
+        # health check will be used").
+        "set high-availability vrrp sync-group MAIN health-check script '/config/vrrp-wan-health'",
+        "set high-availability vrrp sync-group MAIN health-check interval '5'",
+        "set high-availability vrrp sync-group MAIN health-check failure-count '3'",
+        # take and release both exec vrrp-wan-reconcile: one code path, asked at
+        # different moments. `stop` matters as much as `backup` -- a stopped
+        # keepalived is a demotion too, and without it the box would keep the
+        # WAN while holding no VIPs.
+        "set high-availability vrrp sync-group MAIN transition-script master '/config/vrrp-wan-take'",
+        "set high-availability vrrp sync-group MAIN transition-script backup '/config/vrrp-wan-release'",
+        "set high-availability vrrp sync-group MAIN transition-script fault '/config/vrrp-wan-release'",
+        "set high-availability vrrp sync-group MAIN transition-script stop '/config/vrrp-wan-release'",
+        "",
+        "# --- DHCP high-availability ---",
+        "# The thing under test: active-passive should mean exactly one OFFER.",
+        "set service dhcp-server high-availability mode active-passive",
+        f"set service dhcp-server high-availability status {role}",
+        f"set service dhcp-server high-availability name {DHCP_HA_NAME}",
+        f"set service dhcp-server high-availability source-address 172.31.10.{self_o}",
+        f"set service dhcp-server high-availability remote 172.31.10.{peer_o}",
+        "",
+    ]
+
+    inv = json.load(open(os.path.join(MIG, "export", "inventory.json")))
+    dhcp, stats = unifi_to_vyos.build(inv, "sim")
+    out += [l for l in dhcp if l.strip() and not l.startswith("#")]
+    print(f"{role}: {stats['subnets']} subnets, {stats['mappings']} mappings",
+          file=sys.stderr)
+    return out
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--role", choices=("primary", "secondary"), required=True)
+    args = ap.parse_args()
+    sys.stdout.write("\n".join(build(args.role)) + "\n")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

labsim/sim-net-apply.sh

@@ -0,0 +1,71 @@
+#!/usr/bin/env bash
+# Apply -- or drift-check -- the labsim routing config on all four VMs.
+#
+#   ./sim-net-apply.sh check    what the VMs run vs what sim-net-config.py says
+#   ./sim-net-apply.sh apply    push the generated config over the serial console
+#
+# `check` is the one you want most of the time. The whole failure mode this
+# guards against is somebody (including me) fixing something on a VM over SSH
+# and never writing it down, so the next rebuild silently loses it.
+#
+# Applied over the serial console rather than SSH because a freshly installed
+# sim router holds the same addresses as its peer -- there is a window where it
+# is not safely reachable over the network at all. See console-apply.py.
+set -uo pipefail
+HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ACTION="${1:-check}"
+WORK="$(mktemp -d)"; trap 'rm -rf "$WORK"' EXIT
+
+# role : vm : address : regex selecting the subtrees this generator owns
+#
+# primary and secondary now own the SAME subtrees. The secondary's used to omit
+# `interfaces pppoe`, `interfaces bonding`, `nat source` and `protocols
+# failover|static`, so `check` was blind to precisely the WAN config the
+# failover mechanism depends on -- it reported "in sync" for a box that had no
+# WAN at all.
+TARGETS=(
+  "primary:labsim-vyos:172.31.1.252:^set (protocols (bgp|failover|static)|policy (prefix-list|route-map)|nat source rule 1[12]0|interfaces (pppoe|bonding bond0 vif 5[13])|firewall (group interface-group LAN|ipv4|ipv6))"
+  "secondary:labsim-vyos2:172.31.1.253:^set (protocols (bgp|failover|static)|policy (prefix-list|route-map)|nat source rule 1[12]0|interfaces (pppoe|bonding bond0 vif 5[13])|firewall (group interface-group LAN|ipv4|ipv6))"
+  "isp-dhcp:labsim-isp-dhcp:192.168.122.136:^set (interfaces ethernet|nat source|service dhcp-server|firewall ipv4 forward|system host-name)"
+  "isp-pppoe:labsim-isp-pppoe:192.168.122.63:^set (interfaces ethernet|nat source|service pppoe-server|firewall ipv4 forward|system host-name)"
+)
+# Sim-only credential; these VMs hold nothing real and are not reachable from
+# outside the hypervisor.
+SSH_OPTS=(-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null
+          -o LogLevel=ERROR -o PreferredAuthentications=password -o ConnectTimeout=5)
+live() { timeout 30 sshpass -p vyos ssh "${SSH_OPTS[@]}" "vyos@$1" \
+           "/opt/vyatta/bin/vyatta-op-cmd-wrapper show configuration commands" 2>/dev/null; }
+# `vif 53 disable` is RUNTIME state, not config drift. The generators declare it
+# on both routers as the safe resting state (only one box may hold the cloned
+# MAC), and vrrp-wan-reconcile removes it on whichever box currently holds the
+# VIP. Comparing it would therefore report drift on the master for ever, and a
+# check that always cries wolf is a check nobody reads.
+norm() { sed "s/'//g" | grep -v 'hw-id\|offload' \
+           | grep -v 'interfaces bonding bond0 vif 53 disable' | sort -u; }
+
+rc=0
+for t in "${TARGETS[@]}"; do
+  IFS=: read -r role vm addr rx <<<"$t"
+  "$HERE/sim-net-config.py" --role "$role" >"$WORK/$role.conf" 2>/dev/null || {
+    printf '  %-11s GENERATE FAILED\n' "$role"; rc=1; continue; }
+
+  if [ "$ACTION" = apply ]; then
+    printf '  %-11s applying to %s over console...\n' "$role" "$vm"
+    "$HERE/console-apply.py" --vm "$vm" --config "$WORK/$role.conf" || rc=1
+    continue
+  fi
+
+  if ! live "$addr" >"$WORK/$role.live" || [ ! -s "$WORK/$role.live" ]; then
+    printf '  %-11s UNREACHABLE (%s)\n' "$role" "$addr"; rc=1; continue
+  fi
+  grep -E '^set ' "$WORK/$role.conf" | norm >"$WORK/$role.g"
+  grep -E "$rx" "$WORK/$role.live" | norm >"$WORK/$role.l"
+  if d="$(diff "$WORK/$role.g" "$WORK/$role.l")" && [ -z "$d" ]; then
+    printf '  %-11s in sync (%s commands)\n' "$role" "$(wc -l <"$WORK/$role.g")"
+  else
+    printf '  %-11s DRIFT — "<" only in code, ">" only on the VM:\n' "$role"
+    printf '%s\n' "$d" | sed 's/^/      /'
+    rc=1
+  fi
+done
+exit $rc

labsim/sim-net-config.py

@@ -0,0 +1,481 @@
+#!/usr/bin/env python3
+"""Generate the labsim routing + WAN config: BGP, dual WAN, and the two ISP VMs.
+
+`sim-ha-config.py` covers the LAN side of the sim routers (addresses, VRRP,
+conntrack-sync, DHCP). This covers everything that makes the sim a rehearsal for
+production routing rather than just a LAN:
+
+  * eBGP between the sim routers and the k3s nodes, carrying the service range
+  * dual WAN -- DHCP on VLAN 53, PPPoE on VLAN 51 -- with health-checked failover
+  * the two ISP VMs that terminate those WANs and NAT to the real internet
+
+All of it previously existed only as running state on the VMs, applied by hand
+over SSH. Rebuilding a VM lost the rehearsal, and nothing recorded *why* any of
+it was shaped the way it is. That is the entire reason this file exists.
+
+  ./sim-net-config.py --role primary   > r1-net.conf
+  ./console-apply.py --vm labsim-vyos  --config r1-net.conf
+
+or apply all four at once with ./sim-net-apply.sh.
+"""
+from __future__ import annotations
+
+import argparse
+import sys
+
+# ---------------------------------------------------------------------------
+# BGP. Numbers match production so what is proven here ports over unchanged.
+# ---------------------------------------------------------------------------
+ROUTER_AS = 65000
+CLUSTER_AS = 65001
+# The service range Cilium advertises. Chosen against a survey of third-party
+# RFC1918 defaults (docker-desktop, tailscale, k3s, EKS...) so it cannot collide
+# with something we adopt later. Production uses the same /22 -- keep them equal.
+SERVICE_CIDR = "10.61.0.0/22"
+K8S_VLAN = 2
+K8S_NODES = ["172.31.2.11", "172.31.2.12", "172.31.2.13"]
+PEER_GROUP = "K8S"
+PFX_LIST = "K8S-SERVICE-IPS"
+RM_IN, RM_OUT = "K8S-IN", "K8S-OUT"
+
+# IPv6 unicast. Production advertises a public Gateway LoadBalancer /64 out of
+# the HE /48 (2001:470:187e:1e00::/64) and peers over VLAN 2 IPv6. The sim proves
+# the MECHANISM -- ipv6-unicast eBGP, prefix-list6, /128 host routes, ECMP -- with
+# a ULA so nothing here can leak into the real /48. Peering is over VLAN 2 v6
+# (nodes' fd00:2::1x <-> routers' fd00:2::25x, set in sim-ha-config.py /
+# k8s-up.sh), directly connected exactly as v4.
+SERVICE_CIDR_V6 = "fd61:1e00::/64"       # sim analog of prod :1e00::/64 LB pool
+K8S_NODES_V6 = ["fd00:2::11", "fd00:2::12", "fd00:2::13"]
+PEER_GROUP_V6 = "K8S6"
+PFX_LIST_V6 = "K8S-SERVICE-IPS-V6"
+RM_IN_V6, RM_OUT_V6 = "K8S-IN6", "K8S-OUT6"
+# One route per node; ECMP across all three. 4 leaves headroom for a fourth node
+# without a config change.
+MAX_PATHS = 4
+# A safety valve, not a capacity plan: a misconfigured Cilium that starts
+# advertising pod CIDRs should tear the session down, not quietly fill the FIB.
+MAX_PREFIX = 100
+
+# ---------------------------------------------------------------------------
+# Dual WAN. The sim ISPs deliberately use TEST-NET-3 (203.0.113.0/24) and
+# TEST-NET-2 (198.51.100.0/24) from RFC 5737: documentation ranges that are
+# guaranteed never to be real destinations, so a leaked sim route cannot
+# blackhole something that matters.
+# ---------------------------------------------------------------------------
+WAN_DHCP_VLAN = 53          # "10gig-equivalent" -- the primary in production
+WAN_PPPOE_VLAN = 51         # "Vodafone-equivalent" -- the backup
+ISP_DHCP_NET = "203.0.113.0/24"
+ISP_DHCP_GW = "203.0.113.1"
+ISP_DHCP_POOL = ("203.0.113.100", "203.0.113.150")
+ISP_PPPOE_NET = "198.51.100.0/24"
+ISP_PPPOE_GW = "198.51.100.1"
+ISP_PPPOE_POOL = ("198.51.100.100", "198.51.100.150")
+# Sim-only fake credentials. Both ends are in this file on purpose: they
+# authenticate nothing real, and splitting them across a secret store would make
+# the sim unreproducible for no security gain. The PRODUCTION PPPoE password
+# lives in /config/wan-secrets on the router and is never in git.
+PPPOE_USER, PPPOE_PASS = "simdsl", "simpass"
+PPPOE_MTU = 1492            # 1500 - 8 bytes of PPPoE header
+PPPOE_AC = "sim-isp"
+
+# Failover probe targets. NOT 8.8.8.8/8.8.4.4: those are `system name-server`,
+# so a probe failure and a DNS failure would be the same event and the router
+# would flap the WAN every time DNS hiccuped.
+PROBE_TARGETS = ["9.9.9.9", "208.67.222.222"]
+# The bug this shape fixes (WI-8, found here, fixed in production): `ping -I
+# bond0.53` binds the SOURCE address but does not make the kernel use that
+# interface's gateway. On a cold boot where PPPoE won the default route, probes
+# for the 10 gig egressed via PPPoE, succeeded, and the 10 gig was still never
+# selected -- the house ran on the backup line silently. Pinning each target as
+# a /32 via `dhcp-interface` forces the probe onto the line being tested.
+WAN_DHCP_DISTANCE = 210     # NOT `no-default-route`, which blanks new_routers
+PPPOE_DISTANCE = 10         #   in the lease file, leaving failover no gateway
+                            #   to install and silently handing the default
+                            #   route to the backup line.
+
+# One MAC, cloned onto BOTH routers' bond0.53, mirroring production's use of the
+# retired USG's WAN2 MAC to keep its DHCP lease. The sim did not model a shared
+# MAC at all, which is exactly why bond0.53 has to stay on the config plane --
+# only VyOS config can move a MAC between boxes. Locally-administered, sim-only.
+WAN_DHCP_MAC = "02:53:10:61:00:53"
+
+SIM_LAN = "172.31.0.0/16"
+
+# The sim routers' own libvirt-NAT uplink, from before the ISP VMs existed. It
+# is a third default route that does not exist in production and quietly masks
+# WAN failures during a failover test. `--drop-scaffold` removes it.
+SCAFFOLD_IF = "eth2"
+SCAFFOLD_NAT_RULE = 100
+
+
+def bgp(role: str) -> list[str]:
+    """eBGP toward the k3s nodes. Identical on both routers except router-id."""
+    octet = 252 if role == "primary" else 253
+    out = [
+        f"# --- BGP: AS{ROUTER_AS} <-> AS{CLUSTER_AS} (k3s/Cilium) ---",
+        # FRR enforces RFC 8212: an eBGP session with no policy establishes but
+        # exchanges ZERO prefixes, silently. Both directions need a policy or
+        # the session looks perfectly healthy and carries nothing.
+        f"set policy prefix-list {PFX_LIST} rule 10 action permit",
+        f"set policy prefix-list {PFX_LIST} rule 10 prefix {SERVICE_CIDR}",
+        # `le 32` because Cilium advertises individual /32 service addresses out
+        # of the pool, not the aggregate.
+        f"set policy prefix-list {PFX_LIST} rule 10 le 32",
+        f"set policy route-map {RM_IN} rule 10 action permit",
+        f"set policy route-map {RM_IN} rule 10 match ip address prefix-list {PFX_LIST}",
+        # Deny everything outbound. The cluster must never learn a default route
+        # from us -- Cilium would install it and blackhole pod egress.
+        f"set policy route-map {RM_OUT} rule 10 action deny",
+        f"set protocols bgp system-as {ROUTER_AS}",
+        f"set protocols bgp parameters router-id 172.31.{K8S_VLAN}.{octet}",
+        f"set protocols bgp address-family ipv4-unicast maximum-paths ebgp {MAX_PATHS}",
+        f"set protocols bgp peer-group {PEER_GROUP} remote-as {CLUSTER_AS}",
+        f"set protocols bgp peer-group {PEER_GROUP} address-family ipv4-unicast route-map import {RM_IN}",
+        f"set protocols bgp peer-group {PEER_GROUP} address-family ipv4-unicast route-map export {RM_OUT}",
+        f"set protocols bgp peer-group {PEER_GROUP} address-family ipv4-unicast maximum-prefix {MAX_PREFIX}",
+    ]
+    out += [f"set protocols bgp neighbor {n} peer-group {PEER_GROUP}" for n in K8S_NODES]
+    # --- IPv6 unicast: same policy shape, over a separate v6 peer-group ---
+    # `prefix-list6` + `match ipv6 address` are the v6 spellings; `le 128` because
+    # Cilium advertises each Gateway LoadBalancer address as a /128, not the
+    # aggregate. Separate peer-group because the neighbors are v6 addresses; the
+    # export deny is the same safety property (never hand the cluster a default).
+    out += [
+        "",
+        f"set policy prefix-list6 {PFX_LIST_V6} rule 10 action permit",
+        f"set policy prefix-list6 {PFX_LIST_V6} rule 10 prefix {SERVICE_CIDR_V6}",
+        f"set policy prefix-list6 {PFX_LIST_V6} rule 10 le 128",
+        f"set policy route-map {RM_IN_V6} rule 10 action permit",
+        f"set policy route-map {RM_IN_V6} rule 10 match ipv6 address prefix-list {PFX_LIST_V6}",
+        f"set policy route-map {RM_OUT_V6} rule 10 action deny",
+        f"set protocols bgp address-family ipv6-unicast maximum-paths ebgp {MAX_PATHS}",
+        f"set protocols bgp peer-group {PEER_GROUP_V6} remote-as {CLUSTER_AS}",
+        f"set protocols bgp peer-group {PEER_GROUP_V6} address-family ipv6-unicast route-map import {RM_IN_V6}",
+        f"set protocols bgp peer-group {PEER_GROUP_V6} address-family ipv6-unicast route-map export {RM_OUT_V6}",
+        f"set protocols bgp peer-group {PEER_GROUP_V6} address-family ipv6-unicast maximum-prefix {MAX_PREFIX}",
+    ]
+    out += [f"set protocols bgp neighbor {n} peer-group {PEER_GROUP_V6}" for n in K8S_NODES_V6]
+    out.append("")
+    return out
+
+
+def wan(drop_scaffold: bool, role: str = "primary") -> list[str]:
+    """Dual WAN + health-checked failover. IDENTICAL on both routers.
+
+    It used to be primary-only, on the grounds that "two PPPoE clients sharing
+    one credential against a single access concentrator is a different failure
+    mode than anything production has". That was backwards: production has
+    exactly that, and by omitting it the sim could not test the one thing most
+    likely to go wrong. The secondary having no WAN is also why it ended up with
+    zero NAT rules while the primary had ten -- the pair was not comparable.
+
+    Both routers therefore get the same WAN config. What differs is the RESTING
+    STATE, and only for the DHCP line:
+
+      bond0.53  `disable` on BOTH. Its lease is bound to a cloned MAC, and two
+                boxes claiming one MAC is the fault this whole design exists to
+                prevent. vrrp-wan-reconcile removes `disable` on the master.
+      pppoe0    enabled on BOTH, never `disable`d. `disable` unlinks
+                /etc/ppp/peers/pppoe0, which is pppd's own options file, so the
+                promotion path destroyed what it needed. Dialling is gated at
+                the systemd unit instead -- see migration/ppp-vrrp-gate.conf.
+    """
+    out = [
+        "# --- WAN: DHCP (primary) + PPPoE (backup), health-checked ---",
+        f"set interfaces bonding bond0 vif {WAN_PPPOE_VLAN} description "
+        f"'WAN1 Vodafone-equivalent (sim ISP PPPoE)'",
+        f"set interfaces bonding bond0 vif {WAN_DHCP_VLAN} address dhcp",
+        f"set interfaces bonding bond0 vif {WAN_DHCP_VLAN} description "
+        f"'WAN3 10gig-equivalent (sim ISP DHCP)'",
+        f"set interfaces bonding bond0 vif {WAN_DHCP_VLAN} dhcp-options "
+        f"default-route-distance {WAN_DHCP_DISTANCE}",
+        # The cloned MAC. Production clones the old USG's WAN2 MAC so the ISP
+        # keeps handing back the same lease; the sim did not model a shared MAC
+        # at all, which is precisely why bond0.53 must stay on the config plane.
+        # Modelling it lets the sim prove the lease returns to the new master.
+        f"set interfaces bonding bond0 vif {WAN_DHCP_VLAN} mac {WAN_DHCP_MAC}",
+        # Safe at rest on BOTH routers: only the VIP holder enables it.
+        f"set interfaces bonding bond0 vif {WAN_DHCP_VLAN} disable",
+        f"set interfaces pppoe pppoe0 source-interface bond0.{WAN_PPPOE_VLAN}",
+        f"set interfaces pppoe pppoe0 authentication username {PPPOE_USER}",
+        f"set interfaces pppoe pppoe0 authentication password {PPPOE_PASS}",
+        f"set interfaces pppoe pppoe0 default-route-distance {PPPOE_DISTANCE}",
+        f"set interfaces pppoe pppoe0 mtu {PPPOE_MTU}",
+        # The ISP's resolvers would otherwise overwrite ours in resolv.conf every
+        # time the session comes up.
+        "set interfaces pppoe pppoe0 no-peer-dns",
+        "",
+        "# Failover: prefer the DHCP WAN, fall back to PPPoE when probes fail.",
+        f"set protocols failover route 0.0.0.0/0 dhcp-interface bond0.{WAN_DHCP_VLAN} metric 1",
+        f"set protocols failover route 0.0.0.0/0 dhcp-interface bond0.{WAN_DHCP_VLAN} check type icmp",
+        f"set protocols failover route 0.0.0.0/0 dhcp-interface bond0.{WAN_DHCP_VLAN} check timeout 5",
+        # any-available, not all: one unreachable public resolver is a normal
+        # internet event, not a reason to abandon a working 10 gig line.
+        f"set protocols failover route 0.0.0.0/0 dhcp-interface bond0.{WAN_DHCP_VLAN} check policy any-available",
+    ]
+    for t in PROBE_TARGETS:
+        out.append(f"set protocols failover route 0.0.0.0/0 dhcp-interface "
+                   f"bond0.{WAN_DHCP_VLAN} check target {t}")
+    out.append("")
+    out.append("# Pin the probe targets to the line under test (WI-8 -- see above).")
+    for t in PROBE_TARGETS:
+        out.append(f"set protocols static route {t}/32 dhcp-interface bond0.{WAN_DHCP_VLAN}")
+    out += [
+        "",
+        "# Masquerade out of whichever WAN currently holds the default route.",
+        f"set nat source rule 110 outbound-interface name bond0.{WAN_DHCP_VLAN}",
+        f"set nat source rule 110 source address {SIM_LAN}",
+        "set nat source rule 110 translation address masquerade",
+        "set nat source rule 120 outbound-interface name pppoe0",
+        f"set nat source rule 120 source address {SIM_LAN}",
+        "set nat source rule 120 translation address masquerade",
+        "",
+    ]
+    if drop_scaffold:
+        out += [
+            "# Remove the pre-ISP-VM libvirt-NAT uplink: a third default route",
+            "# that has no production equivalent and hides real WAN failures.",
+            f"delete interfaces ethernet {SCAFFOLD_IF} address",
+            f"delete nat source rule {SCAFFOLD_NAT_RULE}",
+            "",
+        ]
+    return out
+
+
+# ---------------------------------------------------------------------------
+# Firewall. The policy is: internal VLANs talk to each other and to the
+# internet; the internet initiates nothing inward.
+#
+# That was already the *effect* of the previous IPv4 ruleset, but it was built
+# as a blacklist -- `default-action accept` plus explicit drops on each WAN
+# interface. The result is identical right up until someone adds a WAN, at
+# which point it is wide open and nothing looks wrong. This is the same policy
+# expressed as a whitelist, so a new interface is closed until it is named.
+# ---------------------------------------------------------------------------
+# Management is `bond0.1`, NOT the bare `bond0`. Every VLAN is tagged and the
+# bond parent carries no subnet at all -- see NATIVE_VLAN in ovs.sh for why.
+#
+# This line is the trap in that change. The address move is the visible part and
+# the part you remember; leaving `bond0` here instead of `bond0.1` means the
+# whole Management VLAN falls outside the LAN group, and with a default-deny
+# ruleset that is every management session and all inter-VLAN routing for VLAN 1
+# dropped the instant the commit lands -- on a router you reach through itself.
+LAN_IFACES = ["bond0.1", "bond0.2", "bond0.3", "bond0.9", "bond0.10", "bond0.200"]
+LAN_GROUP = "LAN"
+
+
+def firewall(wan_dhcp_if: str | None = f"bond0.{WAN_DHCP_VLAN}") -> list[str]:
+    """wan_dhcp_if=None on a router with no DHCP WAN -- a firewall rule naming
+    an interface that does not exist is rejected at commit."""
+    out = [f"# --- firewall: LAN-to-anywhere, internet-to-nothing ---"]
+    # Delete each filter before rebuilding it. `set` on a rule number is
+    # ADDITIVE: if a rule 10 already exists carrying an inbound-interface
+    # constraint, `set ... rule 10 state established` silently ANDs onto it,
+    # and you get a stateful-accept rule that only applies to one interface
+    # pair. Observed in labsim: return traffic from the internet matched
+    # neither that rule nor the LAN rule and hit the default drop, so LAN
+    # hosts could reach nothing outbound. Everything here is one commit, so
+    # nftables is rebuilt atomically -- there is no window with no firewall.
+    out += [f"delete firewall {fam} {hook} filter"
+            for fam in ("ipv4", "ipv6") for hook in ("forward", "input")]
+    out += [f"set firewall group interface-group {LAN_GROUP} interface {i}"
+            for i in LAN_IFACES]
+    out += [
+        "",
+        # INPUT -- traffic terminating ON the router.
+        # Loopback first. Under a default-drop input policy, services talking to
+        # 127.0.0.1 are filtered like anything else, and the failures are
+        # bizarre and hard to attribute. Nothing off-box can forge iif lo.
+        "set firewall ipv4 input filter rule 5 action accept",
+        "set firewall ipv4 input filter rule 5 description 'loopback'",
+        "set firewall ipv4 input filter rule 5 inbound-interface name lo",
+        "set firewall ipv4 input filter rule 10 action accept",
+        "set firewall ipv4 input filter rule 10 description 'established/related'",
+        "set firewall ipv4 input filter rule 10 state established",
+        "set firewall ipv4 input filter rule 10 state related",
+        # One rule covers VRRP, conntrack-sync, kea HA, SSH, DNS and BGP,
+        # because every one of them arrives on a LAN interface. Enumerating the
+        # protocols instead would mean a new firewall rule every time the pair
+        # gains a feature -- and a lockout the day someone forgets.
+        f"set firewall ipv4 input filter rule 20 action accept",
+        f"set firewall ipv4 input filter rule 20 description 'trusted LAN to the router'",
+        f"set firewall ipv4 input filter rule 20 inbound-interface group {LAN_GROUP}",
+        # DHCP client. Lease RENEWAL is unicast UDP to port 68 and conntrack
+        # does not reliably cover it, so without this the WAN keeps working
+        # until the lease expires and then dies -- a delayed failure that looks
+        # nothing like a firewall change.
+        "set firewall ipv4 input filter default-action drop",
+        "",
+        # FORWARD -- traffic passing THROUGH the router.
+        "set firewall ipv4 forward filter rule 10 action accept",
+        "set firewall ipv4 forward filter rule 10 description 'established/related'",
+        "set firewall ipv4 forward filter rule 10 state established",
+        "set firewall ipv4 forward filter rule 10 state related",
+        # Inter-VLAN *and* LAN-to-internet in one rule: both are "came in on a
+        # LAN interface". Deliberately no restriction between internal VLANs --
+        # segmenting them is a separate decision, not a side effect of this one.
+        f"set firewall ipv4 forward filter rule 20 action accept",
+        f"set firewall ipv4 forward filter rule 20 description 'LAN to anywhere (inter-VLAN + internet)'",
+        f"set firewall ipv4 forward filter rule 20 inbound-interface group {LAN_GROUP}",
+        "set firewall ipv4 forward filter default-action drop",
+        "",
+        # IPv6 already runs default-deny. It only lacks the loopback rule.
+        "set firewall ipv6 input filter rule 5 action accept",
+        "set firewall ipv6 input filter rule 5 description 'loopback'",
+        "set firewall ipv6 input filter rule 5 inbound-interface name lo",
+        "set firewall ipv6 input filter rule 10 action accept",
+        "set firewall ipv6 input filter rule 10 description 'replies to our own traffic'",
+        "set firewall ipv6 input filter rule 10 state established",
+        "set firewall ipv6 input filter rule 10 state related",
+        # RFC 4890: filtering ICMPv6 wholesale breaks ND and PMTUD, which
+        # presents as "IPv6 works until something large", not as a block.
+        "set firewall ipv6 input filter rule 20 action accept",
+        "set firewall ipv6 input filter rule 20 description 'ICMPv6 - ND/RA/PMTUD'",
+        "set firewall ipv6 input filter rule 20 protocol icmpv6",
+        f"set firewall ipv6 input filter rule 30 action accept",
+        f"set firewall ipv6 input filter rule 30 description 'trusted LAN to the router'",
+        f"set firewall ipv6 input filter rule 30 inbound-interface group {LAN_GROUP}",
+        "set firewall ipv6 input filter default-action drop",
+        "set firewall ipv6 forward filter rule 10 action accept",
+        "set firewall ipv6 forward filter rule 10 description 'replies to our own traffic'",
+        "set firewall ipv6 forward filter rule 10 state established",
+        "set firewall ipv6 forward filter rule 10 state related",
+        "set firewall ipv6 forward filter rule 20 action accept",
+        "set firewall ipv6 forward filter rule 20 description 'ICMPv6 - ND/RA/PMTUD'",
+        "set firewall ipv6 forward filter rule 20 protocol icmpv6",
+        f"set firewall ipv6 forward filter rule 30 action accept",
+        f"set firewall ipv6 forward filter rule 30 description 'trusted LAN interfaces only'",
+        f"set firewall ipv6 forward filter rule 30 inbound-interface group {LAN_GROUP}",
+        "set firewall ipv6 forward filter default-action drop",
+        "",
+    ]
+    if wan_dhcp_if:
+        dhcp = [
+            "set firewall ipv4 input filter rule 140 action accept",
+            "set firewall ipv4 input filter rule 140 description 'DHCP client lease renewal'",
+            "set firewall ipv4 input filter rule 140 protocol udp",
+            "set firewall ipv4 input filter rule 140 destination port 68",
+            f"set firewall ipv4 input filter rule 140 inbound-interface name {wan_dhcp_if}",
+        ]
+        i = out.index("set firewall ipv4 input filter default-action drop")
+        out[i:i] = dhcp
+    return out
+
+
+def isp_dhcp(wan_if: str, uplink_if: str) -> list[str]:
+    """The 10gig-equivalent ISP: hands out a lease, NATs to the real internet."""
+    return [
+        f"# --- sim ISP: DHCP WAN on VLAN {WAN_DHCP_VLAN} ---",
+        "set system host-name isp-dhcp",
+        f"set interfaces ethernet {wan_if} address {ISP_DHCP_GW}/24",
+        f"set interfaces ethernet {wan_if} description "
+        f"'sim ISP - 10gig-equivalent WAN on VLAN{WAN_DHCP_VLAN}'",
+        f"set interfaces ethernet {uplink_if} address dhcp",
+        f"set interfaces ethernet {uplink_if} description 'uplink to the real internet'",
+        f"set service dhcp-server shared-network-name WAN{WAN_DHCP_VLAN} "
+        f"subnet {ISP_DHCP_NET} subnet-id 1",
+        f"set service dhcp-server shared-network-name WAN{WAN_DHCP_VLAN} "
+        f"subnet {ISP_DHCP_NET} option default-router {ISP_DHCP_GW}",
+        f"set service dhcp-server shared-network-name WAN{WAN_DHCP_VLAN} "
+        f"subnet {ISP_DHCP_NET} option name-server 8.8.8.8",
+        f"set service dhcp-server shared-network-name WAN{WAN_DHCP_VLAN} "
+        f"subnet {ISP_DHCP_NET} range CUST start {ISP_DHCP_POOL[0]}",
+        f"set service dhcp-server shared-network-name WAN{WAN_DHCP_VLAN} "
+        f"subnet {ISP_DHCP_NET} range CUST stop {ISP_DHCP_POOL[1]}",
+        "",
+    ] + _isp_common(uplink_if, ISP_DHCP_NET, "sim ISP: NAT customers to the real internet")
+
+
+def isp_pppoe(wan_if: str, uplink_if: str) -> list[str]:
+    """The Vodafone-equivalent ISP: terminates PPPoE, NATs to the real internet."""
+    return [
+        f"# --- sim ISP: PPPoE WAN on VLAN {WAN_PPPOE_VLAN} ---",
+        "set system host-name isp-pppoe",
+        f"set interfaces ethernet {wan_if} description "
+        f"'sim ISP - Vodafone-equivalent WAN on VLAN{WAN_PPPOE_VLAN} (PPPoE)'",
+        f"set interfaces ethernet {uplink_if} address dhcp",
+        f"set interfaces ethernet {uplink_if} description 'uplink to the real internet'",
+        f"set service pppoe-server access-concentrator {PPPOE_AC}",
+        f"set service pppoe-server interface {wan_if}",
+        f"set service pppoe-server gateway-address {ISP_PPPOE_GW}",
+        "set service pppoe-server authentication mode local",
+        f"set service pppoe-server authentication local-users username {PPPOE_USER} "
+        f"password {PPPOE_PASS}",
+        f"set service pppoe-server client-ip-pool CUST range "
+        f"{ISP_PPPOE_POOL[0]}-{ISP_PPPOE_POOL[1]}",
+        "set service pppoe-server default-pool CUST",
+        "set service pppoe-server name-server 8.8.8.8",
+        "",
+    ] + _isp_common(uplink_if, ISP_PPPOE_NET, "sim ISP: NAT PPPoE customers to the real internet")
+
+
+def _isp_common(uplink_if: str, customer_net: str, desc: str) -> list[str]:
+    return [
+        "set nat source rule 100 description " + f"'{desc}'",
+        f"set nat source rule 100 outbound-interface name {uplink_if}",
+        f"set nat source rule 100 source address {customer_net}",
+        "set nat source rule 100 translation address masquerade",
+        "",
+        # An ISP that drops return traffic is not simulating an ISP. The forward
+        # chain defaults to accept here on purpose -- these VMs model the
+        # internet, and the thing under test is the router's firewall, not this.
+        "set firewall ipv4 forward filter default-action accept",
+        "set firewall ipv4 forward filter rule 10 action accept",
+        "set firewall ipv4 forward filter rule 10 state established",
+        "set firewall ipv4 forward filter rule 10 state related",
+        "set firewall ipv4 forward filter rule 10 description conntrack-engage",
+        "",
+    ]
+
+
+def build(role: str, drop_scaffold: bool, wan_if: str, uplink_if: str) -> list[str]:
+    if role in ("primary", "secondary"):
+        # BOTH routers get the identical WAN. The sim used to give it to the
+        # primary only, reasoning that two PPPoE clients sharing one credential
+        # was "a different failure mode than anything production has" -- but
+        # that IS production, and omitting it meant the failover path was the
+        # one path the sim could not exercise. It also left the pair
+        # incomparable: ten NAT rules on one box, none on the other.
+        #
+        # Safety comes from resting state, not from asymmetry: bond0.53 is
+        # `disable`d on both (cloned MAC), pppoe0 is enabled on both but gated
+        # at the systemd unit. See wan() and migration/ppp-vrrp-gate.conf.
+        return ([f"# labsim routing -- {role}", ""]
+                + bgp(role) + wan(drop_scaffold, role)
+                + firewall(wan_dhcp_if=f"bond0.{WAN_DHCP_VLAN}"))
+    if role == "isp-dhcp":
+        return isp_dhcp(wan_if, uplink_if)
+    return isp_pppoe(wan_if, uplink_if)
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--role", required=True,
+                    choices=("primary", "secondary", "isp-dhcp", "isp-pppoe"))
+    ap.add_argument("--drop-scaffold", action="store_true",
+                    help="also remove the pre-ISP-VM libvirt-NAT uplink (primary only)")
+    # The ISP VMs' interface names depend on PCI enumeration order, which is not
+    # stable across a rebuild: isp-dhcp came up as eth0/eth1 and isp-pppoe as
+    # eth2/eth3 from identical XML. Check with `show interfaces` before applying
+    # rather than trusting these defaults.
+    ap.add_argument("--wan-if", default=None, help="ISP VM: customer-facing NIC")
+    ap.add_argument("--uplink-if", default=None, help="ISP VM: internet-facing NIC")
+    args = ap.parse_args()
+
+    defaults = {"isp-dhcp": ("eth0", "eth1"), "isp-pppoe": ("eth2", "eth3")}
+    w, u = defaults.get(args.role, ("", ""))
+    w, u = args.wan_if or w, args.uplink_if or u
+
+    if args.drop_scaffold and args.role != "primary":
+        print("--drop-scaffold only applies to --role primary", file=sys.stderr)
+        return 2
+
+    lines = build(args.role, args.drop_scaffold, w, u)
+    sys.stdout.write("\n".join(lines) + "\n")
+    n = len([l for l in lines if l.startswith(("set ", "delete "))])
+    print(f"{args.role}: {n} commands", file=sys.stderr)
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

labsim/topology.html

@@ -0,0 +1,181 @@
+<!doctype html>
+<html lang="en">
+<head>
+<meta charset="utf-8">
+<title>labsim — live VLAN topology</title>
+<style>
+  :root {
+    --bg:#0e1116; --panel:#161b22; --line:#30363d; --text:#e6edf3; --dim:#8b949e;
+    --ok:#3fb950; --bad:#f85149; --warn:#d29922; --router:#58a6ff;
+  }
+  * { box-sizing:border-box; }
+  body { margin:0; background:var(--bg); color:var(--text);
+         font:14px/1.5 ui-sans-serif,system-ui,-apple-system,"Segoe UI",sans-serif; }
+  header { display:flex; align-items:baseline; gap:16px; flex-wrap:wrap;
+           padding:14px 20px; border-bottom:1px solid var(--line); }
+  h1 { font-size:16px; margin:0; font-weight:650; letter-spacing:.2px; }
+  .meta { color:var(--dim); font-size:12px; }
+  .pill { padding:2px 8px; border-radius:999px; font-size:12px; font-weight:600; }
+  .pill.ok { background:rgba(63,185,80,.15); color:var(--ok); }
+  .pill.bad { background:rgba(248,81,73,.15); color:var(--bad); }
+  main { display:grid; grid-template-columns:minmax(0,1.35fr) minmax(320px,.65fr);
+         gap:16px; padding:16px 20px; align-items:start; }
+  @media (max-width:1000px){ main { grid-template-columns:1fr; } }
+  .card { background:var(--panel); border:1px solid var(--line); border-radius:10px; padding:14px; }
+  .card h2 { margin:0 0 10px; font-size:13px; font-weight:600; color:var(--dim);
+             text-transform:uppercase; letter-spacing:.6px; }
+  svg { width:100%; height:auto; display:block; }
+  .edge { stroke-width:2.5; transition:stroke .25s, opacity .25s; }
+  .edge.ok { stroke:var(--ok); opacity:.55; }
+  .edge.bad { stroke:var(--bad); opacity:.95; stroke-dasharray:7 5; }
+  .edge:hover { opacity:1; stroke-width:4; }
+  .node circle { fill:#0d1117; stroke-width:2.5; }
+  .node text { text-anchor:middle; font-size:11px; font-weight:600; fill:var(--text); }
+  .node .sub { font-size:9.5px; font-weight:400; fill:var(--dim); }
+  .rtt { font-size:9px; fill:var(--dim); text-anchor:middle; }
+  table { width:100%; border-collapse:collapse; font-size:12.5px; }
+  th,td { text-align:left; padding:5px 8px; border-bottom:1px solid var(--line); }
+  th { color:var(--dim); font-weight:600; font-size:11px; text-transform:uppercase; }
+  td.n { text-align:right; font-variant-numeric:tabular-nums; }
+  .b-ok { color:var(--ok); } .b-bad { color:var(--bad); }
+  .empty { color:var(--dim); padding:10px 4px; }
+  .legend { display:flex; gap:14px; align-items:center; color:var(--dim);
+            font-size:11.5px; margin-top:10px; flex-wrap:wrap; }
+  .swatch { display:inline-block; width:22px; height:0; border-top:2.5px solid; margin-right:5px;
+            vertical-align:middle; }
+</style>
+</head>
+<body>
+<header>
+  <h1>labsim — live VLAN topology</h1>
+  <span id="summary" class="pill ok">…</span>
+  <span class="meta">every path is probed <em>from</em> a VM <em>to</em> every other VM, through the VyOS router</span>
+  <span class="meta" id="clock" style="margin-left:auto"></span>
+</header>
+
+<main>
+  <section class="card">
+    <h2>Mesh — line colour is reachability, label is ICMP RTT</h2>
+    <svg id="topo" viewBox="0 0 720 560" role="img" aria-label="VLAN topology"></svg>
+    <div class="legend">
+      <span><i class="swatch" style="border-color:var(--ok)"></i>reachable</span>
+      <span><i class="swatch" style="border-color:var(--bad); border-top-style:dashed"></i>blocked</span>
+      <span>hover a line for detail · node ring turns red if anything to/from it is blocked</span>
+    </div>
+  </section>
+
+  <aside style="display:grid; gap:16px">
+    <section class="card">
+      <h2>Blocked paths</h2>
+      <div id="blocked"></div>
+    </section>
+    <section class="card">
+      <h2>Latency (ICMP, ms)</h2>
+      <table><thead><tr><th>path</th><th class="n">rtt</th></tr></thead>
+      <tbody id="lat"></tbody></table>
+    </section>
+  </aside>
+</main>
+
+<script>
+const REFRESH_MS = 5000;
+const CX = 360, CY = 250, R = 185;
+
+function polar(i, n) {
+  const a = (i / n) * Math.PI * 2 - Math.PI / 2;
+  return { x: CX + R * Math.cos(a), y: CY + R * Math.sin(a) };
+}
+
+function render(data) {
+  const vlans = data.vlans, res = data.results;
+  const svg = document.getElementById('topo');
+  const n = vlans.length;
+  const pos = vlans.map((_, i) => polar(i, n));
+  let out = '';
+
+  // Router in the middle — every inter-VLAN packet really does traverse it.
+  out += `<circle cx="${CX}" cy="${CY}" r="40" fill="#0d1117" stroke="var(--router)" stroke-width="2.5"/>`;
+  out += `<text x="${CX}" y="${CY-6}" text-anchor="middle" font-size="12" font-weight="700" fill="var(--router)">VyOS</text>`;
+  out += `<text x="${CX}" y="${CY+9}" text-anchor="middle" font-size="8.5" fill="var(--dim)">bond0</text>`;
+  out += `<text x="${CX}" y="${CY+20}" text-anchor="middle" font-size="8.5" fill="var(--dim)">LACP</text>`;
+
+  const bad = new Set();
+  // One line per unordered pair; a pair is bad if EITHER direction fails.
+  for (let i = 0; i < n; i++) {
+    for (let j = i + 1; j < n; j++) {
+      const a = vlans[i].label, b = vlans[j].label;
+      const ab = (res[a] || {})[b] || {}, ba = (res[b] || {})[a] || {};
+      const okAB = ab.icmp === true, okBA = ba.icmp === true;
+      const ok = okAB && okBA;
+      if (!ok) { bad.add(a); bad.add(b); }
+      const rtts = [ab.rtt_ms, ba.rtt_ms].filter(v => typeof v === 'number');
+      const rtt = rtts.length ? (rtts.reduce((s,v)=>s+v,0)/rtts.length) : null;
+      // Place the label ~32% along the edge, not at the midpoint: diagonals of
+      // a 6-node mesh all cross the centre, so midpoint labels stack on top of
+      // the router node. Plus a small perpendicular nudge off the line itself.
+      const dx = pos[j].x - pos[i].x, dy = pos[j].y - pos[i].y;
+      const len = Math.hypot(dx, dy) || 1;
+      const t = 0.32;
+      const mx = pos[i].x + dx * t + (-dy / len) * 8;
+      const my = pos[i].y + dy * t + ( dx / len) * 8;
+      const tip = `${a} ↔ ${b}\n→ ${okAB ? 'ok' : 'BLOCKED'}   ← ${okBA ? 'ok' : 'BLOCKED'}` +
+                  (rtt !== null ? `\nrtt ${rtt.toFixed(2)} ms` : '');
+      out += `<line class="edge ${ok?'ok':'bad'}" x1="${pos[i].x}" y1="${pos[i].y}" x2="${pos[j].x}" y2="${pos[j].y}"><title>${tip}</title></line>`;
+      if (ok && rtt !== null)
+        out += `<text class="rtt" x="${mx}" y="${my}">${rtt.toFixed(2)}</text>`;
+    }
+  }
+
+  vlans.forEach((v, i) => {
+    const p = pos[i], isBad = bad.has(v.label);
+    out += `<g class="node"><circle cx="${p.x}" cy="${p.y}" r="30" stroke="${isBad?'var(--bad)':'var(--ok)'}"/>` +
+           `<text x="${p.x}" y="${p.y-2}">${v.name}</text>` +
+           `<text class="sub" x="${p.x}" y="${p.y+11}">vlan ${v.vid}</text>` +
+           `<text class="sub" x="${p.x}" y="${p.y+47}">${v.ip}</text></g>`;
+  });
+  svg.innerHTML = out;
+
+  // Blocked list — the thing you actually act on.
+  const rows = [];
+  for (const src of vlans) for (const dst of vlans) {
+    if (src.label === dst.label) continue;
+    const d = (res[src.label] || {})[dst.label] || {};
+    for (const proto of ['icmp','tcp22','tcp80'])
+      if (d[proto] === false) rows.push(`${src.label} → ${dst.label} <span style="color:var(--dim)">(${proto})</span>`);
+  }
+  document.getElementById('blocked').innerHTML = rows.length
+    ? `<table><tbody>${rows.map(r=>`<tr><td class="b-bad">${r}</td></tr>`).join('')}</tbody></table>`
+    : `<div class="empty">none — all ${vlans.length*(vlans.length-1)*3} paths open</div>`;
+
+  // Latency table, slowest first.
+  const lat = [];
+  for (const src of vlans) for (const dst of vlans) {
+    if (src.label === dst.label) continue;
+    const d = (res[src.label] || {})[dst.label] || {};
+    if (typeof d.rtt_ms === 'number') lat.push([`${src.label} → ${dst.label}`, d.rtt_ms]);
+  }
+  lat.sort((a,b) => b[1]-a[1]);
+  document.getElementById('lat').innerHTML = lat.slice(0,12)
+    .map(([k,v]) => `<tr><td>${k}</td><td class="n">${v.toFixed(2)}</td></tr>`).join('')
+    || `<tr><td class="empty" colspan="2">no RTT data</td></tr>`;
+
+  const total = data.total, reach = data.reachable;
+  const pill = document.getElementById('summary');
+  pill.textContent = `${reach}/${total} paths open`;
+  pill.className = 'pill ' + (reach === total ? 'ok' : 'bad');
+  document.getElementById('clock').textContent =
+    `updated ${new Date().toLocaleTimeString()} · sweep ${data.sweep_seconds.toFixed(2)}s · refresh ${REFRESH_MS/1000}s`;
+}
+
+async function tick() {
+  try {
+    const r = await fetch('/api/matrix', {cache:'no-store'});
+    render(await r.json());
+  } catch (e) {
+    document.getElementById('clock').textContent = 'exporter unreachable — ' + e;
+  }
+}
+tick(); setInterval(tick, REFRESH_MS);
+</script>
+</body>
+</html>

labsim/vlan-leak-evidence/after-vlan1/capture-parent.txt

@@ -0,0 +1,6 @@
+12:52:35.919490 52:54:00:6d:71:e7 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 346: vlan 1, p 0, ethertype IPv4 (0x0800), 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:6d:71:e7, length 300
+12:52:35.920089 52:54:00:e5:95:a2 > 52:54:00:6d:71:e7, ethertype 802.1Q (0x8100), length 329: vlan 1, p 0, ethertype IPv4 (0x0800), 172.31.1.252.67 > 172.31.1.6.68: BOOTP/DHCP, Reply, length 283
+12:52:35.920307 52:54:00:e5:95:a2 > 52:54:00:6d:71:e7, ethertype 802.1Q (0x8100), length 329: vlan 1, p 0, ethertype IPv4 (0x0800), 172.31.1.252.67 > 172.31.1.7.68: BOOTP/DHCP, Reply, length 283
+12:52:35.922052 52:54:00:6d:71:e7 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 346: vlan 1, p 0, ethertype IPv4 (0x0800), 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:6d:71:e7, length 300
+12:52:35.922509 52:54:00:e5:95:a2 > 52:54:00:6d:71:e7, ethertype 802.1Q (0x8100), length 329: vlan 1, p 0, ethertype IPv4 (0x0800), 172.31.1.252.67 > 172.31.1.6.68: BOOTP/DHCP, Reply, length 283
+12:52:35.923327 52:54:00:e5:95:a2 > 52:54:00:6d:71:e7, ethertype 802.1Q (0x8100), length 329: vlan 1, p 0, ethertype IPv4 (0x0800), 172.31.1.252.67 > 172.31.1.6.68: BOOTP/DHCP, Reply, length 283

labsim/vlan-leak-evidence/after-vlan1/capture-vif.txt

@@ -0,0 +1,6 @@
+12:52:35.919490 52:54:00:6d:71:e7 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:6d:71:e7, length 300
+12:52:35.920081 52:54:00:e5:95:a2 > 52:54:00:6d:71:e7, ethertype IPv4 (0x0800), length 325: 172.31.1.252.67 > 172.31.1.6.68: BOOTP/DHCP, Reply, length 283
+12:52:35.920305 52:54:00:e5:95:a2 > 52:54:00:6d:71:e7, ethertype IPv4 (0x0800), length 325: 172.31.1.252.67 > 172.31.1.7.68: BOOTP/DHCP, Reply, length 283
+12:52:35.922052 52:54:00:6d:71:e7 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:6d:71:e7, length 300
+12:52:35.922507 52:54:00:e5:95:a2 > 52:54:00:6d:71:e7, ethertype IPv4 (0x0800), length 325: 172.31.1.252.67 > 172.31.1.6.68: BOOTP/DHCP, Reply, length 283
+12:52:35.923326 52:54:00:e5:95:a2 > 52:54:00:6d:71:e7, ethertype IPv4 (0x0800), length 325: 172.31.1.252.67 > 172.31.1.6.68: BOOTP/DHCP, Reply, length 283

labsim/vlan-leak-evidence/after-vlan1/client.txt

@@ -0,0 +1,4 @@
+udhcpc: started, v1.37.0
+udhcpc: broadcasting discover
+udhcpc: broadcasting select for 172.31.1.6, server 172.31.1.252
+udhcpc: lease of 172.31.1.6 obtained from 172.31.1.252, lease time 86400

labsim/vlan-leak-evidence/after-vlan1/router-config.txt

@@ -0,0 +1,64 @@
+set high-availability vrrp group native address 172.31.1.1/24
+set high-availability vrrp group native hello-source-address '172.31.1.252'
+set high-availability vrrp group native interface 'bond0.1'
+set high-availability vrrp group native no-preempt
+set high-availability vrrp group native peer-address '172.31.1.253'
+set high-availability vrrp group native priority '200'
+set high-availability vrrp group native vrid '1'
+set high-availability vrrp group vlan2 address 172.31.2.1/24
+set high-availability vrrp group vlan2 hello-source-address '172.31.2.252'
+set high-availability vrrp group vlan2 interface 'bond0.2'
+set high-availability vrrp group vlan2 no-preempt
+set high-availability vrrp group vlan2 peer-address '172.31.2.253'
+set high-availability vrrp group vlan2 priority '200'
+set high-availability vrrp group vlan2 vrid '2'
+set high-availability vrrp group vlan3 address 172.31.3.1/24
+set high-availability vrrp group vlan3 hello-source-address '172.31.3.252'
+set high-availability vrrp group vlan3 interface 'bond0.3'
+set high-availability vrrp group vlan3 no-preempt
+set high-availability vrrp group vlan3 peer-address '172.31.3.253'
+set high-availability vrrp group vlan3 priority '200'
+set high-availability vrrp group vlan3 vrid '3'
+set high-availability vrrp group vlan9 address 172.31.9.1/24
+set high-availability vrrp group vlan9 hello-source-address '172.31.9.252'
+set high-availability vrrp group vlan9 interface 'bond0.9'
+set high-availability vrrp group vlan9 no-preempt
+set high-availability vrrp group vlan9 peer-address '172.31.9.253'
+set high-availability vrrp group vlan9 priority '200'
+set high-availability vrrp group vlan9 vrid '9'
+set high-availability vrrp group vlan10 address 172.31.10.1/23
+set high-availability vrrp group vlan10 hello-source-address '172.31.10.252'
+set high-availability vrrp group vlan10 interface 'bond0.10'
+set high-availability vrrp group vlan10 no-preempt
+set high-availability vrrp group vlan10 peer-address '172.31.10.253'
+set high-availability vrrp group vlan10 priority '200'
+set high-availability vrrp group vlan10 vrid '10'
+set high-availability vrrp group vlan200 address 172.31.200.1/24
+set high-availability vrrp group vlan200 hello-source-address '172.31.200.252'
+set high-availability vrrp group vlan200 interface 'bond0.200'
+set high-availability vrrp group vlan200 no-preempt
+set high-availability vrrp group vlan200 peer-address '172.31.200.253'
+set high-availability vrrp group vlan200 priority '200'
+set high-availability vrrp group vlan200 vrid '200'
+set interfaces bonding bond0 description 'api-batch-test'
+set interfaces bonding bond0 hash-policy 'layer2+3'
+set interfaces bonding bond0 lacp-rate 'fast'
+set interfaces bonding bond0 member interface 'eth0'
+set interfaces bonding bond0 member interface 'eth1'
+set interfaces bonding bond0 mode '802.3ad'
+set interfaces bonding bond0 vif 1 address '172.31.1.252/24'
+set interfaces bonding bond0 vif 1 description 'management'
+set interfaces bonding bond0 vif 2 address '172.31.2.252/24'
+set interfaces bonding bond0 vif 2 description 'k8s'
+set interfaces bonding bond0 vif 3 address '172.31.3.252/24'
+set interfaces bonding bond0 vif 3 description 'kvm'
+set interfaces bonding bond0 vif 9 address '172.31.9.252/24'
+set interfaces bonding bond0 vif 9 description 'private'
+set interfaces bonding bond0 vif 10 address '172.31.10.252/23'
+set interfaces bonding bond0 vif 10 description 'lot'
+set interfaces bonding bond0 vif 51 description 'WAN1 Vodafone-equivalent (sim ISP PPPoE)'
+set interfaces bonding bond0 vif 53 address 'dhcp'
+set interfaces bonding bond0 vif 53 description 'WAN3 10gig-equivalent (sim ISP DHCP)'
+set interfaces bonding bond0 vif 53 dhcp-options default-route-distance '210'
+set interfaces bonding bond0 vif 200 address '172.31.200.252/24'
+set interfaces bonding bond0 vif 200 description 'roomates'

labsim/vlan-leak-evidence/after/capture-parent.txt

@@ -0,0 +1,6 @@
+12:52:14.639170 52:54:00:02:2e:b1 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 346: vlan 3, p 0, ethertype IPv4 (0x0800), 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:02:2e:b1, length 300
+12:52:14.640467 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype 802.1Q (0x8100), length 329: vlan 3, p 0, ethertype IPv4 (0x0800), 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283
+12:52:14.640846 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype 802.1Q (0x8100), length 329: vlan 3, p 0, ethertype IPv4 (0x0800), 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283
+12:52:14.642554 52:54:00:02:2e:b1 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x8100), length 346: vlan 3, p 0, ethertype IPv4 (0x0800), 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:02:2e:b1, length 300
+12:52:14.642766 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype 802.1Q (0x8100), length 329: vlan 3, p 0, ethertype IPv4 (0x0800), 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283
+12:52:14.643056 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype 802.1Q (0x8100), length 329: vlan 3, p 0, ethertype IPv4 (0x0800), 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283

labsim/vlan-leak-evidence/after/capture-vif.txt

@@ -0,0 +1,6 @@
+12:52:14.639170 52:54:00:02:2e:b1 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:02:2e:b1, length 300
+12:52:14.640465 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype IPv4 (0x0800), length 325: 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283
+12:52:14.640845 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype IPv4 (0x0800), length 325: 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283
+12:52:14.642554 52:54:00:02:2e:b1 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 52:54:00:02:2e:b1, length 300
+12:52:14.642764 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype IPv4 (0x0800), length 325: 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283
+12:52:14.643055 52:54:00:e5:95:a2 > 52:54:00:02:2e:b1, ethertype IPv4 (0x0800), length 325: 172.31.3.252.67 > 172.31.3.11.68: BOOTP/DHCP, Reply, length 283

labsim/vlan-leak-evidence/after/client.txt

@@ -0,0 +1,4 @@
+udhcpc: started, v1.37.0
+udhcpc: broadcasting discover
+udhcpc: broadcasting select for 172.31.3.11, server 172.31.3.252
+udhcpc: lease of 172.31.3.11 obtained from 172.31.3.252, lease time 85374

labsim/vlan-leak-evidence/after/router-config.txt

@@ -0,0 +1,64 @@
+set high-availability vrrp group native address 172.31.1.1/24
+set high-availability vrrp group native hello-source-address '172.31.1.252'
+set high-availability vrrp group native interface 'bond0.1'
+set high-availability vrrp group native no-preempt
+set high-availability vrrp group native peer-address '172.31.1.253'
+set high-availability vrrp group native priority '200'
+set high-availability vrrp group native vrid '1'
+set high-availability vrrp group vlan2 address 172.31.2.1/24
+set high-availability vrrp group vlan2 hello-source-address '172.31.2.252'
+set high-availability vrrp group vlan2 interface 'bond0.2'
+set high-availability vrrp group vlan2 no-preempt
+set high-availability vrrp group vlan2 peer-address '172.31.2.253'
+set high-availability vrrp group vlan2 priority '200'
+set high-availability vrrp group vlan2 vrid '2'
+set high-availability vrrp group vlan3 address 172.31.3.1/24
+set high-availability vrrp group vlan3 hello-source-address '172.31.3.252'
+set high-availability vrrp group vlan3 interface 'bond0.3'
+set high-availability vrrp group vlan3 no-preempt
+set high-availability vrrp group vlan3 peer-address '172.31.3.253'
+set high-availability vrrp group vlan3 priority '200'
+set high-availability vrrp group vlan3 vrid '3'
+set high-availability vrrp group vlan9 address 172.31.9.1/24
+set high-availability vrrp group vlan9 hello-source-address '172.31.9.252'
+set high-availability vrrp group vlan9 interface 'bond0.9'
+set high-availability vrrp group vlan9 no-preempt
+set high-availability vrrp group vlan9 peer-address '172.31.9.253'
+set high-availability vrrp group vlan9 priority '200'
+set high-availability vrrp group vlan9 vrid '9'
+set high-availability vrrp group vlan10 address 172.31.10.1/23
+set high-availability vrrp group vlan10 hello-source-address '172.31.10.252'
+set high-availability vrrp group vlan10 interface 'bond0.10'
+set high-availability vrrp group vlan10 no-preempt
+set high-availability vrrp group vlan10 peer-address '172.31.10.253'
+set high-availability vrrp group vlan10 priority '200'
+set high-availability vrrp group vlan10 vrid '10'
+set high-availability vrrp group vlan200 address 172.31.200.1/24
+set high-availability vrrp group vlan200 hello-source-address '172.31.200.252'
+set high-availability vrrp group vlan200 interface 'bond0.200'
+set high-availability vrrp group vlan200 no-preempt
+set high-availability vrrp group vlan200 peer-address '172.31.200.253'
+set high-availability vrrp group vlan200 priority '200'
+set high-availability vrrp group vlan200 vrid '200'
+set interfaces bonding bond0 description 'api-batch-test'
+set interfaces bonding bond0 hash-policy 'layer2+3'
+set interfaces bonding bond0 lacp-rate 'fast'
+set interfaces bonding bond0 member interface 'eth0'
+set interfaces bonding bond0 member interface 'eth1'
+set interfaces bonding bond0 mode '802.3ad'
+set interfaces bonding bond0 vif 1 address '172.31.1.252/24'
+set interfaces bonding bond0 vif 1 description 'management'
+set interfaces bonding bond0 vif 2 address '172.31.2.252/24'
+set interfaces bonding bond0 vif 2 description 'k8s'
+set interfaces bonding bond0 vif 3 address '172.31.3.252/24'
+set interfaces bonding bond0 vif 3 description 'kvm'
+set interfaces bonding bond0 vif 9 address '172.31.9.252/24'
+set interfaces bonding bond0 vif 9 description 'private'
+set interfaces bonding bond0 vif 10 address '172.31.10.252/23'
+set interfaces bonding bond0 vif 10 description 'lot'
+set interfaces bonding bond0 vif 51 description 'WAN1 Vodafone-equivalent (sim ISP PPPoE)'
+set interfaces bonding bond0 vif 53 address 'dhcp'
+set interfaces bonding bond0 vif 53 description 'WAN3 10gig-equivalent (sim ISP DHCP)'
+set interfaces bonding bond0 vif 53 dhcp-options default-route-distance '210'
+set interfaces bonding bond0 vif 200 address '172.31.200.252/24'
+set interfaces bonding bond0 vif 200 description 'roomates'