feat(labsim): libvirt replica of the lab network with LACP + VyOS routing

A throwaway copy of the production VLAN topology so routing and firewall
changes can be tested before they touch the real network. Same VLAN IDs and
roles as UniFi, deliberately different ranges (172.31.<vlan>.0/24) so nothing
here can be mistaken for production.

- OVS fabric: real 802.1Q. Access port per micro VM, host leg per VLAN (.2,
  for SSH only — NOT the VMs' default route, so inter-VLAN tests exercise the
  router rather than the host's routing table), and a trunk portgroup with
  VLAN 1 declared nativeMode='untagged'.
- Six Alpine micro VMs (256MB, copy-on-write overlays on one 176MB image),
  SSH + a hello-world HTTP page naming the VLAN.
- VyOS router installed to disk unattended over the console, with the SAME
  config shape as the VP2440s: two NICs in an LACP bond carrying the trunk,
  VLAN 1 native, bond0.<vlan> holding the .1 gateway on each.
- labsim-matrix.py: full-mesh ICMP/TCP22/TCP80 probe, ~0.2s, --watch
  highlights cells that changed since the last sweep. Guest-side probe is
  python3 (already present via cloud-init) so nothing is installed on VMs
  that have no internet.
- Prometheus + Grafana (anonymous auth, no login) with a provisioned
  dashboard: heatmap plus a state timeline showing exactly when a path
  flipped. Verified end to end: one VyOS rule took sum(labsim_reachable)
  from 90 to 84, blocking precisely kvm<->k8s across all three protocols.

Traps found building this, all now encoded in the scripts:
- virtio-net breaks 802.3ad: the guest's bonding driver reports slaves
  "MII Status: down" despite carrier=1 and never sends an LACPDU, so the bond
  sits in AD_STATE_DEFAULTED. e1000e fixes it with no other change. Matches
  the netdev thread "bonding (IEEE 802.3ad) not working with qemu/virtio".
- OVS defaults bonds to active-backup, which does not speak LACP at all —
  bond_mode=balance-tcp is required.
- LACP deadlock: OVS holds members disabled until negotiation while the
  partner needs carrier before it will send LACPDUs. lacp-fallback-ab breaks it.
- LACPDUs are untagged, so a trunk with no native VLAN has nowhere to put them.
- --boot cdrom,hd re-runs the ISO on every restart, so every commit+save went
  to a live system that evaporated. Install now switches the VM to boot hd.
- cloud-init on Alpine: users stay locked without lock_passwd:false, one
  failing runcmd aborts the rest, busybox here has no httpd applet, and
  start-stop-daemon --exec /usr/bin/python3 matches cloud-init's own python3.
- The user-data heredoc is unquoted, so backticks in a COMMENT were executed
  by the host shell and their output corrupted the YAML. build_seed now
  validates with yaml.safe_load before building the ISO.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH

labsim/router-up.sh

@@ -0,0 +1,124 @@
+#!/bin/bash
+# Add the VyOS router under test to labsim.
+#
+# Mirrors the production VP2440 pair: TWO NICs bonded with LACP carrying a
+# trunk of every VLAN, then bond0.<vlan> sub-interfaces holding the .1 gateway
+# address on each. That is the same config shape the real firewalls run, so a
+# rule tested here means something.
+#
+# NIC model is e1000e, NOT virtio, and that is load-bearing: with virtio the
+# guest's bonding driver reports its slaves "MII Status: down" despite
+# carrier=1 and never emits a single LACPDU, so the bond sits in
+# AD_STATE_DEFAULTED forever. Known issue — see the netdev thread "bonding
+# (IEEE 802.3ad) not working with qemu/virtio"; e1000e fixes it with no other
+# change. 802.3ad also requires the MII link monitor, which virtio cannot back.
+#
+#   host                       OVS "switch"                    VyOS VM
+#   hostv<vlan> (.2) ──────── ovs-labsim ──── lag-vyos ═════ eth0 + eth1
+#                              (tagged)       (LACP, trunk)   └─ bond0.<vlan> = .1
+#
+# Usage: ./router-up.sh            build + install + configure
+#        ./router-up.sh --status   show bond/LACP + interface state
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+source "$SCRIPT_DIR/lib.sh"
+source "$SCRIPT_DIR/ovs.sh"
+
+ROUTER_VM="${ROUTER_VM:-labsim-vyos}"
+ROUTER_MEM="${ROUTER_MEM:-2048}"
+ROUTER_CPUS="${ROUTER_CPUS:-2}"
+ROUTER_DISK_GB="${ROUTER_DISK_GB:-8}"
+VYOS_ISO="${VYOS_ISO:-$IMG_DIR/vyos.iso}"
+VYOS_CACHE="/var/lib/libvirt/images/lab-pxe-cache"
+
+selected_vlans
+
+if [ "${1:-}" = "--status" ]; then
+  ovs_bond_status
+  echo "--- vyos gateway addresses (probed from each host leg) ---"
+  for entry in "${SELECTED[@]}"; do
+    IFS=: read -r vid _n prefix _r <<<"$entry"
+    printf '  vlan %-5s %-16s ' "$vid" "${prefix}.1"
+    ping -c1 -W2 "${prefix}.1" >/dev/null 2>&1 && echo up || echo down
+  done
+  exit 0
+fi
+
+ovs_require
+
+# --- ISO ------------------------------------------------------------------
+if [ ! -f "$VYOS_ISO" ]; then
+  # Reuse the bastion's cached nightly if it is already on this box.
+  if [ -f "$VYOS_CACHE/vyos.iso" ]; then
+    log "reusing cached VyOS ISO"
+    sudo cp "$VYOS_CACHE/vyos.iso" "$VYOS_ISO"
+  else
+    log "resolving latest VyOS nightly ISO..."
+    url="$(curl -sSL https://api.github.com/repos/vyos/vyos-nightly-build/releases/latest \
+      | python3 -c "import json,sys;print(next(a['browser_download_url'] for a in json.load(sys.stdin)['assets'] if a['name'].endswith('generic-amd64.iso')))")"
+    log "downloading $url"
+    sudo curl -sSL --max-time 1800 -o "$VYOS_ISO" "$url"
+  fi
+fi
+[ -f "$VYOS_ISO" ] || die "no VyOS ISO at $VYOS_ISO"
+
+# --- VM -------------------------------------------------------------------
+if virsh_q dominfo "$ROUTER_VM" >/dev/null 2>&1; then
+  log "router VM $ROUTER_VM exists"
+  virsh_q start "$ROUTER_VM" >/dev/null 2>&1 || true
+else
+  log "creating router VM $ROUTER_VM (2 NICs on the trunk, for LACP)"
+  sudo qemu-img create -q -f qcow2 "$IMG_DIR/${ROUTER_VM}.qcow2" "${ROUTER_DISK_GB}G" >/dev/null
+
+  # Two trunk NICs — OVS bonds them after boot (libvirt cannot create bonds).
+  sudo virt-install \
+    --connect "$LIBVIRT_URI" \
+    --name "$ROUTER_VM" \
+    --memory "$ROUTER_MEM" --vcpus "$ROUTER_CPUS" \
+    --disk "path=$IMG_DIR/${ROUTER_VM}.qcow2,format=qcow2,bus=virtio" \
+    --disk "path=$VYOS_ISO,device=cdrom,readonly=on" \
+    --network "network=$OVS_NET,portgroup=trunk,model=e1000e,trustGuestRxFilters=yes" \
+    --network "network=$OVS_NET,portgroup=trunk,model=e1000e,trustGuestRxFilters=yes" \
+    --boot cdrom,hd \
+    --os-variant debian12 \
+    --graphics none --noautoconsole --import >/dev/null
+fi
+
+log "waiting for the live system to boot (VyOS live login)..."
+python3 "$SCRIPT_DIR/router-install.py" --vm "$ROUTER_VM" --phase live || die "live boot failed"
+
+log "installing VyOS to disk (unattended over the console)..."
+python3 "$SCRIPT_DIR/router-install.py" --vm "$ROUTER_VM" --phase install || die "install failed"
+
+# Boot the INSTALLED system from here on. Without this the VM was created with
+# --boot cdrom,hd and every restart re-runs the ISO, so the live system comes
+# back with no config and every `commit; save` silently evaporates.
+log "switching boot to disk and ejecting the install media..."
+virsh_q destroy "$ROUTER_VM" >/dev/null 2>&1 || true
+sleep 2
+sudo virt-xml "$ROUTER_VM" --edit --boot hd >/dev/null
+sudo virt-xml "$ROUTER_VM" --remove-device --disk device=cdrom >/dev/null 2>&1 || true
+virsh_q start "$ROUTER_VM" >/dev/null
+sleep 10
+
+# Bond the taps only now: they are recreated by the restart above, so bonding
+# before this would bond stale interfaces.
+ovs_bond_router "$ROUTER_VM"
+
+log "applying router config (bond0 LACP + VLAN gateways)..."
+python3 "$SCRIPT_DIR/router-install.py" --vm "$ROUTER_VM" --phase configure \
+  --vlans "$(printf '%s\n' "${SELECTED[@]}" | tr '\n' ' ')" || die "configure failed"
+
+log "waiting for LACP to negotiate..."
+for _ in $(seq 1 30); do
+  if sudo ovs-appctl lacp/show "$LAG_NAME" 2>/dev/null | grep -q "current attached"; then
+    log "LACP negotiated"; break
+  fi
+  sleep 5
+done
+
+echo
+ovs_bond_status
+echo
+log "router is up. Check reachability with: $SCRIPT_DIR/labsim-matrix.py --watch 2"