lab/labsim

labsim — libvirt replica of the lab network

A throwaway copy of the production VLAN topology for testing routing, firewall rules and failover without touching the real network. Same VLAN IDs and roles as UniFi, deliberately different IP ranges so nothing can be confused for production.

Topology

Each VLAN is its own isolated libvirt network with one tiny Alpine VM on it.

VLAN	Name	Sim subnet	VM address	Mirrors production
1	management	172.31.1.0/24	172.31.1.10	192.168.1.0/24
2	k8s	172.31.2.0/24	172.31.2.10	192.168.8.0/23
3	kvm	172.31.3.0/24	172.31.3.10	192.168.3.0/24
9	private	172.31.9.0/24	172.31.9.10	10.0.9.0/23
10	lot	172.31.10.0/23	172.31.10.10	10.0.0.0/23
200	roomates	172.31.200.0/24	172.31.200.10	192.168.2.0/24

The sim subnet encodes the VLAN id: 172.31.<vlan>.0/24, with one exception. VLAN 10 is a /23 because every UniFi DHCP reservation lives in LoT and LoT spans 10.0.0.x and 10.0.1.x, which a /24 cannot hold. The mapping stays readable — 10.0.0.46 → 172.31.10.46, 10.0.1.67 → 172.31.11.67.

LoT's host leg is .3, not .2, because 10.0.0.2 is a real reservation (Hubitat) that maps onto 172.31.10.2. .3 is unreserved and sits below the DHCP pool, so it can never be handed out.

vlans.conf therefore takes two optional trailing fields:

vlan_id:name:sim_prefix:real_subnet[:masklen][:host_octet]

defaulting to 24 and 2. k8s and Private are also /23 in production but hold no reservations, so they keep their /24 and their DHCP range is clamped — reported at generation time, never silently.

Address plan, identical on every VLAN:

Address	Role
.1	gateway under test — a router VM you add (not created by default)
.2	host bridge — how you reach the VMs from this workstation
.10	the VLAN's micro VM
.254	reserved for a VRRP VIP, mirroring production

The host sits at .2 purely so you can SSH in. It is deliberately not the VMs' default route — that is .1 — so inter-VLAN tests fail loudly when no router is present instead of being silently served by the host's own routing table. libvirt also installs reject rules that stop these networks forwarding to each other, so traffic between VLANs only works once a router VM bridges them.

Usage

./labsim-up.sh              # bring up every VLAN (idempotent)
./labsim-up.sh 2 3          # only VLANs 2 and 3
./labsim-down.sh            # destroy VMs + networks, keep the base image
./labsim-down.sh --purge    # also delete the downloaded Alpine image

Each VM: 256 MB, 1 vCPU, a copy-on-write overlay on one shared 176 MB Alpine image (so six VMs cost a few MB of disk, not 1 GB).

Access

ssh alpine@172.31.2.10       # normal user (password: labsim)
ssh root@172.31.2.10         # privileged — this image has no sudo
curl http://172.31.2.10/     # hello-world page naming the VLAN

Console, when the network is the thing that is broken:

sudo virsh console labsim-2-k8s     # root / labsim

Watching it

./labsim-matrix.py --watch 2     # terminal grid, changed cells highlighted
./monitoring-up.sh               # topology page + Prometheus + Grafana

Testing the DHCP migration

./labsim-dhcp-test.sh boots throwaway VMs whose MACs are real production MACs and checks each gets the address UniFi reserved for it. MACs are the one piece of production config that transplants verbatim, which is what makes this a test rather than a rehearsal. It is safe because ovs-labsim has no physical NIC — verified with ovs-vsctl show — so a production MAC cannot reach the real LAN.

Apply the config first, from ../migration:

python3 unifi-to-vyos.py --mode sim -o /tmp/sim.conf   # 6 subnets, 31 mappings
# load onto labsim-vyos, then:
./labsim-dhcp-test.sh

Result on VyOS 2026.08 (kea): all four cases pass. The one that mattered: most UniFi reservations sit inside the DHCP pool, and kea honours in-pool host reservations — printer1 received 172.31.10.46 from within the .10.11–.11.254 pool. That was the open question blocking the cutover.

The lease database will lie to you

The script wipes /config/dhcp/dhcp4-leases.csv* before every run, and both halves of that matter:

• Stale leases defeat reservations. Re-running against yesterday's leases, kea handed dynamic addresses to three devices that have reservations. The reservation was present and correct in /run/kea/kea-dhcp4.conf the whole time. Kea saw the reserved address as already leased to "another client" — same MAC, but a different client-id from the earlier boot — and allocated elsewhere. The cutover itself starts with an empty lease database, so this is a testing artifact, but it is worth knowing that a reservation is not an unconditional guarantee once leases exist.
• The * is load-bearing. Kea's memfile backend keeps lease-file-cleanup rotations (dhcp4-leases.csv.2) and restores from them on start, so truncating only the primary file changes nothing.

Both of those first appeared as a passing test. The verdict logic now refuses to score a MAC with more than one lease, because taking the first match had reported an hours-old lease as the current answer and turned three failures into apparent passes.

Still open: whether kea will hand a reserved address to a different client while the reserved device is offline. The negative case here only proves an unreserved MAC gets an unreserved address.

• http://localhost:9101/ — live mesh: a node per VLAN, the router in the middle, one line per pair coloured green/red with the ICMP RTT on it. Hover a line for per-direction detail. Refreshes every 5s. This is the one to watch while changing firewall rules.
• http://localhost:3000/d/labsim-matrix — Grafana (anonymous, no login) for history: when did a path flip, and how has latency moved.
• http://localhost:9101/metrics — labsim_reachable{src,dst,proto} and labsim_rtt_ms{src,dst}.

Routing: BGP, dual WAN, and the ISP VMs

sim-ha-config.py covers the LAN side of the routers. sim-net-config.py covers everything that makes this a rehearsal for production routing:

role	VM	what it generates
primary	labsim-vyos	BGP + dual WAN + health-checked failover
secondary	labsim-vyos2	BGP only
isp-dhcp	labsim-isp-dhcp	10gig-equivalent ISP on VLAN 53
isp-pppoe	labsim-isp-pppoe	Vodafone-equivalent PPPoE ISP on VLAN 51

Both ISP VMs are VyOS with two NICs: one on the OVS trunk facing the sim router, one on libvirt's default network, NATing customers to the real internet. They use RFC 5737 documentation ranges (203.0.113.0/24, 198.51.100.0/24) so a leaked sim route cannot blackhole anything real.

./sim-net-apply.sh check     # VM state vs what the code says — run this first
./sim-net-apply.sh apply     # push generated config over the serial console

check is the important one. All of this previously existed only as running state, applied by hand over SSH; rebuilding a VM lost it, and nothing recorded why any of it was shaped the way it was.

Known gaps vs production

• WAN is on the primary router only. Production has WAN on both. Two PPPoE clients sharing one credential against a single access concentrator is a failure mode production does not have, so the sim does not model it. VRRP and conntrack failover are still exercised.
• ISP VM interface names are not stable across a rebuild — isp-dhcp came up as eth0/eth1 and isp-pppoe as eth2/eth3 from identical XML. Check show interfaces and pass --wan-if / --uplink-if rather than trusting the defaults.
• eth2 on the primary router is a libvirt-NAT uplink predating the ISP VMs: a third default route with no production equivalent that masks real WAN failures during a failover test. --drop-scaffold removes it.
• Committing on isp-pppoe drops the router's PPPoE session, and the client does not redial promptly. After any change there, check pppoe0 on the router and sudo systemctl restart ppp@pppoe0 if it is missing.

Notes for whoever extends this

Things that cost time the first time round, all verified on this image:

• No sudo. Alpine ships doas; cloud-init's sudo: directive is inert here. Use root@ for privileged work.
• cloud-init leaves users locked (!* in /etc/shadow) unless lock_passwd: false, and sshd then refuses key auth for that user.
• One failing runcmd aborts every command after it. Each entry is || true for that reason.
• busybox here has no httpd applet, and the VMs have no internet to apk add one — so the hello-world server is python3 -m http.server (python3 is already present because cloud-init depends on it).
• start-stop-daemon --exec /usr/bin/python3 matches cloud-init's own python3 at boot and refuses to start anything.
• busybox pgrep -f PATTERN matches its own argv, so a "skip if already running" guard always fires. Verified: guard_exit=0 with nothing listening.

Not modelled (yet)

VLANs are separate L2 segments rather than one 802.1Q trunk, so this exercises inter-VLAN routing but not a bond0.<vif> trunk config specifically. A router VM would attach one NIC per VLAN. Adding a tagged-trunk variant is the obvious next step if the bond/vif config itself needs testing.