93fed7826b
Some checks failed
Full run green: baseline (one AC session, held by the VIP holder), clean
failover (pppoe0 moves in 26s, old master releases), 10 gig down (route falls to
pppoe0 and the LAN is back online in 5s), lease expiry (the guard hangs up), and
a missing peers file (NRestarts=0, no loop). Evidence in
labsim/wan-failover-evidence/.
The 10 gig test found a genuine bug in vrrp-wan-health, not in the sim. GRACE
was measured from PROMOTION, so an established master had no grace at all --
after hours of uptime `now - since` far exceeds any window. The first moment
bond0.53 went down while pppoe0 was mid-redial, the master failed its own check,
shed every VIP, and the peer inherited the same WAN outage and did the same. A
brief WAN blip would have flapped the production pair. The stamp is now
refreshed on every healthy tick, so grace measures time since the box last
demonstrably HAD a WAN -- survivable wherever the gap happens, not only just
after a promotion.
Three harness bugs, all the same shape, all of which produced a confident wrong
answer before being caught:
- waiting for "exactly one pppoe0 holder" returns INSTANTLY during a handover,
because it was already true. The useful question is who holds it.
- judging connectivity on a single ping 20s after a link drop reported an
outage that had already healed. Poll, do not sample.
- `-o PreferredAuthentications=password` suits the routers but not the Alpine
LAN VMs, whose sshd offers keyboard-interactive: ssh exited 255 before
running anything and the test read that as "the LAN lost the internet". A
tcpdump on the router showed the pings leaving pppoe0 NATed to
198.51.100.117 and the replies coming back the whole time. An exit code that
can mean "the network is broken" or "I could not log in" is not a
connectivity test, so the check now asserts on what the guest reported.
That last one is why the harness asks the routers and the access concentrator
rather than a client, and why it refuses to run at all while either router still
has a default route via eth2 -- the libvirt-NAT scaffold answers connectivity
checks the WAN under test would have failed.
Still to run: hard failover (destroy the master), and the session-control
replace/deny/disable axis that brackets Vodafone's unknown behaviour and sets
the final GRACE. Nothing applied to production.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
338 lines
16 KiB
Markdown
338 lines
16 KiB
Markdown
# labsim — libvirt replica of the lab network
|
||
|
||
A throwaway copy of the production VLAN topology for testing routing, firewall
|
||
rules and failover **without touching the real network**. Same VLAN IDs and
|
||
roles as UniFi, deliberately different IP ranges so nothing can be confused for
|
||
production.
|
||
|
||
## Topology
|
||
|
||
Each VLAN is its own isolated libvirt network with one tiny Alpine VM on it.
|
||
|
||
| VLAN | Name | Sim subnet | VM address | Mirrors production |
|
||
|-----:|------|------------|-----------|--------------------|
|
||
| 1 | management | 172.31.1.0/24 | 172.31.1.10 | 192.168.1.0/24 |
|
||
| 2 | k8s | 172.31.2.0/24 | 172.31.2.10 | 192.168.8.0/23 |
|
||
| 3 | kvm | 172.31.3.0/24 | 172.31.3.10 | 192.168.3.0/24 |
|
||
| 9 | private | 172.31.9.0/24 | 172.31.9.10 | 10.0.9.0/23 |
|
||
| 10 | lot | **172.31.10.0/23** | 172.31.10.10 | 10.0.0.0/23 |
|
||
| 200 | roomates | 172.31.200.0/24 | 172.31.200.10 | 192.168.2.0/24 |
|
||
|
||
The sim subnet encodes the VLAN id: `172.31.<vlan>.0/24`, with one exception.
|
||
**VLAN 10 is a `/23`** because every UniFi DHCP reservation lives in LoT and LoT
|
||
spans `10.0.0.x` *and* `10.0.1.x`, which a `/24` cannot hold. The mapping stays
|
||
readable — `10.0.0.46 → 172.31.10.46`, `10.0.1.67 → 172.31.11.67`.
|
||
|
||
LoT's host leg is `.3`, not `.2`, because `10.0.0.2` is a real reservation
|
||
(Hubitat) that maps onto `172.31.10.2`. `.3` is unreserved and sits below the
|
||
DHCP pool, so it can never be handed out.
|
||
|
||
`vlans.conf` therefore takes two optional trailing fields:
|
||
|
||
```
|
||
vlan_id:name:sim_prefix:real_subnet[:masklen][:host_octet]
|
||
```
|
||
|
||
defaulting to `24` and `2`. k8s and Private are also `/23` in production but
|
||
hold no reservations, so they keep their `/24` and their DHCP range is clamped
|
||
— reported at generation time, never silently.
|
||
|
||
Address plan, identical on every VLAN:
|
||
|
||
| Address | Role |
|
||
|---------|------|
|
||
| `.1` | gateway under test — a router VM you add (not created by default) |
|
||
| `.2` | host bridge — how you reach the VMs from this workstation |
|
||
| `.10` | the VLAN's micro VM |
|
||
| `.254` | reserved for a VRRP VIP, mirroring production |
|
||
|
||
The host sits at `.2` purely so you can SSH in. It is deliberately **not** the
|
||
VMs' default route — that is `.1` — so inter-VLAN tests fail loudly when no
|
||
router is present instead of being silently served by the host's own routing
|
||
table. libvirt also installs reject rules that stop these networks forwarding
|
||
to each other, so traffic between VLANs only works once a router VM bridges
|
||
them.
|
||
|
||
## Usage
|
||
|
||
```bash
|
||
./labsim-up.sh # bring up every VLAN (idempotent)
|
||
./labsim-up.sh 2 3 # only VLANs 2 and 3
|
||
./labsim-down.sh # destroy VMs + networks, keep the base image
|
||
./labsim-down.sh --purge # also delete the downloaded Alpine image
|
||
```
|
||
|
||
Each VM: 256 MB, 1 vCPU, a copy-on-write overlay on one shared 176 MB Alpine
|
||
image (so six VMs cost a few MB of disk, not 1 GB).
|
||
|
||
## Access
|
||
|
||
```bash
|
||
ssh alpine@172.31.2.10 # normal user (password: labsim)
|
||
ssh root@172.31.2.10 # privileged — this image has no sudo
|
||
curl http://172.31.2.10/ # hello-world page naming the VLAN
|
||
```
|
||
|
||
Console, when the network is the thing that is broken:
|
||
|
||
```bash
|
||
sudo virsh console labsim-2-k8s # root / labsim
|
||
```
|
||
|
||
## Watching it
|
||
|
||
```bash
|
||
./labsim-matrix.py --watch 2 # terminal grid, changed cells highlighted
|
||
./monitoring-up.sh # topology page + Prometheus + Grafana
|
||
```
|
||
|
||
## Testing the DHCP migration
|
||
|
||
`./labsim-dhcp-test.sh` boots throwaway VMs whose MACs are **real production
|
||
MACs** and checks each gets the address UniFi reserved for it. MACs are the one
|
||
piece of production config that transplants verbatim, which is what makes this a
|
||
test rather than a rehearsal. It is safe because `ovs-labsim` has no physical
|
||
NIC — verified with `ovs-vsctl show` — so a production MAC cannot reach the real
|
||
LAN.
|
||
|
||
Apply the config first, from `../migration`:
|
||
|
||
```bash
|
||
python3 unifi-to-vyos.py --mode sim -o /tmp/sim.conf # 6 subnets, 31 mappings
|
||
# load onto labsim-vyos, then:
|
||
./labsim-dhcp-test.sh
|
||
```
|
||
|
||
**Result on VyOS 2026.08 (kea): all four cases pass.** The one that mattered:
|
||
most UniFi reservations sit *inside* the DHCP pool, and **kea honours in-pool
|
||
host reservations** — `printer1` received `172.31.10.46` from within the
|
||
`.10.11–.11.254` pool. That was the open question blocking the cutover.
|
||
|
||
### The lease database will lie to you
|
||
|
||
The script wipes `/config/dhcp/dhcp4-leases.csv*` before every run, and both
|
||
halves of that matter:
|
||
|
||
- **Stale leases defeat reservations.** Re-running against yesterday's leases,
|
||
kea handed dynamic addresses to three devices that have reservations. The
|
||
reservation was present and correct in `/run/kea/kea-dhcp4.conf` the whole
|
||
time. Kea saw the reserved address as already leased to "another client" —
|
||
same MAC, but a different client-id from the earlier boot — and allocated
|
||
elsewhere. The cutover itself starts with an empty lease database, so this is
|
||
a *testing* artifact, but it is worth knowing that a reservation is not an
|
||
unconditional guarantee once leases exist.
|
||
- **The `*` is load-bearing.** Kea's memfile backend keeps lease-file-cleanup
|
||
rotations (`dhcp4-leases.csv.2`) and restores from them on start, so
|
||
truncating only the primary file changes nothing.
|
||
|
||
Both of those first appeared as a *passing* test. The verdict logic now refuses
|
||
to score a MAC with more than one lease, because taking the first match had
|
||
reported an hours-old lease as the current answer and turned three failures
|
||
into apparent passes.
|
||
|
||
Still open: whether kea will hand a *reserved* address to a *different* client
|
||
while the reserved device is offline. The negative case here only proves an
|
||
unreserved MAC gets an unreserved address.
|
||
|
||
- **http://localhost:9101/** — live mesh: a node per VLAN, the router in the
|
||
middle, one line per pair coloured green/red with the ICMP RTT on it. Hover a
|
||
line for per-direction detail. Refreshes every 5s. This is the one to watch
|
||
while changing firewall rules.
|
||
- **http://localhost:3000/d/labsim-matrix** — Grafana (anonymous, no login) for
|
||
*history*: when did a path flip, and how has latency moved.
|
||
- **http://localhost:9101/metrics** — `labsim_reachable{src,dst,proto}` and
|
||
`labsim_rtt_ms{src,dst}`.
|
||
|
||
## WAN follows VRRP, and the PPPoE half of it
|
||
|
||
One consumer ISP account, two routers. The 10 gig line's lease is bound to a
|
||
cloned MAC and the Vodafone line to a single credential, so neither can be live
|
||
on both boxes: the WAN has to move with mastership.
|
||
|
||
The two halves use different control planes, and that asymmetry is the design:
|
||
|
||
| | plane | why |
|
||
|---|---|---|
|
||
| `bond0.53` | VyOS **config** (`disable`) | only config can move a MAC |
|
||
| `pppoe0` | **systemd** unit gate | see below |
|
||
|
||
`set interfaces pppoe pppoe0 disable` cannot work as a resting state.
|
||
`interfaces_pppoe.py` treats `disable` and `delete` identically and **unlinks
|
||
`/etc/ppp/peers/pppoe0`** — which is pppd's own options file. The resting state
|
||
therefore destroyed what the promotion path needed, and `ppp@pppoe0`
|
||
restart-looped against it (47 restarts, zero sessions at the AC). It also makes
|
||
op-mode `connect interface pppoe0` refuse, and puts every failover behind a
|
||
priority-322 commit where one unrelated bad node fails the lot.
|
||
|
||
So `pppoe0` is configured identically and **enabled on both**, and dialling is
|
||
gated by a drop-in:
|
||
|
||
```ini
|
||
ConditionPathExists=/run/vrrp-wan/may-dial
|
||
ConditionPathExists=/etc/ppp/peers/pppoe0
|
||
```
|
||
|
||
`/run` is tmpfs, so the gate is shut at boot. That matters more than it looks:
|
||
with the node enabled, `interfaces_pppoe.py` restarts ppp on **every** commit
|
||
touching the pppoe subtree when the daemon isn't running — so the backup
|
||
actively tries to dial whenever anything commits. The gate is the only thing
|
||
making that a no-op, which is why the reconciler refuses to bless a box whose
|
||
drop-in is missing (`/etc` is per-image; a VyOS upgrade would silently remove
|
||
the protection).
|
||
|
||
`may-dial` is a **lease**, not a flag: `ConditionPathExists` is evaluated at
|
||
start only, so it can prevent a dial but never revoke one. `vrrp-wan-reconcile`
|
||
renews it every 30s; `vrrp-wan-guard` runs every 5s and only ever revokes.
|
||
|
||
### Testing it
|
||
|
||
```sh
|
||
./labsim-pppoe-ha-test.sh --all
|
||
```
|
||
|
||
The verdict is what the **routers** and the **access concentrator** did, never
|
||
what a client happened to get — and the harness refuses to run at all while a
|
||
router still has a default route via `eth2`, because the libvirt-NAT scaffold
|
||
answers connectivity checks that the WAN under test would have failed. The
|
||
invariant it enforces throughout: *the AC never reports two `simdsl` sessions,
|
||
and no two routers ever hold `pppoe0`.*
|
||
|
||
Two failures the harness itself produced, both worth remembering: waiting for
|
||
"exactly one holder" returns instantly during a handover (it was already true),
|
||
and judging connectivity on a single ping 20s after a link drop reports an
|
||
outage that has already healed. Ask **who** holds it, and poll.
|
||
|
||
Evidence in `wan-failover-evidence/`.
|
||
|
||
## Routing: BGP, dual WAN, and the ISP VMs
|
||
|
||
`sim-ha-config.py` covers the LAN side of the routers. `sim-net-config.py`
|
||
covers everything that makes this a rehearsal for production *routing*:
|
||
|
||
| role | VM | what it generates |
|
||
|---|---|---|
|
||
| `primary` | `labsim-vyos` | BGP + dual WAN + health-checked failover |
|
||
| `secondary` | `labsim-vyos2` | BGP only |
|
||
| `isp-dhcp` | `labsim-isp-dhcp` | 10gig-equivalent ISP on VLAN 53 |
|
||
| `isp-pppoe` | `labsim-isp-pppoe` | Vodafone-equivalent PPPoE ISP on VLAN 51 |
|
||
|
||
Both ISP VMs are VyOS with two NICs: one on the OVS trunk facing the sim
|
||
router, one on libvirt's `default` network, NATing customers to the real
|
||
internet. They use RFC 5737 documentation ranges (`203.0.113.0/24`,
|
||
`198.51.100.0/24`) so a leaked sim route cannot blackhole anything real.
|
||
|
||
```sh
|
||
./sim-net-apply.sh check # VM state vs what the code says — run this first
|
||
./sim-net-apply.sh apply # push generated config over the serial console
|
||
```
|
||
|
||
`check` is the important one. All of this previously existed only as running
|
||
state, applied by hand over SSH; rebuilding a VM lost it, and nothing recorded
|
||
why any of it was shaped the way it was.
|
||
|
||
### Known gaps vs production
|
||
|
||
- **WAN is on the primary router only.** Production has WAN on both. Two PPPoE
|
||
clients sharing one credential against a single access concentrator is a
|
||
failure mode production does not have, so the sim does not model it. VRRP and
|
||
conntrack failover are still exercised.
|
||
- **ISP VM interface names are not stable across a rebuild** — `isp-dhcp` came
|
||
up as `eth0`/`eth1` and `isp-pppoe` as `eth2`/`eth3` from identical XML.
|
||
Check `show interfaces` and pass `--wan-if` / `--uplink-if` rather than
|
||
trusting the defaults.
|
||
- **`eth2` on the primary router** is a libvirt-NAT uplink predating the ISP
|
||
VMs: a third default route with no production equivalent that masks real WAN
|
||
failures during a failover test. `--drop-scaffold` removes it.
|
||
- **Committing on `isp-pppoe` drops the router's PPPoE session**, and the
|
||
client does not redial promptly. After any change there, check `pppoe0` on
|
||
the router and `sudo systemctl restart ppp@pppoe0` if it is missing.
|
||
|
||
## The trunk carries every VLAN tagged, including Management
|
||
|
||
There is deliberately **no native/untagged VLAN** on the trunks to the routers,
|
||
and Management lives on `bond0.1`, not on the bare `bond0`.
|
||
|
||
A native VLAN is what puts a subnet on the bond **parent** while every other
|
||
VLAN sits on a sub-interface of it. With `dhcp-socket-type: raw`, kea receives
|
||
each tagged frame *twice* — once on the sub-interface and once on the parent —
|
||
and answers from the parent's pool as well (ISC Kea
|
||
[#1117](https://gitlab.isc.org/isc-projects/kea/-/issues/1117)). A client on
|
||
VLAN 3 gets two OFFERs and keeps whichever arrives first:
|
||
|
||
```
|
||
bond0.3 : 172.31.3.252 → 172.31.3.11 correct
|
||
bond0 : 172.31.1.252 → 172.31.1.8 UNTAGGED, Management pool, wrong
|
||
```
|
||
|
||
`./labsim-vlan-leak-test.sh` makes one client on a tagged VLAN send a DISCOVER
|
||
and captures on the parent and the sub-interface at once. The verdict is how
|
||
many OFFERs the **server** emitted and from which subnets — deliberately not
|
||
"did the client get the right address", because a client picking correctly is
|
||
exactly how this hid. Both orderings were observed across runs, so a passing
|
||
client proves nothing.
|
||
|
||
```sh
|
||
./labsim-vlan-leak-test.sh --vlan 3 # PASS on the current shape
|
||
LABSIM_NATIVE_VLAN=1 ./router-up.sh # restore the old shape...
|
||
./labsim-vlan-leak-test.sh --vlan 3 # ...and it FAILs again
|
||
```
|
||
|
||
Three things this cost, all of which apply to production:
|
||
|
||
- **Kea must be restarted after the address moves.** VyOS does not restart it
|
||
for an interface address change, so it keeps a raw socket bound with the old
|
||
address and the bug survives the fix. In the sim kea had been running since
|
||
16 Aug; the first post-fix test failed for this reason alone and looked like
|
||
the fix simply not working.
|
||
- **The firewall interface-group must move too.** `interface-group LAN` named
|
||
the bare `bond0`; with a default-deny ruleset, moving the address without
|
||
moving the group drops every management session and all VLAN 1 routing.
|
||
- **Duplicate delivery does not stop.** #1117 says only that there is no longer
|
||
a subnet on the parent to match, and that is exactly what happens: two replies
|
||
per DISCOVER, both now from the correct pool. Harmless, but do not read a
|
||
duplicate as a failure.
|
||
|
||
### Tagged and untagged Management coexist
|
||
|
||
Verified directly, and it is what makes the production cutover a rolling change
|
||
rather than an outage: with the primary still untagged on `bond0` and the
|
||
secondary already tagged on `bond0.1`, both routers were reachable, the VIP
|
||
stayed up and a VLAN 1 client kept its gateway. One VLAN is one broadcast
|
||
domain regardless of how each port tags it, so the two firewalls can be
|
||
converted one at a time. See `migration/MANAGEMENT-VLAN-TAGGED.md`.
|
||
|
||
`./vlan1-move-monitor.sh` logs VIP/router liveness once a second during the
|
||
change, because VRRP reconverges and leaves no trace of who held the VIP.
|
||
|
||
## Notes for whoever extends this
|
||
|
||
Things that cost time the first time round, all verified on this image:
|
||
|
||
- **No `sudo`.** Alpine ships `doas`; cloud-init's `sudo:` directive is inert
|
||
here. Use `root@` for privileged work.
|
||
- **cloud-init leaves users locked** (`!*` in `/etc/shadow`) unless
|
||
`lock_passwd: false`, and sshd then refuses key auth for that user.
|
||
- **One failing `runcmd` aborts every command after it.** Each entry is
|
||
`|| true` for that reason.
|
||
- **busybox here has no `httpd` applet**, and the VMs have no internet to
|
||
`apk add` one — so the hello-world server is `python3 -m http.server`
|
||
(python3 is already present because cloud-init depends on it).
|
||
- **`start-stop-daemon --exec /usr/bin/python3` matches cloud-init's own
|
||
python3** at boot and refuses to start anything.
|
||
- **busybox `pgrep -f PATTERN` matches its own argv**, so a "skip if already
|
||
running" guard always fires. Verified: `guard_exit=0` with nothing listening.
|
||
|
||
## Not modelled (yet)
|
||
|
||
- **The secondary's bond was fiction until 2026-09-02.** `ovs_bond_router`'s
|
||
"already bonded, nothing to do" check compared only the trunk VLAN list, not
|
||
the membership. Restarting a VM recreates its taps under new names, so the
|
||
bond sat there holding two interfaces that no longer existed while the router's
|
||
real taps ran in the bridge as two *independent* ports — no LACP, and carrying
|
||
libvirt's own portgroup VLAN config rather than the bond's. It reconciles
|
||
membership now, but the lesson generalises: a sim that reports success is not
|
||
the same as a sim that models the thing.
|
||
- **`labsim-vyos` has a third NIC** on libvirt's `default` network (the scaffold
|
||
uplink, see `--drop-scaffold`). The tap count is filtered to `$OVS_NET` for
|
||
that reason; an unfiltered count is 3 and silently skipped the primary's bond.
|