lab/labsim/README.md

# labsim — libvirt replica of the lab network

A throwaway copy of the production VLAN topology for testing routing, firewall
rules and failover **without touching the real network**. Same VLAN IDs and
roles as UniFi, deliberately different IP ranges so nothing can be confused for
production.

## Topology

Each VLAN is its own isolated libvirt network with one tiny Alpine VM on it.

| VLAN | Name | Sim subnet | VM address | Mirrors production |
|-----:|------|------------|-----------|--------------------|
| 1 | management | 172.31.1.0/24 | 172.31.1.10 | 192.168.1.0/24 |
| 2 | k8s | 172.31.2.0/24 | 172.31.2.10 | 192.168.8.0/23 |
| 3 | kvm | 172.31.3.0/24 | 172.31.3.10 | 192.168.3.0/24 |
| 9 | private | 172.31.9.0/24 | 172.31.9.10 | 10.0.9.0/23 |
| 10 | lot | **172.31.10.0/23** | 172.31.10.10 | 10.0.0.0/23 |
| 200 | roomates | 172.31.200.0/24 | 172.31.200.10 | 192.168.2.0/24 |

The sim subnet encodes the VLAN id: `172.31.<vlan>.0/24`, with one exception.
**VLAN 10 is a `/23`** because every UniFi DHCP reservation lives in LoT and LoT
spans `10.0.0.x` *and* `10.0.1.x`, which a `/24` cannot hold. The mapping stays
readable — `10.0.0.46 → 172.31.10.46`, `10.0.1.67 → 172.31.11.67`.

LoT's host leg is `.3`, not `.2`, because `10.0.0.2` is a real reservation
(Hubitat) that maps onto `172.31.10.2`. `.3` is unreserved and sits below the
DHCP pool, so it can never be handed out.

`vlans.conf` therefore takes two optional trailing fields:

```
vlan_id:name:sim_prefix:real_subnet[:masklen][:host_octet]
```

defaulting to `24` and `2`. k8s and Private are also `/23` in production but
hold no reservations, so they keep their `/24` and their DHCP range is clamped
— reported at generation time, never silently.

Address plan, identical on every VLAN:

| Address | Role |
|---------|------|
| `.1` | gateway under test — a router VM you add (not created by default) |
| `.2` | host bridge — how you reach the VMs from this workstation |
| `.10` | the VLAN's micro VM |
| `.254` | reserved for a VRRP VIP, mirroring production |

The host sits at `.2` purely so you can SSH in. It is deliberately **not** the
VMs' default route — that is `.1` — so inter-VLAN tests fail loudly when no
router is present instead of being silently served by the host's own routing
table. libvirt also installs reject rules that stop these networks forwarding
to each other, so traffic between VLANs only works once a router VM bridges
them.

## Usage

```bash
./labsim-up.sh              # bring up every VLAN (idempotent)
./labsim-up.sh 2 3          # only VLANs 2 and 3
./labsim-down.sh            # destroy VMs + networks, keep the base image
./labsim-down.sh --purge    # also delete the downloaded Alpine image
```

Each VM: 256 MB, 1 vCPU, a copy-on-write overlay on one shared 176 MB Alpine
image (so six VMs cost a few MB of disk, not 1 GB).

## Access

```bash
ssh alpine@172.31.2.10       # normal user (password: labsim)
ssh root@172.31.2.10         # privileged — this image has no sudo
curl http://172.31.2.10/     # hello-world page naming the VLAN
```

Console, when the network is the thing that is broken:

```bash
sudo virsh console labsim-2-k8s     # root / labsim
```

## Watching it

```bash
./labsim-matrix.py --watch 2     # terminal grid, changed cells highlighted
./monitoring-up.sh               # topology page + Prometheus + Grafana
```

## Testing the DHCP migration

`./labsim-dhcp-test.sh` boots throwaway VMs whose MACs are **real production
MACs** and checks each gets the address UniFi reserved for it. MACs are the one
piece of production config that transplants verbatim, which is what makes this a
test rather than a rehearsal. It is safe because `ovs-labsim` has no physical
NIC — verified with `ovs-vsctl show` — so a production MAC cannot reach the real
LAN.

Apply the config first, from `../migration`:

```bash
python3 unifi-to-vyos.py --mode sim -o /tmp/sim.conf   # 6 subnets, 31 mappings
# load onto labsim-vyos, then:
./labsim-dhcp-test.sh
```

**Result on VyOS 2026.08 (kea): all four cases pass.** The one that mattered:
most UniFi reservations sit *inside* the DHCP pool, and **kea honours in-pool
host reservations** — `printer1` received `172.31.10.46` from within the
`.10.11–.11.254` pool. That was the open question blocking the cutover.

### The lease database will lie to you

The script wipes `/config/dhcp/dhcp4-leases.csv*` before every run, and both
halves of that matter:

- **Stale leases defeat reservations.** Re-running against yesterday's leases,
  kea handed dynamic addresses to three devices that have reservations. The
  reservation was present and correct in `/run/kea/kea-dhcp4.conf` the whole
  time. Kea saw the reserved address as already leased to "another client" —
  same MAC, but a different client-id from the earlier boot — and allocated
  elsewhere. The cutover itself starts with an empty lease database, so this is
  a *testing* artifact, but it is worth knowing that a reservation is not an
  unconditional guarantee once leases exist.
- **The `*` is load-bearing.** Kea's memfile backend keeps lease-file-cleanup
  rotations (`dhcp4-leases.csv.2`) and restores from them on start, so
  truncating only the primary file changes nothing.

Both of those first appeared as a *passing* test. The verdict logic now refuses
to score a MAC with more than one lease, because taking the first match had
reported an hours-old lease as the current answer and turned three failures
into apparent passes.

Still open: whether kea will hand a *reserved* address to a *different* client
while the reserved device is offline. The negative case here only proves an
unreserved MAC gets an unreserved address.

- **http://localhost:9101/** — live mesh: a node per VLAN, the router in the
  middle, one line per pair coloured green/red with the ICMP RTT on it. Hover a
  line for per-direction detail. Refreshes every 5s. This is the one to watch
  while changing firewall rules.
- **http://localhost:3000/d/labsim-matrix** — Grafana (anonymous, no login) for
  *history*: when did a path flip, and how has latency moved.
- **http://localhost:9101/metrics** — `labsim_reachable{src,dst,proto}` and
  `labsim_rtt_ms{src,dst}`.

## Routing: BGP, dual WAN, and the ISP VMs

`sim-ha-config.py` covers the LAN side of the routers. `sim-net-config.py`
covers everything that makes this a rehearsal for production *routing*:

| role | VM | what it generates |
|---|---|---|
| `primary` | `labsim-vyos` | BGP + dual WAN + health-checked failover |
| `secondary` | `labsim-vyos2` | BGP only |
| `isp-dhcp` | `labsim-isp-dhcp` | 10gig-equivalent ISP on VLAN 53 |
| `isp-pppoe` | `labsim-isp-pppoe` | Vodafone-equivalent PPPoE ISP on VLAN 51 |

Both ISP VMs are VyOS with two NICs: one on the OVS trunk facing the sim
router, one on libvirt's `default` network, NATing customers to the real
internet. They use RFC 5737 documentation ranges (`203.0.113.0/24`,
`198.51.100.0/24`) so a leaked sim route cannot blackhole anything real.

```sh
./sim-net-apply.sh check     # VM state vs what the code says — run this first
./sim-net-apply.sh apply     # push generated config over the serial console
```

`check` is the important one. All of this previously existed only as running
state, applied by hand over SSH; rebuilding a VM lost it, and nothing recorded
why any of it was shaped the way it was.

### Known gaps vs production

- **WAN is on the primary router only.** Production has WAN on both. Two PPPoE
  clients sharing one credential against a single access concentrator is a
  failure mode production does not have, so the sim does not model it. VRRP and
  conntrack failover are still exercised.
- **ISP VM interface names are not stable across a rebuild** — `isp-dhcp` came
  up as `eth0`/`eth1` and `isp-pppoe` as `eth2`/`eth3` from identical XML.
  Check `show interfaces` and pass `--wan-if` / `--uplink-if` rather than
  trusting the defaults.
- **`eth2` on the primary router** is a libvirt-NAT uplink predating the ISP
  VMs: a third default route with no production equivalent that masks real WAN
  failures during a failover test. `--drop-scaffold` removes it.
- **Committing on `isp-pppoe` drops the router's PPPoE session**, and the
  client does not redial promptly. After any change there, check `pppoe0` on
  the router and `sudo systemctl restart ppp@pppoe0` if it is missing.

## Notes for whoever extends this

Things that cost time the first time round, all verified on this image:

- **No `sudo`.** Alpine ships `doas`; cloud-init's `sudo:` directive is inert
  here. Use `root@` for privileged work.
- **cloud-init leaves users locked** (`!*` in `/etc/shadow`) unless
  `lock_passwd: false`, and sshd then refuses key auth for that user.
- **One failing `runcmd` aborts every command after it.** Each entry is
  `|| true` for that reason.
- **busybox here has no `httpd` applet**, and the VMs have no internet to
  `apk add` one — so the hello-world server is `python3 -m http.server`
  (python3 is already present because cloud-init depends on it).
- **`start-stop-daemon --exec /usr/bin/python3` matches cloud-init's own
  python3** at boot and refuses to start anything.
- **busybox `pgrep -f PATTERN` matches its own argv**, so a "skip if already
  running" guard always fires. Verified: `guard_exit=0` with nothing listening.

## Not modelled (yet)

VLANs are separate L2 segments rather than one 802.1Q trunk, so this exercises
inter-VLAN routing but not a `bond0.<vif>` trunk config specifically. A router
VM would attach one NIC per VLAN. Adding a tagged-trunk variant is the obvious
next step if the bond/vif config itself needs testing.