6c4318d3ae
kea does not inherit UniFi's lease database. At cutover it starts with an empty
view of who holds what, so it can hand an address that is currently in use to a
different device. Reservations are what carry "this device has this address"
across the switch, because they live in config rather than in lease state.
unifi-reserve-all.py creates one per active client, dry run by default. 51
written, 51/51 verified live by reading the records back; the controller now
holds 85 reservations and the generator emits all 85 with unique, valid
hostnames and no duplicate addresses. Active clients with no reservation went
from 48 to 4.
Three guards, each of which caught something real in the dry run:
- VRRP virtual addresses are excluded. UniFi reports them as ordinary client
addresses because the firewalls' bond MACs answer for them, and their
apparent IP flips between the real interface address and the VIP. Without
this, 192.168.1.254 -- the gateway VIP itself -- would have been given a
DHCP reservation.
- The firewalls' own interface MACs are excluded; those are statically
configured routers, not DHCP clients.
- Any address claimed by more than one MAC is dropped rather than guessed
at. This is how the VIPs surfaced in the first place.
Also skipped: addresses already reserved to another MAC, network gateways, and
anything on a network that does not serve DHCP (which excludes the WAN transit
VLANs automatically).
labsim-dhcp-test.sh gained a lease-database flush, and it is not tidiness. Two
findings, both of which first appeared as a PASSING test:
- Re-running against stale leases, kea gave dynamic addresses to three
devices that have reservations. The reservations were present and correct
in kea's own config throughout. Kea saw the reserved address as leased to
"another client" -- same MAC, different client-id from the earlier boot --
and allocated elsewhere. Cutover starts with an empty lease database so
this is a testing artifact, but a reservation is evidently not
unconditional once leases exist.
- Removing only dhcp4-leases.csv does nothing: kea's memfile backend keeps
lease-file-cleanup rotations (.csv.2) and restores from them on start.
The verdict logic no longer takes the first matching lease row. Doing so
reported an hours-old lease as the current answer and scored three failures as
passes, including one where the device had plainly been given a dynamic
address. A MAC with more than one lease is now an explicit failure rather than
a guess.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
170 lines
7.3 KiB
Markdown
170 lines
7.3 KiB
Markdown
# labsim — libvirt replica of the lab network
|
||
|
||
A throwaway copy of the production VLAN topology for testing routing, firewall
|
||
rules and failover **without touching the real network**. Same VLAN IDs and
|
||
roles as UniFi, deliberately different IP ranges so nothing can be confused for
|
||
production.
|
||
|
||
## Topology
|
||
|
||
Each VLAN is its own isolated libvirt network with one tiny Alpine VM on it.
|
||
|
||
| VLAN | Name | Sim subnet | VM address | Mirrors production |
|
||
|-----:|------|------------|-----------|--------------------|
|
||
| 1 | management | 172.31.1.0/24 | 172.31.1.10 | 192.168.1.0/24 |
|
||
| 2 | k8s | 172.31.2.0/24 | 172.31.2.10 | 192.168.8.0/23 |
|
||
| 3 | kvm | 172.31.3.0/24 | 172.31.3.10 | 192.168.3.0/24 |
|
||
| 9 | private | 172.31.9.0/24 | 172.31.9.10 | 10.0.9.0/23 |
|
||
| 10 | lot | **172.31.10.0/23** | 172.31.10.10 | 10.0.0.0/23 |
|
||
| 200 | roomates | 172.31.200.0/24 | 172.31.200.10 | 192.168.2.0/24 |
|
||
|
||
The sim subnet encodes the VLAN id: `172.31.<vlan>.0/24`, with one exception.
|
||
**VLAN 10 is a `/23`** because every UniFi DHCP reservation lives in LoT and LoT
|
||
spans `10.0.0.x` *and* `10.0.1.x`, which a `/24` cannot hold. The mapping stays
|
||
readable — `10.0.0.46 → 172.31.10.46`, `10.0.1.67 → 172.31.11.67`.
|
||
|
||
LoT's host leg is `.3`, not `.2`, because `10.0.0.2` is a real reservation
|
||
(Hubitat) that maps onto `172.31.10.2`. `.3` is unreserved and sits below the
|
||
DHCP pool, so it can never be handed out.
|
||
|
||
`vlans.conf` therefore takes two optional trailing fields:
|
||
|
||
```
|
||
vlan_id:name:sim_prefix:real_subnet[:masklen][:host_octet]
|
||
```
|
||
|
||
defaulting to `24` and `2`. k8s and Private are also `/23` in production but
|
||
hold no reservations, so they keep their `/24` and their DHCP range is clamped
|
||
— reported at generation time, never silently.
|
||
|
||
Address plan, identical on every VLAN:
|
||
|
||
| Address | Role |
|
||
|---------|------|
|
||
| `.1` | gateway under test — a router VM you add (not created by default) |
|
||
| `.2` | host bridge — how you reach the VMs from this workstation |
|
||
| `.10` | the VLAN's micro VM |
|
||
| `.254` | reserved for a VRRP VIP, mirroring production |
|
||
|
||
The host sits at `.2` purely so you can SSH in. It is deliberately **not** the
|
||
VMs' default route — that is `.1` — so inter-VLAN tests fail loudly when no
|
||
router is present instead of being silently served by the host's own routing
|
||
table. libvirt also installs reject rules that stop these networks forwarding
|
||
to each other, so traffic between VLANs only works once a router VM bridges
|
||
them.
|
||
|
||
## Usage
|
||
|
||
```bash
|
||
./labsim-up.sh # bring up every VLAN (idempotent)
|
||
./labsim-up.sh 2 3 # only VLANs 2 and 3
|
||
./labsim-down.sh # destroy VMs + networks, keep the base image
|
||
./labsim-down.sh --purge # also delete the downloaded Alpine image
|
||
```
|
||
|
||
Each VM: 256 MB, 1 vCPU, a copy-on-write overlay on one shared 176 MB Alpine
|
||
image (so six VMs cost a few MB of disk, not 1 GB).
|
||
|
||
## Access
|
||
|
||
```bash
|
||
ssh alpine@172.31.2.10 # normal user (password: labsim)
|
||
ssh root@172.31.2.10 # privileged — this image has no sudo
|
||
curl http://172.31.2.10/ # hello-world page naming the VLAN
|
||
```
|
||
|
||
Console, when the network is the thing that is broken:
|
||
|
||
```bash
|
||
sudo virsh console labsim-2-k8s # root / labsim
|
||
```
|
||
|
||
## Watching it
|
||
|
||
```bash
|
||
./labsim-matrix.py --watch 2 # terminal grid, changed cells highlighted
|
||
./monitoring-up.sh # topology page + Prometheus + Grafana
|
||
```
|
||
|
||
## Testing the DHCP migration
|
||
|
||
`./labsim-dhcp-test.sh` boots throwaway VMs whose MACs are **real production
|
||
MACs** and checks each gets the address UniFi reserved for it. MACs are the one
|
||
piece of production config that transplants verbatim, which is what makes this a
|
||
test rather than a rehearsal. It is safe because `ovs-labsim` has no physical
|
||
NIC — verified with `ovs-vsctl show` — so a production MAC cannot reach the real
|
||
LAN.
|
||
|
||
Apply the config first, from `../migration`:
|
||
|
||
```bash
|
||
python3 unifi-to-vyos.py --mode sim -o /tmp/sim.conf # 6 subnets, 31 mappings
|
||
# load onto labsim-vyos, then:
|
||
./labsim-dhcp-test.sh
|
||
```
|
||
|
||
**Result on VyOS 2026.08 (kea): all four cases pass.** The one that mattered:
|
||
most UniFi reservations sit *inside* the DHCP pool, and **kea honours in-pool
|
||
host reservations** — `printer1` received `172.31.10.46` from within the
|
||
`.10.11–.11.254` pool. That was the open question blocking the cutover.
|
||
|
||
### The lease database will lie to you
|
||
|
||
The script wipes `/config/dhcp/dhcp4-leases.csv*` before every run, and both
|
||
halves of that matter:
|
||
|
||
- **Stale leases defeat reservations.** Re-running against yesterday's leases,
|
||
kea handed dynamic addresses to three devices that have reservations. The
|
||
reservation was present and correct in `/run/kea/kea-dhcp4.conf` the whole
|
||
time. Kea saw the reserved address as already leased to "another client" —
|
||
same MAC, but a different client-id from the earlier boot — and allocated
|
||
elsewhere. The cutover itself starts with an empty lease database, so this is
|
||
a *testing* artifact, but it is worth knowing that a reservation is not an
|
||
unconditional guarantee once leases exist.
|
||
- **The `*` is load-bearing.** Kea's memfile backend keeps lease-file-cleanup
|
||
rotations (`dhcp4-leases.csv.2`) and restores from them on start, so
|
||
truncating only the primary file changes nothing.
|
||
|
||
Both of those first appeared as a *passing* test. The verdict logic now refuses
|
||
to score a MAC with more than one lease, because taking the first match had
|
||
reported an hours-old lease as the current answer and turned three failures
|
||
into apparent passes.
|
||
|
||
Still open: whether kea will hand a *reserved* address to a *different* client
|
||
while the reserved device is offline. The negative case here only proves an
|
||
unreserved MAC gets an unreserved address.
|
||
|
||
- **http://localhost:9101/** — live mesh: a node per VLAN, the router in the
|
||
middle, one line per pair coloured green/red with the ICMP RTT on it. Hover a
|
||
line for per-direction detail. Refreshes every 5s. This is the one to watch
|
||
while changing firewall rules.
|
||
- **http://localhost:3000/d/labsim-matrix** — Grafana (anonymous, no login) for
|
||
*history*: when did a path flip, and how has latency moved.
|
||
- **http://localhost:9101/metrics** — `labsim_reachable{src,dst,proto}` and
|
||
`labsim_rtt_ms{src,dst}`.
|
||
|
||
## Notes for whoever extends this
|
||
|
||
Things that cost time the first time round, all verified on this image:
|
||
|
||
- **No `sudo`.** Alpine ships `doas`; cloud-init's `sudo:` directive is inert
|
||
here. Use `root@` for privileged work.
|
||
- **cloud-init leaves users locked** (`!*` in `/etc/shadow`) unless
|
||
`lock_passwd: false`, and sshd then refuses key auth for that user.
|
||
- **One failing `runcmd` aborts every command after it.** Each entry is
|
||
`|| true` for that reason.
|
||
- **busybox here has no `httpd` applet**, and the VMs have no internet to
|
||
`apk add` one — so the hello-world server is `python3 -m http.server`
|
||
(python3 is already present because cloud-init depends on it).
|
||
- **`start-stop-daemon --exec /usr/bin/python3` matches cloud-init's own
|
||
python3** at boot and refuses to start anything.
|
||
- **busybox `pgrep -f PATTERN` matches its own argv**, so a "skip if already
|
||
running" guard always fires. Verified: `guard_exit=0` with nothing listening.
|
||
|
||
## Not modelled (yet)
|
||
|
||
VLANs are separate L2 segments rather than one 802.1Q trunk, so this exercises
|
||
inter-VLAN routing but not a `bond0.<vif>` trunk config specifically. A router
|
||
VM would attach one NIC per VLAN. Adding a tagged-trunk variant is the obvious
|
||
next step if the bond/vif config itself needs testing.
|