c91e44f796
Some checks failed
CI/CD / lint (pull_request) Failing after 11s
CI/CD / typecheck (pull_request) Failing after 10s
CI/CD / test (pull_request) Failing after 9s
CI/CD / build (pull_request) Has been skipped
CI/CD / publish-rpm (pull_request) Has been skipped
CI/CD / publish-deb (pull_request) Has been skipped
A throwaway copy of the production VLAN topology so routing and firewall changes can be tested before they touch the real network. Same VLAN IDs and roles as UniFi, deliberately different ranges (172.31.<vlan>.0/24) so nothing here can be mistaken for production. - OVS fabric: real 802.1Q. Access port per micro VM, host leg per VLAN (.2, for SSH only — NOT the VMs' default route, so inter-VLAN tests exercise the router rather than the host's routing table), and a trunk portgroup with VLAN 1 declared nativeMode='untagged'. - Six Alpine micro VMs (256MB, copy-on-write overlays on one 176MB image), SSH + a hello-world HTTP page naming the VLAN. - VyOS router installed to disk unattended over the console, with the SAME config shape as the VP2440s: two NICs in an LACP bond carrying the trunk, VLAN 1 native, bond0.<vlan> holding the .1 gateway on each. - labsim-matrix.py: full-mesh ICMP/TCP22/TCP80 probe, ~0.2s, --watch highlights cells that changed since the last sweep. Guest-side probe is python3 (already present via cloud-init) so nothing is installed on VMs that have no internet. - Prometheus + Grafana (anonymous auth, no login) with a provisioned dashboard: heatmap plus a state timeline showing exactly when a path flipped. Verified end to end: one VyOS rule took sum(labsim_reachable) from 90 to 84, blocking precisely kvm<->k8s across all three protocols. Traps found building this, all now encoded in the scripts: - virtio-net breaks 802.3ad: the guest's bonding driver reports slaves "MII Status: down" despite carrier=1 and never sends an LACPDU, so the bond sits in AD_STATE_DEFAULTED. e1000e fixes it with no other change. Matches the netdev thread "bonding (IEEE 802.3ad) not working with qemu/virtio". - OVS defaults bonds to active-backup, which does not speak LACP at all — bond_mode=balance-tcp is required. - LACP deadlock: OVS holds members disabled until negotiation while the partner needs carrier before it will send LACPDUs. lacp-fallback-ab breaks it. - LACPDUs are untagged, so a trunk with no native VLAN has nowhere to put them. - --boot cdrom,hd re-runs the ISO on every restart, so every commit+save went to a live system that evaporated. Install now switches the VM to boot hd. - cloud-init on Alpine: users stay locked without lock_passwd:false, one failing runcmd aborts the rest, busybox here has no httpd applet, and start-stop-daemon --exec /usr/bin/python3 matches cloud-init's own python3. - The user-data heredoc is unquoted, so backticks in a COMMENT were executed by the host shell and their output corrupted the YAML. build_seed now validates with yaml.safe_load before building the ISO. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
89 lines
3.6 KiB
Markdown
89 lines
3.6 KiB
Markdown
# labsim — libvirt replica of the lab network
|
|
|
|
A throwaway copy of the production VLAN topology for testing routing, firewall
|
|
rules and failover **without touching the real network**. Same VLAN IDs and
|
|
roles as UniFi, deliberately different IP ranges so nothing can be confused for
|
|
production.
|
|
|
|
## Topology
|
|
|
|
Each VLAN is its own isolated libvirt network with one tiny Alpine VM on it.
|
|
|
|
| VLAN | Name | Sim subnet | VM address | Mirrors production |
|
|
|-----:|------|------------|-----------|--------------------|
|
|
| 1 | management | 172.31.1.0/24 | 172.31.1.10 | 192.168.1.0/24 |
|
|
| 2 | k8s | 172.31.2.0/24 | 172.31.2.10 | 192.168.8.0/23 |
|
|
| 3 | kvm | 172.31.3.0/24 | 172.31.3.10 | 192.168.3.0/24 |
|
|
| 9 | private | 172.31.9.0/24 | 172.31.9.10 | 10.0.9.0/23 |
|
|
| 10 | lot | 172.31.10.0/24 | 172.31.10.10 | 10.0.0.0/23 |
|
|
| 200 | roomates | 172.31.200.0/24 | 172.31.200.10 | 192.168.2.0/24 |
|
|
|
|
The sim subnet always encodes the VLAN id: `172.31.<vlan>.0/24`.
|
|
|
|
Address plan, identical on every VLAN:
|
|
|
|
| Address | Role |
|
|
|---------|------|
|
|
| `.1` | gateway under test — a router VM you add (not created by default) |
|
|
| `.2` | host bridge — how you reach the VMs from this workstation |
|
|
| `.10` | the VLAN's micro VM |
|
|
| `.254` | reserved for a VRRP VIP, mirroring production |
|
|
|
|
The host sits at `.2` purely so you can SSH in. It is deliberately **not** the
|
|
VMs' default route — that is `.1` — so inter-VLAN tests fail loudly when no
|
|
router is present instead of being silently served by the host's own routing
|
|
table. libvirt also installs reject rules that stop these networks forwarding
|
|
to each other, so traffic between VLANs only works once a router VM bridges
|
|
them.
|
|
|
|
## Usage
|
|
|
|
```bash
|
|
./labsim-up.sh # bring up every VLAN (idempotent)
|
|
./labsim-up.sh 2 3 # only VLANs 2 and 3
|
|
./labsim-down.sh # destroy VMs + networks, keep the base image
|
|
./labsim-down.sh --purge # also delete the downloaded Alpine image
|
|
```
|
|
|
|
Each VM: 256 MB, 1 vCPU, a copy-on-write overlay on one shared 176 MB Alpine
|
|
image (so six VMs cost a few MB of disk, not 1 GB).
|
|
|
|
## Access
|
|
|
|
```bash
|
|
ssh alpine@172.31.2.10 # normal user (password: labsim)
|
|
ssh root@172.31.2.10 # privileged — this image has no sudo
|
|
curl http://172.31.2.10/ # hello-world page naming the VLAN
|
|
```
|
|
|
|
Console, when the network is the thing that is broken:
|
|
|
|
```bash
|
|
sudo virsh console labsim-2-k8s # root / labsim
|
|
```
|
|
|
|
## Notes for whoever extends this
|
|
|
|
Things that cost time the first time round, all verified on this image:
|
|
|
|
- **No `sudo`.** Alpine ships `doas`; cloud-init's `sudo:` directive is inert
|
|
here. Use `root@` for privileged work.
|
|
- **cloud-init leaves users locked** (`!*` in `/etc/shadow`) unless
|
|
`lock_passwd: false`, and sshd then refuses key auth for that user.
|
|
- **One failing `runcmd` aborts every command after it.** Each entry is
|
|
`|| true` for that reason.
|
|
- **busybox here has no `httpd` applet**, and the VMs have no internet to
|
|
`apk add` one — so the hello-world server is `python3 -m http.server`
|
|
(python3 is already present because cloud-init depends on it).
|
|
- **`start-stop-daemon --exec /usr/bin/python3` matches cloud-init's own
|
|
python3** at boot and refuses to start anything.
|
|
- **busybox `pgrep -f PATTERN` matches its own argv**, so a "skip if already
|
|
running" guard always fires. Verified: `guard_exit=0` with nothing listening.
|
|
|
|
## Not modelled (yet)
|
|
|
|
VLANs are separate L2 segments rather than one 802.1Q trunk, so this exercises
|
|
inter-VLAN routing but not a `bond0.<vif>` trunk config specifically. A router
|
|
VM would attach one NIC per VLAN. Adding a tagged-trunk variant is the obvious
|
|
next step if the bond/vif config itself needs testing.
|