a5b36678ed
Some checks failed
CI/CD / lint (pull_request) Failing after 9s
CI/CD / test (pull_request) Failing after 9s
CI/CD / typecheck (pull_request) Failing after 24s
CI/CD / build (pull_request) Has been skipped
CI/CD / publish-rpm (pull_request) Has been skipped
CI/CD / publish-deb (pull_request) Has been skipped
The Grafana heatmap of 1s and 0s said almost nothing, and the state timeline
was an unreadable pile of overlapping series labels. Replaced as the primary
view with a purpose-built page served by the exporter itself.
- Probe now captures ICMP RTT, exposed as labsim_rtt_ms{src,dst}. A path that
is up but slow is a different problem from one that is down, and a pass/fail
grid cannot show it.
- Exporter serves / (topology), /api/matrix (JSON) and /metrics.
- topology.html: node per VLAN in a ring, VyOS router in the centre because
every inter-VLAN packet really does traverse it, one line per pair coloured
green/red with the RTT on it. Hovering gives per-direction state. A node ring
goes red if anything to or from it is blocked. Side panels list blocked paths
and the slowest links. Refreshes every 5s, no dependencies.
Grafana stays for what it is actually good at — history of when a path flipped.
Label placement is deliberate: RTT captions sit ~32% along each edge with a
perpendicular nudge, because every diagonal of a 6-node mesh crosses the centre
and midpoint labels stack on the router node.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
105 lines
4.3 KiB
Markdown
105 lines
4.3 KiB
Markdown
# labsim — libvirt replica of the lab network
|
|
|
|
A throwaway copy of the production VLAN topology for testing routing, firewall
|
|
rules and failover **without touching the real network**. Same VLAN IDs and
|
|
roles as UniFi, deliberately different IP ranges so nothing can be confused for
|
|
production.
|
|
|
|
## Topology
|
|
|
|
Each VLAN is its own isolated libvirt network with one tiny Alpine VM on it.
|
|
|
|
| VLAN | Name | Sim subnet | VM address | Mirrors production |
|
|
|-----:|------|------------|-----------|--------------------|
|
|
| 1 | management | 172.31.1.0/24 | 172.31.1.10 | 192.168.1.0/24 |
|
|
| 2 | k8s | 172.31.2.0/24 | 172.31.2.10 | 192.168.8.0/23 |
|
|
| 3 | kvm | 172.31.3.0/24 | 172.31.3.10 | 192.168.3.0/24 |
|
|
| 9 | private | 172.31.9.0/24 | 172.31.9.10 | 10.0.9.0/23 |
|
|
| 10 | lot | 172.31.10.0/24 | 172.31.10.10 | 10.0.0.0/23 |
|
|
| 200 | roomates | 172.31.200.0/24 | 172.31.200.10 | 192.168.2.0/24 |
|
|
|
|
The sim subnet always encodes the VLAN id: `172.31.<vlan>.0/24`.
|
|
|
|
Address plan, identical on every VLAN:
|
|
|
|
| Address | Role |
|
|
|---------|------|
|
|
| `.1` | gateway under test — a router VM you add (not created by default) |
|
|
| `.2` | host bridge — how you reach the VMs from this workstation |
|
|
| `.10` | the VLAN's micro VM |
|
|
| `.254` | reserved for a VRRP VIP, mirroring production |
|
|
|
|
The host sits at `.2` purely so you can SSH in. It is deliberately **not** the
|
|
VMs' default route — that is `.1` — so inter-VLAN tests fail loudly when no
|
|
router is present instead of being silently served by the host's own routing
|
|
table. libvirt also installs reject rules that stop these networks forwarding
|
|
to each other, so traffic between VLANs only works once a router VM bridges
|
|
them.
|
|
|
|
## Usage
|
|
|
|
```bash
|
|
./labsim-up.sh # bring up every VLAN (idempotent)
|
|
./labsim-up.sh 2 3 # only VLANs 2 and 3
|
|
./labsim-down.sh # destroy VMs + networks, keep the base image
|
|
./labsim-down.sh --purge # also delete the downloaded Alpine image
|
|
```
|
|
|
|
Each VM: 256 MB, 1 vCPU, a copy-on-write overlay on one shared 176 MB Alpine
|
|
image (so six VMs cost a few MB of disk, not 1 GB).
|
|
|
|
## Access
|
|
|
|
```bash
|
|
ssh alpine@172.31.2.10 # normal user (password: labsim)
|
|
ssh root@172.31.2.10 # privileged — this image has no sudo
|
|
curl http://172.31.2.10/ # hello-world page naming the VLAN
|
|
```
|
|
|
|
Console, when the network is the thing that is broken:
|
|
|
|
```bash
|
|
sudo virsh console labsim-2-k8s # root / labsim
|
|
```
|
|
|
|
## Watching it
|
|
|
|
```bash
|
|
./labsim-matrix.py --watch 2 # terminal grid, changed cells highlighted
|
|
./monitoring-up.sh # topology page + Prometheus + Grafana
|
|
```
|
|
|
|
- **http://localhost:9101/** — live mesh: a node per VLAN, the router in the
|
|
middle, one line per pair coloured green/red with the ICMP RTT on it. Hover a
|
|
line for per-direction detail. Refreshes every 5s. This is the one to watch
|
|
while changing firewall rules.
|
|
- **http://localhost:3000/d/labsim-matrix** — Grafana (anonymous, no login) for
|
|
*history*: when did a path flip, and how has latency moved.
|
|
- **http://localhost:9101/metrics** — `labsim_reachable{src,dst,proto}` and
|
|
`labsim_rtt_ms{src,dst}`.
|
|
|
|
## Notes for whoever extends this
|
|
|
|
Things that cost time the first time round, all verified on this image:
|
|
|
|
- **No `sudo`.** Alpine ships `doas`; cloud-init's `sudo:` directive is inert
|
|
here. Use `root@` for privileged work.
|
|
- **cloud-init leaves users locked** (`!*` in `/etc/shadow`) unless
|
|
`lock_passwd: false`, and sshd then refuses key auth for that user.
|
|
- **One failing `runcmd` aborts every command after it.** Each entry is
|
|
`|| true` for that reason.
|
|
- **busybox here has no `httpd` applet**, and the VMs have no internet to
|
|
`apk add` one — so the hello-world server is `python3 -m http.server`
|
|
(python3 is already present because cloud-init depends on it).
|
|
- **`start-stop-daemon --exec /usr/bin/python3` matches cloud-init's own
|
|
python3** at boot and refuses to start anything.
|
|
- **busybox `pgrep -f PATTERN` matches its own argv**, so a "skip if already
|
|
running" guard always fires. Verified: `guard_exit=0` with nothing listening.
|
|
|
|
## Not modelled (yet)
|
|
|
|
VLANs are separate L2 segments rather than one 802.1Q trunk, so this exercises
|
|
inter-VLAN routing but not a `bond0.<vif>` trunk config specifically. A router
|
|
VM would attach one NIC per VLAN. Adding a tagged-trunk variant is the obvious
|
|
next step if the bond/vif config itself needs testing.
|