The Grafana heatmap of 1s and 0s said almost nothing, and the state timeline
was an unreadable pile of overlapping series labels. Replaced as the primary
view with a purpose-built page served by the exporter itself.
- Probe now captures ICMP RTT, exposed as labsim_rtt_ms{src,dst}. A path that
is up but slow is a different problem from one that is down, and a pass/fail
grid cannot show it.
- Exporter serves / (topology), /api/matrix (JSON) and /metrics.
- topology.html: node per VLAN in a ring, VyOS router in the centre because
every inter-VLAN packet really does traverse it, one line per pair coloured
green/red with the RTT on it. Hovering gives per-direction state. A node ring
goes red if anything to or from it is blocked. Side panels list blocked paths
and the slowest links. Refreshes every 5s, no dependencies.
Grafana stays for what it is actually good at — history of when a path flipped.
Label placement is deliberate: RTT captions sit ~32% along each edge with a
perpendicular nudge, because every diagonal of a 6-node mesh crosses the centre
and midpoint labels stack on the router node.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
A throwaway copy of the production VLAN topology so routing and firewall
changes can be tested before they touch the real network. Same VLAN IDs and
roles as UniFi, deliberately different ranges (172.31.<vlan>.0/24) so nothing
here can be mistaken for production.
- OVS fabric: real 802.1Q. Access port per micro VM, host leg per VLAN (.2,
for SSH only — NOT the VMs' default route, so inter-VLAN tests exercise the
router rather than the host's routing table), and a trunk portgroup with
VLAN 1 declared nativeMode='untagged'.
- Six Alpine micro VMs (256MB, copy-on-write overlays on one 176MB image),
SSH + a hello-world HTTP page naming the VLAN.
- VyOS router installed to disk unattended over the console, with the SAME
config shape as the VP2440s: two NICs in an LACP bond carrying the trunk,
VLAN 1 native, bond0.<vlan> holding the .1 gateway on each.
- labsim-matrix.py: full-mesh ICMP/TCP22/TCP80 probe, ~0.2s, --watch
highlights cells that changed since the last sweep. Guest-side probe is
python3 (already present via cloud-init) so nothing is installed on VMs
that have no internet.
- Prometheus + Grafana (anonymous auth, no login) with a provisioned
dashboard: heatmap plus a state timeline showing exactly when a path
flipped. Verified end to end: one VyOS rule took sum(labsim_reachable)
from 90 to 84, blocking precisely kvm<->k8s across all three protocols.
Traps found building this, all now encoded in the scripts:
- virtio-net breaks 802.3ad: the guest's bonding driver reports slaves
"MII Status: down" despite carrier=1 and never sends an LACPDU, so the bond
sits in AD_STATE_DEFAULTED. e1000e fixes it with no other change. Matches
the netdev thread "bonding (IEEE 802.3ad) not working with qemu/virtio".
- OVS defaults bonds to active-backup, which does not speak LACP at all —
bond_mode=balance-tcp is required.
- LACP deadlock: OVS holds members disabled until negotiation while the
partner needs carrier before it will send LACPDUs. lacp-fallback-ab breaks it.
- LACPDUs are untagged, so a trunk with no native VLAN has nowhere to put them.
- --boot cdrom,hd re-runs the ISO on every restart, so every commit+save went
to a live system that evaporated. Install now switches the VM to boot hd.
- cloud-init on Alpine: users stay locked without lock_passwd:false, one
failing runcmd aborts the rest, busybox here has no httpd applet, and
start-stop-daemon --exec /usr/bin/python3 matches cloud-init's own python3.
- The user-data heredoc is unquoted, so backticks in a COMMENT were executed
by the host shell and their output corrupted the YAML. build_seed now
validates with yaml.safe_load before building the ISO.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
