labsim: prove the tagged-Management fix for kea's wrong-pool offers

Kea #1117: with dhcp-socket-type raw, a frame tagged for a sub-interface is
also delivered to the parent's AF_PACKET socket, and if the parent serves a
subnet kea answers from it too. Management being the native VLAN on bond0 is
what gives the parent that subnet. One DISCOVER on VLAN 3 produced two OFFERs,
and in the captures here the WRONG one arrives first as often as not -- which
is why this looked device-dependent rather than like a server bug.

labsim-vlan-leak-test.sh reproduces it and scores the SERVER's offers, not the
client's choice; a client picking correctly is how this hid. Fails on the old
shape, passes on the new one across all six LAN VLANs.

Three things the rehearsal caught that reasoning had not:

  - kea keeps its old raw socket. VyOS does not restart it for an interface
    address change, so the first post-fix test failed and looked exactly like
    the fix not working.
  - interface-group LAN names the bare bond0. Moving the address without
    moving the group drops every management session under default-deny.
  - there is no make-before-break. A port always egresses its native VLAN
    untagged, so while VLAN 1 is native the router can send tagged VLAN 1 but
    never receive it -- verified, the ARP landed on bond0 untagged.

What makes the cutover safe anyway is that tagged and untagged Management
coexist, so the firewalls convert one at a time: 0s of VIP downtime, versus
5m30s if both routers go before the switch does. In that state the healthy
BACKUP does NOT take over -- the sync group holds native BACKUP because the
other VLANs still hear the master.

Also fixes two ways the sim was lying. ovs_bond_router compared only the trunk
VLAN list on re-runs, so a VM restart left the bond holding taps that no longer
existed while the real ones sat in the bridge unbonded -- labsim-vyos2 had no
LACP at all. And the tap count included the primary's libvirt-NAT scaffold NIC,
so the primary's bond was skipped outright.

Runbook: migration/MANAGEMENT-VLAN-TAGGED.md

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH

labsim/README.md

@@ -186,6 +186,63 @@ why any of it was shaped the way it was.
   client does not redial promptly. After any change there, check `pppoe0` on
   the router and `sudo systemctl restart ppp@pppoe0` if it is missing.
 
+## The trunk carries every VLAN tagged, including Management
+
+There is deliberately **no native/untagged VLAN** on the trunks to the routers,
+and Management lives on `bond0.1`, not on the bare `bond0`.
+
+A native VLAN is what puts a subnet on the bond **parent** while every other
+VLAN sits on a sub-interface of it. With `dhcp-socket-type: raw`, kea receives
+each tagged frame *twice* — once on the sub-interface and once on the parent —
+and answers from the parent's pool as well (ISC Kea
+[#1117](https://gitlab.isc.org/isc-projects/kea/-/issues/1117)). A client on
+VLAN 3 gets two OFFERs and keeps whichever arrives first:
+
+```
+bond0.3 :  172.31.3.252 → 172.31.3.11   correct
+bond0   :  172.31.1.252 → 172.31.1.8    UNTAGGED, Management pool, wrong
+```
+
+`./labsim-vlan-leak-test.sh` makes one client on a tagged VLAN send a DISCOVER
+and captures on the parent and the sub-interface at once. The verdict is how
+many OFFERs the **server** emitted and from which subnets — deliberately not
+"did the client get the right address", because a client picking correctly is
+exactly how this hid. Both orderings were observed across runs, so a passing
+client proves nothing.
+
+```sh
+./labsim-vlan-leak-test.sh --vlan 3           # PASS on the current shape
+LABSIM_NATIVE_VLAN=1 ./router-up.sh           # restore the old shape...
+./labsim-vlan-leak-test.sh --vlan 3           # ...and it FAILs again
+```
+
+Three things this cost, all of which apply to production:
+
+- **Kea must be restarted after the address moves.** VyOS does not restart it
+  for an interface address change, so it keeps a raw socket bound with the old
+  address and the bug survives the fix. In the sim kea had been running since
+  16 Aug; the first post-fix test failed for this reason alone and looked like
+  the fix simply not working.
+- **The firewall interface-group must move too.** `interface-group LAN` named
+  the bare `bond0`; with a default-deny ruleset, moving the address without
+  moving the group drops every management session and all VLAN 1 routing.
+- **Duplicate delivery does not stop.** #1117 says only that there is no longer
+  a subnet on the parent to match, and that is exactly what happens: two replies
+  per DISCOVER, both now from the correct pool. Harmless, but do not read a
+  duplicate as a failure.
+
+### Tagged and untagged Management coexist
+
+Verified directly, and it is what makes the production cutover a rolling change
+rather than an outage: with the primary still untagged on `bond0` and the
+secondary already tagged on `bond0.1`, both routers were reachable, the VIP
+stayed up and a VLAN 1 client kept its gateway. One VLAN is one broadcast
+domain regardless of how each port tags it, so the two firewalls can be
+converted one at a time. See `migration/MANAGEMENT-VLAN-TAGGED.md`.
+
+`./vlan1-move-monitor.sh` logs VIP/router liveness once a second during the
+change, because VRRP reconverges and leaves no trace of who held the VIP.
+
 ## Notes for whoever extends this
 
 Things that cost time the first time round, all verified on this image:
@@ -206,7 +263,14 @@ Things that cost time the first time round, all verified on this image:
 
 ## Not modelled (yet)
 
-VLANs are separate L2 segments rather than one 802.1Q trunk, so this exercises
-inter-VLAN routing but not a `bond0.<vif>` trunk config specifically. A router
-VM would attach one NIC per VLAN. Adding a tagged-trunk variant is the obvious
-next step if the bond/vif config itself needs testing.
+- **The secondary's bond was fiction until 2026-09-02.** `ovs_bond_router`'s
+  "already bonded, nothing to do" check compared only the trunk VLAN list, not
+  the membership. Restarting a VM recreates its taps under new names, so the
+  bond sat there holding two interfaces that no longer existed while the router's
+  real taps ran in the bridge as two *independent* ports — no LACP, and carrying
+  libvirt's own portgroup VLAN config rather than the bond's. It reconciles
+  membership now, but the lesson generalises: a sim that reports success is not
+  the same as a sim that models the thing.
+- **`labsim-vyos` has a third NIC** on libvirt's `default` network (the scaffold
+  uplink, see `--drop-scaffold`). The tap count is filtered to `$OVS_NET` for
+  that reason; an unfiltered count is 3 and silently skipped the primary's bond.