michal/lab
1
0
Fork 0
Code Issues Pull Requests 3 Actions Packages Projects Releases Wiki Activity

labsim: prove the tagged-Management fix for kea's wrong-pool offers

Browse Source 
Kea #1117: with dhcp-socket-type raw, a frame tagged for a sub-interface is
also delivered to the parent's AF_PACKET socket, and if the parent serves a
subnet kea answers from it too. Management being the native VLAN on bond0 is
what gives the parent that subnet. One DISCOVER on VLAN 3 produced two OFFERs,
and in the captures here the WRONG one arrives first as often as not -- which
is why this looked device-dependent rather than like a server bug.

labsim-vlan-leak-test.sh reproduces it and scores the SERVER's offers, not the
client's choice; a client picking correctly is how this hid. Fails on the old
shape, passes on the new one across all six LAN VLANs.

Three things the rehearsal caught that reasoning had not:

  - kea keeps its old raw socket. VyOS does not restart it for an interface
    address change, so the first post-fix test failed and looked exactly like
    the fix not working.
  - interface-group LAN names the bare bond0. Moving the address without
    moving the group drops every management session under default-deny.
  - there is no make-before-break. A port always egresses its native VLAN
    untagged, so while VLAN 1 is native the router can send tagged VLAN 1 but
    never receive it -- verified, the ARP landed on bond0 untagged.

What makes the cutover safe anyway is that tagged and untagged Management
coexist, so the firewalls convert one at a time: 0s of VIP downtime, versus
5m30s if both routers go before the switch does. In that state the healthy
BACKUP does NOT take over -- the sync group holds native BACKUP because the
other VLANs still hear the master.

Also fixes two ways the sim was lying. ovs_bond_router compared only the trunk
VLAN list on re-runs, so a VM restart left the bond holding taps that no longer
existed while the real ones sat in the bridge unbonded -- labsim-vyos2 had no
LACP at all. And the tap count included the primary's libvirt-NAT scaffold NIC,
so the primary's bond was skipped outright.

Runbook: migration/MANAGEMENT-VLAN-TAGGED.md

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
This commit is contained in:
Michal
2026-09-02 14:03:44 +01:00
parent 23783b8486
commit 0481c38e09
19 changed files with 737 additions and 40 deletions
Download Patch File Download Diff File Expand all files Collapse all files

137
migration/MANAGEMENT-VLAN-TAGGED.md Normal file
View File

@@ -0,0 +1,137 @@
# Moving Management onto a tagged VLAN
Rehearsed end to end in labsim on 2026-09-02. This is the fix for kea serving
addresses from the wrong VLAN's pool.
## Why
ISC Kea [#1117](https://gitlab.isc.org/isc-projects/kea/-/issues/1117): with
`dhcp-socket-type: raw`, a frame tagged for a sub-interface is **also** delivered
to the parent's `AF_PACKET` socket. If the parent serves a subnet, kea answers
from it too. Ours does — Management is the native/untagged VLAN on `bond0` while
VLANs 2/3/9/10/200 are sub-interfaces of that same bond — so one DISCOVER on
VLAN 3 produces two OFFERs and the *client* decides which to keep:
```
bond0.3 : 192.168.3.14 correct
bond0 : 192.168.1.28 UNTAGGED, Management pool, wrong
```
The fix is to leave **no subnet on the parent**: every VLAN tagged, Management
included, moved from `bond0` to `bond0.1`.
Confirmed in labsim across all six LAN VLANs: fails before, passes after.
`labsim/labsim-vlan-leak-test.sh` is the test; evidence in
`labsim/vlan-leak-evidence/`.
## What must change together
Per router:
| | from | to |
|---|---|---|
| address | `interfaces bonding bond0 address` | `interfaces bonding bond0 vif 1 address` |
| firewall | `interface-group LAN interface bond0` | `... interface bond0.1` |
| VRRP | `vrrp group native interface bond0` | `... interface bond0.1` |
| kea | — | **restart it** (see traps) |
On the switch: Native VLAN = **None** on the trunk to that firewall, with VLAN 1
added to the tagged set.
## The ordering constraint
**There is no overlap state.** An 802.1Q port always egresses its native VLAN
untagged, so while VLAN 1 is native the router can *send* tagged VLAN 1 but can
never *receive* it. Verified: a tagged VLAN 1 ARP sent from the switch arrived on
`bond0` untagged and never on `bond0.1`. Configuring "native VLAN 1 **and** VLAN 1
tagged" as a make-before-break does not work; the switch and router changes for a
given firewall are strictly simultaneous, and that router loses Management in
between.
What makes this safe anyway: **tagged and untagged Management coexist on the
same VLAN.** One VLAN is one broadcast domain no matter how each port tags it, so
the firewalls can be converted one at a time — verified with the primary untagged
and the secondary already tagged, both reachable, VIP up, VLAN 1 clients fine.
Access ports are untouched throughout. The UniFi controller at 192.168.1.5 and
your workstation are on access ports and never traverse the firewall trunks, so
you keep the controller you are making the change from. Only the router being
converted goes dark, and only until its own config lands.
## Procedure
Do the **backup** router first, then fail the VIPs over and do the other. You
need console (JetKVM) on the router being converted — its Management SSH dies the
moment the switch port changes.
For each router in turn:
1. Confirm the *other* router is MASTER and healthy:
`show vrrp` and `sudo /config/vrrp-wan-health; echo $?` (must be 0).
2. Start the monitor from a workstation on an access port:
`labsim/vlan1-move-monitor.sh` (edit the three addresses for production).
3. UniFi: on this firewall's trunk ports, Native VLAN → None, VLAN 1 → tagged.
This router's Management drops now.
4. Over the console, in one commit:
```
set interfaces bonding bond0 vif 1 address '192.168.1.252/24' # .253 on vyos002
set interfaces bonding bond0 vif 1 description 'management'
delete interfaces bonding bond0 address
set firewall group interface-group LAN interface 'bond0.1'
delete firewall group interface-group LAN interface 'bond0'
set high-availability vrrp group native interface 'bond0.1'
commit
save
```
5. `sudo systemctl restart isc-kea-dhcp4-server` — see traps.
6. Verify: Management SSH back, `show vrrp` shows `native` on `bond0.1`, and the
leak test passes.
Then fail back if the VIPs moved (below), and repeat for the other router.
### Measured windows (labsim)
| | |
|---|---|
| this router's own Management unreachable | ~27 s (the console apply) |
| VIP `.1` unreachable, peer already converted | **0 s** |
| VIP `.1` unreachable, converting the current MASTER | ~6 s (VRRP failover) |
| VIP unreachable if you convert both routers before the switch | **5 min 30 s** |
That last row is the failure mode to avoid: with both routers untagged and the
trunks already changed, the VIP is a black hole and **the healthy BACKUP does not
take over**. Its `native` group stays BACKUP because the *other* VLANs still hear
the master, and the sync group holds them together. Redundancy does not help you
here; only ordering does.
## Traps
- **Restart kea.** VyOS does not restart it for an interface address change, so
it keeps a raw socket bound with the old address and keeps emitting the wrong
offers. The first post-fix test in the sim failed for this reason alone and
looked exactly like the fix not working.
- **`interface-group LAN`.** Moving the address without moving the group means
Management falls outside the group, and with default-deny that is every
management session and all VLAN 1 inter-VLAN routing, gone on commit — on a
router you reach through itself. Use `commit-confirm` if you are not on console.
- **The VIPs may move, and `no-preempt` keeps them moved.** Converting a router
restarts keepalived and re-initialises *every* group, not just `native`. In one
rehearsal the priority-100 secondary took all six VIPs and held them while the
priority-200 primary sat at BACKUP; in another the restart was quick enough that
nothing moved. It is non-deterministic — check afterwards, every time.
Fail back with `restart vrrp` **on the router currently holding them**.
- **Duplicate delivery does not stop**, and should not be read as failure. #1117
only promises there is no longer a subnet on the parent to match. Expect two
identical replies per DISCOVER, both from the correct pool.
- **Both firewalls' trunks must end up the same.** If UniFi shares one port
profile between them, changing it converts both at once and you get the 5m30s
row above. Check before you start; use per-port overrides if it does.
## Not covered by the rehearsal
- Whether UniFi's port profile can express "no native VLAN" the way OVS can, and
whether the two firewalls share a profile. Unverified — check on the controller.
- Why the JetKVM consoles specifically accepted the wrong OFFER when a VLAN 3
access port should not receive an untagged VLAN 1 frame at all. Their port
profile likely passes VLAN 1 untagged. Worth confirming, though it does not
change the fix.