Files
lab/migration/MANAGEMENT-VLAN-TAGGED.md
Michal 0481c38e09 labsim: prove the tagged-Management fix for kea's wrong-pool offers
Kea #1117: with dhcp-socket-type raw, a frame tagged for a sub-interface is
also delivered to the parent's AF_PACKET socket, and if the parent serves a
subnet kea answers from it too. Management being the native VLAN on bond0 is
what gives the parent that subnet. One DISCOVER on VLAN 3 produced two OFFERs,
and in the captures here the WRONG one arrives first as often as not -- which
is why this looked device-dependent rather than like a server bug.

labsim-vlan-leak-test.sh reproduces it and scores the SERVER's offers, not the
client's choice; a client picking correctly is how this hid. Fails on the old
shape, passes on the new one across all six LAN VLANs.

Three things the rehearsal caught that reasoning had not:

  - kea keeps its old raw socket. VyOS does not restart it for an interface
    address change, so the first post-fix test failed and looked exactly like
    the fix not working.
  - interface-group LAN names the bare bond0. Moving the address without
    moving the group drops every management session under default-deny.
  - there is no make-before-break. A port always egresses its native VLAN
    untagged, so while VLAN 1 is native the router can send tagged VLAN 1 but
    never receive it -- verified, the ARP landed on bond0 untagged.

What makes the cutover safe anyway is that tagged and untagged Management
coexist, so the firewalls convert one at a time: 0s of VIP downtime, versus
5m30s if both routers go before the switch does. In that state the healthy
BACKUP does NOT take over -- the sync group holds native BACKUP because the
other VLANs still hear the master.

Also fixes two ways the sim was lying. ovs_bond_router compared only the trunk
VLAN list on re-runs, so a VM restart left the bond holding taps that no longer
existed while the real ones sat in the bridge unbonded -- labsim-vyos2 had no
LACP at all. And the tap count included the primary's libvirt-NAT scaffold NIC,
so the primary's bond was skipped outright.

Runbook: migration/MANAGEMENT-VLAN-TAGGED.md

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-09-02 14:03:44 +01:00

6.5 KiB

Moving Management onto a tagged VLAN

Rehearsed end to end in labsim on 2026-09-02. This is the fix for kea serving addresses from the wrong VLAN's pool.

Why

ISC Kea #1117: with dhcp-socket-type: raw, a frame tagged for a sub-interface is also delivered to the parent's AF_PACKET socket. If the parent serves a subnet, kea answers from it too. Ours does — Management is the native/untagged VLAN on bond0 while VLANs 2/3/9/10/200 are sub-interfaces of that same bond — so one DISCOVER on VLAN 3 produces two OFFERs and the client decides which to keep:

bond0.3 :  192.168.3.14  correct
bond0   :  192.168.1.28  UNTAGGED, Management pool, wrong

The fix is to leave no subnet on the parent: every VLAN tagged, Management included, moved from bond0 to bond0.1.

Confirmed in labsim across all six LAN VLANs: fails before, passes after. labsim/labsim-vlan-leak-test.sh is the test; evidence in labsim/vlan-leak-evidence/.

What must change together

Per router:

from to
address interfaces bonding bond0 address interfaces bonding bond0 vif 1 address
firewall interface-group LAN interface bond0 ... interface bond0.1
VRRP vrrp group native interface bond0 ... interface bond0.1
kea restart it (see traps)

On the switch: Native VLAN = None on the trunk to that firewall, with VLAN 1 added to the tagged set.

The ordering constraint

There is no overlap state. An 802.1Q port always egresses its native VLAN untagged, so while VLAN 1 is native the router can send tagged VLAN 1 but can never receive it. Verified: a tagged VLAN 1 ARP sent from the switch arrived on bond0 untagged and never on bond0.1. Configuring "native VLAN 1 and VLAN 1 tagged" as a make-before-break does not work; the switch and router changes for a given firewall are strictly simultaneous, and that router loses Management in between.

What makes this safe anyway: tagged and untagged Management coexist on the same VLAN. One VLAN is one broadcast domain no matter how each port tags it, so the firewalls can be converted one at a time — verified with the primary untagged and the secondary already tagged, both reachable, VIP up, VLAN 1 clients fine.

Access ports are untouched throughout. The UniFi controller at 192.168.1.5 and your workstation are on access ports and never traverse the firewall trunks, so you keep the controller you are making the change from. Only the router being converted goes dark, and only until its own config lands.

Procedure

Do the backup router first, then fail the VIPs over and do the other. You need console (JetKVM) on the router being converted — its Management SSH dies the moment the switch port changes.

For each router in turn:

  1. Confirm the other router is MASTER and healthy: show vrrp and sudo /config/vrrp-wan-health; echo $? (must be 0).
  2. Start the monitor from a workstation on an access port: labsim/vlan1-move-monitor.sh (edit the three addresses for production).
  3. UniFi: on this firewall's trunk ports, Native VLAN → None, VLAN 1 → tagged. This router's Management drops now.
  4. Over the console, in one commit:
    set interfaces bonding bond0 vif 1 address '192.168.1.252/24'   # .253 on vyos002
    set interfaces bonding bond0 vif 1 description 'management'
    delete interfaces bonding bond0 address
    set firewall group interface-group LAN interface 'bond0.1'
    delete firewall group interface-group LAN interface 'bond0'
    set high-availability vrrp group native interface 'bond0.1'
    commit
    save
    
  5. sudo systemctl restart isc-kea-dhcp4-server — see traps.
  6. Verify: Management SSH back, show vrrp shows native on bond0.1, and the leak test passes.

Then fail back if the VIPs moved (below), and repeat for the other router.

Measured windows (labsim)

this router's own Management unreachable ~27 s (the console apply)
VIP .1 unreachable, peer already converted 0 s
VIP .1 unreachable, converting the current MASTER ~6 s (VRRP failover)
VIP unreachable if you convert both routers before the switch 5 min 30 s

That last row is the failure mode to avoid: with both routers untagged and the trunks already changed, the VIP is a black hole and the healthy BACKUP does not take over. Its native group stays BACKUP because the other VLANs still hear the master, and the sync group holds them together. Redundancy does not help you here; only ordering does.

Traps

  • Restart kea. VyOS does not restart it for an interface address change, so it keeps a raw socket bound with the old address and keeps emitting the wrong offers. The first post-fix test in the sim failed for this reason alone and looked exactly like the fix not working.
  • interface-group LAN. Moving the address without moving the group means Management falls outside the group, and with default-deny that is every management session and all VLAN 1 inter-VLAN routing, gone on commit — on a router you reach through itself. Use commit-confirm if you are not on console.
  • The VIPs may move, and no-preempt keeps them moved. Converting a router restarts keepalived and re-initialises every group, not just native. In one rehearsal the priority-100 secondary took all six VIPs and held them while the priority-200 primary sat at BACKUP; in another the restart was quick enough that nothing moved. It is non-deterministic — check afterwards, every time. Fail back with restart vrrp on the router currently holding them.
  • Duplicate delivery does not stop, and should not be read as failure. #1117 only promises there is no longer a subnet on the parent to match. Expect two identical replies per DISCOVER, both from the correct pool.
  • Both firewalls' trunks must end up the same. If UniFi shares one port profile between them, changing it converts both at once and you get the 5m30s row above. Check before you start; use per-port overrides if it does.

Not covered by the rehearsal

  • Whether UniFi's port profile can express "no native VLAN" the way OVS can, and whether the two firewalls share a profile. Unverified — check on the controller.
  • Why the JetKVM consoles specifically accepted the wrong OFFER when a VLAN 3 access port should not receive an untagged VLAN 1 frame at all. Their port profile likely passes VLAN 1 untagged. Worth confirming, though it does not change the fix.