test(labsim): second VyOS router proves DHCP active-passive HA
Answers the question a single router could not, and that would otherwise only
have been discovered at cutover: with kea high-availability active-passive, does
exactly ONE box answer a DHCP request?
Yes. Probing sim VLAN 10 with broadcast-dhcp-discover returns offers from a
single distinct Server Identifier -- 172.31.10.252, the primary. The secondary
runs kea but stays silent. Without this the delta would have put 6 subnets and
84 static-mappings on both boxes with nothing to arbitrate them, and two kea
instances would have raced on every broadcast domain.
Worth noting the raw response count is misleading: nmap reports "Response 1 of
2" because it sends several discovers, and both replies carry the same server
identifier. Counting responses says "2 servers"; counting distinct server
identifiers says "1". The second number is the true one.
labsim now runs a real pair, mirroring production:
router1 172.31.<v>.252 priority 200 DHCP HA primary
router2 172.31.<v>.253 priority 100 DHCP HA secondary
VIP 172.31.<v>.1 floating, held by the master
That required converting router1, which held .1 directly, to .252 plus a
floating VIP -- otherwise it is two routers, not a pair. All six VRRP groups
show MASTER on router1 and BACKUP on router2.
New tooling:
- sim-ha-config.py generates each role's config, reusing unifi-to-vyos.py
--mode sim for the DHCP half so what is proven here and what production
gets share a code path. VLAN 10 correctly carries /23.
- console-apply.py applies config over the serial console, which is necessary
because a freshly installed VyOS holds the same addresses as its peer and
cannot safely be reached over the network at all until reconfigured.
Known sim-only quirk, deliberately not chased: router1 cannot ARP router2 on
the untagged VLAN 1 while every tagged VLAN works, and VRRP forms correctly on
all six groups regardless. Both OVS bonds carry identical vlan_mode/tag/trunks
and the bond MACs differ, so this is OVS bond behaviour on the native VLAN with
two bonds on one bridge -- not a VyOS config problem, and not present in
production, which uses a real switch.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-08-16 23:29:12 +01:00
|
|
|
#!/usr/bin/env python3
|
|
|
|
|
"""Generate the HA config for the labsim VyOS pair.
|
|
|
|
|
|
|
|
|
|
Exists to answer one question that cannot be answered on a single router, and
|
|
|
|
|
that would otherwise only be discovered at cutover: with kea HA active-passive,
|
|
|
|
|
does exactly ONE box answer a DHCP request?
|
|
|
|
|
|
|
|
|
|
Mirrors the production shape so the answer transfers:
|
|
|
|
|
|
|
|
|
|
router1 172.31.<v>.252 priority 200 DHCP HA primary
|
|
|
|
|
router2 172.31.<v>.253 priority 100 DHCP HA secondary
|
|
|
|
|
VIP 172.31.<v>.1 (what clients use as their gateway)
|
|
|
|
|
|
|
|
|
|
Note the sim's LoT VLAN is a /23 like production, so the VIP prefix differs
|
|
|
|
|
there -- getting that wrong produces a config that commits and then behaves
|
|
|
|
|
subtly wrongly, which is worse than a failure.
|
|
|
|
|
|
|
|
|
|
./sim-ha-config.py --role primary > r1.conf
|
|
|
|
|
./sim-ha-config.py --role secondary > r2.conf
|
|
|
|
|
"""
|
|
|
|
|
from __future__ import annotations
|
|
|
|
|
|
|
|
|
|
import argparse
|
|
|
|
|
import importlib.util
|
|
|
|
|
import json
|
|
|
|
|
import os
|
|
|
|
|
import sys
|
|
|
|
|
|
|
|
|
|
HERE = os.path.dirname(os.path.abspath(__file__))
|
|
|
|
|
MIG = os.path.join(HERE, "..", "migration")
|
|
|
|
|
|
|
|
|
|
# Reuse the DHCP/DNS generator rather than hand-writing subnets: the whole
|
|
|
|
|
# point is that what is proven here and what production gets share a code path.
|
|
|
|
|
_spec = importlib.util.spec_from_file_location(
|
|
|
|
|
"unifi_to_vyos", os.path.join(MIG, "unifi-to-vyos.py"))
|
|
|
|
|
unifi_to_vyos = importlib.util.module_from_spec(_spec)
|
|
|
|
|
_spec.loader.exec_module(unifi_to_vyos)
|
|
|
|
|
|
|
|
|
|
# vlan -> (prefix, cidr). LoT is a /23 in the sim, matching production.
|
|
|
|
|
VLANS = {
|
|
|
|
|
1: ("172.31.1", 24),
|
|
|
|
|
2: ("172.31.2", 24),
|
|
|
|
|
3: ("172.31.3", 24),
|
|
|
|
|
9: ("172.31.9", 24),
|
|
|
|
|
10: ("172.31.10", 23),
|
|
|
|
|
200: ("172.31.200", 24),
|
|
|
|
|
}
|
|
|
|
|
DHCP_HA_NAME = "labsim-dhcp-pair" # must not equal either host-name
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def group(vlan: int) -> str:
|
|
|
|
|
return "native" if vlan == 1 else f"vlan{vlan}"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def build(role: str) -> list[str]:
|
|
|
|
|
primary = role == "primary"
|
|
|
|
|
self_o, peer_o = (252, 253) if primary else (253, 252)
|
|
|
|
|
prio = 200 if primary else 100
|
|
|
|
|
out = [f"# labsim VyOS HA -- {role}", ""]
|
|
|
|
|
|
|
|
|
|
for vlan, (pfx, cidr) in VLANS.items():
|
|
|
|
|
g = group(vlan)
|
labsim: prove the tagged-Management fix for kea's wrong-pool offers
Kea #1117: with dhcp-socket-type raw, a frame tagged for a sub-interface is
also delivered to the parent's AF_PACKET socket, and if the parent serves a
subnet kea answers from it too. Management being the native VLAN on bond0 is
what gives the parent that subnet. One DISCOVER on VLAN 3 produced two OFFERs,
and in the captures here the WRONG one arrives first as often as not -- which
is why this looked device-dependent rather than like a server bug.
labsim-vlan-leak-test.sh reproduces it and scores the SERVER's offers, not the
client's choice; a client picking correctly is how this hid. Fails on the old
shape, passes on the new one across all six LAN VLANs.
Three things the rehearsal caught that reasoning had not:
- kea keeps its old raw socket. VyOS does not restart it for an interface
address change, so the first post-fix test failed and looked exactly like
the fix not working.
- interface-group LAN names the bare bond0. Moving the address without
moving the group drops every management session under default-deny.
- there is no make-before-break. A port always egresses its native VLAN
untagged, so while VLAN 1 is native the router can send tagged VLAN 1 but
never receive it -- verified, the ARP landed on bond0 untagged.
What makes the cutover safe anyway is that tagged and untagged Management
coexist, so the firewalls convert one at a time: 0s of VIP downtime, versus
5m30s if both routers go before the switch does. In that state the healthy
BACKUP does NOT take over -- the sync group holds native BACKUP because the
other VLANs still hear the master.
Also fixes two ways the sim was lying. ovs_bond_router compared only the trunk
VLAN list on re-runs, so a VM restart left the bond holding taps that no longer
existed while the real ones sat in the bridge unbonded -- labsim-vyos2 had no
LACP at all. And the tap count included the primary's libvirt-NAT scaffold NIC,
so the primary's bond was skipped outright.
Runbook: migration/MANAGEMENT-VLAN-TAGGED.md
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-09-02 14:03:44 +01:00
|
|
|
# EVERY VLAN is a sub-interface, Management (VLAN 1) included. Putting
|
|
|
|
|
# Management on the bare `bond0` is what gives the parent a subnet, and
|
|
|
|
|
# kea then answers tagged frames from it as well as from the correct
|
|
|
|
|
# sub-interface -- clients on other VLANs get offered a Management
|
|
|
|
|
# address (ISC Kea #1117). See NATIVE_VLAN in ovs.sh; proven by
|
|
|
|
|
# labsim-vlan-leak-test.sh.
|
|
|
|
|
iface = f"bond0 vif {vlan}"
|
test(labsim): second VyOS router proves DHCP active-passive HA
Answers the question a single router could not, and that would otherwise only
have been discovered at cutover: with kea high-availability active-passive, does
exactly ONE box answer a DHCP request?
Yes. Probing sim VLAN 10 with broadcast-dhcp-discover returns offers from a
single distinct Server Identifier -- 172.31.10.252, the primary. The secondary
runs kea but stays silent. Without this the delta would have put 6 subnets and
84 static-mappings on both boxes with nothing to arbitrate them, and two kea
instances would have raced on every broadcast domain.
Worth noting the raw response count is misleading: nmap reports "Response 1 of
2" because it sends several discovers, and both replies carry the same server
identifier. Counting responses says "2 servers"; counting distinct server
identifiers says "1". The second number is the true one.
labsim now runs a real pair, mirroring production:
router1 172.31.<v>.252 priority 200 DHCP HA primary
router2 172.31.<v>.253 priority 100 DHCP HA secondary
VIP 172.31.<v>.1 floating, held by the master
That required converting router1, which held .1 directly, to .252 plus a
floating VIP -- otherwise it is two routers, not a pair. All six VRRP groups
show MASTER on router1 and BACKUP on router2.
New tooling:
- sim-ha-config.py generates each role's config, reusing unifi-to-vyos.py
--mode sim for the DHCP half so what is proven here and what production
gets share a code path. VLAN 10 correctly carries /23.
- console-apply.py applies config over the serial console, which is necessary
because a freshly installed VyOS holds the same addresses as its peer and
cannot safely be reached over the network at all until reconfigured.
Known sim-only quirk, deliberately not chased: router1 cannot ARP router2 on
the untagged VLAN 1 while every tagged VLAN works, and VRRP forms correctly on
all six groups regardless. Both OVS bonds carry identical vlan_mode/tag/trunks
and the bond MACs differ, so this is OVS bond behaviour on the native VLAN with
two bonds on one bridge -- not a VyOS config problem, and not present in
production, which uses a real switch.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-08-16 23:29:12 +01:00
|
|
|
out += [
|
|
|
|
|
f"# VLAN {vlan}",
|
|
|
|
|
# The node's own address replaces the .1 it used to hold directly;
|
|
|
|
|
# .1 becomes the floating VIP, exactly as production will be.
|
|
|
|
|
f"delete interfaces bonding {iface} address",
|
|
|
|
|
f"set interfaces bonding {iface} address '{pfx}.{self_o}/{cidr}'",
|
labsim: prove the tagged-Management fix for kea's wrong-pool offers
Kea #1117: with dhcp-socket-type raw, a frame tagged for a sub-interface is
also delivered to the parent's AF_PACKET socket, and if the parent serves a
subnet kea answers from it too. Management being the native VLAN on bond0 is
what gives the parent that subnet. One DISCOVER on VLAN 3 produced two OFFERs,
and in the captures here the WRONG one arrives first as often as not -- which
is why this looked device-dependent rather than like a server bug.
labsim-vlan-leak-test.sh reproduces it and scores the SERVER's offers, not the
client's choice; a client picking correctly is how this hid. Fails on the old
shape, passes on the new one across all six LAN VLANs.
Three things the rehearsal caught that reasoning had not:
- kea keeps its old raw socket. VyOS does not restart it for an interface
address change, so the first post-fix test failed and looked exactly like
the fix not working.
- interface-group LAN names the bare bond0. Moving the address without
moving the group drops every management session under default-deny.
- there is no make-before-break. A port always egresses its native VLAN
untagged, so while VLAN 1 is native the router can send tagged VLAN 1 but
never receive it -- verified, the ARP landed on bond0 untagged.
What makes the cutover safe anyway is that tagged and untagged Management
coexist, so the firewalls convert one at a time: 0s of VIP downtime, versus
5m30s if both routers go before the switch does. In that state the healthy
BACKUP does NOT take over -- the sync group holds native BACKUP because the
other VLANs still hear the master.
Also fixes two ways the sim was lying. ovs_bond_router compared only the trunk
VLAN list on re-runs, so a VM restart left the bond holding taps that no longer
existed while the real ones sat in the bridge unbonded -- labsim-vyos2 had no
LACP at all. And the tap count included the primary's libvirt-NAT scaffold NIC,
so the primary's bond was skipped outright.
Runbook: migration/MANAGEMENT-VLAN-TAGGED.md
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-09-02 14:03:44 +01:00
|
|
|
f"set high-availability vrrp group {g} interface bond0.{vlan}",
|
test(labsim): second VyOS router proves DHCP active-passive HA
Answers the question a single router could not, and that would otherwise only
have been discovered at cutover: with kea high-availability active-passive, does
exactly ONE box answer a DHCP request?
Yes. Probing sim VLAN 10 with broadcast-dhcp-discover returns offers from a
single distinct Server Identifier -- 172.31.10.252, the primary. The secondary
runs kea but stays silent. Without this the delta would have put 6 subnets and
84 static-mappings on both boxes with nothing to arbitrate them, and two kea
instances would have raced on every broadcast domain.
Worth noting the raw response count is misleading: nmap reports "Response 1 of
2" because it sends several discovers, and both replies carry the same server
identifier. Counting responses says "2 servers"; counting distinct server
identifiers says "1". The second number is the true one.
labsim now runs a real pair, mirroring production:
router1 172.31.<v>.252 priority 200 DHCP HA primary
router2 172.31.<v>.253 priority 100 DHCP HA secondary
VIP 172.31.<v>.1 floating, held by the master
That required converting router1, which held .1 directly, to .252 plus a
floating VIP -- otherwise it is two routers, not a pair. All six VRRP groups
show MASTER on router1 and BACKUP on router2.
New tooling:
- sim-ha-config.py generates each role's config, reusing unifi-to-vyos.py
--mode sim for the DHCP half so what is proven here and what production
gets share a code path. VLAN 10 correctly carries /23.
- console-apply.py applies config over the serial console, which is necessary
because a freshly installed VyOS holds the same addresses as its peer and
cannot safely be reached over the network at all until reconfigured.
Known sim-only quirk, deliberately not chased: router1 cannot ARP router2 on
the untagged VLAN 1 while every tagged VLAN works, and VRRP forms correctly on
all six groups regardless. Both OVS bonds carry identical vlan_mode/tag/trunks
and the bond MACs differ, so this is OVS bond behaviour on the native VLAN with
two bonds on one bridge -- not a VyOS config problem, and not present in
production, which uses a real switch.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-08-16 23:29:12 +01:00
|
|
|
f"set high-availability vrrp group {g} vrid {vlan}",
|
|
|
|
|
f"set high-availability vrrp group {g} address {pfx}.1/{cidr}",
|
|
|
|
|
f"set high-availability vrrp group {g} priority {prio}",
|
|
|
|
|
f"set high-availability vrrp group {g} hello-source-address {pfx}.{self_o}",
|
|
|
|
|
f"set high-availability vrrp group {g} peer-address {pfx}.{peer_o}",
|
|
|
|
|
f"set high-availability vrrp group {g} no-preempt",
|
|
|
|
|
f"set high-availability vrrp sync-group MAIN member {g}",
|
|
|
|
|
"",
|
|
|
|
|
]
|
|
|
|
|
|
|
|
|
|
out += [
|
|
|
|
|
"# --- DHCP high-availability ---",
|
|
|
|
|
"# The thing under test: active-passive should mean exactly one OFFER.",
|
|
|
|
|
"set service dhcp-server high-availability mode active-passive",
|
|
|
|
|
f"set service dhcp-server high-availability status {role}",
|
|
|
|
|
f"set service dhcp-server high-availability name {DHCP_HA_NAME}",
|
|
|
|
|
f"set service dhcp-server high-availability source-address 172.31.10.{self_o}",
|
|
|
|
|
f"set service dhcp-server high-availability remote 172.31.10.{peer_o}",
|
|
|
|
|
"",
|
|
|
|
|
]
|
|
|
|
|
|
|
|
|
|
inv = json.load(open(os.path.join(MIG, "export", "inventory.json")))
|
|
|
|
|
dhcp, stats = unifi_to_vyos.build(inv, "sim")
|
|
|
|
|
out += [l for l in dhcp if l.strip() and not l.startswith("#")]
|
|
|
|
|
print(f"{role}: {stats['subnets']} subnets, {stats['mappings']} mappings",
|
|
|
|
|
file=sys.stderr)
|
|
|
|
|
return out
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
def main() -> int:
|
|
|
|
|
ap = argparse.ArgumentParser()
|
|
|
|
|
ap.add_argument("--role", choices=("primary", "secondary"), required=True)
|
|
|
|
|
args = ap.parse_args()
|
|
|
|
|
sys.stdout.write("\n".join(build(args.role)) + "\n")
|
|
|
|
|
return 0
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
if __name__ == "__main__":
|
|
|
|
|
sys.exit(main())
|