michal/lab
1
0
Fork 0
Code Issues Pull Requests 3 Actions Packages Projects Releases Wiki Activity

feat(migration): complete the VyOS HA stack per the official docs

Browse Source 
Prompted by "I thought we tested HA on libvirt" -- checking rather than
recalling showed the sim has ONE VyOS router with zero high-availability
config. VRRP was configured and running on the real pair, but it is only one of
four parts of what VyOS considers an HA pair.

Against docs.vyos.io (highavailability, conntrack-sync, dhcp-server, and the HA
walkthrough), three gaps are now closed in the delta:

  - VRRP was multicast-only with default preemption. Added unicast
    hello-source-address/peer-address per group, as the walkthrough does, plus
    no-preempt. Without no-preempt a recovered box reclaims the VIP before
    conntrack state has synced and drops every established connection; the docs
    are explicit that preempt-delay must otherwise be >= purge-timeout.
    The per-VLAN node addresses are a table, not derived: VLAN 3 is .4/.5 while
    every other VLAN is .252/.253.
  - DHCP high-availability, which fixes a real defect rather than adding a
    feature. Both boxes carried the full 6 subnets and 84 static-mappings, so
    after cutover two kea instances would have raced on the same broadcast
    domains. Now active-passive with primary/secondary and swapped
    source/remote, syncing over TCP 647 on the LoT addresses. Each subnet
    already carries the unique subnet-id kea HA requires, and the peer name
    deliberately differs from both host-names.
  - conntrack-sync over a dedicated eth3 <-> eth3 link, gated behind
    --conntrack-link because it needs a cable that is not plugged in yet. This
    is what the peer cable is actually for -- VRRP does not want one, since its
    hellos must travel on the segment they protect.

VRRP failover exercised on the production pair, which is free to break today
because nothing uses the .254 VIPs: keepalived stopped on vyos001, all six VIPs
moved to vyos002 within 12s, and returned on restart (preemption still default
on the live boxes). Both boxes clean afterwards, no config drift.

Master delta validated against vyos001's real running config on the sim router
before installing. Installed on both: 6 no-preempt, 6 unicast pairs, DHCP HA
primary/secondary respectively.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
This commit is contained in:
Michal
2026-08-16 22:48:39 +01:00
parent f81c94af43
commit 952f5c66e3
1 changed files with 79 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

81
migration/vyos-mode-delta.py
View File

@@ -58,6 +58,29 @@ DIST_DHCP, DIST_PPPOE = 1, 10
PLACEHOLDER = "@@WAN_PASSWORD@@"
# Per-VLAN interface addresses of each node, read from the live boxes. VRRP
# unicast (hello-source-address/peer-address) needs both ends explicitly, and
# these are NOT derivable from the subnet -- VLAN 3 is .4/.5 while everything
# else is .252/.253.
# vlan: (vyos001, vyos002)
NODE_ADDRS = {
1: ("192.168.1.252", "192.168.1.253"),
2: ("192.168.9.252", "192.168.9.253"),
3: ("192.168.3.4", "192.168.3.5"),
9: ("10.8.0.252", "10.8.0.253"),
10: ("10.0.1.252", "10.0.1.253"),
200: ("192.168.2.252", "192.168.2.253"),
}
# Dedicated point-to-point link for conntrack state sync (eth3 <-> eth3).
CONNTRACK_ADDRS = ("10.255.255.1/30", "10.255.255.2/30")
CONNTRACK_IF = "eth3"
# kea HA talks over TCP 647. The LoT addresses are used because they are stable
# and reachable today without the conntrack cable being plugged in.
DHCP_HA_NAME = "vyos-dhcp-pair" # must NOT equal either system host-name
def vrrp_group(vlan: int) -> str:
"""VRRP group names as configured on the boxes: 'native' for the untagged
@@ -65,8 +88,11 @@ def vrrp_group(vlan: int) -> str:
return "native" if vlan == 1 else f"vlan{vlan}"
def build_delta(inv: dict, priority: int, wan_user: str, with_wan: bool) -> list[str]:
def build_delta(inv: dict, priority: int, wan_user: str, with_wan: bool,
conntrack_link: bool) -> list[str]:
out: list[str] = []
primary = priority >= 200 # vyos001 is the master/primary
self_i, peer_i = (0, 1) if primary else (1, 0)
nets = [n for n in inv["networks"] if n["dhcp_enabled"] and n["subnet"]]
nets.sort(key=unifi_to_vyos.vlan_of)
@@ -98,6 +124,16 @@ def build_delta(inv: dict, priority: int, wan_user: str, with_wan: bool) -> list
out.append(f"delete high-availability vrrp group {grp} address")
out.append(f"set high-availability vrrp group {grp} address {iface.with_prefixlen}")
out.append(f"set high-availability vrrp group {grp} priority {priority}")
own, peer = NODE_ADDRS[vlan] if primary else NODE_ADDRS[vlan][::-1]
# Unicast VRRP: the walkthrough sets both ends explicitly rather than
# relying on multicast, which is more predictable across a switch fabric.
out.append(f"set high-availability vrrp group {grp} hello-source-address {own}")
out.append(f"set high-availability vrrp group {grp} peer-address {peer}")
# Without no-preempt a recovered box reclaims the VIP immediately --
# before conntrack state has synced -- and drops every established
# connection. If preemption is ever wanted, preempt-delay must be >=
# the conntrack-sync purge-timeout.
out.append(f"set high-availability vrrp group {grp} no-preempt")
out += [
"",
@@ -266,6 +302,44 @@ def build_delta(inv: dict, priority: int, wan_user: str, with_wan: bool) -> list
"",
]
# --- DHCP high-availability -------------------------------------------
# Without this BOTH boxes run kea on the same VLANs and race to answer the
# same broadcasts, handing different pool addresses to the same client.
# active-passive so only the primary serves, matching the VRRP shape.
dhcp_self, dhcp_peer = NODE_ADDRS[10][self_i], NODE_ADDRS[10][peer_i]
out += [
"",
"# --- DHCP high-availability --------------------------------",
"# Peers sync leases over TCP 647. Each subnet already carries a",
"# unique subnet-id (keyed on VLAN id), which kea HA requires.",
"set service dhcp-server high-availability mode active-passive",
f"set service dhcp-server high-availability status {'primary' if primary else 'secondary'}",
# The peer name must not collide with either system host-name.
f"set service dhcp-server high-availability name {DHCP_HA_NAME}",
f"set service dhcp-server high-availability source-address {dhcp_self}",
f"set service dhcp-server high-availability remote {dhcp_peer}",
]
if conntrack_link:
# Stateful failover. Without it VRRP moves the address but every
# established connection dies, because the backup has no conntrack
# table. Needs the eth3 <-> eth3 cable physically present.
out += [
"",
"# --- conntrack-sync ----------------------------------------",
"# Dedicated point-to-point link: sync traffic must not compete",
"# with production, and must not die when the LAN does.",
f"set interfaces ethernet {CONNTRACK_IF} address {CONNTRACK_ADDRS[self_i]}",
f"set interfaces ethernet {CONNTRACK_IF} description 'conntrack-sync peer link'",
f"set service conntrack-sync interface {CONNTRACK_IF}",
"set service conntrack-sync failover-mechanism vrrp sync-group MAIN",
"set service conntrack-sync accept-protocol tcp",
"set service conntrack-sync accept-protocol udp",
"set service conntrack-sync accept-protocol icmp",
"set service conntrack-sync mcast-group 225.0.0.50",
]
# DHCP + DNS, from the same generator labsim proved.
dhcp_lines, stats = unifi_to_vyos.build(inv, "prod")
expected = len(inv["reservations"])
@@ -285,6 +359,8 @@ def main() -> int:
ap.add_argument("--raw-networkconf", default=os.path.join(HERE, "export", "rest_networkconf.json"))
ap.add_argument("--with-wan", action="store_true",
help="configure the WAN on this box. Only ONE of the pair may have\n it, because the cloned WAN MAC must be unique.")
ap.add_argument("--conntrack-link", action="store_true",
help="emit conntrack-sync over the eth3 peer link. Requires the\n cable to be physically present on both boxes.")
ap.add_argument("-o", "--out")
ap.add_argument("--emit-secrets", metavar="PATH",
help="write the PPPoE credential to PATH with mode 0600 and exit")
@@ -312,7 +388,8 @@ def main() -> int:
file=sys.stderr)
return 0
lines = build_delta(inv, args.priority, wan["wan_username"], args.with_wan)
lines = build_delta(inv, args.priority, wan["wan_username"], args.with_wan,
args.conntrack_link)
text = "\n".join(lines) + "\n"
# Only a WAN-carrying delta has a credential to placeholder-substitute.