4efd70c987
Some checks failed
PPPoE HA could not work as written, and the reason is structural rather than a
bug: `set interfaces pppoe pppoe0 disable` and `delete` are handled identically
by interfaces_pppoe.py -- both UNLINK /etc/ppp/peers/pppoe0. That path is pppd's
own options file, so the resting state destroyed exactly what the promotion path
needed, and `ppp@pppoe0` restart-looped against it (observed: 47 restarts, zero
sessions at the access concentrator). It also made op-mode `connect interface
pppoe0` unusable, and put every failover behind a priority-322 commit where one
unrelated invalid node fails the whole thing -- which has already taken the
10 gig down once.
pppoe0 is now configured identically and ENABLED on both routers, so the peers
file always exists, and dialling is gated by a drop-in on the unit:
ConditionPathExists=/run/vrrp-wan/may-dial
ConditionPathExists=/etc/ppp/peers/pppoe0
/run is tmpfs, so the gate is shut at boot and neither box can dial before VRRP
has decided. That matters more than it first appears: with the node enabled,
interfaces_pppoe.py restarts ppp on EVERY commit touching the pppoe subtree when
the daemon is not running -- so the backup actively tries to dial whenever
anything commits. The gate is the only thing making that a no-op, which is why
vrrp-wan-reconcile now refuses to bless a box whose drop-in is missing: /etc is
per-image, and a VyOS upgrade would otherwise silently remove the protection.
may-dial is a LEASE, not a flag. ConditionPathExists is evaluated at start only
-- it can prevent a dial, never revoke one -- so a reconciler that stops running
while its box is demoted would keep the one ISP session for ever. The reconciler
renews the lease; a new 5s vrrp-wan-guard revokes it, and only ever revokes. It
fired correctly first time: "GUARD: lease stale (81s > 75s)".
Also: remove-then-stop on release (the file's absence blocks a NEW start that a
concurrent commit would trigger); a flap damper, because two routers that both
believe they hold the VIP will both dial and each dial kills the other's session
-- against a real ISP that is how an account gets rate-limited; and a guard on
`cfg` returning empty under commit-lock contention, which had already produced
one spurious "releasing" on a box that needed nothing.
GRACE 90 -> 180. accel-ppp's dead-peer budget is lcp-echo-interval(30) x
failure(3) = 90s, so the old value sat exactly on the boundary: a hard failover
into an AC that does not replace the stale session would fail its own check,
shed the VIPs, and leave both routers in FAULT.
The sim could not have tested any of this. Both routers now get the identical
WAN -- the secondary had none "because two PPPoE clients sharing one credential
is a different failure mode than anything production has", which is backwards:
that IS production. It also left the pair incomparable, ten NAT rules against
none. Safety now comes from resting state, not asymmetry.
Three more things the sim was hiding:
- the drift check's secondary regex omitted interfaces pppoe/bonding, nat
source and protocols failover, so it reported "in sync" for a box with no
WAN at all;
- the VRRP health-check and transition-script hooks existed on both live VMs
and in NEITHER generator -- the mechanism under test was pure undetected
drift;
- labsim-vyos's only default route was the libvirt-NAT scaffold, so every
"the LAN still has internet" verdict on it was answered by eth2 rather than
the WAN. --drop-scaffold applied; the earlier DHCP-failover proof is being
re-run because of it.
vrrp-wan-install ends the other half of that: the sim's previous proof came from
scripts hand-`sed`-ed in place, so the tested behaviour was not the committed
behaviour. `--check` now makes that a hard failure.
First green run: master holds both WANs, backup released, and the AC reports
exactly ONE session. sim-net-apply.sh check: all four in sync.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
137 lines
6.0 KiB
Python
Executable File
137 lines
6.0 KiB
Python
Executable File
#!/usr/bin/env python3
|
|
"""Generate the HA config for the labsim VyOS pair.
|
|
|
|
Exists to answer one question that cannot be answered on a single router, and
|
|
that would otherwise only be discovered at cutover: with kea HA active-passive,
|
|
does exactly ONE box answer a DHCP request?
|
|
|
|
Mirrors the production shape so the answer transfers:
|
|
|
|
router1 172.31.<v>.252 priority 200 DHCP HA primary
|
|
router2 172.31.<v>.253 priority 100 DHCP HA secondary
|
|
VIP 172.31.<v>.1 (what clients use as their gateway)
|
|
|
|
Note the sim's LoT VLAN is a /23 like production, so the VIP prefix differs
|
|
there -- getting that wrong produces a config that commits and then behaves
|
|
subtly wrongly, which is worse than a failure.
|
|
|
|
./sim-ha-config.py --role primary > r1.conf
|
|
./sim-ha-config.py --role secondary > r2.conf
|
|
"""
|
|
from __future__ import annotations
|
|
|
|
import argparse
|
|
import importlib.util
|
|
import json
|
|
import os
|
|
import sys
|
|
|
|
HERE = os.path.dirname(os.path.abspath(__file__))
|
|
MIG = os.path.join(HERE, "..", "migration")
|
|
|
|
# Reuse the DHCP/DNS generator rather than hand-writing subnets: the whole
|
|
# point is that what is proven here and what production gets share a code path.
|
|
_spec = importlib.util.spec_from_file_location(
|
|
"unifi_to_vyos", os.path.join(MIG, "unifi-to-vyos.py"))
|
|
unifi_to_vyos = importlib.util.module_from_spec(_spec)
|
|
_spec.loader.exec_module(unifi_to_vyos)
|
|
|
|
# vlan -> (prefix, cidr). LoT is a /23 in the sim, matching production.
|
|
VLANS = {
|
|
1: ("172.31.1", 24),
|
|
2: ("172.31.2", 24),
|
|
3: ("172.31.3", 24),
|
|
9: ("172.31.9", 24),
|
|
10: ("172.31.10", 23),
|
|
200: ("172.31.200", 24),
|
|
}
|
|
DHCP_HA_NAME = "labsim-dhcp-pair" # must not equal either host-name
|
|
|
|
|
|
def group(vlan: int) -> str:
|
|
return "native" if vlan == 1 else f"vlan{vlan}"
|
|
|
|
|
|
def build(role: str) -> list[str]:
|
|
primary = role == "primary"
|
|
self_o, peer_o = (252, 253) if primary else (253, 252)
|
|
prio = 200 if primary else 100
|
|
out = [f"# labsim VyOS HA -- {role}", ""]
|
|
|
|
for vlan, (pfx, cidr) in VLANS.items():
|
|
g = group(vlan)
|
|
# EVERY VLAN is a sub-interface, Management (VLAN 1) included. Putting
|
|
# Management on the bare `bond0` is what gives the parent a subnet, and
|
|
# kea then answers tagged frames from it as well as from the correct
|
|
# sub-interface -- clients on other VLANs get offered a Management
|
|
# address (ISC Kea #1117). See NATIVE_VLAN in ovs.sh; proven by
|
|
# labsim-vlan-leak-test.sh.
|
|
iface = f"bond0 vif {vlan}"
|
|
out += [
|
|
f"# VLAN {vlan}",
|
|
# The node's own address replaces the .1 it used to hold directly;
|
|
# .1 becomes the floating VIP, exactly as production will be.
|
|
f"delete interfaces bonding {iface} address",
|
|
f"set interfaces bonding {iface} address '{pfx}.{self_o}/{cidr}'",
|
|
f"set high-availability vrrp group {g} interface bond0.{vlan}",
|
|
f"set high-availability vrrp group {g} vrid {vlan}",
|
|
f"set high-availability vrrp group {g} address {pfx}.1/{cidr}",
|
|
f"set high-availability vrrp group {g} priority {prio}",
|
|
f"set high-availability vrrp group {g} hello-source-address {pfx}.{self_o}",
|
|
f"set high-availability vrrp group {g} peer-address {pfx}.{peer_o}",
|
|
f"set high-availability vrrp group {g} no-preempt",
|
|
f"set high-availability vrrp sync-group MAIN member {g}",
|
|
"",
|
|
]
|
|
|
|
out += [
|
|
"# --- WAN follows VRRP mastership ---",
|
|
# These four hooks and the health check existed on both live sim VMs but
|
|
# in NEITHER generator, so `sim-net-apply.sh check` reported "in sync"
|
|
# while the mechanism under test was pure undetected drift -- exactly
|
|
# the failure mode this file was written to end.
|
|
#
|
|
# The check goes on the SYNC GROUP, not per group: VyOS rejects a
|
|
# per-group check while the group is in a sync group ("Only sync group
|
|
# health check will be used").
|
|
"set high-availability vrrp sync-group MAIN health-check script '/config/vrrp-wan-health'",
|
|
"set high-availability vrrp sync-group MAIN health-check interval '5'",
|
|
"set high-availability vrrp sync-group MAIN health-check failure-count '3'",
|
|
# take and release both exec vrrp-wan-reconcile: one code path, asked at
|
|
# different moments. `stop` matters as much as `backup` -- a stopped
|
|
# keepalived is a demotion too, and without it the box would keep the
|
|
# WAN while holding no VIPs.
|
|
"set high-availability vrrp sync-group MAIN transition-script master '/config/vrrp-wan-take'",
|
|
"set high-availability vrrp sync-group MAIN transition-script backup '/config/vrrp-wan-release'",
|
|
"set high-availability vrrp sync-group MAIN transition-script fault '/config/vrrp-wan-release'",
|
|
"set high-availability vrrp sync-group MAIN transition-script stop '/config/vrrp-wan-release'",
|
|
"",
|
|
"# --- DHCP high-availability ---",
|
|
"# The thing under test: active-passive should mean exactly one OFFER.",
|
|
"set service dhcp-server high-availability mode active-passive",
|
|
f"set service dhcp-server high-availability status {role}",
|
|
f"set service dhcp-server high-availability name {DHCP_HA_NAME}",
|
|
f"set service dhcp-server high-availability source-address 172.31.10.{self_o}",
|
|
f"set service dhcp-server high-availability remote 172.31.10.{peer_o}",
|
|
"",
|
|
]
|
|
|
|
inv = json.load(open(os.path.join(MIG, "export", "inventory.json")))
|
|
dhcp, stats = unifi_to_vyos.build(inv, "sim")
|
|
out += [l for l in dhcp if l.strip() and not l.startswith("#")]
|
|
print(f"{role}: {stats['subnets']} subnets, {stats['mappings']} mappings",
|
|
file=sys.stderr)
|
|
return out
|
|
|
|
|
|
def main() -> int:
|
|
ap = argparse.ArgumentParser()
|
|
ap.add_argument("--role", choices=("primary", "secondary"), required=True)
|
|
args = ap.parse_args()
|
|
sys.stdout.write("\n".join(build(args.role)) + "\n")
|
|
return 0
|
|
|
|
|
|
if __name__ == "__main__":
|
|
sys.exit(main())
|