6987b324f1
Some checks failed
With the matrix actually setting session-control, T4 timed a destroyed
master's takeover at:
replace 26s
deny 148s <-- sizing case
disable 21s
`deny` is the case GRACE exists for: the AC refuses the survivor until its own
dead-peer timer frees the dead session. The session poller caught it happening
-- the destroyed router's session stayed in the table while the survivor's
dials appeared and were rejected, twice, before one took at 148s.
148s is well past the lcp-echo-interval(30) x failure(3) = 90s budget that 180
was sized against, leaving 32s of margin. GRACE=300 is ~2x the worst observed.
Treat 148s as a floor, not a worst case: these are idle 2-vCPU VMs, and the AC
shares an OVS bridge with the routers, so `virsh destroy` removes the port and
accel-ppp sees the peer vanish. A real BRAS over DSL never learns our router
died and waits out longer timers of its own.
The cost is stated in the conf: GRACE is also how long an alive-but-unroutable
master holds every VIP before yielding. bond0.53 covers most of that -- a DHCP
lease satisfies the check in seconds -- so GRACE only dominates when PPPoE is
the last path. Kept the health check's fallback in step, since keepalived runs
it with no environment and that number decides mastership if the conf is ever
missing.
check_invariant no longer fails blind on the AC's session count. That count is
only a proxy for "two of our routers dialled", and only while the AC enforces
single-session; under `disable` it does not, so a destroyed router's session
lingers and the count reads 2 with exactly one live router dialled. The real
invariant -- at most one router holds pppoe0 -- is now the failing one, and the
stale session is reported as a WARN rather than silenced, because it still
occupies the slot at a real ISP and is precisely what made `deny` take 148s.
71 lines
3.4 KiB
Plaintext
71 lines
3.4 KiB
Plaintext
# Settings for the vrrp-wan scripts. Installed to /config/vrrp-wan.conf.
|
|
#
|
|
# Why a file and not systemd Environment=: keepalived invokes vrrp-wan-health
|
|
# with NO environment at all, so an Environment= line in the .service would be
|
|
# read by the reconciler and ignored by the health check -- two sources of truth
|
|
# for the one value that decides who is master. It is also what stops a repeat
|
|
# of 2026-09-05, when the sim's proof was obtained from scripts hand-`sed`-ed in
|
|
# place: /config/vrrp-wan-health differed from git, and a reinstall would have
|
|
# silently reverted the tested behaviour.
|
|
|
|
# The management VIP. "Do I hold this address" IS the definition of master here
|
|
# -- ground truth, not a marker written by a script that may not have run.
|
|
VRRP_WAN_VIP=192.168.1.1
|
|
|
|
# The DHCP WAN sub-interface. Stays on the config plane because its lease is
|
|
# bound to a cloned MAC, which only VyOS config can move.
|
|
WAN_VIF=53
|
|
|
|
# Seconds a new master may go without any WAN before the health check fails it.
|
|
#
|
|
# Must exceed the ISP's stale-session hold-down, or a hard failover blows the
|
|
# window and BOTH routers end up in FAULT -- worse than the outage the check
|
|
# exists to prevent.
|
|
#
|
|
# MEASURED, labsim T4, master destroyed with `virsh destroy`, time until the
|
|
# survivor held a PPPoE session (labsim/wan-failover-evidence/T4-*):
|
|
#
|
|
# session-control=replace 26s
|
|
# session-control=deny 148s <-- worst
|
|
# session-control=disable 21s
|
|
#
|
|
# `deny` is the hostile case and the only one that matters for sizing: the AC
|
|
# refuses the survivor until its own dead-peer timer frees the dead session.
|
|
# The poller caught it happening -- the destroyed router's session stayed in the
|
|
# table while the survivor's dial attempts appeared and were rejected, twice,
|
|
# before it finally got in at 148s.
|
|
#
|
|
# 148s also lands well past the theoretical lcp-echo-interval(30) x
|
|
# failure(3) = 90s budget that 180 was originally sized against, which left only
|
|
# 32s of margin. 300 gives roughly 2x the worst observed, on IDLE 2-vCPU sim
|
|
# VMs; the VP2440s under kea, BGP and conntrack will be slower, and Vodafone's
|
|
# actual policy and timers are unknown.
|
|
#
|
|
# The cost is real and worth stating: this is also how long a master that is
|
|
# alive but genuinely cannot route keeps holding every VIP before yielding --
|
|
# the 2026-09-02 outage shape. That case is mostly covered by bond0.53, which
|
|
# satisfies the check within seconds of getting a DHCP lease; GRACE only
|
|
# dominates when PPPoE is the only path left.
|
|
#
|
|
# Do not lower this below the worst measured handover without re-running
|
|
# `labsim/labsim-pppoe-ha-test.sh --hard`. Before 2026-09-06 that matrix never
|
|
# actually set session-control and reported `deny` at 25s -- a number that did
|
|
# not exist.
|
|
GRACE=300
|
|
|
|
# may-dial is a LEASE, not a flag. vrrp-wan-reconcile renews its mtime every
|
|
# tick; vrrp-wan-guard revokes it once it goes stale. A plain flag survives the
|
|
# reconciler dying, and a router that stops reconciling while demoted would keep
|
|
# dialling for ever.
|
|
LEASE_TTL=75
|
|
|
|
# Flap damper. Two routers that both believe they hold the VIP (a VRRP
|
|
# partition) will both dial; with the AC set to `replace` each dial kills the
|
|
# other's session, the loser's pppd exits non-zero, systemd redials in 5s, and
|
|
# the pair hammers the access concentrator indefinitely. Against a real ISP that
|
|
# is how an account gets rate-limited. More than FLAP_MAX dials in FLAP_WINDOW
|
|
# puts this box in hold-off and logs loudly.
|
|
FLAP_MAX=6
|
|
FLAP_WINDOW=600
|
|
FLAP_HOLDOFF=900
|