4efd70c987
Some checks failed
PPPoE HA could not work as written, and the reason is structural rather than a
bug: `set interfaces pppoe pppoe0 disable` and `delete` are handled identically
by interfaces_pppoe.py -- both UNLINK /etc/ppp/peers/pppoe0. That path is pppd's
own options file, so the resting state destroyed exactly what the promotion path
needed, and `ppp@pppoe0` restart-looped against it (observed: 47 restarts, zero
sessions at the access concentrator). It also made op-mode `connect interface
pppoe0` unusable, and put every failover behind a priority-322 commit where one
unrelated invalid node fails the whole thing -- which has already taken the
10 gig down once.
pppoe0 is now configured identically and ENABLED on both routers, so the peers
file always exists, and dialling is gated by a drop-in on the unit:
ConditionPathExists=/run/vrrp-wan/may-dial
ConditionPathExists=/etc/ppp/peers/pppoe0
/run is tmpfs, so the gate is shut at boot and neither box can dial before VRRP
has decided. That matters more than it first appears: with the node enabled,
interfaces_pppoe.py restarts ppp on EVERY commit touching the pppoe subtree when
the daemon is not running -- so the backup actively tries to dial whenever
anything commits. The gate is the only thing making that a no-op, which is why
vrrp-wan-reconcile now refuses to bless a box whose drop-in is missing: /etc is
per-image, and a VyOS upgrade would otherwise silently remove the protection.
may-dial is a LEASE, not a flag. ConditionPathExists is evaluated at start only
-- it can prevent a dial, never revoke one -- so a reconciler that stops running
while its box is demoted would keep the one ISP session for ever. The reconciler
renews the lease; a new 5s vrrp-wan-guard revokes it, and only ever revokes. It
fired correctly first time: "GUARD: lease stale (81s > 75s)".
Also: remove-then-stop on release (the file's absence blocks a NEW start that a
concurrent commit would trigger); a flap damper, because two routers that both
believe they hold the VIP will both dial and each dial kills the other's session
-- against a real ISP that is how an account gets rate-limited; and a guard on
`cfg` returning empty under commit-lock contention, which had already produced
one spurious "releasing" on a box that needed nothing.
GRACE 90 -> 180. accel-ppp's dead-peer budget is lcp-echo-interval(30) x
failure(3) = 90s, so the old value sat exactly on the boundary: a hard failover
into an AC that does not replace the stale session would fail its own check,
shed the VIPs, and leave both routers in FAULT.
The sim could not have tested any of this. Both routers now get the identical
WAN -- the secondary had none "because two PPPoE clients sharing one credential
is a different failure mode than anything production has", which is backwards:
that IS production. It also left the pair incomparable, ten NAT rules against
none. Safety now comes from resting state, not asymmetry.
Three more things the sim was hiding:
- the drift check's secondary regex omitted interfaces pppoe/bonding, nat
source and protocols failover, so it reported "in sync" for a box with no
WAN at all;
- the VRRP health-check and transition-script hooks existed on both live VMs
and in NEITHER generator -- the mechanism under test was pure undetected
drift;
- labsim-vyos's only default route was the libvirt-NAT scaffold, so every
"the LAN still has internet" verdict on it was answered by eth2 rather than
the WAN. --drop-scaffold applied; the earlier DHCP-failover proof is being
re-run because of it.
vrrp-wan-install ends the other half of that: the sim's previous proof came from
scripts hand-`sed`-ed in place, so the tested behaviour was not the committed
behaviour. `--check` now makes that a hard failure.
First green run: master holds both WANs, backup released, and the AC reports
exactly ONE session. sim-net-apply.sh check: all four in sync.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
132 lines
6.8 KiB
Bash
Executable File
132 lines
6.8 KiB
Bash
Executable File
#!/bin/sh
|
|
# VRRP health check: may THIS router hold the floating IPs?
|
|
#
|
|
# It may only if it can actually carry the WAN. Without this, VRRP decides
|
|
# mastership purely on whether the peer is still advertising -- so a router with
|
|
# no WAN at all happily takes the VIPs and blackholes the entire LAN's internet
|
|
# while looking perfectly healthy. That is not hypothetical: it is the outage of
|
|
# 2026-09-02, reproduced in labsim.
|
|
#
|
|
# ---------------------------------------------------------------------------
|
|
# The first version of this script asked one question: "do I have an address on
|
|
# a WAN interface". That is correct for a pair where both routers hold WAN all
|
|
# the time. Ours cannot: the 10 gig lease is bound to a cloned MAC and the
|
|
# PPPoE line to a single credential, so the WAN follows mastership (see
|
|
# vrrp-wan-take). Against that design the old check DEADLOCKS --
|
|
#
|
|
# may I be master? -> only if I already have WAN
|
|
# do I have WAN? -> only if I am master
|
|
#
|
|
# -- and the backup sits in FAULT for ever. vyos002 sat exactly there, which
|
|
# meant the pair could not fail over at all: the safety check had quietly
|
|
# removed the redundancy it was protecting.
|
|
#
|
|
# So the question is now asked in the right order: enforce "must have WAN" only
|
|
# on the router that is actually HOLDING the VIPs, and give a new master time to
|
|
# bring the WAN up before judging it.
|
|
# ---------------------------------------------------------------------------
|
|
#
|
|
# exit 0 = eligible for MASTER, non-zero = release and let the peer have it.
|
|
|
|
CONF=/config/vrrp-wan.conf
|
|
[ -r "$CONF" ] && . "$CONF"
|
|
STATE=/run/vrrp-wan
|
|
VIP="${VRRP_WAN_VIP:-192.168.1.1}"
|
|
|
|
# Seconds a new master may go without any WAN. Sourced from vrrp-wan.conf; the
|
|
# fallback is deliberately NOT the old 90. accel-ppp's dead-peer budget is
|
|
# lcp-echo-interval(30) x lcp-echo-failure(3) = 90s, so a hard failover into an
|
|
# access concentrator that does not replace the stale session lands exactly on
|
|
# the boundary: the new master fails its own check, sheds the VIPs, and the peer
|
|
# -- in the same position -- does likewise. Both end in FAULT, which is worse
|
|
# than the outage this check exists to prevent.
|
|
GRACE="${GRACE:-180}"
|
|
|
|
# A deliberate hand-over lever.
|
|
#
|
|
# There is no reliable way to MAKE this pair fail over on demand. VyOS offers
|
|
# only `restart vrrp`, and neither that nor `systemctl restart keepalived` is
|
|
# dependable: with advert_int 1 the peer declares the master dead after ~3.6s,
|
|
# and a restart usually finishes inside that window. Measured in labsim -- the
|
|
# same command moved mastership on one run and not on the next three. A
|
|
# fail-back procedure you cannot trigger on purpose is not a procedure.
|
|
#
|
|
# Failing the health check IS the supported way to shed mastership: the sync
|
|
# group goes FAULT, releases every VIP, and the peer takes over -- the same path
|
|
# a genuine WAN loss takes, so the planned drill exercises the real mechanism
|
|
# rather than a special case.
|
|
#
|
|
# touch /run/vrrp-wan/force-fault hand over within failure-count*interval
|
|
# rm /run/vrrp-wan/force-fault become eligible again (no-preempt keeps
|
|
# it BACKUP until the peer hands back)
|
|
#
|
|
# It lives in /run deliberately: a reboot clears it, so a forgotten drill cannot
|
|
# leave a router permanently ineligible.
|
|
[ -f /run/vrrp-wan/force-fault ] && exit 1
|
|
|
|
# Am I holding the VIPs? Asked of REALITY -- is the management VIP actually on
|
|
# this box -- and not of a /run marker.
|
|
#
|
|
# The marker was the first design and it is unsafe: it is written by the VRRP
|
|
# transition script, and in labsim that script silently failed to run on a
|
|
# promotion (VyOS's keepalived-fifo.py helper stopped delivering while
|
|
# keepalived's own notifies kept working). The router then believed it was
|
|
# backup, passed this check, and sat holding every VIP with no WAN -- the exact
|
|
# outage this script exists to prevent, re-created by trusting the reporter
|
|
# instead of the fact.
|
|
if [ -z "$(ip -4 -o addr show 2>/dev/null | grep " ${VIP}/")" ]; then
|
|
# Clear the grace stamp on the way down, HERE, not only in the reconciler.
|
|
# The reconciler runs every 30s; this runs every 5s. A promotion that
|
|
# inherited a stamp from an earlier mastership scored grace = hours, failed
|
|
# immediately, and took the sync group to FAULT ~5s after passing -- with the
|
|
# peer already faulted, that left BOTH routers in FAULT and the LAN with no
|
|
# gateway. The stamp must belong to the CURRENT mastership or it is worse
|
|
# than useless.
|
|
rm -f "$STATE/since" 2>/dev/null
|
|
exit 0
|
|
fi
|
|
|
|
# Start the grace clock HERE, the moment mastership is first observed.
|
|
#
|
|
# It used to be stamped only by vrrp-wan-reconcile, which runs on a 30s timer --
|
|
# so a freshly promoted master reached this check with no stamp, scored grace=0,
|
|
# failed, and went FAULT before it had any chance to bring the WAN up. The peer
|
|
# then found itself alone with no WAN either and did the same. Observed in
|
|
# labsim: BOTH routers in FAULT, nobody holding the VIPs, the LAN with no
|
|
# gateway at all. That is worse than the outage this script exists to prevent,
|
|
# and it would have hit a REAL failover, not just a drill -- the health check
|
|
# runs every 5s and the reconciler had not yet ticked.
|
|
mkdir -p "$STATE" 2>/dev/null
|
|
[ -f "$STATE/since" ] || date +%s > "$STATE/since"
|
|
|
|
# Master with an address on a WAN interface: healthy.
|
|
#
|
|
# Deliberately NOT "can I reach the internet" and NOT "do I have a default
|
|
# route". During a real ISP outage the route disappears on BOTH routers; a check
|
|
# keyed on that would put both into FAULT, nobody would hold the VIPs, and the
|
|
# LAN would lose inter-VLAN routing too -- turning an internet outage into a
|
|
# total one. A DHCP lease survives an ISP outage, so an address still
|
|
# distinguishes "this box structurally cannot route" from "the internet is down
|
|
# right now", which is the distinction that matters.
|
|
# Any WAN counts. Requiring the 10 gig specifically would fault a healthy master
|
|
# during a genuine 10 gig outage and turn a degraded state into a total one --
|
|
# the same reasoning as the default-route note above. Which one satisfied it is
|
|
# recorded for the operator and the test harness, but does not affect the verdict.
|
|
for ifc in bond0.53 pppoe0; do
|
|
if ip -4 addr show dev "$ifc" 2>/dev/null | grep -q 'inet '; then
|
|
echo "$ifc" > "$STATE/wan" 2>/dev/null
|
|
exit 0
|
|
fi
|
|
done
|
|
rm -f "$STATE/wan" 2>/dev/null
|
|
|
|
# Master, no WAN yet, still within the grace window: DHCP negotiation and PPPoE
|
|
# dial-up take real time, and the ISP has to accept the cloned MAC arriving on a
|
|
# different port. Failing here would demote the new master before it ever had a
|
|
# chance, and hand the VIPs straight back -- a flap, not a failover.
|
|
since=$(cat "$STATE/since" 2>/dev/null || echo 0)
|
|
[ $(( $(date +%s) - since )) -lt "$GRACE" ] && exit 0
|
|
|
|
# Master, past grace, still no WAN: release. This is the 2026-09-02 case.
|
|
exit 1
|