2e828b8af2
Some checks failed
A planned failover had no reliable lever. VyOS offers only `restart vrrp`, and
neither that nor `systemctl restart keepalived` is dependable: with advert_int 1
the peer declares the master dead after ~3.6s and a restart usually finishes
inside that window. Measured -- the same command moved mastership on one run and
not on the next three. A fail-back step you cannot trigger on purpose is not a
procedure, and the recovery card depended on one.
The lever is now `touch /run/vrrp-wan/force-fault`: the health check fails, the
sync group sheds every VIP, and the peer takes over. It exercises the same path
a real WAN loss takes rather than a special case, and it lives in /run so a
reboot cannot leave a router permanently ineligible.
Proven end to end in labsim: lever -> mastership moves -> WAN follows -> the old
master releases -> a LAN VM has internet -> the faulted router returns to BACKUP
and is eligible again.
Getting there exposed four real bugs, two of which would have broken a GENUINE
failover, not just the drill:
- The grace stamp was written only by the 30s reconciler, so a freshly
promoted master reached the 5s health check with no stamp, scored grace = 0,
failed instantly and went FAULT. With the peer already faulted that left
BOTH routers in FAULT and the LAN with no gateway at all -- worse than the
outage the check exists to prevent. The check now stamps on promotion.
- And it inherited STALE stamps from an earlier mastership, failing ~5s after
passing. The stamp is now cleared on the way down, by the health check
itself, not only by the reconciler.
- The lock fd leaked into VyOS's config session: `exec 9>` is inherited by the
long-lived unionfs-fuse the session spawns, which never closes it. From the
first config change on, every later reconciler run lost the flock and exited
0 having done nothing -- healthy-looking journal, silently stopped
reconciling. That is how a demoted router kept the WAN. Children now get 9>&-.
- vrrp-wan-apply touched pppoe0 unconditionally. On a box where pppoe0 has no
source-interface VyOS rejects the whole commit ("Physical source-interface
required"), taking the bond0.53 change down with it -- and the script still
returned 0, so the reconciler logged a release that never happened. pppoe0 is
now guarded on existence and the commit's verdict is propagated.
Still NOT applied to production. The pair is single-homed on WAN until it is.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
115 lines
5.7 KiB
Bash
115 lines
5.7 KiB
Bash
#!/bin/sh
|
|
# Make the WAN match VRRP mastership. Idempotent; safe to run every 30s and on
|
|
# every VRRP transition.
|
|
#
|
|
# Why a reconciler and not just transition scripts
|
|
# ------------------------------------------------
|
|
# VyOS delivers `transition-script` through a helper process,
|
|
# /usr/libexec/vyos/system/keepalived-fifo.py, fed by keepalived's notify_fifo.
|
|
# Observed in labsim on 2026-09-02: the primary's Keepalived_vrrp logged
|
|
# "(native) Entering MASTER STATE" for all six instances and the built-in
|
|
# notify_master for conntrack-sync ran -- while the fifo helper logged NOTHING
|
|
# and the master transition script never ran. The helper process was still
|
|
# alive. The result was a router holding every VIP with no WAN at all: the exact
|
|
# 2026-09-02 outage, re-created by the mechanism meant to prevent it.
|
|
#
|
|
# So transition scripts are kept for speed but nothing is trusted to them: this
|
|
# also runs on a timer, and derives everything from ground truth rather than
|
|
# from a marker that only exists if the script it depends on ran.
|
|
#
|
|
# vrrp-wan-reconcile reconcile once
|
|
# vrrp-wan-reconcile --status what it thinks, changing nothing
|
|
#
|
|
# Ground truth for "am I master" is whether the management VIP is really on this
|
|
# box. It is what VRRP actually does, it is observable, and it cannot silently
|
|
# disagree with reality.
|
|
|
|
VIP="${VRRP_WAN_VIP:-192.168.1.1}" # management VIP; sim overrides via env
|
|
WAN_VIF=53 # bond0.53, the DHCP WAN
|
|
STATE=/run/vrrp-wan
|
|
LOCK=/run/vrrp-wan.lock
|
|
|
|
cfg() { /opt/vyatta/bin/vyatta-op-cmd-wrapper show configuration commands 2>/dev/null; }
|
|
holds_vip() { ip -4 -o addr show 2>/dev/null | grep -q " ${VIP}/"; }
|
|
wan_up() { ip -4 addr show "bond0.${WAN_VIF}" 2>/dev/null | grep -q 'inet '; }
|
|
wan_disabled(){ cfg | grep -q "vif ${WAN_VIF} disable"; }
|
|
ppp_disabled(){ cfg | grep -q "pppoe pppoe0 disable"; }
|
|
|
|
# --status must answer WITHOUT sourcing script-template. The template's `exit`
|
|
# is a function that leaves configuration mode, not the shell builtin, so a
|
|
# status run that had sourced it opened and closed a config session on every
|
|
# call -- which is how a read-only query started colliding with the timer and
|
|
# logging "Configuration system temporarily locked due to another commit".
|
|
if [ "${1:-}" = "--status" ]; then
|
|
printf 'vip=%s holds_vip=%s wan_disabled=%s wan_up=%s role=%s\n' \
|
|
"$VIP" "$(holds_vip && echo yes || echo no)" \
|
|
"$(wan_disabled && echo yes || echo no)" \
|
|
"$(wan_up && echo yes || echo no)" \
|
|
"$(cat "$STATE/role" 2>/dev/null || echo unset)"
|
|
exit 0
|
|
fi
|
|
|
|
# One writer. The 30s timer and a VRRP transition can fire together, and two
|
|
# VyOS commits in flight on one box do not queue -- the second fails outright.
|
|
#
|
|
# The lock fd MUST be closed for children (`9>&-` on every call below). Entering
|
|
# VyOS config mode spawns a long-lived unionfs-fuse for the session, and it
|
|
# INHERITS this descriptor and never lets go -- `lsof /run/vrrp-wan.lock` showed
|
|
# it held by unionfs-fuse with fd 9w. From the first config change onward every
|
|
# later run lost the flock and exited 0 without doing anything, so the
|
|
# reconciler looked healthy in the journal ("Finished") while quietly having
|
|
# stopped reconciling. It is how a demoted router kept the WAN.
|
|
exec 9>"$LOCK"
|
|
flock -n 9 || exit 0
|
|
|
|
mkdir -p "$STATE"
|
|
|
|
# Reap config sessions whose owning process is gone. VyOS creates
|
|
# /opt/vyatta/config/tmp/new_config_<pid> (a unionfs mount) per `configure`, and
|
|
# a script that dies inside a session never removes it. One of those holds the
|
|
# commit lock, and from then on EVERY commit fails with "Configuration system
|
|
# temporarily locked due to another commit in progress" -- including the manual
|
|
# one you try in order to fix it. A job on a 30s timer that can leak a session
|
|
# per failure will wedge the router's config system on its own, so it cleans up
|
|
# before it starts. `umount -l` first: the directory is a mount point and plain
|
|
# rm returns "Device or resource busy".
|
|
for d in /opt/vyatta/config/tmp/new_config_*; do
|
|
[ -d "$d" ] || continue
|
|
pid=${d##*_}
|
|
kill -0 "$pid" 2>/dev/null && continue
|
|
umount -l "$d" 2>/dev/null
|
|
rm -rf "$d" 2>/dev/null
|
|
done
|
|
|
|
# The config edit lives in vrrp-wan-apply, because script-template must be the
|
|
# first thing its script does -- sourced any later it terminates the script
|
|
# silently with rc=0. See the header there.
|
|
APPLY=/config/vrrp-wan-apply
|
|
|
|
if holds_vip; then
|
|
echo master > "$STATE/role"
|
|
# Stamp only on entry to master, so the health check's grace window measures
|
|
# time-since-promotion rather than time-since-last-tick.
|
|
[ -f "$STATE/since" ] || date +%s > "$STATE/since"
|
|
wan_disabled || ppp_disabled || exit 0
|
|
logger -t vrrp-wan "MASTER with WAN disabled -> enabling bond0.${WAN_VIF} + pppoe0"
|
|
"$APPLY" enable 9>&-
|
|
else
|
|
echo backup > "$STATE/role"
|
|
rm -f "$STATE/since"
|
|
{ wan_disabled && ppp_disabled; } && exit 0
|
|
# Releasing matters more than taking. A demoted router that keeps the WAN up
|
|
# holds the cloned MAC f0:9f:c2:12:9b:4f on VLAN 53 at the same time as the
|
|
# new master, and the switch sends the ISP's replies to whichever port spoke
|
|
# last -- the WAN-side twin of the eth2 incident.
|
|
logger -t vrrp-wan "not MASTER but WAN enabled -> releasing bond0.${WAN_VIF} + pppoe0"
|
|
"$APPLY" disable 9>&-
|
|
fi
|
|
|
|
# Deliberately no `save`. config.boot keeps `disable` on BOTH routers, so a
|
|
# reboot in any order comes up unable to claim the shared MAC, and only holding
|
|
# the VIP re-enables it. NOTE: any `save` while this box is master (a hand
|
|
# commit, or `pulumi up`) WILL persist the enabled state -- observed in labsim.
|
|
# The Pulumi model asserts `disable` on both routers so an apply puts it back,
|
|
# and vyos:verify reports it as drift if it does not.
|