2e828b8af2
Some checks failed
A planned failover had no reliable lever. VyOS offers only `restart vrrp`, and
neither that nor `systemctl restart keepalived` is dependable: with advert_int 1
the peer declares the master dead after ~3.6s and a restart usually finishes
inside that window. Measured -- the same command moved mastership on one run and
not on the next three. A fail-back step you cannot trigger on purpose is not a
procedure, and the recovery card depended on one.
The lever is now `touch /run/vrrp-wan/force-fault`: the health check fails, the
sync group sheds every VIP, and the peer takes over. It exercises the same path
a real WAN loss takes rather than a special case, and it lives in /run so a
reboot cannot leave a router permanently ineligible.
Proven end to end in labsim: lever -> mastership moves -> WAN follows -> the old
master releases -> a LAN VM has internet -> the faulted router returns to BACKUP
and is eligible again.
Getting there exposed four real bugs, two of which would have broken a GENUINE
failover, not just the drill:
- The grace stamp was written only by the 30s reconciler, so a freshly
promoted master reached the 5s health check with no stamp, scored grace = 0,
failed instantly and went FAULT. With the peer already faulted that left
BOTH routers in FAULT and the LAN with no gateway at all -- worse than the
outage the check exists to prevent. The check now stamps on promotion.
- And it inherited STALE stamps from an earlier mastership, failing ~5s after
passing. The stamp is now cleared on the way down, by the health check
itself, not only by the reconciler.
- The lock fd leaked into VyOS's config session: `exec 9>` is inherited by the
long-lived unionfs-fuse the session spawns, which never closes it. From the
first config change on, every later reconciler run lost the flock and exited
0 having done nothing -- healthy-looking journal, silently stopped
reconciling. That is how a demoted router kept the WAN. Children now get 9>&-.
- vrrp-wan-apply touched pppoe0 unconditionally. On a box where pppoe0 has no
source-interface VyOS rejects the whole commit ("Physical source-interface
required"), taking the bond0.53 change down with it -- and the script still
returned 0, so the reconciler logged a release that never happened. pppoe0 is
now guarded on existence and the commit's verdict is propagated.
Still NOT applied to production. The pair is single-homed on WAN until it is.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
49 lines
2.5 KiB
Plaintext
49 lines
2.5 KiB
Plaintext
#!/bin/vbash
|
|
# Enable or disable the WAN. Split out from vrrp-wan-reconcile for one reason:
|
|
# `source /opt/vyatta/etc/functions/script-template` must be the FIRST thing the
|
|
# script does. Sourced after a few statements -- an if, an exec, a mkdir -- it
|
|
# silently terminated the script; `set -x` showed execution stopping inside the
|
|
# source with no error and rc=0, so the reconciler reported success having done
|
|
# nothing. Only a single assignment may precede it (the template resets the
|
|
# positional parameters, so the mode is captured first), which is the same shape
|
|
# /config/vyos-known-good uses.
|
|
#
|
|
# vrrp-wan-apply enable take the WAN
|
|
# vrrp-wan-apply disable release it
|
|
MODE="${1:-}"
|
|
source /opt/vyatta/etc/functions/script-template
|
|
|
|
WAN_VIF=53
|
|
cfg() { /opt/vyatta/bin/vyatta-op-cmd-wrapper show configuration commands 2>/dev/null; }
|
|
wan_disabled(){ cfg | grep -q "vif ${WAN_VIF} disable"; }
|
|
ppp_disabled(){ cfg | grep -q "pppoe pppoe0 disable"; }
|
|
# Only touch pppoe0 if it is actually configured. `set interfaces pppoe pppoe0
|
|
# disable` on a box that has no pppoe0 CREATES the node with nothing but
|
|
# `disable`, and VyOS then refuses the commit with "Physical source-interface
|
|
# required for pppoe0!" -- taking the bond0.53 change down with it, because one
|
|
# invalid node fails the whole commit. Seen on the labsim secondary, which has
|
|
# no PPPoE; production has it on both, so this would have been an untested path
|
|
# that only ever ran during a failover.
|
|
ppp_exists(){ cfg | grep -q "pppoe pppoe0 source-interface"; }
|
|
|
|
configure
|
|
if [ "$MODE" = enable ]; then
|
|
# Guarded: `delete` of an absent node aborts the whole batch with
|
|
# "Nothing to delete", which left the box detected-but-unfixed.
|
|
wan_disabled && delete interfaces bonding bond0 vif ${WAN_VIF} disable
|
|
ppp_exists && ppp_disabled && delete interfaces pppoe pppoe0 disable
|
|
else
|
|
wan_disabled || set interfaces bonding bond0 vif ${WAN_VIF} disable
|
|
ppp_exists && { ppp_disabled || set interfaces pppoe pppoe0 disable; }
|
|
fi
|
|
# Report the commit's verdict. The script previously ended on `exit` (a
|
|
# script-template function) and returned 0 even after "Commit failed", so the
|
|
# reconciler logged a successful release that had not happened -- the worst kind
|
|
# of failure for something whose whole job is to keep two routers from holding
|
|
# one WAN.
|
|
if commit 2>&1 | tee /tmp/vrrp-wan-commit.log | grep -qi "commit failed"; then
|
|
logger -t vrrp-wan "COMMIT FAILED applying '$MODE' -- see /tmp/vrrp-wan-commit.log"
|
|
exit 1
|
|
fi
|
|
exit
|