vyos: a failover you can actually trigger, and four bugs found triggering it

A planned failover had no reliable lever. VyOS offers only `restart vrrp`, and
neither that nor `systemctl restart keepalived` is dependable: with advert_int 1
the peer declares the master dead after ~3.6s and a restart usually finishes
inside that window. Measured -- the same command moved mastership on one run and
not on the next three. A fail-back step you cannot trigger on purpose is not a
procedure, and the recovery card depended on one.

The lever is now `touch /run/vrrp-wan/force-fault`: the health check fails, the
sync group sheds every VIP, and the peer takes over. It exercises the same path
a real WAN loss takes rather than a special case, and it lives in /run so a
reboot cannot leave a router permanently ineligible.

Proven end to end in labsim: lever -> mastership moves -> WAN follows -> the old
master releases -> a LAN VM has internet -> the faulted router returns to BACKUP
and is eligible again.

Getting there exposed four real bugs, two of which would have broken a GENUINE
failover, not just the drill:

  - The grace stamp was written only by the 30s reconciler, so a freshly
    promoted master reached the 5s health check with no stamp, scored grace = 0,
    failed instantly and went FAULT. With the peer already faulted that left
    BOTH routers in FAULT and the LAN with no gateway at all -- worse than the
    outage the check exists to prevent. The check now stamps on promotion.
  - And it inherited STALE stamps from an earlier mastership, failing ~5s after
    passing. The stamp is now cleared on the way down, by the health check
    itself, not only by the reconciler.
  - The lock fd leaked into VyOS's config session: `exec 9>` is inherited by the
    long-lived unionfs-fuse the session spawns, which never closes it. From the
    first config change on, every later reconciler run lost the flock and exited
    0 having done nothing -- healthy-looking journal, silently stopped
    reconciling. That is how a demoted router kept the WAN. Children now get 9>&-.
  - vrrp-wan-apply touched pppoe0 unconditionally. On a box where pppoe0 has no
    source-interface VyOS rejects the whole commit ("Physical source-interface
    required"), taking the bond0.53 change down with it -- and the script still
    returned 0, so the reconciler logged a release that never happened. pppoe0 is
    now guarded on existence and the commit's verdict is propagated.

Still NOT applied to production. The pair is single-homed on WAN until it is.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH

migration/vrrp-wan-apply

@@ -17,16 +17,32 @@ WAN_VIF=53
 cfg() { /opt/vyatta/bin/vyatta-op-cmd-wrapper show configuration commands 2>/dev/null; }
 wan_disabled(){ cfg | grep -q "vif ${WAN_VIF} disable"; }
 ppp_disabled(){ cfg | grep -q "pppoe pppoe0 disable"; }
+# Only touch pppoe0 if it is actually configured. `set interfaces pppoe pppoe0
+# disable` on a box that has no pppoe0 CREATES the node with nothing but
+# `disable`, and VyOS then refuses the commit with "Physical source-interface
+# required for pppoe0!" -- taking the bond0.53 change down with it, because one
+# invalid node fails the whole commit. Seen on the labsim secondary, which has
+# no PPPoE; production has it on both, so this would have been an untested path
+# that only ever ran during a failover.
+ppp_exists(){ cfg | grep -q "pppoe pppoe0 source-interface"; }
 
 configure
 if [ "$MODE" = enable ]; then
     # Guarded: `delete` of an absent node aborts the whole batch with
     # "Nothing to delete", which left the box detected-but-unfixed.
     wan_disabled && delete interfaces bonding bond0 vif ${WAN_VIF} disable
-    ppp_disabled && delete interfaces pppoe pppoe0 disable
+    ppp_exists && ppp_disabled && delete interfaces pppoe pppoe0 disable
 else
     wan_disabled || set interfaces bonding bond0 vif ${WAN_VIF} disable
-    ppp_disabled || set interfaces pppoe pppoe0 disable
+    ppp_exists && { ppp_disabled || set interfaces pppoe pppoe0 disable; }
+fi
+# Report the commit's verdict. The script previously ended on `exit` (a
+# script-template function) and returned 0 even after "Commit failed", so the
+# reconciler logged a successful release that had not happened -- the worst kind
+# of failure for something whose whole job is to keep two routers from holding
+# one WAN.
+if commit 2>&1 | tee /tmp/vrrp-wan-commit.log | grep -qi "commit failed"; then
+    logger -t vrrp-wan "COMMIT FAILED applying '$MODE' -- see /tmp/vrrp-wan-commit.log"
+    exit 1
 fi
-commit
 exit

migration/vrrp-wan-health

@@ -32,6 +32,28 @@ STATE=/run/vrrp-wan
 GRACE=90          # seconds a new master gets to complete DHCP / PPPoE dial-up
 VIP="${VRRP_WAN_VIP:-192.168.1.1}"
 
+# A deliberate hand-over lever.
+#
+# There is no reliable way to MAKE this pair fail over on demand. VyOS offers
+# only `restart vrrp`, and neither that nor `systemctl restart keepalived` is
+# dependable: with advert_int 1 the peer declares the master dead after ~3.6s,
+# and a restart usually finishes inside that window. Measured in labsim -- the
+# same command moved mastership on one run and not on the next three. A
+# fail-back procedure you cannot trigger on purpose is not a procedure.
+#
+# Failing the health check IS the supported way to shed mastership: the sync
+# group goes FAULT, releases every VIP, and the peer takes over -- the same path
+# a genuine WAN loss takes, so the planned drill exercises the real mechanism
+# rather than a special case.
+#
+#   touch /run/vrrp-wan/force-fault    hand over within failure-count*interval
+#   rm    /run/vrrp-wan/force-fault    become eligible again (no-preempt keeps
+#                                      it BACKUP until the peer hands back)
+#
+# It lives in /run deliberately: a reboot clears it, so a forgotten drill cannot
+# leave a router permanently ineligible.
+[ -f /run/vrrp-wan/force-fault ] && exit 1
+
 # Am I holding the VIPs? Asked of REALITY -- is the management VIP actually on
 # this box -- and not of a /run marker.
 #
@@ -42,7 +64,30 @@ VIP="${VRRP_WAN_VIP:-192.168.1.1}"
 # backup, passed this check, and sat holding every VIP with no WAN -- the exact
 # outage this script exists to prevent, re-created by trusting the reporter
 # instead of the fact.
-[ -n "$(ip -4 -o addr show 2>/dev/null | grep " ${VIP}/")" ] || exit 0
+if [ -z "$(ip -4 -o addr show 2>/dev/null | grep " ${VIP}/")" ]; then
+    # Clear the grace stamp on the way down, HERE, not only in the reconciler.
+    # The reconciler runs every 30s; this runs every 5s. A promotion that
+    # inherited a stamp from an earlier mastership scored grace = hours, failed
+    # immediately, and took the sync group to FAULT ~5s after passing -- with the
+    # peer already faulted, that left BOTH routers in FAULT and the LAN with no
+    # gateway. The stamp must belong to the CURRENT mastership or it is worse
+    # than useless.
+    rm -f "$STATE/since" 2>/dev/null
+    exit 0
+fi
+
+# Start the grace clock HERE, the moment mastership is first observed.
+#
+# It used to be stamped only by vrrp-wan-reconcile, which runs on a 30s timer --
+# so a freshly promoted master reached this check with no stamp, scored grace=0,
+# failed, and went FAULT before it had any chance to bring the WAN up. The peer
+# then found itself alone with no WAN either and did the same. Observed in
+# labsim: BOTH routers in FAULT, nobody holding the VIPs, the LAN with no
+# gateway at all. That is worse than the outage this script exists to prevent,
+# and it would have hit a REAL failover, not just a drill -- the health check
+# runs every 5s and the reconciler had not yet ticked.
+mkdir -p "$STATE" 2>/dev/null
+[ -f "$STATE/since" ] || date +%s > "$STATE/since"
 
 # Master with an address on a WAN interface: healthy.
 #

migration/vrrp-wan-reconcile

@@ -51,6 +51,14 @@ fi
 
 # One writer. The 30s timer and a VRRP transition can fire together, and two
 # VyOS commits in flight on one box do not queue -- the second fails outright.
+#
+# The lock fd MUST be closed for children (`9>&-` on every call below). Entering
+# VyOS config mode spawns a long-lived unionfs-fuse for the session, and it
+# INHERITS this descriptor and never lets go -- `lsof /run/vrrp-wan.lock` showed
+# it held by unionfs-fuse with fd 9w. From the first config change onward every
+# later run lost the flock and exited 0 without doing anything, so the
+# reconciler looked healthy in the journal ("Finished") while quietly having
+# stopped reconciling. It is how a demoted router kept the WAN.
 exec 9>"$LOCK"
 flock -n 9 || exit 0
 
@@ -85,7 +93,7 @@ if holds_vip; then
     [ -f "$STATE/since" ] || date +%s > "$STATE/since"
     wan_disabled || ppp_disabled || exit 0
     logger -t vrrp-wan "MASTER with WAN disabled -> enabling bond0.${WAN_VIF} + pppoe0"
-    "$APPLY" enable
+    "$APPLY" enable 9>&-
 else
     echo backup > "$STATE/role"
     rm -f "$STATE/since"
@@ -95,7 +103,7 @@ else
     # new master, and the switch sends the ISP's replies to whichever port spoke
     # last -- the WAN-side twin of the eth2 incident.
     logger -t vrrp-wan "not MASTER but WAN enabled -> releasing bond0.${WAN_VIF} + pppoe0"
-    "$APPLY" disable
+    "$APPLY" disable 9>&-
 fi
 
 # Deliberately no `save`. config.boot keeps `disable` on BOTH routers, so a