vyos: a failover you can actually trigger, and four bugs found triggering it

A planned failover had no reliable lever. VyOS offers only `restart vrrp`, and
neither that nor `systemctl restart keepalived` is dependable: with advert_int 1
the peer declares the master dead after ~3.6s and a restart usually finishes
inside that window. Measured -- the same command moved mastership on one run and
not on the next three. A fail-back step you cannot trigger on purpose is not a
procedure, and the recovery card depended on one.

The lever is now `touch /run/vrrp-wan/force-fault`: the health check fails, the
sync group sheds every VIP, and the peer takes over. It exercises the same path
a real WAN loss takes rather than a special case, and it lives in /run so a
reboot cannot leave a router permanently ineligible.

Proven end to end in labsim: lever -> mastership moves -> WAN follows -> the old
master releases -> a LAN VM has internet -> the faulted router returns to BACKUP
and is eligible again.

Getting there exposed four real bugs, two of which would have broken a GENUINE
failover, not just the drill:

  - The grace stamp was written only by the 30s reconciler, so a freshly
    promoted master reached the 5s health check with no stamp, scored grace = 0,
    failed instantly and went FAULT. With the peer already faulted that left
    BOTH routers in FAULT and the LAN with no gateway at all -- worse than the
    outage the check exists to prevent. The check now stamps on promotion.
  - And it inherited STALE stamps from an earlier mastership, failing ~5s after
    passing. The stamp is now cleared on the way down, by the health check
    itself, not only by the reconciler.
  - The lock fd leaked into VyOS's config session: `exec 9>` is inherited by the
    long-lived unionfs-fuse the session spawns, which never closes it. From the
    first config change on, every later reconciler run lost the flock and exited
    0 having done nothing -- healthy-looking journal, silently stopped
    reconciling. That is how a demoted router kept the WAN. Children now get 9>&-.
  - vrrp-wan-apply touched pppoe0 unconditionally. On a box where pppoe0 has no
    source-interface VyOS rejects the whole commit ("Physical source-interface
    required"), taking the bond0.53 change down with it -- and the script still
    returned 0, so the reconciler logged a release that never happened. pppoe0 is
    now guarded on existence and the commit's verdict is propagated.

Still NOT applied to production. The pair is single-homed on WAN until it is.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH

migration/vrrp-wan-health

@@ -32,6 +32,28 @@ STATE=/run/vrrp-wan
 GRACE=90          # seconds a new master gets to complete DHCP / PPPoE dial-up
 VIP="${VRRP_WAN_VIP:-192.168.1.1}"
 
+# A deliberate hand-over lever.
+#
+# There is no reliable way to MAKE this pair fail over on demand. VyOS offers
+# only `restart vrrp`, and neither that nor `systemctl restart keepalived` is
+# dependable: with advert_int 1 the peer declares the master dead after ~3.6s,
+# and a restart usually finishes inside that window. Measured in labsim -- the
+# same command moved mastership on one run and not on the next three. A
+# fail-back procedure you cannot trigger on purpose is not a procedure.
+#
+# Failing the health check IS the supported way to shed mastership: the sync
+# group goes FAULT, releases every VIP, and the peer takes over -- the same path
+# a genuine WAN loss takes, so the planned drill exercises the real mechanism
+# rather than a special case.
+#
+#   touch /run/vrrp-wan/force-fault    hand over within failure-count*interval
+#   rm    /run/vrrp-wan/force-fault    become eligible again (no-preempt keeps
+#                                      it BACKUP until the peer hands back)
+#
+# It lives in /run deliberately: a reboot clears it, so a forgotten drill cannot
+# leave a router permanently ineligible.
+[ -f /run/vrrp-wan/force-fault ] && exit 1
+
 # Am I holding the VIPs? Asked of REALITY -- is the management VIP actually on
 # this box -- and not of a /run marker.
 #
@@ -42,7 +64,30 @@ VIP="${VRRP_WAN_VIP:-192.168.1.1}"
 # backup, passed this check, and sat holding every VIP with no WAN -- the exact
 # outage this script exists to prevent, re-created by trusting the reporter
 # instead of the fact.
-[ -n "$(ip -4 -o addr show 2>/dev/null | grep " ${VIP}/")" ] || exit 0
+if [ -z "$(ip -4 -o addr show 2>/dev/null | grep " ${VIP}/")" ]; then
+    # Clear the grace stamp on the way down, HERE, not only in the reconciler.
+    # The reconciler runs every 30s; this runs every 5s. A promotion that
+    # inherited a stamp from an earlier mastership scored grace = hours, failed
+    # immediately, and took the sync group to FAULT ~5s after passing -- with the
+    # peer already faulted, that left BOTH routers in FAULT and the LAN with no
+    # gateway. The stamp must belong to the CURRENT mastership or it is worse
+    # than useless.
+    rm -f "$STATE/since" 2>/dev/null
+    exit 0
+fi
+
+# Start the grace clock HERE, the moment mastership is first observed.
+#
+# It used to be stamped only by vrrp-wan-reconcile, which runs on a 30s timer --
+# so a freshly promoted master reached this check with no stamp, scored grace=0,
+# failed, and went FAULT before it had any chance to bring the WAN up. The peer
+# then found itself alone with no WAN either and did the same. Observed in
+# labsim: BOTH routers in FAULT, nobody holding the VIPs, the LAN with no
+# gateway at all. That is worse than the outage this script exists to prevent,
+# and it would have hit a REAL failover, not just a drill -- the health check
+# runs every 5s and the reconciler had not yet ticked.
+mkdir -p "$STATE" 2>/dev/null
+[ -f "$STATE/since" ] || date +%s > "$STATE/since"
 
 # Master with an address on a WAN interface: healthy.
 #