vyos: a failover you can actually trigger, and four bugs found triggering it

A planned failover had no reliable lever. VyOS offers only `restart vrrp`, and
neither that nor `systemctl restart keepalived` is dependable: with advert_int 1
the peer declares the master dead after ~3.6s and a restart usually finishes
inside that window. Measured -- the same command moved mastership on one run and
not on the next three. A fail-back step you cannot trigger on purpose is not a
procedure, and the recovery card depended on one.

The lever is now `touch /run/vrrp-wan/force-fault`: the health check fails, the
sync group sheds every VIP, and the peer takes over. It exercises the same path
a real WAN loss takes rather than a special case, and it lives in /run so a
reboot cannot leave a router permanently ineligible.

Proven end to end in labsim: lever -> mastership moves -> WAN follows -> the old
master releases -> a LAN VM has internet -> the faulted router returns to BACKUP
and is eligible again.

Getting there exposed four real bugs, two of which would have broken a GENUINE
failover, not just the drill:

  - The grace stamp was written only by the 30s reconciler, so a freshly
    promoted master reached the 5s health check with no stamp, scored grace = 0,
    failed instantly and went FAULT. With the peer already faulted that left
    BOTH routers in FAULT and the LAN with no gateway at all -- worse than the
    outage the check exists to prevent. The check now stamps on promotion.
  - And it inherited STALE stamps from an earlier mastership, failing ~5s after
    passing. The stamp is now cleared on the way down, by the health check
    itself, not only by the reconciler.
  - The lock fd leaked into VyOS's config session: `exec 9>` is inherited by the
    long-lived unionfs-fuse the session spawns, which never closes it. From the
    first config change on, every later reconciler run lost the flock and exited
    0 having done nothing -- healthy-looking journal, silently stopped
    reconciling. That is how a demoted router kept the WAN. Children now get 9>&-.
  - vrrp-wan-apply touched pppoe0 unconditionally. On a box where pppoe0 has no
    source-interface VyOS rejects the whole commit ("Physical source-interface
    required"), taking the bond0.53 change down with it -- and the script still
    returned 0, so the reconciler logged a release that never happened. pppoe0 is
    now guarded on existence and the commit's verdict is propagated.

Still NOT applied to production. The pair is single-homed on WAN until it is.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH

migration/vrrp-wan-apply

@@ -17,16 +17,32 @@ WAN_VIF=53
 cfg() { /opt/vyatta/bin/vyatta-op-cmd-wrapper show configuration commands 2>/dev/null; }
 wan_disabled(){ cfg | grep -q "vif ${WAN_VIF} disable"; }
 ppp_disabled(){ cfg | grep -q "pppoe pppoe0 disable"; }
+# Only touch pppoe0 if it is actually configured. `set interfaces pppoe pppoe0
+# disable` on a box that has no pppoe0 CREATES the node with nothing but
+# `disable`, and VyOS then refuses the commit with "Physical source-interface
+# required for pppoe0!" -- taking the bond0.53 change down with it, because one
+# invalid node fails the whole commit. Seen on the labsim secondary, which has
+# no PPPoE; production has it on both, so this would have been an untested path
+# that only ever ran during a failover.
+ppp_exists(){ cfg | grep -q "pppoe pppoe0 source-interface"; }
 
 configure
 if [ "$MODE" = enable ]; then
     # Guarded: `delete` of an absent node aborts the whole batch with
     # "Nothing to delete", which left the box detected-but-unfixed.
     wan_disabled && delete interfaces bonding bond0 vif ${WAN_VIF} disable
-    ppp_disabled && delete interfaces pppoe pppoe0 disable
+    ppp_exists && ppp_disabled && delete interfaces pppoe pppoe0 disable
 else
     wan_disabled || set interfaces bonding bond0 vif ${WAN_VIF} disable
-    ppp_disabled || set interfaces pppoe pppoe0 disable
+    ppp_exists && { ppp_disabled || set interfaces pppoe pppoe0 disable; }
+fi
+# Report the commit's verdict. The script previously ended on `exit` (a
+# script-template function) and returned 0 even after "Commit failed", so the
+# reconciler logged a successful release that had not happened -- the worst kind
+# of failure for something whose whole job is to keep two routers from holding
+# one WAN.
+if commit 2>&1 | tee /tmp/vrrp-wan-commit.log | grep -qi "commit failed"; then
+    logger -t vrrp-wan "COMMIT FAILED applying '$MODE' -- see /tmp/vrrp-wan-commit.log"
+    exit 1
 fi
-commit
 exit