vyos: a failover you can actually trigger, and four bugs found triggering it

A planned failover had no reliable lever. VyOS offers only `restart vrrp`, and
neither that nor `systemctl restart keepalived` is dependable: with advert_int 1
the peer declares the master dead after ~3.6s and a restart usually finishes
inside that window. Measured -- the same command moved mastership on one run and
not on the next three. A fail-back step you cannot trigger on purpose is not a
procedure, and the recovery card depended on one.

The lever is now `touch /run/vrrp-wan/force-fault`: the health check fails, the
sync group sheds every VIP, and the peer takes over. It exercises the same path
a real WAN loss takes rather than a special case, and it lives in /run so a
reboot cannot leave a router permanently ineligible.

Proven end to end in labsim: lever -> mastership moves -> WAN follows -> the old
master releases -> a LAN VM has internet -> the faulted router returns to BACKUP
and is eligible again.

Getting there exposed four real bugs, two of which would have broken a GENUINE
failover, not just the drill:

  - The grace stamp was written only by the 30s reconciler, so a freshly
    promoted master reached the 5s health check with no stamp, scored grace = 0,
    failed instantly and went FAULT. With the peer already faulted that left
    BOTH routers in FAULT and the LAN with no gateway at all -- worse than the
    outage the check exists to prevent. The check now stamps on promotion.
  - And it inherited STALE stamps from an earlier mastership, failing ~5s after
    passing. The stamp is now cleared on the way down, by the health check
    itself, not only by the reconciler.
  - The lock fd leaked into VyOS's config session: `exec 9>` is inherited by the
    long-lived unionfs-fuse the session spawns, which never closes it. From the
    first config change on, every later reconciler run lost the flock and exited
    0 having done nothing -- healthy-looking journal, silently stopped
    reconciling. That is how a demoted router kept the WAN. Children now get 9>&-.
  - vrrp-wan-apply touched pppoe0 unconditionally. On a box where pppoe0 has no
    source-interface VyOS rejects the whole commit ("Physical source-interface
    required"), taking the bond0.53 change down with it -- and the script still
    returned 0, so the reconciler logged a release that never happened. pppoe0 is
    now guarded on existence and the commit's verdict is propagated.

Still NOT applied to production. The pair is single-homed on WAN until it is.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH

migration/vrrp-wan-reconcile

@@ -51,6 +51,14 @@ fi
 
 # One writer. The 30s timer and a VRRP transition can fire together, and two
 # VyOS commits in flight on one box do not queue -- the second fails outright.
+#
+# The lock fd MUST be closed for children (`9>&-` on every call below). Entering
+# VyOS config mode spawns a long-lived unionfs-fuse for the session, and it
+# INHERITS this descriptor and never lets go -- `lsof /run/vrrp-wan.lock` showed
+# it held by unionfs-fuse with fd 9w. From the first config change onward every
+# later run lost the flock and exited 0 without doing anything, so the
+# reconciler looked healthy in the journal ("Finished") while quietly having
+# stopped reconciling. It is how a demoted router kept the WAN.
 exec 9>"$LOCK"
 flock -n 9 || exit 0
 
@@ -85,7 +93,7 @@ if holds_vip; then
     [ -f "$STATE/since" ] || date +%s > "$STATE/since"
     wan_disabled || ppp_disabled || exit 0
     logger -t vrrp-wan "MASTER with WAN disabled -> enabling bond0.${WAN_VIF} + pppoe0"
-    "$APPLY" enable
+    "$APPLY" enable 9>&-
 else
     echo backup > "$STATE/role"
     rm -f "$STATE/since"
@@ -95,7 +103,7 @@ else
     # new master, and the switch sends the ISP's replies to whichever port spoke
     # last -- the WAN-side twin of the eth2 incident.
     logger -t vrrp-wan "not MASTER but WAN enabled -> releasing bond0.${WAN_VIF} + pppoe0"
-    "$APPLY" disable
+    "$APPLY" disable 9>&-
 fi
 
 # Deliberately no `save`. config.boot keeps `disable` on BOTH routers, so a