labsim: PPPoE HA passes the matrix, and the health check had a real flap bug

Full run green: baseline (one AC session, held by the VIP holder), clean
failover (pppoe0 moves in 26s, old master releases), 10 gig down (route falls to
pppoe0 and the LAN is back online in 5s), lease expiry (the guard hangs up), and
a missing peers file (NRestarts=0, no loop). Evidence in
labsim/wan-failover-evidence/.

The 10 gig test found a genuine bug in vrrp-wan-health, not in the sim. GRACE
was measured from PROMOTION, so an established master had no grace at all --
after hours of uptime `now - since` far exceeds any window. The first moment
bond0.53 went down while pppoe0 was mid-redial, the master failed its own check,
shed every VIP, and the peer inherited the same WAN outage and did the same. A
brief WAN blip would have flapped the production pair. The stamp is now
refreshed on every healthy tick, so grace measures time since the box last
demonstrably HAD a WAN -- survivable wherever the gap happens, not only just
after a promotion.

Three harness bugs, all the same shape, all of which produced a confident wrong
answer before being caught:

  - waiting for "exactly one pppoe0 holder" returns INSTANTLY during a handover,
    because it was already true. The useful question is who holds it.
  - judging connectivity on a single ping 20s after a link drop reported an
    outage that had already healed. Poll, do not sample.
  - `-o PreferredAuthentications=password` suits the routers but not the Alpine
    LAN VMs, whose sshd offers keyboard-interactive: ssh exited 255 before
    running anything and the test read that as "the LAN lost the internet". A
    tcpdump on the router showed the pings leaving pppoe0 NATed to
    198.51.100.117 and the replies coming back the whole time. An exit code that
    can mean "the network is broken" or "I could not log in" is not a
    connectivity test, so the check now asserts on what the guest reported.

That last one is why the harness asks the routers and the access concentrator
rather than a client, and why it refuses to run at all while either router still
has a default route via eth2 -- the libvirt-NAT scaffold answers connectivity
checks the WAN under test would have failed.

Still to run: hard failover (destroy the master), and the session-control
replace/deny/disable axis that brackets Vodafone's unknown behaviour and sets
the final GRACE. Nothing applied to production.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH

labsim/README.md

@@ -143,6 +143,67 @@ unreserved MAC gets an unreserved address.
 - **http://localhost:9101/metrics** — `labsim_reachable{src,dst,proto}` and
   `labsim_rtt_ms{src,dst}`.
 
+## WAN follows VRRP, and the PPPoE half of it
+
+One consumer ISP account, two routers. The 10 gig line's lease is bound to a
+cloned MAC and the Vodafone line to a single credential, so neither can be live
+on both boxes: the WAN has to move with mastership.
+
+The two halves use different control planes, and that asymmetry is the design:
+
+| | plane | why |
+|---|---|---|
+| `bond0.53` | VyOS **config** (`disable`) | only config can move a MAC |
+| `pppoe0` | **systemd** unit gate | see below |
+
+`set interfaces pppoe pppoe0 disable` cannot work as a resting state.
+`interfaces_pppoe.py` treats `disable` and `delete` identically and **unlinks
+`/etc/ppp/peers/pppoe0`** — which is pppd's own options file. The resting state
+therefore destroyed what the promotion path needed, and `ppp@pppoe0`
+restart-looped against it (47 restarts, zero sessions at the AC). It also makes
+op-mode `connect interface pppoe0` refuse, and puts every failover behind a
+priority-322 commit where one unrelated bad node fails the lot.
+
+So `pppoe0` is configured identically and **enabled on both**, and dialling is
+gated by a drop-in:
+
+```ini
+ConditionPathExists=/run/vrrp-wan/may-dial
+ConditionPathExists=/etc/ppp/peers/pppoe0
+```
+
+`/run` is tmpfs, so the gate is shut at boot. That matters more than it looks:
+with the node enabled, `interfaces_pppoe.py` restarts ppp on **every** commit
+touching the pppoe subtree when the daemon isn't running — so the backup
+actively tries to dial whenever anything commits. The gate is the only thing
+making that a no-op, which is why the reconciler refuses to bless a box whose
+drop-in is missing (`/etc` is per-image; a VyOS upgrade would silently remove
+the protection).
+
+`may-dial` is a **lease**, not a flag: `ConditionPathExists` is evaluated at
+start only, so it can prevent a dial but never revoke one. `vrrp-wan-reconcile`
+renews it every 30s; `vrrp-wan-guard` runs every 5s and only ever revokes.
+
+### Testing it
+
+```sh
+./labsim-pppoe-ha-test.sh --all
+```
+
+The verdict is what the **routers** and the **access concentrator** did, never
+what a client happened to get — and the harness refuses to run at all while a
+router still has a default route via `eth2`, because the libvirt-NAT scaffold
+answers connectivity checks that the WAN under test would have failed. The
+invariant it enforces throughout: *the AC never reports two `simdsl` sessions,
+and no two routers ever hold `pppoe0`.*
+
+Two failures the harness itself produced, both worth remembering: waiting for
+"exactly one holder" returns instantly during a handover (it was already true),
+and judging connectivity on a single ping 20s after a link drop reports an
+outage that has already healed. Ask **who** holds it, and poll.
+
+Evidence in `wan-failover-evidence/`.
+
 ## Routing: BGP, dual WAN, and the ISP VMs
 
 `sim-ha-config.py` covers the LAN side of the routers. `sim-net-config.py`

labsim/labsim-pppoe-ha-test.sh

@@ -0,0 +1,230 @@
+#!/bin/bash
+# Does the WAN follow VRRP mastership, and does exactly ONE router ever hold the
+# ISP session?
+#
+# The question is not "did a client get internet". A client can be answered by
+# the wrong path entirely -- for months labsim-vyos's only default route was the
+# libvirt-NAT scaffold on eth2, so every "the LAN still has internet" verdict was
+# answered by eth2 rather than by the WAN under test. This script therefore
+# refuses to run while that is true, and asks its questions of the ROUTERS and
+# the ACCESS CONCENTRATOR, which cannot be answered by accident.
+#
+# The invariant, checked continuously and independently of any individual test:
+#
+#     the AC never reports two `simdsl` sessions, and no two routers ever have a
+#     pppoe0 interface at the same time
+#
+# A run that violates it FAILS regardless of its own verdict, because a single
+# consumer credential is the whole constraint the design exists to satisfy.
+#
+#   ./labsim-pppoe-ha-test.sh --list
+#   ./labsim-pppoe-ha-test.sh T3
+#   ./labsim-pppoe-ha-test.sh --all
+set -uo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+R1="${R1:-172.31.1.252}"; R2="${R2:-172.31.1.253}"
+ISP="${ISP:-192.168.122.63}"          # the fake access concentrator
+LANVM="${LANVM:-172.31.10.10}"
+VIP="${VIP:-172.31.1.1}"
+PW="${VYOS_PW:-vyos}"; LANPW="${LANPW:-labsim}"
+EVID="$SCRIPT_DIR/wan-failover-evidence"
+
+SSH=(-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null
+     -o LogLevel=ERROR -o ConnectTimeout=6 -o PreferredAuthentications=password)
+r()   { timeout 45 sshpass -p "$PW" ssh "${SSH[@]}" "vyos@$1" "${@:2}" 2>/dev/null; }
+isp() { timeout 30 sshpass -p "$PW" ssh "${SSH[@]}" "vyos@$ISP" "$@" 2>/dev/null; }
+# The LAN VMs are Alpine and their sshd offers keyboard-interactive, not
+# `password`. Reusing the routers' option set here made ssh exit 255 BEFORE
+# running anything, and T5 read that as "the LAN lost the internet" while a
+# tcpdump on the router showed the pings flowing out pppoe0 and the replies
+# coming back. An exit code that can mean "the network is broken" or "I could
+# not log in" is not a connectivity test.
+LAN_SSH=(-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null
+         -o LogLevel=ERROR -o ConnectTimeout=6)
+lan() { timeout 45 sshpass -p "$LANPW" ssh "${LAN_SSH[@]}" "root@$LANVM" "$@" 2>/dev/null; }
+
+# Assert on what the guest actually reported, not on ssh's exit status.
+lan_online() { [ "$(lan 'ping -c2 -W3 9.9.9.9 >/dev/null 2>&1 && echo ONLINE')" = ONLINE ]; }
+
+log()  { printf '\033[36m==>\033[0m %s\n' "$*"; }
+pass() { printf '  \033[32mPASS\033[0m %s\n' "$*"; }
+fail() { printf '  \033[31mFAIL\033[0m %s\n' "$*"; FAILED=$((FAILED+1)); }
+FAILED=0
+
+# --- observations ----------------------------------------------------------
+ac_sessions() { isp '/opt/vyatta/bin/vyatta-op-cmd-wrapper show pppoe-server sessions' \
+                  | grep -c ' simdsl ' || true; }
+ac_detail()   { isp '/opt/vyatta/bin/vyatta-op-cmd-wrapper show pppoe-server sessions'; }
+ppp_on()      { r "$1" 'ip -4 addr show pppoe0 2>/dev/null | grep -c inet' | tr -d ' \n'; }
+holder()      { for h in "$R1" "$R2"; do
+                  [ "$(r "$h" "ip -4 -o addr show | grep -c ' ${VIP}/'" | tr -d ' \n')" != 0 ] \
+                    && { echo "$h"; return; }; done; echo none; }
+status()      { r "$1" 'sudo /config/vrrp-wan-reconcile --status'; }
+
+# How many routers currently hold a PPPoE interface. The invariant's other half.
+ppp_holders() { n=0; for h in "$R1" "$R2"; do
+                  [ "$(ppp_on "$h")" != 0 ] && n=$((n+1)); done; echo "$n"; }
+
+check_invariant() {
+    local s p ok=0
+    s="$(ac_sessions)"; p="$(ppp_holders)"
+    [ "${s:-0}" -le 1 ] || { fail "INVARIANT: AC reports $s simdsl sessions"; ok=1; }
+    [ "${p:-0}" -le 1 ] || { fail "INVARIANT: $p routers hold pppoe0"; ok=1; }
+    return $ok
+}
+
+# --- preconditions ---------------------------------------------------------
+# The scaffold check is a hard gate, not a warning. A default route via eth2
+# means the box can reach the internet without the WAN working at all, and every
+# connectivity verdict below would be a lie.
+preflight() {
+    log "preflight"
+    local rc=0
+    for h in "$R1" "$R2"; do
+        if r "$h" 'ip route show default' | grep -q 'dev eth2'; then
+            fail "$h still routes via eth2 (libvirt-NAT scaffold) -- run sim-net-config.py --drop-scaffold"
+            rc=1
+        fi
+        if [ "$(r "$h" '[ -f /etc/systemd/system/ppp@pppoe0.service.d/10-vrrp-wan-gate.conf ] && echo y')" != y ]; then
+            fail "$h is missing the ppp gate drop-in -- run migration/vrrp-wan-install"
+            rc=1
+        fi
+        [ "$(r "$h" 'systemctl is-active vrrp-wan-guard.timer')" = active ] \
+            || { fail "$h vrrp-wan-guard.timer not active"; rc=1; }
+    done
+    [ "$rc" -eq 0 ] && pass "scaffold dropped, gate present, guard running on both"
+    return $rc
+}
+
+settle() {  # wait until exactly one router holds pppoe0, or give up
+    local i
+    for i in $(seq 1 "${1:-24}"); do
+        [ "$(ppp_holders)" = 1 ] && return 0
+        sleep 5
+    done
+    return 1
+}
+
+# Wait until a SPECIFIC router holds pppoe0 and the other does not.
+#
+# The obvious `settle` is wrong for a failover: "exactly one holder" is already
+# true before the handover starts, so it returns instantly and the test reports
+# that nothing moved while the handover is still in flight. Asking who holds it
+# is the only useful form of the question.
+settle_on() {
+    local want="$1" other i
+    other=$([ "$want" = "$R1" ] && echo "$R2" || echo "$R1")
+    for i in $(seq 1 "${2:-30}"); do
+        [ "$(ppp_on "$want")" != 0 ] && [ "$(ppp_on "$other")" = 0 ] && return 0
+        sleep 5
+    done
+    return 1
+}
+
+save_evidence() {
+    local name="$1"; local d="$EVID/$name"; mkdir -p "$d"
+    { echo "=== $(date -Is) ==="; echo "--- AC sessions ---"; ac_detail
+      for h in "$R1" "$R2"; do echo "--- $h ---"; status "$h"
+        r "$h" 'ip -4 -br addr show pppoe0 2>/dev/null; ip route show default; sudo journalctl -t vrrp-wan -n 8 --no-pager'
+      done; } > "$d/state.txt" 2>&1
+    log "evidence -> wan-failover-evidence/$name/"
+}
+
+# --- tests -----------------------------------------------------------------
+T0() {  # baseline
+    log "T0  baseline: exactly one session, held by the VIP holder"
+    local h s; h="$(holder)"; s="$(ac_sessions)"
+    [ "$s" = 1 ] && pass "AC reports 1 session" || fail "AC reports $s sessions"
+    [ "$(ppp_on "$h")" != 0 ] && pass "the VIP holder ($h) is the one dialled" \
+        || fail "VIP holder $h has no pppoe0"
+    local other; other=$([ "$h" = "$R1" ] && echo "$R2" || echo "$R1")
+    [ "$(ppp_on "$other")" = 0 ] && pass "the backup ($other) is not dialled" \
+        || fail "backup $other also holds pppoe0"
+    save_evidence T0-baseline
+}
+
+T3() {  # clean, deliberate failover
+    log "T3  clean failover via force-fault"
+    local from to t0 t1; from="$(holder)"
+    to=$([ "$from" = "$R1" ] && echo "$R2" || echo "$R1")
+    log "    master=$from -> expecting $to"
+    t0=$(date +%s)
+    r "$from" 'sudo mkdir -p /run/vrrp-wan && sudo touch /run/vrrp-wan/force-fault'
+    if settle_on "$to" 30; then
+        t1=$(date +%s)
+        [ "$(ppp_on "$to")" != 0 ] && pass "pppoe0 moved to $to in $((t1-t0))s" \
+            || fail "pppoe0 did not move to $to"
+        [ "$(ppp_on "$from")" = 0 ] && pass "$from released pppoe0" \
+            || fail "$from still holds pppoe0"
+    else
+        fail "never settled to exactly one pppoe0 holder"
+    fi
+    check_invariant
+    save_evidence T3-clean-failover
+    r "$from" 'sudo rm -f /run/vrrp-wan/force-fault'
+    settle 30 >/dev/null
+}
+
+T5() {  # 10gig down -> PPPoE carries traffic
+    log "T5  10 gig down on the master: traffic must survive on pppoe0"
+    local h; h="$(holder)"
+    r "$h" 'sudo ip link set bond0.53 down'
+    sleep 20
+    local via; via="$(r "$h" 'ip route show default' | head -1)"
+    if echo "$via" | grep -q pppoe0; then
+        pass "default route moved to pppoe0: $via"
+    else
+        fail "default route did not move to pppoe0: ${via:-<none>}"
+    fi
+    # Poll, do not sample. Judging connectivity on one ping 20s after the link
+    # dropped failed while the path was still reconverging, and reported "the LAN
+    # lost the internet" for a path that came back moments later. A single
+    # negative sample is the least trustworthy verdict this harness can produce.
+    local ok=no i
+    for i in $(seq 1 12); do
+        lan_online && { ok=yes; break; }
+        sleep 5
+    done
+    [ "$ok" = yes ] && pass "LAN reaches the internet over pppoe0 (after $((i*5))s)" \
+        || fail "LAN never regained the internet with only pppoe0 up (60s)"
+    save_evidence T5-tengig-down
+    r "$h" 'sudo ip link set bond0.53 up'
+    sleep 20
+}
+
+T11() { # a blessed box with no peers file must not restart-loop
+    log "T11 missing peers file must not restart-loop"
+    local h; h="$(holder)"
+    r "$h" 'sudo mv /etc/ppp/peers/pppoe0 /tmp/peers.bak; sudo systemctl restart ppp@pppoe0'
+    sleep 12
+    local n; n="$(r "$h" 'systemctl show ppp@pppoe0 -p NRestarts --value')"
+    [ "${n:-99}" -le 1 ] && pass "NRestarts=$n (gate refused the start)" \
+        || fail "NRestarts=$n -- restart loop is back"
+    r "$h" 'sudo mv /tmp/peers.bak /etc/ppp/peers/pppoe0'
+    save_evidence T11-no-peers-file
+    settle 24 >/dev/null
+}
+
+T8() {  # lease expiry: the guard must hang up a demoted-but-unreconciled box
+    log "T8  lease expiry revokes the session"
+    local h; h="$(holder)"
+    r "$h" 'sudo systemctl stop vrrp-wan-reconcile.timer'
+    r "$h" 'sudo touch -d "-200 seconds" /run/vrrp-wan/may-dial'
+    sleep 12
+    [ "$(ppp_on "$h")" = 0 ] && pass "guard hung up on a stale lease" \
+        || fail "stale lease did not revoke the session"
+    r "$h" 'sudo systemctl start vrrp-wan-reconcile.timer'
+    save_evidence T8-lease-expiry
+    settle 24 >/dev/null
+}
+
+case "${1:---all}" in
+  --list) echo "T0 baseline | T3 clean failover | T5 10gig-down | T8 lease expiry | T11 no-peers-file"; exit 0 ;;
+  --all)  preflight || exit 1; T0; T3; T5; T8; T11 ;;
+  *)      preflight || exit 1; "$1" ;;
+esac
+
+echo
+[ "$FAILED" -eq 0 ] && { echo "ALL PASS"; exit 0; }
+echo "$FAILED check(s) FAILED"; exit 1

labsim/wan-failover-evidence/T0-baseline/state.txt

@@ -0,0 +1,27 @@
+=== 2026-09-05T19:07:59+01:00 ===
+--- AC sessions ---
+ ifname | username |       ip       | ip6 | ip6-dp |    calling-sid    | rate-limit | state  |  uptime  | rx-bytes | tx-bytes 
+--------+----------+----------------+-----+--------+-------------------+------------+--------+----------+----------+----------
+ ppp0   | simdsl   | 198.51.100.117 |     |        | 52:54:00:e5:95:a2 |            | active | 00:01:31 | 1.0 KiB  | 792 B
+--- 172.31.1.252 ---
+vip=172.31.1.1 holds_vip=yes wan_disabled=no wan_up=yes ppp_up=yes ppp_active=yes may_dial=yes lease_age=0 dropin=yes role=master
+pppoe0           UNKNOWN        198.51.100.117 peer 198.51.100.1/32 
+default via 203.0.113.1 dev bond0.53 proto failover metric 1 
+Sep 05 18:01:26 apitest vrrp-wan[2994197]: not MASTER but bond0.53 enabled -> releasing
+Sep 05 18:01:30 apitest vrrp-wan[2995085]: bond0.53 disable commit took 4s
+Sep 05 18:03:25 apitest vrrp-wan[3000708]: MASTER: dialling pppoe0
+Sep 05 18:03:25 apitest vrrp-wan[3000872]: MASTER with bond0.53 disabled -> enabling
+Sep 05 18:03:29 apitest vrrp-wan[3001568]: bond0.53 enable commit took 4s
+Sep 05 18:05:43 apitest vrrp-wan[3005097]: GUARD: lease stale (204s > 75s; is vrrp-wan-reconcile.timer running?) -- hanging up pppoe0
+Sep 05 18:05:53 apitest vrrp-wan[3005490]: MASTER: dialling pppoe0
+Sep 05 18:06:24 apitest vrrp-wan[3007095]: MASTER: dialling pppoe0
+--- 172.31.1.253 ---
+vip=172.31.1.1 holds_vip=no wan_disabled=yes wan_up=no ppp_up=no ppp_active=no may_dial=no lease_age=- dropin=yes role=backup
+Sep 05 17:57:52 vyos vrrp-wan[2797572]: not MASTER but bond0.53 enabled -> releasing
+Sep 05 17:58:16 vyos vrrp-wan[2798935]: bond0.53 disable commit took 24s
+Sep 05 18:01:03 vyos vrrp-wan[2802338]: MASTER: dialling pppoe0
+Sep 05 18:01:03 vyos vrrp-wan[2802489]: MASTER with bond0.53 disabled -> enabling
+Sep 05 18:01:07 vyos vrrp-wan[2803365]: bond0.53 enable commit took 4s
+Sep 05 18:03:02 vyos vrrp-wan[2808661]: not MASTER: hanging up pppoe0
+Sep 05 18:03:02 vyos vrrp-wan[2808822]: not MASTER but bond0.53 enabled -> releasing
+Sep 05 18:03:26 vyos vrrp-wan[2810178]: bond0.53 disable commit took 24s

labsim/wan-failover-evidence/T11-no-peers-file/state.txt

@@ -0,0 +1,25 @@
+=== 2026-09-05T19:10:28+01:00 ===
+--- AC sessions ---
+ ifname | username | ip | ip6 | ip6-dp | calling-sid | rate-limit | state | uptime | rx-bytes | tx-bytes 
+--------+----------+----+-----+--------+-------------+------------+-------+--------+----------+----------
+--- 172.31.1.252 ---
+vip=172.31.1.1 holds_vip=no wan_disabled=yes wan_up=no ppp_up=no ppp_active=no may_dial=no lease_age=- dropin=yes role=backup
+Sep 05 18:03:25 apitest vrrp-wan[3000872]: MASTER with bond0.53 disabled -> enabling
+Sep 05 18:03:29 apitest vrrp-wan[3001568]: bond0.53 enable commit took 4s
+Sep 05 18:05:43 apitest vrrp-wan[3005097]: GUARD: lease stale (204s > 75s; is vrrp-wan-reconcile.timer running?) -- hanging up pppoe0
+Sep 05 18:05:53 apitest vrrp-wan[3005490]: MASTER: dialling pppoe0
+Sep 05 18:06:24 apitest vrrp-wan[3007095]: MASTER: dialling pppoe0
+Sep 05 18:08:15 apitest vrrp-wan[3010461]: not MASTER: hanging up pppoe0
+Sep 05 18:08:16 apitest vrrp-wan[3010622]: not MASTER but bond0.53 enabled -> releasing
+Sep 05 18:08:40 apitest vrrp-wan[3012085]: bond0.53 disable commit took 24s
+--- 172.31.1.253 ---
+vip=172.31.1.1 holds_vip=yes wan_disabled=no wan_up=yes ppp_up=no ppp_active=yes may_dial=yes lease_age=0 dropin=yes role=master
+default via 203.0.113.1 dev bond0.53 proto failover metric 1 
+Sep 05 18:03:02 vyos vrrp-wan[2808822]: not MASTER but bond0.53 enabled -> releasing
+Sep 05 18:03:26 vyos vrrp-wan[2810178]: bond0.53 disable commit took 24s
+Sep 05 18:07:53 vyos vrrp-wan[2815869]: MASTER: dialling pppoe0
+Sep 05 18:07:53 vyos vrrp-wan[2816022]: MASTER with bond0.53 disabled -> enabling
+Sep 05 18:07:57 vyos vrrp-wan[2816731]: bond0.53 enable commit took 4s
+Sep 05 18:09:10 vyos vrrp-wan[2819736]: GUARD: lease stale (204s > 75s; is vrrp-wan-reconcile.timer running?) -- hanging up pppoe0
+Sep 05 18:09:34 vyos vrrp-wan[2820820]: MASTER: dialling pppoe0
+Sep 05 18:10:04 vyos vrrp-wan[2821942]: MASTER: dialling pppoe0

labsim/wan-failover-evidence/T3-clean-failover/state.txt

@@ -0,0 +1,28 @@
+=== 2026-09-05T19:08:34+01:00 ===
+--- AC sessions ---
+ ifname | username |       ip       | ip6 | ip6-dp |    calling-sid    | rate-limit | state  |  uptime  | rx-bytes | tx-bytes 
+--------+----------+----------------+-----+--------+-------------------+------------+--------+----------+----------+----------
+ ppp0   | simdsl   | 198.51.100.118 |     |        | 52:54:00:4e:0b:56 |            | active | 00:00:15 | 590 B    | 280 B
+--- 172.31.1.252 ---
+vip=172.31.1.1 holds_vip=no wan_disabled=no wan_up=yes ppp_up=no ppp_active=no may_dial=no lease_age=- dropin=yes role=backup
+default via 203.0.113.1 dev bond0.53 proto failover metric 1 
+Sep 05 18:03:25 apitest vrrp-wan[3000708]: MASTER: dialling pppoe0
+Sep 05 18:03:25 apitest vrrp-wan[3000872]: MASTER with bond0.53 disabled -> enabling
+Sep 05 18:03:29 apitest vrrp-wan[3001568]: bond0.53 enable commit took 4s
+Sep 05 18:05:43 apitest vrrp-wan[3005097]: GUARD: lease stale (204s > 75s; is vrrp-wan-reconcile.timer running?) -- hanging up pppoe0
+Sep 05 18:05:53 apitest vrrp-wan[3005490]: MASTER: dialling pppoe0
+Sep 05 18:06:24 apitest vrrp-wan[3007095]: MASTER: dialling pppoe0
+Sep 05 18:08:15 apitest vrrp-wan[3010461]: not MASTER: hanging up pppoe0
+Sep 05 18:08:16 apitest vrrp-wan[3010622]: not MASTER but bond0.53 enabled -> releasing
+--- 172.31.1.253 ---
+vip=172.31.1.1 holds_vip=yes wan_disabled=no wan_up=yes ppp_up=yes ppp_active=yes may_dial=yes lease_age=13 dropin=yes role=master
+pppoe0           UNKNOWN        198.51.100.118 peer 198.51.100.1/32 
+default via 203.0.113.1 dev bond0.53 proto failover metric 1 
+Sep 05 18:01:03 vyos vrrp-wan[2802489]: MASTER with bond0.53 disabled -> enabling
+Sep 05 18:01:07 vyos vrrp-wan[2803365]: bond0.53 enable commit took 4s
+Sep 05 18:03:02 vyos vrrp-wan[2808661]: not MASTER: hanging up pppoe0
+Sep 05 18:03:02 vyos vrrp-wan[2808822]: not MASTER but bond0.53 enabled -> releasing
+Sep 05 18:03:26 vyos vrrp-wan[2810178]: bond0.53 disable commit took 24s
+Sep 05 18:07:53 vyos vrrp-wan[2815869]: MASTER: dialling pppoe0
+Sep 05 18:07:53 vyos vrrp-wan[2816022]: MASTER with bond0.53 disabled -> enabling
+Sep 05 18:07:57 vyos vrrp-wan[2816731]: bond0.53 enable commit took 4s

labsim/wan-failover-evidence/T5-tengig-down/state.txt

@@ -0,0 +1,27 @@
+=== 2026-09-05T19:09:05+01:00 ===
+--- AC sessions ---
+ ifname | username |       ip       | ip6 | ip6-dp |    calling-sid    | rate-limit | state  |  uptime  | rx-bytes | tx-bytes 
+--------+----------+----------------+-----+--------+-------------------+------------+--------+----------+----------+----------
+ ppp0   | simdsl   | 198.51.100.118 |     |        | 52:54:00:4e:0b:56 |            | active | 00:00:46 | 758 B    | 448 B
+--- 172.31.1.252 ---
+vip=172.31.1.1 holds_vip=no wan_disabled=yes wan_up=no ppp_up=no ppp_active=no may_dial=no lease_age=- dropin=yes role=backup
+Sep 05 18:03:25 apitest vrrp-wan[3000872]: MASTER with bond0.53 disabled -> enabling
+Sep 05 18:03:29 apitest vrrp-wan[3001568]: bond0.53 enable commit took 4s
+Sep 05 18:05:43 apitest vrrp-wan[3005097]: GUARD: lease stale (204s > 75s; is vrrp-wan-reconcile.timer running?) -- hanging up pppoe0
+Sep 05 18:05:53 apitest vrrp-wan[3005490]: MASTER: dialling pppoe0
+Sep 05 18:06:24 apitest vrrp-wan[3007095]: MASTER: dialling pppoe0
+Sep 05 18:08:15 apitest vrrp-wan[3010461]: not MASTER: hanging up pppoe0
+Sep 05 18:08:16 apitest vrrp-wan[3010622]: not MASTER but bond0.53 enabled -> releasing
+Sep 05 18:08:40 apitest vrrp-wan[3012085]: bond0.53 disable commit took 24s
+--- 172.31.1.253 ---
+vip=172.31.1.1 holds_vip=yes wan_disabled=no wan_up=yes ppp_up=yes ppp_active=yes may_dial=yes lease_age=10 dropin=yes role=master
+pppoe0           UNKNOWN        198.51.100.118 peer 198.51.100.1/32 
+default nhid 483 dev pppoe0 proto static metric 20 
+Sep 05 18:01:03 vyos vrrp-wan[2802489]: MASTER with bond0.53 disabled -> enabling
+Sep 05 18:01:07 vyos vrrp-wan[2803365]: bond0.53 enable commit took 4s
+Sep 05 18:03:02 vyos vrrp-wan[2808661]: not MASTER: hanging up pppoe0
+Sep 05 18:03:02 vyos vrrp-wan[2808822]: not MASTER but bond0.53 enabled -> releasing
+Sep 05 18:03:26 vyos vrrp-wan[2810178]: bond0.53 disable commit took 24s
+Sep 05 18:07:53 vyos vrrp-wan[2815869]: MASTER: dialling pppoe0
+Sep 05 18:07:53 vyos vrrp-wan[2816022]: MASTER with bond0.53 disabled -> enabling
+Sep 05 18:07:57 vyos vrrp-wan[2816731]: bond0.53 enable commit took 4s

labsim/wan-failover-evidence/T8-lease-expiry/state.txt

@@ -0,0 +1,25 @@
+=== 2026-09-05T19:09:47+01:00 ===
+--- AC sessions ---
+ ifname | username | ip | ip6 | ip6-dp | calling-sid | rate-limit | state | uptime | rx-bytes | tx-bytes 
+--------+----------+----+-----+--------+-------------+------------+-------+--------+----------+----------
+--- 172.31.1.252 ---
+vip=172.31.1.1 holds_vip=no wan_disabled=yes wan_up=no ppp_up=no ppp_active=no may_dial=no lease_age=- dropin=yes role=backup
+Sep 05 18:03:25 apitest vrrp-wan[3000872]: MASTER with bond0.53 disabled -> enabling
+Sep 05 18:03:29 apitest vrrp-wan[3001568]: bond0.53 enable commit took 4s
+Sep 05 18:05:43 apitest vrrp-wan[3005097]: GUARD: lease stale (204s > 75s; is vrrp-wan-reconcile.timer running?) -- hanging up pppoe0
+Sep 05 18:05:53 apitest vrrp-wan[3005490]: MASTER: dialling pppoe0
+Sep 05 18:06:24 apitest vrrp-wan[3007095]: MASTER: dialling pppoe0
+Sep 05 18:08:15 apitest vrrp-wan[3010461]: not MASTER: hanging up pppoe0
+Sep 05 18:08:16 apitest vrrp-wan[3010622]: not MASTER but bond0.53 enabled -> releasing
+Sep 05 18:08:40 apitest vrrp-wan[3012085]: bond0.53 disable commit took 24s
+--- 172.31.1.253 ---
+vip=172.31.1.1 holds_vip=yes wan_disabled=no wan_up=yes ppp_up=no ppp_active=no may_dial=no lease_age=- dropin=yes role=master
+default via 203.0.113.1 dev bond0.53 proto failover metric 1 
+Sep 05 18:01:07 vyos vrrp-wan[2803365]: bond0.53 enable commit took 4s
+Sep 05 18:03:02 vyos vrrp-wan[2808661]: not MASTER: hanging up pppoe0
+Sep 05 18:03:02 vyos vrrp-wan[2808822]: not MASTER but bond0.53 enabled -> releasing
+Sep 05 18:03:26 vyos vrrp-wan[2810178]: bond0.53 disable commit took 24s
+Sep 05 18:07:53 vyos vrrp-wan[2815869]: MASTER: dialling pppoe0
+Sep 05 18:07:53 vyos vrrp-wan[2816022]: MASTER with bond0.53 disabled -> enabling
+Sep 05 18:07:57 vyos vrrp-wan[2816731]: bond0.53 enable commit took 4s
+Sep 05 18:09:10 vyos vrrp-wan[2819736]: GUARD: lease stale (204s > 75s; is vrrp-wan-reconcile.timer running?) -- hanging up pppoe0

migration/vrrp-wan-health

@@ -115,15 +115,25 @@ mkdir -p "$STATE" 2>/dev/null
 for ifc in bond0.53 pppoe0; do
     if ip -4 addr show dev "$ifc" 2>/dev/null | grep -q 'inet '; then
         echo "$ifc" > "$STATE/wan" 2>/dev/null
+        # Re-stamp on every healthy tick, so the grace window below measures
+        # time since this box last DEMONSTRABLY had a WAN.
+        date +%s > "$STATE/since" 2>/dev/null
         exit 0
     fi
 done
 rm -f "$STATE/wan" 2>/dev/null
 
-# Master, no WAN yet, still within the grace window: DHCP negotiation and PPPoE
+# No WAN right now, but there was one within GRACE: ride it out.
-# dial-up take real time, and the ISP has to accept the cloned MAC arriving on a
+#
-# different port. Failing here would demote the new master before it ever had a
+# `since` is re-stamped on every healthy tick, so this measures time since the
-# chance, and hand the VIPs straight back -- a flap, not a failover.
+# box last HAD a WAN -- not time since it was promoted. Measuring from promotion
+# was wrong in a way that only shows up on an established master: after hours of
+# uptime `now - since` far exceeds any grace, so the first moment bond0.53 went
+# down and pppoe0 was mid-redial, the master failed its own check, shed every
+# VIP, and the peer -- inheriting the same WAN outage -- did the same. Observed
+# in labsim: taking the 10 gig down flapped the pair instead of falling back to
+# PPPoE. A WAN gap must be survivable wherever it happens, not only just after a
+# promotion.
 since=$(cat "$STATE/since" 2>/dev/null || echo 0)
 [ $(( $(date +%s) - since )) -lt "$GRACE" ] && exit 0