diff --git a/labsim/README.md b/labsim/README.md index ecd6e40..24a7e1b 100644 --- a/labsim/README.md +++ b/labsim/README.md @@ -143,6 +143,67 @@ unreserved MAC gets an unreserved address. - **http://localhost:9101/metrics** — `labsim_reachable{src,dst,proto}` and `labsim_rtt_ms{src,dst}`. +## WAN follows VRRP, and the PPPoE half of it + +One consumer ISP account, two routers. The 10 gig line's lease is bound to a +cloned MAC and the Vodafone line to a single credential, so neither can be live +on both boxes: the WAN has to move with mastership. + +The two halves use different control planes, and that asymmetry is the design: + +| | plane | why | +|---|---|---| +| `bond0.53` | VyOS **config** (`disable`) | only config can move a MAC | +| `pppoe0` | **systemd** unit gate | see below | + +`set interfaces pppoe pppoe0 disable` cannot work as a resting state. +`interfaces_pppoe.py` treats `disable` and `delete` identically and **unlinks +`/etc/ppp/peers/pppoe0`** — which is pppd's own options file. The resting state +therefore destroyed what the promotion path needed, and `ppp@pppoe0` +restart-looped against it (47 restarts, zero sessions at the AC). It also makes +op-mode `connect interface pppoe0` refuse, and puts every failover behind a +priority-322 commit where one unrelated bad node fails the lot. + +So `pppoe0` is configured identically and **enabled on both**, and dialling is +gated by a drop-in: + +```ini +ConditionPathExists=/run/vrrp-wan/may-dial +ConditionPathExists=/etc/ppp/peers/pppoe0 +``` + +`/run` is tmpfs, so the gate is shut at boot. That matters more than it looks: +with the node enabled, `interfaces_pppoe.py` restarts ppp on **every** commit +touching the pppoe subtree when the daemon isn't running — so the backup +actively tries to dial whenever anything commits. The gate is the only thing +making that a no-op, which is why the reconciler refuses to bless a box whose +drop-in is missing (`/etc` is per-image; a VyOS upgrade would silently remove +the protection). + +`may-dial` is a **lease**, not a flag: `ConditionPathExists` is evaluated at +start only, so it can prevent a dial but never revoke one. `vrrp-wan-reconcile` +renews it every 30s; `vrrp-wan-guard` runs every 5s and only ever revokes. + +### Testing it + +```sh +./labsim-pppoe-ha-test.sh --all +``` + +The verdict is what the **routers** and the **access concentrator** did, never +what a client happened to get — and the harness refuses to run at all while a +router still has a default route via `eth2`, because the libvirt-NAT scaffold +answers connectivity checks that the WAN under test would have failed. The +invariant it enforces throughout: *the AC never reports two `simdsl` sessions, +and no two routers ever hold `pppoe0`.* + +Two failures the harness itself produced, both worth remembering: waiting for +"exactly one holder" returns instantly during a handover (it was already true), +and judging connectivity on a single ping 20s after a link drop reports an +outage that has already healed. Ask **who** holds it, and poll. + +Evidence in `wan-failover-evidence/`. + ## Routing: BGP, dual WAN, and the ISP VMs `sim-ha-config.py` covers the LAN side of the routers. `sim-net-config.py` diff --git a/labsim/labsim-pppoe-ha-test.sh b/labsim/labsim-pppoe-ha-test.sh new file mode 100755 index 0000000..2bce68c --- /dev/null +++ b/labsim/labsim-pppoe-ha-test.sh @@ -0,0 +1,230 @@ +#!/bin/bash +# Does the WAN follow VRRP mastership, and does exactly ONE router ever hold the +# ISP session? +# +# The question is not "did a client get internet". A client can be answered by +# the wrong path entirely -- for months labsim-vyos's only default route was the +# libvirt-NAT scaffold on eth2, so every "the LAN still has internet" verdict was +# answered by eth2 rather than by the WAN under test. This script therefore +# refuses to run while that is true, and asks its questions of the ROUTERS and +# the ACCESS CONCENTRATOR, which cannot be answered by accident. +# +# The invariant, checked continuously and independently of any individual test: +# +# the AC never reports two `simdsl` sessions, and no two routers ever have a +# pppoe0 interface at the same time +# +# A run that violates it FAILS regardless of its own verdict, because a single +# consumer credential is the whole constraint the design exists to satisfy. +# +# ./labsim-pppoe-ha-test.sh --list +# ./labsim-pppoe-ha-test.sh T3 +# ./labsim-pppoe-ha-test.sh --all +set -uo pipefail + +SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" +R1="${R1:-172.31.1.252}"; R2="${R2:-172.31.1.253}" +ISP="${ISP:-192.168.122.63}" # the fake access concentrator +LANVM="${LANVM:-172.31.10.10}" +VIP="${VIP:-172.31.1.1}" +PW="${VYOS_PW:-vyos}"; LANPW="${LANPW:-labsim}" +EVID="$SCRIPT_DIR/wan-failover-evidence" + +SSH=(-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null + -o LogLevel=ERROR -o ConnectTimeout=6 -o PreferredAuthentications=password) +r() { timeout 45 sshpass -p "$PW" ssh "${SSH[@]}" "vyos@$1" "${@:2}" 2>/dev/null; } +isp() { timeout 30 sshpass -p "$PW" ssh "${SSH[@]}" "vyos@$ISP" "$@" 2>/dev/null; } +# The LAN VMs are Alpine and their sshd offers keyboard-interactive, not +# `password`. Reusing the routers' option set here made ssh exit 255 BEFORE +# running anything, and T5 read that as "the LAN lost the internet" while a +# tcpdump on the router showed the pings flowing out pppoe0 and the replies +# coming back. An exit code that can mean "the network is broken" or "I could +# not log in" is not a connectivity test. +LAN_SSH=(-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null + -o LogLevel=ERROR -o ConnectTimeout=6) +lan() { timeout 45 sshpass -p "$LANPW" ssh "${LAN_SSH[@]}" "root@$LANVM" "$@" 2>/dev/null; } + +# Assert on what the guest actually reported, not on ssh's exit status. +lan_online() { [ "$(lan 'ping -c2 -W3 9.9.9.9 >/dev/null 2>&1 && echo ONLINE')" = ONLINE ]; } + +log() { printf '\033[36m==>\033[0m %s\n' "$*"; } +pass() { printf ' \033[32mPASS\033[0m %s\n' "$*"; } +fail() { printf ' \033[31mFAIL\033[0m %s\n' "$*"; FAILED=$((FAILED+1)); } +FAILED=0 + +# --- observations ---------------------------------------------------------- +ac_sessions() { isp '/opt/vyatta/bin/vyatta-op-cmd-wrapper show pppoe-server sessions' \ + | grep -c ' simdsl ' || true; } +ac_detail() { isp '/opt/vyatta/bin/vyatta-op-cmd-wrapper show pppoe-server sessions'; } +ppp_on() { r "$1" 'ip -4 addr show pppoe0 2>/dev/null | grep -c inet' | tr -d ' \n'; } +holder() { for h in "$R1" "$R2"; do + [ "$(r "$h" "ip -4 -o addr show | grep -c ' ${VIP}/'" | tr -d ' \n')" != 0 ] \ + && { echo "$h"; return; }; done; echo none; } +status() { r "$1" 'sudo /config/vrrp-wan-reconcile --status'; } + +# How many routers currently hold a PPPoE interface. The invariant's other half. +ppp_holders() { n=0; for h in "$R1" "$R2"; do + [ "$(ppp_on "$h")" != 0 ] && n=$((n+1)); done; echo "$n"; } + +check_invariant() { + local s p ok=0 + s="$(ac_sessions)"; p="$(ppp_holders)" + [ "${s:-0}" -le 1 ] || { fail "INVARIANT: AC reports $s simdsl sessions"; ok=1; } + [ "${p:-0}" -le 1 ] || { fail "INVARIANT: $p routers hold pppoe0"; ok=1; } + return $ok +} + +# --- preconditions --------------------------------------------------------- +# The scaffold check is a hard gate, not a warning. A default route via eth2 +# means the box can reach the internet without the WAN working at all, and every +# connectivity verdict below would be a lie. +preflight() { + log "preflight" + local rc=0 + for h in "$R1" "$R2"; do + if r "$h" 'ip route show default' | grep -q 'dev eth2'; then + fail "$h still routes via eth2 (libvirt-NAT scaffold) -- run sim-net-config.py --drop-scaffold" + rc=1 + fi + if [ "$(r "$h" '[ -f /etc/systemd/system/ppp@pppoe0.service.d/10-vrrp-wan-gate.conf ] && echo y')" != y ]; then + fail "$h is missing the ppp gate drop-in -- run migration/vrrp-wan-install" + rc=1 + fi + [ "$(r "$h" 'systemctl is-active vrrp-wan-guard.timer')" = active ] \ + || { fail "$h vrrp-wan-guard.timer not active"; rc=1; } + done + [ "$rc" -eq 0 ] && pass "scaffold dropped, gate present, guard running on both" + return $rc +} + +settle() { # wait until exactly one router holds pppoe0, or give up + local i + for i in $(seq 1 "${1:-24}"); do + [ "$(ppp_holders)" = 1 ] && return 0 + sleep 5 + done + return 1 +} + +# Wait until a SPECIFIC router holds pppoe0 and the other does not. +# +# The obvious `settle` is wrong for a failover: "exactly one holder" is already +# true before the handover starts, so it returns instantly and the test reports +# that nothing moved while the handover is still in flight. Asking who holds it +# is the only useful form of the question. +settle_on() { + local want="$1" other i + other=$([ "$want" = "$R1" ] && echo "$R2" || echo "$R1") + for i in $(seq 1 "${2:-30}"); do + [ "$(ppp_on "$want")" != 0 ] && [ "$(ppp_on "$other")" = 0 ] && return 0 + sleep 5 + done + return 1 +} + +save_evidence() { + local name="$1"; local d="$EVID/$name"; mkdir -p "$d" + { echo "=== $(date -Is) ==="; echo "--- AC sessions ---"; ac_detail + for h in "$R1" "$R2"; do echo "--- $h ---"; status "$h" + r "$h" 'ip -4 -br addr show pppoe0 2>/dev/null; ip route show default; sudo journalctl -t vrrp-wan -n 8 --no-pager' + done; } > "$d/state.txt" 2>&1 + log "evidence -> wan-failover-evidence/$name/" +} + +# --- tests ----------------------------------------------------------------- +T0() { # baseline + log "T0 baseline: exactly one session, held by the VIP holder" + local h s; h="$(holder)"; s="$(ac_sessions)" + [ "$s" = 1 ] && pass "AC reports 1 session" || fail "AC reports $s sessions" + [ "$(ppp_on "$h")" != 0 ] && pass "the VIP holder ($h) is the one dialled" \ + || fail "VIP holder $h has no pppoe0" + local other; other=$([ "$h" = "$R1" ] && echo "$R2" || echo "$R1") + [ "$(ppp_on "$other")" = 0 ] && pass "the backup ($other) is not dialled" \ + || fail "backup $other also holds pppoe0" + save_evidence T0-baseline +} + +T3() { # clean, deliberate failover + log "T3 clean failover via force-fault" + local from to t0 t1; from="$(holder)" + to=$([ "$from" = "$R1" ] && echo "$R2" || echo "$R1") + log " master=$from -> expecting $to" + t0=$(date +%s) + r "$from" 'sudo mkdir -p /run/vrrp-wan && sudo touch /run/vrrp-wan/force-fault' + if settle_on "$to" 30; then + t1=$(date +%s) + [ "$(ppp_on "$to")" != 0 ] && pass "pppoe0 moved to $to in $((t1-t0))s" \ + || fail "pppoe0 did not move to $to" + [ "$(ppp_on "$from")" = 0 ] && pass "$from released pppoe0" \ + || fail "$from still holds pppoe0" + else + fail "never settled to exactly one pppoe0 holder" + fi + check_invariant + save_evidence T3-clean-failover + r "$from" 'sudo rm -f /run/vrrp-wan/force-fault' + settle 30 >/dev/null +} + +T5() { # 10gig down -> PPPoE carries traffic + log "T5 10 gig down on the master: traffic must survive on pppoe0" + local h; h="$(holder)" + r "$h" 'sudo ip link set bond0.53 down' + sleep 20 + local via; via="$(r "$h" 'ip route show default' | head -1)" + if echo "$via" | grep -q pppoe0; then + pass "default route moved to pppoe0: $via" + else + fail "default route did not move to pppoe0: ${via:-}" + fi + # Poll, do not sample. Judging connectivity on one ping 20s after the link + # dropped failed while the path was still reconverging, and reported "the LAN + # lost the internet" for a path that came back moments later. A single + # negative sample is the least trustworthy verdict this harness can produce. + local ok=no i + for i in $(seq 1 12); do + lan_online && { ok=yes; break; } + sleep 5 + done + [ "$ok" = yes ] && pass "LAN reaches the internet over pppoe0 (after $((i*5))s)" \ + || fail "LAN never regained the internet with only pppoe0 up (60s)" + save_evidence T5-tengig-down + r "$h" 'sudo ip link set bond0.53 up' + sleep 20 +} + +T11() { # a blessed box with no peers file must not restart-loop + log "T11 missing peers file must not restart-loop" + local h; h="$(holder)" + r "$h" 'sudo mv /etc/ppp/peers/pppoe0 /tmp/peers.bak; sudo systemctl restart ppp@pppoe0' + sleep 12 + local n; n="$(r "$h" 'systemctl show ppp@pppoe0 -p NRestarts --value')" + [ "${n:-99}" -le 1 ] && pass "NRestarts=$n (gate refused the start)" \ + || fail "NRestarts=$n -- restart loop is back" + r "$h" 'sudo mv /tmp/peers.bak /etc/ppp/peers/pppoe0' + save_evidence T11-no-peers-file + settle 24 >/dev/null +} + +T8() { # lease expiry: the guard must hang up a demoted-but-unreconciled box + log "T8 lease expiry revokes the session" + local h; h="$(holder)" + r "$h" 'sudo systemctl stop vrrp-wan-reconcile.timer' + r "$h" 'sudo touch -d "-200 seconds" /run/vrrp-wan/may-dial' + sleep 12 + [ "$(ppp_on "$h")" = 0 ] && pass "guard hung up on a stale lease" \ + || fail "stale lease did not revoke the session" + r "$h" 'sudo systemctl start vrrp-wan-reconcile.timer' + save_evidence T8-lease-expiry + settle 24 >/dev/null +} + +case "${1:---all}" in + --list) echo "T0 baseline | T3 clean failover | T5 10gig-down | T8 lease expiry | T11 no-peers-file"; exit 0 ;; + --all) preflight || exit 1; T0; T3; T5; T8; T11 ;; + *) preflight || exit 1; "$1" ;; +esac + +echo +[ "$FAILED" -eq 0 ] && { echo "ALL PASS"; exit 0; } +echo "$FAILED check(s) FAILED"; exit 1 diff --git a/labsim/wan-failover-evidence/T0-baseline/state.txt b/labsim/wan-failover-evidence/T0-baseline/state.txt new file mode 100644 index 0000000..9d409a1 --- /dev/null +++ b/labsim/wan-failover-evidence/T0-baseline/state.txt @@ -0,0 +1,27 @@ +=== 2026-09-05T19:07:59+01:00 === +--- AC sessions --- + ifname | username | ip | ip6 | ip6-dp | calling-sid | rate-limit | state | uptime | rx-bytes | tx-bytes +--------+----------+----------------+-----+--------+-------------------+------------+--------+----------+----------+---------- + ppp0 | simdsl | 198.51.100.117 | | | 52:54:00:e5:95:a2 | | active | 00:01:31 | 1.0 KiB | 792 B +--- 172.31.1.252 --- +vip=172.31.1.1 holds_vip=yes wan_disabled=no wan_up=yes ppp_up=yes ppp_active=yes may_dial=yes lease_age=0 dropin=yes role=master +pppoe0 UNKNOWN 198.51.100.117 peer 198.51.100.1/32 +default via 203.0.113.1 dev bond0.53 proto failover metric 1 +Sep 05 18:01:26 apitest vrrp-wan[2994197]: not MASTER but bond0.53 enabled -> releasing +Sep 05 18:01:30 apitest vrrp-wan[2995085]: bond0.53 disable commit took 4s +Sep 05 18:03:25 apitest vrrp-wan[3000708]: MASTER: dialling pppoe0 +Sep 05 18:03:25 apitest vrrp-wan[3000872]: MASTER with bond0.53 disabled -> enabling +Sep 05 18:03:29 apitest vrrp-wan[3001568]: bond0.53 enable commit took 4s +Sep 05 18:05:43 apitest vrrp-wan[3005097]: GUARD: lease stale (204s > 75s; is vrrp-wan-reconcile.timer running?) -- hanging up pppoe0 +Sep 05 18:05:53 apitest vrrp-wan[3005490]: MASTER: dialling pppoe0 +Sep 05 18:06:24 apitest vrrp-wan[3007095]: MASTER: dialling pppoe0 +--- 172.31.1.253 --- +vip=172.31.1.1 holds_vip=no wan_disabled=yes wan_up=no ppp_up=no ppp_active=no may_dial=no lease_age=- dropin=yes role=backup +Sep 05 17:57:52 vyos vrrp-wan[2797572]: not MASTER but bond0.53 enabled -> releasing +Sep 05 17:58:16 vyos vrrp-wan[2798935]: bond0.53 disable commit took 24s +Sep 05 18:01:03 vyos vrrp-wan[2802338]: MASTER: dialling pppoe0 +Sep 05 18:01:03 vyos vrrp-wan[2802489]: MASTER with bond0.53 disabled -> enabling +Sep 05 18:01:07 vyos vrrp-wan[2803365]: bond0.53 enable commit took 4s +Sep 05 18:03:02 vyos vrrp-wan[2808661]: not MASTER: hanging up pppoe0 +Sep 05 18:03:02 vyos vrrp-wan[2808822]: not MASTER but bond0.53 enabled -> releasing +Sep 05 18:03:26 vyos vrrp-wan[2810178]: bond0.53 disable commit took 24s diff --git a/labsim/wan-failover-evidence/T11-no-peers-file/state.txt b/labsim/wan-failover-evidence/T11-no-peers-file/state.txt new file mode 100644 index 0000000..c44f228 --- /dev/null +++ b/labsim/wan-failover-evidence/T11-no-peers-file/state.txt @@ -0,0 +1,25 @@ +=== 2026-09-05T19:10:28+01:00 === +--- AC sessions --- + ifname | username | ip | ip6 | ip6-dp | calling-sid | rate-limit | state | uptime | rx-bytes | tx-bytes +--------+----------+----+-----+--------+-------------+------------+-------+--------+----------+---------- +--- 172.31.1.252 --- +vip=172.31.1.1 holds_vip=no wan_disabled=yes wan_up=no ppp_up=no ppp_active=no may_dial=no lease_age=- dropin=yes role=backup +Sep 05 18:03:25 apitest vrrp-wan[3000872]: MASTER with bond0.53 disabled -> enabling +Sep 05 18:03:29 apitest vrrp-wan[3001568]: bond0.53 enable commit took 4s +Sep 05 18:05:43 apitest vrrp-wan[3005097]: GUARD: lease stale (204s > 75s; is vrrp-wan-reconcile.timer running?) -- hanging up pppoe0 +Sep 05 18:05:53 apitest vrrp-wan[3005490]: MASTER: dialling pppoe0 +Sep 05 18:06:24 apitest vrrp-wan[3007095]: MASTER: dialling pppoe0 +Sep 05 18:08:15 apitest vrrp-wan[3010461]: not MASTER: hanging up pppoe0 +Sep 05 18:08:16 apitest vrrp-wan[3010622]: not MASTER but bond0.53 enabled -> releasing +Sep 05 18:08:40 apitest vrrp-wan[3012085]: bond0.53 disable commit took 24s +--- 172.31.1.253 --- +vip=172.31.1.1 holds_vip=yes wan_disabled=no wan_up=yes ppp_up=no ppp_active=yes may_dial=yes lease_age=0 dropin=yes role=master +default via 203.0.113.1 dev bond0.53 proto failover metric 1 +Sep 05 18:03:02 vyos vrrp-wan[2808822]: not MASTER but bond0.53 enabled -> releasing +Sep 05 18:03:26 vyos vrrp-wan[2810178]: bond0.53 disable commit took 24s +Sep 05 18:07:53 vyos vrrp-wan[2815869]: MASTER: dialling pppoe0 +Sep 05 18:07:53 vyos vrrp-wan[2816022]: MASTER with bond0.53 disabled -> enabling +Sep 05 18:07:57 vyos vrrp-wan[2816731]: bond0.53 enable commit took 4s +Sep 05 18:09:10 vyos vrrp-wan[2819736]: GUARD: lease stale (204s > 75s; is vrrp-wan-reconcile.timer running?) -- hanging up pppoe0 +Sep 05 18:09:34 vyos vrrp-wan[2820820]: MASTER: dialling pppoe0 +Sep 05 18:10:04 vyos vrrp-wan[2821942]: MASTER: dialling pppoe0 diff --git a/labsim/wan-failover-evidence/T3-clean-failover/state.txt b/labsim/wan-failover-evidence/T3-clean-failover/state.txt new file mode 100644 index 0000000..e0ce455 --- /dev/null +++ b/labsim/wan-failover-evidence/T3-clean-failover/state.txt @@ -0,0 +1,28 @@ +=== 2026-09-05T19:08:34+01:00 === +--- AC sessions --- + ifname | username | ip | ip6 | ip6-dp | calling-sid | rate-limit | state | uptime | rx-bytes | tx-bytes +--------+----------+----------------+-----+--------+-------------------+------------+--------+----------+----------+---------- + ppp0 | simdsl | 198.51.100.118 | | | 52:54:00:4e:0b:56 | | active | 00:00:15 | 590 B | 280 B +--- 172.31.1.252 --- +vip=172.31.1.1 holds_vip=no wan_disabled=no wan_up=yes ppp_up=no ppp_active=no may_dial=no lease_age=- dropin=yes role=backup +default via 203.0.113.1 dev bond0.53 proto failover metric 1 +Sep 05 18:03:25 apitest vrrp-wan[3000708]: MASTER: dialling pppoe0 +Sep 05 18:03:25 apitest vrrp-wan[3000872]: MASTER with bond0.53 disabled -> enabling +Sep 05 18:03:29 apitest vrrp-wan[3001568]: bond0.53 enable commit took 4s +Sep 05 18:05:43 apitest vrrp-wan[3005097]: GUARD: lease stale (204s > 75s; is vrrp-wan-reconcile.timer running?) -- hanging up pppoe0 +Sep 05 18:05:53 apitest vrrp-wan[3005490]: MASTER: dialling pppoe0 +Sep 05 18:06:24 apitest vrrp-wan[3007095]: MASTER: dialling pppoe0 +Sep 05 18:08:15 apitest vrrp-wan[3010461]: not MASTER: hanging up pppoe0 +Sep 05 18:08:16 apitest vrrp-wan[3010622]: not MASTER but bond0.53 enabled -> releasing +--- 172.31.1.253 --- +vip=172.31.1.1 holds_vip=yes wan_disabled=no wan_up=yes ppp_up=yes ppp_active=yes may_dial=yes lease_age=13 dropin=yes role=master +pppoe0 UNKNOWN 198.51.100.118 peer 198.51.100.1/32 +default via 203.0.113.1 dev bond0.53 proto failover metric 1 +Sep 05 18:01:03 vyos vrrp-wan[2802489]: MASTER with bond0.53 disabled -> enabling +Sep 05 18:01:07 vyos vrrp-wan[2803365]: bond0.53 enable commit took 4s +Sep 05 18:03:02 vyos vrrp-wan[2808661]: not MASTER: hanging up pppoe0 +Sep 05 18:03:02 vyos vrrp-wan[2808822]: not MASTER but bond0.53 enabled -> releasing +Sep 05 18:03:26 vyos vrrp-wan[2810178]: bond0.53 disable commit took 24s +Sep 05 18:07:53 vyos vrrp-wan[2815869]: MASTER: dialling pppoe0 +Sep 05 18:07:53 vyos vrrp-wan[2816022]: MASTER with bond0.53 disabled -> enabling +Sep 05 18:07:57 vyos vrrp-wan[2816731]: bond0.53 enable commit took 4s diff --git a/labsim/wan-failover-evidence/T5-tengig-down/state.txt b/labsim/wan-failover-evidence/T5-tengig-down/state.txt new file mode 100644 index 0000000..8261bb1 --- /dev/null +++ b/labsim/wan-failover-evidence/T5-tengig-down/state.txt @@ -0,0 +1,27 @@ +=== 2026-09-05T19:09:05+01:00 === +--- AC sessions --- + ifname | username | ip | ip6 | ip6-dp | calling-sid | rate-limit | state | uptime | rx-bytes | tx-bytes +--------+----------+----------------+-----+--------+-------------------+------------+--------+----------+----------+---------- + ppp0 | simdsl | 198.51.100.118 | | | 52:54:00:4e:0b:56 | | active | 00:00:46 | 758 B | 448 B +--- 172.31.1.252 --- +vip=172.31.1.1 holds_vip=no wan_disabled=yes wan_up=no ppp_up=no ppp_active=no may_dial=no lease_age=- dropin=yes role=backup +Sep 05 18:03:25 apitest vrrp-wan[3000872]: MASTER with bond0.53 disabled -> enabling +Sep 05 18:03:29 apitest vrrp-wan[3001568]: bond0.53 enable commit took 4s +Sep 05 18:05:43 apitest vrrp-wan[3005097]: GUARD: lease stale (204s > 75s; is vrrp-wan-reconcile.timer running?) -- hanging up pppoe0 +Sep 05 18:05:53 apitest vrrp-wan[3005490]: MASTER: dialling pppoe0 +Sep 05 18:06:24 apitest vrrp-wan[3007095]: MASTER: dialling pppoe0 +Sep 05 18:08:15 apitest vrrp-wan[3010461]: not MASTER: hanging up pppoe0 +Sep 05 18:08:16 apitest vrrp-wan[3010622]: not MASTER but bond0.53 enabled -> releasing +Sep 05 18:08:40 apitest vrrp-wan[3012085]: bond0.53 disable commit took 24s +--- 172.31.1.253 --- +vip=172.31.1.1 holds_vip=yes wan_disabled=no wan_up=yes ppp_up=yes ppp_active=yes may_dial=yes lease_age=10 dropin=yes role=master +pppoe0 UNKNOWN 198.51.100.118 peer 198.51.100.1/32 +default nhid 483 dev pppoe0 proto static metric 20 +Sep 05 18:01:03 vyos vrrp-wan[2802489]: MASTER with bond0.53 disabled -> enabling +Sep 05 18:01:07 vyos vrrp-wan[2803365]: bond0.53 enable commit took 4s +Sep 05 18:03:02 vyos vrrp-wan[2808661]: not MASTER: hanging up pppoe0 +Sep 05 18:03:02 vyos vrrp-wan[2808822]: not MASTER but bond0.53 enabled -> releasing +Sep 05 18:03:26 vyos vrrp-wan[2810178]: bond0.53 disable commit took 24s +Sep 05 18:07:53 vyos vrrp-wan[2815869]: MASTER: dialling pppoe0 +Sep 05 18:07:53 vyos vrrp-wan[2816022]: MASTER with bond0.53 disabled -> enabling +Sep 05 18:07:57 vyos vrrp-wan[2816731]: bond0.53 enable commit took 4s diff --git a/labsim/wan-failover-evidence/T8-lease-expiry/state.txt b/labsim/wan-failover-evidence/T8-lease-expiry/state.txt new file mode 100644 index 0000000..45b140d --- /dev/null +++ b/labsim/wan-failover-evidence/T8-lease-expiry/state.txt @@ -0,0 +1,25 @@ +=== 2026-09-05T19:09:47+01:00 === +--- AC sessions --- + ifname | username | ip | ip6 | ip6-dp | calling-sid | rate-limit | state | uptime | rx-bytes | tx-bytes +--------+----------+----+-----+--------+-------------+------------+-------+--------+----------+---------- +--- 172.31.1.252 --- +vip=172.31.1.1 holds_vip=no wan_disabled=yes wan_up=no ppp_up=no ppp_active=no may_dial=no lease_age=- dropin=yes role=backup +Sep 05 18:03:25 apitest vrrp-wan[3000872]: MASTER with bond0.53 disabled -> enabling +Sep 05 18:03:29 apitest vrrp-wan[3001568]: bond0.53 enable commit took 4s +Sep 05 18:05:43 apitest vrrp-wan[3005097]: GUARD: lease stale (204s > 75s; is vrrp-wan-reconcile.timer running?) -- hanging up pppoe0 +Sep 05 18:05:53 apitest vrrp-wan[3005490]: MASTER: dialling pppoe0 +Sep 05 18:06:24 apitest vrrp-wan[3007095]: MASTER: dialling pppoe0 +Sep 05 18:08:15 apitest vrrp-wan[3010461]: not MASTER: hanging up pppoe0 +Sep 05 18:08:16 apitest vrrp-wan[3010622]: not MASTER but bond0.53 enabled -> releasing +Sep 05 18:08:40 apitest vrrp-wan[3012085]: bond0.53 disable commit took 24s +--- 172.31.1.253 --- +vip=172.31.1.1 holds_vip=yes wan_disabled=no wan_up=yes ppp_up=no ppp_active=no may_dial=no lease_age=- dropin=yes role=master +default via 203.0.113.1 dev bond0.53 proto failover metric 1 +Sep 05 18:01:07 vyos vrrp-wan[2803365]: bond0.53 enable commit took 4s +Sep 05 18:03:02 vyos vrrp-wan[2808661]: not MASTER: hanging up pppoe0 +Sep 05 18:03:02 vyos vrrp-wan[2808822]: not MASTER but bond0.53 enabled -> releasing +Sep 05 18:03:26 vyos vrrp-wan[2810178]: bond0.53 disable commit took 24s +Sep 05 18:07:53 vyos vrrp-wan[2815869]: MASTER: dialling pppoe0 +Sep 05 18:07:53 vyos vrrp-wan[2816022]: MASTER with bond0.53 disabled -> enabling +Sep 05 18:07:57 vyos vrrp-wan[2816731]: bond0.53 enable commit took 4s +Sep 05 18:09:10 vyos vrrp-wan[2819736]: GUARD: lease stale (204s > 75s; is vrrp-wan-reconcile.timer running?) -- hanging up pppoe0 diff --git a/migration/vrrp-wan-health b/migration/vrrp-wan-health index e34ef23..f7b1706 100755 --- a/migration/vrrp-wan-health +++ b/migration/vrrp-wan-health @@ -115,15 +115,25 @@ mkdir -p "$STATE" 2>/dev/null for ifc in bond0.53 pppoe0; do if ip -4 addr show dev "$ifc" 2>/dev/null | grep -q 'inet '; then echo "$ifc" > "$STATE/wan" 2>/dev/null + # Re-stamp on every healthy tick, so the grace window below measures + # time since this box last DEMONSTRABLY had a WAN. + date +%s > "$STATE/since" 2>/dev/null exit 0 fi done rm -f "$STATE/wan" 2>/dev/null -# Master, no WAN yet, still within the grace window: DHCP negotiation and PPPoE -# dial-up take real time, and the ISP has to accept the cloned MAC arriving on a -# different port. Failing here would demote the new master before it ever had a -# chance, and hand the VIPs straight back -- a flap, not a failover. +# No WAN right now, but there was one within GRACE: ride it out. +# +# `since` is re-stamped on every healthy tick, so this measures time since the +# box last HAD a WAN -- not time since it was promoted. Measuring from promotion +# was wrong in a way that only shows up on an established master: after hours of +# uptime `now - since` far exceeds any grace, so the first moment bond0.53 went +# down and pppoe0 was mid-redial, the master failed its own check, shed every +# VIP, and the peer -- inheriting the same WAN outage -- did the same. Observed +# in labsim: taking the 10 gig down flapped the pair instead of falling back to +# PPPoE. A WAN gap must be survivable wherever it happens, not only just after a +# promotion. since=$(cat "$STATE/since" 2>/dev/null || echo 0) [ $(( $(date +%s) - since )) -lt "$GRACE" ] && exit 0