labsim: PPPoE HA passes the matrix, and the health check had a real flap bug
Some checks failed
Some checks failed
Full run green: baseline (one AC session, held by the VIP holder), clean
failover (pppoe0 moves in 26s, old master releases), 10 gig down (route falls to
pppoe0 and the LAN is back online in 5s), lease expiry (the guard hangs up), and
a missing peers file (NRestarts=0, no loop). Evidence in
labsim/wan-failover-evidence/.
The 10 gig test found a genuine bug in vrrp-wan-health, not in the sim. GRACE
was measured from PROMOTION, so an established master had no grace at all --
after hours of uptime `now - since` far exceeds any window. The first moment
bond0.53 went down while pppoe0 was mid-redial, the master failed its own check,
shed every VIP, and the peer inherited the same WAN outage and did the same. A
brief WAN blip would have flapped the production pair. The stamp is now
refreshed on every healthy tick, so grace measures time since the box last
demonstrably HAD a WAN -- survivable wherever the gap happens, not only just
after a promotion.
Three harness bugs, all the same shape, all of which produced a confident wrong
answer before being caught:
- waiting for "exactly one pppoe0 holder" returns INSTANTLY during a handover,
because it was already true. The useful question is who holds it.
- judging connectivity on a single ping 20s after a link drop reported an
outage that had already healed. Poll, do not sample.
- `-o PreferredAuthentications=password` suits the routers but not the Alpine
LAN VMs, whose sshd offers keyboard-interactive: ssh exited 255 before
running anything and the test read that as "the LAN lost the internet". A
tcpdump on the router showed the pings leaving pppoe0 NATed to
198.51.100.117 and the replies coming back the whole time. An exit code that
can mean "the network is broken" or "I could not log in" is not a
connectivity test, so the check now asserts on what the guest reported.
That last one is why the harness asks the routers and the access concentrator
rather than a client, and why it refuses to run at all while either router still
has a default route via eth2 -- the libvirt-NAT scaffold answers connectivity
checks the WAN under test would have failed.
Still to run: hard failover (destroy the master), and the session-control
replace/deny/disable axis that brackets Vodafone's unknown behaviour and sets
the final GRACE. Nothing applied to production.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
This commit is contained in:
Michal
230
labsim/labsim-pppoe-ha-test.sh
Executable file
230
labsim/labsim-pppoe-ha-test.sh
Executable file
@@ -0,0 +1,230 @@
|
||||
#!/bin/bash
|
||||
# Does the WAN follow VRRP mastership, and does exactly ONE router ever hold the
|
||||
# ISP session?
|
||||
#
|
||||
# The question is not "did a client get internet". A client can be answered by
|
||||
# the wrong path entirely -- for months labsim-vyos's only default route was the
|
||||
# libvirt-NAT scaffold on eth2, so every "the LAN still has internet" verdict was
|
||||
# answered by eth2 rather than by the WAN under test. This script therefore
|
||||
# refuses to run while that is true, and asks its questions of the ROUTERS and
|
||||
# the ACCESS CONCENTRATOR, which cannot be answered by accident.
|
||||
#
|
||||
# The invariant, checked continuously and independently of any individual test:
|
||||
#
|
||||
# the AC never reports two `simdsl` sessions, and no two routers ever have a
|
||||
# pppoe0 interface at the same time
|
||||
#
|
||||
# A run that violates it FAILS regardless of its own verdict, because a single
|
||||
# consumer credential is the whole constraint the design exists to satisfy.
|
||||
#
|
||||
# ./labsim-pppoe-ha-test.sh --list
|
||||
# ./labsim-pppoe-ha-test.sh T3
|
||||
# ./labsim-pppoe-ha-test.sh --all
|
||||
set -uo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
R1="${R1:-172.31.1.252}"; R2="${R2:-172.31.1.253}"
|
||||
ISP="${ISP:-192.168.122.63}" # the fake access concentrator
|
||||
LANVM="${LANVM:-172.31.10.10}"
|
||||
VIP="${VIP:-172.31.1.1}"
|
||||
PW="${VYOS_PW:-vyos}"; LANPW="${LANPW:-labsim}"
|
||||
EVID="$SCRIPT_DIR/wan-failover-evidence"
|
||||
|
||||
SSH=(-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null
|
||||
-o LogLevel=ERROR -o ConnectTimeout=6 -o PreferredAuthentications=password)
|
||||
r() { timeout 45 sshpass -p "$PW" ssh "${SSH[@]}" "vyos@$1" "${@:2}" 2>/dev/null; }
|
||||
isp() { timeout 30 sshpass -p "$PW" ssh "${SSH[@]}" "vyos@$ISP" "$@" 2>/dev/null; }
|
||||
# The LAN VMs are Alpine and their sshd offers keyboard-interactive, not
|
||||
# `password`. Reusing the routers' option set here made ssh exit 255 BEFORE
|
||||
# running anything, and T5 read that as "the LAN lost the internet" while a
|
||||
# tcpdump on the router showed the pings flowing out pppoe0 and the replies
|
||||
# coming back. An exit code that can mean "the network is broken" or "I could
|
||||
# not log in" is not a connectivity test.
|
||||
LAN_SSH=(-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null
|
||||
-o LogLevel=ERROR -o ConnectTimeout=6)
|
||||
lan() { timeout 45 sshpass -p "$LANPW" ssh "${LAN_SSH[@]}" "root@$LANVM" "$@" 2>/dev/null; }
|
||||
|
||||
# Assert on what the guest actually reported, not on ssh's exit status.
|
||||
lan_online() { [ "$(lan 'ping -c2 -W3 9.9.9.9 >/dev/null 2>&1 && echo ONLINE')" = ONLINE ]; }
|
||||
|
||||
log() { printf '\033[36m==>\033[0m %s\n' "$*"; }
|
||||
pass() { printf ' \033[32mPASS\033[0m %s\n' "$*"; }
|
||||
fail() { printf ' \033[31mFAIL\033[0m %s\n' "$*"; FAILED=$((FAILED+1)); }
|
||||
FAILED=0
|
||||
|
||||
# --- observations ----------------------------------------------------------
|
||||
ac_sessions() { isp '/opt/vyatta/bin/vyatta-op-cmd-wrapper show pppoe-server sessions' \
|
||||
| grep -c ' simdsl ' || true; }
|
||||
ac_detail() { isp '/opt/vyatta/bin/vyatta-op-cmd-wrapper show pppoe-server sessions'; }
|
||||
ppp_on() { r "$1" 'ip -4 addr show pppoe0 2>/dev/null | grep -c inet' | tr -d ' \n'; }
|
||||
holder() { for h in "$R1" "$R2"; do
|
||||
[ "$(r "$h" "ip -4 -o addr show | grep -c ' ${VIP}/'" | tr -d ' \n')" != 0 ] \
|
||||
&& { echo "$h"; return; }; done; echo none; }
|
||||
status() { r "$1" 'sudo /config/vrrp-wan-reconcile --status'; }
|
||||
|
||||
# How many routers currently hold a PPPoE interface. The invariant's other half.
|
||||
ppp_holders() { n=0; for h in "$R1" "$R2"; do
|
||||
[ "$(ppp_on "$h")" != 0 ] && n=$((n+1)); done; echo "$n"; }
|
||||
|
||||
check_invariant() {
|
||||
local s p ok=0
|
||||
s="$(ac_sessions)"; p="$(ppp_holders)"
|
||||
[ "${s:-0}" -le 1 ] || { fail "INVARIANT: AC reports $s simdsl sessions"; ok=1; }
|
||||
[ "${p:-0}" -le 1 ] || { fail "INVARIANT: $p routers hold pppoe0"; ok=1; }
|
||||
return $ok
|
||||
}
|
||||
|
||||
# --- preconditions ---------------------------------------------------------
|
||||
# The scaffold check is a hard gate, not a warning. A default route via eth2
|
||||
# means the box can reach the internet without the WAN working at all, and every
|
||||
# connectivity verdict below would be a lie.
|
||||
preflight() {
|
||||
log "preflight"
|
||||
local rc=0
|
||||
for h in "$R1" "$R2"; do
|
||||
if r "$h" 'ip route show default' | grep -q 'dev eth2'; then
|
||||
fail "$h still routes via eth2 (libvirt-NAT scaffold) -- run sim-net-config.py --drop-scaffold"
|
||||
rc=1
|
||||
fi
|
||||
if [ "$(r "$h" '[ -f /etc/systemd/system/ppp@pppoe0.service.d/10-vrrp-wan-gate.conf ] && echo y')" != y ]; then
|
||||
fail "$h is missing the ppp gate drop-in -- run migration/vrrp-wan-install"
|
||||
rc=1
|
||||
fi
|
||||
[ "$(r "$h" 'systemctl is-active vrrp-wan-guard.timer')" = active ] \
|
||||
|| { fail "$h vrrp-wan-guard.timer not active"; rc=1; }
|
||||
done
|
||||
[ "$rc" -eq 0 ] && pass "scaffold dropped, gate present, guard running on both"
|
||||
return $rc
|
||||
}
|
||||
|
||||
settle() { # wait until exactly one router holds pppoe0, or give up
|
||||
local i
|
||||
for i in $(seq 1 "${1:-24}"); do
|
||||
[ "$(ppp_holders)" = 1 ] && return 0
|
||||
sleep 5
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
# Wait until a SPECIFIC router holds pppoe0 and the other does not.
|
||||
#
|
||||
# The obvious `settle` is wrong for a failover: "exactly one holder" is already
|
||||
# true before the handover starts, so it returns instantly and the test reports
|
||||
# that nothing moved while the handover is still in flight. Asking who holds it
|
||||
# is the only useful form of the question.
|
||||
settle_on() {
|
||||
local want="$1" other i
|
||||
other=$([ "$want" = "$R1" ] && echo "$R2" || echo "$R1")
|
||||
for i in $(seq 1 "${2:-30}"); do
|
||||
[ "$(ppp_on "$want")" != 0 ] && [ "$(ppp_on "$other")" = 0 ] && return 0
|
||||
sleep 5
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
save_evidence() {
|
||||
local name="$1"; local d="$EVID/$name"; mkdir -p "$d"
|
||||
{ echo "=== $(date -Is) ==="; echo "--- AC sessions ---"; ac_detail
|
||||
for h in "$R1" "$R2"; do echo "--- $h ---"; status "$h"
|
||||
r "$h" 'ip -4 -br addr show pppoe0 2>/dev/null; ip route show default; sudo journalctl -t vrrp-wan -n 8 --no-pager'
|
||||
done; } > "$d/state.txt" 2>&1
|
||||
log "evidence -> wan-failover-evidence/$name/"
|
||||
}
|
||||
|
||||
# --- tests -----------------------------------------------------------------
|
||||
T0() { # baseline
|
||||
log "T0 baseline: exactly one session, held by the VIP holder"
|
||||
local h s; h="$(holder)"; s="$(ac_sessions)"
|
||||
[ "$s" = 1 ] && pass "AC reports 1 session" || fail "AC reports $s sessions"
|
||||
[ "$(ppp_on "$h")" != 0 ] && pass "the VIP holder ($h) is the one dialled" \
|
||||
|| fail "VIP holder $h has no pppoe0"
|
||||
local other; other=$([ "$h" = "$R1" ] && echo "$R2" || echo "$R1")
|
||||
[ "$(ppp_on "$other")" = 0 ] && pass "the backup ($other) is not dialled" \
|
||||
|| fail "backup $other also holds pppoe0"
|
||||
save_evidence T0-baseline
|
||||
}
|
||||
|
||||
T3() { # clean, deliberate failover
|
||||
log "T3 clean failover via force-fault"
|
||||
local from to t0 t1; from="$(holder)"
|
||||
to=$([ "$from" = "$R1" ] && echo "$R2" || echo "$R1")
|
||||
log " master=$from -> expecting $to"
|
||||
t0=$(date +%s)
|
||||
r "$from" 'sudo mkdir -p /run/vrrp-wan && sudo touch /run/vrrp-wan/force-fault'
|
||||
if settle_on "$to" 30; then
|
||||
t1=$(date +%s)
|
||||
[ "$(ppp_on "$to")" != 0 ] && pass "pppoe0 moved to $to in $((t1-t0))s" \
|
||||
|| fail "pppoe0 did not move to $to"
|
||||
[ "$(ppp_on "$from")" = 0 ] && pass "$from released pppoe0" \
|
||||
|| fail "$from still holds pppoe0"
|
||||
else
|
||||
fail "never settled to exactly one pppoe0 holder"
|
||||
fi
|
||||
check_invariant
|
||||
save_evidence T3-clean-failover
|
||||
r "$from" 'sudo rm -f /run/vrrp-wan/force-fault'
|
||||
settle 30 >/dev/null
|
||||
}
|
||||
|
||||
T5() { # 10gig down -> PPPoE carries traffic
|
||||
log "T5 10 gig down on the master: traffic must survive on pppoe0"
|
||||
local h; h="$(holder)"
|
||||
r "$h" 'sudo ip link set bond0.53 down'
|
||||
sleep 20
|
||||
local via; via="$(r "$h" 'ip route show default' | head -1)"
|
||||
if echo "$via" | grep -q pppoe0; then
|
||||
pass "default route moved to pppoe0: $via"
|
||||
else
|
||||
fail "default route did not move to pppoe0: ${via:-<none>}"
|
||||
fi
|
||||
# Poll, do not sample. Judging connectivity on one ping 20s after the link
|
||||
# dropped failed while the path was still reconverging, and reported "the LAN
|
||||
# lost the internet" for a path that came back moments later. A single
|
||||
# negative sample is the least trustworthy verdict this harness can produce.
|
||||
local ok=no i
|
||||
for i in $(seq 1 12); do
|
||||
lan_online && { ok=yes; break; }
|
||||
sleep 5
|
||||
done
|
||||
[ "$ok" = yes ] && pass "LAN reaches the internet over pppoe0 (after $((i*5))s)" \
|
||||
|| fail "LAN never regained the internet with only pppoe0 up (60s)"
|
||||
save_evidence T5-tengig-down
|
||||
r "$h" 'sudo ip link set bond0.53 up'
|
||||
sleep 20
|
||||
}
|
||||
|
||||
T11() { # a blessed box with no peers file must not restart-loop
|
||||
log "T11 missing peers file must not restart-loop"
|
||||
local h; h="$(holder)"
|
||||
r "$h" 'sudo mv /etc/ppp/peers/pppoe0 /tmp/peers.bak; sudo systemctl restart ppp@pppoe0'
|
||||
sleep 12
|
||||
local n; n="$(r "$h" 'systemctl show ppp@pppoe0 -p NRestarts --value')"
|
||||
[ "${n:-99}" -le 1 ] && pass "NRestarts=$n (gate refused the start)" \
|
||||
|| fail "NRestarts=$n -- restart loop is back"
|
||||
r "$h" 'sudo mv /tmp/peers.bak /etc/ppp/peers/pppoe0'
|
||||
save_evidence T11-no-peers-file
|
||||
settle 24 >/dev/null
|
||||
}
|
||||
|
||||
T8() { # lease expiry: the guard must hang up a demoted-but-unreconciled box
|
||||
log "T8 lease expiry revokes the session"
|
||||
local h; h="$(holder)"
|
||||
r "$h" 'sudo systemctl stop vrrp-wan-reconcile.timer'
|
||||
r "$h" 'sudo touch -d "-200 seconds" /run/vrrp-wan/may-dial'
|
||||
sleep 12
|
||||
[ "$(ppp_on "$h")" = 0 ] && pass "guard hung up on a stale lease" \
|
||||
|| fail "stale lease did not revoke the session"
|
||||
r "$h" 'sudo systemctl start vrrp-wan-reconcile.timer'
|
||||
save_evidence T8-lease-expiry
|
||||
settle 24 >/dev/null
|
||||
}
|
||||
|
||||
case "${1:---all}" in
|
||||
--list) echo "T0 baseline | T3 clean failover | T5 10gig-down | T8 lease expiry | T11 no-peers-file"; exit 0 ;;
|
||||
--all) preflight || exit 1; T0; T3; T5; T8; T11 ;;
|
||||
*) preflight || exit 1; "$1" ;;
|
||||
esac
|
||||
|
||||
echo
|
||||
[ "$FAILED" -eq 0 ] && { echo "ALL PASS"; exit 0; }
|
||||
echo "$FAILED check(s) FAILED"; exit 1
|
||||
Reference in New Issue
Block a user