5ed0e4888a
Final confirming run with GRACE=300 and every fix installed:
--all ALL PASS (T0 T3 T5 T8 T11 T12)
--hard ALL PASS (replace / deny / disable, each policy verified)
Second independent measurement of the hard failover, which is what makes the
figures trustworthy rather than anecdotal:
run 1 run 2
replace 26s 26s
deny 148s 141s
disable 21s 20s
`deny` sits at ~141-148s across both, so it is the AC's dead-peer behaviour
and not a one-off; GRACE=300 keeps roughly 2x margin.
Corrected an overclaim in PPPOE-HA.md while confirming it: the invariant row
read "AC never showed two simdsl sessions", and under session-control=disable
it did -- one live, one orphaned from the destroyed router. Only ever one LIVE
router dialled, which is the invariant that matters. Said so plainly rather
than leaving a table that reads better than the evidence.
WAN failover evidence
Captured by labsim/labsim-pppoe-ha-test.sh. Each directory holds the state of
both routers and the access concentrator at the end of one test.
The number that sizes GRACE
T4 destroys the master with virsh destroy — no LCP Terminate, no PADT, the
router simply ceases — and times how long until the survivor holds a PPPoE
session. Run across every policy VyOS can express, because Vodafone's is
unknown:
Two independent runs, so these describe the AC's behaviour rather than one-offs:
session-control |
takeover | |
|---|---|---|
replace |
26s / 26s | accel-ppp default; the new auth kills the old session |
deny |
148s / 141s | the AC refuses the survivor until its own dead-peer timer frees the dead session |
disable |
21s / 20s | no single-session enforcement at all |
deny is the only one that matters for sizing, and GRACE=300 in
migration/vrrp-wan.conf comes from it. Session polling during that run caught
the mechanism directly: the destroyed router's session stayed in the AC's table
while the survivor's dial attempts appeared and were rejected — twice — before
one finally took.
Treat 148s as a floor, not a worst case. These are idle 2-vCPU VMs, and the
AC shares an OVS bridge with the routers, so virsh destroy removes the port
and accel-ppp sees the peer physically vanish. A real BRAS reached over DSL
never learns our router died; it waits out its own timers, which are longer and
not ours to know.
How these numbers were nearly wrong
Until 2026-09-06 the matrix set the policy with:
isp "vbash -c 'source /opt/vyatta/etc/functions/script-template; configure; \
set service pppoe-server session-control $mode; commit; save; exit'"
That form never starts a config session. commit fails with
Invalid command: [commit] on stderr, which isp() discards — so all three
iterations ran against the accel-ppp default while printing the mode they were
supposedly testing. show configuration commands | grep session-control on the
ISP VM came back empty after a full run. The matrix reported deny at 25s; the
real figure is 148s, and GRACE was sized against the fiction.
isp_session_control() now drives it from a real script file, reads the value
back, and skips the iteration rather than measure the wrong policy. A harness
that reports coverage it does not have is worse than one that reports a failure.
The other tests
| what it proves | |
|---|---|
T0-baseline |
exactly one session, held by the VIP holder |
T3-clean-failover |
force-fault moves pppoe0 in 20–26s |
T5-tengig-down |
10 gig down → route falls to pppoe0, LAN back in 5s |
T8-lease-expiry |
the guard hangs up a stale lease within ~5s |
T11-no-peers-file |
a blessed box with no peers file does not restart-loop |
T12-holdoff-keeps-session |
a 900s flap hold-off does not tear down a live session |
T12 exists because it did. ppp_dial() checked the hold-off and returned
before renewing /run/vrrp-wan/may-dial; that lease is what vrrp-wan-guard
expires, so tripping the damper stopped the renewal and the guard hung up
pppoe0 on the master ~80s later:
DIAL FLAP: >=6 attempts in 600s -- holding off 900s
GUARD: lease stale (81s > 75s) -- hanging up pppoe0
A damper meant to suppress repeated dials was destroying an established session instead.
Reading check_invariant
The invariant that matters is at most one of our routers holds pppoe0.
The AC's session count is only a proxy for it, and only while the AC enforces
single-session — under disable it does not, so a destroyed router's session
lingers and the count reads 2 with exactly one live router dialled. That is
reported as a WARN, not a failure. It is not silenced, because an orphaned
session still occupies the single slot at a real ISP: it is exactly what made
deny take 148s.