5ed0e4888a
Some checks failed
Final confirming run with GRACE=300 and every fix installed:
--all ALL PASS (T0 T3 T5 T8 T11 T12)
--hard ALL PASS (replace / deny / disable, each policy verified)
Second independent measurement of the hard failover, which is what makes the
figures trustworthy rather than anecdotal:
run 1 run 2
replace 26s 26s
deny 148s 141s
disable 21s 20s
`deny` sits at ~141-148s across both, so it is the AC's dead-peer behaviour
and not a one-off; GRACE=300 keeps roughly 2x margin.
Corrected an overclaim in PPPOE-HA.md while confirming it: the invariant row
read "AC never showed two simdsl sessions", and under session-control=disable
it did -- one live, one orphaned from the destroyed router. Only ever one LIVE
router dialled, which is the invariant that matters. Said so plainly rather
than leaving a table that reads better than the evidence.
86 lines
3.8 KiB
Markdown
86 lines
3.8 KiB
Markdown
# WAN failover evidence
|
||
|
||
Captured by `labsim/labsim-pppoe-ha-test.sh`. Each directory holds the state of
|
||
both routers and the access concentrator at the end of one test.
|
||
|
||
## The number that sizes GRACE
|
||
|
||
`T4` destroys the master with `virsh destroy` — no LCP Terminate, no PADT, the
|
||
router simply ceases — and times how long until the survivor holds a PPPoE
|
||
session. Run across every policy VyOS can express, because Vodafone's is
|
||
unknown:
|
||
|
||
Two independent runs, so these describe the AC's behaviour rather than one-offs:
|
||
|
||
| `session-control` | takeover | |
|
||
|---|---|---|
|
||
| `replace` | 26s / 26s | accel-ppp default; the new auth kills the old session |
|
||
| **`deny`** | **148s / 141s** | the AC refuses the survivor until its own dead-peer timer frees the dead session |
|
||
| `disable` | 21s / 20s | no single-session enforcement at all |
|
||
|
||
`deny` is the only one that matters for sizing, and `GRACE=300` in
|
||
`migration/vrrp-wan.conf` comes from it. Session polling during that run caught
|
||
the mechanism directly: the destroyed router's session stayed in the AC's table
|
||
while the survivor's dial attempts appeared and were rejected — twice — before
|
||
one finally took.
|
||
|
||
**Treat 148s as a floor, not a worst case.** These are idle 2-vCPU VMs, and the
|
||
AC shares an OVS bridge with the routers, so `virsh destroy` removes the port
|
||
and accel-ppp sees the peer physically vanish. A real BRAS reached over DSL
|
||
never learns our router died; it waits out its own timers, which are longer and
|
||
not ours to know.
|
||
|
||
## How these numbers were nearly wrong
|
||
|
||
Until 2026-09-06 the matrix set the policy with:
|
||
|
||
```sh
|
||
isp "vbash -c 'source /opt/vyatta/etc/functions/script-template; configure; \
|
||
set service pppoe-server session-control $mode; commit; save; exit'"
|
||
```
|
||
|
||
That form never starts a config session. `commit` fails with
|
||
`Invalid command: [commit]` on stderr, which `isp()` discards — so all three
|
||
iterations ran against the accel-ppp default while printing the mode they were
|
||
supposedly testing. `show configuration commands | grep session-control` on the
|
||
ISP VM came back empty after a full run. The matrix reported `deny` at 25s; the
|
||
real figure is 148s, and `GRACE` was sized against the fiction.
|
||
|
||
`isp_session_control()` now drives it from a real script file, reads the value
|
||
back, and skips the iteration rather than measure the wrong policy. A harness
|
||
that reports coverage it does not have is worse than one that reports a failure.
|
||
|
||
## The other tests
|
||
|
||
| | what it proves |
|
||
|---|---|
|
||
| `T0-baseline` | exactly one session, held by the VIP holder |
|
||
| `T3-clean-failover` | `force-fault` moves `pppoe0` in 20–26s |
|
||
| `T5-tengig-down` | 10 gig down → route falls to `pppoe0`, LAN back in 5s |
|
||
| `T8-lease-expiry` | the guard hangs up a stale lease within ~5s |
|
||
| `T11-no-peers-file` | a blessed box with no peers file does not restart-loop |
|
||
| `T12-holdoff-keeps-session` | a 900s flap hold-off does **not** tear down a live session |
|
||
|
||
`T12` exists because it did. `ppp_dial()` checked the hold-off and returned
|
||
before renewing `/run/vrrp-wan/may-dial`; that lease is what `vrrp-wan-guard`
|
||
expires, so tripping the damper stopped the renewal and the guard hung up
|
||
`pppoe0` on the master ~80s later:
|
||
|
||
```
|
||
DIAL FLAP: >=6 attempts in 600s -- holding off 900s
|
||
GUARD: lease stale (81s > 75s) -- hanging up pppoe0
|
||
```
|
||
|
||
A damper meant to suppress repeated *dials* was destroying an established
|
||
session instead.
|
||
|
||
## Reading `check_invariant`
|
||
|
||
The invariant that matters is **at most one of our routers holds `pppoe0`**.
|
||
The AC's session count is only a proxy for it, and only while the AC enforces
|
||
single-session — under `disable` it does not, so a destroyed router's session
|
||
lingers and the count reads 2 with exactly one live router dialled. That is
|
||
reported as a `WARN`, not a failure. It is not silenced, because an orphaned
|
||
session still occupies the single slot at a real ISP: it is exactly what made
|
||
`deny` take 148s.
|