97efb5abb2
Some checks failed
The three T4 measurements (replace 26s, deny 148s, disable 21s), why deny is the only one that sizes GRACE, and why 148s is a floor rather than a worst case -- idle VMs, and an AC that shares an OVS bridge so `virsh destroy` lets it see the peer vanish in a way a real BRAS never would. Also writes down the broken setter that made the previous numbers fiction, so the next person to touch the matrix knows to check the policy actually landed.
84 lines
3.7 KiB
Markdown
84 lines
3.7 KiB
Markdown
# WAN failover evidence
|
||
|
||
Captured by `labsim/labsim-pppoe-ha-test.sh`. Each directory holds the state of
|
||
both routers and the access concentrator at the end of one test.
|
||
|
||
## The number that sizes GRACE
|
||
|
||
`T4` destroys the master with `virsh destroy` — no LCP Terminate, no PADT, the
|
||
router simply ceases — and times how long until the survivor holds a PPPoE
|
||
session. Run across every policy VyOS can express, because Vodafone's is
|
||
unknown:
|
||
|
||
| `session-control` | takeover | |
|
||
|---|---|---|
|
||
| `replace` | 26s | accel-ppp default; the new auth kills the old session |
|
||
| **`deny`** | **148s** | the AC refuses the survivor until its own dead-peer timer frees the dead session |
|
||
| `disable` | 21s | no single-session enforcement at all |
|
||
|
||
`deny` is the only one that matters for sizing, and `GRACE=300` in
|
||
`migration/vrrp-wan.conf` comes from it. Session polling during that run caught
|
||
the mechanism directly: the destroyed router's session stayed in the AC's table
|
||
while the survivor's dial attempts appeared and were rejected — twice — before
|
||
one finally took.
|
||
|
||
**Treat 148s as a floor, not a worst case.** These are idle 2-vCPU VMs, and the
|
||
AC shares an OVS bridge with the routers, so `virsh destroy` removes the port
|
||
and accel-ppp sees the peer physically vanish. A real BRAS reached over DSL
|
||
never learns our router died; it waits out its own timers, which are longer and
|
||
not ours to know.
|
||
|
||
## How these numbers were nearly wrong
|
||
|
||
Until 2026-09-06 the matrix set the policy with:
|
||
|
||
```sh
|
||
isp "vbash -c 'source /opt/vyatta/etc/functions/script-template; configure; \
|
||
set service pppoe-server session-control $mode; commit; save; exit'"
|
||
```
|
||
|
||
That form never starts a config session. `commit` fails with
|
||
`Invalid command: [commit]` on stderr, which `isp()` discards — so all three
|
||
iterations ran against the accel-ppp default while printing the mode they were
|
||
supposedly testing. `show configuration commands | grep session-control` on the
|
||
ISP VM came back empty after a full run. The matrix reported `deny` at 25s; the
|
||
real figure is 148s, and `GRACE` was sized against the fiction.
|
||
|
||
`isp_session_control()` now drives it from a real script file, reads the value
|
||
back, and skips the iteration rather than measure the wrong policy. A harness
|
||
that reports coverage it does not have is worse than one that reports a failure.
|
||
|
||
## The other tests
|
||
|
||
| | what it proves |
|
||
|---|---|
|
||
| `T0-baseline` | exactly one session, held by the VIP holder |
|
||
| `T3-clean-failover` | `force-fault` moves `pppoe0` in 20–26s |
|
||
| `T5-tengig-down` | 10 gig down → route falls to `pppoe0`, LAN back in 5s |
|
||
| `T8-lease-expiry` | the guard hangs up a stale lease within ~5s |
|
||
| `T11-no-peers-file` | a blessed box with no peers file does not restart-loop |
|
||
| `T12-holdoff-keeps-session` | a 900s flap hold-off does **not** tear down a live session |
|
||
|
||
`T12` exists because it did. `ppp_dial()` checked the hold-off and returned
|
||
before renewing `/run/vrrp-wan/may-dial`; that lease is what `vrrp-wan-guard`
|
||
expires, so tripping the damper stopped the renewal and the guard hung up
|
||
`pppoe0` on the master ~80s later:
|
||
|
||
```
|
||
DIAL FLAP: >=6 attempts in 600s -- holding off 900s
|
||
GUARD: lease stale (81s > 75s) -- hanging up pppoe0
|
||
```
|
||
|
||
A damper meant to suppress repeated *dials* was destroying an established
|
||
session instead.
|
||
|
||
## Reading `check_invariant`
|
||
|
||
The invariant that matters is **at most one of our routers holds `pppoe0`**.
|
||
The AC's session count is only a proxy for it, and only while the AC enforces
|
||
single-session — under `disable` it does not, so a destroyed router's session
|
||
lingers and the count reads 2 with exactly one live router dialled. That is
|
||
reported as a `WARN`, not a failure. It is not silenced, because an orphaned
|
||
session still occupies the single slot at a real ISP: it is exactly what made
|
||
`deny` take 148s.
|