84 lines
3.7 KiB
Markdown
84 lines
3.7 KiB
Markdown
|
# WAN failover evidence
|
|||
|
|
|
|||
|
|
Captured by `labsim/labsim-pppoe-ha-test.sh`. Each directory holds the state of
|
|||
|
|
both routers and the access concentrator at the end of one test.
|
|||
|
|
|
|||
|
|
## The number that sizes GRACE
|
|||
|
|
|
|||
|
|
`T4` destroys the master with `virsh destroy` — no LCP Terminate, no PADT, the
|
|||
|
|
router simply ceases — and times how long until the survivor holds a PPPoE
|
|||
|
|
session. Run across every policy VyOS can express, because Vodafone's is
|
|||
|
|
unknown:
|
|||
|
|
|
|||
|
|
| `session-control` | takeover | |
|
|||
|
|
|---|---|---|
|
|||
|
|
| `replace` | 26s | accel-ppp default; the new auth kills the old session |
|
|||
|
|
| **`deny`** | **148s** | the AC refuses the survivor until its own dead-peer timer frees the dead session |
|
|||
|
|
| `disable` | 21s | no single-session enforcement at all |
|
|||
|
|
|
|||
|
|
`deny` is the only one that matters for sizing, and `GRACE=300` in
|
|||
|
|
`migration/vrrp-wan.conf` comes from it. Session polling during that run caught
|
|||
|
|
the mechanism directly: the destroyed router's session stayed in the AC's table
|
|||
|
|
while the survivor's dial attempts appeared and were rejected — twice — before
|
|||
|
|
one finally took.
|
|||
|
|
|
|||
|
|
**Treat 148s as a floor, not a worst case.** These are idle 2-vCPU VMs, and the
|
|||
|
|
AC shares an OVS bridge with the routers, so `virsh destroy` removes the port
|
|||
|
|
and accel-ppp sees the peer physically vanish. A real BRAS reached over DSL
|
|||
|
|
never learns our router died; it waits out its own timers, which are longer and
|
|||
|
|
not ours to know.
|
|||
|
|
|
|||
|
|
## How these numbers were nearly wrong
|
|||
|
|
|
|||
|
|
Until 2026-09-06 the matrix set the policy with:
|
|||
|
|
|
|||
|
|
```sh
|
|||
|
|
isp "vbash -c 'source /opt/vyatta/etc/functions/script-template; configure; \
|
|||
|
|
set service pppoe-server session-control $mode; commit; save; exit'"
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
That form never starts a config session. `commit` fails with
|
|||
|
|
`Invalid command: [commit]` on stderr, which `isp()` discards — so all three
|
|||
|
|
iterations ran against the accel-ppp default while printing the mode they were
|
|||
|
|
supposedly testing. `show configuration commands | grep session-control` on the
|
|||
|
|
ISP VM came back empty after a full run. The matrix reported `deny` at 25s; the
|
|||
|
|
real figure is 148s, and `GRACE` was sized against the fiction.
|
|||
|
|
|
|||
|
|
`isp_session_control()` now drives it from a real script file, reads the value
|
|||
|
|
back, and skips the iteration rather than measure the wrong policy. A harness
|
|||
|
|
that reports coverage it does not have is worse than one that reports a failure.
|
|||
|
|
|
|||
|
|
## The other tests
|
|||
|
|
|
|||
|
|
| | what it proves |
|
|||
|
|
|---|---|
|
|||
|
|
| `T0-baseline` | exactly one session, held by the VIP holder |
|
|||
|
|
| `T3-clean-failover` | `force-fault` moves `pppoe0` in 20–26s |
|
|||
|
|
| `T5-tengig-down` | 10 gig down → route falls to `pppoe0`, LAN back in 5s |
|
|||
|
|
| `T8-lease-expiry` | the guard hangs up a stale lease within ~5s |
|
|||
|
|
| `T11-no-peers-file` | a blessed box with no peers file does not restart-loop |
|
|||
|
|
| `T12-holdoff-keeps-session` | a 900s flap hold-off does **not** tear down a live session |
|
|||
|
|
|
|||
|
|
`T12` exists because it did. `ppp_dial()` checked the hold-off and returned
|
|||
|
|
before renewing `/run/vrrp-wan/may-dial`; that lease is what `vrrp-wan-guard`
|
|||
|
|
expires, so tripping the damper stopped the renewal and the guard hung up
|
|||
|
|
`pppoe0` on the master ~80s later:
|
|||
|
|
|
|||
|
|
```
|
|||
|
|
DIAL FLAP: >=6 attempts in 600s -- holding off 900s
|
|||
|
|
GUARD: lease stale (81s > 75s) -- hanging up pppoe0
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
A damper meant to suppress repeated *dials* was destroying an established
|
|||
|
|
session instead.
|
|||
|
|
|
|||
|
|
## Reading `check_invariant`
|
|||
|
|
|
|||
|
|
The invariant that matters is **at most one of our routers holds `pppoe0`**.
|
|||
|
|
The AC's session count is only a proxy for it, and only while the AC enforces
|
|||
|
|
single-session — under `disable` it does not, so a destroyed router's session
|
|||
|
|
lingers and the count reads 2 with exactly one live router dialled. That is
|
|||
|
|
reported as a `WARN`, not a failure. It is not silenced, because an orphaned
|
|||
|
|
session still occupies the single slot at a real ISP: it is exactly what made
|
|||
|
|
`deny` take 148s.
|