# WAN failover evidence

Captured by `labsim/labsim-pppoe-ha-test.sh`. Each directory holds the state of
both routers and the access concentrator at the end of one test.

## The number that sizes GRACE

`T4` destroys the master with `virsh destroy` — no LCP Terminate, no PADT, the
router simply ceases — and times how long until the survivor holds a PPPoE
session. Run across every policy VyOS can express, because Vodafone's is
unknown:

Two independent runs, so these describe the AC's behaviour rather than one-offs:

| `session-control` | takeover | |
|---|---|---|
| `replace` | 26s / 26s | accel-ppp default; the new auth kills the old session |
| **`deny`** | **148s / 141s** | the AC refuses the survivor until its own dead-peer timer frees the dead session |
| `disable` | 21s / 20s | no single-session enforcement at all |

`deny` is the only one that matters for sizing, and `GRACE=300` in
`migration/vrrp-wan.conf` comes from it. Session polling during that run caught
the mechanism directly: the destroyed router's session stayed in the AC's table
while the survivor's dial attempts appeared and were rejected — twice — before
one finally took.

**Treat 148s as a floor, not a worst case.** These are idle 2-vCPU VMs, and the
AC shares an OVS bridge with the routers, so `virsh destroy` removes the port
and accel-ppp sees the peer physically vanish. A real BRAS reached over DSL
never learns our router died; it waits out its own timers, which are longer and
not ours to know.

## How these numbers were nearly wrong

Until 2026-09-06 the matrix set the policy with:

```sh
isp "vbash -c 'source /opt/vyatta/etc/functions/script-template; configure; \
     set service pppoe-server session-control $mode; commit; save; exit'"
```

That form never starts a config session. `commit` fails with
`Invalid command: [commit]` on stderr, which `isp()` discards — so all three
iterations ran against the accel-ppp default while printing the mode they were
supposedly testing. `show configuration commands | grep session-control` on the
ISP VM came back empty after a full run. The matrix reported `deny` at 25s; the
real figure is 148s, and `GRACE` was sized against the fiction.

`isp_session_control()` now drives it from a real script file, reads the value
back, and skips the iteration rather than measure the wrong policy. A harness
that reports coverage it does not have is worse than one that reports a failure.

## The other tests

| | what it proves |
|---|---|
| `T0-baseline` | exactly one session, held by the VIP holder |
| `T3-clean-failover` | `force-fault` moves `pppoe0` in 20–26s |
| `T5-tengig-down` | 10 gig down → route falls to `pppoe0`, LAN back in 5s |
| `T8-lease-expiry` | the guard hangs up a stale lease within ~5s |
| `T11-no-peers-file` | a blessed box with no peers file does not restart-loop |
| `T12-holdoff-keeps-session` | a 900s flap hold-off does **not** tear down a live session |

`T12` exists because it did. `ppp_dial()` checked the hold-off and returned
before renewing `/run/vrrp-wan/may-dial`; that lease is what `vrrp-wan-guard`
expires, so tripping the damper stopped the renewal and the guard hung up
`pppoe0` on the master ~80s later:

```
DIAL FLAP: >=6 attempts in 600s -- holding off 900s
GUARD: lease stale (81s > 75s) -- hanging up pppoe0
```

A damper meant to suppress repeated *dials* was destroying an established
session instead.

## Reading `check_invariant`

The invariant that matters is **at most one of our routers holds `pppoe0`**.
The AC's session count is only a proxy for it, and only while the AC enforces
single-session — under `disable` it does not, so a destroyed router's session
lingers and the count reads 2 with exactly one live router dialled. That is
reported as a `WARN`, not a failure. It is not silenced, because an orphaned
session still occupies the single slot at a real ISP: it is exactly what made
`deny` take 148s.