diff --git a/migration/PPPOE-HA.md b/migration/PPPOE-HA.md index e639eaf..381d151 100644 --- a/migration/PPPOE-HA.md +++ b/migration/PPPOE-HA.md @@ -148,6 +148,42 @@ own reason text — *"with the primary genuinely dead the secondary stays FAULT and nothing holds the gateway. The fix for that is WAN-follows-master, which is a separate change."* This is that change. +## Proven in production — controlled drill, 2026-09-06 + +`migration/wan-drill` force-faulted vyos001 and timed a real failover: + +``` +TAKEOVER OK: vyos002 held the VIP and reached the internet in 52s +FAILBACK OK: 36s +``` + +vyos002 took the VIPs at t+18s and had **both** WANs by t+52s. Failback put +vyos001 back with a WAN in 36s. Total interruption ≈ 88s across two deliberate +transitions. + +**The cloned-MAC lease transfers.** This was the largest untested item in the +whole design — whether the 10 gig ISP would re-issue `87.192.101.48` to +`f0:9f:c2:12:9b:4f` arriving on a different switch port. It did, same address, +within the takeover window. That risk is now closed. + +**Vodafone did not refuse the re-dial**, so its `session-control` behaves like +`replace` rather than the hostile `deny`. `GRACE=300` was sized against the +sim's 148s `deny` case and is therefore comfortable — but it should stay where +it is, because one drill on one evening does not establish the ISP's policy +under all conditions. + +**Vodafone hands out a different IPv4 on every dial**: `83.106.5.72` → +`90.251.153.180` (vyos002) → `90.251.142.103` (vyos001, after failback). +Nothing may be pinned to the PPPoE address. Checked: the HE IPv6 tunnel is +pinned to `87.192.101.48`, which is the **10 gig** (`bond0.53`) and stable +across failover, so `tun0` survived untouched and IPv6 stayed up at 15.5ms. +Anything added later that hardcodes a WAN IP must use the 10 gig one, not +`pppoe0`'s. + +Production takeover (52s) is about twice the sim's `replace` figure (26s), which +is the expected direction: the VP2440s commit under kea, BGP and conntrack while +the sim routers are idle. + ## Outstanding: vyos001's `config.boot` lacks `vif 53 disable` The convention is that **both** `config.boot`s hold `vif 53 disable`, so a @@ -210,11 +246,12 @@ avoids the blip but risks an unbootable router, which is a worse trade. default while printing the mode they were supposedly testing. It reported `deny` at 25s. The real figure is 148s. - **Whether Vodafone honours our LCP Terminate / PADT** on a graceful stop. -- **The cloned-MAC lease** — whether the 10 gig ISP re-issues `87.192.101.48` to - `f0:9f:c2:12:9b:4f` arriving on a different switch port. That risk belongs to - `bond0.53`, not PPPoE, and is the largest untested item in the failover. -- **Real dial time and MTU/MSS under load.** PPPoE was proven on the *USG*; - VyOS dialling Vodafone has never been done. +- ~~**The cloned-MAC lease.**~~ **Answered 2026-09-06**: the drill moved it and + the ISP re-issued `87.192.101.48` to `f0:9f:c2:12:9b:4f` on vyos002's port + within the takeover window. See "Proven in production" above. +- ~~**Whether VyOS can dial Vodafone at all.**~~ **Answered 2026-09-06**: both + routers dialled successfully during the drill. MTU/MSS under sustained load + is still unmeasured, and Vodafone hands out a different IPv4 every dial. - **Timing under load.** The sim routers are idle 2-vCPU VMs; commit latency on the VP2440s under kea + BGP + conntrack will be worse, and commit latency is the dominant term in the `bond0.53` half of a failover.