PPPOE-HA: the drill closed the two biggest unknowns

Controlled failover in production: takeover 52s, failback 36s, ~88s of
interruption across two deliberate transitions.

The cloned-MAC lease TRANSFERS. That was named in this file as "the largest
untested item in any failover" -- whether the 10 gig ISP would re-issue
87.192.101.48 to f0:9f:c2:12:9b:4f arriving on a different switch port. It did,
same address, inside the takeover window. Moved out of "what the sim cannot
prove" rather than left there contradicting the evidence.

VyOS dialling Vodafone also worked from both routers, which had never been done
-- PPPoE was only ever proven on the USG. Vodafone did not refuse either
re-dial, so its session-control behaves like `replace`, not the hostile `deny`.
GRACE stays at 300 anyway: one drill on one evening is not the ISP's policy
under all conditions.

Worth knowing for anything added later: Vodafone hands out a DIFFERENT IPv4 on
every dial (83.106.5.72 -> 90.251.153.180 -> 90.251.142.103). Nothing may be
pinned to the pppoe0 address. Checked the HE IPv6 tunnel specifically, since it
carries a hardcoded source-address -- it is pinned to 87.192.101.48, which is
the 10 gig and stable across failover, so tun0 was untouched and IPv6 stayed up
at 15.5ms.

migration/PPPOE-HA.md

@@ -148,6 +148,42 @@ own reason text — *"with the primary genuinely dead the secondary stays FAULT
 and nothing holds the gateway. The fix for that is WAN-follows-master, which is
 a separate change."* This is that change.
 
+## Proven in production — controlled drill, 2026-09-06
+
+`migration/wan-drill` force-faulted vyos001 and timed a real failover:
+
+```
+TAKEOVER OK: vyos002 held the VIP and reached the internet in 52s
+FAILBACK OK: 36s
+```
+
+vyos002 took the VIPs at t+18s and had **both** WANs by t+52s. Failback put
+vyos001 back with a WAN in 36s. Total interruption ≈ 88s across two deliberate
+transitions.
+
+**The cloned-MAC lease transfers.** This was the largest untested item in the
+whole design — whether the 10 gig ISP would re-issue `87.192.101.48` to
+`f0:9f:c2:12:9b:4f` arriving on a different switch port. It did, same address,
+within the takeover window. That risk is now closed.
+
+**Vodafone did not refuse the re-dial**, so its `session-control` behaves like
+`replace` rather than the hostile `deny`. `GRACE=300` was sized against the
+sim's 148s `deny` case and is therefore comfortable — but it should stay where
+it is, because one drill on one evening does not establish the ISP's policy
+under all conditions.
+
+**Vodafone hands out a different IPv4 on every dial**: `83.106.5.72` →
+`90.251.153.180` (vyos002) → `90.251.142.103` (vyos001, after failback).
+Nothing may be pinned to the PPPoE address. Checked: the HE IPv6 tunnel is
+pinned to `87.192.101.48`, which is the **10 gig** (`bond0.53`) and stable
+across failover, so `tun0` survived untouched and IPv6 stayed up at 15.5ms.
+Anything added later that hardcodes a WAN IP must use the 10 gig one, not
+`pppoe0`'s.
+
+Production takeover (52s) is about twice the sim's `replace` figure (26s), which
+is the expected direction: the VP2440s commit under kea, BGP and conntrack while
+the sim routers are idle.
+
 ## Outstanding: vyos001's `config.boot` lacks `vif 53 disable`
 
 The convention is that **both** `config.boot`s hold `vif 53 disable`, so a
@@ -210,11 +246,12 @@ avoids the blip but risks an unbootable router, which is a worse trade.
   default while printing the mode they were supposedly testing. It reported
   `deny` at 25s. The real figure is 148s.
 - **Whether Vodafone honours our LCP Terminate / PADT** on a graceful stop.
-- **The cloned-MAC lease** — whether the 10 gig ISP re-issues `87.192.101.48` to
+- ~~**The cloned-MAC lease.**~~ **Answered 2026-09-06**: the drill moved it and
-  `f0:9f:c2:12:9b:4f` arriving on a different switch port. That risk belongs to
+  the ISP re-issued `87.192.101.48` to `f0:9f:c2:12:9b:4f` on vyos002's port
-  `bond0.53`, not PPPoE, and is the largest untested item in the failover.
+  within the takeover window. See "Proven in production" above.
-- **Real dial time and MTU/MSS under load.** PPPoE was proven on the *USG*;
+- ~~**Whether VyOS can dial Vodafone at all.**~~ **Answered 2026-09-06**: both
-  VyOS dialling Vodafone has never been done.
+  routers dialled successfully during the drill. MTU/MSS under sustained load
+  is still unmeasured, and Vodafone hands out a different IPv4 every dial.
 - **Timing under load.** The sim routers are idle 2-vCPU VMs; commit latency on
   the VP2440s under kea + BGP + conntrack will be worse, and commit latency is
   the dominant term in the `bond0.53` half of a failover.