labsim/wan-failover-evidence/T4-hard-failover-replace/state.txt

=== hard failover, session-control=replace, 2026-09-05T19:20:56+01:00 ===
master labsim-vyos2 (172.31.1.253) destroyed with virsh destroy -- no PADT sent.

-- survivor took over and dialled --
  172.31.1.252  holds_vip=yes wan_up=yes ppp_up=yes ppp_active=yes may_dial=yes role=master

-- AC: exactly one session throughout, from the survivor's MAC --
  ppp0 simdsl 198.51.100.121 calling-sid 52:54:00:e5:95:a2 active 00:07:01

-- returning router did NOT dial on boot (the gate) --
  ConditionResult=no  ActiveState=inactive  NRestarts=0  pppoe0 absent  may_dial=no
  AC session count after its return: 1

-- caveat --
  A VM restart recreates its taps, leaving the OVS bond holding stale members
  (lacp dead, VLAN 1 unreachable). Re-bonded via ovs_bond_router; the harness
  now does this automatically after virsh start.
labsim: hard failover and reboot safety hold; runbook for the production apply Hard failover proven by destroying the master outright (virsh destroy -- no PADT, the case a graceful stop cannot cover): the survivor took the VIPs, dialled, and the access concentrator showed exactly ONE simdsl session from its MAC for the whole seven minutes. When the destroyed router came back it did NOT dial -- ConditionResult=no, ActiveState=inactive, NRestarts=0, pppoe0 absent, may_dial=no -- and the AC session count stayed at 1. That is the gate doing the one job it exists for, on the path that previously had no protection at all. Two more harness bugs of the same family as the last three, both of which reported a working system as broken: - ppp_on() returned an EMPTY string for an unreachable router, and `[ "" = 0 ]` is false, so the hard-failover wait sat for its full timeout waiting for a DESTROYED box to report zero -- long after the survivor had taken over correctly. Absent now means 0. - a VM restart recreates its taps under new names and the OVS bond keeps the old ones: lacp dies, VLAN 1 goes with it, and the box returns reachable on some VLANs and not others. That looked exactly like a failed failover. It is the same stale-membership fault ovs_bond_router already detects, but nothing ran it after a restart; the harness now does. migration/PPPOE-HA.md is the runbook: what the design is, why `disable` cannot work, the measured numbers, the deploy order (vyos002 first, on its own commit, verified on the wire with tcpdump rather than from state), and the one-line rollback. It also records a model hazard found while writing it. The imported baseline captures the RUNNING state, not the safe one -- vyos001 has no `vif 53 disable` because it happens to be master, vyos002 does. An apply performed while vyos002 held the VIP would therefore enable vyos001's WAN too, putting the cloned MAC on both boxes. An override must assert `vif 53 disable` on BOTH, accepting that an apply then briefly disables the current master's 10 gig until the reconciler restores it. Still not applied to production. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH 2026-09-05 19:21:13 +01:00			`=== hard failover, session-control=replace, 2026-09-05T19:20:56+01:00 ===`
			`master labsim-vyos2 (172.31.1.253) destroyed with virsh destroy -- no PADT sent.`

			`-- survivor took over and dialled --`
			`172.31.1.252 holds_vip=yes wan_up=yes ppp_up=yes ppp_active=yes may_dial=yes role=master`

			`-- AC: exactly one session throughout, from the survivor's MAC --`
			`ppp0 simdsl 198.51.100.121 calling-sid 52:54:00:e5:95:a2 active 00:07:01`

			`-- returning router did NOT dial on boot (the gate) --`
			`ConditionResult=no ActiveState=inactive NRestarts=0 pppoe0 absent may_dial=no`
			`AC session count after its return: 1`

			`-- caveat --`
			`A VM restart recreates its taps, leaving the OVS bond holding stale members`
			`(lacp dead, VLAN 1 unreachable). Re-bonded via ovs_bond_router; the harness`
			`now does this automatically after virsh start.`