labsim: hard failover and reboot safety hold; runbook for the production apply

Hard failover proven by destroying the master outright (virsh destroy -- no PADT,
the case a graceful stop cannot cover): the survivor took the VIPs, dialled, and
the access concentrator showed exactly ONE simdsl session from its MAC for the
whole seven minutes. When the destroyed router came back it did NOT dial --
ConditionResult=no, ActiveState=inactive, NRestarts=0, pppoe0 absent, may_dial=no
-- and the AC session count stayed at 1. That is the gate doing the one job it
exists for, on the path that previously had no protection at all.

Two more harness bugs of the same family as the last three, both of which
reported a working system as broken:

  - ppp_on() returned an EMPTY string for an unreachable router, and `[ "" = 0 ]`
    is false, so the hard-failover wait sat for its full timeout waiting for a
    DESTROYED box to report zero -- long after the survivor had taken over
    correctly. Absent now means 0.
  - a VM restart recreates its taps under new names and the OVS bond keeps the
    old ones: lacp dies, VLAN 1 goes with it, and the box returns reachable on
    some VLANs and not others. That looked exactly like a failed failover. It is
    the same stale-membership fault ovs_bond_router already detects, but nothing
    ran it after a restart; the harness now does.

migration/PPPOE-HA.md is the runbook: what the design is, why `disable` cannot
work, the measured numbers, the deploy order (vyos002 first, on its own commit,
verified on the wire with tcpdump rather than from state), and the one-line
rollback.

It also records a model hazard found while writing it. The imported baseline
captures the RUNNING state, not the safe one -- vyos001 has no `vif 53 disable`
because it happens to be master, vyos002 does. An apply performed while vyos002
held the VIP would therefore enable vyos001's WAN too, putting the cloned MAC on
both boxes. An override must assert `vif 53 disable` on BOTH, accepting that an
apply then briefly disables the current master's 10 gig until the reconciler
restores it.

Still not applied to production.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH

labsim/labsim-pppoe-ha-test.sh

@@ -56,7 +56,12 @@ FAILED=0
 ac_sessions() { isp '/opt/vyatta/bin/vyatta-op-cmd-wrapper show pppoe-server sessions' \
                   | grep -c ' simdsl ' || true; }
 ac_detail()   { isp '/opt/vyatta/bin/vyatta-op-cmd-wrapper show pppoe-server sessions'; }
-ppp_on()      { r "$1" 'ip -4 addr show pppoe0 2>/dev/null | grep -c inet' | tr -d ' \n'; }
+# A destroyed or unreachable router is emphatically NOT holding pppoe0, but ssh
+# returns an EMPTY string rather than 0 -- and `[ "" = 0 ]` is false, so a
+# hard-failover test waited for the dead box to "report" zero and hung until its
+# timeout, long after the survivor had taken over correctly. Default to 0.
+ppp_on()      { local v; v="$(r "$1" 'ip -4 addr show pppoe0 2>/dev/null | grep -c inet' | tr -d ' \n')"
+                echo "${v:-0}"; }
 holder()      { for h in "$R1" "$R2"; do
                   [ "$(r "$h" "ip -4 -o addr show | grep -c ' ${VIP}/'" | tr -d ' \n')" != 0 ] \
                     && { echo "$h"; return; }; done; echo none; }
@@ -219,9 +224,57 @@ T8() {  # lease expiry: the guard must hang up a demoted-but-unreconciled box
     settle 24 >/dev/null
 }
 
+
+T4() {  # hard failover across all three AC session-control policies
+    log "T4  hard failover (destroy the master) x session-control"
+    # Vodafone's policy is unknowable from here, so prove the design survives
+    # every one VyOS can express. `replace` is the accel-ppp default and the
+    # friendly case; `deny` is the hostile one, where the AC refuses the second
+    # session until its own dead-peer timer (lcp-echo-interval 30 x failure 3 =
+    # 90s) frees the first -- which is exactly why GRACE is no longer 90.
+    local mode from to vm t0 t1
+    for mode in replace deny disable; do
+        log "    --- session-control=$mode ---"
+        isp "vbash -c 'source /opt/vyatta/etc/functions/script-template; configure; set service pppoe-server session-control $mode; commit; save; exit'" >/dev/null 2>&1
+        sleep 5
+        from="$(holder)"; to=$([ "$from" = "$R1" ] && echo "$R2" || echo "$R1")
+        vm=$([ "$from" = "$R1" ] && echo labsim-vyos || echo labsim-vyos2)
+        [ "$from" = none ] && { fail "no master before $mode run"; continue; }
+        log "    destroying $vm (master=$from), expecting $to"
+        t0=$(date +%s)
+        sudo virsh destroy "$vm" >/dev/null 2>&1
+        if settle_on "$to" 48; then
+            t1=$(date +%s)
+            pass "$mode: pppoe0 reached $to in $((t1-t0))s"
+        else
+            fail "$mode: $to never dialled within 240s"
+        fi
+        check_invariant
+        save_evidence "T4-hard-failover-$mode"
+        sudo virsh start "$vm" >/dev/null 2>&1
+        # Re-bond. A VM restart recreates its taps under NEW names, and the OVS
+        # bond keeps the old ones -- lacp dies, VLAN 1 goes with it, and the box
+        # comes back reachable on some VLANs but not others. ovs_bond_router
+        # detects the stale membership and rebuilds, but nothing runs it
+        # automatically, so a destroy/start test must do it or the survivor
+        # looks like a failover failure.
+        ( source "$SCRIPT_DIR/lib.sh"; source "$SCRIPT_DIR/ovs.sh"; selected_vlans
+          LAG_NAME=$([ "$vm" = labsim-vyos ] && echo lag-vyos || echo lag-vyos2)
+          ovs_bond_router "$vm" ) >/dev/null 2>&1
+        # Give the returning box time to boot and settle as BACKUP before the
+        # next iteration; it must NOT dial on the way up.
+        sleep 90
+        [ "$(ppp_on "$from")" = 0 ] && pass "$mode: $from did not dial on reboot" \
+            || fail "$mode: $from dialled on reboot (gate failed)"
+    done
+    isp "vbash -c 'source /opt/vyatta/etc/functions/script-template; configure; set service pppoe-server session-control replace; commit; save; exit'" >/dev/null 2>&1
+    log "    AC restored to session-control=replace"
+}
+
 case "${1:---all}" in
-  --list) echo "T0 baseline | T3 clean failover | T5 10gig-down | T8 lease expiry | T11 no-peers-file"; exit 0 ;;
+  --list) echo "T0 baseline | T3 clean failover | T4 hard failover x policy | T5 10gig-down | T8 lease expiry | T11 no-peers-file"; exit 0 ;;
   --all)  preflight || exit 1; T0; T3; T5; T8; T11 ;;
+  --hard) preflight || exit 1; T4 ;;
   *)      preflight || exit 1; "$1" ;;
 esac