6 Commits

Author	SHA1	Message	Date
Michal	6987b324f1	vrrp-wan: size GRACE from the measured hostile failover, not the theory ... With the matrix actually setting session-control, T4 timed a destroyed master's takeover at: replace 26s deny 148s <-- sizing case disable 21s `deny` is the case GRACE exists for: the AC refuses the survivor until its own dead-peer timer frees the dead session. The session poller caught it happening -- the destroyed router's session stayed in the table while the survivor's dials appeared and were rejected, twice, before one took at 148s. 148s is well past the lcp-echo-interval(30) x failure(3) = 90s budget that 180 was sized against, leaving 32s of margin. GRACE=300 is ~2x the worst observed. Treat 148s as a floor, not a worst case: these are idle 2-vCPU VMs, and the AC shares an OVS bridge with the routers, so `virsh destroy` removes the port and accel-ppp sees the peer vanish. A real BRAS over DSL never learns our router died and waits out longer timers of its own. The cost is stated in the conf: GRACE is also how long an alive-but-unroutable master holds every VIP before yielding. bond0.53 covers most of that -- a DHCP lease satisfies the check in seconds -- so GRACE only dominates when PPPoE is the last path. Kept the health check's fallback in step, since keepalived runs it with no environment and that number decides mastership if the conf is ever missing. check_invariant no longer fails blind on the AC's session count. That count is only a proxy for "two of our routers dialled", and only while the AC enforces single-session; under `disable` it does not, so a destroyed router's session lingers and the count reads 2 with exactly one live router dialled. The real invariant -- at most one router holds pppoe0 -- is now the failing one, and the stale session is reported as a WARN rather than silenced, because it still occupies the slot at a real ISP and is precisely what made `deny` take 148s.	2026-09-06 00:25:00 +01:00
Michal	9221c71ff0	labsim: the session-control matrix was never setting session-control ... T4 prints "--- session-control=deny ---" and then measures whatever policy the AC already had. The setter was isp "vbash -c 'source script-template; configure; set ...; commit; save'" and that form does not start a config session at all -- commit dies with "Invalid command: [commit]" on stderr, which isp() discards. `show configuration commands | grep session-control` on the ISP VM returned nothing after a full matrix run: all three iterations had run against the accel-ppp default. The labels were fiction, and a harness that reports coverage it does not have is worse than one that reports a failure. Driving it from a real script FILE works. isp_session_control() does that, reads the value back, and fails the iteration if it disagrees rather than measuring the wrong policy. `session-control` is a valid node here (checked the template dir on VyOS 2026.08.12-0831-rolling), so this was purely the invocation. Staged the two Pulumi overrides in migration/ rather than adding them to kubernetes-deployment: another agent runs `pulumi up` on that repo, so merging `remove: pppoe0 disable` before the gate exists on vyos002 would let it dial and take the single Vodafone session off the live master. Ordering is written at the top of the file.	2026-09-06 00:15:45 +01:00
Michal	4d47b609a2	PPPOE-HA: record the two failure modes found by running the thing ... Both were invisible to inspection and only appeared under the hard matrix: the flap damper tearing down an established session via the lease it stopped renewing, and a missing peers file being completely silent. The second has a production edge worth spelling out -- an unsaved commit reverts on reboot and takes pppoe0 with it, leaving a standby that can never take over while looking perfectly healthy.	2026-09-06 00:11:32 +01:00
Michal	b659e0d47e	vrrp-wan: a flap holdoff must not tear down a live WAN session ... ppp_dial() checked the flap holdoff and returned BEFORE renewing /run/vrrp-wan/may-dial. That lease is what vrrp-wan-guard expires after LEASE_TTL, so tripping the damper stopped the renew and the guard hung up pppoe0 on the MASTER ~80s later. A damper meant to suppress repeated DIALS was tearing down a working WAN instead. Observed in labsim, end to end: DIAL FLAP: >=6 attempts in 600s -- holding off 900s GUARD: lease stale (81s > 75s) -- hanging up pppoe0 An established session now outranks every check below it: ppp_active renews the lease and returns first. Everything after it only decides whether to start a NEW session. Two supporting fixes for how that storm started. The dial attempts were all no-ops because /etc/ppp/peers/pppoe0 was missing, and nothing said so -- systemd logs "skipped because of an unmet condition check" exactly once and the gate looks identical to a healthy backup. ppp_dial() now reports it, and distinguishes "configured but not rendered" (re-commit the subtree) from "no pppoe0 in config at all", which is what a reboot leaves behind when a commit was never saved. That is precisely how the sim secondary lost its WAN. Also `cat | wc -l` rather than `wc -l < file`: redirections are applied left to right, so the missing-file error escapes the 2>/dev/null on every first-ever dial. Harness: T11 copied-then-removed instead of mv, and verifies the restore -- losing that file strands a router permanently, which cost a debugging session. preflight now refuses to run if either router lacks the peers file or the pppoe0 config, since every failover result would otherwise be a false negative blamed on the ISP. New T12 forges a 900s holdoff against a live session and asserts it survives.	2026-09-06 00:04:41 +01:00
Michal	d626750010	labsim: hard failover and reboot safety hold; runbook for the production apply ... Hard failover proven by destroying the master outright (virsh destroy -- no PADT, the case a graceful stop cannot cover): the survivor took the VIPs, dialled, and the access concentrator showed exactly ONE simdsl session from its MAC for the whole seven minutes. When the destroyed router came back it did NOT dial -- ConditionResult=no, ActiveState=inactive, NRestarts=0, pppoe0 absent, may_dial=no -- and the AC session count stayed at 1. That is the gate doing the one job it exists for, on the path that previously had no protection at all. Two more harness bugs of the same family as the last three, both of which reported a working system as broken: - ppp_on() returned an EMPTY string for an unreachable router, and `[ "" = 0 ]` is false, so the hard-failover wait sat for its full timeout waiting for a DESTROYED box to report zero -- long after the survivor had taken over correctly. Absent now means 0. - a VM restart recreates its taps under new names and the OVS bond keeps the old ones: lacp dies, VLAN 1 goes with it, and the box returns reachable on some VLANs and not others. That looked exactly like a failed failover. It is the same stale-membership fault ovs_bond_router already detects, but nothing ran it after a restart; the harness now does. migration/PPPOE-HA.md is the runbook: what the design is, why `disable` cannot work, the measured numbers, the deploy order (vyos002 first, on its own commit, verified on the wire with tcpdump rather than from state), and the one-line rollback. It also records a model hazard found while writing it. The imported baseline captures the RUNNING state, not the safe one -- vyos001 has no `vif 53 disable` because it happens to be master, vyos002 does. An apply performed while vyos002 held the VIP would therefore enable vyos001's WAN too, putting the cloned MAC on both boxes. An override must assert `vif 53 disable` on BOTH, accepting that an apply then briefly disables the current master's 10 gig until the reconciler restores it. Still not applied to production. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH	2026-09-05 19:21:13 +01:00
Michal	93fed7826b	labsim: PPPoE HA passes the matrix, and the health check had a real flap bug ... Full run green: baseline (one AC session, held by the VIP holder), clean failover (pppoe0 moves in 26s, old master releases), 10 gig down (route falls to pppoe0 and the LAN is back online in 5s), lease expiry (the guard hangs up), and a missing peers file (NRestarts=0, no loop). Evidence in labsim/wan-failover-evidence/. The 10 gig test found a genuine bug in vrrp-wan-health, not in the sim. GRACE was measured from PROMOTION, so an established master had no grace at all -- after hours of uptime `now - since` far exceeds any window. The first moment bond0.53 went down while pppoe0 was mid-redial, the master failed its own check, shed every VIP, and the peer inherited the same WAN outage and did the same. A brief WAN blip would have flapped the production pair. The stamp is now refreshed on every healthy tick, so grace measures time since the box last demonstrably HAD a WAN -- survivable wherever the gap happens, not only just after a promotion. Three harness bugs, all the same shape, all of which produced a confident wrong answer before being caught: - waiting for "exactly one pppoe0 holder" returns INSTANTLY during a handover, because it was already true. The useful question is who holds it. - judging connectivity on a single ping 20s after a link drop reported an outage that had already healed. Poll, do not sample. - `-o PreferredAuthentications=password` suits the routers but not the Alpine LAN VMs, whose sshd offers keyboard-interactive: ssh exited 255 before running anything and the test read that as "the LAN lost the internet". A tcpdump on the router showed the pings leaving pppoe0 NATed to 198.51.100.117 and the replies coming back the whole time. An exit code that can mean "the network is broken" or "I could not log in" is not a connectivity test, so the check now asserts on what the guest reported. That last one is why the harness asks the routers and the access concentrator rather than a client, and why it refuses to run at all while either router still has a default route via eth2 -- the libvirt-NAT scaffold answers connectivity checks the WAN under test would have failed. Still to run: hard failover (destroy the master), and the session-control replace/deny/disable axis that brackets Vodafone's unknown behaviour and sets the final GRACE. Nothing applied to production. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH	2026-09-05 19:11:07 +01:00