3 Commits

Author	SHA1	Message	Date
Michal	5ed0e4888a	labsim: both matrices green end to end, numbers reproduced ... Final confirming run with GRACE=300 and every fix installed: --all ALL PASS (T0 T3 T5 T8 T11 T12) --hard ALL PASS (replace / deny / disable, each policy verified) Second independent measurement of the hard failover, which is what makes the figures trustworthy rather than anecdotal: run 1 run 2 replace 26s 26s deny 148s 141s disable 21s 20s `deny` sits at ~141-148s across both, so it is the AC's dead-peer behaviour and not a one-off; GRACE=300 keeps roughly 2x margin. Corrected an overclaim in PPPOE-HA.md while confirming it: the invariant row read "AC never showed two simdsl sessions", and under session-control=disable it did -- one live, one orphaned from the destroyed router. Only ever one LIVE router dialled, which is the invariant that matters. Said so plainly rather than leaving a table that reads better than the evidence.	2026-09-06 00:42:01 +01:00
Michal	6987b324f1	vrrp-wan: size GRACE from the measured hostile failover, not the theory ... With the matrix actually setting session-control, T4 timed a destroyed master's takeover at: replace 26s deny 148s <-- sizing case disable 21s `deny` is the case GRACE exists for: the AC refuses the survivor until its own dead-peer timer frees the dead session. The session poller caught it happening -- the destroyed router's session stayed in the table while the survivor's dials appeared and were rejected, twice, before one took at 148s. 148s is well past the lcp-echo-interval(30) x failure(3) = 90s budget that 180 was sized against, leaving 32s of margin. GRACE=300 is ~2x the worst observed. Treat 148s as a floor, not a worst case: these are idle 2-vCPU VMs, and the AC shares an OVS bridge with the routers, so `virsh destroy` removes the port and accel-ppp sees the peer vanish. A real BRAS over DSL never learns our router died and waits out longer timers of its own. The cost is stated in the conf: GRACE is also how long an alive-but-unroutable master holds every VIP before yielding. bond0.53 covers most of that -- a DHCP lease satisfies the check in seconds -- so GRACE only dominates when PPPoE is the last path. Kept the health check's fallback in step, since keepalived runs it with no environment and that number decides mastership if the conf is ever missing. check_invariant no longer fails blind on the AC's session count. That count is only a proxy for "two of our routers dialled", and only while the AC enforces single-session; under `disable` it does not, so a destroyed router's session lingers and the count reads 2 with exactly one live router dialled. The real invariant -- at most one router holds pppoe0 -- is now the failing one, and the stale session is reported as a WARN rather than silenced, because it still occupies the slot at a real ISP and is precisely what made `deny` take 148s.	2026-09-06 00:25:00 +01:00
Michal	4efd70c987	vyos: move PPPoE off the config plane onto a gated systemd unit ... PPPoE HA could not work as written, and the reason is structural rather than a bug: `set interfaces pppoe pppoe0 disable` and `delete` are handled identically by interfaces_pppoe.py -- both UNLINK /etc/ppp/peers/pppoe0. That path is pppd's own options file, so the resting state destroyed exactly what the promotion path needed, and `ppp@pppoe0` restart-looped against it (observed: 47 restarts, zero sessions at the access concentrator). It also made op-mode `connect interface pppoe0` unusable, and put every failover behind a priority-322 commit where one unrelated invalid node fails the whole thing -- which has already taken the 10 gig down once. pppoe0 is now configured identically and ENABLED on both routers, so the peers file always exists, and dialling is gated by a drop-in on the unit: ConditionPathExists=/run/vrrp-wan/may-dial ConditionPathExists=/etc/ppp/peers/pppoe0 /run is tmpfs, so the gate is shut at boot and neither box can dial before VRRP has decided. That matters more than it first appears: with the node enabled, interfaces_pppoe.py restarts ppp on EVERY commit touching the pppoe subtree when the daemon is not running -- so the backup actively tries to dial whenever anything commits. The gate is the only thing making that a no-op, which is why vrrp-wan-reconcile now refuses to bless a box whose drop-in is missing: /etc is per-image, and a VyOS upgrade would otherwise silently remove the protection. may-dial is a LEASE, not a flag. ConditionPathExists is evaluated at start only -- it can prevent a dial, never revoke one -- so a reconciler that stops running while its box is demoted would keep the one ISP session for ever. The reconciler renews the lease; a new 5s vrrp-wan-guard revokes it, and only ever revokes. It fired correctly first time: "GUARD: lease stale (81s > 75s)". Also: remove-then-stop on release (the file's absence blocks a NEW start that a concurrent commit would trigger); a flap damper, because two routers that both believe they hold the VIP will both dial and each dial kills the other's session -- against a real ISP that is how an account gets rate-limited; and a guard on `cfg` returning empty under commit-lock contention, which had already produced one spurious "releasing" on a box that needed nothing. GRACE 90 -> 180. accel-ppp's dead-peer budget is lcp-echo-interval(30) x failure(3) = 90s, so the old value sat exactly on the boundary: a hard failover into an AC that does not replace the stale session would fail its own check, shed the VIPs, and leave both routers in FAULT. The sim could not have tested any of this. Both routers now get the identical WAN -- the secondary had none "because two PPPoE clients sharing one credential is a different failure mode than anything production has", which is backwards: that IS production. It also left the pair incomparable, ten NAT rules against none. Safety now comes from resting state, not asymmetry. Three more things the sim was hiding: - the drift check's secondary regex omitted interfaces pppoe/bonding, nat source and protocols failover, so it reported "in sync" for a box with no WAN at all; - the VRRP health-check and transition-script hooks existed on both live VMs and in NEITHER generator -- the mechanism under test was pure undetected drift; - labsim-vyos's only default route was the libvirt-NAT scaffold, so every "the LAN still has internet" verdict on it was answered by eth2 rather than the WAN. --drop-scaffold applied; the earlier DHCP-failover proof is being re-run because of it. vrrp-wan-install ends the other half of that: the sim's previous proof came from scripts hand-`sed`-ed in place, so the tested behaviour was not the committed behaviour. `--check` now makes that a hard failure. First green run: master holds both WANs, backup released, and the AC reports exactly ONE session. sim-net-apply.sh check: all four in sync. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH	2026-09-05 18:50:16 +01:00