michal/lab
1
0
Fork 0
Code Issues Pull Requests 3 Actions Packages Projects Releases Wiki Activity
Files
main
lab/labsim/wan-failover-evidence/T4-hard-failover-disable/state.txt

21 lines
1.4 KiB
Plaintext
Raw Permalink Normal View History

labsim: both matrices green end to end, numbers reproduced Final confirming run with GRACE=300 and every fix installed: --all ALL PASS (T0 T3 T5 T8 T11 T12) --hard ALL PASS (replace / deny / disable, each policy verified) Second independent measurement of the hard failover, which is what makes the figures trustworthy rather than anecdotal: run 1 run 2 replace 26s 26s deny 148s 141s disable 21s 20s `deny` sits at ~141-148s across both, so it is the AC's dead-peer behaviour and not a one-off; GRACE=300 keeps roughly 2x margin. Corrected an overclaim in PPPOE-HA.md while confirming it: the invariant row read "AC never showed two simdsl sessions", and under session-control=disable it did -- one live, one orphaned from the destroyed router. Only ever one LIVE router dialled, which is the invariant that matters. Said so plainly rather than leaving a table that reads better than the evidence.
2026-09-06 00:42:01 +01:00
=== 2026-09-06T00:39:29+01:00 ===
vrrp-wan: size GRACE from the measured hostile failover, not the theory With the matrix actually setting session-control, T4 timed a destroyed master's takeover at: replace 26s deny 148s <-- sizing case disable 21s `deny` is the case GRACE exists for: the AC refuses the survivor until its own dead-peer timer frees the dead session. The session poller caught it happening -- the destroyed router's session stayed in the table while the survivor's dials appeared and were rejected, twice, before one took at 148s. 148s is well past the lcp-echo-interval(30) x failure(3) = 90s budget that 180 was sized against, leaving 32s of margin. GRACE=300 is ~2x the worst observed. Treat 148s as a floor, not a worst case: these are idle 2-vCPU VMs, and the AC shares an OVS bridge with the routers, so `virsh destroy` removes the port and accel-ppp sees the peer vanish. A real BRAS over DSL never learns our router died and waits out longer timers of its own. The cost is stated in the conf: GRACE is also how long an alive-but-unroutable master holds every VIP before yielding. bond0.53 covers most of that -- a DHCP lease satisfies the check in seconds -- so GRACE only dominates when PPPoE is the last path. Kept the health check's fallback in step, since keepalived runs it with no environment and that number decides mastership if the conf is ever missing. check_invariant no longer fails blind on the AC's session count. That count is only a proxy for "two of our routers dialled", and only while the AC enforces single-session; under `disable` it does not, so a destroyed router's session lingers and the count reads 2 with exactly one live router dialled. The real invariant -- at most one router holds pppoe0 -- is now the failing one, and the stale session is reported as a WARN rather than silenced, because it still occupies the slot at a real ISP and is precisely what made `deny` take 148s.
2026-09-06 00:25:00 +01:00
--- AC sessions ---
ifname | username | ip | ip6 | ip6-dp | calling-sid | rate-limit | state | uptime | rx-bytes | tx-bytes
--------+----------+----------------+-----+--------+-------------------+------------+--------+----------+----------+----------
labsim: both matrices green end to end, numbers reproduced Final confirming run with GRACE=300 and every fix installed: --all ALL PASS (T0 T3 T5 T8 T11 T12) --hard ALL PASS (replace / deny / disable, each policy verified) Second independent measurement of the hard failover, which is what makes the figures trustworthy rather than anecdotal: run 1 run 2 replace 26s 26s deny 148s 141s disable 21s 20s `deny` sits at ~141-148s across both, so it is the AC's dead-peer behaviour and not a one-off; GRACE=300 keeps roughly 2x margin. Corrected an overclaim in PPPOE-HA.md while confirming it: the invariant row read "AC never showed two simdsl sessions", and under session-control=disable it did -- one live, one orphaned from the destroyed router. Only ever one LIVE router dialled, which is the invariant that matters. Said so plainly rather than leaving a table that reads better than the evidence.
2026-09-06 00:42:01 +01:00
ppp0 | simdsl | 198.51.100.136 | | | 52:54:00:e5:95:a2 | | active | 00:02:31 | 438 B | 204 B
ppp1 | simdsl | 198.51.100.137 | | | 52:54:00:4e:0b:56 | | active | 00:00:26 | 1.0 KiB | 204 B
vrrp-wan: size GRACE from the measured hostile failover, not the theory With the matrix actually setting session-control, T4 timed a destroyed master's takeover at: replace 26s deny 148s <-- sizing case disable 21s `deny` is the case GRACE exists for: the AC refuses the survivor until its own dead-peer timer frees the dead session. The session poller caught it happening -- the destroyed router's session stayed in the table while the survivor's dials appeared and were rejected, twice, before one took at 148s. 148s is well past the lcp-echo-interval(30) x failure(3) = 90s budget that 180 was sized against, leaving 32s of margin. GRACE=300 is ~2x the worst observed. Treat 148s as a floor, not a worst case: these are idle 2-vCPU VMs, and the AC shares an OVS bridge with the routers, so `virsh destroy` removes the port and accel-ppp sees the peer vanish. A real BRAS over DSL never learns our router died and waits out longer timers of its own. The cost is stated in the conf: GRACE is also how long an alive-but-unroutable master holds every VIP before yielding. bond0.53 covers most of that -- a DHCP lease satisfies the check in seconds -- so GRACE only dominates when PPPoE is the last path. Kept the health check's fallback in step, since keepalived runs it with no environment and that number decides mastership if the conf is ever missing. check_invariant no longer fails blind on the AC's session count. That count is only a proxy for "two of our routers dialled", and only while the AC enforces single-session; under `disable` it does not, so a destroyed router's session lingers and the count reads 2 with exactly one live router dialled. The real invariant -- at most one router holds pppoe0 -- is now the failing one, and the stale session is reported as a WARN rather than silenced, because it still occupies the slot at a real ISP and is precisely what made `deny` take 148s.
2026-09-06 00:25:00 +01:00
--- 172.31.1.252 ---
--- 172.31.1.253 ---
labsim: both matrices green end to end, numbers reproduced Final confirming run with GRACE=300 and every fix installed: --all ALL PASS (T0 T3 T5 T8 T11 T12) --hard ALL PASS (replace / deny / disable, each policy verified) Second independent measurement of the hard failover, which is what makes the figures trustworthy rather than anecdotal: run 1 run 2 replace 26s 26s deny 148s 141s disable 21s 20s `deny` sits at ~141-148s across both, so it is the AC's dead-peer behaviour and not a one-off; GRACE=300 keeps roughly 2x margin. Corrected an overclaim in PPPOE-HA.md while confirming it: the invariant row read "AC never showed two simdsl sessions", and under session-control=disable it did -- one live, one orphaned from the destroyed router. Only ever one LIVE router dialled, which is the invariant that matters. Said so plainly rather than leaving a table that reads better than the evidence.
2026-09-06 00:42:01 +01:00
vip=172.31.1.1 holds_vip=yes wan_disabled=no wan_up=yes ppp_up=yes ppp_active=yes may_dial=yes lease_age=5 dropin=yes role=master
pppoe0 UNKNOWN 198.51.100.137 peer 198.51.100.1/32
default nhid 60 dev pppoe0 proto static metric 20
Sep 05 23:27:40 vyos vrrp-wan[16198]: not MASTER but bond0.53 enabled -> releasing
Sep 05 23:27:47 vyos vrrp-wan[16840]: bond0.53 disable commit took 7s
Sep 05 23:32:26 vyos vrrp-wan[23806]: MASTER: dialling pppoe0
Sep 05 23:32:26 vyos vrrp-wan[23956]: MASTER with bond0.53 disabled -> enabling
Sep 05 23:32:31 vyos vrrp-wan[24749]: bond0.53 enable commit took 5s
-- Boot 4b0355878444477fb92738339f85843d --
Sep 05 23:39:03 vyos vrrp-wan[5966]: MASTER: dialling pppoe0
Sep 05 23:39:04 vyos vrrp-wan[6120]: MASTER with bond0.53 disabled -> enabling
Sep 05 23:39:09 vyos vrrp-wan[6914]: bond0.53 enable commit took 5s
Reference in New Issue Copy Permalink