michal/lab
1
0
Fork 0
Code Issues Pull Requests 3 Actions Packages Projects Releases Wiki Activity
Files
bda58545638c4c355bf44da8a60960e271618621
lab/labsim/wan-failover-evidence/T5-tengig-down/state.txt

31 lines
2.1 KiB
Plaintext
Raw Normal View History

labsim: both matrices green end to end, numbers reproduced Final confirming run with GRACE=300 and every fix installed: --all ALL PASS (T0 T3 T5 T8 T11 T12) --hard ALL PASS (replace / deny / disable, each policy verified) Second independent measurement of the hard failover, which is what makes the figures trustworthy rather than anecdotal: run 1 run 2 replace 26s 26s deny 148s 141s disable 21s 20s `deny` sits at ~141-148s across both, so it is the AC's dead-peer behaviour and not a one-off; GRACE=300 keeps roughly 2x margin. Corrected an overclaim in PPPOE-HA.md while confirming it: the invariant row read "AC never showed two simdsl sessions", and under session-control=disable it did -- one live, one orphaned from the destroyed router. Only ever one LIVE router dialled, which is the invariant that matters. Said so plainly rather than leaving a table that reads better than the evidence.
2026-09-06 00:42:01 +01:00
=== 2026-09-06T00:28:31+01:00 ===
labsim: PPPoE HA passes the matrix, and the health check had a real flap bug Full run green: baseline (one AC session, held by the VIP holder), clean failover (pppoe0 moves in 26s, old master releases), 10 gig down (route falls to pppoe0 and the LAN is back online in 5s), lease expiry (the guard hangs up), and a missing peers file (NRestarts=0, no loop). Evidence in labsim/wan-failover-evidence/. The 10 gig test found a genuine bug in vrrp-wan-health, not in the sim. GRACE was measured from PROMOTION, so an established master had no grace at all -- after hours of uptime `now - since` far exceeds any window. The first moment bond0.53 went down while pppoe0 was mid-redial, the master failed its own check, shed every VIP, and the peer inherited the same WAN outage and did the same. A brief WAN blip would have flapped the production pair. The stamp is now refreshed on every healthy tick, so grace measures time since the box last demonstrably HAD a WAN -- survivable wherever the gap happens, not only just after a promotion. Three harness bugs, all the same shape, all of which produced a confident wrong answer before being caught: - waiting for "exactly one pppoe0 holder" returns INSTANTLY during a handover, because it was already true. The useful question is who holds it. - judging connectivity on a single ping 20s after a link drop reported an outage that had already healed. Poll, do not sample. - `-o PreferredAuthentications=password` suits the routers but not the Alpine LAN VMs, whose sshd offers keyboard-interactive: ssh exited 255 before running anything and the test read that as "the LAN lost the internet". A tcpdump on the router showed the pings leaving pppoe0 NATed to 198.51.100.117 and the replies coming back the whole time. An exit code that can mean "the network is broken" or "I could not log in" is not a connectivity test, so the check now asserts on what the guest reported. That last one is why the harness asks the routers and the access concentrator rather than a client, and why it refuses to run at all while either router still has a default route via eth2 -- the libvirt-NAT scaffold answers connectivity checks the WAN under test would have failed. Still to run: hard failover (destroy the master), and the session-control replace/deny/disable axis that brackets Vodafone's unknown behaviour and sets the final GRACE. Nothing applied to production. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-09-05 19:11:07 +01:00
--- AC sessions ---
ifname | username | ip | ip6 | ip6-dp | calling-sid | rate-limit | state | uptime | rx-bytes | tx-bytes
--------+----------+----------------+-----+--------+-------------------+------------+--------+----------+----------+----------
labsim: both matrices green end to end, numbers reproduced Final confirming run with GRACE=300 and every fix installed: --all ALL PASS (T0 T3 T5 T8 T11 T12) --hard ALL PASS (replace / deny / disable, each policy verified) Second independent measurement of the hard failover, which is what makes the figures trustworthy rather than anecdotal: run 1 run 2 replace 26s 26s deny 148s 141s disable 21s 20s `deny` sits at ~141-148s across both, so it is the AC's dead-peer behaviour and not a one-off; GRACE=300 keeps roughly 2x margin. Corrected an overclaim in PPPOE-HA.md while confirming it: the invariant row read "AC never showed two simdsl sessions", and under session-control=disable it did -- one live, one orphaned from the destroyed router. Only ever one LIVE router dialled, which is the invariant that matters. Said so plainly rather than leaving a table that reads better than the evidence.
2026-09-06 00:42:01 +01:00
ppp0 | simdsl | 198.51.100.132 | | | 52:54:00:e5:95:a2 | | active | 00:00:52 | 682 B | 372 B
labsim: PPPoE HA passes the matrix, and the health check had a real flap bug Full run green: baseline (one AC session, held by the VIP holder), clean failover (pppoe0 moves in 26s, old master releases), 10 gig down (route falls to pppoe0 and the LAN is back online in 5s), lease expiry (the guard hangs up), and a missing peers file (NRestarts=0, no loop). Evidence in labsim/wan-failover-evidence/. The 10 gig test found a genuine bug in vrrp-wan-health, not in the sim. GRACE was measured from PROMOTION, so an established master had no grace at all -- after hours of uptime `now - since` far exceeds any window. The first moment bond0.53 went down while pppoe0 was mid-redial, the master failed its own check, shed every VIP, and the peer inherited the same WAN outage and did the same. A brief WAN blip would have flapped the production pair. The stamp is now refreshed on every healthy tick, so grace measures time since the box last demonstrably HAD a WAN -- survivable wherever the gap happens, not only just after a promotion. Three harness bugs, all the same shape, all of which produced a confident wrong answer before being caught: - waiting for "exactly one pppoe0 holder" returns INSTANTLY during a handover, because it was already true. The useful question is who holds it. - judging connectivity on a single ping 20s after a link drop reported an outage that had already healed. Poll, do not sample. - `-o PreferredAuthentications=password` suits the routers but not the Alpine LAN VMs, whose sshd offers keyboard-interactive: ssh exited 255 before running anything and the test read that as "the LAN lost the internet". A tcpdump on the router showed the pings leaving pppoe0 NATed to 198.51.100.117 and the replies coming back the whole time. An exit code that can mean "the network is broken" or "I could not log in" is not a connectivity test, so the check now asserts on what the guest reported. That last one is why the harness asks the routers and the access concentrator rather than a client, and why it refuses to run at all while either router still has a default route via eth2 -- the libvirt-NAT scaffold answers connectivity checks the WAN under test would have failed. Still to run: hard failover (destroy the master), and the session-control replace/deny/disable axis that brackets Vodafone's unknown behaviour and sets the final GRACE. Nothing applied to production. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-09-05 19:11:07 +01:00
--- 172.31.1.252 ---
labsim: both matrices green end to end, numbers reproduced Final confirming run with GRACE=300 and every fix installed: --all ALL PASS (T0 T3 T5 T8 T11 T12) --hard ALL PASS (replace / deny / disable, each policy verified) Second independent measurement of the hard failover, which is what makes the figures trustworthy rather than anecdotal: run 1 run 2 replace 26s 26s deny 148s 141s disable 21s 20s `deny` sits at ~141-148s across both, so it is the AC's dead-peer behaviour and not a one-off; GRACE=300 keeps roughly 2x margin. Corrected an overclaim in PPPOE-HA.md while confirming it: the invariant row read "AC never showed two simdsl sessions", and under session-control=disable it did -- one live, one orphaned from the destroyed router. Only ever one LIVE router dialled, which is the invariant that matters. Said so plainly rather than leaving a table that reads better than the evidence.
2026-09-06 00:42:01 +01:00
vip=172.31.1.1 holds_vip=yes wan_disabled=no wan_up=yes ppp_up=yes ppp_active=yes may_dial=yes lease_age=7 dropin=yes role=master
pppoe0 UNKNOWN 198.51.100.132 peer 198.51.100.1/32
default nhid 58 dev pppoe0 proto static metric 20
Sep 05 23:13:04 apitest vrrp-wan[6413]: MASTER with bond0.53 disabled -> enabling
Sep 05 23:13:08 apitest vrrp-wan[7122]: bond0.53 enable commit took 4s
-- Boot 46727d256db344d6ad4b37c142272071 --
Sep 05 23:18:31 apitest vrrp-wan[6258]: MASTER: dialling pppoe0
Sep 05 23:18:31 apitest vrrp-wan[6411]: MASTER with bond0.53 disabled -> enabling
Sep 05 23:18:36 apitest vrrp-wan[7038]: bond0.53 enable commit took 5s
-- Boot 68bb1e8d56b34d55a511a301c24270fe --
Sep 05 23:27:40 apitest vrrp-wan[12502]: MASTER: dialling pppoe0
Sep 05 23:27:40 apitest vrrp-wan[12654]: MASTER with bond0.53 disabled -> enabling
Sep 05 23:27:45 apitest vrrp-wan[13361]: bond0.53 enable commit took 5s
labsim: PPPoE HA passes the matrix, and the health check had a real flap bug Full run green: baseline (one AC session, held by the VIP holder), clean failover (pppoe0 moves in 26s, old master releases), 10 gig down (route falls to pppoe0 and the LAN is back online in 5s), lease expiry (the guard hangs up), and a missing peers file (NRestarts=0, no loop). Evidence in labsim/wan-failover-evidence/. The 10 gig test found a genuine bug in vrrp-wan-health, not in the sim. GRACE was measured from PROMOTION, so an established master had no grace at all -- after hours of uptime `now - since` far exceeds any window. The first moment bond0.53 went down while pppoe0 was mid-redial, the master failed its own check, shed every VIP, and the peer inherited the same WAN outage and did the same. A brief WAN blip would have flapped the production pair. The stamp is now refreshed on every healthy tick, so grace measures time since the box last demonstrably HAD a WAN -- survivable wherever the gap happens, not only just after a promotion. Three harness bugs, all the same shape, all of which produced a confident wrong answer before being caught: - waiting for "exactly one pppoe0 holder" returns INSTANTLY during a handover, because it was already true. The useful question is who holds it. - judging connectivity on a single ping 20s after a link drop reported an outage that had already healed. Poll, do not sample. - `-o PreferredAuthentications=password` suits the routers but not the Alpine LAN VMs, whose sshd offers keyboard-interactive: ssh exited 255 before running anything and the test read that as "the LAN lost the internet". A tcpdump on the router showed the pings leaving pppoe0 NATed to 198.51.100.117 and the replies coming back the whole time. An exit code that can mean "the network is broken" or "I could not log in" is not a connectivity test, so the check now asserts on what the guest reported. That last one is why the harness asks the routers and the access concentrator rather than a client, and why it refuses to run at all while either router still has a default route via eth2 -- the libvirt-NAT scaffold answers connectivity checks the WAN under test would have failed. Still to run: hard failover (destroy the master), and the session-control replace/deny/disable axis that brackets Vodafone's unknown behaviour and sets the final GRACE. Nothing applied to production. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-09-05 19:11:07 +01:00
--- 172.31.1.253 ---
PPPOE-HA: record the two failure modes found by running the thing Both were invisible to inspection and only appeared under the hard matrix: the flap damper tearing down an established session via the lease it stopped renewing, and a missing peers file being completely silent. The second has a production edge worth spelling out -- an unsaved commit reverts on reboot and takes pppoe0 with it, leaving a standby that can never take over while looking perfectly healthy.
2026-09-06 00:11:32 +01:00
vip=172.31.1.1 holds_vip=no wan_disabled=yes wan_up=no ppp_up=no ppp_active=no may_dial=no lease_age=- dropin=yes role=backup
labsim: both matrices green end to end, numbers reproduced Final confirming run with GRACE=300 and every fix installed: --all ALL PASS (T0 T3 T5 T8 T11 T12) --hard ALL PASS (replace / deny / disable, each policy verified) Second independent measurement of the hard failover, which is what makes the figures trustworthy rather than anecdotal: run 1 run 2 replace 26s 26s deny 148s 141s disable 21s 20s `deny` sits at ~141-148s across both, so it is the AC's dead-peer behaviour and not a one-off; GRACE=300 keeps roughly 2x margin. Corrected an overclaim in PPPOE-HA.md while confirming it: the invariant row read "AC never showed two simdsl sessions", and under session-control=disable it did -- one live, one orphaned from the destroyed router. Only ever one LIVE router dialled, which is the invariant that matters. Said so plainly rather than leaving a table that reads better than the evidence.
2026-09-06 00:42:01 +01:00
Sep 05 23:16:11 vyos vrrp-wan[7190]: MASTER with bond0.53 disabled -> enabling
Sep 05 23:16:15 vyos vrrp-wan[7979]: bond0.53 enable commit took 4s
-- Boot c0e38585d7354ad6ab018800c7f3f6be --
Sep 05 23:22:54 vyos vrrp-wan[5970]: MASTER: dialling pppoe0
Sep 05 23:22:54 vyos vrrp-wan[6126]: MASTER with bond0.53 disabled -> enabling
Sep 05 23:22:59 vyos vrrp-wan[6915]: bond0.53 enable commit took 5s
Sep 05 23:27:40 vyos vrrp-wan[16037]: not MASTER: hanging up pppoe0
Sep 05 23:27:40 vyos vrrp-wan[16198]: not MASTER but bond0.53 enabled -> releasing
Sep 05 23:27:47 vyos vrrp-wan[16840]: bond0.53 disable commit took 7s
Reference in New Issue Copy Permalink