From 5ed0e4888a9c5e9e6077f87410848209709af029 Mon Sep 17 00:00:00 2001 From: Michal Date: Sun, 6 Sep 2026 00:42:01 +0100 Subject: [PATCH] labsim: both matrices green end to end, numbers reproduced Final confirming run with GRACE=300 and every fix installed: --all ALL PASS (T0 T3 T5 T8 T11 T12) --hard ALL PASS (replace / deny / disable, each policy verified) Second independent measurement of the hard failover, which is what makes the figures trustworthy rather than anecdotal: run 1 run 2 replace 26s 26s deny 148s 141s disable 21s 20s `deny` sits at ~141-148s across both, so it is the AC's dead-peer behaviour and not a one-off; GRACE=300 keeps roughly 2x margin. Corrected an overclaim in PPPOE-HA.md while confirming it: the invariant row read "AC never showed two simdsl sessions", and under session-control=disable it did -- one live, one orphaned from the destroyed router. Only ever one LIVE router dialled, which is the invariant that matters. Said so plainly rather than leaving a table that reads better than the evidence. --- labsim/wan-failover-evidence/README.md | 8 ++-- .../T11-no-peers-file/state.txt | 42 ++++++++--------- .../T12-holdoff-keeps-session/state.txt | 45 +++++++++--------- .../T3-clean-failover/state.txt | 44 +++++++++--------- .../T4-hard-failover-deny/state.txt | 27 ++++++----- .../T4-hard-failover-disable/state.txt | 31 ++++++------- .../T4-hard-failover-replace/state.txt | 27 ++++++----- .../T5-tengig-down/state.txt | 46 ++++++++++--------- .../T8-lease-expiry/state.txt | 45 +++++++++--------- migration/PPPOE-HA.md | 16 +++++-- migration/vrrp-wan.conf | 9 ++-- 11 files changed, 177 insertions(+), 163 deletions(-) diff --git a/labsim/wan-failover-evidence/README.md b/labsim/wan-failover-evidence/README.md index da6f20b..9017c39 100644 --- a/labsim/wan-failover-evidence/README.md +++ b/labsim/wan-failover-evidence/README.md @@ -10,11 +10,13 @@ router simply ceases — and times how long until the survivor holds a PPPoE session. Run across every policy VyOS can express, because Vodafone's is unknown: +Two independent runs, so these describe the AC's behaviour rather than one-offs: + | `session-control` | takeover | | |---|---|---| -| `replace` | 26s | accel-ppp default; the new auth kills the old session | -| **`deny`** | **148s** | the AC refuses the survivor until its own dead-peer timer frees the dead session | -| `disable` | 21s | no single-session enforcement at all | +| `replace` | 26s / 26s | accel-ppp default; the new auth kills the old session | +| **`deny`** | **148s / 141s** | the AC refuses the survivor until its own dead-peer timer frees the dead session | +| `disable` | 21s / 20s | no single-session enforcement at all | `deny` is the only one that matters for sizing, and `GRACE=300` in `migration/vrrp-wan.conf` comes from it. Session polling during that run caught diff --git a/labsim/wan-failover-evidence/T11-no-peers-file/state.txt b/labsim/wan-failover-evidence/T11-no-peers-file/state.txt index 4ab41ec..82ac262 100644 --- a/labsim/wan-failover-evidence/T11-no-peers-file/state.txt +++ b/labsim/wan-failover-evidence/T11-no-peers-file/state.txt @@ -1,26 +1,26 @@ -=== 2026-09-06T00:08:14+01:00 === +=== 2026-09-06T00:29:55+01:00 === --- AC sessions --- - ifname | username | ip | ip6 | ip6-dp | calling-sid | rate-limit | state | uptime | rx-bytes | tx-bytes ---------+----------+----+-----+--------+-------------------+------------+-------+----------+----------+---------- - | | | | | 52:54:00:e5:95:a2 | | start | 00:00:00 | 0 B | 0 B + ifname | username | ip | ip6 | ip6-dp | calling-sid | rate-limit | state | uptime | rx-bytes | tx-bytes +--------+----------+----+-----+--------+-------------+------------+-------+--------+----------+---------- --- 172.31.1.252 --- vip=172.31.1.1 holds_vip=yes wan_disabled=no wan_up=no ppp_up=no ppp_active=yes may_dial=yes lease_age=1 dropin=yes role=master -Sep 05 18:11:42 apitest vrrp-wan[3017050]: MASTER with bond0.53 disabled -> enabling -Sep 05 18:11:46 apitest vrrp-wan[3017685]: bond0.53 enable commit took 4s --- Boot 7cb878bdb4c74380ae7869ffd0dad9eb -- -Sep 05 23:06:18 apitest vrrp-wan[18533]: MASTER: dialling pppoe0 -Sep 05 23:06:18 apitest vrrp-wan[18704]: MASTER with bond0.53 disabled -> enabling -Sep 05 23:06:22 apitest vrrp-wan[19398]: bond0.53 enable commit took 4s -Sep 05 23:07:35 apitest vrrp-wan[22111]: GUARD: lease stale (206s > 75s; is vrrp-wan-reconcile.timer running?) -- hanging up pppoe0 -Sep 05 23:07:43 apitest vrrp-wan[22667]: MASTER: dialling pppoe0 -Sep 05 23:08:14 apitest vrrp-wan[24120]: MASTER: dialling pppoe0 +Sep 05 23:18:31 apitest vrrp-wan[6411]: MASTER with bond0.53 disabled -> enabling +Sep 05 23:18:36 apitest vrrp-wan[7038]: bond0.53 enable commit took 5s +-- Boot 68bb1e8d56b34d55a511a301c24270fe -- +Sep 05 23:27:40 apitest vrrp-wan[12502]: MASTER: dialling pppoe0 +Sep 05 23:27:40 apitest vrrp-wan[12654]: MASTER with bond0.53 disabled -> enabling +Sep 05 23:27:45 apitest vrrp-wan[13361]: bond0.53 enable commit took 5s +Sep 05 23:29:01 apitest vrrp-wan[16369]: GUARD: lease stale (200s > 75s; is vrrp-wan-reconcile.timer running?) -- hanging up pppoe0 +Sep 05 23:29:27 apitest vrrp-wan[17398]: MASTER: dialling pppoe0 +Sep 05 23:29:57 apitest vrrp-wan[18610]: MASTER: dialling pppoe0 --- 172.31.1.253 --- vip=172.31.1.1 holds_vip=no wan_disabled=yes wan_up=no ppp_up=no ppp_active=no may_dial=no lease_age=- dropin=yes role=backup -Sep 05 22:56:19 vyos vrrp-wan[196724]: DIAL FLAP: >=6 attempts in 600s -- holding off 900s -Sep 05 22:57:24 vyos vrrp-wan[199011]: MASTER with bond0.53 disabled -> enabling -Sep 05 22:57:28 vyos vrrp-wan[199711]: bond0.53 enable commit took 4s -Sep 05 22:57:40 vyos vrrp-wan[199948]: GUARD: lease stale (81s > 75s; is vrrp-wan-reconcile.timer running?) -- hanging up pppoe0 -Sep 05 23:02:41 vyos vrrp-wan[206596]: MASTER: dialling pppoe0 -Sep 05 23:06:17 vyos vrrp-wan[213698]: not MASTER: hanging up pppoe0 -Sep 05 23:06:18 vyos vrrp-wan[213859]: not MASTER but bond0.53 enabled -> releasing -Sep 05 23:06:23 vyos vrrp-wan[214503]: bond0.53 disable commit took 5s +Sep 05 23:16:11 vyos vrrp-wan[7190]: MASTER with bond0.53 disabled -> enabling +Sep 05 23:16:15 vyos vrrp-wan[7979]: bond0.53 enable commit took 4s +-- Boot c0e38585d7354ad6ab018800c7f3f6be -- +Sep 05 23:22:54 vyos vrrp-wan[5970]: MASTER: dialling pppoe0 +Sep 05 23:22:54 vyos vrrp-wan[6126]: MASTER with bond0.53 disabled -> enabling +Sep 05 23:22:59 vyos vrrp-wan[6915]: bond0.53 enable commit took 5s +Sep 05 23:27:40 vyos vrrp-wan[16037]: not MASTER: hanging up pppoe0 +Sep 05 23:27:40 vyos vrrp-wan[16198]: not MASTER but bond0.53 enabled -> releasing +Sep 05 23:27:47 vyos vrrp-wan[16840]: bond0.53 disable commit took 7s diff --git a/labsim/wan-failover-evidence/T12-holdoff-keeps-session/state.txt b/labsim/wan-failover-evidence/T12-holdoff-keeps-session/state.txt index 777ffdb..2cbe30e 100644 --- a/labsim/wan-failover-evidence/T12-holdoff-keeps-session/state.txt +++ b/labsim/wan-failover-evidence/T12-holdoff-keeps-session/state.txt @@ -1,28 +1,29 @@ -=== 2026-09-06T00:10:11+01:00 === +=== 2026-09-06T00:31:55+01:00 === --- AC sessions --- ifname | username | ip | ip6 | ip6-dp | calling-sid | rate-limit | state | uptime | rx-bytes | tx-bytes --------+----------+----------------+-----+--------+-------------------+------------+--------+----------+----------+---------- - ppp0 | simdsl | 198.51.100.126 | | | 52:54:00:e5:95:a2 | | active | 00:01:57 | 514 B | 204 B + ppp0 | simdsl | 198.51.100.134 | | | 52:54:00:e5:95:a2 | | active | 00:01:58 | 670 B | 555 B --- 172.31.1.252 --- -vip=172.31.1.1 holds_vip=yes wan_disabled=no wan_up=no ppp_up=yes ppp_active=yes may_dial=yes lease_age=23 dropin=yes role=master -pppoe0 UNKNOWN 198.51.100.126 peer 198.51.100.1/32 -default nhid 78 dev pppoe0 proto static metric 20 -Sep 05 18:11:42 apitest vrrp-wan[3017050]: MASTER with bond0.53 disabled -> enabling -Sep 05 18:11:46 apitest vrrp-wan[3017685]: bond0.53 enable commit took 4s --- Boot 7cb878bdb4c74380ae7869ffd0dad9eb -- -Sep 05 23:06:18 apitest vrrp-wan[18533]: MASTER: dialling pppoe0 -Sep 05 23:06:18 apitest vrrp-wan[18704]: MASTER with bond0.53 disabled -> enabling -Sep 05 23:06:22 apitest vrrp-wan[19398]: bond0.53 enable commit took 4s -Sep 05 23:07:35 apitest vrrp-wan[22111]: GUARD: lease stale (206s > 75s; is vrrp-wan-reconcile.timer running?) -- hanging up pppoe0 -Sep 05 23:07:43 apitest vrrp-wan[22667]: MASTER: dialling pppoe0 -Sep 05 23:08:14 apitest vrrp-wan[24120]: MASTER: dialling pppoe0 +vip=172.31.1.1 holds_vip=yes wan_disabled=no wan_up=no ppp_up=yes ppp_active=yes may_dial=yes lease_age=21 dropin=yes role=master +pppoe0 UNKNOWN 198.51.100.134 peer 198.51.100.1/32 +default nhid 74 dev pppoe0 proto static metric 20 +Sep 05 23:18:31 apitest vrrp-wan[6411]: MASTER with bond0.53 disabled -> enabling +Sep 05 23:18:36 apitest vrrp-wan[7038]: bond0.53 enable commit took 5s +-- Boot 68bb1e8d56b34d55a511a301c24270fe -- +Sep 05 23:27:40 apitest vrrp-wan[12502]: MASTER: dialling pppoe0 +Sep 05 23:27:40 apitest vrrp-wan[12654]: MASTER with bond0.53 disabled -> enabling +Sep 05 23:27:45 apitest vrrp-wan[13361]: bond0.53 enable commit took 5s +Sep 05 23:29:01 apitest vrrp-wan[16369]: GUARD: lease stale (200s > 75s; is vrrp-wan-reconcile.timer running?) -- hanging up pppoe0 +Sep 05 23:29:27 apitest vrrp-wan[17398]: MASTER: dialling pppoe0 +Sep 05 23:29:57 apitest vrrp-wan[18610]: MASTER: dialling pppoe0 --- 172.31.1.253 --- vip=172.31.1.1 holds_vip=no wan_disabled=yes wan_up=no ppp_up=no ppp_active=no may_dial=no lease_age=- dropin=yes role=backup -Sep 05 22:56:19 vyos vrrp-wan[196724]: DIAL FLAP: >=6 attempts in 600s -- holding off 900s -Sep 05 22:57:24 vyos vrrp-wan[199011]: MASTER with bond0.53 disabled -> enabling -Sep 05 22:57:28 vyos vrrp-wan[199711]: bond0.53 enable commit took 4s -Sep 05 22:57:40 vyos vrrp-wan[199948]: GUARD: lease stale (81s > 75s; is vrrp-wan-reconcile.timer running?) -- hanging up pppoe0 -Sep 05 23:02:41 vyos vrrp-wan[206596]: MASTER: dialling pppoe0 -Sep 05 23:06:17 vyos vrrp-wan[213698]: not MASTER: hanging up pppoe0 -Sep 05 23:06:18 vyos vrrp-wan[213859]: not MASTER but bond0.53 enabled -> releasing -Sep 05 23:06:23 vyos vrrp-wan[214503]: bond0.53 disable commit took 5s +Sep 05 23:16:11 vyos vrrp-wan[7190]: MASTER with bond0.53 disabled -> enabling +Sep 05 23:16:15 vyos vrrp-wan[7979]: bond0.53 enable commit took 4s +-- Boot c0e38585d7354ad6ab018800c7f3f6be -- +Sep 05 23:22:54 vyos vrrp-wan[5970]: MASTER: dialling pppoe0 +Sep 05 23:22:54 vyos vrrp-wan[6126]: MASTER with bond0.53 disabled -> enabling +Sep 05 23:22:59 vyos vrrp-wan[6915]: bond0.53 enable commit took 5s +Sep 05 23:27:40 vyos vrrp-wan[16037]: not MASTER: hanging up pppoe0 +Sep 05 23:27:40 vyos vrrp-wan[16198]: not MASTER but bond0.53 enabled -> releasing +Sep 05 23:27:47 vyos vrrp-wan[16840]: bond0.53 disable commit took 7s diff --git a/labsim/wan-failover-evidence/T3-clean-failover/state.txt b/labsim/wan-failover-evidence/T3-clean-failover/state.txt index 60988a4..243b36d 100644 --- a/labsim/wan-failover-evidence/T3-clean-failover/state.txt +++ b/labsim/wan-failover-evidence/T3-clean-failover/state.txt @@ -1,28 +1,30 @@ -=== 2026-09-06T00:06:31+01:00 === +=== 2026-09-06T00:27:58+01:00 === --- AC sessions --- ifname | username | ip | ip6 | ip6-dp | calling-sid | rate-limit | state | uptime | rx-bytes | tx-bytes --------+----------+----------------+-----+--------+-------------------+------------+--------+----------+----------+---------- - ppp0 | simdsl | 198.51.100.124 | | | 52:54:00:e5:95:a2 | | active | 00:00:13 | 514 B | 204 B + ppp0 | simdsl | 198.51.100.132 | | | 52:54:00:e5:95:a2 | | active | 00:00:18 | 514 B | 204 B --- 172.31.1.252 --- -vip=172.31.1.1 holds_vip=yes wan_disabled=no wan_up=yes ppp_up=yes ppp_active=yes may_dial=yes lease_age=3 dropin=yes role=master -pppoe0 UNKNOWN 198.51.100.124 peer 198.51.100.1/32 +vip=172.31.1.1 holds_vip=yes wan_disabled=no wan_up=yes ppp_up=yes ppp_active=yes may_dial=yes lease_age=7 dropin=yes role=master +pppoe0 UNKNOWN 198.51.100.132 peer 198.51.100.1/32 default via 203.0.113.1 dev bond0.53 proto failover metric 1 -Sep 05 18:08:16 apitest vrrp-wan[3010622]: not MASTER but bond0.53 enabled -> releasing -Sep 05 18:08:40 apitest vrrp-wan[3012085]: bond0.53 disable commit took 24s -Sep 05 18:11:42 apitest vrrp-wan[3016900]: MASTER: dialling pppoe0 -Sep 05 18:11:42 apitest vrrp-wan[3017050]: MASTER with bond0.53 disabled -> enabling -Sep 05 18:11:46 apitest vrrp-wan[3017685]: bond0.53 enable commit took 4s --- Boot 7cb878bdb4c74380ae7869ffd0dad9eb -- -Sep 05 23:06:18 apitest vrrp-wan[18533]: MASTER: dialling pppoe0 -Sep 05 23:06:18 apitest vrrp-wan[18704]: MASTER with bond0.53 disabled -> enabling -Sep 05 23:06:22 apitest vrrp-wan[19398]: bond0.53 enable commit took 4s +Sep 05 23:13:04 apitest vrrp-wan[6413]: MASTER with bond0.53 disabled -> enabling +Sep 05 23:13:08 apitest vrrp-wan[7122]: bond0.53 enable commit took 4s +-- Boot 46727d256db344d6ad4b37c142272071 -- +Sep 05 23:18:31 apitest vrrp-wan[6258]: MASTER: dialling pppoe0 +Sep 05 23:18:31 apitest vrrp-wan[6411]: MASTER with bond0.53 disabled -> enabling +Sep 05 23:18:36 apitest vrrp-wan[7038]: bond0.53 enable commit took 5s +-- Boot 68bb1e8d56b34d55a511a301c24270fe -- +Sep 05 23:27:40 apitest vrrp-wan[12502]: MASTER: dialling pppoe0 +Sep 05 23:27:40 apitest vrrp-wan[12654]: MASTER with bond0.53 disabled -> enabling +Sep 05 23:27:45 apitest vrrp-wan[13361]: bond0.53 enable commit took 5s --- 172.31.1.253 --- vip=172.31.1.1 holds_vip=no wan_disabled=yes wan_up=no ppp_up=no ppp_active=no may_dial=no lease_age=- dropin=yes role=backup -Sep 05 22:56:19 vyos vrrp-wan[196724]: DIAL FLAP: >=6 attempts in 600s -- holding off 900s -Sep 05 22:57:24 vyos vrrp-wan[199011]: MASTER with bond0.53 disabled -> enabling -Sep 05 22:57:28 vyos vrrp-wan[199711]: bond0.53 enable commit took 4s -Sep 05 22:57:40 vyos vrrp-wan[199948]: GUARD: lease stale (81s > 75s; is vrrp-wan-reconcile.timer running?) -- hanging up pppoe0 -Sep 05 23:02:41 vyos vrrp-wan[206596]: MASTER: dialling pppoe0 -Sep 05 23:06:17 vyos vrrp-wan[213698]: not MASTER: hanging up pppoe0 -Sep 05 23:06:18 vyos vrrp-wan[213859]: not MASTER but bond0.53 enabled -> releasing -Sep 05 23:06:23 vyos vrrp-wan[214503]: bond0.53 disable commit took 5s +Sep 05 23:16:11 vyos vrrp-wan[7190]: MASTER with bond0.53 disabled -> enabling +Sep 05 23:16:15 vyos vrrp-wan[7979]: bond0.53 enable commit took 4s +-- Boot c0e38585d7354ad6ab018800c7f3f6be -- +Sep 05 23:22:54 vyos vrrp-wan[5970]: MASTER: dialling pppoe0 +Sep 05 23:22:54 vyos vrrp-wan[6126]: MASTER with bond0.53 disabled -> enabling +Sep 05 23:22:59 vyos vrrp-wan[6915]: bond0.53 enable commit took 5s +Sep 05 23:27:40 vyos vrrp-wan[16037]: not MASTER: hanging up pppoe0 +Sep 05 23:27:40 vyos vrrp-wan[16198]: not MASTER but bond0.53 enabled -> releasing +Sep 05 23:27:47 vyos vrrp-wan[16840]: bond0.53 disable commit took 7s diff --git a/labsim/wan-failover-evidence/T4-hard-failover-deny/state.txt b/labsim/wan-failover-evidence/T4-hard-failover-deny/state.txt index 8a40ab1..d5a9f6a 100644 --- a/labsim/wan-failover-evidence/T4-hard-failover-deny/state.txt +++ b/labsim/wan-failover-evidence/T4-hard-failover-deny/state.txt @@ -1,20 +1,19 @@ -=== 2026-09-06T00:21:00+01:00 === +=== 2026-09-06T00:37:11+01:00 === --- AC sessions --- ifname | username | ip | ip6 | ip6-dp | calling-sid | rate-limit | state | uptime | rx-bytes | tx-bytes --------+----------+----------------+-----+--------+-------------------+------------+--------+----------+----------+---------- - ppp0 | simdsl | 198.51.100.130 | | | 52:54:00:e5:95:a2 | | active | 00:00:18 | 438 B | 204 B + ppp0 | simdsl | 198.51.100.136 | | | 52:54:00:e5:95:a2 | | active | 00:00:13 | 438 B | 204 B --- 172.31.1.252 --- -vip=172.31.1.1 holds_vip=yes wan_disabled=no wan_up=yes ppp_up=yes ppp_active=yes may_dial=yes lease_age=30 dropin=yes role=master -pppoe0 UNKNOWN 198.51.100.130 peer 198.51.100.1/32 +vip=172.31.1.1 holds_vip=yes wan_disabled=no wan_up=yes ppp_up=yes ppp_active=yes may_dial=yes lease_age=26 dropin=yes role=master +pppoe0 UNKNOWN 198.51.100.136 peer 198.51.100.1/32 default via 203.0.113.1 dev bond0.53 proto failover metric 1 -Sep 05 23:07:43 apitest vrrp-wan[22667]: MASTER: dialling pppoe0 -Sep 05 23:08:14 apitest vrrp-wan[24120]: MASTER: dialling pppoe0 --- Boot c5f23399239b468c8c8b752a4305c515 -- -Sep 05 23:13:04 apitest vrrp-wan[6261]: MASTER: dialling pppoe0 -Sep 05 23:13:04 apitest vrrp-wan[6413]: MASTER with bond0.53 disabled -> enabling -Sep 05 23:13:08 apitest vrrp-wan[7122]: bond0.53 enable commit took 4s --- Boot 46727d256db344d6ad4b37c142272071 -- -Sep 05 23:18:31 apitest vrrp-wan[6258]: MASTER: dialling pppoe0 -Sep 05 23:18:31 apitest vrrp-wan[6411]: MASTER with bond0.53 disabled -> enabling -Sep 05 23:18:36 apitest vrrp-wan[7038]: bond0.53 enable commit took 5s +Sep 05 23:27:40 apitest vrrp-wan[12654]: MASTER with bond0.53 disabled -> enabling +Sep 05 23:27:45 apitest vrrp-wan[13361]: bond0.53 enable commit took 5s +Sep 05 23:29:01 apitest vrrp-wan[16369]: GUARD: lease stale (200s > 75s; is vrrp-wan-reconcile.timer running?) -- hanging up pppoe0 +Sep 05 23:29:27 apitest vrrp-wan[17398]: MASTER: dialling pppoe0 +Sep 05 23:29:57 apitest vrrp-wan[18610]: MASTER: dialling pppoe0 +-- Boot 6efc3c9ba47f455e9668454ee6c2fc37 -- +Sep 05 23:34:48 apitest vrrp-wan[6266]: MASTER: dialling pppoe0 +Sep 05 23:34:49 apitest vrrp-wan[6422]: MASTER with bond0.53 disabled -> enabling +Sep 05 23:34:54 apitest vrrp-wan[7130]: bond0.53 enable commit took 5s --- 172.31.1.253 --- diff --git a/labsim/wan-failover-evidence/T4-hard-failover-disable/state.txt b/labsim/wan-failover-evidence/T4-hard-failover-disable/state.txt index 9e6fb79..ad0d47f 100644 --- a/labsim/wan-failover-evidence/T4-hard-failover-disable/state.txt +++ b/labsim/wan-failover-evidence/T4-hard-failover-disable/state.txt @@ -1,21 +1,20 @@ -=== 2026-09-06T00:23:19+01:00 === +=== 2026-09-06T00:39:29+01:00 === --- AC sessions --- ifname | username | ip | ip6 | ip6-dp | calling-sid | rate-limit | state | uptime | rx-bytes | tx-bytes --------+----------+----------------+-----+--------+-------------------+------------+--------+----------+----------+---------- - ppp0 | simdsl | 198.51.100.130 | | | 52:54:00:e5:95:a2 | | active | 00:02:37 | 438 B | 204 B - ppp1 | simdsl | 198.51.100.131 | | | 52:54:00:4e:0b:56 | | active | 00:00:27 | 1.1 KiB | 204 B + ppp0 | simdsl | 198.51.100.136 | | | 52:54:00:e5:95:a2 | | active | 00:02:31 | 438 B | 204 B + ppp1 | simdsl | 198.51.100.137 | | | 52:54:00:4e:0b:56 | | active | 00:00:26 | 1.0 KiB | 204 B --- 172.31.1.252 --- --- 172.31.1.253 --- -vip=172.31.1.1 holds_vip=yes wan_disabled=no wan_up=yes ppp_up=yes ppp_active=yes may_dial=yes lease_age=15 dropin=yes role=master -pppoe0 UNKNOWN 198.51.100.131 peer 198.51.100.1/32 -default nhid 57 dev pppoe0 proto static metric 20 -Sep 05 23:10:44 vyos vrrp-wan[221212]: MASTER with bond0.53 disabled -> enabling -Sep 05 23:10:48 vyos vrrp-wan[221995]: bond0.53 enable commit took 4s --- Boot 1f635fc6900c4661b3ac6d016d63d293 -- -Sep 05 23:16:10 vyos vrrp-wan[7037]: MASTER: dialling pppoe0 -Sep 05 23:16:11 vyos vrrp-wan[7190]: MASTER with bond0.53 disabled -> enabling -Sep 05 23:16:15 vyos vrrp-wan[7979]: bond0.53 enable commit took 4s --- Boot c0e38585d7354ad6ab018800c7f3f6be -- -Sep 05 23:22:54 vyos vrrp-wan[5970]: MASTER: dialling pppoe0 -Sep 05 23:22:54 vyos vrrp-wan[6126]: MASTER with bond0.53 disabled -> enabling -Sep 05 23:22:59 vyos vrrp-wan[6915]: bond0.53 enable commit took 5s +vip=172.31.1.1 holds_vip=yes wan_disabled=no wan_up=yes ppp_up=yes ppp_active=yes may_dial=yes lease_age=5 dropin=yes role=master +pppoe0 UNKNOWN 198.51.100.137 peer 198.51.100.1/32 +default nhid 60 dev pppoe0 proto static metric 20 +Sep 05 23:27:40 vyos vrrp-wan[16198]: not MASTER but bond0.53 enabled -> releasing +Sep 05 23:27:47 vyos vrrp-wan[16840]: bond0.53 disable commit took 7s +Sep 05 23:32:26 vyos vrrp-wan[23806]: MASTER: dialling pppoe0 +Sep 05 23:32:26 vyos vrrp-wan[23956]: MASTER with bond0.53 disabled -> enabling +Sep 05 23:32:31 vyos vrrp-wan[24749]: bond0.53 enable commit took 5s +-- Boot 4b0355878444477fb92738339f85843d -- +Sep 05 23:39:03 vyos vrrp-wan[5966]: MASTER: dialling pppoe0 +Sep 05 23:39:04 vyos vrrp-wan[6120]: MASTER with bond0.53 disabled -> enabling +Sep 05 23:39:09 vyos vrrp-wan[6914]: bond0.53 enable commit took 5s diff --git a/labsim/wan-failover-evidence/T4-hard-failover-replace/state.txt b/labsim/wan-failover-evidence/T4-hard-failover-replace/state.txt index a675a26..4bae3be 100644 --- a/labsim/wan-failover-evidence/T4-hard-failover-replace/state.txt +++ b/labsim/wan-failover-evidence/T4-hard-failover-replace/state.txt @@ -1,19 +1,18 @@ -=== 2026-09-06T00:16:41+01:00 === +=== 2026-09-06T00:32:54+01:00 === --- AC sessions --- ifname | username | ip | ip6 | ip6-dp | calling-sid | rate-limit | state | uptime | rx-bytes | tx-bytes --------+----------+----------------+-----+--------+-------------------+------------+--------+----------+----------+---------- - ppp0 | simdsl | 198.51.100.129 | | | 52:54:00:4e:0b:56 | | active | 00:00:31 | 438 B | 204 B + ppp0 | simdsl | 198.51.100.135 | | | 52:54:00:4e:0b:56 | | active | 00:00:30 | 438 B | 204 B --- 172.31.1.252 --- --- 172.31.1.253 --- -vip=172.31.1.1 holds_vip=yes wan_disabled=no wan_up=yes ppp_up=yes ppp_active=yes may_dial=yes lease_age=22 dropin=yes role=master -pppoe0 UNKNOWN 198.51.100.129 peer 198.51.100.1/32 -default nhid 62 dev pppoe0 proto static metric 20 -Sep 05 23:06:18 vyos vrrp-wan[213859]: not MASTER but bond0.53 enabled -> releasing -Sep 05 23:06:23 vyos vrrp-wan[214503]: bond0.53 disable commit took 5s -Sep 05 23:10:43 vyos vrrp-wan[221064]: MASTER: dialling pppoe0 -Sep 05 23:10:44 vyos vrrp-wan[221212]: MASTER with bond0.53 disabled -> enabling -Sep 05 23:10:48 vyos vrrp-wan[221995]: bond0.53 enable commit took 4s --- Boot 1f635fc6900c4661b3ac6d016d63d293 -- -Sep 05 23:16:10 vyos vrrp-wan[7037]: MASTER: dialling pppoe0 -Sep 05 23:16:11 vyos vrrp-wan[7190]: MASTER with bond0.53 disabled -> enabling -Sep 05 23:16:15 vyos vrrp-wan[7979]: bond0.53 enable commit took 4s +vip=172.31.1.1 holds_vip=yes wan_disabled=no wan_up=yes ppp_up=yes ppp_active=yes may_dial=yes lease_age=19 dropin=yes role=master +pppoe0 UNKNOWN 198.51.100.135 peer 198.51.100.1/32 +default nhid 77 dev pppoe0 proto static metric 20 +Sep 05 23:22:54 vyos vrrp-wan[6126]: MASTER with bond0.53 disabled -> enabling +Sep 05 23:22:59 vyos vrrp-wan[6915]: bond0.53 enable commit took 5s +Sep 05 23:27:40 vyos vrrp-wan[16037]: not MASTER: hanging up pppoe0 +Sep 05 23:27:40 vyos vrrp-wan[16198]: not MASTER but bond0.53 enabled -> releasing +Sep 05 23:27:47 vyos vrrp-wan[16840]: bond0.53 disable commit took 7s +Sep 05 23:32:26 vyos vrrp-wan[23806]: MASTER: dialling pppoe0 +Sep 05 23:32:26 vyos vrrp-wan[23956]: MASTER with bond0.53 disabled -> enabling +Sep 05 23:32:31 vyos vrrp-wan[24749]: bond0.53 enable commit took 5s diff --git a/labsim/wan-failover-evidence/T5-tengig-down/state.txt b/labsim/wan-failover-evidence/T5-tengig-down/state.txt index 2dd227d..b06f782 100644 --- a/labsim/wan-failover-evidence/T5-tengig-down/state.txt +++ b/labsim/wan-failover-evidence/T5-tengig-down/state.txt @@ -1,28 +1,30 @@ -=== 2026-09-06T00:07:02+01:00 === +=== 2026-09-06T00:28:31+01:00 === --- AC sessions --- ifname | username | ip | ip6 | ip6-dp | calling-sid | rate-limit | state | uptime | rx-bytes | tx-bytes --------+----------+----------------+-----+--------+-------------------+------------+--------+----------+----------+---------- - ppp0 | simdsl | 198.51.100.124 | | | 52:54:00:e5:95:a2 | | active | 00:00:44 | 682 B | 372 B + ppp0 | simdsl | 198.51.100.132 | | | 52:54:00:e5:95:a2 | | active | 00:00:52 | 682 B | 372 B --- 172.31.1.252 --- -vip=172.31.1.1 holds_vip=yes wan_disabled=no wan_up=yes ppp_up=yes ppp_active=yes may_dial=yes lease_age=3 dropin=yes role=master -pppoe0 UNKNOWN 198.51.100.124 peer 198.51.100.1/32 -default nhid 67 dev pppoe0 proto static metric 20 -Sep 05 18:08:16 apitest vrrp-wan[3010622]: not MASTER but bond0.53 enabled -> releasing -Sep 05 18:08:40 apitest vrrp-wan[3012085]: bond0.53 disable commit took 24s -Sep 05 18:11:42 apitest vrrp-wan[3016900]: MASTER: dialling pppoe0 -Sep 05 18:11:42 apitest vrrp-wan[3017050]: MASTER with bond0.53 disabled -> enabling -Sep 05 18:11:46 apitest vrrp-wan[3017685]: bond0.53 enable commit took 4s --- Boot 7cb878bdb4c74380ae7869ffd0dad9eb -- -Sep 05 23:06:18 apitest vrrp-wan[18533]: MASTER: dialling pppoe0 -Sep 05 23:06:18 apitest vrrp-wan[18704]: MASTER with bond0.53 disabled -> enabling -Sep 05 23:06:22 apitest vrrp-wan[19398]: bond0.53 enable commit took 4s +vip=172.31.1.1 holds_vip=yes wan_disabled=no wan_up=yes ppp_up=yes ppp_active=yes may_dial=yes lease_age=7 dropin=yes role=master +pppoe0 UNKNOWN 198.51.100.132 peer 198.51.100.1/32 +default nhid 58 dev pppoe0 proto static metric 20 +Sep 05 23:13:04 apitest vrrp-wan[6413]: MASTER with bond0.53 disabled -> enabling +Sep 05 23:13:08 apitest vrrp-wan[7122]: bond0.53 enable commit took 4s +-- Boot 46727d256db344d6ad4b37c142272071 -- +Sep 05 23:18:31 apitest vrrp-wan[6258]: MASTER: dialling pppoe0 +Sep 05 23:18:31 apitest vrrp-wan[6411]: MASTER with bond0.53 disabled -> enabling +Sep 05 23:18:36 apitest vrrp-wan[7038]: bond0.53 enable commit took 5s +-- Boot 68bb1e8d56b34d55a511a301c24270fe -- +Sep 05 23:27:40 apitest vrrp-wan[12502]: MASTER: dialling pppoe0 +Sep 05 23:27:40 apitest vrrp-wan[12654]: MASTER with bond0.53 disabled -> enabling +Sep 05 23:27:45 apitest vrrp-wan[13361]: bond0.53 enable commit took 5s --- 172.31.1.253 --- vip=172.31.1.1 holds_vip=no wan_disabled=yes wan_up=no ppp_up=no ppp_active=no may_dial=no lease_age=- dropin=yes role=backup -Sep 05 22:56:19 vyos vrrp-wan[196724]: DIAL FLAP: >=6 attempts in 600s -- holding off 900s -Sep 05 22:57:24 vyos vrrp-wan[199011]: MASTER with bond0.53 disabled -> enabling -Sep 05 22:57:28 vyos vrrp-wan[199711]: bond0.53 enable commit took 4s -Sep 05 22:57:40 vyos vrrp-wan[199948]: GUARD: lease stale (81s > 75s; is vrrp-wan-reconcile.timer running?) -- hanging up pppoe0 -Sep 05 23:02:41 vyos vrrp-wan[206596]: MASTER: dialling pppoe0 -Sep 05 23:06:17 vyos vrrp-wan[213698]: not MASTER: hanging up pppoe0 -Sep 05 23:06:18 vyos vrrp-wan[213859]: not MASTER but bond0.53 enabled -> releasing -Sep 05 23:06:23 vyos vrrp-wan[214503]: bond0.53 disable commit took 5s +Sep 05 23:16:11 vyos vrrp-wan[7190]: MASTER with bond0.53 disabled -> enabling +Sep 05 23:16:15 vyos vrrp-wan[7979]: bond0.53 enable commit took 4s +-- Boot c0e38585d7354ad6ab018800c7f3f6be -- +Sep 05 23:22:54 vyos vrrp-wan[5970]: MASTER: dialling pppoe0 +Sep 05 23:22:54 vyos vrrp-wan[6126]: MASTER with bond0.53 disabled -> enabling +Sep 05 23:22:59 vyos vrrp-wan[6915]: bond0.53 enable commit took 5s +Sep 05 23:27:40 vyos vrrp-wan[16037]: not MASTER: hanging up pppoe0 +Sep 05 23:27:40 vyos vrrp-wan[16198]: not MASTER but bond0.53 enabled -> releasing +Sep 05 23:27:47 vyos vrrp-wan[16840]: bond0.53 disable commit took 7s diff --git a/labsim/wan-failover-evidence/T8-lease-expiry/state.txt b/labsim/wan-failover-evidence/T8-lease-expiry/state.txt index 22ea9b0..2acd969 100644 --- a/labsim/wan-failover-evidence/T8-lease-expiry/state.txt +++ b/labsim/wan-failover-evidence/T8-lease-expiry/state.txt @@ -1,26 +1,27 @@ -=== 2026-09-06T00:07:44+01:00 === +=== 2026-09-06T00:29:16+01:00 === --- AC sessions --- - ifname | username | ip | ip6 | ip6-dp | calling-sid | rate-limit | state | uptime | rx-bytes | tx-bytes ---------+----------+----+-----+--------+-------------------+------------+-------+----------+----------+---------- - | | | | | 52:54:00:e5:95:a2 | | start | 00:00:00 | 0 B | 0 B + ifname | username | ip | ip6 | ip6-dp | calling-sid | rate-limit | state | uptime | rx-bytes | tx-bytes +--------+----------+----+-----+--------+-------------+------------+-------+--------+----------+---------- --- 172.31.1.252 --- -vip=172.31.1.1 holds_vip=yes wan_disabled=no wan_up=no ppp_up=no ppp_active=yes may_dial=yes lease_age=2 dropin=yes role=master -Sep 05 18:11:42 apitest vrrp-wan[3016900]: MASTER: dialling pppoe0 -Sep 05 18:11:42 apitest vrrp-wan[3017050]: MASTER with bond0.53 disabled -> enabling -Sep 05 18:11:46 apitest vrrp-wan[3017685]: bond0.53 enable commit took 4s --- Boot 7cb878bdb4c74380ae7869ffd0dad9eb -- -Sep 05 23:06:18 apitest vrrp-wan[18533]: MASTER: dialling pppoe0 -Sep 05 23:06:18 apitest vrrp-wan[18704]: MASTER with bond0.53 disabled -> enabling -Sep 05 23:06:22 apitest vrrp-wan[19398]: bond0.53 enable commit took 4s -Sep 05 23:07:35 apitest vrrp-wan[22111]: GUARD: lease stale (206s > 75s; is vrrp-wan-reconcile.timer running?) -- hanging up pppoe0 -Sep 05 23:07:43 apitest vrrp-wan[22667]: MASTER: dialling pppoe0 +vip=172.31.1.1 holds_vip=yes wan_disabled=no wan_up=no ppp_up=no ppp_active=no may_dial=no lease_age=- dropin=yes role=master +Sep 05 23:13:08 apitest vrrp-wan[7122]: bond0.53 enable commit took 4s +-- Boot 46727d256db344d6ad4b37c142272071 -- +Sep 05 23:18:31 apitest vrrp-wan[6258]: MASTER: dialling pppoe0 +Sep 05 23:18:31 apitest vrrp-wan[6411]: MASTER with bond0.53 disabled -> enabling +Sep 05 23:18:36 apitest vrrp-wan[7038]: bond0.53 enable commit took 5s +-- Boot 68bb1e8d56b34d55a511a301c24270fe -- +Sep 05 23:27:40 apitest vrrp-wan[12502]: MASTER: dialling pppoe0 +Sep 05 23:27:40 apitest vrrp-wan[12654]: MASTER with bond0.53 disabled -> enabling +Sep 05 23:27:45 apitest vrrp-wan[13361]: bond0.53 enable commit took 5s +Sep 05 23:29:01 apitest vrrp-wan[16369]: GUARD: lease stale (200s > 75s; is vrrp-wan-reconcile.timer running?) -- hanging up pppoe0 --- 172.31.1.253 --- vip=172.31.1.1 holds_vip=no wan_disabled=yes wan_up=no ppp_up=no ppp_active=no may_dial=no lease_age=- dropin=yes role=backup -Sep 05 22:56:19 vyos vrrp-wan[196724]: DIAL FLAP: >=6 attempts in 600s -- holding off 900s -Sep 05 22:57:24 vyos vrrp-wan[199011]: MASTER with bond0.53 disabled -> enabling -Sep 05 22:57:28 vyos vrrp-wan[199711]: bond0.53 enable commit took 4s -Sep 05 22:57:40 vyos vrrp-wan[199948]: GUARD: lease stale (81s > 75s; is vrrp-wan-reconcile.timer running?) -- hanging up pppoe0 -Sep 05 23:02:41 vyos vrrp-wan[206596]: MASTER: dialling pppoe0 -Sep 05 23:06:17 vyos vrrp-wan[213698]: not MASTER: hanging up pppoe0 -Sep 05 23:06:18 vyos vrrp-wan[213859]: not MASTER but bond0.53 enabled -> releasing -Sep 05 23:06:23 vyos vrrp-wan[214503]: bond0.53 disable commit took 5s +Sep 05 23:16:11 vyos vrrp-wan[7190]: MASTER with bond0.53 disabled -> enabling +Sep 05 23:16:15 vyos vrrp-wan[7979]: bond0.53 enable commit took 4s +-- Boot c0e38585d7354ad6ab018800c7f3f6be -- +Sep 05 23:22:54 vyos vrrp-wan[5970]: MASTER: dialling pppoe0 +Sep 05 23:22:54 vyos vrrp-wan[6126]: MASTER with bond0.53 disabled -> enabling +Sep 05 23:22:59 vyos vrrp-wan[6915]: bond0.53 enable commit took 5s +Sep 05 23:27:40 vyos vrrp-wan[16037]: not MASTER: hanging up pppoe0 +Sep 05 23:27:40 vyos vrrp-wan[16198]: not MASTER but bond0.53 enabled -> releasing +Sep 05 23:27:47 vyos vrrp-wan[16840]: bond0.53 disable commit took 7s diff --git a/migration/PPPOE-HA.md b/migration/PPPOE-HA.md index 05f755a..993b7e8 100644 --- a/migration/PPPOE-HA.md +++ b/migration/PPPOE-HA.md @@ -67,7 +67,15 @@ either "I do not hold the VIP" or "the lease is stale". | Stale lease | guard hangs up within ~5s | | Missing peers file | `NRestarts=0` — no loop | | Flap holdoff vs live session | session survives a forged 900s holdoff | -| Invariant | AC never showed two `simdsl` sessions | +| Invariant | never more than one router dialled, in any run | + +On the last row, precisely: the AC *did* report two `simdsl` sessions during the +`session-control=disable` run — one live, one orphaned from the router that had +just been destroyed, which that policy does not clean up. Only one live router +was ever dialled. The count is a proxy for the real invariant and only a valid +one while the AC enforces single-session, so it is reported as a `WARN` rather +than silenced: an orphaned session still occupies the single slot at a real ISP, +and that is exactly what made `deny` take 141–148s. ## Two failure modes found by running it, not by reading it @@ -145,9 +153,9 @@ a separate change."* This is that change. | policy | takeover | |---|---| - | `replace` (accel-ppp default) | 26s | - | **`deny`** (hostile) | **148s** | - | `disable` | 21s | + | `replace` (accel-ppp default) | 26s / 26s | + | **`deny`** (hostile) | **148s / 141s** | + | `disable` | 21s / 20s | `deny` is the sizing case: the AC refuses the survivor until its own dead-peer timer frees the dead session, and the poller caught two dial diff --git a/migration/vrrp-wan.conf b/migration/vrrp-wan.conf index fc35b58..deab087 100644 --- a/migration/vrrp-wan.conf +++ b/migration/vrrp-wan.conf @@ -23,11 +23,12 @@ WAN_VIF=53 # exists to prevent. # # MEASURED, labsim T4, master destroyed with `virsh destroy`, time until the -# survivor held a PPPoE session (labsim/wan-failover-evidence/T4-*): +# survivor held a PPPoE session (labsim/wan-failover-evidence/T4-*). Two +# independent runs, so these are the AC's behaviour rather than one-offs: # -# session-control=replace 26s -# session-control=deny 148s <-- worst -# session-control=disable 21s +# session-control=replace 26s / 26s +# session-control=deny 148s / 141s <-- worst +# session-control=disable 21s / 20s # # `deny` is the hostile case and the only one that matters for sizing: the AC # refuses the survivor until its own dead-peer timer frees the dead session.