fd38c05c3e0edc24cd9942517f3ad42e2aae15bb
12 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Michal
|
cb03987c33 |
The drill evidence was silently gitignored by *.log
Some checks failed
PPPOE-HA.md cited migration/drill-evidence/wan-drill-2026-09-06-ipv6.log as the record behind the takeover/failback/HE-call figures, and the repo reported clean -- because the root .gitignore has `*.log`, so the file was never added. The citation was dangling and nobody would have found out until they went looking for the numbers. Renamed to .txt, which is also what this repo already does for evidence: labsim/wan-failover-evidence/ and labsim/vlan-leak-evidence/ store state.txt. The ignore rule is fine; the filename was wrong. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH |
||
|
Michal
|
8a0ef52909 |
The drill measured IPv6 through a failover: zero HE calls, and it followed
Some checks failed
Second drill of the day, after IPv6-follows-master went in. First time the IPv6 behaviour of a failover is a measurement rather than an assertion -- the thing this whole review started from was a 15.5ms figure read outside the window. TAKEOVER OK: 37s IPv6 followed in 37s (v4 37s, gap 0s) FAILBACK OK: 32s IPv6 back in 44s tun0 src : 87.192.101.48 -> 87.192.101.48 unchanged HE updates from vyos001: 0 HE updates from vyos002: 0 Zero HE API calls across a full takeover and failback is the invariant the design rests on, and it now has evidence: the 10 gig lease follows the cloned MAC, so the tunnel endpoint is the same address on whichever router holds the WAN and there is nothing to tell Hurricane Electric. This also exercised the proto-41 accept rule added to vyos002 earlier today. Without it the drill would have shown IPv6 failing to return while tun0 was up and radvd running -- which is precisely how the original gap hid. The two directions are NOT symmetric and the doc says so: IPv6 arrived in the same 5s sample on takeover but trailed by 12s on failback, because the reconciler enables bond0.53 first and v6_take only raises the tunnel once the source address exists. Bounded by one 30s tick. Also noted: the drill samples every 5s, so "gap 0s" means within the same sample, not simultaneous -- and the 37s vs the morning's 52s is a different run of the same IPv4 mechanism, not an improvement. Vodafone handed out a new address again across the drill (90.251.142.103 -> 90.251.152.236), corroborating that nothing may be pinned to the PPPoE address. Production returned to normal: vyos001 MASTER on all six with both WANs, vyos002 BACKUP with tun0 down and radvd stopped, force-fault clear. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH |
||
|
Michal
|
395577850c |
IPv6 was never HA, and the WAN becoming HA is what exposed it
Some checks failed
Reviewed the parked IPv6 task against the PPPoE-HA work of 2026-09-05/06. The
gate that parked it ("WI-8 before IPv6") is cleared, but the same work
invalidated the assumption the IPv6 design rested on.
Verified on the live routers: vyos002 has no tun0, no he-tunnel-follow, no
he-secrets, no VLAN 9 prefix and no route6 ::/0 -- only the pre-staged
default-deny v6 firewall, which is correctly on both. Failover is now automatic
and drill-proven, so every failover takes the whole v6 estate down for as long
as vyos002 holds the VIP.
Four things that came out of checking rather than reading:
- PPPOE-HA.md's "tun0 survived untouched and IPv6 stayed up at 15.5ms" does not
follow from its own premise and is corrected in place. The endpoint address is
stable, but it MOVES to vyos002, which has nothing to decapsulate protocol 41.
wan-drill had no IPv6 check at all, which is why nobody caught it.
- A 22-second near-miss: vif53-pin-boot-disable bounced the 10 gig, he-tunnel-
follow ticked once and saw the PPPoE address, and vyos-failover restored the
route 22s before the second tick would have pointed HE at an address Vodafone
reissues on every dial.
- VyOS does NOT leave a tunnel down when its source-address is absent (the
override's stated reason for leaving IPv6 single-homed). Measured in labsim:
it commits rc=0 and brings the link UP -- a blackhole that attracts the v6
default route. The runtime gate is load-bearing, like the PPPoE gate.
- The RA link-mtu was pinned at 1480 while the tunnel correctly drops to 1472 on
the PPPoE path.
Mechanism, mirroring PPPoE HA -- identical config on both, gated at runtime, no
commit in the failover path:
- vrrp-wan-reconcile: a v6 kernel plane. tun0 and radvd follow the VIP; radvd is
stopped BEFORE the WAN goes so its farewell RA (router-lifetime 0) still has a
path out. The WAN early-exits became if-blocks so the plane runs every tick.
It deliberately does NOT call he-tunnel-follow: that would halve the
hysteresis the near-miss above showed we depend on.
- he-tunnel-follow: a master guard reading the same vrrp-wan.conf VIP, so the
backup copy cannot point HE at its own idle PPPoE line, plus a stubbable
HE_UPDATE_URL.
- vrrp-wan-install carries both, so --check and the upgrade runbook cover IPv6.
- wan-drill measures IPv6 in both timing loops and asserts zero HE API calls
across a router failover.
labsim finally has an HE endpoint, closing the gap the override itself cited as
why this was never rehearsed. Both ISP islands already share the libvirt network,
so that becomes the backbone and HE lives behind it on one address reachable over
either WAN. Proven in the sim: backup tun=DOWN radvd=inactive, master tun=UP
radvd=active, hysteresis then HE call then MTU 1480->1472, and VLAN 9 hosts
autoconfiguring from the RA. The end-to-end v6 datapath is NOT yet proven --
inter-island transit crosses libvirt NAT and the return path is lost. Recorded as
a KNOWN SIM GAP rather than papered over.
The model change is staged, not merged: another agent runs pulumi up on that
repo, and the gate must exist on vyos002 before the tunnel does.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
|
||
|
Michal
|
061b9e3d7e |
Close the last two items: config.boot pinned, and an upgrade runbook
Some checks failed
vif53-pin-boot-disable ran on vyos001. Both config.boots now pin `vif 53 disable` while vyos001's running config keeps the WAN, so a reboot in any order comes up unable to claim the cloned MAC. Cost: a 32s window (10s in the sim -- production commits under kea/BGP/conntrack are slower), bond0.53 re-leased 87.192.101.48 in 5s, and vyos-failover restored the primary route about a minute later. The house rode pppoe0 in between rather than losing the internet. Recorded the shape of that recovery, because I misread it myself: for ~60s after the bounce the default route really is on pppoe0, since vyos-failover only re-adds the bond0.53 route once its probes pass. A fresh `ip route show` in that window looks like a regression and is not one. VYOS-IMAGE-UPGRADE.md is step 9. An image upgrade keeps /config and REPLACES /etc, which silently removes the ppp@pppoe0 gate drop-in -- the only thing stopping the backup from dialling into a single-session account. The reconciler fails closed, so the symptom is "PPPoE never comes up" rather than "both routers dialled", but an upgraded box has no PPPoE until the gate is back. One router at a time, backup first. Deleted the "model hazard to fix before applying" section rather than leaving it: it advised asserting `vif 53 disable` on BOTH routers via an override, which is exactly wrong. That is runtime state owned by vrrp-wan-reconcile, and pinning it would fight the reconciler on every apply and briefly disable the live master's 10 gig each time. Replaced with what is actually done -- follow reality at runtime, hardcode safe at boot and at install time -- and said plainly not to reintroduce it. |
||
|
Michal
|
d08f68e28b |
PPPOE-HA: step 8 done -- model merged, zero drift
Some checks failed
Both overrides are on kubernetes-deployment main (45033dd) and the transition-scripts were applied to the boxes directly rather than left for an unattended `pulumi up` to find. vyos:verify reports both routers in sync. Also records the drill result in the header, since "deployed" and "proven to fail over" are different claims and only the second one is worth much. |
||
|
Michal
|
e1c571d004 |
PPPOE-HA: the drill closed the two biggest unknowns
Some checks failed
Controlled failover in production: takeover 52s, failback 36s, ~88s of interruption across two deliberate transitions. The cloned-MAC lease TRANSFERS. That was named in this file as "the largest untested item in any failover" -- whether the 10 gig ISP would re-issue 87.192.101.48 to f0:9f:c2:12:9b:4f arriving on a different switch port. It did, same address, inside the takeover window. Moved out of "what the sim cannot prove" rather than left there contradicting the evidence. VyOS dialling Vodafone also worked from both routers, which had never been done -- PPPoE was only ever proven on the USG. Vodafone did not refuse either re-dial, so its session-control behaves like `replace`, not the hostile `deny`. GRACE stays at 300 anyway: one drill on one evening is not the ISP's policy under all conditions. Worth knowing for anything added later: Vodafone hands out a DIFFERENT IPv4 on every dial (83.106.5.72 -> 90.251.153.180 -> 90.251.142.103). Nothing may be pinned to the pppoe0 address. Checked the HE IPv6 tunnel specifically, since it carries a hardcoded source-address -- it is pinned to 87.192.101.48, which is the 10 gig and stable across failover, so tun0 was untouched and IPv6 stayed up at 15.5ms. |
||
|
Michal
|
8aa3d0ebaa |
PPPOE-HA: record that vyos001's config.boot lacks vif 53 disable
Some checks failed
Runbook step 7 says confirm it on both; it holds on vyos002 and not on vyos001, whose config.boot dates from 2026-09-02 and predates this work. The exposure is new even though the condition is not: if vyos001 reboots while vyos002 is master holding the 10 gig, the cloned MAC is briefly live on both. That could not happen yesterday, because vyos002 was stuck in FAULT and could never be master. It self-heals in <=30s -- vyos001 comes up BACKUP under no-preempt and the reconciler commits `disable` on the next tick -- so this is a bounded window, not a split. Left as an outstanding item rather than fixed, because writing config.boot means `save`ing the running state, so the fix takes the live 10 gig down briefly and belongs in a maintenance window. Hand-editing config.boot would avoid the blip at the risk of an unbootable router, which is worse. |
||
|
Michal
|
8fce03e705 |
PPPOE-HA: deployed to production
Some checks failed
Mechanism installed on both routers (--check clean against git on both), `interfaces pppoe pppoe0 disable` removed from vyos002, and vyos002 is out of FAULT and holding BACKUP on all six groups for the first time in 3d10h -- the lab has a real standby again. Verified on the wire rather than from state: with a tcpdump running on vyos002's bond0.51 across the commit, ZERO PADI/PADR. The peers file rendered (so it can dial the instant it is promoted) while ConditionResult stayed `no` and NRestarts 0. vyos001's live session was untouched throughout -- same MainPID 2931, same 83.106.5.72, internet 7.6-8.4ms at 0% loss. Not done, deliberately: the controlled failover drill, which interrupts the household's internet, and merging the staged Pulumi override, which lands via another agent's `pulumi up` on a branch they have checked out. |
||
|
Michal
|
5ed0e4888a |
labsim: both matrices green end to end, numbers reproduced
Some checks failed
Final confirming run with GRACE=300 and every fix installed:
--all ALL PASS (T0 T3 T5 T8 T11 T12)
--hard ALL PASS (replace / deny / disable, each policy verified)
Second independent measurement of the hard failover, which is what makes the
figures trustworthy rather than anecdotal:
run 1 run 2
replace 26s 26s
deny 148s 141s
disable 21s 20s
`deny` sits at ~141-148s across both, so it is the AC's dead-peer behaviour
and not a one-off; GRACE=300 keeps roughly 2x margin.
Corrected an overclaim in PPPOE-HA.md while confirming it: the invariant row
read "AC never showed two simdsl sessions", and under session-control=disable
it did -- one live, one orphaned from the destroyed router. Only ever one LIVE
router dialled, which is the invariant that matters. Said so plainly rather
than leaving a table that reads better than the evidence.
|
||
|
Michal
|
6987b324f1 |
vrrp-wan: size GRACE from the measured hostile failover, not the theory
Some checks failed
With the matrix actually setting session-control, T4 timed a destroyed
master's takeover at:
replace 26s
deny 148s <-- sizing case
disable 21s
`deny` is the case GRACE exists for: the AC refuses the survivor until its own
dead-peer timer frees the dead session. The session poller caught it happening
-- the destroyed router's session stayed in the table while the survivor's
dials appeared and were rejected, twice, before one took at 148s.
148s is well past the lcp-echo-interval(30) x failure(3) = 90s budget that 180
was sized against, leaving 32s of margin. GRACE=300 is ~2x the worst observed.
Treat 148s as a floor, not a worst case: these are idle 2-vCPU VMs, and the AC
shares an OVS bridge with the routers, so `virsh destroy` removes the port and
accel-ppp sees the peer vanish. A real BRAS over DSL never learns our router
died and waits out longer timers of its own.
The cost is stated in the conf: GRACE is also how long an alive-but-unroutable
master holds every VIP before yielding. bond0.53 covers most of that -- a DHCP
lease satisfies the check in seconds -- so GRACE only dominates when PPPoE is
the last path. Kept the health check's fallback in step, since keepalived runs
it with no environment and that number decides mastership if the conf is ever
missing.
check_invariant no longer fails blind on the AC's session count. That count is
only a proxy for "two of our routers dialled", and only while the AC enforces
single-session; under `disable` it does not, so a destroyed router's session
lingers and the count reads 2 with exactly one live router dialled. The real
invariant -- at most one router holds pppoe0 -- is now the failing one, and the
stale session is reported as a WARN rather than silenced, because it still
occupies the slot at a real ISP and is precisely what made `deny` take 148s.
|
||
|
Michal
|
4d47b609a2 |
PPPOE-HA: record the two failure modes found by running the thing
Some checks failed
Both were invisible to inspection and only appeared under the hard matrix: the flap damper tearing down an established session via the lease it stopped renewing, and a missing peers file being completely silent. The second has a production edge worth spelling out -- an unsaved commit reverts on reboot and takes pppoe0 with it, leaving a standby that can never take over while looking perfectly healthy. |
||
|
Michal
|
d626750010 |
labsim: hard failover and reboot safety hold; runbook for the production apply
Some checks failed
Hard failover proven by destroying the master outright (virsh destroy -- no PADT,
the case a graceful stop cannot cover): the survivor took the VIPs, dialled, and
the access concentrator showed exactly ONE simdsl session from its MAC for the
whole seven minutes. When the destroyed router came back it did NOT dial --
ConditionResult=no, ActiveState=inactive, NRestarts=0, pppoe0 absent, may_dial=no
-- and the AC session count stayed at 1. That is the gate doing the one job it
exists for, on the path that previously had no protection at all.
Two more harness bugs of the same family as the last three, both of which
reported a working system as broken:
- ppp_on() returned an EMPTY string for an unreachable router, and `[ "" = 0 ]`
is false, so the hard-failover wait sat for its full timeout waiting for a
DESTROYED box to report zero -- long after the survivor had taken over
correctly. Absent now means 0.
- a VM restart recreates its taps under new names and the OVS bond keeps the
old ones: lacp dies, VLAN 1 goes with it, and the box returns reachable on
some VLANs and not others. That looked exactly like a failed failover. It is
the same stale-membership fault ovs_bond_router already detects, but nothing
ran it after a restart; the harness now does.
migration/PPPOE-HA.md is the runbook: what the design is, why `disable` cannot
work, the measured numbers, the deploy order (vyos002 first, on its own commit,
verified on the wire with tcpdump rather than from state), and the one-line
rollback.
It also records a model hazard found while writing it. The imported baseline
captures the RUNNING state, not the safe one -- vyos001 has no `vif 53 disable`
because it happens to be master, vyos002 does. An apply performed while vyos002
held the VIP would therefore enable vyos001's WAN too, putting the cloned MAC on
both boxes. An override must assert `vif 53 disable` on BOTH, accepting that an
apply then briefly disables the current master's 10 gig until the reconciler
restores it.
Still not applied to production.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
|