914135c47d8bff365b52915b0a5737a8f118b3bb
6 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Michal
|
6987b324f1 |
vrrp-wan: size GRACE from the measured hostile failover, not the theory
Some checks failed
With the matrix actually setting session-control, T4 timed a destroyed
master's takeover at:
replace 26s
deny 148s <-- sizing case
disable 21s
`deny` is the case GRACE exists for: the AC refuses the survivor until its own
dead-peer timer frees the dead session. The session poller caught it happening
-- the destroyed router's session stayed in the table while the survivor's
dials appeared and were rejected, twice, before one took at 148s.
148s is well past the lcp-echo-interval(30) x failure(3) = 90s budget that 180
was sized against, leaving 32s of margin. GRACE=300 is ~2x the worst observed.
Treat 148s as a floor, not a worst case: these are idle 2-vCPU VMs, and the AC
shares an OVS bridge with the routers, so `virsh destroy` removes the port and
accel-ppp sees the peer vanish. A real BRAS over DSL never learns our router
died and waits out longer timers of its own.
The cost is stated in the conf: GRACE is also how long an alive-but-unroutable
master holds every VIP before yielding. bond0.53 covers most of that -- a DHCP
lease satisfies the check in seconds -- so GRACE only dominates when PPPoE is
the last path. Kept the health check's fallback in step, since keepalived runs
it with no environment and that number decides mastership if the conf is ever
missing.
check_invariant no longer fails blind on the AC's session count. That count is
only a proxy for "two of our routers dialled", and only while the AC enforces
single-session; under `disable` it does not, so a destroyed router's session
lingers and the count reads 2 with exactly one live router dialled. The real
invariant -- at most one router holds pppoe0 -- is now the failing one, and the
stale session is reported as a WARN rather than silenced, because it still
occupies the slot at a real ISP and is precisely what made `deny` take 148s.
|
||
|
Michal
|
93fed7826b |
labsim: PPPoE HA passes the matrix, and the health check had a real flap bug
Some checks failed
Full run green: baseline (one AC session, held by the VIP holder), clean
failover (pppoe0 moves in 26s, old master releases), 10 gig down (route falls to
pppoe0 and the LAN is back online in 5s), lease expiry (the guard hangs up), and
a missing peers file (NRestarts=0, no loop). Evidence in
labsim/wan-failover-evidence/.
The 10 gig test found a genuine bug in vrrp-wan-health, not in the sim. GRACE
was measured from PROMOTION, so an established master had no grace at all --
after hours of uptime `now - since` far exceeds any window. The first moment
bond0.53 went down while pppoe0 was mid-redial, the master failed its own check,
shed every VIP, and the peer inherited the same WAN outage and did the same. A
brief WAN blip would have flapped the production pair. The stamp is now
refreshed on every healthy tick, so grace measures time since the box last
demonstrably HAD a WAN -- survivable wherever the gap happens, not only just
after a promotion.
Three harness bugs, all the same shape, all of which produced a confident wrong
answer before being caught:
- waiting for "exactly one pppoe0 holder" returns INSTANTLY during a handover,
because it was already true. The useful question is who holds it.
- judging connectivity on a single ping 20s after a link drop reported an
outage that had already healed. Poll, do not sample.
- `-o PreferredAuthentications=password` suits the routers but not the Alpine
LAN VMs, whose sshd offers keyboard-interactive: ssh exited 255 before
running anything and the test read that as "the LAN lost the internet". A
tcpdump on the router showed the pings leaving pppoe0 NATed to
198.51.100.117 and the replies coming back the whole time. An exit code that
can mean "the network is broken" or "I could not log in" is not a
connectivity test, so the check now asserts on what the guest reported.
That last one is why the harness asks the routers and the access concentrator
rather than a client, and why it refuses to run at all while either router still
has a default route via eth2 -- the libvirt-NAT scaffold answers connectivity
checks the WAN under test would have failed.
Still to run: hard failover (destroy the master), and the session-control
replace/deny/disable axis that brackets Vodafone's unknown behaviour and sets
the final GRACE. Nothing applied to production.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
|
||
|
Michal
|
4efd70c987 |
vyos: move PPPoE off the config plane onto a gated systemd unit
Some checks failed
PPPoE HA could not work as written, and the reason is structural rather than a
bug: `set interfaces pppoe pppoe0 disable` and `delete` are handled identically
by interfaces_pppoe.py -- both UNLINK /etc/ppp/peers/pppoe0. That path is pppd's
own options file, so the resting state destroyed exactly what the promotion path
needed, and `ppp@pppoe0` restart-looped against it (observed: 47 restarts, zero
sessions at the access concentrator). It also made op-mode `connect interface
pppoe0` unusable, and put every failover behind a priority-322 commit where one
unrelated invalid node fails the whole thing -- which has already taken the
10 gig down once.
pppoe0 is now configured identically and ENABLED on both routers, so the peers
file always exists, and dialling is gated by a drop-in on the unit:
ConditionPathExists=/run/vrrp-wan/may-dial
ConditionPathExists=/etc/ppp/peers/pppoe0
/run is tmpfs, so the gate is shut at boot and neither box can dial before VRRP
has decided. That matters more than it first appears: with the node enabled,
interfaces_pppoe.py restarts ppp on EVERY commit touching the pppoe subtree when
the daemon is not running -- so the backup actively tries to dial whenever
anything commits. The gate is the only thing making that a no-op, which is why
vrrp-wan-reconcile now refuses to bless a box whose drop-in is missing: /etc is
per-image, and a VyOS upgrade would otherwise silently remove the protection.
may-dial is a LEASE, not a flag. ConditionPathExists is evaluated at start only
-- it can prevent a dial, never revoke one -- so a reconciler that stops running
while its box is demoted would keep the one ISP session for ever. The reconciler
renews the lease; a new 5s vrrp-wan-guard revokes it, and only ever revokes. It
fired correctly first time: "GUARD: lease stale (81s > 75s)".
Also: remove-then-stop on release (the file's absence blocks a NEW start that a
concurrent commit would trigger); a flap damper, because two routers that both
believe they hold the VIP will both dial and each dial kills the other's session
-- against a real ISP that is how an account gets rate-limited; and a guard on
`cfg` returning empty under commit-lock contention, which had already produced
one spurious "releasing" on a box that needed nothing.
GRACE 90 -> 180. accel-ppp's dead-peer budget is lcp-echo-interval(30) x
failure(3) = 90s, so the old value sat exactly on the boundary: a hard failover
into an AC that does not replace the stale session would fail its own check,
shed the VIPs, and leave both routers in FAULT.
The sim could not have tested any of this. Both routers now get the identical
WAN -- the secondary had none "because two PPPoE clients sharing one credential
is a different failure mode than anything production has", which is backwards:
that IS production. It also left the pair incomparable, ten NAT rules against
none. Safety now comes from resting state, not asymmetry.
Three more things the sim was hiding:
- the drift check's secondary regex omitted interfaces pppoe/bonding, nat
source and protocols failover, so it reported "in sync" for a box with no
WAN at all;
- the VRRP health-check and transition-script hooks existed on both live VMs
and in NEITHER generator -- the mechanism under test was pure undetected
drift;
- labsim-vyos's only default route was the libvirt-NAT scaffold, so every
"the LAN still has internet" verdict on it was answered by eth2 rather than
the WAN. --drop-scaffold applied; the earlier DHCP-failover proof is being
re-run because of it.
vrrp-wan-install ends the other half of that: the sim's previous proof came from
scripts hand-`sed`-ed in place, so the tested behaviour was not the committed
behaviour. `--check` now makes that a hard failure.
First green run: master holds both WANs, backup released, and the AC reports
exactly ONE session. sim-net-apply.sh check: all four in sync.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
|
||
|
Michal
|
2e828b8af2 |
vyos: a failover you can actually trigger, and four bugs found triggering it
Some checks failed
A planned failover had no reliable lever. VyOS offers only `restart vrrp`, and
neither that nor `systemctl restart keepalived` is dependable: with advert_int 1
the peer declares the master dead after ~3.6s and a restart usually finishes
inside that window. Measured -- the same command moved mastership on one run and
not on the next three. A fail-back step you cannot trigger on purpose is not a
procedure, and the recovery card depended on one.
The lever is now `touch /run/vrrp-wan/force-fault`: the health check fails, the
sync group sheds every VIP, and the peer takes over. It exercises the same path
a real WAN loss takes rather than a special case, and it lives in /run so a
reboot cannot leave a router permanently ineligible.
Proven end to end in labsim: lever -> mastership moves -> WAN follows -> the old
master releases -> a LAN VM has internet -> the faulted router returns to BACKUP
and is eligible again.
Getting there exposed four real bugs, two of which would have broken a GENUINE
failover, not just the drill:
- The grace stamp was written only by the 30s reconciler, so a freshly
promoted master reached the 5s health check with no stamp, scored grace = 0,
failed instantly and went FAULT. With the peer already faulted that left
BOTH routers in FAULT and the LAN with no gateway at all -- worse than the
outage the check exists to prevent. The check now stamps on promotion.
- And it inherited STALE stamps from an earlier mastership, failing ~5s after
passing. The stamp is now cleared on the way down, by the health check
itself, not only by the reconciler.
- The lock fd leaked into VyOS's config session: `exec 9>` is inherited by the
long-lived unionfs-fuse the session spawns, which never closes it. From the
first config change on, every later reconciler run lost the flock and exited
0 having done nothing -- healthy-looking journal, silently stopped
reconciling. That is how a demoted router kept the WAN. Children now get 9>&-.
- vrrp-wan-apply touched pppoe0 unconditionally. On a box where pppoe0 has no
source-interface VyOS rejects the whole commit ("Physical source-interface
required"), taking the bond0.53 change down with it -- and the script still
returned 0, so the reconciler logged a release that never happened. pppoe0 is
now guarded on existence and the commit's verdict is propagated.
Still NOT applied to production. The pair is single-homed on WAN until it is.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
|
||
|
Michal
|
7bf3f42e19 |
vyos: WAN follows VRRP mastership, rehearsed in labsim
Some checks failed
The ISP is consumer with one static IP, so "both routers hold WAN" is not
available: the 10 gig lease is anchored to a cloned MAC and the PPPoE line to a
single credential. The WAN therefore has to move with mastership.
Proven end to end in the sim: the secondary was promoted, took the WAN, got a
lease, installed a default route, and a LAN VM reached the internet through it
(3/3, 9ms). Demoting released it -- link down, no address. Rebooting the master
converged correctly too: it came back BACKUP with the WAN disabled while the
peer kept it.
The rehearsal earned its keep four times over, and none of these were visible
from reading the docs:
- `transition-script` alone is NOT safe to hang internet on. VyOS delivers it
through keepalived-fifo.py, and on one promotion that helper logged NOTHING
while Keepalived_vrrp logged all six instances entering MASTER and the
built-in notify_master for conntrack-sync ran normally. The result was a
router holding every VIP with no WAN -- the 2026-09-02 outage, recreated by
the mechanism meant to prevent it. Hence vrrp-wan-reconcile on a 30s timer:
the scripts give speed, the timer gives correctness.
- The health check must ask REALITY, not a marker. Keying "am I master" on a
/run file written by the transition script meant that when the script did
not run, the router believed it was backup, passed the check, and kept the
VIPs it could not serve. It now asks whether the VIP is actually on the box.
- The old address-based check DEADLOCKED this design: may-I-be-master required
already having WAN, and only the master gets WAN. That is why vyos002 sat in
FAULT for ever -- the safety check had silently removed the redundancy it
existed to protect.
- script-template must be the FIRST thing a script does. Sourced after an if,
an exec and a mkdir it terminated the script inside the source, rc=0, no
output: the reconciler reported success having done nothing. Hence the split
into vrrp-wan-apply, matching the shape /config/vyos-known-good already uses.
Two hazards found and handled rather than discovered in production:
- A `configure` session whose process dies leaks a unionfs mount under
/opt/vyatta/config/tmp, and one of those holds the commit lock -- after
which every commit fails, including the manual one you try to fix it with.
A 30s job that can leak one per failure wedges the box on its own, so the
reconciler reaps dead sessions before it starts. It cleared 8 on the sim.
- Any `save` while a box is master persists the enabled WAN into config.boot,
so a reboot would claim the shared MAC regardless of VRRP state. Observed:
an ordinary console-apply did exactly this. config.boot must keep `disable`
on BOTH routers; the model asserts it and vyos:verify reports it as drift.
NOT yet applied to production, and it should not be until the remaining item is
settled: a clean, deliberately-triggered failover has been seen via reboot, but
`restart vrrp` twice failed to move mastership at all, so the trigger for a
planned failover is still unproven.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
|
||
|
Michal
|
23783b8486 |
vyos: VRRP health check so the WAN and the gateway VIP cannot separate
Outage of 2026-09-02: vyos002 held every floating gateway IP while vyos001 held
the only working WAN. The LAN had a gateway that could not reach the internet,
and it stayed that way until vyos002 was powered off by hand.
Three causes, none of them bad luck. VRRP had no health check of any kind, so
mastership was decided purely on whether the peer was still advertising and
never on whether this router could route. vyos002 structurally cannot route --
bond0.53 is `disable`d because the 10 gig lease is bound to a cloned MAC that
only one box may hold, and pppoe0 is not up. And `no-preempt` is set on every
group, so once vyos002 took master it kept it even with a healthy priority-200
peer sitting next to it.
Reproduced exactly in labsim: stopping keepalived on the primary moved every VIP
to the WAN-less secondary and LAN internet went from 9ms to 100% loss; restarting
the healthy primary left it BACKUP and the outage in place. Applying this check
self-healed it -- secondary to FAULT, primary to MASTER, internet back.
The check asks "do I have an address on a WAN interface", deliberately not "can
I reach the internet" and not "do I have a default route". During a real ISP
outage the default route disappears on BOTH routers; keying on that would put
both in FAULT, nobody would hold the VIPs, and an internet outage would become a
total one.
Config goes on the SYNC GROUP, not per group: VyOS refuses a per-group check
while the group is in a sync group ("Only sync group health check will be
used"), and sync-group scope is what we want anyway so all VIPs move together.
Known cost, measured: with the primary genuinely dead the secondary stays FAULT
and NOTHING holds the gateway, so inter-VLAN routing stops too. That is the
honest consequence of a backup that cannot route. The fix for it is to make the
WAN follow mastership so the backup CAN route -- next, and rehearsed separately,
since it is the one change that can lose the DHCP lease.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
|