7bf3f42e19450787c0eb627f20b92fbcf672a7c4
2 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Michal
|
7bf3f42e19 |
vyos: WAN follows VRRP mastership, rehearsed in labsim
Some checks failed
The ISP is consumer with one static IP, so "both routers hold WAN" is not
available: the 10 gig lease is anchored to a cloned MAC and the PPPoE line to a
single credential. The WAN therefore has to move with mastership.
Proven end to end in the sim: the secondary was promoted, took the WAN, got a
lease, installed a default route, and a LAN VM reached the internet through it
(3/3, 9ms). Demoting released it -- link down, no address. Rebooting the master
converged correctly too: it came back BACKUP with the WAN disabled while the
peer kept it.
The rehearsal earned its keep four times over, and none of these were visible
from reading the docs:
- `transition-script` alone is NOT safe to hang internet on. VyOS delivers it
through keepalived-fifo.py, and on one promotion that helper logged NOTHING
while Keepalived_vrrp logged all six instances entering MASTER and the
built-in notify_master for conntrack-sync ran normally. The result was a
router holding every VIP with no WAN -- the 2026-09-02 outage, recreated by
the mechanism meant to prevent it. Hence vrrp-wan-reconcile on a 30s timer:
the scripts give speed, the timer gives correctness.
- The health check must ask REALITY, not a marker. Keying "am I master" on a
/run file written by the transition script meant that when the script did
not run, the router believed it was backup, passed the check, and kept the
VIPs it could not serve. It now asks whether the VIP is actually on the box.
- The old address-based check DEADLOCKED this design: may-I-be-master required
already having WAN, and only the master gets WAN. That is why vyos002 sat in
FAULT for ever -- the safety check had silently removed the redundancy it
existed to protect.
- script-template must be the FIRST thing a script does. Sourced after an if,
an exec and a mkdir it terminated the script inside the source, rc=0, no
output: the reconciler reported success having done nothing. Hence the split
into vrrp-wan-apply, matching the shape /config/vyos-known-good already uses.
Two hazards found and handled rather than discovered in production:
- A `configure` session whose process dies leaks a unionfs mount under
/opt/vyatta/config/tmp, and one of those holds the commit lock -- after
which every commit fails, including the manual one you try to fix it with.
A 30s job that can leak one per failure wedges the box on its own, so the
reconciler reaps dead sessions before it starts. It cleared 8 on the sim.
- Any `save` while a box is master persists the enabled WAN into config.boot,
so a reboot would claim the shared MAC regardless of VRRP state. Observed:
an ordinary console-apply did exactly this. config.boot must keep `disable`
on BOTH routers; the model asserts it and vyos:verify reports it as drift.
NOT yet applied to production, and it should not be until the remaining item is
settled: a clean, deliberately-triggered failover has been seen via reboot, but
`restart vrrp` twice failed to move mastership at all, so the trigger for a
planned failover is still unproven.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
|
||
|
Michal
|
23783b8486 |
vyos: VRRP health check so the WAN and the gateway VIP cannot separate
Outage of 2026-09-02: vyos002 held every floating gateway IP while vyos001 held
the only working WAN. The LAN had a gateway that could not reach the internet,
and it stayed that way until vyos002 was powered off by hand.
Three causes, none of them bad luck. VRRP had no health check of any kind, so
mastership was decided purely on whether the peer was still advertising and
never on whether this router could route. vyos002 structurally cannot route --
bond0.53 is `disable`d because the 10 gig lease is bound to a cloned MAC that
only one box may hold, and pppoe0 is not up. And `no-preempt` is set on every
group, so once vyos002 took master it kept it even with a healthy priority-200
peer sitting next to it.
Reproduced exactly in labsim: stopping keepalived on the primary moved every VIP
to the WAN-less secondary and LAN internet went from 9ms to 100% loss; restarting
the healthy primary left it BACKUP and the outage in place. Applying this check
self-healed it -- secondary to FAULT, primary to MASTER, internet back.
The check asks "do I have an address on a WAN interface", deliberately not "can
I reach the internet" and not "do I have a default route". During a real ISP
outage the default route disappears on BOTH routers; keying on that would put
both in FAULT, nobody would hold the VIPs, and an internet outage would become a
total one.
Config goes on the SYNC GROUP, not per group: VyOS refuses a per-group check
while the group is in a sync group ("Only sync group health check will be
used"), and sync-group scope is what we want anyway so all VIPs move together.
Known cost, measured: with the primary genuinely dead the secondary stays FAULT
and NOTHING holds the gateway, so inter-VLAN routing stops too. That is the
honest consequence of a backup that cannot route. The fix for it is to make the
WAN follow mastership so the backup CAN route -- next, and rehearsed separately,
since it is the one change that can lose the DHCP lease.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
|