Files
lab/migration/vrrp-wan-health

25 lines
1.2 KiB
Plaintext
Raw Normal View History

vyos: VRRP health check so the WAN and the gateway VIP cannot separate Outage of 2026-09-02: vyos002 held every floating gateway IP while vyos001 held the only working WAN. The LAN had a gateway that could not reach the internet, and it stayed that way until vyos002 was powered off by hand. Three causes, none of them bad luck. VRRP had no health check of any kind, so mastership was decided purely on whether the peer was still advertising and never on whether this router could route. vyos002 structurally cannot route -- bond0.53 is `disable`d because the 10 gig lease is bound to a cloned MAC that only one box may hold, and pppoe0 is not up. And `no-preempt` is set on every group, so once vyos002 took master it kept it even with a healthy priority-200 peer sitting next to it. Reproduced exactly in labsim: stopping keepalived on the primary moved every VIP to the WAN-less secondary and LAN internet went from 9ms to 100% loss; restarting the healthy primary left it BACKUP and the outage in place. Applying this check self-healed it -- secondary to FAULT, primary to MASTER, internet back. The check asks "do I have an address on a WAN interface", deliberately not "can I reach the internet" and not "do I have a default route". During a real ISP outage the default route disappears on BOTH routers; keying on that would put both in FAULT, nobody would hold the VIPs, and an internet outage would become a total one. Config goes on the SYNC GROUP, not per group: VyOS refuses a per-group check while the group is in a sync group ("Only sync group health check will be used"), and sync-group scope is what we want anyway so all VIPs move together. Known cost, measured: with the primary genuinely dead the secondary stays FAULT and NOTHING holds the gateway, so inter-VLAN routing stops too. That is the honest consequence of a backup that cannot route. The fix for it is to make the WAN follow mastership so the backup CAN route -- next, and rehearsed separately, since it is the one change that can lose the DHCP lease. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-09-02 11:39:44 +01:00
#!/bin/sh
# VRRP health check: may THIS router hold the floating IPs?
#
# It may only if it can actually carry the WAN. Without this, VRRP decides
# mastership purely on whether the peer is still advertising -- so a router with
# no WAN at all happily takes the VIPs and blackholes the entire LAN's internet
# while looking perfectly healthy. That is not hypothetical: it is the outage of
# 2026-09-02, reproduced in labsim.
#
# The test is "do I have an ADDRESS on a WAN interface", deliberately NOT "can I
# reach the internet" and NOT "do I have a default route". During a real ISP
# outage the default route disappears on BOTH routers; a check keyed on that
# would put both into FAULT, nobody would hold the VIPs, and the LAN would lose
# inter-VLAN routing too -- turning an internet outage into a total one. An
# address on a WAN interface distinguishes "this box structurally cannot route"
# (the failure we must prevent) from "the internet happens to be down right now"
# (which the router can do nothing about, and during which it should keep
# serving the LAN).
#
# exit 0 = eligible for MASTER, non-zero = release and let the peer have it.
for ifc in bond0.53 pppoe0; do
ip -4 addr show dev "$ifc" 2>/dev/null | grep -q 'inet ' && exit 0
done
exit 1