7bf3f42e19
Some checks failed
The ISP is consumer with one static IP, so "both routers hold WAN" is not
available: the 10 gig lease is anchored to a cloned MAC and the PPPoE line to a
single credential. The WAN therefore has to move with mastership.
Proven end to end in the sim: the secondary was promoted, took the WAN, got a
lease, installed a default route, and a LAN VM reached the internet through it
(3/3, 9ms). Demoting released it -- link down, no address. Rebooting the master
converged correctly too: it came back BACKUP with the WAN disabled while the
peer kept it.
The rehearsal earned its keep four times over, and none of these were visible
from reading the docs:
- `transition-script` alone is NOT safe to hang internet on. VyOS delivers it
through keepalived-fifo.py, and on one promotion that helper logged NOTHING
while Keepalived_vrrp logged all six instances entering MASTER and the
built-in notify_master for conntrack-sync ran normally. The result was a
router holding every VIP with no WAN -- the 2026-09-02 outage, recreated by
the mechanism meant to prevent it. Hence vrrp-wan-reconcile on a 30s timer:
the scripts give speed, the timer gives correctness.
- The health check must ask REALITY, not a marker. Keying "am I master" on a
/run file written by the transition script meant that when the script did
not run, the router believed it was backup, passed the check, and kept the
VIPs it could not serve. It now asks whether the VIP is actually on the box.
- The old address-based check DEADLOCKED this design: may-I-be-master required
already having WAN, and only the master gets WAN. That is why vyos002 sat in
FAULT for ever -- the safety check had silently removed the redundancy it
existed to protect.
- script-template must be the FIRST thing a script does. Sourced after an if,
an exec and a mkdir it terminated the script inside the source, rc=0, no
output: the reconciler reported success having done nothing. Hence the split
into vrrp-wan-apply, matching the shape /config/vyos-known-good already uses.
Two hazards found and handled rather than discovered in production:
- A `configure` session whose process dies leaks a unionfs mount under
/opt/vyatta/config/tmp, and one of those holds the commit lock -- after
which every commit fails, including the manual one you try to fix it with.
A 30s job that can leak one per failure wedges the box on its own, so the
reconciler reaps dead sessions before it starts. It cleared 8 on the sim.
- Any `save` while a box is master persists the enabled WAN into config.boot,
so a reboot would claim the shared MAC regardless of VRRP state. Observed:
an ordinary console-apply did exactly this. config.boot must keep `disable`
on BOTH routers; the model asserts it and vyos:verify reports it as drift.
NOT yet applied to production, and it should not be until the remaining item is
settled: a clean, deliberately-triggered failover has been seen via reboot, but
`restart vrrp` twice failed to move mastership at all, so the trigger for a
planned failover is still unproven.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
69 lines
3.4 KiB
Bash
Executable File
69 lines
3.4 KiB
Bash
Executable File
#!/bin/sh
|
|
# VRRP health check: may THIS router hold the floating IPs?
|
|
#
|
|
# It may only if it can actually carry the WAN. Without this, VRRP decides
|
|
# mastership purely on whether the peer is still advertising -- so a router with
|
|
# no WAN at all happily takes the VIPs and blackholes the entire LAN's internet
|
|
# while looking perfectly healthy. That is not hypothetical: it is the outage of
|
|
# 2026-09-02, reproduced in labsim.
|
|
#
|
|
# ---------------------------------------------------------------------------
|
|
# The first version of this script asked one question: "do I have an address on
|
|
# a WAN interface". That is correct for a pair where both routers hold WAN all
|
|
# the time. Ours cannot: the 10 gig lease is bound to a cloned MAC and the
|
|
# PPPoE line to a single credential, so the WAN follows mastership (see
|
|
# vrrp-wan-take). Against that design the old check DEADLOCKS --
|
|
#
|
|
# may I be master? -> only if I already have WAN
|
|
# do I have WAN? -> only if I am master
|
|
#
|
|
# -- and the backup sits in FAULT for ever. vyos002 sat exactly there, which
|
|
# meant the pair could not fail over at all: the safety check had quietly
|
|
# removed the redundancy it was protecting.
|
|
#
|
|
# So the question is now asked in the right order: enforce "must have WAN" only
|
|
# on the router that is actually HOLDING the VIPs, and give a new master time to
|
|
# bring the WAN up before judging it.
|
|
# ---------------------------------------------------------------------------
|
|
#
|
|
# exit 0 = eligible for MASTER, non-zero = release and let the peer have it.
|
|
|
|
STATE=/run/vrrp-wan
|
|
GRACE=90 # seconds a new master gets to complete DHCP / PPPoE dial-up
|
|
VIP="${VRRP_WAN_VIP:-192.168.1.1}"
|
|
|
|
# Am I holding the VIPs? Asked of REALITY -- is the management VIP actually on
|
|
# this box -- and not of a /run marker.
|
|
#
|
|
# The marker was the first design and it is unsafe: it is written by the VRRP
|
|
# transition script, and in labsim that script silently failed to run on a
|
|
# promotion (VyOS's keepalived-fifo.py helper stopped delivering while
|
|
# keepalived's own notifies kept working). The router then believed it was
|
|
# backup, passed this check, and sat holding every VIP with no WAN -- the exact
|
|
# outage this script exists to prevent, re-created by trusting the reporter
|
|
# instead of the fact.
|
|
[ -n "$(ip -4 -o addr show 2>/dev/null | grep " ${VIP}/")" ] || exit 0
|
|
|
|
# Master with an address on a WAN interface: healthy.
|
|
#
|
|
# Deliberately NOT "can I reach the internet" and NOT "do I have a default
|
|
# route". During a real ISP outage the route disappears on BOTH routers; a check
|
|
# keyed on that would put both into FAULT, nobody would hold the VIPs, and the
|
|
# LAN would lose inter-VLAN routing too -- turning an internet outage into a
|
|
# total one. A DHCP lease survives an ISP outage, so an address still
|
|
# distinguishes "this box structurally cannot route" from "the internet is down
|
|
# right now", which is the distinction that matters.
|
|
for ifc in bond0.53 pppoe0; do
|
|
ip -4 addr show dev "$ifc" 2>/dev/null | grep -q 'inet ' && exit 0
|
|
done
|
|
|
|
# Master, no WAN yet, still within the grace window: DHCP negotiation and PPPoE
|
|
# dial-up take real time, and the ISP has to accept the cloned MAC arriving on a
|
|
# different port. Failing here would demote the new master before it ever had a
|
|
# chance, and hand the VIPs straight back -- a flap, not a failover.
|
|
since=$(cat "$STATE/since" 2>/dev/null || echo 0)
|
|
[ $(( $(date +%s) - since )) -lt "$GRACE" ] && exit 0
|
|
|
|
# Master, past grace, still no WAN: release. This is the 2026-09-02 case.
|
|
exit 1
|