13dcdff1ef
The WAN now follows VRRP mastership, so a bad failover takes the house offline -- and whoever is debugging it is offline too. The recovery path therefore has to be already on the box, not in a chat log. `/config/wan-panic` works out which router it is running on and does the right thing: on vyos002 it stands down (force-fault), which is what lets vyos001 take the VIPs back; on vyos001 it clears force-fault and says plainly that no-preempt means you must run it on the OTHER box to actually move anything. `status` shows who holds the VIP, which WANs are up and the default route. `undo` stops both timers so nothing moves the WAN again, leaving whatever is currently up exactly as it is. It uses the force-fault lever rather than restarting keepalived because failing the health check is the supported way to shed mastership -- `restart vrrp` is not dependable, since with advert_int 1 the peer declares the master dead in ~3.6s and the restart usually finishes inside that window. Proven in production on vyos002, which carries no traffic: BACKUP -> FAULT in ~20s and back to BACKUP on clearing, with vyos001 untouched and the internet steady at 7.7-8.1ms throughout. The card is installed to /config/RECOVERY-CARD.md on both routers and copied to ~/WAN-RECOVERY-CARD.md, because a card you can only read with working internet is not a card. It leads with how to reach the routers over the LoT leg, which is L2-direct and survives Management/VRRP/routing being broken.
3.0 KiB
RECOVERY CARD — internet is down after a WAN failover
No internet means no Claude. Everything here runs from the routers themselves. Print this or keep it on a phone.
1. Get to a router
The LoT leg is L2-direct on bond0.10. It survives Management, VRRP and
routing being broken:
ssh vyos@10.0.1.252 # vyos001 (normally MASTER, has the WAN)
ssh vyos@10.0.1.253 # vyos002 (normally BACKUP, has nothing)
Password is the usual one. If SSH is dead, use the JetKVM consoles.
2. See what is going on
sudo /config/wan-panic status
Run it on both. You want exactly ONE box saying holds VIP : YES, and that
same box showing a WAN line and a route.
| what you see | what it means |
|---|---|
one box YES with WAN + route |
healthy, look elsewhere for the fault |
one box YES, no WAN, no route |
the failover half-worked — go to §3 |
both YES |
VRRP split — go to §3, run it on vyos002 |
neither YES |
both faulted — go to §4 |
3. Give the WAN back to vyos001
Run this on vyos002 (10.0.1.253). This is the one that matters — vyos002
standing down is what lets vyos001 take over.
sudo /config/wan-panic
Wait ~30s. Then on vyos001 (10.0.1.252):
sudo /config/wan-panic status
Expect holds VIP : YES and a WAN line with bond0.53=… and/or pppoe0=….
Once the house is back online and you want vyos002 to be a standby again:
sudo rm /run/vrrp-wan/force-fault # on vyos002
Leave it set if you would rather have no standby than any more surprises — that is the pre-2026-09-06 arrangement and the house runs fine on it.
4. Stop the mechanism touching anything
If the WAN keeps moving, or you do not trust the automation:
sudo /config/wan-panic undo # on BOTH routers
Stops the reconcile and guard timers. Whatever WAN is up stays up. Nothing will move it again until you re-enable the timers:
sudo systemctl enable --now vrrp-wan-reconcile.timer vrrp-wan-guard.timer
5. Nuclear — put the config back
Only if the config itself is wrong. This reboots the router.
sudo /config/vyos-known-good restore # on BOTH routers
The pinned config is from 2026-09-06, immediately before the PPPoE HA rollout: 1112 lines on vyos001, 1069 on vyos002.
Why the WAN can only be on one box
One ISP account each way. The 10 gig lease is bound to a cloned MAC
(f0:9f:c2:12:9b:4f, the old USG's) and Vodafone is a single-session PPPoE
credential. Two routers holding either at once is worse than one holding
neither — that is why the safe state is "vyos001 has it, vyos002 is inert".
Known gap
vyos001's config.boot does not carry vif 53 disable. If vyos001 reboots
while vyos002 is master, the cloned MAC is briefly live on both. It clears
itself within 30s (the reconciler disables it on the non-master). If you see
MAC flapping on the WAN switch port right after a vyos001 reboot, that is this,
and it will stop on its own.