wan-panic: a one-command revert that works with no internet and no Claude
Some checks failed
Some checks failed
The WAN now follows VRRP mastership, so a bad failover takes the house offline -- and whoever is debugging it is offline too. The recovery path therefore has to be already on the box, not in a chat log. `/config/wan-panic` works out which router it is running on and does the right thing: on vyos002 it stands down (force-fault), which is what lets vyos001 take the VIPs back; on vyos001 it clears force-fault and says plainly that no-preempt means you must run it on the OTHER box to actually move anything. `status` shows who holds the VIP, which WANs are up and the default route. `undo` stops both timers so nothing moves the WAN again, leaving whatever is currently up exactly as it is. It uses the force-fault lever rather than restarting keepalived because failing the health check is the supported way to shed mastership -- `restart vrrp` is not dependable, since with advert_int 1 the peer declares the master dead in ~3.6s and the restart usually finishes inside that window. Proven in production on vyos002, which carries no traffic: BACKUP -> FAULT in ~20s and back to BACKUP on clearing, with vyos001 untouched and the internet steady at 7.7-8.1ms throughout. The card is installed to /config/RECOVERY-CARD.md on both routers and copied to ~/WAN-RECOVERY-CARD.md, because a card you can only read with working internet is not a card. It leads with how to reach the routers over the LoT leg, which is L2-direct and survives Management/VRRP/routing being broken.
This commit is contained in:
99
migration/RECOVERY-CARD-wan-panic.md
Normal file
99
migration/RECOVERY-CARD-wan-panic.md
Normal file
@@ -0,0 +1,99 @@
|
||||
# RECOVERY CARD — internet is down after a WAN failover
|
||||
|
||||
No internet means no Claude. Everything here runs from the routers themselves.
|
||||
**Print this or keep it on a phone.**
|
||||
|
||||
## 1. Get to a router
|
||||
|
||||
The LoT leg is L2-direct on `bond0.10`. It survives Management, VRRP and
|
||||
routing being broken:
|
||||
|
||||
```
|
||||
ssh vyos@10.0.1.252 # vyos001 (normally MASTER, has the WAN)
|
||||
ssh vyos@10.0.1.253 # vyos002 (normally BACKUP, has nothing)
|
||||
```
|
||||
|
||||
Password is the usual one. If SSH is dead, use the JetKVM consoles.
|
||||
|
||||
## 2. See what is going on
|
||||
|
||||
```
|
||||
sudo /config/wan-panic status
|
||||
```
|
||||
|
||||
Run it on both. You want exactly ONE box saying `holds VIP : YES`, and that
|
||||
same box showing a `WAN` line and a `route`.
|
||||
|
||||
| what you see | what it means |
|
||||
|---|---|
|
||||
| one box `YES` with WAN + route | healthy, look elsewhere for the fault |
|
||||
| one box `YES`, **no WAN**, no route | the failover half-worked — go to §3 |
|
||||
| **both** `YES` | VRRP split — go to §3, run it on vyos002 |
|
||||
| **neither** `YES` | both faulted — go to §4 |
|
||||
|
||||
## 3. Give the WAN back to vyos001
|
||||
|
||||
**Run this on vyos002 (`10.0.1.253`).** This is the one that matters — vyos002
|
||||
standing down is what lets vyos001 take over.
|
||||
|
||||
```
|
||||
sudo /config/wan-panic
|
||||
```
|
||||
|
||||
Wait ~30s. Then on vyos001 (`10.0.1.252`):
|
||||
|
||||
```
|
||||
sudo /config/wan-panic status
|
||||
```
|
||||
|
||||
Expect `holds VIP : YES` and a `WAN` line with `bond0.53=…` and/or `pppoe0=…`.
|
||||
|
||||
Once the house is back online and you want vyos002 to be a standby again:
|
||||
|
||||
```
|
||||
sudo rm /run/vrrp-wan/force-fault # on vyos002
|
||||
```
|
||||
|
||||
Leave it set if you would rather have no standby than any more surprises —
|
||||
that is the pre-2026-09-06 arrangement and the house runs fine on it.
|
||||
|
||||
## 4. Stop the mechanism touching anything
|
||||
|
||||
If the WAN keeps moving, or you do not trust the automation:
|
||||
|
||||
```
|
||||
sudo /config/wan-panic undo # on BOTH routers
|
||||
```
|
||||
|
||||
Stops the reconcile and guard timers. Whatever WAN is up **stays** up. Nothing
|
||||
will move it again until you re-enable the timers:
|
||||
|
||||
```
|
||||
sudo systemctl enable --now vrrp-wan-reconcile.timer vrrp-wan-guard.timer
|
||||
```
|
||||
|
||||
## 5. Nuclear — put the config back
|
||||
|
||||
Only if the config itself is wrong. **This reboots the router.**
|
||||
|
||||
```
|
||||
sudo /config/vyos-known-good restore # on BOTH routers
|
||||
```
|
||||
|
||||
The pinned config is from 2026-09-06, immediately before the PPPoE HA rollout:
|
||||
1112 lines on vyos001, 1069 on vyos002.
|
||||
|
||||
## Why the WAN can only be on one box
|
||||
|
||||
One ISP account each way. The 10 gig lease is bound to a cloned MAC
|
||||
(`f0:9f:c2:12:9b:4f`, the old USG's) and Vodafone is a single-session PPPoE
|
||||
credential. Two routers holding either at once is worse than one holding
|
||||
neither — that is why the safe state is "vyos001 has it, vyos002 is inert".
|
||||
|
||||
## Known gap
|
||||
|
||||
vyos001's `config.boot` does not carry `vif 53 disable`. If vyos001 reboots
|
||||
**while vyos002 is master**, the cloned MAC is briefly live on both. It clears
|
||||
itself within 30s (the reconciler disables it on the non-master). If you see
|
||||
MAC flapping on the WAN switch port right after a vyos001 reboot, that is this,
|
||||
and it will stop on its own.
|
||||
Reference in New Issue
Block a user