wan-panic: a one-command revert that works with no internet and no Claude
Some checks failed
CI/CD / typecheck (push) Failing after 8s
CI/CD / test (push) Failing after 8s
CI/CD / lint (push) Failing after 23s
CI/CD / build (push) Has been skipped
CI/CD / publish-rpm (push) Has been skipped
CI/CD / publish-deb (push) Has been skipped

The WAN now follows VRRP mastership, so a bad failover takes the house offline
-- and whoever is debugging it is offline too. The recovery path therefore has
to be already on the box, not in a chat log.

`/config/wan-panic` works out which router it is running on and does the right
thing: on vyos002 it stands down (force-fault), which is what lets vyos001 take
the VIPs back; on vyos001 it clears force-fault and says plainly that
no-preempt means you must run it on the OTHER box to actually move anything.
`status` shows who holds the VIP, which WANs are up and the default route.
`undo` stops both timers so nothing moves the WAN again, leaving whatever is
currently up exactly as it is.

It uses the force-fault lever rather than restarting keepalived because failing
the health check is the supported way to shed mastership -- `restart vrrp` is
not dependable, since with advert_int 1 the peer declares the master dead in
~3.6s and the restart usually finishes inside that window.

Proven in production on vyos002, which carries no traffic: BACKUP -> FAULT in
~20s and back to BACKUP on clearing, with vyos001 untouched and the internet
steady at 7.7-8.1ms throughout.

The card is installed to /config/RECOVERY-CARD.md on both routers and copied
to ~/WAN-RECOVERY-CARD.md, because a card you can only read with working
internet is not a card. It leads with how to reach the routers over the LoT
leg, which is L2-direct and survives Management/VRRP/routing being broken.
This commit is contained in:
Michal
2026-09-06 09:29:23 +01:00
parent 8aa3d0ebaa
commit 13dcdff1ef
2 changed files with 208 additions and 0 deletions

View File

@@ -0,0 +1,99 @@
# RECOVERY CARD — internet is down after a WAN failover
No internet means no Claude. Everything here runs from the routers themselves.
**Print this or keep it on a phone.**
## 1. Get to a router
The LoT leg is L2-direct on `bond0.10`. It survives Management, VRRP and
routing being broken:
```
ssh vyos@10.0.1.252 # vyos001 (normally MASTER, has the WAN)
ssh vyos@10.0.1.253 # vyos002 (normally BACKUP, has nothing)
```
Password is the usual one. If SSH is dead, use the JetKVM consoles.
## 2. See what is going on
```
sudo /config/wan-panic status
```
Run it on both. You want exactly ONE box saying `holds VIP : YES`, and that
same box showing a `WAN` line and a `route`.
| what you see | what it means |
|---|---|
| one box `YES` with WAN + route | healthy, look elsewhere for the fault |
| one box `YES`, **no WAN**, no route | the failover half-worked — go to §3 |
| **both** `YES` | VRRP split — go to §3, run it on vyos002 |
| **neither** `YES` | both faulted — go to §4 |
## 3. Give the WAN back to vyos001
**Run this on vyos002 (`10.0.1.253`).** This is the one that matters — vyos002
standing down is what lets vyos001 take over.
```
sudo /config/wan-panic
```
Wait ~30s. Then on vyos001 (`10.0.1.252`):
```
sudo /config/wan-panic status
```
Expect `holds VIP : YES` and a `WAN` line with `bond0.53=…` and/or `pppoe0=…`.
Once the house is back online and you want vyos002 to be a standby again:
```
sudo rm /run/vrrp-wan/force-fault # on vyos002
```
Leave it set if you would rather have no standby than any more surprises —
that is the pre-2026-09-06 arrangement and the house runs fine on it.
## 4. Stop the mechanism touching anything
If the WAN keeps moving, or you do not trust the automation:
```
sudo /config/wan-panic undo # on BOTH routers
```
Stops the reconcile and guard timers. Whatever WAN is up **stays** up. Nothing
will move it again until you re-enable the timers:
```
sudo systemctl enable --now vrrp-wan-reconcile.timer vrrp-wan-guard.timer
```
## 5. Nuclear — put the config back
Only if the config itself is wrong. **This reboots the router.**
```
sudo /config/vyos-known-good restore # on BOTH routers
```
The pinned config is from 2026-09-06, immediately before the PPPoE HA rollout:
1112 lines on vyos001, 1069 on vyos002.
## Why the WAN can only be on one box
One ISP account each way. The 10 gig lease is bound to a cloned MAC
(`f0:9f:c2:12:9b:4f`, the old USG's) and Vodafone is a single-session PPPoE
credential. Two routers holding either at once is worse than one holding
neither — that is why the safe state is "vyos001 has it, vyos002 is inert".
## Known gap
vyos001's `config.boot` does not carry `vif 53 disable`. If vyos001 reboots
**while vyos002 is master**, the cloned MAC is briefly live on both. It clears
itself within 30s (the reconciler disables it on the non-master). If you see
MAC flapping on the WAN switch port right after a vyos001 reboot, that is this,
and it will stop on its own.