wan-panic: a one-command revert that works with no internet and no Claude
Some checks failed
Some checks failed
The WAN now follows VRRP mastership, so a bad failover takes the house offline -- and whoever is debugging it is offline too. The recovery path therefore has to be already on the box, not in a chat log. `/config/wan-panic` works out which router it is running on and does the right thing: on vyos002 it stands down (force-fault), which is what lets vyos001 take the VIPs back; on vyos001 it clears force-fault and says plainly that no-preempt means you must run it on the OTHER box to actually move anything. `status` shows who holds the VIP, which WANs are up and the default route. `undo` stops both timers so nothing moves the WAN again, leaving whatever is currently up exactly as it is. It uses the force-fault lever rather than restarting keepalived because failing the health check is the supported way to shed mastership -- `restart vrrp` is not dependable, since with advert_int 1 the peer declares the master dead in ~3.6s and the restart usually finishes inside that window. Proven in production on vyos002, which carries no traffic: BACKUP -> FAULT in ~20s and back to BACKUP on clearing, with vyos001 untouched and the internet steady at 7.7-8.1ms throughout. The card is installed to /config/RECOVERY-CARD.md on both routers and copied to ~/WAN-RECOVERY-CARD.md, because a card you can only read with working internet is not a card. It leads with how to reach the routers over the LoT leg, which is L2-direct and survives Management/VRRP/routing being broken.
This commit is contained in:
Michal
99
migration/RECOVERY-CARD-wan-panic.md
Normal file
99
migration/RECOVERY-CARD-wan-panic.md
Normal file
@@ -0,0 +1,99 @@
|
||||
# RECOVERY CARD — internet is down after a WAN failover
|
||||
|
||||
No internet means no Claude. Everything here runs from the routers themselves.
|
||||
**Print this or keep it on a phone.**
|
||||
|
||||
## 1. Get to a router
|
||||
|
||||
The LoT leg is L2-direct on `bond0.10`. It survives Management, VRRP and
|
||||
routing being broken:
|
||||
|
||||
```
|
||||
ssh vyos@10.0.1.252 # vyos001 (normally MASTER, has the WAN)
|
||||
ssh vyos@10.0.1.253 # vyos002 (normally BACKUP, has nothing)
|
||||
```
|
||||
|
||||
Password is the usual one. If SSH is dead, use the JetKVM consoles.
|
||||
|
||||
## 2. See what is going on
|
||||
|
||||
```
|
||||
sudo /config/wan-panic status
|
||||
```
|
||||
|
||||
Run it on both. You want exactly ONE box saying `holds VIP : YES`, and that
|
||||
same box showing a `WAN` line and a `route`.
|
||||
|
||||
| what you see | what it means |
|
||||
|---|---|
|
||||
| one box `YES` with WAN + route | healthy, look elsewhere for the fault |
|
||||
| one box `YES`, **no WAN**, no route | the failover half-worked — go to §3 |
|
||||
| **both** `YES` | VRRP split — go to §3, run it on vyos002 |
|
||||
| **neither** `YES` | both faulted — go to §4 |
|
||||
|
||||
## 3. Give the WAN back to vyos001
|
||||
|
||||
**Run this on vyos002 (`10.0.1.253`).** This is the one that matters — vyos002
|
||||
standing down is what lets vyos001 take over.
|
||||
|
||||
```
|
||||
sudo /config/wan-panic
|
||||
```
|
||||
|
||||
Wait ~30s. Then on vyos001 (`10.0.1.252`):
|
||||
|
||||
```
|
||||
sudo /config/wan-panic status
|
||||
```
|
||||
|
||||
Expect `holds VIP : YES` and a `WAN` line with `bond0.53=…` and/or `pppoe0=…`.
|
||||
|
||||
Once the house is back online and you want vyos002 to be a standby again:
|
||||
|
||||
```
|
||||
sudo rm /run/vrrp-wan/force-fault # on vyos002
|
||||
```
|
||||
|
||||
Leave it set if you would rather have no standby than any more surprises —
|
||||
that is the pre-2026-09-06 arrangement and the house runs fine on it.
|
||||
|
||||
## 4. Stop the mechanism touching anything
|
||||
|
||||
If the WAN keeps moving, or you do not trust the automation:
|
||||
|
||||
```
|
||||
sudo /config/wan-panic undo # on BOTH routers
|
||||
```
|
||||
|
||||
Stops the reconcile and guard timers. Whatever WAN is up **stays** up. Nothing
|
||||
will move it again until you re-enable the timers:
|
||||
|
||||
```
|
||||
sudo systemctl enable --now vrrp-wan-reconcile.timer vrrp-wan-guard.timer
|
||||
```
|
||||
|
||||
## 5. Nuclear — put the config back
|
||||
|
||||
Only if the config itself is wrong. **This reboots the router.**
|
||||
|
||||
```
|
||||
sudo /config/vyos-known-good restore # on BOTH routers
|
||||
```
|
||||
|
||||
The pinned config is from 2026-09-06, immediately before the PPPoE HA rollout:
|
||||
1112 lines on vyos001, 1069 on vyos002.
|
||||
|
||||
## Why the WAN can only be on one box
|
||||
|
||||
One ISP account each way. The 10 gig lease is bound to a cloned MAC
|
||||
(`f0:9f:c2:12:9b:4f`, the old USG's) and Vodafone is a single-session PPPoE
|
||||
credential. Two routers holding either at once is worse than one holding
|
||||
neither — that is why the safe state is "vyos001 has it, vyos002 is inert".
|
||||
|
||||
## Known gap
|
||||
|
||||
vyos001's `config.boot` does not carry `vif 53 disable`. If vyos001 reboots
|
||||
**while vyos002 is master**, the cloned MAC is briefly live on both. It clears
|
||||
itself within 30s (the reconciler disables it on the non-master). If you see
|
||||
MAC flapping on the WAN switch port right after a vyos001 reboot, that is this,
|
||||
and it will stop on its own.
|
||||
109
migration/wan-panic
Normal file
109
migration/wan-panic
Normal file
@@ -0,0 +1,109 @@
|
||||
#!/bin/sh
|
||||
# wan-panic -- get the internet back. No Claude, no internet, no thinking.
|
||||
#
|
||||
# Run it on EITHER router. It works out which box it is and does the right
|
||||
# thing. Safe to run twice, safe to run on both, safe to run when nothing is
|
||||
# wrong.
|
||||
#
|
||||
# sudo /config/wan-panic hand the WAN back to vyos001
|
||||
# sudo /config/wan-panic status show who has what, change nothing
|
||||
# sudo /config/wan-panic undo also stop the whole vrrp-wan mechanism
|
||||
#
|
||||
# WHY THIS EXISTS: the WAN now follows VRRP mastership. If a failover leaves
|
||||
# the wrong box holding the VIPs, or the new master cannot raise a WAN, the
|
||||
# house has no internet -- and whoever is debugging it has no internet either.
|
||||
# So the recovery path must be a single command that is already on the box.
|
||||
#
|
||||
# HOW TO REACH THE ROUTERS WITH THE NETWORK BROKEN: ssh the LoT leg,
|
||||
# ssh vyos@10.0.1.252 (vyos001)
|
||||
# ssh vyos@10.0.1.253 (vyos002)
|
||||
# It is L2-direct on bond0.10 and survives Management, VRRP and routing being
|
||||
# broken. Console via the JetKVMs is the fallback.
|
||||
|
||||
STATE=/run/vrrp-wan
|
||||
HOST="$(cat /etc/hostname 2>/dev/null || hostname)"
|
||||
VIP=192.168.1.1
|
||||
[ -r /config/vrrp-wan.conf ] && . /config/vrrp-wan.conf
|
||||
VIP="${VRRP_WAN_VIP:-$VIP}"
|
||||
|
||||
have_vip() { ip -4 -o addr show 2>/dev/null | grep -q " ${VIP}/"; }
|
||||
wan_list() {
|
||||
for i in bond0.53 pppoe0; do
|
||||
a=$(ip -4 addr show dev "$i" 2>/dev/null | sed -n 's/.*inet \([0-9.]*\).*/\1/p')
|
||||
[ -n "$a" ] && printf '%s=%s ' "$i" "$a"
|
||||
done
|
||||
}
|
||||
|
||||
show() {
|
||||
printf ' host : %s\n' "$HOST"
|
||||
printf ' holds VIP : %s\n' "$(have_vip && echo YES || echo no)"
|
||||
printf ' WAN : %s\n' "$(wan_list)"
|
||||
printf ' route : %s\n' "$(ip route show default 2>/dev/null | head -1)"
|
||||
printf ' force-fault: %s\n' "$([ -f "$STATE/force-fault" ] && echo SET || echo clear)"
|
||||
}
|
||||
|
||||
case "${1:-failback}" in
|
||||
status)
|
||||
show
|
||||
exit 0
|
||||
;;
|
||||
|
||||
undo)
|
||||
# Full stop. Leaves whatever WAN is currently up exactly as it is and
|
||||
# stops anything from moving it again. Use when the mechanism itself is
|
||||
# suspect. `vyos-known-good restore` is the heavier hammer below.
|
||||
systemctl disable --now vrrp-wan-reconcile.timer vrrp-wan-guard.timer 2>/dev/null
|
||||
rm -f "$STATE/force-fault"
|
||||
echo " vrrp-wan timers stopped. Nothing will move the WAN now."
|
||||
echo " The 10 gig / PPPoE stay exactly as they are this second."
|
||||
echo
|
||||
echo " If the config itself is wrong, the heavier hammer is:"
|
||||
echo " sudo /config/vyos-known-good restore"
|
||||
echo " on BOTH routers (it reboots them onto the pinned config)."
|
||||
echo
|
||||
show
|
||||
exit 0
|
||||
;;
|
||||
|
||||
failback)
|
||||
# The common case: put vyos001 back in charge.
|
||||
#
|
||||
# It is done with the force-fault lever, not by restarting keepalived,
|
||||
# because failing the health check is the SUPPORTED way to shed
|
||||
# mastership -- the sync group goes FAULT, releases every VIP, and the
|
||||
# peer takes over by the same path a genuine WAN loss uses. `restart
|
||||
# vrrp` is not dependable: with advert_int 1 the peer declares the master
|
||||
# dead in ~3.6s and a restart usually finishes inside that window.
|
||||
case "$HOST" in
|
||||
*002|*2)
|
||||
# This is the secondary. Stand down so vyos001 can have it back.
|
||||
mkdir -p "$STATE" 2>/dev/null
|
||||
touch "$STATE/force-fault"
|
||||
echo " $HOST: standing down (force-fault SET)."
|
||||
echo " vyos001 should take the VIPs and raise the WAN within ~30s."
|
||||
echo
|
||||
echo " When the dust settles and you WANT this box eligible again:"
|
||||
echo " sudo rm /run/vrrp-wan/force-fault"
|
||||
;;
|
||||
*)
|
||||
# This is the primary. Make sure nothing is holding it back.
|
||||
rm -f "$STATE/force-fault"
|
||||
echo " $HOST: cleared force-fault -- eligible for MASTER."
|
||||
echo " NOTE: VRRP is no-preempt, so if the peer currently holds the"
|
||||
echo " VIPs it KEEPS them. To actually take them back, run this on"
|
||||
echo " the OTHER box (vyos002 / 10.0.1.253):"
|
||||
echo " sudo /config/wan-panic"
|
||||
;;
|
||||
esac
|
||||
# Reconcile now rather than waiting up to 30s for the timer.
|
||||
[ -x /config/vrrp-wan-reconcile ] && /config/vrrp-wan-reconcile 9>&- 2>/dev/null
|
||||
echo
|
||||
show
|
||||
exit 0
|
||||
;;
|
||||
|
||||
*)
|
||||
echo "usage: wan-panic [failback|status|undo]" >&2
|
||||
exit 2
|
||||
;;
|
||||
esac
|
||||
Reference in New Issue
Block a user