From 13dcdff1efa57f49c7c9344ff6c6376d1450fee3 Mon Sep 17 00:00:00 2001 From: Michal Date: Sun, 6 Sep 2026 09:29:23 +0100 Subject: [PATCH] wan-panic: a one-command revert that works with no internet and no Claude The WAN now follows VRRP mastership, so a bad failover takes the house offline -- and whoever is debugging it is offline too. The recovery path therefore has to be already on the box, not in a chat log. `/config/wan-panic` works out which router it is running on and does the right thing: on vyos002 it stands down (force-fault), which is what lets vyos001 take the VIPs back; on vyos001 it clears force-fault and says plainly that no-preempt means you must run it on the OTHER box to actually move anything. `status` shows who holds the VIP, which WANs are up and the default route. `undo` stops both timers so nothing moves the WAN again, leaving whatever is currently up exactly as it is. It uses the force-fault lever rather than restarting keepalived because failing the health check is the supported way to shed mastership -- `restart vrrp` is not dependable, since with advert_int 1 the peer declares the master dead in ~3.6s and the restart usually finishes inside that window. Proven in production on vyos002, which carries no traffic: BACKUP -> FAULT in ~20s and back to BACKUP on clearing, with vyos001 untouched and the internet steady at 7.7-8.1ms throughout. The card is installed to /config/RECOVERY-CARD.md on both routers and copied to ~/WAN-RECOVERY-CARD.md, because a card you can only read with working internet is not a card. It leads with how to reach the routers over the LoT leg, which is L2-direct and survives Management/VRRP/routing being broken. --- migration/RECOVERY-CARD-wan-panic.md | 99 ++++++++++++++++++++++++ migration/wan-panic | 109 +++++++++++++++++++++++++++ 2 files changed, 208 insertions(+) create mode 100644 migration/RECOVERY-CARD-wan-panic.md create mode 100644 migration/wan-panic diff --git a/migration/RECOVERY-CARD-wan-panic.md b/migration/RECOVERY-CARD-wan-panic.md new file mode 100644 index 0000000..7d16ff7 --- /dev/null +++ b/migration/RECOVERY-CARD-wan-panic.md @@ -0,0 +1,99 @@ +# RECOVERY CARD — internet is down after a WAN failover + +No internet means no Claude. Everything here runs from the routers themselves. +**Print this or keep it on a phone.** + +## 1. Get to a router + +The LoT leg is L2-direct on `bond0.10`. It survives Management, VRRP and +routing being broken: + +``` +ssh vyos@10.0.1.252 # vyos001 (normally MASTER, has the WAN) +ssh vyos@10.0.1.253 # vyos002 (normally BACKUP, has nothing) +``` + +Password is the usual one. If SSH is dead, use the JetKVM consoles. + +## 2. See what is going on + +``` +sudo /config/wan-panic status +``` + +Run it on both. You want exactly ONE box saying `holds VIP : YES`, and that +same box showing a `WAN` line and a `route`. + +| what you see | what it means | +|---|---| +| one box `YES` with WAN + route | healthy, look elsewhere for the fault | +| one box `YES`, **no WAN**, no route | the failover half-worked — go to §3 | +| **both** `YES` | VRRP split — go to §3, run it on vyos002 | +| **neither** `YES` | both faulted — go to §4 | + +## 3. Give the WAN back to vyos001 + +**Run this on vyos002 (`10.0.1.253`).** This is the one that matters — vyos002 +standing down is what lets vyos001 take over. + +``` +sudo /config/wan-panic +``` + +Wait ~30s. Then on vyos001 (`10.0.1.252`): + +``` +sudo /config/wan-panic status +``` + +Expect `holds VIP : YES` and a `WAN` line with `bond0.53=…` and/or `pppoe0=…`. + +Once the house is back online and you want vyos002 to be a standby again: + +``` +sudo rm /run/vrrp-wan/force-fault # on vyos002 +``` + +Leave it set if you would rather have no standby than any more surprises — +that is the pre-2026-09-06 arrangement and the house runs fine on it. + +## 4. Stop the mechanism touching anything + +If the WAN keeps moving, or you do not trust the automation: + +``` +sudo /config/wan-panic undo # on BOTH routers +``` + +Stops the reconcile and guard timers. Whatever WAN is up **stays** up. Nothing +will move it again until you re-enable the timers: + +``` +sudo systemctl enable --now vrrp-wan-reconcile.timer vrrp-wan-guard.timer +``` + +## 5. Nuclear — put the config back + +Only if the config itself is wrong. **This reboots the router.** + +``` +sudo /config/vyos-known-good restore # on BOTH routers +``` + +The pinned config is from 2026-09-06, immediately before the PPPoE HA rollout: +1112 lines on vyos001, 1069 on vyos002. + +## Why the WAN can only be on one box + +One ISP account each way. The 10 gig lease is bound to a cloned MAC +(`f0:9f:c2:12:9b:4f`, the old USG's) and Vodafone is a single-session PPPoE +credential. Two routers holding either at once is worse than one holding +neither — that is why the safe state is "vyos001 has it, vyos002 is inert". + +## Known gap + +vyos001's `config.boot` does not carry `vif 53 disable`. If vyos001 reboots +**while vyos002 is master**, the cloned MAC is briefly live on both. It clears +itself within 30s (the reconciler disables it on the non-master). If you see +MAC flapping on the WAN switch port right after a vyos001 reboot, that is this, +and it will stop on its own. diff --git a/migration/wan-panic b/migration/wan-panic new file mode 100644 index 0000000..5122f9a --- /dev/null +++ b/migration/wan-panic @@ -0,0 +1,109 @@ +#!/bin/sh +# wan-panic -- get the internet back. No Claude, no internet, no thinking. +# +# Run it on EITHER router. It works out which box it is and does the right +# thing. Safe to run twice, safe to run on both, safe to run when nothing is +# wrong. +# +# sudo /config/wan-panic hand the WAN back to vyos001 +# sudo /config/wan-panic status show who has what, change nothing +# sudo /config/wan-panic undo also stop the whole vrrp-wan mechanism +# +# WHY THIS EXISTS: the WAN now follows VRRP mastership. If a failover leaves +# the wrong box holding the VIPs, or the new master cannot raise a WAN, the +# house has no internet -- and whoever is debugging it has no internet either. +# So the recovery path must be a single command that is already on the box. +# +# HOW TO REACH THE ROUTERS WITH THE NETWORK BROKEN: ssh the LoT leg, +# ssh vyos@10.0.1.252 (vyos001) +# ssh vyos@10.0.1.253 (vyos002) +# It is L2-direct on bond0.10 and survives Management, VRRP and routing being +# broken. Console via the JetKVMs is the fallback. + +STATE=/run/vrrp-wan +HOST="$(cat /etc/hostname 2>/dev/null || hostname)" +VIP=192.168.1.1 +[ -r /config/vrrp-wan.conf ] && . /config/vrrp-wan.conf +VIP="${VRRP_WAN_VIP:-$VIP}" + +have_vip() { ip -4 -o addr show 2>/dev/null | grep -q " ${VIP}/"; } +wan_list() { + for i in bond0.53 pppoe0; do + a=$(ip -4 addr show dev "$i" 2>/dev/null | sed -n 's/.*inet \([0-9.]*\).*/\1/p') + [ -n "$a" ] && printf '%s=%s ' "$i" "$a" + done +} + +show() { + printf ' host : %s\n' "$HOST" + printf ' holds VIP : %s\n' "$(have_vip && echo YES || echo no)" + printf ' WAN : %s\n' "$(wan_list)" + printf ' route : %s\n' "$(ip route show default 2>/dev/null | head -1)" + printf ' force-fault: %s\n' "$([ -f "$STATE/force-fault" ] && echo SET || echo clear)" +} + +case "${1:-failback}" in +status) + show + exit 0 + ;; + +undo) + # Full stop. Leaves whatever WAN is currently up exactly as it is and + # stops anything from moving it again. Use when the mechanism itself is + # suspect. `vyos-known-good restore` is the heavier hammer below. + systemctl disable --now vrrp-wan-reconcile.timer vrrp-wan-guard.timer 2>/dev/null + rm -f "$STATE/force-fault" + echo " vrrp-wan timers stopped. Nothing will move the WAN now." + echo " The 10 gig / PPPoE stay exactly as they are this second." + echo + echo " If the config itself is wrong, the heavier hammer is:" + echo " sudo /config/vyos-known-good restore" + echo " on BOTH routers (it reboots them onto the pinned config)." + echo + show + exit 0 + ;; + +failback) + # The common case: put vyos001 back in charge. + # + # It is done with the force-fault lever, not by restarting keepalived, + # because failing the health check is the SUPPORTED way to shed + # mastership -- the sync group goes FAULT, releases every VIP, and the + # peer takes over by the same path a genuine WAN loss uses. `restart + # vrrp` is not dependable: with advert_int 1 the peer declares the master + # dead in ~3.6s and a restart usually finishes inside that window. + case "$HOST" in + *002|*2) + # This is the secondary. Stand down so vyos001 can have it back. + mkdir -p "$STATE" 2>/dev/null + touch "$STATE/force-fault" + echo " $HOST: standing down (force-fault SET)." + echo " vyos001 should take the VIPs and raise the WAN within ~30s." + echo + echo " When the dust settles and you WANT this box eligible again:" + echo " sudo rm /run/vrrp-wan/force-fault" + ;; + *) + # This is the primary. Make sure nothing is holding it back. + rm -f "$STATE/force-fault" + echo " $HOST: cleared force-fault -- eligible for MASTER." + echo " NOTE: VRRP is no-preempt, so if the peer currently holds the" + echo " VIPs it KEEPS them. To actually take them back, run this on" + echo " the OTHER box (vyos002 / 10.0.1.253):" + echo " sudo /config/wan-panic" + ;; + esac + # Reconcile now rather than waiting up to 30s for the timer. + [ -x /config/vrrp-wan-reconcile ] && /config/vrrp-wan-reconcile 9>&- 2>/dev/null + echo + show + exit 0 + ;; + +*) + echo "usage: wan-panic [failback|status|undo]" >&2 + exit 2 + ;; +esac