Close the last two items: config.boot pinned, and an upgrade runbook
Some checks failed
Some checks failed
vif53-pin-boot-disable ran on vyos001. Both config.boots now pin `vif 53 disable` while vyos001's running config keeps the WAN, so a reboot in any order comes up unable to claim the cloned MAC. Cost: a 32s window (10s in the sim -- production commits under kea/BGP/conntrack are slower), bond0.53 re-leased 87.192.101.48 in 5s, and vyos-failover restored the primary route about a minute later. The house rode pppoe0 in between rather than losing the internet. Recorded the shape of that recovery, because I misread it myself: for ~60s after the bounce the default route really is on pppoe0, since vyos-failover only re-adds the bond0.53 route once its probes pass. A fresh `ip route show` in that window looks like a regression and is not one. VYOS-IMAGE-UPGRADE.md is step 9. An image upgrade keeps /config and REPLACES /etc, which silently removes the ppp@pppoe0 gate drop-in -- the only thing stopping the backup from dialling into a single-session account. The reconciler fails closed, so the symptom is "PPPoE never comes up" rather than "both routers dialled", but an upgraded box has no PPPoE until the gate is back. One router at a time, backup first. Deleted the "model hazard to fix before applying" section rather than leaving it: it advised asserting `vif 53 disable` on BOTH routers via an override, which is exactly wrong. That is runtime state owned by vrrp-wan-reconcile, and pinning it would fight the reconciler on every apply and briefly disable the live master's 10 gig each time. Replaced with what is actually done -- follow reality at runtime, hardcode safe at boot and at install time -- and said plainly not to reintroduce it.
This commit is contained in:
82
migration/VYOS-IMAGE-UPGRADE.md
Normal file
82
migration/VYOS-IMAGE-UPGRADE.md
Normal file
@@ -0,0 +1,82 @@
|
||||
# Upgrading a VyOS image on the router pair
|
||||
|
||||
VyOS keeps `/config` across an image upgrade and **replaces `/etc`**. Everything
|
||||
in `/config` survives; anything the WAN mechanism put in `/etc` does not.
|
||||
|
||||
## What an upgrade silently removes
|
||||
|
||||
`/etc/systemd/system/ppp@pppoe0.service.d/10-vrrp-wan-gate.conf` — the gate.
|
||||
|
||||
That drop-in is the only thing stopping the **backup** from dialling. With
|
||||
`pppoe0` enabled on both routers (which it is, by design), `interfaces_pppoe.py`
|
||||
restarts ppp on every commit touching the pppoe subtree when the daemon is not
|
||||
running. Without the gate, the backup dials on the next `pulumi up`, hand
|
||||
commit, or boot-time config load — and Vodafone is a single-session account, so
|
||||
it takes the session off the live master.
|
||||
|
||||
`vrrp-wan-reconcile` fails **closed** here: it refuses to dial at all when the
|
||||
drop-in is missing, and logs
|
||||
|
||||
```
|
||||
REFUSING to dial: gate drop-in ... is missing (VyOS upgrade?)
|
||||
```
|
||||
|
||||
so the symptom is "PPPoE never comes up", not "both routers dialled". That is
|
||||
the safe direction, but it does mean an upgraded router has no PPPoE until the
|
||||
gate is reinstalled.
|
||||
|
||||
The systemd **units and timers** also live in `/etc` and go the same way.
|
||||
|
||||
## Do this, one router at a time
|
||||
|
||||
Never both at once — the surviving router must be able to hold the VIPs.
|
||||
|
||||
1. **Upgrade the BACKUP first.** Confirm which it is:
|
||||
```
|
||||
sudo /config/wan-panic status # "holds VIP : no"
|
||||
```
|
||||
2. Install the image and reboot as normal.
|
||||
3. **Reinstall the mechanism** from a checkout of `lab`:
|
||||
```
|
||||
migration/vrrp-wan-install --vip 192.168.1.1 --host vyos@<router>
|
||||
migration/vrrp-wan-install --check --host vyos@<router> # must be clean
|
||||
```
|
||||
4. **Verify the gate is shut and nothing dialled:**
|
||||
```
|
||||
systemctl show ppp@pppoe0 -p ConditionResult -p ActiveState -p NRestarts
|
||||
```
|
||||
Want `ConditionResult=no`, `ActiveState=inactive`, `NRestarts=0`. Check the
|
||||
wire too, not just state: `sudo tcpdump -i bond0.51 -nn pppoed` — no PADI.
|
||||
5. Confirm it reaches **BACKUP**, not FAULT:
|
||||
```
|
||||
show vrrp
|
||||
```
|
||||
FAULT on every group means the health check is failing — most likely
|
||||
`/config/vrrp-wan-health` did not get reinstalled, or `vrrp-wan.conf` is
|
||||
missing so `GRACE` and the VIP fall back to defaults.
|
||||
6. Confirm `config.boot` still pins the safe resting state:
|
||||
```
|
||||
sudo /config/vif53-pin-boot-disable --check # config.boot disable : 1
|
||||
```
|
||||
7. Only once the upgraded box is a healthy BACKUP, fail over and repeat for the
|
||||
other router. `migration/wan-drill` does that unattended, or by hand:
|
||||
`sudo /config/wan-panic` on the box you want to give up mastership.
|
||||
|
||||
## After both are done
|
||||
|
||||
```
|
||||
migration/vrrp-wan-install --check --host vyos@10.0.1.252
|
||||
migration/vrrp-wan-install --check --host vyos@10.0.1.253
|
||||
cd kubernetes-deployment && npm run vyos:export -- --router vyos001=10.0.1.252 --router vyos002=10.0.1.253
|
||||
npm run vyos:render && npm run vyos:verify -- --router vyos001=10.0.1.252 --router vyos002=10.0.1.253
|
||||
```
|
||||
|
||||
Both should read "in sync". If the export shows `interfaces pppoe pppoe0
|
||||
disable` coming back, something reinstated it — the
|
||||
`pppoe-gated-not-config-disabled` override exists to prevent exactly that, so
|
||||
check it is still in `infra/vyos/subtrees/overrides.json`.
|
||||
|
||||
## If it goes wrong
|
||||
|
||||
`migration/RECOVERY-CARD-wan-panic.md`, or `/config/RECOVERY-CARD.md` on either
|
||||
router. Short version, on vyos002: `sudo /config/wan-panic`.
|
||||
Reference in New Issue
Block a user