michal/lab
1
0
Fork 0
Code Issues Pull Requests 3 Actions Packages Projects Releases Wiki Activity
Files
061b9e3d7e1a36278434e11bbb16b8cd44c3be2e
lab/migration/VYOS-IMAGE-UPGRADE.md
Michal 061b9e3d7e
Some checks failed
CI/CD / lint (push) Failing after 8s
Details
CI/CD / test (push) Failing after 9s
Details
CI/CD / typecheck (push) Failing after 24s
Details
CI/CD / build (push) Has been skipped
Details
CI/CD / publish-rpm (push) Has been skipped
Details
CI/CD / publish-deb (push) Has been skipped
Details
Close the last two items: config.boot pinned, and an upgrade runbook 
vif53-pin-boot-disable ran on vyos001. Both config.boots now pin
`vif 53 disable` while vyos001's running config keeps the WAN, so a reboot in
any order comes up unable to claim the cloned MAC. Cost: a 32s window (10s in
the sim -- production commits under kea/BGP/conntrack are slower), bond0.53
re-leased 87.192.101.48 in 5s, and vyos-failover restored the primary route
about a minute later. The house rode pppoe0 in between rather than losing the
internet.

Recorded the shape of that recovery, because I misread it myself: for ~60s
after the bounce the default route really is on pppoe0, since vyos-failover
only re-adds the bond0.53 route once its probes pass. A fresh `ip route show`
in that window looks like a regression and is not one.

VYOS-IMAGE-UPGRADE.md is step 9. An image upgrade keeps /config and REPLACES
/etc, which silently removes the ppp@pppoe0 gate drop-in -- the only thing
stopping the backup from dialling into a single-session account. The reconciler
fails closed, so the symptom is "PPPoE never comes up" rather than "both
routers dialled", but an upgraded box has no PPPoE until the gate is back.
One router at a time, backup first.

Deleted the "model hazard to fix before applying" section rather than leaving
it: it advised asserting `vif 53 disable` on BOTH routers via an override,
which is exactly wrong. That is runtime state owned by vrrp-wan-reconcile, and
pinning it would fight the reconciler on every apply and briefly disable the
live master's 10 gig each time. Replaced with what is actually done -- follow
reality at runtime, hardcode safe at boot and at install time -- and said
plainly not to reintroduce it.
2026-09-06 10:39:59 +01:00

3.3 KiB
Raw Blame History

Upgrading a VyOS image on the router pair

VyOS keeps /config across an image upgrade and replaces /etc. Everything in /config survives; anything the WAN mechanism put in /etc does not.

What an upgrade silently removes

/etc/systemd/system/ppp@pppoe0.service.d/10-vrrp-wan-gate.conf — the gate.

That drop-in is the only thing stopping the backup from dialling. With pppoe0 enabled on both routers (which it is, by design), interfaces_pppoe.py restarts ppp on every commit touching the pppoe subtree when the daemon is not running. Without the gate, the backup dials on the next pulumi up, hand commit, or boot-time config load — and Vodafone is a single-session account, so it takes the session off the live master.

vrrp-wan-reconcile fails closed here: it refuses to dial at all when the drop-in is missing, and logs

REFUSING to dial: gate drop-in ... is missing (VyOS upgrade?)

so the symptom is "PPPoE never comes up", not "both routers dialled". That is the safe direction, but it does mean an upgraded router has no PPPoE until the gate is reinstalled.

The systemd units and timers also live in /etc and go the same way.

Do this, one router at a time

Never both at once — the surviving router must be able to hold the VIPs.

  1. Upgrade the BACKUP first. Confirm which it is: 
    sudo /config/wan-panic status          # "holds VIP : no"
  2. Install the image and reboot as normal.
  3. Reinstall the mechanism from a checkout of lab: 
    migration/vrrp-wan-install --vip 192.168.1.1 --host vyos@<router>
migration/vrrp-wan-install --check  --host vyos@<router>     # must be clean
  4. Verify the gate is shut and nothing dialled: 
    systemctl show ppp@pppoe0 -p ConditionResult -p ActiveState -p NRestarts
    Want ConditionResult=no, ActiveState=inactive, NRestarts=0. Check the wire too, not just state: sudo tcpdump -i bond0.51 -nn pppoed — no PADI.
  5. Confirm it reaches BACKUP, not FAULT: 
    show vrrp
    FAULT on every group means the health check is failing — most likely /config/vrrp-wan-health did not get reinstalled, or vrrp-wan.conf is missing so GRACE and the VIP fall back to defaults.
  6. Confirm config.boot still pins the safe resting state: 
    sudo /config/vif53-pin-boot-disable --check    # config.boot disable : 1
  7. Only once the upgraded box is a healthy BACKUP, fail over and repeat for the other router. migration/wan-drill does that unattended, or by hand: sudo /config/wan-panic on the box you want to give up mastership.

After both are done

migration/vrrp-wan-install --check --host vyos@10.0.1.252
migration/vrrp-wan-install --check --host vyos@10.0.1.253
cd kubernetes-deployment && npm run vyos:export -- --router vyos001=10.0.1.252 --router vyos002=10.0.1.253
npm run vyos:render && npm run vyos:verify -- --router vyos001=10.0.1.252 --router vyos002=10.0.1.253

Both should read "in sync". If the export shows interfaces pppoe pppoe0 disable coming back, something reinstated it — the pppoe-gated-not-config-disabled override exists to prevent exactly that, so check it is still in infra/vyos/subtrees/overrides.json.

If it goes wrong

migration/RECOVERY-CARD-wan-panic.md, or /config/RECOVERY-CARD.md on either router. Short version, on vyos002: sudo /config/wan-panic.

Reference in New Issue View Git Blame Copy Permalink