Close the last two items: config.boot pinned, and an upgrade runbook
vif53-pin-boot-disable ran on vyos001. Both config.boots now pin
`vif 53 disable` while vyos001's running config keeps the WAN, so a reboot in
any order comes up unable to claim the cloned MAC. Cost: a 32s window (10s in
the sim -- production commits under kea/BGP/conntrack are slower), bond0.53
re-leased 87.192.101.48 in 5s, and vyos-failover restored the primary route
about a minute later. The house rode pppoe0 in between rather than losing the
internet.
Recorded the shape of that recovery, because I misread it myself: for ~60s
after the bounce the default route really is on pppoe0, since vyos-failover
only re-adds the bond0.53 route once its probes pass. A fresh `ip route show`
in that window looks like a regression and is not one.
VYOS-IMAGE-UPGRADE.md is step 9. An image upgrade keeps /config and REPLACES
/etc, which silently removes the ppp@pppoe0 gate drop-in -- the only thing
stopping the backup from dialling into a single-session account. The reconciler
fails closed, so the symptom is "PPPoE never comes up" rather than "both
routers dialled", but an upgraded box has no PPPoE until the gate is back.
One router at a time, backup first.
Deleted the "model hazard to fix before applying" section rather than leaving
it: it advised asserting `vif 53 disable` on BOTH routers via an override,
which is exactly wrong. That is runtime state owned by vrrp-wan-reconcile, and
pinning it would fight the reconciler on every apply and briefly disable the
live master's 10 gig each time. Replaced with what is actually done -- follow
reality at runtime, hardcode safe at boot and at install time -- and said
plainly not to reintroduce it.
2026-09-06 10:39:59 +01:00
|
|
|
# Upgrading a VyOS image on the router pair
|
|
|
|
|
|
|
|
|
|
VyOS keeps `/config` across an image upgrade and **replaces `/etc`**. Everything
|
|
|
|
|
in `/config` survives; anything the WAN mechanism put in `/etc` does not.
|
|
|
|
|
|
|
|
|
|
## What an upgrade silently removes
|
|
|
|
|
|
|
|
|
|
`/etc/systemd/system/ppp@pppoe0.service.d/10-vrrp-wan-gate.conf` — the gate.
|
|
|
|
|
|
|
|
|
|
That drop-in is the only thing stopping the **backup** from dialling. With
|
|
|
|
|
`pppoe0` enabled on both routers (which it is, by design), `interfaces_pppoe.py`
|
|
|
|
|
restarts ppp on every commit touching the pppoe subtree when the daemon is not
|
|
|
|
|
running. Without the gate, the backup dials on the next `pulumi up`, hand
|
|
|
|
|
commit, or boot-time config load — and Vodafone is a single-session account, so
|
|
|
|
|
it takes the session off the live master.
|
|
|
|
|
|
|
|
|
|
`vrrp-wan-reconcile` fails **closed** here: it refuses to dial at all when the
|
|
|
|
|
drop-in is missing, and logs
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
REFUSING to dial: gate drop-in ... is missing (VyOS upgrade?)
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
so the symptom is "PPPoE never comes up", not "both routers dialled". That is
|
|
|
|
|
the safe direction, but it does mean an upgraded router has no PPPoE until the
|
|
|
|
|
gate is reinstalled.
|
|
|
|
|
|
|
|
|
|
The systemd **units and timers** also live in `/etc` and go the same way.
|
|
|
|
|
|
|
|
|
|
## Do this, one router at a time
|
|
|
|
|
|
|
|
|
|
Never both at once — the surviving router must be able to hold the VIPs.
|
|
|
|
|
|
|
|
|
|
1. **Upgrade the BACKUP first.** Confirm which it is:
|
|
|
|
|
```
|
|
|
|
|
sudo /config/wan-panic status # "holds VIP : no"
|
|
|
|
|
```
|
|
|
|
|
2. Install the image and reboot as normal.
|
|
|
|
|
3. **Reinstall the mechanism** from a checkout of `lab`:
|
|
|
|
|
```
|
|
|
|
|
migration/vrrp-wan-install --vip 192.168.1.1 --host vyos@<router>
|
|
|
|
|
migration/vrrp-wan-install --check --host vyos@<router> # must be clean
|
|
|
|
|
```
|
|
|
|
|
4. **Verify the gate is shut and nothing dialled:**
|
|
|
|
|
```
|
|
|
|
|
systemctl show ppp@pppoe0 -p ConditionResult -p ActiveState -p NRestarts
|
|
|
|
|
```
|
|
|
|
|
Want `ConditionResult=no`, `ActiveState=inactive`, `NRestarts=0`. Check the
|
|
|
|
|
wire too, not just state: `sudo tcpdump -i bond0.51 -nn pppoed` — no PADI.
|
|
|
|
|
5. Confirm it reaches **BACKUP**, not FAULT:
|
|
|
|
|
```
|
|
|
|
|
show vrrp
|
|
|
|
|
```
|
|
|
|
|
FAULT on every group means the health check is failing — most likely
|
|
|
|
|
`/config/vrrp-wan-health` did not get reinstalled, or `vrrp-wan.conf` is
|
|
|
|
|
missing so `GRACE` and the VIP fall back to defaults.
|
IPv6 was never HA, and the WAN becoming HA is what exposed it
Reviewed the parked IPv6 task against the PPPoE-HA work of 2026-09-05/06. The
gate that parked it ("WI-8 before IPv6") is cleared, but the same work
invalidated the assumption the IPv6 design rested on.
Verified on the live routers: vyos002 has no tun0, no he-tunnel-follow, no
he-secrets, no VLAN 9 prefix and no route6 ::/0 -- only the pre-staged
default-deny v6 firewall, which is correctly on both. Failover is now automatic
and drill-proven, so every failover takes the whole v6 estate down for as long
as vyos002 holds the VIP.
Four things that came out of checking rather than reading:
- PPPOE-HA.md's "tun0 survived untouched and IPv6 stayed up at 15.5ms" does not
follow from its own premise and is corrected in place. The endpoint address is
stable, but it MOVES to vyos002, which has nothing to decapsulate protocol 41.
wan-drill had no IPv6 check at all, which is why nobody caught it.
- A 22-second near-miss: vif53-pin-boot-disable bounced the 10 gig, he-tunnel-
follow ticked once and saw the PPPoE address, and vyos-failover restored the
route 22s before the second tick would have pointed HE at an address Vodafone
reissues on every dial.
- VyOS does NOT leave a tunnel down when its source-address is absent (the
override's stated reason for leaving IPv6 single-homed). Measured in labsim:
it commits rc=0 and brings the link UP -- a blackhole that attracts the v6
default route. The runtime gate is load-bearing, like the PPPoE gate.
- The RA link-mtu was pinned at 1480 while the tunnel correctly drops to 1472 on
the PPPoE path.
Mechanism, mirroring PPPoE HA -- identical config on both, gated at runtime, no
commit in the failover path:
- vrrp-wan-reconcile: a v6 kernel plane. tun0 and radvd follow the VIP; radvd is
stopped BEFORE the WAN goes so its farewell RA (router-lifetime 0) still has a
path out. The WAN early-exits became if-blocks so the plane runs every tick.
It deliberately does NOT call he-tunnel-follow: that would halve the
hysteresis the near-miss above showed we depend on.
- he-tunnel-follow: a master guard reading the same vrrp-wan.conf VIP, so the
backup copy cannot point HE at its own idle PPPoE line, plus a stubbable
HE_UPDATE_URL.
- vrrp-wan-install carries both, so --check and the upgrade runbook cover IPv6.
- wan-drill measures IPv6 in both timing loops and asserts zero HE API calls
across a router failover.
labsim finally has an HE endpoint, closing the gap the override itself cited as
why this was never rehearsed. Both ISP islands already share the libvirt network,
so that becomes the backbone and HE lives behind it on one address reachable over
either WAN. Proven in the sim: backup tun=DOWN radvd=inactive, master tun=UP
radvd=active, hysteresis then HE call then MTU 1480->1472, and VLAN 9 hosts
autoconfiguring from the RA. The end-to-end v6 datapath is NOT yet proven --
inter-island transit crosses libvirt NAT and the return path is lost. Recorded as
a KNOWN SIM GAP rather than papered over.
The model change is staged, not merged: another agent runs pulumi up on that
repo, and the gate must exist on vyos002 before the tunnel does.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-09-06 14:30:44 +01:00
|
|
|
5b. **Check IPv6 came back with it.** `vrrp-wan-install` now carries
|
|
|
|
|
`he-tunnel-follow`, so `--check` covers it, but `/config/he-secrets` is a
|
|
|
|
|
secret placed by Pulumi and is only checked for *presence*:
|
|
|
|
|
```
|
|
|
|
|
sudo /config/he-tunnel-follow status # role, tunnel src, MTU
|
|
|
|
|
```
|
|
|
|
|
Want the box's own role, and — on the master — a tunnel source equal to the
|
|
|
|
|
**10 gig** address with MTU 1480. `/config` survives an upgrade, so the
|
|
|
|
|
`system task-scheduler` entry that runs this every minute survives too; it is
|
|
|
|
|
the units and the ppp gate in `/etc` that do not.
|
Close the last two items: config.boot pinned, and an upgrade runbook
vif53-pin-boot-disable ran on vyos001. Both config.boots now pin
`vif 53 disable` while vyos001's running config keeps the WAN, so a reboot in
any order comes up unable to claim the cloned MAC. Cost: a 32s window (10s in
the sim -- production commits under kea/BGP/conntrack are slower), bond0.53
re-leased 87.192.101.48 in 5s, and vyos-failover restored the primary route
about a minute later. The house rode pppoe0 in between rather than losing the
internet.
Recorded the shape of that recovery, because I misread it myself: for ~60s
after the bounce the default route really is on pppoe0, since vyos-failover
only re-adds the bond0.53 route once its probes pass. A fresh `ip route show`
in that window looks like a regression and is not one.
VYOS-IMAGE-UPGRADE.md is step 9. An image upgrade keeps /config and REPLACES
/etc, which silently removes the ppp@pppoe0 gate drop-in -- the only thing
stopping the backup from dialling into a single-session account. The reconciler
fails closed, so the symptom is "PPPoE never comes up" rather than "both
routers dialled", but an upgraded box has no PPPoE until the gate is back.
One router at a time, backup first.
Deleted the "model hazard to fix before applying" section rather than leaving
it: it advised asserting `vif 53 disable` on BOTH routers via an override,
which is exactly wrong. That is runtime state owned by vrrp-wan-reconcile, and
pinning it would fight the reconciler on every apply and briefly disable the
live master's 10 gig each time. Replaced with what is actually done -- follow
reality at runtime, hardcode safe at boot and at install time -- and said
plainly not to reintroduce it.
2026-09-06 10:39:59 +01:00
|
|
|
6. Confirm `config.boot` still pins the safe resting state:
|
|
|
|
|
```
|
|
|
|
|
sudo /config/vif53-pin-boot-disable --check # config.boot disable : 1
|
|
|
|
|
```
|
|
|
|
|
7. Only once the upgraded box is a healthy BACKUP, fail over and repeat for the
|
|
|
|
|
other router. `migration/wan-drill` does that unattended, or by hand:
|
|
|
|
|
`sudo /config/wan-panic` on the box you want to give up mastership.
|
|
|
|
|
|
|
|
|
|
## After both are done
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
migration/vrrp-wan-install --check --host vyos@10.0.1.252
|
|
|
|
|
migration/vrrp-wan-install --check --host vyos@10.0.1.253
|
|
|
|
|
cd kubernetes-deployment && npm run vyos:export -- --router vyos001=10.0.1.252 --router vyos002=10.0.1.253
|
|
|
|
|
npm run vyos:render && npm run vyos:verify -- --router vyos001=10.0.1.252 --router vyos002=10.0.1.253
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Both should read "in sync". If the export shows `interfaces pppoe pppoe0
|
|
|
|
|
disable` coming back, something reinstated it — the
|
|
|
|
|
`pppoe-gated-not-config-disabled` override exists to prevent exactly that, so
|
|
|
|
|
check it is still in `infra/vyos/subtrees/overrides.json`.
|
|
|
|
|
|
|
|
|
|
## If it goes wrong
|
|
|
|
|
|
|
|
|
|
`migration/RECOVERY-CARD-wan-panic.md`, or `/config/RECOVERY-CARD.md` on either
|
|
|
|
|
router. Short version, on vyos002: `sudo /config/wan-panic`.
|