diff --git a/migration/PPPOE-HA.md b/migration/PPPOE-HA.md index f586d46..40492e1 100644 --- a/migration/PPPOE-HA.md +++ b/migration/PPPOE-HA.md @@ -112,7 +112,7 @@ sim secondary lost its WAN and spent hours looking like an ISP problem. After any hand commit to the pppoe subtree, `save` — or the next reboot produces a standby that can never take over. -## Deploying — steps 1–8 done 2026-09-06, 9 outstanding +## Deploying — steps 1–9 done 2026-09-06 1. `sudo /config/vyos-known-good save` on both. 2. `migration/vrrp-wan-install --vip 192.168.1.1 --host vyos@10.0.1.253` then the @@ -125,10 +125,8 @@ standby that can never take over. 5. Verify vyos002 did **not** dial — check on the wire, not from state: `sudo tcpdump -i bond0.51 -nn pppoed` should show no PADI. Then `confirm`; `save`. 6. vyos001: nothing to change; it already has `pppoe0` enabled. -7. Confirm `vif 53 disable` is in **both** `config.boot`s. - **It is not.** vyos002 has it; vyos001 does not (its `config.boot` dates from - 2026-09-02 and predates this work). See "Outstanding: vyos001's config.boot" - below — it self-heals, but it should still be fixed. +7. Confirm `vif 53 disable` is in **both** `config.boot`s. vyos001's lacked it; + fixed with `migration/vif53-pin-boot-disable` — see below. 8. **Done 2026-09-06** (`kubernetes-deployment@45033dd`, on `main`). Both overrides merged, transition-scripts applied to both boxes by hand rather than left as drift, and `vyos:verify` is clean: 533 / 512 nodes, zero drift. @@ -140,6 +138,9 @@ standby that can never take over. vyos001. Run `npm run vyos:export && npm run vyos:render` first so the model follows whichever router actually holds the WAN. 9. Add the drop-in re-install to the VyOS image-upgrade runbook. + **Done** — `migration/VYOS-IMAGE-UPGRADE.md`. An upgrade replaces `/etc` and + so removes the gate; the reconciler fails closed, giving "PPPoE never + dials" rather than "both routers dial". Step 4 is the one that matters most and is worth stopping on. vyos002 has been in **FAULT on all six groups for over three days** — verified again while @@ -187,32 +188,47 @@ Production takeover (52s) is about twice the sim's `replace` figure (26s), which is the expected direction: the VP2440s commit under kea, BGP and conntrack while the sim routers are idle. -## Outstanding: vyos001's `config.boot` lacks `vif 53 disable` +## config.boot pins `vif 53 disable` on both — fixed 2026-09-06 -The convention is that **both** `config.boot`s hold `vif 53 disable`, so a -reboot in any order comes up unable to claim the cloned MAC and the reconciler -then enables it on whichever box holds the VIP. vyos002 satisfies this. -vyos001 does not — its `config.boot` is from 2026-09-02 and predates this work, -so this is pre-existing rather than introduced here. +The convention is that **both** `config.boot`s hold `vif 53 disable`, so a reboot +in any order comes up unable to claim the cloned MAC and the reconciler enables +it on whichever box holds the VIP. vyos001's did not; its `config.boot` predated +this work. -What it now exposes: if vyos001 reboots *while vyos002 is master and holding the -10 gig*, vyos001 comes up with `bond0.53` enabled and the same cloned MAC -`f0:9f:c2:12:9b:4f` is briefly live on both boxes. This mattered less before -today, because vyos002 was stuck in FAULT and could never be master. It can be -now. +There is no clean way to express "boot disabled, run enabled" in VyOS: **`save` +writes the RUNNING config, not the candidate.** Setting the node, saving and +discarding was tested in labsim and `config.boot` came back *without* `disable`, +the WAN untouched. So the node must genuinely be disabled, saved, and +re-enabled. `migration/vif53-pin-boot-disable` does exactly that, is idempotent, +and no-ops on a box that already has it. -It **self-heals within 30s**: vyos001 comes up BACKUP (priority 200 but -`no-preempt`), the reconciler sees "not master but `bond0.53` enabled", and -commits `disable`. So the exposure is a ≤30s duplicate MAC on the WAN segment, -not a permanent split. +Cost, measured on vyos001: a **32s** window (10s in the sim — production commits +under kea/BGP/conntrack are slower), `bond0.53` re-leased `87.192.101.48` 5s +after re-enable, and `vyos-failover` restored the primary route about a minute +later: -Fixing it properly is not free, which is why it is listed rather than done: -`config.boot` can only be written by `save`, and `save` writes the *running* -state — so you must `set … vif 53 disable`, `commit` (which **takes the 10 gig -down**, dropping the default route onto `pppoe0`), `save`, then `delete … disable` -and `commit` **without** saving. That is a brief, deliberate WAN interruption on -the live master and belongs in a maintenance window. Hand-editing `config.boot` -avoids the blip but risks an unbootable router, which is a worse trade. +``` +09:32:23 ip route del 0.0.0.0/0 ... dev bond0.53 +09:32:32 Check fail for route 0.0.0.0/0 interface "bond0.53" +09:33:23 ip route add 0.0.0.0/0 via 87.192.96.1 dev bond0.53 metric 1 proto failover +``` + +The house rode `pppoe0` for that minute rather than losing the internet, which +is the T5 path working. Note the shape of that recovery before reading a fresh +`ip route show` as a regression: for ~60s after the bounce the default really is +on `pppoe0`, because `vyos-failover` only re-adds the `bond0.53` route once its +probes pass again. + +**A race worth knowing about.** The first sim run collided with +`vrrp-wan-reconcile`'s own commit — *"Configuration system temporarily locked due +to another commit in progress"* — and the `save` landed while the **re-enable did +not**, leaving the master with its 10 gig down. The script now takes the +reconciler's `/run/vrrp-wan.lock` (which the reconciler skips a tick rather than +block on), with `9>&-` so the config session's unionfs child cannot inherit it. +Even the bad run ended correctly — the reconciler logged *"MASTER with bond0.53 +disabled -> enabling"* and repaired it in 4s — so a half-completed run is +survivable by design. The script no longer leans on that, and verifies the +re-enable rather than reporting a success it did not achieve. **Rollback**, from either box: `set interfaces pppoe pppoe0 disable` on both and `rm /run/vrrp-wan/may-dial`. That restores today's behaviour exactly. @@ -259,12 +275,27 @@ avoids the blip but risks an unbootable router, which is a worse trade. the VP2440s under kea + BGP + conntrack will be worse, and commit latency is the dominant term in the `bond0.53` half of a failover. -## A model hazard to fix before applying +## How the model handles the asymmetry — resolved -The imported baseline records the **running** state, not the safe one: vyos001 -has no `vif 53 disable` (it is master), vyos002 does. So an apply performed while -vyos002 held the VIP would enable vyos001's WAN as well, putting the cloned MAC -on both boxes. An override must assert `vif 53 disable` on **both** routers, with -the reconciler re-enabling whichever holds the VIP. Note the consequence: an -apply then briefly disables the current master's 10 gig until the reconciler -restores it (≤30s). +An earlier draft of this file said an override "must assert `vif 53 disable` on +**both** routers". **That advice was wrong and has been removed**; do not +reintroduce it. `vif 53 disable` is *runtime* state owned by +`vrrp-wan-reconcile`, keyed on who holds the management VIP, so pinning it in +the model would fight the reconciler on every apply and would briefly disable +the live master's 10 gig each time. + +What is actually done, and why it is safe: + +- **Runtime (Pulumi): follow reality.** Run `npm run vyos:export && npm run + vyos:render` immediately before any `pulumi up` touching vyos. Whichever + router currently holds the WAN keeps it; the apply is a no-op on that node. + There is deliberately **no** override for `vif 53 disable`. +- **Boot (`config.boot`): hardcode safe.** Both routers pin `vif 53 disable`, + so a reboot in any order comes up unable to claim the cloned MAC and the + reconciler enables it on whoever holds the VIP. See the section above. +- **Install time (PXE, nothing to follow).** `migration/vyos-mode-delta.py` + emits `vif 53 disable` for a box with no WAN, and deliberately does *not* + emit `pppoe pppoe0 disable`. + +Verified 2026-09-06: `vyos:verify` reports both routers in sync, 533 and 512 +nodes, zero drift. diff --git a/migration/VYOS-IMAGE-UPGRADE.md b/migration/VYOS-IMAGE-UPGRADE.md new file mode 100644 index 0000000..cb3e3c9 --- /dev/null +++ b/migration/VYOS-IMAGE-UPGRADE.md @@ -0,0 +1,82 @@ +# Upgrading a VyOS image on the router pair + +VyOS keeps `/config` across an image upgrade and **replaces `/etc`**. Everything +in `/config` survives; anything the WAN mechanism put in `/etc` does not. + +## What an upgrade silently removes + +`/etc/systemd/system/ppp@pppoe0.service.d/10-vrrp-wan-gate.conf` — the gate. + +That drop-in is the only thing stopping the **backup** from dialling. With +`pppoe0` enabled on both routers (which it is, by design), `interfaces_pppoe.py` +restarts ppp on every commit touching the pppoe subtree when the daemon is not +running. Without the gate, the backup dials on the next `pulumi up`, hand +commit, or boot-time config load — and Vodafone is a single-session account, so +it takes the session off the live master. + +`vrrp-wan-reconcile` fails **closed** here: it refuses to dial at all when the +drop-in is missing, and logs + +``` +REFUSING to dial: gate drop-in ... is missing (VyOS upgrade?) +``` + +so the symptom is "PPPoE never comes up", not "both routers dialled". That is +the safe direction, but it does mean an upgraded router has no PPPoE until the +gate is reinstalled. + +The systemd **units and timers** also live in `/etc` and go the same way. + +## Do this, one router at a time + +Never both at once — the surviving router must be able to hold the VIPs. + +1. **Upgrade the BACKUP first.** Confirm which it is: + ``` + sudo /config/wan-panic status # "holds VIP : no" + ``` +2. Install the image and reboot as normal. +3. **Reinstall the mechanism** from a checkout of `lab`: + ``` + migration/vrrp-wan-install --vip 192.168.1.1 --host vyos@ + migration/vrrp-wan-install --check --host vyos@ # must be clean + ``` +4. **Verify the gate is shut and nothing dialled:** + ``` + systemctl show ppp@pppoe0 -p ConditionResult -p ActiveState -p NRestarts + ``` + Want `ConditionResult=no`, `ActiveState=inactive`, `NRestarts=0`. Check the + wire too, not just state: `sudo tcpdump -i bond0.51 -nn pppoed` — no PADI. +5. Confirm it reaches **BACKUP**, not FAULT: + ``` + show vrrp + ``` + FAULT on every group means the health check is failing — most likely + `/config/vrrp-wan-health` did not get reinstalled, or `vrrp-wan.conf` is + missing so `GRACE` and the VIP fall back to defaults. +6. Confirm `config.boot` still pins the safe resting state: + ``` + sudo /config/vif53-pin-boot-disable --check # config.boot disable : 1 + ``` +7. Only once the upgraded box is a healthy BACKUP, fail over and repeat for the + other router. `migration/wan-drill` does that unattended, or by hand: + `sudo /config/wan-panic` on the box you want to give up mastership. + +## After both are done + +``` +migration/vrrp-wan-install --check --host vyos@10.0.1.252 +migration/vrrp-wan-install --check --host vyos@10.0.1.253 +cd kubernetes-deployment && npm run vyos:export -- --router vyos001=10.0.1.252 --router vyos002=10.0.1.253 +npm run vyos:render && npm run vyos:verify -- --router vyos001=10.0.1.252 --router vyos002=10.0.1.253 +``` + +Both should read "in sync". If the export shows `interfaces pppoe pppoe0 +disable` coming back, something reinstated it — the +`pppoe-gated-not-config-disabled` override exists to prevent exactly that, so +check it is still in `infra/vyos/subtrees/overrides.json`. + +## If it goes wrong + +`migration/RECOVERY-CARD-wan-panic.md`, or `/config/RECOVERY-CARD.md` on either +router. Short version, on vyos002: `sudo /config/wan-panic`.