diff --git a/migration/PPPOE-HA.md b/migration/PPPOE-HA.md index d3b74a8..e639eaf 100644 --- a/migration/PPPOE-HA.md +++ b/migration/PPPOE-HA.md @@ -111,7 +111,7 @@ sim secondary lost its WAN and spent hours looking like an ISP problem. After any hand commit to the pppoe subtree, `save` — or the next reboot produces a standby that can never take over. -## Deploying (not yet done) +## Deploying — steps 1–7 done 2026-09-06, 8–9 outstanding 1. `sudo /config/vyos-known-good save` on both. 2. `migration/vrrp-wan-install --vip 192.168.1.1 --host vyos@10.0.1.253` then the @@ -125,6 +125,9 @@ standby that can never take over. `sudo tcpdump -i bond0.51 -nn pppoed` should show no PADI. Then `confirm`; `save`. 6. vyos001: nothing to change; it already has `pppoe0` enabled. 7. Confirm `vif 53 disable` is in **both** `config.boot`s. + **It is not.** vyos002 has it; vyos001 does not (its `config.boot` dates from + 2026-09-02 and predates this work). See "Outstanding: vyos001's config.boot" + below — it self-heals, but it should still be fixed. 8. **Only now** merge `migration/pulumi-override-pppoe-gated.json` into `kubernetes-deployment` `infra/vyos/subtrees/overrides.json`. It is staged here, unapplied, on purpose: another agent runs `pulumi up` on that repo, so @@ -145,6 +148,33 @@ own reason text — *"with the primary genuinely dead the secondary stays FAULT and nothing holds the gateway. The fix for that is WAN-follows-master, which is a separate change."* This is that change. +## Outstanding: vyos001's `config.boot` lacks `vif 53 disable` + +The convention is that **both** `config.boot`s hold `vif 53 disable`, so a +reboot in any order comes up unable to claim the cloned MAC and the reconciler +then enables it on whichever box holds the VIP. vyos002 satisfies this. +vyos001 does not — its `config.boot` is from 2026-09-02 and predates this work, +so this is pre-existing rather than introduced here. + +What it now exposes: if vyos001 reboots *while vyos002 is master and holding the +10 gig*, vyos001 comes up with `bond0.53` enabled and the same cloned MAC +`f0:9f:c2:12:9b:4f` is briefly live on both boxes. This mattered less before +today, because vyos002 was stuck in FAULT and could never be master. It can be +now. + +It **self-heals within 30s**: vyos001 comes up BACKUP (priority 200 but +`no-preempt`), the reconciler sees "not master but `bond0.53` enabled", and +commits `disable`. So the exposure is a ≤30s duplicate MAC on the WAN segment, +not a permanent split. + +Fixing it properly is not free, which is why it is listed rather than done: +`config.boot` can only be written by `save`, and `save` writes the *running* +state — so you must `set … vif 53 disable`, `commit` (which **takes the 10 gig +down**, dropping the default route onto `pppoe0`), `save`, then `delete … disable` +and `commit` **without** saving. That is a brief, deliberate WAN interruption on +the live master and belongs in a maintenance window. Hand-editing `config.boot` +avoids the blip but risks an unbootable router, which is a worse trade. + **Rollback**, from either box: `set interfaces pppoe pppoe0 disable` on both and `rm /run/vrrp-wan/may-dial`. That restores today's behaviour exactly.