From 8aa3d0ebaae6469002e6e68ef3a8f117fe171539 Mon Sep 17 00:00:00 2001 From: Michal Date: Sun, 6 Sep 2026 01:27:07 +0100 Subject: [PATCH] PPPOE-HA: record that vyos001's config.boot lacks `vif 53 disable` Runbook step 7 says confirm it on both; it holds on vyos002 and not on vyos001, whose config.boot dates from 2026-09-02 and predates this work. The exposure is new even though the condition is not: if vyos001 reboots while vyos002 is master holding the 10 gig, the cloned MAC is briefly live on both. That could not happen yesterday, because vyos002 was stuck in FAULT and could never be master. It self-heals in <=30s -- vyos001 comes up BACKUP under no-preempt and the reconciler commits `disable` on the next tick -- so this is a bounded window, not a split. Left as an outstanding item rather than fixed, because writing config.boot means `save`ing the running state, so the fix takes the live 10 gig down briefly and belongs in a maintenance window. Hand-editing config.boot would avoid the blip at the risk of an unbootable router, which is worse. --- migration/PPPOE-HA.md | 32 +++++++++++++++++++++++++++++++- 1 file changed, 31 insertions(+), 1 deletion(-) diff --git a/migration/PPPOE-HA.md b/migration/PPPOE-HA.md index d3b74a8..e639eaf 100644 --- a/migration/PPPOE-HA.md +++ b/migration/PPPOE-HA.md @@ -111,7 +111,7 @@ sim secondary lost its WAN and spent hours looking like an ISP problem. After any hand commit to the pppoe subtree, `save` — or the next reboot produces a standby that can never take over. -## Deploying (not yet done) +## Deploying — steps 1–7 done 2026-09-06, 8–9 outstanding 1. `sudo /config/vyos-known-good save` on both. 2. `migration/vrrp-wan-install --vip 192.168.1.1 --host vyos@10.0.1.253` then the @@ -125,6 +125,9 @@ standby that can never take over. `sudo tcpdump -i bond0.51 -nn pppoed` should show no PADI. Then `confirm`; `save`. 6. vyos001: nothing to change; it already has `pppoe0` enabled. 7. Confirm `vif 53 disable` is in **both** `config.boot`s. + **It is not.** vyos002 has it; vyos001 does not (its `config.boot` dates from + 2026-09-02 and predates this work). See "Outstanding: vyos001's config.boot" + below — it self-heals, but it should still be fixed. 8. **Only now** merge `migration/pulumi-override-pppoe-gated.json` into `kubernetes-deployment` `infra/vyos/subtrees/overrides.json`. It is staged here, unapplied, on purpose: another agent runs `pulumi up` on that repo, so @@ -145,6 +148,33 @@ own reason text — *"with the primary genuinely dead the secondary stays FAULT and nothing holds the gateway. The fix for that is WAN-follows-master, which is a separate change."* This is that change. +## Outstanding: vyos001's `config.boot` lacks `vif 53 disable` + +The convention is that **both** `config.boot`s hold `vif 53 disable`, so a +reboot in any order comes up unable to claim the cloned MAC and the reconciler +then enables it on whichever box holds the VIP. vyos002 satisfies this. +vyos001 does not — its `config.boot` is from 2026-09-02 and predates this work, +so this is pre-existing rather than introduced here. + +What it now exposes: if vyos001 reboots *while vyos002 is master and holding the +10 gig*, vyos001 comes up with `bond0.53` enabled and the same cloned MAC +`f0:9f:c2:12:9b:4f` is briefly live on both boxes. This mattered less before +today, because vyos002 was stuck in FAULT and could never be master. It can be +now. + +It **self-heals within 30s**: vyos001 comes up BACKUP (priority 200 but +`no-preempt`), the reconciler sees "not master but `bond0.53` enabled", and +commits `disable`. So the exposure is a ≤30s duplicate MAC on the WAN segment, +not a permanent split. + +Fixing it properly is not free, which is why it is listed rather than done: +`config.boot` can only be written by `save`, and `save` writes the *running* +state — so you must `set … vif 53 disable`, `commit` (which **takes the 10 gig +down**, dropping the default route onto `pppoe0`), `save`, then `delete … disable` +and `commit` **without** saving. That is a brief, deliberate WAN interruption on +the live master and belongs in a maintenance window. Hand-editing `config.boot` +avoids the blip but risks an unbootable router, which is a worse trade. + **Rollback**, from either box: `set interfaces pppoe pppoe0 disable` on both and `rm /run/vrrp-wan/may-dial`. That restores today's behaviour exactly.