PPPOE-HA: record that vyos001's config.boot lacks vif 53 disable
Some checks failed
CI/CD / lint (push) Failing after 8s
CI/CD / test (push) Failing after 8s
CI/CD / typecheck (push) Failing after 22s
CI/CD / build (push) Has been skipped
CI/CD / publish-rpm (push) Has been skipped
CI/CD / publish-deb (push) Has been skipped

Runbook step 7 says confirm it on both; it holds on vyos002 and not on
vyos001, whose config.boot dates from 2026-09-02 and predates this work.

The exposure is new even though the condition is not: if vyos001 reboots while
vyos002 is master holding the 10 gig, the cloned MAC is briefly live on both.
That could not happen yesterday, because vyos002 was stuck in FAULT and could
never be master.

It self-heals in <=30s -- vyos001 comes up BACKUP under no-preempt and the
reconciler commits `disable` on the next tick -- so this is a bounded window,
not a split. Left as an outstanding item rather than fixed, because writing
config.boot means `save`ing the running state, so the fix takes the live 10 gig
down briefly and belongs in a maintenance window. Hand-editing config.boot
would avoid the blip at the risk of an unbootable router, which is worse.
This commit is contained in:
Michal
2026-09-06 01:27:07 +01:00
parent 8fce03e705
commit 8aa3d0ebaa

View File

@@ -111,7 +111,7 @@ sim secondary lost its WAN and spent hours looking like an ISP problem. After
any hand commit to the pppoe subtree, `save` — or the next reboot produces a
standby that can never take over.
## Deploying (not yet done)
## Deploying — steps 17 done 2026-09-06, 89 outstanding
1. `sudo /config/vyos-known-good save` on both.
2. `migration/vrrp-wan-install --vip 192.168.1.1 --host vyos@10.0.1.253` then the
@@ -125,6 +125,9 @@ standby that can never take over.
`sudo tcpdump -i bond0.51 -nn pppoed` should show no PADI. Then `confirm`; `save`.
6. vyos001: nothing to change; it already has `pppoe0` enabled.
7. Confirm `vif 53 disable` is in **both** `config.boot`s.
**It is not.** vyos002 has it; vyos001 does not (its `config.boot` dates from
2026-09-02 and predates this work). See "Outstanding: vyos001's config.boot"
below — it self-heals, but it should still be fixed.
8. **Only now** merge `migration/pulumi-override-pppoe-gated.json` into
`kubernetes-deployment` `infra/vyos/subtrees/overrides.json`. It is staged
here, unapplied, on purpose: another agent runs `pulumi up` on that repo, so
@@ -145,6 +148,33 @@ own reason text — *"with the primary genuinely dead the secondary stays FAULT
and nothing holds the gateway. The fix for that is WAN-follows-master, which is
a separate change."* This is that change.
## Outstanding: vyos001's `config.boot` lacks `vif 53 disable`
The convention is that **both** `config.boot`s hold `vif 53 disable`, so a
reboot in any order comes up unable to claim the cloned MAC and the reconciler
then enables it on whichever box holds the VIP. vyos002 satisfies this.
vyos001 does not — its `config.boot` is from 2026-09-02 and predates this work,
so this is pre-existing rather than introduced here.
What it now exposes: if vyos001 reboots *while vyos002 is master and holding the
10 gig*, vyos001 comes up with `bond0.53` enabled and the same cloned MAC
`f0:9f:c2:12:9b:4f` is briefly live on both boxes. This mattered less before
today, because vyos002 was stuck in FAULT and could never be master. It can be
now.
It **self-heals within 30s**: vyos001 comes up BACKUP (priority 200 but
`no-preempt`), the reconciler sees "not master but `bond0.53` enabled", and
commits `disable`. So the exposure is a ≤30s duplicate MAC on the WAN segment,
not a permanent split.
Fixing it properly is not free, which is why it is listed rather than done:
`config.boot` can only be written by `save`, and `save` writes the *running*
state — so you must `set … vif 53 disable`, `commit` (which **takes the 10 gig
down**, dropping the default route onto `pppoe0`), `save`, then `delete … disable`
and `commit` **without** saving. That is a brief, deliberate WAN interruption on
the live master and belongs in a maintenance window. Hand-editing `config.boot`
avoids the blip but risks an unbootable router, which is a worse trade.
**Rollback**, from either box: `set interfaces pppoe pppoe0 disable` on both and
`rm /run/vrrp-wan/may-dial`. That restores today's behaviour exactly.