Close the last two items: config.boot pinned, and an upgrade runbook
Some checks failed
Some checks failed
vif53-pin-boot-disable ran on vyos001. Both config.boots now pin `vif 53 disable` while vyos001's running config keeps the WAN, so a reboot in any order comes up unable to claim the cloned MAC. Cost: a 32s window (10s in the sim -- production commits under kea/BGP/conntrack are slower), bond0.53 re-leased 87.192.101.48 in 5s, and vyos-failover restored the primary route about a minute later. The house rode pppoe0 in between rather than losing the internet. Recorded the shape of that recovery, because I misread it myself: for ~60s after the bounce the default route really is on pppoe0, since vyos-failover only re-adds the bond0.53 route once its probes pass. A fresh `ip route show` in that window looks like a regression and is not one. VYOS-IMAGE-UPGRADE.md is step 9. An image upgrade keeps /config and REPLACES /etc, which silently removes the ppp@pppoe0 gate drop-in -- the only thing stopping the backup from dialling into a single-session account. The reconciler fails closed, so the symptom is "PPPoE never comes up" rather than "both routers dialled", but an upgraded box has no PPPoE until the gate is back. One router at a time, backup first. Deleted the "model hazard to fix before applying" section rather than leaving it: it advised asserting `vif 53 disable` on BOTH routers via an override, which is exactly wrong. That is runtime state owned by vrrp-wan-reconcile, and pinning it would fight the reconciler on every apply and briefly disable the live master's 10 gig each time. Replaced with what is actually done -- follow reality at runtime, hardcode safe at boot and at install time -- and said plainly not to reintroduce it.
This commit is contained in:
Michal
@@ -112,7 +112,7 @@ sim secondary lost its WAN and spent hours looking like an ISP problem. After
|
||||
any hand commit to the pppoe subtree, `save` — or the next reboot produces a
|
||||
standby that can never take over.
|
||||
|
||||
## Deploying — steps 1–8 done 2026-09-06, 9 outstanding
|
||||
## Deploying — steps 1–9 done 2026-09-06
|
||||
|
||||
1. `sudo /config/vyos-known-good save` on both.
|
||||
2. `migration/vrrp-wan-install --vip 192.168.1.1 --host vyos@10.0.1.253` then the
|
||||
@@ -125,10 +125,8 @@ standby that can never take over.
|
||||
5. Verify vyos002 did **not** dial — check on the wire, not from state:
|
||||
`sudo tcpdump -i bond0.51 -nn pppoed` should show no PADI. Then `confirm`; `save`.
|
||||
6. vyos001: nothing to change; it already has `pppoe0` enabled.
|
||||
7. Confirm `vif 53 disable` is in **both** `config.boot`s.
|
||||
**It is not.** vyos002 has it; vyos001 does not (its `config.boot` dates from
|
||||
2026-09-02 and predates this work). See "Outstanding: vyos001's config.boot"
|
||||
below — it self-heals, but it should still be fixed.
|
||||
7. Confirm `vif 53 disable` is in **both** `config.boot`s. vyos001's lacked it;
|
||||
fixed with `migration/vif53-pin-boot-disable` — see below.
|
||||
8. **Done 2026-09-06** (`kubernetes-deployment@45033dd`, on `main`). Both
|
||||
overrides merged, transition-scripts applied to both boxes by hand rather
|
||||
than left as drift, and `vyos:verify` is clean: 533 / 512 nodes, zero drift.
|
||||
@@ -140,6 +138,9 @@ standby that can never take over.
|
||||
vyos001. Run `npm run vyos:export && npm run vyos:render` first so the model
|
||||
follows whichever router actually holds the WAN.
|
||||
9. Add the drop-in re-install to the VyOS image-upgrade runbook.
|
||||
**Done** — `migration/VYOS-IMAGE-UPGRADE.md`. An upgrade replaces `/etc` and
|
||||
so removes the gate; the reconciler fails closed, giving "PPPoE never
|
||||
dials" rather than "both routers dial".
|
||||
|
||||
Step 4 is the one that matters most and is worth stopping on. vyos002 has been
|
||||
in **FAULT on all six groups for over three days** — verified again while
|
||||
@@ -187,32 +188,47 @@ Production takeover (52s) is about twice the sim's `replace` figure (26s), which
|
||||
is the expected direction: the VP2440s commit under kea, BGP and conntrack while
|
||||
the sim routers are idle.
|
||||
|
||||
## Outstanding: vyos001's `config.boot` lacks `vif 53 disable`
|
||||
## config.boot pins `vif 53 disable` on both — fixed 2026-09-06
|
||||
|
||||
The convention is that **both** `config.boot`s hold `vif 53 disable`, so a
|
||||
reboot in any order comes up unable to claim the cloned MAC and the reconciler
|
||||
then enables it on whichever box holds the VIP. vyos002 satisfies this.
|
||||
vyos001 does not — its `config.boot` is from 2026-09-02 and predates this work,
|
||||
so this is pre-existing rather than introduced here.
|
||||
The convention is that **both** `config.boot`s hold `vif 53 disable`, so a reboot
|
||||
in any order comes up unable to claim the cloned MAC and the reconciler enables
|
||||
it on whichever box holds the VIP. vyos001's did not; its `config.boot` predated
|
||||
this work.
|
||||
|
||||
What it now exposes: if vyos001 reboots *while vyos002 is master and holding the
|
||||
10 gig*, vyos001 comes up with `bond0.53` enabled and the same cloned MAC
|
||||
`f0:9f:c2:12:9b:4f` is briefly live on both boxes. This mattered less before
|
||||
today, because vyos002 was stuck in FAULT and could never be master. It can be
|
||||
now.
|
||||
There is no clean way to express "boot disabled, run enabled" in VyOS: **`save`
|
||||
writes the RUNNING config, not the candidate.** Setting the node, saving and
|
||||
discarding was tested in labsim and `config.boot` came back *without* `disable`,
|
||||
the WAN untouched. So the node must genuinely be disabled, saved, and
|
||||
re-enabled. `migration/vif53-pin-boot-disable` does exactly that, is idempotent,
|
||||
and no-ops on a box that already has it.
|
||||
|
||||
It **self-heals within 30s**: vyos001 comes up BACKUP (priority 200 but
|
||||
`no-preempt`), the reconciler sees "not master but `bond0.53` enabled", and
|
||||
commits `disable`. So the exposure is a ≤30s duplicate MAC on the WAN segment,
|
||||
not a permanent split.
|
||||
Cost, measured on vyos001: a **32s** window (10s in the sim — production commits
|
||||
under kea/BGP/conntrack are slower), `bond0.53` re-leased `87.192.101.48` 5s
|
||||
after re-enable, and `vyos-failover` restored the primary route about a minute
|
||||
later:
|
||||
|
||||
Fixing it properly is not free, which is why it is listed rather than done:
|
||||
`config.boot` can only be written by `save`, and `save` writes the *running*
|
||||
state — so you must `set … vif 53 disable`, `commit` (which **takes the 10 gig
|
||||
down**, dropping the default route onto `pppoe0`), `save`, then `delete … disable`
|
||||
and `commit` **without** saving. That is a brief, deliberate WAN interruption on
|
||||
the live master and belongs in a maintenance window. Hand-editing `config.boot`
|
||||
avoids the blip but risks an unbootable router, which is a worse trade.
|
||||
```
|
||||
09:32:23 ip route del 0.0.0.0/0 ... dev bond0.53
|
||||
09:32:32 Check fail for route 0.0.0.0/0 interface "bond0.53"
|
||||
09:33:23 ip route add 0.0.0.0/0 via 87.192.96.1 dev bond0.53 metric 1 proto failover
|
||||
```
|
||||
|
||||
The house rode `pppoe0` for that minute rather than losing the internet, which
|
||||
is the T5 path working. Note the shape of that recovery before reading a fresh
|
||||
`ip route show` as a regression: for ~60s after the bounce the default really is
|
||||
on `pppoe0`, because `vyos-failover` only re-adds the `bond0.53` route once its
|
||||
probes pass again.
|
||||
|
||||
**A race worth knowing about.** The first sim run collided with
|
||||
`vrrp-wan-reconcile`'s own commit — *"Configuration system temporarily locked due
|
||||
to another commit in progress"* — and the `save` landed while the **re-enable did
|
||||
not**, leaving the master with its 10 gig down. The script now takes the
|
||||
reconciler's `/run/vrrp-wan.lock` (which the reconciler skips a tick rather than
|
||||
block on), with `9>&-` so the config session's unionfs child cannot inherit it.
|
||||
Even the bad run ended correctly — the reconciler logged *"MASTER with bond0.53
|
||||
disabled -> enabling"* and repaired it in 4s — so a half-completed run is
|
||||
survivable by design. The script no longer leans on that, and verifies the
|
||||
re-enable rather than reporting a success it did not achieve.
|
||||
|
||||
**Rollback**, from either box: `set interfaces pppoe pppoe0 disable` on both and
|
||||
`rm /run/vrrp-wan/may-dial`. That restores today's behaviour exactly.
|
||||
@@ -259,12 +275,27 @@ avoids the blip but risks an unbootable router, which is a worse trade.
|
||||
the VP2440s under kea + BGP + conntrack will be worse, and commit latency is
|
||||
the dominant term in the `bond0.53` half of a failover.
|
||||
|
||||
## A model hazard to fix before applying
|
||||
## How the model handles the asymmetry — resolved
|
||||
|
||||
The imported baseline records the **running** state, not the safe one: vyos001
|
||||
has no `vif 53 disable` (it is master), vyos002 does. So an apply performed while
|
||||
vyos002 held the VIP would enable vyos001's WAN as well, putting the cloned MAC
|
||||
on both boxes. An override must assert `vif 53 disable` on **both** routers, with
|
||||
the reconciler re-enabling whichever holds the VIP. Note the consequence: an
|
||||
apply then briefly disables the current master's 10 gig until the reconciler
|
||||
restores it (≤30s).
|
||||
An earlier draft of this file said an override "must assert `vif 53 disable` on
|
||||
**both** routers". **That advice was wrong and has been removed**; do not
|
||||
reintroduce it. `vif 53 disable` is *runtime* state owned by
|
||||
`vrrp-wan-reconcile`, keyed on who holds the management VIP, so pinning it in
|
||||
the model would fight the reconciler on every apply and would briefly disable
|
||||
the live master's 10 gig each time.
|
||||
|
||||
What is actually done, and why it is safe:
|
||||
|
||||
- **Runtime (Pulumi): follow reality.** Run `npm run vyos:export && npm run
|
||||
vyos:render` immediately before any `pulumi up` touching vyos. Whichever
|
||||
router currently holds the WAN keeps it; the apply is a no-op on that node.
|
||||
There is deliberately **no** override for `vif 53 disable`.
|
||||
- **Boot (`config.boot`): hardcode safe.** Both routers pin `vif 53 disable`,
|
||||
so a reboot in any order comes up unable to claim the cloned MAC and the
|
||||
reconciler enables it on whoever holds the VIP. See the section above.
|
||||
- **Install time (PXE, nothing to follow).** `migration/vyos-mode-delta.py`
|
||||
emits `vif 53 disable` for a box with no WAN, and deliberately does *not*
|
||||
emit `pppoe pppoe0 disable`.
|
||||
|
||||
Verified 2026-09-06: `vyos:verify` reports both routers in sync, 533 and 512
|
||||
nodes, zero drift.
|
||||
|
||||
Reference in New Issue
Block a user