labsim: hard failover and reboot safety hold; runbook for the production apply
Hard failover proven by destroying the master outright (virsh destroy -- no PADT,
the case a graceful stop cannot cover): the survivor took the VIPs, dialled, and
the access concentrator showed exactly ONE simdsl session from its MAC for the
whole seven minutes. When the destroyed router came back it did NOT dial --
ConditionResult=no, ActiveState=inactive, NRestarts=0, pppoe0 absent, may_dial=no
-- and the AC session count stayed at 1. That is the gate doing the one job it
exists for, on the path that previously had no protection at all.
Two more harness bugs of the same family as the last three, both of which
reported a working system as broken:
- ppp_on() returned an EMPTY string for an unreachable router, and `[ "" = 0 ]`
is false, so the hard-failover wait sat for its full timeout waiting for a
DESTROYED box to report zero -- long after the survivor had taken over
correctly. Absent now means 0.
- a VM restart recreates its taps under new names and the OVS bond keeps the
old ones: lacp dies, VLAN 1 goes with it, and the box returns reachable on
some VLANs and not others. That looked exactly like a failed failover. It is
the same stale-membership fault ovs_bond_router already detects, but nothing
ran it after a restart; the harness now does.
migration/PPPOE-HA.md is the runbook: what the design is, why `disable` cannot
work, the measured numbers, the deploy order (vyos002 first, on its own commit,
verified on the wire with tcpdump rather than from state), and the one-line
rollback.
It also records a model hazard found while writing it. The imported baseline
captures the RUNNING state, not the safe one -- vyos001 has no `vif 53 disable`
because it happens to be master, vyos002 does. An apply performed while vyos002
held the VIP would therefore enable vyos001's WAN too, putting the cloned MAC on
both boxes. An override must assert `vif 53 disable` on BOTH, accepting that an
apply then briefly disables the current master's 10 gig until the reconciler
restores it.
Still not applied to production.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-09-05 19:21:13 +01:00
|
|
|
|
# PPPoE high availability
|
|
|
|
|
|
|
2026-09-06 01:26:38 +01:00
|
|
|
|
Proven in labsim. **Deployed to production 2026-09-06** — mechanism on both
|
|
|
|
|
|
routers, `pppoe0 disable` removed from vyos002, vyos002 out of FAULT and in
|
2026-09-06 09:42:12 +01:00
|
|
|
|
BACKUP, Pulumi model merged, and a controlled failover drill passed
|
|
|
|
|
|
(takeover 52s, failback 36s). `vyos:verify` reports both routers in sync.
|
labsim: hard failover and reboot safety hold; runbook for the production apply
Hard failover proven by destroying the master outright (virsh destroy -- no PADT,
the case a graceful stop cannot cover): the survivor took the VIPs, dialled, and
the access concentrator showed exactly ONE simdsl session from its MAC for the
whole seven minutes. When the destroyed router came back it did NOT dial --
ConditionResult=no, ActiveState=inactive, NRestarts=0, pppoe0 absent, may_dial=no
-- and the AC session count stayed at 1. That is the gate doing the one job it
exists for, on the path that previously had no protection at all.
Two more harness bugs of the same family as the last three, both of which
reported a working system as broken:
- ppp_on() returned an EMPTY string for an unreachable router, and `[ "" = 0 ]`
is false, so the hard-failover wait sat for its full timeout waiting for a
DESTROYED box to report zero -- long after the survivor had taken over
correctly. Absent now means 0.
- a VM restart recreates its taps under new names and the OVS bond keeps the
old ones: lacp dies, VLAN 1 goes with it, and the box returns reachable on
some VLANs and not others. That looked exactly like a failed failover. It is
the same stale-membership fault ovs_bond_router already detects, but nothing
ran it after a restart; the harness now does.
migration/PPPOE-HA.md is the runbook: what the design is, why `disable` cannot
work, the measured numbers, the deploy order (vyos002 first, on its own commit,
verified on the wire with tcpdump rather than from state), and the one-line
rollback.
It also records a model hazard found while writing it. The imported baseline
captures the RUNNING state, not the safe one -- vyos001 has no `vif 53 disable`
because it happens to be master, vyos002 does. An apply performed while vyos002
held the VIP would therefore enable vyos001's WAN too, putting the cloned MAC on
both boxes. An override must assert `vif 53 disable` on BOTH, accepting that an
apply then briefly disables the current master's 10 gig until the reconciler
restores it.
Still not applied to production.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-09-05 19:21:13 +01:00
|
|
|
|
|
|
|
|
|
|
## What it does
|
|
|
|
|
|
|
|
|
|
|
|
One consumer ISP account, two routers. The 10 gig lease is bound to a cloned MAC
|
|
|
|
|
|
(`f0:9f:c2:12:9b:4f`, the retired USG's) and the Vodafone line to a single
|
|
|
|
|
|
credential, so neither may be live on both boxes. The WAN follows VRRP
|
|
|
|
|
|
mastership — but the two halves use different control planes, and that is the
|
|
|
|
|
|
whole design:
|
|
|
|
|
|
|
|
|
|
|
|
| | plane | why |
|
|
|
|
|
|
|---|---|---|
|
|
|
|
|
|
| `bond0.53` (10 gig) | VyOS **config** (`disable`) | only config can move a MAC |
|
|
|
|
|
|
| `pppoe0` (Vodafone) | **systemd** unit gate | see below |
|
|
|
|
|
|
|
|
|
|
|
|
## Why PPPoE cannot live on the config plane
|
|
|
|
|
|
|
|
|
|
|
|
`interfaces_pppoe.py` treats `disable` and `delete` identically: both **unlink
|
|
|
|
|
|
`/etc/ppp/peers/pppoe0`**, call `PPPoEIf.remove()` (withdrawing the FRR default
|
|
|
|
|
|
route) and stop the unit. That path is pppd's own options file
|
|
|
|
|
|
(`ExecStart=/usr/sbin/pppd call %I`), so the resting state destroyed exactly what
|
|
|
|
|
|
the promotion path needed. `ppp@pppoe0` then restart-looped against the missing
|
|
|
|
|
|
file — 47 restarts observed, zero sessions at the access concentrator — and
|
|
|
|
|
|
never tripped systemd's limiter, because `RestartSec=5s` against the default
|
|
|
|
|
|
10s/5-burst window is only two restarts per interval.
|
|
|
|
|
|
|
|
|
|
|
|
It also made op-mode `connect interface pppoe0` unusable (it refuses without the
|
|
|
|
|
|
peers file), and put every failover behind a priority-322 commit where one
|
|
|
|
|
|
unrelated invalid node fails the whole thing.
|
|
|
|
|
|
|
|
|
|
|
|
## The gate
|
|
|
|
|
|
|
|
|
|
|
|
`pppoe0` is configured identically and **enabled on both** routers, so the peers
|
|
|
|
|
|
file always exists. Dialling is gated by
|
|
|
|
|
|
`/etc/systemd/system/ppp@pppoe0.service.d/10-vrrp-wan-gate.conf`:
|
|
|
|
|
|
|
|
|
|
|
|
```ini
|
|
|
|
|
|
ConditionPathExists=/run/vrrp-wan/may-dial
|
|
|
|
|
|
ConditionPathExists=/etc/ppp/peers/pppoe0
|
|
|
|
|
|
StartLimitIntervalSec=600
|
|
|
|
|
|
StartLimitBurst=6
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
`/run` is tmpfs, so the gate is shut at boot and neither box can dial before VRRP
|
|
|
|
|
|
has decided. **This is load-bearing, not a nicety:** with the node enabled,
|
|
|
|
|
|
`interfaces_pppoe.py` restarts ppp on *every* commit touching the pppoe subtree
|
|
|
|
|
|
when the daemon is not running — so the backup actively tries to dial whenever
|
|
|
|
|
|
anything commits (`pulumi up`, a hand commit, the boot-time config load). The
|
|
|
|
|
|
gate is the only thing making that a no-op, which is why `vrrp-wan-reconcile`
|
|
|
|
|
|
**refuses to bless a box whose drop-in is missing**: `/etc` is per-image, so a
|
|
|
|
|
|
VyOS upgrade silently removes the protection, and failing closed turns that into
|
|
|
|
|
|
"PPPoE never dials" rather than "both routers dial".
|
|
|
|
|
|
|
|
|
|
|
|
`may-dial` is a **lease, not a flag**. `ConditionPathExists` is evaluated at
|
|
|
|
|
|
start only — it can prevent a dial, never revoke one. `vrrp-wan-reconcile`
|
|
|
|
|
|
renews it every 30s; `vrrp-wan-guard` runs every 5s and only ever revokes, on
|
|
|
|
|
|
either "I do not hold the VIP" or "the lease is stale".
|
|
|
|
|
|
|
|
|
|
|
|
## Measured in labsim
|
|
|
|
|
|
|
|
|
|
|
|
| | |
|
|
|
|
|
|
|---|---|
|
2026-09-06 00:11:32 +01:00
|
|
|
|
| Clean failover (`force-fault`) | `pppoe0` moves in **20-26s**, reproducible |
|
labsim: hard failover and reboot safety hold; runbook for the production apply
Hard failover proven by destroying the master outright (virsh destroy -- no PADT,
the case a graceful stop cannot cover): the survivor took the VIPs, dialled, and
the access concentrator showed exactly ONE simdsl session from its MAC for the
whole seven minutes. When the destroyed router came back it did NOT dial --
ConditionResult=no, ActiveState=inactive, NRestarts=0, pppoe0 absent, may_dial=no
-- and the AC session count stayed at 1. That is the gate doing the one job it
exists for, on the path that previously had no protection at all.
Two more harness bugs of the same family as the last three, both of which
reported a working system as broken:
- ppp_on() returned an EMPTY string for an unreachable router, and `[ "" = 0 ]`
is false, so the hard-failover wait sat for its full timeout waiting for a
DESTROYED box to report zero -- long after the survivor had taken over
correctly. Absent now means 0.
- a VM restart recreates its taps under new names and the OVS bond keeps the
old ones: lacp dies, VLAN 1 goes with it, and the box returns reachable on
some VLANs and not others. That looked exactly like a failed failover. It is
the same stale-membership fault ovs_bond_router already detects, but nothing
ran it after a restart; the harness now does.
migration/PPPOE-HA.md is the runbook: what the design is, why `disable` cannot
work, the measured numbers, the deploy order (vyos002 first, on its own commit,
verified on the wire with tcpdump rather than from state), and the one-line
rollback.
It also records a model hazard found while writing it. The imported baseline
captures the RUNNING state, not the safe one -- vyos001 has no `vif 53 disable`
because it happens to be master, vyos002 does. An apply performed while vyos002
held the VIP would therefore enable vyos001's WAN too, putting the cloned MAC on
both boxes. An override must assert `vif 53 disable` on BOTH, accepting that an
apply then briefly disables the current master's 10 gig until the reconciler
restores it.
Still not applied to production.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-09-05 19:21:13 +01:00
|
|
|
|
| 10 gig down → PPPoE | route falls to `pppoe0`; LAN back online in **5s** |
|
|
|
|
|
|
| Stale lease | guard hangs up within ~5s |
|
|
|
|
|
|
| Missing peers file | `NRestarts=0` — no loop |
|
2026-09-06 00:11:32 +01:00
|
|
|
|
| Flap holdoff vs live session | session survives a forged 900s holdoff |
|
2026-09-06 00:42:01 +01:00
|
|
|
|
| Invariant | never more than one router dialled, in any run |
|
|
|
|
|
|
|
|
|
|
|
|
On the last row, precisely: the AC *did* report two `simdsl` sessions during the
|
|
|
|
|
|
`session-control=disable` run — one live, one orphaned from the router that had
|
|
|
|
|
|
just been destroyed, which that policy does not clean up. Only one live router
|
|
|
|
|
|
was ever dialled. The count is a proxy for the real invariant and only a valid
|
|
|
|
|
|
one while the AC enforces single-session, so it is reported as a `WARN` rather
|
|
|
|
|
|
than silenced: an orphaned session still occupies the single slot at a real ISP,
|
|
|
|
|
|
and that is exactly what made `deny` take 141–148s.
|
labsim: hard failover and reboot safety hold; runbook for the production apply
Hard failover proven by destroying the master outright (virsh destroy -- no PADT,
the case a graceful stop cannot cover): the survivor took the VIPs, dialled, and
the access concentrator showed exactly ONE simdsl session from its MAC for the
whole seven minutes. When the destroyed router came back it did NOT dial --
ConditionResult=no, ActiveState=inactive, NRestarts=0, pppoe0 absent, may_dial=no
-- and the AC session count stayed at 1. That is the gate doing the one job it
exists for, on the path that previously had no protection at all.
Two more harness bugs of the same family as the last three, both of which
reported a working system as broken:
- ppp_on() returned an EMPTY string for an unreachable router, and `[ "" = 0 ]`
is false, so the hard-failover wait sat for its full timeout waiting for a
DESTROYED box to report zero -- long after the survivor had taken over
correctly. Absent now means 0.
- a VM restart recreates its taps under new names and the OVS bond keeps the
old ones: lacp dies, VLAN 1 goes with it, and the box returns reachable on
some VLANs and not others. That looked exactly like a failed failover. It is
the same stale-membership fault ovs_bond_router already detects, but nothing
ran it after a restart; the harness now does.
migration/PPPOE-HA.md is the runbook: what the design is, why `disable` cannot
work, the measured numbers, the deploy order (vyos002 first, on its own commit,
verified on the wire with tcpdump rather than from state), and the one-line
rollback.
It also records a model hazard found while writing it. The imported baseline
captures the RUNNING state, not the safe one -- vyos001 has no `vif 53 disable`
because it happens to be master, vyos002 does. An apply performed while vyos002
held the VIP would therefore enable vyos001's WAN too, putting the cloned MAC on
both boxes. An override must assert `vif 53 disable` on BOTH, accepting that an
apply then briefly disables the current master's 10 gig until the reconciler
restores it.
Still not applied to production.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-09-05 19:21:13 +01:00
|
|
|
|
|
2026-09-06 00:11:32 +01:00
|
|
|
|
## Two failure modes found by running it, not by reading it
|
|
|
|
|
|
|
|
|
|
|
|
**The flap damper tore down a healthy WAN.** `ppp_dial()` checked the hold-off
|
|
|
|
|
|
and returned *before* renewing `may-dial`. That file is a lease the guard
|
|
|
|
|
|
expires after `LEASE_TTL`, so tripping the damper stopped the renewal and the
|
|
|
|
|
|
guard hung up `pppoe0` **on the master** ~80s later:
|
|
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
|
DIAL FLAP: >=6 attempts in 600s -- holding off 900s
|
|
|
|
|
|
GUARD: lease stale (81s > 75s) -- hanging up pppoe0
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
A damper meant to suppress repeated *dials* was destroying an established
|
|
|
|
|
|
session instead. An active session now renews the lease and returns before
|
|
|
|
|
|
every other check; everything below only decides whether to start a **new**
|
|
|
|
|
|
session. T12 is the regression test.
|
|
|
|
|
|
|
|
|
|
|
|
**A missing peers file is silent.** `/etc/ppp/peers/pppoe0` is both pppd's
|
|
|
|
|
|
options file and the gate's second condition, and it is only written by a commit
|
|
|
|
|
|
that touches the pppoe subtree. Without it systemd logs
|
|
|
|
|
|
`skipped because of an unmet condition check` exactly once and then nothing —
|
|
|
|
|
|
a router that cannot dial at all looks identical to a healthy backup.
|
|
|
|
|
|
`ppp_dial()` now says so on every tick, and distinguishes the two causes:
|
|
|
|
|
|
configured-but-not-rendered (re-commit the subtree) versus no `pppoe0` in the
|
|
|
|
|
|
config at all.
|
|
|
|
|
|
|
|
|
|
|
|
The second cause is the one to watch in production: **a commit that was never
|
|
|
|
|
|
`save`d reverts on reboot and takes `pppoe0` with it.** That is exactly how the
|
|
|
|
|
|
sim secondary lost its WAN and spent hours looking like an ISP problem. After
|
|
|
|
|
|
any hand commit to the pppoe subtree, `save` — or the next reboot produces a
|
|
|
|
|
|
standby that can never take over.
|
|
|
|
|
|
|
2026-09-06 09:42:12 +01:00
|
|
|
|
## Deploying — steps 1–8 done 2026-09-06, 9 outstanding
|
labsim: hard failover and reboot safety hold; runbook for the production apply
Hard failover proven by destroying the master outright (virsh destroy -- no PADT,
the case a graceful stop cannot cover): the survivor took the VIPs, dialled, and
the access concentrator showed exactly ONE simdsl session from its MAC for the
whole seven minutes. When the destroyed router came back it did NOT dial --
ConditionResult=no, ActiveState=inactive, NRestarts=0, pppoe0 absent, may_dial=no
-- and the AC session count stayed at 1. That is the gate doing the one job it
exists for, on the path that previously had no protection at all.
Two more harness bugs of the same family as the last three, both of which
reported a working system as broken:
- ppp_on() returned an EMPTY string for an unreachable router, and `[ "" = 0 ]`
is false, so the hard-failover wait sat for its full timeout waiting for a
DESTROYED box to report zero -- long after the survivor had taken over
correctly. Absent now means 0.
- a VM restart recreates its taps under new names and the OVS bond keeps the
old ones: lacp dies, VLAN 1 goes with it, and the box returns reachable on
some VLANs and not others. That looked exactly like a failed failover. It is
the same stale-membership fault ovs_bond_router already detects, but nothing
ran it after a restart; the harness now does.
migration/PPPOE-HA.md is the runbook: what the design is, why `disable` cannot
work, the measured numbers, the deploy order (vyos002 first, on its own commit,
verified on the wire with tcpdump rather than from state), and the one-line
rollback.
It also records a model hazard found while writing it. The imported baseline
captures the RUNNING state, not the safe one -- vyos001 has no `vif 53 disable`
because it happens to be master, vyos002 does. An apply performed while vyos002
held the VIP would therefore enable vyos001's WAN too, putting the cloned MAC on
both boxes. An override must assert `vif 53 disable` on BOTH, accepting that an
apply then briefly disables the current master's 10 gig until the reconciler
restores it.
Still not applied to production.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-09-05 19:21:13 +01:00
|
|
|
|
|
|
|
|
|
|
1. `sudo /config/vyos-known-good save` on both.
|
|
|
|
|
|
2. `migration/vrrp-wan-install --vip 192.168.1.1 --host vyos@10.0.1.253` then the
|
|
|
|
|
|
same for `.252`. Then `--check` on both. **No config change yet** — verify
|
|
|
|
|
|
nothing dials.
|
|
|
|
|
|
3. Confirm `/config/wan-secrets` is present and identical on both.
|
|
|
|
|
|
4. **vyos002 first** (the non-master), on its own commit — `interfaces pppoe` is
|
|
|
|
|
|
priority 322 and one bad node fails everything:
|
|
|
|
|
|
`delete interfaces pppoe pppoe0 disable`, `commit-confirm 10`.
|
|
|
|
|
|
5. Verify vyos002 did **not** dial — check on the wire, not from state:
|
|
|
|
|
|
`sudo tcpdump -i bond0.51 -nn pppoed` should show no PADI. Then `confirm`; `save`.
|
|
|
|
|
|
6. vyos001: nothing to change; it already has `pppoe0` enabled.
|
|
|
|
|
|
7. Confirm `vif 53 disable` is in **both** `config.boot`s.
|
2026-09-06 01:27:07 +01:00
|
|
|
|
**It is not.** vyos002 has it; vyos001 does not (its `config.boot` dates from
|
|
|
|
|
|
2026-09-02 and predates this work). See "Outstanding: vyos001's config.boot"
|
|
|
|
|
|
below — it self-heals, but it should still be fixed.
|
2026-09-06 09:42:12 +01:00
|
|
|
|
8. **Done 2026-09-06** (`kubernetes-deployment@45033dd`, on `main`). Both
|
|
|
|
|
|
overrides merged, transition-scripts applied to both boxes by hand rather
|
|
|
|
|
|
than left as drift, and `vyos:verify` is clean: 533 / 512 nodes, zero drift.
|
|
|
|
|
|
The staging file `migration/pulumi-override-pppoe-gated.json` is kept as the
|
|
|
|
|
|
record of why the ordering mattered. It was staged, unapplied, on purpose: another agent runs `pulumi up` on that repo, so
|
vrrp-wan: size GRACE from the measured hostile failover, not the theory
With the matrix actually setting session-control, T4 timed a destroyed
master's takeover at:
replace 26s
deny 148s <-- sizing case
disable 21s
`deny` is the case GRACE exists for: the AC refuses the survivor until its own
dead-peer timer frees the dead session. The session poller caught it happening
-- the destroyed router's session stayed in the table while the survivor's
dials appeared and were rejected, twice, before one took at 148s.
148s is well past the lcp-echo-interval(30) x failure(3) = 90s budget that 180
was sized against, leaving 32s of margin. GRACE=300 is ~2x the worst observed.
Treat 148s as a floor, not a worst case: these are idle 2-vCPU VMs, and the AC
shares an OVS bridge with the routers, so `virsh destroy` removes the port and
accel-ppp sees the peer vanish. A real BRAS over DSL never learns our router
died and waits out longer timers of its own.
The cost is stated in the conf: GRACE is also how long an alive-but-unroutable
master holds every VIP before yielding. bond0.53 covers most of that -- a DHCP
lease satisfies the check in seconds -- so GRACE only dominates when PPPoE is
the last path. Kept the health check's fallback in step, since keepalived runs
it with no environment and that number decides mastership if the conf is ever
missing.
check_invariant no longer fails blind on the AC's session count. That count is
only a proxy for "two of our routers dialled", and only while the AC enforces
single-session; under `disable` it does not, so a destroyed router's session
lingers and the count reads 2 with exactly one live router dialled. The real
invariant -- at most one router holds pppoe0 -- is now the failing one, and the
stale session is reported as a WARN rather than silenced, because it still
occupies the slot at a real ISP and is precisely what made `deny` take 148s.
2026-09-06 00:25:00 +01:00
|
|
|
|
merging it *is* a production change made by someone else at a time you do not
|
|
|
|
|
|
choose. Removing `pppoe0 disable` from vyos002 before the gate exists there
|
|
|
|
|
|
lets it dial on the next commit and take the single Vodafone session off
|
|
|
|
|
|
vyos001. Run `npm run vyos:export && npm run vyos:render` first so the model
|
|
|
|
|
|
follows whichever router actually holds the WAN.
|
|
|
|
|
|
9. Add the drop-in re-install to the VyOS image-upgrade runbook.
|
|
|
|
|
|
|
|
|
|
|
|
Step 4 is the one that matters most and is worth stopping on. vyos002 has been
|
|
|
|
|
|
in **FAULT on all six groups for over three days** — verified again while
|
|
|
|
|
|
writing this, alongside vyos001 holding `192.168.1.1` on `bond0.1` and
|
|
|
|
|
|
`pppoe0` up on `83.106.5.72`. Until vyos002 reaches BACKUP there is no standby
|
|
|
|
|
|
at all: if vyos001 died today nothing would pick up the gateway VIPs. The
|
|
|
|
|
|
existing `vrrp-health-check-wan-present` override predicted exactly this in its
|
|
|
|
|
|
own reason text — *"with the primary genuinely dead the secondary stays FAULT
|
|
|
|
|
|
and nothing holds the gateway. The fix for that is WAN-follows-master, which is
|
|
|
|
|
|
a separate change."* This is that change.
|
labsim: hard failover and reboot safety hold; runbook for the production apply
Hard failover proven by destroying the master outright (virsh destroy -- no PADT,
the case a graceful stop cannot cover): the survivor took the VIPs, dialled, and
the access concentrator showed exactly ONE simdsl session from its MAC for the
whole seven minutes. When the destroyed router came back it did NOT dial --
ConditionResult=no, ActiveState=inactive, NRestarts=0, pppoe0 absent, may_dial=no
-- and the AC session count stayed at 1. That is the gate doing the one job it
exists for, on the path that previously had no protection at all.
Two more harness bugs of the same family as the last three, both of which
reported a working system as broken:
- ppp_on() returned an EMPTY string for an unreachable router, and `[ "" = 0 ]`
is false, so the hard-failover wait sat for its full timeout waiting for a
DESTROYED box to report zero -- long after the survivor had taken over
correctly. Absent now means 0.
- a VM restart recreates its taps under new names and the OVS bond keeps the
old ones: lacp dies, VLAN 1 goes with it, and the box returns reachable on
some VLANs and not others. That looked exactly like a failed failover. It is
the same stale-membership fault ovs_bond_router already detects, but nothing
ran it after a restart; the harness now does.
migration/PPPOE-HA.md is the runbook: what the design is, why `disable` cannot
work, the measured numbers, the deploy order (vyos002 first, on its own commit,
verified on the wire with tcpdump rather than from state), and the one-line
rollback.
It also records a model hazard found while writing it. The imported baseline
captures the RUNNING state, not the safe one -- vyos001 has no `vif 53 disable`
because it happens to be master, vyos002 does. An apply performed while vyos002
held the VIP would therefore enable vyos001's WAN too, putting the cloned MAC on
both boxes. An override must assert `vif 53 disable` on BOTH, accepting that an
apply then briefly disables the current master's 10 gig until the reconciler
restores it.
Still not applied to production.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-09-05 19:21:13 +01:00
|
|
|
|
|
PPPOE-HA: the drill closed the two biggest unknowns
Controlled failover in production: takeover 52s, failback 36s, ~88s of
interruption across two deliberate transitions.
The cloned-MAC lease TRANSFERS. That was named in this file as "the largest
untested item in any failover" -- whether the 10 gig ISP would re-issue
87.192.101.48 to f0:9f:c2:12:9b:4f arriving on a different switch port. It did,
same address, inside the takeover window. Moved out of "what the sim cannot
prove" rather than left there contradicting the evidence.
VyOS dialling Vodafone also worked from both routers, which had never been done
-- PPPoE was only ever proven on the USG. Vodafone did not refuse either
re-dial, so its session-control behaves like `replace`, not the hostile `deny`.
GRACE stays at 300 anyway: one drill on one evening is not the ISP's policy
under all conditions.
Worth knowing for anything added later: Vodafone hands out a DIFFERENT IPv4 on
every dial (83.106.5.72 -> 90.251.153.180 -> 90.251.142.103). Nothing may be
pinned to the pppoe0 address. Checked the HE IPv6 tunnel specifically, since it
carries a hardcoded source-address -- it is pinned to 87.192.101.48, which is
the 10 gig and stable across failover, so tun0 was untouched and IPv6 stayed up
at 15.5ms.
2026-09-06 09:38:20 +01:00
|
|
|
|
## Proven in production — controlled drill, 2026-09-06
|
|
|
|
|
|
|
|
|
|
|
|
`migration/wan-drill` force-faulted vyos001 and timed a real failover:
|
|
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
|
TAKEOVER OK: vyos002 held the VIP and reached the internet in 52s
|
|
|
|
|
|
FAILBACK OK: 36s
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
vyos002 took the VIPs at t+18s and had **both** WANs by t+52s. Failback put
|
|
|
|
|
|
vyos001 back with a WAN in 36s. Total interruption ≈ 88s across two deliberate
|
|
|
|
|
|
transitions.
|
|
|
|
|
|
|
|
|
|
|
|
**The cloned-MAC lease transfers.** This was the largest untested item in the
|
|
|
|
|
|
whole design — whether the 10 gig ISP would re-issue `87.192.101.48` to
|
|
|
|
|
|
`f0:9f:c2:12:9b:4f` arriving on a different switch port. It did, same address,
|
|
|
|
|
|
within the takeover window. That risk is now closed.
|
|
|
|
|
|
|
|
|
|
|
|
**Vodafone did not refuse the re-dial**, so its `session-control` behaves like
|
|
|
|
|
|
`replace` rather than the hostile `deny`. `GRACE=300` was sized against the
|
|
|
|
|
|
sim's 148s `deny` case and is therefore comfortable — but it should stay where
|
|
|
|
|
|
it is, because one drill on one evening does not establish the ISP's policy
|
|
|
|
|
|
under all conditions.
|
|
|
|
|
|
|
|
|
|
|
|
**Vodafone hands out a different IPv4 on every dial**: `83.106.5.72` →
|
|
|
|
|
|
`90.251.153.180` (vyos002) → `90.251.142.103` (vyos001, after failback).
|
|
|
|
|
|
Nothing may be pinned to the PPPoE address. Checked: the HE IPv6 tunnel is
|
|
|
|
|
|
pinned to `87.192.101.48`, which is the **10 gig** (`bond0.53`) and stable
|
|
|
|
|
|
across failover, so `tun0` survived untouched and IPv6 stayed up at 15.5ms.
|
|
|
|
|
|
Anything added later that hardcodes a WAN IP must use the 10 gig one, not
|
|
|
|
|
|
`pppoe0`'s.
|
|
|
|
|
|
|
|
|
|
|
|
Production takeover (52s) is about twice the sim's `replace` figure (26s), which
|
|
|
|
|
|
is the expected direction: the VP2440s commit under kea, BGP and conntrack while
|
|
|
|
|
|
the sim routers are idle.
|
|
|
|
|
|
|
2026-09-06 01:27:07 +01:00
|
|
|
|
## Outstanding: vyos001's `config.boot` lacks `vif 53 disable`
|
|
|
|
|
|
|
|
|
|
|
|
The convention is that **both** `config.boot`s hold `vif 53 disable`, so a
|
|
|
|
|
|
reboot in any order comes up unable to claim the cloned MAC and the reconciler
|
|
|
|
|
|
then enables it on whichever box holds the VIP. vyos002 satisfies this.
|
|
|
|
|
|
vyos001 does not — its `config.boot` is from 2026-09-02 and predates this work,
|
|
|
|
|
|
so this is pre-existing rather than introduced here.
|
|
|
|
|
|
|
|
|
|
|
|
What it now exposes: if vyos001 reboots *while vyos002 is master and holding the
|
|
|
|
|
|
10 gig*, vyos001 comes up with `bond0.53` enabled and the same cloned MAC
|
|
|
|
|
|
`f0:9f:c2:12:9b:4f` is briefly live on both boxes. This mattered less before
|
|
|
|
|
|
today, because vyos002 was stuck in FAULT and could never be master. It can be
|
|
|
|
|
|
now.
|
|
|
|
|
|
|
|
|
|
|
|
It **self-heals within 30s**: vyos001 comes up BACKUP (priority 200 but
|
|
|
|
|
|
`no-preempt`), the reconciler sees "not master but `bond0.53` enabled", and
|
|
|
|
|
|
commits `disable`. So the exposure is a ≤30s duplicate MAC on the WAN segment,
|
|
|
|
|
|
not a permanent split.
|
|
|
|
|
|
|
|
|
|
|
|
Fixing it properly is not free, which is why it is listed rather than done:
|
|
|
|
|
|
`config.boot` can only be written by `save`, and `save` writes the *running*
|
|
|
|
|
|
state — so you must `set … vif 53 disable`, `commit` (which **takes the 10 gig
|
|
|
|
|
|
down**, dropping the default route onto `pppoe0`), `save`, then `delete … disable`
|
|
|
|
|
|
and `commit` **without** saving. That is a brief, deliberate WAN interruption on
|
|
|
|
|
|
the live master and belongs in a maintenance window. Hand-editing `config.boot`
|
|
|
|
|
|
avoids the blip but risks an unbootable router, which is a worse trade.
|
|
|
|
|
|
|
labsim: hard failover and reboot safety hold; runbook for the production apply
Hard failover proven by destroying the master outright (virsh destroy -- no PADT,
the case a graceful stop cannot cover): the survivor took the VIPs, dialled, and
the access concentrator showed exactly ONE simdsl session from its MAC for the
whole seven minutes. When the destroyed router came back it did NOT dial --
ConditionResult=no, ActiveState=inactive, NRestarts=0, pppoe0 absent, may_dial=no
-- and the AC session count stayed at 1. That is the gate doing the one job it
exists for, on the path that previously had no protection at all.
Two more harness bugs of the same family as the last three, both of which
reported a working system as broken:
- ppp_on() returned an EMPTY string for an unreachable router, and `[ "" = 0 ]`
is false, so the hard-failover wait sat for its full timeout waiting for a
DESTROYED box to report zero -- long after the survivor had taken over
correctly. Absent now means 0.
- a VM restart recreates its taps under new names and the OVS bond keeps the
old ones: lacp dies, VLAN 1 goes with it, and the box returns reachable on
some VLANs and not others. That looked exactly like a failed failover. It is
the same stale-membership fault ovs_bond_router already detects, but nothing
ran it after a restart; the harness now does.
migration/PPPOE-HA.md is the runbook: what the design is, why `disable` cannot
work, the measured numbers, the deploy order (vyos002 first, on its own commit,
verified on the wire with tcpdump rather than from state), and the one-line
rollback.
It also records a model hazard found while writing it. The imported baseline
captures the RUNNING state, not the safe one -- vyos001 has no `vif 53 disable`
because it happens to be master, vyos002 does. An apply performed while vyos002
held the VIP would therefore enable vyos001's WAN too, putting the cloned MAC on
both boxes. An override must assert `vif 53 disable` on BOTH, accepting that an
apply then briefly disables the current master's 10 gig until the reconciler
restores it.
Still not applied to production.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-09-05 19:21:13 +01:00
|
|
|
|
**Rollback**, from either box: `set interfaces pppoe pppoe0 disable` on both and
|
|
|
|
|
|
`rm /run/vrrp-wan/may-dial`. That restores today's behaviour exactly.
|
|
|
|
|
|
|
|
|
|
|
|
## What the sim cannot prove
|
|
|
|
|
|
|
vrrp-wan: size GRACE from the measured hostile failover, not the theory
With the matrix actually setting session-control, T4 timed a destroyed
master's takeover at:
replace 26s
deny 148s <-- sizing case
disable 21s
`deny` is the case GRACE exists for: the AC refuses the survivor until its own
dead-peer timer frees the dead session. The session poller caught it happening
-- the destroyed router's session stayed in the table while the survivor's
dials appeared and were rejected, twice, before one took at 148s.
148s is well past the lcp-echo-interval(30) x failure(3) = 90s budget that 180
was sized against, leaving 32s of margin. GRACE=300 is ~2x the worst observed.
Treat 148s as a floor, not a worst case: these are idle 2-vCPU VMs, and the AC
shares an OVS bridge with the routers, so `virsh destroy` removes the port and
accel-ppp sees the peer vanish. A real BRAS over DSL never learns our router
died and waits out longer timers of its own.
The cost is stated in the conf: GRACE is also how long an alive-but-unroutable
master holds every VIP before yielding. bond0.53 covers most of that -- a DHCP
lease satisfies the check in seconds -- so GRACE only dominates when PPPoE is
the last path. Kept the health check's fallback in step, since keepalived runs
it with no environment and that number decides mastership if the conf is ever
missing.
check_invariant no longer fails blind on the AC's session count. That count is
only a proxy for "two of our routers dialled", and only while the AC enforces
single-session; under `disable` it does not, so a destroyed router's session
lingers and the count reads 2 with exactly one live router dialled. The real
invariant -- at most one router holds pppoe0 -- is now the failing one, and the
stale session is reported as a WARN rather than silenced, because it still
occupies the slot at a real ISP and is precisely what made `deny` take 148s.
2026-09-06 00:25:00 +01:00
|
|
|
|
- **Vodafone's `session-control`.** The matrix now brackets it properly.
|
|
|
|
|
|
Destroying the master and timing the survivor's session:
|
|
|
|
|
|
|
|
|
|
|
|
| policy | takeover |
|
|
|
|
|
|
|---|---|
|
2026-09-06 00:42:01 +01:00
|
|
|
|
| `replace` (accel-ppp default) | 26s / 26s |
|
|
|
|
|
|
| **`deny`** (hostile) | **148s / 141s** |
|
|
|
|
|
|
| `disable` | 21s / 20s |
|
vrrp-wan: size GRACE from the measured hostile failover, not the theory
With the matrix actually setting session-control, T4 timed a destroyed
master's takeover at:
replace 26s
deny 148s <-- sizing case
disable 21s
`deny` is the case GRACE exists for: the AC refuses the survivor until its own
dead-peer timer frees the dead session. The session poller caught it happening
-- the destroyed router's session stayed in the table while the survivor's
dials appeared and were rejected, twice, before one took at 148s.
148s is well past the lcp-echo-interval(30) x failure(3) = 90s budget that 180
was sized against, leaving 32s of margin. GRACE=300 is ~2x the worst observed.
Treat 148s as a floor, not a worst case: these are idle 2-vCPU VMs, and the AC
shares an OVS bridge with the routers, so `virsh destroy` removes the port and
accel-ppp sees the peer vanish. A real BRAS over DSL never learns our router
died and waits out longer timers of its own.
The cost is stated in the conf: GRACE is also how long an alive-but-unroutable
master holds every VIP before yielding. bond0.53 covers most of that -- a DHCP
lease satisfies the check in seconds -- so GRACE only dominates when PPPoE is
the last path. Kept the health check's fallback in step, since keepalived runs
it with no environment and that number decides mastership if the conf is ever
missing.
check_invariant no longer fails blind on the AC's session count. That count is
only a proxy for "two of our routers dialled", and only while the AC enforces
single-session; under `disable` it does not, so a destroyed router's session
lingers and the count reads 2 with exactly one live router dialled. The real
invariant -- at most one router holds pppoe0 -- is now the failing one, and the
stale session is reported as a WARN rather than silenced, because it still
occupies the slot at a real ISP and is precisely what made `deny` take 148s.
2026-09-06 00:25:00 +01:00
|
|
|
|
|
|
|
|
|
|
`deny` is the sizing case: the AC refuses the survivor until its own
|
|
|
|
|
|
dead-peer timer frees the dead session, and the poller caught two dial
|
|
|
|
|
|
attempts being rejected before one succeeded. `GRACE` is set from that — see
|
|
|
|
|
|
`migration/vrrp-wan.conf`. This still cannot tell you which policy Vodafone
|
|
|
|
|
|
runs, and account rate-limiting or lockout on repeated dials has no sim
|
|
|
|
|
|
analogue at all; the flap damper (6 dials / 600s → 15 min hold-off) exists
|
|
|
|
|
|
for that.
|
|
|
|
|
|
|
|
|
|
|
|
Treat the 148s as a floor rather than a worst case. These are idle 2-vCPU
|
|
|
|
|
|
VMs, and the AC shares an OVS bridge with the routers, so `virsh destroy`
|
|
|
|
|
|
removes the port and accel-ppp sees the peer physically vanish. A real BRAS
|
|
|
|
|
|
reached over DSL does not learn that our router died — it waits out its own
|
|
|
|
|
|
timers, which are longer and not ours to know.
|
|
|
|
|
|
|
|
|
|
|
|
Worth knowing how close this came to being missed: until 2026-09-06 the
|
|
|
|
|
|
matrix set the policy with `vbash -c 'source script-template; configure;
|
|
|
|
|
|
...; commit'`, which never starts a config session. `commit` failed to
|
|
|
|
|
|
stderr, the helper discarded it, and all three iterations ran against the
|
|
|
|
|
|
default while printing the mode they were supposedly testing. It reported
|
|
|
|
|
|
`deny` at 25s. The real figure is 148s.
|
labsim: hard failover and reboot safety hold; runbook for the production apply
Hard failover proven by destroying the master outright (virsh destroy -- no PADT,
the case a graceful stop cannot cover): the survivor took the VIPs, dialled, and
the access concentrator showed exactly ONE simdsl session from its MAC for the
whole seven minutes. When the destroyed router came back it did NOT dial --
ConditionResult=no, ActiveState=inactive, NRestarts=0, pppoe0 absent, may_dial=no
-- and the AC session count stayed at 1. That is the gate doing the one job it
exists for, on the path that previously had no protection at all.
Two more harness bugs of the same family as the last three, both of which
reported a working system as broken:
- ppp_on() returned an EMPTY string for an unreachable router, and `[ "" = 0 ]`
is false, so the hard-failover wait sat for its full timeout waiting for a
DESTROYED box to report zero -- long after the survivor had taken over
correctly. Absent now means 0.
- a VM restart recreates its taps under new names and the OVS bond keeps the
old ones: lacp dies, VLAN 1 goes with it, and the box returns reachable on
some VLANs and not others. That looked exactly like a failed failover. It is
the same stale-membership fault ovs_bond_router already detects, but nothing
ran it after a restart; the harness now does.
migration/PPPOE-HA.md is the runbook: what the design is, why `disable` cannot
work, the measured numbers, the deploy order (vyos002 first, on its own commit,
verified on the wire with tcpdump rather than from state), and the one-line
rollback.
It also records a model hazard found while writing it. The imported baseline
captures the RUNNING state, not the safe one -- vyos001 has no `vif 53 disable`
because it happens to be master, vyos002 does. An apply performed while vyos002
held the VIP would therefore enable vyos001's WAN too, putting the cloned MAC on
both boxes. An override must assert `vif 53 disable` on BOTH, accepting that an
apply then briefly disables the current master's 10 gig until the reconciler
restores it.
Still not applied to production.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-09-05 19:21:13 +01:00
|
|
|
|
- **Whether Vodafone honours our LCP Terminate / PADT** on a graceful stop.
|
PPPOE-HA: the drill closed the two biggest unknowns
Controlled failover in production: takeover 52s, failback 36s, ~88s of
interruption across two deliberate transitions.
The cloned-MAC lease TRANSFERS. That was named in this file as "the largest
untested item in any failover" -- whether the 10 gig ISP would re-issue
87.192.101.48 to f0:9f:c2:12:9b:4f arriving on a different switch port. It did,
same address, inside the takeover window. Moved out of "what the sim cannot
prove" rather than left there contradicting the evidence.
VyOS dialling Vodafone also worked from both routers, which had never been done
-- PPPoE was only ever proven on the USG. Vodafone did not refuse either
re-dial, so its session-control behaves like `replace`, not the hostile `deny`.
GRACE stays at 300 anyway: one drill on one evening is not the ISP's policy
under all conditions.
Worth knowing for anything added later: Vodafone hands out a DIFFERENT IPv4 on
every dial (83.106.5.72 -> 90.251.153.180 -> 90.251.142.103). Nothing may be
pinned to the pppoe0 address. Checked the HE IPv6 tunnel specifically, since it
carries a hardcoded source-address -- it is pinned to 87.192.101.48, which is
the 10 gig and stable across failover, so tun0 was untouched and IPv6 stayed up
at 15.5ms.
2026-09-06 09:38:20 +01:00
|
|
|
|
- ~~**The cloned-MAC lease.**~~ **Answered 2026-09-06**: the drill moved it and
|
|
|
|
|
|
the ISP re-issued `87.192.101.48` to `f0:9f:c2:12:9b:4f` on vyos002's port
|
|
|
|
|
|
within the takeover window. See "Proven in production" above.
|
|
|
|
|
|
- ~~**Whether VyOS can dial Vodafone at all.**~~ **Answered 2026-09-06**: both
|
|
|
|
|
|
routers dialled successfully during the drill. MTU/MSS under sustained load
|
|
|
|
|
|
is still unmeasured, and Vodafone hands out a different IPv4 every dial.
|
labsim: hard failover and reboot safety hold; runbook for the production apply
Hard failover proven by destroying the master outright (virsh destroy -- no PADT,
the case a graceful stop cannot cover): the survivor took the VIPs, dialled, and
the access concentrator showed exactly ONE simdsl session from its MAC for the
whole seven minutes. When the destroyed router came back it did NOT dial --
ConditionResult=no, ActiveState=inactive, NRestarts=0, pppoe0 absent, may_dial=no
-- and the AC session count stayed at 1. That is the gate doing the one job it
exists for, on the path that previously had no protection at all.
Two more harness bugs of the same family as the last three, both of which
reported a working system as broken:
- ppp_on() returned an EMPTY string for an unreachable router, and `[ "" = 0 ]`
is false, so the hard-failover wait sat for its full timeout waiting for a
DESTROYED box to report zero -- long after the survivor had taken over
correctly. Absent now means 0.
- a VM restart recreates its taps under new names and the OVS bond keeps the
old ones: lacp dies, VLAN 1 goes with it, and the box returns reachable on
some VLANs and not others. That looked exactly like a failed failover. It is
the same stale-membership fault ovs_bond_router already detects, but nothing
ran it after a restart; the harness now does.
migration/PPPOE-HA.md is the runbook: what the design is, why `disable` cannot
work, the measured numbers, the deploy order (vyos002 first, on its own commit,
verified on the wire with tcpdump rather than from state), and the one-line
rollback.
It also records a model hazard found while writing it. The imported baseline
captures the RUNNING state, not the safe one -- vyos001 has no `vif 53 disable`
because it happens to be master, vyos002 does. An apply performed while vyos002
held the VIP would therefore enable vyos001's WAN too, putting the cloned MAC on
both boxes. An override must assert `vif 53 disable` on BOTH, accepting that an
apply then briefly disables the current master's 10 gig until the reconciler
restores it.
Still not applied to production.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-09-05 19:21:13 +01:00
|
|
|
|
- **Timing under load.** The sim routers are idle 2-vCPU VMs; commit latency on
|
|
|
|
|
|
the VP2440s under kea + BGP + conntrack will be worse, and commit latency is
|
|
|
|
|
|
the dominant term in the `bond0.53` half of a failover.
|
|
|
|
|
|
|
|
|
|
|
|
## A model hazard to fix before applying
|
|
|
|
|
|
|
|
|
|
|
|
The imported baseline records the **running** state, not the safe one: vyos001
|
|
|
|
|
|
has no `vif 53 disable` (it is master), vyos002 does. So an apply performed while
|
|
|
|
|
|
vyos002 held the VIP would enable vyos001's WAN as well, putting the cloned MAC
|
|
|
|
|
|
on both boxes. An override must assert `vif 53 disable` on **both** routers, with
|
|
|
|
|
|
the reconciler re-enabling whichever holds the VIP. Note the consequence: an
|
|
|
|
|
|
apply then briefly disables the current master's 10 gig until the reconciler
|
|
|
|
|
|
restores it (≤30s).
|