vif53-pin-boot-disable ran on vyos001. Both config.boots now pin
`vif 53 disable` while vyos001's running config keeps the WAN, so a reboot in
any order comes up unable to claim the cloned MAC. Cost: a 32s window (10s in
the sim -- production commits under kea/BGP/conntrack are slower), bond0.53
re-leased 87.192.101.48 in 5s, and vyos-failover restored the primary route
about a minute later. The house rode pppoe0 in between rather than losing the
internet.
Recorded the shape of that recovery, because I misread it myself: for ~60s
after the bounce the default route really is on pppoe0, since vyos-failover
only re-adds the bond0.53 route once its probes pass. A fresh `ip route show`
in that window looks like a regression and is not one.
VYOS-IMAGE-UPGRADE.md is step 9. An image upgrade keeps /config and REPLACES
/etc, which silently removes the ppp@pppoe0 gate drop-in -- the only thing
stopping the backup from dialling into a single-session account. The reconciler
fails closed, so the symptom is "PPPoE never comes up" rather than "both
routers dialled", but an upgraded box has no PPPoE until the gate is back.
One router at a time, backup first.
Deleted the "model hazard to fix before applying" section rather than leaving
it: it advised asserting `vif 53 disable` on BOTH routers via an override,
which is exactly wrong. That is runtime state owned by vrrp-wan-reconcile, and
pinning it would fight the reconciler on every apply and briefly disable the
live master's 10 gig each time. Replaced with what is actually done -- follow
reality at runtime, hardcode safe at boot and at install time -- and said
plainly not to reintroduce it.
Both overrides are on kubernetes-deployment main (45033dd) and the
transition-scripts were applied to the boxes directly rather than left for an
unattended `pulumi up` to find. vyos:verify reports both routers in sync.
Also records the drill result in the header, since "deployed" and "proven to
fail over" are different claims and only the second one is worth much.
Controlled failover in production: takeover 52s, failback 36s, ~88s of
interruption across two deliberate transitions.
The cloned-MAC lease TRANSFERS. That was named in this file as "the largest
untested item in any failover" -- whether the 10 gig ISP would re-issue
87.192.101.48 to f0:9f:c2:12:9b:4f arriving on a different switch port. It did,
same address, inside the takeover window. Moved out of "what the sim cannot
prove" rather than left there contradicting the evidence.
VyOS dialling Vodafone also worked from both routers, which had never been done
-- PPPoE was only ever proven on the USG. Vodafone did not refuse either
re-dial, so its session-control behaves like `replace`, not the hostile `deny`.
GRACE stays at 300 anyway: one drill on one evening is not the ISP's policy
under all conditions.
Worth knowing for anything added later: Vodafone hands out a DIFFERENT IPv4 on
every dial (83.106.5.72 -> 90.251.153.180 -> 90.251.142.103). Nothing may be
pinned to the pppoe0 address. Checked the HE IPv6 tunnel specifically, since it
carries a hardcoded source-address -- it is pinned to 87.192.101.48, which is
the 10 gig and stable across failover, so tun0 was untouched and IPv6 stayed up
at 15.5ms.
Runbook step 7 says confirm it on both; it holds on vyos002 and not on
vyos001, whose config.boot dates from 2026-09-02 and predates this work.
The exposure is new even though the condition is not: if vyos001 reboots while
vyos002 is master holding the 10 gig, the cloned MAC is briefly live on both.
That could not happen yesterday, because vyos002 was stuck in FAULT and could
never be master.
It self-heals in <=30s -- vyos001 comes up BACKUP under no-preempt and the
reconciler commits `disable` on the next tick -- so this is a bounded window,
not a split. Left as an outstanding item rather than fixed, because writing
config.boot means `save`ing the running state, so the fix takes the live 10 gig
down briefly and belongs in a maintenance window. Hand-editing config.boot
would avoid the blip at the risk of an unbootable router, which is worse.
Mechanism installed on both routers (--check clean against git on both),
`interfaces pppoe pppoe0 disable` removed from vyos002, and vyos002 is out of
FAULT and holding BACKUP on all six groups for the first time in 3d10h -- the
lab has a real standby again.
Verified on the wire rather than from state: with a tcpdump running on
vyos002's bond0.51 across the commit, ZERO PADI/PADR. The peers file rendered
(so it can dial the instant it is promoted) while ConditionResult stayed `no`
and NRestarts 0. vyos001's live session was untouched throughout -- same
MainPID 2931, same 83.106.5.72, internet 7.6-8.4ms at 0% loss.
Not done, deliberately: the controlled failover drill, which interrupts the
household's internet, and merging the staged Pulumi override, which lands via
another agent's `pulumi up` on a branch they have checked out.
Final confirming run with GRACE=300 and every fix installed:
--all ALL PASS (T0 T3 T5 T8 T11 T12)
--hard ALL PASS (replace / deny / disable, each policy verified)
Second independent measurement of the hard failover, which is what makes the
figures trustworthy rather than anecdotal:
run 1 run 2
replace 26s 26s
deny 148s 141s
disable 21s 20s
`deny` sits at ~141-148s across both, so it is the AC's dead-peer behaviour
and not a one-off; GRACE=300 keeps roughly 2x margin.
Corrected an overclaim in PPPOE-HA.md while confirming it: the invariant row
read "AC never showed two simdsl sessions", and under session-control=disable
it did -- one live, one orphaned from the destroyed router. Only ever one LIVE
router dialled, which is the invariant that matters. Said so plainly rather
than leaving a table that reads better than the evidence.
With the matrix actually setting session-control, T4 timed a destroyed
master's takeover at:
replace 26s
deny 148s <-- sizing case
disable 21s
`deny` is the case GRACE exists for: the AC refuses the survivor until its own
dead-peer timer frees the dead session. The session poller caught it happening
-- the destroyed router's session stayed in the table while the survivor's
dials appeared and were rejected, twice, before one took at 148s.
148s is well past the lcp-echo-interval(30) x failure(3) = 90s budget that 180
was sized against, leaving 32s of margin. GRACE=300 is ~2x the worst observed.
Treat 148s as a floor, not a worst case: these are idle 2-vCPU VMs, and the AC
shares an OVS bridge with the routers, so `virsh destroy` removes the port and
accel-ppp sees the peer vanish. A real BRAS over DSL never learns our router
died and waits out longer timers of its own.
The cost is stated in the conf: GRACE is also how long an alive-but-unroutable
master holds every VIP before yielding. bond0.53 covers most of that -- a DHCP
lease satisfies the check in seconds -- so GRACE only dominates when PPPoE is
the last path. Kept the health check's fallback in step, since keepalived runs
it with no environment and that number decides mastership if the conf is ever
missing.
check_invariant no longer fails blind on the AC's session count. That count is
only a proxy for "two of our routers dialled", and only while the AC enforces
single-session; under `disable` it does not, so a destroyed router's session
lingers and the count reads 2 with exactly one live router dialled. The real
invariant -- at most one router holds pppoe0 -- is now the failing one, and the
stale session is reported as a WARN rather than silenced, because it still
occupies the slot at a real ISP and is precisely what made `deny` take 148s.
Both were invisible to inspection and only appeared under the hard matrix:
the flap damper tearing down an established session via the lease it stopped
renewing, and a missing peers file being completely silent. The second has a
production edge worth spelling out -- an unsaved commit reverts on reboot and
takes pppoe0 with it, leaving a standby that can never take over while
looking perfectly healthy.
Hard failover proven by destroying the master outright (virsh destroy -- no PADT,
the case a graceful stop cannot cover): the survivor took the VIPs, dialled, and
the access concentrator showed exactly ONE simdsl session from its MAC for the
whole seven minutes. When the destroyed router came back it did NOT dial --
ConditionResult=no, ActiveState=inactive, NRestarts=0, pppoe0 absent, may_dial=no
-- and the AC session count stayed at 1. That is the gate doing the one job it
exists for, on the path that previously had no protection at all.
Two more harness bugs of the same family as the last three, both of which
reported a working system as broken:
- ppp_on() returned an EMPTY string for an unreachable router, and `[ "" = 0 ]`
is false, so the hard-failover wait sat for its full timeout waiting for a
DESTROYED box to report zero -- long after the survivor had taken over
correctly. Absent now means 0.
- a VM restart recreates its taps under new names and the OVS bond keeps the
old ones: lacp dies, VLAN 1 goes with it, and the box returns reachable on
some VLANs and not others. That looked exactly like a failed failover. It is
the same stale-membership fault ovs_bond_router already detects, but nothing
ran it after a restart; the harness now does.
migration/PPPOE-HA.md is the runbook: what the design is, why `disable` cannot
work, the measured numbers, the deploy order (vyos002 first, on its own commit,
verified on the wire with tcpdump rather than from state), and the one-line
rollback.
It also records a model hazard found while writing it. The imported baseline
captures the RUNNING state, not the safe one -- vyos001 has no `vif 53 disable`
because it happens to be master, vyos002 does. An apply performed while vyos002
held the VIP would therefore enable vyos001's WAN too, putting the cloned MAC on
both boxes. An override must assert `vif 53 disable` on BOTH, accepting that an
apply then briefly disables the current master's 10 gig until the reconciler
restores it.
Still not applied to production.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
