4d47b609a2fb98f45c9d92034cb23d604dadda70
4 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Michal
|
4d47b609a2 |
PPPOE-HA: record the two failure modes found by running the thing
Some checks failed
Both were invisible to inspection and only appeared under the hard matrix: the flap damper tearing down an established session via the lease it stopped renewing, and a missing peers file being completely silent. The second has a production edge worth spelling out -- an unsaved commit reverts on reboot and takes pppoe0 with it, leaving a standby that can never take over while looking perfectly healthy. |
||
|
Michal
|
b659e0d47e |
vrrp-wan: a flap holdoff must not tear down a live WAN session
Some checks failed
ppp_dial() checked the flap holdoff and returned BEFORE renewing /run/vrrp-wan/may-dial. That lease is what vrrp-wan-guard expires after LEASE_TTL, so tripping the damper stopped the renew and the guard hung up pppoe0 on the MASTER ~80s later. A damper meant to suppress repeated DIALS was tearing down a working WAN instead. Observed in labsim, end to end: DIAL FLAP: >=6 attempts in 600s -- holding off 900s GUARD: lease stale (81s > 75s) -- hanging up pppoe0 An established session now outranks every check below it: ppp_active renews the lease and returns first. Everything after it only decides whether to start a NEW session. Two supporting fixes for how that storm started. The dial attempts were all no-ops because /etc/ppp/peers/pppoe0 was missing, and nothing said so -- systemd logs "skipped because of an unmet condition check" exactly once and the gate looks identical to a healthy backup. ppp_dial() now reports it, and distinguishes "configured but not rendered" (re-commit the subtree) from "no pppoe0 in config at all", which is what a reboot leaves behind when a commit was never saved. That is precisely how the sim secondary lost its WAN. Also `cat | wc -l` rather than `wc -l < file`: redirections are applied left to right, so the missing-file error escapes the 2>/dev/null on every first-ever dial. Harness: T11 copied-then-removed instead of mv, and verifies the restore -- losing that file strands a router permanently, which cost a debugging session. preflight now refuses to run if either router lacks the peers file or the pppoe0 config, since every failover result would otherwise be a false negative blamed on the ISP. New T12 forges a 900s holdoff against a live session and asserts it survives. |
||
|
Michal
|
d626750010 |
labsim: hard failover and reboot safety hold; runbook for the production apply
Some checks failed
Hard failover proven by destroying the master outright (virsh destroy -- no PADT,
the case a graceful stop cannot cover): the survivor took the VIPs, dialled, and
the access concentrator showed exactly ONE simdsl session from its MAC for the
whole seven minutes. When the destroyed router came back it did NOT dial --
ConditionResult=no, ActiveState=inactive, NRestarts=0, pppoe0 absent, may_dial=no
-- and the AC session count stayed at 1. That is the gate doing the one job it
exists for, on the path that previously had no protection at all.
Two more harness bugs of the same family as the last three, both of which
reported a working system as broken:
- ppp_on() returned an EMPTY string for an unreachable router, and `[ "" = 0 ]`
is false, so the hard-failover wait sat for its full timeout waiting for a
DESTROYED box to report zero -- long after the survivor had taken over
correctly. Absent now means 0.
- a VM restart recreates its taps under new names and the OVS bond keeps the
old ones: lacp dies, VLAN 1 goes with it, and the box returns reachable on
some VLANs and not others. That looked exactly like a failed failover. It is
the same stale-membership fault ovs_bond_router already detects, but nothing
ran it after a restart; the harness now does.
migration/PPPOE-HA.md is the runbook: what the design is, why `disable` cannot
work, the measured numbers, the deploy order (vyos002 first, on its own commit,
verified on the wire with tcpdump rather than from state), and the one-line
rollback.
It also records a model hazard found while writing it. The imported baseline
captures the RUNNING state, not the safe one -- vyos001 has no `vif 53 disable`
because it happens to be master, vyos002 does. An apply performed while vyos002
held the VIP would therefore enable vyos001's WAN too, putting the cloned MAC on
both boxes. An override must assert `vif 53 disable` on BOTH, accepting that an
apply then briefly disables the current master's 10 gig until the reconciler
restores it.
Still not applied to production.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
|
||
|
Michal
|
93fed7826b |
labsim: PPPoE HA passes the matrix, and the health check had a real flap bug
Some checks failed
Full run green: baseline (one AC session, held by the VIP holder), clean
failover (pppoe0 moves in 26s, old master releases), 10 gig down (route falls to
pppoe0 and the LAN is back online in 5s), lease expiry (the guard hangs up), and
a missing peers file (NRestarts=0, no loop). Evidence in
labsim/wan-failover-evidence/.
The 10 gig test found a genuine bug in vrrp-wan-health, not in the sim. GRACE
was measured from PROMOTION, so an established master had no grace at all --
after hours of uptime `now - since` far exceeds any window. The first moment
bond0.53 went down while pppoe0 was mid-redial, the master failed its own check,
shed every VIP, and the peer inherited the same WAN outage and did the same. A
brief WAN blip would have flapped the production pair. The stamp is now
refreshed on every healthy tick, so grace measures time since the box last
demonstrably HAD a WAN -- survivable wherever the gap happens, not only just
after a promotion.
Three harness bugs, all the same shape, all of which produced a confident wrong
answer before being caught:
- waiting for "exactly one pppoe0 holder" returns INSTANTLY during a handover,
because it was already true. The useful question is who holds it.
- judging connectivity on a single ping 20s after a link drop reported an
outage that had already healed. Poll, do not sample.
- `-o PreferredAuthentications=password` suits the routers but not the Alpine
LAN VMs, whose sshd offers keyboard-interactive: ssh exited 255 before
running anything and the test read that as "the LAN lost the internet". A
tcpdump on the router showed the pings leaving pppoe0 NATed to
198.51.100.117 and the replies coming back the whole time. An exit code that
can mean "the network is broken" or "I could not log in" is not a
connectivity test, so the check now asserts on what the guest reported.
That last one is why the harness asks the routers and the access concentrator
rather than a client, and why it refuses to run at all while either router still
has a default route via eth2 -- the libvirt-NAT scaffold answers connectivity
checks the WAN under test would have failed.
Still to run: hard failover (destroy the master), and the session-control
replace/deny/disable axis that brackets Vodafone's unknown behaviour and sets
the final GRACE. Nothing applied to production.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
|