4d47b609a2
Some checks failed
Both were invisible to inspection and only appeared under the hard matrix: the flap damper tearing down an established session via the lease it stopped renewing, and a missing peers file being completely silent. The second has a production edge worth spelling out -- an unsaved commit reverts on reboot and takes pppoe0 with it, leaving a standby that can never take over while looking perfectly healthy.
150 lines
7.3 KiB
Markdown
150 lines
7.3 KiB
Markdown
# PPPoE high availability
|
|
|
|
Proven in labsim. **Not applied to production.**
|
|
|
|
## What it does
|
|
|
|
One consumer ISP account, two routers. The 10 gig lease is bound to a cloned MAC
|
|
(`f0:9f:c2:12:9b:4f`, the retired USG's) and the Vodafone line to a single
|
|
credential, so neither may be live on both boxes. The WAN follows VRRP
|
|
mastership — but the two halves use different control planes, and that is the
|
|
whole design:
|
|
|
|
| | plane | why |
|
|
|---|---|---|
|
|
| `bond0.53` (10 gig) | VyOS **config** (`disable`) | only config can move a MAC |
|
|
| `pppoe0` (Vodafone) | **systemd** unit gate | see below |
|
|
|
|
## Why PPPoE cannot live on the config plane
|
|
|
|
`interfaces_pppoe.py` treats `disable` and `delete` identically: both **unlink
|
|
`/etc/ppp/peers/pppoe0`**, call `PPPoEIf.remove()` (withdrawing the FRR default
|
|
route) and stop the unit. That path is pppd's own options file
|
|
(`ExecStart=/usr/sbin/pppd call %I`), so the resting state destroyed exactly what
|
|
the promotion path needed. `ppp@pppoe0` then restart-looped against the missing
|
|
file — 47 restarts observed, zero sessions at the access concentrator — and
|
|
never tripped systemd's limiter, because `RestartSec=5s` against the default
|
|
10s/5-burst window is only two restarts per interval.
|
|
|
|
It also made op-mode `connect interface pppoe0` unusable (it refuses without the
|
|
peers file), and put every failover behind a priority-322 commit where one
|
|
unrelated invalid node fails the whole thing.
|
|
|
|
## The gate
|
|
|
|
`pppoe0` is configured identically and **enabled on both** routers, so the peers
|
|
file always exists. Dialling is gated by
|
|
`/etc/systemd/system/ppp@pppoe0.service.d/10-vrrp-wan-gate.conf`:
|
|
|
|
```ini
|
|
ConditionPathExists=/run/vrrp-wan/may-dial
|
|
ConditionPathExists=/etc/ppp/peers/pppoe0
|
|
StartLimitIntervalSec=600
|
|
StartLimitBurst=6
|
|
```
|
|
|
|
`/run` is tmpfs, so the gate is shut at boot and neither box can dial before VRRP
|
|
has decided. **This is load-bearing, not a nicety:** with the node enabled,
|
|
`interfaces_pppoe.py` restarts ppp on *every* commit touching the pppoe subtree
|
|
when the daemon is not running — so the backup actively tries to dial whenever
|
|
anything commits (`pulumi up`, a hand commit, the boot-time config load). The
|
|
gate is the only thing making that a no-op, which is why `vrrp-wan-reconcile`
|
|
**refuses to bless a box whose drop-in is missing**: `/etc` is per-image, so a
|
|
VyOS upgrade silently removes the protection, and failing closed turns that into
|
|
"PPPoE never dials" rather than "both routers dial".
|
|
|
|
`may-dial` is a **lease, not a flag**. `ConditionPathExists` is evaluated at
|
|
start only — it can prevent a dial, never revoke one. `vrrp-wan-reconcile`
|
|
renews it every 30s; `vrrp-wan-guard` runs every 5s and only ever revokes, on
|
|
either "I do not hold the VIP" or "the lease is stale".
|
|
|
|
## Measured in labsim
|
|
|
|
| | |
|
|
|---|---|
|
|
| Clean failover (`force-fault`) | `pppoe0` moves in **20-26s**, reproducible |
|
|
| 10 gig down → PPPoE | route falls to `pppoe0`; LAN back online in **5s** |
|
|
| Stale lease | guard hangs up within ~5s |
|
|
| Missing peers file | `NRestarts=0` — no loop |
|
|
| Flap holdoff vs live session | session survives a forged 900s holdoff |
|
|
| Invariant | AC never showed two `simdsl` sessions |
|
|
|
|
## Two failure modes found by running it, not by reading it
|
|
|
|
**The flap damper tore down a healthy WAN.** `ppp_dial()` checked the hold-off
|
|
and returned *before* renewing `may-dial`. That file is a lease the guard
|
|
expires after `LEASE_TTL`, so tripping the damper stopped the renewal and the
|
|
guard hung up `pppoe0` **on the master** ~80s later:
|
|
|
|
```
|
|
DIAL FLAP: >=6 attempts in 600s -- holding off 900s
|
|
GUARD: lease stale (81s > 75s) -- hanging up pppoe0
|
|
```
|
|
|
|
A damper meant to suppress repeated *dials* was destroying an established
|
|
session instead. An active session now renews the lease and returns before
|
|
every other check; everything below only decides whether to start a **new**
|
|
session. T12 is the regression test.
|
|
|
|
**A missing peers file is silent.** `/etc/ppp/peers/pppoe0` is both pppd's
|
|
options file and the gate's second condition, and it is only written by a commit
|
|
that touches the pppoe subtree. Without it systemd logs
|
|
`skipped because of an unmet condition check` exactly once and then nothing —
|
|
a router that cannot dial at all looks identical to a healthy backup.
|
|
`ppp_dial()` now says so on every tick, and distinguishes the two causes:
|
|
configured-but-not-rendered (re-commit the subtree) versus no `pppoe0` in the
|
|
config at all.
|
|
|
|
The second cause is the one to watch in production: **a commit that was never
|
|
`save`d reverts on reboot and takes `pppoe0` with it.** That is exactly how the
|
|
sim secondary lost its WAN and spent hours looking like an ISP problem. After
|
|
any hand commit to the pppoe subtree, `save` — or the next reboot produces a
|
|
standby that can never take over.
|
|
|
|
## Deploying (not yet done)
|
|
|
|
1. `sudo /config/vyos-known-good save` on both.
|
|
2. `migration/vrrp-wan-install --vip 192.168.1.1 --host vyos@10.0.1.253` then the
|
|
same for `.252`. Then `--check` on both. **No config change yet** — verify
|
|
nothing dials.
|
|
3. Confirm `/config/wan-secrets` is present and identical on both.
|
|
4. **vyos002 first** (the non-master), on its own commit — `interfaces pppoe` is
|
|
priority 322 and one bad node fails everything:
|
|
`delete interfaces pppoe pppoe0 disable`, `commit-confirm 10`.
|
|
5. Verify vyos002 did **not** dial — check on the wire, not from state:
|
|
`sudo tcpdump -i bond0.51 -nn pppoed` should show no PADI. Then `confirm`; `save`.
|
|
6. vyos001: nothing to change; it already has `pppoe0` enabled.
|
|
7. Confirm `vif 53 disable` is in **both** `config.boot`s.
|
|
8. Add the drop-in re-install to the VyOS image-upgrade runbook.
|
|
|
|
**Rollback**, from either box: `set interfaces pppoe pppoe0 disable` on both and
|
|
`rm /run/vrrp-wan/may-dial`. That restores today's behaviour exactly.
|
|
|
|
## What the sim cannot prove
|
|
|
|
- **Vodafone's `session-control`.** The sim's accel-ppp defaults to `replace`, so
|
|
a new auth kills the old session immediately. A real BRAS may `deny` and hold
|
|
the session for its own dead-peer timer. The matrix runs all three modes to
|
|
bracket the risk; it cannot tell you which one you will meet, and account
|
|
rate-limiting or lockout on repeated dials has no sim analogue at all. The
|
|
flap damper (6 dials / 600s → 15 min hold-off) exists for that.
|
|
- **Whether Vodafone honours our LCP Terminate / PADT** on a graceful stop.
|
|
- **The cloned-MAC lease** — whether the 10 gig ISP re-issues `87.192.101.48` to
|
|
`f0:9f:c2:12:9b:4f` arriving on a different switch port. That risk belongs to
|
|
`bond0.53`, not PPPoE, and is the largest untested item in the failover.
|
|
- **Real dial time and MTU/MSS under load.** PPPoE was proven on the *USG*;
|
|
VyOS dialling Vodafone has never been done.
|
|
- **Timing under load.** The sim routers are idle 2-vCPU VMs; commit latency on
|
|
the VP2440s under kea + BGP + conntrack will be worse, and commit latency is
|
|
the dominant term in the `bond0.53` half of a failover.
|
|
|
|
## A model hazard to fix before applying
|
|
|
|
The imported baseline records the **running** state, not the safe one: vyos001
|
|
has no `vif 53 disable` (it is master), vyos002 does. So an apply performed while
|
|
vyos002 held the VIP would enable vyos001's WAN as well, putting the cloned MAC
|
|
on both boxes. An override must assert `vif 53 disable` on **both** routers, with
|
|
the reconciler re-enabling whichever holds the VIP. Note the consequence: an
|
|
apply then briefly disables the current master's 10 gig until the reconciler
|
|
restores it (≤30s).
|