vyos: move PPPoE off the config plane onto a gated systemd unit
Some checks failed
Some checks failed
PPPoE HA could not work as written, and the reason is structural rather than a
bug: `set interfaces pppoe pppoe0 disable` and `delete` are handled identically
by interfaces_pppoe.py -- both UNLINK /etc/ppp/peers/pppoe0. That path is pppd's
own options file, so the resting state destroyed exactly what the promotion path
needed, and `ppp@pppoe0` restart-looped against it (observed: 47 restarts, zero
sessions at the access concentrator). It also made op-mode `connect interface
pppoe0` unusable, and put every failover behind a priority-322 commit where one
unrelated invalid node fails the whole thing -- which has already taken the
10 gig down once.
pppoe0 is now configured identically and ENABLED on both routers, so the peers
file always exists, and dialling is gated by a drop-in on the unit:
ConditionPathExists=/run/vrrp-wan/may-dial
ConditionPathExists=/etc/ppp/peers/pppoe0
/run is tmpfs, so the gate is shut at boot and neither box can dial before VRRP
has decided. That matters more than it first appears: with the node enabled,
interfaces_pppoe.py restarts ppp on EVERY commit touching the pppoe subtree when
the daemon is not running -- so the backup actively tries to dial whenever
anything commits. The gate is the only thing making that a no-op, which is why
vrrp-wan-reconcile now refuses to bless a box whose drop-in is missing: /etc is
per-image, and a VyOS upgrade would otherwise silently remove the protection.
may-dial is a LEASE, not a flag. ConditionPathExists is evaluated at start only
-- it can prevent a dial, never revoke one -- so a reconciler that stops running
while its box is demoted would keep the one ISP session for ever. The reconciler
renews the lease; a new 5s vrrp-wan-guard revokes it, and only ever revokes. It
fired correctly first time: "GUARD: lease stale (81s > 75s)".
Also: remove-then-stop on release (the file's absence blocks a NEW start that a
concurrent commit would trigger); a flap damper, because two routers that both
believe they hold the VIP will both dial and each dial kills the other's session
-- against a real ISP that is how an account gets rate-limited; and a guard on
`cfg` returning empty under commit-lock contention, which had already produced
one spurious "releasing" on a box that needed nothing.
GRACE 90 -> 180. accel-ppp's dead-peer budget is lcp-echo-interval(30) x
failure(3) = 90s, so the old value sat exactly on the boundary: a hard failover
into an AC that does not replace the stale session would fail its own check,
shed the VIPs, and leave both routers in FAULT.
The sim could not have tested any of this. Both routers now get the identical
WAN -- the secondary had none "because two PPPoE clients sharing one credential
is a different failure mode than anything production has", which is backwards:
that IS production. It also left the pair incomparable, ten NAT rules against
none. Safety now comes from resting state, not asymmetry.
Three more things the sim was hiding:
- the drift check's secondary regex omitted interfaces pppoe/bonding, nat
source and protocols failover, so it reported "in sync" for a box with no
WAN at all;
- the VRRP health-check and transition-script hooks existed on both live VMs
and in NEITHER generator -- the mechanism under test was pure undetected
drift;
- labsim-vyos's only default route was the libvirt-NAT scaffold, so every
"the LAN still has internet" verdict on it was answered by eth2 rather than
the WAN. --drop-scaffold applied; the earlier DHCP-failover proof is being
re-run because of it.
vrrp-wan-install ends the other half of that: the sim's previous proof came from
scripts hand-`sed`-ed in place, so the tested behaviour was not the committed
behaviour. `--check` now makes that a hard failure.
First green run: master holds both WANs, backup released, and the AC reports
exactly ONE session. sim-net-apply.sh check: all four in sync.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
This commit is contained in:
Michal
@@ -2,77 +2,87 @@
|
||||
# Make the WAN match VRRP mastership. Idempotent; safe to run every 30s and on
|
||||
# every VRRP transition.
|
||||
#
|
||||
# Why a reconciler and not just transition scripts
|
||||
# ------------------------------------------------
|
||||
# VyOS delivers `transition-script` through a helper process,
|
||||
# /usr/libexec/vyos/system/keepalived-fifo.py, fed by keepalived's notify_fifo.
|
||||
# Observed in labsim on 2026-09-02: the primary's Keepalived_vrrp logged
|
||||
# "(native) Entering MASTER STATE" for all six instances and the built-in
|
||||
# notify_master for conntrack-sync ran -- while the fifo helper logged NOTHING
|
||||
# and the master transition script never ran. The helper process was still
|
||||
# alive. The result was a router holding every VIP with no WAN at all: the exact
|
||||
# 2026-09-02 outage, re-created by the mechanism meant to prevent it.
|
||||
# Two WANs, two different control planes, for a reason:
|
||||
#
|
||||
# So transition scripts are kept for speed but nothing is trusted to them: this
|
||||
# also runs on a timer, and derives everything from ground truth rather than
|
||||
# from a marker that only exists if the script it depends on ran.
|
||||
# bond0.53 (10 gig, DHCP) -- CONFIG plane. Its lease is bound to a cloned MAC
|
||||
# (f0:9f:c2:12:9b:4f, the old USG's), and only VyOS
|
||||
# config can move a MAC. One commit per failover.
|
||||
# pppoe0 (Vodafone) -- SYSTEMD plane. Gated by a drop-in on
|
||||
# ppp@pppoe0; see migration/ppp-vrrp-gate.conf.
|
||||
# No commit, no config lock, no `save`.
|
||||
#
|
||||
# PPPoE used to be on the config plane too, via `set interfaces pppoe pppoe0
|
||||
# disable`. That could not work: `disable` unlinks /etc/ppp/peers/pppoe0, which
|
||||
# is pppd's options file, so the promotion path deleted the very thing it needed
|
||||
# and left the unit restart-looping (observed: 47 restarts, zero sessions at the
|
||||
# access concentrator).
|
||||
#
|
||||
# Why a reconciler and not just transition scripts: VyOS delivers
|
||||
# `transition-script` through keepalived-fifo.py, and on 2026-09-02 that helper
|
||||
# logged NOTHING for a promotion while Keepalived_vrrp logged all six instances
|
||||
# entering MASTER. A router held every VIP with no WAN -- the outage, recreated
|
||||
# by the mechanism meant to prevent it. Scripts give speed; the timer gives
|
||||
# correctness.
|
||||
#
|
||||
# vrrp-wan-reconcile reconcile once
|
||||
# vrrp-wan-reconcile --status what it thinks, changing nothing
|
||||
#
|
||||
# Ground truth for "am I master" is whether the management VIP is really on this
|
||||
# box. It is what VRRP actually does, it is observable, and it cannot silently
|
||||
# disagree with reality.
|
||||
|
||||
VIP="${VRRP_WAN_VIP:-192.168.1.1}" # management VIP; sim overrides via env
|
||||
WAN_VIF=53 # bond0.53, the DHCP WAN
|
||||
CONF=/config/vrrp-wan.conf
|
||||
[ -r "$CONF" ] && . "$CONF"
|
||||
VIP="${VRRP_WAN_VIP:-192.168.1.1}"
|
||||
WAN_VIF="${WAN_VIF:-53}"
|
||||
FLAP_MAX="${FLAP_MAX:-6}"
|
||||
FLAP_WINDOW="${FLAP_WINDOW:-600}"
|
||||
FLAP_HOLDOFF="${FLAP_HOLDOFF:-900}"
|
||||
|
||||
STATE=/run/vrrp-wan
|
||||
LOCK=/run/vrrp-wan.lock
|
||||
APPLY=/config/vrrp-wan-apply
|
||||
DROPIN=/etc/systemd/system/ppp@pppoe0.service.d/10-vrrp-wan-gate.conf
|
||||
|
||||
cfg() { /opt/vyatta/bin/vyatta-op-cmd-wrapper show configuration commands 2>/dev/null; }
|
||||
holds_vip() { ip -4 -o addr show 2>/dev/null | grep -q " ${VIP}/"; }
|
||||
wan_up() { ip -4 addr show "bond0.${WAN_VIF}" 2>/dev/null | grep -q 'inet '; }
|
||||
wan_disabled(){ cfg | grep -q "vif ${WAN_VIF} disable"; }
|
||||
ppp_disabled(){ cfg | grep -q "pppoe pppoe0 disable"; }
|
||||
ppp_up() { ip -4 addr show pppoe0 2>/dev/null | grep -q 'inet '; }
|
||||
ppp_active() { systemctl is-active --quiet ppp@pppoe0 2>/dev/null; }
|
||||
lease_age() { s=$(stat -c %Y "$STATE/may-dial" 2>/dev/null) || return 1
|
||||
echo $(( $(date +%s) - s )); }
|
||||
|
||||
# NOTE: there is deliberately no ppp_disabled(). pppoe0 is now ENABLED in config
|
||||
# on both routers, so such a test would be permanently false and the backup
|
||||
# early-exit below would never fire -- entering config mode every 30s for ever,
|
||||
# committing nothing. That exact shape was already live on the sim secondary,
|
||||
# whose /tmp/vrrp-wan-commit.log read "No configuration changes to commit" while
|
||||
# the script reported success.
|
||||
wan_disabled(){ cfg | grep -q "vif ${WAN_VIF} disable"; }
|
||||
|
||||
# --status must answer WITHOUT sourcing script-template. The template's `exit`
|
||||
# is a function that leaves configuration mode, not the shell builtin, so a
|
||||
# status run that had sourced it opened and closed a config session on every
|
||||
# call -- which is how a read-only query started colliding with the timer and
|
||||
# logging "Configuration system temporarily locked due to another commit".
|
||||
if [ "${1:-}" = "--status" ]; then
|
||||
printf 'vip=%s holds_vip=%s wan_disabled=%s wan_up=%s role=%s\n' \
|
||||
printf 'vip=%s holds_vip=%s wan_disabled=%s wan_up=%s ppp_up=%s ppp_active=%s may_dial=%s lease_age=%s dropin=%s role=%s\n' \
|
||||
"$VIP" "$(holds_vip && echo yes || echo no)" \
|
||||
"$(wan_disabled && echo yes || echo no)" \
|
||||
"$(wan_up && echo yes || echo no)" \
|
||||
"$(ppp_up && echo yes || echo no)" \
|
||||
"$(ppp_active && echo yes || echo no)" \
|
||||
"$([ -f "$STATE/may-dial" ] && echo yes || echo no)" \
|
||||
"$(lease_age 2>/dev/null || echo -)" \
|
||||
"$([ -f "$DROPIN" ] && echo yes || echo MISSING)" \
|
||||
"$(cat "$STATE/role" 2>/dev/null || echo unset)"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# One writer. The 30s timer and a VRRP transition can fire together, and two
|
||||
# VyOS commits in flight on one box do not queue -- the second fails outright.
|
||||
#
|
||||
# The lock fd MUST be closed for children (`9>&-` on every call below). Entering
|
||||
# VyOS config mode spawns a long-lived unionfs-fuse for the session, and it
|
||||
# INHERITS this descriptor and never lets go -- `lsof /run/vrrp-wan.lock` showed
|
||||
# it held by unionfs-fuse with fd 9w. From the first config change onward every
|
||||
# later run lost the flock and exited 0 without doing anything, so the
|
||||
# reconciler looked healthy in the journal ("Finished") while quietly having
|
||||
# stopped reconciling. It is how a demoted router kept the WAN.
|
||||
# One writer. The lock fd MUST be closed for children (`9>&-` on every call):
|
||||
# entering VyOS config mode spawns a long-lived unionfs-fuse for the session
|
||||
# which INHERITS the descriptor and never releases it, so from the first commit
|
||||
# onward every later run lost the flock and exited 0 having done nothing. That
|
||||
# is how a demoted router kept the WAN.
|
||||
exec 9>"$LOCK"
|
||||
flock -n 9 || exit 0
|
||||
|
||||
mkdir -p "$STATE"
|
||||
|
||||
# Reap config sessions whose owning process is gone. VyOS creates
|
||||
# /opt/vyatta/config/tmp/new_config_<pid> (a unionfs mount) per `configure`, and
|
||||
# a script that dies inside a session never removes it. One of those holds the
|
||||
# commit lock, and from then on EVERY commit fails with "Configuration system
|
||||
# temporarily locked due to another commit in progress" -- including the manual
|
||||
# one you try in order to fix it. A job on a 30s timer that can leak a session
|
||||
# per failure will wedge the router's config system on its own, so it cleans up
|
||||
# before it starts. `umount -l` first: the directory is a mount point and plain
|
||||
# rm returns "Device or resource busy".
|
||||
# Reap config sessions whose owning process is gone. VyOS leaves a unionfs mount
|
||||
# per `configure`, one of them holds the commit lock, and after that EVERY commit
|
||||
# fails -- including the manual one you try in order to fix it.
|
||||
for d in /opt/vyatta/config/tmp/new_config_*; do
|
||||
[ -d "$d" ] || continue
|
||||
pid=${d##*_}
|
||||
@@ -81,34 +91,84 @@ for d in /opt/vyatta/config/tmp/new_config_*; do
|
||||
rm -rf "$d" 2>/dev/null
|
||||
done
|
||||
|
||||
# The config edit lives in vrrp-wan-apply, because script-template must be the
|
||||
# first thing its script does -- sourced any later it terminates the script
|
||||
# silently with rc=0. See the header there.
|
||||
APPLY=/config/vrrp-wan-apply
|
||||
|
||||
if holds_vip; then
|
||||
echo master > "$STATE/role"
|
||||
# Stamp only on entry to master, so the health check's grace window measures
|
||||
# time-since-promotion rather than time-since-last-tick.
|
||||
[ -f "$STATE/since" ] || date +%s > "$STATE/since"
|
||||
wan_disabled || ppp_disabled || exit 0
|
||||
logger -t vrrp-wan "MASTER with WAN disabled -> enabling bond0.${WAN_VIF} + pppoe0"
|
||||
"$APPLY" enable 9>&-
|
||||
else
|
||||
echo backup > "$STATE/role"
|
||||
rm -f "$STATE/since"
|
||||
{ wan_disabled && ppp_disabled; } && exit 0
|
||||
# Releasing matters more than taking. A demoted router that keeps the WAN up
|
||||
# holds the cloned MAC f0:9f:c2:12:9b:4f on VLAN 53 at the same time as the
|
||||
# new master, and the switch sends the ISP's replies to whichever port spoke
|
||||
# last -- the WAN-side twin of the eth2 incident.
|
||||
logger -t vrrp-wan "not MASTER but WAN enabled -> releasing bond0.${WAN_VIF} + pppoe0"
|
||||
"$APPLY" disable 9>&-
|
||||
# `show configuration commands` has been observed returning EMPTY transiently
|
||||
# under commit-lock contention. Every grep against it then reads false, which on
|
||||
# the master path looks like "the WAN is disabled" and triggers a pointless
|
||||
# commit -- one such spurious "releasing" was logged on a box where both WANs
|
||||
# were already in the right state. A real config is ~500 lines; refuse to act on
|
||||
# a suspiciously short one.
|
||||
if [ "$(cfg | wc -l)" -lt 50 ]; then
|
||||
logger -t vrrp-wan "config read returned <50 lines; skipping this tick"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Deliberately no `save`. config.boot keeps `disable` on BOTH routers, so a
|
||||
# reboot in any order comes up unable to claim the shared MAC, and only holding
|
||||
# the VIP re-enables it. NOTE: any `save` while this box is master (a hand
|
||||
# commit, or `pulumi up`) WILL persist the enabled state -- observed in labsim.
|
||||
# The Pulumi model asserts `disable` on both routers so an apply puts it back,
|
||||
# and vyos:verify reports it as drift if it does not.
|
||||
# --- PPPoE: the systemd plane ---------------------------------------------
|
||||
ppp_dial() {
|
||||
# Refuse to bless a box whose gate is missing. /etc is per-image, so a VyOS
|
||||
# upgrade silently drops the drop-in -- and without it BOTH routers dial on
|
||||
# the next commit that touches the pppoe subtree. Failing closed turns a
|
||||
# silent loss of protection into "PPPoE never comes up", the safe direction.
|
||||
if [ ! -f "$DROPIN" ]; then
|
||||
logger -t vrrp-wan "REFUSING to dial: gate drop-in $DROPIN is missing (VyOS upgrade?)"
|
||||
return 1
|
||||
fi
|
||||
now=$(date +%s)
|
||||
if [ -f "$STATE/holdoff" ] && [ "$now" -lt "$(cat "$STATE/holdoff" 2>/dev/null || echo 0)" ]; then
|
||||
return 1
|
||||
fi
|
||||
touch "$STATE/may-dial" # renew the lease every tick
|
||||
ppp_active && return 0
|
||||
# Trim the dial log to the window, then decide.
|
||||
if [ -f "$STATE/dials" ]; then
|
||||
awk -v c="$((now - FLAP_WINDOW))" '$1 > c' "$STATE/dials" > "$STATE/dials.new" 2>/dev/null
|
||||
mv "$STATE/dials.new" "$STATE/dials" 2>/dev/null
|
||||
fi
|
||||
if [ "$(wc -l < "$STATE/dials" 2>/dev/null || echo 0)" -ge "$FLAP_MAX" ]; then
|
||||
echo $((now + FLAP_HOLDOFF)) > "$STATE/holdoff"
|
||||
logger -t vrrp-wan "DIAL FLAP: >=${FLAP_MAX} attempts in ${FLAP_WINDOW}s -- holding off ${FLAP_HOLDOFF}s"
|
||||
return 1
|
||||
fi
|
||||
echo "$now" >> "$STATE/dials"
|
||||
logger -t vrrp-wan "MASTER: dialling pppoe0"
|
||||
systemctl reset-failed ppp@pppoe0 2>/dev/null
|
||||
# `systemctl start` exits 0 even when a Condition blocks the start, so its
|
||||
# return code proves nothing. is-active is the only honest answer.
|
||||
systemctl start ppp@pppoe0 2>/dev/null
|
||||
}
|
||||
|
||||
ppp_release() {
|
||||
# Order matters: revoke the lease FIRST, then stop. The file's absence blocks
|
||||
# any NEW start (including one a concurrent VyOS commit would trigger); the
|
||||
# stop kills the process that already exists. Stopping first leaves a window
|
||||
# in which a commit re-dials a box that is being demoted.
|
||||
rm -f "$STATE/may-dial"
|
||||
ppp_active || return 0
|
||||
logger -t vrrp-wan "not MASTER: hanging up pppoe0"
|
||||
systemctl stop ppp@pppoe0 2>/dev/null
|
||||
}
|
||||
|
||||
# --- decide ----------------------------------------------------------------
|
||||
if holds_vip; then
|
||||
echo master > "$STATE/role"
|
||||
[ -f "$STATE/since" ] || date +%s > "$STATE/since"
|
||||
ppp_dial
|
||||
wan_disabled || exit 0
|
||||
logger -t vrrp-wan "MASTER with bond0.${WAN_VIF} disabled -> enabling"
|
||||
t0=$(date +%s)
|
||||
"$APPLY" enable 9>&-
|
||||
logger -t vrrp-wan "bond0.${WAN_VIF} enable commit took $(( $(date +%s) - t0 ))s"
|
||||
else
|
||||
echo backup > "$STATE/role"
|
||||
rm -f "$STATE/since" "$STATE/holdoff"
|
||||
ppp_release
|
||||
wan_disabled && exit 0
|
||||
logger -t vrrp-wan "not MASTER but bond0.${WAN_VIF} enabled -> releasing"
|
||||
t0=$(date +%s)
|
||||
"$APPLY" disable 9>&-
|
||||
logger -t vrrp-wan "bond0.${WAN_VIF} disable commit took $(( $(date +%s) - t0 ))s"
|
||||
fi
|
||||
|
||||
# No `save`, deliberately. config.boot keeps `vif 53 disable` on BOTH routers, so
|
||||
# a reboot in any order comes up unable to claim the cloned MAC. PPPoE needs no
|
||||
# such convention any more: with the gate, config.boot is safe by construction
|
||||
# and a stray `save` cannot make both boxes dial.
|
||||
|
||||
Reference in New Issue
Block a user