vrrp-wan: a flap holdoff must not tear down a live WAN session
Some checks failed
Some checks failed
ppp_dial() checked the flap holdoff and returned BEFORE renewing /run/vrrp-wan/may-dial. That lease is what vrrp-wan-guard expires after LEASE_TTL, so tripping the damper stopped the renew and the guard hung up pppoe0 on the MASTER ~80s later. A damper meant to suppress repeated DIALS was tearing down a working WAN instead. Observed in labsim, end to end: DIAL FLAP: >=6 attempts in 600s -- holding off 900s GUARD: lease stale (81s > 75s) -- hanging up pppoe0 An established session now outranks every check below it: ppp_active renews the lease and returns first. Everything after it only decides whether to start a NEW session. Two supporting fixes for how that storm started. The dial attempts were all no-ops because /etc/ppp/peers/pppoe0 was missing, and nothing said so -- systemd logs "skipped because of an unmet condition check" exactly once and the gate looks identical to a healthy backup. ppp_dial() now reports it, and distinguishes "configured but not rendered" (re-commit the subtree) from "no pppoe0 in config at all", which is what a reboot leaves behind when a commit was never saved. That is precisely how the sim secondary lost its WAN. Also `cat | wc -l` rather than `wc -l < file`: redirections are applied left to right, so the missing-file error escapes the 2>/dev/null on every first-ever dial. Harness: T11 copied-then-removed instead of mv, and verifies the restore -- losing that file strands a router permanently, which cost a debugging session. preflight now refuses to run if either router lacks the peers file or the pppoe0 config, since every failover result would otherwise be a false negative blamed on the ISP. New T12 forges a 900s holdoff against a live session and asserts it survives.
This commit is contained in:
@@ -104,6 +104,19 @@ fi
|
||||
|
||||
# --- PPPoE: the systemd plane ---------------------------------------------
|
||||
ppp_dial() {
|
||||
# An ESTABLISHED session outranks every guard below, and this must be the
|
||||
# first thing here. `may-dial` is a lease the guard expires after
|
||||
# LEASE_TTL, so any early `return 1` before this renew silently hands the
|
||||
# guard a live session to kill.
|
||||
#
|
||||
# Observed in labsim: the flap damper tripped, returned early, the lease
|
||||
# went stale at 81s > 75s and the guard hung up pppoe0 ON THE MASTER --
|
||||
# a damper meant to suppress repeat DIALS tore down a working WAN instead.
|
||||
# Everything below only decides whether to start a NEW session.
|
||||
if ppp_active; then
|
||||
touch "$STATE/may-dial"
|
||||
return 0
|
||||
fi
|
||||
# Refuse to bless a box whose gate is missing. /etc is per-image, so a VyOS
|
||||
# upgrade silently drops the drop-in -- and without it BOTH routers dial on
|
||||
# the next commit that touches the pppoe subtree. Failing closed turns a
|
||||
@@ -112,18 +125,36 @@ ppp_dial() {
|
||||
logger -t vrrp-wan "REFUSING to dial: gate drop-in $DROPIN is missing (VyOS upgrade?)"
|
||||
return 1
|
||||
fi
|
||||
# The peers file is pppd's options file AND the gate's second condition, so
|
||||
# without it this box silently never dials: systemd logs "skipped because of
|
||||
# an unmet condition check" once and nothing else complains. Only a commit
|
||||
# that touches the pppoe subtree re-renders it.
|
||||
#
|
||||
# Seen in labsim: config applied but never `save`d, the router rebooted, and
|
||||
# came back with no pppoe0 node at all -- so no peers file, and a master that
|
||||
# dialled every tick into silence. Say so loudly rather than looking healthy.
|
||||
if [ ! -f /etc/ppp/peers/pppoe0 ]; then
|
||||
if cfg | grep -q "interfaces pppoe pppoe0 source-interface"; then
|
||||
logger -t vrrp-wan "CANNOT dial: pppoe0 is configured but /etc/ppp/peers/pppoe0 is missing -- re-commit the pppoe subtree to re-render it"
|
||||
else
|
||||
logger -t vrrp-wan "CANNOT dial: no pppoe0 in config (did a reboot revert an unsaved commit?)"
|
||||
fi
|
||||
return 1
|
||||
fi
|
||||
now=$(date +%s)
|
||||
if [ -f "$STATE/holdoff" ] && [ "$now" -lt "$(cat "$STATE/holdoff" 2>/dev/null || echo 0)" ]; then
|
||||
return 1
|
||||
fi
|
||||
touch "$STATE/may-dial" # renew the lease every tick
|
||||
ppp_active && return 0
|
||||
# Trim the dial log to the window, then decide.
|
||||
if [ -f "$STATE/dials" ]; then
|
||||
awk -v c="$((now - FLAP_WINDOW))" '$1 > c' "$STATE/dials" > "$STATE/dials.new" 2>/dev/null
|
||||
mv "$STATE/dials.new" "$STATE/dials" 2>/dev/null
|
||||
fi
|
||||
if [ "$(wc -l < "$STATE/dials" 2>/dev/null || echo 0)" -ge "$FLAP_MAX" ]; then
|
||||
# `cat | wc`, not `wc -l < file`: the shell applies redirections left to
|
||||
# right, so a missing file fails the `<` BEFORE `2>/dev/null` is in effect
|
||||
# and dash prints "No such file or directory" on every first-ever dial.
|
||||
if [ "$(cat "$STATE/dials" 2>/dev/null | wc -l)" -ge "$FLAP_MAX" ]; then
|
||||
echo $((now + FLAP_HOLDOFF)) > "$STATE/holdoff"
|
||||
logger -t vrrp-wan "DIAL FLAP: >=${FLAP_MAX} attempts in ${FLAP_WINDOW}s -- holding off ${FLAP_HOLDOFF}s"
|
||||
return 1
|
||||
|
||||
Reference in New Issue
Block a user