michal/lab
1
0
Fork 0
Code Issues Pull Requests 3 Actions Packages Projects Releases Wiki Activity

IPv6 was never HA, and the WAN becoming HA is what exposed it
Some checks failed
CI/CD / lint (push) Failing after 9s
Details
CI/CD / test (push) Failing after 9s
Details
CI/CD / typecheck (push) Failing after 26s
Details
CI/CD / build (push) Has been skipped
Details
CI/CD / publish-rpm (push) Has been skipped
Details
CI/CD / publish-deb (push) Has been skipped
Details

Browse Source 
Reviewed the parked IPv6 task against the PPPoE-HA work of 2026-09-05/06. The
gate that parked it ("WI-8 before IPv6") is cleared, but the same work
invalidated the assumption the IPv6 design rested on.

Verified on the live routers: vyos002 has no tun0, no he-tunnel-follow, no
he-secrets, no VLAN 9 prefix and no route6 ::/0 -- only the pre-staged
default-deny v6 firewall, which is correctly on both. Failover is now automatic
and drill-proven, so every failover takes the whole v6 estate down for as long
as vyos002 holds the VIP.

Four things that came out of checking rather than reading:

- PPPOE-HA.md's "tun0 survived untouched and IPv6 stayed up at 15.5ms" does not
  follow from its own premise and is corrected in place. The endpoint address is
  stable, but it MOVES to vyos002, which has nothing to decapsulate protocol 41.
  wan-drill had no IPv6 check at all, which is why nobody caught it.
- A 22-second near-miss: vif53-pin-boot-disable bounced the 10 gig, he-tunnel-
  follow ticked once and saw the PPPoE address, and vyos-failover restored the
  route 22s before the second tick would have pointed HE at an address Vodafone
  reissues on every dial.
- VyOS does NOT leave a tunnel down when its source-address is absent (the
  override's stated reason for leaving IPv6 single-homed). Measured in labsim:
  it commits rc=0 and brings the link UP -- a blackhole that attracts the v6
  default route. The runtime gate is load-bearing, like the PPPoE gate.
- The RA link-mtu was pinned at 1480 while the tunnel correctly drops to 1472 on
  the PPPoE path.

Mechanism, mirroring PPPoE HA -- identical config on both, gated at runtime, no
commit in the failover path:

- vrrp-wan-reconcile: a v6 kernel plane. tun0 and radvd follow the VIP; radvd is
  stopped BEFORE the WAN goes so its farewell RA (router-lifetime 0) still has a
  path out. The WAN early-exits became if-blocks so the plane runs every tick.
  It deliberately does NOT call he-tunnel-follow: that would halve the
  hysteresis the near-miss above showed we depend on.
- he-tunnel-follow: a master guard reading the same vrrp-wan.conf VIP, so the
  backup copy cannot point HE at its own idle PPPoE line, plus a stubbable
  HE_UPDATE_URL.
- vrrp-wan-install carries both, so --check and the upgrade runbook cover IPv6.
- wan-drill measures IPv6 in both timing loops and asserts zero HE API calls
  across a router failover.

labsim finally has an HE endpoint, closing the gap the override itself cited as
why this was never rehearsed. Both ISP islands already share the libvirt network,
so that becomes the backbone and HE lives behind it on one address reachable over
either WAN. Proven in the sim: backup tun=DOWN radvd=inactive, master tun=UP
radvd=active, hysteresis then HE call then MTU 1480->1472, and VLAN 9 hosts
autoconfiguring from the RA. The end-to-end v6 datapath is NOT yet proven --
inter-island transit crosses libvirt NAT and the return path is lost. Recorded as
a KNOWN SIM GAP rather than papered over.

The model change is staged, not merged: another agent runs pulumi up on that
repo, and the gate must exist on vyos002 before the tunnel does.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
This commit is contained in:
Michal
2026-09-06 14:30:44 +01:00
parent 061b9e3d7e
commit 395577850c
11 changed files with 1216 additions and 27 deletions
Download Patch File Download Diff File Expand all files Collapse all files

32
migration/PPPOE-HA.md
View File

@@ -178,11 +178,33 @@ under all conditions.
**Vodafone hands out a different IPv4 on every dial**: `83.106.5.72` →
`90.251.153.180` (vyos002) → `90.251.142.103` (vyos001, after failback).
Nothing may be pinned to the PPPoE address. Checked: the HE IPv6 tunnel is
pinned to `87.192.101.48`, which is the **10 gig** (`bond0.53`) and stable
across failover, so `tun0` survived untouched and IPv6 stayed up at 15.5ms.
Anything added later that hardcodes a WAN IP must use the 10 gig one, not
`pppoe0`'s.
Nothing may be pinned to the PPPoE address. The HE IPv6 tunnel is pinned to
`87.192.101.48`, which is the **10 gig** (`bond0.53`) and stable across
failover. Anything added later that hardcodes a WAN IP must use the 10 gig one,
not `pppoe0`'s.
> **CORRECTED 2026-09-06.** This paragraph originally continued "so `tun0`
> survived untouched and IPv6 stayed up at 15.5ms." **That conclusion was
> wrong, and it should never have been recorded as a result.** The premise is
> right — the endpoint address is stable — but it does not follow. During
> takeover the reconciler disables `bond0.53` on the demoted box, so
> `87.192.101.48` *leaves vyos001 and appears on vyos002*, and vyos002 has no
> `tun0` at all: no tunnel, no `he-tunnel-follow`, no `/config/he-secrets`, no
> VLAN 9 prefix, no `route6 ::/0`. Inbound protocol 41 from HE lands on a router
> with nothing to decapsulate it. vyos001's own journal for the drill window
> reads `08:36:01 he-tunnel-follow: no default route; refusing to guess`.
>
> The reading was taken either side of the window, not through it — `wan-drill`
> contained **no IPv6 check of any kind**, and neither did any other part of the
> mechanism. That is now fixed: the drill probes IPv6 in both timing loops and
> asserts that a router-level failover makes **zero** HE API calls. Until a
> drill produces that figure, the IPv6 behaviour of a failover is *unmeasured*,
> not "fine".
>
> The gap itself is real and open: **IPv6 is single-homed on vyos001 while the
> WAN beneath it is HA.** Step 0 of the re-sequenced IPv6 plan closes it; see
> `migration/pulumi-override-he-tunnel-both.json` and
> `labsim/labsim-ipv6-ha-test.sh`.
Production takeover (52s) is about twice the sim's `replace` figure (26s), which
is the expected direction: the VP2440s commit under kea, BGP and conntrack while

10
migration/VYOS-IMAGE-UPGRADE.md
View File

@@ -54,6 +54,16 @@ Never both at once — the surviving router must be able to hold the VIPs.
FAULT on every group means the health check is failing — most likely
`/config/vrrp-wan-health` did not get reinstalled, or `vrrp-wan.conf` is
missing so `GRACE` and the VIP fall back to defaults.
5b. **Check IPv6 came back with it.** `vrrp-wan-install` now carries
`he-tunnel-follow`, so `--check` covers it, but `/config/he-secrets` is a
secret placed by Pulumi and is only checked for *presence*:
```
sudo /config/he-tunnel-follow status # role, tunnel src, MTU
```
Want the box's own role, and — on the master — a tunnel source equal to the
**10 gig** address with MTU 1480. `/config` survives an upgrade, so the
`system task-scheduler` entry that runs this every minute survives too; it is
the units and the ppp gate in `/etc` that do not.
6. Confirm `config.boot` still pins the safe resting state:
```
sudo /config/vif53-pin-boot-disable --check # config.boot disable : 1

56
migration/he-tunnel-follow
View File

@@ -7,6 +7,13 @@
# that presents as "some sites are broken", which is far worse to diagnose than
# a clean one.
#
# Installed on BOTH routers and gated on VRRP mastership: the backup exits
# immediately, and vrrp-wan-reconcile brings tun0 up and calls this script the
# moment it takes the VIP. The HE endpoint itself needs no update when the WAN
# moves between routers -- 87.192.101.48 is the 10 gig lease bound to the cloned
# MAC, so it follows the VIP to the other box unchanged (proven by the 2026-09-06
# drill). HE only has to be told about the WITHIN-box fall back to PPPoE.
#
# Changes are made at KERNEL level (`ip tunnel change`), not in VyOS config, on
# purpose:
# - no commit per WAN flip, so a flapping line cannot churn the config;
@@ -27,6 +34,27 @@ set -uo pipefail
TUNNEL="${TUNNEL:-tun0}"
SECRETS="${SECRETS:-/config/he-secrets}"
STATE="${STATE:-/run/he-tunnel-follow.state}"
ROLE_STATE="${ROLE_STATE:-/run/he-tunnel-follow.role}"
# HE's update endpoint, as a variable so labsim can point it at a stub. The sim
# has no public IPv4 and no HE account, which is the whole reason the tunnel was
# never rehearsed; with this the sim can exercise the HE-side half too.
HE_UPDATE_URL="${HE_UPDATE_URL:-https://ipv4.tunnelbroker.net/nic/update}"
# This script is installed on BOTH routers -- the same principle as the PPPoE
# gate: configured identically everywhere, gated at runtime. So it must know
# when it is the backup. Left ungated, the backup copy either dies on "no
# default route" every minute, or, far worse, sees its own idle pppoe0 address
# and points the HE endpoint at it. Vodafone hands out a different IPv4 on every
# dial, so that is an IPv6 blackhole plus a wasted write against a rate-limited
# API -- and it would fire on the backup, where nobody is looking.
#
# The VIP comes from the same /config/vrrp-wan.conf the reconciler and the
# health check read, so there is exactly one definition of "master" on the box.
WAN_CONF="${WAN_CONF:-/config/vrrp-wan.conf}"
# shellcheck disable=SC1090
[ -r "$WAN_CONF" ] && . "$WAN_CONF"
VIP="${VRRP_WAN_VIP:-192.168.1.1}"
# 6in4 costs 20 bytes. The 10 gig path is 1500 -> 1480; PPPoE is 1492 -> 1472.
# Getting this wrong is the classic "IPv6 works until something large" failure.
declare -A WAN_MTU=( ["bond0.53"]=1480 ["pppoe0"]=1472 )
@@ -37,6 +65,7 @@ HYSTERESIS="${HYSTERESIS:-2}"
log() { logger -t he-tunnel-follow -- "$*"; printf ' %s\n' "$*"; }
die() { logger -t he-tunnel-follow -p user.err -- "$*"; printf ' ERROR: %s\n' "$*" >&2; exit 1; }
holds_vip() { ip -4 -o addr show 2>/dev/null | grep -q " ${VIP}/"; }
active_wan() { ip -4 route show default 2>/dev/null | awk '/^default/{for(i=1;i<=NF;i++) if($i=="dev") print $(i+1); exit}'; }
addr_of() { ip -4 -br addr show "$1" 2>/dev/null | awk '{print $3}' | cut -d/ -f1; }
tunnel_src() { ip tunnel show "$TUNNEL" 2>/dev/null | sed -nE 's/.* local ([0-9.]+).*/\1/p'; }
@@ -60,7 +89,7 @@ he_update() {
--data-urlencode "password=$HE_UPDATE_KEY" \
--data-urlencode "hostname=$HE_TUNNEL_ID" \
--data-urlencode "myip=$ip" \
"https://ipv4.tunnelbroker.net/nic/update" 2>&1)"
"$HE_UPDATE_URL" 2>&1)"
# dyndns protocol: "good <ip>" or "nochg <ip>" are both success.
case "$out" in
good*|nochg*) log "HE endpoint set to $ip ($out)"; return 0 ;;
@@ -68,9 +97,31 @@ he_update() {
esac
}
# Log only when the role CHANGES. On a 1-minute timer an unconditional line
# would be 1440 entries a day on the backup, which is how a real message gets
# lost. The marker lives in /run, so a reboot re-announces the role once.
note_role() {
local role="$1" last=""
[ -r "$ROLE_STATE" ] && read -r last < "$ROLE_STATE"
[ "$last" = "$role" ] && return 0
echo "$role" > "$ROLE_STATE"
log "role is now $role"
}
reconcile() {
local dry="${1:-}"
local wan src want_mtu cur_src cur_mtu
# The backup owns nothing here. vrrp-wan-reconcile holds tun0 down on this box
# and will run this script itself the moment it takes the VIP, so there is
# nothing to do and nothing to say.
if ! holds_vip; then
note_role backup
rm -f "$STATE" # start a promoted box with a clean hysteresis count
return 0
fi
note_role master
wan="$(active_wan)"; [ -n "$wan" ] || die "no default route; refusing to guess"
src="$(addr_of "$wan")"; [ -n "$src" ] || die "no IPv4 address on $wan"
want_mtu="${WAN_MTU[$wan]:-}"
@@ -110,10 +161,13 @@ reconcile() {
case "${1:-status}" in
status)
wan="$(active_wan)"
printf ' role : %s (vip %s)\n' "$(holds_vip && echo master || echo backup)" "$VIP"
printf ' active WAN : %s\n' "${wan:-<none>}"
printf ' wan addr : %s\n' "$(addr_of "${wan:-lo}")"
printf ' tunnel : %s\n' "$(ip -br link show "$TUNNEL" 2>/dev/null | awk '{print $2}' || echo '<absent>')"
printf ' tunnel src : %s\n' "$(tunnel_src)"
printf ' tunnel mtu : %s (want %s)\n' "$(tunnel_mtu)" "${WAN_MTU[${wan:-}]:-?}"
printf ' he endpoint: %s\n' "$HE_UPDATE_URL"
[ -r "$SECRETS" ] && printf ' credentials: present\n' || printf ' credentials: MISSING (%s)\n' "$SECRETS"
;;
run) reconcile "${2:-}" ;;

188
migration/pulumi-override-he-tunnel-both.json Normal file
View File

@@ -0,0 +1,188 @@
{
"id": "he-ipv6-tunnel",
"reason": "6in4 tunnel to Hurricane Electric, bringing 2001:470:187e::/48 in, on BOTH routers. SUPERSEDES he-ipv6-tunnel-vyos001, whose reason said 'vyos002 carries bond0.53 disabled, so the source address does not exist there and the tunnel would simply stay down -- adding it there is for a later takeover story, not now.' Both halves of that expired on 2026-09-06. `vif 53 disable` is no longer a property of vyos002: it is RUNTIME state owned by vrrp-wan-reconcile, keyed on who holds the management VIP, and deliberately absent from this model (see pppoe-gated-not-config-disabled and PPPOE-HA.md). And the takeover story shipped -- a drill moved the WAN and back, 52s and 36s. IPv6 did not follow it, so every failover took the whole v6 estate down for as long as vyos002 held the VIP. WHY THIS IS CHEAP: 87.192.101.48 is the 10 gig lease bound to the cloned MAC, and the drill proved the ISP re-issues THE SAME address to that MAC on the other router's port. The tunnel endpoint is therefore stable across the pair, so a router-level failover needs no HE API call at all -- only the tunnel present on both boxes and live on exactly one. HE updates remain solely for the within-box fall back to PPPoE, which /config/he-tunnel-follow already handles. WHY THE RUNTIME GATE IS LOAD-BEARING, not a nicety: measured in labsim 2026-09-06, VyOS ACCEPTS a tunnel whose source-address does not exist on the box (commit rc=0) and brings the link UP anyway. It is a blackhole that will happily attract the v6 default route -- not the inert node the old reason assumed. vrrp-wan-reconcile holds tun0 down on the backup and brings it up on the master, at kernel level, with no commit in the failover path. Verified in the sim: backup tun=DOWN radvd=inactive, master tun=UP radvd=active. MTU 1480, not 1500: 6in4 adds a 20-byte outer IPv4 header. Leave it at 1500 and IPv6 appears to work while large transfers hang. The RA link-mtu is 1472 -- the PPPoE figure -- deliberately, on BOTH routers: it cannot be reconciled at runtime because it needs a commit, so advertise the lower of the two paths and be correct on either WAN. Production previously pinned 1480 and was silently wrong whenever the WAN fell back. bond0.9 takes ::1 on vyos001 and ::2 on vyos002 -- NOT the same address: SLAAC hosts take their gateway from the advertising router's link-local, so the global address need not move, and duplicating it would only produce a DAD conflict. default-preference is high on vyos001 and low on vyos002 so that if both ever advertise at once -- radvd's config is rendered into /run and a booting backup starts it before VRRP has decided -- hosts prefer the normal master, while a genuinely dead vyos001 still leaves vyos002 as the only router on the link. No firewall change is needed: the IPv6 firewall accepts only from interface-group LAN, so tun0 is untrusted by default. It is already default-deny on BOTH routers -- that ordering held. REHEARSAL: labsim/labsim-he-endpoint.sh now builds a fake HE endpoint and stub tunnelbroker API, closing the 'the sim has no public IPv4 and no HE endpoint' gap the old reason cited as why this was never tested. labsim/labsim-ipv6-ha-test.sh is the matrix. Mechanism proven there; the end-to-end v6 datapath is not yet, because the sim's inter-island transit crosses libvirt NAT -- see that file's KNOWN SIM GAP header.",
"set": [
{
"path": [
"interfaces",
"tunnel",
"tun0",
"encapsulation"
],
"value": "sit"
},
{
"path": [
"interfaces",
"tunnel",
"tun0",
"source-address"
],
"value": "87.192.101.48"
},
{
"path": [
"interfaces",
"tunnel",
"tun0",
"remote"
],
"value": "216.66.88.98"
},
{
"path": [
"interfaces",
"tunnel",
"tun0",
"address"
],
"value": "2001:470:1f1c:f6::2/64"
},
{
"path": [
"interfaces",
"tunnel",
"tun0",
"mtu"
],
"value": "1480"
},
{
"path": [
"interfaces",
"tunnel",
"tun0",
"description"
],
"value": "HE 6in4 tunnel - 2001:470:187e::/48"
},
{
"path": [
"protocols",
"static",
"route6",
"::/0",
"next-hop",
"2001:470:1f1c:f6::1"
],
"value": {}
},
{
"path": [
"system",
"task-scheduler",
"task",
"he-tunnel-follow",
"executable",
"path"
],
"value": "/config/he-tunnel-follow"
},
{
"path": [
"system",
"task-scheduler",
"task",
"he-tunnel-follow",
"executable",
"arguments"
],
"value": "run"
},
{
"path": [
"system",
"task-scheduler",
"task",
"he-tunnel-follow",
"interval"
],
"value": "1m"
}
],
"perRouter": {
"vyos001": [
{
"path": [
"interfaces",
"bonding",
"bond0",
"vif",
"9",
"address"
],
"value": "2001:470:187e:9::1/64"
},
{
"path": [
"service",
"router-advert",
"interface",
"bond0.9",
"default-preference"
],
"value": "high"
}
],
"vyos002": [
{
"path": [
"interfaces",
"bonding",
"bond0",
"vif",
"9",
"address"
],
"value": "2001:470:187e:9::2/64"
},
{
"path": [
"service",
"router-advert",
"interface",
"bond0.9",
"default-preference"
],
"value": "low"
}
]
},
"sharedRouterAdvert": [
{
"path": [
"service",
"router-advert",
"interface",
"bond0.9",
"link-mtu"
],
"value": "1472"
},
{
"path": [
"service",
"router-advert",
"interface",
"bond0.9",
"prefix",
"2001:470:187e:9::/64",
"preferred-lifetime"
],
"value": "604800"
},
{
"path": [
"service",
"router-advert",
"interface",
"bond0.9",
"prefix",
"2001:470:187e:9::/64",
"valid-lifetime"
],
"value": "2592000"
}
],
"_staging_note": "STAGED, NOT MERGED -- on purpose, exactly like migration/pulumi-override-pppoe-gated.json was. Another agent runs `pulumi up` on kubernetes-deployment, so merging this IS a production change made by someone else at a time you do not choose. Order of operations, and it matters: (1) place /config/he-secrets on vyos002 and run migration/vrrp-wan-install on BOTH routers, so the runtime gate that holds tun0 down exists there BEFORE the tunnel does; (2) run `npm run vyos:export && npm run vyos:render` immediately before merging, so the model follows whichever router actually holds the WAN; (3) merge. Reversed, vyos002 gets a tunnel with no gate and an UP blackhole interface that attracts the v6 default route. The perRouter/sharedRouterAdvert shape here is descriptive -- fold it into whatever form overrides.json actually supports for per-router values when merging; the vyos001-only precedent is the `routers` key."
}

14
migration/vrrp-wan-install
View File

@@ -45,7 +45,13 @@ put() { # copy a file to the target
# script -> destination. take/release are hooks keepalived calls; both exec the
# reconciler, so there is one code path.
SCRIPTS="vrrp-wan-reconcile vrrp-wan-apply vrrp-wan-health vrrp-wan-guard vrrp-wan-take vrrp-wan-release"
#
# he-tunnel-follow is part of the mechanism, not a separate thing: the reconciler
# owns tun0's link state and that script owns its source address and MTU. Listing
# it here is what makes `--check` catch drift on it and what makes the VyOS
# image-upgrade runbook reinstall it -- IPv6 was previously the one half of the
# WAN story that no installer knew about.
SCRIPTS="vrrp-wan-reconcile vrrp-wan-apply vrrp-wan-health vrrp-wan-guard vrrp-wan-take vrrp-wan-release he-tunnel-follow"
UNITS="vrrp-wan-reconcile.service vrrp-wan-reconcile.timer vrrp-wan-guard.service vrrp-wan-guard.timer"
GATE_DIR=/etc/systemd/system/ppp@pppoe0.service.d
GATE=$GATE_DIR/10-vrrp-wan-gate.conf
@@ -66,6 +72,12 @@ if [ "$MODE" = check ]; then
remote_gate=$(run "md5sum $GATE 2>/dev/null | cut -d' ' -f1")
[ "$gate_sum" = "$remote_gate" ] || { echo " DRIFT $GATE (a VyOS upgrade wipes /etc -- both routers would dial)"; rc=1; }
run "[ -r /config/vrrp-wan.conf ]" || { echo " MISSING /config/vrrp-wan.conf"; rc=1; }
# Secrets are placed by Pulumi (infra/vyos/secretsFile.ts), never by this
# installer, so check presence only -- there is no correct content to compare
# against and printing a diff of credentials would be worse than useless.
# Without it he-tunnel-follow cannot re-point the tunnel when the WAN falls
# back to PPPoE, which fails silently: IPv4 keeps working and IPv6 goes dark.
run "[ -r /config/he-secrets ]" || { echo " MISSING /config/he-secrets (IPv6 cannot follow a WAN change)"; rc=1; }
for t in vrrp-wan-reconcile.timer vrrp-wan-guard.timer; do
[ "$(run "systemctl is-enabled $t 2>/dev/null")" = enabled ] || { echo " NOT ENABLED $t"; rc=1; }
done

97
migration/vrrp-wan-reconcile
View File

@@ -39,6 +39,8 @@ STATE=/run/vrrp-wan
LOCK=/run/vrrp-wan.lock
APPLY=/config/vrrp-wan-apply
DROPIN=/etc/systemd/system/ppp@pppoe0.service.d/10-vrrp-wan-gate.conf
V6_TUNNEL="${V6_TUNNEL:-tun0}"
RADVD_CONF="${RADVD_CONF:-/run/radvd/radvd.conf}"
cfg() { /opt/vyatta/bin/vyatta-op-cmd-wrapper show configuration commands 2>/dev/null; }
holds_vip() { ip -4 -o addr show 2>/dev/null | grep -q " ${VIP}/"; }
@@ -57,7 +59,7 @@ lease_age() { s=$(stat -c %Y "$STATE/may-dial" 2>/dev/null) || return 1
wan_disabled(){ cfg | grep -q "vif ${WAN_VIF} disable"; }
if [ "${1:-}" = "--status" ]; then
printf 'vip=%s holds_vip=%s wan_disabled=%s wan_up=%s ppp_up=%s ppp_active=%s may_dial=%s lease_age=%s dropin=%s role=%s\n' \
printf 'vip=%s holds_vip=%s wan_disabled=%s wan_up=%s ppp_up=%s ppp_active=%s may_dial=%s lease_age=%s dropin=%s role=%s tun=%s radvd=%s\n' \
"$VIP" "$(holds_vip && echo yes || echo no)" \
"$(wan_disabled && echo yes || echo no)" \
"$(wan_up && echo yes || echo no)" \
@@ -66,7 +68,9 @@ if [ "${1:-}" = "--status" ]; then
"$([ -f "$STATE/may-dial" ] && echo yes || echo no)" \
"$(lease_age 2>/dev/null || echo -)" \
"$([ -f "$DROPIN" ] && echo yes || echo MISSING)" \
"$(cat "$STATE/role" 2>/dev/null || echo unset)"
"$(cat "$STATE/role" 2>/dev/null || echo unset)" \
"$(ip -br link show "$V6_TUNNEL" 2>/dev/null | awk '{print $2}' || echo absent)" \
"$(systemctl is-active radvd 2>/dev/null || echo inactive)"
exit 0
fi
@@ -178,25 +182,94 @@ ppp_release() {
systemctl stop ppp@pppoe0 2>/dev/null
}
# --- IPv6: the kernel plane -------------------------------------------------
# The HE 6in4 tunnel and the VLAN 9 router advertisements have to follow
# mastership too, or a failover keeps IPv4 and silently drops IPv6 -- the
# partial outage that presents as "some sites are broken".
#
# A THIRD plane, and deliberately not either of the other two. Not config,
# because nothing here needs a commit (unlike the cloned MAC) and a commit per
# transition is the cost the pppoe0 half exists to avoid. Not the systemd gate,
# because there is no equivalent of a peers file to destroy.
#
# What makes this cheap: the tunnel is anchored to 87.192.101.48, the 10 gig
# lease bound to the cloned MAC, so it follows the VIP to the other router
# UNCHANGED. A router-level failover therefore needs no HE API call at all --
# only the link brought up on the box that now owns the address.
#
# Note what is deliberately NOT done here: this does not invoke
# he-tunnel-follow. That script has its own 1-minute task-scheduler cadence and
# a 2-tick hysteresis, and on 2026-09-06 that hysteresis was the only thing that
# stopped a routine `vif53-pin-boot-disable` run from pointing HE at a PPPoE
# address -- by 22 seconds. Calling it from a 30s reconciler as well would halve
# the window it needs. Its job is the WITHIN-box fall back to PPPoE; ours is
# link state.
v6_take() {
# Absent on a router that has no tunnel in its config -- which is every
# router until the model change lands. No-op there rather than complain.
[ -e "/sys/class/net/$V6_TUNNEL" ] || return 0
# Needs SOME WAN address to source from. Either line will do: if bond0.53 is
# down but pppoe0 is up, he-tunnel-follow re-points the tunnel on its own
# schedule, and holding the link down until then would turn a degraded path
# into no path.
wan_up || ppp_up || return 0
ip link show "$V6_TUNNEL" 2>/dev/null | grep -q 'state DOWN' && {
logger -t vrrp-wan "MASTER: bringing $V6_TUNNEL up"
ip link set "$V6_TUNNEL" up 2>/dev/null
}
# radvd's config is rendered into /run by the VyOS commit, so on a box with
# no router-advert node there is nothing to start.
[ -f "$RADVD_CONF" ] || return 0
systemctl is-active --quiet radvd 2>/dev/null && return 0
logger -t vrrp-wan "MASTER: starting radvd"
systemctl start radvd 2>/dev/null
}
v6_release() {
[ -e "/sys/class/net/$V6_TUNNEL" ] || return 0
# radvd FIRST, and this ordering is the point: on a graceful stop it emits a
# final advertisement with router-lifetime 0, which is what tells VLAN 9
# hosts to stop using this box as their default router. Kill the daemon
# after tearing things down and they keep a dead gateway until the RA
# lifetime expires on its own.
if systemctl is-active --quiet radvd 2>/dev/null; then
logger -t vrrp-wan "not MASTER: stopping radvd (deprecates the v6 gateway)"
systemctl stop radvd 2>/dev/null
fi
ip link show "$V6_TUNNEL" 2>/dev/null | grep -q 'state DOWN' && return 0
logger -t vrrp-wan "not MASTER: bringing $V6_TUNNEL down"
ip link set "$V6_TUNNEL" down 2>/dev/null
}
# --- decide ----------------------------------------------------------------
if holds_vip; then
echo master > "$STATE/role"
[ -f "$STATE/since" ] || date +%s > "$STATE/since"
ppp_dial
wan_disabled || exit 0
logger -t vrrp-wan "MASTER with bond0.${WAN_VIF} disabled -> enabling"
t0=$(date +%s)
"$APPLY" enable 9>&-
logger -t vrrp-wan "bond0.${WAN_VIF} enable commit took $(( $(date +%s) - t0 ))s"
# The WAN block is now an `if` rather than an early exit, so the IPv6 plane
# below is reached on EVERY tick and not only on the one that enables the
# WAN. On the promotion tick bond0.53 has no address yet, so v6_take no-ops
# and the next tick takes it.
if wan_disabled; then
logger -t vrrp-wan "MASTER with bond0.${WAN_VIF} disabled -> enabling"
t0=$(date +%s)
"$APPLY" enable 9>&-
logger -t vrrp-wan "bond0.${WAN_VIF} enable commit took $(( $(date +%s) - t0 ))s"
fi
v6_take
else
echo backup > "$STATE/role"
rm -f "$STATE/since" "$STATE/holdoff"
ppp_release
wan_disabled && exit 0
logger -t vrrp-wan "not MASTER but bond0.${WAN_VIF} enabled -> releasing"
t0=$(date +%s)
"$APPLY" disable 9>&-
logger -t vrrp-wan "bond0.${WAN_VIF} disable commit took $(( $(date +%s) - t0 ))s"
# Before the WAN goes, not after: once bond0.53 is disabled the source
# address is gone and radvd's farewell advertisement has no path out.
v6_release
if ! wan_disabled; then
logger -t vrrp-wan "not MASTER but bond0.${WAN_VIF} enabled -> releasing"
t0=$(date +%s)
"$APPLY" disable 9>&-
logger -t vrrp-wan "bond0.${WAN_VIF} disable commit took $(( $(date +%s) - t0 ))s"
fi
fi
# No `save`, deliberately. config.boot keeps `vif 53 disable` on BOTH routers, so

70
migration/wan-drill
View File

@@ -36,6 +36,22 @@ holder() { for h in "$P1" "$P2"; do
wan_of() { r "$1" 'for i in bond0.53 pppoe0; do a=$(ip -4 addr show dev $i 2>/dev/null | sed -n "s/.*inet \([0-9.]*\).*/\1/p"); [ -n "$a" ] && printf "%s=%s " $i $a; done'; }
online() { [ "$(r "$1" 'ping -c1 -W2 9.9.9.9 >/dev/null 2>&1 && echo y' | tr -d ' \n')" = y ]; }
# --- IPv6 -------------------------------------------------------------------
# Until 2026-09-06 this drill measured IPv4 only, and PPPOE-HA.md recorded that
# "IPv6 stayed up" through a failover on the strength of a reading taken outside
# the window. It cannot have: the reconciler disables bond0.53 on the demoted
# box, so 87.192.101.48 moves to the survivor, and inbound protocol 41 from HE
# then lands on whichever router owns the tunnel. Measure it rather than assume.
#
# Quad9 again, so the v6 result is comparable with the v4 one on the line above.
online6() { [ "$(r "$1" 'ping -6 -c1 -W2 2620:fe::fe >/dev/null 2>&1 && echo y' | tr -d ' \n')" = y ]; }
# Tunnel link state and source address. The source is the interesting half: it
# should be IDENTICAL before and after a router-level failover, because the 10
# gig lease follows the cloned MAC. A changed source means something called the
# HE API during the drill, which a router failover must never need to do.
tun_of() { r "$1" 'ip tunnel show tun0 2>/dev/null | sed -nE "s/.* local ([0-9.]+).*/\1/p"' | tr -d ' \n'; }
he_calls(){ r "$1" 'sudo journalctl -t he-tunnel-follow --since "'"$2"'" --no-pager 2>/dev/null | grep -c "HE endpoint set to"' | tr -d ' \n'; }
cleanup() {
say "--- cleanup (always runs) ---"
r "$P2" 'sudo /config/wan-drill-watchdog disarm' >/dev/null
@@ -52,11 +68,17 @@ cleanup() {
trap cleanup EXIT
say "=== pre-flight ==="
DRILL_START="$(date '+%Y-%m-%d %H:%M:%S')"
start_holder="$(holder)"
say "holder now : $start_holder"
say "$P1 WAN : $(wan_of $P1)"
say "$P2 WAN : $(wan_of $P2)"
online "$P1" && say "internet : UP via $P1" || { say "internet ALREADY DOWN -- refusing to drill"; exit 1; }
tun_before="$(tun_of $P1)"; tun2_before="$(tun_of $P2)"
say "$P1 tun0 src : ${tun_before:-<no tunnel>}"
say "$P2 tun0 src : ${tun2_before:-<no tunnel -- IPv6 cannot survive a failover>}"
if online6 "$P1"; then say "IPv6 : UP via $P1"
else say "IPv6 : DOWN on $P1 before we start -- v6 figures below are not meaningful"; fi
[ "$start_holder" = "$P1" ] || { say "expected $P1 to hold the VIP, got $start_holder -- refusing"; exit 1; }
if [ "$DRY" = 1 ]; then
@@ -71,14 +93,19 @@ say "=== DRILL: force-faulting $P1 ==="
t0=$(date +%s)
r "$P1" 'sudo touch /run/vrrp-wan/force-fault'
took=""
took=""; took6=""
while [ $(( $(date +%s) - t0 )) -lt "$TAKEOVER_BUDGET" ]; do
sleep 5
h="$(holder)"; w="$(wan_of $P2)"
say " t+$(( $(date +%s) - t0 ))s holder=$h vyos002_wan=[$w]"
if [ "$h" = "$P2" ] && [ -n "$w" ] && online "$P2"; then
took=$(( $(date +%s) - t0 )); break
# v6 keeps being probed after v4 comes back, because the two recover
# independently and the gap between them IS the number this drill exists to
# produce. Stop only when both are up, or the budget runs out.
[ -z "$took6" ] && online6 "$P2" && took6=$(( $(date +%s) - t0 ))
say " t+$(( $(date +%s) - t0 ))s holder=$h vyos002_wan=[$w] v6=$([ -n "$took6" ] && echo up || echo down)"
if [ -z "$took" ] && [ "$h" = "$P2" ] && [ -n "$w" ] && online "$P2"; then
took=$(( $(date +%s) - t0 ))
fi
[ -n "$took" ] && [ -n "$took6" ] && break
done
if [ -n "$took" ]; then
@@ -86,20 +113,47 @@ if [ -n "$took" ]; then
else
say "*** TAKEOVER FAILED within ${TAKEOVER_BUDGET}s -- failing back ***"
fi
if [ -n "$took6" ] && [ -n "$took" ]; then
say "*** IPv6 followed in ${took6}s (v4 ${took}s, gap $(( took6 - took ))s) ***"
elif [ -n "$took6" ]; then
say "*** IPv6 followed in ${took6}s, but IPv4 never did ***"
else
say "*** IPv6 did NOT return within ${TAKEOVER_BUDGET}s on $P2 -- the v6 estate is down for the whole takeover ***"
fi
say "=== failing back to $P1 ==="
t1=$(date +%s)
r "$P2" 'sudo /config/wan-drill-watchdog disarm' >/dev/null
r "$P1" 'sudo rm -f /run/vrrp-wan/force-fault'
r "$P2" 'sudo touch /run/vrrp-wan/force-fault'
back=""
back=""; back6=""
while [ $(( $(date +%s) - t1 )) -lt "$FAILBACK_BUDGET" ]; do
sleep 5
h="$(holder)"; w="$(wan_of $P1)"
say " t+$(( $(date +%s) - t1 ))s holder=$h vyos001_wan=[$w]"
if [ "$h" = "$P1" ] && [ -n "$w" ] && online "$P1"; then
back=$(( $(date +%s) - t1 )); break
[ -z "$back6" ] && online6 "$P1" && back6=$(( $(date +%s) - t1 ))
say " t+$(( $(date +%s) - t1 ))s holder=$h vyos001_wan=[$w] v6=$([ -n "$back6" ] && echo up || echo down)"
if [ -z "$back" ] && [ "$h" = "$P1" ] && [ -n "$w" ] && online "$P1"; then
back=$(( $(date +%s) - t1 ))
fi
[ -n "$back" ] && [ -n "$back6" ] && break
done
[ -n "$back" ] && say "*** FAILBACK OK in ${back}s ***" \
|| say "*** FAILBACK FAILED -- cleanup will clear both levers ***"
[ -n "$back6" ] && say "*** IPv6 back in ${back6}s ***" \
|| say "*** IPv6 did NOT return within ${FAILBACK_BUDGET}s on $P1 ***"
# The invariant a router-level failover must satisfy: the HE endpoint is never
# touched. 87.192.101.48 follows the cloned MAC to the other box, so the tunnel
# source is the same address on either router and there is nothing to tell HE.
# A non-zero count here means something re-pointed the tunnel at a PPPoE address
# -- which Vodafone re-issues on every dial, so it would be wrong within minutes.
say "=== IPv6 invariants ==="
tun_after="$(tun_of $P1)"
say "tun0 src : ${tun_before:-none} -> ${tun_after:-none}"
[ "$tun_before" = "$tun_after" ] && say " OK: tunnel source unchanged across the drill" \
|| say " *** CHANGED -- a router failover should never move the HE endpoint ***"
for h in "$P1" "$P2"; do
n="$(he_calls "$h" "$DRILL_START")"
say "HE updates from $h since ${DRILL_START}: ${n:-?}"
[ "${n:-0}" = 0 ] || say " *** $h called the HE API during a router failover -- it should not need to ***"
done