IPv6 was never HA, and the WAN becoming HA is what exposed it
Some checks failed
Some checks failed
Reviewed the parked IPv6 task against the PPPoE-HA work of 2026-09-05/06. The
gate that parked it ("WI-8 before IPv6") is cleared, but the same work
invalidated the assumption the IPv6 design rested on.
Verified on the live routers: vyos002 has no tun0, no he-tunnel-follow, no
he-secrets, no VLAN 9 prefix and no route6 ::/0 -- only the pre-staged
default-deny v6 firewall, which is correctly on both. Failover is now automatic
and drill-proven, so every failover takes the whole v6 estate down for as long
as vyos002 holds the VIP.
Four things that came out of checking rather than reading:
- PPPOE-HA.md's "tun0 survived untouched and IPv6 stayed up at 15.5ms" does not
follow from its own premise and is corrected in place. The endpoint address is
stable, but it MOVES to vyos002, which has nothing to decapsulate protocol 41.
wan-drill had no IPv6 check at all, which is why nobody caught it.
- A 22-second near-miss: vif53-pin-boot-disable bounced the 10 gig, he-tunnel-
follow ticked once and saw the PPPoE address, and vyos-failover restored the
route 22s before the second tick would have pointed HE at an address Vodafone
reissues on every dial.
- VyOS does NOT leave a tunnel down when its source-address is absent (the
override's stated reason for leaving IPv6 single-homed). Measured in labsim:
it commits rc=0 and brings the link UP -- a blackhole that attracts the v6
default route. The runtime gate is load-bearing, like the PPPoE gate.
- The RA link-mtu was pinned at 1480 while the tunnel correctly drops to 1472 on
the PPPoE path.
Mechanism, mirroring PPPoE HA -- identical config on both, gated at runtime, no
commit in the failover path:
- vrrp-wan-reconcile: a v6 kernel plane. tun0 and radvd follow the VIP; radvd is
stopped BEFORE the WAN goes so its farewell RA (router-lifetime 0) still has a
path out. The WAN early-exits became if-blocks so the plane runs every tick.
It deliberately does NOT call he-tunnel-follow: that would halve the
hysteresis the near-miss above showed we depend on.
- he-tunnel-follow: a master guard reading the same vrrp-wan.conf VIP, so the
backup copy cannot point HE at its own idle PPPoE line, plus a stubbable
HE_UPDATE_URL.
- vrrp-wan-install carries both, so --check and the upgrade runbook cover IPv6.
- wan-drill measures IPv6 in both timing loops and asserts zero HE API calls
across a router failover.
labsim finally has an HE endpoint, closing the gap the override itself cited as
why this was never rehearsed. Both ISP islands already share the libvirt network,
so that becomes the backbone and HE lives behind it on one address reachable over
either WAN. Proven in the sim: backup tun=DOWN radvd=inactive, master tun=UP
radvd=active, hysteresis then HE call then MTU 1480->1472, and VLAN 9 hosts
autoconfiguring from the RA. The end-to-end v6 datapath is NOT yet proven --
inter-island transit crosses libvirt NAT and the return path is lost. Recorded as
a KNOWN SIM GAP rather than papered over.
The model change is staged, not merged: another agent runs pulumi up on that
repo, and the gate must exist on vyos002 before the tunnel does.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
This commit is contained in:
Michal
43
labsim/ipv6-ha-evidence/V0-baseline/state.txt
Normal file
43
labsim/ipv6-ha-evidence/V0-baseline/state.txt
Normal file
@@ -0,0 +1,43 @@
|
|||||||
|
=== 2026-09-06T14:25:25+01:00 ===
|
||||||
|
--- HE endpoint ---
|
||||||
|
HE address : 192.0.2.10/32
|
||||||
|
he-sim : he-sim: ipv6/ip remote 198.51.100.137 local 192.0.2.10 ttl 64 6rd-prefix 2002::/16
|
||||||
|
he-sim v6 : 2001:db8:1f1c:f6::1/64
|
||||||
|
API : running
|
||||||
|
API bound : nohost
|
||||||
|
API calls : 4
|
||||||
|
route back : 198.51.100.0/24 via 192.168.122.63 dev eth1
|
||||||
|
--- 172.31.1.252 ---
|
||||||
|
vip=172.31.1.1 holds_vip=no wan_disabled=yes wan_up=no ppp_up=no ppp_active=no may_dial=no lease_age=- dropin=yes role=backup tun=DOWN radvd=inactive
|
||||||
|
inactive
|
||||||
|
tun0@NONE DOWN 203.0.113.108 <POINTOPOINT,NOARP>
|
||||||
|
tun0: ipv6/ip remote 192.0.2.10 local 203.0.113.108 ttl 64 tos inherit 6rd-prefix 2002::/16
|
||||||
|
inactive
|
||||||
|
Sep 05 23:27:45 apitest vrrp-wan[13361]: bond0.53 enable commit took 5s
|
||||||
|
Sep 05 23:29:01 apitest vrrp-wan[16369]: GUARD: lease stale (200s > 75s; is vrrp-wan-reconcile.timer running?) -- hanging up pppoe0
|
||||||
|
Sep 05 23:29:27 apitest vrrp-wan[17398]: MASTER: dialling pppoe0
|
||||||
|
Sep 05 23:29:57 apitest vrrp-wan[18610]: MASTER: dialling pppoe0
|
||||||
|
-- Boot 6efc3c9ba47f455e9668454ee6c2fc37 --
|
||||||
|
Sep 05 23:34:48 apitest vrrp-wan[6266]: MASTER: dialling pppoe0
|
||||||
|
Sep 05 23:34:49 apitest vrrp-wan[6422]: MASTER with bond0.53 disabled -> enabling
|
||||||
|
Sep 05 23:34:54 apitest vrrp-wan[7130]: bond0.53 enable commit took 5s
|
||||||
|
-- Boot 866611afd7c542d4bf8c5117978dcd4a --
|
||||||
|
Sep 06 13:21:54 apitest vrrp-wan[544215]: not MASTER: stopping radvd (deprecates the v6 gateway)
|
||||||
|
Sep 06 13:21:54 apitest vrrp-wan[544221]: not MASTER: bringing tun0 down
|
||||||
|
Sep 06 13:22:01 apitest he-tunnel-follow[544377]: role is now backup
|
||||||
|
--- 172.31.1.253 ---
|
||||||
|
vip=172.31.1.1 holds_vip=yes wan_disabled=no wan_up=yes ppp_up=yes ppp_active=yes may_dial=yes lease_age=25 dropin=yes role=master tun=UNKNOWN radvd=active
|
||||||
|
tun0@NONE UNKNOWN 198.51.100.137 <POINTOPOINT,NOARP,UP,LOWER_UP>
|
||||||
|
tun0: ipv6/ip remote 192.0.2.10 local 198.51.100.137 ttl 64 tos inherit 6rd-prefix 2002::/16
|
||||||
|
default nhid 111 via 2001:db8:1f1c:f6::1 dev tun0 proto static metric 20 pref medium
|
||||||
|
active
|
||||||
|
Sep 06 09:29:21 vyos vrrp-wan[422441]: MASTER with bond0.53 disabled -> enabling
|
||||||
|
Sep 06 09:29:25 vyos vrrp-wan[423516]: bond0.53 enable commit took 4s
|
||||||
|
Sep 06 13:22:01 vyos he-tunnel-follow[593096]: role is now master
|
||||||
|
Sep 06 13:22:01 vyos he-tunnel-follow[593109]: change seen (203.0.113.108 -> 198.51.100.137) but waiting for stability (1/2)
|
||||||
|
Sep 06 13:23:01 vyos he-tunnel-follow[594799]: HE endpoint set to 198.51.100.137 (good 198.51.100.137)
|
||||||
|
Sep 06 13:23:01 vyos he-tunnel-follow[594805]: moved tun0 to pppoe0: src 203.0.113.108 -> 198.51.100.137, mtu 1480 -> 1472
|
||||||
|
Sep 06 13:24:02 vyos he-tunnel-follow[595700]: in sync: tun0 via pppoe0 src 198.51.100.137 mtu 1472
|
||||||
|
Sep 06 13:24:26 vyos he-tunnel-follow[595886]: in sync: tun0 via pppoe0 src 198.51.100.137 mtu 1472
|
||||||
|
Sep 06 13:24:26 vyos he-tunnel-follow[595904]: in sync: tun0 via pppoe0 src 198.51.100.137 mtu 1472
|
||||||
|
Sep 06 13:25:01 vyos he-tunnel-follow[596311]: in sync: tun0 via pppoe0 src 198.51.100.137 mtu 1472
|
||||||
23
labsim/ipv6-ha-evidence/mechanism-2026-09-06.txt
Normal file
23
labsim/ipv6-ha-evidence/mechanism-2026-09-06.txt
Normal file
@@ -0,0 +1,23 @@
|
|||||||
|
=== mechanism evidence, labsim, 2026-09-06T14:28:11+01:00 ===
|
||||||
|
--- 172.31.1.252 ---
|
||||||
|
vip=172.31.1.1 holds_vip=no wan_disabled=yes wan_up=no ppp_up=no ppp_active=no may_dial=no lease_age=- dropin=yes role=backup tun=DOWN radvd=inactive
|
||||||
|
inactive
|
||||||
|
tun0@NONE DOWN 203.0.113.108 <POINTOPOINT,NOARP>
|
||||||
|
inactive
|
||||||
|
Sep 06 13:22:01 apitest he-tunnel-follow[544377]: role is now backup
|
||||||
|
--- 172.31.1.253 ---
|
||||||
|
vip=172.31.1.1 holds_vip=yes wan_disabled=no wan_up=yes ppp_up=yes ppp_active=yes may_dial=yes lease_age=26 dropin=yes role=master tun=UNKNOWN radvd=active
|
||||||
|
tun0@NONE UNKNOWN 198.51.100.137 <POINTOPOINT,NOARP,UP,LOWER_UP>
|
||||||
|
active
|
||||||
|
Sep 06 13:25:01 vyos he-tunnel-follow[596311]: in sync: tun0 via pppoe0 src 198.51.100.137 mtu 1472
|
||||||
|
Sep 06 13:26:01 vyos he-tunnel-follow[598274]: in sync: tun0 via pppoe0 src 198.51.100.137 mtu 1472
|
||||||
|
Sep 06 13:27:01 vyos he-tunnel-follow[599393]: in sync: tun0 via pppoe0 src 198.51.100.137 mtu 1472
|
||||||
|
Sep 06 13:28:01 vyos he-tunnel-follow[600596]: in sync: tun0 via pppoe0 src 198.51.100.137 mtu 1472
|
||||||
|
--- HE endpoint ---
|
||||||
|
HE address : 192.0.2.10/32
|
||||||
|
he-sim : he-sim: ipv6/ip remote 198.51.100.137 local 192.0.2.10 ttl 64 6rd-prefix 2002::/16
|
||||||
|
he-sim v6 : 2001:db8:1f1c:f6::1/64
|
||||||
|
API : running
|
||||||
|
API bound : nohost
|
||||||
|
API calls : 5
|
||||||
|
route back : 198.51.100.0/24 via 192.168.122.63 dev eth1
|
||||||
280
labsim/labsim-he-endpoint.sh
Executable file
280
labsim/labsim-he-endpoint.sh
Executable file
@@ -0,0 +1,280 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# Build a fake Hurricane Electric 6in4 endpoint inside labsim.
|
||||||
|
#
|
||||||
|
# WHY THIS EXISTS
|
||||||
|
# The production HE tunnel was never rehearsed. The override that introduced it
|
||||||
|
# says so in its own reason text -- "the sim has no public IPv4 and no HE
|
||||||
|
# endpoint, so there is nothing to tunnel to" -- and that gap is why IPv6 was
|
||||||
|
# the one half of the WAN story with no matrix behind it. Then the WAN became
|
||||||
|
# HA and IPv6 did not follow, which nobody caught, because nothing tests it.
|
||||||
|
#
|
||||||
|
# THE PROBLEM THIS HAD TO SOLVE
|
||||||
|
# The sim's two WANs are isolated islands. Verified:
|
||||||
|
#
|
||||||
|
# 203.0.113.1 from 203.0.113.107 : OK <- 10 gig analogue
|
||||||
|
# 203.0.113.1 from 198.51.100.137 : unreachable
|
||||||
|
# 198.51.100.1 from 198.51.100.137 : OK <- PPPoE analogue
|
||||||
|
# 198.51.100.1 from 203.0.113.107 : unreachable
|
||||||
|
#
|
||||||
|
# A 6in4 tunnel has ONE remote address, and production never changes it -- so an
|
||||||
|
# endpoint reachable over only one WAN could not rehearse the case that matters
|
||||||
|
# most: the WITHIN-box fall back from the 10 gig to PPPoE, where he-tunnel-follow
|
||||||
|
# re-points the tunnel and calls the HE API. That is exactly where the 2026-09-06
|
||||||
|
# near-miss lived.
|
||||||
|
#
|
||||||
|
# So the sim needs a minimal "internet": both ISP boxes already sit on the
|
||||||
|
# libvirt default network (192.168.122.0/24) and both forward, so that becomes
|
||||||
|
# the backbone, and HE lives on a single address behind it, reachable over
|
||||||
|
# either WAN. No new VMs, no new networks.
|
||||||
|
#
|
||||||
|
# 192.0.2.10 "HE" -- on isp-dhcp, reached from the PPPoE island via
|
||||||
|
# 192.168.122.136, and directly from the 10 gig island
|
||||||
|
#
|
||||||
|
# EVERYTHING HERE IS KERNEL-LEVEL, not VyOS config. The ISP boxes are scaffold,
|
||||||
|
# not the thing under test: `ip` commands leave no config to drift, no commit to
|
||||||
|
# fail, and a reboot cleans up. The ROUTER side is deliberately the opposite --
|
||||||
|
# it goes through real VyOS config, because "will VyOS commit a tunnel whose
|
||||||
|
# source-address does not exist on this box?" is one of the questions.
|
||||||
|
#
|
||||||
|
# ./labsim-he-endpoint.sh up build it
|
||||||
|
# ./labsim-he-endpoint.sh down tear it down
|
||||||
|
# ./labsim-he-endpoint.sh status what is live
|
||||||
|
# ./labsim-he-endpoint.sh calls how many times the HE API was called
|
||||||
|
# ./labsim-he-endpoint.sh point IP point the endpoint by hand (sim bookkeeping)
|
||||||
|
set -uo pipefail
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||||
|
DHCP_ISP="${DHCP_ISP:-192.168.122.136}" # owns the 10 gig segment, hosts "HE"
|
||||||
|
PPPOE_ISP="${PPPOE_ISP:-192.168.122.63}" # owns the PPPoE segment
|
||||||
|
PW="${VYOS_PW:-vyos}"
|
||||||
|
|
||||||
|
# TEST-NET-1 for the endpoint, and the documentation prefix for v6. Production
|
||||||
|
# uses 2001:470:187e::/48 from HE; the sim mirrors its SHAPE
|
||||||
|
# (2001:db8:187e:<vlan>::/64) so the scheme is exercised, not just the tunnel.
|
||||||
|
HE_ADDR="${HE_ADDR:-192.0.2.10}"
|
||||||
|
HE_LINK6="${HE_LINK6:-2001:db8:1f1c:f6::1}" # HE side of the tunnel /64
|
||||||
|
RT_LINK6="${RT_LINK6:-2001:db8:1f1c:f6::2}" # router side
|
||||||
|
SITE6="${SITE6:-2001:db8:187e::/48}" # routed to the router side
|
||||||
|
TENGIG_NET="${TENGIG_NET:-203.0.113.0/24}"
|
||||||
|
# Any valid address that will never be a router WAN -- see the tunnel creation
|
||||||
|
# below for why this cannot be 0.0.0.0.
|
||||||
|
PLACEHOLDER_REMOTE="${PLACEHOLDER_REMOTE:-203.0.113.1}"
|
||||||
|
PPPOE_NET="${PPPOE_NET:-198.51.100.0/24}"
|
||||||
|
|
||||||
|
SSH=(-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null
|
||||||
|
-o LogLevel=ERROR -o ConnectTimeout=6 -o PreferredAuthentications=password)
|
||||||
|
dhcp_isp() { timeout 40 sshpass -p "$PW" ssh "${SSH[@]}" "vyos@$DHCP_ISP" "$@" 2>/dev/null; }
|
||||||
|
pppoe_isp() { timeout 40 sshpass -p "$PW" ssh "${SSH[@]}" "vyos@$PPPOE_ISP" "$@" 2>/dev/null; }
|
||||||
|
|
||||||
|
log() { printf '\033[0;36m[he-sim]\033[0m %s\n' "$*"; }
|
||||||
|
die() { printf '\033[0;31m[he-sim]\033[0m %s\n' "$*" >&2; exit 1; }
|
||||||
|
|
||||||
|
# --- the stub tunnelbroker API ---------------------------------------------
|
||||||
|
# HE's real endpoint is a dyndns-style updater that re-points the tunnel's remote
|
||||||
|
# address. This is that, in 40 lines, so the sim can exercise the HE-SIDE half of
|
||||||
|
# a failover -- the half production can never safely test.
|
||||||
|
#
|
||||||
|
# It logs every call to /run/he-sim-api.log, which is what lets the matrix assert
|
||||||
|
# the invariant that matters: a ROUTER-level failover must call this ZERO times,
|
||||||
|
# because the 10 gig address follows the cloned MAC to the other box unchanged.
|
||||||
|
API_PY='
|
||||||
|
import http.server, subprocess, urllib.parse, datetime, sys, re
|
||||||
|
|
||||||
|
TUN = "he-sim"
|
||||||
|
LOCAL = sys.argv[1] if len(sys.argv) > 1 else "192.0.2.10"
|
||||||
|
V6_LOCAL = sys.argv[2] if len(sys.argv) > 2 else "2001:db8:1f1c:f6::1/64"
|
||||||
|
V6_PEER = sys.argv[3] if len(sys.argv) > 3 else "2001:db8:1f1c:f6::2"
|
||||||
|
SITE6 = sys.argv[4] if len(sys.argv) > 4 else "2001:db8:187e::/48"
|
||||||
|
LOG = "/run/he-sim-api.log"
|
||||||
|
|
||||||
|
def note(msg):
|
||||||
|
with open(LOG, "a") as f:
|
||||||
|
f.write("%s %s\n" % (datetime.datetime.now().isoformat(timespec="seconds"), msg))
|
||||||
|
|
||||||
|
def current_remote():
|
||||||
|
out = subprocess.run(["ip", "tunnel", "show", TUN], capture_output=True, text=True).stdout
|
||||||
|
m = re.search(r"remote ([0-9.]+)", out)
|
||||||
|
return m.group(1) if m else None
|
||||||
|
|
||||||
|
class H(http.server.BaseHTTPRequestHandler):
|
||||||
|
def reply(self, body):
|
||||||
|
b = body.encode()
|
||||||
|
self.send_response(200)
|
||||||
|
self.send_header("Content-Type", "text/plain")
|
||||||
|
self.send_header("Content-Length", str(len(b)))
|
||||||
|
self.end_headers()
|
||||||
|
self.wfile.write(b)
|
||||||
|
|
||||||
|
def handle_update(self, qs):
|
||||||
|
q = urllib.parse.parse_qs(qs)
|
||||||
|
ip = (q.get("myip") or [""])[0]
|
||||||
|
if not ip:
|
||||||
|
note("REFUSED no myip"); return self.reply("nohost")
|
||||||
|
cur = current_remote()
|
||||||
|
if cur == ip:
|
||||||
|
note("nochg %s" % ip); return self.reply("nochg %s" % ip)
|
||||||
|
rc = subprocess.run(["ip", "tunnel", "change", TUN, "mode", "sit",
|
||||||
|
"local", LOCAL, "remote", ip],
|
||||||
|
capture_output=True, text=True)
|
||||||
|
if rc.returncode != 0:
|
||||||
|
# Recreate rather than report a success we did not achieve. This is
|
||||||
|
# the path that a multipoint tunnel takes; keeping it means a stub
|
||||||
|
# that cannot silently no-op.
|
||||||
|
note("change failed (%s) -- recreating" % rc.stderr.strip())
|
||||||
|
subprocess.run(["ip", "tunnel", "del", TUN], check=False)
|
||||||
|
subprocess.run(["ip", "tunnel", "add", TUN, "mode", "sit",
|
||||||
|
"local", LOCAL, "remote", ip, "ttl", "64"], check=False)
|
||||||
|
subprocess.run(["ip", "link", "set", TUN, "up", "mtu", "1480"], check=False)
|
||||||
|
subprocess.run(["ip", "-6", "addr", "replace", V6_LOCAL, "dev", TUN], check=False)
|
||||||
|
subprocess.run(["ip", "-6", "route", "replace", SITE6, "via", V6_PEER,
|
||||||
|
"dev", TUN], check=False)
|
||||||
|
# Verify rather than trust: read the remote back.
|
||||||
|
got = current_remote()
|
||||||
|
if got != ip:
|
||||||
|
note("FAILED to point %s at %s (reads %s)" % (TUN, ip, got))
|
||||||
|
return self.reply("dnserr")
|
||||||
|
note("good %s (was %s)" % (ip, cur))
|
||||||
|
self.reply("good %s" % ip)
|
||||||
|
|
||||||
|
def do_GET(self):
|
||||||
|
u = urllib.parse.urlparse(self.path)
|
||||||
|
if u.path == "/nic/update": self.handle_update(u.query)
|
||||||
|
else: self.reply("badauth")
|
||||||
|
|
||||||
|
def do_POST(self):
|
||||||
|
n = int(self.headers.get("Content-Length") or 0)
|
||||||
|
self.handle_update(self.rfile.read(n).decode())
|
||||||
|
|
||||||
|
def log_message(self, *a): pass
|
||||||
|
|
||||||
|
http.server.HTTPServer(("0.0.0.0", 80), H).serve_forever()
|
||||||
|
'
|
||||||
|
|
||||||
|
up() {
|
||||||
|
log "backbone: teaching each ISP box how to reach the other island"
|
||||||
|
# isp-dhcp owns HE and must be able to answer a router that arrived over
|
||||||
|
# PPPoE, so it needs a route back to that island via the backbone.
|
||||||
|
dhcp_isp "sudo ip route replace $PPPOE_NET via $PPPOE_ISP" \
|
||||||
|
|| die "could not add the PPPoE-island route on isp-dhcp"
|
||||||
|
# isp-pppoe must forward its clients' traffic for HE across the backbone.
|
||||||
|
pppoe_isp "sudo ip route replace $HE_ADDR/32 via $DHCP_ISP" \
|
||||||
|
|| die "could not add the HE route on isp-pppoe"
|
||||||
|
|
||||||
|
log "HE endpoint: $HE_ADDR on isp-dhcp"
|
||||||
|
# A dummy interface, not a loopback alias: `ip tunnel` wants a real local
|
||||||
|
# address and a dummy is the honest way to have one that is not tied to
|
||||||
|
# either WAN segment -- which is the point, HE is neither.
|
||||||
|
dhcp_isp "sudo modprobe dummy 2>/dev/null;
|
||||||
|
sudo ip link add he-lo type dummy 2>/dev/null;
|
||||||
|
sudo ip link set he-lo up;
|
||||||
|
sudo ip addr replace $HE_ADDR/32 dev he-lo"
|
||||||
|
|
||||||
|
log "6in4 tunnel he-sim: local $HE_ADDR, remote set by the API on demand"
|
||||||
|
# A PLACEHOLDER remote, not 0.0.0.0. A sit tunnel created with `remote any`
|
||||||
|
# is multipoint (6rd-shaped), and `ip tunnel change` then refuses to convert
|
||||||
|
# it to point-to-point -- "add tunnel he-sim failed: Invalid argument". The
|
||||||
|
# API's update silently did nothing, so the stub logged "good", production's
|
||||||
|
# he-tunnel-follow logged success, and the tunnel still pointed nowhere.
|
||||||
|
# Created point-to-point from the start, `change` works.
|
||||||
|
dhcp_isp "sudo ip tunnel del he-sim 2>/dev/null;
|
||||||
|
sudo ip tunnel add he-sim mode sit local $HE_ADDR remote $PLACEHOLDER_REMOTE ttl 64;
|
||||||
|
sudo ip link set he-sim up mtu 1480;
|
||||||
|
sudo ip -6 addr replace $HE_LINK6/64 dev he-sim;
|
||||||
|
sudo ip -6 route replace $SITE6 via $RT_LINK6 dev he-sim;
|
||||||
|
sudo sysctl -qw net.ipv6.conf.all.forwarding=1"
|
||||||
|
|
||||||
|
log "stub tunnelbroker API on $HE_ADDR:80"
|
||||||
|
printf '%s' "$API_PY" | dhcp_isp "cat > /tmp/he-sim-api.py"
|
||||||
|
# Launch from a script FILE, not an inline ssh command. The remote login
|
||||||
|
# shell is vbash, and a multi-line inlined `sudo setsid nohup ... &` through
|
||||||
|
# it silently ran nothing at all: no process, no /run/he-sim-api.out, and a
|
||||||
|
# `pgrep -f he-sim-api.py` status check that reported "running" because the
|
||||||
|
# unbracketed pattern matched its OWN ssh command line. Two self-inflicted
|
||||||
|
# illusions stacked on each other.
|
||||||
|
#
|
||||||
|
# This repo already learned this once -- see isp_session_control() in
|
||||||
|
# labsim-pppoe-ha-test.sh, where driving vbash inline made every iteration
|
||||||
|
# of the T4 matrix test the wrong policy while printing the right one.
|
||||||
|
printf '%s\n' \
|
||||||
|
'#!/bin/sh' \
|
||||||
|
'# started detached so it outlives the ssh session that launched it' \
|
||||||
|
'pkill -f "he-sim-api[.]py" 2>/dev/null' \
|
||||||
|
'rm -f /run/he-sim-api.log /run/he-sim-api.out' \
|
||||||
|
"exec setsid python3 /tmp/he-sim-api.py $HE_ADDR '$HE_LINK6/64' $RT_LINK6 $SITE6 >/run/he-sim-api.out 2>&1 </dev/null &" \
|
||||||
|
| dhcp_isp "cat > /tmp/he-sim-start.sh"
|
||||||
|
dhcp_isp "chmod +x /tmp/he-sim-start.sh && sudo /tmp/he-sim-start.sh" >/dev/null
|
||||||
|
# Poll for the bind rather than sleeping a guessed interval.
|
||||||
|
local i probe=""
|
||||||
|
for i in $(seq 1 10); do
|
||||||
|
sleep 1
|
||||||
|
probe="$(dhcp_isp "curl -sS --max-time 3 'http://$HE_ADDR/nic/update' 2>&1")"
|
||||||
|
[ "$probe" = nohost ] && break
|
||||||
|
done
|
||||||
|
case "$probe" in
|
||||||
|
nohost) log "API answering (returned 'nohost' for a call with no myip -- correct)" ;;
|
||||||
|
*) die "stub API not answering on $HE_ADDR:80 (got: ${probe:-<nothing>})" ;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
log "up. Router side is NOT configured by this script -- that is real VyOS"
|
||||||
|
log "config and belongs to the matrix; see labsim-ipv6-ha-test.sh --setup."
|
||||||
|
}
|
||||||
|
|
||||||
|
down() {
|
||||||
|
log "tearing down"
|
||||||
|
dhcp_isp "sudo pkill -f 'he-sim-api[.]py' 2>/dev/null;
|
||||||
|
sudo ip tunnel del he-sim 2>/dev/null;
|
||||||
|
sudo ip link del he-lo 2>/dev/null;
|
||||||
|
sudo ip route del $PPPOE_NET via $PPPOE_ISP 2>/dev/null" >/dev/null
|
||||||
|
pppoe_isp "sudo ip route del $HE_ADDR/32 via $DHCP_ISP 2>/dev/null" >/dev/null
|
||||||
|
log "down"
|
||||||
|
}
|
||||||
|
|
||||||
|
status() {
|
||||||
|
printf ' HE address : %s\n' "$(dhcp_isp "ip -4 -br addr show he-lo 2>/dev/null | awk '{print \$3}'" || echo '<absent>')"
|
||||||
|
printf ' he-sim : %s\n' "$(dhcp_isp "ip tunnel show he-sim 2>/dev/null" || echo '<absent>')"
|
||||||
|
printf ' he-sim v6 : %s\n' "$(dhcp_isp "ip -6 -br addr show he-sim 2>/dev/null | awk '{print \$3}'" || echo '-')"
|
||||||
|
# Bracketed so the pattern cannot match the ssh command line carrying it --
|
||||||
|
# unbracketed, this reported "running" while nothing was listening at all.
|
||||||
|
printf ' API : %s\n' "$(dhcp_isp "pgrep -f 'he-sim-api[.]py' >/dev/null && echo running || echo stopped")"
|
||||||
|
printf ' API bound : %s\n' "$(dhcp_isp "curl -sS --max-time 3 'http://$HE_ADDR/nic/update' 2>/dev/null" || echo 'NOT ANSWERING')"
|
||||||
|
printf ' API calls : %s\n' "$(dhcp_isp "grep -c . /run/he-sim-api.log 2>/dev/null" || echo 0)"
|
||||||
|
printf ' route back : %s\n' "$(dhcp_isp "ip route show $PPPOE_NET 2>/dev/null" || echo '<none>')"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Count of endpoint-CHANGING calls. `nochg` does not count: production's
|
||||||
|
# he-tunnel-follow re-sends the same address happily and HE treats it as a
|
||||||
|
# no-op, so only a real move is evidence that something re-pointed the tunnel.
|
||||||
|
calls() { dhcp_isp "grep -c ' good ' /run/he-sim-api.log 2>/dev/null" | tr -d ' \n'; }
|
||||||
|
|
||||||
|
# Point the endpoint at an address WITHOUT going through the API, and without
|
||||||
|
# counting as an API call.
|
||||||
|
#
|
||||||
|
# Needed because rebuilding the sim endpoint resets its remote, while the
|
||||||
|
# routers' he-tunnel-follow still reads "in sync" and therefore never re-asserts
|
||||||
|
# -- it only calls HE when its own LOCAL source changes, and has no way to learn
|
||||||
|
# that the far end drifted. The real HE does not forget, so this is sim
|
||||||
|
# bookkeeping, not a behaviour production needs. Keeping it out of the call
|
||||||
|
# counter is the point: the matrix asserts on that counter.
|
||||||
|
point() {
|
||||||
|
local ip="$1"
|
||||||
|
[ -n "$ip" ] || die "usage: $0 point <ipv4>"
|
||||||
|
dhcp_isp "sudo ip tunnel change he-sim mode sit local $HE_ADDR remote $ip 2>/dev/null \
|
||||||
|
|| { sudo ip tunnel del he-sim 2>/dev/null;
|
||||||
|
sudo ip tunnel add he-sim mode sit local $HE_ADDR remote $ip ttl 64;
|
||||||
|
sudo ip link set he-sim up mtu 1480;
|
||||||
|
sudo ip -6 addr replace $HE_LINK6/64 dev he-sim;
|
||||||
|
sudo ip -6 route replace $SITE6 via $RT_LINK6 dev he-sim; }"
|
||||||
|
local got
|
||||||
|
got="$(dhcp_isp "ip tunnel show he-sim 2>/dev/null | sed -nE 's/.* remote ([0-9.]+).*/\\1/p'" | tr -d ' \n')"
|
||||||
|
[ "$got" = "$ip" ] || die "endpoint still points at ${got:-nothing}, wanted $ip"
|
||||||
|
log "endpoint now points at $ip"
|
||||||
|
}
|
||||||
|
|
||||||
|
case "${1:-status}" in
|
||||||
|
up) up ;;
|
||||||
|
down) down ;;
|
||||||
|
status) status ;;
|
||||||
|
calls) calls; echo ;;
|
||||||
|
point) point "${2:-}" ;;
|
||||||
|
*) die "usage: $0 {up|down|status|calls}" ;;
|
||||||
|
esac
|
||||||
430
labsim/labsim-ipv6-ha-test.sh
Executable file
430
labsim/labsim-ipv6-ha-test.sh
Executable file
@@ -0,0 +1,430 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# Does IPv6 follow VRRP mastership, and does it do so WITHOUT touching HE?
|
||||||
|
#
|
||||||
|
# The WAN became HA on 2026-09-06 and IPv6 did not follow it. Nothing caught
|
||||||
|
# that, because nothing tested it: wan-drill measured IPv4 only, and PPPOE-HA.md
|
||||||
|
# recorded "IPv6 stayed up at 15.5ms" from a reading taken outside the failover
|
||||||
|
# window. This is the matrix that would have caught it.
|
||||||
|
#
|
||||||
|
# TWO INVARIANTS, checked independently of any individual test:
|
||||||
|
#
|
||||||
|
# 1. At most ONE router ever has a live tunnel. An UP tunnel on a box that
|
||||||
|
# does not own the source address is not harmless -- see V4, it is a
|
||||||
|
# BLACKHOLE that will happily attract the v6 default route.
|
||||||
|
# 2. A ROUTER-level failover calls the HE API ZERO times. The 10 gig address
|
||||||
|
# is bound to a cloned MAC and follows the VIP to the other box unchanged,
|
||||||
|
# so there is nothing to tell HE. A non-zero count means something
|
||||||
|
# re-pointed the tunnel at a PPPoE address -- which the ISP re-issues on
|
||||||
|
# every dial, so it would be wrong within minutes.
|
||||||
|
#
|
||||||
|
# KNOWN SIM GAP, 2026-09-06 -- read this before believing a v6_online failure.
|
||||||
|
# The MECHANISM is proven here: tun0 up on the master and down on the backup,
|
||||||
|
# radvd following mastership, he-tunnel-follow's master guard, its hysteresis,
|
||||||
|
# its HE call and the 1480->1472 MTU switch, and VLAN 9 hosts autoconfiguring
|
||||||
|
# from the RA (observed: real SLAAC traffic from 2001:db8:187e:9::/64 arriving
|
||||||
|
# at the endpoint encapsulated).
|
||||||
|
#
|
||||||
|
# What is NOT yet proven is the end-to-end v6 DATAPATH, because the sim's
|
||||||
|
# "internet" is asymmetric: 6in4 packets from the PPPoE island reach the
|
||||||
|
# endpoint with an outer source of 192.168.122.1 -- the libvirt host's NAT --
|
||||||
|
# so HE's replies go back to the tunnel remote by a path with no NAT state and
|
||||||
|
# are lost. Forward works, return does not.
|
||||||
|
#
|
||||||
|
# That is a topology fault in the scaffold, not in the thing under test. Fixing
|
||||||
|
# it means giving the two ISP islands a real transit path that does not traverse
|
||||||
|
# libvirt NAT. Until then, treat v6_online failures as UNPROVEN rather than as
|
||||||
|
# evidence the design is wrong -- and do not let that ambiguity leak into
|
||||||
|
# production sign-off, which is exactly the mistake wan-drill made by asserting
|
||||||
|
# "IPv6 stayed up" from a reading taken outside the window.
|
||||||
|
#
|
||||||
|
# Requires the fake HE endpoint: ./labsim-he-endpoint.sh up
|
||||||
|
#
|
||||||
|
# ./labsim-ipv6-ha-test.sh --setup configure the router side (once)
|
||||||
|
# ./labsim-ipv6-ha-test.sh --list
|
||||||
|
# ./labsim-ipv6-ha-test.sh V1
|
||||||
|
# ./labsim-ipv6-ha-test.sh --all
|
||||||
|
set -uo pipefail
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||||
|
R1="${R1:-172.31.1.252}"; R2="${R2:-172.31.1.253}"
|
||||||
|
VIP="${VIP:-172.31.1.1}"
|
||||||
|
LAN9="${LAN9:-172.31.9.10}" # VLAN 9 client, for RA tests
|
||||||
|
HE_ADDR="${HE_ADDR:-192.0.2.10}"
|
||||||
|
HE_LINK6="${HE_LINK6:-2001:db8:1f1c:f6::1}"
|
||||||
|
RT_LINK6="${RT_LINK6:-2001:db8:1f1c:f6::2}"
|
||||||
|
V9_PREFIX="${V9_PREFIX:-2001:db8:187e:9}"
|
||||||
|
# Mirrors production: one MAC, one lease, whichever router holds the VIP.
|
||||||
|
WAN_MAC="${WAN_MAC:-02:9f:c2:12:9b:4f}"
|
||||||
|
PW="${VYOS_PW:-vyos}"; LANPW="${LANPW:-labsim}"
|
||||||
|
EVID="$SCRIPT_DIR/ipv6-ha-evidence"
|
||||||
|
HE="$SCRIPT_DIR/labsim-he-endpoint.sh"
|
||||||
|
|
||||||
|
SSH=(-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null
|
||||||
|
-o LogLevel=ERROR -o ConnectTimeout=6 -o PreferredAuthentications=password)
|
||||||
|
r() { timeout 45 sshpass -p "$PW" ssh "${SSH[@]}" "vyos@$1" "${@:2}" 2>/dev/null; }
|
||||||
|
# The Alpine LAN VMs do not offer `password` auth -- reusing the routers' option
|
||||||
|
# set makes ssh exit 255 before running anything, which reads as "the network is
|
||||||
|
# broken". Same trap as lan() in labsim-pppoe-ha-test.sh.
|
||||||
|
LAN_SSH=(-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null
|
||||||
|
-o LogLevel=ERROR -o ConnectTimeout=6)
|
||||||
|
lan9() { timeout 30 sshpass -p "$LANPW" ssh "${LAN_SSH[@]}" "root@$LAN9" "$@" 2>/dev/null; }
|
||||||
|
|
||||||
|
log() { printf '\033[36m==>\033[0m %s\n' "$*"; }
|
||||||
|
pass() { printf ' \033[32mPASS\033[0m %s\n' "$*"; }
|
||||||
|
fail() { printf ' \033[31mFAIL\033[0m %s\n' "$*"; FAILED=$((FAILED+1)); }
|
||||||
|
warn() { printf ' \033[33mWARN\033[0m %s\n' "$*"; }
|
||||||
|
FAILED=0
|
||||||
|
|
||||||
|
# --- observations ----------------------------------------------------------
|
||||||
|
# A destroyed or unreachable router is emphatically NOT holding the tunnel, but
|
||||||
|
# ssh returns an EMPTY string, and `[ "" = 0 ]` is false -- the pppoe matrix hung
|
||||||
|
# on exactly this waiting for a dead box to report zero. Default everything to 0.
|
||||||
|
tun_state() { local v; v="$(r "$1" "ip -br link show tun0 2>/dev/null | awk '{print \$2}'" | tr -d ' \n')"; echo "${v:-absent}"; }
|
||||||
|
tun_src() { r "$1" 'ip tunnel show tun0 2>/dev/null | sed -nE "s/.* local ([0-9.]+).*/\1/p"' | tr -d ' \n'; }
|
||||||
|
# Whichever router currently holds the 10 gig lease -- under the cloned MAC only
|
||||||
|
# one ever does. Read rather than assumed: the address changes when the MAC does.
|
||||||
|
tengig_addr() { local h a; for h in "$R1" "$R2"; do
|
||||||
|
a="$(r "$h" 'ip -4 addr show bond0.53 2>/dev/null | sed -nE "s/.*inet ([0-9.]+).*/\1/p"' | tr -d ' \n')"
|
||||||
|
[ -n "$a" ] && { echo "$a"; return; }
|
||||||
|
done; echo ""; }
|
||||||
|
tun_mtu() { local v; v="$(r "$1" 'cat /sys/class/net/tun0/mtu 2>/dev/null' | tr -d ' \n')"; echo "${v:-0}"; }
|
||||||
|
holder() { for h in "$R1" "$R2"; do
|
||||||
|
[ "$(r "$h" "ip -4 -o addr show | grep -c ' ${VIP}/'" | tr -d ' \n')" != 0 ] \
|
||||||
|
&& { echo "$h"; return; }; done; echo none; }
|
||||||
|
# Ask the ROUTER, not a client: a client can be answered by the wrong path.
|
||||||
|
v6_online() { [ "$(r "$1" "ping -6 -c1 -W3 $HE_LINK6 >/dev/null 2>&1 && echo y" | tr -d ' \n')" = y ]; }
|
||||||
|
radvd_on() { [ "$(r "$1" 'systemctl is-active radvd 2>/dev/null' | tr -d ' \n')" = active ]; }
|
||||||
|
he_calls() { timeout 60 "$HE" calls | tr -d ' \n'; }
|
||||||
|
|
||||||
|
# How many routers have a tunnel that is UP. Invariant 1's numerator.
|
||||||
|
tun_holders() { local n=0 h; for h in "$R1" "$R2"; do
|
||||||
|
case "$(tun_state "$h")" in UP|UNKNOWN) n=$((n+1)) ;; esac
|
||||||
|
done; echo "$n"; }
|
||||||
|
|
||||||
|
check_invariants() {
|
||||||
|
local t ok=0
|
||||||
|
t="$(tun_holders)"
|
||||||
|
[ "${t:-0}" -le 1 ] || { fail "INVARIANT: $t routers have tun0 up"; ok=1; }
|
||||||
|
return $ok
|
||||||
|
}
|
||||||
|
|
||||||
|
save_evidence() {
|
||||||
|
local name="$1"; local d="$EVID/$name"; mkdir -p "$d"
|
||||||
|
{ echo "=== $(date -Is) ==="
|
||||||
|
echo "--- HE endpoint ---"; timeout 60 "$HE" status
|
||||||
|
for h in "$R1" "$R2"; do echo "--- $h ---"
|
||||||
|
r "$h" 'sudo /config/vrrp-wan-reconcile --status 2>/dev/null
|
||||||
|
ip -br link show tun0 2>/dev/null; ip tunnel show tun0 2>/dev/null
|
||||||
|
ip -6 route show default; systemctl is-active radvd
|
||||||
|
sudo journalctl -t vrrp-wan -t he-tunnel-follow -n 10 --no-pager'
|
||||||
|
done; } > "$d/state.txt" 2>&1
|
||||||
|
log "evidence -> ipv6-ha-evidence/$name/"
|
||||||
|
}
|
||||||
|
|
||||||
|
# --- setup ------------------------------------------------------------------
|
||||||
|
# Applied through REAL VyOS config, unlike the ISP-side scaffold, because "will
|
||||||
|
# VyOS accept this?" is one of the questions being asked.
|
||||||
|
vyos_apply() { # host, then set-lines on stdin
|
||||||
|
local h="$1"
|
||||||
|
{ printf '#!/bin/vbash\nsource /opt/vyatta/etc/functions/script-template\nconfigure\n'
|
||||||
|
cat
|
||||||
|
printf 'commit\nsave\nexit\n'
|
||||||
|
} | timeout 90 sshpass -p "$PW" ssh "${SSH[@]}" "vyos@$h" \
|
||||||
|
'cat > /tmp/v6-apply.sh && chmod +x /tmp/v6-apply.sh && sudo /tmp/v6-apply.sh' 2>&1 | tail -3
|
||||||
|
# `vbash -c` never starts a config session and commit fails to stderr, which
|
||||||
|
# a helper like this discards -- the T4 matrix in labsim-pppoe-ha-test.sh ran
|
||||||
|
# its whole policy sweep against the default while printing the mode it
|
||||||
|
# thought it was testing. Always read the value back.
|
||||||
|
}
|
||||||
|
|
||||||
|
setup() {
|
||||||
|
log "--setup: router-side IPv6, both routers"
|
||||||
|
|
||||||
|
# THE CLONED MAC, first and on its own. Production pins f0:9f:c2:12:9b:4f on
|
||||||
|
# vif 53 so the 10 gig lease follows the VIP and the tunnel source is the
|
||||||
|
# SAME address on either box. The sim never had it -- each router took its
|
||||||
|
# own lease -- so the sim could not reproduce the one property the whole
|
||||||
|
# IPv6-HA design leans on, and invariant 2 would have been untestable here.
|
||||||
|
local h pref v9
|
||||||
|
for h in "$R1" "$R2"; do
|
||||||
|
log " $h: pinning the cloned WAN MAC $WAN_MAC"
|
||||||
|
vyos_apply "$h" <<EOF
|
||||||
|
set interfaces bonding bond0 vif 53 mac '$WAN_MAC'
|
||||||
|
EOF
|
||||||
|
done
|
||||||
|
|
||||||
|
# A new MAC means a NEW lease, so the tunnel source cannot be hardcoded --
|
||||||
|
# discover it. Hardcoding the pre-change address here would have configured
|
||||||
|
# every tunnel with a source neither router owns, i.e. the V4 blackhole, on
|
||||||
|
# both boxes, while the matrix reported setup success.
|
||||||
|
local src="" i
|
||||||
|
for i in $(seq 1 30); do
|
||||||
|
src="$(tengig_addr)"; [ -n "$src" ] && break
|
||||||
|
sleep 5
|
||||||
|
done
|
||||||
|
[ -n "$src" ] || { fail "no router took a 10 gig lease after the MAC change -- cannot set a tunnel source"; return 1; }
|
||||||
|
log " 10 gig lease under the cloned MAC: $src"
|
||||||
|
|
||||||
|
# Credentials pointing at the stub. HE_UPDATE_URL is read AFTER the secrets
|
||||||
|
# file is sourced, so putting it here overrides the production default
|
||||||
|
# without the script needing a sim-specific branch.
|
||||||
|
for h in "$R1" "$R2"; do
|
||||||
|
printf 'HE_USER=sim\nHE_UPDATE_KEY=sim\nHE_TUNNEL_ID=1\nHE_UPDATE_URL=http://%s/nic/update\n' "$HE_ADDR" \
|
||||||
|
| timeout 30 sshpass -p "$PW" ssh "${SSH[@]}" "vyos@$h" \
|
||||||
|
'cat > /tmp/he-secrets && sudo install -o root -g vyattacfg -m 0640 /tmp/he-secrets /config/he-secrets' >/dev/null
|
||||||
|
done
|
||||||
|
|
||||||
|
for h in "$R1" "$R2"; do
|
||||||
|
[ "$h" = "$R1" ] && { pref=high; v9=1; } || { pref=low; v9=2; }
|
||||||
|
log " $h (RA preference $pref, bond0.9 ::${v9})"
|
||||||
|
vyos_apply "$h" <<EOF
|
||||||
|
set interfaces tunnel tun0 encapsulation 'sit'
|
||||||
|
set interfaces tunnel tun0 source-address '$src'
|
||||||
|
set interfaces tunnel tun0 remote '$HE_ADDR'
|
||||||
|
set interfaces tunnel tun0 address '$RT_LINK6/64'
|
||||||
|
set interfaces tunnel tun0 mtu '1480'
|
||||||
|
set protocols static route6 ::/0 next-hop '$HE_LINK6'
|
||||||
|
set interfaces bonding bond0 vif 9 address '${V9_PREFIX}::${v9}/64'
|
||||||
|
set service router-advert interface bond0.9 prefix ${V9_PREFIX}::/64 preferred-lifetime '604800'
|
||||||
|
set service router-advert interface bond0.9 link-mtu '1472'
|
||||||
|
set service router-advert interface bond0.9 default-preference '$pref'
|
||||||
|
set system task-scheduler task he-tunnel-follow executable path '/config/he-tunnel-follow'
|
||||||
|
set system task-scheduler task he-tunnel-follow executable arguments 'run'
|
||||||
|
set system task-scheduler task he-tunnel-follow interval '1m'
|
||||||
|
EOF
|
||||||
|
done
|
||||||
|
# RA link-mtu is 1472, the PPPoE figure, on BOTH -- deliberately not 1480.
|
||||||
|
# It cannot be reconciled at runtime (it needs a commit, and the tunnel plane
|
||||||
|
# is commit-free on purpose), so advertise the lower of the two paths and be
|
||||||
|
# correct on either WAN. Production pinned 1480 and was wrong whenever the
|
||||||
|
# WAN fell back.
|
||||||
|
log " setup done -- run V0 to check the baseline"
|
||||||
|
}
|
||||||
|
|
||||||
|
# --- tests ------------------------------------------------------------------
|
||||||
|
V0() { # baseline
|
||||||
|
log "V0 baseline: the VIP holder owns the tunnel, the backup does not"
|
||||||
|
local h o; h="$(holder)"; o=$([ "$h" = "$R1" ] && echo "$R2" || echo "$R1")
|
||||||
|
[ "$h" = none ] && { fail "no VIP holder"; return; }
|
||||||
|
case "$(tun_state "$h")" in UP|UNKNOWN) pass "master $h has tun0 up" ;;
|
||||||
|
*) fail "master $h tun0 is $(tun_state "$h")" ;; esac
|
||||||
|
case "$(tun_state "$o")" in DOWN|absent) pass "backup $o tun0 is $(tun_state "$o")" ;;
|
||||||
|
*) fail "backup $o tun0 is $(tun_state "$o") -- it should be held down" ;; esac
|
||||||
|
v6_online "$h" && pass "master reaches HE over v6" || fail "master has no IPv6"
|
||||||
|
radvd_on "$h" && pass "master is advertising on VLAN 9" || fail "master radvd not running"
|
||||||
|
radvd_on "$o" && fail "backup is ALSO advertising -- two default routers on VLAN 9" \
|
||||||
|
|| pass "backup is not advertising"
|
||||||
|
check_invariants
|
||||||
|
save_evidence V0-baseline
|
||||||
|
}
|
||||||
|
|
||||||
|
V1() { # clean failover: v6 follows, and HE is never called
|
||||||
|
log "V1 clean failover: IPv6 follows, HE API untouched"
|
||||||
|
local from to t0 before after i
|
||||||
|
from="$(holder)"; to=$([ "$from" = "$R1" ] && echo "$R2" || echo "$R1")
|
||||||
|
before="$(he_calls)"
|
||||||
|
log " master=$from -> expecting $to (HE calls so far: ${before:-0})"
|
||||||
|
t0=$(date +%s)
|
||||||
|
r "$from" 'sudo mkdir -p /run/vrrp-wan && sudo touch /run/vrrp-wan/force-fault'
|
||||||
|
local took=""
|
||||||
|
for i in $(seq 1 36); do
|
||||||
|
sleep 5
|
||||||
|
[ "$(holder)" = "$to" ] && v6_online "$to" && { took=$(( $(date +%s) - t0 )); break; }
|
||||||
|
done
|
||||||
|
[ -n "$took" ] && pass "IPv6 reached $to in ${took}s" \
|
||||||
|
|| fail "IPv6 never followed to $to within 180s"
|
||||||
|
case "$(tun_state "$from")" in DOWN|absent) pass "$from released its tunnel" ;;
|
||||||
|
*) fail "$from still has tun0 $(tun_state "$from") -- blackhole risk" ;; esac
|
||||||
|
after="$(he_calls)"
|
||||||
|
# THE invariant this test exists for.
|
||||||
|
[ "${after:-0}" = "${before:-0}" ] \
|
||||||
|
&& pass "HE API not called (${after:-0} total) -- the address followed the MAC" \
|
||||||
|
|| fail "HE API called $(( ${after:-0} - ${before:-0} )) time(s) during a ROUTER failover"
|
||||||
|
check_invariants
|
||||||
|
save_evidence V1-clean-failover
|
||||||
|
r "$from" 'sudo rm -f /run/vrrp-wan/force-fault'
|
||||||
|
sleep 40
|
||||||
|
}
|
||||||
|
|
||||||
|
V2() { # 10 gig down on the master: HE must be told, exactly once
|
||||||
|
log "V2 10 gig down: he-tunnel-follow re-points the tunnel and tells HE"
|
||||||
|
local h before after mtu i ok=no
|
||||||
|
h="$(holder)"; before="$(he_calls)"
|
||||||
|
local tengig; tengig="$(tengig_addr)"
|
||||||
|
r "$h" 'sudo ip link set bond0.53 down'
|
||||||
|
# he-tunnel-follow runs on a 1m task-scheduler with a 2-tick hysteresis, so
|
||||||
|
# allow well past 2 minutes before calling it a failure.
|
||||||
|
for i in $(seq 1 30); do
|
||||||
|
sleep 10
|
||||||
|
[ "$(tun_src "$h")" != "$tengig" ] && { ok=yes; break; }
|
||||||
|
done
|
||||||
|
[ "$ok" = yes ] && pass "tunnel source moved to $(tun_src "$h") after $((i*10))s" \
|
||||||
|
|| fail "tunnel source never left the 10 gig address"
|
||||||
|
mtu="$(tun_mtu "$h")"
|
||||||
|
[ "$mtu" = 1472 ] && pass "MTU dropped to 1472 for the PPPoE path" \
|
||||||
|
|| fail "MTU is $mtu, want 1472 -- large transfers will hang"
|
||||||
|
after="$(he_calls)"
|
||||||
|
[ "$(( ${after:-0} - ${before:-0} ))" -ge 1 ] \
|
||||||
|
&& pass "HE API called $(( ${after:-0} - ${before:-0} )) time(s), as it must be here" \
|
||||||
|
|| fail "HE was never told -- it still points at an address this box no longer has"
|
||||||
|
v6_online "$h" && pass "IPv6 still up over PPPoE" || fail "IPv6 down on the PPPoE path"
|
||||||
|
save_evidence V2-tengig-down
|
||||||
|
r "$h" 'sudo ip link set bond0.53 up'
|
||||||
|
sleep 60
|
||||||
|
}
|
||||||
|
|
||||||
|
V3() { # a cold backup must not advertise, dial, or blackhole
|
||||||
|
log "V3 cold backup: no tunnel, no RA, no HE call"
|
||||||
|
local h o before after
|
||||||
|
h="$(holder)"; o=$([ "$h" = "$R1" ] && echo "$R2" || echo "$R1")
|
||||||
|
before="$(he_calls)"
|
||||||
|
r "$o" 'sudo systemctl restart vrrp-wan-reconcile.service' >/dev/null
|
||||||
|
sleep 20
|
||||||
|
case "$(tun_state "$o")" in DOWN|absent) pass "backup tunnel stays $(tun_state "$o")" ;;
|
||||||
|
*) fail "backup brought tun0 up while not holding the VIP" ;; esac
|
||||||
|
radvd_on "$o" && fail "backup is advertising on VLAN 9" || pass "backup is silent on VLAN 9"
|
||||||
|
after="$(he_calls)"
|
||||||
|
[ "${after:-0}" = "${before:-0}" ] && pass "backup made no HE call" \
|
||||||
|
|| fail "the BACKUP called the HE API -- it would point HE at its own idle line"
|
||||||
|
save_evidence V3-cold-backup
|
||||||
|
}
|
||||||
|
|
||||||
|
V4() { # the assumption the production override was built on
|
||||||
|
log "V4 a tunnel whose source-address is absent: what does VyOS actually do?"
|
||||||
|
# The production override says such a tunnel "would simply stay down", and
|
||||||
|
# treats that as the reason it was safe to leave IPv6 single-homed. Measured
|
||||||
|
# on the sim backup 2026-09-06: the commit SUCCEEDS and the link comes up
|
||||||
|
# anyway -- it is a blackhole, not an inert node. That is why the runtime
|
||||||
|
# gate is load-bearing rather than a nicety, exactly like the PPPoE gate.
|
||||||
|
local o h; h="$(holder)"; o=$([ "$h" = "$R1" ] && echo "$R2" || echo "$R1")
|
||||||
|
r "$o" 'sudo ip link set tun0 up' >/dev/null; sleep 3
|
||||||
|
case "$(tun_state "$o")" in
|
||||||
|
UP|UNKNOWN) pass "confirmed: VyOS leaves it UP with no source address (blackhole)" ;;
|
||||||
|
*) warn "this VyOS version keeps it $(tun_state "$o") -- the override's assumption holds here; re-check the production version before relying on it" ;;
|
||||||
|
esac
|
||||||
|
v6_online "$o" && fail "the backup somehow reached HE -- two live tunnels" \
|
||||||
|
|| pass "and it carries nothing, as expected"
|
||||||
|
# Hand it straight back to the reconciler rather than leaving it up.
|
||||||
|
r "$o" 'sudo systemctl restart vrrp-wan-reconcile.service' >/dev/null
|
||||||
|
sleep 15
|
||||||
|
case "$(tun_state "$o")" in DOWN|absent) pass "the reconciler put it back down" ;;
|
||||||
|
*) fail "the reconciler did NOT re-close the gate -- this is the load-bearing bit" ;; esac
|
||||||
|
save_evidence V4-absent-source-address
|
||||||
|
}
|
||||||
|
|
||||||
|
V5() { # never two live tunnels, even mid-transition
|
||||||
|
log "V5 both routers momentarily master: never two live tunnels"
|
||||||
|
local from to i worst=0 n
|
||||||
|
from="$(holder)"; to=$([ "$from" = "$R1" ] && echo "$R2" || echo "$R1")
|
||||||
|
r "$from" 'sudo mkdir -p /run/vrrp-wan && sudo touch /run/vrrp-wan/force-fault'
|
||||||
|
# Sample THROUGH the transition rather than at the ends. The interesting
|
||||||
|
# window is the one where both boxes briefly think they are in charge.
|
||||||
|
for i in $(seq 1 24); do
|
||||||
|
n="$(tun_holders)"; [ "${n:-0}" -gt "$worst" ] && worst="$n"
|
||||||
|
sleep 5
|
||||||
|
done
|
||||||
|
[ "$worst" -le 1 ] && pass "at most $worst live tunnel throughout the transition" \
|
||||||
|
|| fail "saw $worst live tunnels at once -- HE would receive two claimants"
|
||||||
|
r "$from" 'sudo rm -f /run/vrrp-wan/force-fault'
|
||||||
|
save_evidence V5-transition-invariant
|
||||||
|
sleep 40
|
||||||
|
}
|
||||||
|
|
||||||
|
V6() { # RA deprecation: does a VLAN 9 host drop the dead gateway?
|
||||||
|
log "V6 RA deprecation: the client must stop using a demoted router"
|
||||||
|
local h before
|
||||||
|
h="$(holder)"
|
||||||
|
before="$(lan9 'ip -6 route show default 2>/dev/null | head -1')"
|
||||||
|
if [ -z "$before" ]; then
|
||||||
|
warn "VLAN 9 client has no IPv6 default route -- SLAAC may not have run; skipping"
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
log " client default was: $before"
|
||||||
|
r "$h" 'sudo mkdir -p /run/vrrp-wan && sudo touch /run/vrrp-wan/force-fault'
|
||||||
|
sleep 45
|
||||||
|
local after; after="$(lan9 'ip -6 route show default 2>/dev/null | head -1')"
|
||||||
|
log " client default now: ${after:-<none>}"
|
||||||
|
# radvd emits a final RA with router-lifetime 0 on a graceful stop. Either
|
||||||
|
# the client moved to the new master or it dropped the route entirely; both
|
||||||
|
# are correct. Still pointing at the demoted box is not.
|
||||||
|
if [ "$after" = "$before" ]; then
|
||||||
|
fail "client still points at the demoted router -- the farewell RA did not land"
|
||||||
|
else
|
||||||
|
pass "client stopped using the demoted router"
|
||||||
|
fi
|
||||||
|
r "$h" 'sudo rm -f /run/vrrp-wan/force-fault'
|
||||||
|
save_evidence V6-ra-deprecation
|
||||||
|
sleep 40
|
||||||
|
}
|
||||||
|
|
||||||
|
V7() { # replay the 2026-09-06 near-miss, but slowly
|
||||||
|
log "V7 slow WAN restore: does the hysteresis still hold?"
|
||||||
|
# On 2026-09-06 vif53-pin-boot-disable bounced the 10 gig, he-tunnel-follow
|
||||||
|
# ticked once and saw the PPPoE address, and vyos-failover restored the 10
|
||||||
|
# gig 22 SECONDS before the second tick would have pushed HE at an address
|
||||||
|
# Vodafone reissues on every dial. 22s of margin is not a safety property.
|
||||||
|
# Here the restore is deliberately slower than the hysteresis window.
|
||||||
|
local h before after tengig
|
||||||
|
h="$(holder)"; before="$(he_calls)"; tengig="$(tengig_addr)"
|
||||||
|
r "$h" 'sudo ip link set bond0.53 down'
|
||||||
|
sleep 200 # > 2 ticks of a 1m scheduler
|
||||||
|
r "$h" 'sudo ip link set bond0.53 up'
|
||||||
|
sleep 90
|
||||||
|
after="$(he_calls)"
|
||||||
|
if [ "$(( ${after:-0} - ${before:-0} ))" -ge 1 ]; then
|
||||||
|
warn "HE was updated $(( ${after:-0} - ${before:-0} )) time(s) and then had to move back -- this is the 2026-09-06 shape, now reproduced deliberately. Either widen HYSTERESIS or make vif53-pin-boot-disable hold he-tunnel-follow off for the bounce."
|
||||||
|
else
|
||||||
|
pass "no HE churn across a slow WAN bounce"
|
||||||
|
fi
|
||||||
|
# Whatever happened, the tunnel must end up back on the 10 gig.
|
||||||
|
local i
|
||||||
|
for i in $(seq 1 24); do
|
||||||
|
[ "$(tun_src "$h")" = "$tengig" ] && break
|
||||||
|
sleep 10
|
||||||
|
done
|
||||||
|
[ "$(tun_src "$h")" = "$tengig" ] && pass "tunnel returned to the 10 gig address" \
|
||||||
|
|| fail "tunnel stuck on $(tun_src "$h") after the 10 gig came back"
|
||||||
|
save_evidence V7-slow-restore
|
||||||
|
}
|
||||||
|
|
||||||
|
preflight() {
|
||||||
|
log "preflight"
|
||||||
|
local rc=0
|
||||||
|
[ "$(timeout 60 "$HE" status | grep -c 'API bound : nohost')" = 1 ] \
|
||||||
|
|| { fail "the fake HE endpoint is not answering -- run ./labsim-he-endpoint.sh up"; rc=1; }
|
||||||
|
local h
|
||||||
|
for h in "$R1" "$R2"; do
|
||||||
|
[ "$(tun_state "$h")" = absent ] \
|
||||||
|
&& { fail "$h has no tun0 -- run --setup first"; rc=1; }
|
||||||
|
[ "$(r "$h" 'systemctl is-active vrrp-wan-reconcile.timer')" = active ] \
|
||||||
|
|| { fail "$h vrrp-wan-reconcile.timer not active"; rc=1; }
|
||||||
|
# The reconciler must be the version that knows about the v6 plane, or
|
||||||
|
# every result below measures the OLD behaviour while printing the new
|
||||||
|
# test names -- the failure mode this repo has already been bitten by.
|
||||||
|
r "$h" 'grep -q v6_take /config/vrrp-wan-reconcile' \
|
||||||
|
|| { fail "$h has a vrrp-wan-reconcile with no IPv6 plane -- run migration/vrrp-wan-install"; rc=1; }
|
||||||
|
done
|
||||||
|
# Line the sim's endpoint up with whoever actually holds the tunnel. The real
|
||||||
|
# HE remembers where it was pointed; a rebuilt sim endpoint does not, and
|
||||||
|
# he-tunnel-follow will not re-assert because from its side nothing changed.
|
||||||
|
# Does NOT count as an API call, so the invariant-2 assertions stay honest.
|
||||||
|
local m src
|
||||||
|
m="$(holder)"; src="$(tun_src "$m")"
|
||||||
|
if [ -n "$src" ]; then
|
||||||
|
timeout 60 "$HE" point "$src" >/dev/null 2>&1 \
|
||||||
|
|| { fail "could not point the sim HE endpoint at $src"; rc=1; }
|
||||||
|
fi
|
||||||
|
[ "$rc" -eq 0 ] && pass "HE endpoint answering and pointed at $src, both routers have tun0 and a v6-aware reconciler"
|
||||||
|
return $rc
|
||||||
|
}
|
||||||
|
|
||||||
|
case "${1:---all}" in
|
||||||
|
--list) echo "V0 baseline | V1 clean failover | V2 10gig-down | V3 cold backup | V4 absent source-address | V5 transition invariant | V6 RA deprecation | V7 slow restore"; exit 0 ;;
|
||||||
|
--setup) setup; exit 0 ;;
|
||||||
|
--all) preflight || exit 1; V0; V1; V2; V3; V4; V5; V6; V7 ;;
|
||||||
|
*) preflight || exit 1; "$1" ;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
echo
|
||||||
|
[ "$FAILED" -eq 0 ] && { echo "ALL PASS"; exit 0; }
|
||||||
|
echo "$FAILED check(s) FAILED"; exit 1
|
||||||
@@ -178,11 +178,33 @@ under all conditions.
|
|||||||
|
|
||||||
**Vodafone hands out a different IPv4 on every dial**: `83.106.5.72` →
|
**Vodafone hands out a different IPv4 on every dial**: `83.106.5.72` →
|
||||||
`90.251.153.180` (vyos002) → `90.251.142.103` (vyos001, after failback).
|
`90.251.153.180` (vyos002) → `90.251.142.103` (vyos001, after failback).
|
||||||
Nothing may be pinned to the PPPoE address. Checked: the HE IPv6 tunnel is
|
Nothing may be pinned to the PPPoE address. The HE IPv6 tunnel is pinned to
|
||||||
pinned to `87.192.101.48`, which is the **10 gig** (`bond0.53`) and stable
|
`87.192.101.48`, which is the **10 gig** (`bond0.53`) and stable across
|
||||||
across failover, so `tun0` survived untouched and IPv6 stayed up at 15.5ms.
|
failover. Anything added later that hardcodes a WAN IP must use the 10 gig one,
|
||||||
Anything added later that hardcodes a WAN IP must use the 10 gig one, not
|
not `pppoe0`'s.
|
||||||
`pppoe0`'s.
|
|
||||||
|
> **CORRECTED 2026-09-06.** This paragraph originally continued "so `tun0`
|
||||||
|
> survived untouched and IPv6 stayed up at 15.5ms." **That conclusion was
|
||||||
|
> wrong, and it should never have been recorded as a result.** The premise is
|
||||||
|
> right — the endpoint address is stable — but it does not follow. During
|
||||||
|
> takeover the reconciler disables `bond0.53` on the demoted box, so
|
||||||
|
> `87.192.101.48` *leaves vyos001 and appears on vyos002*, and vyos002 has no
|
||||||
|
> `tun0` at all: no tunnel, no `he-tunnel-follow`, no `/config/he-secrets`, no
|
||||||
|
> VLAN 9 prefix, no `route6 ::/0`. Inbound protocol 41 from HE lands on a router
|
||||||
|
> with nothing to decapsulate it. vyos001's own journal for the drill window
|
||||||
|
> reads `08:36:01 he-tunnel-follow: no default route; refusing to guess`.
|
||||||
|
>
|
||||||
|
> The reading was taken either side of the window, not through it — `wan-drill`
|
||||||
|
> contained **no IPv6 check of any kind**, and neither did any other part of the
|
||||||
|
> mechanism. That is now fixed: the drill probes IPv6 in both timing loops and
|
||||||
|
> asserts that a router-level failover makes **zero** HE API calls. Until a
|
||||||
|
> drill produces that figure, the IPv6 behaviour of a failover is *unmeasured*,
|
||||||
|
> not "fine".
|
||||||
|
>
|
||||||
|
> The gap itself is real and open: **IPv6 is single-homed on vyos001 while the
|
||||||
|
> WAN beneath it is HA.** Step 0 of the re-sequenced IPv6 plan closes it; see
|
||||||
|
> `migration/pulumi-override-he-tunnel-both.json` and
|
||||||
|
> `labsim/labsim-ipv6-ha-test.sh`.
|
||||||
|
|
||||||
Production takeover (52s) is about twice the sim's `replace` figure (26s), which
|
Production takeover (52s) is about twice the sim's `replace` figure (26s), which
|
||||||
is the expected direction: the VP2440s commit under kea, BGP and conntrack while
|
is the expected direction: the VP2440s commit under kea, BGP and conntrack while
|
||||||
|
|||||||
@@ -54,6 +54,16 @@ Never both at once — the surviving router must be able to hold the VIPs.
|
|||||||
FAULT on every group means the health check is failing — most likely
|
FAULT on every group means the health check is failing — most likely
|
||||||
`/config/vrrp-wan-health` did not get reinstalled, or `vrrp-wan.conf` is
|
`/config/vrrp-wan-health` did not get reinstalled, or `vrrp-wan.conf` is
|
||||||
missing so `GRACE` and the VIP fall back to defaults.
|
missing so `GRACE` and the VIP fall back to defaults.
|
||||||
|
5b. **Check IPv6 came back with it.** `vrrp-wan-install` now carries
|
||||||
|
`he-tunnel-follow`, so `--check` covers it, but `/config/he-secrets` is a
|
||||||
|
secret placed by Pulumi and is only checked for *presence*:
|
||||||
|
```
|
||||||
|
sudo /config/he-tunnel-follow status # role, tunnel src, MTU
|
||||||
|
```
|
||||||
|
Want the box's own role, and — on the master — a tunnel source equal to the
|
||||||
|
**10 gig** address with MTU 1480. `/config` survives an upgrade, so the
|
||||||
|
`system task-scheduler` entry that runs this every minute survives too; it is
|
||||||
|
the units and the ppp gate in `/etc` that do not.
|
||||||
6. Confirm `config.boot` still pins the safe resting state:
|
6. Confirm `config.boot` still pins the safe resting state:
|
||||||
```
|
```
|
||||||
sudo /config/vif53-pin-boot-disable --check # config.boot disable : 1
|
sudo /config/vif53-pin-boot-disable --check # config.boot disable : 1
|
||||||
|
|||||||
@@ -7,6 +7,13 @@
|
|||||||
# that presents as "some sites are broken", which is far worse to diagnose than
|
# that presents as "some sites are broken", which is far worse to diagnose than
|
||||||
# a clean one.
|
# a clean one.
|
||||||
#
|
#
|
||||||
|
# Installed on BOTH routers and gated on VRRP mastership: the backup exits
|
||||||
|
# immediately, and vrrp-wan-reconcile brings tun0 up and calls this script the
|
||||||
|
# moment it takes the VIP. The HE endpoint itself needs no update when the WAN
|
||||||
|
# moves between routers -- 87.192.101.48 is the 10 gig lease bound to the cloned
|
||||||
|
# MAC, so it follows the VIP to the other box unchanged (proven by the 2026-09-06
|
||||||
|
# drill). HE only has to be told about the WITHIN-box fall back to PPPoE.
|
||||||
|
#
|
||||||
# Changes are made at KERNEL level (`ip tunnel change`), not in VyOS config, on
|
# Changes are made at KERNEL level (`ip tunnel change`), not in VyOS config, on
|
||||||
# purpose:
|
# purpose:
|
||||||
# - no commit per WAN flip, so a flapping line cannot churn the config;
|
# - no commit per WAN flip, so a flapping line cannot churn the config;
|
||||||
@@ -27,6 +34,27 @@ set -uo pipefail
|
|||||||
TUNNEL="${TUNNEL:-tun0}"
|
TUNNEL="${TUNNEL:-tun0}"
|
||||||
SECRETS="${SECRETS:-/config/he-secrets}"
|
SECRETS="${SECRETS:-/config/he-secrets}"
|
||||||
STATE="${STATE:-/run/he-tunnel-follow.state}"
|
STATE="${STATE:-/run/he-tunnel-follow.state}"
|
||||||
|
ROLE_STATE="${ROLE_STATE:-/run/he-tunnel-follow.role}"
|
||||||
|
|
||||||
|
# HE's update endpoint, as a variable so labsim can point it at a stub. The sim
|
||||||
|
# has no public IPv4 and no HE account, which is the whole reason the tunnel was
|
||||||
|
# never rehearsed; with this the sim can exercise the HE-side half too.
|
||||||
|
HE_UPDATE_URL="${HE_UPDATE_URL:-https://ipv4.tunnelbroker.net/nic/update}"
|
||||||
|
|
||||||
|
# This script is installed on BOTH routers -- the same principle as the PPPoE
|
||||||
|
# gate: configured identically everywhere, gated at runtime. So it must know
|
||||||
|
# when it is the backup. Left ungated, the backup copy either dies on "no
|
||||||
|
# default route" every minute, or, far worse, sees its own idle pppoe0 address
|
||||||
|
# and points the HE endpoint at it. Vodafone hands out a different IPv4 on every
|
||||||
|
# dial, so that is an IPv6 blackhole plus a wasted write against a rate-limited
|
||||||
|
# API -- and it would fire on the backup, where nobody is looking.
|
||||||
|
#
|
||||||
|
# The VIP comes from the same /config/vrrp-wan.conf the reconciler and the
|
||||||
|
# health check read, so there is exactly one definition of "master" on the box.
|
||||||
|
WAN_CONF="${WAN_CONF:-/config/vrrp-wan.conf}"
|
||||||
|
# shellcheck disable=SC1090
|
||||||
|
[ -r "$WAN_CONF" ] && . "$WAN_CONF"
|
||||||
|
VIP="${VRRP_WAN_VIP:-192.168.1.1}"
|
||||||
# 6in4 costs 20 bytes. The 10 gig path is 1500 -> 1480; PPPoE is 1492 -> 1472.
|
# 6in4 costs 20 bytes. The 10 gig path is 1500 -> 1480; PPPoE is 1492 -> 1472.
|
||||||
# Getting this wrong is the classic "IPv6 works until something large" failure.
|
# Getting this wrong is the classic "IPv6 works until something large" failure.
|
||||||
declare -A WAN_MTU=( ["bond0.53"]=1480 ["pppoe0"]=1472 )
|
declare -A WAN_MTU=( ["bond0.53"]=1480 ["pppoe0"]=1472 )
|
||||||
@@ -37,6 +65,7 @@ HYSTERESIS="${HYSTERESIS:-2}"
|
|||||||
log() { logger -t he-tunnel-follow -- "$*"; printf ' %s\n' "$*"; }
|
log() { logger -t he-tunnel-follow -- "$*"; printf ' %s\n' "$*"; }
|
||||||
die() { logger -t he-tunnel-follow -p user.err -- "$*"; printf ' ERROR: %s\n' "$*" >&2; exit 1; }
|
die() { logger -t he-tunnel-follow -p user.err -- "$*"; printf ' ERROR: %s\n' "$*" >&2; exit 1; }
|
||||||
|
|
||||||
|
holds_vip() { ip -4 -o addr show 2>/dev/null | grep -q " ${VIP}/"; }
|
||||||
active_wan() { ip -4 route show default 2>/dev/null | awk '/^default/{for(i=1;i<=NF;i++) if($i=="dev") print $(i+1); exit}'; }
|
active_wan() { ip -4 route show default 2>/dev/null | awk '/^default/{for(i=1;i<=NF;i++) if($i=="dev") print $(i+1); exit}'; }
|
||||||
addr_of() { ip -4 -br addr show "$1" 2>/dev/null | awk '{print $3}' | cut -d/ -f1; }
|
addr_of() { ip -4 -br addr show "$1" 2>/dev/null | awk '{print $3}' | cut -d/ -f1; }
|
||||||
tunnel_src() { ip tunnel show "$TUNNEL" 2>/dev/null | sed -nE 's/.* local ([0-9.]+).*/\1/p'; }
|
tunnel_src() { ip tunnel show "$TUNNEL" 2>/dev/null | sed -nE 's/.* local ([0-9.]+).*/\1/p'; }
|
||||||
@@ -60,7 +89,7 @@ he_update() {
|
|||||||
--data-urlencode "password=$HE_UPDATE_KEY" \
|
--data-urlencode "password=$HE_UPDATE_KEY" \
|
||||||
--data-urlencode "hostname=$HE_TUNNEL_ID" \
|
--data-urlencode "hostname=$HE_TUNNEL_ID" \
|
||||||
--data-urlencode "myip=$ip" \
|
--data-urlencode "myip=$ip" \
|
||||||
"https://ipv4.tunnelbroker.net/nic/update" 2>&1)"
|
"$HE_UPDATE_URL" 2>&1)"
|
||||||
# dyndns protocol: "good <ip>" or "nochg <ip>" are both success.
|
# dyndns protocol: "good <ip>" or "nochg <ip>" are both success.
|
||||||
case "$out" in
|
case "$out" in
|
||||||
good*|nochg*) log "HE endpoint set to $ip ($out)"; return 0 ;;
|
good*|nochg*) log "HE endpoint set to $ip ($out)"; return 0 ;;
|
||||||
@@ -68,9 +97,31 @@ he_update() {
|
|||||||
esac
|
esac
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# Log only when the role CHANGES. On a 1-minute timer an unconditional line
|
||||||
|
# would be 1440 entries a day on the backup, which is how a real message gets
|
||||||
|
# lost. The marker lives in /run, so a reboot re-announces the role once.
|
||||||
|
note_role() {
|
||||||
|
local role="$1" last=""
|
||||||
|
[ -r "$ROLE_STATE" ] && read -r last < "$ROLE_STATE"
|
||||||
|
[ "$last" = "$role" ] && return 0
|
||||||
|
echo "$role" > "$ROLE_STATE"
|
||||||
|
log "role is now $role"
|
||||||
|
}
|
||||||
|
|
||||||
reconcile() {
|
reconcile() {
|
||||||
local dry="${1:-}"
|
local dry="${1:-}"
|
||||||
local wan src want_mtu cur_src cur_mtu
|
local wan src want_mtu cur_src cur_mtu
|
||||||
|
|
||||||
|
# The backup owns nothing here. vrrp-wan-reconcile holds tun0 down on this box
|
||||||
|
# and will run this script itself the moment it takes the VIP, so there is
|
||||||
|
# nothing to do and nothing to say.
|
||||||
|
if ! holds_vip; then
|
||||||
|
note_role backup
|
||||||
|
rm -f "$STATE" # start a promoted box with a clean hysteresis count
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
note_role master
|
||||||
|
|
||||||
wan="$(active_wan)"; [ -n "$wan" ] || die "no default route; refusing to guess"
|
wan="$(active_wan)"; [ -n "$wan" ] || die "no default route; refusing to guess"
|
||||||
src="$(addr_of "$wan")"; [ -n "$src" ] || die "no IPv4 address on $wan"
|
src="$(addr_of "$wan")"; [ -n "$src" ] || die "no IPv4 address on $wan"
|
||||||
want_mtu="${WAN_MTU[$wan]:-}"
|
want_mtu="${WAN_MTU[$wan]:-}"
|
||||||
@@ -110,10 +161,13 @@ reconcile() {
|
|||||||
case "${1:-status}" in
|
case "${1:-status}" in
|
||||||
status)
|
status)
|
||||||
wan="$(active_wan)"
|
wan="$(active_wan)"
|
||||||
|
printf ' role : %s (vip %s)\n' "$(holds_vip && echo master || echo backup)" "$VIP"
|
||||||
printf ' active WAN : %s\n' "${wan:-<none>}"
|
printf ' active WAN : %s\n' "${wan:-<none>}"
|
||||||
printf ' wan addr : %s\n' "$(addr_of "${wan:-lo}")"
|
printf ' wan addr : %s\n' "$(addr_of "${wan:-lo}")"
|
||||||
|
printf ' tunnel : %s\n' "$(ip -br link show "$TUNNEL" 2>/dev/null | awk '{print $2}' || echo '<absent>')"
|
||||||
printf ' tunnel src : %s\n' "$(tunnel_src)"
|
printf ' tunnel src : %s\n' "$(tunnel_src)"
|
||||||
printf ' tunnel mtu : %s (want %s)\n' "$(tunnel_mtu)" "${WAN_MTU[${wan:-}]:-?}"
|
printf ' tunnel mtu : %s (want %s)\n' "$(tunnel_mtu)" "${WAN_MTU[${wan:-}]:-?}"
|
||||||
|
printf ' he endpoint: %s\n' "$HE_UPDATE_URL"
|
||||||
[ -r "$SECRETS" ] && printf ' credentials: present\n' || printf ' credentials: MISSING (%s)\n' "$SECRETS"
|
[ -r "$SECRETS" ] && printf ' credentials: present\n' || printf ' credentials: MISSING (%s)\n' "$SECRETS"
|
||||||
;;
|
;;
|
||||||
run) reconcile "${2:-}" ;;
|
run) reconcile "${2:-}" ;;
|
||||||
|
|||||||
188
migration/pulumi-override-he-tunnel-both.json
Normal file
188
migration/pulumi-override-he-tunnel-both.json
Normal file
@@ -0,0 +1,188 @@
|
|||||||
|
{
|
||||||
|
"id": "he-ipv6-tunnel",
|
||||||
|
"reason": "6in4 tunnel to Hurricane Electric, bringing 2001:470:187e::/48 in, on BOTH routers. SUPERSEDES he-ipv6-tunnel-vyos001, whose reason said 'vyos002 carries bond0.53 disabled, so the source address does not exist there and the tunnel would simply stay down -- adding it there is for a later takeover story, not now.' Both halves of that expired on 2026-09-06. `vif 53 disable` is no longer a property of vyos002: it is RUNTIME state owned by vrrp-wan-reconcile, keyed on who holds the management VIP, and deliberately absent from this model (see pppoe-gated-not-config-disabled and PPPOE-HA.md). And the takeover story shipped -- a drill moved the WAN and back, 52s and 36s. IPv6 did not follow it, so every failover took the whole v6 estate down for as long as vyos002 held the VIP. WHY THIS IS CHEAP: 87.192.101.48 is the 10 gig lease bound to the cloned MAC, and the drill proved the ISP re-issues THE SAME address to that MAC on the other router's port. The tunnel endpoint is therefore stable across the pair, so a router-level failover needs no HE API call at all -- only the tunnel present on both boxes and live on exactly one. HE updates remain solely for the within-box fall back to PPPoE, which /config/he-tunnel-follow already handles. WHY THE RUNTIME GATE IS LOAD-BEARING, not a nicety: measured in labsim 2026-09-06, VyOS ACCEPTS a tunnel whose source-address does not exist on the box (commit rc=0) and brings the link UP anyway. It is a blackhole that will happily attract the v6 default route -- not the inert node the old reason assumed. vrrp-wan-reconcile holds tun0 down on the backup and brings it up on the master, at kernel level, with no commit in the failover path. Verified in the sim: backup tun=DOWN radvd=inactive, master tun=UP radvd=active. MTU 1480, not 1500: 6in4 adds a 20-byte outer IPv4 header. Leave it at 1500 and IPv6 appears to work while large transfers hang. The RA link-mtu is 1472 -- the PPPoE figure -- deliberately, on BOTH routers: it cannot be reconciled at runtime because it needs a commit, so advertise the lower of the two paths and be correct on either WAN. Production previously pinned 1480 and was silently wrong whenever the WAN fell back. bond0.9 takes ::1 on vyos001 and ::2 on vyos002 -- NOT the same address: SLAAC hosts take their gateway from the advertising router's link-local, so the global address need not move, and duplicating it would only produce a DAD conflict. default-preference is high on vyos001 and low on vyos002 so that if both ever advertise at once -- radvd's config is rendered into /run and a booting backup starts it before VRRP has decided -- hosts prefer the normal master, while a genuinely dead vyos001 still leaves vyos002 as the only router on the link. No firewall change is needed: the IPv6 firewall accepts only from interface-group LAN, so tun0 is untrusted by default. It is already default-deny on BOTH routers -- that ordering held. REHEARSAL: labsim/labsim-he-endpoint.sh now builds a fake HE endpoint and stub tunnelbroker API, closing the 'the sim has no public IPv4 and no HE endpoint' gap the old reason cited as why this was never tested. labsim/labsim-ipv6-ha-test.sh is the matrix. Mechanism proven there; the end-to-end v6 datapath is not yet, because the sim's inter-island transit crosses libvirt NAT -- see that file's KNOWN SIM GAP header.",
|
||||||
|
"set": [
|
||||||
|
{
|
||||||
|
"path": [
|
||||||
|
"interfaces",
|
||||||
|
"tunnel",
|
||||||
|
"tun0",
|
||||||
|
"encapsulation"
|
||||||
|
],
|
||||||
|
"value": "sit"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"path": [
|
||||||
|
"interfaces",
|
||||||
|
"tunnel",
|
||||||
|
"tun0",
|
||||||
|
"source-address"
|
||||||
|
],
|
||||||
|
"value": "87.192.101.48"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"path": [
|
||||||
|
"interfaces",
|
||||||
|
"tunnel",
|
||||||
|
"tun0",
|
||||||
|
"remote"
|
||||||
|
],
|
||||||
|
"value": "216.66.88.98"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"path": [
|
||||||
|
"interfaces",
|
||||||
|
"tunnel",
|
||||||
|
"tun0",
|
||||||
|
"address"
|
||||||
|
],
|
||||||
|
"value": "2001:470:1f1c:f6::2/64"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"path": [
|
||||||
|
"interfaces",
|
||||||
|
"tunnel",
|
||||||
|
"tun0",
|
||||||
|
"mtu"
|
||||||
|
],
|
||||||
|
"value": "1480"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"path": [
|
||||||
|
"interfaces",
|
||||||
|
"tunnel",
|
||||||
|
"tun0",
|
||||||
|
"description"
|
||||||
|
],
|
||||||
|
"value": "HE 6in4 tunnel - 2001:470:187e::/48"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"path": [
|
||||||
|
"protocols",
|
||||||
|
"static",
|
||||||
|
"route6",
|
||||||
|
"::/0",
|
||||||
|
"next-hop",
|
||||||
|
"2001:470:1f1c:f6::1"
|
||||||
|
],
|
||||||
|
"value": {}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"path": [
|
||||||
|
"system",
|
||||||
|
"task-scheduler",
|
||||||
|
"task",
|
||||||
|
"he-tunnel-follow",
|
||||||
|
"executable",
|
||||||
|
"path"
|
||||||
|
],
|
||||||
|
"value": "/config/he-tunnel-follow"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"path": [
|
||||||
|
"system",
|
||||||
|
"task-scheduler",
|
||||||
|
"task",
|
||||||
|
"he-tunnel-follow",
|
||||||
|
"executable",
|
||||||
|
"arguments"
|
||||||
|
],
|
||||||
|
"value": "run"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"path": [
|
||||||
|
"system",
|
||||||
|
"task-scheduler",
|
||||||
|
"task",
|
||||||
|
"he-tunnel-follow",
|
||||||
|
"interval"
|
||||||
|
],
|
||||||
|
"value": "1m"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"perRouter": {
|
||||||
|
"vyos001": [
|
||||||
|
{
|
||||||
|
"path": [
|
||||||
|
"interfaces",
|
||||||
|
"bonding",
|
||||||
|
"bond0",
|
||||||
|
"vif",
|
||||||
|
"9",
|
||||||
|
"address"
|
||||||
|
],
|
||||||
|
"value": "2001:470:187e:9::1/64"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"path": [
|
||||||
|
"service",
|
||||||
|
"router-advert",
|
||||||
|
"interface",
|
||||||
|
"bond0.9",
|
||||||
|
"default-preference"
|
||||||
|
],
|
||||||
|
"value": "high"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"vyos002": [
|
||||||
|
{
|
||||||
|
"path": [
|
||||||
|
"interfaces",
|
||||||
|
"bonding",
|
||||||
|
"bond0",
|
||||||
|
"vif",
|
||||||
|
"9",
|
||||||
|
"address"
|
||||||
|
],
|
||||||
|
"value": "2001:470:187e:9::2/64"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"path": [
|
||||||
|
"service",
|
||||||
|
"router-advert",
|
||||||
|
"interface",
|
||||||
|
"bond0.9",
|
||||||
|
"default-preference"
|
||||||
|
],
|
||||||
|
"value": "low"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"sharedRouterAdvert": [
|
||||||
|
{
|
||||||
|
"path": [
|
||||||
|
"service",
|
||||||
|
"router-advert",
|
||||||
|
"interface",
|
||||||
|
"bond0.9",
|
||||||
|
"link-mtu"
|
||||||
|
],
|
||||||
|
"value": "1472"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"path": [
|
||||||
|
"service",
|
||||||
|
"router-advert",
|
||||||
|
"interface",
|
||||||
|
"bond0.9",
|
||||||
|
"prefix",
|
||||||
|
"2001:470:187e:9::/64",
|
||||||
|
"preferred-lifetime"
|
||||||
|
],
|
||||||
|
"value": "604800"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"path": [
|
||||||
|
"service",
|
||||||
|
"router-advert",
|
||||||
|
"interface",
|
||||||
|
"bond0.9",
|
||||||
|
"prefix",
|
||||||
|
"2001:470:187e:9::/64",
|
||||||
|
"valid-lifetime"
|
||||||
|
],
|
||||||
|
"value": "2592000"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"_staging_note": "STAGED, NOT MERGED -- on purpose, exactly like migration/pulumi-override-pppoe-gated.json was. Another agent runs `pulumi up` on kubernetes-deployment, so merging this IS a production change made by someone else at a time you do not choose. Order of operations, and it matters: (1) place /config/he-secrets on vyos002 and run migration/vrrp-wan-install on BOTH routers, so the runtime gate that holds tun0 down exists there BEFORE the tunnel does; (2) run `npm run vyos:export && npm run vyos:render` immediately before merging, so the model follows whichever router actually holds the WAN; (3) merge. Reversed, vyos002 gets a tunnel with no gate and an UP blackhole interface that attracts the v6 default route. The perRouter/sharedRouterAdvert shape here is descriptive -- fold it into whatever form overrides.json actually supports for per-router values when merging; the vyos001-only precedent is the `routers` key."
|
||||||
|
}
|
||||||
@@ -45,7 +45,13 @@ put() { # copy a file to the target
|
|||||||
|
|
||||||
# script -> destination. take/release are hooks keepalived calls; both exec the
|
# script -> destination. take/release are hooks keepalived calls; both exec the
|
||||||
# reconciler, so there is one code path.
|
# reconciler, so there is one code path.
|
||||||
SCRIPTS="vrrp-wan-reconcile vrrp-wan-apply vrrp-wan-health vrrp-wan-guard vrrp-wan-take vrrp-wan-release"
|
#
|
||||||
|
# he-tunnel-follow is part of the mechanism, not a separate thing: the reconciler
|
||||||
|
# owns tun0's link state and that script owns its source address and MTU. Listing
|
||||||
|
# it here is what makes `--check` catch drift on it and what makes the VyOS
|
||||||
|
# image-upgrade runbook reinstall it -- IPv6 was previously the one half of the
|
||||||
|
# WAN story that no installer knew about.
|
||||||
|
SCRIPTS="vrrp-wan-reconcile vrrp-wan-apply vrrp-wan-health vrrp-wan-guard vrrp-wan-take vrrp-wan-release he-tunnel-follow"
|
||||||
UNITS="vrrp-wan-reconcile.service vrrp-wan-reconcile.timer vrrp-wan-guard.service vrrp-wan-guard.timer"
|
UNITS="vrrp-wan-reconcile.service vrrp-wan-reconcile.timer vrrp-wan-guard.service vrrp-wan-guard.timer"
|
||||||
GATE_DIR=/etc/systemd/system/ppp@pppoe0.service.d
|
GATE_DIR=/etc/systemd/system/ppp@pppoe0.service.d
|
||||||
GATE=$GATE_DIR/10-vrrp-wan-gate.conf
|
GATE=$GATE_DIR/10-vrrp-wan-gate.conf
|
||||||
@@ -66,6 +72,12 @@ if [ "$MODE" = check ]; then
|
|||||||
remote_gate=$(run "md5sum $GATE 2>/dev/null | cut -d' ' -f1")
|
remote_gate=$(run "md5sum $GATE 2>/dev/null | cut -d' ' -f1")
|
||||||
[ "$gate_sum" = "$remote_gate" ] || { echo " DRIFT $GATE (a VyOS upgrade wipes /etc -- both routers would dial)"; rc=1; }
|
[ "$gate_sum" = "$remote_gate" ] || { echo " DRIFT $GATE (a VyOS upgrade wipes /etc -- both routers would dial)"; rc=1; }
|
||||||
run "[ -r /config/vrrp-wan.conf ]" || { echo " MISSING /config/vrrp-wan.conf"; rc=1; }
|
run "[ -r /config/vrrp-wan.conf ]" || { echo " MISSING /config/vrrp-wan.conf"; rc=1; }
|
||||||
|
# Secrets are placed by Pulumi (infra/vyos/secretsFile.ts), never by this
|
||||||
|
# installer, so check presence only -- there is no correct content to compare
|
||||||
|
# against and printing a diff of credentials would be worse than useless.
|
||||||
|
# Without it he-tunnel-follow cannot re-point the tunnel when the WAN falls
|
||||||
|
# back to PPPoE, which fails silently: IPv4 keeps working and IPv6 goes dark.
|
||||||
|
run "[ -r /config/he-secrets ]" || { echo " MISSING /config/he-secrets (IPv6 cannot follow a WAN change)"; rc=1; }
|
||||||
for t in vrrp-wan-reconcile.timer vrrp-wan-guard.timer; do
|
for t in vrrp-wan-reconcile.timer vrrp-wan-guard.timer; do
|
||||||
[ "$(run "systemctl is-enabled $t 2>/dev/null")" = enabled ] || { echo " NOT ENABLED $t"; rc=1; }
|
[ "$(run "systemctl is-enabled $t 2>/dev/null")" = enabled ] || { echo " NOT ENABLED $t"; rc=1; }
|
||||||
done
|
done
|
||||||
|
|||||||
@@ -39,6 +39,8 @@ STATE=/run/vrrp-wan
|
|||||||
LOCK=/run/vrrp-wan.lock
|
LOCK=/run/vrrp-wan.lock
|
||||||
APPLY=/config/vrrp-wan-apply
|
APPLY=/config/vrrp-wan-apply
|
||||||
DROPIN=/etc/systemd/system/ppp@pppoe0.service.d/10-vrrp-wan-gate.conf
|
DROPIN=/etc/systemd/system/ppp@pppoe0.service.d/10-vrrp-wan-gate.conf
|
||||||
|
V6_TUNNEL="${V6_TUNNEL:-tun0}"
|
||||||
|
RADVD_CONF="${RADVD_CONF:-/run/radvd/radvd.conf}"
|
||||||
|
|
||||||
cfg() { /opt/vyatta/bin/vyatta-op-cmd-wrapper show configuration commands 2>/dev/null; }
|
cfg() { /opt/vyatta/bin/vyatta-op-cmd-wrapper show configuration commands 2>/dev/null; }
|
||||||
holds_vip() { ip -4 -o addr show 2>/dev/null | grep -q " ${VIP}/"; }
|
holds_vip() { ip -4 -o addr show 2>/dev/null | grep -q " ${VIP}/"; }
|
||||||
@@ -57,7 +59,7 @@ lease_age() { s=$(stat -c %Y "$STATE/may-dial" 2>/dev/null) || return 1
|
|||||||
wan_disabled(){ cfg | grep -q "vif ${WAN_VIF} disable"; }
|
wan_disabled(){ cfg | grep -q "vif ${WAN_VIF} disable"; }
|
||||||
|
|
||||||
if [ "${1:-}" = "--status" ]; then
|
if [ "${1:-}" = "--status" ]; then
|
||||||
printf 'vip=%s holds_vip=%s wan_disabled=%s wan_up=%s ppp_up=%s ppp_active=%s may_dial=%s lease_age=%s dropin=%s role=%s\n' \
|
printf 'vip=%s holds_vip=%s wan_disabled=%s wan_up=%s ppp_up=%s ppp_active=%s may_dial=%s lease_age=%s dropin=%s role=%s tun=%s radvd=%s\n' \
|
||||||
"$VIP" "$(holds_vip && echo yes || echo no)" \
|
"$VIP" "$(holds_vip && echo yes || echo no)" \
|
||||||
"$(wan_disabled && echo yes || echo no)" \
|
"$(wan_disabled && echo yes || echo no)" \
|
||||||
"$(wan_up && echo yes || echo no)" \
|
"$(wan_up && echo yes || echo no)" \
|
||||||
@@ -66,7 +68,9 @@ if [ "${1:-}" = "--status" ]; then
|
|||||||
"$([ -f "$STATE/may-dial" ] && echo yes || echo no)" \
|
"$([ -f "$STATE/may-dial" ] && echo yes || echo no)" \
|
||||||
"$(lease_age 2>/dev/null || echo -)" \
|
"$(lease_age 2>/dev/null || echo -)" \
|
||||||
"$([ -f "$DROPIN" ] && echo yes || echo MISSING)" \
|
"$([ -f "$DROPIN" ] && echo yes || echo MISSING)" \
|
||||||
"$(cat "$STATE/role" 2>/dev/null || echo unset)"
|
"$(cat "$STATE/role" 2>/dev/null || echo unset)" \
|
||||||
|
"$(ip -br link show "$V6_TUNNEL" 2>/dev/null | awk '{print $2}' || echo absent)" \
|
||||||
|
"$(systemctl is-active radvd 2>/dev/null || echo inactive)"
|
||||||
exit 0
|
exit 0
|
||||||
fi
|
fi
|
||||||
|
|
||||||
@@ -178,25 +182,94 @@ ppp_release() {
|
|||||||
systemctl stop ppp@pppoe0 2>/dev/null
|
systemctl stop ppp@pppoe0 2>/dev/null
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# --- IPv6: the kernel plane -------------------------------------------------
|
||||||
|
# The HE 6in4 tunnel and the VLAN 9 router advertisements have to follow
|
||||||
|
# mastership too, or a failover keeps IPv4 and silently drops IPv6 -- the
|
||||||
|
# partial outage that presents as "some sites are broken".
|
||||||
|
#
|
||||||
|
# A THIRD plane, and deliberately not either of the other two. Not config,
|
||||||
|
# because nothing here needs a commit (unlike the cloned MAC) and a commit per
|
||||||
|
# transition is the cost the pppoe0 half exists to avoid. Not the systemd gate,
|
||||||
|
# because there is no equivalent of a peers file to destroy.
|
||||||
|
#
|
||||||
|
# What makes this cheap: the tunnel is anchored to 87.192.101.48, the 10 gig
|
||||||
|
# lease bound to the cloned MAC, so it follows the VIP to the other router
|
||||||
|
# UNCHANGED. A router-level failover therefore needs no HE API call at all --
|
||||||
|
# only the link brought up on the box that now owns the address.
|
||||||
|
#
|
||||||
|
# Note what is deliberately NOT done here: this does not invoke
|
||||||
|
# he-tunnel-follow. That script has its own 1-minute task-scheduler cadence and
|
||||||
|
# a 2-tick hysteresis, and on 2026-09-06 that hysteresis was the only thing that
|
||||||
|
# stopped a routine `vif53-pin-boot-disable` run from pointing HE at a PPPoE
|
||||||
|
# address -- by 22 seconds. Calling it from a 30s reconciler as well would halve
|
||||||
|
# the window it needs. Its job is the WITHIN-box fall back to PPPoE; ours is
|
||||||
|
# link state.
|
||||||
|
v6_take() {
|
||||||
|
# Absent on a router that has no tunnel in its config -- which is every
|
||||||
|
# router until the model change lands. No-op there rather than complain.
|
||||||
|
[ -e "/sys/class/net/$V6_TUNNEL" ] || return 0
|
||||||
|
# Needs SOME WAN address to source from. Either line will do: if bond0.53 is
|
||||||
|
# down but pppoe0 is up, he-tunnel-follow re-points the tunnel on its own
|
||||||
|
# schedule, and holding the link down until then would turn a degraded path
|
||||||
|
# into no path.
|
||||||
|
wan_up || ppp_up || return 0
|
||||||
|
ip link show "$V6_TUNNEL" 2>/dev/null | grep -q 'state DOWN' && {
|
||||||
|
logger -t vrrp-wan "MASTER: bringing $V6_TUNNEL up"
|
||||||
|
ip link set "$V6_TUNNEL" up 2>/dev/null
|
||||||
|
}
|
||||||
|
# radvd's config is rendered into /run by the VyOS commit, so on a box with
|
||||||
|
# no router-advert node there is nothing to start.
|
||||||
|
[ -f "$RADVD_CONF" ] || return 0
|
||||||
|
systemctl is-active --quiet radvd 2>/dev/null && return 0
|
||||||
|
logger -t vrrp-wan "MASTER: starting radvd"
|
||||||
|
systemctl start radvd 2>/dev/null
|
||||||
|
}
|
||||||
|
|
||||||
|
v6_release() {
|
||||||
|
[ -e "/sys/class/net/$V6_TUNNEL" ] || return 0
|
||||||
|
# radvd FIRST, and this ordering is the point: on a graceful stop it emits a
|
||||||
|
# final advertisement with router-lifetime 0, which is what tells VLAN 9
|
||||||
|
# hosts to stop using this box as their default router. Kill the daemon
|
||||||
|
# after tearing things down and they keep a dead gateway until the RA
|
||||||
|
# lifetime expires on its own.
|
||||||
|
if systemctl is-active --quiet radvd 2>/dev/null; then
|
||||||
|
logger -t vrrp-wan "not MASTER: stopping radvd (deprecates the v6 gateway)"
|
||||||
|
systemctl stop radvd 2>/dev/null
|
||||||
|
fi
|
||||||
|
ip link show "$V6_TUNNEL" 2>/dev/null | grep -q 'state DOWN' && return 0
|
||||||
|
logger -t vrrp-wan "not MASTER: bringing $V6_TUNNEL down"
|
||||||
|
ip link set "$V6_TUNNEL" down 2>/dev/null
|
||||||
|
}
|
||||||
|
|
||||||
# --- decide ----------------------------------------------------------------
|
# --- decide ----------------------------------------------------------------
|
||||||
if holds_vip; then
|
if holds_vip; then
|
||||||
echo master > "$STATE/role"
|
echo master > "$STATE/role"
|
||||||
[ -f "$STATE/since" ] || date +%s > "$STATE/since"
|
[ -f "$STATE/since" ] || date +%s > "$STATE/since"
|
||||||
ppp_dial
|
ppp_dial
|
||||||
wan_disabled || exit 0
|
# The WAN block is now an `if` rather than an early exit, so the IPv6 plane
|
||||||
logger -t vrrp-wan "MASTER with bond0.${WAN_VIF} disabled -> enabling"
|
# below is reached on EVERY tick and not only on the one that enables the
|
||||||
t0=$(date +%s)
|
# WAN. On the promotion tick bond0.53 has no address yet, so v6_take no-ops
|
||||||
"$APPLY" enable 9>&-
|
# and the next tick takes it.
|
||||||
logger -t vrrp-wan "bond0.${WAN_VIF} enable commit took $(( $(date +%s) - t0 ))s"
|
if wan_disabled; then
|
||||||
|
logger -t vrrp-wan "MASTER with bond0.${WAN_VIF} disabled -> enabling"
|
||||||
|
t0=$(date +%s)
|
||||||
|
"$APPLY" enable 9>&-
|
||||||
|
logger -t vrrp-wan "bond0.${WAN_VIF} enable commit took $(( $(date +%s) - t0 ))s"
|
||||||
|
fi
|
||||||
|
v6_take
|
||||||
else
|
else
|
||||||
echo backup > "$STATE/role"
|
echo backup > "$STATE/role"
|
||||||
rm -f "$STATE/since" "$STATE/holdoff"
|
rm -f "$STATE/since" "$STATE/holdoff"
|
||||||
ppp_release
|
ppp_release
|
||||||
wan_disabled && exit 0
|
# Before the WAN goes, not after: once bond0.53 is disabled the source
|
||||||
logger -t vrrp-wan "not MASTER but bond0.${WAN_VIF} enabled -> releasing"
|
# address is gone and radvd's farewell advertisement has no path out.
|
||||||
t0=$(date +%s)
|
v6_release
|
||||||
"$APPLY" disable 9>&-
|
if ! wan_disabled; then
|
||||||
logger -t vrrp-wan "bond0.${WAN_VIF} disable commit took $(( $(date +%s) - t0 ))s"
|
logger -t vrrp-wan "not MASTER but bond0.${WAN_VIF} enabled -> releasing"
|
||||||
|
t0=$(date +%s)
|
||||||
|
"$APPLY" disable 9>&-
|
||||||
|
logger -t vrrp-wan "bond0.${WAN_VIF} disable commit took $(( $(date +%s) - t0 ))s"
|
||||||
|
fi
|
||||||
fi
|
fi
|
||||||
|
|
||||||
# No `save`, deliberately. config.boot keeps `vif 53 disable` on BOTH routers, so
|
# No `save`, deliberately. config.boot keeps `vif 53 disable` on BOTH routers, so
|
||||||
|
|||||||
@@ -36,6 +36,22 @@ holder() { for h in "$P1" "$P2"; do
|
|||||||
wan_of() { r "$1" 'for i in bond0.53 pppoe0; do a=$(ip -4 addr show dev $i 2>/dev/null | sed -n "s/.*inet \([0-9.]*\).*/\1/p"); [ -n "$a" ] && printf "%s=%s " $i $a; done'; }
|
wan_of() { r "$1" 'for i in bond0.53 pppoe0; do a=$(ip -4 addr show dev $i 2>/dev/null | sed -n "s/.*inet \([0-9.]*\).*/\1/p"); [ -n "$a" ] && printf "%s=%s " $i $a; done'; }
|
||||||
online() { [ "$(r "$1" 'ping -c1 -W2 9.9.9.9 >/dev/null 2>&1 && echo y' | tr -d ' \n')" = y ]; }
|
online() { [ "$(r "$1" 'ping -c1 -W2 9.9.9.9 >/dev/null 2>&1 && echo y' | tr -d ' \n')" = y ]; }
|
||||||
|
|
||||||
|
# --- IPv6 -------------------------------------------------------------------
|
||||||
|
# Until 2026-09-06 this drill measured IPv4 only, and PPPOE-HA.md recorded that
|
||||||
|
# "IPv6 stayed up" through a failover on the strength of a reading taken outside
|
||||||
|
# the window. It cannot have: the reconciler disables bond0.53 on the demoted
|
||||||
|
# box, so 87.192.101.48 moves to the survivor, and inbound protocol 41 from HE
|
||||||
|
# then lands on whichever router owns the tunnel. Measure it rather than assume.
|
||||||
|
#
|
||||||
|
# Quad9 again, so the v6 result is comparable with the v4 one on the line above.
|
||||||
|
online6() { [ "$(r "$1" 'ping -6 -c1 -W2 2620:fe::fe >/dev/null 2>&1 && echo y' | tr -d ' \n')" = y ]; }
|
||||||
|
# Tunnel link state and source address. The source is the interesting half: it
|
||||||
|
# should be IDENTICAL before and after a router-level failover, because the 10
|
||||||
|
# gig lease follows the cloned MAC. A changed source means something called the
|
||||||
|
# HE API during the drill, which a router failover must never need to do.
|
||||||
|
tun_of() { r "$1" 'ip tunnel show tun0 2>/dev/null | sed -nE "s/.* local ([0-9.]+).*/\1/p"' | tr -d ' \n'; }
|
||||||
|
he_calls(){ r "$1" 'sudo journalctl -t he-tunnel-follow --since "'"$2"'" --no-pager 2>/dev/null | grep -c "HE endpoint set to"' | tr -d ' \n'; }
|
||||||
|
|
||||||
cleanup() {
|
cleanup() {
|
||||||
say "--- cleanup (always runs) ---"
|
say "--- cleanup (always runs) ---"
|
||||||
r "$P2" 'sudo /config/wan-drill-watchdog disarm' >/dev/null
|
r "$P2" 'sudo /config/wan-drill-watchdog disarm' >/dev/null
|
||||||
@@ -52,11 +68,17 @@ cleanup() {
|
|||||||
trap cleanup EXIT
|
trap cleanup EXIT
|
||||||
|
|
||||||
say "=== pre-flight ==="
|
say "=== pre-flight ==="
|
||||||
|
DRILL_START="$(date '+%Y-%m-%d %H:%M:%S')"
|
||||||
start_holder="$(holder)"
|
start_holder="$(holder)"
|
||||||
say "holder now : $start_holder"
|
say "holder now : $start_holder"
|
||||||
say "$P1 WAN : $(wan_of $P1)"
|
say "$P1 WAN : $(wan_of $P1)"
|
||||||
say "$P2 WAN : $(wan_of $P2)"
|
say "$P2 WAN : $(wan_of $P2)"
|
||||||
online "$P1" && say "internet : UP via $P1" || { say "internet ALREADY DOWN -- refusing to drill"; exit 1; }
|
online "$P1" && say "internet : UP via $P1" || { say "internet ALREADY DOWN -- refusing to drill"; exit 1; }
|
||||||
|
tun_before="$(tun_of $P1)"; tun2_before="$(tun_of $P2)"
|
||||||
|
say "$P1 tun0 src : ${tun_before:-<no tunnel>}"
|
||||||
|
say "$P2 tun0 src : ${tun2_before:-<no tunnel -- IPv6 cannot survive a failover>}"
|
||||||
|
if online6 "$P1"; then say "IPv6 : UP via $P1"
|
||||||
|
else say "IPv6 : DOWN on $P1 before we start -- v6 figures below are not meaningful"; fi
|
||||||
[ "$start_holder" = "$P1" ] || { say "expected $P1 to hold the VIP, got $start_holder -- refusing"; exit 1; }
|
[ "$start_holder" = "$P1" ] || { say "expected $P1 to hold the VIP, got $start_holder -- refusing"; exit 1; }
|
||||||
|
|
||||||
if [ "$DRY" = 1 ]; then
|
if [ "$DRY" = 1 ]; then
|
||||||
@@ -71,14 +93,19 @@ say "=== DRILL: force-faulting $P1 ==="
|
|||||||
t0=$(date +%s)
|
t0=$(date +%s)
|
||||||
r "$P1" 'sudo touch /run/vrrp-wan/force-fault'
|
r "$P1" 'sudo touch /run/vrrp-wan/force-fault'
|
||||||
|
|
||||||
took=""
|
took=""; took6=""
|
||||||
while [ $(( $(date +%s) - t0 )) -lt "$TAKEOVER_BUDGET" ]; do
|
while [ $(( $(date +%s) - t0 )) -lt "$TAKEOVER_BUDGET" ]; do
|
||||||
sleep 5
|
sleep 5
|
||||||
h="$(holder)"; w="$(wan_of $P2)"
|
h="$(holder)"; w="$(wan_of $P2)"
|
||||||
say " t+$(( $(date +%s) - t0 ))s holder=$h vyos002_wan=[$w]"
|
# v6 keeps being probed after v4 comes back, because the two recover
|
||||||
if [ "$h" = "$P2" ] && [ -n "$w" ] && online "$P2"; then
|
# independently and the gap between them IS the number this drill exists to
|
||||||
took=$(( $(date +%s) - t0 )); break
|
# produce. Stop only when both are up, or the budget runs out.
|
||||||
|
[ -z "$took6" ] && online6 "$P2" && took6=$(( $(date +%s) - t0 ))
|
||||||
|
say " t+$(( $(date +%s) - t0 ))s holder=$h vyos002_wan=[$w] v6=$([ -n "$took6" ] && echo up || echo down)"
|
||||||
|
if [ -z "$took" ] && [ "$h" = "$P2" ] && [ -n "$w" ] && online "$P2"; then
|
||||||
|
took=$(( $(date +%s) - t0 ))
|
||||||
fi
|
fi
|
||||||
|
[ -n "$took" ] && [ -n "$took6" ] && break
|
||||||
done
|
done
|
||||||
|
|
||||||
if [ -n "$took" ]; then
|
if [ -n "$took" ]; then
|
||||||
@@ -86,20 +113,47 @@ if [ -n "$took" ]; then
|
|||||||
else
|
else
|
||||||
say "*** TAKEOVER FAILED within ${TAKEOVER_BUDGET}s -- failing back ***"
|
say "*** TAKEOVER FAILED within ${TAKEOVER_BUDGET}s -- failing back ***"
|
||||||
fi
|
fi
|
||||||
|
if [ -n "$took6" ] && [ -n "$took" ]; then
|
||||||
|
say "*** IPv6 followed in ${took6}s (v4 ${took}s, gap $(( took6 - took ))s) ***"
|
||||||
|
elif [ -n "$took6" ]; then
|
||||||
|
say "*** IPv6 followed in ${took6}s, but IPv4 never did ***"
|
||||||
|
else
|
||||||
|
say "*** IPv6 did NOT return within ${TAKEOVER_BUDGET}s on $P2 -- the v6 estate is down for the whole takeover ***"
|
||||||
|
fi
|
||||||
|
|
||||||
say "=== failing back to $P1 ==="
|
say "=== failing back to $P1 ==="
|
||||||
t1=$(date +%s)
|
t1=$(date +%s)
|
||||||
r "$P2" 'sudo /config/wan-drill-watchdog disarm' >/dev/null
|
r "$P2" 'sudo /config/wan-drill-watchdog disarm' >/dev/null
|
||||||
r "$P1" 'sudo rm -f /run/vrrp-wan/force-fault'
|
r "$P1" 'sudo rm -f /run/vrrp-wan/force-fault'
|
||||||
r "$P2" 'sudo touch /run/vrrp-wan/force-fault'
|
r "$P2" 'sudo touch /run/vrrp-wan/force-fault'
|
||||||
back=""
|
back=""; back6=""
|
||||||
while [ $(( $(date +%s) - t1 )) -lt "$FAILBACK_BUDGET" ]; do
|
while [ $(( $(date +%s) - t1 )) -lt "$FAILBACK_BUDGET" ]; do
|
||||||
sleep 5
|
sleep 5
|
||||||
h="$(holder)"; w="$(wan_of $P1)"
|
h="$(holder)"; w="$(wan_of $P1)"
|
||||||
say " t+$(( $(date +%s) - t1 ))s holder=$h vyos001_wan=[$w]"
|
[ -z "$back6" ] && online6 "$P1" && back6=$(( $(date +%s) - t1 ))
|
||||||
if [ "$h" = "$P1" ] && [ -n "$w" ] && online "$P1"; then
|
say " t+$(( $(date +%s) - t1 ))s holder=$h vyos001_wan=[$w] v6=$([ -n "$back6" ] && echo up || echo down)"
|
||||||
back=$(( $(date +%s) - t1 )); break
|
if [ -z "$back" ] && [ "$h" = "$P1" ] && [ -n "$w" ] && online "$P1"; then
|
||||||
|
back=$(( $(date +%s) - t1 ))
|
||||||
fi
|
fi
|
||||||
|
[ -n "$back" ] && [ -n "$back6" ] && break
|
||||||
done
|
done
|
||||||
[ -n "$back" ] && say "*** FAILBACK OK in ${back}s ***" \
|
[ -n "$back" ] && say "*** FAILBACK OK in ${back}s ***" \
|
||||||
|| say "*** FAILBACK FAILED -- cleanup will clear both levers ***"
|
|| say "*** FAILBACK FAILED -- cleanup will clear both levers ***"
|
||||||
|
[ -n "$back6" ] && say "*** IPv6 back in ${back6}s ***" \
|
||||||
|
|| say "*** IPv6 did NOT return within ${FAILBACK_BUDGET}s on $P1 ***"
|
||||||
|
|
||||||
|
# The invariant a router-level failover must satisfy: the HE endpoint is never
|
||||||
|
# touched. 87.192.101.48 follows the cloned MAC to the other box, so the tunnel
|
||||||
|
# source is the same address on either router and there is nothing to tell HE.
|
||||||
|
# A non-zero count here means something re-pointed the tunnel at a PPPoE address
|
||||||
|
# -- which Vodafone re-issues on every dial, so it would be wrong within minutes.
|
||||||
|
say "=== IPv6 invariants ==="
|
||||||
|
tun_after="$(tun_of $P1)"
|
||||||
|
say "tun0 src : ${tun_before:-none} -> ${tun_after:-none}"
|
||||||
|
[ "$tun_before" = "$tun_after" ] && say " OK: tunnel source unchanged across the drill" \
|
||||||
|
|| say " *** CHANGED -- a router failover should never move the HE endpoint ***"
|
||||||
|
for h in "$P1" "$P2"; do
|
||||||
|
n="$(he_calls "$h" "$DRILL_START")"
|
||||||
|
say "HE updates from $h since ${DRILL_START}: ${n:-?}"
|
||||||
|
[ "${n:-0}" = 0 ] || say " *** $h called the HE API during a router failover -- it should not need to ***"
|
||||||
|
done
|
||||||
|
|||||||
Reference in New Issue
Block a user