VLAN 2 IPv6 applied to production: RA proven, MAC reservations do NOT match
Some checks failed
Some checks failed
The unattended window's W1/W2. Both routers now carry the VLAN 2 IPv6 config (addresses ::1/::2, RA with managed-flag + no-autonomous-flag + link-mtu 1472, DHCPv6 reservations for all five nodes), applied via migration/vlan2-v6-apply with a vlan2-v6-watchdog armed on both routers throughout. IPv4 untouched: internet up, 6 MASTER / 6 BACKUP, 5/5 nodes Ready, watchdogs quiet. default-lifetime 0 on purpose -- addressing without egress. Turning on v6 egress moves image pulls onto a tunnel of unmeasured throughput, and the person who would notice is away. One line to flip when attended. THE WINDOW'S QUESTION IS ANSWERED, HALF YES, HALF NO: YES: NetworkManager follows the managed flag. worker0 logged dhcp6 (eno1): activation: beginning transaction minutes after the RA appeared. "ipv6.method=auto will do DHCPv6" is now evidence, not inference. NO: the reservations never match. Every packet kea logs is [no hwaddr info] -- the clients identify with DUID-UUID (and one DUID-LLT), kea derives no MAC from any of them, so hw-address reservations cannot match and no node got its reserved address. VyOS accepting `static-mapping mac` renders valid kea config that simply never matches these clients. The "one source of truth with IPv4" addressing scheme does not survive contact with DHCPv6; options (DUID keys, kea mac-sources, dynamic range + discovery, or SLAAC) are written up in migration/window-evidence/2026-09-06-dhcpv6.txt for an attended decision. FOUND LIVE AND FIXED IN THE SAME WINDOW: `service dhcpv6-server` with no listen-interface renders kea6 with interfaces: ["*"] -- a DHCPv6 server on EVERY VLAN. kea was answering an unrelated device on bond0.10 within seconds of the first apply. Same family as the kea IPv4 cross-VLAN bug (ISC #1117). Now pinned to bond0.2 on both routers. Also in this commit, three self-inflicted script bugs found by their own failures: log() wrote progress lines into the captured config stream (VyOS rejected each as "Invalid command", leaving the two routers correct but NOT identical); "Invalid command" was missing from the failure patterns so that run reported success; and the MAC lookup matched its own freshly-created v6 reservations on the second run, returning doubled MACs. All three fixed, both routers converged and diffed identical. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
This commit is contained in:
189
migration/vlan2-v6-apply
Executable file
189
migration/vlan2-v6-apply
Executable file
@@ -0,0 +1,189 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# Give VLAN 2 (the k8s VLAN) IPv6: addresses, router advertisements and DHCPv6
|
||||||
|
# reservations. Production half of dual-stack phase 2b.
|
||||||
|
#
|
||||||
|
# Rehearsed first as labsim/labsim-dualstack-net.sh, which is where the four
|
||||||
|
# VyOS facts below were paid for rather than guessed.
|
||||||
|
#
|
||||||
|
# ADDRESSING ONLY -- NOT EGRESS. The prefix is advertised with
|
||||||
|
# `default-lifetime 0`, so nodes get their reserved addresses but neither router
|
||||||
|
# becomes an IPv6 default router. Turning on real v6 egress moves cluster image
|
||||||
|
# pulls onto the HE tunnel (1480, or 1472 on PPPoE) whose throughput has never
|
||||||
|
# been measured, and that is not a thing to switch on unattended. Flipping it is
|
||||||
|
# one line: `set service router-advert interface bond0.2 default-lifetime '1800'`
|
||||||
|
# plus a default-preference, once somebody is watching.
|
||||||
|
#
|
||||||
|
# LISTEN-INTERFACE IS NOT OPTIONAL. Without it kea6 renders
|
||||||
|
# `interfaces: [ "*" ]` and serves DHCPv6 on EVERY VLAN, not just this one.
|
||||||
|
# Observed in production the moment this was first applied: kea started
|
||||||
|
# answering SOLICIT/REQUEST from an unrelated device on bond0.10 (LoT). That is
|
||||||
|
# a DHCPv6 server switched on estate-wide as a side effect of configuring one
|
||||||
|
# VLAN -- the same family of mistake as the kea IPv4 cross-VLAN bug (ISC #1117)
|
||||||
|
# this estate already fought. Pin the interface.
|
||||||
|
#
|
||||||
|
# WHY DHCPv6 RATHER THAN SLAAC: k3s resolves node-ip once at start-up, so a
|
||||||
|
# node's address must be knowable in advance and stable. The estate already
|
||||||
|
# answers that for IPv4 with kea reservations keyed on MAC; IPv6 answers it the
|
||||||
|
# same way, from the same MACs, so there is one source of truth. VyOS's
|
||||||
|
# static-mapping accepts `mac` as well as `duid`, which is what makes that
|
||||||
|
# possible -- DHCPv6 normally keys on a client-generated DUID.
|
||||||
|
#
|
||||||
|
# vlan2-v6-apply plan print what would be applied, change nothing
|
||||||
|
# vlan2-v6-apply apply apply to both routers
|
||||||
|
# vlan2-v6-apply verify what the routers and nodes now hold
|
||||||
|
# vlan2-v6-apply revert remove it again
|
||||||
|
set -uo pipefail
|
||||||
|
|
||||||
|
R1="${R1:-10.0.1.252}" # vyos001 -> ::1
|
||||||
|
R2="${R2:-10.0.1.253}" # vyos002 -> ::2
|
||||||
|
PW="${VYOS_PW:-vyos}"
|
||||||
|
V6_PREFIX="${V6_PREFIX:-2001:470:187e:2}"
|
||||||
|
LINK_MTU="${LINK_MTU:-1472}"
|
||||||
|
SUBNET_ID="${SUBNET_ID:-2}" # VyOS requires a unique id per DHCPv6 subnet
|
||||||
|
SHARED_NET="${SHARED_NET:-TheLab-k8s}"
|
||||||
|
|
||||||
|
# name:v4-host-octet -- the IPv6 host part mirrors the IPv4 one so a reservation
|
||||||
|
# is readable next to its twin. MACs are read from the LIVE IPv4 reservations at
|
||||||
|
# run time, never duplicated here: one source of truth, and a node that is
|
||||||
|
# re-homed cannot end up with a stale v6 mapping.
|
||||||
|
NODES=(worker0-k8s0:23 worker1-k8s0:13 worker2-k8s0:25 spark-2935:12 aitopatom-3a1c:27)
|
||||||
|
|
||||||
|
SSH=(-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null
|
||||||
|
-o LogLevel=ERROR -o ConnectTimeout=8)
|
||||||
|
|
||||||
|
# stderr, NOT stdout. build_mappings() is captured with $(...) and log() lines
|
||||||
|
# went straight into the config stream, where VyOS rejected each one as
|
||||||
|
# "Invalid command: [[0" -- ANSI escapes and all. The valid sets still applied,
|
||||||
|
# so the routers ended up correct but NOT identical: different lines were lost
|
||||||
|
# on each. A progress message is not data; keep it off the data channel.
|
||||||
|
log() { printf '\033[0;36m[vlan2-v6]\033[0m %s\n' "$*" >&2; }
|
||||||
|
die() { printf '\033[0;31m[vlan2-v6]\033[0m %s\n' "$*" >&2; exit 1; }
|
||||||
|
|
||||||
|
r() { timeout 45 ssh "${SSH[@]}" "vyos@$1" "${@:2}" 2>/dev/null; }
|
||||||
|
|
||||||
|
# Drive VyOS from a script FILE with plain commit + save.
|
||||||
|
# - `vbash -c` never starts a config session; the commit fails to stderr and a
|
||||||
|
# helper discards it, so the run reports success having changed nothing.
|
||||||
|
# - `commit-confirm` hangs non-interactively and strands an orphaned
|
||||||
|
# config-mgmt commit_confirm holding the config lock.
|
||||||
|
# - vbash exits 0 even when the commit fails, so the OUTPUT is the only honest
|
||||||
|
# signal. Read it.
|
||||||
|
vyos_apply() {
|
||||||
|
local h="$1" out
|
||||||
|
out="$({ printf '#!/bin/vbash\nsource /opt/vyatta/etc/functions/script-template\nconfigure\n'
|
||||||
|
cat
|
||||||
|
printf 'commit\nsave\nexit\n'
|
||||||
|
} | timeout 150 ssh "${SSH[@]}" "vyos@$h" \
|
||||||
|
'cat > /tmp/vlan2-v6.sh && chmod +x /tmp/vlan2-v6.sh && sudo /tmp/vlan2-v6.sh' 2>&1)"
|
||||||
|
printf '%s\n' "$out" | grep -vE '^\s*$' | sed 's/^/ /' | tail -8
|
||||||
|
# "Invalid command" was NOT in this list the first time, so a run that fed
|
||||||
|
# rubbish to VyOS reported success. vbash exits 0 regardless, so every
|
||||||
|
# rejection shape has to be named explicitly.
|
||||||
|
printf '%s' "$out" | grep -qiE 'Commit failed|\[\[.*\]\] failed|Set failed|Invalid command' && return 1
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
|
||||||
|
# Read each node's MAC out of the live IPv4 reservation.
|
||||||
|
#
|
||||||
|
# Scoped to the IPv4 subnet on purpose. Once this script has run once, the node
|
||||||
|
# has TWO `static-mapping <name> mac` lines -- the v4 one and the v6 one it just
|
||||||
|
# created -- and an unscoped match returned both concatenated
|
||||||
|
# ("9c:76:0e:49:e9:179c:76:0e:49:e9:17"), which VyOS then rejected as an invalid
|
||||||
|
# value. Self-inflicted on the second run: the lookup has to name the subnet it
|
||||||
|
# means, or the script poisons its own input as soon as it succeeds.
|
||||||
|
V4_SUBNET="${V4_SUBNET:-192.168.8.0/23}"
|
||||||
|
mac_of() {
|
||||||
|
r "$R1" "/opt/vyatta/bin/vyatta-op-cmd-wrapper show configuration commands 2>/dev/null \
|
||||||
|
| grep -F 'subnet $V4_SUBNET' \
|
||||||
|
| sed -n \"s/.*static-mapping $1 mac '\\(.*\\)'/\\1/p\"" | tr -d ' \n'
|
||||||
|
}
|
||||||
|
|
||||||
|
build_mappings() {
|
||||||
|
local out="" name hextet mac
|
||||||
|
for entry in "${NODES[@]}"; do
|
||||||
|
name="${entry%%:*}"; hextet="${entry##*:}"
|
||||||
|
mac="$(mac_of "$name")"
|
||||||
|
[ -n "$mac" ] || die "no IPv4 reservation found for $name -- refusing to invent a MAC"
|
||||||
|
log " $name $mac -> ${V6_PREFIX}::${hextet}"
|
||||||
|
[ -n "$out" ] && out+=$'\n'
|
||||||
|
out+="set service dhcpv6-server shared-network-name ${SHARED_NET} subnet ${V6_PREFIX}::/64 static-mapping ${name} mac '${mac}'
|
||||||
|
set service dhcpv6-server shared-network-name ${SHARED_NET} subnet ${V6_PREFIX}::/64 static-mapping ${name} ipv6-address '${V6_PREFIX}::${hextet}'"
|
||||||
|
done
|
||||||
|
printf '%s' "$out"
|
||||||
|
}
|
||||||
|
|
||||||
|
# One commit per router, and `no-autonomous-flag` is in it. That is not a
|
||||||
|
# style choice: turning autonomous off LATER does not retract addresses already
|
||||||
|
# formed, so a prefix advertised even briefly without it leaves every node
|
||||||
|
# holding an unreserved EUI-64 address with a 30-day lifetime. Observed in the
|
||||||
|
# sim. VLAN 2 has no IPv6 today, so this is the one chance to get it right.
|
||||||
|
config_for() {
|
||||||
|
local self="$1" mappings="$2"
|
||||||
|
cat <<EOF
|
||||||
|
set interfaces bonding bond0 vif 2 address '${V6_PREFIX}::${self}/64'
|
||||||
|
set service router-advert interface bond0.2 prefix ${V6_PREFIX}::/64 no-autonomous-flag
|
||||||
|
set service router-advert interface bond0.2 prefix ${V6_PREFIX}::/64 valid-lifetime '2592000'
|
||||||
|
set service router-advert interface bond0.2 prefix ${V6_PREFIX}::/64 preferred-lifetime '604800'
|
||||||
|
set service router-advert interface bond0.2 managed-flag
|
||||||
|
set service router-advert interface bond0.2 link-mtu '${LINK_MTU}'
|
||||||
|
set service router-advert interface bond0.2 default-lifetime '0'
|
||||||
|
set service dhcpv6-server listen-interface bond0.2
|
||||||
|
set service dhcpv6-server shared-network-name ${SHARED_NET} subnet ${V6_PREFIX}::/64 subnet-id '${SUBNET_ID}'
|
||||||
|
set service dhcpv6-server shared-network-name ${SHARED_NET} subnet ${V6_PREFIX}::/64 interface 'bond0.2'
|
||||||
|
${mappings}
|
||||||
|
EOF
|
||||||
|
}
|
||||||
|
|
||||||
|
cmd_plan() {
|
||||||
|
log "reading MACs from the live IPv4 reservations"
|
||||||
|
local m; m="$(build_mappings)" || exit 1
|
||||||
|
echo; log "--- vyos001 (${V6_PREFIX}::1) ---"; config_for 1 "$m" | sed 's/^/ /'
|
||||||
|
echo; log "--- vyos002 (${V6_PREFIX}::2) ---"; config_for 2 "$m" | sed 's/^/ /'
|
||||||
|
}
|
||||||
|
|
||||||
|
cmd_apply() {
|
||||||
|
log "reading MACs from the live IPv4 reservations"
|
||||||
|
local m; m="$(build_mappings)" || exit 1
|
||||||
|
local h self
|
||||||
|
for h in "$R1" "$R2"; do
|
||||||
|
[ "$h" = "$R1" ] && self=1 || self=2
|
||||||
|
log "applying to $h (${V6_PREFIX}::${self})"
|
||||||
|
config_for "$self" "$m" | vyos_apply "$h" \
|
||||||
|
|| die "commit failed on $h -- NOTE: VyOS commits node groups independently, so re-read the config rather than assuming rollback"
|
||||||
|
done
|
||||||
|
log "applied. Nodes should take their reservations within a few minutes."
|
||||||
|
}
|
||||||
|
|
||||||
|
cmd_verify() {
|
||||||
|
local h
|
||||||
|
for h in "$R1" "$R2"; do
|
||||||
|
printf ' --- %s ---\n' "$h"
|
||||||
|
printf ' bond0.2 v6 : %s\n' "$(r "$h" 'ip -6 -br addr show bond0.2 | tr -s " "')"
|
||||||
|
printf ' radvd : %s (inactive on the backup is CORRECT)\n' "$(r "$h" 'systemctl is-active radvd')"
|
||||||
|
printf ' kea-dhcp6 : %s\n' "$(r "$h" 'c=$(ps -ef | grep -c "[k]ea-dhcp6"); [ "$c" -gt 0 ] && echo running || echo "NOT running"')"
|
||||||
|
printf ' role : %s\n' "$(r "$h" 'sudo /config/vrrp-wan-reconcile --status 2>/dev/null | grep -o "role=[a-z]*"')"
|
||||||
|
printf ' IPv4 sane : %s\n' "$(r "$h" 'ip -4 route show default | head -1')"
|
||||||
|
done
|
||||||
|
}
|
||||||
|
|
||||||
|
cmd_revert() {
|
||||||
|
local h self
|
||||||
|
for h in "$R1" "$R2"; do
|
||||||
|
[ "$h" = "$R1" ] && self=1 || self=2
|
||||||
|
log "reverting $h"
|
||||||
|
vyos_apply "$h" <<EOF
|
||||||
|
delete service dhcpv6-server
|
||||||
|
delete service router-advert interface bond0.2
|
||||||
|
delete interfaces bonding bond0 vif 2 address '${V6_PREFIX}::${self}/64'
|
||||||
|
EOF
|
||||||
|
done
|
||||||
|
log "reverted"
|
||||||
|
}
|
||||||
|
|
||||||
|
case "${1:-plan}" in
|
||||||
|
plan) cmd_plan ;;
|
||||||
|
apply) cmd_apply ;;
|
||||||
|
verify) cmd_verify ;;
|
||||||
|
revert) cmd_revert ;;
|
||||||
|
*) die "usage: $0 {plan|apply|verify|revert}" ;;
|
||||||
|
esac
|
||||||
133
migration/vlan2-v6-watchdog
Executable file
133
migration/vlan2-v6-watchdog
Executable file
@@ -0,0 +1,133 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# Auto-revert the VLAN 2 IPv6 change if IPv4 stops working.
|
||||||
|
#
|
||||||
|
# The change it guards is additive and IPv6-only, so in theory it cannot affect
|
||||||
|
# IPv4 at all. This exists because "in theory" is exactly what has been wrong
|
||||||
|
# repeatedly, and because it is applied in an unattended window: the person who
|
||||||
|
# would notice is away, and a router that has lost IPv4 takes the house and the
|
||||||
|
# cluster with it.
|
||||||
|
#
|
||||||
|
# Modelled on wan-drill-watchdog: arm before the risky thing, disarm after, and
|
||||||
|
# in between let the ROUTER decide for itself rather than depending on anything
|
||||||
|
# off-box. A watchdog that needs the network it is protecting is not a watchdog.
|
||||||
|
#
|
||||||
|
# vlan2-v6-watchdog arm [seconds] start watching (default 2400 = 40 min)
|
||||||
|
# vlan2-v6-watchdog disarm stop
|
||||||
|
# vlan2-v6-watchdog status is it armed, and what has it seen
|
||||||
|
#
|
||||||
|
# Runs ON the router, out of /config, detached via setsid so it outlives the ssh
|
||||||
|
# session that started it.
|
||||||
|
set -uo pipefail
|
||||||
|
|
||||||
|
STATE=/run/vlan2-v6-watchdog
|
||||||
|
PIDF=$STATE/pid
|
||||||
|
LOGF=$STATE/log
|
||||||
|
V6_PREFIX="${V6_PREFIX:-2001:470:187e:2}"
|
||||||
|
|
||||||
|
# How long IPv4 must be continuously bad before reverting. Long enough that a
|
||||||
|
# commit's own brief disruption, or one lost probe, does not trigger it; short
|
||||||
|
# enough to matter. The reconciler ticks at 30s, so 90s is three chances.
|
||||||
|
GRACE="${GRACE:-90}"
|
||||||
|
PROBE="${PROBE:-9.9.9.9}"
|
||||||
|
|
||||||
|
log() { mkdir -p "$STATE"; printf '%s %s\n' "$(date -Is)" "$*" >> "$LOGF"; }
|
||||||
|
|
||||||
|
# IPv4 health, asked of the box itself -- and it MUST be role-aware.
|
||||||
|
#
|
||||||
|
# The first version required a default route plus internet on both routers, and
|
||||||
|
# would have reverted on vyos002 within 90s of being armed. That box is the
|
||||||
|
# gated BACKUP: by design it holds no VIP, has bond0.53 disabled and pppoe0
|
||||||
|
# down, so it has no default route and no internet, and that is the correct
|
||||||
|
# resting state rather than a fault. Caught at 20s of the 90s grace.
|
||||||
|
#
|
||||||
|
# So: the master must be able to reach the internet. The backup only has to
|
||||||
|
# still be on the network and able to see its peer -- which is what would
|
||||||
|
# actually be at risk if a VLAN 2 change went wrong.
|
||||||
|
VIP="${VIP:-192.168.1.1}"
|
||||||
|
holds_vip() { ip -4 -o addr show 2>/dev/null | grep -q " ${VIP}/"; }
|
||||||
|
|
||||||
|
v4_ok() {
|
||||||
|
if holds_vip; then
|
||||||
|
ip -4 route show default 2>/dev/null | grep -q . || return 1
|
||||||
|
ping -c1 -W2 "$PROBE" >/dev/null 2>&1
|
||||||
|
else
|
||||||
|
# Backup: management address present and the VIP answers. If the VIP has
|
||||||
|
# gone too then the pair has a bigger problem than this change, and
|
||||||
|
# reverting an IPv6 addition would not help -- so this deliberately does
|
||||||
|
# not fire on peer loss alone.
|
||||||
|
ip -4 -o addr show bond0.1 2>/dev/null | grep -q 'inet ' || return 1
|
||||||
|
ping -c1 -W2 "$VIP" >/dev/null 2>&1
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
revert() {
|
||||||
|
log "REVERTING: IPv4 has been down for ${GRACE}s"
|
||||||
|
# A script file, not vbash -c: the latter never starts a config session and
|
||||||
|
# the commit fails to stderr where nobody sees it.
|
||||||
|
cat > /tmp/vlan2-v6-revert.sh <<'REOF'
|
||||||
|
#!/bin/vbash
|
||||||
|
source /opt/vyatta/etc/functions/script-template
|
||||||
|
configure
|
||||||
|
delete service dhcpv6-server
|
||||||
|
delete service router-advert interface bond0.2
|
||||||
|
commit
|
||||||
|
save
|
||||||
|
exit
|
||||||
|
REOF
|
||||||
|
chmod +x /tmp/vlan2-v6-revert.sh
|
||||||
|
/tmp/vlan2-v6-revert.sh >> "$LOGF" 2>&1
|
||||||
|
# The interface address is deleted separately: it is the one node whose
|
||||||
|
# removal could plausibly disturb something else, so it goes last and its
|
||||||
|
# failure does not block the rest.
|
||||||
|
for n in 1 2; do
|
||||||
|
ip -6 addr del "${V6_PREFIX}::${n}/64" dev bond0.2 2>/dev/null
|
||||||
|
done
|
||||||
|
log "revert done; IPv4 now $(v4_ok && echo OK || echo STILL BAD)"
|
||||||
|
}
|
||||||
|
|
||||||
|
watch_loop() {
|
||||||
|
local deadline=$(( $(date +%s) + $1 )) bad=0
|
||||||
|
log "armed for $1s (grace ${GRACE}s, probe ${PROBE})"
|
||||||
|
while [ "$(date +%s)" -lt "$deadline" ]; do
|
||||||
|
if v4_ok; then
|
||||||
|
[ "$bad" -ne 0 ] && log "IPv4 recovered after ${bad}s"
|
||||||
|
bad=0
|
||||||
|
else
|
||||||
|
bad=$((bad + 10))
|
||||||
|
log "IPv4 bad (${bad}s/${GRACE}s)"
|
||||||
|
if [ "$bad" -ge "$GRACE" ]; then
|
||||||
|
revert
|
||||||
|
log "disarming after revert"
|
||||||
|
rm -f "$PIDF"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
sleep 10
|
||||||
|
done
|
||||||
|
log "expired without incident"
|
||||||
|
rm -f "$PIDF"
|
||||||
|
}
|
||||||
|
|
||||||
|
case "${1:-status}" in
|
||||||
|
arm)
|
||||||
|
mkdir -p "$STATE"
|
||||||
|
[ -f "$PIDF" ] && kill -0 "$(cat "$PIDF")" 2>/dev/null && { echo "already armed"; exit 0; }
|
||||||
|
setsid "$0" _run "${2:-2400}" >/dev/null 2>&1 < /dev/null &
|
||||||
|
sleep 1
|
||||||
|
[ -f "$PIDF" ] && echo " armed (pid $(cat "$PIDF"), ${2:-2400}s)" || echo " FAILED to arm"
|
||||||
|
;;
|
||||||
|
_run)
|
||||||
|
echo $$ > "$PIDF"
|
||||||
|
watch_loop "${2:-2400}"
|
||||||
|
;;
|
||||||
|
disarm)
|
||||||
|
if [ -f "$PIDF" ]; then kill "$(cat "$PIDF")" 2>/dev/null; rm -f "$PIDF"; echo " disarmed"
|
||||||
|
else echo " not armed"; fi
|
||||||
|
;;
|
||||||
|
status)
|
||||||
|
if [ -f "$PIDF" ] && kill -0 "$(cat "$PIDF")" 2>/dev/null; then echo " armed (pid $(cat "$PIDF"))"
|
||||||
|
else echo " not armed"; fi
|
||||||
|
echo " --- log ---"; tail -12 "$LOGF" 2>/dev/null | sed 's/^/ /' || echo " (none)"
|
||||||
|
;;
|
||||||
|
*) echo "usage: $0 {arm [seconds]|disarm|status}" >&2; exit 2 ;;
|
||||||
|
esac
|
||||||
35
migration/window-evidence/2026-09-06-baseline.txt
Normal file
35
migration/window-evidence/2026-09-06-baseline.txt
Normal file
@@ -0,0 +1,35 @@
|
|||||||
|
=== W0 baseline 2026-09-06T22:25:20+01:00 ===
|
||||||
|
--- 10.0.1.252 ---
|
||||||
|
vyos001
|
||||||
|
holds VIP : YES
|
||||||
|
WAN : bond0.53=87.192.101.48 pppoe0=90.251.152.236
|
||||||
|
route : default via 87.192.96.1 dev bond0.53 proto failover metric 1
|
||||||
|
internet: UP
|
||||||
|
ipv6 : UP
|
||||||
|
vrrp : 6 MASTER
|
||||||
|
vlan2 v6 present already? : 0
|
||||||
|
--- 10.0.1.253 ---
|
||||||
|
vyos002
|
||||||
|
holds VIP : no
|
||||||
|
WAN :
|
||||||
|
route :
|
||||||
|
internet: DOWN
|
||||||
|
ipv6 : DOWN
|
||||||
|
vrrp : 6 BACKUP
|
||||||
|
vlan2 v6 present already? : 0
|
||||||
|
--- cluster ---
|
||||||
|
aitopatom-3a1c Ready
|
||||||
|
spark-2935 Ready
|
||||||
|
worker0-k8s0.ad.itaz.eu Ready
|
||||||
|
worker1-k8s0.ad.itaz.eu Ready
|
||||||
|
worker2-k8s0.ad.itaz.eu Ready
|
||||||
|
--- known-good save ---
|
||||||
|
[0;36m[known-good][0m pinned 1118 lines as known-good on vyos001
|
||||||
|
[0;36m[known-good][0m restore with: /config/vyos-known-good restore
|
||||||
|
[0;36m[known-good][0m pinned 1119 lines as known-good on vyos002
|
||||||
|
[0;36m[known-good][0m restore with: /config/vyos-known-good restore
|
||||||
|
--- vrrp-wan-install --check ---
|
||||||
|
10.0.1.252: vrrp-wan in sync
|
||||||
|
10.0.1.253: vrrp-wan in sync
|
||||||
|
vyos001: in sync (533 nodes)
|
||||||
|
vyos002: in sync (534 nodes)
|
||||||
56
migration/window-evidence/2026-09-06-dhcpv6.txt
Normal file
56
migration/window-evidence/2026-09-06-dhcpv6.txt
Normal file
@@ -0,0 +1,56 @@
|
|||||||
|
=== W2: does a production node take a DHCPv6 reservation? ===
|
||||||
|
Window of 2026-09-06, operator offline.
|
||||||
|
|
||||||
|
SHORT ANSWER: the RA half works; the RESERVATION half does not, and the reason
|
||||||
|
is structural rather than a misconfiguration.
|
||||||
|
|
||||||
|
WHAT WORKS
|
||||||
|
- RA is emitted on bond0.2 with AdvManagedFlag on, AdvAutonomous off,
|
||||||
|
AdvLinkMTU 1472, AdvDefaultLifetime 0 (deliberately not a default router).
|
||||||
|
- NetworkManager on worker0 SAW it and acted:
|
||||||
|
dhcp6 (eno1): activation: beginning transaction (timeout in 45 seconds)
|
||||||
|
So "ipv6.method=auto follows the managed flag" is now EVIDENCE, not
|
||||||
|
inference. That was the open question this window set out to close.
|
||||||
|
- kea-dhcp6 runs on both routers and receives SOLICITs on bond0.2.
|
||||||
|
|
||||||
|
WHAT DOES NOT WORK, AND WHY
|
||||||
|
Every DHCPv6 packet kea logs is annotated [no hwaddr info]:
|
||||||
|
|
||||||
|
duid=[00:04:69:d9:30:64:c3:4e:...] [no hwaddr info] <- DUID-UUID (type 4)
|
||||||
|
duid=[00:04:25:11:e0:0a:7b:c4:...] [no hwaddr info] <- DUID-UUID
|
||||||
|
duid=[00:01:00:01:31:78:0a:ae:...] [no hwaddr info] <- DUID-LLT (type 1)
|
||||||
|
|
||||||
|
DHCPv6 identifies a client by DUID, not by MAC. kea can sometimes DERIVE a MAC
|
||||||
|
(its mac-sources machinery), but here it derives nothing -- including from a
|
||||||
|
DUID-LLT, which does embed one. So `hw-address` reservations cannot match, and
|
||||||
|
a node that solicits gets no reserved address.
|
||||||
|
|
||||||
|
VyOS accepting `static-mapping <name> mac` is therefore necessary but NOT
|
||||||
|
sufficient: it renders valid kea config ("hw-address": "78:55:36:08:28:fb",
|
||||||
|
confirmed in /run/kea/kea-dhcp6.conf) that simply never matches these clients.
|
||||||
|
The plan's assumption -- "reservations keyed on MAC, one source of truth with
|
||||||
|
IPv4" -- does not survive contact with DHCPv6.
|
||||||
|
|
||||||
|
OPTIONS, none free, for an attended session:
|
||||||
|
a) Key reservations on DUID (VyOS static-mapping supports `duid`). Works, but a
|
||||||
|
DUID is client-generated: it is a SECOND source of truth, it must be
|
||||||
|
harvested per node, and it changes if a node is rebuilt -- which is exactly
|
||||||
|
the "out of the box for new nodes" property we were trying to preserve.
|
||||||
|
b) Set kea `mac-sources` explicitly and see whether any method recovers a MAC
|
||||||
|
from these clients. Cheap to try; may simply not work for DUID-UUID.
|
||||||
|
c) Drop reservations: hand out a dynamic range and have labctl DISCOVER the
|
||||||
|
node's address rather than predict it. The k3s preflight (914135c) already
|
||||||
|
refuses to write a node-ip the node does not hold, so discovery is safe --
|
||||||
|
but the address is then not knowable before the node boots.
|
||||||
|
d) SLAAC with no-autonomous cleared, and accept EUI-64 addresses. Stable per
|
||||||
|
NIC and needs no server state at all, but the address is not ours to choose.
|
||||||
|
|
||||||
|
A SECOND FINDING, unrelated and worse if unnoticed
|
||||||
|
`service dhcpv6-server` with no listen-interface renders kea6 with
|
||||||
|
interfaces: [ "*" ] -- a DHCPv6 server on EVERY VLAN. Observed live: kea began
|
||||||
|
answering SOLICIT/REQUEST from an unrelated device on bond0.10 (LoT) within
|
||||||
|
seconds of the first apply. Same family as the kea IPv4 cross-VLAN bug (ISC
|
||||||
|
#1117) this estate already fought. Fixed by pinning
|
||||||
|
`set service dhcpv6-server listen-interface bond0.2` plus a subnet-level
|
||||||
|
`interface bond0.2`; both routers now render interfaces: ["bond0.2"].
|
||||||
|
|
||||||
Reference in New Issue
Block a user