3c933b96e0
Some checks failed
The unattended window's W1/W2. Both routers now carry the VLAN 2 IPv6 config (addresses ::1/::2, RA with managed-flag + no-autonomous-flag + link-mtu 1472, DHCPv6 reservations for all five nodes), applied via migration/vlan2-v6-apply with a vlan2-v6-watchdog armed on both routers throughout. IPv4 untouched: internet up, 6 MASTER / 6 BACKUP, 5/5 nodes Ready, watchdogs quiet. default-lifetime 0 on purpose -- addressing without egress. Turning on v6 egress moves image pulls onto a tunnel of unmeasured throughput, and the person who would notice is away. One line to flip when attended. THE WINDOW'S QUESTION IS ANSWERED, HALF YES, HALF NO: YES: NetworkManager follows the managed flag. worker0 logged dhcp6 (eno1): activation: beginning transaction minutes after the RA appeared. "ipv6.method=auto will do DHCPv6" is now evidence, not inference. NO: the reservations never match. Every packet kea logs is [no hwaddr info] -- the clients identify with DUID-UUID (and one DUID-LLT), kea derives no MAC from any of them, so hw-address reservations cannot match and no node got its reserved address. VyOS accepting `static-mapping mac` renders valid kea config that simply never matches these clients. The "one source of truth with IPv4" addressing scheme does not survive contact with DHCPv6; options (DUID keys, kea mac-sources, dynamic range + discovery, or SLAAC) are written up in migration/window-evidence/2026-09-06-dhcpv6.txt for an attended decision. FOUND LIVE AND FIXED IN THE SAME WINDOW: `service dhcpv6-server` with no listen-interface renders kea6 with interfaces: ["*"] -- a DHCPv6 server on EVERY VLAN. kea was answering an unrelated device on bond0.10 within seconds of the first apply. Same family as the kea IPv4 cross-VLAN bug (ISC #1117). Now pinned to bond0.2 on both routers. Also in this commit, three self-inflicted script bugs found by their own failures: log() wrote progress lines into the captured config stream (VyOS rejected each as "Invalid command", leaving the two routers correct but NOT identical); "Invalid command" was missing from the failure patterns so that run reported success; and the MAC lookup matched its own freshly-created v6 reservations on the second run, returning doubled MACs. All three fixed, both routers converged and diffed identical. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
190 lines
9.1 KiB
Bash
Executable File
190 lines
9.1 KiB
Bash
Executable File
#!/bin/bash
|
|
# Give VLAN 2 (the k8s VLAN) IPv6: addresses, router advertisements and DHCPv6
|
|
# reservations. Production half of dual-stack phase 2b.
|
|
#
|
|
# Rehearsed first as labsim/labsim-dualstack-net.sh, which is where the four
|
|
# VyOS facts below were paid for rather than guessed.
|
|
#
|
|
# ADDRESSING ONLY -- NOT EGRESS. The prefix is advertised with
|
|
# `default-lifetime 0`, so nodes get their reserved addresses but neither router
|
|
# becomes an IPv6 default router. Turning on real v6 egress moves cluster image
|
|
# pulls onto the HE tunnel (1480, or 1472 on PPPoE) whose throughput has never
|
|
# been measured, and that is not a thing to switch on unattended. Flipping it is
|
|
# one line: `set service router-advert interface bond0.2 default-lifetime '1800'`
|
|
# plus a default-preference, once somebody is watching.
|
|
#
|
|
# LISTEN-INTERFACE IS NOT OPTIONAL. Without it kea6 renders
|
|
# `interfaces: [ "*" ]` and serves DHCPv6 on EVERY VLAN, not just this one.
|
|
# Observed in production the moment this was first applied: kea started
|
|
# answering SOLICIT/REQUEST from an unrelated device on bond0.10 (LoT). That is
|
|
# a DHCPv6 server switched on estate-wide as a side effect of configuring one
|
|
# VLAN -- the same family of mistake as the kea IPv4 cross-VLAN bug (ISC #1117)
|
|
# this estate already fought. Pin the interface.
|
|
#
|
|
# WHY DHCPv6 RATHER THAN SLAAC: k3s resolves node-ip once at start-up, so a
|
|
# node's address must be knowable in advance and stable. The estate already
|
|
# answers that for IPv4 with kea reservations keyed on MAC; IPv6 answers it the
|
|
# same way, from the same MACs, so there is one source of truth. VyOS's
|
|
# static-mapping accepts `mac` as well as `duid`, which is what makes that
|
|
# possible -- DHCPv6 normally keys on a client-generated DUID.
|
|
#
|
|
# vlan2-v6-apply plan print what would be applied, change nothing
|
|
# vlan2-v6-apply apply apply to both routers
|
|
# vlan2-v6-apply verify what the routers and nodes now hold
|
|
# vlan2-v6-apply revert remove it again
|
|
set -uo pipefail
|
|
|
|
R1="${R1:-10.0.1.252}" # vyos001 -> ::1
|
|
R2="${R2:-10.0.1.253}" # vyos002 -> ::2
|
|
PW="${VYOS_PW:-vyos}"
|
|
V6_PREFIX="${V6_PREFIX:-2001:470:187e:2}"
|
|
LINK_MTU="${LINK_MTU:-1472}"
|
|
SUBNET_ID="${SUBNET_ID:-2}" # VyOS requires a unique id per DHCPv6 subnet
|
|
SHARED_NET="${SHARED_NET:-TheLab-k8s}"
|
|
|
|
# name:v4-host-octet -- the IPv6 host part mirrors the IPv4 one so a reservation
|
|
# is readable next to its twin. MACs are read from the LIVE IPv4 reservations at
|
|
# run time, never duplicated here: one source of truth, and a node that is
|
|
# re-homed cannot end up with a stale v6 mapping.
|
|
NODES=(worker0-k8s0:23 worker1-k8s0:13 worker2-k8s0:25 spark-2935:12 aitopatom-3a1c:27)
|
|
|
|
SSH=(-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null
|
|
-o LogLevel=ERROR -o ConnectTimeout=8)
|
|
|
|
# stderr, NOT stdout. build_mappings() is captured with $(...) and log() lines
|
|
# went straight into the config stream, where VyOS rejected each one as
|
|
# "Invalid command: [[0" -- ANSI escapes and all. The valid sets still applied,
|
|
# so the routers ended up correct but NOT identical: different lines were lost
|
|
# on each. A progress message is not data; keep it off the data channel.
|
|
log() { printf '\033[0;36m[vlan2-v6]\033[0m %s\n' "$*" >&2; }
|
|
die() { printf '\033[0;31m[vlan2-v6]\033[0m %s\n' "$*" >&2; exit 1; }
|
|
|
|
r() { timeout 45 ssh "${SSH[@]}" "vyos@$1" "${@:2}" 2>/dev/null; }
|
|
|
|
# Drive VyOS from a script FILE with plain commit + save.
|
|
# - `vbash -c` never starts a config session; the commit fails to stderr and a
|
|
# helper discards it, so the run reports success having changed nothing.
|
|
# - `commit-confirm` hangs non-interactively and strands an orphaned
|
|
# config-mgmt commit_confirm holding the config lock.
|
|
# - vbash exits 0 even when the commit fails, so the OUTPUT is the only honest
|
|
# signal. Read it.
|
|
vyos_apply() {
|
|
local h="$1" out
|
|
out="$({ printf '#!/bin/vbash\nsource /opt/vyatta/etc/functions/script-template\nconfigure\n'
|
|
cat
|
|
printf 'commit\nsave\nexit\n'
|
|
} | timeout 150 ssh "${SSH[@]}" "vyos@$h" \
|
|
'cat > /tmp/vlan2-v6.sh && chmod +x /tmp/vlan2-v6.sh && sudo /tmp/vlan2-v6.sh' 2>&1)"
|
|
printf '%s\n' "$out" | grep -vE '^\s*$' | sed 's/^/ /' | tail -8
|
|
# "Invalid command" was NOT in this list the first time, so a run that fed
|
|
# rubbish to VyOS reported success. vbash exits 0 regardless, so every
|
|
# rejection shape has to be named explicitly.
|
|
printf '%s' "$out" | grep -qiE 'Commit failed|\[\[.*\]\] failed|Set failed|Invalid command' && return 1
|
|
return 0
|
|
}
|
|
|
|
# Read each node's MAC out of the live IPv4 reservation.
|
|
#
|
|
# Scoped to the IPv4 subnet on purpose. Once this script has run once, the node
|
|
# has TWO `static-mapping <name> mac` lines -- the v4 one and the v6 one it just
|
|
# created -- and an unscoped match returned both concatenated
|
|
# ("9c:76:0e:49:e9:179c:76:0e:49:e9:17"), which VyOS then rejected as an invalid
|
|
# value. Self-inflicted on the second run: the lookup has to name the subnet it
|
|
# means, or the script poisons its own input as soon as it succeeds.
|
|
V4_SUBNET="${V4_SUBNET:-192.168.8.0/23}"
|
|
mac_of() {
|
|
r "$R1" "/opt/vyatta/bin/vyatta-op-cmd-wrapper show configuration commands 2>/dev/null \
|
|
| grep -F 'subnet $V4_SUBNET' \
|
|
| sed -n \"s/.*static-mapping $1 mac '\\(.*\\)'/\\1/p\"" | tr -d ' \n'
|
|
}
|
|
|
|
build_mappings() {
|
|
local out="" name hextet mac
|
|
for entry in "${NODES[@]}"; do
|
|
name="${entry%%:*}"; hextet="${entry##*:}"
|
|
mac="$(mac_of "$name")"
|
|
[ -n "$mac" ] || die "no IPv4 reservation found for $name -- refusing to invent a MAC"
|
|
log " $name $mac -> ${V6_PREFIX}::${hextet}"
|
|
[ -n "$out" ] && out+=$'\n'
|
|
out+="set service dhcpv6-server shared-network-name ${SHARED_NET} subnet ${V6_PREFIX}::/64 static-mapping ${name} mac '${mac}'
|
|
set service dhcpv6-server shared-network-name ${SHARED_NET} subnet ${V6_PREFIX}::/64 static-mapping ${name} ipv6-address '${V6_PREFIX}::${hextet}'"
|
|
done
|
|
printf '%s' "$out"
|
|
}
|
|
|
|
# One commit per router, and `no-autonomous-flag` is in it. That is not a
|
|
# style choice: turning autonomous off LATER does not retract addresses already
|
|
# formed, so a prefix advertised even briefly without it leaves every node
|
|
# holding an unreserved EUI-64 address with a 30-day lifetime. Observed in the
|
|
# sim. VLAN 2 has no IPv6 today, so this is the one chance to get it right.
|
|
config_for() {
|
|
local self="$1" mappings="$2"
|
|
cat <<EOF
|
|
set interfaces bonding bond0 vif 2 address '${V6_PREFIX}::${self}/64'
|
|
set service router-advert interface bond0.2 prefix ${V6_PREFIX}::/64 no-autonomous-flag
|
|
set service router-advert interface bond0.2 prefix ${V6_PREFIX}::/64 valid-lifetime '2592000'
|
|
set service router-advert interface bond0.2 prefix ${V6_PREFIX}::/64 preferred-lifetime '604800'
|
|
set service router-advert interface bond0.2 managed-flag
|
|
set service router-advert interface bond0.2 link-mtu '${LINK_MTU}'
|
|
set service router-advert interface bond0.2 default-lifetime '0'
|
|
set service dhcpv6-server listen-interface bond0.2
|
|
set service dhcpv6-server shared-network-name ${SHARED_NET} subnet ${V6_PREFIX}::/64 subnet-id '${SUBNET_ID}'
|
|
set service dhcpv6-server shared-network-name ${SHARED_NET} subnet ${V6_PREFIX}::/64 interface 'bond0.2'
|
|
${mappings}
|
|
EOF
|
|
}
|
|
|
|
cmd_plan() {
|
|
log "reading MACs from the live IPv4 reservations"
|
|
local m; m="$(build_mappings)" || exit 1
|
|
echo; log "--- vyos001 (${V6_PREFIX}::1) ---"; config_for 1 "$m" | sed 's/^/ /'
|
|
echo; log "--- vyos002 (${V6_PREFIX}::2) ---"; config_for 2 "$m" | sed 's/^/ /'
|
|
}
|
|
|
|
cmd_apply() {
|
|
log "reading MACs from the live IPv4 reservations"
|
|
local m; m="$(build_mappings)" || exit 1
|
|
local h self
|
|
for h in "$R1" "$R2"; do
|
|
[ "$h" = "$R1" ] && self=1 || self=2
|
|
log "applying to $h (${V6_PREFIX}::${self})"
|
|
config_for "$self" "$m" | vyos_apply "$h" \
|
|
|| die "commit failed on $h -- NOTE: VyOS commits node groups independently, so re-read the config rather than assuming rollback"
|
|
done
|
|
log "applied. Nodes should take their reservations within a few minutes."
|
|
}
|
|
|
|
cmd_verify() {
|
|
local h
|
|
for h in "$R1" "$R2"; do
|
|
printf ' --- %s ---\n' "$h"
|
|
printf ' bond0.2 v6 : %s\n' "$(r "$h" 'ip -6 -br addr show bond0.2 | tr -s " "')"
|
|
printf ' radvd : %s (inactive on the backup is CORRECT)\n' "$(r "$h" 'systemctl is-active radvd')"
|
|
printf ' kea-dhcp6 : %s\n' "$(r "$h" 'c=$(ps -ef | grep -c "[k]ea-dhcp6"); [ "$c" -gt 0 ] && echo running || echo "NOT running"')"
|
|
printf ' role : %s\n' "$(r "$h" 'sudo /config/vrrp-wan-reconcile --status 2>/dev/null | grep -o "role=[a-z]*"')"
|
|
printf ' IPv4 sane : %s\n' "$(r "$h" 'ip -4 route show default | head -1')"
|
|
done
|
|
}
|
|
|
|
cmd_revert() {
|
|
local h self
|
|
for h in "$R1" "$R2"; do
|
|
[ "$h" = "$R1" ] && self=1 || self=2
|
|
log "reverting $h"
|
|
vyos_apply "$h" <<EOF
|
|
delete service dhcpv6-server
|
|
delete service router-advert interface bond0.2
|
|
delete interfaces bonding bond0 vif 2 address '${V6_PREFIX}::${self}/64'
|
|
EOF
|
|
done
|
|
log "reverted"
|
|
}
|
|
|
|
case "${1:-plan}" in
|
|
plan) cmd_plan ;;
|
|
apply) cmd_apply ;;
|
|
verify) cmd_verify ;;
|
|
revert) cmd_revert ;;
|
|
*) die "usage: $0 {plan|apply|verify|revert}" ;;
|
|
esac
|