aitopatom-3a1c (the DGX that refused my key) reached via michal@ with passwordless
sudo, took its reserved ::27 immediately on the eui64 flip. All five cluster nodes
now hold their exact VLAN 2 reservations, 5/5 Ready.
Node addressing is done and enforced (provisioning default + fleet drop-in). What
remains for IPv6 is the egress flip and the cluster conversion, both attended.
Docmost IPv6 page updated to match.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
Michal pushed back on "arm64 nodes behave differently" -- correctly. worker2 is
aarch64 and binds fine, so architecture was a coincidence. Chased it to the real
cause, proven by the kea ALLOC_ENGINE log.
The subnet is reservations-only, no dynamic pool. Every node sends DUID-UUID (no
MAC), so kea can only match the hw-address reservation by deriving the MAC from
the source link-local -- which works ONLY for EUI-64 link-locals. worker0/worker2
have ipv6.addr-gen-mode=eui64; worker1/spark use the default (stable-privacy,
RFC 7217), whose link-local embeds no MAC. kea derives nothing, no reservation
matches, no pool exists to fall back to:
ALLOC_ENGINE_V6_ALLOC_FAIL_NO_POOLS: no pools were available
So the "MAC reservation" scheme is really a link-local-EUI-64 scheme, and only
works where every node uses EUI-64 link-locals -- not the modern NM default.
Fix options (attended) written up in the evidence file: enforce
addr-gen-mode=eui64 fleet-wide (smallest, keeps one source of truth), DUID keys,
or dynamic pool + discovery. This is the keying decision the plan flagged, now
with a precise cause behind it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
VLAN 2 IPv6 live on both routers and in the model. IPv4 untouched (6 MASTER /
6 BACKUP, default route on bond0.53, watchdog 0 reverts, 5/5 Ready). worker0 and
worker2 hold their reserved ::23/::25; the two arm64 nodes are the open item.
Watchdog disarmed. Full handoff at the top of window-evidence/2026-09-06-final.txt.
vyos002 showing internet DOWN is the correct resting state of the gated backup,
not a fault.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
Commit 3c933b9 concluded "MAC reservations do NOT match". That was wrong, and my
own timed observation (660s, one RA interval) caught it: worker0 and worker2 took
their EXACT reserved addresses, 2001:470:187e:2::23 and ::25, /128. Direct check
confirmed. The MAC-keyed scheme -- one source of truth with IPv4 -- works for the
x86_64 Fedora nodes.
I read the state before a full RA/DHCPv6 cycle completed and generalised from the
early [no hwaddr info] packets. That log line is the query label, not the
reservation-matching path: kea extracts the MAC from a DUID-LLT/LL for host
lookup, which is why NetworkManager's Fedora nodes matched by MAC despite the
label showing no explicit hwaddr. Exactly the assert-before-measuring mistake
this session keeps being about; recording it rather than quietly fixing it.
Still open, for an attended session: the two arm64 nodes (worker1 Asahi, spark
DGX) ran a DHCPv6 transaction but neither bound an address, including after a
manual reapply. Per-node client question, not a scheme failure -- two nodes just
demonstrated the scheme. Not touched unattended.
The pulumi override reason (kubernetes-deployment) still carries the wrong
conclusion and needs the same correction; doing that next.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
The unattended window's W1/W2. Both routers now carry the VLAN 2 IPv6 config
(addresses ::1/::2, RA with managed-flag + no-autonomous-flag + link-mtu 1472,
DHCPv6 reservations for all five nodes), applied via migration/vlan2-v6-apply
with a vlan2-v6-watchdog armed on both routers throughout. IPv4 untouched:
internet up, 6 MASTER / 6 BACKUP, 5/5 nodes Ready, watchdogs quiet.
default-lifetime 0 on purpose -- addressing without egress. Turning on v6
egress moves image pulls onto a tunnel of unmeasured throughput, and the person
who would notice is away. One line to flip when attended.
THE WINDOW'S QUESTION IS ANSWERED, HALF YES, HALF NO:
YES: NetworkManager follows the managed flag. worker0 logged
dhcp6 (eno1): activation: beginning transaction
minutes after the RA appeared. "ipv6.method=auto will do DHCPv6" is now
evidence, not inference.
NO: the reservations never match. Every packet kea logs is [no hwaddr info] --
the clients identify with DUID-UUID (and one DUID-LLT), kea derives no MAC from
any of them, so hw-address reservations cannot match and no node got its
reserved address. VyOS accepting `static-mapping mac` renders valid kea config
that simply never matches these clients. The "one source of truth with IPv4"
addressing scheme does not survive contact with DHCPv6; options (DUID keys, kea
mac-sources, dynamic range + discovery, or SLAAC) are written up in
migration/window-evidence/2026-09-06-dhcpv6.txt for an attended decision.
FOUND LIVE AND FIXED IN THE SAME WINDOW: `service dhcpv6-server` with no
listen-interface renders kea6 with interfaces: ["*"] -- a DHCPv6 server on
EVERY VLAN. kea was answering an unrelated device on bond0.10 within seconds of
the first apply. Same family as the kea IPv4 cross-VLAN bug (ISC #1117). Now
pinned to bond0.2 on both routers.
Also in this commit, three self-inflicted script bugs found by their own
failures: log() wrote progress lines into the captured config stream (VyOS
rejected each as "Invalid command", leaving the two routers correct but NOT
identical); "Invalid command" was missing from the failure patterns so that run
reported success; and the MAC lookup matched its own freshly-created v6
reservations on the second run, returning doubled MACs. All three fixed, both
routers converged and diffed identical.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
