ROOT CAUSE found: EUI-64 vs stable-privacy link-local, not arch or DUID

Michal pushed back on "arm64 nodes behave differently" -- correctly. worker2 is
aarch64 and binds fine, so architecture was a coincidence. Chased it to the real
cause, proven by the kea ALLOC_ENGINE log.

The subnet is reservations-only, no dynamic pool. Every node sends DUID-UUID (no
MAC), so kea can only match the hw-address reservation by deriving the MAC from
the source link-local -- which works ONLY for EUI-64 link-locals. worker0/worker2
have ipv6.addr-gen-mode=eui64; worker1/spark use the default (stable-privacy,
RFC 7217), whose link-local embeds no MAC. kea derives nothing, no reservation
matches, no pool exists to fall back to:

  ALLOC_ENGINE_V6_ALLOC_FAIL_NO_POOLS: no pools were available

So the "MAC reservation" scheme is really a link-local-EUI-64 scheme, and only
works where every node uses EUI-64 link-locals -- not the modern NM default.

Fix options (attended) written up in the evidence file: enforce
addr-gen-mode=eui64 fleet-wide (smallest, keeps one source of truth), DUID keys,
or dynamic pool + discovery. This is the keying decision the plan flagged, now
with a precise cause behind it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH

migration/window-evidence/2026-09-06-dhcpv6.txt

@@ -24,16 +24,46 @@ NetworkManager on the Fedora nodes uses such a DUID, so kea matched ::23 and ::2
 by MAC even though the query label showed no explicit hwaddr. The earlier
 DUID-UUID packets I saw were from other clients, and led me to over-generalise.
 
-NOT YET WORKING: the two arm64 nodes
+ROOT CAUSE OF THE TWO NON-BINDING NODES (proven, and it is NOT architecture)
-    worker1-k8s0  192.168.8.13  (Fedora Asahi Remix, arm64)  -> no v6 address
+Corrects an earlier claim in git that called this an "arm64" / "per-node client"
-    spark-2935    192.168.8.12  (NVIDIA DGX, arm64)          -> no v6 address
+issue. worker2 is aarch64 and bound fine, so arch was a coincidence. The real
-Both ran a DHCPv6 transaction (NetworkManager "dhcp6: beginning transaction")
+chain, proven by the kea ALLOC_ENGINE log:
-but neither completed to a bound address, including after a manual reapply.
+
-Undetermined -- could be the 45s transaction timeout vs the 600s RA interval, a
+  1. The VLAN 2 subnet is RESERVATIONS-ONLY -- no dynamic pool (confirmed in the
-DUID type kea cannot extract a MAC from on these two, or something arm/vendor-OS
+     rendered kea6 config: pools NONE, 5 reservations).
-specific. This is a PER-NODE CLIENT question for an attended session, not a
+  2. Every node sends DUID-UUID (type 00:04), which carries no MAC -- hence
-failure of the reservation scheme (which two nodes just demonstrated). Do not
+     "[no hwaddr info]" on every packet. kea therefore cannot match the
-hand-configure these unattended.
+     hw-address reservation from the DUID.
+  3. kea falls back to deriving the MAC from the packet's SOURCE LINK-LOCAL,
+     which only works when that address is EUI-64 (embeds the MAC).
+  4. worker0/worker2 have ipv6.addr-gen-mode=eui64, so their link-local is
+     EUI-64 (fe80::7a55:36ff:fe08:28fb embeds 78:55:36:08:28:fb) -> MAC derived
+     -> reservation matches -> they get ::23 / ::25.
+  5. worker1 (end0) and spark (enP7s7) use addr-gen-mode=default =
+     STABLE-PRIVACY (RFC 7217): fe80::ed79:7863:8c2c:1d4c embeds no MAC -> kea
+     derives nothing -> no reservation match -> and with no dynamic pool there
+     is nothing else to hand out:
+
+       ALLOC_ENGINE_V6_ALLOC_FAIL_SHARED_NETWORK: 1 subnets have no matching pools
+       ALLOC_ENGINE_V6_ALLOC_FAIL_NO_POOLS: no pools were available
+
+  So the MAC-reservation scheme is really a LINK-LOCAL-EUI-64 scheme in disguise.
+  It works only where every node uses EUI-64 link-locals, which is NOT the modern
+  NetworkManager default.
+
+FIX OPTIONS (attended decision)
+  a) Enforce ipv6.addr-gen-mode=eui64 on the cluster NICs fleet-wide and at
+     provision time. Smallest change, keeps one-source-of-truth-with-IPv4, and
+     is a node setting we already control via NM/labctl. Cost: EUI-64 leaks the
+     MAC into the address (irrelevant for infra nodes) and it must be enforced or
+     a future node silently fails to bind -- the exact trap that produced this.
+  b) Key reservations on DUID instead. Robust to link-local mode, but a DUID is
+     client-generated, a second source of truth, and changes on reinstall.
+  c) Dynamic pool + labctl discovery (node-ip refuses an absent address already,
+     914135c), giving up the "address knowable before boot" property.
+  Not (d) a dynamic pool ALONGSIDE reservations: an unmatched node would then get
+  SOME address, not its reserved one, so node-ip becomes unpredictable -- worse
+  than failing loudly.
 
 RA HALF (unchanged, and it was always the safe finding)
 RA on bond0.2 carries AdvManagedFlag on, AdvAutonomous off, AdvLinkMTU 1472,