From 7c2cbfaf31d4e3e44e4c741613a68d48294c0cfe Mon Sep 17 00:00:00 2001 From: Michal Date: Sun, 6 Sep 2026 22:58:42 +0100 Subject: [PATCH] ROOT CAUSE found: EUI-64 vs stable-privacy link-local, not arch or DUID Michal pushed back on "arm64 nodes behave differently" -- correctly. worker2 is aarch64 and binds fine, so architecture was a coincidence. Chased it to the real cause, proven by the kea ALLOC_ENGINE log. The subnet is reservations-only, no dynamic pool. Every node sends DUID-UUID (no MAC), so kea can only match the hw-address reservation by deriving the MAC from the source link-local -- which works ONLY for EUI-64 link-locals. worker0/worker2 have ipv6.addr-gen-mode=eui64; worker1/spark use the default (stable-privacy, RFC 7217), whose link-local embeds no MAC. kea derives nothing, no reservation matches, no pool exists to fall back to: ALLOC_ENGINE_V6_ALLOC_FAIL_NO_POOLS: no pools were available So the "MAC reservation" scheme is really a link-local-EUI-64 scheme, and only works where every node uses EUI-64 link-locals -- not the modern NM default. Fix options (attended) written up in the evidence file: enforce addr-gen-mode=eui64 fleet-wide (smallest, keeps one source of truth), DUID keys, or dynamic pool + discovery. This is the keying decision the plan flagged, now with a precise cause behind it. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH --- .../window-evidence/2026-09-06-dhcpv6.txt | 50 +++++++++++++++---- 1 file changed, 40 insertions(+), 10 deletions(-) diff --git a/migration/window-evidence/2026-09-06-dhcpv6.txt b/migration/window-evidence/2026-09-06-dhcpv6.txt index 0fa7539..7951266 100644 --- a/migration/window-evidence/2026-09-06-dhcpv6.txt +++ b/migration/window-evidence/2026-09-06-dhcpv6.txt @@ -24,16 +24,46 @@ NetworkManager on the Fedora nodes uses such a DUID, so kea matched ::23 and ::2 by MAC even though the query label showed no explicit hwaddr. The earlier DUID-UUID packets I saw were from other clients, and led me to over-generalise. -NOT YET WORKING: the two arm64 nodes - worker1-k8s0 192.168.8.13 (Fedora Asahi Remix, arm64) -> no v6 address - spark-2935 192.168.8.12 (NVIDIA DGX, arm64) -> no v6 address -Both ran a DHCPv6 transaction (NetworkManager "dhcp6: beginning transaction") -but neither completed to a bound address, including after a manual reapply. -Undetermined -- could be the 45s transaction timeout vs the 600s RA interval, a -DUID type kea cannot extract a MAC from on these two, or something arm/vendor-OS -specific. This is a PER-NODE CLIENT question for an attended session, not a -failure of the reservation scheme (which two nodes just demonstrated). Do not -hand-configure these unattended. +ROOT CAUSE OF THE TWO NON-BINDING NODES (proven, and it is NOT architecture) +Corrects an earlier claim in git that called this an "arm64" / "per-node client" +issue. worker2 is aarch64 and bound fine, so arch was a coincidence. The real +chain, proven by the kea ALLOC_ENGINE log: + + 1. The VLAN 2 subnet is RESERVATIONS-ONLY -- no dynamic pool (confirmed in the + rendered kea6 config: pools NONE, 5 reservations). + 2. Every node sends DUID-UUID (type 00:04), which carries no MAC -- hence + "[no hwaddr info]" on every packet. kea therefore cannot match the + hw-address reservation from the DUID. + 3. kea falls back to deriving the MAC from the packet's SOURCE LINK-LOCAL, + which only works when that address is EUI-64 (embeds the MAC). + 4. worker0/worker2 have ipv6.addr-gen-mode=eui64, so their link-local is + EUI-64 (fe80::7a55:36ff:fe08:28fb embeds 78:55:36:08:28:fb) -> MAC derived + -> reservation matches -> they get ::23 / ::25. + 5. worker1 (end0) and spark (enP7s7) use addr-gen-mode=default = + STABLE-PRIVACY (RFC 7217): fe80::ed79:7863:8c2c:1d4c embeds no MAC -> kea + derives nothing -> no reservation match -> and with no dynamic pool there + is nothing else to hand out: + + ALLOC_ENGINE_V6_ALLOC_FAIL_SHARED_NETWORK: 1 subnets have no matching pools + ALLOC_ENGINE_V6_ALLOC_FAIL_NO_POOLS: no pools were available + + So the MAC-reservation scheme is really a LINK-LOCAL-EUI-64 scheme in disguise. + It works only where every node uses EUI-64 link-locals, which is NOT the modern + NetworkManager default. + +FIX OPTIONS (attended decision) + a) Enforce ipv6.addr-gen-mode=eui64 on the cluster NICs fleet-wide and at + provision time. Smallest change, keeps one-source-of-truth-with-IPv4, and + is a node setting we already control via NM/labctl. Cost: EUI-64 leaks the + MAC into the address (irrelevant for infra nodes) and it must be enforced or + a future node silently fails to bind -- the exact trap that produced this. + b) Key reservations on DUID instead. Robust to link-local mode, but a DUID is + client-generated, a second source of truth, and changes on reinstall. + c) Dynamic pool + labctl discovery (node-ip refuses an absent address already, + 914135c), giving up the "address knowable before boot" property. + Not (d) a dynamic pool ALONGSIDE reservations: an unmatched node would then get + SOME address, not its reserved one, so node-ip becomes unpredictable -- worse + than failing loudly. RA HALF (unchanged, and it was always the safe finding) RA on bond0.2 carries AdvManagedFlag on, AdvAutonomous off, AdvLinkMTU 1472,