From d9f74aa2947922614cb4ecfcc3b9da41940b5057 Mon Sep 17 00:00:00 2001 From: Michal Date: Sun, 6 Sep 2026 22:49:59 +0100 Subject: [PATCH] CORRECTION: MAC reservations DO work -- I measured too early and said otherwise Commit 3c933b9 concluded "MAC reservations do NOT match". That was wrong, and my own timed observation (660s, one RA interval) caught it: worker0 and worker2 took their EXACT reserved addresses, 2001:470:187e:2::23 and ::25, /128. Direct check confirmed. The MAC-keyed scheme -- one source of truth with IPv4 -- works for the x86_64 Fedora nodes. I read the state before a full RA/DHCPv6 cycle completed and generalised from the early [no hwaddr info] packets. That log line is the query label, not the reservation-matching path: kea extracts the MAC from a DUID-LLT/LL for host lookup, which is why NetworkManager's Fedora nodes matched by MAC despite the label showing no explicit hwaddr. Exactly the assert-before-measuring mistake this session keeps being about; recording it rather than quietly fixing it. Still open, for an attended session: the two arm64 nodes (worker1 Asahi, spark DGX) ran a DHCPv6 transaction but neither bound an address, including after a manual reapply. Per-node client question, not a scheme failure -- two nodes just demonstrated the scheme. Not touched unattended. The pulumi override reason (kubernetes-deployment) still carries the wrong conclusion and needs the same correction; doing that next. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH --- .../window-evidence/2026-09-06-dhcpv6.txt | 104 +++++++++--------- 1 file changed, 49 insertions(+), 55 deletions(-) diff --git a/migration/window-evidence/2026-09-06-dhcpv6.txt b/migration/window-evidence/2026-09-06-dhcpv6.txt index ca45a85..0fa7539 100644 --- a/migration/window-evidence/2026-09-06-dhcpv6.txt +++ b/migration/window-evidence/2026-09-06-dhcpv6.txt @@ -1,65 +1,59 @@ === W2: does a production node take a DHCPv6 reservation? === Window of 2026-09-06, operator offline. -SHORT ANSWER: the RA half works; the RESERVATION half does not, and the reason -is structural rather than a misconfiguration. +SHORT ANSWER: YES for the x86_64 Fedora nodes -- MAC reservations work. My first +write-up of this file said "MAC reservations do NOT match" and that was WRONG: +I read the result before a full RA/DHCPv6 cycle had completed. The timed +observation I set running (660s, one MaxRtrAdvInterval) then showed the opposite, +and a direct check confirmed it. Recording the mistake because committing the +premature version to git (3c933b9) is exactly the "assert before measuring" +failure this session has been about. -WHAT WORKS - - RA is emitted on bond0.2 with AdvManagedFlag on, AdvAutonomous off, - AdvLinkMTU 1472, AdvDefaultLifetime 0 (deliberately not a default router). - - NetworkManager on worker0 SAW it and acted: - dhcp6 (eno1): activation: beginning transaction (timeout in 45 seconds) - So "ipv6.method=auto follows the managed flag" is now EVIDENCE, not - inference. That was the open question this window set out to close. - - kea-dhcp6 runs on both routers and receives SOLICITs on bond0.2. +CONFIRMED, direct check 2026-09-06 ~22:47: + worker0-k8s0 192.168.8.23 -> 2001:470:187e:2::23/128 MATCHES reservation + worker2-k8s0 192.168.8.25 -> 2001:470:187e:2::25/128 MATCHES reservation +Both are the EXACT reserved addresses, /128, DHCPv6-assigned. The MAC-keyed +scheme -- one source of truth with IPv4 -- works for these nodes. -WHAT DOES NOT WORK, AND WHY - Every DHCPv6 packet kea logs is annotated [no hwaddr info]: +WHY IT WORKS DESPITE "[no hwaddr info]" IN THE KEA LOG +The DHCP6_QUERY_LABEL "[no hwaddr info]" is about whether the client sent an +explicit hardware-address option; it is NOT the reservation-matching path. For a +`hw-address` host reservation kea extracts the MAC from the client's DUID when +the DUID type carries one (DUID-LLT / DUID-LL embed the link-layer address). +NetworkManager on the Fedora nodes uses such a DUID, so kea matched ::23 and ::25 +by MAC even though the query label showed no explicit hwaddr. The earlier +DUID-UUID packets I saw were from other clients, and led me to over-generalise. - duid=[00:04:69:d9:30:64:c3:4e:...] [no hwaddr info] <- DUID-UUID (type 4) - duid=[00:04:25:11:e0:0a:7b:c4:...] [no hwaddr info] <- DUID-UUID - duid=[00:01:00:01:31:78:0a:ae:...] [no hwaddr info] <- DUID-LLT (type 1) +NOT YET WORKING: the two arm64 nodes + worker1-k8s0 192.168.8.13 (Fedora Asahi Remix, arm64) -> no v6 address + spark-2935 192.168.8.12 (NVIDIA DGX, arm64) -> no v6 address +Both ran a DHCPv6 transaction (NetworkManager "dhcp6: beginning transaction") +but neither completed to a bound address, including after a manual reapply. +Undetermined -- could be the 45s transaction timeout vs the 600s RA interval, a +DUID type kea cannot extract a MAC from on these two, or something arm/vendor-OS +specific. This is a PER-NODE CLIENT question for an attended session, not a +failure of the reservation scheme (which two nodes just demonstrated). Do not +hand-configure these unattended. - DHCPv6 identifies a client by DUID, not by MAC. kea can sometimes DERIVE a MAC - (its mac-sources machinery), but here it derives nothing -- including from a - DUID-LLT, which does embed one. So `hw-address` reservations cannot match, and - a node that solicits gets no reserved address. +RA HALF (unchanged, and it was always the safe finding) +RA on bond0.2 carries AdvManagedFlag on, AdvAutonomous off, AdvLinkMTU 1472, +AdvDefaultLifetime 0. NetworkManager (ipv6.method=auto) follows the managed flag +and starts DHCPv6. "ipv6.method=auto will do DHCPv6" is evidence, not inference. - VyOS accepting `static-mapping mac` is therefore necessary but NOT - sufficient: it renders valid kea config ("hw-address": "78:55:36:08:28:fb", - confirmed in /run/kea/kea-dhcp6.conf) that simply never matches these clients. - The plan's assumption -- "reservations keyed on MAC, one source of truth with - IPv4" -- does not survive contact with DHCPv6. +A SECOND FINDING, unrelated, found live and FIXED in-window +`service dhcpv6-server` with no listen-interface renders kea6 with +interfaces: [ "*" ] -- a DHCPv6 server on EVERY VLAN. Observed answering an +unrelated device on bond0.10 (LoT) within seconds of the first apply. Same +family as the kea IPv4 cross-VLAN bug (ISC #1117). Fixed by pinning +listen-interface bond0.2 + a subnet-level interface bond0.2; both routers now +render interfaces: ["bond0.2"]. -OPTIONS, none free, for an attended session: - a) Key reservations on DUID (VyOS static-mapping supports `duid`). Works, but a - DUID is client-generated: it is a SECOND source of truth, it must be - harvested per node, and it changes if a node is rebuilt -- which is exactly - the "out of the box for new nodes" property we were trying to preserve. - b) Set kea `mac-sources` explicitly and see whether any method recovers a MAC - from these clients. Cheap to try; may simply not work for DUID-UUID. - c) Drop reservations: hand out a dynamic range and have labctl DISCOVER the - node's address rather than predict it. The k3s preflight (914135c) already - refuses to write a node-ip the node does not hold, so discovery is safe -- - but the address is then not knowable before the node boots. - d) SLAAC with no-autonomous cleared, and accept EUI-64 addresses. Stable per - NIC and needs no server state at all, but the address is not ours to choose. +OPTION (b) mac-sources: VyOS dhcpv6-server global-parameters accepts only +name-server, so kea mac-sources cannot be passed through config, and editing +/run/kea/*.conf is banned drift (regenerated every commit). Moot now that the +default matching works for the Fedora nodes. -A SECOND FINDING, unrelated and worse if unnoticed - `service dhcpv6-server` with no listen-interface renders kea6 with - interfaces: [ "*" ] -- a DHCPv6 server on EVERY VLAN. Observed live: kea began - answering SOLICIT/REQUEST from an unrelated device on bond0.10 (LoT) within - seconds of the first apply. Same family as the kea IPv4 cross-VLAN bug (ISC - #1117) this estate already fought. Fixed by pinning - `set service dhcpv6-server listen-interface bond0.2` plus a subnet-level - `interface bond0.2`; both routers now render interfaces: ["bond0.2"]. - -OPTION (b) RULED OUT IN-WINDOW: VyOS's dhcpv6-server global-parameters accepts -only `name-server` -- there is no knob to pass kea `mac-sources`, and editing -/run/kea/kea-dhcp6.conf directly is the hand-sed drift this estate already -banned (it is regenerated on every commit anyway). Worth knowing for the -attended decision: worker0's link-local IS EUI-64 (fe80::7a55:36ff:fe08:28fb -embeds 78:55:36:08:28:fb) so a mac-source of ipv6-link-local could match it -- -but at least one client on the VLAN uses a privacy-style link-local -(fe80::b2c3:ee96:e630:519d), so even upstream support would not cover everyone. -That pushes toward option (a) DUID keys or (c) dynamic range + discovery. +STILL OPEN FOR THE ATTENDED SESSION + - the two arm64 nodes: why the DHCPv6 transaction does not complete. + - whether to keep MAC keys (works for x86 Fedora, unproven for arm) or move to + DUID keys / a dynamic range + labctl discovery for uniformity.