michal/lab
1
0
Fork 0
Code Issues Pull Requests 3 Actions Packages Projects Releases Wiki Activity

CORRECTION: MAC reservations DO work -- I measured too early and said otherwise
Some checks failed
CI/CD / typecheck (push) Failing after 9s
Details
CI/CD / test (push) Failing after 9s
Details
CI/CD / lint (push) Failing after 25s
Details
CI/CD / build (push) Has been skipped
Details
CI/CD / publish-rpm (push) Has been skipped
Details
CI/CD / publish-deb (push) Has been skipped
Details

Browse Source 
Commit 3c933b9 concluded "MAC reservations do NOT match". That was wrong, and my
own timed observation (660s, one RA interval) caught it: worker0 and worker2 took
their EXACT reserved addresses, 2001:470:187e:2::23 and ::25, /128. Direct check
confirmed. The MAC-keyed scheme -- one source of truth with IPv4 -- works for the
x86_64 Fedora nodes.

I read the state before a full RA/DHCPv6 cycle completed and generalised from the
early [no hwaddr info] packets. That log line is the query label, not the
reservation-matching path: kea extracts the MAC from a DUID-LLT/LL for host
lookup, which is why NetworkManager's Fedora nodes matched by MAC despite the
label showing no explicit hwaddr. Exactly the assert-before-measuring mistake
this session keeps being about; recording it rather than quietly fixing it.

Still open, for an attended session: the two arm64 nodes (worker1 Asahi, spark
DGX) ran a DHCPv6 transaction but neither bound an address, including after a
manual reapply. Per-node client question, not a scheme failure -- two nodes just
demonstrated the scheme. Not touched unattended.

The pulumi override reason (kubernetes-deployment) still carries the wrong
conclusion and needs the same correction; doing that next.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
This commit is contained in:
Michal
2026-09-06 22:49:59 +01:00
parent c392bb9233
commit d9f74aa294
1 changed files with 49 additions and 55 deletions
Download Patch File Download Diff File Expand all files Collapse all files

104
migration/window-evidence/2026-09-06-dhcpv6.txt
View File

@@ -1,65 +1,59 @@
=== W2: does a production node take a DHCPv6 reservation? ===
Window of 2026-09-06, operator offline.
SHORT ANSWER: the RA half works; the RESERVATION half does not, and the reason
is structural rather than a misconfiguration.
SHORT ANSWER: YES for the x86_64 Fedora nodes -- MAC reservations work. My first
write-up of this file said "MAC reservations do NOT match" and that was WRONG:
I read the result before a full RA/DHCPv6 cycle had completed. The timed
observation I set running (660s, one MaxRtrAdvInterval) then showed the opposite,
and a direct check confirmed it. Recording the mistake because committing the
premature version to git (3c933b9) is exactly the "assert before measuring"
failure this session has been about.
WHAT WORKS
- RA is emitted on bond0.2 with AdvManagedFlag on, AdvAutonomous off,
AdvLinkMTU 1472, AdvDefaultLifetime 0 (deliberately not a default router).
- NetworkManager on worker0 SAW it and acted:
dhcp6 (eno1): activation: beginning transaction (timeout in 45 seconds)
So "ipv6.method=auto follows the managed flag" is now EVIDENCE, not
inference. That was the open question this window set out to close.
- kea-dhcp6 runs on both routers and receives SOLICITs on bond0.2.
CONFIRMED, direct check 2026-09-06 ~22:47:
worker0-k8s0 192.168.8.23 -> 2001:470:187e:2::23/128 MATCHES reservation
worker2-k8s0 192.168.8.25 -> 2001:470:187e:2::25/128 MATCHES reservation
Both are the EXACT reserved addresses, /128, DHCPv6-assigned. The MAC-keyed
scheme -- one source of truth with IPv4 -- works for these nodes.
WHAT DOES NOT WORK, AND WHY
Every DHCPv6 packet kea logs is annotated [no hwaddr info]:
WHY IT WORKS DESPITE "[no hwaddr info]" IN THE KEA LOG
The DHCP6_QUERY_LABEL "[no hwaddr info]" is about whether the client sent an
explicit hardware-address option; it is NOT the reservation-matching path. For a
`hw-address` host reservation kea extracts the MAC from the client's DUID when
the DUID type carries one (DUID-LLT / DUID-LL embed the link-layer address).
NetworkManager on the Fedora nodes uses such a DUID, so kea matched ::23 and ::25
by MAC even though the query label showed no explicit hwaddr. The earlier
DUID-UUID packets I saw were from other clients, and led me to over-generalise.
duid=[00:04:69:d9:30:64:c3:4e:...] [no hwaddr info] <- DUID-UUID (type 4)
duid=[00:04:25:11:e0:0a:7b:c4:...] [no hwaddr info] <- DUID-UUID
duid=[00:01:00:01:31:78:0a:ae:...] [no hwaddr info] <- DUID-LLT (type 1)
NOT YET WORKING: the two arm64 nodes
worker1-k8s0 192.168.8.13 (Fedora Asahi Remix, arm64) -> no v6 address
spark-2935 192.168.8.12 (NVIDIA DGX, arm64) -> no v6 address
Both ran a DHCPv6 transaction (NetworkManager "dhcp6: beginning transaction")
but neither completed to a bound address, including after a manual reapply.
Undetermined -- could be the 45s transaction timeout vs the 600s RA interval, a
DUID type kea cannot extract a MAC from on these two, or something arm/vendor-OS
specific. This is a PER-NODE CLIENT question for an attended session, not a
failure of the reservation scheme (which two nodes just demonstrated). Do not
hand-configure these unattended.
DHCPv6 identifies a client by DUID, not by MAC. kea can sometimes DERIVE a MAC
(its mac-sources machinery), but here it derives nothing -- including from a
DUID-LLT, which does embed one. So `hw-address` reservations cannot match, and
a node that solicits gets no reserved address.
RA HALF (unchanged, and it was always the safe finding)
RA on bond0.2 carries AdvManagedFlag on, AdvAutonomous off, AdvLinkMTU 1472,
AdvDefaultLifetime 0. NetworkManager (ipv6.method=auto) follows the managed flag
and starts DHCPv6. "ipv6.method=auto will do DHCPv6" is evidence, not inference.
VyOS accepting `static-mapping <name> mac` is therefore necessary but NOT
sufficient: it renders valid kea config ("hw-address": "78:55:36:08:28:fb",
confirmed in /run/kea/kea-dhcp6.conf) that simply never matches these clients.
The plan's assumption -- "reservations keyed on MAC, one source of truth with
IPv4" -- does not survive contact with DHCPv6.
OPTIONS, none free, for an attended session:
a) Key reservations on DUID (VyOS static-mapping supports `duid`). Works, but a
DUID is client-generated: it is a SECOND source of truth, it must be
harvested per node, and it changes if a node is rebuilt -- which is exactly
the "out of the box for new nodes" property we were trying to preserve.
b) Set kea `mac-sources` explicitly and see whether any method recovers a MAC
from these clients. Cheap to try; may simply not work for DUID-UUID.
c) Drop reservations: hand out a dynamic range and have labctl DISCOVER the
node's address rather than predict it. The k3s preflight (914135c) already
refuses to write a node-ip the node does not hold, so discovery is safe --
but the address is then not knowable before the node boots.
d) SLAAC with no-autonomous cleared, and accept EUI-64 addresses. Stable per
NIC and needs no server state at all, but the address is not ours to choose.
A SECOND FINDING, unrelated and worse if unnoticed
A SECOND FINDING, unrelated, found live and FIXED in-window
`service dhcpv6-server` with no listen-interface renders kea6 with
interfaces: [ "*" ] -- a DHCPv6 server on EVERY VLAN. Observed live: kea began
answering SOLICIT/REQUEST from an unrelated device on bond0.10 (LoT) within
seconds of the first apply. Same family as the kea IPv4 cross-VLAN bug (ISC
#1117) this estate already fought. Fixed by pinning
`set service dhcpv6-server listen-interface bond0.2` plus a subnet-level
`interface bond0.2`; both routers now render interfaces: ["bond0.2"].
interfaces: [ "*" ] -- a DHCPv6 server on EVERY VLAN. Observed answering an
unrelated device on bond0.10 (LoT) within seconds of the first apply. Same
family as the kea IPv4 cross-VLAN bug (ISC #1117). Fixed by pinning
listen-interface bond0.2 + a subnet-level interface bond0.2; both routers now
render interfaces: ["bond0.2"].
OPTION (b) RULED OUT IN-WINDOW: VyOS's dhcpv6-server global-parameters accepts
only `name-server` -- there is no knob to pass kea `mac-sources`, and editing
/run/kea/kea-dhcp6.conf directly is the hand-sed drift this estate already
banned (it is regenerated on every commit anyway). Worth knowing for the
attended decision: worker0's link-local IS EUI-64 (fe80::7a55:36ff:fe08:28fb
embeds 78:55:36:08:28:fb) so a mac-source of ipv6-link-local could match it --
but at least one client on the VLAN uses a privacy-style link-local
(fe80::b2c3:ee96:e630:519d), so even upstream support would not cover everyone.
That pushes toward option (a) DUID keys or (c) dynamic range + discovery.
OPTION (b) mac-sources: VyOS dhcpv6-server global-parameters accepts only
name-server, so kea mac-sources cannot be passed through config, and editing
/run/kea/*.conf is banned drift (regenerated every commit). Moot now that the
default matching works for the Fedora nodes.
STILL OPEN FOR THE ATTENDED SESSION
- the two arm64 nodes: why the DHCPv6 transaction does not complete.
- whether to keep MAC keys (works for x86 Fedora, unproven for arm) or move to
DUID keys / a dynamic range + labctl discovery for uniformity.