7c2cbfaf31
Some checks failed
Michal pushed back on "arm64 nodes behave differently" -- correctly. worker2 is aarch64 and binds fine, so architecture was a coincidence. Chased it to the real cause, proven by the kea ALLOC_ENGINE log. The subnet is reservations-only, no dynamic pool. Every node sends DUID-UUID (no MAC), so kea can only match the hw-address reservation by deriving the MAC from the source link-local -- which works ONLY for EUI-64 link-locals. worker0/worker2 have ipv6.addr-gen-mode=eui64; worker1/spark use the default (stable-privacy, RFC 7217), whose link-local embeds no MAC. kea derives nothing, no reservation matches, no pool exists to fall back to: ALLOC_ENGINE_V6_ALLOC_FAIL_NO_POOLS: no pools were available So the "MAC reservation" scheme is really a link-local-EUI-64 scheme, and only works where every node uses EUI-64 link-locals -- not the modern NM default. Fix options (attended) written up in the evidence file: enforce addr-gen-mode=eui64 fleet-wide (smallest, keeps one source of truth), DUID keys, or dynamic pool + discovery. This is the keying decision the plan flagged, now with a precise cause behind it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
90 lines
5.2 KiB
Plaintext
90 lines
5.2 KiB
Plaintext
=== W2: does a production node take a DHCPv6 reservation? ===
|
|
Window of 2026-09-06, operator offline.
|
|
|
|
SHORT ANSWER: YES for the x86_64 Fedora nodes -- MAC reservations work. My first
|
|
write-up of this file said "MAC reservations do NOT match" and that was WRONG:
|
|
I read the result before a full RA/DHCPv6 cycle had completed. The timed
|
|
observation I set running (660s, one MaxRtrAdvInterval) then showed the opposite,
|
|
and a direct check confirmed it. Recording the mistake because committing the
|
|
premature version to git (3c933b9) is exactly the "assert before measuring"
|
|
failure this session has been about.
|
|
|
|
CONFIRMED, direct check 2026-09-06 ~22:47:
|
|
worker0-k8s0 192.168.8.23 -> 2001:470:187e:2::23/128 MATCHES reservation
|
|
worker2-k8s0 192.168.8.25 -> 2001:470:187e:2::25/128 MATCHES reservation
|
|
Both are the EXACT reserved addresses, /128, DHCPv6-assigned. The MAC-keyed
|
|
scheme -- one source of truth with IPv4 -- works for these nodes.
|
|
|
|
WHY IT WORKS DESPITE "[no hwaddr info]" IN THE KEA LOG
|
|
The DHCP6_QUERY_LABEL "[no hwaddr info]" is about whether the client sent an
|
|
explicit hardware-address option; it is NOT the reservation-matching path. For a
|
|
`hw-address` host reservation kea extracts the MAC from the client's DUID when
|
|
the DUID type carries one (DUID-LLT / DUID-LL embed the link-layer address).
|
|
NetworkManager on the Fedora nodes uses such a DUID, so kea matched ::23 and ::25
|
|
by MAC even though the query label showed no explicit hwaddr. The earlier
|
|
DUID-UUID packets I saw were from other clients, and led me to over-generalise.
|
|
|
|
ROOT CAUSE OF THE TWO NON-BINDING NODES (proven, and it is NOT architecture)
|
|
Corrects an earlier claim in git that called this an "arm64" / "per-node client"
|
|
issue. worker2 is aarch64 and bound fine, so arch was a coincidence. The real
|
|
chain, proven by the kea ALLOC_ENGINE log:
|
|
|
|
1. The VLAN 2 subnet is RESERVATIONS-ONLY -- no dynamic pool (confirmed in the
|
|
rendered kea6 config: pools NONE, 5 reservations).
|
|
2. Every node sends DUID-UUID (type 00:04), which carries no MAC -- hence
|
|
"[no hwaddr info]" on every packet. kea therefore cannot match the
|
|
hw-address reservation from the DUID.
|
|
3. kea falls back to deriving the MAC from the packet's SOURCE LINK-LOCAL,
|
|
which only works when that address is EUI-64 (embeds the MAC).
|
|
4. worker0/worker2 have ipv6.addr-gen-mode=eui64, so their link-local is
|
|
EUI-64 (fe80::7a55:36ff:fe08:28fb embeds 78:55:36:08:28:fb) -> MAC derived
|
|
-> reservation matches -> they get ::23 / ::25.
|
|
5. worker1 (end0) and spark (enP7s7) use addr-gen-mode=default =
|
|
STABLE-PRIVACY (RFC 7217): fe80::ed79:7863:8c2c:1d4c embeds no MAC -> kea
|
|
derives nothing -> no reservation match -> and with no dynamic pool there
|
|
is nothing else to hand out:
|
|
|
|
ALLOC_ENGINE_V6_ALLOC_FAIL_SHARED_NETWORK: 1 subnets have no matching pools
|
|
ALLOC_ENGINE_V6_ALLOC_FAIL_NO_POOLS: no pools were available
|
|
|
|
So the MAC-reservation scheme is really a LINK-LOCAL-EUI-64 scheme in disguise.
|
|
It works only where every node uses EUI-64 link-locals, which is NOT the modern
|
|
NetworkManager default.
|
|
|
|
FIX OPTIONS (attended decision)
|
|
a) Enforce ipv6.addr-gen-mode=eui64 on the cluster NICs fleet-wide and at
|
|
provision time. Smallest change, keeps one-source-of-truth-with-IPv4, and
|
|
is a node setting we already control via NM/labctl. Cost: EUI-64 leaks the
|
|
MAC into the address (irrelevant for infra nodes) and it must be enforced or
|
|
a future node silently fails to bind -- the exact trap that produced this.
|
|
b) Key reservations on DUID instead. Robust to link-local mode, but a DUID is
|
|
client-generated, a second source of truth, and changes on reinstall.
|
|
c) Dynamic pool + labctl discovery (node-ip refuses an absent address already,
|
|
914135c), giving up the "address knowable before boot" property.
|
|
Not (d) a dynamic pool ALONGSIDE reservations: an unmatched node would then get
|
|
SOME address, not its reserved one, so node-ip becomes unpredictable -- worse
|
|
than failing loudly.
|
|
|
|
RA HALF (unchanged, and it was always the safe finding)
|
|
RA on bond0.2 carries AdvManagedFlag on, AdvAutonomous off, AdvLinkMTU 1472,
|
|
AdvDefaultLifetime 0. NetworkManager (ipv6.method=auto) follows the managed flag
|
|
and starts DHCPv6. "ipv6.method=auto will do DHCPv6" is evidence, not inference.
|
|
|
|
A SECOND FINDING, unrelated, found live and FIXED in-window
|
|
`service dhcpv6-server` with no listen-interface renders kea6 with
|
|
interfaces: [ "*" ] -- a DHCPv6 server on EVERY VLAN. Observed answering an
|
|
unrelated device on bond0.10 (LoT) within seconds of the first apply. Same
|
|
family as the kea IPv4 cross-VLAN bug (ISC #1117). Fixed by pinning
|
|
listen-interface bond0.2 + a subnet-level interface bond0.2; both routers now
|
|
render interfaces: ["bond0.2"].
|
|
|
|
OPTION (b) mac-sources: VyOS dhcpv6-server global-parameters accepts only
|
|
name-server, so kea mac-sources cannot be passed through config, and editing
|
|
/run/kea/*.conf is banned drift (regenerated every commit). Moot now that the
|
|
default matching works for the Fedora nodes.
|
|
|
|
STILL OPEN FOR THE ATTENDED SESSION
|
|
- the two arm64 nodes: why the DHCPv6 transaction does not complete.
|
|
- whether to keep MAC keys (works for x86 Fedora, unproven for arm) or move to
|
|
DUID keys / a dynamic range + labctl discovery for uniformity.
|