VLAN 2 IPv6 applied to production: RA proven, MAC reservations do NOT match
The unattended window's W1/W2. Both routers now carry the VLAN 2 IPv6 config
(addresses ::1/::2, RA with managed-flag + no-autonomous-flag + link-mtu 1472,
DHCPv6 reservations for all five nodes), applied via migration/vlan2-v6-apply
with a vlan2-v6-watchdog armed on both routers throughout. IPv4 untouched:
internet up, 6 MASTER / 6 BACKUP, 5/5 nodes Ready, watchdogs quiet.
default-lifetime 0 on purpose -- addressing without egress. Turning on v6
egress moves image pulls onto a tunnel of unmeasured throughput, and the person
who would notice is away. One line to flip when attended.
THE WINDOW'S QUESTION IS ANSWERED, HALF YES, HALF NO:
YES: NetworkManager follows the managed flag. worker0 logged
dhcp6 (eno1): activation: beginning transaction
minutes after the RA appeared. "ipv6.method=auto will do DHCPv6" is now
evidence, not inference.
NO: the reservations never match. Every packet kea logs is [no hwaddr info] --
the clients identify with DUID-UUID (and one DUID-LLT), kea derives no MAC from
any of them, so hw-address reservations cannot match and no node got its
reserved address. VyOS accepting `static-mapping mac` renders valid kea config
that simply never matches these clients. The "one source of truth with IPv4"
addressing scheme does not survive contact with DHCPv6; options (DUID keys, kea
mac-sources, dynamic range + discovery, or SLAAC) are written up in
migration/window-evidence/2026-09-06-dhcpv6.txt for an attended decision.
FOUND LIVE AND FIXED IN THE SAME WINDOW: `service dhcpv6-server` with no
listen-interface renders kea6 with interfaces: ["*"] -- a DHCPv6 server on
EVERY VLAN. kea was answering an unrelated device on bond0.10 within seconds of
the first apply. Same family as the kea IPv4 cross-VLAN bug (ISC #1117). Now
pinned to bond0.2 on both routers.
Also in this commit, three self-inflicted script bugs found by their own
failures: log() wrote progress lines into the captured config stream (VyOS
rejected each as "Invalid command", leaving the two routers correct but NOT
identical); "Invalid command" was missing from the failure patterns so that run
reported success; and the MAC lookup matched its own freshly-created v6
reservations on the second run, returning doubled MACs. All three fixed, both
routers converged and diffed identical.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-09-06 22:36:47 +01:00
|
|
|
=== W2: does a production node take a DHCPv6 reservation? ===
|
|
|
|
|
Window of 2026-09-06, operator offline.
|
|
|
|
|
|
CORRECTION: MAC reservations DO work -- I measured too early and said otherwise
Commit 3c933b9 concluded "MAC reservations do NOT match". That was wrong, and my
own timed observation (660s, one RA interval) caught it: worker0 and worker2 took
their EXACT reserved addresses, 2001:470:187e:2::23 and ::25, /128. Direct check
confirmed. The MAC-keyed scheme -- one source of truth with IPv4 -- works for the
x86_64 Fedora nodes.
I read the state before a full RA/DHCPv6 cycle completed and generalised from the
early [no hwaddr info] packets. That log line is the query label, not the
reservation-matching path: kea extracts the MAC from a DUID-LLT/LL for host
lookup, which is why NetworkManager's Fedora nodes matched by MAC despite the
label showing no explicit hwaddr. Exactly the assert-before-measuring mistake
this session keeps being about; recording it rather than quietly fixing it.
Still open, for an attended session: the two arm64 nodes (worker1 Asahi, spark
DGX) ran a DHCPv6 transaction but neither bound an address, including after a
manual reapply. Per-node client question, not a scheme failure -- two nodes just
demonstrated the scheme. Not touched unattended.
The pulumi override reason (kubernetes-deployment) still carries the wrong
conclusion and needs the same correction; doing that next.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-09-06 22:49:59 +01:00
|
|
|
SHORT ANSWER: YES for the x86_64 Fedora nodes -- MAC reservations work. My first
|
|
|
|
|
write-up of this file said "MAC reservations do NOT match" and that was WRONG:
|
|
|
|
|
I read the result before a full RA/DHCPv6 cycle had completed. The timed
|
|
|
|
|
observation I set running (660s, one MaxRtrAdvInterval) then showed the opposite,
|
|
|
|
|
and a direct check confirmed it. Recording the mistake because committing the
|
|
|
|
|
premature version to git (3c933b9) is exactly the "assert before measuring"
|
|
|
|
|
failure this session has been about.
|
|
|
|
|
|
|
|
|
|
CONFIRMED, direct check 2026-09-06 ~22:47:
|
|
|
|
|
worker0-k8s0 192.168.8.23 -> 2001:470:187e:2::23/128 MATCHES reservation
|
|
|
|
|
worker2-k8s0 192.168.8.25 -> 2001:470:187e:2::25/128 MATCHES reservation
|
|
|
|
|
Both are the EXACT reserved addresses, /128, DHCPv6-assigned. The MAC-keyed
|
|
|
|
|
scheme -- one source of truth with IPv4 -- works for these nodes.
|
|
|
|
|
|
|
|
|
|
WHY IT WORKS DESPITE "[no hwaddr info]" IN THE KEA LOG
|
|
|
|
|
The DHCP6_QUERY_LABEL "[no hwaddr info]" is about whether the client sent an
|
|
|
|
|
explicit hardware-address option; it is NOT the reservation-matching path. For a
|
|
|
|
|
`hw-address` host reservation kea extracts the MAC from the client's DUID when
|
|
|
|
|
the DUID type carries one (DUID-LLT / DUID-LL embed the link-layer address).
|
|
|
|
|
NetworkManager on the Fedora nodes uses such a DUID, so kea matched ::23 and ::25
|
|
|
|
|
by MAC even though the query label showed no explicit hwaddr. The earlier
|
|
|
|
|
DUID-UUID packets I saw were from other clients, and led me to over-generalise.
|
|
|
|
|
|
ROOT CAUSE found: EUI-64 vs stable-privacy link-local, not arch or DUID
Michal pushed back on "arm64 nodes behave differently" -- correctly. worker2 is
aarch64 and binds fine, so architecture was a coincidence. Chased it to the real
cause, proven by the kea ALLOC_ENGINE log.
The subnet is reservations-only, no dynamic pool. Every node sends DUID-UUID (no
MAC), so kea can only match the hw-address reservation by deriving the MAC from
the source link-local -- which works ONLY for EUI-64 link-locals. worker0/worker2
have ipv6.addr-gen-mode=eui64; worker1/spark use the default (stable-privacy,
RFC 7217), whose link-local embeds no MAC. kea derives nothing, no reservation
matches, no pool exists to fall back to:
ALLOC_ENGINE_V6_ALLOC_FAIL_NO_POOLS: no pools were available
So the "MAC reservation" scheme is really a link-local-EUI-64 scheme, and only
works where every node uses EUI-64 link-locals -- not the modern NM default.
Fix options (attended) written up in the evidence file: enforce
addr-gen-mode=eui64 fleet-wide (smallest, keeps one source of truth), DUID keys,
or dynamic pool + discovery. This is the keying decision the plan flagged, now
with a precise cause behind it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-09-06 22:58:42 +01:00
|
|
|
ROOT CAUSE OF THE TWO NON-BINDING NODES (proven, and it is NOT architecture)
|
|
|
|
|
Corrects an earlier claim in git that called this an "arm64" / "per-node client"
|
|
|
|
|
issue. worker2 is aarch64 and bound fine, so arch was a coincidence. The real
|
|
|
|
|
chain, proven by the kea ALLOC_ENGINE log:
|
|
|
|
|
|
|
|
|
|
1. The VLAN 2 subnet is RESERVATIONS-ONLY -- no dynamic pool (confirmed in the
|
|
|
|
|
rendered kea6 config: pools NONE, 5 reservations).
|
|
|
|
|
2. Every node sends DUID-UUID (type 00:04), which carries no MAC -- hence
|
|
|
|
|
"[no hwaddr info]" on every packet. kea therefore cannot match the
|
|
|
|
|
hw-address reservation from the DUID.
|
|
|
|
|
3. kea falls back to deriving the MAC from the packet's SOURCE LINK-LOCAL,
|
|
|
|
|
which only works when that address is EUI-64 (embeds the MAC).
|
|
|
|
|
4. worker0/worker2 have ipv6.addr-gen-mode=eui64, so their link-local is
|
|
|
|
|
EUI-64 (fe80::7a55:36ff:fe08:28fb embeds 78:55:36:08:28:fb) -> MAC derived
|
|
|
|
|
-> reservation matches -> they get ::23 / ::25.
|
|
|
|
|
5. worker1 (end0) and spark (enP7s7) use addr-gen-mode=default =
|
|
|
|
|
STABLE-PRIVACY (RFC 7217): fe80::ed79:7863:8c2c:1d4c embeds no MAC -> kea
|
|
|
|
|
derives nothing -> no reservation match -> and with no dynamic pool there
|
|
|
|
|
is nothing else to hand out:
|
|
|
|
|
|
|
|
|
|
ALLOC_ENGINE_V6_ALLOC_FAIL_SHARED_NETWORK: 1 subnets have no matching pools
|
|
|
|
|
ALLOC_ENGINE_V6_ALLOC_FAIL_NO_POOLS: no pools were available
|
|
|
|
|
|
|
|
|
|
So the MAC-reservation scheme is really a LINK-LOCAL-EUI-64 scheme in disguise.
|
|
|
|
|
It works only where every node uses EUI-64 link-locals, which is NOT the modern
|
|
|
|
|
NetworkManager default.
|
|
|
|
|
|
|
|
|
|
FIX OPTIONS (attended decision)
|
|
|
|
|
a) Enforce ipv6.addr-gen-mode=eui64 on the cluster NICs fleet-wide and at
|
|
|
|
|
provision time. Smallest change, keeps one-source-of-truth-with-IPv4, and
|
|
|
|
|
is a node setting we already control via NM/labctl. Cost: EUI-64 leaks the
|
|
|
|
|
MAC into the address (irrelevant for infra nodes) and it must be enforced or
|
|
|
|
|
a future node silently fails to bind -- the exact trap that produced this.
|
|
|
|
|
b) Key reservations on DUID instead. Robust to link-local mode, but a DUID is
|
|
|
|
|
client-generated, a second source of truth, and changes on reinstall.
|
|
|
|
|
c) Dynamic pool + labctl discovery (node-ip refuses an absent address already,
|
|
|
|
|
914135c), giving up the "address knowable before boot" property.
|
|
|
|
|
Not (d) a dynamic pool ALONGSIDE reservations: an unmatched node would then get
|
|
|
|
|
SOME address, not its reserved one, so node-ip becomes unpredictable -- worse
|
|
|
|
|
than failing loudly.
|
CORRECTION: MAC reservations DO work -- I measured too early and said otherwise
Commit 3c933b9 concluded "MAC reservations do NOT match". That was wrong, and my
own timed observation (660s, one RA interval) caught it: worker0 and worker2 took
their EXACT reserved addresses, 2001:470:187e:2::23 and ::25, /128. Direct check
confirmed. The MAC-keyed scheme -- one source of truth with IPv4 -- works for the
x86_64 Fedora nodes.
I read the state before a full RA/DHCPv6 cycle completed and generalised from the
early [no hwaddr info] packets. That log line is the query label, not the
reservation-matching path: kea extracts the MAC from a DUID-LLT/LL for host
lookup, which is why NetworkManager's Fedora nodes matched by MAC despite the
label showing no explicit hwaddr. Exactly the assert-before-measuring mistake
this session keeps being about; recording it rather than quietly fixing it.
Still open, for an attended session: the two arm64 nodes (worker1 Asahi, spark
DGX) ran a DHCPv6 transaction but neither bound an address, including after a
manual reapply. Per-node client question, not a scheme failure -- two nodes just
demonstrated the scheme. Not touched unattended.
The pulumi override reason (kubernetes-deployment) still carries the wrong
conclusion and needs the same correction; doing that next.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-09-06 22:49:59 +01:00
|
|
|
|
|
|
|
|
RA HALF (unchanged, and it was always the safe finding)
|
|
|
|
|
RA on bond0.2 carries AdvManagedFlag on, AdvAutonomous off, AdvLinkMTU 1472,
|
|
|
|
|
AdvDefaultLifetime 0. NetworkManager (ipv6.method=auto) follows the managed flag
|
|
|
|
|
and starts DHCPv6. "ipv6.method=auto will do DHCPv6" is evidence, not inference.
|
|
|
|
|
|
|
|
|
|
A SECOND FINDING, unrelated, found live and FIXED in-window
|
|
|
|
|
`service dhcpv6-server` with no listen-interface renders kea6 with
|
|
|
|
|
interfaces: [ "*" ] -- a DHCPv6 server on EVERY VLAN. Observed answering an
|
|
|
|
|
unrelated device on bond0.10 (LoT) within seconds of the first apply. Same
|
|
|
|
|
family as the kea IPv4 cross-VLAN bug (ISC #1117). Fixed by pinning
|
|
|
|
|
listen-interface bond0.2 + a subnet-level interface bond0.2; both routers now
|
|
|
|
|
render interfaces: ["bond0.2"].
|
|
|
|
|
|
|
|
|
|
OPTION (b) mac-sources: VyOS dhcpv6-server global-parameters accepts only
|
|
|
|
|
name-server, so kea mac-sources cannot be passed through config, and editing
|
|
|
|
|
/run/kea/*.conf is banned drift (regenerated every commit). Moot now that the
|
|
|
|
|
default matching works for the Fedora nodes.
|
|
|
|
|
|
|
|
|
|
STILL OPEN FOR THE ATTENDED SESSION
|
|
|
|
|
- the two arm64 nodes: why the DHCPv6 transaction does not complete.
|
|
|
|
|
- whether to keep MAC keys (works for x86 Fedora, unproven for arm) or move to
|
|
|
|
|
DUID keys / a dynamic range + labctl discovery for uniformity.
|
2026-09-06 23:18:55 +01:00
|
|
|
|
|
|
|
|
=== CLOSED 2026-09-06: all five nodes bound, aitopatom done ===
|
|
|
|
|
addr-gen-mode=eui64 rolled out fleet-wide + baked into provisioning (6c94371).
|
|
|
|
|
aitopatom-3a1c reached via michal@ (root@ refused key), took ::27 immediately on
|
|
|
|
|
the flip. Final: worker0 ::23, worker2 ::25, worker1 ::13, spark ::12,
|
|
|
|
|
aitopatom ::27 -- all match their reservations, cluster 5/5 Ready.
|
|
|
|
|
The keying scheme (MAC + enforced EUI-64) is proven and enforced; remaining IPv6
|
|
|
|
|
work is egress + the cluster conversion, both attended.
|