Rehearsed kubernetes -> cluster-pool on the 3-node labsim cluster, which is the
transition production faces. The switch itself is undramatic: agents stayed up,
the operator adopted the pool, and the agent-not-ready taint deadlock did NOT
occur. That deadlock is specific to ADDING IPv6 -- the agent blocks on an IPv6
pod CIDR that does not exist yet. A v4-only mode switch does not hit it.
The real hazard is quieter. The operator does not preserve which node held which
/24: two nodes swapped CIDRs. Their existing pods kept their old addresses,
which now fall outside the node's range, so every other node routes that prefix
to the wrong node. Cross-node ping to those pods dropped 100% while every pod
stayed Running and every node stayed Ready. Nothing in `kubectl get pods` shows
it.
So "pods kept the same address" is the FAILURE signal here, not the reassurance
it looks like. cilium-ipam-switch.sh verify now flags pods sitting outside their
node's CIDR, which is the check that decides whether a recycle is optional
(it is not) or mandatory (it is).
Recycling every deploy/ds/sts restored it: all pods back inside their node CIDR,
cross-node ping 0% loss. Sequence proven end to end:
preflight -> apply -> restart operator then agents -> unstick if needed ->
recycle all workloads -> verify
Also fixed the recycle hint the script printed: `kubectl rollout restart
deploy,ds,sts -A` is not valid (`unknown shorthand flag: 'A'`), so anyone
following it under pressure would have got an error instead of a recycle.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
k3s documents that dual-stack "cannot be enabled on an existing cluster". Rather
than accept that for a 145-day-old production cluster, build both shapes and
diff them. dualstack-lab.sh builds a single-node IPv4 cluster and a native
dual-stack one, takes a reflink copy of the IPv4 disk so a failed conversion
costs 90 seconds to undo, converts in place, and diffs the results.
Result: the conversion works. etcd data is never touched.
The documented blocker is real but narrower than stated. Cilium reports it
exactly -- `required IPv6 PodCIDR not available` -- because node.spec.podCIDRs
is assigned at join and is immutable, and the Kubernetes IPAM controller will
not add a second family later. That objection only holds while Cilium runs
ipam=kubernetes and therefore reads that field. Switching to cluster-pool IPAM
moves pod CIDR allocation into the CiliumNode CRD, where the operator hands out
both families on a cluster that was born IPv4-only.
Sequence that works, in order:
1. k3s unit gains --cluster-cidr/--service-cidr/--node-ip with both families.
k3s validates the two CIDRs together and refuses to start on a mismatch
("must share the same IP version"), so a partial edit crash-loops rather
than coming up half-configured. That is the safe failure mode.
2. The ServiceCIDR object picks up the IPv6 range on restart -- this is
upstream's supported "single-to-dual-stack preserving the primary
ServiceCIDR" path, and existing Services keep their IPv4 addresses.
3. Cilium to ipam=cluster-pool with an IPv6 pool, then DELETE the CiliumNode
so the operator reallocates; it will not add a family to an existing one.
4. Expect a deadlock here: the agent will not go ready without a pod CIDR, so
the node keeps the node.cilium.io/agent-not-ready taint, so the new
operator that would assign the CIDR cannot schedule. Remove the taint by
hand once to break it.
Verified on the converted cluster: pod with 10.42.0.125 AND fd00:42::4843, and
a PreferDualStack Service holding 10.43.122.115 AND fd00:43::72a3.
The only field that still differs from a native build is node.spec.podCIDRs,
which stays IPv4-only -- immutable, and unused once Cilium owns IPAM. CiliumNode
podCIDRs and ServiceCIDR are identical to the native cluster.
Not yet answered: this is one node. Whether a 3-server etcd cluster converts as
cleanly, and what a rejoining agent does, is the next experiment.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
- Add UserKnownHostsFile=/dev/null to SSH in debug and reprovision commands
- Track install state in log follower so it doesn't exit prematurely on "installed"
- Reorder bastion status check to prioritize active queue over stale installed state
- Update .gitignore with task file entries
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Kickstart installs on real hardware failed silently — no error reporting,
only 3 progress callbacks, zero log streaming. This overhaul makes every
install fully observable.
Kickstart improvements:
- Error trapping in %pre and %post (trap ERR sends failure details to bastion)
- 12+ granular progress stages (was 3): SSH, hostname, k3s prep, EFI boot, metadata
- Background log streamer: tails %post output and batch-sends to /api/log
- bastion_log() function for explicit log lines from kickstart scripts
Bastion API:
- POST /api/log — receives raw log lines from kickstart (single or batch)
- InstallLogBuffer — per-MAC ring buffer (2000 lines) + file persistence
- GET /api/logs/:mac — now returns log_lines + log_total alongside stages
- SSE /api/logs/:mac/follow — uses named events (event: stage vs event: log)
- Progress events forwarded to labd via bastion-progress WebSocket message
- Post-provision k3s logs routed through progressBus (was console-only)
dnsmasq fixes found during VM testing:
- HTTP Boot filename: ipxe-real.efi → ipxe.efi (leftover from old 2-stage approach)
- pxe-service directives: only in proxy mode (breaks OVMF PXE in full mode)
- PXEClient vendor class echo for UEFI firmware compatibility
Integration tests:
- PXE boot test: blank UEFI VM → dnsmasq → HTTP Boot → iPXE → bastion → install
- ISO boot test: blank VM boots from bastion-generated ISO → same flow
- Shared helpers: pxe-network (no DHCP, nftables fix), pxe-vm (UEFI + ISO boot)
- test-provision.sh: runs both PXE + ISO tests with prerequisite checks
- 250GB sparse QCOW2 disk (LVM layout needs ~204GB)
201 unit tests passing (11 new).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
