Re-ran the 3-node rehearsal, this time from a state that matches production
rather than one I had accidentally skewed.
The earlier "two nodes swapped CIDRs" result was an artefact of my own setup:
labsim had been running cluster-pool with per-node CIDRs that differed from
node.spec, I flipped it to ipam=kubernetes (which resyncs CiliumNode from
node.spec), and flipping back therefore looked like a renumber. Production has
only ever run ipam=kubernetes, and all five nodes were checked: CiliumNode and
node.spec agree everywhere.
From that matching state the switch is close to a non-event: per-node CIDRs
unchanged, every pod still inside its node's range, nothing stranded, no recycle
required. The only disruption is the cilium DaemonSet restarting itself -- one
agent sat in Init:0/6 and one node briefly took the agent-not-ready taint, both
of which cleared on their own. Cross-node connectivity verified after.
So the recycle is CONDITIONAL, not a fixed step, and `verify` is what decides.
Documented both cases in the script header, because the dangerous one is silent:
stranded pods report Running and Ready while being unreachable.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
Rehearsed kubernetes -> cluster-pool on the 3-node labsim cluster, which is the
transition production faces. The switch itself is undramatic: agents stayed up,
the operator adopted the pool, and the agent-not-ready taint deadlock did NOT
occur. That deadlock is specific to ADDING IPv6 -- the agent blocks on an IPv6
pod CIDR that does not exist yet. A v4-only mode switch does not hit it.
The real hazard is quieter. The operator does not preserve which node held which
/24: two nodes swapped CIDRs. Their existing pods kept their old addresses,
which now fall outside the node's range, so every other node routes that prefix
to the wrong node. Cross-node ping to those pods dropped 100% while every pod
stayed Running and every node stayed Ready. Nothing in `kubectl get pods` shows
it.
So "pods kept the same address" is the FAILURE signal here, not the reassurance
it looks like. cilium-ipam-switch.sh verify now flags pods sitting outside their
node's CIDR, which is the check that decides whether a recycle is optional
(it is not) or mandatory (it is).
Recycling every deploy/ds/sts restored it: all pods back inside their node CIDR,
cross-node ping 0% loss. Sequence proven end to end:
preflight -> apply -> restart operator then agents -> unstick if needed ->
recycle all workloads -> verify
Also fixed the recycle hint the script printed: `kubectl rollout restart
deploy,ds,sts -A` is not valid (`unknown shorthand flag: 'A'`), so anyone
following it under pressure would have got an error instead of a recycle.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
