Files
lab/labsim/cilium-ipam-switch.sh
Michal e8679f45b5 labsim: rehearse the IPAM switch on 3 nodes, and catch the trap in it
Rehearsed kubernetes -> cluster-pool on the 3-node labsim cluster, which is the
transition production faces. The switch itself is undramatic: agents stayed up,
the operator adopted the pool, and the agent-not-ready taint deadlock did NOT
occur. That deadlock is specific to ADDING IPv6 -- the agent blocks on an IPv6
pod CIDR that does not exist yet. A v4-only mode switch does not hit it.

The real hazard is quieter. The operator does not preserve which node held which
/24: two nodes swapped CIDRs. Their existing pods kept their old addresses,
which now fall outside the node's range, so every other node routes that prefix
to the wrong node. Cross-node ping to those pods dropped 100% while every pod
stayed Running and every node stayed Ready. Nothing in `kubectl get pods` shows
it.

So "pods kept the same address" is the FAILURE signal here, not the reassurance
it looks like. cilium-ipam-switch.sh verify now flags pods sitting outside their
node's CIDR, which is the check that decides whether a recycle is optional
(it is not) or mandatory (it is).

Recycling every deploy/ds/sts restored it: all pods back inside their node CIDR,
cross-node ping 0% loss. Sequence proven end to end:
  preflight -> apply -> restart operator then agents -> unstick if needed ->
  recycle all workloads -> verify

Also fixed the recycle hint the script printed: `kubectl rollout restart
deploy,ds,sts -A` is not valid (`unknown shorthand flag: 'A'`), so anyone
following it under pressure would have got an error instead of a recycle.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-08-24 23:13:20 +01:00

6.3 KiB
Executable File