labsim: rehearse the IPAM switch on 3 nodes, and catch the trap in it
Rehearsed kubernetes -> cluster-pool on the 3-node labsim cluster, which is the transition production faces. The switch itself is undramatic: agents stayed up, the operator adopted the pool, and the agent-not-ready taint deadlock did NOT occur. That deadlock is specific to ADDING IPv6 -- the agent blocks on an IPv6 pod CIDR that does not exist yet. A v4-only mode switch does not hit it. The real hazard is quieter. The operator does not preserve which node held which /24: two nodes swapped CIDRs. Their existing pods kept their old addresses, which now fall outside the node's range, so every other node routes that prefix to the wrong node. Cross-node ping to those pods dropped 100% while every pod stayed Running and every node stayed Ready. Nothing in `kubectl get pods` shows it. So "pods kept the same address" is the FAILURE signal here, not the reassurance it looks like. cilium-ipam-switch.sh verify now flags pods sitting outside their node's CIDR, which is the check that decides whether a recycle is optional (it is not) or mandatory (it is). Recycling every deploy/ds/sts restored it: all pods back inside their node CIDR, cross-node ping 0% loss. Sequence proven end to end: preflight -> apply -> restart operator then agents -> unstick if needed -> recycle all workloads -> verify Also fixed the recycle hint the script printed: `kubectl rollout restart deploy,ds,sts -A` is not valid (`unknown shorthand flag: 'A'`), so anyone following it under pressure would have got an error instead of a recycle. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
This commit is contained in:
3
.gitignore
vendored
3
.gitignore
vendored
@@ -34,3 +34,6 @@ bastion/asahi-repo/*.zip
|
||||
|
||||
# Regenerated by labsim/dualstack-lab.sh; derived state, not source.
|
||||
labsim/dualstack-evidence/
|
||||
|
||||
# Runtime snapshots from labsim/cilium-ipam-switch.sh
|
||||
labsim/.ipam-switch-state/
|
||||
|
||||
Reference in New Issue
Block a user