From 11344eab9245a1fc463a901b580683d7c22c08a9 Mon Sep 17 00:00:00 2001 From: Michal Date: Mon, 24 Aug 2026 23:20:16 +0100 Subject: [PATCH] labsim: the IPAM switch needs no pod recycle when the CIDR sources agree Re-ran the 3-node rehearsal, this time from a state that matches production rather than one I had accidentally skewed. The earlier "two nodes swapped CIDRs" result was an artefact of my own setup: labsim had been running cluster-pool with per-node CIDRs that differed from node.spec, I flipped it to ipam=kubernetes (which resyncs CiliumNode from node.spec), and flipping back therefore looked like a renumber. Production has only ever run ipam=kubernetes, and all five nodes were checked: CiliumNode and node.spec agree everywhere. From that matching state the switch is close to a non-event: per-node CIDRs unchanged, every pod still inside its node's range, nothing stranded, no recycle required. The only disruption is the cilium DaemonSet restarting itself -- one agent sat in Init:0/6 and one node briefly took the agent-not-ready taint, both of which cleared on their own. Cross-node connectivity verified after. So the recycle is CONDITIONAL, not a fixed step, and `verify` is what decides. Documented both cases in the script header, because the dangerous one is silent: stranded pods report Running and Ready while being unreachable. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH --- labsim/cilium-ipam-switch.sh | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/labsim/cilium-ipam-switch.sh b/labsim/cilium-ipam-switch.sh index e703343..144cd0f 100755 --- a/labsim/cilium-ipam-switch.sh +++ b/labsim/cilium-ipam-switch.sh @@ -12,6 +12,25 @@ # ./cilium-ipam-switch.sh verify compare against preflight, report renumbering # # KUBECONFIG=... ./cilium-ipam-switch.sh preflight +# +# Whether a recycle is needed is CONDITIONAL, and `verify` is what decides it. +# +# The operator does not preserve which node held which /24 -- it adopts whatever +# CiliumNode.spec.ipam.podCIDRs already says. So: +# +# * If CiliumNode already agrees with node.spec.podCIDRs on every node -- which +# is the case for any cluster that has only ever run ipam=kubernetes, because +# the operator syncs one from the other -- the pool adopts the existing +# allocation, no node is renumbered, and NO pod recycle is needed. Verified +# on the 3-node labsim cluster: CIDRs unchanged, nothing stranded, the only +# blip was the cilium DaemonSet restarting itself. +# +# * If the two sources DISAGREE, nodes can swap /24s. Their running pods keep +# addresses that no longer fall inside the node's range, every other node +# routes that prefix to the wrong node, and those pods go unreachable +# cross-node while still showing Running. Then a full recycle is mandatory. +# +# Do not skip `verify` on the assumption of the good case. Run it and read it. set -uo pipefail SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" STATE="${STATE:-$SCRIPT_DIR/.ipam-switch-state}"