labsim: the IPAM switch needs no pod recycle when the CIDR sources agree

Re-ran the 3-node rehearsal, this time from a state that matches production
rather than one I had accidentally skewed.

The earlier "two nodes swapped CIDRs" result was an artefact of my own setup:
labsim had been running cluster-pool with per-node CIDRs that differed from
node.spec, I flipped it to ipam=kubernetes (which resyncs CiliumNode from
node.spec), and flipping back therefore looked like a renumber. Production has
only ever run ipam=kubernetes, and all five nodes were checked: CiliumNode and
node.spec agree everywhere.

From that matching state the switch is close to a non-event: per-node CIDRs
unchanged, every pod still inside its node's range, nothing stranded, no recycle
required. The only disruption is the cilium DaemonSet restarting itself -- one
agent sat in Init:0/6 and one node briefly took the agent-not-ready taint, both
of which cleared on their own. Cross-node connectivity verified after.

So the recycle is CONDITIONAL, not a fixed step, and `verify` is what decides.
Documented both cases in the script header, because the dangerous one is silent:
stranded pods report Running and Ready while being unreachable.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH

labsim/cilium-ipam-switch.sh

@@ -12,6 +12,25 @@
 #   ./cilium-ipam-switch.sh verify      compare against preflight, report renumbering
 #
 #   KUBECONFIG=... ./cilium-ipam-switch.sh preflight
+#
+# Whether a recycle is needed is CONDITIONAL, and `verify` is what decides it.
+#
+# The operator does not preserve which node held which /24 -- it adopts whatever
+# CiliumNode.spec.ipam.podCIDRs already says. So:
+#
+#   * If CiliumNode already agrees with node.spec.podCIDRs on every node -- which
+#     is the case for any cluster that has only ever run ipam=kubernetes, because
+#     the operator syncs one from the other -- the pool adopts the existing
+#     allocation, no node is renumbered, and NO pod recycle is needed. Verified
+#     on the 3-node labsim cluster: CIDRs unchanged, nothing stranded, the only
+#     blip was the cilium DaemonSet restarting itself.
+#
+#   * If the two sources DISAGREE, nodes can swap /24s. Their running pods keep
+#     addresses that no longer fall inside the node's range, every other node
+#     routes that prefix to the wrong node, and those pods go unreachable
+#     cross-node while still showing Running. Then a full recycle is mandatory.
+#
+# Do not skip `verify` on the assumption of the good case. Run it and read it.
 set -uo pipefail
 SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
 STATE="${STATE:-$SCRIPT_DIR/.ipam-switch-state}"