michal/lab
1
0
Fork 0
Code Issues Pull Requests 3 Actions Packages Projects Releases Wiki Activity

labsim: rehearse the IPAM switch on 3 nodes, and catch the trap in it

Browse Source 
Rehearsed kubernetes -> cluster-pool on the 3-node labsim cluster, which is the
transition production faces. The switch itself is undramatic: agents stayed up,
the operator adopted the pool, and the agent-not-ready taint deadlock did NOT
occur. That deadlock is specific to ADDING IPv6 -- the agent blocks on an IPv6
pod CIDR that does not exist yet. A v4-only mode switch does not hit it.

The real hazard is quieter. The operator does not preserve which node held which
/24: two nodes swapped CIDRs. Their existing pods kept their old addresses,
which now fall outside the node's range, so every other node routes that prefix
to the wrong node. Cross-node ping to those pods dropped 100% while every pod
stayed Running and every node stayed Ready. Nothing in `kubectl get pods` shows
it.

So "pods kept the same address" is the FAILURE signal here, not the reassurance
it looks like. cilium-ipam-switch.sh verify now flags pods sitting outside their
node's CIDR, which is the check that decides whether a recycle is optional
(it is not) or mandatory (it is).

Recycling every deploy/ds/sts restored it: all pods back inside their node CIDR,
cross-node ping 0% loss. Sequence proven end to end:
  preflight -> apply -> restart operator then agents -> unstick if needed ->
  recycle all workloads -> verify

Also fixed the recycle hint the script printed: `kubectl rollout restart
deploy,ds,sts -A` is not valid (`unknown shorthand flag: 'A'`), so anyone
following it under pressure would have got an error instead of a recycle.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
This commit is contained in:
Michal
2026-08-24 23:13:20 +01:00
parent 527e0798ae
commit e8679f45b5
2 changed files with 128 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
.gitignore vendored
View File

@@ -34,3 +34,6 @@ bastion/asahi-repo/*.zip
# Regenerated by labsim/dualstack-lab.sh; derived state, not source. # Regenerated by labsim/dualstack-lab.sh; derived state, not source.
labsim/dualstack-evidence/ labsim/dualstack-evidence/
# Runtime snapshots from labsim/cilium-ipam-switch.sh
labsim/.ipam-switch-state/

125
labsim/cilium-ipam-switch.sh Executable file
View File

@@ -0,0 +1,125 @@
#!/usr/bin/env bash
# Procedure around a Cilium IPAM mode change. Works against any cluster, so the
# rehearsal in labsim and the real thing in production run the SAME steps.
#
# It deliberately does NOT change the mode itself. In labsim that is `helm
# upgrade`; in production Pulumi owns the release and a script racing it would
# just reintroduce drift. What this owns is everything around the apply -- the
# evidence, the deadlock, and the verdict.
#
# ./cilium-ipam-switch.sh preflight record what the cluster looks like now
# ./cilium-ipam-switch.sh unstick break the agent-not-ready taint deadlock
# ./cilium-ipam-switch.sh verify compare against preflight, report renumbering
#
# KUBECONFIG=... ./cilium-ipam-switch.sh preflight
set -uo pipefail
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
STATE="${STATE:-$SCRIPT_DIR/.ipam-switch-state}"
K="kubectl"
say() { printf '\033[0;36m[ipam]\033[0m %s\n' "$*"; }
warn() { printf '\033[1;33m[ipam]\033[0m %s\n' "$*" >&2; }
snapshot() {
echo "## nodes"
$K get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.podCIDRs}{"\n"}{end}' 2>/dev/null
echo "## ciliumnodes"
$K get ciliumnode -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.ipam.podCIDRs}{"\n"}{end}' 2>/dev/null
echo "## pods"
$K get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}{"\t"}{.status.podIP}{"\n"}{end}' 2>/dev/null \
| grep -vP '\t$' | sort
echo "## ipam"
$K -n kube-system get cm cilium-config -o jsonpath='{.data.ipam}' 2>/dev/null; echo
}
cmd_preflight() {
mkdir -p "$STATE"
snapshot > "$STATE/before.txt"
say "recorded $(grep -c . "$STATE/before.txt") lines -> $STATE/before.txt"
say "mode now: $(sed -n '/^## ipam/,$p' "$STATE/before.txt" | tail -1)"
# The pod inventory is the rollback reference: if the switch renumbers, this
# is the only record of what an address USED to be.
say "pods on the pod network: $(sed -n '/^## pods/,/^## ipam/p' "$STATE/before.txt" | grep -c '10\.')"
}
# The deadlock, in one place because it WILL happen and doing it by hand under
# time pressure is how the wrong node gets untainted:
# agent has no pod CIDR -> agent not ready -> node keeps
# node.cilium.io/agent-not-ready:NoSchedule -> the operator that would assign
# the CIDR cannot schedule -> agent still has no pod CIDR.
# Removing the taint is safe: it exists to keep normal workloads off a node
# without working networking, and the operator is precisely the thing that fixes
# that. Kubernetes re-adds it on the next agent restart.
cmd_unstick() {
local stuck=0
for n in $($K get nodes -o name 2>/dev/null); do
$K get "$n" -o jsonpath='{.spec.taints[*].key}' 2>/dev/null | grep -q 'agent-not-ready' || continue
warn "${n#node/} carries agent-not-ready; removing so the operator can schedule"
$K taint "$n" node.cilium.io/agent-not-ready- >/dev/null 2>&1 && stuck=$((stuck+1))
done
[ "$stuck" -eq 0 ] && say "no node was stuck" || say "cleared $stuck node(s)"
local pend
pend="$($K -n kube-system get pods -l io.cilium/app=operator --no-headers 2>/dev/null | grep -c Pending)"
[ "${pend:-0}" -gt 0 ] && warn "$pend operator pod(s) still Pending — check tolerations, not just taints"
return 0
}
cmd_verify() {
[ -f "$STATE/before.txt" ] || { warn "no preflight snapshot; nothing to compare"; return 1; }
snapshot > "$STATE/after.txt"
echo
say "mode: $(sed -n '/^## ipam/,$p' "$STATE/before.txt" | tail -1) -> $(sed -n '/^## ipam/,$p' "$STATE/after.txt" | tail -1)"
# The question that decides the size of the maintenance window: did per-node
# CIDRs survive, or was every node renumbered (and every pod with it)?
local moved=0
while IFS=$'\t' read -r node cidr; do
[ -z "${node:-}" ] && continue
local now; now="$(sed -n '/^## ciliumnodes/,/^## pods/p' "$STATE/after.txt" | awk -F'\t' -v n="$node" '$1==n{print $2}')"
if [ -n "$now" ] && [ "$now" != "$cidr" ]; then
printf ' %-16s %s -> %s\n' "$node" "$cidr" "$now"; moved=$((moved+1))
fi
done < <(sed -n '/^## ciliumnodes/,/^## pods/p' "$STATE/before.txt" | grep -P '\t')
if [ "$moved" -eq 0 ]; then
say "per-node CIDRs UNCHANGED — the pool adopted the existing allocation"
else
warn "$moved node(s) renumbered — every pod on them must be recycled"
fi
local before after same
before="$(sed -n '/^## pods/,/^## ipam/p' "$STATE/before.txt" | grep -P '\t10\.' | wc -l)"
after="$(sed -n '/^## pods/,/^## ipam/p' "$STATE/after.txt" | grep -P '\t10\.' | wc -l)"
same="$(comm -12 <(sed -n '/^## pods/,/^## ipam/p' "$STATE/before.txt" | grep -P '\t10\.' | sort) \
<(sed -n '/^## pods/,/^## ipam/p' "$STATE/after.txt" | grep -P '\t10\.' | sort) | wc -l)"
say "pods: $before before, $after after, $same kept the SAME address"
# Keeping the address is NOT the good outcome. If a node's CIDR moved, its
# existing pods keep IPs that no longer fall inside it, every other node routes
# that prefix to the WRONG node, and those pods go unreachable cross-node while
# looking perfectly healthy. Observed in labsim: two nodes swapped CIDRs and
# cross-node ping to their pods dropped 100%, with every pod still Running.
# This is the check that decides whether a recycle is optional or mandatory.
local stranded=0
while read -r ns name ip node; do
[ -z "${node:-}" ] && continue
local cidr; cidr="$($K get ciliumnode "$node" -o jsonpath='{.spec.ipam.podCIDRs[0]}' 2>/dev/null)"
[ -z "$cidr" ] && continue
case "$ip" in
"${cidr%.*/*}".*) ;;
*) printf ' STRANDED %-40s %-15s on %s (now %s)\n' "$ns/$name" "$ip" "$node" "$cidr"; stranded=$((stranded+1)) ;;
esac
done < <($K get pods -A -o jsonpath='{range .items[?(@.status.podIP)]}{.metadata.namespace}{" "}{.metadata.name}{" "}{.status.podIP}{" "}{.spec.nodeName}{"\n"}{end}' 2>/dev/null | grep -E ' 10\.')
if [ "$stranded" -gt 0 ]; then
warn "$stranded pod(s) sit OUTSIDE their node CIDR — unreachable cross-node until recycled"
warn "recycle: for ns in $(kubectl get ns -o name | cut -d/ -f2); do kubectl -n $ns rollout restart deploy,ds,sts 2>/dev/null; done"
else
say "every pod is inside its node CIDR — no recycle needed"
fi
say "not-Running pods: $($K get pods -A --no-headers 2>/dev/null | grep -vcE 'Running|Completed')"
}
case "${1:-}" in
preflight) cmd_preflight ;;
unstick) cmd_unstick ;;
verify) cmd_verify ;;
*) sed -n '2,16p' "$0"; exit 1 ;;
esac