From 97ae6dea8948f95ea91c5e086dd80f6264d60be7 Mon Sep 17 00:00:00 2001
From: Michal <michal@itaz.eu>
Date: Tue, 8 Sep 2026 11:07:51 +0100
Subject: [PATCH] labsim etcd harness: the real fix -- protect-kernel-defaults
 sysctls

The crash-loop was NOT etcd starvation (my earlier diagnosis was wrong). The
generated config sets `protect-kernel-defaults: true`, which makes the kubelet
refuse to start unless vm.overcommit_memory=1, kernel.panic=10 and
kernel.panic_on_oops=1 are set:

  Failed to start ContainerManager err="invalid kernel flag: vm/overcommit_memory
  expected 1 actual 0, kernel/panic expected 10 actual 0, ..."

k3s then exited 1 and restart-looped, which downstream looked exactly like etcd
re-initialising and the apiserver flapping -- so it read as a CPU/etcd problem
when it was a missing-sysctl problem. Production sets these via install.ks.ts +
sysctl.ts (applyCisHardening); the sim's sysctl.d was missing them. Added the
byte-for-byte CIS set.

Result: fresh build, all three servers k3s=active with 0 restarts, 3-node
embedded-etcd cluster formed and stable, apiserver responsive. NotReady is
expected (no CNI yet). The etcd timer tuning stays as cheap nested-virt
insurance but was not the fix; its comment is corrected.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
---
 labsim/labsim-k8s-etcd.sh | 38 +++++++++++++++++++++++++-------------
 1 file changed, 25 insertions(+), 13 deletions(-)

diff --git a/labsim/labsim-k8s-etcd.sh b/labsim/labsim-k8s-etcd.sh
index 839741d..f826599 100755
--- a/labsim/labsim-k8s-etcd.sh
+++ b/labsim/labsim-k8s-etcd.sh
@@ -146,12 +146,26 @@ $audit_ind
     content: |
       br_netfilter
       overlay
-  - path: /etc/sysctl.d/99-k8s.conf
+  # The CIS sysctls the generated config's `protect-kernel-defaults: true`
+  # REQUIRES -- byte-for-byte from labctl's sysctl.ts (applyCisHardening), plus
+  # v6 forwarding. Without vm.overcommit_memory=1 / kernel.panic=10 /
+  # kernel.panic_on_oops=1 the kubelet REFUSES to start ("invalid kernel flag"),
+  # k3s exits 1 and crash-loops -- which presents downstream as etcd
+  # re-initialising and the apiserver flapping, i.e. it looks like an etcd/CPU
+  # problem when it is not. In production these come from install.ks.ts + this
+  # operation; the sim must set them too.
+  - path: /etc/sysctl.d/90-k3s-cis.conf
     content: |
-      net.ipv4.ip_forward = 1
-      net.ipv6.conf.all.forwarding = 1
-      net.bridge.bridge-nf-call-iptables = 1
+      net.bridge.bridge-nf-call-iptables  = 1
       net.bridge.bridge-nf-call-ip6tables = 1
+      net.ipv4.ip_forward                 = 1
+      net.ipv6.conf.all.forwarding        = 1
+      vm.panic_on_oom                     = 0
+      vm.overcommit_memory                = 1
+      kernel.panic                        = 10
+      kernel.panic_on_oops                = 1
+      fs.inotify.max_user_instances       = 524288
+      fs.inotify.max_user_watches         = 524288
 
 runcmd:
   - [ modprobe, br_netfilter ]
@@ -171,15 +185,13 @@ runcmd:
     # exactly as production. The token is passed via env for the join; on node 1
     # it seeds the cluster token.
     #
-    # The etcd-arg tuning is LAB-ONLY and deliberately NOT in the production
-    # config.yaml/k3s-config.ts: three embedded-etcd servers on a busy labsim
-    # host starve etcd of CPU, and stock etcd timers (100ms heartbeat / 1s
-    # election) then trigger a constant leader-re-election storm that never
-    # settles ("storage is (re)initializing", k3s stuck activating). Relaxing
-    # the timers lets etcd tolerate the lab's scheduling latency. It changes
-    # NOTHING the dual-stack conversion exercises (config.yaml CIDR handling,
-    # ServiceCIDR pickup, quorum across a restart) -- it just keeps etcd alive
-    # under nested-virt jitter. See dualstack-evidence/etcd-harness-2026-09-07.md.
+    # The etcd-arg tuning is LAB-ONLY (not in production's config): relaxed
+    # heartbeat/election timers so etcd tolerates nested-virt scheduling jitter.
+    # NB: this was NOT what fixed the first run's failure -- that was the missing
+    # protect-kernel-defaults sysctls above, which crash-looped the kubelet and
+    # only LOOKED like etcd instability. The tuning is kept as cheap defensive
+    # insurance for a busy host; it changes nothing the conversion test
+    # exercises.
     curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="server --etcd-arg=heartbeat-interval=500 --etcd-arg=election-timeout=5000" K3S_TOKEN="$TOKEN" sh -
 EOF