The crash-loop was NOT etcd starvation (my earlier diagnosis was wrong). The
generated config sets `protect-kernel-defaults: true`, which makes the kubelet
refuse to start unless vm.overcommit_memory=1, kernel.panic=10 and
kernel.panic_on_oops=1 are set:
Failed to start ContainerManager err="invalid kernel flag: vm/overcommit_memory
expected 1 actual 0, kernel/panic expected 10 actual 0, ..."
k3s then exited 1 and restart-looped, which downstream looked exactly like etcd
re-initialising and the apiserver flapping -- so it read as a CPU/etcd problem
when it was a missing-sysctl problem. Production sets these via install.ks.ts +
sysctl.ts (applyCisHardening); the sim's sysctl.d was missing them. Added the
byte-for-byte CIS set.
Result: fresh build, all three servers k3s=active with 0 restarts, 3-node
embedded-etcd cluster formed and stable, apiserver responsive. NotReady is
expected (no CNI yet). The etcd timer tuning stays as cheap nested-virt
insurance but was not the fix; its comment is corrected.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
First run of the 3-server harness: the config path works (VMs boot, config.yaml
rendered by the production generator, k3s installs, etcd forms), but three full
control-plane servers starved etcd of CPU on a labsim host running 18 VMs (load
8+). etcd lost leader continuously -> "storage is (re)initializing" -> k3s stuck
activating -> datastore reset. Not a harness bug; host capacity + etcd's latency
intolerance.
Fixes for a clean next run:
- LAB-ONLY etcd tuning (heartbeat-interval=500, election-timeout=5000) in the
install exec, clearly marked as not-production and not affecting what the
conversion test exercises -- the standard remedy for etcd under nested-virt.
- Shut the other labsim k8s/dualstack VMs before `up` (done at teardown).
Also fixed a real sim drift found en route: the sim MASTER router lacked the NAT
masquerade rules the backup had, so LAN had no internet whenever .253 was master.
Added masquerade for 172.31.0.0/16 out bond0.53 + pppoe0.
Full write-up: labsim/dualstack-evidence/etcd-harness-2026-09-07.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
The rehearsal vehicle for cluster dual-stack. Unlike k8s-up.sh (1 server + 2
agents, inline INSTALL_K3S_EXEC), this builds 3 control-plane servers with
embedded etcd, each configured through /etc/rancher/k3s/config.yaml rendered by
labctl's own generate*Config via bin/render-config.js -- the exact production
code path. Node 1 cluster-inits; 2/3 join as etcd members (role=infra + server
URL, same shape as production worker1/worker2).
Carries the audit-policy.yaml the generated config references (byte-identical to
audit-policy.ts) -- without it the apiserver silently crash-loops. Validates the
embedded config.yaml as YAML before building each seed ISO.
Next: rehearse the dual-stack conversion on it (rolling config.yaml change,
quorum, mixed control plane, ServiceCIDR pickup).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
