The fleet's disk layout was decided by literal role-name comparisons at 17
sites across 8 files. That is why `labcontroller` got no rancher LV (it
matched neither "infra" nor "worker"), why labmaster carried an empty 728 GiB
Longhorn LV, and why a 62 GiB container image tipped that node into
DiskPressure and evicted bastion, labd, cockroach and openbao.
Three generators each hardcoded their own sizes and disagreed: the kickstart
said rancher=120G, ubuntu-autoinstall said 20G, asahi-firstboot said 20G under
a comment asserting the sizes matched install.ks.ts. They did not.
ROLE_REGISTRY already carried `k3s` and `parent` and nothing read them. This
makes capabilities authoritative, moves every size into one shared module, and
adds schema v2 (fixed 250G rancher / 300G longhorn, never --grow). Existing
machines stay v1 forever: schema is detected from the on-disk VG tag, so a v1
box stays v1 even when v2 is requested.
Proven on real VMs, not just unit tests: all four topologies installed and had
their layouts read back off the machines; markers in /home, /srv,
/var/lib/longhorn and /var/lib/rancher survived reinstall; and a 4-node cluster
formed, took real data, and had all three servers rolled one at a time through
evacuate / wipe / reinstall / rejoin with checksums verified before and after
each roll. Drill: 10/10, evidence in migration/drill-evidence/.
Production defects the drill found, none of which a unit test could reach:
- sshOpts() capped every operation at 30s while drainNode asked kubectl for
--timeout=15m, so the transport killed the drain after evicting most pods and
returned a transcript that read like success. `labctl app k3s evacuate` could
never have drained a real node.
- generateAgentConfig put node-role.kubernetes.io/worker in the kubelet's
node-label list, which NodeRestriction rejects, so no agent installed through
K3sModule could start. systemd reported only "did not take the steps required
by its unit configuration".
- toOpContext built K3sConfig by enumerating three keys and silently dropped
the rest, so the k3s version pin did nothing and --allow-single-replica was a
no-op that claimed to override a data-loss guard.
- The Longhorn disk annotation was mangled by shell quoting; with
createDefaultDiskLabeledNodes the node then gets no disk at all, joins, goes
Ready, and silently hosts no replica.
- evacuate drained before evicting, which blocks on Longhorn's PDBs and would
destroy a single-replica volume's only source.
- etcd member ids were parsed from JSON, losing uint64 precision and emitting
decimal where `member remove` needs hex -- at the moment a node is stopped
and about to be wiped.
- get.k3s.io installs latest, two minor versions ahead of the fleet, so a
reinstalled node would rejoin on a newer k3s than its peers and roll an
unplanned two-version control-plane upgrade. K3sConfig.k3sVersion now pins it.
- `labctl app k3s install --role x` mapped anything that was not "worker" to
"infra", so the runbook's own `--role k8s-combo` would have installed a
different role with different disk expectations and no warning.
Also: /var/lib/longhorn/replicas must never be deleted, only its contents, or
every replica placed on the node dies on a non-recursive mkdir -- for ever,
while the node reports Ready, Schedulable and healthy.
etcd is now in the kickstart's package set for k3s server roles, because
`k3s etcdctl` is not a command and a member that needs to recover its own
membership has no internet.
542 unit tests / 39 files.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
PPPOE-HA.md cited migration/drill-evidence/wan-drill-2026-09-06-ipv6.log as the
record behind the takeover/failback/HE-call figures, and the repo reported clean
-- because the root .gitignore has `*.log`, so the file was never added. The
citation was dangling and nobody would have found out until they went looking for
the numbers.
Renamed to .txt, which is also what this repo already does for evidence:
labsim/wan-failover-evidence/ and labsim/vlan-leak-evidence/ store state.txt.
The ignore rule is fine; the filename was wrong.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
