v2 partition schema: capability model, fixed sizing, and a proven rolling migration
Some checks failed
CI/CD / typecheck (push) Failing after 9s
CI/CD / test (push) Failing after 10s
CI/CD / lint (push) Failing after 24s
CI/CD / build (push) Has been skipped
CI/CD / publish-rpm (push) Has been skipped
CI/CD / publish-deb (push) Has been skipped

The fleet's disk layout was decided by literal role-name comparisons at 17
sites across 8 files. That is why `labcontroller` got no rancher LV (it
matched neither "infra" nor "worker"), why labmaster carried an empty 728 GiB
Longhorn LV, and why a 62 GiB container image tipped that node into
DiskPressure and evicted bastion, labd, cockroach and openbao.

Three generators each hardcoded their own sizes and disagreed: the kickstart
said rancher=120G, ubuntu-autoinstall said 20G, asahi-firstboot said 20G under
a comment asserting the sizes matched install.ks.ts. They did not.

ROLE_REGISTRY already carried `k3s` and `parent` and nothing read them. This
makes capabilities authoritative, moves every size into one shared module, and
adds schema v2 (fixed 250G rancher / 300G longhorn, never --grow). Existing
machines stay v1 forever: schema is detected from the on-disk VG tag, so a v1
box stays v1 even when v2 is requested.

Proven on real VMs, not just unit tests: all four topologies installed and had
their layouts read back off the machines; markers in /home, /srv,
/var/lib/longhorn and /var/lib/rancher survived reinstall; and a 4-node cluster
formed, took real data, and had all three servers rolled one at a time through
evacuate / wipe / reinstall / rejoin with checksums verified before and after
each roll. Drill: 10/10, evidence in migration/drill-evidence/.

Production defects the drill found, none of which a unit test could reach:

- sshOpts() capped every operation at 30s while drainNode asked kubectl for
  --timeout=15m, so the transport killed the drain after evicting most pods and
  returned a transcript that read like success. `labctl app k3s evacuate` could
  never have drained a real node.
- generateAgentConfig put node-role.kubernetes.io/worker in the kubelet's
  node-label list, which NodeRestriction rejects, so no agent installed through
  K3sModule could start. systemd reported only "did not take the steps required
  by its unit configuration".
- toOpContext built K3sConfig by enumerating three keys and silently dropped
  the rest, so the k3s version pin did nothing and --allow-single-replica was a
  no-op that claimed to override a data-loss guard.
- The Longhorn disk annotation was mangled by shell quoting; with
  createDefaultDiskLabeledNodes the node then gets no disk at all, joins, goes
  Ready, and silently hosts no replica.
- evacuate drained before evicting, which blocks on Longhorn's PDBs and would
  destroy a single-replica volume's only source.
- etcd member ids were parsed from JSON, losing uint64 precision and emitting
  decimal where `member remove` needs hex -- at the moment a node is stopped
  and about to be wiped.
- get.k3s.io installs latest, two minor versions ahead of the fleet, so a
  reinstalled node would rejoin on a newer k3s than its peers and roll an
  unplanned two-version control-plane upgrade. K3sConfig.k3sVersion now pins it.
- `labctl app k3s install --role x` mapped anything that was not "worker" to
  "infra", so the runbook's own `--role k8s-combo` would have installed a
  different role with different disk expectations and no warning.

Also: /var/lib/longhorn/replicas must never be deleted, only its contents, or
every replica placed on the node dies on a non-recursive mkdir -- for ever,
while the node reports Ready, Schedulable and healthy.

etcd is now in the kickstart's package set for k3s server roles, because
`k3s etcdctl` is not a command and a member that needs to recover its own
membership has no internet.

542 unit tests / 39 files.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Michal
2026-10-09 22:33:59 +01:00
parent c729275961
commit 84836cb8f2
60 changed files with 7457 additions and 96 deletions

View File

@@ -0,0 +1,92 @@
import { describe, it, expect } from "vitest";
import { v2Layout, v2AllocatedMib, V2_BOOT, V2_RANCHER_MIB, V2_LONGHORN_MIB } from "../src/types/layout.js";
import { SUPPORTED_ROLES, roleInfo } from "../src/types/state.js";
/** A 1 TB disk is the smallest we will ever buy. 1e12 bytes in MiB. */
const ONE_TB_MIB = Math.floor(1_000_000_000_000 / (1024 * 1024)); // 953674
const VG_MIB_ON_1TB = ONE_TB_MIB - V2_BOOT.efiMib - V2_BOOT.bootMib;
describe("v2 layout fits the smallest disk we will buy", () => {
it.each(SUPPORTED_ROLES)("%s fits in a 1 TB disk with headroom left over", (role) => {
const used = v2AllocatedMib(role);
expect(used).toBeLessThan(VG_MIB_ON_1TB);
// The entire point of v2 is to STOP consuming the disk. Demand real slack.
const freeMib = VG_MIB_ON_1TB - used;
expect(freeMib).toBeGreaterThan(100 * 1024); // >100 GiB unallocated
});
it("the heaviest role still leaves >200 GiB free on 1 TB", () => {
const worst = Math.max(...SUPPORTED_ROLES.map((r) => v2AllocatedMib(r)));
expect(VG_MIB_ON_1TB - worst).toBeGreaterThan(200 * 1024);
});
});
describe("v2 gates volumes by capability, not role name", () => {
it.each(SUPPORTED_ROLES)("%s gets a rancher LV iff it runs k3s", (role) => {
const hasRancher = v2Layout(role).some((lv) => lv.name === "rancher");
expect(hasRancher).toBe(roleInfo(role).k3sMode !== null);
});
it("labcontroller gets a rancher LV — the v1 bug this schema fixes", () => {
const lvs = v2Layout("labcontroller");
const rancher = lvs.find((lv) => lv.name === "rancher");
expect(rancher).toBeDefined();
expect(rancher!.sizeMib).toBe(V2_RANCHER_MIB);
});
it("vanilla gets neither rancher nor longhorn", () => {
const names = v2Layout("vanilla").map((lv) => lv.name);
expect(names).not.toContain("rancher");
expect(names).not.toContain("longhorn");
});
it("a control-plane-only role would get no longhorn LV", () => {
for (const role of SUPPORTED_ROLES) {
if (roleInfo(role).controlPlaneOnly) {
expect(v2Layout(role).map((l) => l.name)).not.toContain("longhorn");
}
}
});
});
describe("v2 never over-allocates", () => {
it("no volume is grow/unbounded — every size is a concrete number", () => {
for (const role of SUPPORTED_ROLES) {
for (const lv of v2Layout(role)) {
expect(Number.isFinite(lv.sizeMib)).toBe(true);
expect(lv.sizeMib).toBeGreaterThan(0);
}
}
});
it("longhorn is fixed at 300 GiB, not grown", () => {
const lh = v2Layout("worker").find((lv) => lv.name === "longhorn");
expect(lh?.sizeMib).toBe(V2_LONGHORN_MIB);
expect(V2_LONGHORN_MIB).toBe(300 * 1024);
});
it("rancher is sized for an image store AND local-path PVCs", () => {
// A single ROCm vLLM image measured 62 GiB extracted on 2026-09-17.
expect(V2_RANCHER_MIB).toBeGreaterThan(200 * 1024);
});
});
describe("Asahi variant", () => {
it("omitRoot drops the root LV — the vendor installer already made one", () => {
const names = v2Layout("infra", { omitRoot: true }).map((lv) => lv.name);
expect(names).not.toContain("root");
expect(names).toContain("var");
});
it("omitRoot frees the root allocation and nothing else", () => {
const delta = v2AllocatedMib("infra") - v2AllocatedMib("infra", { omitRoot: true });
expect(delta).toBe(32768);
});
});
describe("layout is a pure value, safe to render three times", () => {
it("mutating a returned spec does not affect the next call", () => {
v2Layout("worker")[0]!.sizeMib = 999;
expect(v2Layout("worker")[0]!.sizeMib).not.toBe(999);
});
});