v2 partition schema: capability model, fixed sizing, and a proven rolling migration

The fleet's disk layout was decided by literal role-name comparisons at 17
sites across 8 files. That is why `labcontroller` got no rancher LV (it
matched neither "infra" nor "worker"), why labmaster carried an empty 728 GiB
Longhorn LV, and why a 62 GiB container image tipped that node into
DiskPressure and evicted bastion, labd, cockroach and openbao.

Three generators each hardcoded their own sizes and disagreed: the kickstart
said rancher=120G, ubuntu-autoinstall said 20G, asahi-firstboot said 20G under
a comment asserting the sizes matched install.ks.ts. They did not.

ROLE_REGISTRY already carried `k3s` and `parent` and nothing read them. This
makes capabilities authoritative, moves every size into one shared module, and
adds schema v2 (fixed 250G rancher / 300G longhorn, never --grow). Existing
machines stay v1 forever: schema is detected from the on-disk VG tag, so a v1
box stays v1 even when v2 is requested.

Proven on real VMs, not just unit tests: all four topologies installed and had
their layouts read back off the machines; markers in /home, /srv,
/var/lib/longhorn and /var/lib/rancher survived reinstall; and a 4-node cluster
formed, took real data, and had all three servers rolled one at a time through
evacuate / wipe / reinstall / rejoin with checksums verified before and after
each roll. Drill: 10/10, evidence in migration/drill-evidence/.

Production defects the drill found, none of which a unit test could reach:

- sshOpts() capped every operation at 30s while drainNode asked kubectl for
  --timeout=15m, so the transport killed the drain after evicting most pods and
  returned a transcript that read like success. `labctl app k3s evacuate` could
  never have drained a real node.
- generateAgentConfig put node-role.kubernetes.io/worker in the kubelet's
  node-label list, which NodeRestriction rejects, so no agent installed through
  K3sModule could start. systemd reported only "did not take the steps required
  by its unit configuration".
- toOpContext built K3sConfig by enumerating three keys and silently dropped
  the rest, so the k3s version pin did nothing and --allow-single-replica was a
  no-op that claimed to override a data-loss guard.
- The Longhorn disk annotation was mangled by shell quoting; with
  createDefaultDiskLabeledNodes the node then gets no disk at all, joins, goes
  Ready, and silently hosts no replica.
- evacuate drained before evicting, which blocks on Longhorn's PDBs and would
  destroy a single-replica volume's only source.
- etcd member ids were parsed from JSON, losing uint64 precision and emitting
  decimal where `member remove` needs hex -- at the moment a node is stopped
  and about to be wiped.
- get.k3s.io installs latest, two minor versions ahead of the fleet, so a
  reinstalled node would rejoin on a newer k3s than its peers and roll an
  unplanned two-version control-plane upgrade. K3sConfig.k3sVersion now pins it.
- `labctl app k3s install --role x` mapped anything that was not "worker" to
  "infra", so the runbook's own `--role k8s-combo` would have installed a
  different role with different disk expectations and no warning.

Also: /var/lib/longhorn/replicas must never be deleted, only its contents, or
every replica placed on the node dies on a non-recursive mkdir -- for ever,
while the node reports Ready, Schedulable and healthy.

etcd is now in the kickstart's package set for k3s server roles, because
`k3s etcdctl` is not a command and a member that needs to recover its own
membership has no internet.

542 unit tests / 39 files.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

bastion/src/shared/tests/capabilities.test.ts

@@ -0,0 +1,129 @@
+import { describe, it, expect } from "vitest";
+import {
+  SUPPORTED_ROLES,
+  ROLE_REGISTRY,
+  roleInfo,
+  runsK3s,
+  isK3sServer,
+  isManaged,
+  hostsLonghorn,
+} from "../src/types/state.js";
+import type { Role } from "../src/types/state.js";
+
+/**
+ * The capability lookups exist to replace role-name comparisons scattered across
+ * the codebase. Introducing them must change NO behaviour — so these tests pin
+ * each helper against the literal expression it replaces, copied verbatim from
+ * the call site it came from.
+ *
+ * If one of these fails, the refactor changed behaviour somewhere. That is the
+ * whole point of the file.
+ */
+/**
+ * Only the FOUR legacy roles have behaviour to preserve. The topology profiles
+ * (standalone-cluster, k8s-*) are new, v2-only, and have no historical
+ * expression to be compatible with -- pinning them against `role === "worker"`
+ * would assert nonsense.
+ */
+const LEGACY_ROLES = ["vanilla", "worker", "infra", "labcontroller"] as const;
+
+describe("capability helpers reproduce the expressions they replace", () => {
+  // Verbatim from the call sites, as of the commit that introduced this file.
+  const legacy = {
+    // k3s-install.ts:7, k3s-config.ts:7, render-config.ts:25,
+    // k3s-module.ts:65, health/k3s-service.ts:5, post-provision.ts:153
+    isServer: (r: Role) => r === "infra" || r === "labcontroller",
+    // ROLE_REGISTRY.k3s — the field that already existed and nobody read
+    runsK3s: (r: Role) => ROLE_REGISTRY.find((x) => x.name === r)!.k3s,
+    // install.ks.ts:41, ubuntu-autoinstall.ts:34
+    hasLonghorn: (r: Role) => r === "worker",
+    // install.ks.ts:48
+    isVanilla: (r: Role) => r === "vanilla",
+  };
+
+  it.each(LEGACY_ROLES)("isK3sServer matches the legacy expression for %s", (role) => {
+    expect(isK3sServer(role)).toBe(legacy.isServer(role));
+  });
+
+  it.each(LEGACY_ROLES)("runsK3s matches ROLE_REGISTRY.k3s for %s", (role) => {
+    expect(runsK3s(role)).toBe(legacy.runsK3s(role));
+  });
+
+  it.each(LEGACY_ROLES)("hostsLonghorn matches the legacy expression for %s", (role) => {
+    expect(hostsLonghorn(role)).toBe(legacy.hasLonghorn(role));
+  });
+
+  it.each(LEGACY_ROLES)("k3sMode === null matches isVanilla for %s", (role) => {
+    expect(roleInfo(role).k3sMode === null).toBe(legacy.isVanilla(role));
+  });
+
+  it("every supported role is managed today (nothing is onboarded yet)", () => {
+    for (const role of SUPPORTED_ROLES) expect(isManaged(role)).toBe(true);
+  });
+});
+
+/**
+ * The topology profiles. These are asserted directly rather than against a
+ * legacy expression, because they describe cluster SHAPE -- which is the thing
+ * role names were supposed to convey and did not.
+ */
+describe("topology profiles", () => {
+  it("standalone-cluster is a server that also holds storage", () => {
+    expect(isK3sServer("standalone-cluster")).toBe(true);
+    // Longhorn at replica 1: a cluster of one cannot do 3, but keeping the
+    // StorageClass present keeps manifests portable across clusters.
+    expect(hostsLonghorn("standalone-cluster")).toBe(true);
+    expect(roleInfo("standalone-cluster").controlPlaneOnly).toBe(false);
+  });
+
+  it("k8s-controller is the only role that runs k3s and holds no storage", () => {
+    expect(isK3sServer("k8s-controller")).toBe(true);
+    expect(hostsLonghorn("k8s-controller")).toBe(false);
+    expect(roleInfo("k8s-controller").controlPlaneOnly).toBe(true);
+  });
+
+  it("k8s-worker is an agent with storage", () => {
+    expect(isK3sServer("k8s-worker")).toBe(false);
+    expect(runsK3s("k8s-worker")).toBe(true);
+    expect(hostsLonghorn("k8s-worker")).toBe(true);
+  });
+
+  it("k8s-combo is both control plane and storage — unlike k8s-controller", () => {
+    expect(isK3sServer("k8s-combo")).toBe(true);
+    expect(hostsLonghorn("k8s-combo")).toBe(true);
+    expect(roleInfo("k8s-combo").controlPlaneOnly).toBe(false);
+  });
+
+  it("k8s-combo and k8s-controller differ ONLY in storage duty", () => {
+    const combo = roleInfo("k8s-combo");
+    const ctrl = roleInfo("k8s-controller");
+    expect(combo.k3sMode).toBe(ctrl.k3sMode);
+    expect(combo.hostsLonghorn).not.toBe(ctrl.hostsLonghorn);
+  });
+});
+
+describe("registry integrity", () => {
+  it("every SUPPORTED_ROLE has a registry entry", () => {
+    for (const role of SUPPORTED_ROLES) expect(() => roleInfo(role)).not.toThrow();
+  });
+
+  it("registry has no entries outside SUPPORTED_ROLES", () => {
+    for (const info of ROLE_REGISTRY) expect(SUPPORTED_ROLES).toContain(info.name);
+  });
+
+  it("k3s boolean and k3sMode never disagree", () => {
+    for (const info of ROLE_REGISTRY) {
+      expect(info.k3s).toBe(info.k3sMode !== null);
+    }
+  });
+
+  it("an unknown role throws rather than silently defaulting", () => {
+    expect(() => roleInfo("does-not-exist" as Role)).toThrow(/unknown role/);
+  });
+
+  it("a control-plane-only role never hosts Longhorn", () => {
+    for (const info of ROLE_REGISTRY) {
+      if (info.controlPlaneOnly) expect(info.hostsLonghorn).toBe(false);
+    }
+  });
+});

bastion/src/shared/tests/layout.test.ts

@@ -0,0 +1,92 @@
+import { describe, it, expect } from "vitest";
+import { v2Layout, v2AllocatedMib, V2_BOOT, V2_RANCHER_MIB, V2_LONGHORN_MIB } from "../src/types/layout.js";
+import { SUPPORTED_ROLES, roleInfo } from "../src/types/state.js";
+
+/** A 1 TB disk is the smallest we will ever buy. 1e12 bytes in MiB. */
+const ONE_TB_MIB = Math.floor(1_000_000_000_000 / (1024 * 1024)); // 953674
+const VG_MIB_ON_1TB = ONE_TB_MIB - V2_BOOT.efiMib - V2_BOOT.bootMib;
+
+describe("v2 layout fits the smallest disk we will buy", () => {
+  it.each(SUPPORTED_ROLES)("%s fits in a 1 TB disk with headroom left over", (role) => {
+    const used = v2AllocatedMib(role);
+    expect(used).toBeLessThan(VG_MIB_ON_1TB);
+    // The entire point of v2 is to STOP consuming the disk. Demand real slack.
+    const freeMib = VG_MIB_ON_1TB - used;
+    expect(freeMib).toBeGreaterThan(100 * 1024); // >100 GiB unallocated
+  });
+
+  it("the heaviest role still leaves >200 GiB free on 1 TB", () => {
+    const worst = Math.max(...SUPPORTED_ROLES.map((r) => v2AllocatedMib(r)));
+    expect(VG_MIB_ON_1TB - worst).toBeGreaterThan(200 * 1024);
+  });
+});
+
+describe("v2 gates volumes by capability, not role name", () => {
+  it.each(SUPPORTED_ROLES)("%s gets a rancher LV iff it runs k3s", (role) => {
+    const hasRancher = v2Layout(role).some((lv) => lv.name === "rancher");
+    expect(hasRancher).toBe(roleInfo(role).k3sMode !== null);
+  });
+
+  it("labcontroller gets a rancher LV — the v1 bug this schema fixes", () => {
+    const lvs = v2Layout("labcontroller");
+    const rancher = lvs.find((lv) => lv.name === "rancher");
+    expect(rancher).toBeDefined();
+    expect(rancher!.sizeMib).toBe(V2_RANCHER_MIB);
+  });
+
+  it("vanilla gets neither rancher nor longhorn", () => {
+    const names = v2Layout("vanilla").map((lv) => lv.name);
+    expect(names).not.toContain("rancher");
+    expect(names).not.toContain("longhorn");
+  });
+
+  it("a control-plane-only role would get no longhorn LV", () => {
+    for (const role of SUPPORTED_ROLES) {
+      if (roleInfo(role).controlPlaneOnly) {
+        expect(v2Layout(role).map((l) => l.name)).not.toContain("longhorn");
+      }
+    }
+  });
+});
+
+describe("v2 never over-allocates", () => {
+  it("no volume is grow/unbounded — every size is a concrete number", () => {
+    for (const role of SUPPORTED_ROLES) {
+      for (const lv of v2Layout(role)) {
+        expect(Number.isFinite(lv.sizeMib)).toBe(true);
+        expect(lv.sizeMib).toBeGreaterThan(0);
+      }
+    }
+  });
+
+  it("longhorn is fixed at 300 GiB, not grown", () => {
+    const lh = v2Layout("worker").find((lv) => lv.name === "longhorn");
+    expect(lh?.sizeMib).toBe(V2_LONGHORN_MIB);
+    expect(V2_LONGHORN_MIB).toBe(300 * 1024);
+  });
+
+  it("rancher is sized for an image store AND local-path PVCs", () => {
+    // A single ROCm vLLM image measured 62 GiB extracted on 2026-09-17.
+    expect(V2_RANCHER_MIB).toBeGreaterThan(200 * 1024);
+  });
+});
+
+describe("Asahi variant", () => {
+  it("omitRoot drops the root LV — the vendor installer already made one", () => {
+    const names = v2Layout("infra", { omitRoot: true }).map((lv) => lv.name);
+    expect(names).not.toContain("root");
+    expect(names).toContain("var");
+  });
+
+  it("omitRoot frees the root allocation and nothing else", () => {
+    const delta = v2AllocatedMib("infra") - v2AllocatedMib("infra", { omitRoot: true });
+    expect(delta).toBe(32768);
+  });
+});
+
+describe("layout is a pure value, safe to render three times", () => {
+  it("mutating a returned spec does not affect the next call", () => {
+    v2Layout("worker")[0]!.sizeMib = 999;
+    expect(v2Layout("worker")[0]!.sizeMib).not.toBe(999);
+  });
+});