v2 partition schema: capability model, fixed sizing, and a proven rolling migration

The fleet's disk layout was decided by literal role-name comparisons at 17
sites across 8 files. That is why `labcontroller` got no rancher LV (it
matched neither "infra" nor "worker"), why labmaster carried an empty 728 GiB
Longhorn LV, and why a 62 GiB container image tipped that node into
DiskPressure and evicted bastion, labd, cockroach and openbao.

Three generators each hardcoded their own sizes and disagreed: the kickstart
said rancher=120G, ubuntu-autoinstall said 20G, asahi-firstboot said 20G under
a comment asserting the sizes matched install.ks.ts. They did not.

ROLE_REGISTRY already carried `k3s` and `parent` and nothing read them. This
makes capabilities authoritative, moves every size into one shared module, and
adds schema v2 (fixed 250G rancher / 300G longhorn, never --grow). Existing
machines stay v1 forever: schema is detected from the on-disk VG tag, so a v1
box stays v1 even when v2 is requested.

Proven on real VMs, not just unit tests: all four topologies installed and had
their layouts read back off the machines; markers in /home, /srv,
/var/lib/longhorn and /var/lib/rancher survived reinstall; and a 4-node cluster
formed, took real data, and had all three servers rolled one at a time through
evacuate / wipe / reinstall / rejoin with checksums verified before and after
each roll. Drill: 10/10, evidence in migration/drill-evidence/.

Production defects the drill found, none of which a unit test could reach:

- sshOpts() capped every operation at 30s while drainNode asked kubectl for
  --timeout=15m, so the transport killed the drain after evicting most pods and
  returned a transcript that read like success. `labctl app k3s evacuate` could
  never have drained a real node.
- generateAgentConfig put node-role.kubernetes.io/worker in the kubelet's
  node-label list, which NodeRestriction rejects, so no agent installed through
  K3sModule could start. systemd reported only "did not take the steps required
  by its unit configuration".
- toOpContext built K3sConfig by enumerating three keys and silently dropped
  the rest, so the k3s version pin did nothing and --allow-single-replica was a
  no-op that claimed to override a data-loss guard.
- The Longhorn disk annotation was mangled by shell quoting; with
  createDefaultDiskLabeledNodes the node then gets no disk at all, joins, goes
  Ready, and silently hosts no replica.
- evacuate drained before evicting, which blocks on Longhorn's PDBs and would
  destroy a single-replica volume's only source.
- etcd member ids were parsed from JSON, losing uint64 precision and emitting
  decimal where `member remove` needs hex -- at the moment a node is stopped
  and about to be wiped.
- get.k3s.io installs latest, two minor versions ahead of the fleet, so a
  reinstalled node would rejoin on a newer k3s than its peers and roll an
  unplanned two-version control-plane upgrade. K3sConfig.k3sVersion now pins it.
- `labctl app k3s install --role x` mapped anything that was not "worker" to
  "infra", so the runbook's own `--role k8s-combo` would have installed a
  different role with different disk expectations and no warning.

Also: /var/lib/longhorn/replicas must never be deleted, only its contents, or
every replica placed on the node dies on a non-recursive mkdir -- for ever,
while the node reports Ready, Schedulable and healthy.

etcd is now in the kickstart's package set for k3s server roles, because
`k3s etcdctl` is not a command and a member that needs to recover its own
membership has no internet.

542 unit tests / 39 files.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

bastion/tests/integration/reinstall.test.ts

@@ -0,0 +1,138 @@
+// Integration tests: reinstalling a machine must not destroy its data, and
+// must not silently change its disk layout.
+//
+// This is the risk the whole v2 change carries. Every machine in the fleet was
+// installed with v1, and `labctl provision reprovision` has to keep reproducing
+// v1 on those boxes forever. If a reinstall quietly switched them to v2 it
+// would recreate the volume group and take every Longhorn replica with it.
+//
+// Ported from test-reprovision.sh, which had the right idea (marker files in
+// each preserved volume) but only ever ran by hand and was never in CI.
+//
+// Run: sudo -E npx vitest run -c tests/integration/vitest.config.ts -t "reinstall"
+
+import { describe, it, expect, beforeAll, afterAll } from "vitest";
+import { existsSync, readFileSync } from "node:fs";
+import { homedir } from "node:os";
+import { join } from "node:path";
+import { v2Layout } from "../../src/shared/src/types/layout.js";
+import {
+  startTestBastion, provisionVm, reprovisionVm, readLvs, readSchemaTag,
+  writeMarkers, readMarkers, destroyVms, type Harness,
+} from "./helpers/provision-harness.js";
+import { destroyPxeNetwork } from "./helpers/pxe-network.js";
+
+const HTTP_PORT = 8095;
+const SSH_USER = "lab";
+const DISK_GB = 1000;
+
+/** Mount points a reinstall is supposed to preserve on a v1 worker. */
+const PRESERVED = ["/home", "/srv", "/var/lib/longhorn", "/var/lib/rancher"];
+
+function findSshKey(): { pubKey: string; keyPath: string } {
+  const homes = [homedir()];
+  const sudoUser = process.env["SUDO_USER"];
+  if (sudoUser) homes.push(join("/home", sudoUser));
+  for (const home of homes) {
+    for (const name of ["id_ed25519", "id_ecdsa", "id_rsa"]) {
+      const keyPath = join(home, ".ssh", name);
+      if (existsSync(keyPath) && existsSync(`${keyPath}.pub`)) {
+        return { pubKey: readFileSync(`${keyPath}.pub`, "utf-8").trim(), keyPath };
+      }
+    }
+  }
+  throw new Error("no SSH keypair found");
+}
+
+describe("reinstall preserves data and layout", () => {
+  let h: Harness;
+  const created: string[] = [];
+
+  beforeAll(async () => {
+    const { pubKey, keyPath } = findSshKey();
+    h = await startTestBastion({
+      httpPort: HTTP_PORT, sshUser: SSH_USER, sshPubKey: pubKey,
+      sshKeyPath: keyPath, label: "reinstall",
+    });
+  }, 20 * 60_000);
+
+  afterAll(async () => {
+    destroyVms(...created);
+    try { await h?.close(); } catch { /* closed */ }
+    try { destroyPxeNetwork(); } catch { /* gone */ }
+  }, 5 * 60_000);
+
+  it("v1 worker: data survives and the layout stays v1", async () => {
+    const name = "re-v1-worker";
+    created.push(name);
+
+    // Install at v1 -- i.e. exactly what every machine in the fleet is today.
+    let vm = await provisionVm(h, { name, role: "worker", diskGb: DISK_GB });
+
+    const before = readLvs(h, vm);
+    expect(before["root"], "v1 root is 33G, not v2's 32G").toBeGreaterThan(33000);
+    expect(before["rancher"], "v1 worker gets a 120G rancher LV").toBeGreaterThan(120000 - 8);
+    expect(readSchemaTag(h, vm), "a v1 disk must NOT carry the v2 tag").not.toContain("lab-schema-v2");
+
+    const token = writeMarkers(h, vm, PRESERVED);
+
+    // Reinstall. No schemaVersion passed -- the disk decides.
+    vm = await reprovisionVm(h, vm, "worker");
+
+    const markers = readMarkers(h, vm, PRESERVED);
+    for (const mount of PRESERVED) {
+      expect(markers[mount], `${mount} must survive a reinstall`).toBe(token);
+    }
+
+    // And the layout must be untouched.
+    const after = readLvs(h, vm);
+    expect(Object.keys(after).sort()).toEqual(Object.keys(before).sort());
+    expect(after["root"]).toBe(before["root"]);
+    expect(after["rancher"]).toBe(before["rancher"]);
+    expect(readSchemaTag(h, vm)).not.toContain("lab-schema-v2");
+  }, 120 * 60_000);
+
+  it("v2 worker: data survives and the layout stays v2", async () => {
+    const name = "re-v2-worker";
+    created.push(name);
+
+    let vm = await provisionVm(h, { name, role: "k8s-worker", schemaVersion: 2, diskGb: DISK_GB });
+    const before = readLvs(h, vm);
+    expect(readSchemaTag(h, vm)).toContain("lab-schema-v2");
+
+    const token = writeMarkers(h, vm, PRESERVED);
+    vm = await reprovisionVm(h, vm, "k8s-worker");
+
+    const markers = readMarkers(h, vm, PRESERVED);
+    for (const mount of PRESERVED) {
+      expect(markers[mount], `${mount} must survive a reinstall`).toBe(token);
+    }
+
+    const after = readLvs(h, vm);
+    const expected = Object.fromEntries(v2Layout("k8s-worker").map((lv) => [lv.name, lv.sizeMib]));
+    expect(Object.keys(after).sort()).toEqual(Object.keys(expected).sort());
+    expect(after["root"]).toBe(before["root"]);
+    // The tag must survive too, or the NEXT reinstall would fall back to v1.
+    expect(readSchemaTag(h, vm)).toContain("lab-schema-v2");
+  }, 120 * 60_000);
+
+  it("a v1 machine is NOT upgraded to v2 just because v2 was requested", async () => {
+    const name = "re-v1-sticky";
+    created.push(name);
+
+    let vm = await provisionVm(h, { name, role: "worker", diskGb: DISK_GB });
+    const before = readLvs(h, vm);
+    const token = writeMarkers(h, vm, ["/home", "/srv"]);
+
+    // Ask for v2 on a disk that is v1. %pre must ignore it: silently
+    // re-laying-out a live machine is how data gets destroyed.
+    vm = await reprovisionVm(h, vm, "worker", 2);
+
+    const after = readLvs(h, vm);
+    expect(after["root"], "root must stay at the v1 size").toBe(before["root"]);
+    expect(readSchemaTag(h, vm), "still not tagged v2").not.toContain("lab-schema-v2");
+    const markers = readMarkers(h, vm, ["/home", "/srv"]);
+    expect(markers["/home"]).toBe(token);
+    expect(markers["/srv"]).toBe(token);
+  }, 120 * 60_000);
+});