v2 partition schema: capability model, fixed sizing, and a proven rolling migration

The fleet's disk layout was decided by literal role-name comparisons at 17
sites across 8 files. That is why `labcontroller` got no rancher LV (it
matched neither "infra" nor "worker"), why labmaster carried an empty 728 GiB
Longhorn LV, and why a 62 GiB container image tipped that node into
DiskPressure and evicted bastion, labd, cockroach and openbao.

Three generators each hardcoded their own sizes and disagreed: the kickstart
said rancher=120G, ubuntu-autoinstall said 20G, asahi-firstboot said 20G under
a comment asserting the sizes matched install.ks.ts. They did not.

ROLE_REGISTRY already carried `k3s` and `parent` and nothing read them. This
makes capabilities authoritative, moves every size into one shared module, and
adds schema v2 (fixed 250G rancher / 300G longhorn, never --grow). Existing
machines stay v1 forever: schema is detected from the on-disk VG tag, so a v1
box stays v1 even when v2 is requested.

Proven on real VMs, not just unit tests: all four topologies installed and had
their layouts read back off the machines; markers in /home, /srv,
/var/lib/longhorn and /var/lib/rancher survived reinstall; and a 4-node cluster
formed, took real data, and had all three servers rolled one at a time through
evacuate / wipe / reinstall / rejoin with checksums verified before and after
each roll. Drill: 10/10, evidence in migration/drill-evidence/.

Production defects the drill found, none of which a unit test could reach:

- sshOpts() capped every operation at 30s while drainNode asked kubectl for
  --timeout=15m, so the transport killed the drain after evicting most pods and
  returned a transcript that read like success. `labctl app k3s evacuate` could
  never have drained a real node.
- generateAgentConfig put node-role.kubernetes.io/worker in the kubelet's
  node-label list, which NodeRestriction rejects, so no agent installed through
  K3sModule could start. systemd reported only "did not take the steps required
  by its unit configuration".
- toOpContext built K3sConfig by enumerating three keys and silently dropped
  the rest, so the k3s version pin did nothing and --allow-single-replica was a
  no-op that claimed to override a data-loss guard.
- The Longhorn disk annotation was mangled by shell quoting; with
  createDefaultDiskLabeledNodes the node then gets no disk at all, joins, goes
  Ready, and silently hosts no replica.
- evacuate drained before evicting, which blocks on Longhorn's PDBs and would
  destroy a single-replica volume's only source.
- etcd member ids were parsed from JSON, losing uint64 precision and emitting
  decimal where `member remove` needs hex -- at the moment a node is stopped
  and about to be wiped.
- get.k3s.io installs latest, two minor versions ahead of the fleet, so a
  reinstalled node would rejoin on a newer k3s than its peers and roll an
  unplanned two-version control-plane upgrade. K3sConfig.k3sVersion now pins it.
- `labctl app k3s install --role x` mapped anything that was not "worker" to
  "infra", so the runbook's own `--role k8s-combo` would have installed a
  different role with different disk expectations and no warning.

Also: /var/lib/longhorn/replicas must never be deleted, only its contents, or
every replica placed on the node dies on a non-recursive mkdir -- for ever,
while the node reports Ready, Schedulable and healthy.

etcd is now in the kickstart's package set for k3s server roles, because
`k3s etcdctl` is not a command and a member that needs to recover its own
membership has no internet.

542 unit tests / 39 files.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

migration/RECOVERY-CARD-schema-v2.md

@@ -0,0 +1,176 @@
+# Recovery card — v2 layout migration, one node at a time
+
+Print or keep open. **A node is being wiped. If the cluster goes down there is
+no Claude and possibly no internet.** Everything you need is on this page.
+
+---
+
+## The one thing that matters
+
+**`k3s etcdctl` is NOT a command.** It does not exist (`No help topic for
+'etcdctl'`). You need the real binary plus k3s's certs. Paste this once on
+whichever server you are using:
+
+```
+e() {
+  local b=/var/lib/rancher/k3s/server/tls/etcd
+  ETCDCTL_API=3 etcdctl --cacert=$b/server-ca.crt \
+    --cert=$b/server-client.crt --key=$b/server-client.key \
+    --endpoints=https://127.0.0.1:2379 "$@"
+}
+```
+
+(A function, not an alias — aliases do not expand inside scripts. Verified
+working on worker0.)
+
+If `etcdctl: command not found`, install it: `dnf install -y etcd` — which needs
+working DNS and a reachable mirror, so do not count on it here. Nodes
+reinstalled with the current kickstart already have it (`etcd` is in the package
+set for k3s server roles precisely so this step is never needed).
+
+Then:
+
+```
+e member list -w table
+```
+
+There are **3 etcd members. Quorum is 2. You have no margin.**
+
+Two naming traps, both of which make a correct-looking command match nothing:
+
+- **Kubernetes node names are FQDNs.** `worker0-k8s0.ad.itaz.eu`, never
+  `worker0`. The same name is used by `kubectl` and by `nodes.longhorn.io`.
+  Get it from `kubectl get nodes`; do not type it.
+- **etcd member names add a random suffix** on top of that:
+  `worker0-k8s0.ad.itaz.eu-e26aacc5`. Match on the prefix.
+
+One node down is survivable. Two is a dead API server and a dead cluster. So:
+**one node at a time**, and never start the next while anything is degraded.
+
+Reach the servers directly:
+
+```
+ssh root@192.168.8.23     # worker0   (Fedora x86)
+ssh root@192.168.8.25     # worker2   (Fedora aarch64)
+ssh root@192.168.8.13     # worker1   (Asahi aarch64 — migrate LAST)
+```
+
+---
+
+## Before you touch anything
+
+```
+kubectl get nodes                                        # all Ready
+e member list                                            # exactly 3, all started
+e endpoint health                                        # healthy
+kubectl -n longhorn-system get volumes.longhorn.io | grep -v healthy   # nothing
+```
+
+All four clean, or do not start.
+
+---
+
+## Order
+
+**worker0 → worker2 → worker1.** worker1 is last: vendor-owned partition table,
+4.6 G root, undersized rancher LV, and a hand-made bind mount a reinstall
+destroys.
+
+---
+
+## The loop
+
+```
+# 1. Ask first. It refuses in SECONDS if it cannot finish.
+labctl app k3s evacuate <node>
+```
+
+**If it refuses, it is right.** It means either no node can legally take a
+replica, or the disk has no room. Do not force it. Add the N+1 node first.
+
+```
+# 2. Node is empty. Prove the data is fine BEFORE destroying anything.
+kubectl -n longhorn-system get replicas.longhorn.io -o wide | grep <node>   # none
+#    restart something using a Longhorn PVC; check it reads back.
+
+# 3. Irreversible from here.
+ssh <node> systemctl stop k3s
+e member remove <id>             # member names are "<node>-<8 hex>"
+kubectl delete node <node>
+e member list                    # 2, both healthy  <-- CHECK THIS
+kubectl get --raw /readyz        # 'ok' from each survivor
+
+# 4. Wipe.
+labctl provision reprovision <node>
+
+# 5. Rejoin AGAINST A SURVIVOR, pinned to the cluster's version.
+labctl app k3s install <node> --role k8s-combo \
+  --k3s-server https://<survivor-ip>:6443 --k3s-token <token> \
+  --k3s-version <what the others run>     # kubectl get nodes -o wide
+e member list                    # EXACTLY 3  <-- CHECK THIS
+
+# 6. Back in service, then wait for the rebuild.
+labctl app k3s admit <node>
+kubectl -n longhorn-system get volumes.longhorn.io    # all healthy
+```
+
+---
+
+## When something is wrong
+
+| Symptom | Do this |
+|---|---|
+| Evacuate refused | Believe it. Read the named volume. Add capacity or a node. |
+| Evacuate timed out | It already cancelled itself and restored scheduling. If it says it could not: `labctl app k3s admit <node>` |
+| Aborted midway, node cordoned | `labctl app k3s admit <node>` |
+| Rejoined node hosts no replicas | It has no Longhorn disk. `labctl app k3s admit <node>` |
+| Replicas land but stay `stopped` (`desireState=running`, no `failedAt`) | `/var/lib/longhorn/replicas` is missing while `longhorn-disk.cfg` exists, so Longhorn thinks the disk is initialised and the replica process dies on `mkdir`. `mkdir -p /var/lib/longhorn/replicas`. **Never delete that directory** — delete its contents. |
+| Rejoined node is a newer k3s than its peers | You forgot `--k3s-version`. More than one minor of skew between control-plane members is unsupported; reinstall it pinned. |
+| `member list` shows 4 | `e member remove <dead-id>` — do this now |
+| Node wiped, will not rejoin | Restore from the Longhorn backup target. |
+| API server gone (2 members down) | You lost quorum. See below. |
+
+### Lost quorum
+
+Do not panic-restart things. From the one surviving server:
+
+```
+systemctl stop k3s
+k3s server --cluster-reset
+systemctl start k3s
+```
+
+This rebuilds a single-member cluster from that node's data. The other two
+rejoin afterwards as fresh members; their
+`/var/lib/rancher/k3s/server/db` must be removed before they will.
+
+---
+
+## worker1 only (Asahi)
+
+After its reinstall, put this back in `/etc/fstab` or the node runs out of root
+space the first time it pulls a large image:
+
+```
+/var/lib/longhorn/k3s-containerd /var/lib/rancher/k3s/agent/containerd none bind,x-systemd.requires-mounts-for=/var/lib/longhorn,x-systemd.requires-mounts-for=/var/lib/rancher 0 0
+```
+
+```
+mkdir -p /var/lib/longhorn/k3s-containerd
+systemctl daemon-reload && mount -a
+```
+
+---
+
+## After all three
+
+Re-apply by hand — `pulumi up` does NOT restore these:
+
+- aitopatom Longhorn `allowScheduling: false`
+- Prometheus 2-replica pin
+- multus 512Mi memory limit
+- every `local-path` PVC (model caches re-download; nothing else does)
+
+Then check the sum: 3 x 300 GiB = 900 GiB real, against ~1550 GiB scheduled.
+Right-size Prometheus (376 GiB provisioned, ~8 GiB actual) or set
+`storageOverProvisioningPercentage` deliberately.