# Rolling migration to the v2 disk layout Moving `worker0`, `worker1` and `worker2` from the v1 layout to v2, one at a time, without losing data or quorum. **Status: NOT YET EXECUTED against the fleet.** The mechanism is proven by `bastion/tests/integration/rolling-drill.test.ts`, which passed **10/10** on four VMs on 2026-09-30 — including a backup to S3 and a byte-identical restore. Evidence: `drill-evidence/schema-v2-2026-09-30.txt`. --- ## The one thing to understand first **You cannot start by evacuating an existing node.** Two independent reasons, both read off the live cluster: 1. **Placement.** `replica-soft-anti-affinity` is `false`, so a volume puts at most one replica per node. There are exactly **3** Longhorn-schedulable nodes (`worker0`, `worker1`, `worker2` — `aitopatom` and `spark` have `allowScheduling: false` deliberately). **37 volumes want 3 replicas.** Evict any one node and the other two already hold a copy each, so the third has nowhere legal to go. Eviction can never reach zero. 2. **Capacity**, which blocks it even if placement did not: | node | Longhorn disk | used | free | |---|---|---|---| | worker0 | 727.6 GiB | 91.5 | ~636 | | worker1 | 664.7 GiB | 114.3 | ~550 | | **worker2** | **99.9 GiB** | 51.1 | **~49** | worker2 cannot absorb worker0's 91.5 GiB. No setting changes that. So: **bring the new box up first (N+1), then evacuate an old one onto it.** `labctl app k3s evacuate` now refuses in seconds with the offending volume named, rather than discovering this an hour in. If it refuses, it is right. --- ## Capacity arithmetic, stated plainly - Today: 727.6 + 664.7 + 99.9 = **~1.49 TiB** of Longhorn disk. - After v2 (fixed 300 GiB per node): 3 x 300 = **900 GiB** — a **reduction of ~590 GiB**. - Real usage is **~257 GiB**, so it fits with ~640 GiB headroom. - But **1550 GiB is already *scheduled*** (794 + 558 + 198), which only works because `storageOverProvisioningPercentage` is **500**. Against 900 GiB real that needs >=172% and leaves no room for the next volume. **Action:** before migrating the last node, right-size the oversized PVCs (Prometheus is 376 GiB provisioned against ~8 GiB actual) or set the over-provisioning percentage deliberately. Do not discover this when a disk fills. --- ## Preamble — do these once, before touching any node 1. **Backup target.** Configured in `deployments/longhorn/` and `Pulumi.homelab.yaml` (SeaweedFS S3 at `10.0.0.194:8333` — a different machine from the cluster, which is what makes it a backup rather than another copy of the same failure domain). TrueNAS was rejected on evidence: port 2049 is CLOSED, so NFS would need the export creating first. **`defaultSettings.backupTarget` DOES NOT WORK on Longhorn 1.8.** The target moved onto the `backuptargets.longhorn.io/default` CR (`spec.backupTargetURL`, `spec.credentialSecret`); the helm value is silently ignored, no `backup-target` Setting object is created, and the chart reports success with no backup target at all. The Pulumi code therefore patches the CR via `apiextensions.CustomResourcePatch`, and the RecurringJobs depend on that patch so the first scheduled backup cannot fire into nothing. Before relying on it: - create the `longhorn-backup` bucket; - apply with `--target`, **not** a bare `pulumi up` — the stack carries unrelated drift (at one point 12 resources, including a replacement of the openbao watchdog ConfigMap). Target these four: ``` kubernetes:core/v1:Secret::longhorn-backup-credentials kubernetes:helm.sh/v3:Release::longhorn ...Release$...:BackupTarget::longhorn-backup-target ...Release$...:RecurringJob::longhorn-job-daily-backup ...Release$...:RecurringJob::longhorn-job-hourly-snapshot ``` (`scripts/pulumi.sh` refuses an unclean tree; commit and merge rather than setting `PULUMI_ALLOW_UNMERGED=1`.) - confirm Longhorn accepted it — `available: true` is the appliance saying it could reach the endpoint AND authenticate: ``` kubectl -n longhorn-system get backuptargets.longhorn.io default \ -o custom-columns=URL:.spec.backupTargetURL,AVAIL:.status.available ``` - take one backup, then **restore it to a new volume and check the contents.** An unverified backup is not a backup. The restore is the gate — not the `pulumi up`, and not `available: true`. Note: this cluster has **no CSI VolumeSnapshot API** (no `snapshot.storage.k8s.io` resources at all), so backup tooling must use Longhorn's native CRs — `snapshots.longhorn.io`, `backups.longhorn.io`, and a `fromBackup` StorageClass for restores. The drill does exactly this. 2. **The 21 single-replica volumes.** One copy each, and until step 1 is done, no other copy anywhere. For each one on the node you are about to evacuate: ``` kubectl -n longhorn-system patch volumes.longhorn.io \ --type=merge -p '{"spec":{"numberOfReplicas":3}}' ``` wait for `status.robustness: healthy`, *then* evacuate. Restore the count afterwards if you want it back. `evacuate` refuses single-replica volumes by default. `--allow-single-replica` overrides it and means "I accept that a failed rebuild is permanent loss". Detached volumes are listed separately by the pre-check: Longhorn must attach them to rebuild, so they need attention before, not during. 3. **Pin the k3s version, or the migration upgrades the cluster by accident.** `get.k3s.io` installs **latest**. The fleet runs **v1.34.6+k3s1**; latest at the time of writing is **v1.36.4+k3s1** — two minor versions ahead. A node reinstalled by this procedure rejoins on a newer k3s than the two members it is joining, and Kubernetes supports only **one** minor of skew between control-plane members. Roll three nodes and you have performed an unplanned, unstaged, two-version upgrade one node at a time. Pass `--k3s-version v1.34.6+k3s1` (check the real value with `kubectl get nodes -o wide`). Upgrade deliberately, afterwards, as its own change. 4. **Add the N+1 node.** Provision the new box with `schemaVersion: 2`, join it, and confirm it has a schedulable Longhorn disk: ``` kubectl -n longhorn-system get nodes.longhorn.io ``` A node with no disk hosts no replicas while looking perfectly healthy. If it has none, `labctl app k3s admit ` repairs it. --- ## Node order 1. **worker0** — plain Fedora x86, the most ordinary box. Go first. 2. **worker2** — Fedora aarch64. Its 100 GiB Longhorn disk becomes 300 GiB under v2, so this node *gains* room. 3. **worker1** — **LAST, and a different animal.** Apple Silicon under Asahi: the partition table belongs to the vendor installer, root is 4.6 GiB, the rancher LV is undersized at 20 GiB, and it carries a containerd bind mount that exists in no template or script: ``` /var/lib/longhorn/k3s-containerd /var/lib/rancher/k3s/agent/containerd none \ bind,x-systemd.requires-mounts-for=/var/lib/longhorn,x-systemd.requires-mounts-for=/var/lib/rancher 0 0 ``` A reinstall destroys that silently. Re-create it, or the node runs out of root space the first time it pulls a large image. Quorum: there are **exactly 3 etcd members**, so quorum is 2 and the fleet tolerates **exactly one member down with no margin**. One node at a time, never two, and never start a roll while anything is already degraded. --- ## Per-node loop Gate at every step. If one fails, stop — do not roll the next node on top of a half-migrated one. ``` # 0. GATE: cluster is whole before you start kubectl get nodes kubectl -n longhorn-system get volumes.longhorn.io | grep -v healthy # expect none # `k3s etcdctl` is NOT a command. Define this first (see the recovery card): # e() { local b=/var/lib/rancher/k3s/server/tls/etcd # ETCDCTL_API=3 etcdctl --cacert=$b/server-ca.crt \ # --cert=$b/server-client.crt --key=$b/server-client.key \ # --endpoints=https://127.0.0.1:2379 "$@"; } e member list # expect 3, all started e endpoint health # 1. Will it even work? labctl app k3s evacuate # refuses fast if it cannot complete # 2. (it evacuates: cordon -> check -> evict+wait -> drain -> verify) kubectl -n longhorn-system get replicas.longhorn.io -o wide | grep # expect none # 3. Prove the data is fine BEFORE destroying anything # Restart something that uses a Longhorn PVC and check it reads back. # 4. The irreversible part — as late as possible ssh systemctl stop k3s e member remove # member names are "-<8 hex>", not "" kubectl delete node e member list # expect 2, both healthy kubectl get --raw /readyz # from each survivor: ok # 5. Wipe and reinstall labctl provision reprovision # the DISK decides the schema, not you # 6. Rejoin — against a SURVIVOR, not against itself, PINNED labctl app k3s install --role k8s-combo \ --k3s-server https://:6443 --k3s-token \ --k3s-version v1.34.6+k3s1 # MUST match the cluster; see preamble step 3 e member list # expect EXACTLY 3 # 7. Back into service labctl app k3s admit # 8. Wait for the rebuild, then verify the data again kubectl -n longhorn-system get volumes.longhorn.io # all healthy, 3 replicas # And check the node actually took replicas BACK — running, not merely assigned: kubectl -n longhorn-system get replicas.longhorn.io -o wide | grep ``` **Step 6's "exactly 3" is not a formality.** A reinstalled server rejoins as a *new* etcd member. If the old record survives you get 4 members, quorum rises from 2-of-3 to 3-of-4, and a cluster that tolerated one failure now tolerates none — while every dashboard still says green. That is why step 4 removes the member explicitly instead of trusting the rejoin to clean up. **Step 8's "running, not merely assigned" is also not a formality.** A replica can sit at `currentState=stopped` / `desireState=running` with no `failedAt` indefinitely. The node looks Ready and Schedulable and hosts nothing, and the *next* node's eviction then has nowhere to drain and stalls for its full hour. --- ## What `pulumi up` puts back, and what it does not **Restored automatically.** Node labels and taints — both `NodePatch` sites use `pulumi.com/patchForce`, so they genuinely self-heal. **NOT restored — re-apply by hand:** - aitopatom's Longhorn `allowScheduling: false` - the Prometheus 2-replica pin - the multus 512Mi memory limit - worker1's containerd bind mount (above) - every `local-path` PVC. Model caches re-download; the rest do not come back. --- ## If it goes wrong - **Evacuation refused.** Believe it. Read the named volume; it is telling you either "no node can take this replica" or "the disk does not have room". Add capacity rather than forcing it. - **Evacuation timed out.** The node is left schedulable again and the eviction cancelled automatically; the error names which volumes were stuck and in what state. If that cleanup itself failed, the error says so — run `labctl app k3s admit `. - **Aborted midway, node still cordoned.** `labctl app k3s admit ` uncordons, clears the eviction and restores the Longhorn disk. - **Replicas land on a node but stay `stopped`.** `/var/lib/longhorn/replicas` is missing while `longhorn-disk.cfg` exists, so Longhorn considers the disk already initialised and never recreates the directory; the replica process then fails on a non-recursive `mkdir` every time, permanently. `mkdir -p /var/lib/longhorn/replicas`. **Never delete that directory** — delete its contents. `labctl app k3s admit ` now ensures it exists. - **Rejoined node hosts no replicas.** It has no Longhorn disk. `admit` repairs it; `kubectl -n longhorn-system get nodes.longhorn.io -o yaml` confirms. - **Node wiped and cannot rejoin.** This is the case the backup target exists for. Restore the volumes from backup. If step 1 of the preamble was skipped, there is no recovery path — which is precisely why it is step 1. --- ## Known gaps - Every node reports `RequiredPackages=False — Missing packages: [nfs-utils cryptsetup]` and `KernelModulesLoaded=False`. Uniform across the fleet and not blocking today, but it would block an NFS backup target and encrypted volumes. The S3 target does not need `nfs-utils`. - The drill ran on VMs with ~3 GiB of test data; production has ~257 GiB. It proves **mechanism**, not timing. Keep the 60-minute eviction timeout generous. - `worker1` has never been VM-tested and cannot be — it needs Apple Silicon.