lab/migration/ROLLING-MIGRATION-RUNBOOK.md

Rolling migration to the v2 disk layout

Moving worker0, worker1 and worker2 from the v1 layout to v2, one at a time, without losing data or quorum.

Status: NOT YET EXECUTED against the fleet. The mechanism is proven by bastion/tests/integration/rolling-drill.test.ts, which passed 10/10 on four VMs on 2026-09-30 — including a backup to S3 and a byte-identical restore. Evidence: drill-evidence/schema-v2-2026-09-30.txt.

---

The one thing to understand first

You cannot start by evacuating an existing node. Two independent reasons, both read off the live cluster:

1. Placement. replica-soft-anti-affinity is false, so a volume puts at most one replica per node. There are exactly 3 Longhorn-schedulable nodes (worker0, worker1, worker2 — aitopatom and spark have allowScheduling: false deliberately). 37 volumes want 3 replicas. Evict any one node and the other two already hold a copy each, so the third has nowhere legal to go. Eviction can never reach zero.

2. Capacity, which blocks it even if placement did not:

   node	Longhorn disk	used	free
   worker0	727.6 GiB	91.5	~636
   worker1	664.7 GiB	114.3	~550
   worker2	99.9 GiB	51.1	~49

   worker2 cannot absorb worker0's 91.5 GiB. No setting changes that.

So: bring the new box up first (N+1), then evacuate an old one onto it.

labctl app k3s evacuate now refuses in seconds with the offending volume named, rather than discovering this an hour in. If it refuses, it is right.

---

Capacity arithmetic, stated plainly

• Today: 727.6 + 664.7 + 99.9 = ~1.49 TiB of Longhorn disk.
• After v2 (fixed 300 GiB per node): 3 x 300 = 900 GiB — a reduction of ~590 GiB.
• Real usage is ~257 GiB, so it fits with ~640 GiB headroom.
• But 1550 GiB is already scheduled (794 + 558 + 198), which only works because storageOverProvisioningPercentage is 500. Against 900 GiB real that needs >=172% and leaves no room for the next volume.

Action: before migrating the last node, right-size the oversized PVCs (Prometheus is 376 GiB provisioned against ~8 GiB actual) or set the over-provisioning percentage deliberately. Do not discover this when a disk fills.

---

Preamble — do these once, before touching any node

1. Backup target. Configured in deployments/longhorn/ and Pulumi.homelab.yaml (SeaweedFS S3 at 10.0.0.194:8333 — a different machine from the cluster, which is what makes it a backup rather than another copy of the same failure domain). TrueNAS was rejected on evidence: port 2049 is CLOSED, so NFS would need the export creating first.

   defaultSettings.backupTarget DOES NOT WORK on Longhorn 1.8. The target moved onto the backuptargets.longhorn.io/default CR (spec.backupTargetURL, spec.credentialSecret); the helm value is silently ignored, no backup-target Setting object is created, and the chart reports success with no backup target at all. The Pulumi code therefore patches the CR via apiextensions.CustomResourcePatch, and the RecurringJobs depend on that patch so the first scheduled backup cannot fire into nothing.

   Before relying on it:

   • create the longhorn-backup bucket;
   • apply with --target, not a bare pulumi up — the stack carries unrelated drift (at one point 12 resources, including a replacement of the openbao watchdog ConfigMap). Target these four:

     kubernetes:core/v1:Secret::longhorn-backup-credentials
     kubernetes:helm.sh/v3:Release::longhorn
     ...Release$...:BackupTarget::longhorn-backup-target
     ...Release$...:RecurringJob::longhorn-job-daily-backup
     ...Release$...:RecurringJob::longhorn-job-hourly-snapshot
     

     (scripts/pulumi.sh refuses an unclean tree; commit and merge rather than setting PULUMI_ALLOW_UNMERGED=1.)
   • confirm Longhorn accepted it — available: true is the appliance saying it could reach the endpoint AND authenticate:

     kubectl -n longhorn-system get backuptargets.longhorn.io default \
       -o custom-columns=URL:.spec.backupTargetURL,AVAIL:.status.available
     

   • take one backup, then restore it to a new volume and check the contents. An unverified backup is not a backup. The restore is the gate — not the pulumi up, and not available: true.

   Note: this cluster has no CSI VolumeSnapshot API (no snapshot.storage.k8s.io resources at all), so backup tooling must use Longhorn's native CRs — snapshots.longhorn.io, backups.longhorn.io, and a fromBackup StorageClass for restores. The drill does exactly this.

2. The 21 single-replica volumes. One copy each, and until step 1 is done, no other copy anywhere. For each one on the node you are about to evacuate:

   kubectl -n longhorn-system patch volumes.longhorn.io <pv> \
     --type=merge -p '{"spec":{"numberOfReplicas":3}}'
   

   wait for status.robustness: healthy, then evacuate. Restore the count afterwards if you want it back.

   evacuate refuses single-replica volumes by default. --allow-single-replica overrides it and means "I accept that a failed rebuild is permanent loss".

   Detached volumes are listed separately by the pre-check: Longhorn must attach them to rebuild, so they need attention before, not during.

3. Pin the k3s version, or the migration upgrades the cluster by accident.

   get.k3s.io installs latest. The fleet runs v1.34.6+k3s1; latest at the time of writing is v1.36.4+k3s1 — two minor versions ahead. A node reinstalled by this procedure rejoins on a newer k3s than the two members it is joining, and Kubernetes supports only one minor of skew between control-plane members. Roll three nodes and you have performed an unplanned, unstaged, two-version upgrade one node at a time.

   Pass --k3s-version v1.34.6+k3s1 (check the real value with kubectl get nodes -o wide). Upgrade deliberately, afterwards, as its own change.

4. Add the N+1 node. Provision the new box with schemaVersion: 2, join it, and confirm it has a schedulable Longhorn disk:

   kubectl -n longhorn-system get nodes.longhorn.io
   

   A node with no disk hosts no replicas while looking perfectly healthy. If it has none, labctl app k3s admit <node> repairs it.

---

Node order

1. worker0 — plain Fedora x86, the most ordinary box. Go first.
2. worker2 — Fedora aarch64. Its 100 GiB Longhorn disk becomes 300 GiB under v2, so this node gains room.
3. worker1 — LAST, and a different animal. Apple Silicon under Asahi: the partition table belongs to the vendor installer, root is 4.6 GiB, the rancher LV is undersized at 20 GiB, and it carries a containerd bind mount that exists in no template or script:

   /var/lib/longhorn/k3s-containerd /var/lib/rancher/k3s/agent/containerd none \
     bind,x-systemd.requires-mounts-for=/var/lib/longhorn,x-systemd.requires-mounts-for=/var/lib/rancher 0 0
   

   A reinstall destroys that silently. Re-create it, or the node runs out of root space the first time it pulls a large image.

Quorum: there are exactly 3 etcd members, so quorum is 2 and the fleet tolerates exactly one member down with no margin. One node at a time, never two, and never start a roll while anything is already degraded.

---

Per-node loop

Gate at every step. If one fails, stop — do not roll the next node on top of a half-migrated one.

# 0. GATE: cluster is whole before you start
kubectl get nodes
kubectl -n longhorn-system get volumes.longhorn.io | grep -v healthy   # expect none
# `k3s etcdctl` is NOT a command. Define this first (see the recovery card):
#   e() { local b=/var/lib/rancher/k3s/server/tls/etcd
#         ETCDCTL_API=3 etcdctl --cacert=$b/server-ca.crt \
#           --cert=$b/server-client.crt --key=$b/server-client.key \
#           --endpoints=https://127.0.0.1:2379 "$@"; }
e member list                  # expect 3, all started
e endpoint health

# 1. Will it even work?
labctl app k3s evacuate <node>      # refuses fast if it cannot complete

# 2. (it evacuates: cordon -> check -> evict+wait -> drain -> verify)
kubectl -n longhorn-system get replicas.longhorn.io -o wide | grep <node>   # expect none

# 3. Prove the data is fine BEFORE destroying anything
#    Restart something that uses a Longhorn PVC and check it reads back.

# 4. The irreversible part — as late as possible
ssh <node> systemctl stop k3s
e member remove <id-of-node>   # member names are "<node>-<8 hex>", not "<node>"
kubectl delete node <node>
e member list                  # expect 2, both healthy
kubectl get --raw /readyz      # from each survivor: ok

# 5. Wipe and reinstall
labctl provision reprovision <node>    # the DISK decides the schema, not you

# 6. Rejoin — against a SURVIVOR, not against itself, PINNED
labctl app k3s install <node> --role k8s-combo \
  --k3s-server https://<survivor-ip>:6443 --k3s-token <token> \
  --k3s-version v1.34.6+k3s1      # MUST match the cluster; see preamble step 3
e member list                  # expect EXACTLY 3

# 7. Back into service
labctl app k3s admit <node>

# 8. Wait for the rebuild, then verify the data again
kubectl -n longhorn-system get volumes.longhorn.io    # all healthy, 3 replicas
#    And check the node actually took replicas BACK — running, not merely assigned:
kubectl -n longhorn-system get replicas.longhorn.io -o wide | grep <node>

Step 6's "exactly 3" is not a formality. A reinstalled server rejoins as a new etcd member. If the old record survives you get 4 members, quorum rises from 2-of-3 to 3-of-4, and a cluster that tolerated one failure now tolerates none — while every dashboard still says green. That is why step 4 removes the member explicitly instead of trusting the rejoin to clean up.

Step 8's "running, not merely assigned" is also not a formality. A replica can sit at currentState=stopped / desireState=running with no failedAt indefinitely. The node looks Ready and Schedulable and hosts nothing, and the next node's eviction then has nowhere to drain and stalls for its full hour.

---

What pulumi up puts back, and what it does not

Restored automatically. Node labels and taints — both NodePatch sites use pulumi.com/patchForce, so they genuinely self-heal.

NOT restored — re-apply by hand:

• aitopatom's Longhorn allowScheduling: false
• the Prometheus 2-replica pin
• the multus 512Mi memory limit
• worker1's containerd bind mount (above)
• every local-path PVC. Model caches re-download; the rest do not come back.

---

If it goes wrong

• Evacuation refused. Believe it. Read the named volume; it is telling you either "no node can take this replica" or "the disk does not have room". Add capacity rather than forcing it.
• Evacuation timed out. The node is left schedulable again and the eviction cancelled automatically; the error names which volumes were stuck and in what state. If that cleanup itself failed, the error says so — run labctl app k3s admit <node>.
• Aborted midway, node still cordoned. labctl app k3s admit <node> uncordons, clears the eviction and restores the Longhorn disk.
• Replicas land on a node but stay stopped. /var/lib/longhorn/replicas is missing while longhorn-disk.cfg exists, so Longhorn considers the disk already initialised and never recreates the directory; the replica process then fails on a non-recursive mkdir every time, permanently. mkdir -p /var/lib/longhorn/replicas. Never delete that directory — delete its contents. labctl app k3s admit <node> now ensures it exists.
• Rejoined node hosts no replicas. It has no Longhorn disk. admit repairs it; kubectl -n longhorn-system get nodes.longhorn.io <node> -o yaml confirms.
• Node wiped and cannot rejoin. This is the case the backup target exists for. Restore the volumes from backup. If step 1 of the preamble was skipped, there is no recovery path — which is precisely why it is step 1.

---

Known gaps

• Every node reports RequiredPackages=False — Missing packages: [nfs-utils cryptsetup] and KernelModulesLoaded=False. Uniform across the fleet and not blocking today, but it would block an NFS backup target and encrypted volumes. The S3 target does not need nfs-utils.
• The drill ran on VMs with ~3 GiB of test data; production has ~257 GiB. It proves mechanism, not timing. Keep the 60-minute eviction timeout generous.
• worker1 has never been VM-tested and cannot be — it needs Apple Silicon.