lab/migration/ROLLING-MIGRATION-RUNBOOK.md

# Rolling migration to the v2 disk layout

Moving `worker0`, `worker1` and `worker2` from the v1 layout to v2, one at a
time, without losing data or quorum.

**Status: NOT YET EXECUTED against the fleet.** The mechanism is proven by
`bastion/tests/integration/rolling-drill.test.ts`, which passed **10/10** on
four VMs on 2026-09-30 — including a backup to S3 and a byte-identical restore.
Evidence: `drill-evidence/schema-v2-2026-09-30.txt`.

---

## The one thing to understand first

**You cannot start by evacuating an existing node.** Two independent reasons,
both read off the live cluster:

1. **Placement.** `replica-soft-anti-affinity` is `false`, so a volume puts at
   most one replica per node. There are exactly **3** Longhorn-schedulable nodes
   (`worker0`, `worker1`, `worker2` — `aitopatom` and `spark` have
   `allowScheduling: false` deliberately). **37 volumes want 3 replicas.** Evict
   any one node and the other two already hold a copy each, so the third has
   nowhere legal to go. Eviction can never reach zero.

2. **Capacity**, which blocks it even if placement did not:

   | node | Longhorn disk | used | free |
   |---|---|---|---|
   | worker0 | 727.6 GiB | 91.5 | ~636 |
   | worker1 | 664.7 GiB | 114.3 | ~550 |
   | **worker2** | **99.9 GiB** | 51.1 | **~49** |

   worker2 cannot absorb worker0's 91.5 GiB. No setting changes that.

So: **bring the new box up first (N+1), then evacuate an old one onto it.**

`labctl app k3s evacuate` now refuses in seconds with the offending volume
named, rather than discovering this an hour in. If it refuses, it is right.

---

## Capacity arithmetic, stated plainly

- Today: 727.6 + 664.7 + 99.9 = **~1.49 TiB** of Longhorn disk.
- After v2 (fixed 300 GiB per node): 3 x 300 = **900 GiB** — a **reduction of
  ~590 GiB**.
- Real usage is **~257 GiB**, so it fits with ~640 GiB headroom.
- But **1550 GiB is already *scheduled*** (794 + 558 + 198), which only works
  because `storageOverProvisioningPercentage` is **500**. Against 900 GiB real
  that needs >=172% and leaves no room for the next volume.

**Action:** before migrating the last node, right-size the oversized PVCs
(Prometheus is 376 GiB provisioned against ~8 GiB actual) or set the
over-provisioning percentage deliberately. Do not discover this when a disk
fills.

---

## Preamble — do these once, before touching any node

1. **Backup target.** Configured in `deployments/longhorn/` and
   `Pulumi.homelab.yaml` (SeaweedFS S3 at `10.0.0.194:8333` — a different
   machine from the cluster, which is what makes it a backup rather than another
   copy of the same failure domain). TrueNAS was rejected on evidence: port 2049
   is CLOSED, so NFS would need the export creating first.

   **`defaultSettings.backupTarget` DOES NOT WORK on Longhorn 1.8.** The target
   moved onto the `backuptargets.longhorn.io/default` CR
   (`spec.backupTargetURL`, `spec.credentialSecret`); the helm value is silently
   ignored, no `backup-target` Setting object is created, and the chart reports
   success with no backup target at all. The Pulumi code therefore patches the
   CR via `apiextensions.CustomResourcePatch`, and the RecurringJobs depend on
   that patch so the first scheduled backup cannot fire into nothing.

   Before relying on it:
   - create the `longhorn-backup` bucket;
   - apply with `--target`, **not** a bare `pulumi up` — the stack carries
     unrelated drift (at one point 12 resources, including a replacement of the
     openbao watchdog ConfigMap). Target these four:
     ```
     kubernetes:core/v1:Secret::longhorn-backup-credentials
     kubernetes:helm.sh/v3:Release::longhorn
     ...Release$...:BackupTarget::longhorn-backup-target
     ...Release$...:RecurringJob::longhorn-job-daily-backup
     ...Release$...:RecurringJob::longhorn-job-hourly-snapshot
     ```
     (`scripts/pulumi.sh` refuses an unclean tree; commit and merge rather than
     setting `PULUMI_ALLOW_UNMERGED=1`.)
   - confirm Longhorn accepted it — `available: true` is the appliance saying it
     could reach the endpoint AND authenticate:
     ```
     kubectl -n longhorn-system get backuptargets.longhorn.io default \
       -o custom-columns=URL:.spec.backupTargetURL,AVAIL:.status.available
     ```
   - take one backup, then **restore it to a new volume and check the contents.**
     An unverified backup is not a backup. The restore is the gate — not the
     `pulumi up`, and not `available: true`.

   Note: this cluster has **no CSI VolumeSnapshot API** (no
   `snapshot.storage.k8s.io` resources at all), so backup tooling must use
   Longhorn's native CRs — `snapshots.longhorn.io`, `backups.longhorn.io`, and a
   `fromBackup` StorageClass for restores. The drill does exactly this.

2. **The 21 single-replica volumes.** One copy each, and until step 1 is done, no
   other copy anywhere. For each one on the node you are about to evacuate:
   ```
   kubectl -n longhorn-system patch volumes.longhorn.io <pv> \
     --type=merge -p '{"spec":{"numberOfReplicas":3}}'
   ```
   wait for `status.robustness: healthy`, *then* evacuate. Restore the count
   afterwards if you want it back.

   `evacuate` refuses single-replica volumes by default.
   `--allow-single-replica` overrides it and means "I accept that a failed
   rebuild is permanent loss".

   Detached volumes are listed separately by the pre-check: Longhorn must attach
   them to rebuild, so they need attention before, not during.

3. **Pin the k3s version, or the migration upgrades the cluster by accident.**

   `get.k3s.io` installs **latest**. The fleet runs **v1.34.6+k3s1**; latest at
   the time of writing is **v1.36.4+k3s1** — two minor versions ahead. A node
   reinstalled by this procedure rejoins on a newer k3s than the two members it
   is joining, and Kubernetes supports only **one** minor of skew between
   control-plane members. Roll three nodes and you have performed an unplanned,
   unstaged, two-version upgrade one node at a time.

   Pass `--k3s-version v1.34.6+k3s1` (check the real value with
   `kubectl get nodes -o wide`). Upgrade deliberately, afterwards, as its own
   change.

4. **Add the N+1 node.** Provision the new box with `schemaVersion: 2`, join it,
   and confirm it has a schedulable Longhorn disk:
   ```
   kubectl -n longhorn-system get nodes.longhorn.io
   ```
   A node with no disk hosts no replicas while looking perfectly healthy. If it
   has none, `labctl app k3s admit <node>` repairs it.

---

## Node order

1. **worker0** — plain Fedora x86, the most ordinary box. Go first.
2. **worker2** — Fedora aarch64. Its 100 GiB Longhorn disk becomes 300 GiB
   under v2, so this node *gains* room.
3. **worker1** — **LAST, and a different animal.** Apple Silicon under Asahi:
   the partition table belongs to the vendor installer, root is 4.6 GiB, the
   rancher LV is undersized at 20 GiB, and it carries a containerd bind mount
   that exists in no template or script:
   ```
   /var/lib/longhorn/k3s-containerd /var/lib/rancher/k3s/agent/containerd none \
     bind,x-systemd.requires-mounts-for=/var/lib/longhorn,x-systemd.requires-mounts-for=/var/lib/rancher 0 0
   ```
   A reinstall destroys that silently. Re-create it, or the node runs out of
   root space the first time it pulls a large image.

Quorum: there are **exactly 3 etcd members**, so quorum is 2 and the fleet
tolerates **exactly one member down with no margin**. One node at a time, never
two, and never start a roll while anything is already degraded.

---

## Per-node loop

Gate at every step. If one fails, stop — do not roll the next node on top of a
half-migrated one.

```
# 0. GATE: cluster is whole before you start
kubectl get nodes
kubectl -n longhorn-system get volumes.longhorn.io | grep -v healthy   # expect none
# `k3s etcdctl` is NOT a command. Define this first (see the recovery card):
#   e() { local b=/var/lib/rancher/k3s/server/tls/etcd
#         ETCDCTL_API=3 etcdctl --cacert=$b/server-ca.crt \
#           --cert=$b/server-client.crt --key=$b/server-client.key \
#           --endpoints=https://127.0.0.1:2379 "$@"; }
e member list                  # expect 3, all started
e endpoint health

# 1. Will it even work?
labctl app k3s evacuate <node>      # refuses fast if it cannot complete

# 2. (it evacuates: cordon -> check -> evict+wait -> drain -> verify)
kubectl -n longhorn-system get replicas.longhorn.io -o wide | grep <node>   # expect none

# 3. Prove the data is fine BEFORE destroying anything
#    Restart something that uses a Longhorn PVC and check it reads back.

# 4. The irreversible part — as late as possible
ssh <node> systemctl stop k3s
e member remove <id-of-node>   # member names are "<node>-<8 hex>", not "<node>"
kubectl delete node <node>
e member list                  # expect 2, both healthy
kubectl get --raw /readyz      # from each survivor: ok

# 5. Wipe and reinstall
labctl provision reprovision <node>    # the DISK decides the schema, not you

# 6. Rejoin — against a SURVIVOR, not against itself, PINNED
labctl app k3s install <node> --role k8s-combo \
  --k3s-server https://<survivor-ip>:6443 --k3s-token <token> \
  --k3s-version v1.34.6+k3s1      # MUST match the cluster; see preamble step 3
e member list                  # expect EXACTLY 3

# 7. Back into service
labctl app k3s admit <node>

# 8. Wait for the rebuild, then verify the data again
kubectl -n longhorn-system get volumes.longhorn.io    # all healthy, 3 replicas
#    And check the node actually took replicas BACK — running, not merely assigned:
kubectl -n longhorn-system get replicas.longhorn.io -o wide | grep <node>
```

**Step 6's "exactly 3" is not a formality.** A reinstalled server rejoins as a
*new* etcd member. If the old record survives you get 4 members, quorum rises
from 2-of-3 to 3-of-4, and a cluster that tolerated one failure now tolerates
none — while every dashboard still says green. That is why step 4 removes the
member explicitly instead of trusting the rejoin to clean up.

**Step 8's "running, not merely assigned" is also not a formality.** A replica
can sit at `currentState=stopped` / `desireState=running` with no `failedAt`
indefinitely. The node looks Ready and Schedulable and hosts nothing, and the
*next* node's eviction then has nowhere to drain and stalls for its full hour.

---

## What `pulumi up` puts back, and what it does not

**Restored automatically.** Node labels and taints — both `NodePatch` sites use
`pulumi.com/patchForce`, so they genuinely self-heal.

**NOT restored — re-apply by hand:**

- aitopatom's Longhorn `allowScheduling: false`
- the Prometheus 2-replica pin
- the multus 512Mi memory limit
- worker1's containerd bind mount (above)
- every `local-path` PVC. Model caches re-download; the rest do not come back.

---

## If it goes wrong

- **Evacuation refused.** Believe it. Read the named volume; it is telling you
  either "no node can take this replica" or "the disk does not have room". Add
  capacity rather than forcing it.
- **Evacuation timed out.** The node is left schedulable again and the eviction
  cancelled automatically; the error names which volumes were stuck and in what
  state. If that cleanup itself failed, the error says so — run
  `labctl app k3s admit <node>`.
- **Aborted midway, node still cordoned.** `labctl app k3s admit <node>`
  uncordons, clears the eviction and restores the Longhorn disk.
- **Replicas land on a node but stay `stopped`.** `/var/lib/longhorn/replicas`
  is missing while `longhorn-disk.cfg` exists, so Longhorn considers the disk
  already initialised and never recreates the directory; the replica process
  then fails on a non-recursive `mkdir` every time, permanently.
  `mkdir -p /var/lib/longhorn/replicas`. **Never delete that directory** —
  delete its contents. `labctl app k3s admit <node>` now ensures it exists.
- **Rejoined node hosts no replicas.** It has no Longhorn disk. `admit` repairs
  it; `kubectl -n longhorn-system get nodes.longhorn.io <node> -o yaml` confirms.
- **Node wiped and cannot rejoin.** This is the case the backup target exists
  for. Restore the volumes from backup. If step 1 of the preamble was skipped,
  there is no recovery path — which is precisely why it is step 1.

---

## Known gaps

- Every node reports `RequiredPackages=False — Missing packages: [nfs-utils
  cryptsetup]` and `KernelModulesLoaded=False`. Uniform across the fleet and not
  blocking today, but it would block an NFS backup target and encrypted volumes.
  The S3 target does not need `nfs-utils`.
- The drill ran on VMs with ~3 GiB of test data; production has ~257 GiB. It
  proves **mechanism**, not timing. Keep the 60-minute eviction timeout generous.
- `worker1` has never been VM-tested and cannot be — it needs Apple Silicon.