lab/migration/RECOVERY-CARD-schema-v2.md

Recovery card — v2 layout migration, one node at a time

Print or keep open. A node is being wiped. If the cluster goes down there is no Claude and possibly no internet. Everything you need is on this page.

---

The one thing that matters

k3s etcdctl is NOT a command. It does not exist (No help topic for 'etcdctl'). You need the real binary plus k3s's certs. Paste this once on whichever server you are using:

e() {
  local b=/var/lib/rancher/k3s/server/tls/etcd
  ETCDCTL_API=3 etcdctl --cacert=$b/server-ca.crt \
    --cert=$b/server-client.crt --key=$b/server-client.key \
    --endpoints=https://127.0.0.1:2379 "$@"
}

(A function, not an alias — aliases do not expand inside scripts. Verified working on worker0.)

If etcdctl: command not found, install it: dnf install -y etcd — which needs working DNS and a reachable mirror, so do not count on it here. Nodes reinstalled with the current kickstart already have it (etcd is in the package set for k3s server roles precisely so this step is never needed).

Then:

e member list -w table

There are 3 etcd members. Quorum is 2. You have no margin.

Two naming traps, both of which make a correct-looking command match nothing:

• Kubernetes node names are FQDNs. worker0-k8s0.ad.itaz.eu, never worker0. The same name is used by kubectl and by nodes.longhorn.io. Get it from kubectl get nodes; do not type it.
• etcd member names add a random suffix on top of that: worker0-k8s0.ad.itaz.eu-e26aacc5. Match on the prefix.

One node down is survivable. Two is a dead API server and a dead cluster. So: one node at a time, and never start the next while anything is degraded.

Reach the servers directly:

ssh root@192.168.8.23     # worker0   (Fedora x86)
ssh root@192.168.8.25     # worker2   (Fedora aarch64)
ssh root@192.168.8.13     # worker1   (Asahi aarch64 — migrate LAST)

---

Before you touch anything

kubectl get nodes                                        # all Ready
e member list                                            # exactly 3, all started
e endpoint health                                        # healthy
kubectl -n longhorn-system get volumes.longhorn.io | grep -v healthy   # nothing

All four clean, or do not start.

---

Order

worker0 → worker2 → worker1. worker1 is last: vendor-owned partition table, 4.6 G root, undersized rancher LV, and a hand-made bind mount a reinstall destroys.

---

The loop

# 1. Ask first. It refuses in SECONDS if it cannot finish.
labctl app k3s evacuate <node>

If it refuses, it is right. It means either no node can legally take a replica, or the disk has no room. Do not force it. Add the N+1 node first.

# 2. Node is empty. Prove the data is fine BEFORE destroying anything.
kubectl -n longhorn-system get replicas.longhorn.io -o wide | grep <node>   # none
#    restart something using a Longhorn PVC; check it reads back.

# 3. Irreversible from here.
ssh <node> systemctl stop k3s
e member remove <id>             # member names are "<node>-<8 hex>"
kubectl delete node <node>
e member list                    # 2, both healthy  <-- CHECK THIS
kubectl get --raw /readyz        # 'ok' from each survivor

# 4. Wipe.
labctl provision reprovision <node>

# 5. Rejoin AGAINST A SURVIVOR, pinned to the cluster's version.
labctl app k3s install <node> --role k8s-combo \
  --k3s-server https://<survivor-ip>:6443 --k3s-token <token> \
  --k3s-version <what the others run>     # kubectl get nodes -o wide
e member list                    # EXACTLY 3  <-- CHECK THIS

# 6. Back in service, then wait for the rebuild.
labctl app k3s admit <node>
kubectl -n longhorn-system get volumes.longhorn.io    # all healthy

---

When something is wrong

Symptom	Do this
Evacuate refused	Believe it. Read the named volume. Add capacity or a node.
Evacuate timed out	It already cancelled itself and restored scheduling. If it says it could not: labctl app k3s admit <node>
Aborted midway, node cordoned	labctl app k3s admit <node>
Rejoined node hosts no replicas	It has no Longhorn disk. labctl app k3s admit <node>
Replicas land but stay stopped (desireState=running, no failedAt)	/var/lib/longhorn/replicas is missing while longhorn-disk.cfg exists, so Longhorn thinks the disk is initialised and the replica process dies on mkdir. mkdir -p /var/lib/longhorn/replicas. Never delete that directory — delete its contents.
Rejoined node is a newer k3s than its peers	You forgot --k3s-version. More than one minor of skew between control-plane members is unsupported; reinstall it pinned.
member list shows 4	e member remove <dead-id> — do this now
Node wiped, will not rejoin	Restore from the Longhorn backup target.
API server gone (2 members down)	You lost quorum. See below.

Lost quorum

Do not panic-restart things. From the one surviving server:

systemctl stop k3s
k3s server --cluster-reset
systemctl start k3s

This rebuilds a single-member cluster from that node's data. The other two rejoin afterwards as fresh members; their /var/lib/rancher/k3s/server/db must be removed before they will.

---

worker1 only (Asahi)

After its reinstall, put this back in /etc/fstab or the node runs out of root space the first time it pulls a large image:

/var/lib/longhorn/k3s-containerd /var/lib/rancher/k3s/agent/containerd none bind,x-systemd.requires-mounts-for=/var/lib/longhorn,x-systemd.requires-mounts-for=/var/lib/rancher 0 0

mkdir -p /var/lib/longhorn/k3s-containerd
systemctl daemon-reload && mount -a

---

After all three

Re-apply by hand — pulumi up does NOT restore these:

• aitopatom Longhorn allowScheduling: false
• Prometheus 2-replica pin
• multus 512Mi memory limit
• every local-path PVC (model caches re-download; nothing else does)

Then check the sum: 3 x 300 GiB = 900 GiB real, against ~1550 GiB scheduled. Right-size Prometheus (376 GiB provisioned, ~8 GiB actual) or set storageOverProvisioningPercentage deliberately.