lab/migration/RECOVERY-CARD-schema-v2.md

# Recovery card — v2 layout migration, one node at a time

Print or keep open. **A node is being wiped. If the cluster goes down there is
no Claude and possibly no internet.** Everything you need is on this page.

---

## The one thing that matters

**`k3s etcdctl` is NOT a command.** It does not exist (`No help topic for
'etcdctl'`). You need the real binary plus k3s's certs. Paste this once on
whichever server you are using:

```
e() {
  local b=/var/lib/rancher/k3s/server/tls/etcd
  ETCDCTL_API=3 etcdctl --cacert=$b/server-ca.crt \
    --cert=$b/server-client.crt --key=$b/server-client.key \
    --endpoints=https://127.0.0.1:2379 "$@"
}
```

(A function, not an alias — aliases do not expand inside scripts. Verified
working on worker0.)

If `etcdctl: command not found`, install it: `dnf install -y etcd` — which needs
working DNS and a reachable mirror, so do not count on it here. Nodes
reinstalled with the current kickstart already have it (`etcd` is in the package
set for k3s server roles precisely so this step is never needed).

Then:

```
e member list -w table
```

There are **3 etcd members. Quorum is 2. You have no margin.**

Two naming traps, both of which make a correct-looking command match nothing:

- **Kubernetes node names are FQDNs.** `worker0-k8s0.ad.itaz.eu`, never
  `worker0`. The same name is used by `kubectl` and by `nodes.longhorn.io`.
  Get it from `kubectl get nodes`; do not type it.
- **etcd member names add a random suffix** on top of that:
  `worker0-k8s0.ad.itaz.eu-e26aacc5`. Match on the prefix.

One node down is survivable. Two is a dead API server and a dead cluster. So:
**one node at a time**, and never start the next while anything is degraded.

Reach the servers directly:

```
ssh root@192.168.8.23     # worker0   (Fedora x86)
ssh root@192.168.8.25     # worker2   (Fedora aarch64)
ssh root@192.168.8.13     # worker1   (Asahi aarch64 — migrate LAST)
```

---

## Before you touch anything

```
kubectl get nodes                                        # all Ready
e member list                                            # exactly 3, all started
e endpoint health                                        # healthy
kubectl -n longhorn-system get volumes.longhorn.io | grep -v healthy   # nothing
```

All four clean, or do not start.

---

## Order

**worker0 → worker2 → worker1.** worker1 is last: vendor-owned partition table,
4.6 G root, undersized rancher LV, and a hand-made bind mount a reinstall
destroys.

---

## The loop

```
# 1. Ask first. It refuses in SECONDS if it cannot finish.
labctl app k3s evacuate <node>
```

**If it refuses, it is right.** It means either no node can legally take a
replica, or the disk has no room. Do not force it. Add the N+1 node first.

```
# 2. Node is empty. Prove the data is fine BEFORE destroying anything.
kubectl -n longhorn-system get replicas.longhorn.io -o wide | grep <node>   # none
#    restart something using a Longhorn PVC; check it reads back.

# 3. Irreversible from here.
ssh <node> systemctl stop k3s
e member remove <id>             # member names are "<node>-<8 hex>"
kubectl delete node <node>
e member list                    # 2, both healthy  <-- CHECK THIS
kubectl get --raw /readyz        # 'ok' from each survivor

# 4. Wipe.
labctl provision reprovision <node>

# 5. Rejoin AGAINST A SURVIVOR, pinned to the cluster's version.
labctl app k3s install <node> --role k8s-combo \
  --k3s-server https://<survivor-ip>:6443 --k3s-token <token> \
  --k3s-version <what the others run>     # kubectl get nodes -o wide
e member list                    # EXACTLY 3  <-- CHECK THIS

# 6. Back in service, then wait for the rebuild.
labctl app k3s admit <node>
kubectl -n longhorn-system get volumes.longhorn.io    # all healthy
```

---

## When something is wrong

| Symptom | Do this |
|---|---|
| Evacuate refused | Believe it. Read the named volume. Add capacity or a node. |
| Evacuate timed out | It already cancelled itself and restored scheduling. If it says it could not: `labctl app k3s admit <node>` |
| Aborted midway, node cordoned | `labctl app k3s admit <node>` |
| Rejoined node hosts no replicas | It has no Longhorn disk. `labctl app k3s admit <node>` |
| Replicas land but stay `stopped` (`desireState=running`, no `failedAt`) | `/var/lib/longhorn/replicas` is missing while `longhorn-disk.cfg` exists, so Longhorn thinks the disk is initialised and the replica process dies on `mkdir`. `mkdir -p /var/lib/longhorn/replicas`. **Never delete that directory** — delete its contents. |
| Rejoined node is a newer k3s than its peers | You forgot `--k3s-version`. More than one minor of skew between control-plane members is unsupported; reinstall it pinned. |
| `member list` shows 4 | `e member remove <dead-id>` — do this now |
| Node wiped, will not rejoin | Restore from the Longhorn backup target. |
| API server gone (2 members down) | You lost quorum. See below. |

### Lost quorum

Do not panic-restart things. From the one surviving server:

```
systemctl stop k3s
k3s server --cluster-reset
systemctl start k3s
```

This rebuilds a single-member cluster from that node's data. The other two
rejoin afterwards as fresh members; their
`/var/lib/rancher/k3s/server/db` must be removed before they will.

---

## worker1 only (Asahi)

After its reinstall, put this back in `/etc/fstab` or the node runs out of root
space the first time it pulls a large image:

```
/var/lib/longhorn/k3s-containerd /var/lib/rancher/k3s/agent/containerd none bind,x-systemd.requires-mounts-for=/var/lib/longhorn,x-systemd.requires-mounts-for=/var/lib/rancher 0 0
```

```
mkdir -p /var/lib/longhorn/k3s-containerd
systemctl daemon-reload && mount -a
```

---

## After all three

Re-apply by hand — `pulumi up` does NOT restore these:

- aitopatom Longhorn `allowScheduling: false`
- Prometheus 2-replica pin
- multus 512Mi memory limit
- every `local-path` PVC (model caches re-download; nothing else does)

Then check the sum: 3 x 300 GiB = 900 GiB real, against ~1550 GiB scheduled.
Right-size Prometheus (376 GiB provisioned, ~8 GiB actual) or set
`storageOverProvisioningPercentage` deliberately.