# Recovery card — v2 layout migration, one node at a time Print or keep open. **A node is being wiped. If the cluster goes down there is no Claude and possibly no internet.** Everything you need is on this page. --- ## The one thing that matters **`k3s etcdctl` is NOT a command.** It does not exist (`No help topic for 'etcdctl'`). You need the real binary plus k3s's certs. Paste this once on whichever server you are using: ``` e() { local b=/var/lib/rancher/k3s/server/tls/etcd ETCDCTL_API=3 etcdctl --cacert=$b/server-ca.crt \ --cert=$b/server-client.crt --key=$b/server-client.key \ --endpoints=https://127.0.0.1:2379 "$@" } ``` (A function, not an alias — aliases do not expand inside scripts. Verified working on worker0.) If `etcdctl: command not found`, install it: `dnf install -y etcd` — which needs working DNS and a reachable mirror, so do not count on it here. Nodes reinstalled with the current kickstart already have it (`etcd` is in the package set for k3s server roles precisely so this step is never needed). Then: ``` e member list -w table ``` There are **3 etcd members. Quorum is 2. You have no margin.** Two naming traps, both of which make a correct-looking command match nothing: - **Kubernetes node names are FQDNs.** `worker0-k8s0.ad.itaz.eu`, never `worker0`. The same name is used by `kubectl` and by `nodes.longhorn.io`. Get it from `kubectl get nodes`; do not type it. - **etcd member names add a random suffix** on top of that: `worker0-k8s0.ad.itaz.eu-e26aacc5`. Match on the prefix. One node down is survivable. Two is a dead API server and a dead cluster. So: **one node at a time**, and never start the next while anything is degraded. Reach the servers directly: ``` ssh root@192.168.8.23 # worker0 (Fedora x86) ssh root@192.168.8.25 # worker2 (Fedora aarch64) ssh root@192.168.8.13 # worker1 (Asahi aarch64 — migrate LAST) ``` --- ## Before you touch anything ``` kubectl get nodes # all Ready e member list # exactly 3, all started e endpoint health # healthy kubectl -n longhorn-system get volumes.longhorn.io | grep -v healthy # nothing ``` All four clean, or do not start. --- ## Order **worker0 → worker2 → worker1.** worker1 is last: vendor-owned partition table, 4.6 G root, undersized rancher LV, and a hand-made bind mount a reinstall destroys. --- ## The loop ``` # 1. Ask first. It refuses in SECONDS if it cannot finish. labctl app k3s evacuate ``` **If it refuses, it is right.** It means either no node can legally take a replica, or the disk has no room. Do not force it. Add the N+1 node first. ``` # 2. Node is empty. Prove the data is fine BEFORE destroying anything. kubectl -n longhorn-system get replicas.longhorn.io -o wide | grep # none # restart something using a Longhorn PVC; check it reads back. # 3. Irreversible from here. ssh systemctl stop k3s e member remove # member names are "-<8 hex>" kubectl delete node e member list # 2, both healthy <-- CHECK THIS kubectl get --raw /readyz # 'ok' from each survivor # 4. Wipe. labctl provision reprovision # 5. Rejoin AGAINST A SURVIVOR, pinned to the cluster's version. labctl app k3s install --role k8s-combo \ --k3s-server https://:6443 --k3s-token \ --k3s-version # kubectl get nodes -o wide e member list # EXACTLY 3 <-- CHECK THIS # 6. Back in service, then wait for the rebuild. labctl app k3s admit kubectl -n longhorn-system get volumes.longhorn.io # all healthy ``` --- ## When something is wrong | Symptom | Do this | |---|---| | Evacuate refused | Believe it. Read the named volume. Add capacity or a node. | | Evacuate timed out | It already cancelled itself and restored scheduling. If it says it could not: `labctl app k3s admit ` | | Aborted midway, node cordoned | `labctl app k3s admit ` | | Rejoined node hosts no replicas | It has no Longhorn disk. `labctl app k3s admit ` | | Replicas land but stay `stopped` (`desireState=running`, no `failedAt`) | `/var/lib/longhorn/replicas` is missing while `longhorn-disk.cfg` exists, so Longhorn thinks the disk is initialised and the replica process dies on `mkdir`. `mkdir -p /var/lib/longhorn/replicas`. **Never delete that directory** — delete its contents. | | Rejoined node is a newer k3s than its peers | You forgot `--k3s-version`. More than one minor of skew between control-plane members is unsupported; reinstall it pinned. | | `member list` shows 4 | `e member remove ` — do this now | | Node wiped, will not rejoin | Restore from the Longhorn backup target. | | API server gone (2 members down) | You lost quorum. See below. | ### Lost quorum Do not panic-restart things. From the one surviving server: ``` systemctl stop k3s k3s server --cluster-reset systemctl start k3s ``` This rebuilds a single-member cluster from that node's data. The other two rejoin afterwards as fresh members; their `/var/lib/rancher/k3s/server/db` must be removed before they will. --- ## worker1 only (Asahi) After its reinstall, put this back in `/etc/fstab` or the node runs out of root space the first time it pulls a large image: ``` /var/lib/longhorn/k3s-containerd /var/lib/rancher/k3s/agent/containerd none bind,x-systemd.requires-mounts-for=/var/lib/longhorn,x-systemd.requires-mounts-for=/var/lib/rancher 0 0 ``` ``` mkdir -p /var/lib/longhorn/k3s-containerd systemctl daemon-reload && mount -a ``` --- ## After all three Re-apply by hand — `pulumi up` does NOT restore these: - aitopatom Longhorn `allowScheduling: false` - Prometheus 2-replica pin - multus 512Mi memory limit - every `local-path` PVC (model caches re-download; nothing else does) Then check the sum: 3 x 300 GiB = 900 GiB real, against ~1550 GiB scheduled. Right-size Prometheus (376 GiB provisioned, ~8 GiB actual) or set `storageOverProvisioningPercentage` deliberately.