michal/lab
1
0
Fork 0
Code Issues Pull Requests 3 Actions Packages Projects Releases Wiki Activity

v2 partition schema: capability model, fixed sizing, and a proven rolling migration
Some checks failed
CI/CD / typecheck (push) Failing after 9s
Details
CI/CD / test (push) Failing after 10s
Details
CI/CD / lint (push) Failing after 24s
Details
CI/CD / build (push) Has been skipped
Details
CI/CD / publish-rpm (push) Has been skipped
Details
CI/CD / publish-deb (push) Has been skipped
Details

Browse Source 
The fleet's disk layout was decided by literal role-name comparisons at 17
sites across 8 files. That is why `labcontroller` got no rancher LV (it
matched neither "infra" nor "worker"), why labmaster carried an empty 728 GiB
Longhorn LV, and why a 62 GiB container image tipped that node into
DiskPressure and evicted bastion, labd, cockroach and openbao.

Three generators each hardcoded their own sizes and disagreed: the kickstart
said rancher=120G, ubuntu-autoinstall said 20G, asahi-firstboot said 20G under
a comment asserting the sizes matched install.ks.ts. They did not.

ROLE_REGISTRY already carried `k3s` and `parent` and nothing read them. This
makes capabilities authoritative, moves every size into one shared module, and
adds schema v2 (fixed 250G rancher / 300G longhorn, never --grow). Existing
machines stay v1 forever: schema is detected from the on-disk VG tag, so a v1
box stays v1 even when v2 is requested.

Proven on real VMs, not just unit tests: all four topologies installed and had
their layouts read back off the machines; markers in /home, /srv,
/var/lib/longhorn and /var/lib/rancher survived reinstall; and a 4-node cluster
formed, took real data, and had all three servers rolled one at a time through
evacuate / wipe / reinstall / rejoin with checksums verified before and after
each roll. Drill: 10/10, evidence in migration/drill-evidence/.

Production defects the drill found, none of which a unit test could reach:

- sshOpts() capped every operation at 30s while drainNode asked kubectl for
  --timeout=15m, so the transport killed the drain after evicting most pods and
  returned a transcript that read like success. `labctl app k3s evacuate` could
  never have drained a real node.
- generateAgentConfig put node-role.kubernetes.io/worker in the kubelet's
  node-label list, which NodeRestriction rejects, so no agent installed through
  K3sModule could start. systemd reported only "did not take the steps required
  by its unit configuration".
- toOpContext built K3sConfig by enumerating three keys and silently dropped
  the rest, so the k3s version pin did nothing and --allow-single-replica was a
  no-op that claimed to override a data-loss guard.
- The Longhorn disk annotation was mangled by shell quoting; with
  createDefaultDiskLabeledNodes the node then gets no disk at all, joins, goes
  Ready, and silently hosts no replica.
- evacuate drained before evicting, which blocks on Longhorn's PDBs and would
  destroy a single-replica volume's only source.
- etcd member ids were parsed from JSON, losing uint64 precision and emitting
  decimal where `member remove` needs hex -- at the moment a node is stopped
  and about to be wiped.
- get.k3s.io installs latest, two minor versions ahead of the fleet, so a
  reinstalled node would rejoin on a newer k3s than its peers and roll an
  unplanned two-version control-plane upgrade. K3sConfig.k3sVersion now pins it.
- `labctl app k3s install --role x` mapped anything that was not "worker" to
  "infra", so the runbook's own `--role k8s-combo` would have installed a
  different role with different disk expectations and no warning.

Also: /var/lib/longhorn/replicas must never be deleted, only its contents, or
every replica placed on the node dies on a non-recursive mkdir -- for ever,
while the node reports Ready, Schedulable and healthy.

etcd is now in the kickstart's package set for k3s server roles, because
`k3s etcdctl` is not a command and a member that needs to recover its own
membership has no internet.

542 unit tests / 39 files.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Michal
2026-10-09 22:33:59 +01:00
parent c729275961
commit 84836cb8f2
60 changed files with 7457 additions and 96 deletions
Download Patch File Download Diff File Expand all files Collapse all files

176
migration/RECOVERY-CARD-schema-v2.md Normal file
View File

@@ -0,0 +1,176 @@
# Recovery card — v2 layout migration, one node at a time
Print or keep open. **A node is being wiped. If the cluster goes down there is
no Claude and possibly no internet.** Everything you need is on this page.
---
## The one thing that matters
**`k3s etcdctl` is NOT a command.** It does not exist (`No help topic for
'etcdctl'`). You need the real binary plus k3s's certs. Paste this once on
whichever server you are using:
```
e() {
local b=/var/lib/rancher/k3s/server/tls/etcd
ETCDCTL_API=3 etcdctl --cacert=$b/server-ca.crt \
--cert=$b/server-client.crt --key=$b/server-client.key \
--endpoints=https://127.0.0.1:2379 "$@"
}
```
(A function, not an alias — aliases do not expand inside scripts. Verified
working on worker0.)
If `etcdctl: command not found`, install it: `dnf install -y etcd` — which needs
working DNS and a reachable mirror, so do not count on it here. Nodes
reinstalled with the current kickstart already have it (`etcd` is in the package
set for k3s server roles precisely so this step is never needed).
Then:
```
e member list -w table
```
There are **3 etcd members. Quorum is 2. You have no margin.**
Two naming traps, both of which make a correct-looking command match nothing:
- **Kubernetes node names are FQDNs.** `worker0-k8s0.ad.itaz.eu`, never
`worker0`. The same name is used by `kubectl` and by `nodes.longhorn.io`.
Get it from `kubectl get nodes`; do not type it.
- **etcd member names add a random suffix** on top of that:
`worker0-k8s0.ad.itaz.eu-e26aacc5`. Match on the prefix.
One node down is survivable. Two is a dead API server and a dead cluster. So:
**one node at a time**, and never start the next while anything is degraded.
Reach the servers directly:
```
ssh root@192.168.8.23 # worker0 (Fedora x86)
ssh root@192.168.8.25 # worker2 (Fedora aarch64)
ssh root@192.168.8.13 # worker1 (Asahi aarch64 — migrate LAST)
```
---
## Before you touch anything
```
kubectl get nodes # all Ready
e member list # exactly 3, all started
e endpoint health # healthy
kubectl -n longhorn-system get volumes.longhorn.io | grep -v healthy # nothing
```
All four clean, or do not start.
---
## Order
**worker0 → worker2 → worker1.** worker1 is last: vendor-owned partition table,
4.6 G root, undersized rancher LV, and a hand-made bind mount a reinstall
destroys.
---
## The loop
```
# 1. Ask first. It refuses in SECONDS if it cannot finish.
labctl app k3s evacuate <node>
```
**If it refuses, it is right.** It means either no node can legally take a
replica, or the disk has no room. Do not force it. Add the N+1 node first.
```
# 2. Node is empty. Prove the data is fine BEFORE destroying anything.
kubectl -n longhorn-system get replicas.longhorn.io -o wide | grep <node> # none
# restart something using a Longhorn PVC; check it reads back.
# 3. Irreversible from here.
ssh <node> systemctl stop k3s
e member remove <id> # member names are "<node>-<8 hex>"
kubectl delete node <node>
e member list # 2, both healthy <-- CHECK THIS
kubectl get --raw /readyz # 'ok' from each survivor
# 4. Wipe.
labctl provision reprovision <node>
# 5. Rejoin AGAINST A SURVIVOR, pinned to the cluster's version.
labctl app k3s install <node> --role k8s-combo \
--k3s-server https://<survivor-ip>:6443 --k3s-token <token> \
--k3s-version <what the others run> # kubectl get nodes -o wide
e member list # EXACTLY 3 <-- CHECK THIS
# 6. Back in service, then wait for the rebuild.
labctl app k3s admit <node>
kubectl -n longhorn-system get volumes.longhorn.io # all healthy
```
---
## When something is wrong
| Symptom | Do this |
|---|---|
| Evacuate refused | Believe it. Read the named volume. Add capacity or a node. |
| Evacuate timed out | It already cancelled itself and restored scheduling. If it says it could not: `labctl app k3s admit <node>` |
| Aborted midway, node cordoned | `labctl app k3s admit <node>` |
| Rejoined node hosts no replicas | It has no Longhorn disk. `labctl app k3s admit <node>` |
| Replicas land but stay `stopped` (`desireState=running`, no `failedAt`) | `/var/lib/longhorn/replicas` is missing while `longhorn-disk.cfg` exists, so Longhorn thinks the disk is initialised and the replica process dies on `mkdir`. `mkdir -p /var/lib/longhorn/replicas`. **Never delete that directory** — delete its contents. |
| Rejoined node is a newer k3s than its peers | You forgot `--k3s-version`. More than one minor of skew between control-plane members is unsupported; reinstall it pinned. |
| `member list` shows 4 | `e member remove <dead-id>` — do this now |
| Node wiped, will not rejoin | Restore from the Longhorn backup target. |
| API server gone (2 members down) | You lost quorum. See below. |
### Lost quorum
Do not panic-restart things. From the one surviving server:
```
systemctl stop k3s
k3s server --cluster-reset
systemctl start k3s
```
This rebuilds a single-member cluster from that node's data. The other two
rejoin afterwards as fresh members; their
`/var/lib/rancher/k3s/server/db` must be removed before they will.
---
## worker1 only (Asahi)
After its reinstall, put this back in `/etc/fstab` or the node runs out of root
space the first time it pulls a large image:
```
/var/lib/longhorn/k3s-containerd /var/lib/rancher/k3s/agent/containerd none bind,x-systemd.requires-mounts-for=/var/lib/longhorn,x-systemd.requires-mounts-for=/var/lib/rancher 0 0
```
```
mkdir -p /var/lib/longhorn/k3s-containerd
systemctl daemon-reload && mount -a
```
---
## After all three
Re-apply by hand — `pulumi up` does NOT restore these:
- aitopatom Longhorn `allowScheduling: false`
- Prometheus 2-replica pin
- multus 512Mi memory limit
- every `local-path` PVC (model caches re-download; nothing else does)
Then check the sum: 3 x 300 GiB = 900 GiB real, against ~1550 GiB scheduled.
Right-size Prometheus (376 GiB provisioned, ~8 GiB actual) or set
`storageOverProvisioningPercentage` deliberately.

277
migration/ROLLING-MIGRATION-RUNBOOK.md Normal file
View File

@@ -0,0 +1,277 @@
# Rolling migration to the v2 disk layout
Moving `worker0`, `worker1` and `worker2` from the v1 layout to v2, one at a
time, without losing data or quorum.
**Status: NOT YET EXECUTED against the fleet.** The mechanism is proven by
`bastion/tests/integration/rolling-drill.test.ts`, which passed **10/10** on
four VMs on 2026-09-30 — including a backup to S3 and a byte-identical restore.
Evidence: `drill-evidence/schema-v2-2026-09-30.txt`.
---
## The one thing to understand first
**You cannot start by evacuating an existing node.** Two independent reasons,
both read off the live cluster:
1. **Placement.** `replica-soft-anti-affinity` is `false`, so a volume puts at
most one replica per node. There are exactly **3** Longhorn-schedulable nodes
(`worker0`, `worker1`, `worker2` — `aitopatom` and `spark` have
`allowScheduling: false` deliberately). **37 volumes want 3 replicas.** Evict
any one node and the other two already hold a copy each, so the third has
nowhere legal to go. Eviction can never reach zero.
2. **Capacity**, which blocks it even if placement did not:
| node | Longhorn disk | used | free |
|---|---|---|---|
| worker0 | 727.6 GiB | 91.5 | ~636 |
| worker1 | 664.7 GiB | 114.3 | ~550 |
| **worker2** | **99.9 GiB** | 51.1 | **~49** |
worker2 cannot absorb worker0's 91.5 GiB. No setting changes that.
So: **bring the new box up first (N+1), then evacuate an old one onto it.**
`labctl app k3s evacuate` now refuses in seconds with the offending volume
named, rather than discovering this an hour in. If it refuses, it is right.
---
## Capacity arithmetic, stated plainly
- Today: 727.6 + 664.7 + 99.9 = **~1.49 TiB** of Longhorn disk.
- After v2 (fixed 300 GiB per node): 3 x 300 = **900 GiB** — a **reduction of
~590 GiB**.
- Real usage is **~257 GiB**, so it fits with ~640 GiB headroom.
- But **1550 GiB is already *scheduled*** (794 + 558 + 198), which only works
because `storageOverProvisioningPercentage` is **500**. Against 900 GiB real
that needs >=172% and leaves no room for the next volume.
**Action:** before migrating the last node, right-size the oversized PVCs
(Prometheus is 376 GiB provisioned against ~8 GiB actual) or set the
over-provisioning percentage deliberately. Do not discover this when a disk
fills.
---
## Preamble — do these once, before touching any node
1. **Backup target.** Configured in `deployments/longhorn/` and
`Pulumi.homelab.yaml` (SeaweedFS S3 at `10.0.0.194:8333` — a different
machine from the cluster, which is what makes it a backup rather than another
copy of the same failure domain). TrueNAS was rejected on evidence: port 2049
is CLOSED, so NFS would need the export creating first.
**`defaultSettings.backupTarget` DOES NOT WORK on Longhorn 1.8.** The target
moved onto the `backuptargets.longhorn.io/default` CR
(`spec.backupTargetURL`, `spec.credentialSecret`); the helm value is silently
ignored, no `backup-target` Setting object is created, and the chart reports
success with no backup target at all. The Pulumi code therefore patches the
CR via `apiextensions.CustomResourcePatch`, and the RecurringJobs depend on
that patch so the first scheduled backup cannot fire into nothing.
Before relying on it:
- create the `longhorn-backup` bucket;
- apply with `--target`, **not** a bare `pulumi up` — the stack carries
unrelated drift (at one point 12 resources, including a replacement of the
openbao watchdog ConfigMap). Target these four:
```
kubernetes:core/v1:Secret::longhorn-backup-credentials
kubernetes:helm.sh/v3:Release::longhorn
...Release$...:BackupTarget::longhorn-backup-target
...Release$...:RecurringJob::longhorn-job-daily-backup
...Release$...:RecurringJob::longhorn-job-hourly-snapshot
```
(`scripts/pulumi.sh` refuses an unclean tree; commit and merge rather than
setting `PULUMI_ALLOW_UNMERGED=1`.)
- confirm Longhorn accepted it — `available: true` is the appliance saying it
could reach the endpoint AND authenticate:
```
kubectl -n longhorn-system get backuptargets.longhorn.io default \
-o custom-columns=URL:.spec.backupTargetURL,AVAIL:.status.available
```
- take one backup, then **restore it to a new volume and check the contents.**
An unverified backup is not a backup. The restore is the gate — not the
`pulumi up`, and not `available: true`.
Note: this cluster has **no CSI VolumeSnapshot API** (no
`snapshot.storage.k8s.io` resources at all), so backup tooling must use
Longhorn's native CRs — `snapshots.longhorn.io`, `backups.longhorn.io`, and a
`fromBackup` StorageClass for restores. The drill does exactly this.
2. **The 21 single-replica volumes.** One copy each, and until step 1 is done, no
other copy anywhere. For each one on the node you are about to evacuate:
```
kubectl -n longhorn-system patch volumes.longhorn.io <pv> \
--type=merge -p '{"spec":{"numberOfReplicas":3}}'
```
wait for `status.robustness: healthy`, *then* evacuate. Restore the count
afterwards if you want it back.
`evacuate` refuses single-replica volumes by default.
`--allow-single-replica` overrides it and means "I accept that a failed
rebuild is permanent loss".
Detached volumes are listed separately by the pre-check: Longhorn must attach
them to rebuild, so they need attention before, not during.
3. **Pin the k3s version, or the migration upgrades the cluster by accident.**
`get.k3s.io` installs **latest**. The fleet runs **v1.34.6+k3s1**; latest at
the time of writing is **v1.36.4+k3s1** — two minor versions ahead. A node
reinstalled by this procedure rejoins on a newer k3s than the two members it
is joining, and Kubernetes supports only **one** minor of skew between
control-plane members. Roll three nodes and you have performed an unplanned,
unstaged, two-version upgrade one node at a time.
Pass `--k3s-version v1.34.6+k3s1` (check the real value with
`kubectl get nodes -o wide`). Upgrade deliberately, afterwards, as its own
change.
4. **Add the N+1 node.** Provision the new box with `schemaVersion: 2`, join it,
and confirm it has a schedulable Longhorn disk:
```
kubectl -n longhorn-system get nodes.longhorn.io
```
A node with no disk hosts no replicas while looking perfectly healthy. If it
has none, `labctl app k3s admit <node>` repairs it.
---
## Node order
1. **worker0** — plain Fedora x86, the most ordinary box. Go first.
2. **worker2** — Fedora aarch64. Its 100 GiB Longhorn disk becomes 300 GiB
under v2, so this node *gains* room.
3. **worker1** — **LAST, and a different animal.** Apple Silicon under Asahi:
the partition table belongs to the vendor installer, root is 4.6 GiB, the
rancher LV is undersized at 20 GiB, and it carries a containerd bind mount
that exists in no template or script:
```
/var/lib/longhorn/k3s-containerd /var/lib/rancher/k3s/agent/containerd none \
bind,x-systemd.requires-mounts-for=/var/lib/longhorn,x-systemd.requires-mounts-for=/var/lib/rancher 0 0
```
A reinstall destroys that silently. Re-create it, or the node runs out of
root space the first time it pulls a large image.
Quorum: there are **exactly 3 etcd members**, so quorum is 2 and the fleet
tolerates **exactly one member down with no margin**. One node at a time, never
two, and never start a roll while anything is already degraded.
---
## Per-node loop
Gate at every step. If one fails, stop — do not roll the next node on top of a
half-migrated one.
```
# 0. GATE: cluster is whole before you start
kubectl get nodes
kubectl -n longhorn-system get volumes.longhorn.io | grep -v healthy # expect none
# `k3s etcdctl` is NOT a command. Define this first (see the recovery card):
# e() { local b=/var/lib/rancher/k3s/server/tls/etcd
# ETCDCTL_API=3 etcdctl --cacert=$b/server-ca.crt \
# --cert=$b/server-client.crt --key=$b/server-client.key \
# --endpoints=https://127.0.0.1:2379 "$@"; }
e member list # expect 3, all started
e endpoint health
# 1. Will it even work?
labctl app k3s evacuate <node> # refuses fast if it cannot complete
# 2. (it evacuates: cordon -> check -> evict+wait -> drain -> verify)
kubectl -n longhorn-system get replicas.longhorn.io -o wide | grep <node> # expect none
# 3. Prove the data is fine BEFORE destroying anything
# Restart something that uses a Longhorn PVC and check it reads back.
# 4. The irreversible part — as late as possible
ssh <node> systemctl stop k3s
e member remove <id-of-node> # member names are "<node>-<8 hex>", not "<node>"
kubectl delete node <node>
e member list # expect 2, both healthy
kubectl get --raw /readyz # from each survivor: ok
# 5. Wipe and reinstall
labctl provision reprovision <node> # the DISK decides the schema, not you
# 6. Rejoin — against a SURVIVOR, not against itself, PINNED
labctl app k3s install <node> --role k8s-combo \
--k3s-server https://<survivor-ip>:6443 --k3s-token <token> \
--k3s-version v1.34.6+k3s1 # MUST match the cluster; see preamble step 3
e member list # expect EXACTLY 3
# 7. Back into service
labctl app k3s admit <node>
# 8. Wait for the rebuild, then verify the data again
kubectl -n longhorn-system get volumes.longhorn.io # all healthy, 3 replicas
# And check the node actually took replicas BACK — running, not merely assigned:
kubectl -n longhorn-system get replicas.longhorn.io -o wide | grep <node>
```
**Step 6's "exactly 3" is not a formality.** A reinstalled server rejoins as a
*new* etcd member. If the old record survives you get 4 members, quorum rises
from 2-of-3 to 3-of-4, and a cluster that tolerated one failure now tolerates
none — while every dashboard still says green. That is why step 4 removes the
member explicitly instead of trusting the rejoin to clean up.
**Step 8's "running, not merely assigned" is also not a formality.** A replica
can sit at `currentState=stopped` / `desireState=running` with no `failedAt`
indefinitely. The node looks Ready and Schedulable and hosts nothing, and the
*next* node's eviction then has nowhere to drain and stalls for its full hour.
---
## What `pulumi up` puts back, and what it does not
**Restored automatically.** Node labels and taints — both `NodePatch` sites use
`pulumi.com/patchForce`, so they genuinely self-heal.
**NOT restored — re-apply by hand:**
- aitopatom's Longhorn `allowScheduling: false`
- the Prometheus 2-replica pin
- the multus 512Mi memory limit
- worker1's containerd bind mount (above)
- every `local-path` PVC. Model caches re-download; the rest do not come back.
---
## If it goes wrong
- **Evacuation refused.** Believe it. Read the named volume; it is telling you
either "no node can take this replica" or "the disk does not have room". Add
capacity rather than forcing it.
- **Evacuation timed out.** The node is left schedulable again and the eviction
cancelled automatically; the error names which volumes were stuck and in what
state. If that cleanup itself failed, the error says so — run
`labctl app k3s admit <node>`.
- **Aborted midway, node still cordoned.** `labctl app k3s admit <node>`
uncordons, clears the eviction and restores the Longhorn disk.
- **Replicas land on a node but stay `stopped`.** `/var/lib/longhorn/replicas`
is missing while `longhorn-disk.cfg` exists, so Longhorn considers the disk
already initialised and never recreates the directory; the replica process
then fails on a non-recursive `mkdir` every time, permanently.
`mkdir -p /var/lib/longhorn/replicas`. **Never delete that directory** —
delete its contents. `labctl app k3s admit <node>` now ensures it exists.
- **Rejoined node hosts no replicas.** It has no Longhorn disk. `admit` repairs
it; `kubectl -n longhorn-system get nodes.longhorn.io <node> -o yaml` confirms.
- **Node wiped and cannot rejoin.** This is the case the backup target exists
for. Restore the volumes from backup. If step 1 of the preamble was skipped,
there is no recovery path — which is precisely why it is step 1.
---
## Known gaps
- Every node reports `RequiredPackages=False — Missing packages: [nfs-utils
cryptsetup]` and `KernelModulesLoaded=False`. Uniform across the fleet and not
blocking today, but it would block an NFS backup target and encrypted volumes.
The S3 target does not need `nfs-utils`.
- The drill ran on VMs with ~3 GiB of test data; production has ~257 GiB. It
proves **mechanism**, not timing. Keep the 60-minute eviction timeout generous.
- `worker1` has never been VM-tested and cannot be — it needs Apple Silicon.

123
migration/drill-evidence/schema-v2-2026-09-30.txt Normal file
View File

@@ -0,0 +1,123 @@
Rolling migration drill — schema v1 -> v2, WITH backup/restore and version pin
run finished: 2026-09-30T02:26:27+01:00 host: thebeast
k3s: v1.34.6+k3s1 (PINNED and asserted on every node) longhorn: 1.8.1
topology: 3 x k8s-combo (server+etcd+storage) + 1 x k8s-worker (storage spare, N+1)
backup target: s3://longhorn-drill-backup@us-east-1/ via SeaweedFS at 10.0.0.194:8333
(separate bucket from production's longhorn-backup)
=== checkpoints ===
[drill] ==> provision
[drill] <== provision (943s)
[drill] ==> form cluster
[00:22:37] [snapshot] after cluster-init: 1/1 Ready, etcd members=1 healthy=true
[00:24:42] [snapshot] after joining drill-s2: 2/2 Ready, etcd members=2 healthy=true
[00:25:54] [snapshot] after joining drill-s3: 3/3 Ready, etcd members=3 healthy=true
[00:26:55] [snapshot] formed: 4/4 Ready, etcd members=3 healthy=true
[drill] all 4 nodes on pinned v1.34.6+k3s1
[drill] <== form cluster (348s)
[drill] ==> install longhorn
[00:26:56] [longhorn] creating longhorn-system with privileged PSA labels
[00:26:56] [longhorn] configuring backup target s3://longhorn-drill-backup@us-east-1/
[00:26:56] [longhorn] installing chart 1.8.1 via k3s helm controller
[00:28:11] [longhorn] ready after 0s — 4/4 with schedulable disk: drill-s1.pxe-test.local,drill-s2.pxe-test.local,drill-s3.pxe-test.local,drill-spare.pxe-test.local
[drill] <== install longhorn (75s)
[drill] ==> seed data
[00:28:11] [longhorn] creating drill-r3 (2Gi, 3 replicas)
[00:28:11] [longhorn] creating drill-r1 (1Gi, 1 replicas)
[00:29:01] [longhorn] writing 512 MiB to drill-r3
[00:29:04] [longhorn] drill-r3 digest b8171779defe286b...
[00:29:05] [longhorn] writing 128 MiB to drill-r1
[00:29:06] [longhorn] drill-r1 digest 89213fddc2738c8d...
[00:29:40] [longhorn] drill-r3 verified intact across a detach/attach (pod drill-r3-754b4fbc7d-6ntwp -> drill-r3-754b4fbc7d-xrgds)
[00:30:13] [longhorn] drill-r1 verified intact across a detach/attach (pod drill-r1-949b946f8-4vbr6 -> drill-r1-949b946f8-pt6kv)
[drill] <== seed data (123s)
[drill] ==> backup and restore
[00:30:14] [longhorn] backup target set to s3://longhorn-drill-backup@us-east-1/
[00:30:15] [longhorn] backup target available after 0s
[00:30:16] [longhorn] snapshotting drill-r3 (pvc-4eeea364-2ed3-41fa-abc4-0176fb13ba3e)
[00:30:21] [longhorn] backing up drill-r3 to the backup target
[00:30:43] [longhorn] backup drill-bak-mundcj51 completed after 21s
[00:30:44] [longhorn] restoring drill-bak-mundcj51 into drill-restored
[drill] restored digest b8171779defe286b... matches original
[drill] <== backup and restore (108s)
[drill] ==> negative pre-check
[drill] <== negative pre-check (34s)
[drill] ==> roll drill-s1
[00:32:37] [snapshot] before rolling drill-s1: 4/4 Ready, etcd members=3 healthy=true
[drill-s1] All replicas evicted from drill-s1.pxe-test.local after 32s
[drill-s1] Drained drill-s1.pxe-test.local
[00:33:20] [longhorn] drill-r3 healthy on 3 nodes after 1s
[00:34:16] [longhorn] drill-r3 verified intact across a detach/attach (pod drill-r3-754b4fbc7d-xrgds -> drill-r3-754b4fbc7d-mhlj7)
[00:34:49] [longhorn] drill-r1 verified intact across a detach/attach (pod drill-r1-949b946f8-pt6kv -> drill-r1-949b946f8-6kqct)
[00:34:50] [etcd] removed member drill-s1.pxe-test.local-e58f4aaa (9575fbd07a6c3ca1)
[00:35:37] [snapshot] drill-s1 removed: 3/3 Ready, etcd members=2 healthy=true
[00:46:06] [snapshot] drill-s1 rejoined: 4/4 Ready, etcd members=3 healthy=true
[00:46:37] [longhorn] drill-r3 healthy on 3 nodes after 1s
[00:47:53] [longhorn] drill-r3 verified intact across a detach/attach (pod drill-r3-754b4fbc7d-mhlj7 -> drill-r3-754b4fbc7d-qhn2c)
[00:48:58] [longhorn] drill-r1 verified intact across a detach/attach (pod drill-r1-949b946f8-6kqct -> drill-r1-949b946f8-zj9lv)
[drill] after rolling drill-s1: 2 RUNNING replica(s) on the node
[drill] <== roll drill-s1 (982s)
[drill] ==> roll drill-s2
[00:48:59] [snapshot] before rolling drill-s2: 4/4 Ready, etcd members=3 healthy=true
[drill-s2] All replicas evicted from drill-s2.pxe-test.local after 47s
[drill-s2] Drained drill-s2.pxe-test.local
[00:50:31] [longhorn] drill-r3 healthy on 3 nodes after 1s
[00:51:41] [longhorn] drill-r3 verified intact across a detach/attach (pod drill-r3-754b4fbc7d-qhn2c -> drill-r3-754b4fbc7d-9bfqw)
[00:52:15] [longhorn] drill-r1 verified intact across a detach/attach (pod drill-r1-949b946f8-zj9lv -> drill-r1-949b946f8-7k9pn)
[00:52:16] [etcd] removed member drill-s2.pxe-test.local-943a6633 (2dfcaab9db591b03)
[00:53:02] [snapshot] drill-s2 removed: 3/3 Ready, etcd members=2 healthy=true
[01:04:02] [snapshot] drill-s2 rejoined: 4/4 Ready, etcd members=3 healthy=true
[01:04:32] [longhorn] drill-r3 healthy on 3 nodes after 1s
[01:05:38] [longhorn] drill-r3 verified intact across a detach/attach (pod drill-r3-754b4fbc7d-9bfqw -> drill-r3-754b4fbc7d-tf7z2)
[01:06:46] [longhorn] drill-r1 verified intact across a detach/attach (pod drill-r1-949b946f8-7k9pn -> drill-r1-949b946f8-cx7rx)
[drill] after rolling drill-s2: 2 RUNNING replica(s) on the node
[drill] <== roll drill-s2 (1069s)
[drill] ==> roll drill-s3
[01:06:48] [snapshot] before rolling drill-s3: 4/4 Ready, etcd members=3 healthy=true
[drill-s3] All replicas evicted from drill-s3.pxe-test.local after 31s
[drill-s3] Drained drill-s3.pxe-test.local
[01:07:59] [longhorn] drill-r3 healthy on 3 nodes after 1s
[01:09:14] [longhorn] drill-r3 verified intact across a detach/attach (pod drill-r3-754b4fbc7d-tf7z2 -> drill-r3-754b4fbc7d-qv54j)
[01:10:11] [longhorn] drill-r1 verified intact across a detach/attach (pod drill-r1-949b946f8-cx7rx -> drill-r1-949b946f8-sm8z7)
[01:10:13] [etcd] removed member drill-s3.pxe-test.local-9bf0ac84 (d47166c003b5115c)
[01:11:01] [snapshot] drill-s3 removed: 3/3 Ready, etcd members=2 healthy=true
[01:22:13] [snapshot] drill-s3 rejoined: 4/4 Ready, etcd members=3 healthy=true
[01:22:44] [longhorn] drill-r3 healthy on 3 nodes after 1s
[01:23:50] [longhorn] drill-r3 verified intact across a detach/attach (pod drill-r3-754b4fbc7d-qv54j -> drill-r3-754b4fbc7d-wbz8c)
[01:24:50] [longhorn] drill-r1 verified intact across a detach/attach (pod drill-r1-949b946f8-sm8z7 -> drill-r1-949b946f8-s5j48)
[drill] after rolling drill-s3: 2 RUNNING replica(s) on the node
[drill] <== roll drill-s3 (1084s)
[drill] ==> final verification
[01:24:51] [snapshot] end of drill: 4/4 Ready, etcd members=3 healthy=true
[01:25:26] [longhorn] drill-r3 verified intact across a detach/attach (pod drill-r3-754b4fbc7d-wbz8c -> drill-r3-754b4fbc7d-nj5jp)
[01:25:59] [longhorn] drill-r1 verified intact across a detach/attach (pod drill-r1-949b946f8-s5j48 -> drill-r1-949b946f8-xfbcq)
[drill] <== final verification (69s)
[01:26:00] [snapshot] FINAL: 4/4 Ready, etcd members=3 healthy=true
[drill] digests: drill-r3=b8171779defe286b drill-r1=89213fddc2738c8d
=== result ===
✓ provisions four v2 nodes 943264ms
✓ forms one cluster with three etcd members plus a storage agent 348143ms
✓ installs Longhorn with a schedulable disk on every node 75260ms
✓ writes known data and proves it survives a detach/attach 122827ms
✓ backs up a volume to S3 and restores it byte-identical 108172ms
✓ REFUSES to evacuate when there is nowhere for replicas to go 33846ms
✓ rolls drill-s1: evacuate, wipe, reinstall, rejoin, verify 982060ms
✓ rolls drill-s2: evacuate, wipe, reinstall, rejoin, verify 1068887ms
✓ rolls drill-s3: evacuate, wipe, reinstall, rejoin, verify 1083812ms
✓ ends with a whole cluster and the original bytes 68503ms
Test Files 1 passed | 10 skipped (11)
Tests 10 passed | 104 skipped (114)
VERDICT: PASSED (10/10).
- All four nodes ran the PINNED k3s v1.34.6+k3s1, asserted from kubeletVersion.
- Backup target reachable and authenticated; 512 MiB backed up to S3 and RESTORED
into a separate volume byte-identical to the original digest.
- Three storage+etcd nodes rolled through evacuate/wipe/reinstall/rejoin one at a
time; checksums identical before and after every roll; etcd held exactly 3 healthy
members at every gate, 2 only while a node was deliberately absent, never 4.
- Each reinstalled node came back hosting RUNNING replicas.
NOTE: defaultSettings.backupTarget is IGNORED by Longhorn 1.8 — the target lives on
the backuptargets.longhorn.io/default CR. Setting it via helm values reports success
and produces no backup target. Both the Pulumi code and this drill patch the CR.