Rolling migration drill — schema v1 -> v2, WITH backup/restore and version pin run finished: 2026-09-30T02:26:27+01:00 host: thebeast k3s: v1.34.6+k3s1 (PINNED and asserted on every node) longhorn: 1.8.1 topology: 3 x k8s-combo (server+etcd+storage) + 1 x k8s-worker (storage spare, N+1) backup target: s3://longhorn-drill-backup@us-east-1/ via SeaweedFS at 10.0.0.194:8333 (separate bucket from production's longhorn-backup) === checkpoints === [drill] ==> provision [drill] <== provision (943s) [drill] ==> form cluster [00:22:37] [snapshot] after cluster-init: 1/1 Ready, etcd members=1 healthy=true [00:24:42] [snapshot] after joining drill-s2: 2/2 Ready, etcd members=2 healthy=true [00:25:54] [snapshot] after joining drill-s3: 3/3 Ready, etcd members=3 healthy=true [00:26:55] [snapshot] formed: 4/4 Ready, etcd members=3 healthy=true [drill] all 4 nodes on pinned v1.34.6+k3s1 [drill] <== form cluster (348s) [drill] ==> install longhorn [00:26:56] [longhorn] creating longhorn-system with privileged PSA labels [00:26:56] [longhorn] configuring backup target s3://longhorn-drill-backup@us-east-1/ [00:26:56] [longhorn] installing chart 1.8.1 via k3s helm controller [00:28:11] [longhorn] ready after 0s — 4/4 with schedulable disk: drill-s1.pxe-test.local,drill-s2.pxe-test.local,drill-s3.pxe-test.local,drill-spare.pxe-test.local [drill] <== install longhorn (75s) [drill] ==> seed data [00:28:11] [longhorn] creating drill-r3 (2Gi, 3 replicas) [00:28:11] [longhorn] creating drill-r1 (1Gi, 1 replicas) [00:29:01] [longhorn] writing 512 MiB to drill-r3 [00:29:04] [longhorn] drill-r3 digest b8171779defe286b... [00:29:05] [longhorn] writing 128 MiB to drill-r1 [00:29:06] [longhorn] drill-r1 digest 89213fddc2738c8d... [00:29:40] [longhorn] drill-r3 verified intact across a detach/attach (pod drill-r3-754b4fbc7d-6ntwp -> drill-r3-754b4fbc7d-xrgds) [00:30:13] [longhorn] drill-r1 verified intact across a detach/attach (pod drill-r1-949b946f8-4vbr6 -> drill-r1-949b946f8-pt6kv) [drill] <== seed data (123s) [drill] ==> backup and restore [00:30:14] [longhorn] backup target set to s3://longhorn-drill-backup@us-east-1/ [00:30:15] [longhorn] backup target available after 0s [00:30:16] [longhorn] snapshotting drill-r3 (pvc-4eeea364-2ed3-41fa-abc4-0176fb13ba3e) [00:30:21] [longhorn] backing up drill-r3 to the backup target [00:30:43] [longhorn] backup drill-bak-mundcj51 completed after 21s [00:30:44] [longhorn] restoring drill-bak-mundcj51 into drill-restored [drill] restored digest b8171779defe286b... matches original [drill] <== backup and restore (108s) [drill] ==> negative pre-check [drill] <== negative pre-check (34s) [drill] ==> roll drill-s1 [00:32:37] [snapshot] before rolling drill-s1: 4/4 Ready, etcd members=3 healthy=true [drill-s1] All replicas evicted from drill-s1.pxe-test.local after 32s [drill-s1] Drained drill-s1.pxe-test.local [00:33:20] [longhorn] drill-r3 healthy on 3 nodes after 1s [00:34:16] [longhorn] drill-r3 verified intact across a detach/attach (pod drill-r3-754b4fbc7d-xrgds -> drill-r3-754b4fbc7d-mhlj7) [00:34:49] [longhorn] drill-r1 verified intact across a detach/attach (pod drill-r1-949b946f8-pt6kv -> drill-r1-949b946f8-6kqct) [00:34:50] [etcd] removed member drill-s1.pxe-test.local-e58f4aaa (9575fbd07a6c3ca1) [00:35:37] [snapshot] drill-s1 removed: 3/3 Ready, etcd members=2 healthy=true [00:46:06] [snapshot] drill-s1 rejoined: 4/4 Ready, etcd members=3 healthy=true [00:46:37] [longhorn] drill-r3 healthy on 3 nodes after 1s [00:47:53] [longhorn] drill-r3 verified intact across a detach/attach (pod drill-r3-754b4fbc7d-mhlj7 -> drill-r3-754b4fbc7d-qhn2c) [00:48:58] [longhorn] drill-r1 verified intact across a detach/attach (pod drill-r1-949b946f8-6kqct -> drill-r1-949b946f8-zj9lv) [drill] after rolling drill-s1: 2 RUNNING replica(s) on the node [drill] <== roll drill-s1 (982s) [drill] ==> roll drill-s2 [00:48:59] [snapshot] before rolling drill-s2: 4/4 Ready, etcd members=3 healthy=true [drill-s2] All replicas evicted from drill-s2.pxe-test.local after 47s [drill-s2] Drained drill-s2.pxe-test.local [00:50:31] [longhorn] drill-r3 healthy on 3 nodes after 1s [00:51:41] [longhorn] drill-r3 verified intact across a detach/attach (pod drill-r3-754b4fbc7d-qhn2c -> drill-r3-754b4fbc7d-9bfqw) [00:52:15] [longhorn] drill-r1 verified intact across a detach/attach (pod drill-r1-949b946f8-zj9lv -> drill-r1-949b946f8-7k9pn) [00:52:16] [etcd] removed member drill-s2.pxe-test.local-943a6633 (2dfcaab9db591b03) [00:53:02] [snapshot] drill-s2 removed: 3/3 Ready, etcd members=2 healthy=true [01:04:02] [snapshot] drill-s2 rejoined: 4/4 Ready, etcd members=3 healthy=true [01:04:32] [longhorn] drill-r3 healthy on 3 nodes after 1s [01:05:38] [longhorn] drill-r3 verified intact across a detach/attach (pod drill-r3-754b4fbc7d-9bfqw -> drill-r3-754b4fbc7d-tf7z2) [01:06:46] [longhorn] drill-r1 verified intact across a detach/attach (pod drill-r1-949b946f8-7k9pn -> drill-r1-949b946f8-cx7rx) [drill] after rolling drill-s2: 2 RUNNING replica(s) on the node [drill] <== roll drill-s2 (1069s) [drill] ==> roll drill-s3 [01:06:48] [snapshot] before rolling drill-s3: 4/4 Ready, etcd members=3 healthy=true [drill-s3] All replicas evicted from drill-s3.pxe-test.local after 31s [drill-s3] Drained drill-s3.pxe-test.local [01:07:59] [longhorn] drill-r3 healthy on 3 nodes after 1s [01:09:14] [longhorn] drill-r3 verified intact across a detach/attach (pod drill-r3-754b4fbc7d-tf7z2 -> drill-r3-754b4fbc7d-qv54j) [01:10:11] [longhorn] drill-r1 verified intact across a detach/attach (pod drill-r1-949b946f8-cx7rx -> drill-r1-949b946f8-sm8z7) [01:10:13] [etcd] removed member drill-s3.pxe-test.local-9bf0ac84 (d47166c003b5115c) [01:11:01] [snapshot] drill-s3 removed: 3/3 Ready, etcd members=2 healthy=true [01:22:13] [snapshot] drill-s3 rejoined: 4/4 Ready, etcd members=3 healthy=true [01:22:44] [longhorn] drill-r3 healthy on 3 nodes after 1s [01:23:50] [longhorn] drill-r3 verified intact across a detach/attach (pod drill-r3-754b4fbc7d-qv54j -> drill-r3-754b4fbc7d-wbz8c) [01:24:50] [longhorn] drill-r1 verified intact across a detach/attach (pod drill-r1-949b946f8-sm8z7 -> drill-r1-949b946f8-s5j48) [drill] after rolling drill-s3: 2 RUNNING replica(s) on the node [drill] <== roll drill-s3 (1084s) [drill] ==> final verification [01:24:51] [snapshot] end of drill: 4/4 Ready, etcd members=3 healthy=true [01:25:26] [longhorn] drill-r3 verified intact across a detach/attach (pod drill-r3-754b4fbc7d-wbz8c -> drill-r3-754b4fbc7d-nj5jp) [01:25:59] [longhorn] drill-r1 verified intact across a detach/attach (pod drill-r1-949b946f8-s5j48 -> drill-r1-949b946f8-xfbcq) [drill] <== final verification (69s) [01:26:00] [snapshot] FINAL: 4/4 Ready, etcd members=3 healthy=true [drill] digests: drill-r3=b8171779defe286b drill-r1=89213fddc2738c8d === result === ✓ provisions four v2 nodes 943264ms ✓ forms one cluster with three etcd members plus a storage agent 348143ms ✓ installs Longhorn with a schedulable disk on every node 75260ms ✓ writes known data and proves it survives a detach/attach 122827ms ✓ backs up a volume to S3 and restores it byte-identical 108172ms ✓ REFUSES to evacuate when there is nowhere for replicas to go 33846ms ✓ rolls drill-s1: evacuate, wipe, reinstall, rejoin, verify 982060ms ✓ rolls drill-s2: evacuate, wipe, reinstall, rejoin, verify 1068887ms ✓ rolls drill-s3: evacuate, wipe, reinstall, rejoin, verify 1083812ms ✓ ends with a whole cluster and the original bytes 68503ms Test Files 1 passed | 10 skipped (11) Tests 10 passed | 104 skipped (114) VERDICT: PASSED (10/10). - All four nodes ran the PINNED k3s v1.34.6+k3s1, asserted from kubeletVersion. - Backup target reachable and authenticated; 512 MiB backed up to S3 and RESTORED into a separate volume byte-identical to the original digest. - Three storage+etcd nodes rolled through evacuate/wipe/reinstall/rejoin one at a time; checksums identical before and after every roll; etcd held exactly 3 healthy members at every gate, 2 only while a node was deliberately absent, never 4. - Each reinstalled node came back hosting RUNNING replicas. NOTE: defaultSettings.backupTarget is IGNORED by Longhorn 1.8 — the target lives on the backuptargets.longhorn.io/default CR. Setting it via helm values reports success and produces no backup target. Both the Pulumi code and this drill patch the CR.