michal/lab
1
0
Fork 0
Code Issues Pull Requests 3 Actions Packages Projects Releases Wiki Activity
Files
5d5fc5cb7c9855e14c8542d494a89dfaed1f100c
lab/migration/drill-evidence/schema-v2-2026-09-30.txt

124 lines
8.2 KiB
Plaintext
Raw Normal View History

v2 partition schema: capability model, fixed sizing, and a proven rolling migration The fleet's disk layout was decided by literal role-name comparisons at 17 sites across 8 files. That is why `labcontroller` got no rancher LV (it matched neither "infra" nor "worker"), why labmaster carried an empty 728 GiB Longhorn LV, and why a 62 GiB container image tipped that node into DiskPressure and evicted bastion, labd, cockroach and openbao. Three generators each hardcoded their own sizes and disagreed: the kickstart said rancher=120G, ubuntu-autoinstall said 20G, asahi-firstboot said 20G under a comment asserting the sizes matched install.ks.ts. They did not. ROLE_REGISTRY already carried `k3s` and `parent` and nothing read them. This makes capabilities authoritative, moves every size into one shared module, and adds schema v2 (fixed 250G rancher / 300G longhorn, never --grow). Existing machines stay v1 forever: schema is detected from the on-disk VG tag, so a v1 box stays v1 even when v2 is requested. Proven on real VMs, not just unit tests: all four topologies installed and had their layouts read back off the machines; markers in /home, /srv, /var/lib/longhorn and /var/lib/rancher survived reinstall; and a 4-node cluster formed, took real data, and had all three servers rolled one at a time through evacuate / wipe / reinstall / rejoin with checksums verified before and after each roll. Drill: 10/10, evidence in migration/drill-evidence/. Production defects the drill found, none of which a unit test could reach: - sshOpts() capped every operation at 30s while drainNode asked kubectl for --timeout=15m, so the transport killed the drain after evicting most pods and returned a transcript that read like success. `labctl app k3s evacuate` could never have drained a real node. - generateAgentConfig put node-role.kubernetes.io/worker in the kubelet's node-label list, which NodeRestriction rejects, so no agent installed through K3sModule could start. systemd reported only "did not take the steps required by its unit configuration". - toOpContext built K3sConfig by enumerating three keys and silently dropped the rest, so the k3s version pin did nothing and --allow-single-replica was a no-op that claimed to override a data-loss guard. - The Longhorn disk annotation was mangled by shell quoting; with createDefaultDiskLabeledNodes the node then gets no disk at all, joins, goes Ready, and silently hosts no replica. - evacuate drained before evicting, which blocks on Longhorn's PDBs and would destroy a single-replica volume's only source. - etcd member ids were parsed from JSON, losing uint64 precision and emitting decimal where `member remove` needs hex -- at the moment a node is stopped and about to be wiped. - get.k3s.io installs latest, two minor versions ahead of the fleet, so a reinstalled node would rejoin on a newer k3s than its peers and roll an unplanned two-version control-plane upgrade. K3sConfig.k3sVersion now pins it. - `labctl app k3s install --role x` mapped anything that was not "worker" to "infra", so the runbook's own `--role k8s-combo` would have installed a different role with different disk expectations and no warning. Also: /var/lib/longhorn/replicas must never be deleted, only its contents, or every replica placed on the node dies on a non-recursive mkdir -- for ever, while the node reports Ready, Schedulable and healthy. etcd is now in the kickstart's package set for k3s server roles, because `k3s etcdctl` is not a command and a member that needs to recover its own membership has no internet. 542 unit tests / 39 files. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-10-09 22:33:59 +01:00
Rolling migration drill — schema v1 -> v2, WITH backup/restore and version pin
run finished: 2026-09-30T02:26:27+01:00 host: thebeast
k3s: v1.34.6+k3s1 (PINNED and asserted on every node) longhorn: 1.8.1
topology: 3 x k8s-combo (server+etcd+storage) + 1 x k8s-worker (storage spare, N+1)
backup target: s3://longhorn-drill-backup@us-east-1/ via SeaweedFS at 10.0.0.194:8333
(separate bucket from production's longhorn-backup)
=== checkpoints ===
[drill] ==> provision
[drill] <== provision (943s)
[drill] ==> form cluster
[00:22:37] [snapshot] after cluster-init: 1/1 Ready, etcd members=1 healthy=true
[00:24:42] [snapshot] after joining drill-s2: 2/2 Ready, etcd members=2 healthy=true
[00:25:54] [snapshot] after joining drill-s3: 3/3 Ready, etcd members=3 healthy=true
[00:26:55] [snapshot] formed: 4/4 Ready, etcd members=3 healthy=true
[drill] all 4 nodes on pinned v1.34.6+k3s1
[drill] <== form cluster (348s)
[drill] ==> install longhorn
[00:26:56] [longhorn] creating longhorn-system with privileged PSA labels
[00:26:56] [longhorn] configuring backup target s3://longhorn-drill-backup@us-east-1/
[00:26:56] [longhorn] installing chart 1.8.1 via k3s helm controller
[00:28:11] [longhorn] ready after 0s — 4/4 with schedulable disk: drill-s1.pxe-test.local,drill-s2.pxe-test.local,drill-s3.pxe-test.local,drill-spare.pxe-test.local
[drill] <== install longhorn (75s)
[drill] ==> seed data
[00:28:11] [longhorn] creating drill-r3 (2Gi, 3 replicas)
[00:28:11] [longhorn] creating drill-r1 (1Gi, 1 replicas)
[00:29:01] [longhorn] writing 512 MiB to drill-r3
[00:29:04] [longhorn] drill-r3 digest b8171779defe286b...
[00:29:05] [longhorn] writing 128 MiB to drill-r1
[00:29:06] [longhorn] drill-r1 digest 89213fddc2738c8d...
[00:29:40] [longhorn] drill-r3 verified intact across a detach/attach (pod drill-r3-754b4fbc7d-6ntwp -> drill-r3-754b4fbc7d-xrgds)
[00:30:13] [longhorn] drill-r1 verified intact across a detach/attach (pod drill-r1-949b946f8-4vbr6 -> drill-r1-949b946f8-pt6kv)
[drill] <== seed data (123s)
[drill] ==> backup and restore
[00:30:14] [longhorn] backup target set to s3://longhorn-drill-backup@us-east-1/
[00:30:15] [longhorn] backup target available after 0s
[00:30:16] [longhorn] snapshotting drill-r3 (pvc-4eeea364-2ed3-41fa-abc4-0176fb13ba3e)
[00:30:21] [longhorn] backing up drill-r3 to the backup target
[00:30:43] [longhorn] backup drill-bak-mundcj51 completed after 21s
[00:30:44] [longhorn] restoring drill-bak-mundcj51 into drill-restored
[drill] restored digest b8171779defe286b... matches original
[drill] <== backup and restore (108s)
[drill] ==> negative pre-check
[drill] <== negative pre-check (34s)
[drill] ==> roll drill-s1
[00:32:37] [snapshot] before rolling drill-s1: 4/4 Ready, etcd members=3 healthy=true
[drill-s1] All replicas evicted from drill-s1.pxe-test.local after 32s
[drill-s1] Drained drill-s1.pxe-test.local
[00:33:20] [longhorn] drill-r3 healthy on 3 nodes after 1s
[00:34:16] [longhorn] drill-r3 verified intact across a detach/attach (pod drill-r3-754b4fbc7d-xrgds -> drill-r3-754b4fbc7d-mhlj7)
[00:34:49] [longhorn] drill-r1 verified intact across a detach/attach (pod drill-r1-949b946f8-pt6kv -> drill-r1-949b946f8-6kqct)
[00:34:50] [etcd] removed member drill-s1.pxe-test.local-e58f4aaa (9575fbd07a6c3ca1)
[00:35:37] [snapshot] drill-s1 removed: 3/3 Ready, etcd members=2 healthy=true
[00:46:06] [snapshot] drill-s1 rejoined: 4/4 Ready, etcd members=3 healthy=true
[00:46:37] [longhorn] drill-r3 healthy on 3 nodes after 1s
[00:47:53] [longhorn] drill-r3 verified intact across a detach/attach (pod drill-r3-754b4fbc7d-mhlj7 -> drill-r3-754b4fbc7d-qhn2c)
[00:48:58] [longhorn] drill-r1 verified intact across a detach/attach (pod drill-r1-949b946f8-6kqct -> drill-r1-949b946f8-zj9lv)
[drill] after rolling drill-s1: 2 RUNNING replica(s) on the node
[drill] <== roll drill-s1 (982s)
[drill] ==> roll drill-s2
[00:48:59] [snapshot] before rolling drill-s2: 4/4 Ready, etcd members=3 healthy=true
[drill-s2] All replicas evicted from drill-s2.pxe-test.local after 47s
[drill-s2] Drained drill-s2.pxe-test.local
[00:50:31] [longhorn] drill-r3 healthy on 3 nodes after 1s
[00:51:41] [longhorn] drill-r3 verified intact across a detach/attach (pod drill-r3-754b4fbc7d-qhn2c -> drill-r3-754b4fbc7d-9bfqw)
[00:52:15] [longhorn] drill-r1 verified intact across a detach/attach (pod drill-r1-949b946f8-zj9lv -> drill-r1-949b946f8-7k9pn)
[00:52:16] [etcd] removed member drill-s2.pxe-test.local-943a6633 (2dfcaab9db591b03)
[00:53:02] [snapshot] drill-s2 removed: 3/3 Ready, etcd members=2 healthy=true
[01:04:02] [snapshot] drill-s2 rejoined: 4/4 Ready, etcd members=3 healthy=true
[01:04:32] [longhorn] drill-r3 healthy on 3 nodes after 1s
[01:05:38] [longhorn] drill-r3 verified intact across a detach/attach (pod drill-r3-754b4fbc7d-9bfqw -> drill-r3-754b4fbc7d-tf7z2)
[01:06:46] [longhorn] drill-r1 verified intact across a detach/attach (pod drill-r1-949b946f8-7k9pn -> drill-r1-949b946f8-cx7rx)
[drill] after rolling drill-s2: 2 RUNNING replica(s) on the node
[drill] <== roll drill-s2 (1069s)
[drill] ==> roll drill-s3
[01:06:48] [snapshot] before rolling drill-s3: 4/4 Ready, etcd members=3 healthy=true
[drill-s3] All replicas evicted from drill-s3.pxe-test.local after 31s
[drill-s3] Drained drill-s3.pxe-test.local
[01:07:59] [longhorn] drill-r3 healthy on 3 nodes after 1s
[01:09:14] [longhorn] drill-r3 verified intact across a detach/attach (pod drill-r3-754b4fbc7d-tf7z2 -> drill-r3-754b4fbc7d-qv54j)
[01:10:11] [longhorn] drill-r1 verified intact across a detach/attach (pod drill-r1-949b946f8-cx7rx -> drill-r1-949b946f8-sm8z7)
[01:10:13] [etcd] removed member drill-s3.pxe-test.local-9bf0ac84 (d47166c003b5115c)
[01:11:01] [snapshot] drill-s3 removed: 3/3 Ready, etcd members=2 healthy=true
[01:22:13] [snapshot] drill-s3 rejoined: 4/4 Ready, etcd members=3 healthy=true
[01:22:44] [longhorn] drill-r3 healthy on 3 nodes after 1s
[01:23:50] [longhorn] drill-r3 verified intact across a detach/attach (pod drill-r3-754b4fbc7d-qv54j -> drill-r3-754b4fbc7d-wbz8c)
[01:24:50] [longhorn] drill-r1 verified intact across a detach/attach (pod drill-r1-949b946f8-sm8z7 -> drill-r1-949b946f8-s5j48)
[drill] after rolling drill-s3: 2 RUNNING replica(s) on the node
[drill] <== roll drill-s3 (1084s)
[drill] ==> final verification
[01:24:51] [snapshot] end of drill: 4/4 Ready, etcd members=3 healthy=true
[01:25:26] [longhorn] drill-r3 verified intact across a detach/attach (pod drill-r3-754b4fbc7d-wbz8c -> drill-r3-754b4fbc7d-nj5jp)
[01:25:59] [longhorn] drill-r1 verified intact across a detach/attach (pod drill-r1-949b946f8-s5j48 -> drill-r1-949b946f8-xfbcq)
[drill] <== final verification (69s)
[01:26:00] [snapshot] FINAL: 4/4 Ready, etcd members=3 healthy=true
[drill] digests: drill-r3=b8171779defe286b drill-r1=89213fddc2738c8d
=== result ===
✓ provisions four v2 nodes 943264ms
✓ forms one cluster with three etcd members plus a storage agent 348143ms
✓ installs Longhorn with a schedulable disk on every node 75260ms
✓ writes known data and proves it survives a detach/attach 122827ms
✓ backs up a volume to S3 and restores it byte-identical 108172ms
✓ REFUSES to evacuate when there is nowhere for replicas to go 33846ms
✓ rolls drill-s1: evacuate, wipe, reinstall, rejoin, verify 982060ms
✓ rolls drill-s2: evacuate, wipe, reinstall, rejoin, verify 1068887ms
✓ rolls drill-s3: evacuate, wipe, reinstall, rejoin, verify 1083812ms
✓ ends with a whole cluster and the original bytes 68503ms
Test Files 1 passed | 10 skipped (11)
Tests 10 passed | 104 skipped (114)
VERDICT: PASSED (10/10).
- All four nodes ran the PINNED k3s v1.34.6+k3s1, asserted from kubeletVersion.
- Backup target reachable and authenticated; 512 MiB backed up to S3 and RESTORED
into a separate volume byte-identical to the original digest.
- Three storage+etcd nodes rolled through evacuate/wipe/reinstall/rejoin one at a
time; checksums identical before and after every roll; etcd held exactly 3 healthy
members at every gate, 2 only while a node was deliberately absent, never 4.
- Each reinstalled node came back hosting RUNNING replicas.
NOTE: defaultSettings.backupTarget is IGNORED by Longhorn 1.8 — the target lives on
the backuptargets.longhorn.io/default CR. Setting it via helm values reports success
and produces no backup target. Both the Pulumi code and this drill patch the CR.
Reference in New Issue Copy Permalink