v2 partition schema: capability model, fixed sizing, and a proven rolling migration
Some checks failed
Some checks failed
The fleet's disk layout was decided by literal role-name comparisons at 17 sites across 8 files. That is why `labcontroller` got no rancher LV (it matched neither "infra" nor "worker"), why labmaster carried an empty 728 GiB Longhorn LV, and why a 62 GiB container image tipped that node into DiskPressure and evicted bastion, labd, cockroach and openbao. Three generators each hardcoded their own sizes and disagreed: the kickstart said rancher=120G, ubuntu-autoinstall said 20G, asahi-firstboot said 20G under a comment asserting the sizes matched install.ks.ts. They did not. ROLE_REGISTRY already carried `k3s` and `parent` and nothing read them. This makes capabilities authoritative, moves every size into one shared module, and adds schema v2 (fixed 250G rancher / 300G longhorn, never --grow). Existing machines stay v1 forever: schema is detected from the on-disk VG tag, so a v1 box stays v1 even when v2 is requested. Proven on real VMs, not just unit tests: all four topologies installed and had their layouts read back off the machines; markers in /home, /srv, /var/lib/longhorn and /var/lib/rancher survived reinstall; and a 4-node cluster formed, took real data, and had all three servers rolled one at a time through evacuate / wipe / reinstall / rejoin with checksums verified before and after each roll. Drill: 10/10, evidence in migration/drill-evidence/. Production defects the drill found, none of which a unit test could reach: - sshOpts() capped every operation at 30s while drainNode asked kubectl for --timeout=15m, so the transport killed the drain after evicting most pods and returned a transcript that read like success. `labctl app k3s evacuate` could never have drained a real node. - generateAgentConfig put node-role.kubernetes.io/worker in the kubelet's node-label list, which NodeRestriction rejects, so no agent installed through K3sModule could start. systemd reported only "did not take the steps required by its unit configuration". - toOpContext built K3sConfig by enumerating three keys and silently dropped the rest, so the k3s version pin did nothing and --allow-single-replica was a no-op that claimed to override a data-loss guard. - The Longhorn disk annotation was mangled by shell quoting; with createDefaultDiskLabeledNodes the node then gets no disk at all, joins, goes Ready, and silently hosts no replica. - evacuate drained before evicting, which blocks on Longhorn's PDBs and would destroy a single-replica volume's only source. - etcd member ids were parsed from JSON, losing uint64 precision and emitting decimal where `member remove` needs hex -- at the moment a node is stopped and about to be wiped. - get.k3s.io installs latest, two minor versions ahead of the fleet, so a reinstalled node would rejoin on a newer k3s than its peers and roll an unplanned two-version control-plane upgrade. K3sConfig.k3sVersion now pins it. - `labctl app k3s install --role x` mapped anything that was not "worker" to "infra", so the runbook's own `--role k8s-combo` would have installed a different role with different disk expectations and no warning. Also: /var/lib/longhorn/replicas must never be deleted, only its contents, or every replica placed on the node dies on a non-recursive mkdir -- for ever, while the node reports Ready, Schedulable and healthy. etcd is now in the kickstart's package set for k3s server roles, because `k3s etcdctl` is not a command and a member that needs to recover its own membership has no internet. 542 unit tests / 39 files. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -56,6 +56,17 @@ export function ensurePxeNetwork(): void {
|
||||
|
||||
try { unlinkSync(xmlPath); } catch { /* ignore */ }
|
||||
|
||||
// `virsh net-start` returns before the bridge is usable. The bastion's
|
||||
// dnsmasq starts about a second later and binds to this device by name; if
|
||||
// the interface is not up and addressed yet it fails with "error binding
|
||||
// DHCP socket to device virbr-pxe", and because that failure is
|
||||
// asynchronous the harness carries on with NO DHCP AT ALL -- every VM then
|
||||
// sits in discovery until it times out, six minutes later, looking like a
|
||||
// PXE problem rather than a startup-order one.
|
||||
//
|
||||
// Cheap to wait, expensive not to.
|
||||
waitForBridge();
|
||||
|
||||
log(`Network ${PXE_NETWORK_NAME} created and active`);
|
||||
}
|
||||
|
||||
@@ -64,6 +75,32 @@ export function ensurePxeNetwork(): void {
|
||||
deleteNftablesRejectRules();
|
||||
}
|
||||
|
||||
/**
|
||||
* Block until the bridge exists AND carries the gateway address.
|
||||
*
|
||||
* Checking only that the device exists is not enough: it appears before the
|
||||
* address is assigned, and dnsmasq needs the address to bind.
|
||||
*/
|
||||
export function waitForBridge(timeoutMs = 30_000): void {
|
||||
const start = Date.now();
|
||||
let last = "";
|
||||
while (Date.now() - start < timeoutMs) {
|
||||
const r = spawnSync("ip", ["-br", "addr", "show", PXE_BRIDGE], { encoding: "utf-8", stdio: "pipe" });
|
||||
last = (r.stdout ?? "").trim();
|
||||
if (r.status === 0 && last.includes(`${PXE_GATEWAY}/`)) {
|
||||
log(`Bridge ${PXE_BRIDGE} ready with ${PXE_GATEWAY} after ${Date.now() - start}ms`);
|
||||
return;
|
||||
}
|
||||
// Busy-wait briefly; this normally resolves in well under a second.
|
||||
const until = Date.now() + 250;
|
||||
while (Date.now() < until) { /* spin */ }
|
||||
}
|
||||
throw new Error(
|
||||
`Bridge ${PXE_BRIDGE} did not come up with ${PXE_GATEWAY} within ${timeoutMs}ms ` +
|
||||
`(last: ${last || "device absent"}). dnsmasq cannot bind, so PXE would silently not work.`,
|
||||
);
|
||||
}
|
||||
|
||||
/** Delete libvirt's nftables reject rules for our bridge so host→VM traffic works.
|
||||
* Must be called after every VM start/restart — libvirt recreates them. */
|
||||
export function deleteNftablesRejectRules(): void {
|
||||
|
||||
Reference in New Issue
Block a user