v2 partition schema: capability model, fixed sizing, and a proven rolling migration
Some checks failed
Some checks failed
The fleet's disk layout was decided by literal role-name comparisons at 17 sites across 8 files. That is why `labcontroller` got no rancher LV (it matched neither "infra" nor "worker"), why labmaster carried an empty 728 GiB Longhorn LV, and why a 62 GiB container image tipped that node into DiskPressure and evicted bastion, labd, cockroach and openbao. Three generators each hardcoded their own sizes and disagreed: the kickstart said rancher=120G, ubuntu-autoinstall said 20G, asahi-firstboot said 20G under a comment asserting the sizes matched install.ks.ts. They did not. ROLE_REGISTRY already carried `k3s` and `parent` and nothing read them. This makes capabilities authoritative, moves every size into one shared module, and adds schema v2 (fixed 250G rancher / 300G longhorn, never --grow). Existing machines stay v1 forever: schema is detected from the on-disk VG tag, so a v1 box stays v1 even when v2 is requested. Proven on real VMs, not just unit tests: all four topologies installed and had their layouts read back off the machines; markers in /home, /srv, /var/lib/longhorn and /var/lib/rancher survived reinstall; and a 4-node cluster formed, took real data, and had all three servers rolled one at a time through evacuate / wipe / reinstall / rejoin with checksums verified before and after each roll. Drill: 10/10, evidence in migration/drill-evidence/. Production defects the drill found, none of which a unit test could reach: - sshOpts() capped every operation at 30s while drainNode asked kubectl for --timeout=15m, so the transport killed the drain after evicting most pods and returned a transcript that read like success. `labctl app k3s evacuate` could never have drained a real node. - generateAgentConfig put node-role.kubernetes.io/worker in the kubelet's node-label list, which NodeRestriction rejects, so no agent installed through K3sModule could start. systemd reported only "did not take the steps required by its unit configuration". - toOpContext built K3sConfig by enumerating three keys and silently dropped the rest, so the k3s version pin did nothing and --allow-single-replica was a no-op that claimed to override a data-loss guard. - The Longhorn disk annotation was mangled by shell quoting; with createDefaultDiskLabeledNodes the node then gets no disk at all, joins, goes Ready, and silently hosts no replica. - evacuate drained before evicting, which blocks on Longhorn's PDBs and would destroy a single-replica volume's only source. - etcd member ids were parsed from JSON, losing uint64 precision and emitting decimal where `member remove` needs hex -- at the moment a node is stopped and about to be wiped. - get.k3s.io installs latest, two minor versions ahead of the fleet, so a reinstalled node would rejoin on a newer k3s than its peers and roll an unplanned two-version control-plane upgrade. K3sConfig.k3sVersion now pins it. - `labctl app k3s install --role x` mapped anything that was not "worker" to "infra", so the runbook's own `--role k8s-combo` would have installed a different role with different disk expectations and no warning. Also: /var/lib/longhorn/replicas must never be deleted, only its contents, or every replica placed on the node dies on a non-recursive mkdir -- for ever, while the node reports Ready, Schedulable and healthy. etcd is now in the kickstart's package set for k3s server roles, because `k3s etcdctl` is not a command and a member that needs to recover its own membership has no internet. 542 unit tests / 39 files. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Michal
164
bastion/tests/integration/cluster-smoke.test.ts
Normal file
164
bastion/tests/integration/cluster-smoke.test.ts
Normal file
@@ -0,0 +1,164 @@
|
||||
// Integration test: form a real 3-server k3s cluster from PXE-installed VMs.
|
||||
//
|
||||
// WHY THIS EXISTS SEPARATELY FROM THE DRILL
|
||||
// -----------------------------------------
|
||||
// The rolling drill is a ~3 hour test. Almost everything that can go wrong in
|
||||
// its first thirty minutes goes wrong here instead, in forty-five: the token
|
||||
// handoff, the cluster-init/join discriminator, the etcd bootstrap race, root
|
||||
// SSH for the module, and whether post-provision was actually suppressed.
|
||||
// Debugging those inside a three-hour test is how you spend a day per attempt.
|
||||
//
|
||||
// It asserts on ETCD MEMBERSHIP, not just `kubectl get nodes`. A node object
|
||||
// and an etcd member are different things, and the whole hazard in a rolling
|
||||
// reinstall is that they can disagree.
|
||||
//
|
||||
// Prerequisites: libvirtd, OVMF, ipxe-bootimgs-x86, sudo. Set FEDORA_MIRROR to
|
||||
// a local mirror or each install pulls the full package set over the internet.
|
||||
//
|
||||
// sudo -E npx vitest run -c tests/integration/vitest.config.ts -t "cluster smoke"
|
||||
|
||||
import { describe, it, expect, beforeAll, afterAll } from "vitest";
|
||||
import { existsSync, readFileSync } from "node:fs";
|
||||
import { homedir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import {
|
||||
startTestBastion, provisionVms, destroyVms,
|
||||
type Harness, type ProvisionedVm,
|
||||
} from "./helpers/provision-harness.js";
|
||||
import {
|
||||
formCluster, clusterSnapshot, printSnapshot, etcdMembers, etcdHealthy,
|
||||
allServersReady, pickJoinTarget, kubectlOn, k8sNodeName,
|
||||
} from "./helpers/k3s-cluster.js";
|
||||
import { destroyPxeNetwork } from "./helpers/pxe-network.js";
|
||||
|
||||
const HTTP_PORT = 8097;
|
||||
const SSH_USER = "lab";
|
||||
const DISK_GB = 1000;
|
||||
const NODES = ["smoke-a", "smoke-b", "smoke-c"];
|
||||
|
||||
function findSshKey(): { pubKey: string; keyPath: string } {
|
||||
const homes = [homedir()];
|
||||
const sudoUser = process.env["SUDO_USER"];
|
||||
if (sudoUser) homes.push(join("/home", sudoUser));
|
||||
for (const home of homes) {
|
||||
for (const name of ["id_ed25519", "id_ecdsa", "id_rsa"]) {
|
||||
const keyPath = join(home, ".ssh", name);
|
||||
if (existsSync(keyPath) && existsSync(`${keyPath}.pub`)) {
|
||||
return { pubKey: readFileSync(`${keyPath}.pub`, "utf-8").trim(), keyPath };
|
||||
}
|
||||
}
|
||||
}
|
||||
throw new Error("no SSH keypair found");
|
||||
}
|
||||
|
||||
describe("cluster smoke: three servers, one cluster", () => {
|
||||
let h: Harness;
|
||||
let vms: ProvisionedVm[] = [];
|
||||
|
||||
/**
|
||||
* Kubernetes node names, NOT libvirt domain names.
|
||||
*
|
||||
* k3s registers a node by FQDN, so the VM `smoke-a` is the node
|
||||
* `smoke-a.pxe-test.local`. Asserting against the VM names would compare two
|
||||
* different namespaces and fail for a reason that has nothing to do with the
|
||||
* cluster.
|
||||
*/
|
||||
const nodeNames = (): string[] => vms.map((v) => k8sNodeName(h, v)).sort();
|
||||
|
||||
beforeAll(async () => {
|
||||
const { pubKey, keyPath } = findSshKey();
|
||||
destroyVms(...NODES);
|
||||
h = await startTestBastion({
|
||||
httpPort: HTTP_PORT, sshUser: SSH_USER, sshPubKey: pubKey, sshKeyPath: keyPath,
|
||||
label: "cluster-smoke",
|
||||
// Mandatory. Left on, post-provision installs each node into its OWN
|
||||
// cluster and this test would be measuring three clusters of one.
|
||||
autoK3s: false,
|
||||
});
|
||||
}, 20 * 60_000);
|
||||
|
||||
afterAll(async () => {
|
||||
// Dump state before anything is destroyed — the VMs are the only place
|
||||
// this information exists, and they are about to stop existing.
|
||||
try {
|
||||
if (vms.length > 0) printSnapshot("final", clusterSnapshot(h, vms[0]!));
|
||||
} catch { /* cluster may already be gone */ }
|
||||
destroyVms(...NODES);
|
||||
try { destroyPxeNetwork(); } catch { /* shared */ }
|
||||
await h?.close();
|
||||
}, 10 * 60_000);
|
||||
|
||||
it("provisions three v2 combo nodes", async () => {
|
||||
vms = await provisionVms(h, NODES.map((name) => ({
|
||||
name, role: "k8s-combo" as const, schemaVersion: 2 as const,
|
||||
diskGb: DISK_GB, memory: 6144, vcpus: 4,
|
||||
})));
|
||||
expect(vms).toHaveLength(3);
|
||||
for (const vm of vms) expect(vm.ip).toMatch(/^\d+\.\d+\.\d+\.\d+$/);
|
||||
}, 60 * 60_000);
|
||||
|
||||
it("forms ONE cluster, not three", async () => {
|
||||
await formCluster(h, vms, { role: "k8s-combo" });
|
||||
|
||||
const snap = clusterSnapshot(h, vms[0]!);
|
||||
printSnapshot("after formation", snap);
|
||||
|
||||
// The precise failure this guards: three separate single-node clusters
|
||||
// each report 1/1 Ready and look perfectly healthy in isolation.
|
||||
expect(snap.nodes).toHaveLength(3);
|
||||
expect(snap.readyCount).toBe(3);
|
||||
expect(snap.nodes.map((n) => n.name).sort()).toEqual(nodeNames());
|
||||
}, 90 * 60_000);
|
||||
|
||||
it("has exactly three healthy etcd members", () => {
|
||||
const members = etcdMembers(h, vms[0]!);
|
||||
// .node, not .name: etcd names a member "<node>-<8 hex>", so comparing
|
||||
// raw names against node names never matches.
|
||||
expect(members.map((m) => m.node).sort()).toEqual(nodeNames());
|
||||
expect(members).toHaveLength(3);
|
||||
expect(members.every((m) => m.started)).toBe(true);
|
||||
expect(etcdHealthy(h, vms[0]!)).toBe(true);
|
||||
}, 5 * 60_000);
|
||||
|
||||
it("agrees about membership from EVERY node, not just the first", () => {
|
||||
// Asked of each server independently: a split brain is three nodes that
|
||||
// each believe they are correct.
|
||||
for (const vm of vms) {
|
||||
const members = etcdMembers(h, vm).map((m) => m.node).sort();
|
||||
expect(members, `etcd membership as seen by ${vm.name}`).toEqual(nodeNames());
|
||||
const nodes = clusterSnapshot(h, vm).nodes.map((n) => n.name).sort();
|
||||
expect(nodes, `node list as seen by ${vm.name}`).toEqual(nodeNames());
|
||||
}
|
||||
}, 10 * 60_000);
|
||||
|
||||
it("reports every server ready", () => {
|
||||
const ready = allServersReady(h, vms);
|
||||
expect(ready.ok, ready.detail).toBe(true);
|
||||
}, 5 * 60_000);
|
||||
|
||||
it("makes all three control-plane and etcd", () => {
|
||||
const snap = clusterSnapshot(h, vms[0]!);
|
||||
for (const n of snap.nodes) {
|
||||
expect(n.roles, `${n.name} roles`).toContain("control-plane");
|
||||
expect(n.roles, `${n.name} roles`).toContain("etcd");
|
||||
}
|
||||
}, 5 * 60_000);
|
||||
|
||||
it("can pick a join target other than a given node", () => {
|
||||
// The primitive the rolling drill depends on: after the first server is
|
||||
// wiped it must rejoin against a survivor.
|
||||
const target = pickJoinTarget(h, vms, vms[0]!);
|
||||
expect(target.name).not.toBe(vms[0]!.name);
|
||||
}, 5 * 60_000);
|
||||
|
||||
it("carries the Longhorn disk label on every node", () => {
|
||||
// Set via k3s node-label. Without it Longhorn creates no disk on the node,
|
||||
// and the node then hosts zero replicas while looking entirely healthy.
|
||||
const out = kubectlOn(h, vms[0]!,
|
||||
`get nodes -o jsonpath='{range .items[*]}{.metadata.name}={.metadata.labels.node\\.longhorn\\.io/create-default-disk}{"\\n"}{end}'`);
|
||||
for (const vm of vms) {
|
||||
expect(out, `${vm.name} create-default-disk label`)
|
||||
.toContain(`${k8sNodeName(h, vm)}=config`);
|
||||
}
|
||||
}, 5 * 60_000);
|
||||
});
|
||||
427
bastion/tests/integration/helpers/k3s-cluster.ts
Normal file
427
bastion/tests/integration/helpers/k3s-cluster.ts
Normal file
@@ -0,0 +1,427 @@
|
||||
// Form and inspect a real multi-node k3s cluster from an integration test.
|
||||
//
|
||||
// WHY THIS EXISTS
|
||||
// ---------------
|
||||
// Nothing in the repo decides which node bootstraps a cluster and which nodes
|
||||
// join it. The production generator has the right discriminator --
|
||||
// generateServerConfig emits `cluster-init: true` when no server URL is set and
|
||||
// `server:`+`token:` when one is -- but the choice is an INPUT, and the only
|
||||
// caller that ever supplied it was a hand-run labsim shell script.
|
||||
//
|
||||
// So a test that wants three servers in one cluster has to sequence it. That
|
||||
// sequencing is here rather than in a test file because the rolling drill is
|
||||
// not the only thing that will want it: an etcd-recovery test (recoverEtcdMember
|
||||
// still has no caller), a dual-stack conversion and any HA upgrade test all
|
||||
// need exactly these primitives.
|
||||
//
|
||||
// Everything drives K3sModule, not shell. That is deliberate and is the same
|
||||
// rule labsim follows by shelling out to the compiled render-config binary: a
|
||||
// test that reimplements the install proves the test works, not the product.
|
||||
|
||||
import { execSync } from "node:child_process";
|
||||
import type { Role } from "../../../src/shared/src/types/state.js";
|
||||
import { K3sModule } from "../../../src/modules/modules/k3s/src/k3s-module.js";
|
||||
import type { ModuleContext } from "../../../src/modules/src/types.js";
|
||||
import { log } from "./libvirt.js";
|
||||
import { parseEtcdMembers, type EtcdMember } from "../../../src/modules/modules/k3s/src/operations/etcd-members.js";
|
||||
import { sshExec } from "./ssh.js";
|
||||
import type { Harness, ProvisionedVm } from "./provision-harness.js";
|
||||
|
||||
export const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));
|
||||
|
||||
/**
|
||||
* A pre-shared cluster token, the way labsim does it.
|
||||
*
|
||||
* The alternative is scraping /var/lib/rancher/k3s/server/node-token off the
|
||||
* first server, and nothing in the repo does that. It is also the wrong shape
|
||||
* for a rolling migration: the first server gets wiped and rejoins, so the
|
||||
* token has to outlive it and be knowable before it exists.
|
||||
*/
|
||||
export const DRILL_TOKEN = "lab-drill-shared-token";
|
||||
|
||||
/**
|
||||
* k3s operations run unprefixed `mkdir -p /etc/rancher/k3s` and
|
||||
* `systemctl restart k3s`, so they need root. install.ks.ts provisions
|
||||
* `sshkey --username=root` and sets PermitRootLogin prohibit-password, so root
|
||||
* SSH works on every machine this harness builds. The harness's own `lab` user
|
||||
* stays for the sudo-based reads (readLvs, markers).
|
||||
*/
|
||||
export const MODULE_SSH_USER = "root";
|
||||
|
||||
/**
|
||||
* The k3s version the drill runs, pinned to what the fleet runs.
|
||||
*
|
||||
* get.k3s.io installs LATEST, and latest is currently two minor versions ahead
|
||||
* of production (v1.36.4 vs v1.34.6). Two things follow, and both matter:
|
||||
*
|
||||
* - a rehearsal on a different minor version is rehearsing a different
|
||||
* system, and Longhorn in particular is only qualified against a range of
|
||||
* Kubernetes versions;
|
||||
* - it is the same trap production would hit. A node reinstalled mid-migration
|
||||
* comes back on latest while its peers stay put, and Kubernetes supports
|
||||
* only ONE minor of skew between control-plane members.
|
||||
*
|
||||
* Override with LAB_K3S_VERSION to test an upgrade deliberately.
|
||||
*/
|
||||
export const PINNED_K3S_VERSION = process.env["LAB_K3S_VERSION"] ?? "v1.34.6+k3s1";
|
||||
|
||||
/** Build the ModuleContext K3sModule expects for one VM. */
|
||||
export function moduleCtx(
|
||||
h: Harness,
|
||||
vm: ProvisionedVm,
|
||||
role: Role,
|
||||
config: Record<string, unknown> = {},
|
||||
): ModuleContext {
|
||||
return {
|
||||
hostname: vm.name,
|
||||
ip: vm.ip,
|
||||
role,
|
||||
os: "fedora-43",
|
||||
arch: "x86_64",
|
||||
sshUser: MODULE_SSH_USER,
|
||||
sshKeyPath: h.sshKeyPath,
|
||||
config,
|
||||
};
|
||||
}
|
||||
|
||||
/** Run a command on a node as root, throwing with context on failure. */
|
||||
export function nodeExec(h: Harness, vm: ProvisionedVm, cmd: string, timeoutMs = 120_000): string {
|
||||
const r = sshExec(vm.ip, MODULE_SSH_USER, cmd, { keyPath: h.sshKeyPath, timeout: timeoutMs });
|
||||
if (r.exitCode !== 0) {
|
||||
throw new Error(`[${vm.name}] ${cmd} failed (${r.exitCode}): ${r.stderr.trim() || r.stdout.trim()}`);
|
||||
}
|
||||
return r.stdout;
|
||||
}
|
||||
|
||||
/** Same, but returns the result instead of throwing. */
|
||||
export function tryNodeExec(h: Harness, vm: ProvisionedVm, cmd: string, timeoutMs = 120_000) {
|
||||
return sshExec(vm.ip, MODULE_SSH_USER, cmd, { keyPath: h.sshKeyPath, timeout: timeoutMs });
|
||||
}
|
||||
|
||||
/**
|
||||
* The name Kubernetes knows this VM by.
|
||||
*
|
||||
* NOT the libvirt domain name. k3s registers a node under its FQDN, so a VM
|
||||
* called `smoke-a` appears as `smoke-a.pxe-test.local`. Passing the VM name to
|
||||
* `kubectl get node` silently finds nothing, and every wait built on it times
|
||||
* out instead of failing usefully. Read from the node itself, the same way the
|
||||
* production operations do (`hostname -f`), so the two can never disagree.
|
||||
*/
|
||||
const nodeNameCache = new Map<string, string>();
|
||||
export function k8sNodeName(h: Harness, vm: ProvisionedVm): string {
|
||||
const cached = nodeNameCache.get(vm.name);
|
||||
if (cached !== undefined) return cached;
|
||||
const name = nodeExec(h, vm, "hostname -f 2>/dev/null || hostname").trim();
|
||||
if (name === "") throw new Error(`[${vm.name}] could not resolve node name`);
|
||||
nodeNameCache.set(vm.name, name);
|
||||
return name;
|
||||
}
|
||||
|
||||
/** Forget cached names — a reinstalled machine re-registers. */
|
||||
export function forgetNodeName(vm: ProvisionedVm): void {
|
||||
nodeNameCache.delete(vm.name);
|
||||
}
|
||||
|
||||
/** kubectl against a server node. */
|
||||
export function kubectlOn(h: Harness, server: ProvisionedVm, args: string, timeoutMs = 120_000): string {
|
||||
return nodeExec(h, server, `k3s kubectl ${args}`, timeoutMs);
|
||||
}
|
||||
|
||||
export function kubectlJson<T>(h: Harness, server: ProvisionedVm, args: string): T {
|
||||
return JSON.parse(kubectlOn(h, server, `${args} -o json`)) as T;
|
||||
}
|
||||
|
||||
/**
|
||||
* Wait until a server's API answers /ping.
|
||||
*
|
||||
* Straight from labsim-k8s-etcd.sh, whose comment is the whole justification:
|
||||
* a joiner that starts before the first server has finished bootstrapping etcd
|
||||
* races it, and the failure looks like etcd instability rather than a startup
|
||||
* order problem. Probed from the HOST, so it does not depend on the joining
|
||||
* node being reachable yet.
|
||||
*/
|
||||
export async function waitForApiPing(vm: ProvisionedVm, timeoutMs = 450_000): Promise<void> {
|
||||
const start = Date.now();
|
||||
let lastErr = "";
|
||||
while (Date.now() - start < timeoutMs) {
|
||||
try {
|
||||
const out = execSync(
|
||||
`curl -sk --max-time 5 https://${vm.ip}:6443/ping || true`,
|
||||
{ encoding: "utf-8", timeout: 15_000 },
|
||||
).trim();
|
||||
if (out === "pong") {
|
||||
log(`[${vm.name}] API answering after ${Math.round((Date.now() - start) / 1000)}s`);
|
||||
return;
|
||||
}
|
||||
lastErr = out;
|
||||
} catch (e) {
|
||||
lastErr = e instanceof Error ? e.message : String(e);
|
||||
}
|
||||
await sleep(5_000);
|
||||
}
|
||||
throw new Error(`[${vm.name}] API never answered /ping in ${timeoutMs / 1000}s (last: ${lastErr || "no response"})`);
|
||||
}
|
||||
|
||||
/** Wait for a node to appear Ready in the cluster, as seen from `server`. */
|
||||
export async function waitForNodeReady(
|
||||
h: Harness,
|
||||
server: ProvisionedVm,
|
||||
/** The VM to wait for; its Kubernetes name is resolved from the machine. */
|
||||
target: ProvisionedVm,
|
||||
// Generous: a node is NotReady until its CNI is up, and on the cluster-init
|
||||
// server that means waiting for Cilium to install and pull its images. Five
|
||||
// minutes is enough on a warm host and not enough on a cold one, and the
|
||||
// cost of being wrong is a wasted multi-hour run.
|
||||
timeoutMs = 900_000,
|
||||
): Promise<void> {
|
||||
const nodeName = k8sNodeName(h, target);
|
||||
const start = Date.now();
|
||||
let last = "";
|
||||
while (Date.now() - start < timeoutMs) {
|
||||
const r = tryNodeExec(h, server,
|
||||
`k3s kubectl get node ${nodeName} -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}'`);
|
||||
last = r.stdout.trim();
|
||||
if (last === "True") {
|
||||
log(`[${nodeName}] Ready after ${Math.round((Date.now() - start) / 1000)}s`);
|
||||
return;
|
||||
}
|
||||
await sleep(5_000);
|
||||
}
|
||||
throw new Error(`[${nodeName}] never became Ready in ${timeoutMs / 1000}s (last status: ${last || "absent"})`);
|
||||
}
|
||||
|
||||
/**
|
||||
* etcdctl invocation.
|
||||
*
|
||||
* NOT `k3s etcdctl` -- that is not a subcommand ("No help topic for 'etcdctl'",
|
||||
* verified on a live v1.34 node). The binary comes from the `etcd` rpm, which
|
||||
* our kickstart does NOT install, so ensureEtcdctl() has to put it there.
|
||||
* Cert paths verified against a real node.
|
||||
*/
|
||||
const ETCD_BASE = "/var/lib/rancher/k3s/server/tls/etcd";
|
||||
const ETCDCTL =
|
||||
`ETCDCTL_API=3 etcdctl ` +
|
||||
`--cacert=${ETCD_BASE}/server-ca.crt ` +
|
||||
`--cert=${ETCD_BASE}/server-client.crt ` +
|
||||
`--key=${ETCD_BASE}/server-client.key ` +
|
||||
`--endpoints=https://127.0.0.1:2379`;
|
||||
|
||||
/** Install etcdctl if the node does not have it. Idempotent, ~30s once. */
|
||||
export function ensureEtcdctl(h: Harness, server: ProvisionedVm): void {
|
||||
const has = tryNodeExec(h, server, "command -v etcdctl");
|
||||
if (has.exitCode === 0) return;
|
||||
log(`[${server.name}] installing etcdctl (the etcd rpm; kickstart does not include it)`);
|
||||
nodeExec(h, server, "dnf install -y etcd >/dev/null 2>&1 || dnf install -y etcd-ctl >/dev/null 2>&1", 300_000);
|
||||
nodeExec(h, server, "command -v etcdctl");
|
||||
}
|
||||
|
||||
|
||||
/**
|
||||
* etcd membership, read with etcdctl against k3s's own certs.
|
||||
*
|
||||
* Read rather than inferred from `kubectl get nodes`: a node object can be
|
||||
* deleted while its etcd member survives, and that stale member is exactly the
|
||||
* failure a rolling reinstall creates. Counting nodes would not see it.
|
||||
*/
|
||||
export function etcdMembers(h: Harness, server: ProvisionedVm): EtcdMember[] {
|
||||
// NOT `-w json`. Two reasons, and the second is the dangerous one:
|
||||
//
|
||||
// 1. `member remove` requires the ID in HEX; the JSON writer emits decimal.
|
||||
// 2. etcd member IDs are uint64. JSON.parse turns them into JS numbers,
|
||||
// which silently lose precision above 2^53 -- the ID that comes back is
|
||||
// an APPROXIMATION of the real one. Passing that to `member remove`
|
||||
// fails with "value out of range" if you are lucky, and in principle
|
||||
// could match some other member if you are not. This happens at the one
|
||||
// moment it must not: the node is already stopped and about to be wiped.
|
||||
//
|
||||
// The default writer prints the ID as hex text, so it never becomes a
|
||||
// number at all:
|
||||
// 825852d3683bc406, started, worker2-...-5f7ad0c9, https://...:2380, ...
|
||||
return parseEtcdMembers(nodeExec(h, server, `${ETCDCTL} member list`));
|
||||
}
|
||||
|
||||
|
||||
/** Are all etcd endpoints healthy? Answers the question quorum actually asks. */
|
||||
export function etcdHealthy(h: Harness, server: ProvisionedVm): boolean {
|
||||
const r = tryNodeExec(h, server, `${ETCDCTL} endpoint health`);
|
||||
return r.exitCode === 0 && r.stdout.includes("is healthy");
|
||||
}
|
||||
|
||||
/**
|
||||
* Remove an etcd member by node name.
|
||||
*
|
||||
* Must happen BEFORE the node is wiped. k3s-install.ts removes the local
|
||||
* datastore when a server joins, so a reinstalled node rejoins as a NEW member
|
||||
* while the old record persists: 3 members becomes 4, quorum rises from 2 to 3,
|
||||
* and the cluster that tolerated one failure now tolerates none.
|
||||
*/
|
||||
export function removeEtcdMember(h: Harness, peer: ProvisionedVm, nodeName: string): boolean {
|
||||
const members = etcdMembers(h, peer);
|
||||
const target = members.find((m) => m.node === nodeName || m.name === nodeName);
|
||||
if (!target) {
|
||||
log(`[etcd] no member for ${nodeName} (have: ${members.map((m) => m.node).join(",")})`);
|
||||
return false;
|
||||
}
|
||||
nodeExec(h, peer, `${ETCDCTL} member remove ${target.id}`);
|
||||
log(`[etcd] removed member ${target.name} (${target.id})`);
|
||||
return true;
|
||||
}
|
||||
|
||||
export interface ClusterSnapshot {
|
||||
nodes: { name: string; ready: boolean; roles: string }[];
|
||||
readyCount: number;
|
||||
etcdMembers: number;
|
||||
etcdHealthy: boolean;
|
||||
}
|
||||
|
||||
/** One read of cluster state, in the shape labsim's cmd_status prints. */
|
||||
export function clusterSnapshot(h: Harness, server: ProvisionedVm): ClusterSnapshot {
|
||||
const raw = kubectlJson<{ items: {
|
||||
metadata: { name: string; labels?: Record<string, string> };
|
||||
status: { conditions?: { type: string; status: string }[] };
|
||||
}[] }>(h, server, "get nodes");
|
||||
|
||||
const nodes = raw.items.map((n) => ({
|
||||
name: n.metadata.name,
|
||||
ready: (n.status.conditions ?? []).some((c) => c.type === "Ready" && c.status === "True"),
|
||||
roles: Object.keys(n.metadata.labels ?? {})
|
||||
.filter((k) => k.startsWith("node-role.kubernetes.io/"))
|
||||
.map((k) => k.replace("node-role.kubernetes.io/", ""))
|
||||
.sort().join(",") || "<none>",
|
||||
}));
|
||||
|
||||
let members = 0;
|
||||
let healthy = false;
|
||||
try {
|
||||
members = etcdMembers(h, server).length;
|
||||
healthy = etcdHealthy(h, server);
|
||||
} catch { /* an agent, or the API is down — report zero rather than throw */ }
|
||||
|
||||
return { nodes, readyCount: nodes.filter((n) => n.ready).length, etcdMembers: members, etcdHealthy: healthy };
|
||||
}
|
||||
|
||||
/**
|
||||
* Print a snapshot. Called liberally during a drill: the VMs are destroyed at
|
||||
* the end, so this transcript is the only surviving record of what happened.
|
||||
*/
|
||||
export function printSnapshot(label: string, s: ClusterSnapshot): void {
|
||||
log(`[snapshot] ${label}: ${s.readyCount}/${s.nodes.length} Ready, ` +
|
||||
`etcd members=${s.etcdMembers} healthy=${s.etcdHealthy}`);
|
||||
for (const n of s.nodes) log(` ${n.ready ? "Ready " : "NotReady"} ${n.name} (${n.roles})`);
|
||||
}
|
||||
|
||||
/**
|
||||
* Install k3s on one node via the production module.
|
||||
*
|
||||
* Pass no serverUrl to bootstrap the cluster (the token still seeds), or a
|
||||
* serverUrl to join an existing one.
|
||||
*/
|
||||
export async function installK3sNode(h: Harness, vm: ProvisionedVm, opts: {
|
||||
role: Role;
|
||||
serverUrl?: string;
|
||||
token?: string;
|
||||
}): Promise<void> {
|
||||
const mod = new K3sModule();
|
||||
const config: Record<string, unknown> = {
|
||||
k3sToken: opts.token ?? DRILL_TOKEN,
|
||||
k3sVersion: PINNED_K3S_VERSION,
|
||||
...(opts.serverUrl === undefined ? {} : { k3sServerUrl: opts.serverUrl }),
|
||||
};
|
||||
|
||||
log(`[${vm.name}] installing k3s (${opts.role}${opts.serverUrl ? ` -> ${opts.serverUrl}` : " — cluster-init"})`);
|
||||
const install = await mod.install(moduleCtx(h, vm, opts.role, config));
|
||||
for (const line of install.output) log(` [${vm.name}] ${line}`);
|
||||
if (!install.success) {
|
||||
throw new Error(`[${vm.name}] k3s install failed: ${install.errors.join("; ")}`);
|
||||
}
|
||||
|
||||
const configure = await mod.configure(moduleCtx(h, vm, opts.role, config));
|
||||
for (const line of configure.output) log(` [${vm.name}] ${line}`);
|
||||
if (!configure.success) {
|
||||
throw new Error(`[${vm.name}] k3s configure failed: ${configure.errors.join("; ")}`);
|
||||
}
|
||||
}
|
||||
|
||||
export function serverUrlOf(vm: ProvisionedVm): string {
|
||||
return `https://${vm.ip}:6443`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Form a cluster from a list of server VMs, strictly in order.
|
||||
*
|
||||
* The first bootstraps etcd and installs the CNI (k3s-module runs networking
|
||||
* only on the cluster-init node -- joiners get Cilium via its DaemonSet). The
|
||||
* rest join one at a time, each gated on the previous node being Ready.
|
||||
*
|
||||
* Serial on purpose. labsim gets away with starting joiners together because
|
||||
* its nodes are cloud-init VMs that poll /ping themselves; here the few extra
|
||||
* minutes buy determinism in the one place where etcd bootstrap genuinely
|
||||
* races.
|
||||
*/
|
||||
export async function formCluster(h: Harness, servers: ProvisionedVm[], opts: {
|
||||
role?: Role;
|
||||
token?: string;
|
||||
} = {}): Promise<void> {
|
||||
if (servers.length === 0) throw new Error("formCluster: no servers");
|
||||
const role = opts.role ?? "k8s-combo";
|
||||
const token = opts.token ?? DRILL_TOKEN;
|
||||
const first = servers[0]!;
|
||||
|
||||
await installK3sNode(h, first, { role, token });
|
||||
await waitForApiPing(first);
|
||||
await waitForNodeReady(h, first, first);
|
||||
ensureEtcdctl(h, first);
|
||||
printSnapshot("after cluster-init", clusterSnapshot(h, first));
|
||||
|
||||
for (const vm of servers.slice(1)) {
|
||||
await waitForApiPing(first);
|
||||
await installK3sNode(h, vm, { role, token, serverUrl: serverUrlOf(first) });
|
||||
await waitForNodeReady(h, first, vm);
|
||||
ensureEtcdctl(h, vm);
|
||||
printSnapshot(`after joining ${vm.name}`, clusterSnapshot(h, first));
|
||||
}
|
||||
}
|
||||
|
||||
/** Join an agent. Agents cannot start without a server URL and token. */
|
||||
export async function joinAgent(h: Harness, vm: ProvisionedVm, opts: {
|
||||
server: ProvisionedVm;
|
||||
role?: Role;
|
||||
token?: string;
|
||||
}): Promise<void> {
|
||||
await waitForApiPing(opts.server);
|
||||
await installK3sNode(h, vm, {
|
||||
role: opts.role ?? "k8s-worker",
|
||||
token: opts.token ?? DRILL_TOKEN,
|
||||
serverUrl: serverUrlOf(opts.server),
|
||||
});
|
||||
await waitForNodeReady(h, opts.server, vm);
|
||||
}
|
||||
|
||||
/**
|
||||
* A healthy server that is NOT the node being worked on.
|
||||
*
|
||||
* Needed because the first server is itself rolled during a migration: after
|
||||
* it is wiped it must rejoin against a survivor, so "the first node" cannot be
|
||||
* the join target and cannot be where cluster state is read from.
|
||||
*/
|
||||
export function pickJoinTarget(
|
||||
h: Harness,
|
||||
servers: ProvisionedVm[],
|
||||
exclude: ProvisionedVm,
|
||||
): ProvisionedVm {
|
||||
for (const vm of servers) {
|
||||
if (vm.name === exclude.name) continue;
|
||||
const r = tryNodeExec(h, vm, "k3s kubectl get --raw /readyz");
|
||||
if (r.exitCode === 0 && r.stdout.trim() === "ok") return vm;
|
||||
}
|
||||
throw new Error(`No healthy server other than ${exclude.name} — cluster has no quorum to join against`);
|
||||
}
|
||||
|
||||
/** Every server reports /readyz ok. Asserted positively, not inferred. */
|
||||
export function allServersReady(h: Harness, servers: ProvisionedVm[]): { ok: boolean; detail: string } {
|
||||
const results = servers.map((vm) => {
|
||||
const r = tryNodeExec(h, vm, "k3s kubectl get --raw /readyz");
|
||||
return `${vm.name}=${r.exitCode === 0 ? r.stdout.trim() : `ERR(${r.exitCode})`}`;
|
||||
});
|
||||
return { ok: results.every((r) => r.endsWith("=ok")), detail: results.join(" ") };
|
||||
}
|
||||
666
bastion/tests/integration/helpers/longhorn.ts
Normal file
666
bastion/tests/integration/helpers/longhorn.ts
Normal file
@@ -0,0 +1,666 @@
|
||||
// Install Longhorn on a test cluster and put verifiable data in it.
|
||||
//
|
||||
// WHAT THIS IS FOR
|
||||
// ----------------
|
||||
// The rolling drill's whole claim is "the bytes survived". That needs data
|
||||
// whose integrity can be checked independently of Longhorn's own opinion of
|
||||
// itself -- Longhorn reporting `robustness: healthy` is not evidence that the
|
||||
// file is intact, it is evidence that Longhorn thinks it is.
|
||||
//
|
||||
// So: write a known blob, record its sha256, and re-verify THROUGH A POD
|
||||
// RESTART every time. The restart is not incidental. It forces a real
|
||||
// detach/attach, frequently onto a different node, which is the only way to
|
||||
// show the data followed the volume rather than sitting in a page cache on the
|
||||
// node that wrote it.
|
||||
//
|
||||
// Longhorn version is pinned to the fleet's (1.8.1) so the drill rehearses what
|
||||
// production runs, not what happens to be latest.
|
||||
|
||||
import type { Harness, ProvisionedVm } from "./provision-harness.js";
|
||||
import { log } from "./libvirt.js";
|
||||
import { nodeExec, tryNodeExec, kubectlOn, kubectlJson, sleep, k8sNodeName } from "./k3s-cluster.js";
|
||||
|
||||
export const LONGHORN_VERSION = "1.8.1";
|
||||
export const LONGHORN_NS = "longhorn-system";
|
||||
|
||||
/**
|
||||
* Install Longhorn via k3s's built-in helm controller.
|
||||
*
|
||||
* A HelmChart manifest dropped in the server manifests directory is how k3s
|
||||
* installs charts natively, so this needs no helm binary on the host and no
|
||||
* kubeconfig plumbing.
|
||||
*
|
||||
* The namespace is created FIRST, with privileged Pod Security labels. Longhorn
|
||||
* needs hostPath and privileged containers; under the default `restricted`
|
||||
* enforcement its DaemonSets are rejected and the install fails in a way that
|
||||
* looks like a chart problem.
|
||||
*
|
||||
* `createDefaultDiskLabeledNodes: true` matches production, and is the setting
|
||||
* that makes the node label and disk annotation load-bearing: without both, a
|
||||
* node gets no disk and silently hosts no replicas.
|
||||
*/
|
||||
/**
|
||||
* S3 backup target for the drill, from the environment.
|
||||
*
|
||||
* Absent means the backup/restore phase is skipped rather than silently
|
||||
* "passing" -- a backup test that quietly does nothing is worse than no test.
|
||||
* The run script exports these; see DRILL_S3_* in run-drill.sh.
|
||||
*/
|
||||
export interface BackupTargetOpts {
|
||||
/** e.g. s3://longhorn-drill-backup@us-east-1/ */
|
||||
target: string;
|
||||
endpoint: string;
|
||||
accessKey: string;
|
||||
secretKey: string;
|
||||
}
|
||||
|
||||
export function backupTargetFromEnv(): BackupTargetOpts | undefined {
|
||||
const target = process.env["DRILL_S3_TARGET"];
|
||||
const endpoint = process.env["DRILL_S3_ENDPOINT"];
|
||||
const accessKey = process.env["DRILL_S3_ACCESS_KEY"];
|
||||
const secretKey = process.env["DRILL_S3_SECRET_KEY"];
|
||||
if (target === undefined || endpoint === undefined
|
||||
|| accessKey === undefined || secretKey === undefined) return undefined;
|
||||
return { target, endpoint, accessKey, secretKey };
|
||||
}
|
||||
|
||||
export const BACKUP_SECRET = "longhorn-backup-credentials";
|
||||
|
||||
export async function installLonghorn(h: Harness, server: ProvisionedVm, opts: {
|
||||
defaultReplicas?: number;
|
||||
backup?: BackupTargetOpts;
|
||||
} = {}): Promise<void> {
|
||||
const replicas = opts.defaultReplicas ?? 3;
|
||||
|
||||
log(`[longhorn] creating ${LONGHORN_NS} with privileged PSA labels`);
|
||||
nodeExec(h, server, `k3s kubectl apply -f - <<'EOF'
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: ${LONGHORN_NS}
|
||||
labels:
|
||||
pod-security.kubernetes.io/enforce: privileged
|
||||
pod-security.kubernetes.io/audit: privileged
|
||||
pod-security.kubernetes.io/warn: privileged
|
||||
EOF`);
|
||||
|
||||
if (opts.backup !== undefined) {
|
||||
// Secret BEFORE the chart: Longhorn reads it by name from a defaultSetting,
|
||||
// and a missing secret makes the first backup fail with a credential error
|
||||
// that reads like a network problem.
|
||||
//
|
||||
// AWS_ENDPOINTS is what makes this work against a non-AWS gateway; without
|
||||
// it the driver resolves real AWS endpoints.
|
||||
log(`[longhorn] configuring backup target ${opts.backup.target}`);
|
||||
// tryNodeExec, not nodeExec: nodeExec throws with the whole command in the
|
||||
// message, and this command contains the S3 secret. A credential landing in
|
||||
// a log that then gets pasted into drill evidence is not acceptable, so the
|
||||
// failure is reported without the payload.
|
||||
const secretApplied = tryNodeExec(h, server, `k3s kubectl -n ${LONGHORN_NS} apply -f - <<'EOF'
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: ${BACKUP_SECRET}
|
||||
namespace: ${LONGHORN_NS}
|
||||
stringData:
|
||||
AWS_ACCESS_KEY_ID: ${opts.backup.accessKey}
|
||||
AWS_SECRET_ACCESS_KEY: ${opts.backup.secretKey}
|
||||
AWS_ENDPOINTS: ${opts.backup.endpoint}
|
||||
EOF`);
|
||||
if (secretApplied.exitCode !== 0) {
|
||||
throw new Error(
|
||||
`[${server.name}] could not create the ${BACKUP_SECRET} secret ` +
|
||||
`(exit ${secretApplied.exitCode}). Command withheld: it contains credentials. ` +
|
||||
`stderr: ${secretApplied.stderr.trim().slice(0, 200)}`);
|
||||
}
|
||||
}
|
||||
|
||||
log(`[longhorn] installing chart ${LONGHORN_VERSION} via k3s helm controller`);
|
||||
nodeExec(h, server, `cat > /var/lib/rancher/k3s/server/manifests/longhorn.yaml <<'EOF'
|
||||
apiVersion: helm.cattle.io/v1
|
||||
kind: HelmChart
|
||||
metadata:
|
||||
name: longhorn
|
||||
namespace: kube-system
|
||||
spec:
|
||||
chart: longhorn
|
||||
repo: https://charts.longhorn.io
|
||||
version: ${LONGHORN_VERSION}
|
||||
targetNamespace: ${LONGHORN_NS}
|
||||
valuesContent: |-
|
||||
defaultSettings:
|
||||
defaultDataPath: /var/lib/longhorn
|
||||
defaultReplicaCount: ${replicas}
|
||||
createDefaultDiskLabeledNodes: true
|
||||
defaultDataLocality: best-effort
|
||||
replicaSoftAntiAffinity: false
|
||||
replicaReplenishmentWaitInterval: 600
|
||||
orphanAutoDeletion: true
|
||||
persistence:
|
||||
defaultClass: true
|
||||
defaultClassReplicaCount: ${replicas}
|
||||
defaultFsType: ext4
|
||||
longhornUI:
|
||||
replicas: 1
|
||||
EOF`);
|
||||
}
|
||||
|
||||
/**
|
||||
* Every expected node has a Longhorn disk that is actually schedulable.
|
||||
*
|
||||
* Takes the VMs, not names, and resolves each one's Kubernetes name itself.
|
||||
* Longhorn keys its node records by the FQDN the kubelet registered
|
||||
* (`drill-s1.pxe-test.local`), never the libvirt domain name (`drill-s1`), so
|
||||
* a caller passing VM names matches nothing and waits out the full timeout
|
||||
* while the cluster is in fact perfectly healthy.
|
||||
*/
|
||||
export async function waitForLonghornReady(
|
||||
h: Harness,
|
||||
server: ProvisionedVm,
|
||||
vms: ProvisionedVm[],
|
||||
timeoutMs = 20 * 60_000,
|
||||
): Promise<void> {
|
||||
const expectedNodes = vms.map((vm) => k8sNodeName(h, vm));
|
||||
const start = Date.now();
|
||||
let last = "";
|
||||
while (Date.now() - start < timeoutMs) {
|
||||
const r = tryNodeExec(h, server,
|
||||
`k3s kubectl -n ${LONGHORN_NS} get nodes.longhorn.io -o json`, 60_000);
|
||||
if (r.exitCode === 0) {
|
||||
try {
|
||||
const parsed = JSON.parse(r.stdout) as { items: {
|
||||
metadata: { name: string };
|
||||
status?: { diskStatus?: Record<string, { conditions?: { type: string; status: string }[] }> };
|
||||
}[] };
|
||||
const ready = parsed.items.filter((n) => {
|
||||
const disks = Object.values(n.status?.diskStatus ?? {});
|
||||
return disks.length > 0 && disks.some((d) =>
|
||||
(d.conditions ?? []).some((c) => c.type === "Schedulable" && c.status === "True"));
|
||||
}).map((n) => n.metadata.name);
|
||||
|
||||
last = `${ready.length}/${expectedNodes.length} with schedulable disk: ${ready.join(",") || "none"}`;
|
||||
if (expectedNodes.every((n) => ready.includes(n))) {
|
||||
log(`[longhorn] ready after ${Math.round((Date.now() - start) / 1000)}s — ${last}`);
|
||||
return;
|
||||
}
|
||||
} catch { /* API not serving the CRD yet */ }
|
||||
}
|
||||
await sleep(15_000);
|
||||
}
|
||||
|
||||
// Say WHY, per node, instead of just "not ready". A node with no disk has a
|
||||
// specific cause and the record knows it; without this the only way to find
|
||||
// out is to catch the cluster before teardown, which is a race you lose.
|
||||
const detail: string[] = [];
|
||||
try {
|
||||
const raw = kubectlJson<{ items: {
|
||||
metadata: { name: string; annotations?: Record<string, string>; labels?: Record<string, string> };
|
||||
spec?: { disks?: Record<string, unknown>; allowScheduling?: boolean };
|
||||
status?: {
|
||||
conditions?: { type: string; status: string; reason?: string; message?: string }[];
|
||||
diskStatus?: Record<string, { conditions?: { type: string; status: string; reason?: string; message?: string }[] }>;
|
||||
};
|
||||
}[] }>(h, server, `-n ${LONGHORN_NS} get nodes.longhorn.io`);
|
||||
|
||||
for (const n of raw.items) {
|
||||
const disks = Object.keys(n.spec?.disks ?? {});
|
||||
const conds = (n.status?.conditions ?? [])
|
||||
.filter((c) => c.status !== "True")
|
||||
.map((c) => `${c.type}=${c.status}(${c.reason ?? ""}: ${(c.message ?? "").slice(0, 80)})`);
|
||||
const diskConds = Object.entries(n.status?.diskStatus ?? {}).flatMap(([dn, d]) =>
|
||||
(d.conditions ?? []).filter((c) => c.status !== "True")
|
||||
.map((c) => `${dn}:${c.type}=${c.status}(${c.reason ?? ""}: ${(c.message ?? "").slice(0, 80)})`));
|
||||
const ann = n.metadata.annotations?.["node.longhorn.io/default-disks-config"] ?? "<absent>";
|
||||
const lbl = n.metadata.labels?.["node.longhorn.io/create-default-disk"] ?? "<absent>";
|
||||
detail.push(
|
||||
` ${n.metadata.name}: disks=[${disks.join(",")}] label=${lbl} ann=${ann} ` +
|
||||
`${conds.length > 0 ? `nodeConds=${conds.join(";")} ` : ""}` +
|
||||
`${diskConds.length > 0 ? `diskConds=${diskConds.join(";")}` : ""}`);
|
||||
}
|
||||
} catch (e) {
|
||||
detail.push(` (could not read Longhorn nodes: ${e instanceof Error ? e.message : String(e)})`);
|
||||
}
|
||||
|
||||
throw new Error(
|
||||
`Longhorn not ready in ${timeoutMs / 60000} min (${last}).\n` +
|
||||
`A node with no disk usually means the create-default-disk label or the\n` +
|
||||
`default-disks-config annotation is missing, or the path is not mounted.\n` +
|
||||
detail.join("\n"));
|
||||
}
|
||||
|
||||
/** Create a PVC plus a Deployment that mounts it at /data. */
|
||||
export function createDataWorkload(h: Harness, server: ProvisionedVm, opts: {
|
||||
name: string;
|
||||
sizeGi: number;
|
||||
replicas: number;
|
||||
}): void {
|
||||
log(`[longhorn] creating ${opts.name} (${opts.sizeGi}Gi, ${opts.replicas} replicas)`);
|
||||
nodeExec(h, server, `k3s kubectl apply -f - <<'EOF'
|
||||
apiVersion: storage.k8s.io/v1
|
||||
kind: StorageClass
|
||||
metadata:
|
||||
name: lh-${opts.name}
|
||||
provisioner: driver.longhorn.io
|
||||
allowVolumeExpansion: true
|
||||
parameters:
|
||||
numberOfReplicas: "${opts.replicas}"
|
||||
staleReplicaTimeout: "30"
|
||||
fsType: ext4
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: ${opts.name}
|
||||
spec:
|
||||
accessModes: [ReadWriteOnce]
|
||||
storageClassName: lh-${opts.name}
|
||||
resources:
|
||||
requests:
|
||||
storage: ${opts.sizeGi}Gi
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: ${opts.name}
|
||||
spec:
|
||||
replicas: 1
|
||||
strategy:
|
||||
type: Recreate
|
||||
selector:
|
||||
matchLabels: {app: ${opts.name}}
|
||||
template:
|
||||
metadata:
|
||||
labels: {app: ${opts.name}}
|
||||
spec:
|
||||
# Pod Security Admission enforces "restricted" on this namespace (the k3s
|
||||
# hardening group applies the label), so a bare pod spec is REJECTED
|
||||
# outright -- the Deployment reports 0/1 with no pod and no obvious
|
||||
# reason unless you read the ReplicaSet events. Every one of these
|
||||
# fields is required by "restricted"; dropping any brings it back.
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 1000
|
||||
runAsGroup: 1000
|
||||
# fsGroup so the non-root user can actually write to the Longhorn
|
||||
# volume, which is otherwise owned by root.
|
||||
fsGroup: 1000
|
||||
seccompProfile: {type: RuntimeDefault}
|
||||
containers:
|
||||
- name: shell
|
||||
image: busybox:1.36
|
||||
command: ["sh", "-c", "sleep infinity"]
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
capabilities: {drop: ["ALL"]}
|
||||
volumeMounts:
|
||||
- {name: data, mountPath: /data}
|
||||
volumes:
|
||||
- name: data
|
||||
persistentVolumeClaim: {claimName: ${opts.name}}
|
||||
EOF`);
|
||||
}
|
||||
|
||||
/** Wait for the workload's pod to be Running and its volume attached. */
|
||||
export async function waitForWorkload(
|
||||
h: Harness, server: ProvisionedVm, name: string, timeoutMs = 10 * 60_000,
|
||||
): Promise<void> {
|
||||
const start = Date.now();
|
||||
let last = "";
|
||||
while (Date.now() - start < timeoutMs) {
|
||||
const r = tryNodeExec(h, server,
|
||||
`k3s kubectl get pods -l app=${name} -o jsonpath='{.items[*].status.phase}'`);
|
||||
last = r.stdout.trim();
|
||||
if (last === "Running") {
|
||||
// Ready, not merely scheduled: the volume mount happens before Ready.
|
||||
const ready = tryNodeExec(h, server,
|
||||
`k3s kubectl get pods -l app=${name} -o jsonpath='{.items[*].status.containerStatuses[*].ready}'`);
|
||||
if (ready.stdout.trim() === "true") return;
|
||||
}
|
||||
await sleep(5_000);
|
||||
}
|
||||
throw new Error(`workload ${name} not running in ${timeoutMs / 60000} min (last phase: ${last || "none"})`);
|
||||
}
|
||||
|
||||
function podOf(h: Harness, server: ProvisionedVm, name: string): string {
|
||||
const pod = kubectlOn(h, server, `get pods -l app=${name} -o jsonpath='{.items[0].metadata.name}'`).trim();
|
||||
if (!pod) throw new Error(`no pod for workload ${name}`);
|
||||
return pod;
|
||||
}
|
||||
|
||||
/**
|
||||
* Write a random blob, fsync it, and record its digest.
|
||||
*
|
||||
* conv=fsync matters: without it the write can still be in the guest's page
|
||||
* cache, and a checksum taken immediately afterwards proves nothing about what
|
||||
* reached the replicas.
|
||||
*/
|
||||
export async function writeBlob(h: Harness, server: ProvisionedVm, opts: {
|
||||
name: string; mib: number;
|
||||
}): Promise<string> {
|
||||
const pod = podOf(h, server, opts.name);
|
||||
log(`[longhorn] writing ${opts.mib} MiB to ${opts.name}`);
|
||||
nodeExec(h, server,
|
||||
`k3s kubectl exec ${pod} -- sh -c ` +
|
||||
`"dd if=/dev/urandom of=/data/blob bs=1M count=${opts.mib} conv=fsync 2>/dev/null && ` +
|
||||
`cd /data && sha256sum blob > blob.sha256 && sync"`,
|
||||
15 * 60_000);
|
||||
|
||||
const digest = kubectlOn(h, server, `exec ${pod} -- cat /data/blob.sha256`).trim().split(/\s+/)[0] ?? "";
|
||||
if (!/^[0-9a-f]{64}$/.test(digest)) throw new Error(`bad digest for ${opts.name}: ${digest}`);
|
||||
log(`[longhorn] ${opts.name} digest ${digest.slice(0, 16)}...`);
|
||||
return digest;
|
||||
}
|
||||
|
||||
/**
|
||||
* Restart the pod, then verify the blob.
|
||||
*
|
||||
* The restart is the point. It detaches and reattaches the volume, so a pass
|
||||
* means the bytes came back from Longhorn rather than from wherever the writer
|
||||
* left them.
|
||||
*/
|
||||
export async function verifyBlob(h: Harness, server: ProvisionedVm, opts: {
|
||||
name: string; expectDigest: string;
|
||||
}): Promise<void> {
|
||||
const before = podOf(h, server, opts.name);
|
||||
kubectlOn(h, server, `delete pod ${before} --wait=true --timeout=5m`, 6 * 60_000);
|
||||
await waitForWorkload(h, server, opts.name);
|
||||
const after = podOf(h, server, opts.name);
|
||||
if (after === before) throw new Error(`${opts.name}: pod was not actually replaced`);
|
||||
|
||||
// sha256sum -c proves the file matches what was recorded IN the volume...
|
||||
const check = tryNodeExec(h, server,
|
||||
`k3s kubectl exec ${after} -- sh -c "cd /data && sha256sum -c blob.sha256"`, 15 * 60_000);
|
||||
if (check.exitCode !== 0) {
|
||||
throw new Error(`${opts.name}: DATA CORRUPTION — sha256sum -c failed: ${check.stdout}${check.stderr}`);
|
||||
}
|
||||
|
||||
// ...and this proves the recorded digest is still the one we started with,
|
||||
// which -c alone would not catch if both file and checksum were replaced.
|
||||
const actual = kubectlOn(h, server,
|
||||
`exec ${after} -- sh -c "cd /data && sha256sum blob"`, 15 * 60_000).trim().split(/\s+/)[0];
|
||||
if (actual !== opts.expectDigest) {
|
||||
throw new Error(`${opts.name}: DIGEST CHANGED — expected ${opts.expectDigest}, got ${actual}`);
|
||||
}
|
||||
log(`[longhorn] ${opts.name} verified intact across a detach/attach (pod ${before} -> ${after})`);
|
||||
}
|
||||
|
||||
export interface VolumeState {
|
||||
pv: string;
|
||||
state: string;
|
||||
robustness: string;
|
||||
wanted: number;
|
||||
replicaNodes: string[];
|
||||
healthyReplicas: number;
|
||||
}
|
||||
|
||||
/** Longhorn's view of the volume backing a PVC. */
|
||||
export function volumeState(h: Harness, server: ProvisionedVm, pvcName: string): VolumeState {
|
||||
const pv = kubectlOn(h, server, `get pvc ${pvcName} -o jsonpath='{.spec.volumeName}'`).trim();
|
||||
if (!pv) throw new Error(`PVC ${pvcName} has no bound volume`);
|
||||
|
||||
const vol = kubectlJson<{
|
||||
spec: { numberOfReplicas?: number };
|
||||
status?: { state?: string; robustness?: string };
|
||||
}>(h, server, `-n ${LONGHORN_NS} get volumes.longhorn.io ${pv}`);
|
||||
|
||||
const reps = kubectlJson<{ items: {
|
||||
spec: { nodeID?: string; volumeName?: string };
|
||||
status?: { currentState?: string; started?: boolean };
|
||||
}[] }>(h, server, `-n ${LONGHORN_NS} get replicas.longhorn.io`);
|
||||
|
||||
const mine = reps.items.filter((r) => r.spec.volumeName === pv);
|
||||
return {
|
||||
pv,
|
||||
state: vol.status?.state ?? "unknown",
|
||||
robustness: vol.status?.robustness ?? "unknown",
|
||||
wanted: vol.spec.numberOfReplicas ?? 0,
|
||||
replicaNodes: mine.map((r) => r.spec.nodeID ?? "").filter(Boolean).sort(),
|
||||
healthyReplicas: mine.filter((r) => r.status?.currentState === "running").length,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Wait for a volume to be healthy on the expected number of distinct nodes.
|
||||
*
|
||||
* Logs rebuild progress while waiting, so a slow rebuild reads as progress
|
||||
* rather than as an opaque hour of silence.
|
||||
*/
|
||||
export async function waitForVolumeHealthy(h: Harness, server: ProvisionedVm, opts: {
|
||||
pvc: string; expectReplicas: number; timeoutMs?: number;
|
||||
}): Promise<VolumeState> {
|
||||
const timeoutMs = opts.timeoutMs ?? 60 * 60_000;
|
||||
const start = Date.now();
|
||||
let last = "";
|
||||
while (Date.now() - start < timeoutMs) {
|
||||
let st: VolumeState | null = null;
|
||||
try { st = volumeState(h, server, opts.pvc); } catch { /* transient */ }
|
||||
|
||||
if (st) {
|
||||
const distinct = new Set(st.replicaNodes).size;
|
||||
last = `${st.state}/${st.robustness} on ${distinct} node(s): ${st.replicaNodes.join(",")}`;
|
||||
if (st.robustness === "healthy" && distinct >= opts.expectReplicas) {
|
||||
log(`[longhorn] ${opts.pvc} healthy on ${distinct} nodes after ${Math.round((Date.now() - start) / 1000)}s`);
|
||||
return st;
|
||||
}
|
||||
const prog = tryNodeExec(h, server,
|
||||
`k3s kubectl -n ${LONGHORN_NS} get engines.longhorn.io -o jsonpath=` +
|
||||
`'{range .items[*]}{.spec.volumeName}{" "}{.status.rebuildStatus}{"\\n"}{end}'`);
|
||||
const line = prog.stdout.split("\n").find((l) => l.includes(st!.pv) && l.trim() !== st!.pv);
|
||||
log(`[longhorn] ${opts.pvc}: ${last}${line ? ` | rebuild ${line.replace(st.pv, "").trim()}` : ""}`);
|
||||
}
|
||||
await sleep(20_000);
|
||||
}
|
||||
throw new Error(`${opts.pvc} not healthy on ${opts.expectReplicas} nodes in ${timeoutMs / 60000} min (last: ${last})`);
|
||||
}
|
||||
|
||||
/**
|
||||
* Wait until Longhorn reports the backup target usable.
|
||||
*
|
||||
* `available: true` is the appliance saying it could reach the endpoint AND
|
||||
* authenticate. Backing up without checking this first produces a failure that
|
||||
* looks like a volume problem.
|
||||
*/
|
||||
export async function setBackupTarget(
|
||||
h: Harness, server: ProvisionedVm, backup: BackupTargetOpts, timeoutMs = 5 * 60_000,
|
||||
): Promise<void> {
|
||||
// PATCH the CR; do NOT set defaultSettings.backupTarget.
|
||||
//
|
||||
// In Longhorn 1.8 the backup target moved out of settings and onto
|
||||
// `backuptargets.longhorn.io/default`. The helm value is SILENTLY IGNORED --
|
||||
// no `backup-target` Setting object is created at all, the default
|
||||
// BackupTarget keeps an empty URL, and everything reports success. Observed
|
||||
// on a fresh 1.8.1 install here and confirmed against the live fleet.
|
||||
const deadline = Date.now() + timeoutMs;
|
||||
while (Date.now() < deadline) {
|
||||
const exists = tryNodeExec(h, server,
|
||||
`k3s kubectl -n ${LONGHORN_NS} get backuptargets.longhorn.io default -o name`);
|
||||
if (exists.exitCode === 0 && exists.stdout.trim() !== "") break;
|
||||
await sleep(5_000);
|
||||
}
|
||||
|
||||
const patch = JSON.stringify({
|
||||
spec: { backupTargetURL: backup.target, credentialSecret: BACKUP_SECRET, pollInterval: "5m0s" },
|
||||
});
|
||||
nodeExec(h, server,
|
||||
`k3s kubectl -n ${LONGHORN_NS} patch backuptargets.longhorn.io default --type=merge -p '${patch}'`);
|
||||
log(`[longhorn] backup target set to ${backup.target}`);
|
||||
}
|
||||
|
||||
export async function waitForBackupTarget(
|
||||
h: Harness, server: ProvisionedVm, timeoutMs = 5 * 60_000,
|
||||
): Promise<void> {
|
||||
const start = Date.now();
|
||||
let last = "";
|
||||
while (Date.now() - start < timeoutMs) {
|
||||
const r = tryNodeExec(h, server,
|
||||
`k3s kubectl -n ${LONGHORN_NS} get backuptargets.longhorn.io default ` +
|
||||
`-o jsonpath='{.spec.backupTargetURL}|{.status.available}|{.status.conditions[*].message}'`);
|
||||
last = r.stdout.trim();
|
||||
if (r.exitCode === 0 && last.includes("|true|")) {
|
||||
log(`[longhorn] backup target available after ${Math.round((Date.now() - start) / 1000)}s`);
|
||||
return;
|
||||
}
|
||||
await sleep(10_000);
|
||||
}
|
||||
throw new Error(
|
||||
`Backup target not available in ${timeoutMs / 60000} min (url|available|conditions = ${last}). ` +
|
||||
`Longhorn could not reach or authenticate to the endpoint.`);
|
||||
}
|
||||
|
||||
/**
|
||||
* Snapshot a volume and back it up, returning the backup's name.
|
||||
*
|
||||
* Longhorn's own CRs, not the CSI VolumeSnapshot API -- this cluster has no
|
||||
* snapshot.storage.k8s.io resources at all, so VolumeSnapshot/VolumeSnapshotClass
|
||||
* simply do not exist here. Any backup tooling has to use these.
|
||||
*/
|
||||
export async function backupVolume(
|
||||
h: Harness, server: ProvisionedVm, pvc: string, timeoutMs = 20 * 60_000,
|
||||
): Promise<{ backup: string; volume: string }> {
|
||||
const pv = volumeState(h, server, pvc).pv;
|
||||
const snapName = `drill-snap-${Date.now().toString(36)}`;
|
||||
|
||||
log(`[longhorn] snapshotting ${pvc} (${pv})`);
|
||||
nodeExec(h, server, `k3s kubectl -n ${LONGHORN_NS} apply -f - <<'EOF'
|
||||
apiVersion: longhorn.io/v1beta2
|
||||
kind: Snapshot
|
||||
metadata:
|
||||
name: ${snapName}
|
||||
namespace: ${LONGHORN_NS}
|
||||
spec:
|
||||
volume: ${pv}
|
||||
createSnapshot: true
|
||||
EOF`);
|
||||
|
||||
// The snapshot has to exist before a backup can reference it.
|
||||
const snapDeadline = Date.now() + 5 * 60_000;
|
||||
while (Date.now() < snapDeadline) {
|
||||
const r = tryNodeExec(h, server,
|
||||
`k3s kubectl -n ${LONGHORN_NS} get snapshots.longhorn.io ${snapName} -o jsonpath='{.status.readyToUse}'`);
|
||||
if (r.stdout.trim() === "true") break;
|
||||
await sleep(5_000);
|
||||
}
|
||||
|
||||
const backupName = `drill-bak-${Date.now().toString(36)}`;
|
||||
log(`[longhorn] backing up ${pvc} to the backup target`);
|
||||
nodeExec(h, server, `k3s kubectl -n ${LONGHORN_NS} apply -f - <<'EOF'
|
||||
apiVersion: longhorn.io/v1beta2
|
||||
kind: Backup
|
||||
metadata:
|
||||
name: ${backupName}
|
||||
namespace: ${LONGHORN_NS}
|
||||
labels:
|
||||
backup-volume: ${pv}
|
||||
spec:
|
||||
snapshotName: ${snapName}
|
||||
EOF`);
|
||||
|
||||
const start = Date.now();
|
||||
let last = "";
|
||||
while (Date.now() - start < timeoutMs) {
|
||||
const r = tryNodeExec(h, server,
|
||||
`k3s kubectl -n ${LONGHORN_NS} get backups.longhorn.io ${backupName} ` +
|
||||
`-o jsonpath='{.status.state}|{.status.progress}|{.status.error}'`);
|
||||
last = r.stdout.trim();
|
||||
if (last.startsWith("Completed")) {
|
||||
log(`[longhorn] backup ${backupName} completed after ${Math.round((Date.now() - start) / 1000)}s`);
|
||||
return { backup: backupName, volume: pv };
|
||||
}
|
||||
if (last.includes("Error")) {
|
||||
throw new Error(`Backup ${backupName} failed: ${last}`);
|
||||
}
|
||||
await sleep(10_000);
|
||||
}
|
||||
throw new Error(`Backup ${backupName} did not complete in ${timeoutMs / 60000} min (state|progress|error = ${last})`);
|
||||
}
|
||||
|
||||
/**
|
||||
* Restore a backup into a brand-new PVC and workload.
|
||||
*
|
||||
* Uses the `fromBackup` StorageClass parameter, which is the supported route
|
||||
* when the CSI VolumeSnapshot API is unavailable. The restored volume is a
|
||||
* separate volume with its own replicas -- nothing about the original is
|
||||
* touched, so a failed restore cannot damage the source.
|
||||
*/
|
||||
export async function restoreFromBackup(h: Harness, server: ProvisionedVm, opts: {
|
||||
backup: string;
|
||||
name: string;
|
||||
sizeGi: number;
|
||||
}): Promise<void> {
|
||||
const url = nodeExec(h, server,
|
||||
`k3s kubectl -n ${LONGHORN_NS} get backups.longhorn.io ${opts.backup} -o jsonpath='{.status.url}'`).trim();
|
||||
if (url === "") throw new Error(`Backup ${opts.backup} has no status.url; cannot restore from it`);
|
||||
log(`[longhorn] restoring ${opts.backup} into ${opts.name}`);
|
||||
|
||||
nodeExec(h, server, `k3s kubectl apply -f - <<'EOF'
|
||||
apiVersion: storage.k8s.io/v1
|
||||
kind: StorageClass
|
||||
metadata:
|
||||
name: lh-${opts.name}
|
||||
provisioner: driver.longhorn.io
|
||||
allowVolumeExpansion: true
|
||||
parameters:
|
||||
numberOfReplicas: "2"
|
||||
staleReplicaTimeout: "30"
|
||||
fsType: ext4
|
||||
fromBackup: "${url}"
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: ${opts.name}
|
||||
spec:
|
||||
accessModes: [ReadWriteOnce]
|
||||
storageClassName: lh-${opts.name}
|
||||
resources:
|
||||
requests:
|
||||
storage: ${opts.sizeGi}Gi
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: ${opts.name}
|
||||
spec:
|
||||
replicas: 1
|
||||
strategy:
|
||||
type: Recreate
|
||||
selector:
|
||||
matchLabels: {app: ${opts.name}}
|
||||
template:
|
||||
metadata:
|
||||
labels: {app: ${opts.name}}
|
||||
spec:
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 1000
|
||||
runAsGroup: 1000
|
||||
fsGroup: 1000
|
||||
seccompProfile: {type: RuntimeDefault}
|
||||
containers:
|
||||
- name: shell
|
||||
image: busybox:1.36
|
||||
command: ["sh", "-c", "sleep infinity"]
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
capabilities: {drop: ["ALL"]}
|
||||
volumeMounts:
|
||||
- {name: data, mountPath: /data}
|
||||
volumes:
|
||||
- name: data
|
||||
persistentVolumeClaim: {claimName: ${opts.name}}
|
||||
EOF`);
|
||||
}
|
||||
|
||||
/**
|
||||
* Read a blob's digest from a workload, without restarting it.
|
||||
*
|
||||
* Used on a RESTORED volume, where the point is to compare bytes against the
|
||||
* original's recorded digest rather than to prove durability again.
|
||||
*/
|
||||
export function readBlobDigest(h: Harness, server: ProvisionedVm, name: string): string {
|
||||
const pod = kubectlOn(h, server, `get pods -l app=${name} -o jsonpath='{.items[0].metadata.name}'`).trim();
|
||||
if (pod === "") throw new Error(`no pod for workload ${name}`);
|
||||
const out = kubectlOn(h, server,
|
||||
`exec ${pod} -- sh -c "cd /data && sha256sum blob"`, 15 * 60_000).trim();
|
||||
return out.split(/\s+/)[0] ?? "";
|
||||
}
|
||||
534
bastion/tests/integration/helpers/provision-harness.ts
Normal file
534
bastion/tests/integration/helpers/provision-harness.ts
Normal file
@@ -0,0 +1,534 @@
|
||||
// Reusable bastion + VM provisioning harness for partition-layout tests.
|
||||
//
|
||||
// pxe-provision.test.ts grew a ~200-line beforeAll that stands up a real
|
||||
// bastion, caches Fedora boot artifacts, and drives one VM through discovery
|
||||
// and install. Testing a partition MATRIX needs that same sequence for several
|
||||
// VMs and several roles, so it lives here instead of being copy-pasted per
|
||||
// topology.
|
||||
|
||||
import { existsSync, mkdirSync, copyFileSync, symlinkSync, writeFileSync } from "node:fs";
|
||||
import { execSync } from "node:child_process";
|
||||
import { join } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
import type { BastionConfig } from "../../../src/shared/src/types/config.js";
|
||||
import type { Role } from "../../../src/shared/src/types/state.js";
|
||||
import { log, waitForSsh } from "./libvirt.js";
|
||||
import { ensurePxeNetwork, deleteNftablesRejectRules, PXE_NETWORK_NAME, PXE_BRIDGE, PXE_GATEWAY, PXE_SUBNET } from "./pxe-network.js";
|
||||
import { createPxeVm, destroyPxeVm, getVmMac, rebootPxeVm } from "./pxe-vm.js";
|
||||
import { sshExec } from "./ssh.js";
|
||||
|
||||
export const DISCOVERY_TIMEOUT_MS = 6 * 60_000;
|
||||
export const INSTALL_TIMEOUT_MS = 60 * 60_000;
|
||||
export const SSH_TIMEOUT_MS = 12 * 60_000;
|
||||
|
||||
export const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));
|
||||
|
||||
export interface Harness {
|
||||
config: BastionConfig;
|
||||
close: () => Promise<void>;
|
||||
httpPort: number;
|
||||
sshUser: string;
|
||||
sshKeyPath: string;
|
||||
}
|
||||
|
||||
async function pollUntil<T>(url: string, check: (d: T) => boolean, timeoutMs: number, intervalMs = 5000): Promise<T> {
|
||||
const start = Date.now();
|
||||
let lastErr = "";
|
||||
while (Date.now() - start < timeoutMs) {
|
||||
try {
|
||||
const res = await fetch(url);
|
||||
if (res.ok) {
|
||||
const data = (await res.json()) as T;
|
||||
if (check(data)) return data;
|
||||
}
|
||||
} catch (e) { lastErr = e instanceof Error ? e.message : String(e); }
|
||||
await sleep(intervalMs);
|
||||
}
|
||||
throw new Error(`Timeout after ${timeoutMs}ms polling ${url}${lastErr ? ` (last error: ${lastErr})` : ""}`);
|
||||
}
|
||||
|
||||
/**
|
||||
* Stand up a real bastion on the isolated PXE network.
|
||||
*
|
||||
* FEDORA_MIRROR is honoured (config.ts now reads it) -- point it at a local
|
||||
* mirror before running a multi-VM matrix, or every install pulls the full
|
||||
* package set from download.fedoraproject.org.
|
||||
*/
|
||||
export async function startTestBastion(opts: {
|
||||
httpPort: number;
|
||||
sshUser: string;
|
||||
sshPubKey: string;
|
||||
sshKeyPath: string;
|
||||
label: string;
|
||||
/**
|
||||
* Leave the post-provision k3s auto-install ON (the default) or suppress it.
|
||||
*
|
||||
* Any test that forms a cluster ITSELF must pass false. The auto-install runs
|
||||
* a bare `INSTALL_K3S_EXEC=server|agent` with no server URL and no token, so
|
||||
* three combo nodes become three separate single-node clusters -- and it
|
||||
* stops k3s and deletes CNI state minutes after an install lands, landing on
|
||||
* top of whatever the test is doing by then.
|
||||
*/
|
||||
autoK3s?: boolean;
|
||||
}): Promise<Harness> {
|
||||
ensurePxeNetwork();
|
||||
|
||||
const testDir = join(tmpdir(), `lab-${opts.label}-${Date.now()}`);
|
||||
for (const sub of ["tftp", "http", "logs"]) mkdirSync(join(testDir, sub), { recursive: true });
|
||||
|
||||
const { createApp } = await import("../../../src/bastion/src/server.js");
|
||||
const { loadConfig } = await import("../../../src/bastion/src/config.js");
|
||||
const { generateDnsmasqConf, startDnsmasq, stopDnsmasq } = await import("../../../src/bastion/src/services/dnsmasq.js");
|
||||
const { generateDiscoverKickstart } = await import("../../../src/bastion/src/services/kickstart-generator.js");
|
||||
const { renderBootIpxe } = await import("../../../src/bastion/src/templates/boot.ipxe.js");
|
||||
|
||||
const config = loadConfig({
|
||||
bastionDir: testDir,
|
||||
httpPort: opts.httpPort,
|
||||
iface: "virbr-pxe",
|
||||
serverIp: PXE_GATEWAY,
|
||||
network: `${PXE_SUBNET}.0`,
|
||||
gateway: PXE_GATEWAY,
|
||||
dhcpMode: "full",
|
||||
dhcpRangeStart: `${PXE_SUBNET}.100`,
|
||||
dhcpRangeEnd: `${PXE_SUBNET}.200`,
|
||||
domain: "pxe-test.local",
|
||||
sshKeys: [opts.sshPubKey],
|
||||
adminUser: opts.sshUser,
|
||||
...(opts.autoK3s === undefined ? {} : { autoK3s: opts.autoK3s }),
|
||||
});
|
||||
|
||||
const ipxeSrc = "/usr/share/ipxe/ipxe-snponly-x86_64.efi";
|
||||
if (!existsSync(ipxeSrc)) throw new Error(`iPXE not found: ${ipxeSrc}. Install: sudo dnf install ipxe-bootimgs-x86`);
|
||||
const ipxeDest = join(config.tftpDir, "ipxe.efi");
|
||||
copyFileSync(ipxeSrc, ipxeDest);
|
||||
|
||||
const cacheDir = "/var/lib/libvirt/images/lab-pxe-cache";
|
||||
execSync(`mkdir -p "${cacheDir}"`, { stdio: "pipe" });
|
||||
const kernel = join(cacheDir, `vmlinuz-${config.fedoraVersion}`);
|
||||
const initrd = join(cacheDir, `initrd-${config.fedoraVersion}.img`);
|
||||
if (!existsSync(kernel)) {
|
||||
log(`Downloading Fedora ${config.fedoraVersion} kernel from ${config.fedoraMirror}...`);
|
||||
execSync(`curl -sS -L -f -o "${kernel}" "${config.fedoraMirror}/images/pxeboot/vmlinuz"`, { stdio: "inherit", timeout: 300_000 });
|
||||
}
|
||||
if (!existsSync(initrd)) {
|
||||
log(`Downloading Fedora ${config.fedoraVersion} initrd...`);
|
||||
execSync(`curl -sS -L -f -o "${initrd}" "${config.fedoraMirror}/images/pxeboot/initrd.img"`, { stdio: "inherit", timeout: 300_000 });
|
||||
}
|
||||
copyFileSync(kernel, join(config.httpDir, "vmlinuz"));
|
||||
copyFileSync(initrd, join(config.httpDir, "initrd.img"));
|
||||
try { symlinkSync(ipxeDest, join(config.httpDir, "ipxe.efi")); } catch { /* exists */ }
|
||||
|
||||
writeFileSync(join(config.httpDir, "discover.ks"), generateDiscoverKickstart(config));
|
||||
writeFileSync(join(config.httpDir, "boot.ipxe"), renderBootIpxe({ serverIp: config.serverIp, httpPort: config.httpPort }));
|
||||
generateDnsmasqConf(config);
|
||||
|
||||
// Kill dnsmasq processes left behind by EARLIER TEST RUNS.
|
||||
//
|
||||
// startTestBastion's close() shuts down the HTTP app but never stopped
|
||||
// dnsmasq, so every run leaked one. A leaked dnsmasq keeps :67 on the bridge
|
||||
// and keeps answering DHCP -- with the PREVIOUS run's config, pointing PXE
|
||||
// clients at a bastion port that is no longer listening. The next run then
|
||||
// sees perfect DHCP (offer, request, ack, even the right bootfile name) and
|
||||
// no machine ever reaches it. That is not a failure mode anyone guesses from
|
||||
// "discovery timed out".
|
||||
killOrphanedDnsmasq();
|
||||
|
||||
const { app, syslog } = createApp(config);
|
||||
await app.listen({ port: config.httpPort, host: "0.0.0.0" });
|
||||
syslog.start();
|
||||
// NOT awaited, deliberately.
|
||||
//
|
||||
// startDnsmasq returns the execa child handle, and because the function is
|
||||
// `async` the promise it returns ADOPTS that thenable -- which settles only
|
||||
// when dnsmasq EXITS. With --no-daemon that is never, so `await` here hangs
|
||||
// the harness silently and forever, with the log frozen right after
|
||||
// "TFTP root is ...". (Learned by doing exactly that.) The rejection still
|
||||
// has to be handled or an early bind failure becomes an unhandled rejection.
|
||||
startDnsmasq(config).catch((err) => log(`dnsmasq: ${err instanceof Error ? err.message : String(err)}`));
|
||||
await sleep(1000);
|
||||
|
||||
// Prove DHCP is actually listening before handing the harness back.
|
||||
//
|
||||
// startDnsmasq's rejection is swallowed into a log line, so a dnsmasq that
|
||||
// failed to bind looks exactly like one that started. The symptom then
|
||||
// arrives six minutes later as four VMs timing out in discovery, which
|
||||
// reads as a PXE or kickstart problem rather than "there was never any
|
||||
// DHCP". Fail here, immediately, with the real reason.
|
||||
await assertDhcpListening(join(testDir, "dnsmasq.conf"));
|
||||
log(`Bastion up on :${opts.httpPort} (mirror: ${config.fedoraMirror}, autoK3s=${config.autoK3s})`);
|
||||
|
||||
return {
|
||||
config,
|
||||
httpPort: opts.httpPort,
|
||||
sshUser: opts.sshUser,
|
||||
sshKeyPath: opts.sshKeyPath,
|
||||
close: async () => {
|
||||
// Stop OUR dnsmasq. close() never used to, which is how a leaked one
|
||||
// ended up owning :67 and serving the PREVIOUS run's TFTP root and
|
||||
// bastion port to the NEXT run's VMs -- they booted toward a dead port
|
||||
// and simply never appeared, while DHCP looked perfect.
|
||||
try { stopDnsmasq(); } catch { /* already gone */ }
|
||||
await app.close();
|
||||
// Belt and braces: if that did not stop it, sweep by config path.
|
||||
killOrphanedDnsmasq();
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Kill any dnsmasq started by a previous (or current) lab test run.
|
||||
*
|
||||
* Matched on the config path, which is always /tmp/lab-<label>-<ts>/dnsmasq.conf
|
||||
* -- so this never touches libvirt's own dnsmasq for the default network, or
|
||||
* anything else on the host.
|
||||
*/
|
||||
function killOrphanedDnsmasq(): void {
|
||||
try {
|
||||
// Match on the process NAME first (pgrep -x), then inspect each one's
|
||||
// arguments. A `pgrep -f 'dnsmasq.*/tmp/lab-'` matches the shell running
|
||||
// that very pattern, so the naive version can kill its own shell -- and
|
||||
// libvirt's dnsmasq for the default network must never be touched either.
|
||||
const pids = execSync("pgrep -x dnsmasq 2>/dev/null || true", { encoding: "utf-8" })
|
||||
.trim().split("\n").filter((x) => /^\d+$/.test(x.trim()));
|
||||
let killed = 0;
|
||||
for (const pid of pids) {
|
||||
let args = "";
|
||||
try {
|
||||
args = execSync(`tr '\\0' ' ' < /proc/${pid.trim()}/cmdline 2>/dev/null || true`, { encoding: "utf-8" });
|
||||
} catch { continue; }
|
||||
if (!args.includes("--conf-file=/tmp/lab-")) continue;
|
||||
log(`Killing orphaned test dnsmasq (pid ${pid.trim()}) — it would answer DHCP with stale config`);
|
||||
try { execSync(`sudo kill -9 ${pid.trim()} 2>/dev/null || kill -9 ${pid.trim()} 2>/dev/null || true`); } catch { /* gone */ }
|
||||
killed++;
|
||||
}
|
||||
// Give the socket a moment to be released before we bind it.
|
||||
if (killed > 0) execSync("sleep 1");
|
||||
} catch { /* nothing to kill */ }
|
||||
}
|
||||
|
||||
/**
|
||||
* Is something serving DHCP on the PXE bridge?
|
||||
*
|
||||
* `ss -lunp` shows UDP listeners; dnsmasq binds port 67. Checked on the bridge
|
||||
* address specifically, because a dnsmasq bound to the wrong interface is the
|
||||
* failure being guarded against.
|
||||
*/
|
||||
async function assertDhcpListening(confPath: string, timeoutMs = 20_000): Promise<void> {
|
||||
const start = Date.now();
|
||||
let last = "";
|
||||
while (Date.now() - start < timeoutMs) {
|
||||
try {
|
||||
// Check for OUR dnsmasq by its config path, not merely for something on
|
||||
// :67. A leaked dnsmasq from a previous run also holds :67 and also
|
||||
// answers DHCP -- with the wrong bastion port -- so a port check alone
|
||||
// reports success for precisely the situation it is meant to catch.
|
||||
const pids = execSync("pgrep -x dnsmasq 2>/dev/null || true", { encoding: "utf-8" })
|
||||
.trim().split("\n").filter((x) => /^\d+$/.test(x.trim()));
|
||||
last = pids.filter((pid) => {
|
||||
try {
|
||||
return execSync(`tr '\\0' ' ' < /proc/${pid.trim()}/cmdline 2>/dev/null || true`, { encoding: "utf-8" })
|
||||
.includes(confPath);
|
||||
} catch { return false; }
|
||||
}).join(",");
|
||||
if (last !== "") {
|
||||
const ports = execSync("ss -lun 2>/dev/null || true", { encoding: "utf-8" });
|
||||
if (/:67\b/.test(ports)) {
|
||||
log("dnsmasq (ours) is serving DHCP on :67");
|
||||
return;
|
||||
}
|
||||
}
|
||||
} catch { /* retry */ }
|
||||
await sleep(1000);
|
||||
}
|
||||
throw new Error(
|
||||
`dnsmasq for ${confPath} is not serving DHCP after ${timeoutMs / 1000}s. ` +
|
||||
`It most likely failed to bind to ${PXE_BRIDGE} -- check the bastion log for ` +
|
||||
`"error binding DHCP socket", and check for a leaked dnsmasq from an earlier run ` +
|
||||
`(pgrep -af 'dnsmasq.*--conf-file=/tmp/lab-'). Without this every VM sits in ` +
|
||||
`discovery until it times out, six minutes later, looking like a PXE problem.`,
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Wait for the installer to actually start, rebooting into PXE if it does not.
|
||||
*
|
||||
* "Started" means the bastion has seen anything at all for this MAC beyond the
|
||||
* queued state -- a dispatch, a progress line, a status. A VM that never gets
|
||||
* that far is stuck before Anaconda, and only a reboot will shift it.
|
||||
*/
|
||||
async function waitForInstallStart(
|
||||
base: string,
|
||||
mac: string,
|
||||
name: string,
|
||||
attemptMs = 6 * 60_000,
|
||||
attempts = 3,
|
||||
): Promise<void> {
|
||||
for (let attempt = 1; attempt <= attempts; attempt++) {
|
||||
try {
|
||||
// NOT `status === "installing"`. A machine that has merely been QUEUED
|
||||
// already reports that -- the bastion sets it the moment the install is
|
||||
// dispatched, before anything has booted. Using it would match instantly
|
||||
// and this retry would never fire.
|
||||
//
|
||||
// The honest signals that Anaconda is alive are `progress` leaving
|
||||
// "queued", or log lines arriving over syslog.
|
||||
await pollUntil<{ status?: string; progress?: string; log_total?: number }>(
|
||||
`${base}/api/logs/${encodeURIComponent(mac)}`,
|
||||
(d) => d.status === "installed"
|
||||
|| (d.progress !== undefined && d.progress !== "queued")
|
||||
|| (d.log_total ?? 0) > 0,
|
||||
attemptMs, 5_000);
|
||||
if (attempt > 1) log(`[${name}] PXE boot took on attempt ${attempt}`);
|
||||
return;
|
||||
} catch {
|
||||
if (attempt === attempts) {
|
||||
throw new Error(
|
||||
`[${name}] installer never started after ${attempts} PXE boots ` +
|
||||
`(${attemptMs / 60000} min each). The VM is stuck before Anaconda — ` +
|
||||
`check TFTP and the iPXE chainload.`);
|
||||
}
|
||||
log(`[${name}] no installer activity in ${attemptMs / 60000} min; rebooting into PXE (attempt ${attempt + 1}/${attempts})`);
|
||||
rebootPxeVm(name);
|
||||
deleteNftablesRejectRules();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export interface ProvisionedVm { name: string; mac: string; ip: string }
|
||||
|
||||
/**
|
||||
* Serial ports are a global resource on the host, so hand out a distinct one
|
||||
* per VM. Without this only one VM can run at a time -- qemu refuses to start
|
||||
* the second with "Address already in use".
|
||||
*/
|
||||
let nextSerialPort = 4600;
|
||||
|
||||
/** Create a blank VM, let it discover, queue the install, wait for SSH. */
|
||||
export async function provisionVm(h: Harness, spec: {
|
||||
name: string;
|
||||
role: Role;
|
||||
schemaVersion?: 1 | 2;
|
||||
diskGb: number;
|
||||
memory?: number;
|
||||
vcpus?: number;
|
||||
}): Promise<ProvisionedVm> {
|
||||
const base = `http://${PXE_GATEWAY}:${h.httpPort}`;
|
||||
log(`[${spec.name}] creating VM (${spec.diskGb}GB disk, role=${spec.role}, schema=v${spec.schemaVersion ?? 1})`);
|
||||
createPxeVm({
|
||||
name: spec.name,
|
||||
memory: spec.memory ?? 4096,
|
||||
vcpus: spec.vcpus ?? 4,
|
||||
diskSize: spec.diskGb,
|
||||
network: PXE_NETWORK_NAME,
|
||||
serialPort: nextSerialPort++,
|
||||
});
|
||||
deleteNftablesRejectRules();
|
||||
|
||||
const mac = getVmMac(spec.name);
|
||||
if (!mac) throw new Error(`[${spec.name}] no MAC`);
|
||||
|
||||
log(`[${spec.name}] waiting for discovery (${mac})...`);
|
||||
await pollUntil<{ discovered: Record<string, unknown> }>(
|
||||
`${base}/api/machines`, (d) => mac in (d.discovered ?? {}), DISCOVERY_TIMEOUT_MS);
|
||||
|
||||
log(`[${spec.name}] queueing install...`);
|
||||
const res = await fetch(`${base}/api/install`, {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({
|
||||
mac, hostname: spec.name, disk: "", role: spec.role,
|
||||
...(spec.schemaVersion ? { schemaVersion: spec.schemaVersion } : {}),
|
||||
}),
|
||||
});
|
||||
if (!res.ok) throw new Error(`[${spec.name}] install queue rejected: ${res.status} ${await res.text()}`);
|
||||
|
||||
rebootPxeVm(spec.name);
|
||||
deleteNftablesRejectRules();
|
||||
|
||||
// Make sure the PXE boot actually TOOK before settling in for the install.
|
||||
//
|
||||
// A UEFI VM that fails to chainload just retries DHCP quietly, for ever.
|
||||
// Observed: one VM of four sat in a DHCP loop for an HOUR with no TFTP
|
||||
// fetch, then began installing seconds after the 60-minute install timeout
|
||||
// had already fired -- so the run failed on a machine that was, by then,
|
||||
// working. TFTP is UDP and four VMs contend for it, so this is a question of
|
||||
// when rather than if.
|
||||
//
|
||||
// Rebooting is cheap and a boot that has not started within a few minutes is
|
||||
// not going to start on its own.
|
||||
await waitForInstallStart(base, mac, spec.name);
|
||||
|
||||
log(`[${spec.name}] installing (up to ${INSTALL_TIMEOUT_MS / 60000} min)...`);
|
||||
await pollUntil<{ status?: string }>(
|
||||
`${base}/api/logs/${encodeURIComponent(mac)}`, (d) => d.status === "installed", INSTALL_TIMEOUT_MS, 10_000);
|
||||
|
||||
// The installed record carries the DHCP address the machine ended up with.
|
||||
const machines = await pollUntil<{ installed: Record<string, { ip?: string }> }>(
|
||||
`${base}/api/machines`, (d) => Boolean(d.installed?.[mac]?.ip), 2 * 60_000);
|
||||
const ip = machines.installed[mac]!.ip!;
|
||||
|
||||
try {
|
||||
if (execSync(`sudo virsh domstate ${spec.name}`, { encoding: "utf-8" }).trim() === "shut off") {
|
||||
execSync(`sudo virsh start ${spec.name}`, { stdio: "pipe" });
|
||||
}
|
||||
} catch { /* already running */ }
|
||||
deleteNftablesRejectRules();
|
||||
|
||||
log(`[${spec.name}] waiting for SSH at ${ip}...`);
|
||||
await waitForSsh(ip, h.sshUser, SSH_TIMEOUT_MS, h.sshKeyPath);
|
||||
log(`[${spec.name}] up at ${ip}`);
|
||||
return { name: spec.name, mac, ip };
|
||||
}
|
||||
|
||||
/**
|
||||
* Provision several VMs at once.
|
||||
*
|
||||
* Installs are ~12 minutes each and almost entirely I/O and network wait, so
|
||||
* four in sequence is an hour that four in parallel is twenty minutes. They are
|
||||
* staggered because the expensive, contended part is the start: PXE, DHCP and
|
||||
* the first package fetches all land at once otherwise.
|
||||
*
|
||||
* Rejects only after every VM has settled, so a single failure does not leave
|
||||
* the others half-built and unreferenced -- the caller still gets the names it
|
||||
* needs to tear down.
|
||||
*/
|
||||
export async function provisionVms(
|
||||
h: Harness,
|
||||
specs: Parameters<typeof provisionVm>[1][],
|
||||
opts: { staggerMs?: number } = {},
|
||||
): Promise<ProvisionedVm[]> {
|
||||
const stagger = opts.staggerMs ?? 60_000;
|
||||
const settled = await Promise.allSettled(
|
||||
specs.map(async (spec, i) => {
|
||||
await sleep(i * stagger);
|
||||
return provisionVm(h, spec);
|
||||
}),
|
||||
);
|
||||
|
||||
const failures = settled
|
||||
.map((r, i) => (r.status === "rejected" ? `${specs[i]?.name}: ${String(r.reason)}` : null))
|
||||
.filter((x): x is string => x !== null);
|
||||
if (failures.length > 0) {
|
||||
throw new Error(`provisionVms: ${failures.length}/${specs.length} failed\n ${failures.join("\n ")}`);
|
||||
}
|
||||
return settled.map((r) => (r as PromiseFulfilledResult<ProvisionedVm>).value);
|
||||
}
|
||||
|
||||
/** LV name -> size in MiB, read off the installed machine. */
|
||||
export function readLvs(h: Harness, vm: ProvisionedVm): Record<string, number> {
|
||||
const res = sshExec(vm.ip, h.sshUser,
|
||||
"sudo lvs labvg --noheadings --units m --nosuffix -o lv_name,lv_size --separator=,",
|
||||
{ keyPath: h.sshKeyPath });
|
||||
if (res.exitCode !== 0) throw new Error(`[${vm.name}] lvs failed: ${res.stderr}`);
|
||||
const map: Record<string, number> = {};
|
||||
for (const line of res.stdout.split("\n")) {
|
||||
const [name, size] = line.trim().split(",");
|
||||
if (name && size) map[name] = Math.round(Number(size));
|
||||
}
|
||||
// Log what actually landed on disk. The VMs are destroyed in afterAll, so
|
||||
// without this the only record of the real layout is the pass/fail bit.
|
||||
const summary = Object.entries(map)
|
||||
.sort(([a], [b]) => a.localeCompare(b))
|
||||
.map(([n, m]) => `${n}=${(m / 1024).toFixed(0)}G`)
|
||||
.join(" ");
|
||||
log(`[${vm.name}] on-disk LVs: ${summary}`);
|
||||
return map;
|
||||
}
|
||||
|
||||
/** The VG tag that records which schema this disk was built with. */
|
||||
export function readSchemaTag(h: Harness, vm: ProvisionedVm): string {
|
||||
return sshExec(vm.ip, h.sshUser, "sudo vgs --noheadings -o vg_tags labvg",
|
||||
{ keyPath: h.sshKeyPath }).stdout.trim();
|
||||
}
|
||||
|
||||
export function destroyVms(...names: string[]): void {
|
||||
for (const n of names) { try { destroyPxeVm(n); } catch { /* already gone */ } }
|
||||
}
|
||||
|
||||
/**
|
||||
* Reinstall a machine that is already installed, the way
|
||||
* `labctl provision reprovision` does: queue another install for the same MAC
|
||||
* and reboot it into PXE.
|
||||
*
|
||||
* The schema is deliberately NOT passed. A reinstall must reproduce whatever
|
||||
* layout is already on the disk -- %pre reads the lab-schema-v2 LVM tag rather
|
||||
* than trusting the caller -- and passing one here would hide a regression in
|
||||
* exactly that logic.
|
||||
*/
|
||||
export async function reprovisionVm(
|
||||
h: Harness,
|
||||
vm: ProvisionedVm,
|
||||
role: Role,
|
||||
/**
|
||||
* Only set this to prove it is IGNORED. A reinstall takes its layout from the
|
||||
* disk, not from the request; passing a schema here is how you test that.
|
||||
*/
|
||||
requestSchemaVersion?: 1 | 2,
|
||||
): Promise<ProvisionedVm> {
|
||||
const base = `http://${PXE_GATEWAY}:${h.httpPort}`;
|
||||
log(`[${vm.name}] reprovisioning (role=${role}, schema: whatever is on disk)...`);
|
||||
|
||||
const res = await fetch(`${base}/api/install`, {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({
|
||||
mac: vm.mac, hostname: vm.name, disk: "", role,
|
||||
...(requestSchemaVersion ? { schemaVersion: requestSchemaVersion } : {}),
|
||||
}),
|
||||
});
|
||||
if (!res.ok) throw new Error(`[${vm.name}] reinstall queue rejected: ${res.status} ${await res.text()}`);
|
||||
|
||||
rebootPxeVm(vm.name);
|
||||
deleteNftablesRejectRules();
|
||||
|
||||
log(`[${vm.name}] reinstalling...`);
|
||||
await pollUntil<{ status?: string }>(
|
||||
`${base}/api/logs/${encodeURIComponent(vm.mac)}`,
|
||||
(d) => d.status === "installed", INSTALL_TIMEOUT_MS, 10_000);
|
||||
|
||||
const machines = await pollUntil<{ installed: Record<string, { ip?: string }> }>(
|
||||
`${base}/api/machines`, (d) => Boolean(d.installed?.[vm.mac]?.ip), 2 * 60_000);
|
||||
const ip = machines.installed[vm.mac]!.ip!;
|
||||
|
||||
try {
|
||||
if (execSync(`sudo virsh domstate ${vm.name}`, { encoding: "utf-8" }).trim() === "shut off") {
|
||||
execSync(`sudo virsh start ${vm.name}`, { stdio: "pipe" });
|
||||
}
|
||||
} catch { /* already running */ }
|
||||
deleteNftablesRejectRules();
|
||||
|
||||
await waitForSsh(ip, h.sshUser, SSH_TIMEOUT_MS, h.sshKeyPath);
|
||||
log(`[${vm.name}] back up at ${ip} after reinstall`);
|
||||
return { ...vm, ip };
|
||||
}
|
||||
|
||||
/**
|
||||
* Drop a marker file into each mount point that a reinstall is supposed to
|
||||
* preserve. Ported from test-reprovision.sh, which proved the idea but only
|
||||
* ever ran by hand.
|
||||
*/
|
||||
export function writeMarkers(h: Harness, vm: ProvisionedVm, mounts: string[]): string {
|
||||
const token = `preserve-${Date.now()}-${Math.random().toString(36).slice(2, 8)}`;
|
||||
for (const m of mounts) {
|
||||
const r = sshExec(vm.ip, h.sshUser,
|
||||
`sudo sh -c 'echo ${token} > ${m}/LAB_PRESERVE_TEST.txt && sync'`,
|
||||
{ keyPath: h.sshKeyPath });
|
||||
if (r.exitCode !== 0) throw new Error(`[${vm.name}] could not write marker in ${m}: ${r.stderr}`);
|
||||
}
|
||||
log(`[${vm.name}] wrote marker ${token} into ${mounts.join(", ")}`);
|
||||
return token;
|
||||
}
|
||||
|
||||
/** Which mount points still carry the marker. */
|
||||
export function readMarkers(h: Harness, vm: ProvisionedVm, mounts: string[]): Record<string, string | null> {
|
||||
const out: Record<string, string | null> = {};
|
||||
for (const m of mounts) {
|
||||
const r = sshExec(vm.ip, h.sshUser, `sudo cat ${m}/LAB_PRESERVE_TEST.txt 2>/dev/null || true`,
|
||||
{ keyPath: h.sshKeyPath });
|
||||
const v = r.stdout.trim();
|
||||
out[m] = v === "" ? null : v;
|
||||
}
|
||||
log(`[${vm.name}] markers after reinstall: ${JSON.stringify(out)}`);
|
||||
return out;
|
||||
}
|
||||
@@ -56,6 +56,17 @@ export function ensurePxeNetwork(): void {
|
||||
|
||||
try { unlinkSync(xmlPath); } catch { /* ignore */ }
|
||||
|
||||
// `virsh net-start` returns before the bridge is usable. The bastion's
|
||||
// dnsmasq starts about a second later and binds to this device by name; if
|
||||
// the interface is not up and addressed yet it fails with "error binding
|
||||
// DHCP socket to device virbr-pxe", and because that failure is
|
||||
// asynchronous the harness carries on with NO DHCP AT ALL -- every VM then
|
||||
// sits in discovery until it times out, six minutes later, looking like a
|
||||
// PXE problem rather than a startup-order one.
|
||||
//
|
||||
// Cheap to wait, expensive not to.
|
||||
waitForBridge();
|
||||
|
||||
log(`Network ${PXE_NETWORK_NAME} created and active`);
|
||||
}
|
||||
|
||||
@@ -64,6 +75,32 @@ export function ensurePxeNetwork(): void {
|
||||
deleteNftablesRejectRules();
|
||||
}
|
||||
|
||||
/**
|
||||
* Block until the bridge exists AND carries the gateway address.
|
||||
*
|
||||
* Checking only that the device exists is not enough: it appears before the
|
||||
* address is assigned, and dnsmasq needs the address to bind.
|
||||
*/
|
||||
export function waitForBridge(timeoutMs = 30_000): void {
|
||||
const start = Date.now();
|
||||
let last = "";
|
||||
while (Date.now() - start < timeoutMs) {
|
||||
const r = spawnSync("ip", ["-br", "addr", "show", PXE_BRIDGE], { encoding: "utf-8", stdio: "pipe" });
|
||||
last = (r.stdout ?? "").trim();
|
||||
if (r.status === 0 && last.includes(`${PXE_GATEWAY}/`)) {
|
||||
log(`Bridge ${PXE_BRIDGE} ready with ${PXE_GATEWAY} after ${Date.now() - start}ms`);
|
||||
return;
|
||||
}
|
||||
// Busy-wait briefly; this normally resolves in well under a second.
|
||||
const until = Date.now() + 250;
|
||||
while (Date.now() < until) { /* spin */ }
|
||||
}
|
||||
throw new Error(
|
||||
`Bridge ${PXE_BRIDGE} did not come up with ${PXE_GATEWAY} within ${timeoutMs}ms ` +
|
||||
`(last: ${last || "device absent"}). dnsmasq cannot bind, so PXE would silently not work.`,
|
||||
);
|
||||
}
|
||||
|
||||
/** Delete libvirt's nftables reject rules for our bridge so host→VM traffic works.
|
||||
* Must be called after every VM start/restart — libvirt recreates them. */
|
||||
export function deleteNftablesRejectRules(): void {
|
||||
|
||||
@@ -28,6 +28,13 @@ export interface PxeVmConfig {
|
||||
vcpus: number;
|
||||
diskSize: number; // GB
|
||||
network: string; // libvirt network name
|
||||
/**
|
||||
* Serial console TCP port. Defaults to 4555 for backwards compatibility with
|
||||
* the single-VM tests, but it MUST be unique per concurrently-running VM:
|
||||
* qemu fails with "Failed to find an available port: Address already in use"
|
||||
* and the domain never starts.
|
||||
*/
|
||||
serialPort?: number;
|
||||
arch?: "x86_64" | "aarch64";
|
||||
/**
|
||||
* Extra NICs enumerated BEFORE the PXE NIC, on a network with no route to
|
||||
@@ -86,8 +93,8 @@ export function createPxeVm(config: PxeVmConfig): void {
|
||||
// Graphics for debugging (VNC, connect with virt-viewer if needed)
|
||||
"--graphics=vnc,listen=127.0.0.1",
|
||||
// Serial console via TCP — allows exec without network/SSH
|
||||
// Connect: socat - TCP:127.0.0.1:4555
|
||||
"--serial=tcp,host=127.0.0.1:4555,mode=bind,protocol=telnet",
|
||||
// Connect: socat - TCP:127.0.0.1:<serialPort>
|
||||
`--serial=tcp,host=127.0.0.1:${config.serialPort ?? 4555},mode=bind,protocol=telnet`,
|
||||
];
|
||||
|
||||
if (arch === "aarch64") {
|
||||
@@ -96,7 +103,7 @@ export function createPxeVm(config: PxeVmConfig): void {
|
||||
|
||||
log(`Running: virt-install --name=${config.name} --boot=uefi,network ...`);
|
||||
run(virtInstallArgs.join(" "), { timeout: 30_000 });
|
||||
log(`PXE VM ${config.name} created (serial: telnet 127.0.0.1 4555)`);
|
||||
log(`PXE VM ${config.name} created (serial: telnet 127.0.0.1 ${config.serialPort ?? 4555})`);
|
||||
}
|
||||
|
||||
/** Destroy a PXE VM and clean up its disk. */
|
||||
|
||||
145
bastion/tests/integration/partition-layout.test.ts
Normal file
145
bastion/tests/integration/partition-layout.test.ts
Normal file
@@ -0,0 +1,145 @@
|
||||
// Integration tests: the v2 disk layout, on real VMs.
|
||||
//
|
||||
// Unit tests prove the kickstart RENDERS the right layout. These prove anaconda
|
||||
// ACTUALLY PRODUCES it -- which is a different claim, and the one that matters:
|
||||
// the labmaster incident was caused by what landed on disk, not by what was
|
||||
// written into a template.
|
||||
//
|
||||
// Every VM gets a 1 TB sparse disk, because "fits a 1 TB disk with headroom" is
|
||||
// the constraint v2 was designed around. qcow2 is sparse, so a 1 TB disk costs
|
||||
// ~10-15 GB of real space.
|
||||
//
|
||||
// Prerequisites: libvirtd, OVMF, ipxe-bootimgs-x86, sudo, and a Fedora mirror.
|
||||
// Set FEDORA_MIRROR to a local mirror -- otherwise each install pulls the full
|
||||
// package set over the internet and the matrix takes hours.
|
||||
//
|
||||
// Run one scenario at a time:
|
||||
// sudo -E npx vitest run -c tests/integration/vitest.config.ts -t "vanilla"
|
||||
|
||||
import { describe, it, expect, beforeAll, afterAll } from "vitest";
|
||||
import { existsSync, readFileSync } from "node:fs";
|
||||
import { homedir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { v2Layout } from "../../src/shared/src/types/layout.js";
|
||||
import type { Role } from "../../src/shared/src/types/state.js";
|
||||
import {
|
||||
startTestBastion, provisionVm, readLvs, readSchemaTag, destroyVms,
|
||||
type Harness, type ProvisionedVm,
|
||||
} from "./helpers/provision-harness.js";
|
||||
import { destroyPxeNetwork } from "./helpers/pxe-network.js";
|
||||
|
||||
const HTTP_PORT = 8096;
|
||||
const SSH_USER = "lab";
|
||||
const DISK_GB = 1000; // the smallest disk we will ever buy
|
||||
|
||||
function findSshKey(): { pubKey: string; keyPath: string } {
|
||||
const homes = [homedir()];
|
||||
const sudoUser = process.env["SUDO_USER"];
|
||||
if (sudoUser) homes.push(join("/home", sudoUser));
|
||||
for (const home of homes) {
|
||||
for (const name of ["id_ed25519", "id_ecdsa", "id_rsa"]) {
|
||||
const keyPath = join(home, ".ssh", name);
|
||||
if (existsSync(keyPath) && existsSync(`${keyPath}.pub`)) {
|
||||
return { pubKey: readFileSync(`${keyPath}.pub`, "utf-8").trim(), keyPath };
|
||||
}
|
||||
}
|
||||
}
|
||||
throw new Error("no SSH keypair found");
|
||||
}
|
||||
|
||||
/** Assert the on-disk volumes match the shared v2 definition for this role. */
|
||||
function expectV2Layout(h: Harness, vm: ProvisionedVm, role: Role): void {
|
||||
const actual = readLvs(h, vm);
|
||||
const expected = v2Layout(role);
|
||||
|
||||
expect(Object.keys(actual).sort(), `${vm.name} volume set`).toEqual(
|
||||
expected.map((lv) => lv.name).sort(),
|
||||
);
|
||||
for (const spec of expected) {
|
||||
// LVM rounds to extent boundaries (4 MiB), so allow a small delta.
|
||||
expect(actual[spec.name], `${vm.name}/${spec.name} size`).toBeGreaterThanOrEqual(spec.sizeMib - 4);
|
||||
expect(actual[spec.name], `${vm.name}/${spec.name} size`).toBeLessThanOrEqual(spec.sizeMib + 4);
|
||||
}
|
||||
expect(readSchemaTag(h, vm), `${vm.name} schema tag`).toContain("lab-schema-v2");
|
||||
}
|
||||
|
||||
describe("v2 partition layout on real hardware", () => {
|
||||
let h: Harness;
|
||||
const created: string[] = [];
|
||||
|
||||
beforeAll(async () => {
|
||||
const { pubKey, keyPath } = findSshKey();
|
||||
h = await startTestBastion({
|
||||
httpPort: HTTP_PORT, sshUser: SSH_USER, sshPubKey: pubKey,
|
||||
sshKeyPath: keyPath, label: "partition",
|
||||
});
|
||||
}, 20 * 60_000);
|
||||
|
||||
afterAll(async () => {
|
||||
destroyVms(...created);
|
||||
try { await h?.close(); } catch { /* already closed */ }
|
||||
try { destroyPxeNetwork(); } catch { /* already gone */ }
|
||||
}, 5 * 60_000);
|
||||
|
||||
// --- Topology 1: vanilla -------------------------------------------------
|
||||
it("vanilla: base volumes only, no rancher, no longhorn", async () => {
|
||||
const name = "v2-vanilla";
|
||||
created.push(name);
|
||||
const vm = await provisionVm(h, { name, role: "vanilla", schemaVersion: 2, diskGb: DISK_GB });
|
||||
expectV2Layout(h, vm, "vanilla");
|
||||
|
||||
const lvs = readLvs(h, vm);
|
||||
expect(lvs["rancher"]).toBeUndefined();
|
||||
expect(lvs["longhorn"]).toBeUndefined();
|
||||
}, 60 * 60_000);
|
||||
|
||||
// --- Topology 2: standalone cluster --------------------------------------
|
||||
it("standalone-cluster: single node with its own control plane and storage", async () => {
|
||||
const name = "v2-standalone";
|
||||
created.push(name);
|
||||
const vm = await provisionVm(h, { name, role: "standalone-cluster", schemaVersion: 2, diskGb: DISK_GB, memory: 6144 });
|
||||
expectV2Layout(h, vm, "standalone-cluster");
|
||||
|
||||
const lvs = readLvs(h, vm);
|
||||
// A cluster of one still gets a Longhorn volume, so the `longhorn`
|
||||
// StorageClass exists and manifests stay portable across clusters.
|
||||
expect(lvs["longhorn"]).toBeDefined();
|
||||
expect(lvs["rancher"]).toBeDefined();
|
||||
}, 60 * 60_000);
|
||||
|
||||
// --- Topology 3: 3x combo (control plane AND worker on every node) -------
|
||||
it("k8s-combo x3: every node is control plane and storage", async () => {
|
||||
const names = ["v2-combo-1", "v2-combo-2", "v2-combo-3"];
|
||||
created.push(...names);
|
||||
const vms: ProvisionedVm[] = [];
|
||||
for (const name of names) {
|
||||
vms.push(await provisionVm(h, { name, role: "k8s-combo", schemaVersion: 2, diskGb: DISK_GB }));
|
||||
}
|
||||
for (const vm of vms) {
|
||||
expectV2Layout(h, vm, "k8s-combo");
|
||||
const lvs = readLvs(h, vm);
|
||||
expect(lvs["longhorn"], `${vm.name} holds storage`).toBeDefined();
|
||||
expect(lvs["rancher"], `${vm.name} has an image store`).toBeDefined();
|
||||
}
|
||||
}, 180 * 60_000);
|
||||
|
||||
// --- Topology 4: 3 controllers + 2 workers -------------------------------
|
||||
it("3x k8s-controller + 2x k8s-worker: storage only on the workers", async () => {
|
||||
const ctrls = ["v2-ctrl-1", "v2-ctrl-2", "v2-ctrl-3"];
|
||||
const workers = ["v2-wrk-1", "v2-wrk-2"];
|
||||
created.push(...ctrls, ...workers);
|
||||
|
||||
for (const name of ctrls) {
|
||||
const vm = await provisionVm(h, { name, role: "k8s-controller", schemaVersion: 2, diskGb: DISK_GB });
|
||||
expectV2Layout(h, vm, "k8s-controller");
|
||||
// The distinguishing property of a controller: image store, no storage.
|
||||
expect(readLvs(h, vm)["longhorn"], `${name} must not hold replicas`).toBeUndefined();
|
||||
expect(readLvs(h, vm)["rancher"]).toBeDefined();
|
||||
}
|
||||
for (const name of workers) {
|
||||
const vm = await provisionVm(h, { name, role: "k8s-worker", schemaVersion: 2, diskGb: DISK_GB });
|
||||
expectV2Layout(h, vm, "k8s-worker");
|
||||
expect(readLvs(h, vm)["longhorn"], `${name} must hold replicas`).toBeDefined();
|
||||
}
|
||||
}, 300 * 60_000);
|
||||
});
|
||||
138
bastion/tests/integration/reinstall.test.ts
Normal file
138
bastion/tests/integration/reinstall.test.ts
Normal file
@@ -0,0 +1,138 @@
|
||||
// Integration tests: reinstalling a machine must not destroy its data, and
|
||||
// must not silently change its disk layout.
|
||||
//
|
||||
// This is the risk the whole v2 change carries. Every machine in the fleet was
|
||||
// installed with v1, and `labctl provision reprovision` has to keep reproducing
|
||||
// v1 on those boxes forever. If a reinstall quietly switched them to v2 it
|
||||
// would recreate the volume group and take every Longhorn replica with it.
|
||||
//
|
||||
// Ported from test-reprovision.sh, which had the right idea (marker files in
|
||||
// each preserved volume) but only ever ran by hand and was never in CI.
|
||||
//
|
||||
// Run: sudo -E npx vitest run -c tests/integration/vitest.config.ts -t "reinstall"
|
||||
|
||||
import { describe, it, expect, beforeAll, afterAll } from "vitest";
|
||||
import { existsSync, readFileSync } from "node:fs";
|
||||
import { homedir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { v2Layout } from "../../src/shared/src/types/layout.js";
|
||||
import {
|
||||
startTestBastion, provisionVm, reprovisionVm, readLvs, readSchemaTag,
|
||||
writeMarkers, readMarkers, destroyVms, type Harness,
|
||||
} from "./helpers/provision-harness.js";
|
||||
import { destroyPxeNetwork } from "./helpers/pxe-network.js";
|
||||
|
||||
const HTTP_PORT = 8095;
|
||||
const SSH_USER = "lab";
|
||||
const DISK_GB = 1000;
|
||||
|
||||
/** Mount points a reinstall is supposed to preserve on a v1 worker. */
|
||||
const PRESERVED = ["/home", "/srv", "/var/lib/longhorn", "/var/lib/rancher"];
|
||||
|
||||
function findSshKey(): { pubKey: string; keyPath: string } {
|
||||
const homes = [homedir()];
|
||||
const sudoUser = process.env["SUDO_USER"];
|
||||
if (sudoUser) homes.push(join("/home", sudoUser));
|
||||
for (const home of homes) {
|
||||
for (const name of ["id_ed25519", "id_ecdsa", "id_rsa"]) {
|
||||
const keyPath = join(home, ".ssh", name);
|
||||
if (existsSync(keyPath) && existsSync(`${keyPath}.pub`)) {
|
||||
return { pubKey: readFileSync(`${keyPath}.pub`, "utf-8").trim(), keyPath };
|
||||
}
|
||||
}
|
||||
}
|
||||
throw new Error("no SSH keypair found");
|
||||
}
|
||||
|
||||
describe("reinstall preserves data and layout", () => {
|
||||
let h: Harness;
|
||||
const created: string[] = [];
|
||||
|
||||
beforeAll(async () => {
|
||||
const { pubKey, keyPath } = findSshKey();
|
||||
h = await startTestBastion({
|
||||
httpPort: HTTP_PORT, sshUser: SSH_USER, sshPubKey: pubKey,
|
||||
sshKeyPath: keyPath, label: "reinstall",
|
||||
});
|
||||
}, 20 * 60_000);
|
||||
|
||||
afterAll(async () => {
|
||||
destroyVms(...created);
|
||||
try { await h?.close(); } catch { /* closed */ }
|
||||
try { destroyPxeNetwork(); } catch { /* gone */ }
|
||||
}, 5 * 60_000);
|
||||
|
||||
it("v1 worker: data survives and the layout stays v1", async () => {
|
||||
const name = "re-v1-worker";
|
||||
created.push(name);
|
||||
|
||||
// Install at v1 -- i.e. exactly what every machine in the fleet is today.
|
||||
let vm = await provisionVm(h, { name, role: "worker", diskGb: DISK_GB });
|
||||
|
||||
const before = readLvs(h, vm);
|
||||
expect(before["root"], "v1 root is 33G, not v2's 32G").toBeGreaterThan(33000);
|
||||
expect(before["rancher"], "v1 worker gets a 120G rancher LV").toBeGreaterThan(120000 - 8);
|
||||
expect(readSchemaTag(h, vm), "a v1 disk must NOT carry the v2 tag").not.toContain("lab-schema-v2");
|
||||
|
||||
const token = writeMarkers(h, vm, PRESERVED);
|
||||
|
||||
// Reinstall. No schemaVersion passed -- the disk decides.
|
||||
vm = await reprovisionVm(h, vm, "worker");
|
||||
|
||||
const markers = readMarkers(h, vm, PRESERVED);
|
||||
for (const mount of PRESERVED) {
|
||||
expect(markers[mount], `${mount} must survive a reinstall`).toBe(token);
|
||||
}
|
||||
|
||||
// And the layout must be untouched.
|
||||
const after = readLvs(h, vm);
|
||||
expect(Object.keys(after).sort()).toEqual(Object.keys(before).sort());
|
||||
expect(after["root"]).toBe(before["root"]);
|
||||
expect(after["rancher"]).toBe(before["rancher"]);
|
||||
expect(readSchemaTag(h, vm)).not.toContain("lab-schema-v2");
|
||||
}, 120 * 60_000);
|
||||
|
||||
it("v2 worker: data survives and the layout stays v2", async () => {
|
||||
const name = "re-v2-worker";
|
||||
created.push(name);
|
||||
|
||||
let vm = await provisionVm(h, { name, role: "k8s-worker", schemaVersion: 2, diskGb: DISK_GB });
|
||||
const before = readLvs(h, vm);
|
||||
expect(readSchemaTag(h, vm)).toContain("lab-schema-v2");
|
||||
|
||||
const token = writeMarkers(h, vm, PRESERVED);
|
||||
vm = await reprovisionVm(h, vm, "k8s-worker");
|
||||
|
||||
const markers = readMarkers(h, vm, PRESERVED);
|
||||
for (const mount of PRESERVED) {
|
||||
expect(markers[mount], `${mount} must survive a reinstall`).toBe(token);
|
||||
}
|
||||
|
||||
const after = readLvs(h, vm);
|
||||
const expected = Object.fromEntries(v2Layout("k8s-worker").map((lv) => [lv.name, lv.sizeMib]));
|
||||
expect(Object.keys(after).sort()).toEqual(Object.keys(expected).sort());
|
||||
expect(after["root"]).toBe(before["root"]);
|
||||
// The tag must survive too, or the NEXT reinstall would fall back to v1.
|
||||
expect(readSchemaTag(h, vm)).toContain("lab-schema-v2");
|
||||
}, 120 * 60_000);
|
||||
|
||||
it("a v1 machine is NOT upgraded to v2 just because v2 was requested", async () => {
|
||||
const name = "re-v1-sticky";
|
||||
created.push(name);
|
||||
|
||||
let vm = await provisionVm(h, { name, role: "worker", diskGb: DISK_GB });
|
||||
const before = readLvs(h, vm);
|
||||
const token = writeMarkers(h, vm, ["/home", "/srv"]);
|
||||
|
||||
// Ask for v2 on a disk that is v1. %pre must ignore it: silently
|
||||
// re-laying-out a live machine is how data gets destroyed.
|
||||
vm = await reprovisionVm(h, vm, "worker", 2);
|
||||
|
||||
const after = readLvs(h, vm);
|
||||
expect(after["root"], "root must stay at the v1 size").toBe(before["root"]);
|
||||
expect(readSchemaTag(h, vm), "still not tagged v2").not.toContain("lab-schema-v2");
|
||||
const markers = readMarkers(h, vm, ["/home", "/srv"]);
|
||||
expect(markers["/home"]).toBe(token);
|
||||
expect(markers["/srv"]).toBe(token);
|
||||
}, 120 * 60_000);
|
||||
});
|
||||
623
bastion/tests/integration/rolling-drill.test.ts
Normal file
623
bastion/tests/integration/rolling-drill.test.ts
Normal file
@@ -0,0 +1,623 @@
|
||||
// THE MIGRATION DRILL: roll three storage+etcd nodes through
|
||||
// evacuate -> wipe -> reinstall -> rejoin, one at a time, with data intact.
|
||||
//
|
||||
// WHAT IT IS PROVING
|
||||
// ------------------
|
||||
// The v1 -> v2 layout migration cannot preserve data through the wipe: %pre
|
||||
// only preserves when labvg already has the expected LV names, and a new layout
|
||||
// recreates the VG, taking the clearpart branch. So the data has to leave the
|
||||
// node first and come back afterwards. This asserts, on real machines, that it
|
||||
// does -- by sha256, either side of every step.
|
||||
//
|
||||
// WHY FOUR NODES FOR A THREE-NODE ROLL
|
||||
// ------------------------------------
|
||||
// With strict anti-affinity (replica-soft-anti-affinity=false, what the fleet
|
||||
// runs) a volume puts at most one replica per node, so a 3-replica volume on a
|
||||
// 3-node cluster CANNOT be evicted: the two survivors already hold a copy each
|
||||
// and cannot take the third. Eviction would wait for its full timeout and fail.
|
||||
// The real fleet has exactly this shape, plus a 100 GiB disk on worker2 with
|
||||
// ~49 GiB free -- so capacity blocks it independently of anti-affinity.
|
||||
//
|
||||
// The fix is the one already planned for production: bring the NEW box up
|
||||
// first, then evacuate an old one onto it. The fourth node here is an AGENT, so
|
||||
// etcd stays a 3-member cluster and every roll runs against production's real
|
||||
// quorum arithmetic (2 of 3, no margin).
|
||||
//
|
||||
// ORDER OF THE DANGEROUS PART
|
||||
// ---------------------------
|
||||
// The irreversible pair -- etcd member removal and node deletion -- happens as
|
||||
// late as possible, immediately before the wipe. Anything that fails before it
|
||||
// has destroyed nothing.
|
||||
//
|
||||
// sudo -E npx vitest run -c tests/integration/vitest.config.ts -t "rolling drill"
|
||||
|
||||
import { describe, it, expect, beforeAll, afterAll } from "vitest";
|
||||
import { existsSync, readFileSync } from "node:fs";
|
||||
import { homedir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { K3sModule } from "../../src/modules/modules/k3s/src/k3s-module.js";
|
||||
import {
|
||||
startTestBastion, provisionVms, reprovisionVm, readLvs, destroyVms,
|
||||
type Harness, type ProvisionedVm,
|
||||
} from "./helpers/provision-harness.js";
|
||||
import { PINNED_K3S_VERSION } from "./helpers/k3s-cluster.js";
|
||||
import {
|
||||
formCluster, joinAgent, installK3sNode, serverUrlOf, moduleCtx,
|
||||
clusterSnapshot, printSnapshot, etcdMembers, etcdHealthy, allServersReady,
|
||||
pickJoinTarget, kubectlOn, kubectlJson, nodeExec, removeEtcdMember, waitForNodeReady,
|
||||
waitForApiPing, sleep, DRILL_TOKEN, k8sNodeName, forgetNodeName,
|
||||
} from "./helpers/k3s-cluster.js";
|
||||
import {
|
||||
installLonghorn, waitForLonghornReady, createDataWorkload, waitForWorkload,
|
||||
writeBlob, verifyBlob, volumeState, waitForVolumeHealthy, LONGHORN_NS,
|
||||
backupTargetFromEnv, setBackupTarget, waitForBackupTarget, backupVolume, restoreFromBackup,
|
||||
readBlobDigest,
|
||||
} from "./helpers/longhorn.js";
|
||||
import { destroyPxeNetwork } from "./helpers/pxe-network.js";
|
||||
|
||||
const HTTP_PORT = 8098;
|
||||
const SSH_USER = "lab";
|
||||
const DISK_GB = 1000;
|
||||
|
||||
const SERVERS = ["drill-s1", "drill-s2", "drill-s3"];
|
||||
const SPARE = "drill-spare";
|
||||
const ALL = [...SERVERS, SPARE];
|
||||
|
||||
/** The replica-3 volume: the one whose rebuild the roll depends on. */
|
||||
const R3 = "drill-r3";
|
||||
/** The replica-1 volume: single copy, the sharpest risk in the real fleet. */
|
||||
const R1 = "drill-r1";
|
||||
|
||||
function findSshKey(): { pubKey: string; keyPath: string } {
|
||||
const homes = [homedir()];
|
||||
const sudoUser = process.env["SUDO_USER"];
|
||||
if (sudoUser) homes.push(join("/home", sudoUser));
|
||||
for (const home of homes) {
|
||||
for (const name of ["id_ed25519", "id_ecdsa", "id_rsa"]) {
|
||||
const keyPath = join(home, ".ssh", name);
|
||||
if (existsSync(keyPath) && existsSync(`${keyPath}.pub`)) {
|
||||
return { pubKey: readFileSync(`${keyPath}.pub`, "utf-8").trim(), keyPath };
|
||||
}
|
||||
}
|
||||
}
|
||||
throw new Error("no SSH keypair found");
|
||||
}
|
||||
|
||||
describe("rolling drill: migrate three nodes without losing data", () => {
|
||||
let h: Harness;
|
||||
let vms: ProvisionedVm[] = [];
|
||||
let spare: ProvisionedVm;
|
||||
let servers: ProvisionedVm[] = [];
|
||||
let digestR3 = "";
|
||||
let digestR1 = "";
|
||||
/**
|
||||
* S3 backup target, from DRILL_S3_* in the environment.
|
||||
*
|
||||
* Undefined means the backup/restore phase SKIPS rather than passing
|
||||
* vacuously. A backup test that quietly does nothing is worse than none.
|
||||
*/
|
||||
const backupTarget = backupTargetFromEnv();
|
||||
|
||||
/**
|
||||
* Sticky abort. Once a step fails, the remaining steps SKIP rather than run
|
||||
* against a cluster that is already wrong -- a failed roll leaves a node
|
||||
* mid-migration, and rolling the next one on top of that turns one recoverable
|
||||
* problem into an unrecoverable one.
|
||||
*/
|
||||
let aborted: string | null = null;
|
||||
/**
|
||||
* `ctx` is vitest's test context. It is used to SKIP downstream steps rather
|
||||
* than let them return quietly — a silent return marks the test PASSED, so a
|
||||
* drill that died in provisioning reports "1 failed | 8 passed" and the eight
|
||||
* steps that never ran look like evidence. They are not evidence.
|
||||
*/
|
||||
const step = async (
|
||||
ctx: { skip: (note?: string) => void },
|
||||
name: string,
|
||||
fn: () => Promise<void>,
|
||||
): Promise<void> => {
|
||||
if (aborted !== null) {
|
||||
console.log(`[drill] SKIPPED ${name} (aborted at: ${aborted})`); // eslint-disable-line no-console
|
||||
ctx.skip(`drill aborted at: ${aborted}`);
|
||||
return;
|
||||
}
|
||||
const t0 = Date.now();
|
||||
console.log(`[drill] ==> ${name}`); // eslint-disable-line no-console
|
||||
try {
|
||||
await fn();
|
||||
console.log(`[drill] <== ${name} (${Math.round((Date.now() - t0) / 1000)}s)`); // eslint-disable-line no-console
|
||||
} catch (e) {
|
||||
aborted = name;
|
||||
console.error(`[drill] !!! ${name} FAILED: ${e instanceof Error ? e.message : String(e)}`); // eslint-disable-line no-console
|
||||
throw e;
|
||||
}
|
||||
};
|
||||
|
||||
/**
|
||||
* The runbook's replica-1 pre-flight, rehearsed rather than described.
|
||||
*
|
||||
* A single-replica volume rebuilds from the only copy that exists, so
|
||||
* `evacuate` refuses it by default. The documented answer is to raise it to
|
||||
* 3, wait for healthy, evacuate, and put it back afterwards. Doing that here
|
||||
* means the drill exercises the procedure operators will actually follow
|
||||
* instead of quietly bypassing it with the override flag.
|
||||
*/
|
||||
const raiseReplicas = async (peer: ProvisionedVm, pvc: string, to: number): Promise<boolean> => {
|
||||
const st = volumeState(h, peer, pvc);
|
||||
kubectlOn(h, peer, `-n ${LONGHORN_NS} patch volumes.longhorn.io ${st.pv} ` +
|
||||
`--type=merge -p '{"spec":{"numberOfReplicas":${to}}}'`);
|
||||
if (to <= 1) return true;
|
||||
try {
|
||||
// BOUNDED, and allowed to fail.
|
||||
//
|
||||
// Raising a volume's replica count only converges if every target node
|
||||
// will actually start a replica. A node that has just been wiped and
|
||||
// rejoined may not: observed here, a replica sat at
|
||||
// currentState=stopped / desireState=running / failedAt='' for forty
|
||||
// minutes with no rebuild and no error, while the volume stayed
|
||||
// degraded at 2-of-3 RW. Blocking the whole drill on that turns a
|
||||
// Longhorn quirk into a total failure and proves nothing.
|
||||
await waitForVolumeHealthy(h, peer, { pvc, expectReplicas: to, timeoutMs: 10 * 60_000 });
|
||||
return true;
|
||||
} catch (e) {
|
||||
console.warn( // eslint-disable-line no-console
|
||||
`[drill] ${pvc} did not reach ${to} healthy replicas in 10 min: ` +
|
||||
`${e instanceof Error ? e.message : String(e)}`);
|
||||
return false;
|
||||
}
|
||||
};
|
||||
|
||||
/** Is this volume's only replica on the node we are about to evacuate? */
|
||||
const replicaOn = (peer: ProvisionedVm, pvc: string, node: string): boolean =>
|
||||
volumeState(h, peer, pvc).replicaNodes.includes(node);
|
||||
|
||||
/** Read cluster state from a node that is NOT the one being worked on. */
|
||||
const survivor = (exclude?: ProvisionedVm): ProvisionedVm =>
|
||||
exclude ? pickJoinTarget(h, servers, exclude) : servers[0]!;
|
||||
|
||||
beforeAll(async () => {
|
||||
const { pubKey, keyPath } = findSshKey();
|
||||
destroyVms(...ALL);
|
||||
h = await startTestBastion({
|
||||
httpPort: HTTP_PORT, sshUser: SSH_USER, sshPubKey: pubKey, sshKeyPath: keyPath,
|
||||
label: "rolling-drill",
|
||||
autoK3s: false, // we sequence cluster formation ourselves
|
||||
});
|
||||
}, 20 * 60_000);
|
||||
|
||||
afterAll(async () => {
|
||||
// Diagnostics BEFORE teardown: these VMs are the only record of what
|
||||
// happened and they are about to be deleted.
|
||||
try {
|
||||
const s = servers[0];
|
||||
if (s) {
|
||||
printSnapshot("FINAL", clusterSnapshot(h, s));
|
||||
for (const pvc of [R3, R1]) {
|
||||
try {
|
||||
const st = volumeState(h, s, pvc);
|
||||
console.log(`[drill] ${pvc}: ${st.state}/${st.robustness} on ${st.replicaNodes.join(",")}`); // eslint-disable-line no-console
|
||||
} catch { /* volume may be gone */ }
|
||||
}
|
||||
console.log(`[drill] digests: ${R3}=${digestR3.slice(0, 16)} ${R1}=${digestR1.slice(0, 16)}`); // eslint-disable-line no-console
|
||||
}
|
||||
} catch { /* cluster may be unreachable */ }
|
||||
|
||||
// Best-effort: clear any eviction we may have left pending. Harmless on a
|
||||
// VM about to be destroyed, but this is the same call the runbook makes on
|
||||
// a real node, so exercising it here is free coverage.
|
||||
for (const vm of servers) {
|
||||
try { await new K3sModule().admit(moduleCtx(h, vm, "k8s-combo")); } catch { /* ignore */ }
|
||||
}
|
||||
|
||||
if (process.env["LAB_DRILL_KEEP_VMS"] !== "1") {
|
||||
destroyVms(...ALL);
|
||||
try { destroyPxeNetwork(); } catch { /* shared */ }
|
||||
} else {
|
||||
console.log("[drill] LAB_DRILL_KEEP_VMS=1 — leaving VMs up for inspection"); // eslint-disable-line no-console
|
||||
}
|
||||
await h?.close();
|
||||
}, 20 * 60_000);
|
||||
|
||||
it("provisions four v2 nodes", async (ctx) => {
|
||||
await step(ctx, "provision", async () => {
|
||||
vms = await provisionVms(h, [
|
||||
...SERVERS.map((name) => ({
|
||||
name, role: "k8s-combo" as const, schemaVersion: 2 as const,
|
||||
diskGb: DISK_GB, memory: 6144, vcpus: 4,
|
||||
})),
|
||||
{ name: SPARE, role: "k8s-worker" as const, schemaVersion: 2 as const,
|
||||
diskGb: DISK_GB, memory: 4096, vcpus: 2 },
|
||||
]);
|
||||
servers = vms.filter((v) => SERVERS.includes(v.name));
|
||||
spare = vms.find((v) => v.name === SPARE)!;
|
||||
expect(servers).toHaveLength(3);
|
||||
|
||||
// Every node must actually have the longhorn LV, or nothing later works.
|
||||
for (const vm of vms) expect(Object.keys(readLvs(h, vm))).toContain("longhorn");
|
||||
});
|
||||
}, 90 * 60_000);
|
||||
|
||||
it("forms one cluster with three etcd members plus a storage agent", async (ctx) => {
|
||||
await step(ctx, "form cluster", async () => {
|
||||
await formCluster(h, servers, { role: "k8s-combo" });
|
||||
await joinAgent(h, spare, { server: servers[0]!, role: "k8s-worker" });
|
||||
|
||||
const snap = clusterSnapshot(h, servers[0]!);
|
||||
printSnapshot("formed", snap);
|
||||
expect(snap.nodes).toHaveLength(4);
|
||||
expect(snap.readyCount).toBe(4);
|
||||
// The spare is an agent on purpose: etcd stays at 3 so each roll runs
|
||||
// against production's real quorum arithmetic.
|
||||
expect(etcdMembers(h, servers[0]!)).toHaveLength(3);
|
||||
expect(etcdHealthy(h, servers[0]!)).toBe(true);
|
||||
|
||||
// Every node must run the PINNED version, not whatever get.k3s.io calls
|
||||
// latest today. Setting k3sVersion is not the same as it taking effect:
|
||||
// toOpContext used to drop the field silently, so the pin was ignored and
|
||||
// nodes came up two minor versions ahead of the fleet. Assert the
|
||||
// outcome, on the machines.
|
||||
const versions = kubectlJson<{ items: {
|
||||
metadata: { name: string };
|
||||
status: { nodeInfo: { kubeletVersion: string } };
|
||||
}[] }>(h, servers[0]!, "get nodes");
|
||||
expect(versions.items).toHaveLength(4);
|
||||
for (const n of versions.items) {
|
||||
expect(n.status.nodeInfo.kubeletVersion,
|
||||
`${n.metadata.name} must run the pinned ${PINNED_K3S_VERSION}`)
|
||||
.toBe(PINNED_K3S_VERSION);
|
||||
}
|
||||
console.log(`[drill] all 4 nodes on pinned ${PINNED_K3S_VERSION}`); // eslint-disable-line no-console
|
||||
});
|
||||
}, 120 * 60_000);
|
||||
|
||||
it("installs Longhorn with a schedulable disk on every node", async (ctx) => {
|
||||
await step(ctx, "install longhorn", async () => {
|
||||
await installLonghorn(h, servers[0]!, {
|
||||
defaultReplicas: 3,
|
||||
...(backupTarget === undefined ? {} : { backup: backupTarget }),
|
||||
});
|
||||
|
||||
// Give Longhorn time to register the nodes, then run the PRODUCTION
|
||||
// repair path on each one.
|
||||
//
|
||||
// configureLonghornDisk runs during configure(), which happens BEFORE
|
||||
// Longhorn is installed -- there is no node record to repair yet, so it
|
||||
// only sets the annotation and returns. If Longhorn then misses that
|
||||
// annotation for any reason, nothing ever retries, and the node joins,
|
||||
// goes Ready, and silently hosts no replicas. Observed: one node in four
|
||||
// came up with no disk while its label and annotation were byte-identical
|
||||
// to its siblings'.
|
||||
//
|
||||
// admit() is exactly what the runbook tells an operator to run, and it
|
||||
// exercises the verify-and-repair branch that is otherwise only reachable
|
||||
// after a wipe-and-rejoin.
|
||||
await sleep(60_000);
|
||||
for (const vm of vms) {
|
||||
const role = vm.name === SPARE ? "k8s-worker" : "k8s-combo";
|
||||
const res = await new K3sModule().admit(moduleCtx(h, vm, role));
|
||||
for (const line of res.output) console.log(` [${vm.name}] ${line}`); // eslint-disable-line no-console
|
||||
}
|
||||
|
||||
// All four VMs, including the agent: the spare having a schedulable disk
|
||||
// is the whole reason eviction can complete at all.
|
||||
await waitForLonghornReady(h, servers[0]!, vms);
|
||||
|
||||
// Read it back rather than trusting the wait: every node must have a
|
||||
// disk, or replicas silently never land there.
|
||||
//
|
||||
// -o json and parse, NOT jsonpath. A jsonpath containing {"\n"} has to
|
||||
// survive a TS template literal AND a shell, and getting either escape
|
||||
// wrong yields `unterminated quoted string` at runtime — which is how
|
||||
// this assertion failed the first time it ran.
|
||||
const lhNodes = kubectlJson<{ items: {
|
||||
metadata: { name: string };
|
||||
spec: { disks?: Record<string, unknown> };
|
||||
}[] }>(h, servers[0]!, `-n ${LONGHORN_NS} get nodes.longhorn.io`);
|
||||
|
||||
for (const vm of vms) {
|
||||
const name = k8sNodeName(h, vm);
|
||||
const rec = lhNodes.items.find((n) => n.metadata.name === name);
|
||||
expect(rec, `${name} has no Longhorn node record`).toBeDefined();
|
||||
const disks = Object.keys(rec?.spec.disks ?? {});
|
||||
expect(disks.length, `${name} has no Longhorn disk — the default-disks-config ` +
|
||||
`annotation never parsed, so this node would silently host no replicas`)
|
||||
.toBeGreaterThan(0);
|
||||
}
|
||||
});
|
||||
}, 40 * 60_000);
|
||||
|
||||
it("writes known data and proves it survives a detach/attach", async (ctx) => {
|
||||
await step(ctx, "seed data", async () => {
|
||||
const s = servers[0]!;
|
||||
createDataWorkload(h, s, { name: R3, sizeGi: 2, replicas: 3 });
|
||||
createDataWorkload(h, s, { name: R1, sizeGi: 1, replicas: 1 });
|
||||
await waitForWorkload(h, s, R3);
|
||||
await waitForWorkload(h, s, R1);
|
||||
|
||||
digestR3 = await writeBlob(h, s, { name: R3, mib: 512 });
|
||||
digestR1 = await writeBlob(h, s, { name: R1, mib: 128 });
|
||||
|
||||
// Record the digest only AFTER it has survived one restart. Otherwise
|
||||
// the baseline could be a page-cache artefact and every later
|
||||
// comparison would be against a value that was never really on disk.
|
||||
await verifyBlob(h, s, { name: R3, expectDigest: digestR3 });
|
||||
await verifyBlob(h, s, { name: R1, expectDigest: digestR1 });
|
||||
|
||||
const st = volumeState(h, s, R3);
|
||||
expect(st.robustness).toBe("healthy");
|
||||
expect(new Set(st.replicaNodes).size).toBe(3);
|
||||
});
|
||||
}, 60 * 60_000);
|
||||
|
||||
it("backs up a volume to S3 and restores it byte-identical", async (ctx) => {
|
||||
await step(ctx, "backup and restore", async () => {
|
||||
const s = servers[0]!;
|
||||
|
||||
if (backupTarget === undefined) {
|
||||
// Deliberately loud. A backup phase that silently passes because no
|
||||
// target was configured is exactly the kind of green tick that makes
|
||||
// people believe they have backups when they do not.
|
||||
throw new Error(
|
||||
"no backup target configured (DRILL_S3_TARGET / _ENDPOINT / _ACCESS_KEY / _SECRET_KEY). " +
|
||||
"The backup/restore phase cannot prove anything without one.");
|
||||
}
|
||||
|
||||
// Set it on the CR. defaultSettings.backupTarget does nothing in 1.8 --
|
||||
// see setBackupTarget for why.
|
||||
await setBackupTarget(h, s, backupTarget);
|
||||
|
||||
// `available: true` is Longhorn saying it reached the endpoint AND
|
||||
// authenticated. Backing up before checking gives a failure that reads
|
||||
// like a volume problem.
|
||||
await waitForBackupTarget(h, s);
|
||||
|
||||
const { backup } = await backupVolume(h, s, R3);
|
||||
|
||||
// Restore into a SEPARATE volume. The original is untouched, so a failed
|
||||
// restore cannot damage the source -- which is the property that makes
|
||||
// this safe to rehearse at all.
|
||||
await restoreFromBackup(h, s, { backup, name: "drill-restored", sizeGi: 2 });
|
||||
await waitForWorkload(h, s, "drill-restored");
|
||||
|
||||
// The whole point: the bytes that came back are the bytes that went in.
|
||||
// Compared against the digest recorded BEFORE any of this, which was
|
||||
// itself only trusted after surviving a detach/attach.
|
||||
const restored = readBlobDigest(h, s, "drill-restored");
|
||||
expect(restored, "restored volume digest must equal the original").toBe(digestR3);
|
||||
console.log(`[drill] restored digest ${restored.slice(0, 16)}... matches original`); // eslint-disable-line no-console
|
||||
|
||||
// And the restored volume is a real, independent volume.
|
||||
const st = volumeState(h, s, "drill-restored");
|
||||
expect(st.pv).not.toBe(volumeState(h, s, R3).pv);
|
||||
expect(st.state).toBe("attached");
|
||||
});
|
||||
}, 45 * 60_000);
|
||||
|
||||
it("REFUSES to evacuate when there is nowhere for replicas to go", async (ctx) => {
|
||||
await step(ctx, "negative pre-check", async () => {
|
||||
const s = servers[0]!;
|
||||
|
||||
// Must be a node that actually HOLDS an r3 replica, and not the one we
|
||||
// read cluster state from. r3 has 3 replicas across 4 nodes, so one node
|
||||
// has none -- picking blindly makes this test pass or fail on placement
|
||||
// luck rather than on the behaviour it is meant to pin.
|
||||
// replicaNodes are KUBERNETES node names (smoke-a.pxe-test.local), not
|
||||
// libvirt domain names, so resolve before comparing.
|
||||
const r3Nodes = volumeState(h, s, R3).replicaNodes;
|
||||
const target = servers.find(
|
||||
(v) => v.name !== s.name && r3Nodes.includes(k8sNodeName(h, v)));
|
||||
expect(target, `no server other than ${s.name} holds an ${R3} replica (${r3Nodes.join(",")})`)
|
||||
.toBeDefined();
|
||||
|
||||
// Take the spare out of service: now 3 schedulable nodes for a 3-replica
|
||||
// volume, which is exactly the real fleet's shape.
|
||||
kubectlOn(h, s, `-n ${LONGHORN_NS} patch nodes.longhorn.io ${k8sNodeName(h, spare)} ` +
|
||||
`--type=merge -p '{"spec":{"allowScheduling":false}}'`);
|
||||
await sleep(10_000);
|
||||
|
||||
const t0 = Date.now();
|
||||
const res = await new K3sModule().evacuate(moduleCtx(h, target!, "k8s-combo"));
|
||||
const elapsed = (Date.now() - t0) / 1000;
|
||||
|
||||
expect(res.success, "evacuation should have been refused").toBe(false);
|
||||
// Fast is the whole point: the old behaviour was a 60-minute wait.
|
||||
expect(elapsed, `refused in ${elapsed}s`).toBeLessThan(300);
|
||||
expect(res.errors.join(" ")).toMatch(/cannot complete|cannot be placed|wants 3 replicas/);
|
||||
|
||||
// And it must not have touched anything on the way to saying no.
|
||||
const lhNode = kubectlOn(h, s,
|
||||
`-n ${LONGHORN_NS} get nodes.longhorn.io ${k8sNodeName(h, target!)} ` +
|
||||
`-o jsonpath='{.spec.evictionRequested}'`).trim();
|
||||
expect(lhNode, "refusal must not leave an eviction pending").not.toBe("true");
|
||||
|
||||
// Put the spare back and un-cordon whatever the attempt cordoned.
|
||||
kubectlOn(h, s, `-n ${LONGHORN_NS} patch nodes.longhorn.io ${k8sNodeName(h, spare)} ` +
|
||||
`--type=merge -p '{"spec":{"allowScheduling":true}}'`);
|
||||
await new K3sModule().admit(moduleCtx(h, target!, "k8s-combo"));
|
||||
await sleep(15_000);
|
||||
});
|
||||
}, 30 * 60_000);
|
||||
|
||||
// The roll itself. Serial by construction -- each `it` depends on the last.
|
||||
for (const name of SERVERS) {
|
||||
it(`rolls ${name}: evacuate, wipe, reinstall, rejoin, verify`, async (ctx) => {
|
||||
await step(ctx, `roll ${name}`, async () => {
|
||||
const target = vms.find((v) => v.name === name)!;
|
||||
const peer = survivor(target);
|
||||
// Kubernetes and Longhorn both know this machine by its FQDN, not by
|
||||
// the libvirt domain name used everywhere else in this file.
|
||||
const nodeName = k8sNodeName(h, target);
|
||||
|
||||
// --- GATE: never start a roll on a cluster that is already degraded.
|
||||
const pre = clusterSnapshot(h, peer);
|
||||
printSnapshot(`before rolling ${name}`, pre);
|
||||
expect(pre.etcdMembers, "etcd membership before roll").toBe(3);
|
||||
expect(pre.etcdHealthy, "etcd health before roll").toBe(true);
|
||||
expect(pre.readyCount, "all nodes Ready before roll").toBe(4);
|
||||
|
||||
// --- Replica-1 pre-flight, exactly as the runbook prescribes.
|
||||
//
|
||||
// If drill-r1's single replica is on this node, evacuate MUST refuse:
|
||||
// moving it means rebuilding from the only copy in existence, and this
|
||||
// fleet has no backup target. Assert that refusal, then apply the
|
||||
// documented mitigation rather than overriding the guard.
|
||||
const r1Here = replicaOn(peer, R1, nodeName);
|
||||
let r1Raised = false;
|
||||
if (r1Here) {
|
||||
const refused = await new K3sModule().evacuate(moduleCtx(h, target, "k8s-combo"));
|
||||
expect(refused.success, "single-replica volume should have blocked evacuation").toBe(false);
|
||||
expect(refused.errors.join(" ")).toMatch(/single replica/);
|
||||
await new K3sModule().admit(moduleCtx(h, target, "k8s-combo"));
|
||||
r1Raised = await raiseReplicas(peer, R1, 3);
|
||||
}
|
||||
|
||||
// --- Evacuate (cordon -> check -> evict+wait -> drain -> verify).
|
||||
//
|
||||
// If raising drill-r1 did not converge, fall back to the documented
|
||||
// override rather than stalling. The guard itself is already proven by
|
||||
// the refusal above and by the negative pre-check; what this roll is
|
||||
// here to prove is the migration.
|
||||
const evacConfig = r1Here && !r1Raised ? { evictionAllowSingleReplica: true } : {};
|
||||
if (r1Here && !r1Raised) {
|
||||
console.warn( // eslint-disable-line no-console
|
||||
"[drill] proceeding with --allow-single-replica because the raise did not converge");
|
||||
}
|
||||
const evac = await new K3sModule().evacuate(moduleCtx(h, target, "k8s-combo", evacConfig));
|
||||
for (const line of evac.output) console.log(` [${name}] ${line}`); // eslint-disable-line no-console
|
||||
expect(evac.success, `evacuate ${name}: ${evac.errors.join("; ")}`).toBe(true);
|
||||
|
||||
// Replicas really gone, and the r3 volume really rebuilt elsewhere.
|
||||
const afterEvac = await waitForVolumeHealthy(h, peer, { pvc: R3, expectReplicas: 3 });
|
||||
expect(afterEvac.replicaNodes).not.toContain(nodeName);
|
||||
|
||||
// --- Data intact after evacuation, BEFORE anything is destroyed.
|
||||
await verifyBlob(h, peer, { name: R3, expectDigest: digestR3 });
|
||||
await verifyBlob(h, peer, { name: R1, expectDigest: digestR1 });
|
||||
|
||||
// --- The irreversible part starts here and not before.
|
||||
nodeExec(h, target, "systemctl stop k3s || true");
|
||||
const removed = removeEtcdMember(h, peer, nodeName);
|
||||
expect(removed, `etcd member ${nodeName} should have existed`).toBe(true);
|
||||
kubectlOn(h, peer, `delete node ${nodeName} --wait=true --timeout=5m`, 6 * 60_000);
|
||||
|
||||
// Quorum asserted POSITIVELY: 2 of 3 is survivable, silence is not proof.
|
||||
const mid = clusterSnapshot(h, peer);
|
||||
printSnapshot(`${name} removed`, mid);
|
||||
expect(mid.etcdMembers, "etcd must be down to exactly 2 members").toBe(2);
|
||||
expect(mid.etcdHealthy, "surviving etcd must still be healthy").toBe(true);
|
||||
const ready = allServersReady(h, servers.filter((v) => v.name !== name));
|
||||
expect(ready.ok, `survivors /readyz: ${ready.detail}`).toBe(true);
|
||||
|
||||
// --- Wipe and reinstall. The disk decides the schema, not the request.
|
||||
//
|
||||
// reprovisionVm returns a NEW record: the machine can come back on a
|
||||
// different DHCP address. Every later reference must use that record,
|
||||
// and the shared arrays have to be updated too or `survivor()` will
|
||||
// hand out a stale IP on the next roll.
|
||||
const rolled = await reprovisionVm(h, target, "k8s-combo");
|
||||
vms = vms.map((v) => (v.name === name ? rolled : v));
|
||||
servers = servers.map((v) => (v.name === name ? rolled : v));
|
||||
// A reinstalled machine re-registers; do not trust the cached name.
|
||||
forgetNodeName(rolled);
|
||||
|
||||
// v2 reinstall PRESERVES /var/lib/longhorn, so without this the next
|
||||
// step would measure replica re-adoption, not a real rebuild. The
|
||||
// actual v1->v2 migration recreates the VG and has no such leftovers.
|
||||
//
|
||||
// Clear the CONTENTS, never the directory itself. Longhorn's replica
|
||||
// process does a non-recursive `mkdir /host/var/lib/longhorn/replicas/
|
||||
// <name>`, which fails outright when the parent is missing:
|
||||
//
|
||||
// level=fatal msg="Error running start replica command"
|
||||
// error="mkdir /host/var/lib/longhorn/replicas/pvc-...: ..."
|
||||
//
|
||||
// Every replica scheduled onto the node then dies instantly and
|
||||
// permanently, while the node looks entirely healthy: longhorn-disk.cfg
|
||||
// survives the wipe, so the disk keeps its UUID and reports Ready and
|
||||
// Schedulable. The next node's eviction can never drain onto it and
|
||||
// stalls for its full hour. Two earlier drill runs died exactly here.
|
||||
nodeExec(h, rolled,
|
||||
"rm -rf /var/lib/longhorn/replicas/* /var/lib/longhorn/*.tmp 2>/dev/null; " +
|
||||
"mkdir -p /var/lib/longhorn/replicas && ls -d /var/lib/longhorn/replicas");
|
||||
|
||||
// --- Rejoin against a SURVIVOR, not against itself.
|
||||
await waitForApiPing(peer);
|
||||
await installK3sNode(h, rolled, {
|
||||
role: "k8s-combo", token: DRILL_TOKEN, serverUrl: serverUrlOf(peer),
|
||||
});
|
||||
await waitForNodeReady(h, peer, rolled);
|
||||
|
||||
// Exactly 3. Four would mean the old member survived the rejoin, which
|
||||
// silently raises quorum from 2-of-3 to 3-of-4 and removes all margin.
|
||||
const post = clusterSnapshot(h, peer);
|
||||
printSnapshot(`${name} rejoined`, post);
|
||||
expect(post.etcdMembers, "stale etcd member left behind").toBe(3);
|
||||
expect(post.etcdHealthy).toBe(true);
|
||||
expect(post.readyCount).toBe(4);
|
||||
|
||||
// --- Back into service, then wait for the replica to rebuild onto it.
|
||||
const admit = await new K3sModule().admit(moduleCtx(h, rolled, "k8s-combo"));
|
||||
for (const line of admit.output) console.log(` [${name}] ${line}`); // eslint-disable-line no-console
|
||||
expect(admit.success, `admit ${name}: ${admit.errors.join("; ")}`).toBe(true);
|
||||
|
||||
const healthy = await waitForVolumeHealthy(h, peer, { pvc: R3, expectReplicas: 3 });
|
||||
expect(new Set(healthy.replicaNodes).size).toBe(3);
|
||||
|
||||
// --- And the data is still the data.
|
||||
await verifyBlob(h, peer, { name: R3, expectDigest: digestR3 });
|
||||
await verifyBlob(h, peer, { name: R1, expectDigest: digestR1 });
|
||||
|
||||
// Put drill-r1 back to a single replica, so the next roll faces the
|
||||
// same hazard rather than a progressively safer cluster.
|
||||
if (r1Here && r1Raised) await raiseReplicas(peer, R1, 1);
|
||||
|
||||
// --- Does the rolled node actually take replicas BACK?
|
||||
//
|
||||
// Roll 1 originally "passed" with drill-r3 healthy on three nodes that
|
||||
// did not include the node just rolled -- the assertion counted
|
||||
// distinct nodes without ever checking that the reinstalled one was
|
||||
// among them. A node that rejoins, reports Ready and Schedulable, and
|
||||
// then hosts nothing is the exact failure this whole exercise exists to
|
||||
// catch, so record it explicitly.
|
||||
// RUNNING, not merely assigned. The first version of this check used
|
||||
// replicaNodes (which comes from spec.nodeID) and happily passed on a
|
||||
// node holding two replicas that were both state=stopped -- exactly the
|
||||
// condition that makes the NEXT eviction impossible, because Longhorn
|
||||
// keeps scheduling replacements onto a node that cannot start them.
|
||||
const running = kubectlJson<{ items: {
|
||||
spec: { nodeID?: string };
|
||||
status?: { currentState?: string };
|
||||
}[] }>(h, peer, `-n ${LONGHORN_NS} get replicas.longhorn.io`)
|
||||
.items.filter((r) => r.spec.nodeID === nodeName && r.status?.currentState === "running");
|
||||
console.log( // eslint-disable-line no-console
|
||||
`[drill] after rolling ${name}: ${running.length} RUNNING replica(s) on the node`);
|
||||
expect(running.length,
|
||||
`${name} rejoined but hosts no RUNNING replica — it would look healthy and ` +
|
||||
`silently take no storage, and the next eviction could never drain onto it`)
|
||||
.toBeGreaterThan(0);
|
||||
});
|
||||
}, 240 * 60_000);
|
||||
}
|
||||
|
||||
it("ends with a whole cluster and the original bytes", async (ctx) => {
|
||||
await step(ctx, "final verification", async () => {
|
||||
const s = servers[0]!;
|
||||
const snap = clusterSnapshot(h, s);
|
||||
printSnapshot("end of drill", snap);
|
||||
|
||||
expect(snap.nodes).toHaveLength(4);
|
||||
expect(snap.readyCount).toBe(4);
|
||||
expect(snap.etcdMembers).toBe(3);
|
||||
expect(snap.etcdHealthy).toBe(true);
|
||||
|
||||
const r3 = volumeState(h, s, R3);
|
||||
expect(r3.robustness).toBe("healthy");
|
||||
expect(new Set(r3.replicaNodes).size).toBe(3);
|
||||
|
||||
// One last restart-and-check, so the final claim is about bytes read
|
||||
// back through a fresh attach rather than a cached mount.
|
||||
await verifyBlob(h, s, { name: R3, expectDigest: digestR3 });
|
||||
await verifyBlob(h, s, { name: R1, expectDigest: digestR1 });
|
||||
});
|
||||
}, 60 * 60_000);
|
||||
});
|
||||
Reference in New Issue
Block a user