v2 partition schema: capability model, fixed sizing, and a proven rolling migration
Some checks failed
Some checks failed
The fleet's disk layout was decided by literal role-name comparisons at 17 sites across 8 files. That is why `labcontroller` got no rancher LV (it matched neither "infra" nor "worker"), why labmaster carried an empty 728 GiB Longhorn LV, and why a 62 GiB container image tipped that node into DiskPressure and evicted bastion, labd, cockroach and openbao. Three generators each hardcoded their own sizes and disagreed: the kickstart said rancher=120G, ubuntu-autoinstall said 20G, asahi-firstboot said 20G under a comment asserting the sizes matched install.ks.ts. They did not. ROLE_REGISTRY already carried `k3s` and `parent` and nothing read them. This makes capabilities authoritative, moves every size into one shared module, and adds schema v2 (fixed 250G rancher / 300G longhorn, never --grow). Existing machines stay v1 forever: schema is detected from the on-disk VG tag, so a v1 box stays v1 even when v2 is requested. Proven on real VMs, not just unit tests: all four topologies installed and had their layouts read back off the machines; markers in /home, /srv, /var/lib/longhorn and /var/lib/rancher survived reinstall; and a 4-node cluster formed, took real data, and had all three servers rolled one at a time through evacuate / wipe / reinstall / rejoin with checksums verified before and after each roll. Drill: 10/10, evidence in migration/drill-evidence/. Production defects the drill found, none of which a unit test could reach: - sshOpts() capped every operation at 30s while drainNode asked kubectl for --timeout=15m, so the transport killed the drain after evicting most pods and returned a transcript that read like success. `labctl app k3s evacuate` could never have drained a real node. - generateAgentConfig put node-role.kubernetes.io/worker in the kubelet's node-label list, which NodeRestriction rejects, so no agent installed through K3sModule could start. systemd reported only "did not take the steps required by its unit configuration". - toOpContext built K3sConfig by enumerating three keys and silently dropped the rest, so the k3s version pin did nothing and --allow-single-replica was a no-op that claimed to override a data-loss guard. - The Longhorn disk annotation was mangled by shell quoting; with createDefaultDiskLabeledNodes the node then gets no disk at all, joins, goes Ready, and silently hosts no replica. - evacuate drained before evicting, which blocks on Longhorn's PDBs and would destroy a single-replica volume's only source. - etcd member ids were parsed from JSON, losing uint64 precision and emitting decimal where `member remove` needs hex -- at the moment a node is stopped and about to be wiped. - get.k3s.io installs latest, two minor versions ahead of the fleet, so a reinstalled node would rejoin on a newer k3s than its peers and roll an unplanned two-version control-plane upgrade. K3sConfig.k3sVersion now pins it. - `labctl app k3s install --role x` mapped anything that was not "worker" to "infra", so the runbook's own `--role k8s-combo` would have installed a different role with different disk expectations and no warning. Also: /var/lib/longhorn/replicas must never be deleted, only its contents, or every replica placed on the node dies on a non-recursive mkdir -- for ever, while the node reports Ready, Schedulable and healthy. etcd is now in the kickstart's package set for k3s server roles, because `k3s etcdctl` is not a command and a member that needs to recover its own membership has no internet. 542 unit tests / 39 files. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Michal
427
bastion/tests/integration/helpers/k3s-cluster.ts
Normal file
427
bastion/tests/integration/helpers/k3s-cluster.ts
Normal file
@@ -0,0 +1,427 @@
|
||||
// Form and inspect a real multi-node k3s cluster from an integration test.
|
||||
//
|
||||
// WHY THIS EXISTS
|
||||
// ---------------
|
||||
// Nothing in the repo decides which node bootstraps a cluster and which nodes
|
||||
// join it. The production generator has the right discriminator --
|
||||
// generateServerConfig emits `cluster-init: true` when no server URL is set and
|
||||
// `server:`+`token:` when one is -- but the choice is an INPUT, and the only
|
||||
// caller that ever supplied it was a hand-run labsim shell script.
|
||||
//
|
||||
// So a test that wants three servers in one cluster has to sequence it. That
|
||||
// sequencing is here rather than in a test file because the rolling drill is
|
||||
// not the only thing that will want it: an etcd-recovery test (recoverEtcdMember
|
||||
// still has no caller), a dual-stack conversion and any HA upgrade test all
|
||||
// need exactly these primitives.
|
||||
//
|
||||
// Everything drives K3sModule, not shell. That is deliberate and is the same
|
||||
// rule labsim follows by shelling out to the compiled render-config binary: a
|
||||
// test that reimplements the install proves the test works, not the product.
|
||||
|
||||
import { execSync } from "node:child_process";
|
||||
import type { Role } from "../../../src/shared/src/types/state.js";
|
||||
import { K3sModule } from "../../../src/modules/modules/k3s/src/k3s-module.js";
|
||||
import type { ModuleContext } from "../../../src/modules/src/types.js";
|
||||
import { log } from "./libvirt.js";
|
||||
import { parseEtcdMembers, type EtcdMember } from "../../../src/modules/modules/k3s/src/operations/etcd-members.js";
|
||||
import { sshExec } from "./ssh.js";
|
||||
import type { Harness, ProvisionedVm } from "./provision-harness.js";
|
||||
|
||||
export const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));
|
||||
|
||||
/**
|
||||
* A pre-shared cluster token, the way labsim does it.
|
||||
*
|
||||
* The alternative is scraping /var/lib/rancher/k3s/server/node-token off the
|
||||
* first server, and nothing in the repo does that. It is also the wrong shape
|
||||
* for a rolling migration: the first server gets wiped and rejoins, so the
|
||||
* token has to outlive it and be knowable before it exists.
|
||||
*/
|
||||
export const DRILL_TOKEN = "lab-drill-shared-token";
|
||||
|
||||
/**
|
||||
* k3s operations run unprefixed `mkdir -p /etc/rancher/k3s` and
|
||||
* `systemctl restart k3s`, so they need root. install.ks.ts provisions
|
||||
* `sshkey --username=root` and sets PermitRootLogin prohibit-password, so root
|
||||
* SSH works on every machine this harness builds. The harness's own `lab` user
|
||||
* stays for the sudo-based reads (readLvs, markers).
|
||||
*/
|
||||
export const MODULE_SSH_USER = "root";
|
||||
|
||||
/**
|
||||
* The k3s version the drill runs, pinned to what the fleet runs.
|
||||
*
|
||||
* get.k3s.io installs LATEST, and latest is currently two minor versions ahead
|
||||
* of production (v1.36.4 vs v1.34.6). Two things follow, and both matter:
|
||||
*
|
||||
* - a rehearsal on a different minor version is rehearsing a different
|
||||
* system, and Longhorn in particular is only qualified against a range of
|
||||
* Kubernetes versions;
|
||||
* - it is the same trap production would hit. A node reinstalled mid-migration
|
||||
* comes back on latest while its peers stay put, and Kubernetes supports
|
||||
* only ONE minor of skew between control-plane members.
|
||||
*
|
||||
* Override with LAB_K3S_VERSION to test an upgrade deliberately.
|
||||
*/
|
||||
export const PINNED_K3S_VERSION = process.env["LAB_K3S_VERSION"] ?? "v1.34.6+k3s1";
|
||||
|
||||
/** Build the ModuleContext K3sModule expects for one VM. */
|
||||
export function moduleCtx(
|
||||
h: Harness,
|
||||
vm: ProvisionedVm,
|
||||
role: Role,
|
||||
config: Record<string, unknown> = {},
|
||||
): ModuleContext {
|
||||
return {
|
||||
hostname: vm.name,
|
||||
ip: vm.ip,
|
||||
role,
|
||||
os: "fedora-43",
|
||||
arch: "x86_64",
|
||||
sshUser: MODULE_SSH_USER,
|
||||
sshKeyPath: h.sshKeyPath,
|
||||
config,
|
||||
};
|
||||
}
|
||||
|
||||
/** Run a command on a node as root, throwing with context on failure. */
|
||||
export function nodeExec(h: Harness, vm: ProvisionedVm, cmd: string, timeoutMs = 120_000): string {
|
||||
const r = sshExec(vm.ip, MODULE_SSH_USER, cmd, { keyPath: h.sshKeyPath, timeout: timeoutMs });
|
||||
if (r.exitCode !== 0) {
|
||||
throw new Error(`[${vm.name}] ${cmd} failed (${r.exitCode}): ${r.stderr.trim() || r.stdout.trim()}`);
|
||||
}
|
||||
return r.stdout;
|
||||
}
|
||||
|
||||
/** Same, but returns the result instead of throwing. */
|
||||
export function tryNodeExec(h: Harness, vm: ProvisionedVm, cmd: string, timeoutMs = 120_000) {
|
||||
return sshExec(vm.ip, MODULE_SSH_USER, cmd, { keyPath: h.sshKeyPath, timeout: timeoutMs });
|
||||
}
|
||||
|
||||
/**
|
||||
* The name Kubernetes knows this VM by.
|
||||
*
|
||||
* NOT the libvirt domain name. k3s registers a node under its FQDN, so a VM
|
||||
* called `smoke-a` appears as `smoke-a.pxe-test.local`. Passing the VM name to
|
||||
* `kubectl get node` silently finds nothing, and every wait built on it times
|
||||
* out instead of failing usefully. Read from the node itself, the same way the
|
||||
* production operations do (`hostname -f`), so the two can never disagree.
|
||||
*/
|
||||
const nodeNameCache = new Map<string, string>();
|
||||
export function k8sNodeName(h: Harness, vm: ProvisionedVm): string {
|
||||
const cached = nodeNameCache.get(vm.name);
|
||||
if (cached !== undefined) return cached;
|
||||
const name = nodeExec(h, vm, "hostname -f 2>/dev/null || hostname").trim();
|
||||
if (name === "") throw new Error(`[${vm.name}] could not resolve node name`);
|
||||
nodeNameCache.set(vm.name, name);
|
||||
return name;
|
||||
}
|
||||
|
||||
/** Forget cached names — a reinstalled machine re-registers. */
|
||||
export function forgetNodeName(vm: ProvisionedVm): void {
|
||||
nodeNameCache.delete(vm.name);
|
||||
}
|
||||
|
||||
/** kubectl against a server node. */
|
||||
export function kubectlOn(h: Harness, server: ProvisionedVm, args: string, timeoutMs = 120_000): string {
|
||||
return nodeExec(h, server, `k3s kubectl ${args}`, timeoutMs);
|
||||
}
|
||||
|
||||
export function kubectlJson<T>(h: Harness, server: ProvisionedVm, args: string): T {
|
||||
return JSON.parse(kubectlOn(h, server, `${args} -o json`)) as T;
|
||||
}
|
||||
|
||||
/**
|
||||
* Wait until a server's API answers /ping.
|
||||
*
|
||||
* Straight from labsim-k8s-etcd.sh, whose comment is the whole justification:
|
||||
* a joiner that starts before the first server has finished bootstrapping etcd
|
||||
* races it, and the failure looks like etcd instability rather than a startup
|
||||
* order problem. Probed from the HOST, so it does not depend on the joining
|
||||
* node being reachable yet.
|
||||
*/
|
||||
export async function waitForApiPing(vm: ProvisionedVm, timeoutMs = 450_000): Promise<void> {
|
||||
const start = Date.now();
|
||||
let lastErr = "";
|
||||
while (Date.now() - start < timeoutMs) {
|
||||
try {
|
||||
const out = execSync(
|
||||
`curl -sk --max-time 5 https://${vm.ip}:6443/ping || true`,
|
||||
{ encoding: "utf-8", timeout: 15_000 },
|
||||
).trim();
|
||||
if (out === "pong") {
|
||||
log(`[${vm.name}] API answering after ${Math.round((Date.now() - start) / 1000)}s`);
|
||||
return;
|
||||
}
|
||||
lastErr = out;
|
||||
} catch (e) {
|
||||
lastErr = e instanceof Error ? e.message : String(e);
|
||||
}
|
||||
await sleep(5_000);
|
||||
}
|
||||
throw new Error(`[${vm.name}] API never answered /ping in ${timeoutMs / 1000}s (last: ${lastErr || "no response"})`);
|
||||
}
|
||||
|
||||
/** Wait for a node to appear Ready in the cluster, as seen from `server`. */
|
||||
export async function waitForNodeReady(
|
||||
h: Harness,
|
||||
server: ProvisionedVm,
|
||||
/** The VM to wait for; its Kubernetes name is resolved from the machine. */
|
||||
target: ProvisionedVm,
|
||||
// Generous: a node is NotReady until its CNI is up, and on the cluster-init
|
||||
// server that means waiting for Cilium to install and pull its images. Five
|
||||
// minutes is enough on a warm host and not enough on a cold one, and the
|
||||
// cost of being wrong is a wasted multi-hour run.
|
||||
timeoutMs = 900_000,
|
||||
): Promise<void> {
|
||||
const nodeName = k8sNodeName(h, target);
|
||||
const start = Date.now();
|
||||
let last = "";
|
||||
while (Date.now() - start < timeoutMs) {
|
||||
const r = tryNodeExec(h, server,
|
||||
`k3s kubectl get node ${nodeName} -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}'`);
|
||||
last = r.stdout.trim();
|
||||
if (last === "True") {
|
||||
log(`[${nodeName}] Ready after ${Math.round((Date.now() - start) / 1000)}s`);
|
||||
return;
|
||||
}
|
||||
await sleep(5_000);
|
||||
}
|
||||
throw new Error(`[${nodeName}] never became Ready in ${timeoutMs / 1000}s (last status: ${last || "absent"})`);
|
||||
}
|
||||
|
||||
/**
|
||||
* etcdctl invocation.
|
||||
*
|
||||
* NOT `k3s etcdctl` -- that is not a subcommand ("No help topic for 'etcdctl'",
|
||||
* verified on a live v1.34 node). The binary comes from the `etcd` rpm, which
|
||||
* our kickstart does NOT install, so ensureEtcdctl() has to put it there.
|
||||
* Cert paths verified against a real node.
|
||||
*/
|
||||
const ETCD_BASE = "/var/lib/rancher/k3s/server/tls/etcd";
|
||||
const ETCDCTL =
|
||||
`ETCDCTL_API=3 etcdctl ` +
|
||||
`--cacert=${ETCD_BASE}/server-ca.crt ` +
|
||||
`--cert=${ETCD_BASE}/server-client.crt ` +
|
||||
`--key=${ETCD_BASE}/server-client.key ` +
|
||||
`--endpoints=https://127.0.0.1:2379`;
|
||||
|
||||
/** Install etcdctl if the node does not have it. Idempotent, ~30s once. */
|
||||
export function ensureEtcdctl(h: Harness, server: ProvisionedVm): void {
|
||||
const has = tryNodeExec(h, server, "command -v etcdctl");
|
||||
if (has.exitCode === 0) return;
|
||||
log(`[${server.name}] installing etcdctl (the etcd rpm; kickstart does not include it)`);
|
||||
nodeExec(h, server, "dnf install -y etcd >/dev/null 2>&1 || dnf install -y etcd-ctl >/dev/null 2>&1", 300_000);
|
||||
nodeExec(h, server, "command -v etcdctl");
|
||||
}
|
||||
|
||||
|
||||
/**
|
||||
* etcd membership, read with etcdctl against k3s's own certs.
|
||||
*
|
||||
* Read rather than inferred from `kubectl get nodes`: a node object can be
|
||||
* deleted while its etcd member survives, and that stale member is exactly the
|
||||
* failure a rolling reinstall creates. Counting nodes would not see it.
|
||||
*/
|
||||
export function etcdMembers(h: Harness, server: ProvisionedVm): EtcdMember[] {
|
||||
// NOT `-w json`. Two reasons, and the second is the dangerous one:
|
||||
//
|
||||
// 1. `member remove` requires the ID in HEX; the JSON writer emits decimal.
|
||||
// 2. etcd member IDs are uint64. JSON.parse turns them into JS numbers,
|
||||
// which silently lose precision above 2^53 -- the ID that comes back is
|
||||
// an APPROXIMATION of the real one. Passing that to `member remove`
|
||||
// fails with "value out of range" if you are lucky, and in principle
|
||||
// could match some other member if you are not. This happens at the one
|
||||
// moment it must not: the node is already stopped and about to be wiped.
|
||||
//
|
||||
// The default writer prints the ID as hex text, so it never becomes a
|
||||
// number at all:
|
||||
// 825852d3683bc406, started, worker2-...-5f7ad0c9, https://...:2380, ...
|
||||
return parseEtcdMembers(nodeExec(h, server, `${ETCDCTL} member list`));
|
||||
}
|
||||
|
||||
|
||||
/** Are all etcd endpoints healthy? Answers the question quorum actually asks. */
|
||||
export function etcdHealthy(h: Harness, server: ProvisionedVm): boolean {
|
||||
const r = tryNodeExec(h, server, `${ETCDCTL} endpoint health`);
|
||||
return r.exitCode === 0 && r.stdout.includes("is healthy");
|
||||
}
|
||||
|
||||
/**
|
||||
* Remove an etcd member by node name.
|
||||
*
|
||||
* Must happen BEFORE the node is wiped. k3s-install.ts removes the local
|
||||
* datastore when a server joins, so a reinstalled node rejoins as a NEW member
|
||||
* while the old record persists: 3 members becomes 4, quorum rises from 2 to 3,
|
||||
* and the cluster that tolerated one failure now tolerates none.
|
||||
*/
|
||||
export function removeEtcdMember(h: Harness, peer: ProvisionedVm, nodeName: string): boolean {
|
||||
const members = etcdMembers(h, peer);
|
||||
const target = members.find((m) => m.node === nodeName || m.name === nodeName);
|
||||
if (!target) {
|
||||
log(`[etcd] no member for ${nodeName} (have: ${members.map((m) => m.node).join(",")})`);
|
||||
return false;
|
||||
}
|
||||
nodeExec(h, peer, `${ETCDCTL} member remove ${target.id}`);
|
||||
log(`[etcd] removed member ${target.name} (${target.id})`);
|
||||
return true;
|
||||
}
|
||||
|
||||
export interface ClusterSnapshot {
|
||||
nodes: { name: string; ready: boolean; roles: string }[];
|
||||
readyCount: number;
|
||||
etcdMembers: number;
|
||||
etcdHealthy: boolean;
|
||||
}
|
||||
|
||||
/** One read of cluster state, in the shape labsim's cmd_status prints. */
|
||||
export function clusterSnapshot(h: Harness, server: ProvisionedVm): ClusterSnapshot {
|
||||
const raw = kubectlJson<{ items: {
|
||||
metadata: { name: string; labels?: Record<string, string> };
|
||||
status: { conditions?: { type: string; status: string }[] };
|
||||
}[] }>(h, server, "get nodes");
|
||||
|
||||
const nodes = raw.items.map((n) => ({
|
||||
name: n.metadata.name,
|
||||
ready: (n.status.conditions ?? []).some((c) => c.type === "Ready" && c.status === "True"),
|
||||
roles: Object.keys(n.metadata.labels ?? {})
|
||||
.filter((k) => k.startsWith("node-role.kubernetes.io/"))
|
||||
.map((k) => k.replace("node-role.kubernetes.io/", ""))
|
||||
.sort().join(",") || "<none>",
|
||||
}));
|
||||
|
||||
let members = 0;
|
||||
let healthy = false;
|
||||
try {
|
||||
members = etcdMembers(h, server).length;
|
||||
healthy = etcdHealthy(h, server);
|
||||
} catch { /* an agent, or the API is down — report zero rather than throw */ }
|
||||
|
||||
return { nodes, readyCount: nodes.filter((n) => n.ready).length, etcdMembers: members, etcdHealthy: healthy };
|
||||
}
|
||||
|
||||
/**
|
||||
* Print a snapshot. Called liberally during a drill: the VMs are destroyed at
|
||||
* the end, so this transcript is the only surviving record of what happened.
|
||||
*/
|
||||
export function printSnapshot(label: string, s: ClusterSnapshot): void {
|
||||
log(`[snapshot] ${label}: ${s.readyCount}/${s.nodes.length} Ready, ` +
|
||||
`etcd members=${s.etcdMembers} healthy=${s.etcdHealthy}`);
|
||||
for (const n of s.nodes) log(` ${n.ready ? "Ready " : "NotReady"} ${n.name} (${n.roles})`);
|
||||
}
|
||||
|
||||
/**
|
||||
* Install k3s on one node via the production module.
|
||||
*
|
||||
* Pass no serverUrl to bootstrap the cluster (the token still seeds), or a
|
||||
* serverUrl to join an existing one.
|
||||
*/
|
||||
export async function installK3sNode(h: Harness, vm: ProvisionedVm, opts: {
|
||||
role: Role;
|
||||
serverUrl?: string;
|
||||
token?: string;
|
||||
}): Promise<void> {
|
||||
const mod = new K3sModule();
|
||||
const config: Record<string, unknown> = {
|
||||
k3sToken: opts.token ?? DRILL_TOKEN,
|
||||
k3sVersion: PINNED_K3S_VERSION,
|
||||
...(opts.serverUrl === undefined ? {} : { k3sServerUrl: opts.serverUrl }),
|
||||
};
|
||||
|
||||
log(`[${vm.name}] installing k3s (${opts.role}${opts.serverUrl ? ` -> ${opts.serverUrl}` : " — cluster-init"})`);
|
||||
const install = await mod.install(moduleCtx(h, vm, opts.role, config));
|
||||
for (const line of install.output) log(` [${vm.name}] ${line}`);
|
||||
if (!install.success) {
|
||||
throw new Error(`[${vm.name}] k3s install failed: ${install.errors.join("; ")}`);
|
||||
}
|
||||
|
||||
const configure = await mod.configure(moduleCtx(h, vm, opts.role, config));
|
||||
for (const line of configure.output) log(` [${vm.name}] ${line}`);
|
||||
if (!configure.success) {
|
||||
throw new Error(`[${vm.name}] k3s configure failed: ${configure.errors.join("; ")}`);
|
||||
}
|
||||
}
|
||||
|
||||
export function serverUrlOf(vm: ProvisionedVm): string {
|
||||
return `https://${vm.ip}:6443`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Form a cluster from a list of server VMs, strictly in order.
|
||||
*
|
||||
* The first bootstraps etcd and installs the CNI (k3s-module runs networking
|
||||
* only on the cluster-init node -- joiners get Cilium via its DaemonSet). The
|
||||
* rest join one at a time, each gated on the previous node being Ready.
|
||||
*
|
||||
* Serial on purpose. labsim gets away with starting joiners together because
|
||||
* its nodes are cloud-init VMs that poll /ping themselves; here the few extra
|
||||
* minutes buy determinism in the one place where etcd bootstrap genuinely
|
||||
* races.
|
||||
*/
|
||||
export async function formCluster(h: Harness, servers: ProvisionedVm[], opts: {
|
||||
role?: Role;
|
||||
token?: string;
|
||||
} = {}): Promise<void> {
|
||||
if (servers.length === 0) throw new Error("formCluster: no servers");
|
||||
const role = opts.role ?? "k8s-combo";
|
||||
const token = opts.token ?? DRILL_TOKEN;
|
||||
const first = servers[0]!;
|
||||
|
||||
await installK3sNode(h, first, { role, token });
|
||||
await waitForApiPing(first);
|
||||
await waitForNodeReady(h, first, first);
|
||||
ensureEtcdctl(h, first);
|
||||
printSnapshot("after cluster-init", clusterSnapshot(h, first));
|
||||
|
||||
for (const vm of servers.slice(1)) {
|
||||
await waitForApiPing(first);
|
||||
await installK3sNode(h, vm, { role, token, serverUrl: serverUrlOf(first) });
|
||||
await waitForNodeReady(h, first, vm);
|
||||
ensureEtcdctl(h, vm);
|
||||
printSnapshot(`after joining ${vm.name}`, clusterSnapshot(h, first));
|
||||
}
|
||||
}
|
||||
|
||||
/** Join an agent. Agents cannot start without a server URL and token. */
|
||||
export async function joinAgent(h: Harness, vm: ProvisionedVm, opts: {
|
||||
server: ProvisionedVm;
|
||||
role?: Role;
|
||||
token?: string;
|
||||
}): Promise<void> {
|
||||
await waitForApiPing(opts.server);
|
||||
await installK3sNode(h, vm, {
|
||||
role: opts.role ?? "k8s-worker",
|
||||
token: opts.token ?? DRILL_TOKEN,
|
||||
serverUrl: serverUrlOf(opts.server),
|
||||
});
|
||||
await waitForNodeReady(h, opts.server, vm);
|
||||
}
|
||||
|
||||
/**
|
||||
* A healthy server that is NOT the node being worked on.
|
||||
*
|
||||
* Needed because the first server is itself rolled during a migration: after
|
||||
* it is wiped it must rejoin against a survivor, so "the first node" cannot be
|
||||
* the join target and cannot be where cluster state is read from.
|
||||
*/
|
||||
export function pickJoinTarget(
|
||||
h: Harness,
|
||||
servers: ProvisionedVm[],
|
||||
exclude: ProvisionedVm,
|
||||
): ProvisionedVm {
|
||||
for (const vm of servers) {
|
||||
if (vm.name === exclude.name) continue;
|
||||
const r = tryNodeExec(h, vm, "k3s kubectl get --raw /readyz");
|
||||
if (r.exitCode === 0 && r.stdout.trim() === "ok") return vm;
|
||||
}
|
||||
throw new Error(`No healthy server other than ${exclude.name} — cluster has no quorum to join against`);
|
||||
}
|
||||
|
||||
/** Every server reports /readyz ok. Asserted positively, not inferred. */
|
||||
export function allServersReady(h: Harness, servers: ProvisionedVm[]): { ok: boolean; detail: string } {
|
||||
const results = servers.map((vm) => {
|
||||
const r = tryNodeExec(h, vm, "k3s kubectl get --raw /readyz");
|
||||
return `${vm.name}=${r.exitCode === 0 ? r.stdout.trim() : `ERR(${r.exitCode})`}`;
|
||||
});
|
||||
return { ok: results.every((r) => r.endsWith("=ok")), detail: results.join(" ") };
|
||||
}
|
||||
666
bastion/tests/integration/helpers/longhorn.ts
Normal file
666
bastion/tests/integration/helpers/longhorn.ts
Normal file
@@ -0,0 +1,666 @@
|
||||
// Install Longhorn on a test cluster and put verifiable data in it.
|
||||
//
|
||||
// WHAT THIS IS FOR
|
||||
// ----------------
|
||||
// The rolling drill's whole claim is "the bytes survived". That needs data
|
||||
// whose integrity can be checked independently of Longhorn's own opinion of
|
||||
// itself -- Longhorn reporting `robustness: healthy` is not evidence that the
|
||||
// file is intact, it is evidence that Longhorn thinks it is.
|
||||
//
|
||||
// So: write a known blob, record its sha256, and re-verify THROUGH A POD
|
||||
// RESTART every time. The restart is not incidental. It forces a real
|
||||
// detach/attach, frequently onto a different node, which is the only way to
|
||||
// show the data followed the volume rather than sitting in a page cache on the
|
||||
// node that wrote it.
|
||||
//
|
||||
// Longhorn version is pinned to the fleet's (1.8.1) so the drill rehearses what
|
||||
// production runs, not what happens to be latest.
|
||||
|
||||
import type { Harness, ProvisionedVm } from "./provision-harness.js";
|
||||
import { log } from "./libvirt.js";
|
||||
import { nodeExec, tryNodeExec, kubectlOn, kubectlJson, sleep, k8sNodeName } from "./k3s-cluster.js";
|
||||
|
||||
export const LONGHORN_VERSION = "1.8.1";
|
||||
export const LONGHORN_NS = "longhorn-system";
|
||||
|
||||
/**
|
||||
* Install Longhorn via k3s's built-in helm controller.
|
||||
*
|
||||
* A HelmChart manifest dropped in the server manifests directory is how k3s
|
||||
* installs charts natively, so this needs no helm binary on the host and no
|
||||
* kubeconfig plumbing.
|
||||
*
|
||||
* The namespace is created FIRST, with privileged Pod Security labels. Longhorn
|
||||
* needs hostPath and privileged containers; under the default `restricted`
|
||||
* enforcement its DaemonSets are rejected and the install fails in a way that
|
||||
* looks like a chart problem.
|
||||
*
|
||||
* `createDefaultDiskLabeledNodes: true` matches production, and is the setting
|
||||
* that makes the node label and disk annotation load-bearing: without both, a
|
||||
* node gets no disk and silently hosts no replicas.
|
||||
*/
|
||||
/**
|
||||
* S3 backup target for the drill, from the environment.
|
||||
*
|
||||
* Absent means the backup/restore phase is skipped rather than silently
|
||||
* "passing" -- a backup test that quietly does nothing is worse than no test.
|
||||
* The run script exports these; see DRILL_S3_* in run-drill.sh.
|
||||
*/
|
||||
export interface BackupTargetOpts {
|
||||
/** e.g. s3://longhorn-drill-backup@us-east-1/ */
|
||||
target: string;
|
||||
endpoint: string;
|
||||
accessKey: string;
|
||||
secretKey: string;
|
||||
}
|
||||
|
||||
export function backupTargetFromEnv(): BackupTargetOpts | undefined {
|
||||
const target = process.env["DRILL_S3_TARGET"];
|
||||
const endpoint = process.env["DRILL_S3_ENDPOINT"];
|
||||
const accessKey = process.env["DRILL_S3_ACCESS_KEY"];
|
||||
const secretKey = process.env["DRILL_S3_SECRET_KEY"];
|
||||
if (target === undefined || endpoint === undefined
|
||||
|| accessKey === undefined || secretKey === undefined) return undefined;
|
||||
return { target, endpoint, accessKey, secretKey };
|
||||
}
|
||||
|
||||
export const BACKUP_SECRET = "longhorn-backup-credentials";
|
||||
|
||||
export async function installLonghorn(h: Harness, server: ProvisionedVm, opts: {
|
||||
defaultReplicas?: number;
|
||||
backup?: BackupTargetOpts;
|
||||
} = {}): Promise<void> {
|
||||
const replicas = opts.defaultReplicas ?? 3;
|
||||
|
||||
log(`[longhorn] creating ${LONGHORN_NS} with privileged PSA labels`);
|
||||
nodeExec(h, server, `k3s kubectl apply -f - <<'EOF'
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: ${LONGHORN_NS}
|
||||
labels:
|
||||
pod-security.kubernetes.io/enforce: privileged
|
||||
pod-security.kubernetes.io/audit: privileged
|
||||
pod-security.kubernetes.io/warn: privileged
|
||||
EOF`);
|
||||
|
||||
if (opts.backup !== undefined) {
|
||||
// Secret BEFORE the chart: Longhorn reads it by name from a defaultSetting,
|
||||
// and a missing secret makes the first backup fail with a credential error
|
||||
// that reads like a network problem.
|
||||
//
|
||||
// AWS_ENDPOINTS is what makes this work against a non-AWS gateway; without
|
||||
// it the driver resolves real AWS endpoints.
|
||||
log(`[longhorn] configuring backup target ${opts.backup.target}`);
|
||||
// tryNodeExec, not nodeExec: nodeExec throws with the whole command in the
|
||||
// message, and this command contains the S3 secret. A credential landing in
|
||||
// a log that then gets pasted into drill evidence is not acceptable, so the
|
||||
// failure is reported without the payload.
|
||||
const secretApplied = tryNodeExec(h, server, `k3s kubectl -n ${LONGHORN_NS} apply -f - <<'EOF'
|
||||
apiVersion: v1
|
||||
kind: Secret
|
||||
metadata:
|
||||
name: ${BACKUP_SECRET}
|
||||
namespace: ${LONGHORN_NS}
|
||||
stringData:
|
||||
AWS_ACCESS_KEY_ID: ${opts.backup.accessKey}
|
||||
AWS_SECRET_ACCESS_KEY: ${opts.backup.secretKey}
|
||||
AWS_ENDPOINTS: ${opts.backup.endpoint}
|
||||
EOF`);
|
||||
if (secretApplied.exitCode !== 0) {
|
||||
throw new Error(
|
||||
`[${server.name}] could not create the ${BACKUP_SECRET} secret ` +
|
||||
`(exit ${secretApplied.exitCode}). Command withheld: it contains credentials. ` +
|
||||
`stderr: ${secretApplied.stderr.trim().slice(0, 200)}`);
|
||||
}
|
||||
}
|
||||
|
||||
log(`[longhorn] installing chart ${LONGHORN_VERSION} via k3s helm controller`);
|
||||
nodeExec(h, server, `cat > /var/lib/rancher/k3s/server/manifests/longhorn.yaml <<'EOF'
|
||||
apiVersion: helm.cattle.io/v1
|
||||
kind: HelmChart
|
||||
metadata:
|
||||
name: longhorn
|
||||
namespace: kube-system
|
||||
spec:
|
||||
chart: longhorn
|
||||
repo: https://charts.longhorn.io
|
||||
version: ${LONGHORN_VERSION}
|
||||
targetNamespace: ${LONGHORN_NS}
|
||||
valuesContent: |-
|
||||
defaultSettings:
|
||||
defaultDataPath: /var/lib/longhorn
|
||||
defaultReplicaCount: ${replicas}
|
||||
createDefaultDiskLabeledNodes: true
|
||||
defaultDataLocality: best-effort
|
||||
replicaSoftAntiAffinity: false
|
||||
replicaReplenishmentWaitInterval: 600
|
||||
orphanAutoDeletion: true
|
||||
persistence:
|
||||
defaultClass: true
|
||||
defaultClassReplicaCount: ${replicas}
|
||||
defaultFsType: ext4
|
||||
longhornUI:
|
||||
replicas: 1
|
||||
EOF`);
|
||||
}
|
||||
|
||||
/**
|
||||
* Every expected node has a Longhorn disk that is actually schedulable.
|
||||
*
|
||||
* Takes the VMs, not names, and resolves each one's Kubernetes name itself.
|
||||
* Longhorn keys its node records by the FQDN the kubelet registered
|
||||
* (`drill-s1.pxe-test.local`), never the libvirt domain name (`drill-s1`), so
|
||||
* a caller passing VM names matches nothing and waits out the full timeout
|
||||
* while the cluster is in fact perfectly healthy.
|
||||
*/
|
||||
export async function waitForLonghornReady(
|
||||
h: Harness,
|
||||
server: ProvisionedVm,
|
||||
vms: ProvisionedVm[],
|
||||
timeoutMs = 20 * 60_000,
|
||||
): Promise<void> {
|
||||
const expectedNodes = vms.map((vm) => k8sNodeName(h, vm));
|
||||
const start = Date.now();
|
||||
let last = "";
|
||||
while (Date.now() - start < timeoutMs) {
|
||||
const r = tryNodeExec(h, server,
|
||||
`k3s kubectl -n ${LONGHORN_NS} get nodes.longhorn.io -o json`, 60_000);
|
||||
if (r.exitCode === 0) {
|
||||
try {
|
||||
const parsed = JSON.parse(r.stdout) as { items: {
|
||||
metadata: { name: string };
|
||||
status?: { diskStatus?: Record<string, { conditions?: { type: string; status: string }[] }> };
|
||||
}[] };
|
||||
const ready = parsed.items.filter((n) => {
|
||||
const disks = Object.values(n.status?.diskStatus ?? {});
|
||||
return disks.length > 0 && disks.some((d) =>
|
||||
(d.conditions ?? []).some((c) => c.type === "Schedulable" && c.status === "True"));
|
||||
}).map((n) => n.metadata.name);
|
||||
|
||||
last = `${ready.length}/${expectedNodes.length} with schedulable disk: ${ready.join(",") || "none"}`;
|
||||
if (expectedNodes.every((n) => ready.includes(n))) {
|
||||
log(`[longhorn] ready after ${Math.round((Date.now() - start) / 1000)}s — ${last}`);
|
||||
return;
|
||||
}
|
||||
} catch { /* API not serving the CRD yet */ }
|
||||
}
|
||||
await sleep(15_000);
|
||||
}
|
||||
|
||||
// Say WHY, per node, instead of just "not ready". A node with no disk has a
|
||||
// specific cause and the record knows it; without this the only way to find
|
||||
// out is to catch the cluster before teardown, which is a race you lose.
|
||||
const detail: string[] = [];
|
||||
try {
|
||||
const raw = kubectlJson<{ items: {
|
||||
metadata: { name: string; annotations?: Record<string, string>; labels?: Record<string, string> };
|
||||
spec?: { disks?: Record<string, unknown>; allowScheduling?: boolean };
|
||||
status?: {
|
||||
conditions?: { type: string; status: string; reason?: string; message?: string }[];
|
||||
diskStatus?: Record<string, { conditions?: { type: string; status: string; reason?: string; message?: string }[] }>;
|
||||
};
|
||||
}[] }>(h, server, `-n ${LONGHORN_NS} get nodes.longhorn.io`);
|
||||
|
||||
for (const n of raw.items) {
|
||||
const disks = Object.keys(n.spec?.disks ?? {});
|
||||
const conds = (n.status?.conditions ?? [])
|
||||
.filter((c) => c.status !== "True")
|
||||
.map((c) => `${c.type}=${c.status}(${c.reason ?? ""}: ${(c.message ?? "").slice(0, 80)})`);
|
||||
const diskConds = Object.entries(n.status?.diskStatus ?? {}).flatMap(([dn, d]) =>
|
||||
(d.conditions ?? []).filter((c) => c.status !== "True")
|
||||
.map((c) => `${dn}:${c.type}=${c.status}(${c.reason ?? ""}: ${(c.message ?? "").slice(0, 80)})`));
|
||||
const ann = n.metadata.annotations?.["node.longhorn.io/default-disks-config"] ?? "<absent>";
|
||||
const lbl = n.metadata.labels?.["node.longhorn.io/create-default-disk"] ?? "<absent>";
|
||||
detail.push(
|
||||
` ${n.metadata.name}: disks=[${disks.join(",")}] label=${lbl} ann=${ann} ` +
|
||||
`${conds.length > 0 ? `nodeConds=${conds.join(";")} ` : ""}` +
|
||||
`${diskConds.length > 0 ? `diskConds=${diskConds.join(";")}` : ""}`);
|
||||
}
|
||||
} catch (e) {
|
||||
detail.push(` (could not read Longhorn nodes: ${e instanceof Error ? e.message : String(e)})`);
|
||||
}
|
||||
|
||||
throw new Error(
|
||||
`Longhorn not ready in ${timeoutMs / 60000} min (${last}).\n` +
|
||||
`A node with no disk usually means the create-default-disk label or the\n` +
|
||||
`default-disks-config annotation is missing, or the path is not mounted.\n` +
|
||||
detail.join("\n"));
|
||||
}
|
||||
|
||||
/** Create a PVC plus a Deployment that mounts it at /data. */
|
||||
export function createDataWorkload(h: Harness, server: ProvisionedVm, opts: {
|
||||
name: string;
|
||||
sizeGi: number;
|
||||
replicas: number;
|
||||
}): void {
|
||||
log(`[longhorn] creating ${opts.name} (${opts.sizeGi}Gi, ${opts.replicas} replicas)`);
|
||||
nodeExec(h, server, `k3s kubectl apply -f - <<'EOF'
|
||||
apiVersion: storage.k8s.io/v1
|
||||
kind: StorageClass
|
||||
metadata:
|
||||
name: lh-${opts.name}
|
||||
provisioner: driver.longhorn.io
|
||||
allowVolumeExpansion: true
|
||||
parameters:
|
||||
numberOfReplicas: "${opts.replicas}"
|
||||
staleReplicaTimeout: "30"
|
||||
fsType: ext4
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: ${opts.name}
|
||||
spec:
|
||||
accessModes: [ReadWriteOnce]
|
||||
storageClassName: lh-${opts.name}
|
||||
resources:
|
||||
requests:
|
||||
storage: ${opts.sizeGi}Gi
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: ${opts.name}
|
||||
spec:
|
||||
replicas: 1
|
||||
strategy:
|
||||
type: Recreate
|
||||
selector:
|
||||
matchLabels: {app: ${opts.name}}
|
||||
template:
|
||||
metadata:
|
||||
labels: {app: ${opts.name}}
|
||||
spec:
|
||||
# Pod Security Admission enforces "restricted" on this namespace (the k3s
|
||||
# hardening group applies the label), so a bare pod spec is REJECTED
|
||||
# outright -- the Deployment reports 0/1 with no pod and no obvious
|
||||
# reason unless you read the ReplicaSet events. Every one of these
|
||||
# fields is required by "restricted"; dropping any brings it back.
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 1000
|
||||
runAsGroup: 1000
|
||||
# fsGroup so the non-root user can actually write to the Longhorn
|
||||
# volume, which is otherwise owned by root.
|
||||
fsGroup: 1000
|
||||
seccompProfile: {type: RuntimeDefault}
|
||||
containers:
|
||||
- name: shell
|
||||
image: busybox:1.36
|
||||
command: ["sh", "-c", "sleep infinity"]
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
capabilities: {drop: ["ALL"]}
|
||||
volumeMounts:
|
||||
- {name: data, mountPath: /data}
|
||||
volumes:
|
||||
- name: data
|
||||
persistentVolumeClaim: {claimName: ${opts.name}}
|
||||
EOF`);
|
||||
}
|
||||
|
||||
/** Wait for the workload's pod to be Running and its volume attached. */
|
||||
export async function waitForWorkload(
|
||||
h: Harness, server: ProvisionedVm, name: string, timeoutMs = 10 * 60_000,
|
||||
): Promise<void> {
|
||||
const start = Date.now();
|
||||
let last = "";
|
||||
while (Date.now() - start < timeoutMs) {
|
||||
const r = tryNodeExec(h, server,
|
||||
`k3s kubectl get pods -l app=${name} -o jsonpath='{.items[*].status.phase}'`);
|
||||
last = r.stdout.trim();
|
||||
if (last === "Running") {
|
||||
// Ready, not merely scheduled: the volume mount happens before Ready.
|
||||
const ready = tryNodeExec(h, server,
|
||||
`k3s kubectl get pods -l app=${name} -o jsonpath='{.items[*].status.containerStatuses[*].ready}'`);
|
||||
if (ready.stdout.trim() === "true") return;
|
||||
}
|
||||
await sleep(5_000);
|
||||
}
|
||||
throw new Error(`workload ${name} not running in ${timeoutMs / 60000} min (last phase: ${last || "none"})`);
|
||||
}
|
||||
|
||||
function podOf(h: Harness, server: ProvisionedVm, name: string): string {
|
||||
const pod = kubectlOn(h, server, `get pods -l app=${name} -o jsonpath='{.items[0].metadata.name}'`).trim();
|
||||
if (!pod) throw new Error(`no pod for workload ${name}`);
|
||||
return pod;
|
||||
}
|
||||
|
||||
/**
|
||||
* Write a random blob, fsync it, and record its digest.
|
||||
*
|
||||
* conv=fsync matters: without it the write can still be in the guest's page
|
||||
* cache, and a checksum taken immediately afterwards proves nothing about what
|
||||
* reached the replicas.
|
||||
*/
|
||||
export async function writeBlob(h: Harness, server: ProvisionedVm, opts: {
|
||||
name: string; mib: number;
|
||||
}): Promise<string> {
|
||||
const pod = podOf(h, server, opts.name);
|
||||
log(`[longhorn] writing ${opts.mib} MiB to ${opts.name}`);
|
||||
nodeExec(h, server,
|
||||
`k3s kubectl exec ${pod} -- sh -c ` +
|
||||
`"dd if=/dev/urandom of=/data/blob bs=1M count=${opts.mib} conv=fsync 2>/dev/null && ` +
|
||||
`cd /data && sha256sum blob > blob.sha256 && sync"`,
|
||||
15 * 60_000);
|
||||
|
||||
const digest = kubectlOn(h, server, `exec ${pod} -- cat /data/blob.sha256`).trim().split(/\s+/)[0] ?? "";
|
||||
if (!/^[0-9a-f]{64}$/.test(digest)) throw new Error(`bad digest for ${opts.name}: ${digest}`);
|
||||
log(`[longhorn] ${opts.name} digest ${digest.slice(0, 16)}...`);
|
||||
return digest;
|
||||
}
|
||||
|
||||
/**
|
||||
* Restart the pod, then verify the blob.
|
||||
*
|
||||
* The restart is the point. It detaches and reattaches the volume, so a pass
|
||||
* means the bytes came back from Longhorn rather than from wherever the writer
|
||||
* left them.
|
||||
*/
|
||||
export async function verifyBlob(h: Harness, server: ProvisionedVm, opts: {
|
||||
name: string; expectDigest: string;
|
||||
}): Promise<void> {
|
||||
const before = podOf(h, server, opts.name);
|
||||
kubectlOn(h, server, `delete pod ${before} --wait=true --timeout=5m`, 6 * 60_000);
|
||||
await waitForWorkload(h, server, opts.name);
|
||||
const after = podOf(h, server, opts.name);
|
||||
if (after === before) throw new Error(`${opts.name}: pod was not actually replaced`);
|
||||
|
||||
// sha256sum -c proves the file matches what was recorded IN the volume...
|
||||
const check = tryNodeExec(h, server,
|
||||
`k3s kubectl exec ${after} -- sh -c "cd /data && sha256sum -c blob.sha256"`, 15 * 60_000);
|
||||
if (check.exitCode !== 0) {
|
||||
throw new Error(`${opts.name}: DATA CORRUPTION — sha256sum -c failed: ${check.stdout}${check.stderr}`);
|
||||
}
|
||||
|
||||
// ...and this proves the recorded digest is still the one we started with,
|
||||
// which -c alone would not catch if both file and checksum were replaced.
|
||||
const actual = kubectlOn(h, server,
|
||||
`exec ${after} -- sh -c "cd /data && sha256sum blob"`, 15 * 60_000).trim().split(/\s+/)[0];
|
||||
if (actual !== opts.expectDigest) {
|
||||
throw new Error(`${opts.name}: DIGEST CHANGED — expected ${opts.expectDigest}, got ${actual}`);
|
||||
}
|
||||
log(`[longhorn] ${opts.name} verified intact across a detach/attach (pod ${before} -> ${after})`);
|
||||
}
|
||||
|
||||
export interface VolumeState {
|
||||
pv: string;
|
||||
state: string;
|
||||
robustness: string;
|
||||
wanted: number;
|
||||
replicaNodes: string[];
|
||||
healthyReplicas: number;
|
||||
}
|
||||
|
||||
/** Longhorn's view of the volume backing a PVC. */
|
||||
export function volumeState(h: Harness, server: ProvisionedVm, pvcName: string): VolumeState {
|
||||
const pv = kubectlOn(h, server, `get pvc ${pvcName} -o jsonpath='{.spec.volumeName}'`).trim();
|
||||
if (!pv) throw new Error(`PVC ${pvcName} has no bound volume`);
|
||||
|
||||
const vol = kubectlJson<{
|
||||
spec: { numberOfReplicas?: number };
|
||||
status?: { state?: string; robustness?: string };
|
||||
}>(h, server, `-n ${LONGHORN_NS} get volumes.longhorn.io ${pv}`);
|
||||
|
||||
const reps = kubectlJson<{ items: {
|
||||
spec: { nodeID?: string; volumeName?: string };
|
||||
status?: { currentState?: string; started?: boolean };
|
||||
}[] }>(h, server, `-n ${LONGHORN_NS} get replicas.longhorn.io`);
|
||||
|
||||
const mine = reps.items.filter((r) => r.spec.volumeName === pv);
|
||||
return {
|
||||
pv,
|
||||
state: vol.status?.state ?? "unknown",
|
||||
robustness: vol.status?.robustness ?? "unknown",
|
||||
wanted: vol.spec.numberOfReplicas ?? 0,
|
||||
replicaNodes: mine.map((r) => r.spec.nodeID ?? "").filter(Boolean).sort(),
|
||||
healthyReplicas: mine.filter((r) => r.status?.currentState === "running").length,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Wait for a volume to be healthy on the expected number of distinct nodes.
|
||||
*
|
||||
* Logs rebuild progress while waiting, so a slow rebuild reads as progress
|
||||
* rather than as an opaque hour of silence.
|
||||
*/
|
||||
export async function waitForVolumeHealthy(h: Harness, server: ProvisionedVm, opts: {
|
||||
pvc: string; expectReplicas: number; timeoutMs?: number;
|
||||
}): Promise<VolumeState> {
|
||||
const timeoutMs = opts.timeoutMs ?? 60 * 60_000;
|
||||
const start = Date.now();
|
||||
let last = "";
|
||||
while (Date.now() - start < timeoutMs) {
|
||||
let st: VolumeState | null = null;
|
||||
try { st = volumeState(h, server, opts.pvc); } catch { /* transient */ }
|
||||
|
||||
if (st) {
|
||||
const distinct = new Set(st.replicaNodes).size;
|
||||
last = `${st.state}/${st.robustness} on ${distinct} node(s): ${st.replicaNodes.join(",")}`;
|
||||
if (st.robustness === "healthy" && distinct >= opts.expectReplicas) {
|
||||
log(`[longhorn] ${opts.pvc} healthy on ${distinct} nodes after ${Math.round((Date.now() - start) / 1000)}s`);
|
||||
return st;
|
||||
}
|
||||
const prog = tryNodeExec(h, server,
|
||||
`k3s kubectl -n ${LONGHORN_NS} get engines.longhorn.io -o jsonpath=` +
|
||||
`'{range .items[*]}{.spec.volumeName}{" "}{.status.rebuildStatus}{"\\n"}{end}'`);
|
||||
const line = prog.stdout.split("\n").find((l) => l.includes(st!.pv) && l.trim() !== st!.pv);
|
||||
log(`[longhorn] ${opts.pvc}: ${last}${line ? ` | rebuild ${line.replace(st.pv, "").trim()}` : ""}`);
|
||||
}
|
||||
await sleep(20_000);
|
||||
}
|
||||
throw new Error(`${opts.pvc} not healthy on ${opts.expectReplicas} nodes in ${timeoutMs / 60000} min (last: ${last})`);
|
||||
}
|
||||
|
||||
/**
|
||||
* Wait until Longhorn reports the backup target usable.
|
||||
*
|
||||
* `available: true` is the appliance saying it could reach the endpoint AND
|
||||
* authenticate. Backing up without checking this first produces a failure that
|
||||
* looks like a volume problem.
|
||||
*/
|
||||
export async function setBackupTarget(
|
||||
h: Harness, server: ProvisionedVm, backup: BackupTargetOpts, timeoutMs = 5 * 60_000,
|
||||
): Promise<void> {
|
||||
// PATCH the CR; do NOT set defaultSettings.backupTarget.
|
||||
//
|
||||
// In Longhorn 1.8 the backup target moved out of settings and onto
|
||||
// `backuptargets.longhorn.io/default`. The helm value is SILENTLY IGNORED --
|
||||
// no `backup-target` Setting object is created at all, the default
|
||||
// BackupTarget keeps an empty URL, and everything reports success. Observed
|
||||
// on a fresh 1.8.1 install here and confirmed against the live fleet.
|
||||
const deadline = Date.now() + timeoutMs;
|
||||
while (Date.now() < deadline) {
|
||||
const exists = tryNodeExec(h, server,
|
||||
`k3s kubectl -n ${LONGHORN_NS} get backuptargets.longhorn.io default -o name`);
|
||||
if (exists.exitCode === 0 && exists.stdout.trim() !== "") break;
|
||||
await sleep(5_000);
|
||||
}
|
||||
|
||||
const patch = JSON.stringify({
|
||||
spec: { backupTargetURL: backup.target, credentialSecret: BACKUP_SECRET, pollInterval: "5m0s" },
|
||||
});
|
||||
nodeExec(h, server,
|
||||
`k3s kubectl -n ${LONGHORN_NS} patch backuptargets.longhorn.io default --type=merge -p '${patch}'`);
|
||||
log(`[longhorn] backup target set to ${backup.target}`);
|
||||
}
|
||||
|
||||
export async function waitForBackupTarget(
|
||||
h: Harness, server: ProvisionedVm, timeoutMs = 5 * 60_000,
|
||||
): Promise<void> {
|
||||
const start = Date.now();
|
||||
let last = "";
|
||||
while (Date.now() - start < timeoutMs) {
|
||||
const r = tryNodeExec(h, server,
|
||||
`k3s kubectl -n ${LONGHORN_NS} get backuptargets.longhorn.io default ` +
|
||||
`-o jsonpath='{.spec.backupTargetURL}|{.status.available}|{.status.conditions[*].message}'`);
|
||||
last = r.stdout.trim();
|
||||
if (r.exitCode === 0 && last.includes("|true|")) {
|
||||
log(`[longhorn] backup target available after ${Math.round((Date.now() - start) / 1000)}s`);
|
||||
return;
|
||||
}
|
||||
await sleep(10_000);
|
||||
}
|
||||
throw new Error(
|
||||
`Backup target not available in ${timeoutMs / 60000} min (url|available|conditions = ${last}). ` +
|
||||
`Longhorn could not reach or authenticate to the endpoint.`);
|
||||
}
|
||||
|
||||
/**
|
||||
* Snapshot a volume and back it up, returning the backup's name.
|
||||
*
|
||||
* Longhorn's own CRs, not the CSI VolumeSnapshot API -- this cluster has no
|
||||
* snapshot.storage.k8s.io resources at all, so VolumeSnapshot/VolumeSnapshotClass
|
||||
* simply do not exist here. Any backup tooling has to use these.
|
||||
*/
|
||||
export async function backupVolume(
|
||||
h: Harness, server: ProvisionedVm, pvc: string, timeoutMs = 20 * 60_000,
|
||||
): Promise<{ backup: string; volume: string }> {
|
||||
const pv = volumeState(h, server, pvc).pv;
|
||||
const snapName = `drill-snap-${Date.now().toString(36)}`;
|
||||
|
||||
log(`[longhorn] snapshotting ${pvc} (${pv})`);
|
||||
nodeExec(h, server, `k3s kubectl -n ${LONGHORN_NS} apply -f - <<'EOF'
|
||||
apiVersion: longhorn.io/v1beta2
|
||||
kind: Snapshot
|
||||
metadata:
|
||||
name: ${snapName}
|
||||
namespace: ${LONGHORN_NS}
|
||||
spec:
|
||||
volume: ${pv}
|
||||
createSnapshot: true
|
||||
EOF`);
|
||||
|
||||
// The snapshot has to exist before a backup can reference it.
|
||||
const snapDeadline = Date.now() + 5 * 60_000;
|
||||
while (Date.now() < snapDeadline) {
|
||||
const r = tryNodeExec(h, server,
|
||||
`k3s kubectl -n ${LONGHORN_NS} get snapshots.longhorn.io ${snapName} -o jsonpath='{.status.readyToUse}'`);
|
||||
if (r.stdout.trim() === "true") break;
|
||||
await sleep(5_000);
|
||||
}
|
||||
|
||||
const backupName = `drill-bak-${Date.now().toString(36)}`;
|
||||
log(`[longhorn] backing up ${pvc} to the backup target`);
|
||||
nodeExec(h, server, `k3s kubectl -n ${LONGHORN_NS} apply -f - <<'EOF'
|
||||
apiVersion: longhorn.io/v1beta2
|
||||
kind: Backup
|
||||
metadata:
|
||||
name: ${backupName}
|
||||
namespace: ${LONGHORN_NS}
|
||||
labels:
|
||||
backup-volume: ${pv}
|
||||
spec:
|
||||
snapshotName: ${snapName}
|
||||
EOF`);
|
||||
|
||||
const start = Date.now();
|
||||
let last = "";
|
||||
while (Date.now() - start < timeoutMs) {
|
||||
const r = tryNodeExec(h, server,
|
||||
`k3s kubectl -n ${LONGHORN_NS} get backups.longhorn.io ${backupName} ` +
|
||||
`-o jsonpath='{.status.state}|{.status.progress}|{.status.error}'`);
|
||||
last = r.stdout.trim();
|
||||
if (last.startsWith("Completed")) {
|
||||
log(`[longhorn] backup ${backupName} completed after ${Math.round((Date.now() - start) / 1000)}s`);
|
||||
return { backup: backupName, volume: pv };
|
||||
}
|
||||
if (last.includes("Error")) {
|
||||
throw new Error(`Backup ${backupName} failed: ${last}`);
|
||||
}
|
||||
await sleep(10_000);
|
||||
}
|
||||
throw new Error(`Backup ${backupName} did not complete in ${timeoutMs / 60000} min (state|progress|error = ${last})`);
|
||||
}
|
||||
|
||||
/**
|
||||
* Restore a backup into a brand-new PVC and workload.
|
||||
*
|
||||
* Uses the `fromBackup` StorageClass parameter, which is the supported route
|
||||
* when the CSI VolumeSnapshot API is unavailable. The restored volume is a
|
||||
* separate volume with its own replicas -- nothing about the original is
|
||||
* touched, so a failed restore cannot damage the source.
|
||||
*/
|
||||
export async function restoreFromBackup(h: Harness, server: ProvisionedVm, opts: {
|
||||
backup: string;
|
||||
name: string;
|
||||
sizeGi: number;
|
||||
}): Promise<void> {
|
||||
const url = nodeExec(h, server,
|
||||
`k3s kubectl -n ${LONGHORN_NS} get backups.longhorn.io ${opts.backup} -o jsonpath='{.status.url}'`).trim();
|
||||
if (url === "") throw new Error(`Backup ${opts.backup} has no status.url; cannot restore from it`);
|
||||
log(`[longhorn] restoring ${opts.backup} into ${opts.name}`);
|
||||
|
||||
nodeExec(h, server, `k3s kubectl apply -f - <<'EOF'
|
||||
apiVersion: storage.k8s.io/v1
|
||||
kind: StorageClass
|
||||
metadata:
|
||||
name: lh-${opts.name}
|
||||
provisioner: driver.longhorn.io
|
||||
allowVolumeExpansion: true
|
||||
parameters:
|
||||
numberOfReplicas: "2"
|
||||
staleReplicaTimeout: "30"
|
||||
fsType: ext4
|
||||
fromBackup: "${url}"
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: ${opts.name}
|
||||
spec:
|
||||
accessModes: [ReadWriteOnce]
|
||||
storageClassName: lh-${opts.name}
|
||||
resources:
|
||||
requests:
|
||||
storage: ${opts.sizeGi}Gi
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: ${opts.name}
|
||||
spec:
|
||||
replicas: 1
|
||||
strategy:
|
||||
type: Recreate
|
||||
selector:
|
||||
matchLabels: {app: ${opts.name}}
|
||||
template:
|
||||
metadata:
|
||||
labels: {app: ${opts.name}}
|
||||
spec:
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 1000
|
||||
runAsGroup: 1000
|
||||
fsGroup: 1000
|
||||
seccompProfile: {type: RuntimeDefault}
|
||||
containers:
|
||||
- name: shell
|
||||
image: busybox:1.36
|
||||
command: ["sh", "-c", "sleep infinity"]
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
capabilities: {drop: ["ALL"]}
|
||||
volumeMounts:
|
||||
- {name: data, mountPath: /data}
|
||||
volumes:
|
||||
- name: data
|
||||
persistentVolumeClaim: {claimName: ${opts.name}}
|
||||
EOF`);
|
||||
}
|
||||
|
||||
/**
|
||||
* Read a blob's digest from a workload, without restarting it.
|
||||
*
|
||||
* Used on a RESTORED volume, where the point is to compare bytes against the
|
||||
* original's recorded digest rather than to prove durability again.
|
||||
*/
|
||||
export function readBlobDigest(h: Harness, server: ProvisionedVm, name: string): string {
|
||||
const pod = kubectlOn(h, server, `get pods -l app=${name} -o jsonpath='{.items[0].metadata.name}'`).trim();
|
||||
if (pod === "") throw new Error(`no pod for workload ${name}`);
|
||||
const out = kubectlOn(h, server,
|
||||
`exec ${pod} -- sh -c "cd /data && sha256sum blob"`, 15 * 60_000).trim();
|
||||
return out.split(/\s+/)[0] ?? "";
|
||||
}
|
||||
534
bastion/tests/integration/helpers/provision-harness.ts
Normal file
534
bastion/tests/integration/helpers/provision-harness.ts
Normal file
@@ -0,0 +1,534 @@
|
||||
// Reusable bastion + VM provisioning harness for partition-layout tests.
|
||||
//
|
||||
// pxe-provision.test.ts grew a ~200-line beforeAll that stands up a real
|
||||
// bastion, caches Fedora boot artifacts, and drives one VM through discovery
|
||||
// and install. Testing a partition MATRIX needs that same sequence for several
|
||||
// VMs and several roles, so it lives here instead of being copy-pasted per
|
||||
// topology.
|
||||
|
||||
import { existsSync, mkdirSync, copyFileSync, symlinkSync, writeFileSync } from "node:fs";
|
||||
import { execSync } from "node:child_process";
|
||||
import { join } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
import type { BastionConfig } from "../../../src/shared/src/types/config.js";
|
||||
import type { Role } from "../../../src/shared/src/types/state.js";
|
||||
import { log, waitForSsh } from "./libvirt.js";
|
||||
import { ensurePxeNetwork, deleteNftablesRejectRules, PXE_NETWORK_NAME, PXE_BRIDGE, PXE_GATEWAY, PXE_SUBNET } from "./pxe-network.js";
|
||||
import { createPxeVm, destroyPxeVm, getVmMac, rebootPxeVm } from "./pxe-vm.js";
|
||||
import { sshExec } from "./ssh.js";
|
||||
|
||||
export const DISCOVERY_TIMEOUT_MS = 6 * 60_000;
|
||||
export const INSTALL_TIMEOUT_MS = 60 * 60_000;
|
||||
export const SSH_TIMEOUT_MS = 12 * 60_000;
|
||||
|
||||
export const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));
|
||||
|
||||
export interface Harness {
|
||||
config: BastionConfig;
|
||||
close: () => Promise<void>;
|
||||
httpPort: number;
|
||||
sshUser: string;
|
||||
sshKeyPath: string;
|
||||
}
|
||||
|
||||
async function pollUntil<T>(url: string, check: (d: T) => boolean, timeoutMs: number, intervalMs = 5000): Promise<T> {
|
||||
const start = Date.now();
|
||||
let lastErr = "";
|
||||
while (Date.now() - start < timeoutMs) {
|
||||
try {
|
||||
const res = await fetch(url);
|
||||
if (res.ok) {
|
||||
const data = (await res.json()) as T;
|
||||
if (check(data)) return data;
|
||||
}
|
||||
} catch (e) { lastErr = e instanceof Error ? e.message : String(e); }
|
||||
await sleep(intervalMs);
|
||||
}
|
||||
throw new Error(`Timeout after ${timeoutMs}ms polling ${url}${lastErr ? ` (last error: ${lastErr})` : ""}`);
|
||||
}
|
||||
|
||||
/**
|
||||
* Stand up a real bastion on the isolated PXE network.
|
||||
*
|
||||
* FEDORA_MIRROR is honoured (config.ts now reads it) -- point it at a local
|
||||
* mirror before running a multi-VM matrix, or every install pulls the full
|
||||
* package set from download.fedoraproject.org.
|
||||
*/
|
||||
export async function startTestBastion(opts: {
|
||||
httpPort: number;
|
||||
sshUser: string;
|
||||
sshPubKey: string;
|
||||
sshKeyPath: string;
|
||||
label: string;
|
||||
/**
|
||||
* Leave the post-provision k3s auto-install ON (the default) or suppress it.
|
||||
*
|
||||
* Any test that forms a cluster ITSELF must pass false. The auto-install runs
|
||||
* a bare `INSTALL_K3S_EXEC=server|agent` with no server URL and no token, so
|
||||
* three combo nodes become three separate single-node clusters -- and it
|
||||
* stops k3s and deletes CNI state minutes after an install lands, landing on
|
||||
* top of whatever the test is doing by then.
|
||||
*/
|
||||
autoK3s?: boolean;
|
||||
}): Promise<Harness> {
|
||||
ensurePxeNetwork();
|
||||
|
||||
const testDir = join(tmpdir(), `lab-${opts.label}-${Date.now()}`);
|
||||
for (const sub of ["tftp", "http", "logs"]) mkdirSync(join(testDir, sub), { recursive: true });
|
||||
|
||||
const { createApp } = await import("../../../src/bastion/src/server.js");
|
||||
const { loadConfig } = await import("../../../src/bastion/src/config.js");
|
||||
const { generateDnsmasqConf, startDnsmasq, stopDnsmasq } = await import("../../../src/bastion/src/services/dnsmasq.js");
|
||||
const { generateDiscoverKickstart } = await import("../../../src/bastion/src/services/kickstart-generator.js");
|
||||
const { renderBootIpxe } = await import("../../../src/bastion/src/templates/boot.ipxe.js");
|
||||
|
||||
const config = loadConfig({
|
||||
bastionDir: testDir,
|
||||
httpPort: opts.httpPort,
|
||||
iface: "virbr-pxe",
|
||||
serverIp: PXE_GATEWAY,
|
||||
network: `${PXE_SUBNET}.0`,
|
||||
gateway: PXE_GATEWAY,
|
||||
dhcpMode: "full",
|
||||
dhcpRangeStart: `${PXE_SUBNET}.100`,
|
||||
dhcpRangeEnd: `${PXE_SUBNET}.200`,
|
||||
domain: "pxe-test.local",
|
||||
sshKeys: [opts.sshPubKey],
|
||||
adminUser: opts.sshUser,
|
||||
...(opts.autoK3s === undefined ? {} : { autoK3s: opts.autoK3s }),
|
||||
});
|
||||
|
||||
const ipxeSrc = "/usr/share/ipxe/ipxe-snponly-x86_64.efi";
|
||||
if (!existsSync(ipxeSrc)) throw new Error(`iPXE not found: ${ipxeSrc}. Install: sudo dnf install ipxe-bootimgs-x86`);
|
||||
const ipxeDest = join(config.tftpDir, "ipxe.efi");
|
||||
copyFileSync(ipxeSrc, ipxeDest);
|
||||
|
||||
const cacheDir = "/var/lib/libvirt/images/lab-pxe-cache";
|
||||
execSync(`mkdir -p "${cacheDir}"`, { stdio: "pipe" });
|
||||
const kernel = join(cacheDir, `vmlinuz-${config.fedoraVersion}`);
|
||||
const initrd = join(cacheDir, `initrd-${config.fedoraVersion}.img`);
|
||||
if (!existsSync(kernel)) {
|
||||
log(`Downloading Fedora ${config.fedoraVersion} kernel from ${config.fedoraMirror}...`);
|
||||
execSync(`curl -sS -L -f -o "${kernel}" "${config.fedoraMirror}/images/pxeboot/vmlinuz"`, { stdio: "inherit", timeout: 300_000 });
|
||||
}
|
||||
if (!existsSync(initrd)) {
|
||||
log(`Downloading Fedora ${config.fedoraVersion} initrd...`);
|
||||
execSync(`curl -sS -L -f -o "${initrd}" "${config.fedoraMirror}/images/pxeboot/initrd.img"`, { stdio: "inherit", timeout: 300_000 });
|
||||
}
|
||||
copyFileSync(kernel, join(config.httpDir, "vmlinuz"));
|
||||
copyFileSync(initrd, join(config.httpDir, "initrd.img"));
|
||||
try { symlinkSync(ipxeDest, join(config.httpDir, "ipxe.efi")); } catch { /* exists */ }
|
||||
|
||||
writeFileSync(join(config.httpDir, "discover.ks"), generateDiscoverKickstart(config));
|
||||
writeFileSync(join(config.httpDir, "boot.ipxe"), renderBootIpxe({ serverIp: config.serverIp, httpPort: config.httpPort }));
|
||||
generateDnsmasqConf(config);
|
||||
|
||||
// Kill dnsmasq processes left behind by EARLIER TEST RUNS.
|
||||
//
|
||||
// startTestBastion's close() shuts down the HTTP app but never stopped
|
||||
// dnsmasq, so every run leaked one. A leaked dnsmasq keeps :67 on the bridge
|
||||
// and keeps answering DHCP -- with the PREVIOUS run's config, pointing PXE
|
||||
// clients at a bastion port that is no longer listening. The next run then
|
||||
// sees perfect DHCP (offer, request, ack, even the right bootfile name) and
|
||||
// no machine ever reaches it. That is not a failure mode anyone guesses from
|
||||
// "discovery timed out".
|
||||
killOrphanedDnsmasq();
|
||||
|
||||
const { app, syslog } = createApp(config);
|
||||
await app.listen({ port: config.httpPort, host: "0.0.0.0" });
|
||||
syslog.start();
|
||||
// NOT awaited, deliberately.
|
||||
//
|
||||
// startDnsmasq returns the execa child handle, and because the function is
|
||||
// `async` the promise it returns ADOPTS that thenable -- which settles only
|
||||
// when dnsmasq EXITS. With --no-daemon that is never, so `await` here hangs
|
||||
// the harness silently and forever, with the log frozen right after
|
||||
// "TFTP root is ...". (Learned by doing exactly that.) The rejection still
|
||||
// has to be handled or an early bind failure becomes an unhandled rejection.
|
||||
startDnsmasq(config).catch((err) => log(`dnsmasq: ${err instanceof Error ? err.message : String(err)}`));
|
||||
await sleep(1000);
|
||||
|
||||
// Prove DHCP is actually listening before handing the harness back.
|
||||
//
|
||||
// startDnsmasq's rejection is swallowed into a log line, so a dnsmasq that
|
||||
// failed to bind looks exactly like one that started. The symptom then
|
||||
// arrives six minutes later as four VMs timing out in discovery, which
|
||||
// reads as a PXE or kickstart problem rather than "there was never any
|
||||
// DHCP". Fail here, immediately, with the real reason.
|
||||
await assertDhcpListening(join(testDir, "dnsmasq.conf"));
|
||||
log(`Bastion up on :${opts.httpPort} (mirror: ${config.fedoraMirror}, autoK3s=${config.autoK3s})`);
|
||||
|
||||
return {
|
||||
config,
|
||||
httpPort: opts.httpPort,
|
||||
sshUser: opts.sshUser,
|
||||
sshKeyPath: opts.sshKeyPath,
|
||||
close: async () => {
|
||||
// Stop OUR dnsmasq. close() never used to, which is how a leaked one
|
||||
// ended up owning :67 and serving the PREVIOUS run's TFTP root and
|
||||
// bastion port to the NEXT run's VMs -- they booted toward a dead port
|
||||
// and simply never appeared, while DHCP looked perfect.
|
||||
try { stopDnsmasq(); } catch { /* already gone */ }
|
||||
await app.close();
|
||||
// Belt and braces: if that did not stop it, sweep by config path.
|
||||
killOrphanedDnsmasq();
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Kill any dnsmasq started by a previous (or current) lab test run.
|
||||
*
|
||||
* Matched on the config path, which is always /tmp/lab-<label>-<ts>/dnsmasq.conf
|
||||
* -- so this never touches libvirt's own dnsmasq for the default network, or
|
||||
* anything else on the host.
|
||||
*/
|
||||
function killOrphanedDnsmasq(): void {
|
||||
try {
|
||||
// Match on the process NAME first (pgrep -x), then inspect each one's
|
||||
// arguments. A `pgrep -f 'dnsmasq.*/tmp/lab-'` matches the shell running
|
||||
// that very pattern, so the naive version can kill its own shell -- and
|
||||
// libvirt's dnsmasq for the default network must never be touched either.
|
||||
const pids = execSync("pgrep -x dnsmasq 2>/dev/null || true", { encoding: "utf-8" })
|
||||
.trim().split("\n").filter((x) => /^\d+$/.test(x.trim()));
|
||||
let killed = 0;
|
||||
for (const pid of pids) {
|
||||
let args = "";
|
||||
try {
|
||||
args = execSync(`tr '\\0' ' ' < /proc/${pid.trim()}/cmdline 2>/dev/null || true`, { encoding: "utf-8" });
|
||||
} catch { continue; }
|
||||
if (!args.includes("--conf-file=/tmp/lab-")) continue;
|
||||
log(`Killing orphaned test dnsmasq (pid ${pid.trim()}) — it would answer DHCP with stale config`);
|
||||
try { execSync(`sudo kill -9 ${pid.trim()} 2>/dev/null || kill -9 ${pid.trim()} 2>/dev/null || true`); } catch { /* gone */ }
|
||||
killed++;
|
||||
}
|
||||
// Give the socket a moment to be released before we bind it.
|
||||
if (killed > 0) execSync("sleep 1");
|
||||
} catch { /* nothing to kill */ }
|
||||
}
|
||||
|
||||
/**
|
||||
* Is something serving DHCP on the PXE bridge?
|
||||
*
|
||||
* `ss -lunp` shows UDP listeners; dnsmasq binds port 67. Checked on the bridge
|
||||
* address specifically, because a dnsmasq bound to the wrong interface is the
|
||||
* failure being guarded against.
|
||||
*/
|
||||
async function assertDhcpListening(confPath: string, timeoutMs = 20_000): Promise<void> {
|
||||
const start = Date.now();
|
||||
let last = "";
|
||||
while (Date.now() - start < timeoutMs) {
|
||||
try {
|
||||
// Check for OUR dnsmasq by its config path, not merely for something on
|
||||
// :67. A leaked dnsmasq from a previous run also holds :67 and also
|
||||
// answers DHCP -- with the wrong bastion port -- so a port check alone
|
||||
// reports success for precisely the situation it is meant to catch.
|
||||
const pids = execSync("pgrep -x dnsmasq 2>/dev/null || true", { encoding: "utf-8" })
|
||||
.trim().split("\n").filter((x) => /^\d+$/.test(x.trim()));
|
||||
last = pids.filter((pid) => {
|
||||
try {
|
||||
return execSync(`tr '\\0' ' ' < /proc/${pid.trim()}/cmdline 2>/dev/null || true`, { encoding: "utf-8" })
|
||||
.includes(confPath);
|
||||
} catch { return false; }
|
||||
}).join(",");
|
||||
if (last !== "") {
|
||||
const ports = execSync("ss -lun 2>/dev/null || true", { encoding: "utf-8" });
|
||||
if (/:67\b/.test(ports)) {
|
||||
log("dnsmasq (ours) is serving DHCP on :67");
|
||||
return;
|
||||
}
|
||||
}
|
||||
} catch { /* retry */ }
|
||||
await sleep(1000);
|
||||
}
|
||||
throw new Error(
|
||||
`dnsmasq for ${confPath} is not serving DHCP after ${timeoutMs / 1000}s. ` +
|
||||
`It most likely failed to bind to ${PXE_BRIDGE} -- check the bastion log for ` +
|
||||
`"error binding DHCP socket", and check for a leaked dnsmasq from an earlier run ` +
|
||||
`(pgrep -af 'dnsmasq.*--conf-file=/tmp/lab-'). Without this every VM sits in ` +
|
||||
`discovery until it times out, six minutes later, looking like a PXE problem.`,
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Wait for the installer to actually start, rebooting into PXE if it does not.
|
||||
*
|
||||
* "Started" means the bastion has seen anything at all for this MAC beyond the
|
||||
* queued state -- a dispatch, a progress line, a status. A VM that never gets
|
||||
* that far is stuck before Anaconda, and only a reboot will shift it.
|
||||
*/
|
||||
async function waitForInstallStart(
|
||||
base: string,
|
||||
mac: string,
|
||||
name: string,
|
||||
attemptMs = 6 * 60_000,
|
||||
attempts = 3,
|
||||
): Promise<void> {
|
||||
for (let attempt = 1; attempt <= attempts; attempt++) {
|
||||
try {
|
||||
// NOT `status === "installing"`. A machine that has merely been QUEUED
|
||||
// already reports that -- the bastion sets it the moment the install is
|
||||
// dispatched, before anything has booted. Using it would match instantly
|
||||
// and this retry would never fire.
|
||||
//
|
||||
// The honest signals that Anaconda is alive are `progress` leaving
|
||||
// "queued", or log lines arriving over syslog.
|
||||
await pollUntil<{ status?: string; progress?: string; log_total?: number }>(
|
||||
`${base}/api/logs/${encodeURIComponent(mac)}`,
|
||||
(d) => d.status === "installed"
|
||||
|| (d.progress !== undefined && d.progress !== "queued")
|
||||
|| (d.log_total ?? 0) > 0,
|
||||
attemptMs, 5_000);
|
||||
if (attempt > 1) log(`[${name}] PXE boot took on attempt ${attempt}`);
|
||||
return;
|
||||
} catch {
|
||||
if (attempt === attempts) {
|
||||
throw new Error(
|
||||
`[${name}] installer never started after ${attempts} PXE boots ` +
|
||||
`(${attemptMs / 60000} min each). The VM is stuck before Anaconda — ` +
|
||||
`check TFTP and the iPXE chainload.`);
|
||||
}
|
||||
log(`[${name}] no installer activity in ${attemptMs / 60000} min; rebooting into PXE (attempt ${attempt + 1}/${attempts})`);
|
||||
rebootPxeVm(name);
|
||||
deleteNftablesRejectRules();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export interface ProvisionedVm { name: string; mac: string; ip: string }
|
||||
|
||||
/**
|
||||
* Serial ports are a global resource on the host, so hand out a distinct one
|
||||
* per VM. Without this only one VM can run at a time -- qemu refuses to start
|
||||
* the second with "Address already in use".
|
||||
*/
|
||||
let nextSerialPort = 4600;
|
||||
|
||||
/** Create a blank VM, let it discover, queue the install, wait for SSH. */
|
||||
export async function provisionVm(h: Harness, spec: {
|
||||
name: string;
|
||||
role: Role;
|
||||
schemaVersion?: 1 | 2;
|
||||
diskGb: number;
|
||||
memory?: number;
|
||||
vcpus?: number;
|
||||
}): Promise<ProvisionedVm> {
|
||||
const base = `http://${PXE_GATEWAY}:${h.httpPort}`;
|
||||
log(`[${spec.name}] creating VM (${spec.diskGb}GB disk, role=${spec.role}, schema=v${spec.schemaVersion ?? 1})`);
|
||||
createPxeVm({
|
||||
name: spec.name,
|
||||
memory: spec.memory ?? 4096,
|
||||
vcpus: spec.vcpus ?? 4,
|
||||
diskSize: spec.diskGb,
|
||||
network: PXE_NETWORK_NAME,
|
||||
serialPort: nextSerialPort++,
|
||||
});
|
||||
deleteNftablesRejectRules();
|
||||
|
||||
const mac = getVmMac(spec.name);
|
||||
if (!mac) throw new Error(`[${spec.name}] no MAC`);
|
||||
|
||||
log(`[${spec.name}] waiting for discovery (${mac})...`);
|
||||
await pollUntil<{ discovered: Record<string, unknown> }>(
|
||||
`${base}/api/machines`, (d) => mac in (d.discovered ?? {}), DISCOVERY_TIMEOUT_MS);
|
||||
|
||||
log(`[${spec.name}] queueing install...`);
|
||||
const res = await fetch(`${base}/api/install`, {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({
|
||||
mac, hostname: spec.name, disk: "", role: spec.role,
|
||||
...(spec.schemaVersion ? { schemaVersion: spec.schemaVersion } : {}),
|
||||
}),
|
||||
});
|
||||
if (!res.ok) throw new Error(`[${spec.name}] install queue rejected: ${res.status} ${await res.text()}`);
|
||||
|
||||
rebootPxeVm(spec.name);
|
||||
deleteNftablesRejectRules();
|
||||
|
||||
// Make sure the PXE boot actually TOOK before settling in for the install.
|
||||
//
|
||||
// A UEFI VM that fails to chainload just retries DHCP quietly, for ever.
|
||||
// Observed: one VM of four sat in a DHCP loop for an HOUR with no TFTP
|
||||
// fetch, then began installing seconds after the 60-minute install timeout
|
||||
// had already fired -- so the run failed on a machine that was, by then,
|
||||
// working. TFTP is UDP and four VMs contend for it, so this is a question of
|
||||
// when rather than if.
|
||||
//
|
||||
// Rebooting is cheap and a boot that has not started within a few minutes is
|
||||
// not going to start on its own.
|
||||
await waitForInstallStart(base, mac, spec.name);
|
||||
|
||||
log(`[${spec.name}] installing (up to ${INSTALL_TIMEOUT_MS / 60000} min)...`);
|
||||
await pollUntil<{ status?: string }>(
|
||||
`${base}/api/logs/${encodeURIComponent(mac)}`, (d) => d.status === "installed", INSTALL_TIMEOUT_MS, 10_000);
|
||||
|
||||
// The installed record carries the DHCP address the machine ended up with.
|
||||
const machines = await pollUntil<{ installed: Record<string, { ip?: string }> }>(
|
||||
`${base}/api/machines`, (d) => Boolean(d.installed?.[mac]?.ip), 2 * 60_000);
|
||||
const ip = machines.installed[mac]!.ip!;
|
||||
|
||||
try {
|
||||
if (execSync(`sudo virsh domstate ${spec.name}`, { encoding: "utf-8" }).trim() === "shut off") {
|
||||
execSync(`sudo virsh start ${spec.name}`, { stdio: "pipe" });
|
||||
}
|
||||
} catch { /* already running */ }
|
||||
deleteNftablesRejectRules();
|
||||
|
||||
log(`[${spec.name}] waiting for SSH at ${ip}...`);
|
||||
await waitForSsh(ip, h.sshUser, SSH_TIMEOUT_MS, h.sshKeyPath);
|
||||
log(`[${spec.name}] up at ${ip}`);
|
||||
return { name: spec.name, mac, ip };
|
||||
}
|
||||
|
||||
/**
|
||||
* Provision several VMs at once.
|
||||
*
|
||||
* Installs are ~12 minutes each and almost entirely I/O and network wait, so
|
||||
* four in sequence is an hour that four in parallel is twenty minutes. They are
|
||||
* staggered because the expensive, contended part is the start: PXE, DHCP and
|
||||
* the first package fetches all land at once otherwise.
|
||||
*
|
||||
* Rejects only after every VM has settled, so a single failure does not leave
|
||||
* the others half-built and unreferenced -- the caller still gets the names it
|
||||
* needs to tear down.
|
||||
*/
|
||||
export async function provisionVms(
|
||||
h: Harness,
|
||||
specs: Parameters<typeof provisionVm>[1][],
|
||||
opts: { staggerMs?: number } = {},
|
||||
): Promise<ProvisionedVm[]> {
|
||||
const stagger = opts.staggerMs ?? 60_000;
|
||||
const settled = await Promise.allSettled(
|
||||
specs.map(async (spec, i) => {
|
||||
await sleep(i * stagger);
|
||||
return provisionVm(h, spec);
|
||||
}),
|
||||
);
|
||||
|
||||
const failures = settled
|
||||
.map((r, i) => (r.status === "rejected" ? `${specs[i]?.name}: ${String(r.reason)}` : null))
|
||||
.filter((x): x is string => x !== null);
|
||||
if (failures.length > 0) {
|
||||
throw new Error(`provisionVms: ${failures.length}/${specs.length} failed\n ${failures.join("\n ")}`);
|
||||
}
|
||||
return settled.map((r) => (r as PromiseFulfilledResult<ProvisionedVm>).value);
|
||||
}
|
||||
|
||||
/** LV name -> size in MiB, read off the installed machine. */
|
||||
export function readLvs(h: Harness, vm: ProvisionedVm): Record<string, number> {
|
||||
const res = sshExec(vm.ip, h.sshUser,
|
||||
"sudo lvs labvg --noheadings --units m --nosuffix -o lv_name,lv_size --separator=,",
|
||||
{ keyPath: h.sshKeyPath });
|
||||
if (res.exitCode !== 0) throw new Error(`[${vm.name}] lvs failed: ${res.stderr}`);
|
||||
const map: Record<string, number> = {};
|
||||
for (const line of res.stdout.split("\n")) {
|
||||
const [name, size] = line.trim().split(",");
|
||||
if (name && size) map[name] = Math.round(Number(size));
|
||||
}
|
||||
// Log what actually landed on disk. The VMs are destroyed in afterAll, so
|
||||
// without this the only record of the real layout is the pass/fail bit.
|
||||
const summary = Object.entries(map)
|
||||
.sort(([a], [b]) => a.localeCompare(b))
|
||||
.map(([n, m]) => `${n}=${(m / 1024).toFixed(0)}G`)
|
||||
.join(" ");
|
||||
log(`[${vm.name}] on-disk LVs: ${summary}`);
|
||||
return map;
|
||||
}
|
||||
|
||||
/** The VG tag that records which schema this disk was built with. */
|
||||
export function readSchemaTag(h: Harness, vm: ProvisionedVm): string {
|
||||
return sshExec(vm.ip, h.sshUser, "sudo vgs --noheadings -o vg_tags labvg",
|
||||
{ keyPath: h.sshKeyPath }).stdout.trim();
|
||||
}
|
||||
|
||||
export function destroyVms(...names: string[]): void {
|
||||
for (const n of names) { try { destroyPxeVm(n); } catch { /* already gone */ } }
|
||||
}
|
||||
|
||||
/**
|
||||
* Reinstall a machine that is already installed, the way
|
||||
* `labctl provision reprovision` does: queue another install for the same MAC
|
||||
* and reboot it into PXE.
|
||||
*
|
||||
* The schema is deliberately NOT passed. A reinstall must reproduce whatever
|
||||
* layout is already on the disk -- %pre reads the lab-schema-v2 LVM tag rather
|
||||
* than trusting the caller -- and passing one here would hide a regression in
|
||||
* exactly that logic.
|
||||
*/
|
||||
export async function reprovisionVm(
|
||||
h: Harness,
|
||||
vm: ProvisionedVm,
|
||||
role: Role,
|
||||
/**
|
||||
* Only set this to prove it is IGNORED. A reinstall takes its layout from the
|
||||
* disk, not from the request; passing a schema here is how you test that.
|
||||
*/
|
||||
requestSchemaVersion?: 1 | 2,
|
||||
): Promise<ProvisionedVm> {
|
||||
const base = `http://${PXE_GATEWAY}:${h.httpPort}`;
|
||||
log(`[${vm.name}] reprovisioning (role=${role}, schema: whatever is on disk)...`);
|
||||
|
||||
const res = await fetch(`${base}/api/install`, {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({
|
||||
mac: vm.mac, hostname: vm.name, disk: "", role,
|
||||
...(requestSchemaVersion ? { schemaVersion: requestSchemaVersion } : {}),
|
||||
}),
|
||||
});
|
||||
if (!res.ok) throw new Error(`[${vm.name}] reinstall queue rejected: ${res.status} ${await res.text()}`);
|
||||
|
||||
rebootPxeVm(vm.name);
|
||||
deleteNftablesRejectRules();
|
||||
|
||||
log(`[${vm.name}] reinstalling...`);
|
||||
await pollUntil<{ status?: string }>(
|
||||
`${base}/api/logs/${encodeURIComponent(vm.mac)}`,
|
||||
(d) => d.status === "installed", INSTALL_TIMEOUT_MS, 10_000);
|
||||
|
||||
const machines = await pollUntil<{ installed: Record<string, { ip?: string }> }>(
|
||||
`${base}/api/machines`, (d) => Boolean(d.installed?.[vm.mac]?.ip), 2 * 60_000);
|
||||
const ip = machines.installed[vm.mac]!.ip!;
|
||||
|
||||
try {
|
||||
if (execSync(`sudo virsh domstate ${vm.name}`, { encoding: "utf-8" }).trim() === "shut off") {
|
||||
execSync(`sudo virsh start ${vm.name}`, { stdio: "pipe" });
|
||||
}
|
||||
} catch { /* already running */ }
|
||||
deleteNftablesRejectRules();
|
||||
|
||||
await waitForSsh(ip, h.sshUser, SSH_TIMEOUT_MS, h.sshKeyPath);
|
||||
log(`[${vm.name}] back up at ${ip} after reinstall`);
|
||||
return { ...vm, ip };
|
||||
}
|
||||
|
||||
/**
|
||||
* Drop a marker file into each mount point that a reinstall is supposed to
|
||||
* preserve. Ported from test-reprovision.sh, which proved the idea but only
|
||||
* ever ran by hand.
|
||||
*/
|
||||
export function writeMarkers(h: Harness, vm: ProvisionedVm, mounts: string[]): string {
|
||||
const token = `preserve-${Date.now()}-${Math.random().toString(36).slice(2, 8)}`;
|
||||
for (const m of mounts) {
|
||||
const r = sshExec(vm.ip, h.sshUser,
|
||||
`sudo sh -c 'echo ${token} > ${m}/LAB_PRESERVE_TEST.txt && sync'`,
|
||||
{ keyPath: h.sshKeyPath });
|
||||
if (r.exitCode !== 0) throw new Error(`[${vm.name}] could not write marker in ${m}: ${r.stderr}`);
|
||||
}
|
||||
log(`[${vm.name}] wrote marker ${token} into ${mounts.join(", ")}`);
|
||||
return token;
|
||||
}
|
||||
|
||||
/** Which mount points still carry the marker. */
|
||||
export function readMarkers(h: Harness, vm: ProvisionedVm, mounts: string[]): Record<string, string | null> {
|
||||
const out: Record<string, string | null> = {};
|
||||
for (const m of mounts) {
|
||||
const r = sshExec(vm.ip, h.sshUser, `sudo cat ${m}/LAB_PRESERVE_TEST.txt 2>/dev/null || true`,
|
||||
{ keyPath: h.sshKeyPath });
|
||||
const v = r.stdout.trim();
|
||||
out[m] = v === "" ? null : v;
|
||||
}
|
||||
log(`[${vm.name}] markers after reinstall: ${JSON.stringify(out)}`);
|
||||
return out;
|
||||
}
|
||||
@@ -56,6 +56,17 @@ export function ensurePxeNetwork(): void {
|
||||
|
||||
try { unlinkSync(xmlPath); } catch { /* ignore */ }
|
||||
|
||||
// `virsh net-start` returns before the bridge is usable. The bastion's
|
||||
// dnsmasq starts about a second later and binds to this device by name; if
|
||||
// the interface is not up and addressed yet it fails with "error binding
|
||||
// DHCP socket to device virbr-pxe", and because that failure is
|
||||
// asynchronous the harness carries on with NO DHCP AT ALL -- every VM then
|
||||
// sits in discovery until it times out, six minutes later, looking like a
|
||||
// PXE problem rather than a startup-order one.
|
||||
//
|
||||
// Cheap to wait, expensive not to.
|
||||
waitForBridge();
|
||||
|
||||
log(`Network ${PXE_NETWORK_NAME} created and active`);
|
||||
}
|
||||
|
||||
@@ -64,6 +75,32 @@ export function ensurePxeNetwork(): void {
|
||||
deleteNftablesRejectRules();
|
||||
}
|
||||
|
||||
/**
|
||||
* Block until the bridge exists AND carries the gateway address.
|
||||
*
|
||||
* Checking only that the device exists is not enough: it appears before the
|
||||
* address is assigned, and dnsmasq needs the address to bind.
|
||||
*/
|
||||
export function waitForBridge(timeoutMs = 30_000): void {
|
||||
const start = Date.now();
|
||||
let last = "";
|
||||
while (Date.now() - start < timeoutMs) {
|
||||
const r = spawnSync("ip", ["-br", "addr", "show", PXE_BRIDGE], { encoding: "utf-8", stdio: "pipe" });
|
||||
last = (r.stdout ?? "").trim();
|
||||
if (r.status === 0 && last.includes(`${PXE_GATEWAY}/`)) {
|
||||
log(`Bridge ${PXE_BRIDGE} ready with ${PXE_GATEWAY} after ${Date.now() - start}ms`);
|
||||
return;
|
||||
}
|
||||
// Busy-wait briefly; this normally resolves in well under a second.
|
||||
const until = Date.now() + 250;
|
||||
while (Date.now() < until) { /* spin */ }
|
||||
}
|
||||
throw new Error(
|
||||
`Bridge ${PXE_BRIDGE} did not come up with ${PXE_GATEWAY} within ${timeoutMs}ms ` +
|
||||
`(last: ${last || "device absent"}). dnsmasq cannot bind, so PXE would silently not work.`,
|
||||
);
|
||||
}
|
||||
|
||||
/** Delete libvirt's nftables reject rules for our bridge so host→VM traffic works.
|
||||
* Must be called after every VM start/restart — libvirt recreates them. */
|
||||
export function deleteNftablesRejectRules(): void {
|
||||
|
||||
@@ -28,6 +28,13 @@ export interface PxeVmConfig {
|
||||
vcpus: number;
|
||||
diskSize: number; // GB
|
||||
network: string; // libvirt network name
|
||||
/**
|
||||
* Serial console TCP port. Defaults to 4555 for backwards compatibility with
|
||||
* the single-VM tests, but it MUST be unique per concurrently-running VM:
|
||||
* qemu fails with "Failed to find an available port: Address already in use"
|
||||
* and the domain never starts.
|
||||
*/
|
||||
serialPort?: number;
|
||||
arch?: "x86_64" | "aarch64";
|
||||
/**
|
||||
* Extra NICs enumerated BEFORE the PXE NIC, on a network with no route to
|
||||
@@ -86,8 +93,8 @@ export function createPxeVm(config: PxeVmConfig): void {
|
||||
// Graphics for debugging (VNC, connect with virt-viewer if needed)
|
||||
"--graphics=vnc,listen=127.0.0.1",
|
||||
// Serial console via TCP — allows exec without network/SSH
|
||||
// Connect: socat - TCP:127.0.0.1:4555
|
||||
"--serial=tcp,host=127.0.0.1:4555,mode=bind,protocol=telnet",
|
||||
// Connect: socat - TCP:127.0.0.1:<serialPort>
|
||||
`--serial=tcp,host=127.0.0.1:${config.serialPort ?? 4555},mode=bind,protocol=telnet`,
|
||||
];
|
||||
|
||||
if (arch === "aarch64") {
|
||||
@@ -96,7 +103,7 @@ export function createPxeVm(config: PxeVmConfig): void {
|
||||
|
||||
log(`Running: virt-install --name=${config.name} --boot=uefi,network ...`);
|
||||
run(virtInstallArgs.join(" "), { timeout: 30_000 });
|
||||
log(`PXE VM ${config.name} created (serial: telnet 127.0.0.1 4555)`);
|
||||
log(`PXE VM ${config.name} created (serial: telnet 127.0.0.1 ${config.serialPort ?? 4555})`);
|
||||
}
|
||||
|
||||
/** Destroy a PXE VM and clean up its disk. */
|
||||
|
||||
Reference in New Issue
Block a user