v2 partition schema: capability model, fixed sizing, and a proven rolling migration
Some checks failed
Some checks failed
The fleet's disk layout was decided by literal role-name comparisons at 17 sites across 8 files. That is why `labcontroller` got no rancher LV (it matched neither "infra" nor "worker"), why labmaster carried an empty 728 GiB Longhorn LV, and why a 62 GiB container image tipped that node into DiskPressure and evicted bastion, labd, cockroach and openbao. Three generators each hardcoded their own sizes and disagreed: the kickstart said rancher=120G, ubuntu-autoinstall said 20G, asahi-firstboot said 20G under a comment asserting the sizes matched install.ks.ts. They did not. ROLE_REGISTRY already carried `k3s` and `parent` and nothing read them. This makes capabilities authoritative, moves every size into one shared module, and adds schema v2 (fixed 250G rancher / 300G longhorn, never --grow). Existing machines stay v1 forever: schema is detected from the on-disk VG tag, so a v1 box stays v1 even when v2 is requested. Proven on real VMs, not just unit tests: all four topologies installed and had their layouts read back off the machines; markers in /home, /srv, /var/lib/longhorn and /var/lib/rancher survived reinstall; and a 4-node cluster formed, took real data, and had all three servers rolled one at a time through evacuate / wipe / reinstall / rejoin with checksums verified before and after each roll. Drill: 10/10, evidence in migration/drill-evidence/. Production defects the drill found, none of which a unit test could reach: - sshOpts() capped every operation at 30s while drainNode asked kubectl for --timeout=15m, so the transport killed the drain after evicting most pods and returned a transcript that read like success. `labctl app k3s evacuate` could never have drained a real node. - generateAgentConfig put node-role.kubernetes.io/worker in the kubelet's node-label list, which NodeRestriction rejects, so no agent installed through K3sModule could start. systemd reported only "did not take the steps required by its unit configuration". - toOpContext built K3sConfig by enumerating three keys and silently dropped the rest, so the k3s version pin did nothing and --allow-single-replica was a no-op that claimed to override a data-loss guard. - The Longhorn disk annotation was mangled by shell quoting; with createDefaultDiskLabeledNodes the node then gets no disk at all, joins, goes Ready, and silently hosts no replica. - evacuate drained before evicting, which blocks on Longhorn's PDBs and would destroy a single-replica volume's only source. - etcd member ids were parsed from JSON, losing uint64 precision and emitting decimal where `member remove` needs hex -- at the moment a node is stopped and about to be wiped. - get.k3s.io installs latest, two minor versions ahead of the fleet, so a reinstalled node would rejoin on a newer k3s than its peers and roll an unplanned two-version control-plane upgrade. K3sConfig.k3sVersion now pins it. - `labctl app k3s install --role x` mapped anything that was not "worker" to "infra", so the runbook's own `--role k8s-combo` would have installed a different role with different disk expectations and no warning. Also: /var/lib/longhorn/replicas must never be deleted, only its contents, or every replica placed on the node dies on a non-recursive mkdir -- for ever, while the node reports Ready, Schedulable and healthy. etcd is now in the kickstart's package set for k3s server roles, because `k3s etcdctl` is not a command and a member that needs to recover its own membership has no internet. 542 unit tests / 39 files. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Michal
31
bastion/src/modules/modules/k3s/src/groups/admit.ts
Normal file
31
bastion/src/modules/modules/k3s/src/groups/admit.ts
Normal file
@@ -0,0 +1,31 @@
|
||||
// Return an evacuated node to service. The inverse of the evacuate group.
|
||||
//
|
||||
// Needed for two different journeys that end in the same place:
|
||||
// - a node that was wiped, reinstalled and has just rejoined the cluster;
|
||||
// - a node whose evacuation was aborted and must simply be put back.
|
||||
//
|
||||
// Order matters here too. configureLonghornDisk runs AFTER eviction is cleared,
|
||||
// because a node that is still marked for eviction will have any disk we give
|
||||
// it immediately drained again.
|
||||
|
||||
import type { OperationContext, OperationResult, OperationGroup } from "../types.js";
|
||||
import { runSequential } from "../utils.js";
|
||||
import { uncordonNode } from "../operations/node-evacuate.js";
|
||||
import { clearLonghornEviction, assertNodeSchedulable } from "../operations/node-admit.js";
|
||||
import { configureLonghornDisk } from "../operations/longhorn-disk.js";
|
||||
|
||||
export const admitGroup: OperationGroup = {
|
||||
name: "admit",
|
||||
description: "Uncordon a node, clear its eviction request and restore its Longhorn disk",
|
||||
operations: [
|
||||
{ name: "Uncordon node", fn: uncordonNode },
|
||||
{ name: "Clear Longhorn eviction", fn: clearLonghornEviction },
|
||||
{ name: "Restore Longhorn default disk", fn: configureLonghornDisk },
|
||||
{ name: "Verify node is back in service", fn: assertNodeSchedulable },
|
||||
],
|
||||
};
|
||||
|
||||
export async function runAdmit(ctx: OperationContext): Promise<OperationResult[]> {
|
||||
ctx.log("Returning node to service...");
|
||||
return runSequential(ctx, admitGroup.operations);
|
||||
}
|
||||
54
bastion/src/modules/modules/k3s/src/groups/evacuate.ts
Normal file
54
bastion/src/modules/modules/k3s/src/groups/evacuate.ts
Normal file
@@ -0,0 +1,54 @@
|
||||
// Evacuate a node so it can be wiped and reinstalled with a different layout.
|
||||
//
|
||||
// The ORDER is the operation:
|
||||
//
|
||||
// 1. Cordon stop new work landing on a node that is about to leave.
|
||||
// 2. Check can eviction even finish? Seconds to answer, an hour to
|
||||
// discover the hard way.
|
||||
// 3. Evict move the replicas and WAIT for them to rebuild elsewhere.
|
||||
// 4. Drain now move the workloads.
|
||||
// 5. Verify re-read the cluster; do not trust that 1-4 worked.
|
||||
//
|
||||
// EVICT BEFORE DRAIN, which is the opposite of the obvious order and the one
|
||||
// thing here most likely to be "tidied" back into a bug. `drain --force` does
|
||||
// not override PodDisruptionBudgets, and longhorn-manager holds a PDB on the
|
||||
// instance-manager pod for as long as volumes are attached or degraded. Drain
|
||||
// first and you stop the instance-manager that serves the replicas you are
|
||||
// about to evict: the drain blocks for its full 15-minute timeout, and if it
|
||||
// did succeed it would have removed the only source a single-replica volume
|
||||
// could rebuild from.
|
||||
//
|
||||
// Removing the etcd member is NOT part of this group -- it costs you the API
|
||||
// server the rest of the procedure needs, so it belongs immediately before the
|
||||
// wipe, not here.
|
||||
//
|
||||
// This group deliberately does not reinstall anything. It leaves the node
|
||||
// cordoned, drained and empty, and a human (or `labctl provision reprovision`)
|
||||
// takes it from there.
|
||||
|
||||
import type { OperationContext, OperationResult, OperationGroup } from "../types.js";
|
||||
import { runSequential } from "../utils.js";
|
||||
import {
|
||||
cordonNode,
|
||||
drainNode,
|
||||
evictLonghornReplicas,
|
||||
assertEvictionCompletable,
|
||||
assertNodeEvacuated,
|
||||
} from "../operations/node-evacuate.js";
|
||||
|
||||
export const evacuateGroup: OperationGroup = {
|
||||
name: "evacuate",
|
||||
description: "Cordon, evict storage from and drain a node so it can be safely wiped",
|
||||
operations: [
|
||||
{ name: "Cordon node", fn: cordonNode },
|
||||
{ name: "Check eviction can complete", fn: assertEvictionCompletable },
|
||||
{ name: "Evict Longhorn replicas (waits for rebuild)", fn: evictLonghornReplicas },
|
||||
{ name: "Drain workloads", fn: drainNode },
|
||||
{ name: "Verify node is empty", fn: assertNodeEvacuated },
|
||||
],
|
||||
};
|
||||
|
||||
export async function runEvacuate(ctx: OperationContext): Promise<OperationResult[]> {
|
||||
ctx.log("Evacuating node before wipe...");
|
||||
return runSequential(ctx, evacuateGroup.operations);
|
||||
}
|
||||
@@ -4,6 +4,7 @@ import type { OperationContext, OperationResult, OperationGroup } from "../types
|
||||
import { runSequential } from "../utils.js";
|
||||
import { writeK3sConfig } from "../operations/k3s-config.js";
|
||||
import { installK3sBinary } from "../operations/k3s-install.js";
|
||||
import { labelWorkerRole } from "../operations/worker-label.js";
|
||||
|
||||
export const k3sAgentGroup: OperationGroup = {
|
||||
name: "k3s-agent",
|
||||
@@ -11,6 +12,8 @@ export const k3sAgentGroup: OperationGroup = {
|
||||
operations: [
|
||||
{ name: "Write k3s config", fn: writeK3sConfig },
|
||||
{ name: "Install k3s agent", fn: installK3sBinary },
|
||||
// After registration, never before: the kubelet may not set this itself.
|
||||
{ name: "Label node as worker", fn: labelWorkerRole },
|
||||
],
|
||||
};
|
||||
|
||||
|
||||
@@ -1,8 +1,9 @@
|
||||
import type { Operation, OperationResult } from "../types.js";
|
||||
import { isServiceActive } from "../utils.js";
|
||||
import { isK3sServer } from "@lab/shared";
|
||||
|
||||
export const checkK3sService: Operation = async (ctx): Promise<OperationResult> => {
|
||||
const isServer = ctx.config.role === "infra" || ctx.config.role === "labcontroller";
|
||||
const isServer = isK3sServer(ctx.config.role);
|
||||
const service = isServer ? "k3s" : "k3s-agent";
|
||||
const active = await isServiceActive(ctx, service);
|
||||
return { success: active, changed: false, message: active ? `${service} is active` : `${service} is not active` };
|
||||
|
||||
@@ -1,17 +1,21 @@
|
||||
// k3s module — install phase.
|
||||
// Installs k3s with CIS-hardened configuration and Cilium CNI.
|
||||
|
||||
import { isK3sServer, type Role } from "@lab/shared";
|
||||
|
||||
export interface K3sInstallContext {
|
||||
hostname: string;
|
||||
ip: string;
|
||||
role: string; // "infra" = server, "worker" = agent
|
||||
role: Role; // server vs agent is derived via isK3sServer(), never by name
|
||||
k3sServerUrl?: string; // Required for agent role
|
||||
k3sToken?: string; // Required for agent role
|
||||
}
|
||||
|
||||
/** Generate the shell script that installs k3s on a target machine. */
|
||||
export function generateInstallScript(ctx: K3sInstallContext): string {
|
||||
const isServer = ctx.role === "infra";
|
||||
// Was `ctx.role === "infra"`, which silently treated a `labcontroller` -- a
|
||||
// role that runs a k3s SERVER -- as an agent. Derived from the registry now.
|
||||
const isServer = isK3sServer(ctx.role);
|
||||
|
||||
return `#!/bin/bash
|
||||
set -euo pipefail
|
||||
|
||||
@@ -6,21 +6,35 @@ import type { OperationContext, K3sConfig, OperationResult } from "./types.js";
|
||||
import { sshExec } from "../../../src/ssh.js";
|
||||
import { aggregateResults } from "./utils.js";
|
||||
import { runHostPrep } from "./groups/host-prep.js";
|
||||
import { runEvacuate } from "./groups/evacuate.js";
|
||||
import { runAdmit } from "./groups/admit.js";
|
||||
import { runK3sServer } from "./groups/k3s-server.js";
|
||||
import { runK3sAgent } from "./groups/k3s-agent.js";
|
||||
import { runNetworking } from "./groups/networking.js";
|
||||
import { runHardening } from "./groups/hardening.js";
|
||||
import { runSequential } from "./utils.js";
|
||||
import * as health from "./health/index.js";
|
||||
import { isK3sServer } from "@lab/shared";
|
||||
|
||||
function toOpContext(ctx: ModuleContext): OperationContext {
|
||||
// Pass the WHOLE config bag through, then pin the fields that come from the
|
||||
// ModuleContext itself.
|
||||
//
|
||||
// This used to enumerate three keys -- k3sServerUrl, k3sToken, tlsSans -- and
|
||||
// silently drop every other one. Anything a caller set beyond those three
|
||||
// vanished with no error and no warning: k3sVersion (so a pinned version was
|
||||
// ignored and get.k3s.io installed LATEST), evictionAllowSingleReplica (so
|
||||
// `labctl app k3s evacuate --allow-single-replica` was a no-op that claimed
|
||||
// to override a data-loss guard), and the dual-stack fields ipv6 /
|
||||
// clusterCidr / serviceCidr.
|
||||
//
|
||||
// Enumerating is the bug: every field added to K3sConfig has to be
|
||||
// remembered here, and forgetting produces silence rather than a type error.
|
||||
const config: K3sConfig = {
|
||||
...(ctx.config as Partial<K3sConfig>),
|
||||
hostname: ctx.hostname,
|
||||
ip: ctx.ip,
|
||||
role: ctx.role as K3sConfig["role"],
|
||||
k3sServerUrl: ctx.config["k3sServerUrl"] as string | undefined,
|
||||
k3sToken: ctx.config["k3sToken"] as string | undefined,
|
||||
tlsSans: ctx.config["tlsSans"] as string[] | undefined,
|
||||
};
|
||||
return {
|
||||
config,
|
||||
@@ -35,7 +49,16 @@ function toOpContext(ctx: ModuleContext): OperationContext {
|
||||
},
|
||||
os: ctx.os,
|
||||
arch: ctx.arch,
|
||||
log: (_msg) => { /* collected via results */ },
|
||||
// Emit progress LIVE, on stderr.
|
||||
//
|
||||
// This used to be a no-op "collected via results", but results only carry
|
||||
// an operation's final message. Anything an operation reports WHILE it
|
||||
// runs was therefore thrown away -- including evictLonghornReplicas's
|
||||
// per-poll "N replica(s) still on <node>", which is the only visibility
|
||||
// into an eviction that can legitimately take an hour. When one stalled,
|
||||
// there was no way to tell which volumes were stuck or whether progress
|
||||
// was being made at all.
|
||||
log: (msg) => { console.error(` [${ctx.hostname}] ${msg}`); }, // eslint-disable-line no-console
|
||||
};
|
||||
}
|
||||
|
||||
@@ -62,7 +85,7 @@ export class K3sModule implements Module {
|
||||
async install(ctx: ModuleContext): Promise<ModuleResult> {
|
||||
const start = performance.now();
|
||||
const opCtx = toOpContext(ctx);
|
||||
const isServer = ctx.role === "infra" || ctx.role === "labcontroller";
|
||||
const isServer = isK3sServer(ctx.role);
|
||||
|
||||
// Phase 1: Host preparation
|
||||
const prepResults = await runHostPrep(opCtx);
|
||||
@@ -95,6 +118,31 @@ export class K3sModule implements Module {
|
||||
return toModuleResult("configure", results, start);
|
||||
}
|
||||
|
||||
/**
|
||||
* Evacuate the node so it can be wiped. Does NOT reinstall anything --
|
||||
* it leaves the node cordoned, drained and empty.
|
||||
*/
|
||||
async evacuate(ctx: ModuleContext): Promise<ModuleResult> {
|
||||
const start = performance.now();
|
||||
const opCtx = toOpContext(ctx);
|
||||
const results = await runEvacuate(opCtx);
|
||||
return toModuleResult("evacuate", results, start);
|
||||
}
|
||||
|
||||
/**
|
||||
* Return an evacuated node to service: uncordon, clear Longhorn's eviction
|
||||
* request, restore its default disk.
|
||||
*
|
||||
* The counterpart to evacuate(), and the only supported way out of an
|
||||
* evacuation that was aborted partway.
|
||||
*/
|
||||
async admit(ctx: ModuleContext): Promise<ModuleResult> {
|
||||
const start = performance.now();
|
||||
const opCtx = toOpContext(ctx);
|
||||
const results = await runAdmit(opCtx);
|
||||
return toModuleResult("admit", results, start);
|
||||
}
|
||||
|
||||
async health(ctx: ModuleContext): Promise<ModuleResult> {
|
||||
const start = performance.now();
|
||||
const opCtx = toOpContext(ctx);
|
||||
|
||||
@@ -0,0 +1,60 @@
|
||||
// Parsing `etcdctl member list`.
|
||||
//
|
||||
// WHY NOT `-w json`
|
||||
// -----------------
|
||||
// Two reasons, and the second is the dangerous one:
|
||||
//
|
||||
// 1. `member remove` requires the id in HEX. The JSON writer emits decimal.
|
||||
// 2. etcd member ids are uint64. JSON.parse turns them into JS numbers, which
|
||||
// lose precision above 2^53 -- so the id that comes back is an
|
||||
// APPROXIMATION of the real one.
|
||||
//
|
||||
// Observed for real: `member remove 13323819137391910000` ->
|
||||
// "bad member ID arg (strconv.ParseUint: value out of range), expecting ID in
|
||||
// Hex". That happens at the worst possible moment, with the node already
|
||||
// stopped and about to be wiped. Failing loudly is the lucky outcome; a
|
||||
// mangled id that happened to match another member would remove the WRONG one.
|
||||
//
|
||||
// The default writer prints the id as hex TEXT, so it never becomes a number.
|
||||
|
||||
export interface EtcdMember {
|
||||
/** Raw member name: "<node>-<8 hex>", e.g. "worker0.lab-e26aacc5". */
|
||||
name: string;
|
||||
/** The node name with etcd's random suffix stripped. */
|
||||
node: string;
|
||||
/** Hex id, exactly as etcd prints it — never round-tripped through a number. */
|
||||
id: string;
|
||||
started: boolean;
|
||||
}
|
||||
|
||||
/**
|
||||
* etcd names a member `<node>-<8 hex>`. Comparing a raw member name against a
|
||||
* node name without stripping that suffix silently never matches.
|
||||
*/
|
||||
export function etcdNodeName(memberName: string): string {
|
||||
return memberName.replace(/-[0-9a-f]{8}$/, "");
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse the default `etcdctl member list` writer:
|
||||
*
|
||||
* 825852d3683bc406, started, worker2.lab-5f7ad0c9, https://..:2380, https://..:2379, false
|
||||
* <hex id> , status , <node>-<8 hex> , peer , client , learner
|
||||
*/
|
||||
export function parseEtcdMembers(out: string): EtcdMember[] {
|
||||
return out.split("\n")
|
||||
.map((l) => l.trim())
|
||||
.filter((l) => l !== "")
|
||||
.map((line) => {
|
||||
const cols = line.split(",").map((c) => c.trim());
|
||||
const name = cols[2] ?? "";
|
||||
return {
|
||||
name,
|
||||
node: etcdNodeName(name),
|
||||
id: cols[0] ?? "",
|
||||
started: (cols[1] ?? "") === "started",
|
||||
};
|
||||
})
|
||||
// Drop anything without a hex id: blank lines, warnings, stray output.
|
||||
.filter((m) => /^[0-9a-f]+$/i.test(m.id));
|
||||
}
|
||||
@@ -24,3 +24,10 @@ export type {
|
||||
RecoverEtcdMemberOptions,
|
||||
RecoverEtcdMemberResult,
|
||||
} from "./etcd-recover.js";
|
||||
export {
|
||||
cordonNode, uncordonNode, drainNode, evictLonghornReplicas,
|
||||
assertEvictionCompletable, assertNodeEvacuated, checkEvictionCompletable,
|
||||
} from "./node-evacuate.js";
|
||||
export type { EvictionFeasibility } from "./node-evacuate.js";
|
||||
export { clearLonghornEviction, assertNodeSchedulable } from "./node-admit.js";
|
||||
export { labelWorkerRole } from "./worker-label.js";
|
||||
|
||||
@@ -2,10 +2,7 @@
|
||||
|
||||
import type { Operation, OperationResult, K3sConfig } from "../types.js";
|
||||
import { sshOpts, writeRemoteFile } from "../utils.js";
|
||||
|
||||
function isServerRole(role: string): boolean {
|
||||
return role === "infra" || role === "labcontroller";
|
||||
}
|
||||
import { isK3sServer } from "@lab/shared";
|
||||
|
||||
/**
|
||||
* The address-family block: `cluster-cidr`, `service-cidr` and `node-ip`.
|
||||
@@ -101,9 +98,28 @@ ${tlsSans.map((s) => ` - "${s}"`).join("\n")}
|
||||
// node being unreachable over v6 while the node itself reads Ready.
|
||||
// It takes no cluster/service CIDRs -- those are server-side only.
|
||||
export function generateAgentConfig(config: K3sConfig): string {
|
||||
// NO node-role.kubernetes.io/worker LABEL HERE, deliberately.
|
||||
//
|
||||
// A kubelet is not permitted to assign itself labels in the
|
||||
// `node-role.kubernetes.io/` namespace -- the servers run with
|
||||
// `enable-admission-plugins=NodeRestriction`, which rejects the registration
|
||||
// outright. The agent then never finishes registering, k3s-agent.service
|
||||
// never signals readiness, and systemd reports:
|
||||
//
|
||||
// Job for k3s-agent.service failed because the service did not take the
|
||||
// steps required by its unit configuration.
|
||||
//
|
||||
// which says nothing about labels and sends you looking at the network, the
|
||||
// token, or the kubelet's sysctls. Every agent installed through this module
|
||||
// failed this way; the estate's existing workers only escaped it because
|
||||
// post-provision installs them with no config.yaml at all and their role
|
||||
// label was applied separately, server-side.
|
||||
//
|
||||
// `node.longhorn.io/...` is fine: it is outside the kubernetes.io namespace,
|
||||
// so the kubelet may set it. The worker role label is applied after the node
|
||||
// registers, by labelWorkerRole below.
|
||||
return `${addressFamilyLines(config, { cidrs: false })}protect-kernel-defaults: true
|
||||
node-label:
|
||||
- "node-role.kubernetes.io/worker=true"
|
||||
- "node.longhorn.io/create-default-disk=config"
|
||||
kubelet-arg:
|
||||
- "protect-kernel-defaults=true"
|
||||
@@ -145,7 +161,7 @@ export const writeK3sConfig: Operation = async (ctx): Promise<OperationResult> =
|
||||
|
||||
await ctx.ssh.exec("mkdir -p /etc/rancher/k3s", sshOpts(ctx));
|
||||
|
||||
const content = isServerRole(ctx.config.role)
|
||||
const content = isK3sServer(ctx.config.role)
|
||||
? generateServerConfig(ctx.config)
|
||||
: generateAgentConfig(ctx.config);
|
||||
|
||||
|
||||
@@ -2,13 +2,19 @@
|
||||
|
||||
import type { Operation, OperationResult } from "../types.js";
|
||||
import { sshOpts } from "../utils.js";
|
||||
|
||||
function isServerRole(role: string): boolean {
|
||||
return role === "infra" || role === "labcontroller";
|
||||
}
|
||||
import { isK3sServer } from "@lab/shared";
|
||||
|
||||
export const installK3sBinary: Operation = async (ctx): Promise<OperationResult> => {
|
||||
const isServer = isServerRole(ctx.config.role);
|
||||
const isServer = isK3sServer(ctx.config.role);
|
||||
|
||||
// Pin the version when asked. get.k3s.io installs LATEST by default, so a
|
||||
// node reinstalled during a rolling migration comes back on whatever shipped
|
||||
// since -- potentially several minor versions ahead of the cluster it is
|
||||
// about to rejoin, which Kubernetes does not support between control-plane
|
||||
// members.
|
||||
const versionEnv = ctx.config.k3sVersion !== undefined && ctx.config.k3sVersion !== ""
|
||||
? `INSTALL_K3S_VERSION="${ctx.config.k3sVersion}" `
|
||||
: "";
|
||||
|
||||
// Check if already installed
|
||||
const version = await ctx.ssh.exec("k3s --version 2>/dev/null", sshOpts(ctx));
|
||||
@@ -24,12 +30,21 @@ export const installK3sBinary: Operation = async (ctx): Promise<OperationResult>
|
||||
);
|
||||
}
|
||||
|
||||
// If joining an existing cluster, pass K3S_URL and K3S_TOKEN
|
||||
const joinEnv = ctx.config.k3sServerUrl && ctx.config.k3sToken
|
||||
? `K3S_URL="${ctx.config.k3sServerUrl}" K3S_TOKEN="${ctx.config.k3sToken}"`
|
||||
: "";
|
||||
// Token and URL are independent. A token WITHOUT a URL seeds the cluster
|
||||
// token on the cluster-init node, so every later joiner can be given the
|
||||
// same pre-shared value instead of scraping node-token off the first
|
||||
// server -- which matters here because that first server gets wiped and
|
||||
// rejoined during a rolling migration. URL only counts when a token comes
|
||||
// with it; a URL alone cannot authenticate a join.
|
||||
//
|
||||
// The token deliberately does NOT go in config.yaml: writeRemoteFile diffs
|
||||
// that file, so a token appearing there would restart k3s on every node.
|
||||
const joinEnv = [
|
||||
ctx.config.k3sServerUrl && ctx.config.k3sToken ? `K3S_URL="${ctx.config.k3sServerUrl}"` : "",
|
||||
ctx.config.k3sToken ? `K3S_TOKEN="${ctx.config.k3sToken}"` : "",
|
||||
].filter(Boolean).join(" ");
|
||||
const result = await ctx.ssh.exec(
|
||||
`curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="server" INSTALL_K3S_SKIP_SELINUX_RPM=true ${joinEnv} sh -`,
|
||||
`curl -sfL https://get.k3s.io | ${versionEnv}INSTALL_K3S_EXEC="server" INSTALL_K3S_SKIP_SELINUX_RPM=true ${joinEnv} sh -`,
|
||||
{ timeoutMs: 300_000 },
|
||||
);
|
||||
if (result.exitCode !== 0) {
|
||||
@@ -50,7 +65,7 @@ export const installK3sBinary: Operation = async (ctx): Promise<OperationResult>
|
||||
};
|
||||
}
|
||||
const result = await ctx.ssh.exec(
|
||||
`curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="agent" K3S_URL="${ctx.config.k3sServerUrl}" K3S_TOKEN="${ctx.config.k3sToken}" sh -`,
|
||||
`curl -sfL https://get.k3s.io | ${versionEnv}INSTALL_K3S_EXEC="agent" K3S_URL="${ctx.config.k3sServerUrl}" K3S_TOKEN="${ctx.config.k3sToken}" sh -`,
|
||||
{ timeoutMs: 300_000 },
|
||||
);
|
||||
if (result.exitCode !== 0) {
|
||||
|
||||
@@ -1,50 +1,206 @@
|
||||
// Annotate nodes with Longhorn default disk config when /var/lib/longhorn exists.
|
||||
// The label is set in k3s config (node-label), but the annotation must be applied via kubectl.
|
||||
// Give a node a Longhorn disk at /var/lib/longhorn.
|
||||
//
|
||||
// The cluster runs `createDefaultDiskLabeledNodes: true`, which means Longhorn
|
||||
// creates a disk ONLY on nodes carrying the label
|
||||
// `node.longhorn.io/create-default-disk=config` (set via k3s node-label) AND
|
||||
// the annotation `node.longhorn.io/default-disks-config` (set here). Miss
|
||||
// either and Longhorn gives the node no disk at all: it joins, goes Ready,
|
||||
// accepts pods, and silently hosts zero replicas.
|
||||
//
|
||||
// WHY THERE IS A REPAIR PATH
|
||||
// --------------------------
|
||||
// Longhorn reads that annotation when it FIRST creates the node's
|
||||
// `nodes.longhorn.io` record. This operation runs during configure(), which on
|
||||
// a rejoining node is after the record already exists -- so the annotation
|
||||
// lands too late and is never read. That is invisible until a rebuild is
|
||||
// expected and never arrives, roughly forty minutes into a migration.
|
||||
//
|
||||
// So: annotate, then read back, and if there is still no disk, write it
|
||||
// directly into spec.disks. When Longhorn does honour the annotation the
|
||||
// repair is a no-op.
|
||||
|
||||
import type { Operation, OperationResult } from "../types.js";
|
||||
import type { Operation, OperationContext, OperationResult } from "../types.js";
|
||||
import { sshOpts } from "../utils.js";
|
||||
import { sshExec as remoteSshExec } from "../../../../src/ssh.js";
|
||||
|
||||
const DISK_PATH = "/var/lib/longhorn";
|
||||
/** Fixed key so a repeated repair updates one disk rather than adding more. */
|
||||
const DISK_KEY = "default-disk-lab";
|
||||
const VERIFY_INTERVAL_MS = 5_000;
|
||||
/** How long to wait for Longhorn to create the node's record after it joins. */
|
||||
const REGISTER_TIMEOUT_MS = 180_000;
|
||||
/** How long to then wait for Longhorn to give that record a disk. */
|
||||
const DISK_TIMEOUT_MS = 60_000;
|
||||
|
||||
const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));
|
||||
|
||||
/**
|
||||
* Run kubectl from wherever it can be reached: locally on a server, otherwise
|
||||
* over SSH to the configured server (an agent has no local kubectl).
|
||||
*/
|
||||
async function kubectlAnywhere(
|
||||
ctx: OperationContext,
|
||||
args: string,
|
||||
): Promise<{ ok: boolean; out: string }> {
|
||||
const local = await ctx.ssh.exec(`k3s kubectl ${args} 2>&1`, sshOpts(ctx));
|
||||
if (local.exitCode === 0) return { ok: true, out: local.stdout };
|
||||
|
||||
if (ctx.config.k3sServerUrl) {
|
||||
try {
|
||||
const serverHost = new URL(ctx.config.k3sServerUrl).hostname;
|
||||
const remote = await remoteSshExec(serverHost, "root", `k3s kubectl ${args}`, {
|
||||
...(ctx.ssh.keyPath ? { keyPath: ctx.ssh.keyPath } : {}),
|
||||
timeoutMs: 15_000,
|
||||
});
|
||||
if (remote.exitCode === 0) return { ok: true, out: remote.stdout };
|
||||
return { ok: false, out: remote.stdout || remote.stderr };
|
||||
} catch {
|
||||
/* fall through */
|
||||
}
|
||||
}
|
||||
return { ok: false, out: local.stdout || local.stderr };
|
||||
}
|
||||
|
||||
/** Does the node's Longhorn record exist yet? */
|
||||
async function nodeRecordExists(ctx: OperationContext, nodeName: string): Promise<boolean> {
|
||||
const res = await kubectlAnywhere(ctx, `-n longhorn-system get nodes.longhorn.io "${nodeName}" -o name`);
|
||||
return res.ok && res.out.trim() !== "";
|
||||
}
|
||||
|
||||
/** Does the node's Longhorn record carry a disk at DISK_PATH? */
|
||||
async function hasDisk(ctx: OperationContext, nodeName: string): Promise<boolean> {
|
||||
const res = await kubectlAnywhere(
|
||||
ctx,
|
||||
`-n longhorn-system get nodes.longhorn.io "${nodeName}" ` +
|
||||
`-o jsonpath='{range .spec.disks.*}{.path}{"\\n"}{end}'`,
|
||||
);
|
||||
if (!res.ok) return false;
|
||||
return res.out.split("\n").map((l) => l.trim()).includes(DISK_PATH);
|
||||
}
|
||||
|
||||
export const configureLonghornDisk: Operation = async (ctx): Promise<OperationResult> => {
|
||||
// Check if /var/lib/longhorn exists on this node
|
||||
const check = await ctx.ssh.exec("test -d /var/lib/longhorn && echo yes || echo no", sshOpts(ctx));
|
||||
const check = await ctx.ssh.exec(`test -d ${DISK_PATH} && echo yes || echo no`, sshOpts(ctx));
|
||||
if (check.stdout.trim() !== "yes") {
|
||||
return { success: true, changed: false, message: "No /var/lib/longhorn directory — skipping Longhorn disk config" };
|
||||
return { success: true, changed: false, message: `No ${DISK_PATH} directory — skipping Longhorn disk config` };
|
||||
}
|
||||
|
||||
// Find the node name (hostname as registered in k3s)
|
||||
// Longhorn writes replicas into <path>/replicas and creates that directory
|
||||
// only when it FIRST initialises the disk. If the directory later goes away
|
||||
// while longhorn-disk.cfg remains -- someone clearing space, a partial
|
||||
// restore, a wipe that preserved the LV -- Longhorn considers the disk
|
||||
// already initialised and never recreates it. The replica process then dies
|
||||
// on a non-recursive mkdir:
|
||||
//
|
||||
// level=fatal msg="Error running start replica command"
|
||||
// error="mkdir /host/var/lib/longhorn/replicas/pvc-...: ..."
|
||||
//
|
||||
// every single time, for ever, while the node reports Ready and Schedulable
|
||||
// and its disk looks healthy. Cheap to guarantee, expensive to diagnose.
|
||||
await ctx.ssh.exec(`mkdir -p ${DISK_PATH}/replicas`, sshOpts(ctx));
|
||||
|
||||
const nodeNameResult = await ctx.ssh.exec("hostname -f 2>/dev/null || hostname", sshOpts(ctx));
|
||||
const nodeName = nodeNameResult.stdout.trim();
|
||||
|
||||
const annotation = JSON.stringify([{ path: "/var/lib/longhorn", allowScheduling: true }]);
|
||||
|
||||
// Try kubectl locally first (works on server nodes)
|
||||
const result = await ctx.ssh.exec(
|
||||
`k3s kubectl annotate node "${nodeName}" "node.longhorn.io/default-disks-config=${annotation}" --overwrite 2>&1 || true`,
|
||||
sshOpts(ctx),
|
||||
// SINGLE quotes around the whole key=value, not double.
|
||||
//
|
||||
// The value is JSON and therefore full of double quotes. Wrapping it in
|
||||
// double quotes ends the quoted string at the first inner one, and the shell
|
||||
// strips the rest, so the node gets
|
||||
// [{path:/var/lib/longhorn,allowScheduling:true}]
|
||||
// instead of
|
||||
// [{"path":"/var/lib/longhorn","allowScheduling":true}]
|
||||
// -- valid-looking, unparseable, and silent. Longhorn then creates NO disk on
|
||||
// the node, which surfaces much later as "why does nothing schedule here".
|
||||
// The JSON contains no single quotes, so single-quoting is safe.
|
||||
const annotation = JSON.stringify([{ path: DISK_PATH, allowScheduling: true }]);
|
||||
const annotated = await kubectlAnywhere(
|
||||
ctx,
|
||||
`annotate node "${nodeName}" 'node.longhorn.io/default-disks-config=${annotation}' --overwrite`,
|
||||
);
|
||||
|
||||
if (result.stdout.includes("annotated") || result.stdout.includes("unchanged")) {
|
||||
return { success: true, changed: true, message: `Longhorn disk annotation applied to ${nodeName}` };
|
||||
// No Longhorn at all (or no way to reach the API) — nothing more to do.
|
||||
const lh = await kubectlAnywhere(ctx, "get ns longhorn-system -o name");
|
||||
if (!lh.ok) {
|
||||
return annotated.ok
|
||||
? { success: true, changed: true, message: `Longhorn disk annotation applied to ${nodeName} (Longhorn not installed yet)` }
|
||||
: { success: true, changed: false, message: "Longhorn disk label set (annotation requires server kubectl)" };
|
||||
}
|
||||
|
||||
// For worker/agent nodes without local kubectl: apply via the server
|
||||
if (ctx.config.k3sServerUrl) {
|
||||
// The CLI has SSH access to the server — use sshExec from there
|
||||
const serverHost = new URL(ctx.config.k3sServerUrl).hostname;
|
||||
try {
|
||||
const remoteResult = await remoteSshExec(
|
||||
serverHost, "root",
|
||||
`k3s kubectl annotate node "${nodeName}" "node.longhorn.io/default-disks-config=${annotation}" --overwrite`,
|
||||
{ ...(ctx.ssh.keyPath ? { keyPath: ctx.ssh.keyPath } : {}), timeoutMs: 15_000 },
|
||||
);
|
||||
if (remoteResult.stdout.includes("annotated") || remoteResult.stdout.includes("unchanged")) {
|
||||
return { success: true, changed: true, message: `Longhorn disk annotation applied to ${nodeName} (via server)` };
|
||||
}
|
||||
} catch {
|
||||
// Fall through to manual instruction
|
||||
// Longhorn registration is ASYNCHRONOUS. On a node that has just joined --
|
||||
// which is exactly the case after a wipe-and-rejoin -- the nodes.longhorn.io
|
||||
// record does not exist yet, so there is nothing to inspect and nothing to
|
||||
// repair. Waiting a fixed 60s and then giving up meant the repair never ran
|
||||
// on the one path that needs it most, and the caller's schedulability gate
|
||||
// then failed immediately with "Longhorn reports no disk".
|
||||
//
|
||||
// So: wait for the RECORD first, generously, and only then worry about the
|
||||
// disk on it.
|
||||
const registerDeadline = Date.now() + REGISTER_TIMEOUT_MS;
|
||||
let registered = false;
|
||||
while (Date.now() < registerDeadline) {
|
||||
if (await hasDisk(ctx, nodeName)) {
|
||||
return { success: true, changed: annotated.ok, message: `${nodeName} has a Longhorn disk at ${DISK_PATH}` };
|
||||
}
|
||||
if (await nodeRecordExists(ctx, nodeName)) { registered = true; break; }
|
||||
await sleep(VERIFY_INTERVAL_MS);
|
||||
}
|
||||
|
||||
return { success: true, changed: false, message: "Longhorn disk label set (annotation requires server kubectl)" };
|
||||
if (!registered) {
|
||||
return {
|
||||
success: true,
|
||||
changed: annotated.ok,
|
||||
message: `${nodeName} not registered with Longhorn within ${REGISTER_TIMEOUT_MS / 1000}s; ` +
|
||||
`annotation is in place for when it is`,
|
||||
};
|
||||
}
|
||||
|
||||
// Registered. Give Longhorn a chance to act on the annotation by itself
|
||||
// before overriding it.
|
||||
const diskDeadline = Date.now() + DISK_TIMEOUT_MS;
|
||||
while (Date.now() < diskDeadline) {
|
||||
if (await hasDisk(ctx, nodeName)) {
|
||||
return { success: true, changed: annotated.ok, message: `${nodeName} has a Longhorn disk at ${DISK_PATH}` };
|
||||
}
|
||||
await sleep(VERIFY_INTERVAL_MS);
|
||||
}
|
||||
|
||||
// The record exists and still has no disk: the annotation was applied too
|
||||
// late to be read. Write it in.
|
||||
ctx.log(`${nodeName} has no Longhorn disk after annotating; writing spec.disks directly`);
|
||||
const patch = JSON.stringify({
|
||||
spec: {
|
||||
disks: {
|
||||
[DISK_KEY]: {
|
||||
path: DISK_PATH,
|
||||
allowScheduling: true,
|
||||
evictionRequested: false,
|
||||
storageReserved: 0,
|
||||
tags: [],
|
||||
diskType: "filesystem",
|
||||
},
|
||||
},
|
||||
},
|
||||
});
|
||||
const repaired = await kubectlAnywhere(
|
||||
ctx,
|
||||
`-n longhorn-system patch nodes.longhorn.io "${nodeName}" --type=merge -p '${patch}'`,
|
||||
);
|
||||
if (!repaired.ok) {
|
||||
return {
|
||||
success: false,
|
||||
changed: false,
|
||||
message: `${nodeName} has no Longhorn disk and could not be repaired`,
|
||||
error: `${repaired.out}. Without a disk this node will never host a replica.`,
|
||||
};
|
||||
}
|
||||
|
||||
if (await hasDisk(ctx, nodeName)) {
|
||||
return { success: true, changed: true, message: `Longhorn disk repaired on ${nodeName} (${DISK_PATH})` };
|
||||
}
|
||||
return {
|
||||
success: false,
|
||||
changed: true,
|
||||
message: `Patched spec.disks on ${nodeName} but Longhorn still reports no disk at ${DISK_PATH}`,
|
||||
error: "Check longhorn-manager logs; the node will not host replicas until this is resolved.",
|
||||
};
|
||||
};
|
||||
|
||||
136
bastion/src/modules/modules/k3s/src/operations/node-admit.ts
Normal file
136
bastion/src/modules/modules/k3s/src/operations/node-admit.ts
Normal file
@@ -0,0 +1,136 @@
|
||||
// Bring a node back into service. The inverse of node-evacuate.ts.
|
||||
//
|
||||
// WHY THIS EXISTS
|
||||
// ---------------
|
||||
// Evacuation is not symmetric with itself. It leaves three marks on a node --
|
||||
// a kubernetes cordon, Longhorn's `allowScheduling:false` + `evictionRequested`,
|
||||
// and (on a node that was wiped and rejoined) a MISSING default Longhorn disk.
|
||||
// Undoing a cordon undoes only the first. Without the other two the node comes
|
||||
// back Ready, accepts pods, and quietly hosts no replicas at all -- which looks
|
||||
// like success right up until you notice storage never rebalanced onto it.
|
||||
//
|
||||
// This is also the only supported way out of an aborted evacuation.
|
||||
|
||||
import type { Operation, OperationContext, OperationResult } from "../types.js";
|
||||
import { sshOpts } from "../utils.js";
|
||||
|
||||
async function nodeName(ctx: OperationContext): Promise<string> {
|
||||
const r = await ctx.ssh.exec("hostname -f 2>/dev/null || hostname", sshOpts(ctx));
|
||||
return r.stdout.trim();
|
||||
}
|
||||
|
||||
async function kubectl(ctx: OperationContext, args: string): Promise<{ ok: boolean; out: string }> {
|
||||
const res = await ctx.ssh.exec(`k3s kubectl ${args} 2>&1`, sshOpts(ctx));
|
||||
return { ok: res.exitCode === 0, out: res.exitCode === 0 ? res.stdout : (res.stdout || res.stderr) };
|
||||
}
|
||||
|
||||
/**
|
||||
* Clear Longhorn's eviction request and re-enable scheduling.
|
||||
*
|
||||
* Both fields in one merge patch, mirroring how eviction sets them: Longhorn's
|
||||
* validating webhook rejects `evictionRequested:true` on a schedulable node, so
|
||||
* the pair has to move together in both directions.
|
||||
*/
|
||||
export const clearLonghornEviction: Operation = async (ctx): Promise<OperationResult> => {
|
||||
const name = await nodeName(ctx);
|
||||
|
||||
const lh = await kubectl(ctx, "get ns longhorn-system -o name");
|
||||
if (!lh.ok) {
|
||||
return { success: true, changed: false, message: "Longhorn not installed — nothing to clear" };
|
||||
}
|
||||
|
||||
const before = await kubectl(
|
||||
ctx,
|
||||
`-n longhorn-system get nodes.longhorn.io "${name}" -o jsonpath='{.spec.allowScheduling},{.spec.evictionRequested}'`,
|
||||
);
|
||||
if (!before.ok) {
|
||||
return { success: true, changed: false, message: `${name} has no Longhorn node record yet` };
|
||||
}
|
||||
if (before.out.trim() === "true,false") {
|
||||
return { success: true, changed: false, message: `${name} already accepts replicas` };
|
||||
}
|
||||
|
||||
const patch = JSON.stringify({ spec: { allowScheduling: true, evictionRequested: false } });
|
||||
const res = await kubectl(
|
||||
ctx,
|
||||
`-n longhorn-system patch nodes.longhorn.io "${name}" --type=merge -p '${patch}'`,
|
||||
);
|
||||
return res.ok
|
||||
? { success: true, changed: true, message: `${name} accepts replicas again` }
|
||||
: { success: false, changed: false, message: `Could not clear eviction on ${name}`, error: res.out };
|
||||
};
|
||||
|
||||
/**
|
||||
* Confirm the node can actually host replicas again.
|
||||
*
|
||||
* Reads back from Longhorn rather than trusting the patches above, because the
|
||||
* failure this catches is a node that is Ready and schedulable in kubernetes
|
||||
* while having no Longhorn disk at all -- the exact state a rejoined node lands
|
||||
* in when its default-disk annotation was applied too late.
|
||||
*/
|
||||
const SCHEDULABLE_TIMEOUT_MS = 120_000;
|
||||
const SCHEDULABLE_INTERVAL_MS = 5_000;
|
||||
const sleep = (ms: number): Promise<void> => new Promise((r) => setTimeout(r, ms));
|
||||
|
||||
export const assertNodeSchedulable: Operation = async (ctx): Promise<OperationResult> => {
|
||||
// Poll rather than check once.
|
||||
//
|
||||
// Longhorn registers a node and attaches its disk ASYNCHRONOUSLY, and this
|
||||
// runs moments after a wipe-and-rejoin. Checking instantly reports "Longhorn
|
||||
// reports no disk on this node" for a node that is simply a few seconds from
|
||||
// being fine -- which is a false alarm on the exact path the gate exists to
|
||||
// protect. Still fails if it never settles.
|
||||
const deadline = Date.now() + SCHEDULABLE_TIMEOUT_MS;
|
||||
let problems: string[] = [];
|
||||
for (;;) {
|
||||
problems = await schedulableProblems(ctx);
|
||||
if (problems.length === 0 || Date.now() >= deadline) break;
|
||||
await sleep(SCHEDULABLE_INTERVAL_MS);
|
||||
}
|
||||
|
||||
const name = await nodeName(ctx);
|
||||
return problems.length === 0
|
||||
? { success: true, changed: false, message: `${name} is back in service` }
|
||||
: {
|
||||
success: false,
|
||||
changed: false,
|
||||
message: `${name} is NOT fully back in service after ${SCHEDULABLE_TIMEOUT_MS / 1000}s`,
|
||||
error: problems.join("; "),
|
||||
};
|
||||
};
|
||||
|
||||
/** One pass of the schedulability checks. Returns the reasons it is not. */
|
||||
async function schedulableProblems(ctx: OperationContext): Promise<string[]> {
|
||||
const name = await nodeName(ctx);
|
||||
const problems: string[] = [];
|
||||
|
||||
const sched = await kubectl(ctx, `get node "${name}" -o jsonpath='{.spec.unschedulable}'`);
|
||||
if (sched.ok && sched.out.trim() === "true") problems.push("node is still cordoned");
|
||||
|
||||
const lh = await kubectl(ctx, "get ns longhorn-system -o name");
|
||||
if (lh.ok) {
|
||||
const spec = await kubectl(
|
||||
ctx,
|
||||
`-n longhorn-system get nodes.longhorn.io "${name}" -o jsonpath='{.spec.allowScheduling},{.spec.evictionRequested}'`,
|
||||
);
|
||||
if (!spec.ok) {
|
||||
problems.push("no Longhorn node record");
|
||||
} else {
|
||||
const [allow, evicting] = spec.out.trim().split(",");
|
||||
if (allow !== "true") problems.push("Longhorn scheduling is still disabled");
|
||||
if (evicting === "true") problems.push("Longhorn eviction is still requested");
|
||||
}
|
||||
|
||||
// A node with no disk hosts no replicas, however healthy it otherwise looks.
|
||||
const disks = await kubectl(
|
||||
ctx,
|
||||
`-n longhorn-system get nodes.longhorn.io "${name}" ` +
|
||||
`-o jsonpath='{range .status.diskStatus.*}{.conditions[?(@.type=="Schedulable")].status}{"\\n"}{end}'`,
|
||||
);
|
||||
const states = disks.ok ? disks.out.split("\n").map((l) => l.trim()).filter(Boolean) : [];
|
||||
if (states.length === 0) problems.push("Longhorn reports no disk on this node");
|
||||
else if (!states.includes("True")) problems.push(`no schedulable disk (states: ${states.join(",")})`);
|
||||
}
|
||||
|
||||
return problems;
|
||||
}
|
||||
547
bastion/src/modules/modules/k3s/src/operations/node-evacuate.ts
Normal file
547
bastion/src/modules/modules/k3s/src/operations/node-evacuate.ts
Normal file
@@ -0,0 +1,547 @@
|
||||
// Evacuate a node before it is wiped and reinstalled.
|
||||
//
|
||||
// WHY THIS EXISTS
|
||||
// ---------------
|
||||
// The kickstart's %pre can preserve data across a reinstall, but ONLY when the
|
||||
// disk layout is unchanged: it keys off `labvg` already existing with the
|
||||
// expected LV names. Migrating a machine to the v2 layout means recreating the
|
||||
// volume group, which takes the `clearpart --all` branch and destroys every
|
||||
// Longhorn replica on that node.
|
||||
//
|
||||
// So a layout migration cannot rely on disk-level preservation. The node has to
|
||||
// be EVACUATED first -- replicas rebuilt elsewhere, workloads moved, etcd
|
||||
// membership surrendered -- and then wiped. These operations are the missing
|
||||
// half; before this file the repo had no drain, no cordon, and no replica
|
||||
// eviction anywhere.
|
||||
//
|
||||
// Ordering matters and is not interchangeable:
|
||||
// cordon -> check -> evict replicas (WAIT for rebuild) -> drain -> remove etcd member
|
||||
//
|
||||
// Two ordering facts, both learned the hard way:
|
||||
// - Removing the etcd member first costs you the API server you need for the
|
||||
// rest of the procedure.
|
||||
// - EVICT BEFORE DRAIN. `drain --force` does not override PodDisruptionBudgets,
|
||||
// and longhorn-manager keeps a PDB on the instance-manager pod while volumes
|
||||
// are attached or degraded. Draining a storage node first therefore stops the
|
||||
// instance-manager that serves the very replicas being evicted: the drain
|
||||
// blocks for its full timeout, and if it did succeed it would remove the only
|
||||
// source a single-replica volume could rebuild from.
|
||||
|
||||
import type { Operation, OperationContext, OperationResult } from "../types.js";
|
||||
import { sshOpts } from "../utils.js";
|
||||
|
||||
/** Longhorn's own default; a rebuild of a large replica set can exceed it. */
|
||||
const REPLICA_REBUILD_TIMEOUT_MS = 60 * 60_000;
|
||||
const POLL_INTERVAL_MS = 15_000;
|
||||
/** How often to report "still waiting" when the replica count is not moving. */
|
||||
const HEARTBEAT_INTERVAL_MS = 3 * 60_000;
|
||||
|
||||
const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));
|
||||
|
||||
async function nodeName(ctx: OperationContext): Promise<string> {
|
||||
const r = await ctx.ssh.exec("hostname -f 2>/dev/null || hostname", sshOpts(ctx));
|
||||
return r.stdout.trim();
|
||||
}
|
||||
|
||||
/**
|
||||
* kubectl, from wherever it is reachable.
|
||||
*
|
||||
* An agent has no local kubectl, so the caller must be pointed at a server.
|
||||
* Returns null when there is no way to reach the API, so callers can report
|
||||
* "could not do this" instead of silently continuing.
|
||||
*/
|
||||
async function kubectl(
|
||||
ctx: OperationContext,
|
||||
args: string,
|
||||
/**
|
||||
* Override the SSH timeout for this call.
|
||||
*
|
||||
* sshOpts() is a flat 30 seconds, which is right for a status read and
|
||||
* completely wrong for anything that does work. A `kubectl drain` asking for
|
||||
* --timeout=15m gets killed by the transport at 30s and returns non-zero
|
||||
* having ALREADY evicted most of the pods -- so it looks like a drain
|
||||
* failure, with a transcript that reads like success, on every real node.
|
||||
*/
|
||||
timeoutMs?: number,
|
||||
): Promise<{ ok: boolean; out: string }> {
|
||||
const opts = timeoutMs === undefined ? sshOpts(ctx) : { timeoutMs };
|
||||
const local = await ctx.ssh.exec(`k3s kubectl ${args} 2>&1`, opts);
|
||||
if (local.exitCode === 0) return { ok: true, out: local.stdout };
|
||||
return { ok: false, out: local.stdout || local.stderr };
|
||||
}
|
||||
|
||||
/** Mark the node unschedulable. Safe and instantly reversible. */
|
||||
export const cordonNode: Operation = async (ctx): Promise<OperationResult> => {
|
||||
const name = await nodeName(ctx);
|
||||
const res = await kubectl(ctx, `cordon "${name}"`);
|
||||
if (!res.ok) {
|
||||
return { success: false, changed: false, message: `Could not cordon ${name}`, error: res.out };
|
||||
}
|
||||
const already = res.out.includes("already cordoned");
|
||||
return {
|
||||
success: true,
|
||||
changed: !already,
|
||||
message: already ? `${name} was already cordoned` : `Cordoned ${name}`,
|
||||
};
|
||||
};
|
||||
|
||||
export const uncordonNode: Operation = async (ctx): Promise<OperationResult> => {
|
||||
const name = await nodeName(ctx);
|
||||
const res = await kubectl(ctx, `uncordon "${name}"`);
|
||||
return res.ok
|
||||
? { success: true, changed: true, message: `Uncordoned ${name}` }
|
||||
: { success: false, changed: false, message: `Could not uncordon ${name}`, error: res.out };
|
||||
};
|
||||
|
||||
/**
|
||||
* Move workloads off the node.
|
||||
*
|
||||
* --ignore-daemonsets because a DaemonSet pod cannot be evicted by definition;
|
||||
* --delete-emptydir-data because refusing to delete emptyDir is what makes an
|
||||
* unattended drain hang forever, and a node about to be wiped is going to lose
|
||||
* that data regardless.
|
||||
*/
|
||||
export const drainNode: Operation = async (ctx): Promise<OperationResult> => {
|
||||
const name = await nodeName(ctx);
|
||||
// The SSH timeout must outlive kubectl's own --timeout, or the transport
|
||||
// kills a drain that is working and the result is indistinguishable from a
|
||||
// drain that failed.
|
||||
const DRAIN_TIMEOUT_MS = 15 * 60_000;
|
||||
const res = await kubectl(
|
||||
ctx,
|
||||
`drain "${name}" --ignore-daemonsets --delete-emptydir-data --force --timeout=15m`,
|
||||
DRAIN_TIMEOUT_MS + 60_000,
|
||||
);
|
||||
if (res.ok) return { success: true, changed: true, message: `Drained ${name}` };
|
||||
|
||||
// Surface the END of the transcript. The first line of a drain is always
|
||||
// "node/x already cordoned", which tells you nothing and hides the reason.
|
||||
const tail = res.out.split("\n").map((l) => l.trim()).filter(Boolean).slice(-6).join(" | ");
|
||||
return {
|
||||
success: false,
|
||||
changed: false,
|
||||
message: `Drain failed on ${name}`,
|
||||
error: tail === "" ? res.out : `...${tail}`,
|
||||
};
|
||||
};
|
||||
|
||||
/** kubectl -o json, parsed. Returns null when the read or the parse failed. */
|
||||
async function kubectlJson<T>(ctx: OperationContext, args: string): Promise<T | null> {
|
||||
const res = await kubectl(ctx, `${args} -o json`);
|
||||
if (!res.ok) return null;
|
||||
try {
|
||||
return JSON.parse(res.out) as T;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/** A Longhorn setting's value, or the supplied default when it is unreadable. */
|
||||
async function longhornSetting(ctx: OperationContext, name: string, fallback: string): Promise<string> {
|
||||
const res = await kubectl(ctx, `-n longhorn-system get settings.longhorn.io ${name} -o jsonpath='{.value}'`);
|
||||
const v = res.ok ? res.out.trim() : "";
|
||||
return v === "" ? fallback : v;
|
||||
}
|
||||
|
||||
interface LhNode {
|
||||
metadata: { name: string };
|
||||
spec: { allowScheduling?: boolean };
|
||||
status?: { diskStatus?: Record<string, {
|
||||
storageAvailable?: number; storageMaximum?: number;
|
||||
storageScheduled?: number; storageReserved?: number;
|
||||
}> };
|
||||
}
|
||||
interface LhReplica {
|
||||
metadata: { name: string };
|
||||
spec: { nodeID?: string; volumeName?: string };
|
||||
}
|
||||
interface LhVolume {
|
||||
metadata: { name: string };
|
||||
spec: { numberOfReplicas?: number; size?: number | string };
|
||||
status?: { state?: string; robustness?: string };
|
||||
}
|
||||
|
||||
export interface EvictionFeasibility {
|
||||
completable: boolean;
|
||||
/** Reasons eviction can never finish. Each names the offending volume. */
|
||||
blockers: string[];
|
||||
/** Things that will work but are dangerous. */
|
||||
warnings: string[];
|
||||
replicasMoving: number;
|
||||
bytesMoving: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Can eviction of this node actually COMPLETE?
|
||||
*
|
||||
* evictLonghornReplicas waits for the node's replica count to reach zero. If
|
||||
* there is nowhere for a replica to go, that wait can never end -- it burns the
|
||||
* full hour-long timeout and then fails, which is an expensive way to discover
|
||||
* a fact that is knowable in a couple of seconds.
|
||||
*
|
||||
* It is knowable because Longhorn's placement rules are simple:
|
||||
*
|
||||
* - With strict anti-affinity (replica-soft-anti-affinity=false, the default
|
||||
* and what this fleet runs) a volume may put at most ONE replica on a node.
|
||||
* So a volume wanting N replicas needs N schedulable nodes OTHER than the
|
||||
* one being evacuated. A 3-replica volume on a 3-node cluster is therefore
|
||||
* unevictable -- the two survivors already hold a copy each and cannot take
|
||||
* the third.
|
||||
* - The scheduler admits a replica against over-provisioned space, but the
|
||||
* rebuild writes real bytes to a real disk. Those are different numbers,
|
||||
* and at 500% over-provisioning the first can pass while the second fails.
|
||||
*
|
||||
* Read-only, a handful of API calls, finishes in seconds.
|
||||
*/
|
||||
export async function checkEvictionCompletable(
|
||||
ctx: OperationContext,
|
||||
name: string,
|
||||
): Promise<EvictionFeasibility> {
|
||||
const blockers: string[] = [];
|
||||
const warnings: string[] = [];
|
||||
|
||||
const nodes = await kubectlJson<{ items: LhNode[] }>(ctx, "-n longhorn-system get nodes.longhorn.io");
|
||||
const replicas = await kubectlJson<{ items: LhReplica[] }>(ctx, "-n longhorn-system get replicas.longhorn.io");
|
||||
const volumes = await kubectlJson<{ items: LhVolume[] }>(ctx, "-n longhorn-system get volumes.longhorn.io");
|
||||
if (!nodes || !replicas || !volumes) {
|
||||
return {
|
||||
completable: false,
|
||||
blockers: ["Could not read Longhorn nodes/replicas/volumes — refusing to guess"],
|
||||
warnings, replicasMoving: 0, bytesMoving: 0,
|
||||
};
|
||||
}
|
||||
|
||||
const softAntiAffinity = (await longhornSetting(ctx, "replica-soft-anti-affinity", "false")) === "true";
|
||||
const overProvisionPct = Number(await longhornSetting(ctx, "storage-over-provisioning-percentage", "100")) || 100;
|
||||
|
||||
const mine = replicas.items.filter((r) => r.spec.nodeID === name);
|
||||
if (mine.length === 0) {
|
||||
return { completable: true, blockers, warnings, replicasMoving: 0, bytesMoving: 0 };
|
||||
}
|
||||
|
||||
const volumeByName = new Map(volumes.items.map((v) => [v.metadata.name, v]));
|
||||
// Candidates are schedulable Longhorn nodes that are not the one leaving.
|
||||
const candidates = nodes.items.filter(
|
||||
(n) => n.metadata.name !== name && n.spec.allowScheduling !== false,
|
||||
);
|
||||
|
||||
const diskFree = (n: LhNode): number =>
|
||||
Object.values(n.status?.diskStatus ?? {}).reduce((a, d) => a + (d.storageAvailable ?? 0), 0);
|
||||
const schedulableFree = (n: LhNode): number =>
|
||||
Object.values(n.status?.diskStatus ?? {}).reduce((a, d) => {
|
||||
const usable = ((d.storageMaximum ?? 0) - (d.storageReserved ?? 0)) * (overProvisionPct / 100);
|
||||
return a + Math.max(0, usable - (d.storageScheduled ?? 0));
|
||||
}, 0);
|
||||
|
||||
let bytesMoving = 0;
|
||||
|
||||
for (const rep of mine) {
|
||||
const volName = rep.spec.volumeName ?? "";
|
||||
const vol = volumeByName.get(volName);
|
||||
if (!vol) {
|
||||
warnings.push(`replica ${rep.metadata.name} has no volume ${volName} — orphan?`);
|
||||
continue;
|
||||
}
|
||||
const want = vol.spec.numberOfReplicas ?? 1;
|
||||
const size = Number(vol.spec.size ?? 0);
|
||||
bytesMoving += size;
|
||||
|
||||
// (1) Placement. Strict anti-affinity means one replica per node per volume.
|
||||
if (!softAntiAffinity && candidates.length < want) {
|
||||
blockers.push(
|
||||
`volume ${volName} wants ${want} replicas but only ${candidates.length} other schedulable ` +
|
||||
`node(s) exist — with strict anti-affinity its replica cannot be placed. Add a schedulable node.`,
|
||||
);
|
||||
continue;
|
||||
}
|
||||
|
||||
// (2) Scheduler capacity, at the configured over-provisioning percentage.
|
||||
const schedulableFit = candidates.filter((n) => schedulableFree(n) >= size);
|
||||
if (schedulableFit.length === 0) {
|
||||
blockers.push(
|
||||
`volume ${volName} (${gib(size)}) does not fit the scheduler on any candidate node ` +
|
||||
`even at ${overProvisionPct}% over-provisioning.`,
|
||||
);
|
||||
continue;
|
||||
}
|
||||
|
||||
// (3) Real capacity. The scheduler counts over-provisioned space; a rebuild
|
||||
// writes real bytes. This is the check that catches a small disk hiding
|
||||
// behind a generous over-provisioning percentage.
|
||||
const realFit = candidates.filter((n) => diskFree(n) >= size);
|
||||
if (realFit.length === 0) {
|
||||
const best = Math.max(0, ...candidates.map(diskFree));
|
||||
blockers.push(
|
||||
`volume ${volName} needs ${gib(size)} but the roomiest candidate node has ${gib(best)} ` +
|
||||
`actually free. storage-over-provisioning-percentage=${overProvisionPct} lets the scheduler ` +
|
||||
`accept this; the disk does not.`,
|
||||
);
|
||||
continue;
|
||||
}
|
||||
|
||||
// (4) Volumes with no second copy, and volumes that cannot rebuild at all.
|
||||
if (want === 1) {
|
||||
warnings.push(
|
||||
`volume ${volName} has a single replica — evicting it rebuilds from the only copy that exists`,
|
||||
);
|
||||
}
|
||||
if (vol.status?.state === "detached") {
|
||||
warnings.push(`volume ${volName} is detached — Longhorn must attach it to rebuild`);
|
||||
}
|
||||
if (vol.status?.robustness === "degraded") {
|
||||
warnings.push(`volume ${volName} is already degraded before eviction starts`);
|
||||
}
|
||||
}
|
||||
|
||||
return { completable: blockers.length === 0, blockers, warnings, replicasMoving: mine.length, bytesMoving };
|
||||
}
|
||||
|
||||
function gib(bytes: number): string {
|
||||
return `${(bytes / 2 ** 30).toFixed(1)} GiB`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Standalone gate: refuse fast and loudly rather than hang for an hour.
|
||||
*
|
||||
* Exposed as its own Operation so it can be run on its own -- "could I evacuate
|
||||
* this node if I wanted to?" is a question worth answering without starting.
|
||||
*/
|
||||
export const assertEvictionCompletable: Operation = async (ctx): Promise<OperationResult> => {
|
||||
const name = await nodeName(ctx);
|
||||
const lh = await kubectl(ctx, "get ns longhorn-system -o name");
|
||||
if (!lh.ok) {
|
||||
return { success: true, changed: false, message: "Longhorn not installed — nothing to evict" };
|
||||
}
|
||||
|
||||
const f = await checkEvictionCompletable(ctx, name);
|
||||
for (const w of f.warnings) ctx.log(` WARNING: ${w}`);
|
||||
|
||||
if (!f.completable) {
|
||||
return {
|
||||
success: false,
|
||||
changed: false,
|
||||
message: `Eviction of ${name} cannot complete`,
|
||||
error: f.blockers.join("; "),
|
||||
};
|
||||
}
|
||||
return {
|
||||
success: true,
|
||||
changed: false,
|
||||
message: f.replicasMoving === 0
|
||||
? `${name} holds no replicas`
|
||||
: `${f.replicasMoving} replica(s) (${gib(f.bytesMoving)}) can be placed elsewhere`,
|
||||
};
|
||||
};
|
||||
|
||||
/**
|
||||
* Ask Longhorn to move this node's replicas elsewhere, then WAIT.
|
||||
*
|
||||
* The wait is the entire point. Setting evictionRequested and immediately
|
||||
* wiping the disk destroys the replicas mid-rebuild, which on a 3-replica
|
||||
* volume takes you to 1 and on the 2-replica Prometheus volume takes you to 0.
|
||||
*
|
||||
* Returns changed:false with an explanation when Longhorn is not installed,
|
||||
* so a standalone or storage-free node is a no-op rather than an error.
|
||||
*/
|
||||
export const evictLonghornReplicas: Operation = async (ctx): Promise<OperationResult> => {
|
||||
const name = await nodeName(ctx);
|
||||
|
||||
const hasLonghorn = await kubectl(ctx, "get ns longhorn-system -o name");
|
||||
if (!hasLonghorn.ok) {
|
||||
return { success: true, changed: false, message: "Longhorn not installed — nothing to evict" };
|
||||
}
|
||||
|
||||
// Refuse in seconds rather than discover it after an hour of polling.
|
||||
// Inside the operation, not only in the group, so `labctl app k3s evacuate`
|
||||
// gets the same protection as the drill does.
|
||||
const feasible = await checkEvictionCompletable(ctx, name);
|
||||
for (const w of feasible.warnings) ctx.log(` WARNING: ${w}`);
|
||||
if (!feasible.completable) {
|
||||
return {
|
||||
success: false,
|
||||
changed: false,
|
||||
message: `Eviction of ${name} cannot complete — not starting it`,
|
||||
error: feasible.blockers.join("; "),
|
||||
};
|
||||
}
|
||||
// A single-replica volume rebuilds from the only copy in existence, and this
|
||||
// fleet has no backup target. That is a decision for a human, not a default.
|
||||
const singles = feasible.warnings.filter((w) => w.includes("single replica"));
|
||||
if (singles.length > 0 && ctx.config.evictionAllowSingleReplica !== true) {
|
||||
return {
|
||||
success: false,
|
||||
changed: false,
|
||||
message: `${name} holds single-replica volume(s); refusing to evict`,
|
||||
error: `${singles.join("; ")}. Raise them to 3 replicas and wait for healthy, ` +
|
||||
`or pass evictionAllowSingleReplica to accept the risk.`,
|
||||
};
|
||||
}
|
||||
|
||||
const patch = JSON.stringify({
|
||||
spec: { allowScheduling: false, evictionRequested: true },
|
||||
});
|
||||
const req = await kubectl(
|
||||
ctx,
|
||||
`-n longhorn-system patch nodes.longhorn.io "${name}" --type=merge -p '${patch}'`,
|
||||
);
|
||||
if (!req.ok) {
|
||||
return { success: false, changed: false, message: `Could not request eviction for ${name}`, error: req.out };
|
||||
}
|
||||
|
||||
ctx.log(`Eviction requested for ${name}; waiting for replicas to rebuild elsewhere...`);
|
||||
const start = Date.now();
|
||||
let last = -1;
|
||||
let lastHeartbeat = Date.now();
|
||||
try {
|
||||
while (Date.now() - start < REPLICA_REBUILD_TIMEOUT_MS) {
|
||||
const res = await kubectl(
|
||||
ctx,
|
||||
`-n longhorn-system get replicas.longhorn.io -o jsonpath='{range .items[?(@.spec.nodeID=="${name}")]}{.metadata.name}{"\\n"}{end}'`,
|
||||
);
|
||||
const remaining = res.ok ? res.out.split("\n").map((l) => l.trim()).filter(Boolean).length : -1;
|
||||
if (remaining === 0) {
|
||||
return {
|
||||
success: true,
|
||||
changed: true,
|
||||
message: `All replicas evicted from ${name} after ${Math.round((Date.now() - start) / 1000)}s`,
|
||||
};
|
||||
}
|
||||
// Log on change, and otherwise every few minutes. A silent hour is
|
||||
// indistinguishable from a hung process; a heartbeat that carries the
|
||||
// elapsed time makes "stalled" visible while it is happening rather than
|
||||
// only in the timeout message.
|
||||
const elapsedS = Math.round((Date.now() - start) / 1000);
|
||||
if (remaining !== last) {
|
||||
ctx.log(` ${remaining} replica(s) still on ${name} (${elapsedS}s elapsed)`);
|
||||
last = remaining;
|
||||
lastHeartbeat = Date.now();
|
||||
} else if (Date.now() - lastHeartbeat >= HEARTBEAT_INTERVAL_MS) {
|
||||
ctx.log(` still waiting: ${remaining} replica(s) on ${name}, no change in ` +
|
||||
`${Math.round((Date.now() - lastHeartbeat) / 60_000)} min (${elapsedS}s elapsed)`);
|
||||
lastHeartbeat = Date.now();
|
||||
}
|
||||
await sleep(POLL_INTERVAL_MS);
|
||||
}
|
||||
} catch (err) {
|
||||
// Never leave the node eviction-pending because of a transient failure.
|
||||
await cancelEviction(ctx, name);
|
||||
throw err;
|
||||
}
|
||||
|
||||
// Timed out. Say exactly what is stuck before restoring the node, because
|
||||
// after the cancel the eviction flags are gone and the evidence with them.
|
||||
const stuck = await describeStuckReplicas(ctx, name);
|
||||
|
||||
// Put the node back the way we found it: an abandoned eviction leaves it
|
||||
// unschedulable with Longhorn still trying to drain it, which is safe but
|
||||
// sticky, and nothing else would ever clear it.
|
||||
const cancelled = await cancelEviction(ctx, name);
|
||||
return {
|
||||
success: false,
|
||||
changed: true,
|
||||
message: `Timed out waiting for replica eviction from ${name}`,
|
||||
error: `${last} replica(s) still present after ${REPLICA_REBUILD_TIMEOUT_MS / 60000} min. ` +
|
||||
`Wiping now would destroy them.\nStill on ${name}:\n${stuck}\n` +
|
||||
(cancelled
|
||||
? `Eviction request has been cancelled and scheduling restored.`
|
||||
: `WARNING: could not cancel the eviction request — clear it with ` +
|
||||
`\`labctl app k3s admit ${name}\`.`),
|
||||
};
|
||||
};
|
||||
|
||||
/**
|
||||
* Describe the replicas still sitting on a node, and why they are not moving.
|
||||
*
|
||||
* An eviction that times out after an hour used to report only a count. That
|
||||
* is not enough to act on: the question is always WHICH volume is stuck and
|
||||
* what state its replica is in. A replica with `desireState=running` and
|
||||
* `currentState=stopped` and no `failedAt` is Longhorn wanting to start it and
|
||||
* not doing so -- a very different problem from a volume that cannot be placed.
|
||||
*/
|
||||
async function describeStuckReplicas(ctx: OperationContext, name: string): Promise<string> {
|
||||
const reps = await kubectlJson<{ items: {
|
||||
metadata: { name: string };
|
||||
spec: { nodeID?: string; volumeName?: string; failedAt?: string; desireState?: string };
|
||||
status?: { currentState?: string };
|
||||
}[] }>(ctx, "-n longhorn-system get replicas.longhorn.io");
|
||||
if (!reps) return " (could not read replicas)";
|
||||
|
||||
const mine = reps.items.filter((r) => r.spec.nodeID === name);
|
||||
if (mine.length === 0) return " (none)";
|
||||
|
||||
const vols = await kubectlJson<{ items: {
|
||||
metadata: { name: string };
|
||||
spec: { numberOfReplicas?: number };
|
||||
status?: { state?: string; robustness?: string };
|
||||
}[] }>(ctx, "-n longhorn-system get volumes.longhorn.io");
|
||||
const byName = new Map((vols?.items ?? []).map((v) => [v.metadata.name, v]));
|
||||
|
||||
return mine.map((r) => {
|
||||
const v = byName.get(r.spec.volumeName ?? "");
|
||||
return ` ${r.spec.volumeName ?? "?"}: replica ${r.metadata.name.slice(-12)} ` +
|
||||
`currentState=${r.status?.currentState ?? "?"} desireState=${r.spec.desireState ?? "?"} ` +
|
||||
`failedAt=${r.spec.failedAt === undefined || r.spec.failedAt === "" ? "(none)" : r.spec.failedAt} | ` +
|
||||
`volume ${v?.status?.state ?? "?"}/${v?.status?.robustness ?? "?"} ` +
|
||||
`wants ${v?.spec.numberOfReplicas ?? "?"} replicas`;
|
||||
}).join("\n");
|
||||
}
|
||||
|
||||
/**
|
||||
* Undo the eviction request. Returns false if it could not be undone, which the
|
||||
* caller must surface -- silently failing here leaves a node nobody can use.
|
||||
*/
|
||||
async function cancelEviction(ctx: OperationContext, name: string): Promise<boolean> {
|
||||
const patch = JSON.stringify({ spec: { allowScheduling: true, evictionRequested: false } });
|
||||
const res = await kubectl(
|
||||
ctx,
|
||||
`-n longhorn-system patch nodes.longhorn.io "${name}" --type=merge -p '${patch}'`,
|
||||
);
|
||||
if (res.ok) ctx.log(`Cancelled eviction request on ${name} and restored scheduling`);
|
||||
return res.ok;
|
||||
}
|
||||
|
||||
/**
|
||||
* Verify the node holds nothing before it is wiped.
|
||||
*
|
||||
* A separate, cheap operation on purpose: it is the gate that turns "we ran the
|
||||
* eviction steps" into "the node is actually empty", and it is the one thing
|
||||
* worth asserting twice. Reading back from the thing that does the work, not
|
||||
* from the request we sent it.
|
||||
*/
|
||||
export const assertNodeEvacuated: Operation = async (ctx): Promise<OperationResult> => {
|
||||
const name = await nodeName(ctx);
|
||||
const problems: string[] = [];
|
||||
|
||||
const sched = await kubectl(ctx, `get node "${name}" -o jsonpath='{.spec.unschedulable}'`);
|
||||
if (sched.ok && sched.out.trim() !== "true") problems.push("node is still schedulable");
|
||||
|
||||
const pods = await kubectl(
|
||||
ctx,
|
||||
`get pods -A --field-selector spec.nodeName="${name}",status.phase=Running ` +
|
||||
`-o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}{"\\n"}{end}'`,
|
||||
);
|
||||
if (pods.ok) {
|
||||
const running = pods.out.split("\n").map((l) => l.trim()).filter(Boolean);
|
||||
// DaemonSet pods legitimately remain; anything else means the drain lied.
|
||||
if (running.length > 0) ctx.log(` ${running.length} pod(s) remain (expected: DaemonSets only)`);
|
||||
}
|
||||
|
||||
const lh = await kubectl(ctx, "get ns longhorn-system -o name");
|
||||
if (lh.ok) {
|
||||
const reps = await kubectl(
|
||||
ctx,
|
||||
`-n longhorn-system get replicas.longhorn.io -o jsonpath='{range .items[?(@.spec.nodeID=="${name}")]}{.metadata.name}{"\\n"}{end}'`,
|
||||
);
|
||||
const n = reps.ok ? reps.out.split("\n").map((l) => l.trim()).filter(Boolean).length : 0;
|
||||
if (n > 0) problems.push(`${n} Longhorn replica(s) still on this node`);
|
||||
}
|
||||
|
||||
return problems.length === 0
|
||||
? { success: true, changed: false, message: `${name} is evacuated and safe to wipe` }
|
||||
: {
|
||||
success: false,
|
||||
changed: false,
|
||||
message: `${name} is NOT safe to wipe`,
|
||||
error: problems.join("; "),
|
||||
};
|
||||
};
|
||||
@@ -0,0 +1,66 @@
|
||||
// Apply the worker role label from the SERVER side, after the node registers.
|
||||
//
|
||||
// WHY THIS IS NOT A NODE-LABEL
|
||||
// ----------------------------
|
||||
// `node-role.kubernetes.io/worker=true` cannot be set by the kubelet itself.
|
||||
// The servers enable the NodeRestriction admission plugin, which forbids a
|
||||
// kubelet from assigning itself any label in the `node-role.kubernetes.io/`
|
||||
// namespace. Putting it in the agent's `node-label:` list makes registration
|
||||
// fail, and the resulting message --
|
||||
//
|
||||
// Job for k3s-agent.service failed because the service did not take the
|
||||
// steps required by its unit configuration.
|
||||
//
|
||||
// names neither labels nor admission, so it reads as a networking, token or
|
||||
// sysctl problem. It is none of those.
|
||||
//
|
||||
// The label is cosmetic (it populates the ROLES column in `kubectl get nodes`)
|
||||
// so failing to apply it is a warning, never an error.
|
||||
|
||||
import type { Operation, OperationContext, OperationResult } from "../types.js";
|
||||
import { sshOpts } from "../utils.js";
|
||||
import { sshExec as remoteSshExec } from "../../../../src/ssh.js";
|
||||
import { isK3sServer } from "@lab/shared";
|
||||
|
||||
const WORKER_LABEL = "node-role.kubernetes.io/worker=true";
|
||||
|
||||
async function nodeName(ctx: OperationContext): Promise<string> {
|
||||
const r = await ctx.ssh.exec("hostname -f 2>/dev/null || hostname", sshOpts(ctx));
|
||||
return r.stdout.trim();
|
||||
}
|
||||
|
||||
export const labelWorkerRole: Operation = async (ctx): Promise<OperationResult> => {
|
||||
if (isK3sServer(ctx.config.role)) {
|
||||
return { success: true, changed: false, message: "Server node — k3s sets its own roles" };
|
||||
}
|
||||
|
||||
const name = await nodeName(ctx);
|
||||
|
||||
// An agent has no local kubectl, so this has to run on a server.
|
||||
const local = await ctx.ssh.exec(`k3s kubectl label node "${name}" ${WORKER_LABEL} --overwrite 2>&1`, sshOpts(ctx));
|
||||
if (local.exitCode === 0) {
|
||||
return { success: true, changed: true, message: `Labelled ${name} as worker` };
|
||||
}
|
||||
|
||||
if (ctx.config.k3sServerUrl !== undefined && ctx.config.k3sServerUrl !== "") {
|
||||
try {
|
||||
const serverHost = new URL(ctx.config.k3sServerUrl).hostname;
|
||||
const remote = await remoteSshExec(
|
||||
serverHost, "root",
|
||||
`k3s kubectl label node "${name}" ${WORKER_LABEL} --overwrite`,
|
||||
{ ...(ctx.ssh.keyPath ? { keyPath: ctx.ssh.keyPath } : {}), timeoutMs: 30_000 },
|
||||
);
|
||||
if (remote.exitCode === 0) {
|
||||
return { success: true, changed: true, message: `Labelled ${name} as worker (via server)` };
|
||||
}
|
||||
} catch { /* fall through to the warning */ }
|
||||
}
|
||||
|
||||
// Cosmetic: the ROLES column will read <none>. Not worth failing an install.
|
||||
return {
|
||||
success: true,
|
||||
changed: false,
|
||||
message: `Could not apply the worker role label to ${name} (cosmetic; set it with ` +
|
||||
`\`kubectl label node ${name} ${WORKER_LABEL}\`)`,
|
||||
};
|
||||
};
|
||||
@@ -17,6 +17,29 @@ export interface K3sConfig {
|
||||
// Additional TLS SANs for API server certificate
|
||||
tlsSans?: string[] | undefined;
|
||||
|
||||
/**
|
||||
* Exact k3s version to install, e.g. "v1.34.6+k3s1".
|
||||
*
|
||||
* Unset means whatever get.k3s.io currently calls latest, which is how a
|
||||
* reinstalled node silently ends up AHEAD of the cluster it is rejoining.
|
||||
* Kubernetes supports only one minor version of skew between control-plane
|
||||
* members; a node reinstalled two minors ahead of its peers is an
|
||||
* unsupported cluster, and a rolling migration reinstalls nodes one at a
|
||||
* time by design. Pin it to the version the cluster already runs.
|
||||
*/
|
||||
k3sVersion?: string | undefined;
|
||||
|
||||
/**
|
||||
* Permit evacuating a node that holds single-replica Longhorn volumes.
|
||||
*
|
||||
* Off by default and deliberately awkward. Evicting the only replica of a
|
||||
* volume rebuilds it from the sole copy in existence; with no backup target
|
||||
* configured, a failure during that rebuild is permanent data loss. Raising
|
||||
* the volume to 3 replicas and waiting for it to go healthy is almost always
|
||||
* the right move instead.
|
||||
*/
|
||||
evictionAllowSingleReplica?: boolean | undefined;
|
||||
|
||||
/**
|
||||
* IPv6 address of this node on the cluster VLAN. Its PRESENCE is what makes a
|
||||
* node dual-stack: supply it and `node-ip` is emitted as `<v4>,<v6>`; omit it
|
||||
|
||||
@@ -0,0 +1,75 @@
|
||||
// The agent must NOT try to self-assign a node-role label.
|
||||
//
|
||||
// A kubelet may not set labels in the `node-role.kubernetes.io/` namespace;
|
||||
// the servers run NodeRestriction, which rejects the registration. The agent
|
||||
// then never finishes registering and systemd reports only:
|
||||
//
|
||||
// Job for k3s-agent.service failed because the service did not take the
|
||||
// steps required by its unit configuration.
|
||||
//
|
||||
// That message names neither labels nor admission control, so the cost of
|
||||
// reintroducing this is hours of looking at the network and the join token.
|
||||
// Observed twice for real: on v2-wrk-1, and on the drill's storage agent.
|
||||
|
||||
import { describe, it, expect } from "vitest";
|
||||
import { generateAgentConfig, generateServerConfig } from "../src/operations/k3s-config.js";
|
||||
import { k3sAgentGroup } from "../src/groups/k3s-agent.js";
|
||||
import { mockCtx, OK, stdout, expectCommand } from "./helpers.js";
|
||||
import { labelWorkerRole } from "../src/operations/worker-label.js";
|
||||
|
||||
const base = { hostname: "w1.lab", ip: "10.0.0.9", role: "k8s-worker" as const };
|
||||
|
||||
describe("agent config", () => {
|
||||
it("does NOT put a node-role label in node-label", () => {
|
||||
const cfg = generateAgentConfig({ ...base });
|
||||
expect(cfg).not.toContain("node-role.kubernetes.io");
|
||||
});
|
||||
|
||||
it("keeps the Longhorn label, which IS allowed (outside kubernetes.io)", () => {
|
||||
expect(generateAgentConfig({ ...base })).toContain("node.longhorn.io/create-default-disk=config");
|
||||
});
|
||||
|
||||
it("server config does not set node-role either — k3s assigns those itself", () => {
|
||||
const cfg = generateServerConfig({ ...base, role: "k8s-combo", hostname: "s1.lab" });
|
||||
expect(cfg).not.toContain("node-role.kubernetes.io");
|
||||
});
|
||||
|
||||
it("the agent group applies the worker label AFTER installing", () => {
|
||||
const names = k3sAgentGroup.operations.map((o) => o.name);
|
||||
const install = names.findIndex((n) => /Install k3s agent/i.test(n));
|
||||
const label = names.findIndex((n) => /Label node as worker/i.test(n));
|
||||
expect(install).toBeGreaterThanOrEqual(0);
|
||||
expect(label).toBeGreaterThan(install);
|
||||
});
|
||||
});
|
||||
|
||||
describe("labelWorkerRole", () => {
|
||||
it("labels an agent from the server side", async () => {
|
||||
const ctx = mockCtx({ role: "k8s-worker" });
|
||||
ctx.ssh.exec.mockImplementation(async (cmd: string) => {
|
||||
if (cmd.startsWith("hostname")) return stdout("w1.lab\n");
|
||||
return OK;
|
||||
});
|
||||
const r = await labelWorkerRole(ctx);
|
||||
expect(r.success).toBe(true);
|
||||
expectCommand(ctx.ssh, /label node "w1\.lab" node-role\.kubernetes\.io\/worker=true/);
|
||||
});
|
||||
|
||||
it("is a no-op on a server — k3s sets control-plane/etcd itself", async () => {
|
||||
const r = await labelWorkerRole(mockCtx({ role: "k8s-combo" }));
|
||||
expect(r.success).toBe(true);
|
||||
expect(r.changed).toBe(false);
|
||||
});
|
||||
|
||||
it("never fails an install over a cosmetic label", async () => {
|
||||
const ctx = mockCtx({ role: "k8s-worker" });
|
||||
ctx.ssh.exec.mockImplementation(async (cmd: string) => {
|
||||
if (cmd.startsWith("hostname")) return stdout("w1.lab\n");
|
||||
return { exitCode: 1, stdout: "", stderr: "no kubectl here" };
|
||||
});
|
||||
const r = await labelWorkerRole(ctx);
|
||||
expect(r.success, "a missing ROLES column must not fail a node install").toBe(true);
|
||||
expect(r.changed).toBe(false);
|
||||
expect(r.message).toMatch(/cosmetic/);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,58 @@
|
||||
// Parsing `etcdctl member list`.
|
||||
//
|
||||
// WHY NOT -w json
|
||||
// ---------------
|
||||
// etcd member IDs are uint64. JSON.parse turns them into JS numbers, which
|
||||
// lose precision above 2^53, so the id that comes back is an APPROXIMATION of
|
||||
// the real one -- and `member remove` wants HEX anyway, while the JSON writer
|
||||
// emits decimal. Observed for real: `member remove 13323819137391910000`
|
||||
// -> "bad member ID arg (value out of range), expecting ID in Hex".
|
||||
//
|
||||
// That happens at the worst possible moment: the node is already stopped and
|
||||
// about to be wiped. A mangled id fails loudly if you are lucky; in principle
|
||||
// it could match a different member, which would remove the wrong one.
|
||||
//
|
||||
// The default writer prints the id as hex TEXT, so it never becomes a number.
|
||||
// The fixture below is real output from a production node.
|
||||
|
||||
import { describe, it, expect } from "vitest";
|
||||
import { parseEtcdMembers } from "../src/operations/etcd-members.js";
|
||||
|
||||
const REAL = `825852d3683bc406, started, worker2-k8s0.ad.itaz.eu-5f7ad0c9, https://192.168.8.25:2380, https://192.168.8.25:2379, false
|
||||
a6170c0755307cea, started, worker1-k8s0.ad.itaz.eu-82b4fe24, https://192.168.8.13:2380, https://192.168.8.13:2379, false
|
||||
ad1fe97d38022a78, started, worker0-k8s0.ad.itaz.eu-e26aacc5, https://192.168.8.23:2380, https://192.168.8.23:2379, false`;
|
||||
|
||||
describe("parseEtcdMembers", () => {
|
||||
it("keeps ids as exact hex text, never as numbers", () => {
|
||||
const m = parseEtcdMembers(REAL);
|
||||
expect(m.map((x) => x.id)).toEqual([
|
||||
"825852d3683bc406", "a6170c0755307cea", "ad1fe97d38022a78",
|
||||
]);
|
||||
// The precision trap, stated as an assertion: every one of these exceeds
|
||||
// Number.MAX_SAFE_INTEGER, so a numeric round-trip would corrupt them.
|
||||
for (const x of m) {
|
||||
expect(Number(BigInt(`0x${x.id}`)) > Number.MAX_SAFE_INTEGER).toBe(true);
|
||||
expect(BigInt(`0x${x.id}`).toString(16)).toBe(x.id);
|
||||
}
|
||||
});
|
||||
|
||||
it("strips etcd's random name suffix to give the node name", () => {
|
||||
expect(parseEtcdMembers(REAL).map((x) => x.node)).toEqual([
|
||||
"worker2-k8s0.ad.itaz.eu", "worker1-k8s0.ad.itaz.eu", "worker0-k8s0.ad.itaz.eu",
|
||||
]);
|
||||
});
|
||||
|
||||
it("reports started status", () => {
|
||||
expect(parseEtcdMembers(REAL).every((x) => x.started)).toBe(true);
|
||||
const unstarted = parseEtcdMembers("abc123, unstarted, , https://x:2380, , false");
|
||||
expect(unstarted[0]?.started).toBe(false);
|
||||
});
|
||||
|
||||
it("ignores blank lines and anything without a hex id", () => {
|
||||
expect(parseEtcdMembers(`\n \nWARNING: something\n${REAL}\n`)).toHaveLength(3);
|
||||
});
|
||||
|
||||
it("returns nothing for empty output rather than a bogus member", () => {
|
||||
expect(parseEtcdMembers("")).toEqual([]);
|
||||
});
|
||||
});
|
||||
158
bastion/src/modules/modules/k3s/tests/join-env.test.ts
Normal file
158
bastion/src/modules/modules/k3s/tests/join-env.test.ts
Normal file
@@ -0,0 +1,158 @@
|
||||
// Tests for how K3S_URL / K3S_TOKEN reach the k3s installer.
|
||||
//
|
||||
// Token and URL are independent, and the distinction is load-bearing:
|
||||
//
|
||||
// token only -> SEEDS the cluster token on the cluster-init node, so
|
||||
// every later joiner can use the same pre-shared value.
|
||||
// url + token -> JOINS an existing cluster.
|
||||
// url only -> cannot authenticate; must not be passed.
|
||||
//
|
||||
// Before this was fixed the installer emitted join env ONLY when both were
|
||||
// present, so a cluster-init node always self-generated its token and there
|
||||
// was no way to pre-share one. A `toContain("K3S_TOKEN")` assertion would pass
|
||||
// against the broken version for the url+token case, so each case is asserted
|
||||
// on the exact command.
|
||||
|
||||
import { describe, it, expect } from "vitest";
|
||||
import { mockCtx, stdout, OK } from "./helpers.js";
|
||||
import { installK3sBinary } from "../src/operations/k3s-install.js";
|
||||
|
||||
/** The `curl ... | sh -` line the operation ran. */
|
||||
function installCommand(ssh: ReturnType<typeof mockCtx>["ssh"]): string {
|
||||
const calls = ssh.exec.mock.calls.map((c: [string, unknown?]) => c[0]);
|
||||
const found = calls.find((c: string) => c.includes("get.k3s.io"));
|
||||
if (!found) throw new Error(`No install command issued. Commands:\n${calls.join("\n")}`);
|
||||
return found;
|
||||
}
|
||||
|
||||
/** k3s --version fails (not installed), everything after succeeds. */
|
||||
function primeServerInstall(ctx: ReturnType<typeof mockCtx>): void {
|
||||
ctx.ssh.exec.mockResolvedValue(OK);
|
||||
ctx.ssh.exec.mockResolvedValueOnce({ exitCode: 1, stdout: "", stderr: "" }); // k3s --version
|
||||
}
|
||||
|
||||
describe("installK3sBinary — server join environment", () => {
|
||||
it("token alone seeds the cluster token and does NOT pass a server URL", async () => {
|
||||
const ctx = mockCtx({ role: "k8s-combo", k3sToken: "pre-shared-token" });
|
||||
primeServerInstall(ctx);
|
||||
ctx.ssh.exec.mockResolvedValue(stdout("NAME STATUS\nnode Ready"));
|
||||
|
||||
await installK3sBinary(ctx);
|
||||
const cmd = installCommand(ctx.ssh);
|
||||
|
||||
expect(cmd).toContain('K3S_TOKEN="pre-shared-token"');
|
||||
expect(cmd).not.toContain("K3S_URL");
|
||||
expect(cmd).toContain('INSTALL_K3S_EXEC="server"');
|
||||
});
|
||||
|
||||
it("url + token joins an existing cluster", async () => {
|
||||
const ctx = mockCtx({
|
||||
role: "k8s-combo",
|
||||
k3sServerUrl: "https://10.0.0.11:6443",
|
||||
k3sToken: "pre-shared-token",
|
||||
});
|
||||
primeServerInstall(ctx);
|
||||
ctx.ssh.exec.mockResolvedValue(stdout("NAME STATUS\nnode Ready"));
|
||||
|
||||
await installK3sBinary(ctx);
|
||||
const cmd = installCommand(ctx.ssh);
|
||||
|
||||
expect(cmd).toContain('K3S_URL="https://10.0.0.11:6443"');
|
||||
expect(cmd).toContain('K3S_TOKEN="pre-shared-token"');
|
||||
});
|
||||
|
||||
it("neither url nor token passes no join environment at all", async () => {
|
||||
const ctx = mockCtx({ role: "k8s-combo" });
|
||||
primeServerInstall(ctx);
|
||||
ctx.ssh.exec.mockResolvedValue(stdout("NAME STATUS\nnode Ready"));
|
||||
|
||||
await installK3sBinary(ctx);
|
||||
const cmd = installCommand(ctx.ssh);
|
||||
|
||||
expect(cmd).not.toContain("K3S_URL");
|
||||
expect(cmd).not.toContain("K3S_TOKEN");
|
||||
});
|
||||
|
||||
it("a URL without a token is ignored — it cannot authenticate a join", async () => {
|
||||
const ctx = mockCtx({ role: "k8s-combo", k3sServerUrl: "https://10.0.0.11:6443" });
|
||||
primeServerInstall(ctx);
|
||||
ctx.ssh.exec.mockResolvedValue(stdout("NAME STATUS\nnode Ready"));
|
||||
|
||||
await installK3sBinary(ctx);
|
||||
expect(installCommand(ctx.ssh)).not.toContain("K3S_URL");
|
||||
});
|
||||
});
|
||||
|
||||
describe("installK3sBinary — datastore wipe guard", () => {
|
||||
// Wiping server/{tls,cred,db} is correct when JOINING (stale certs are fatal:
|
||||
// "newer than datastore") and catastrophic on a cluster-init node, where it
|
||||
// would destroy the etcd database the node is about to bootstrap from.
|
||||
const WIPE = "rm -rf /var/lib/rancher/k3s/server/tls";
|
||||
|
||||
it("wipes stale server state when joining", async () => {
|
||||
const ctx = mockCtx({
|
||||
role: "k8s-combo",
|
||||
k3sServerUrl: "https://10.0.0.11:6443",
|
||||
k3sToken: "t",
|
||||
});
|
||||
primeServerInstall(ctx);
|
||||
ctx.ssh.exec.mockResolvedValue(stdout("NAME STATUS\nnode Ready"));
|
||||
|
||||
await installK3sBinary(ctx);
|
||||
const calls = ctx.ssh.exec.mock.calls.map((c: [string, unknown?]) => c[0]);
|
||||
expect(calls.some((c: string) => c.includes(WIPE))).toBe(true);
|
||||
});
|
||||
|
||||
it("does NOT wipe the datastore when only seeding a token (cluster-init)", async () => {
|
||||
const ctx = mockCtx({ role: "k8s-combo", k3sToken: "pre-shared-token" });
|
||||
primeServerInstall(ctx);
|
||||
ctx.ssh.exec.mockResolvedValue(stdout("NAME STATUS\nnode Ready"));
|
||||
|
||||
await installK3sBinary(ctx);
|
||||
const calls = ctx.ssh.exec.mock.calls.map((c: [string, unknown?]) => c[0]);
|
||||
expect(calls.some((c: string) => c.includes(WIPE))).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe("installK3sBinary — agent", () => {
|
||||
it("refuses to install an agent with no server URL or token", async () => {
|
||||
const ctx = mockCtx({ role: "k8s-worker" });
|
||||
primeServerInstall(ctx);
|
||||
|
||||
const result = await installK3sBinary(ctx);
|
||||
expect(result.success).toBe(false);
|
||||
expect(result.message).toContain("Agent requires");
|
||||
});
|
||||
});
|
||||
|
||||
describe("installK3sBinary — version pinning", () => {
|
||||
// get.k3s.io installs LATEST. During a rolling migration that means a
|
||||
// reinstalled node rejoins on a newer k3s than its peers; Kubernetes supports
|
||||
// only one minor of skew between control-plane members, and at the time of
|
||||
// writing latest was TWO ahead of the fleet (v1.36.4 vs v1.34.6).
|
||||
it("pins the version when one is given", async () => {
|
||||
const ctx = mockCtx({ role: "k8s-combo", k3sToken: "t", k3sVersion: "v1.34.6+k3s1" });
|
||||
primeServerInstall(ctx);
|
||||
ctx.ssh.exec.mockResolvedValue(stdout("NAME STATUS\nnode Ready"));
|
||||
await installK3sBinary(ctx);
|
||||
expect(installCommand(ctx.ssh)).toContain('INSTALL_K3S_VERSION="v1.34.6+k3s1"');
|
||||
});
|
||||
|
||||
it("pins the agent install too", async () => {
|
||||
const ctx = mockCtx({
|
||||
role: "k8s-worker", k3sServerUrl: "https://10.0.0.11:6443",
|
||||
k3sToken: "t", k3sVersion: "v1.34.6+k3s1",
|
||||
});
|
||||
primeServerInstall(ctx);
|
||||
await installK3sBinary(ctx);
|
||||
expect(installCommand(ctx.ssh)).toContain('INSTALL_K3S_VERSION="v1.34.6+k3s1"');
|
||||
});
|
||||
|
||||
it("omits the variable entirely when unset, preserving today's behaviour", async () => {
|
||||
const ctx = mockCtx({ role: "k8s-combo", k3sToken: "t" });
|
||||
primeServerInstall(ctx);
|
||||
ctx.ssh.exec.mockResolvedValue(stdout("NAME STATUS\nnode Ready"));
|
||||
await installK3sBinary(ctx);
|
||||
expect(installCommand(ctx.ssh)).not.toContain("INSTALL_K3S_VERSION");
|
||||
});
|
||||
});
|
||||
188
bastion/src/modules/modules/k3s/tests/longhorn-disk.test.ts
Normal file
188
bastion/src/modules/modules/k3s/tests/longhorn-disk.test.ts
Normal file
@@ -0,0 +1,188 @@
|
||||
// Tests for giving a node its Longhorn disk.
|
||||
//
|
||||
// The cluster runs createDefaultDiskLabeledNodes:true, so a node gets a disk
|
||||
// only if it carries BOTH the create-default-disk label and the
|
||||
// default-disks-config annotation. Longhorn reads that annotation when it
|
||||
// first creates the node's record — which means on a node that is REJOINING,
|
||||
// the record already exists and the annotation is never read. The node comes
|
||||
// back Ready, accepts pods, and hosts no replicas at all.
|
||||
//
|
||||
// That failure is invisible until a rebuild is expected and does not arrive,
|
||||
// so these tests pin the read-back and the repair, not just the annotate.
|
||||
|
||||
import { describe, it, expect, vi, afterEach } from "vitest";
|
||||
import { mockCtx, OK, FAIL, stdout, expectCommand, expectNoCommand } from "./helpers.js";
|
||||
import { configureLonghornDisk } from "../src/operations/longhorn-disk.js";
|
||||
|
||||
const HOST = stdout("node1.lab\n");
|
||||
const DISK = "/var/lib/longhorn";
|
||||
|
||||
interface Opts {
|
||||
/** Does Longhorn create the disk on its own (i.e. it read the annotation)? */
|
||||
longhornCreatesDisk?: boolean;
|
||||
/** Has longhorn-manager registered this node yet? */
|
||||
recordExists?: boolean;
|
||||
hasDir?: boolean;
|
||||
noLonghorn?: boolean;
|
||||
patchFails?: boolean;
|
||||
/** Does the disk show up after a direct spec.disks patch? */
|
||||
repairWorks?: boolean;
|
||||
}
|
||||
|
||||
function ctxFor(o: Opts = {}) {
|
||||
// Model the real object rather than a script of reads: a disk exists if
|
||||
// Longhorn made one, or if our repair patch landed and took effect.
|
||||
let patched = false;
|
||||
const ctx = mockCtx();
|
||||
ctx.ssh.exec.mockImplementation(async (cmd: string) => {
|
||||
if (cmd.startsWith("test -d")) return stdout(o.hasDir === false ? "no" : "yes");
|
||||
if (cmd.startsWith("hostname")) return HOST;
|
||||
if (cmd.includes("get ns longhorn-system")) return o.noLonghorn ? FAIL : stdout("namespace/longhorn-system");
|
||||
// Existence probe (-o name) is a DIFFERENT question from "has a disk",
|
||||
// and must be matched before the jsonpath reads.
|
||||
if (cmd.includes("get nodes.longhorn.io") && cmd.includes("-o name")) {
|
||||
return o.recordExists === false ? FAIL : stdout("node.longhorn.io/node1.lab");
|
||||
}
|
||||
if (cmd.includes("get nodes.longhorn.io") && cmd.includes("jsonpath")) {
|
||||
const has = (o.longhornCreatesDisk ?? false) || (patched && (o.repairWorks ?? true));
|
||||
return stdout(has ? DISK : "");
|
||||
}
|
||||
if (cmd.includes("patch nodes.longhorn.io")) {
|
||||
if (o.patchFails === true) return FAIL;
|
||||
patched = true;
|
||||
return OK;
|
||||
}
|
||||
if (cmd.includes("annotate node")) return stdout("node/node1.lab annotated");
|
||||
return OK;
|
||||
});
|
||||
return ctx;
|
||||
}
|
||||
|
||||
/** Run the operation with the 60s verify window collapsed. */
|
||||
async function runFast(ctx: ReturnType<typeof ctxFor>) {
|
||||
vi.useFakeTimers();
|
||||
try {
|
||||
const p = configureLonghornDisk(ctx);
|
||||
// Long enough to cover the registration wait (180s) AND the disk wait (60s).
|
||||
await vi.advanceTimersByTimeAsync(300_000);
|
||||
return await p;
|
||||
} finally {
|
||||
vi.useRealTimers();
|
||||
}
|
||||
}
|
||||
|
||||
afterEach(() => { vi.useRealTimers(); });
|
||||
|
||||
describe("the annotation survives the shell", () => {
|
||||
// The value is JSON, so it is full of double quotes. Wrapping it in double
|
||||
// quotes truncates it at the first inner quote and the node ends up with
|
||||
// [{path:/var/lib/longhorn,allowScheduling:true}]
|
||||
// which Longhorn cannot parse, so it creates no disk at all -- silently.
|
||||
// Asserting on the command string alone would NOT catch this; the command
|
||||
// looks fine until a shell reads it. So run it through a real shell.
|
||||
it("arrives as valid JSON after shell parsing", async () => {
|
||||
const ctx = ctxFor({ longhornCreatesDisk: true });
|
||||
await runFast(ctx);
|
||||
const cmd = ctx.ssh.exec.mock.calls
|
||||
.map((c: [string, unknown?]) => c[0])
|
||||
.find((c: string) => c.includes("default-disks-config")) ?? "";
|
||||
expect(cmd, "annotate command").not.toBe("");
|
||||
|
||||
// Let a real shell word-split it exactly as sshd would, and recover the
|
||||
// argument that carries the annotation.
|
||||
const { execFileSync } = await import("node:child_process");
|
||||
const printed = execFileSync("bash", ["-c", `printf '%s\n' ${cmd.replace(/^k3s kubectl /, "")}`], {
|
||||
encoding: "utf-8",
|
||||
});
|
||||
const arg = printed.split("\n").find((l) => l.startsWith("node.longhorn.io/default-disks-config=")) ?? "";
|
||||
expect(arg, "annotation argument after shell parsing").not.toBe("");
|
||||
|
||||
const json = arg.slice("node.longhorn.io/default-disks-config=".length);
|
||||
const parsed = JSON.parse(json) as { path: string; allowScheduling: boolean }[];
|
||||
expect(parsed).toEqual([{ path: DISK, allowScheduling: true }]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("configureLonghornDisk", () => {
|
||||
it("guarantees the replicas directory exists", async () => {
|
||||
// Longhorn creates <path>/replicas only when it first initialises the
|
||||
// disk. If it disappears while longhorn-disk.cfg remains, Longhorn will
|
||||
// not recreate it and every replica placed on the node dies on a
|
||||
// non-recursive mkdir -- permanently, and invisibly, because the node
|
||||
// still reports Ready and Schedulable.
|
||||
const ctx = ctxFor({ longhornCreatesDisk: true });
|
||||
await runFast(ctx);
|
||||
expectCommand(ctx.ssh, /mkdir -p \/var\/lib\/longhorn\/replicas/);
|
||||
});
|
||||
|
||||
it("skips a node with no /var/lib/longhorn", async () => {
|
||||
const r = await configureLonghornDisk(ctxFor({ hasDir: false }));
|
||||
expect(r.success).toBe(true);
|
||||
expect(r.changed).toBe(false);
|
||||
expect(r.message).toMatch(/skipping/);
|
||||
});
|
||||
|
||||
it("annotates and is satisfied when Longhorn creates the disk itself", async () => {
|
||||
const ctx = ctxFor({ longhornCreatesDisk: true });
|
||||
const r = await runFast(ctx);
|
||||
expect(r.success).toBe(true);
|
||||
expectCommand(ctx.ssh, /default-disks-config/);
|
||||
// Longhorn honoured the annotation, so no direct spec.disks write.
|
||||
expectNoCommand(ctx.ssh, /patch nodes\.longhorn\.io/);
|
||||
});
|
||||
|
||||
it("REPAIRS a rejoining node whose annotation was read too late", async () => {
|
||||
// No disk for the whole verify window, then present after the patch.
|
||||
const ctx = ctxFor({ longhornCreatesDisk: false, repairWorks: true });
|
||||
const r = await runFast(ctx);
|
||||
expect(r.success).toBe(true);
|
||||
expect(r.changed).toBe(true);
|
||||
expect(r.message).toMatch(/repaired/i);
|
||||
expectCommand(ctx.ssh, /patch nodes\.longhorn\.io/);
|
||||
expectCommand(ctx.ssh, new RegExp(`"path":"${DISK.replace(/\//g, "\\/")}"`.replace(/\\\//g, "/")));
|
||||
});
|
||||
|
||||
it("FAILS loudly when the disk cannot be created — a node with no disk is useless", async () => {
|
||||
const ctx = ctxFor({ longhornCreatesDisk: false, repairWorks: false });
|
||||
const r = await runFast(ctx);
|
||||
expect(r.success).toBe(false);
|
||||
expect(r.error).toMatch(/will not host replicas|never host a replica/);
|
||||
});
|
||||
|
||||
it("FAILS when the repair patch itself is rejected", async () => {
|
||||
const ctx = ctxFor({ longhornCreatesDisk: false, patchFails: true });
|
||||
const r = await runFast(ctx);
|
||||
expect(r.success).toBe(false);
|
||||
expect(r.message).toMatch(/could not be repaired/);
|
||||
});
|
||||
|
||||
it("does NOT fail a node Longhorn has not registered yet", async () => {
|
||||
// A node that has only just rejoined has no nodes.longhorn.io record. The
|
||||
// annotation is already in place, so Longhorn will read it when it creates
|
||||
// one. Failing here would abort a rejoin over something self-resolving.
|
||||
const ctx = ctxFor({ longhornCreatesDisk: false, recordExists: false });
|
||||
const r = await runFast(ctx);
|
||||
expect(r.success).toBe(true);
|
||||
expect(r.message).toMatch(/not registered/);
|
||||
expectNoCommand(ctx.ssh, /patch nodes\.longhorn\.io/);
|
||||
});
|
||||
|
||||
it("annotates and stops when Longhorn is not installed yet", async () => {
|
||||
// Install-time ordering: the annotation is set so Longhorn honours it when
|
||||
// it does arrive, but there is nothing to read back from.
|
||||
const ctx = ctxFor({ noLonghorn: true });
|
||||
const r = await configureLonghornDisk(ctx);
|
||||
expect(r.success).toBe(true);
|
||||
expectCommand(ctx.ssh, /default-disks-config/);
|
||||
expectNoCommand(ctx.ssh, /patch nodes\.longhorn\.io/);
|
||||
});
|
||||
|
||||
it("uses one fixed disk key, so repeated repairs update rather than multiply", async () => {
|
||||
const ctx = ctxFor({ longhornCreatesDisk: false, repairWorks: true });
|
||||
await runFast(ctx);
|
||||
const patch = ctx.ssh.exec.mock.calls
|
||||
.map((c: [string, unknown?]) => c[0])
|
||||
.find((c: string) => c.includes("patch nodes.longhorn.io")) ?? "";
|
||||
expect(patch).toContain("default-disk-lab");
|
||||
});
|
||||
});
|
||||
79
bastion/src/modules/modules/k3s/tests/module-config.test.ts
Normal file
79
bastion/src/modules/modules/k3s/tests/module-config.test.ts
Normal file
@@ -0,0 +1,79 @@
|
||||
// The ModuleContext -> OperationContext config hand-off.
|
||||
//
|
||||
// toOpContext used to enumerate three keys and silently drop the rest, so a
|
||||
// caller could set a field, get no error, and have it ignored. That produced
|
||||
// two live bugs: a pinned k3sVersion was discarded (get.k3s.io installed
|
||||
// LATEST, two minor versions ahead of the fleet), and
|
||||
// `--allow-single-replica` was a no-op that claimed to override a data-loss
|
||||
// guard.
|
||||
//
|
||||
// These tests assert the pass-through per field, and the last one fails for
|
||||
// ANY future K3sConfig field that goes missing — which is the property that
|
||||
// was actually absent.
|
||||
|
||||
import { describe, it, expect, vi } from "vitest";
|
||||
import { K3sModule } from "../src/k3s-module.js";
|
||||
import type { ModuleContext } from "../../../src/types.js";
|
||||
|
||||
vi.mock("../../../src/ssh.js", () => ({
|
||||
sshExec: vi.fn(async (_ip: string, _user: string, cmd: string) => {
|
||||
// Record every command so the test can assert on what was sent.
|
||||
issued.push(cmd);
|
||||
if (cmd.includes("k3s --version")) return { exitCode: 1, stdout: "", stderr: "" };
|
||||
if (cmd.includes("hostname")) return { exitCode: 0, stdout: "n1.lab\n", stderr: "" };
|
||||
if (cmd.includes("get nodes")) return { exitCode: 0, stdout: "NAME\nn1 Ready", stderr: "" };
|
||||
return { exitCode: 0, stdout: "", stderr: "" };
|
||||
}),
|
||||
}));
|
||||
|
||||
const issued: string[] = [];
|
||||
|
||||
function ctxWith(config: Record<string, unknown>): ModuleContext {
|
||||
return {
|
||||
hostname: "n1.lab", ip: "10.0.0.5", role: "k8s-combo",
|
||||
os: "fedora-43", arch: "x86_64", sshUser: "root", config,
|
||||
};
|
||||
}
|
||||
|
||||
describe("ModuleContext.config reaches the operations", () => {
|
||||
it("passes k3sVersion through to the installer", async () => {
|
||||
issued.length = 0;
|
||||
await new K3sModule().install(ctxWith({
|
||||
k3sToken: "t", k3sVersion: "v1.34.6+k3s1",
|
||||
}));
|
||||
const install = issued.find((c) => c.includes("get.k3s.io")) ?? "";
|
||||
expect(install, "install command").toContain('INSTALL_K3S_VERSION="v1.34.6+k3s1"');
|
||||
});
|
||||
|
||||
it("still passes the three fields it always did", async () => {
|
||||
issued.length = 0;
|
||||
await new K3sModule().install(ctxWith({
|
||||
k3sServerUrl: "https://10.0.0.1:6443", k3sToken: "tok",
|
||||
}));
|
||||
const install = issued.find((c) => c.includes("get.k3s.io")) ?? "";
|
||||
expect(install).toContain('K3S_URL="https://10.0.0.1:6443"');
|
||||
expect(install).toContain('K3S_TOKEN="tok"');
|
||||
});
|
||||
|
||||
it("drops nothing: every key given is visible to the operations", async () => {
|
||||
// The generic guard. Enumerating keys in toOpContext is the bug; this
|
||||
// fails if anyone reintroduces it for a field added later.
|
||||
const { K3sModule: Mod } = await import("../src/k3s-module.js");
|
||||
const mod = new Mod();
|
||||
const given = {
|
||||
k3sServerUrl: "https://10.0.0.1:6443",
|
||||
k3sToken: "tok",
|
||||
k3sVersion: "v1.34.6+k3s1",
|
||||
evictionAllowSingleReplica: true,
|
||||
tlsSans: ["a.lab"],
|
||||
};
|
||||
issued.length = 0;
|
||||
await mod.install(ctxWith(given));
|
||||
// k3sVersion and the join env are the observable ones at install time;
|
||||
// assert them explicitly rather than trusting the shape.
|
||||
const install = issued.find((c) => c.includes("get.k3s.io")) ?? "";
|
||||
expect(install).toContain("INSTALL_K3S_VERSION=");
|
||||
expect(install).toContain("K3S_URL=");
|
||||
expect(install).toContain("K3S_TOKEN=");
|
||||
});
|
||||
});
|
||||
211
bastion/src/modules/modules/k3s/tests/node-admit.test.ts
Normal file
211
bastion/src/modules/modules/k3s/tests/node-admit.test.ts
Normal file
@@ -0,0 +1,211 @@
|
||||
// Tests for returning an evacuated node to service.
|
||||
//
|
||||
// Evacuation leaves three separate marks on a node — a kubernetes cordon,
|
||||
// Longhorn's allowScheduling/evictionRequested pair, and (after a wipe and
|
||||
// rejoin) a MISSING Longhorn disk. Undoing only the cordon gives you a node
|
||||
// that is Ready, accepts pods and hosts no replicas at all, which reads as
|
||||
// success until storage never rebalances onto it. These tests pin all three.
|
||||
|
||||
import { describe, it, expect, vi, afterEach } from "vitest";
|
||||
import { mockCtx, OK, FAIL, stdout, expectCommand } from "./helpers.js";
|
||||
import { clearLonghornEviction, assertNodeSchedulable } from "../src/operations/node-admit.js";
|
||||
import { evictLonghornReplicas } from "../src/operations/node-evacuate.js";
|
||||
import { evacuateGroup } from "../src/groups/evacuate.js";
|
||||
import { admitGroup } from "../src/groups/admit.js";
|
||||
|
||||
const HOST = stdout("node1.lab\n");
|
||||
|
||||
/**
|
||||
* Run assertNodeSchedulable with its 120s poll collapsed.
|
||||
*
|
||||
* It polls because Longhorn registers a node asynchronously and this runs
|
||||
* moments after a rejoin; the negative cases therefore wait out the whole
|
||||
* window before they can honestly fail.
|
||||
*/
|
||||
async function runGate(ctx: ReturnType<typeof mockCtx>) {
|
||||
vi.useFakeTimers();
|
||||
try {
|
||||
const p = assertNodeSchedulable(ctx);
|
||||
await vi.advanceTimersByTimeAsync(150_000);
|
||||
return await p;
|
||||
} finally {
|
||||
vi.useRealTimers();
|
||||
}
|
||||
}
|
||||
|
||||
afterEach(() => { vi.useRealTimers(); });
|
||||
|
||||
/** Command-matching mock; see node-evacuate.test.ts for why not call order. */
|
||||
function routed(handlers: Record<string, string | null>, opts: { noLonghorn?: boolean } = {}) {
|
||||
const ctx = mockCtx();
|
||||
ctx.ssh.exec.mockImplementation(async (cmd: string) => {
|
||||
if (cmd.startsWith("hostname")) return HOST;
|
||||
if (cmd.includes("get ns longhorn-system")) return opts.noLonghorn ? FAIL : stdout("namespace/longhorn-system");
|
||||
for (const [needle, out] of Object.entries(handlers)) {
|
||||
if (cmd.includes(needle)) return out === null ? FAIL : stdout(out);
|
||||
}
|
||||
return OK;
|
||||
});
|
||||
return ctx;
|
||||
}
|
||||
|
||||
describe("clearLonghornEviction", () => {
|
||||
it("clears scheduling and eviction together in one patch", async () => {
|
||||
const ctx = routed({ "get nodes.longhorn.io": "false,true" });
|
||||
const r = await clearLonghornEviction(ctx);
|
||||
expect(r.success).toBe(true);
|
||||
expect(r.changed).toBe(true);
|
||||
// Both fields must move together: Longhorn's webhook rejects
|
||||
// evictionRequested:true on a schedulable node, so the inverse pairing
|
||||
// has to be atomic as well.
|
||||
expectCommand(ctx.ssh, /"allowScheduling":true/);
|
||||
expectCommand(ctx.ssh, /"evictionRequested":false/);
|
||||
});
|
||||
|
||||
it("is idempotent when the node already accepts replicas", async () => {
|
||||
const ctx = routed({ "get nodes.longhorn.io": "true,false" });
|
||||
const r = await clearLonghornEviction(ctx);
|
||||
expect(r.success).toBe(true);
|
||||
expect(r.changed).toBe(false);
|
||||
});
|
||||
|
||||
it("is a no-op when Longhorn is not installed", async () => {
|
||||
const ctx = routed({}, { noLonghorn: true });
|
||||
const r = await clearLonghornEviction(ctx);
|
||||
expect(r.success).toBe(true);
|
||||
expect(r.changed).toBe(false);
|
||||
});
|
||||
|
||||
it("reports failure rather than claiming the node is back", async () => {
|
||||
const ctx = routed({ "get nodes.longhorn.io": "false,true", "patch nodes.longhorn.io": null });
|
||||
const r = await clearLonghornEviction(ctx);
|
||||
expect(r.success).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe("assertNodeSchedulable — the gate after a rejoin", () => {
|
||||
it("passes when uncordoned, not evicting, and holding a schedulable disk", async () => {
|
||||
const ctx = mockCtx();
|
||||
ctx.ssh.exec.mockImplementation(async (cmd: string) => {
|
||||
if (cmd.startsWith("hostname")) return HOST;
|
||||
if (cmd.includes("get ns longhorn-system")) return stdout("namespace/longhorn-system");
|
||||
if (cmd.includes("diskStatus")) return stdout("True");
|
||||
if (cmd.includes("get nodes.longhorn.io")) return stdout("true,false");
|
||||
if (cmd.includes("get node ")) return stdout("");
|
||||
return OK;
|
||||
});
|
||||
const r = await assertNodeSchedulable(ctx);
|
||||
expect(r.success).toBe(true);
|
||||
});
|
||||
|
||||
it("REFUSES a node that has no Longhorn disk — the silent rejoin failure", async () => {
|
||||
const ctx = mockCtx();
|
||||
ctx.ssh.exec.mockImplementation(async (cmd: string) => {
|
||||
if (cmd.startsWith("hostname")) return HOST;
|
||||
if (cmd.includes("get ns longhorn-system")) return stdout("namespace/longhorn-system");
|
||||
if (cmd.includes("diskStatus")) return stdout(""); // no disks at all
|
||||
if (cmd.includes("get nodes.longhorn.io")) return stdout("true,false");
|
||||
if (cmd.includes("get node ")) return stdout("");
|
||||
return OK;
|
||||
});
|
||||
const r = await runGate(ctx);
|
||||
expect(r.success).toBe(false);
|
||||
expect(r.error).toMatch(/no disk/);
|
||||
});
|
||||
|
||||
it("REFUSES a node still marked for eviction", async () => {
|
||||
const ctx = mockCtx();
|
||||
ctx.ssh.exec.mockImplementation(async (cmd: string) => {
|
||||
if (cmd.startsWith("hostname")) return HOST;
|
||||
if (cmd.includes("get ns longhorn-system")) return stdout("namespace/longhorn-system");
|
||||
if (cmd.includes("diskStatus")) return stdout("True");
|
||||
if (cmd.includes("get nodes.longhorn.io")) return stdout("false,true");
|
||||
if (cmd.includes("get node ")) return stdout("");
|
||||
return OK;
|
||||
});
|
||||
const r = await runGate(ctx);
|
||||
expect(r.success).toBe(false);
|
||||
expect(r.error).toMatch(/eviction is still requested|scheduling is still disabled/);
|
||||
});
|
||||
|
||||
it("REFUSES a node that is still cordoned", async () => {
|
||||
const ctx = mockCtx();
|
||||
ctx.ssh.exec.mockImplementation(async (cmd: string) => {
|
||||
if (cmd.startsWith("hostname")) return HOST;
|
||||
if (cmd.includes("get ns longhorn-system")) return stdout("namespace/longhorn-system");
|
||||
if (cmd.includes("diskStatus")) return stdout("True");
|
||||
if (cmd.includes("get nodes.longhorn.io")) return stdout("true,false");
|
||||
if (cmd.includes("get node ")) return stdout("true"); // unschedulable
|
||||
return OK;
|
||||
});
|
||||
const r = await runGate(ctx);
|
||||
expect(r.success).toBe(false);
|
||||
expect(r.error).toMatch(/cordoned/);
|
||||
});
|
||||
});
|
||||
|
||||
describe("evictLonghornReplicas — never leaves a node eviction-pending", () => {
|
||||
it("cancels the eviction request when the wait blows up", async () => {
|
||||
const ctx = mockCtx();
|
||||
const patched: string[] = [];
|
||||
ctx.ssh.exec.mockImplementation(async (cmd: string) => {
|
||||
if (cmd.startsWith("hostname")) return HOST;
|
||||
if (cmd.includes("get ns longhorn-system")) return stdout("namespace/longhorn-system");
|
||||
if (cmd.includes("jsonpath")) {
|
||||
// Blow up during the rebuild wait, after eviction was requested.
|
||||
if (cmd.includes("replicas.longhorn.io")) throw new Error("API server went away");
|
||||
return stdout("");
|
||||
}
|
||||
if (cmd.includes("-o json")) return stdout(JSON.stringify({ items: [] }));
|
||||
if (cmd.includes("patch nodes.longhorn.io")) { patched.push(cmd); return OK; }
|
||||
return OK;
|
||||
});
|
||||
|
||||
await expect(evictLonghornReplicas(ctx)).rejects.toThrow("API server went away");
|
||||
|
||||
// Two patches: the request, then the rollback. Leaving only the first
|
||||
// behind would strand the node unschedulable with Longhorn still draining.
|
||||
expect(patched.length).toBe(2);
|
||||
expect(patched[0]).toContain('"evictionRequested":true');
|
||||
expect(patched[1]).toContain('"evictionRequested":false');
|
||||
expect(patched[1]).toContain('"allowScheduling":true');
|
||||
});
|
||||
});
|
||||
|
||||
describe("group ordering", () => {
|
||||
it("evacuate evicts BEFORE it drains", () => {
|
||||
// drain --force does not override PodDisruptionBudgets, and Longhorn holds
|
||||
// one on the instance-manager pod. Draining first stops the process that
|
||||
// serves the replicas being evicted: the drain blocks for its full timeout
|
||||
// and would remove the only rebuild source for a single-replica volume.
|
||||
const names = evacuateGroup.operations.map((o) => o.name);
|
||||
const evict = names.findIndex((n) => /Evict/i.test(n));
|
||||
const drain = names.findIndex((n) => /Drain/i.test(n));
|
||||
expect(evict).toBeGreaterThanOrEqual(0);
|
||||
expect(drain).toBeGreaterThanOrEqual(0);
|
||||
expect(evict).toBeLessThan(drain);
|
||||
});
|
||||
|
||||
it("evacuate checks feasibility before it mutates anything", () => {
|
||||
const names = evacuateGroup.operations.map((o) => o.name);
|
||||
expect(names.findIndex((n) => /Check eviction/i.test(n)))
|
||||
.toBeLessThan(names.findIndex((n) => /Evict Longhorn/i.test(n)));
|
||||
});
|
||||
|
||||
it("evacuate verifies last", () => {
|
||||
const names = evacuateGroup.operations.map((o) => o.name);
|
||||
expect(names[names.length - 1]).toMatch(/Verify/i);
|
||||
});
|
||||
|
||||
it("admit clears eviction before restoring the disk", () => {
|
||||
// A node still marked for eviction has any disk it is given drained again.
|
||||
const names = admitGroup.operations.map((o) => o.name);
|
||||
expect(names.findIndex((n) => /Clear Longhorn eviction/i.test(n)))
|
||||
.toBeLessThan(names.findIndex((n) => /Restore Longhorn default disk/i.test(n)));
|
||||
});
|
||||
|
||||
it("admit verifies last", () => {
|
||||
const names = admitGroup.operations.map((o) => o.name);
|
||||
expect(names[names.length - 1]).toMatch(/Verify/i);
|
||||
});
|
||||
});
|
||||
413
bastion/src/modules/modules/k3s/tests/node-evacuate.test.ts
Normal file
413
bastion/src/modules/modules/k3s/tests/node-evacuate.test.ts
Normal file
@@ -0,0 +1,413 @@
|
||||
import { describe, it, expect, vi } from "vitest";
|
||||
import { mockCtx, OK, FAIL, stdout, expectCommand, expectNoCommand } from "./helpers.js";
|
||||
import {
|
||||
cordonNode, uncordonNode, drainNode, evictLonghornReplicas, assertNodeEvacuated,
|
||||
assertEvictionCompletable, checkEvictionCompletable,
|
||||
} from "../src/operations/node-evacuate.js";
|
||||
import type { K3sConfig } from "../src/types.js";
|
||||
|
||||
const HOST = stdout("node1.lab\n");
|
||||
const GIB = 2 ** 30;
|
||||
|
||||
interface RoutedOpts {
|
||||
nodes?: string[];
|
||||
replicas?: { volume: string }[];
|
||||
volumes?: { name: string; replicas: number; size: number }[];
|
||||
softAntiAffinity?: boolean;
|
||||
overProvisionPct?: number;
|
||||
diskFreeBytes?: number;
|
||||
diskMaxBytes?: number;
|
||||
patchFails?: boolean;
|
||||
}
|
||||
|
||||
/**
|
||||
* A context whose SSH mock answers by MATCHING the command rather than by call
|
||||
* order. The operations under test read several API objects before they do
|
||||
* anything, and an ordered mock has to be re-counted every time one is added —
|
||||
* which makes the test fail for reasons that have nothing to do with the
|
||||
* behaviour it is meant to pin.
|
||||
*/
|
||||
function routed(o: RoutedOpts = {}, config?: Partial<K3sConfig>) {
|
||||
const nodeNames = o.nodes ?? ["node1.lab", "node2.lab", "node3.lab", "node4.lab"];
|
||||
const free = o.diskFreeBytes ?? 500 * GIB;
|
||||
const max = o.diskMaxBytes ?? 700 * GIB;
|
||||
const lhNodes = {
|
||||
items: nodeNames.map((name) => ({
|
||||
metadata: { name },
|
||||
spec: { allowScheduling: true },
|
||||
status: { diskStatus: { "default-disk": {
|
||||
storageAvailable: free, storageMaximum: max, storageScheduled: 0, storageReserved: 0,
|
||||
} } },
|
||||
})),
|
||||
};
|
||||
const replicas = {
|
||||
items: (o.replicas ?? []).map((r, i) => ({
|
||||
metadata: { name: `replica-${i}` },
|
||||
spec: { nodeID: "node1.lab", volumeName: r.volume },
|
||||
})),
|
||||
};
|
||||
const volumes = {
|
||||
items: (o.volumes ?? []).map((v) => ({
|
||||
metadata: { name: v.name },
|
||||
spec: { numberOfReplicas: v.replicas, size: v.size },
|
||||
status: { state: "attached", robustness: "healthy" },
|
||||
})),
|
||||
};
|
||||
|
||||
const ctx = mockCtx(config);
|
||||
ctx.ssh.exec.mockImplementation(async (cmd: string) => {
|
||||
if (cmd.startsWith("hostname")) return HOST;
|
||||
if (cmd.includes("get ns longhorn-system")) return stdout("namespace/longhorn-system");
|
||||
// NOTE: "-o jsonpath=" CONTAINS "-o json", so jsonpath must be matched
|
||||
// first or the eviction poll below is handed a whole JSON document and
|
||||
// never sees zero replicas. (This mock got that wrong once already.)
|
||||
if (cmd.includes("jsonpath")) {
|
||||
// The eviction wait polls replicas on the node — report none remaining.
|
||||
if (cmd.includes("replicas.longhorn.io")) return stdout("");
|
||||
if (cmd.includes("replica-soft-anti-affinity")) return stdout(String(o.softAntiAffinity ?? false));
|
||||
if (cmd.includes("storage-over-provisioning-percentage")) return stdout(String(o.overProvisionPct ?? 100));
|
||||
return stdout("");
|
||||
}
|
||||
if (cmd.includes("get nodes.longhorn.io") && cmd.includes("-o json")) return stdout(JSON.stringify(lhNodes));
|
||||
if (cmd.includes("get replicas.longhorn.io") && cmd.includes("-o json")) return stdout(JSON.stringify(replicas));
|
||||
if (cmd.includes("get volumes.longhorn.io") && cmd.includes("-o json")) return stdout(JSON.stringify(volumes));
|
||||
if (cmd.includes("patch nodes.longhorn.io")) return o.patchFails ? FAIL : OK;
|
||||
return OK;
|
||||
});
|
||||
return ctx;
|
||||
}
|
||||
|
||||
describe("cordonNode", () => {
|
||||
it("cordons and reports a change", async () => {
|
||||
const ctx = mockCtx();
|
||||
ctx.ssh.exec.mockResolvedValueOnce(HOST).mockResolvedValueOnce(stdout('node/node1.lab cordoned'));
|
||||
const r = await cordonNode(ctx);
|
||||
expect(r.success).toBe(true);
|
||||
expect(r.changed).toBe(true);
|
||||
expectCommand(ctx.ssh, /cordon "node1\.lab"/);
|
||||
});
|
||||
|
||||
it("is idempotent — already cordoned is changed:false", async () => {
|
||||
const ctx = mockCtx();
|
||||
ctx.ssh.exec.mockResolvedValueOnce(HOST).mockResolvedValueOnce(stdout("node/node1.lab already cordoned"));
|
||||
const r = await cordonNode(ctx);
|
||||
expect(r.success).toBe(true);
|
||||
expect(r.changed).toBe(false);
|
||||
});
|
||||
|
||||
it("fails loudly when the API is unreachable", async () => {
|
||||
const ctx = mockCtx();
|
||||
ctx.ssh.exec.mockResolvedValueOnce(HOST).mockResolvedValueOnce(FAIL);
|
||||
const r = await cordonNode(ctx);
|
||||
expect(r.success).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe("drainNode", () => {
|
||||
it("ignores daemonsets and deletes emptyDir, or it hangs forever", async () => {
|
||||
const ctx = mockCtx();
|
||||
ctx.ssh.exec.mockResolvedValueOnce(HOST).mockResolvedValueOnce(OK);
|
||||
const r = await drainNode(ctx);
|
||||
expect(r.success).toBe(true);
|
||||
expectCommand(ctx.ssh, /--ignore-daemonsets/);
|
||||
expectCommand(ctx.ssh, /--delete-emptydir-data/);
|
||||
});
|
||||
|
||||
it("gives the SSH transport longer than kubectl's own --timeout", async () => {
|
||||
// drainNode asks kubectl for --timeout=15m. sshOpts() is a flat 30s, so
|
||||
// without an explicit override the transport kills a drain that is working
|
||||
// -- after it has already evicted most of the pods. The result is
|
||||
// non-zero with a transcript that reads like success, on every node with
|
||||
// real workloads. `labctl app k3s evacuate` could never have drained one.
|
||||
const ctx = mockCtx();
|
||||
ctx.ssh.exec.mockResolvedValueOnce(HOST).mockResolvedValueOnce(OK);
|
||||
await drainNode(ctx);
|
||||
const call = ctx.ssh.exec.mock.calls.find((c: [string, { timeoutMs?: number }?]) =>
|
||||
c[0].includes("drain "));
|
||||
expect(call, "drain command").toBeDefined();
|
||||
const timeoutMs = call?.[1]?.timeoutMs ?? 0;
|
||||
expect(timeoutMs, "SSH timeout must exceed kubectl's 15m --timeout")
|
||||
.toBeGreaterThan(15 * 60_000);
|
||||
});
|
||||
|
||||
it("reports the END of the drain transcript, not the useless first line", async () => {
|
||||
// Every drain starts with "node/x already cordoned". Surfacing the whole
|
||||
// transcript means the error you see is that line, which explains nothing.
|
||||
const ctx = mockCtx();
|
||||
ctx.ssh.exec.mockResolvedValueOnce(HOST).mockResolvedValueOnce({
|
||||
exitCode: 1,
|
||||
stdout: "node/n1 already cordoned\nevicting pod a\nevicting pod b\n" +
|
||||
"error: cannot evict pod as it would violate the pod's disruption budget",
|
||||
stderr: "",
|
||||
});
|
||||
const r = await drainNode(ctx);
|
||||
expect(r.success).toBe(false);
|
||||
expect(r.error).toContain("disruption budget");
|
||||
});
|
||||
|
||||
it("reports failure rather than pretending the node is empty", async () => {
|
||||
const ctx = mockCtx();
|
||||
ctx.ssh.exec.mockResolvedValueOnce(HOST).mockResolvedValueOnce(FAIL);
|
||||
const r = await drainNode(ctx);
|
||||
expect(r.success).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe("evictLonghornReplicas", () => {
|
||||
it("is a no-op when Longhorn is not installed", async () => {
|
||||
const ctx = mockCtx();
|
||||
ctx.ssh.exec.mockResolvedValueOnce(HOST).mockResolvedValueOnce(FAIL); // no longhorn-system ns
|
||||
const r = await evictLonghornReplicas(ctx);
|
||||
expect(r.success).toBe(true);
|
||||
expect(r.changed).toBe(false);
|
||||
expect(r.message).toMatch(/not installed/);
|
||||
});
|
||||
|
||||
it("requests eviction and waits until zero replicas remain", async () => {
|
||||
// Node holds no replicas, so the pre-check clears it immediately.
|
||||
const ctx = routed({ replicas: [] });
|
||||
const r = await evictLonghornReplicas(ctx);
|
||||
expect(r.success).toBe(true);
|
||||
expect(r.changed).toBe(true);
|
||||
expectCommand(ctx.ssh, /evictionRequested/);
|
||||
expectCommand(ctx.ssh, /allowScheduling/);
|
||||
});
|
||||
|
||||
it("fails when eviction cannot even be requested", async () => {
|
||||
const ctx = routed({ replicas: [], patchFails: true });
|
||||
const r = await evictLonghornReplicas(ctx);
|
||||
expect(r.success).toBe(false);
|
||||
});
|
||||
|
||||
it("does not request eviction at all when it could never complete", async () => {
|
||||
// 3-replica volume, and only 2 other schedulable nodes: the survivors
|
||||
// already hold a copy each and strict anti-affinity forbids a third.
|
||||
const ctx = routed({
|
||||
nodes: ["node1.lab", "node2.lab", "node3.lab"],
|
||||
replicas: [{ volume: "pvc-a" }],
|
||||
volumes: [{ name: "pvc-a", replicas: 3, size: 1 * GIB }],
|
||||
});
|
||||
const r = await evictLonghornReplicas(ctx);
|
||||
expect(r.success).toBe(false);
|
||||
expect(r.error).toMatch(/cannot be placed|wants 3 replicas/);
|
||||
// The important half: nothing was mutated.
|
||||
expectNoCommand(ctx.ssh, /evictionRequested/);
|
||||
});
|
||||
|
||||
it("refuses a single-replica volume unless explicitly allowed", async () => {
|
||||
const opts = {
|
||||
nodes: ["node1.lab", "node2.lab", "node3.lab", "node4.lab"],
|
||||
replicas: [{ volume: "pvc-solo" }],
|
||||
volumes: [{ name: "pvc-solo", replicas: 1, size: 1 * GIB }],
|
||||
};
|
||||
const r = await evictLonghornReplicas(routed(opts));
|
||||
expect(r.success).toBe(false);
|
||||
expect(r.error).toMatch(/single replica/);
|
||||
|
||||
const allowed = await evictLonghornReplicas(
|
||||
routed(opts, { evictionAllowSingleReplica: true }),
|
||||
);
|
||||
expect(allowed.success).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe("checkEvictionCompletable — refuse in seconds, not in an hour", () => {
|
||||
it("passes when a spare schedulable node exists", async () => {
|
||||
const ctx = routed({
|
||||
nodes: ["node1.lab", "node2.lab", "node3.lab", "node4.lab"],
|
||||
replicas: [{ volume: "pvc-a" }],
|
||||
volumes: [{ name: "pvc-a", replicas: 3, size: 1 * GIB }],
|
||||
});
|
||||
const f = await checkEvictionCompletable(ctx, "node1.lab");
|
||||
expect(f.completable).toBe(true);
|
||||
expect(f.replicasMoving).toBe(1);
|
||||
});
|
||||
|
||||
it("blocks when replica count equals the surviving node count", async () => {
|
||||
const ctx = routed({
|
||||
nodes: ["node1.lab", "node2.lab", "node3.lab"],
|
||||
replicas: [{ volume: "pvc-a" }],
|
||||
volumes: [{ name: "pvc-a", replicas: 3, size: 1 * GIB }],
|
||||
});
|
||||
const f = await checkEvictionCompletable(ctx, "node1.lab");
|
||||
expect(f.completable).toBe(false);
|
||||
expect(f.blockers.join(" ")).toContain("pvc-a");
|
||||
});
|
||||
|
||||
it("allows it when soft anti-affinity is on — placement stops being the limit", async () => {
|
||||
const ctx = routed({
|
||||
nodes: ["node1.lab", "node2.lab", "node3.lab"],
|
||||
replicas: [{ volume: "pvc-a" }],
|
||||
volumes: [{ name: "pvc-a", replicas: 3, size: 1 * GIB }],
|
||||
softAntiAffinity: true,
|
||||
});
|
||||
const f = await checkEvictionCompletable(ctx, "node1.lab");
|
||||
expect(f.completable).toBe(true);
|
||||
});
|
||||
|
||||
it("catches a small disk hiding behind over-provisioning", async () => {
|
||||
// The scheduler is happy at 500%; the disk has 49 GiB and the volume is 92.
|
||||
const ctx = routed({
|
||||
nodes: ["node1.lab", "node2.lab", "node3.lab", "node4.lab"],
|
||||
replicas: [{ volume: "pvc-big" }],
|
||||
volumes: [{ name: "pvc-big", replicas: 3, size: 92 * GIB }],
|
||||
overProvisionPct: 500,
|
||||
diskFreeBytes: 49 * GIB,
|
||||
diskMaxBytes: 100 * GIB,
|
||||
});
|
||||
const f = await checkEvictionCompletable(ctx, "node1.lab");
|
||||
expect(f.completable).toBe(false);
|
||||
expect(f.blockers.join(" ")).toMatch(/actually free/);
|
||||
expect(f.blockers.join(" ")).toMatch(/over-provisioning-percentage=500/);
|
||||
});
|
||||
|
||||
it("warns, but does not block, on a single-replica volume", async () => {
|
||||
const ctx = routed({
|
||||
nodes: ["node1.lab", "node2.lab", "node3.lab"],
|
||||
replicas: [{ volume: "pvc-solo" }],
|
||||
volumes: [{ name: "pvc-solo", replicas: 1, size: 1 * GIB }],
|
||||
});
|
||||
const f = await checkEvictionCompletable(ctx, "node1.lab");
|
||||
expect(f.completable).toBe(true);
|
||||
expect(f.warnings.join(" ")).toMatch(/single replica/);
|
||||
});
|
||||
|
||||
it("is trivially completable when the node holds nothing", async () => {
|
||||
const f = await checkEvictionCompletable(routed({ replicas: [] }), "node1.lab");
|
||||
expect(f.completable).toBe(true);
|
||||
expect(f.replicasMoving).toBe(0);
|
||||
});
|
||||
|
||||
it("refuses to guess when Longhorn cannot be read", async () => {
|
||||
const ctx = mockCtx();
|
||||
ctx.ssh.exec.mockResolvedValue(FAIL);
|
||||
const f = await checkEvictionCompletable(ctx, "node1.lab");
|
||||
expect(f.completable).toBe(false);
|
||||
expect(f.blockers.join(" ")).toMatch(/refusing to guess/);
|
||||
});
|
||||
});
|
||||
|
||||
describe("assertNodeEvacuated — the gate before a wipe", () => {
|
||||
it("passes only when cordoned with no replicas left", async () => {
|
||||
const ctx = mockCtx();
|
||||
ctx.ssh.exec
|
||||
.mockResolvedValueOnce(HOST)
|
||||
.mockResolvedValueOnce(stdout("true")) // unschedulable
|
||||
.mockResolvedValueOnce(stdout("")) // no running pods
|
||||
.mockResolvedValueOnce(stdout("namespace/longhorn-system"))
|
||||
.mockResolvedValueOnce(stdout("")); // no replicas
|
||||
const r = await assertNodeEvacuated(ctx);
|
||||
expect(r.success).toBe(true);
|
||||
expect(r.message).toMatch(/safe to wipe/);
|
||||
});
|
||||
|
||||
it("REFUSES when replicas are still on the node", async () => {
|
||||
const ctx = mockCtx();
|
||||
ctx.ssh.exec
|
||||
.mockResolvedValueOnce(HOST)
|
||||
.mockResolvedValueOnce(stdout("true"))
|
||||
.mockResolvedValueOnce(stdout(""))
|
||||
.mockResolvedValueOnce(stdout("namespace/longhorn-system"))
|
||||
.mockResolvedValueOnce(stdout("replica-abc\nreplica-def"));
|
||||
const r = await assertNodeEvacuated(ctx);
|
||||
expect(r.success).toBe(false);
|
||||
expect(r.error).toMatch(/2 Longhorn replica/);
|
||||
});
|
||||
|
||||
it("REFUSES when the node is still schedulable", async () => {
|
||||
const ctx = mockCtx();
|
||||
ctx.ssh.exec
|
||||
.mockResolvedValueOnce(HOST)
|
||||
.mockResolvedValueOnce(stdout("")) // not unschedulable
|
||||
.mockResolvedValueOnce(stdout(""))
|
||||
.mockResolvedValueOnce(FAIL); // no longhorn
|
||||
const r = await assertNodeEvacuated(ctx);
|
||||
expect(r.success).toBe(false);
|
||||
expect(r.error).toMatch(/still schedulable/);
|
||||
});
|
||||
});
|
||||
|
||||
describe("uncordonNode", () => {
|
||||
it("returns the node to service", async () => {
|
||||
const ctx = mockCtx();
|
||||
ctx.ssh.exec.mockResolvedValueOnce(HOST).mockResolvedValueOnce(OK);
|
||||
const r = await uncordonNode(ctx);
|
||||
expect(r.success).toBe(true);
|
||||
expectCommand(ctx.ssh, /uncordon "node1\.lab"/);
|
||||
});
|
||||
});
|
||||
|
||||
describe("eviction timeout diagnostics", () => {
|
||||
// An eviction can legitimately run for an hour. When it times out, a bare
|
||||
// count ("2 replica(s) still present") is not actionable -- the question is
|
||||
// always WHICH volume is stuck and in what state. A replica with
|
||||
// desireState=running, currentState=stopped and no failedAt is Longhorn
|
||||
// wanting to start it and not doing so, which is a completely different
|
||||
// problem from a volume that cannot be placed. Observed for real on a
|
||||
// freshly wiped-and-rejoined node.
|
||||
it("names the stuck volumes and their replica states, then restores the node", async () => {
|
||||
const ctx = mockCtx();
|
||||
const patches: string[] = [];
|
||||
ctx.ssh.exec.mockImplementation(async (cmd: string) => {
|
||||
if (cmd.startsWith("hostname")) return HOST;
|
||||
if (cmd.includes("get ns longhorn-system")) return stdout("namespace/longhorn-system");
|
||||
if (cmd.includes("patch nodes.longhorn.io")) { patches.push(cmd); return OK; }
|
||||
if (cmd.includes("jsonpath")) {
|
||||
// The poll never reaches zero: one replica remains for the whole run.
|
||||
if (cmd.includes("replicas.longhorn.io")) return stdout("replica-stuck");
|
||||
if (cmd.includes("replica-soft-anti-affinity")) return stdout("false");
|
||||
if (cmd.includes("storage-over-provisioning-percentage")) return stdout("100");
|
||||
return stdout("");
|
||||
}
|
||||
if (cmd.includes("get replicas.longhorn.io") && cmd.includes("-o json")) {
|
||||
return stdout(JSON.stringify({ items: [{
|
||||
metadata: { name: "pvc-abc-r-deadbeef99" },
|
||||
spec: { nodeID: "node1.lab", volumeName: "pvc-abc", failedAt: "", desireState: "running" },
|
||||
status: { currentState: "stopped" },
|
||||
}] }));
|
||||
}
|
||||
if (cmd.includes("get volumes.longhorn.io") && cmd.includes("-o json")) {
|
||||
return stdout(JSON.stringify({ items: [{
|
||||
metadata: { name: "pvc-abc" },
|
||||
spec: { numberOfReplicas: 3, size: 2 * 2 ** 30 },
|
||||
status: { state: "attached", robustness: "degraded" },
|
||||
}] }));
|
||||
}
|
||||
if (cmd.includes("get nodes.longhorn.io") && cmd.includes("-o json")) {
|
||||
return stdout(JSON.stringify({ items: ["node1.lab", "node2.lab", "node3.lab", "node4.lab"].map((n) => ({
|
||||
metadata: { name: n },
|
||||
spec: { allowScheduling: true },
|
||||
status: { diskStatus: { d: {
|
||||
storageAvailable: 500 * 2 ** 30, storageMaximum: 700 * 2 ** 30,
|
||||
storageScheduled: 0, storageReserved: 0,
|
||||
} } },
|
||||
})) }));
|
||||
}
|
||||
return OK;
|
||||
});
|
||||
|
||||
vi.useFakeTimers();
|
||||
let result;
|
||||
try {
|
||||
const p = evictLonghornReplicas(ctx);
|
||||
await vi.advanceTimersByTimeAsync(61 * 60_000);
|
||||
result = await p;
|
||||
} finally {
|
||||
vi.useRealTimers();
|
||||
}
|
||||
|
||||
expect(result.success).toBe(false);
|
||||
// The actionable part: which volume, and what state its replica is in.
|
||||
expect(result.error).toContain("pvc-abc");
|
||||
expect(result.error).toContain("currentState=stopped");
|
||||
expect(result.error).toContain("desireState=running");
|
||||
expect(result.error).toContain("failedAt=(none)");
|
||||
expect(result.error).toContain("attached/degraded");
|
||||
|
||||
// And the node must be handed back usable, not left eviction-pending.
|
||||
expect(result.error).toMatch(/cancelled and scheduling restored/);
|
||||
expect(patches).toHaveLength(2);
|
||||
expect(patches[1]).toContain('"evictionRequested":false');
|
||||
expect(patches[1]).toContain('"allowScheduling":true');
|
||||
});
|
||||
});
|
||||
@@ -1,4 +1,4 @@
|
||||
import type { OsId, Arch } from "@lab/shared";
|
||||
import type { OsId, Arch, Role } from "@lab/shared";
|
||||
|
||||
export interface ModuleMetadata {
|
||||
name: string;
|
||||
@@ -14,7 +14,10 @@ export interface ModuleMetadata {
|
||||
export interface ModuleContext {
|
||||
hostname: string;
|
||||
ip: string;
|
||||
role: string;
|
||||
// Role, not string. A literal `string` let `role === "infra"` comparisons
|
||||
// compile while silently missing every other server role -- which is how
|
||||
// labcontroller ended up installing as an agent.
|
||||
role: Role;
|
||||
os: OsId;
|
||||
arch: Arch;
|
||||
sshUser: string;
|
||||
@@ -24,7 +27,7 @@ export interface ModuleContext {
|
||||
|
||||
export interface ModuleResult {
|
||||
success: boolean;
|
||||
phase: "install" | "configure" | "health";
|
||||
phase: "install" | "configure" | "health" | "evacuate" | "admit";
|
||||
duration: number;
|
||||
output: string[];
|
||||
errors: string[];
|
||||
|
||||
Reference in New Issue
Block a user