lab/bastion/docs/ARCHITECTURE.md

Lab Platform Architecture

Overview

A bare-metal and hybrid cloud infrastructure platform for automated machine provisioning, Kubernetes cluster management, and fleet operations. The platform discovers hardware via PXE boot, installs operating systems unattended, deploys k3s clusters, and provides centralized management through a CLI and API.

Components:

• bastion -- PXE boot server (DHCP/TFTP/HTTP) for machine discovery and OS installation
• labd -- Master daemon for multi-bastion aggregation, persistent state, agent management
• labctl -- CLI tool for operators (kubectl-style interface)
• lab-agent -- Daemon on provisioned servers for remote execution and monitoring
• modules -- Declarative configuration system (k3s, labcontroller)

---

Architecture

                           labctl (CLI)
                               |
                        labd (master daemon)
                        /    |    \
                  bastion1  bastion2  ...     (PXE provisioning)
                    /   \      |
              [machines]  [machines]           (bare metal)
                  |           |
              lab-agent   lab-agent            (remote exec)

Communication Patterns

Path	Protocol	Auth
labctl -> labd	HTTP/HTTPS	mTLS cert (future: token)
bastion -> labd	WebSocket	Join token enrollment
lab-agent -> labd	WebSocket	mTLS certificate
machine -> bastion	HTTP	None (local network)
Anaconda -> bastion	HTTP + UDP syslog	None (install-time)
labctl -> bastion	HTTP	None (standalone mode)

Standalone vs Centralized

The bastion can operate in two modes:

1. Standalone -- single bastion, state in local JSON file, CLI talks directly to bastion HTTP API
2. Centralized -- bastion registers with labd via WebSocket, state aggregated in CockroachDB, CLI talks to labd which routes commands to the correct bastion

---

Machine Lifecycle

                     PXE boot
                        |
               +--------v--------+
               |    DISCOVERED   |  Hardware inventory collected
               +---------+-------+
                         |
              labctl provision install
                         |
               +---------v-------+
               |  INSTALL_QUEUE  |  Waiting for next PXE boot
               +---------+-------+
                         |
                   PXE boot (Anaconda)
                         |
               +---------v-------+
               |   INSTALLING    |  Progress: partitioning -> packages -> post-install
               +---------+-------+
                         |
               +---------v-------+
               |    INSTALLED    |  OS ready, SSH accessible
               +---------+-------+
                         |
              labctl app k3s install
                         |
               +---------v-------+
               |   K3S RUNNING   |  Kubernetes node operational
               +--------+--------+
                        |
              labctl provision reprovision
                        |
                   (back to INSTALL_QUEUE)

Side paths:

• DEBUG -- labctl provision debug boots Anaconda rescue mode for diagnostics
• FORGET -- labctl provision forget removes machine from all state

---

Multi-architecture PXE

The bastion serves both x86_64 and aarch64 over the network. Nothing about this is operator-configured -- there is no --arch flag, by design.

How a client's architecture is decided

1. DHCP option 93 (Client System Architecture) picks the bootloader. dnsmasq matches it and hands out a matching iPXE binary:

   Option 93	Client	Served
   0	x86 BIOS	undionly.kpxe (TFTP)
   7, 9	x64 UEFI	ipxe.efi (TFTP)
   11	ARM64 UEFI	ipxe-arm64.efi (TFTP)
   16	x64 UEFI HTTP Boot	http://…/ipxe.efi
   19	ARM64 UEFI HTTP Boot	http://…/ipxe-arm64.efi

   Values come from the IANA Processor Architecture Types registry. Note 19, not 20 -- 20 is pc/at bios boot from http. EDK2/AAVMF prefers HTTP Boot over TFTP PXE, so the iPXE binaries are staged in both tftpDir and httpDir (symlinked by main.ts).

2. /dispatch picks the kernel. Option 93 never reaches the HTTP endpoint, so boot.ipxe passes iPXE's own ${buildarch} as ?arch=. resolveArch() prefers, in order: the tracked machine record → the reported ?arch= → the configured default. The record wins because it is what we observed on the machine itself.

Artifact naming

x86_64 keeps the original unsuffixed paths so its rendered iPXE scripts are unchanged; everything else is suffixed. kernelPath() / initrdPath() in templates/boot.ipxe.ts are the single source of truth, used by both the templates and main.ts staging.

arch	kernel	initrd
x86_64	/vmlinuz	/initrd.img
aarch64	/vmlinuz-aarch64	/initrd-aarch64.img

tests/ipxe-x86-regression.test.ts pins the x86_64 output against a golden fixture.

arm64 gotchas

• LoadFile2 is mandatory. arm64 has no HdrS boot protocol; the kernel's EFI stub fetches the initrd over the UEFI EFI_LOAD_FILE2_PROTOCOL. An iPXE build without it accepts the initrd line, silently drops it, and the kernel panics with VFS: Unable to mount root fs on unknown-block(0,0). Fedora's ipxe-bootimgs-aarch64 implements it; the integration test asserts this up front so the failure names itself instead of looking like a disk problem.
• nomodeset is x86-only. On arm64 there is no VGA path to fall back to. aarch64 gets console=tty0 console=ttyAMA0,115200 instead — the last console= wins for /dev/console, so serial is the interactive one.
• Ubuntu is x86_64-only. releases.ubuntu.com publishes no arm64 netboot artifacts. osSupportsArch() encodes this, and both the install guard and /dispatch refuse the combination rather than serving an x86 kernel to an ARM machine.

---

Onboarding classification (vendor OS)

Machines carry an onboard field: "pxe" (default) or "ssh", plus vendor_os naming what they run. classifyOnboard() in @lab/shared sets it from DMI identity, with known hardware also matched by MAC — a machine can sit in state for a long time with no DMI, and a DMI-only rule would fail open exactly where it matters.

onboard: "ssh" means we cannot rebuild this machine's OS. Installs are refused at both entry points (/api/install and the labd command-install handler) with an error naming the machine and pointing at provision debug. Rescue is never guarded — being unable to reinstall a machine is precisely when a rescue shell is needed.

This is a fact about the machine, not a blocklist. The refusal follows from "no image in our pipeline restores vendor_os", so adding a DGX OS image to the pipeline is what unblocks the DGX Sparks — no entry needs deleting.

Current classifications: NVIDIA DGX Spark (spark-2935, spark-3a1c) → dgx-os.

---

Packages

Monorepo Structure

TypeScript ESM monorepo with pnpm workspaces. Six packages:

Package	Role	Key Tech
@lab/shared	Types, protocol, constants	-
@lab/bastion	PXE server	Fastify, dnsmasq
@lab/cli	CLI binary	Commander.js
@lab/labd	Master daemon	Fastify, Prisma, CockroachDB
@lab/agent	Server agent	WebSocket
@lab/modules	Config modules	SSH, k8s-client

@lab/shared

Core type system shared by all packages.

State Model:

interface BastionState {
  discovered:    Record<MAC, HardwareInfo>
  install_queue: Record<MAC, InstallConfig>
  installed:     Record<MAC, InstalledInfo>
  debug:         Record<MAC, DebugConfig>
}

Roles:

• vanilla -- OS only, no k3s, no cluster services
• worker -- k3s agent + Longhorn storage (joins existing cluster)
• infra -- k3s server + etcd (control plane node)
• labcontroller -- infra + bastion + labd + CockroachDB (self-sufficient)

OS Support:

• fedora-43 -- Anaconda kickstart installer
• ubuntu-26.04 -- cloud-init autoinstall

Protocol: Discriminated union message types for WebSocket communication between agents, bastions, and labd. Type guards and parsers for runtime validation.

@lab/bastion

PXE boot server that handles the physical provisioning lifecycle.

Services:

• StateManager -- JSON file persistence with immutable update pattern
• SyslogListener -- UDP syslog receiver (port 5514) for Anaconda install logs
• InstallLogBuffer -- In-memory ring buffer + disk persistence per machine
• BastionConnection -- WebSocket client to labd for centralized mode
• dnsmasq management (spawn, config generation, proxy/full DHCP)
• Network auto-detection (interface, IP, subnet, gateway)
• ISO builder (xorriso + mtools for non-PXE machines)

HTTP Routes:

Endpoint	Purpose
GET /dispatch?mac=	Dynamic iPXE script (discover/install/debug/local-boot)
GET /ks?mac=	Per-machine Anaconda kickstart
GET /debug.ks	Rescue mode kickstart
GET /debug-setup.sh	nc listener setup script for rescue shell
GET /discover.ks	Hardware discovery kickstart
POST /api/discover	Hardware inventory report
POST /api/install	Queue machine for install
POST /api/progress	Install progress callback
POST /api/log	Raw log line ingestion
POST /api/debug	Queue debug/rescue mode
GET /api/machines	List all machines
GET /api/logs/:mac	Install logs + progress
GET /api/logs/:mac/follow	SSE stream of progress events
DELETE /api/machines/:mac	Forget machine

Templates:

• boot.ipxe.ts -- iPXE scripts for each boot mode (discover, install, debug, pxe-boot-debug, local-boot)
• install.ks.ts -- Full Fedora kickstart with LVM, SSH, k3s prereqs, progress callbacks, SysRq keys
• debug.ks.ts -- Minimal rescue kickstart (SSH via inst.sshd)
• ubuntu-autoinstall.ts -- cloud-init for Ubuntu
• dnsmasq.conf.ts -- DHCP/TFTP configuration

Boot Dispatch Logic:

1. debug[mac]?        -> renderDebugIpxe (auto-clear after serving)
2. install_queue[mac]? -> renderInstallIpxe
3. installed[mac]?    -> renderLocalBootIpxe (exit to disk)
4. unknown            -> renderDiscoverIpxe

@lab/labd

Central management daemon. Aggregates multiple bastions, stores persistent state in CockroachDB, relays commands, manages agent fleet.

Database (Prisma + CockroachDB):

• Server -- hostname, MAC, IP, role, status, cloud, environment, labels
• Bastion -- hostname, network, serverIp, lastHeartbeat
• Agent -- certificate, enrollment, heartbeat
• Cluster -- name, cloud, environment, kubeconfig (encrypted)
• User / Role / Permission -- RBAC (action:cloud:env:server matrix)
• JoinToken -- one-time/reusable enrollment tokens
• AuditLog -- action, resource, result, timestamp

Key Services:

• BastionRegistry -- in-memory registry of connected bastions, state aggregation, MAC-to-bastion routing
• AgentRegistry -- connected agents, heartbeat tracking
• MessageRouter -- command relay between CLI/agents and bastions

Command Routing:

CLI: labctl provision install <mac> <hostname>
  -> POST /api/machines/install
  -> labd finds bastion that knows this MAC
  -> WebSocket: {type: "command-install", mac, hostname, disk, role}
  -> bastion updates install_queue
  -> WebSocket: {type: "command-response", status: "ok"}
  -> HTTP response to CLI

@lab/cli (labctl)

Operator CLI. Commander.js binary, distributed as RPM/DEB or standalone bun-compiled executable.

Command Groups:

labctl init bastion standalone start|stop|status
labctl provision list|install|reprovision|forget|debug|logs|makeiso
labctl app k3s install|health|list
labctl config list|get|set|path
labctl login
labctl doctor
labctl roles

Key Features:

• Target resolution: hostname, MAC, or IP -> machine lookup
• SSH reboot into PXE for reprovision/debug (efibootmgr --bootnext)
• Follow mode: labctl provision logs <target> -f (5s polling)
• Shell completions: bash, fish

@lab/modules

Declarative configuration modules with three-phase lifecycle: install -> configure -> health.

k3s Module:

• 5 operation groups: host-prep, networking, k3s-server, k3s-agent, hardening
• 15+ individual operations: kernel modules, sysctl, firewall, Cilium CNI, SELinux, audit policy, pod security, cert checks
• Health checks: service running, node ready, API health, pod status, Cilium status, secrets encryption
• SSH execution backend with progress callbacks

@lab/agent

Daemon on provisioned servers. WebSocket to labd for:

• Heartbeat (hostname, uptime, CPU/mem usage)
• Command execution (with stdout/stderr streaming)
• Log streaming (journalctl relay)
• mTLS certificate enrollment and rotation

---

Disk Layout

LVM Partitioning (labvg)

All roles share a common LVM layout. The kickstart %pre auto-detects the install disk (NVMe preferred, then SATA, skipping USB/removable).

Volume	Size	FS	Reprovision
/boot/efi	600 MB	vfat	Reused
/boot	3 GB	ext4	Reused
swap	27 GB	swap	Recreated
/ (root)	33 GB	xfs	Recreated
/var	100 GB	xfs	Recreated
/var/log	10 GB	xfs	Recreated
/home	10 GB	xfs	Preserved
/srv	20 GB	xfs	Preserved
/var/lib/longhorn	remaining	xfs	Preserved (worker)
/var/lib/rancher	20 GB	xfs	Preserved (infra)
/tmp	4 GB	tmpfs	-

Reprovision detection: if labvg VG exists, reuse EFI/boot partitions and preserve data volumes.

---

Kickstart Features

The Fedora kickstart template (install.ks.ts) includes:

• Dynamic disk detection -- %pre probes NVMe/SATA/virtio, skips USB/removable, supports both fresh install and reprovision
• Progress callbacks -- curl -sf POST /api/progress at each stage (partitioning, post-install substeps, complete)
• Anaconda syslog forwarding -- logging --host --port streams real-time install logs to bastion
• SSH hardening -- key-only auth, root login via pubkey only, admin user with passwordless sudo
• Network-first boot order -- efibootmgr reorders boot entries so PXE is always first (bastion controls every reboot)
• SysRq magic keys -- kernel.sysrq=1 for emergency reboot via KVM keyboard
• Role-specific setup:
  • vanilla: chronyd only
  • worker/infra: kernel modules (br_netfilter, overlay), sysctl (ip_forward, inotify), firewalld disabled, k3s binary installed
  • infra: k3s server binary pre-installed

What is NOT in the kickstart:

• console=ttyS0 -- causes 30s-per-step boot timeout on hardware without physical serial UART (discovered 2026-03-30, see docs/pxe-boot-debugging-2026-03-30.md)
• Background log streamer (tail -f) -- prevents Anaconda from syncing filesystem, causes %post writes to not persist

---

Deployment

Container Images

bastion (Dockerfile.bastion):

• Base: Fedora 43 (needs dnsmasq, iPXE)
• Multi-stage: Alpine build -> Fedora runtime
• iPXE rebuilt from source (SNP driver for EFI)
• hostNetwork in k8s (DHCP needs raw sockets)
• Capabilities: NET_ADMIN, NET_RAW

labd (Dockerfile.labd):

• Base: Alpine (minimal)
• Multi-stage build with Prisma client generation
• Runs as non-root node user

Kubernetes (k3s)

Namespace: lab-infra
  Deployment: bastion (hostNetwork, PVC for /data, host SSH keys)
  ConfigMap: bastion-config (env vars)
  Secret: bastion-join-token
  PVC: bastion-state (local-path)

Namespace: lab-system
  Deployment: labd
  Service: labd (NodePort 30100)
  StatefulSet: cockroachdb-0

CLI Distribution

Built with nfpm as RPM/DEB. Includes:

• /usr/bin/labctl (bun-compiled standalone binary)
• /usr/share/bash-completion/completions/labctl
• /usr/share/fish/vendor_completions.d/labctl.fish

Config: ~/.labctl/config.yaml with labdUrl, output format, default cloud/environment.

---

Build & Release

# Development
pnpm install && pnpm build      # Compile all packages
pnpm test:run                    # Unit tests (vitest)
npx tsc --noEmit                 # Type check

# Deploy
bash scripts/deploy.sh all       # Build containers + RPM, push, restart pods
bash scripts/deploy.sh bastion   # Just bastion
bash scripts/deploy.sh labd      # Just labd
bash scripts/deploy.sh labctl    # Just CLI (local RPM install)

# Container builds
bash scripts/build-bastion.sh --platforms linux/amd64 --push latest
bash scripts/build-labd.sh --platforms linux/amd64 --push latest
bash scripts/build-rpm.sh        # RPM + DEB packages

# Integration tests (require libvirt, sudo)
sudo tests/integration/run-pxe-test.sh

Registry: mysources.co.uk (Gitea at 10.0.0.194:3012)

---

Testing

Unit Tests

• Kickstart rendering (ksvalidator syntax check, partition layout, role-specific sections)
• State management (load, save, update, debug field)
• Dispatch routing (correct iPXE script for each machine state)
• Syslog listener (UDP receive, IP->MAC resolution, RFC 3164 parsing)

Integration Tests (libvirt VMs)

• pxe-provision.test.ts -- Full end-to-end: create VM -> PXE discovery -> queue install -> Anaconda install -> SSH verification -> systemd health -> SELinux enforcing -> boot order check
• iso-provision.test.ts -- ISO boot for non-PXE machines
• k3s-single-node.test.ts -- Post-provision k3s installation and health
• VM screenshot capture during boot for debugging

---

Security

• mTLS for agent-labd communication (certificate enrollment via join tokens)
• SSH key-only auth on provisioned machines (no password auth)
• SELinux enforcing verified in integration tests
• RBAC (planned): action:cloud:environment:server permission matrix
• Audit logging (planned): every mutation tracked in CockroachDB
• Network-first boot order prevents machines from booting without bastion approval
• SysRq keys enabled for emergency reboot without SSH access

---

Known Issues & Lessons Learned

Serial Console Boot Delay (2026-03-30)

console=ttyS0,115200n8 in kernel cmdline causes 30-second timeout at every systemd boot phase on hardware without a physical serial UART. Root cause: systemd blocks writing to non-existent UART. Fix: removed from kickstart entirely.

Anaconda %post Log Streamer

Background tail -f in kickstart %post prevents Anaconda from syncing the filesystem. All file writes in %post appear to succeed but are lost on reboot. Fix: removed background log streamer, replaced with Anaconda's built-in logging --host --port syslog forwarding.

Disk Auto-Detection

Hardcoded /dev/sda default broke NVMe-only machines. Fix: default to empty string (auto-detect) which triggers the %pre disk probe logic.

Anaconda Rescue Mode Limitations

%pre and %post sections do not execute in inst.rescue mode. SSH in rescue mode is provided by Anaconda's inst.sshd kernel parameter + sshpw kickstart directive. Manual setup via curl bastion:8080/debug-setup.sh | bash for nc listener.

Unresolved (2026-08-11): rescue SSH has never been observed working. Adding the first integration coverage for provision debug (tests/integration/pxe-rescue.test.ts) showed the rescue environment coming up correctly — the bastion serves the kernel and initrd, Anaconda boots, fetches debug.ks, and reaches its installer environment — but nothing ever listens on port 22. Confirmed on aarch64 by probing the port for 30 minutes while the Anaconda environment was demonstrably running (NetworkManager, polkitd, rsyslog all up), and reproduced on x86_64 with KVM, so it is not architecture-specific and not an emulation artefact.

This is orthogonal to the multi-architecture work: the same failure occurs on x86_64, which that work does not touch. Leads worth checking, in order:

• Does inst.sshd actually start sshd in inst.rescue mode, or only in install mode? The port never opens, so this is the prime suspect — an auth problem would still show an open port.
• sshkey may apply only to the installed system, leaving the installer environment password-only via sshpw. That would matter once sshd does listen: the test authenticates key-only (BatchMode=yes).
• The %anaconda-context directives in debug.ks may be skipped entirely when a kickstart is supplied alongside inst.rescue.

Until this is resolved, provision debug gets you a booted rescue environment on the console (including on arm64), but not an SSH shell. The debug-setup.sh nc-listener path is the documented workaround and is unaffected.

---

Planned Work (Taskmaster)

13 tasks in queue, all pending:

1. #72 Expand Prisma schema with resource relationships (Network, ServerNic, ServerDisk, ClusterMember)
2. #73 State persistence service (bastion state -> CockroachDB)
3. #74 State loading from labd on bastion startup
4. #75 Fix bastion --dir env var default
5. #76 Resource type registry with aliases (kubectl-style)
6. #77 labctl get <resource> command
7. #78 labctl describe <resource> command
8. #79 labctl create/delete commands
9. #80 Refactor provision commands to kubectl-style
10. #81 Server and resource API endpoints in labd
11. #82 RBAC permission checks in CLI
12. #83 Audit logging for resource operations
13. #84 Update CLI entry point and help text

Additional items not in taskmaster:

• Ubuntu autoinstall disk auto-detect (still defaults to /dev/sda)
• Verify inst.sshd works end-to-end in rescue mode
• k3s cluster join vs new cluster distinction in labctl app k3s install
• arm64 container build (iPXE cross-compilation broken)