lab/bastion/docs/ARCHITECTURE.md

# Lab Platform Architecture

## Overview

A bare-metal and hybrid cloud infrastructure platform for automated machine provisioning, Kubernetes cluster management, and fleet operations. The platform discovers hardware via PXE boot, installs operating systems unattended, deploys k3s clusters, and provides centralized management through a CLI and API.

**Components:**
- **bastion** -- PXE boot server (DHCP/TFTP/HTTP) for machine discovery and OS installation
- **labd** -- Master daemon for multi-bastion aggregation, persistent state, agent management
- **labctl** -- CLI tool for operators (kubectl-style interface)
- **lab-agent** -- Daemon on provisioned servers for remote execution and monitoring
- **modules** -- Declarative configuration system (k3s, labcontroller)

---

## Architecture

```
                           labctl (CLI)
                               |
                        labd (master daemon)
                        /    |    \
                  bastion1  bastion2  ...     (PXE provisioning)
                    /   \      |
              [machines]  [machines]           (bare metal)
                  |           |
              lab-agent   lab-agent            (remote exec)
```

### Communication Patterns

| Path | Protocol | Auth |
|------|----------|------|
| labctl -> labd | HTTP/HTTPS | mTLS cert (future: token) |
| bastion -> labd | WebSocket | Join token enrollment |
| lab-agent -> labd | WebSocket | mTLS certificate |
| machine -> bastion | HTTP | None (local network) |
| Anaconda -> bastion | HTTP + UDP syslog | None (install-time) |
| labctl -> bastion | HTTP | None (standalone mode) |

### Standalone vs Centralized

The bastion can operate in two modes:

1. **Standalone** -- single bastion, state in local JSON file, CLI talks directly to bastion HTTP API
2. **Centralized** -- bastion registers with labd via WebSocket, state aggregated in CockroachDB, CLI talks to labd which routes commands to the correct bastion

---

## Machine Lifecycle

```
                     PXE boot
                        |
               +--------v--------+
               |    DISCOVERED   |  Hardware inventory collected
               +---------+-------+
                         |
              labctl provision install
                         |
               +---------v-------+
               |  INSTALL_QUEUE  |  Waiting for next PXE boot
               +---------+-------+
                         |
                   PXE boot (Anaconda)
                         |
               +---------v-------+
               |   INSTALLING    |  Progress: partitioning -> packages -> post-install
               +---------+-------+
                         |
               +---------v-------+
               |    INSTALLED    |  OS ready, SSH accessible
               +---------+-------+
                         |
              labctl app k3s install
                         |
               +---------v-------+
               |   K3S RUNNING   |  Kubernetes node operational
               +--------+--------+
                        |
              labctl provision reprovision
                        |
                   (back to INSTALL_QUEUE)
```

Side paths:
- **DEBUG** -- `labctl provision debug` boots Anaconda rescue mode for diagnostics
- **FORGET** -- `labctl provision forget` removes machine from all state

---

## Multi-architecture PXE

The bastion serves both `x86_64` and `aarch64` over the network. Nothing about this is
operator-configured -- there is no `--arch` flag, by design.

### How a client's architecture is decided

1. **DHCP option 93** (Client System Architecture) picks the *bootloader*. dnsmasq matches
   it and hands out a matching iPXE binary:

   | Option 93 | Client | Served |
   |---|---|---|
   | `0` | x86 BIOS | `undionly.kpxe` (TFTP) |
   | `7`, `9` | x64 UEFI | `ipxe.efi` (TFTP) |
   | `11` | **ARM64 UEFI** | `ipxe-arm64.efi` (TFTP) |
   | `16` | x64 UEFI HTTP Boot | `http://…/ipxe.efi` |
   | `19` | **ARM64 UEFI HTTP Boot** | `http://…/ipxe-arm64.efi` |

   Values come from the IANA Processor Architecture Types registry. Note `19`, not `20` --
   `20` is *pc/at bios boot from http*. EDK2/AAVMF prefers HTTP Boot over TFTP PXE, so the
   iPXE binaries are staged in **both** `tftpDir` and `httpDir` (symlinked by `main.ts`).

2. **`/dispatch` picks the kernel.** Option 93 never reaches the HTTP endpoint, so
   `boot.ipxe` passes iPXE's own `${buildarch}` as `?arch=`. `resolveArch()` prefers, in
   order: the tracked machine record → the reported `?arch=` → the configured default.
   The record wins because it is what we observed on the machine itself.

### Artifact naming

`x86_64` keeps the original unsuffixed paths so its rendered iPXE scripts are unchanged;
everything else is suffixed. `kernelPath()` / `initrdPath()` in `templates/boot.ipxe.ts`
are the single source of truth, used by both the templates and `main.ts` staging.

| arch | kernel | initrd |
|---|---|---|
| `x86_64` | `/vmlinuz` | `/initrd.img` |
| `aarch64` | `/vmlinuz-aarch64` | `/initrd-aarch64.img` |

`tests/ipxe-x86-regression.test.ts` pins the x86_64 output against a golden fixture.

### arm64 gotchas

- **LoadFile2 is mandatory.** arm64 has no `HdrS` boot protocol; the kernel's EFI stub
  fetches the initrd over the UEFI `EFI_LOAD_FILE2_PROTOCOL`. An iPXE build without it
  accepts the `initrd` line, silently drops it, and the kernel panics with
  `VFS: Unable to mount root fs on unknown-block(0,0)`. Fedora's
  `ipxe-bootimgs-aarch64` implements it; the integration test asserts this up front so
  the failure names itself instead of looking like a disk problem.
- **`nomodeset` is x86-only.** On arm64 there is no VGA path to fall back to. aarch64 gets
  `console=tty0 console=ttyAMA0,115200` instead — the last `console=` wins for
  `/dev/console`, so serial is the interactive one.
- **Ubuntu is x86_64-only.** `releases.ubuntu.com` publishes no arm64 netboot artifacts.
  `osSupportsArch()` encodes this, and both the install guard and `/dispatch` refuse the
  combination rather than serving an x86 kernel to an ARM machine.

---

## Onboarding classification (vendor OS)

Machines carry an `onboard` field: `"pxe"` (default) or `"ssh"`, plus `vendor_os` naming
what they run. `classifyOnboard()` in `@lab/shared` sets it from DMI identity, with known
hardware also matched by MAC — a machine can sit in state for a long time with no DMI, and
a DMI-only rule would fail open exactly where it matters.

`onboard: "ssh"` means *we cannot rebuild this machine's OS*. Installs are refused at both
entry points (`/api/install` and the labd `command-install` handler) with an error naming
the machine and pointing at `provision debug`. **Rescue is never guarded** — being unable
to reinstall a machine is precisely when a rescue shell is needed.

This is a fact about the machine, not a blocklist. The refusal follows from "no image in
our pipeline restores `vendor_os`", so adding a DGX OS image to the pipeline is what
unblocks the DGX Sparks — no entry needs deleting.

Current classifications: NVIDIA DGX Spark (`spark-2935`, `spark-3a1c`) → `dgx-os`.

---

## Packages

### Monorepo Structure

TypeScript ESM monorepo with pnpm workspaces. Six packages:

| Package | Role | Key Tech |
|---------|------|----------|
| `@lab/shared` | Types, protocol, constants | - |
| `@lab/bastion` | PXE server | Fastify, dnsmasq |
| `@lab/cli` | CLI binary | Commander.js |
| `@lab/labd` | Master daemon | Fastify, Prisma, CockroachDB |
| `@lab/agent` | Server agent | WebSocket |
| `@lab/modules` | Config modules | SSH, k8s-client |

### @lab/shared

Core type system shared by all packages.

**State Model:**
```typescript
interface BastionState {
  discovered:    Record<MAC, HardwareInfo>
  install_queue: Record<MAC, InstallConfig>
  installed:     Record<MAC, InstalledInfo>
  debug:         Record<MAC, DebugConfig>
}
```

**Roles:**
- `vanilla` -- OS only, no k3s, no cluster services
- `worker` -- k3s agent + Longhorn storage (joins existing cluster)
- `infra` -- k3s server + etcd (control plane node)
- `labcontroller` -- infra + bastion + labd + CockroachDB (self-sufficient)

**OS Support:**
- `fedora-43` -- Anaconda kickstart installer
- `ubuntu-26.04` -- cloud-init autoinstall

**Protocol:** Discriminated union message types for WebSocket communication between agents, bastions, and labd. Type guards and parsers for runtime validation.

### @lab/bastion

PXE boot server that handles the physical provisioning lifecycle.

**Services:**
- `StateManager` -- JSON file persistence with immutable update pattern
- `SyslogListener` -- UDP syslog receiver (port 5514) for Anaconda install logs
- `InstallLogBuffer` -- In-memory ring buffer + disk persistence per machine
- `BastionConnection` -- WebSocket client to labd for centralized mode
- dnsmasq management (spawn, config generation, proxy/full DHCP)
- Network auto-detection (interface, IP, subnet, gateway)
- ISO builder (xorriso + mtools for non-PXE machines)

**HTTP Routes:**

| Endpoint | Purpose |
|----------|---------|
| `GET /dispatch?mac=` | Dynamic iPXE script (discover/install/debug/local-boot) |
| `GET /ks?mac=` | Per-machine Anaconda kickstart |
| `GET /debug.ks` | Rescue mode kickstart |
| `GET /debug-setup.sh` | nc listener setup script for rescue shell |
| `GET /discover.ks` | Hardware discovery kickstart |
| `POST /api/discover` | Hardware inventory report |
| `POST /api/install` | Queue machine for install |
| `POST /api/progress` | Install progress callback |
| `POST /api/log` | Raw log line ingestion |
| `POST /api/debug` | Queue debug/rescue mode |
| `GET /api/machines` | List all machines |
| `GET /api/logs/:mac` | Install logs + progress |
| `GET /api/logs/:mac/follow` | SSE stream of progress events |
| `DELETE /api/machines/:mac` | Forget machine |

**Templates:**
- `boot.ipxe.ts` -- iPXE scripts for each boot mode (discover, install, debug, pxe-boot-debug, local-boot)
- `install.ks.ts` -- Full Fedora kickstart with LVM, SSH, k3s prereqs, progress callbacks, SysRq keys
- `debug.ks.ts` -- Minimal rescue kickstart (SSH via inst.sshd)
- `ubuntu-autoinstall.ts` -- cloud-init for Ubuntu
- `dnsmasq.conf.ts` -- DHCP/TFTP configuration

**Boot Dispatch Logic:**
```
1. debug[mac]?        -> renderDebugIpxe (auto-clear after serving)
2. install_queue[mac]? -> renderInstallIpxe
3. installed[mac]?    -> renderLocalBootIpxe (exit to disk)
4. unknown            -> renderDiscoverIpxe
```

### @lab/labd

Central management daemon. Aggregates multiple bastions, stores persistent state in CockroachDB, relays commands, manages agent fleet.

**Database (Prisma + CockroachDB):**
- `Server` -- hostname, MAC, IP, role, status, cloud, environment, labels
- `Bastion` -- hostname, network, serverIp, lastHeartbeat
- `Agent` -- certificate, enrollment, heartbeat
- `Cluster` -- name, cloud, environment, kubeconfig (encrypted)
- `User` / `Role` / `Permission` -- RBAC (action:cloud:env:server matrix)
- `JoinToken` -- one-time/reusable enrollment tokens
- `AuditLog` -- action, resource, result, timestamp

**Key Services:**
- `BastionRegistry` -- in-memory registry of connected bastions, state aggregation, MAC-to-bastion routing
- `AgentRegistry` -- connected agents, heartbeat tracking
- `MessageRouter` -- command relay between CLI/agents and bastions

**Command Routing:**
```
CLI: labctl provision install <mac> <hostname>
  -> POST /api/machines/install
  -> labd finds bastion that knows this MAC
  -> WebSocket: {type: "command-install", mac, hostname, disk, role}
  -> bastion updates install_queue
  -> WebSocket: {type: "command-response", status: "ok"}
  -> HTTP response to CLI
```

### @lab/cli (labctl)

Operator CLI. Commander.js binary, distributed as RPM/DEB or standalone bun-compiled executable.

**Command Groups:**

```
labctl init bastion standalone start|stop|status
labctl provision list|install|reprovision|forget|debug|logs|makeiso
labctl app k3s install|health|list
labctl config list|get|set|path
labctl login
labctl doctor
labctl roles
```

**Key Features:**
- Target resolution: hostname, MAC, or IP -> machine lookup
- SSH reboot into PXE for reprovision/debug (efibootmgr --bootnext)
- Follow mode: `labctl provision logs <target> -f` (5s polling)
- Shell completions: bash, fish

### @lab/modules

Declarative configuration modules with three-phase lifecycle: install -> configure -> health.

**k3s Module:**
- 5 operation groups: host-prep, networking, k3s-server, k3s-agent, hardening
- 15+ individual operations: kernel modules, sysctl, firewall, Cilium CNI, SELinux, audit policy, pod security, cert checks
- Health checks: service running, node ready, API health, pod status, Cilium status, secrets encryption
- SSH execution backend with progress callbacks

### @lab/agent

Daemon on provisioned servers. WebSocket to labd for:
- Heartbeat (hostname, uptime, CPU/mem usage)
- Command execution (with stdout/stderr streaming)
- Log streaming (journalctl relay)
- mTLS certificate enrollment and rotation

---

## Disk Layout

### LVM Partitioning (labvg)

All roles share a common LVM layout. The kickstart `%pre` auto-detects the install disk (NVMe preferred, then SATA, skipping USB/removable).

| Volume | Size | FS | Reprovision |
|--------|------|-----|-------------|
| `/boot/efi` | 600 MB | vfat | Reused |
| `/boot` | 3 GB | ext4 | Reused |
| `swap` | 27 GB | swap | Recreated |
| `/` (root) | 33 GB | xfs | Recreated |
| `/var` | 100 GB | xfs | Recreated |
| `/var/log` | 10 GB | xfs | Recreated |
| `/home` | 10 GB | xfs | **Preserved** |
| `/srv` | 20 GB | xfs | **Preserved** |
| `/var/lib/longhorn` | remaining | xfs | **Preserved** (worker) |
| `/var/lib/rancher` | 20 GB | xfs | **Preserved** (infra) |
| `/tmp` | 4 GB | tmpfs | - |

Reprovision detection: if `labvg` VG exists, reuse EFI/boot partitions and preserve data volumes.

---

## Kickstart Features

The Fedora kickstart template (`install.ks.ts`) includes:

- **Dynamic disk detection** -- `%pre` probes NVMe/SATA/virtio, skips USB/removable, supports both fresh install and reprovision
- **Progress callbacks** -- `curl -sf POST /api/progress` at each stage (partitioning, post-install substeps, complete)
- **Anaconda syslog forwarding** -- `logging --host --port` streams real-time install logs to bastion
- **SSH hardening** -- key-only auth, root login via pubkey only, admin user with passwordless sudo
- **Network-first boot order** -- `efibootmgr` reorders boot entries so PXE is always first (bastion controls every reboot)
- **SysRq magic keys** -- `kernel.sysrq=1` for emergency reboot via KVM keyboard
- **Role-specific setup:**
  - `vanilla`: chronyd only
  - `worker`/`infra`: kernel modules (br_netfilter, overlay), sysctl (ip_forward, inotify), firewalld disabled, k3s binary installed
  - `infra`: k3s server binary pre-installed

**What is NOT in the kickstart:**
- `console=ttyS0` -- causes 30s-per-step boot timeout on hardware without physical serial UART (discovered 2026-03-30, see docs/pxe-boot-debugging-2026-03-30.md)
- Background log streamer (`tail -f`) -- prevents Anaconda from syncing filesystem, causes %post writes to not persist

---

## Deployment

### Container Images

**bastion** (`Dockerfile.bastion`):
- Base: Fedora 43 (needs dnsmasq, iPXE)
- Multi-stage: Alpine build -> Fedora runtime
- iPXE rebuilt from source (SNP driver for EFI)
- hostNetwork in k8s (DHCP needs raw sockets)
- Capabilities: NET_ADMIN, NET_RAW

**labd** (`Dockerfile.labd`):
- Base: Alpine (minimal)
- Multi-stage build with Prisma client generation
- Runs as non-root `node` user

### Kubernetes (k3s)

```
Namespace: lab-infra
  Deployment: bastion (hostNetwork, PVC for /data, host SSH keys)
  ConfigMap: bastion-config (env vars)
  Secret: bastion-join-token
  PVC: bastion-state (local-path)

Namespace: lab-system
  Deployment: labd
  Service: labd (NodePort 30100)
  StatefulSet: cockroachdb-0
```

### CLI Distribution

Built with `nfpm` as RPM/DEB. Includes:
- `/usr/bin/labctl` (bun-compiled standalone binary)
- `/usr/share/bash-completion/completions/labctl`
- `/usr/share/fish/vendor_completions.d/labctl.fish`

Config: `~/.labctl/config.yaml` with `labdUrl`, output format, default cloud/environment.

---

## Build & Release

```bash
# Development
pnpm install && pnpm build      # Compile all packages
pnpm test:run                    # Unit tests (vitest)
npx tsc --noEmit                 # Type check

# Deploy
bash scripts/deploy.sh all       # Build containers + RPM, push, restart pods
bash scripts/deploy.sh bastion   # Just bastion
bash scripts/deploy.sh labd      # Just labd
bash scripts/deploy.sh labctl    # Just CLI (local RPM install)

# Container builds
bash scripts/build-bastion.sh --platforms linux/amd64 --push latest
bash scripts/build-labd.sh --platforms linux/amd64 --push latest
bash scripts/build-rpm.sh        # RPM + DEB packages

# Integration tests (require libvirt, sudo)
sudo tests/integration/run-pxe-test.sh
```

Registry: `mysources.co.uk` (Gitea at 10.0.0.194:3012)

---

## Testing

### Unit Tests
- Kickstart rendering (ksvalidator syntax check, partition layout, role-specific sections)
- State management (load, save, update, debug field)
- Dispatch routing (correct iPXE script for each machine state)
- Syslog listener (UDP receive, IP->MAC resolution, RFC 3164 parsing)

### Integration Tests (libvirt VMs)
- **pxe-provision.test.ts** -- Full end-to-end: create VM -> PXE discovery -> queue install -> Anaconda install -> SSH verification -> systemd health -> SELinux enforcing -> boot order check
- **iso-provision.test.ts** -- ISO boot for non-PXE machines
- **k3s-single-node.test.ts** -- Post-provision k3s installation and health
- VM screenshot capture during boot for debugging

---

## Security

- **mTLS** for agent-labd communication (certificate enrollment via join tokens)
- **SSH key-only auth** on provisioned machines (no password auth)
- **SELinux enforcing** verified in integration tests
- **RBAC** (planned): action:cloud:environment:server permission matrix
- **Audit logging** (planned): every mutation tracked in CockroachDB
- **Network-first boot order** prevents machines from booting without bastion approval
- **SysRq keys** enabled for emergency reboot without SSH access

---

## Known Issues & Lessons Learned

### Serial Console Boot Delay (2026-03-30)
`console=ttyS0,115200n8` in kernel cmdline causes 30-second timeout at every systemd boot phase on hardware without a physical serial UART. Root cause: systemd blocks writing to non-existent UART. Fix: removed from kickstart entirely.

### Anaconda %post Log Streamer
Background `tail -f` in kickstart `%post` prevents Anaconda from syncing the filesystem. All file writes in %post appear to succeed but are lost on reboot. Fix: removed background log streamer, replaced with Anaconda's built-in `logging --host --port` syslog forwarding.

### Disk Auto-Detection
Hardcoded `/dev/sda` default broke NVMe-only machines. Fix: default to empty string (auto-detect) which triggers the `%pre` disk probe logic.

### Anaconda Rescue Mode Limitations
`%pre` and `%post` sections do not execute in `inst.rescue` mode. SSH in rescue mode is provided by Anaconda's `inst.sshd` kernel parameter + `sshpw` kickstart directive. Manual setup via `curl bastion:8080/debug-setup.sh | bash` for nc listener.

**Unresolved (2026-08-11): rescue SSH has never been observed working.** Adding the first
integration coverage for `provision debug` (`tests/integration/pxe-rescue.test.ts`) showed the
rescue environment coming up correctly — the bastion serves the kernel and initrd, Anaconda
boots, fetches `debug.ks`, and reaches its installer environment — but **nothing ever listens on
port 22**.

Strength of the evidence, stated precisely because it decides where to look next:
- **aarch64 — direct.** Port 22 probed every 20s for 30 minutes while the Anaconda installer
  environment was demonstrably running (NetworkManager, polkitd, rsyslog on the console). Never
  opened.
- **x86_64 — corroborating, not conclusive.** One clean KVM run (943s) where SSH never became
  available inside a 15-minute budget. That VM's progress into the rescue environment was *not*
  observed — vitest's final reporter discards the streamed log — so it is consistent with the
  aarch64 result but does not independently prove it. Re-run with `KEEP_VM=1` and probe port 22
  directly to settle it.

If the x86_64 result holds up, this is orthogonal to the multi-architecture work, since x86_64 is
untouched by it. Leads worth checking, in order:
- Does `inst.sshd` actually start `sshd` in `inst.rescue` mode, or only in install mode? The
  port never opens, so this is the prime suspect — an auth problem would still show an open port.
- `sshkey` may apply only to the *installed* system, leaving the installer environment
  password-only via `sshpw`. That would matter once sshd does listen: the test authenticates
  key-only (`BatchMode=yes`).
- The `%anaconda`-context directives in `debug.ks` may be skipped entirely when a kickstart is
  supplied alongside `inst.rescue`.

Until this is resolved, `provision debug` gets you a booted rescue environment on the console
(including on arm64), but not an SSH shell. The `debug-setup.sh` nc-listener path is the
documented workaround and is unaffected.

---

## Planned Work (Taskmaster)

13 tasks in queue, all pending:

1. **#72** Expand Prisma schema with resource relationships (Network, ServerNic, ServerDisk, ClusterMember)
2. **#73** State persistence service (bastion state -> CockroachDB)
3. **#74** State loading from labd on bastion startup
4. **#75** Fix bastion --dir env var default
5. **#76** Resource type registry with aliases (kubectl-style)
6. **#77** `labctl get <resource>` command
7. **#78** `labctl describe <resource>` command
8. **#79** `labctl create/delete` commands
9. **#80** Refactor provision commands to kubectl-style
10. **#81** Server and resource API endpoints in labd
11. **#82** RBAC permission checks in CLI
12. **#83** Audit logging for resource operations
13. **#84** Update CLI entry point and help text

Additional items not in taskmaster:
- Ubuntu autoinstall disk auto-detect (still defaults to /dev/sda)
- Verify `inst.sshd` works end-to-end in rescue mode
- k3s cluster join vs new cluster distinction in `labctl app k3s install`
- arm64 container build (iPXE cross-compilation broken)