lab/STATUS.md

# labctl Platform — Implementation Status

## What This Document Is

An honest assessment of what code exists, what works, what is stubbed, and what
hasn't been started — measured against the PRD phases.

---

## Architecture Overview (as built)

```
labctl CLI ──HTTP──▶ bastion (PXE server)     ← WORKING
labctl CLI ──HTTP──▶ labd (master daemon)     ← PARTIALLY WORKING
                       │
                       ├── CockroachDB/Prisma  ← SCHEMA DEFINED, NOT DEPLOYED
                       ├── /ws/agent WebSocket  ← ACCEPTS CONNECTIONS, DOES NOT ROUTE
                       └── mTLS CA              ← NOT IMPLEMENTED

lab-agent ──WS──▶ labd                        ← LIBRARY CODE, NO DAEMON BINARY
```

---

## Package Inventory

| Package | Lines of Source | Tests | Status |
|---------|---------------|-------|--------|
| @lab/shared | ~200 | 0 | Complete — types, protocol, errors |
| @lab/bastion | ~800 | 32 | **Production-ready** — PXE discovery, install, reprovision |
| @lab/cli | ~600 | 0 (uses bastion tests) | Complete — all commands implemented |
| @lab/labd | ~500 | 2 | Partial — routes exist, core features stubbed |
| @lab/agent | ~300 | 0 | Library only — no daemon binary |

All 5 packages compile. 32 tests pass.

---

## Phase 1: Foundation

### DONE — Working in production

| Feature | Code | How It Works |
|---------|------|-------------|
| PXE bastion server | `src/bastion/` | Fastify HTTP + dnsmasq DHCP/TFTP. Machines PXE boot, get iPXE script from `/dispatch?mac=XX`, chain to discovery or install kickstart. State persisted to JSON file. |
| Machine discovery | `routes/dispatch.ts`, `templates/discover.ks.ts` | Unknown MACs get a mini-kickstart that boots a RAM-only Fedora, scrapes hardware via `/proc`, `/sys`, `dmidecode`, POSTs to `/api/discover`, then reboots. No disk touch. |
| Machine installation | `routes/api.ts`, `templates/install.ks.ts` | Queue a MAC via `POST /api/install`. Next PXE boot gets a full Kickstart with LVM partitioning (worker: longhorn LV, infra: rancher LV), SSH keys, k3s kernel prereqs, progress callbacks. |
| Reprovision with data preservation | `commands/reprovision.ts`, `install.ks.ts` | `%pre` script detects existing LVM. Reformats `/`, `/var`, `/boot` but preserves `/home`, `/srv`, `/var/lib/longhorn`, `/var/lib/rancher`. |
| CLI: init/provision commands | `src/cli/src/commands/` | `labctl init bastion standalone start/stop/status`, `labctl provision list/install/reprovision/forget`. All talk to bastion HTTP API. |
| CLI: config management | `config/index.ts`, `commands/config.ts` | `labctl config list/get/set/path`. YAML config at `~/.labctl/config.yaml` with env var overrides. |
| labd scaffold | `src/labd/` | Fastify server with health, server listing, token management routes. Prisma schema for all models. Starts with or without database. |
| Prisma schema | `prisma/schema.prisma` | 10 models: Server, Agent, User, Role, Permission, UserRole, JoinToken, AuditLog, PulumiRun, Cluster. CockroachDB provider. |
| Database seeding | `prisma/seed.ts` | Creates admin/viewer/operator roles with proper allow/deny permissions. Idempotent via upsert. |
| Multi-arch builds + packaging | `nfpm.yaml`, `scripts/` | nfpm config for RPM/DEB. Bun compile for standalone binary (102MB labctl in `dist/`). |
| Gitea CI/CD | `.gitea/` (on remote) | Lint → typecheck → test → build → publish pipeline on mysources.co.uk. |

### DONE — Code exists, not yet connected end-to-end

| Feature | Code | What's Real | What's Missing |
|---------|------|------------|----------------|
| lab-agent connection library | `lab-agent/src/services/connection.ts` | `AgentConnection` class: WebSocket to labd, heartbeat (10s), exponential backoff reconnect (1-30s), state machine (disconnected/connecting/connected/reconnecting), handles server-shutdown messages. | **No daemon binary.** This is a library — nothing starts it. No systemd unit. No enrollment flow. |
| lab-agent command executor | `lab-agent/src/services/executor.ts` | `CommandExecutor` class: `spawn()` with timeout handling (SIGTERM then SIGKILL after 5s), stdout/stderr streaming via EventEmitter, stdin writing, signal forwarding. | **Not wired to WebSocket.** The executor and connection don't talk to each other. No message dispatch. |
| Agent registry (labd) | `labd/src/services/agent-registry.ts` | `AgentRegistry`: in-memory Map tracking by serverId and hostname, lifecycle events, heartbeat updates. Singleton exported. | **Not used by /ws/agent handler.** The WebSocket handler in `server.ts` just logs messages — it doesn't call `agentRegistry.register()`. |
| Message router (labd) | `labd/src/services/message-router.ts` | `MessageRouter`: handler registration, pending request tracking with timeouts, streaming support, log subscription, agent cleanup on disconnect. | **Not used.** `server.ts` doesn't call `messageRouter.handleMessage()`. The router exists but is dead code. |
| Token management | `labd/src/routes/auth.ts` | Create, list, revoke join tokens. Validates one-time vs reusable, expiry, revocation. Marks tokens as used. | Token validation works. **But enrollment returns `certificatePem: null`** — no actual certificate is issued. |
| CLI API client | `cli/src/api/client.ts` | `LabdClient` with mTLS support, typed methods for servers/tokens/health/enrollment. | Works for REST endpoints. **No CLI commands use it yet** — existing commands still talk directly to bastion HTTP. |
| CLI WebSocket streaming | `cli/src/api/websocket.ts` | `streamExec()` and `streamLogs()` functions. | **No `labctl exec` or `labctl logs` commands exist.** The streaming code has no consumer. |
| Zod validation | `labd/src/validation/` | Schemas for createToken, enrollment, serverFilters, createRole, permission patterns. Middleware for body/query validation. | **Not applied to routes.** The schemas and middleware exist but no route uses `preHandler: [validateBody(schema)]`. |
| Encryption service | `labd/src/services/encryption.ts` | AES-256-GCM with scrypt key derivation. Encrypt/decrypt roundtrip. Singleton from `CA_ENCRYPTION_KEY` env var. | **Not used anywhere.** No CA key is encrypted, no kubeconfig is stored. |
| Graceful shutdown | `labd/src/services/shutdown.ts` | SIGTERM/SIGINT handlers, agent notification, message router cleanup, DB disconnect, force exit timer. | Works but agent notification is a no-op since no agents are registered (see above). |
| Rate limiting | `labd/src/middleware/rate-limit.ts` | `@fastify/rate-limit`: 100/min global, 10/min for enrollment, 20/min for tokens. | **Wired up in `server.ts`.** This actually works. |
| Health checks | `labd/src/routes/health.ts` | `/healthz`, `/health`, `/health/detailed`, `/health/live`, `/health/ready`. Checks DB latency and agent count. | Works. Returns `agents: { connected: 0 }` since no agents ever register. |
| Error hierarchy | `shared/src/errors/` | `LabError`, `NotFoundError`, `PermissionDeniedError`, `ValidationError`, `AgentNotConnectedError`. | **Not used in routes.** Routes still use inline `reply.code(404).send({error: ...})`. |
| Table formatting | `cli/src/utils/table.ts` | `printTable`, `formatStatus`, `formatRelativeTime`, predefined column sets. | **Not used by existing commands.** `provision list` has its own inline formatting. |
| Resource parsing | `cli/src/utils/resource.ts` | Parse `server/labmaster`, `app/kube-system/nginx` format. | **Not used.** No commands accept `type/name` arguments yet. |
| Doctor command | `cli/src/commands/doctor.ts` | Config, cert, connectivity diagnostics. | Works standalone. |
| Login command | `cli/src/commands/login.ts` | Generates EC keypair, prompts for token, POSTs to `/api/auth/user-enroll`. | **labd has no `/api/auth/user-enroll` endpoint.** Only `/api/auth/enroll` exists (for agents). Login will 404. |

### NOT DONE — Phase 1 items from PRD with no code

| Feature | PRD Description | Status |
|---------|----------------|--------|
| Certificate Authority | Built-in CA in labd. Generate root CA, sign CSRs, revoke certs, rotate. | **Nothing.** No CA code. No X.509 operations. No `@peculiar/x509` dependency. `EncryptionService` exists but it's for data-at-rest, not PKI. |
| RBAC engine | Middleware that checks permissions on every request. Deny overrides allow. | **Nothing.** `auth.ts` middleware is a placeholder. No route checks permissions. Anyone can call any endpoint. |
| Audit logging | Log every action with user, session, action, resource, result, duration. | **Nothing.** `AuditLog` Prisma model exists but nothing writes to it. No audit middleware. |
| `labctl exec` | Remote command execution via labd → agent WebSocket relay. | **Nothing.** No `exec` CLI command. The executor library exists in lab-agent but isn't connected. |
| `labctl logs` | Resource-scoped log streaming (server, app, bastion, audit). | **Nothing.** No `logs` CLI command. |
| `labctl get servers` | List servers from labd with filters. | **Nothing.** No `get` CLI command. The API client has `getServers()` but no command calls it. |
| Smoke test stack | `podman-compose` with CockroachDB + labd + 2 agents, testing enrollment/heartbeat/exec/RBAC. | **Nothing.** `stack/docker-compose.yml` exists but only runs bastion + CockroachDB, not labd or agents. |
| Agent enrollment during PXE | Embed join token in kickstart, agent auto-enrolls on first boot. | **Nothing.** Kickstart installs k3s prereqs but doesn't install or start lab-agent. |

---

## Phase 2: Deployment

**Nothing from Phase 2 has been built.**

| Feature | Status |
|---------|--------|
| Reprovision labmaster as labmaster.ad.itaz.eu | Not done — manual operation |
| Deploy k3s with Cilium CNI | Not done — kickstart only sets up kernel prereqs, leaves a comment "run `curl -sfL https://get.k3s.io`" |
| Deploy CockroachDB on k3s | Not done — `docker-compose.yml` runs it in-memory for dev, no k8s manifests for CRDB |
| Deploy labd on k3s | **K8s manifests exist** (`deploy/k8s/labd/base/`) — Deployment, Service, ConfigMap, HPA, PDB. But no CockroachDB to connect to and no TLS configured. |
| Deploy bastion as managed app | Not done — bastion runs standalone, no Pulumi chart |
| Auto-enroll agents during PXE | Not done — no agent install in kickstart, no token embedding |

---

## Phase 3: Infrastructure as Code

**Nothing from Phase 3 has been built.**

| Feature | Status |
|---------|--------|
| Module system | Not done — no `module.yaml`, no module loader |
| Pulumi charts | Not done — no Pulumi dependency, no chart structure |
| `labctl apps install/upgrade/rollback` | Not done — no `apps` command |
| `labctl apply -f` | Not done — no `apply` command |
| `kubectl proxy` (audited) | Not done — no kubectl proxy |
| Kubeconfig store (encrypted) | `EncryptionService` exists but nothing uses it. `Cluster.kubeconfigEnc` field exists in Prisma but nothing reads/writes it. |

---

## Phase 4: Multi-Cloud

**Nothing from Phase 4 has been built.**

| Feature | Status |
|---------|--------|
| AWS provider | Not done |
| Reusable join tokens for ASGs | Token model supports `reusable` type, but no AWS integration |
| Cilium Cluster Mesh | Not done |
| Ephemeral test environments | Not done |
| Grafana Loki | Not done |

---

## Infrastructure Files

| File | Status |
|------|--------|
| `Dockerfile.labd` | Exists. Multi-stage Alpine build. Would work if you `docker build` it. |
| `Dockerfile.bastion` | Exists. Multi-stage Fedora build. Would work. |
| `.dockerignore` | Exists. |
| `deploy/k8s/labd/base/` | Kustomize manifests for labd (Deployment, Service, ConfigMap, HPA, PDB). Points at a non-existent CockroachDB and has no TLS. |
| `stack/docker-compose.yml` | Runs bastion + CockroachDB for local dev. Works. |
| `nfpm.yaml` | RPM/DEB packaging config. Works with `nfpm pkg`. |

---

## The Disconnection Problem

The core issue is that many services were built in isolation but never wired together:

```
┌─────────────────────────────────────────────────────────┐
│  BUILT BUT NOT CONNECTED                                │
│                                                         │
│  AgentConnection ──✗──▶ /ws/agent handler               │
│  CommandExecutor ──✗──▶ MessageRouter                   │
│  MessageRouter   ──✗──▶ /ws/agent handler               │
│  AgentRegistry   ──✗──▶ /ws/agent handler               │
│  Zod schemas     ──✗──▶ Route preHandlers               │
│  Error classes   ──✗──▶ Route error handling             │
│  LabdClient      ──✗──▶ CLI commands (get/exec/logs)    │
│  Table formatting──✗──▶ CLI commands                    │
│  Resource parsing──✗──▶ CLI commands                    │
│  EncryptionService──✗──▶ CA / kubeconfig storage        │
│  Login command   ──✗──▶ /api/auth/user-enroll (missing) │
│  Audit logging   ──✗──▶ Any middleware                  │
│  RBAC engine     ──✗──▶ Any middleware                  │
└─────────────────────────────────────────────────────────┘
```

---

## What Actually Works End-to-End Today

1. **PXE boot a bare-metal machine:**
   ```
   labctl init bastion standalone start
   # Machine PXE boots → discovered automatically
   labctl provision list
   labctl provision install AA:BB:CC:DD:EE:FF worker-1 --role worker
   # Machine reboots → installs Fedora → reports complete
   ```

2. **Manage bastion lifecycle:**
   ```
   labctl init bastion standalone status
   labctl init bastion standalone stop
   ```

3. **Start labd (without database):**
   ```
   LABD_PORT=3100 tsx src/labd/src/main.ts
   # Starts with stub DB, health endpoint works, token/server routes return errors
   ```

4. **Start labd (with CockroachDB):**
   ```
   docker-compose -f stack/docker-compose.yml up cockroachdb
   DATABASE_URL=postgresql://root@localhost:26257/lab tsx src/labd/src/main.ts
   # Token creation/listing/revocation works
   # Server listing works (empty until agents register)
   ```

5. **CLI diagnostics:**
   ```
   labctl doctor
   labctl config list
   labctl version
   ```

That's it. No agent communication, no remote exec, no log streaming, no RBAC, no certificates.

---

## Recommended Next Steps (to make Phase 1 actually work)

### Priority 1: Wire up the agent connection
1. Update `/ws/agent` handler to use `agentRegistry.register()` and `messageRouter.handleMessage()`
2. Create lab-agent daemon binary that uses `AgentConnection` + `CommandExecutor`
3. Create systemd unit for lab-agent

### Priority 2: Certificate Authority
1. Add `@peculiar/x509` dependency
2. Implement CA service: generate root CA, sign CSRs
3. Wire enrollment route to actually sign and return certificates
4. Store CA key encrypted using `EncryptionService`

### Priority 3: RBAC + Audit
1. Create RBAC middleware that checks `Permission` table
2. Create audit middleware that writes to `AuditLog`
3. Apply both to all routes

### Priority 4: CLI commands for labd
1. `labctl get servers` using `LabdClient.getServers()`
2. `labctl exec server/<name>` using `streamExec()`
3. `labctl logs server/<name>` using `streamLogs()`

### Priority 5: Smoke test stack
1. Update `docker-compose.yml` to include labd + 2 agents
2. Write integration tests for enrollment → heartbeat → exec → logs