Groundwork for replacing the USG with the VyOS pair without anything on the
network noticing. Three pieces:
migration/unifi-export.py pulls 13 endpoints off the classic controller into
timestamped JSON plus a normalised inventory: 11 networks, 31 DHCP
reservations, 4 port forwards, 2 firewall rules, 0 static routes. Two things
this turned up that a naive export would have lost:
- 23 of the 31 reservations carry no network_id at all -- UniFi simply does
not store the binding -- so they are resolved by subnet containment
instead. Without that, three quarters of the reservations have no subnet
to be placed in.
- 30 of the 31 sit INSIDE the DHCP pool, which UniFi's dhcpd tolerates and
which is flagged as a warning rather than discovered at cutover.
migration/unifi-to-vyos.py turns that inventory into VyOS `set` commands for
DHCP and DNS only -- the services the USG owns that VyOS must reproduce. Not a
general converter. --prod and --sim come from one code path so the config
proven in the sim and the config applied to the firewalls cannot drift. Prod
mode hard-fails if any reservation is missing, since a silent drop is the
failure mode that matters.
DNS is included because the USG resolves for 5 of 6 VLANs today: UniFi hands
out the gateway's own address whenever dhcpd_dns is empty, verified by
labmaster resolving against 192.168.8.1. Replacing the USG without a forwarder
would take DNS away from those VLANs entirely.
labsim/labsim-dhcp-test.sh proves it by booting throwaway VMs with real
production MACs -- the one piece of production config that transplants
verbatim. Safe because ovs-labsim has no physical NIC, so those MACs cannot
reach the real LAN.
Result on VyOS 2026.08 (kea), 4/4: printer1 got 172.31.10.46 from inside the
pool, sonoff-matter got 172.31.11.67 across the /23 boundary, Hubitat got its
out-of-pool .2, and an unreserved MAC got an unreserved address. kea honours
in-pool host reservations -- the open question blocking the cutover.
Supporting changes to labsim:
- VLAN 10 widened to /23. Every reservation is in LoT and LoT spans 10.0.0.x
and 10.0.1.x, which a /24 cannot represent.
- LoT's host leg moved to .3, because 10.0.0.2 is a real reservation
(Hubitat) that maps onto the host's own address.
- vlans.conf gained optional masklen and host_octet fields, defaulting to
24 and 2 so the other five VLANs are untouched.
- Fixed /etc/network/interfaces hardcoding 255.255.255.0. That file is what
actually takes effect on these Alpine guests -- cloud-init's
network-config is ignored -- so any non-/24 VLAN was silently wrong.
Two generator bugs found by VyOS rejecting the output: static-mapping names
are validated as hostnames, so underscores fail; and two devices named
"espressif" plus two named "thebeast" collided into single names, which would
have overwritten one reservation with another's address.
The raw export holds WiFi passphrases and the WAN PPPoE credentials and is
gitignored.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
Replace the CIS-style disableSwap op with enableSwap: activate the
labvg-swap LV with an fstab entry (kubelet runs failSwapOn=false; zram
stays the fast tier, the LV is overflow before OOM kill). Add
growRancherLv: extend labvg/rancher to 120G when the VG has free space,
covering nodes installed before the kickstart sizing change and vanilla
nodes converted to k8s; skips with a clear message when the VG is full.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017f6jyeeDqP4ufyeL3UER9w
The 20G /var/lib/rancher LV (k3s imageFs) idled at 85% used from
steady-state images alone; one ~5G image pull tripped imagefs eviction
and evicted unrelated pods (2026-08-14 DiskPressure incident). Create
the LV for both worker and infra roles at 120G — it must be sized here
because longhorn's --grow consumes all remaining VG space, making
post-install lvextend impossible.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017f6jyeeDqP4ufyeL3UER9w
The Grafana heatmap of 1s and 0s said almost nothing, and the state timeline
was an unreadable pile of overlapping series labels. Replaced as the primary
view with a purpose-built page served by the exporter itself.
- Probe now captures ICMP RTT, exposed as labsim_rtt_ms{src,dst}. A path that
is up but slow is a different problem from one that is down, and a pass/fail
grid cannot show it.
- Exporter serves / (topology), /api/matrix (JSON) and /metrics.
- topology.html: node per VLAN in a ring, VyOS router in the centre because
every inter-VLAN packet really does traverse it, one line per pair coloured
green/red with the RTT on it. Hovering gives per-direction state. A node ring
goes red if anything to or from it is blocked. Side panels list blocked paths
and the slowest links. Refreshes every 5s, no dependencies.
Grafana stays for what it is actually good at — history of when a path flipped.
Label placement is deliberate: RTT captions sit ~32% along each edge with a
perpendicular nudge, because every diagonal of a 6-node mesh crosses the centre
and midpoint labels stack on the router node.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
A throwaway copy of the production VLAN topology so routing and firewall
changes can be tested before they touch the real network. Same VLAN IDs and
roles as UniFi, deliberately different ranges (172.31.<vlan>.0/24) so nothing
here can be mistaken for production.
- OVS fabric: real 802.1Q. Access port per micro VM, host leg per VLAN (.2,
for SSH only — NOT the VMs' default route, so inter-VLAN tests exercise the
router rather than the host's routing table), and a trunk portgroup with
VLAN 1 declared nativeMode='untagged'.
- Six Alpine micro VMs (256MB, copy-on-write overlays on one 176MB image),
SSH + a hello-world HTTP page naming the VLAN.
- VyOS router installed to disk unattended over the console, with the SAME
config shape as the VP2440s: two NICs in an LACP bond carrying the trunk,
VLAN 1 native, bond0.<vlan> holding the .1 gateway on each.
- labsim-matrix.py: full-mesh ICMP/TCP22/TCP80 probe, ~0.2s, --watch
highlights cells that changed since the last sweep. Guest-side probe is
python3 (already present via cloud-init) so nothing is installed on VMs
that have no internet.
- Prometheus + Grafana (anonymous auth, no login) with a provisioned
dashboard: heatmap plus a state timeline showing exactly when a path
flipped. Verified end to end: one VyOS rule took sum(labsim_reachable)
from 90 to 84, blocking precisely kvm<->k8s across all three protocols.
Traps found building this, all now encoded in the scripts:
- virtio-net breaks 802.3ad: the guest's bonding driver reports slaves
"MII Status: down" despite carrier=1 and never sends an LACPDU, so the bond
sits in AD_STATE_DEFAULTED. e1000e fixes it with no other change. Matches
the netdev thread "bonding (IEEE 802.3ad) not working with qemu/virtio".
- OVS defaults bonds to active-backup, which does not speak LACP at all —
bond_mode=balance-tcp is required.
- LACP deadlock: OVS holds members disabled until negotiation while the
partner needs carrier before it will send LACPDUs. lacp-fallback-ab breaks it.
- LACPDUs are untagged, so a trunk with no native VLAN has nowhere to put them.
- --boot cdrom,hd re-runs the ISO on every restart, so every commit+save went
to a live system that evaporated. Install now switches the VM to boot hd.
- cloud-init on Alpine: users stay locked without lock_passwd:false, one
failing runcmd aborts the rest, busybox here has no httpd applet, and
start-stop-daemon --exec /usr/bin/python3 matches cloud-init's own python3.
- The user-data heredoc is unquoted, so backticks in a COMMENT were executed
by the host shell and their output corrupted the YAML. build_seed now
validates with yaml.safe_load before building the ISO.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
Both Protectli VP2440s failed to install on real hardware: they fetched
kernel+initrd and then went silent. The console showed why —
Looking for a connected Ethernet interface ... e2 ? e3 ? e4 ? e5 ?
Connected e4 found
Connected e5 found
[4.595647] igc 0000:02:00.0 e2: NIC Link is Up
IP-Config: e4 ... no response after 15 secs - giving up
Unable to find a live file system on the network
live-boot picks the first *connected* interface. The i40e SFP+ pair links
before the igc copper port (up at 4.6s), so it chose the fiber ports, which
have no DHCP, and never tried the NIC that actually PXE booted.
Fix: pass BOOTIF=01-<mac> on the kernel cmdline. live-boot's
Device_from_bootif() (verified present in this image) matches it against
/sys/class/net and sets DEVICE directly. The MAC comes from the dispatch
key — i.e. exactly the NIC that PXE booted — which is more reliable than
iPXE's ${net0} on a box where the booting NIC may not be net0.
Why the integration test missed it: the VM had ONE NIC, so "first connected
interface" was trivially correct, and virtio links instantly so there was no
negotiation race. createPxeVm now takes decoyNics, attaching extra NICs
ahead of the PXE NIC on a network with no route to the bastion; the VyOS
test uses 2. Without BOOTIF that reproduces the hardware failure. getVmMac
is network-aware so it still returns the booting NIC.
Also: the bastion had every clue and said nothing — it logged INSTALL
STARTED, served kernel+initrd, then nothing for 7 minutes. dispatch now
stamps dispatched_at, and /api/logs/:mac returns stalled_for_s / stalled
(8 min threshold, sized for the ~600MB squashfs fetch), so a machine wedged
before the installer environment comes up is diagnosable without a console.
Verified on hardware: both firewalls installed, bond0 802.3ad + VLANs
2/3/9/10/200 + VRRP (priority 200/100, VIP .254 per VLAN) applied, and
/config/lab-provisioned written.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
pnpm completions:check was failing: labctl.fish/bash were stale. The
generated --os choices still listed only fedora-43 and ubuntu-26.04
(missing vyos-rolling since the OsId union gained it), and none of the
--vyos-*/--vlan flags were present.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
Ports the Fedora provisioning features that matter for a router onto the
VyOS path, and adds the libvirt integration test that proves them.
- Live install logs: the driver streams the installer pty (ANSI-stripped,
batched, best-effort) to POST /api/log, so `labctl provision logs -f`
works during a VyOS install the way Anaconda's syslog does for Fedora.
- installed.ip: report "ready at <ip>" -- the exact detail format
routes/api.ts parses -- using the static mgmt address when known, else
the live DHCP address. Without it VyOS machines landed with an empty IP,
breaking provision list, logs-by-IP, recheck and reprovision.
api.ts also guards the complete handler: VyOS boxes get the "vyos" SSH
hint and never trigger the k3s post-provision.
- EFI network-first boot order: port of the Fedora %post efibootmgr step,
run from the live env after install (NVRAM, not disk). Best-effort.
- Reinstall semantics: VyOS's installer already carries the previous
config and SSH host keys forward -- the analog of Fedora's LV
preservation -- so that stays the default. New --vyos-fresh-config
overwrites the installed config.boot with the generated one instead,
via a post-install target mount that also writes /config/lab-provisioned
(mirrors Fedora's /etc/lab-provisioned, survives image upgrades).
- reprovision/recheck default to the "vyos" SSH user for VyOS machines.
Two hangs found by the VM test and fixed:
- On reinstall the installer asks "Would you like to copy data to the new
image?" (search_previous_installation). Unanswered, the driver blocked
on stdin until its stall timeout -- a silent 15-minute hang.
- The RAID regex missed "Would you like to choose two disks for RAID-1
mirroring?", which would wedge any multi-disk box. Both prompts default
to yes, so a miss also risks an unwanted mirror.
Both are now covered by a unit test asserting all 17 installer prompts
match exactly one rule -- verified to fail against the unfixed code, so
this class of bug is caught in a second instead of a 45-minute VM run.
tests/integration/vyos-provision.test.ts: fresh install, reinstall
preserves config + /config data, and freshConfig override. All 8 pass
against the real nightly ISO (EXIT=0). 273 unit tests pass; no new lint
errors in touched files.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
VyOS ships no unattended installer (install_image() is unconditionally
interactive; --no-prompt is wired only to 'add'), so the automation is
injected through live-config's hooks component: iPXE boots the live
kernel with fetch= and live-config.hooks=, the hook fetches a generated
per-MAC Python driver, and the driver builds config.boot, stages the
rootfs, and drives the interactive installer over a pty.
Bastion:
- vyos-boot.ipxe template (no 'nonetworking' — breaks the hook fetch;
no console=ttyS0 — 30s/systemd-phase on UART-less boards)
- /vyos/autoinstall.sh + /vyos/install.py routes (per-MAC driver with
the config spec baked in as base64)
- vyos-config-spec: bond0 (802.3ad) + tagged VLANs + VRRP groups
(vrid = VLAN id) + sync group + hw-id pinning by MAC + SSH keys;
config built from the image's own config.boot.default via
vyos.configtree, version footer reattached via component_version
- prepareVyosArtifacts: extract kernel/initrd/squashfs from the nightly
ISO with xorriso; initrd picked by size from regular files only;
ISO URL "latest" resolves the newest vyos-nightly-build GH release
(downloads.vyos.io no longer serves direct ISOs)
Verified end-to-end in a libvirt VM against the real nightly ISO —
installed system boots with bond/VRRP/hw-id config applied and no
migrations. Fixes found by the VM run, encoded in code comments:
config.boot.default lives at /usr/share/vyos at hook time; fetch= boot
has no medium so the rootfs is symlinked to the installer's expected
path; reboot must be --force (the hook is a child of the still-starting
live-config unit); installer disk answers are full /dev paths; zram0
passes the 2GB min-disk filter so the disk is always pinned.
CLI/labd: vyos spec threaded through provision install (--vyos-* and
--vlan/--vlan-vip flags with guards), labd install route, protocol
command-install, and the bastion's direct /api/install.
268 unit tests pass; no new lint errors in touched files.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
The macvlan/VLAN-10 foundation HA depends on (Multus meta-CNI + a lan10
sub-interface + reference CNI plugins) was applied by hand during the HA
migration. Codify both as idempotent lab operations in the networking group,
after installCilium (which already sets cni.exclusive=false + bpf.vlanBypass={10}).
A fresh cluster now reproduces the full macvlan stack.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The v2.0 Phase 1 commit (04faa07) added AuthService, RbacService,
ResourceStore, AuditService, the bearer auth middleware, and the
v2-auth/environments/resources route files, but createApp() never
registered any of them. They sat in the codebase as dead code: a
running labd would 404 on /api/auth/login, /api/resources, /api/events,
etc.
Wiring (server.ts)
- Instantiate AuthService, RbacService, ResourceStore, AuditService at
app creation. Cast DbClient to PrismaClient (the runtime db is a real
PrismaClient; DbClient is a structural shim).
- Start AuditService timer, register an onClose hook to stop it on
shutdown so we never lose the last batch.
- Register v2 routes inside a Fastify scope with the bearer-auth
middleware as preHandler. v1 routes (registered on the root scope)
are unaffected so existing labd clients keep working.
AuditService (audit.ts)
- Expose flushPending() so tests can deterministically observe events
without leaning on the 5-second flush interval. Implementation
delegates to the existing private flush().
Smoke tests (v2-smoke.test.ts, 11 cases)
- Bootstrap: first POST /api/auth/login with empty users creates the
admin (role=ADMIN, hashed password), returns a 64-hex token, marks
isBootstrap=true, emits an auth_bootstrap audit event. Second login
uses the normal flow. Wrong password returns 401 and audits failure.
Missing credentials returns 400.
- RBAC: missing/empty/invalid bearer tokens return 401. ADMIN role
bypasses RBAC. A non-admin with no role bindings gets 403 with
"no matching role binding". A user with an env-A binding is denied
for env-B resources.
- Audit: bootstrap event is queryable via /api/events?correlation=...
Explicit parent/child chain (shared correlationId, parentEventId)
is preserved across emits.
All 246 workspace tests pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
bastion serve/stop default for --dir was hardcoded to /tmp/lab-bastion.
Now reads BASTION_DIR from env if set, so a deployed bastion daemon
can run from a persistent directory without callers having to pass
--dir on every invocation.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The v2.0 Phase 1 commit (04faa07) introduced the @lab/core package but
the labd Dockerfile still only copied @lab/shared and @lab/labd, so the
container build would fail to resolve @lab/core imports.
Both stages updated:
- Builder: copy @lab/core package.json/tsconfig + src, add it to the
build order between @lab/shared and @lab/labd.
- Runtime: copy @lab/core dist and package.json into the final image.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The worker0-k8s0 bug: when labd restarts, the in-memory installed map
is lost. The next DHCP/PXE re-discovery for that MAC ran an upsert that
wrote status="discovered", silently downgrading the DB record from
"online" or "offline" and erasing the machine's known hostname/role
identity from the CLI view.
- server.ts: drop status="discovered" from the upsert update branch so
re-discovery cannot downgrade an installed record.
- routes/bastions.ts (/api/machines): when the DB knows a real
hostname+role for a MAC currently only in live.discovered, promote
it back to live.installed so the CLI sees the right state. Also
reordered the live-vs-DB fallback so DB online/offline maps to
live.installed and the discovered branch is the else.
- tests: 3 new vitest cases covering promotion, fresh-discovery, and
unknown-MAC fallback.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two changes prompted by today's etcd raft panic on worker1-k8s0
(tocommit out of range, lost-write on follower) and the cascading
disk pressure that surfaced underneath it.
Audit logs to journald
- kube-apiserver now uses audit-log-path=- so audit events flow to
k3s.service stdout and into journald instead of growing files in
/var/log/kubernetes. The previous setup combined apiserver's
internal rotation with a logrotate *.log glob that double-rotated
the rotated files into permanent orphans (observed: 7+ GB).
- New journald-limits operation writes a SystemMaxUse=2G drop-in so
audit volume cannot fill /var/log even under bursty load.
- log-rotation operation repurposed to decommission the obsolete
logrotate rule and reap leftover audit files. Idempotent: no-op
on fresh installs.
Etcd member recovery
- New recoverEtcdMember(broken, peer, hostname) codifies the
documented k3s recovery: stop k3s, etcdctl member remove, wipe
/var/lib/rancher/k3s/server/{db,tls,cred}, restart, poll for
rejoin. Refuses to operate when cluster size < 3 to preserve
quorum.
Tests
- 7 new unit tests covering both decommission paths and the
recovery procedure (54 total, all green).
- install.test.ts asserts the file-based audit args are gone.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Project tracking for labctl v2.0 platform design. Includes P1 (arch doc update),
P2 (SSH emergency mode, Prometheus metrics), and P3 (graph viz, import, secrets rotation)
items from the CEO and eng review sessions.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add `labctl provision recheck` to refresh hardware info via SSH
- Preserve hardware info in InstalledInfo when install completes
- Fix /ks-auto: run nested %pre scripts from included kickstarts
- Add command-discover WebSocket routing for hw info updates
- Fix k3s join: clean stale TLS/cred when joining existing cluster
- Add --tls-verify=false for internal HTTP registry pushes
- Add fix-ssh-root.sh script for root SSH access on all nodes
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add state-aware kickstart dispatch for machines that boot from ISO
(no PXE/network at UEFI level). Replaces hardcoded discover.ks.
- /ks-auto: %pre detects MAC, queries /api/machine-state/<mac>,
writes discover or install kickstart to /tmp/dynamic.ks,
main body %include's it
- /api/machine-state/<mac>: simple state endpoint returning
unknown|discovered|queued|installing|installed|debug
- ISO kernel cmdline updated: discover.ks → ks-auto
- Handles: discovery (first boot), install (queued), debug modes
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Skip Cilium install for joining servers (already in cluster via daemonset)
- Longhorn annotation for workers: SSH to server node from CLI to apply
kubectl annotation (workers don't have kubectl access)
- Default SSH user for k3s/app commands changed to 'root' (operations
need root privileges, using 'lab' user broke installs)
- k3s server config: cluster-init for initial server, server+token for joins
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Server config now uses cluster-init: true for initial server (enables
embedded etcd). Joining servers get server: + token: in config.
- Cilium install already checks for existing installation, so joining
servers skip it gracefully (the "release name in use" error is non-fatal)
Cluster rebuilt as etcd HA:
worker0-k8s0 control-plane,etcd (initial server, cluster-init)
worker1-k8s0 control-plane,etcd (joined server, Mac Studio aarch64)
spark-2935 worker (DGX Spark, aarch64)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Replace sed with grep -v / awk for fstab manipulation (Asahi Fedora's
sed doesn't support \| delimiter or \? quantifier)
- Use idempotent write_lab_fstab function: removes all old entries first,
comments out conflicting btrfs subvol entries, adds fresh LVM entries
- Fix sed for SSH hardening: use #* instead of \? (POSIX compatible)
- Tested on Mac Studio: no duplicate fstab entries after multiple runs
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Firstboot script defaults admin user to 'lab' instead of bastion's
config.adminUser (which was 'michal' from host system)
- iSCSI OS detection uses case-insensitive match for 'fedora'
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The firstboot script now auto-detects hostname (from hostnamectl) and
MAC address (from first UP interface) at runtime. No URL query parameters
required — just `curl bastion/asahi/firstboot.sh | sudo bash`.
Fixes the shell escaping issue where `&` in query params broke curl piping.
Updated labctl provision asahi instructions accordingly.
Tested on Mac Studio (worker1-k8s0): hostname, MAC, and bastion
registration all auto-detected correctly.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Previously the reprovision path exited early after re-mounting LVs,
skipping hostname setup, admin user creation, metadata, and bastion
registration. Now both paths fall through to the common post-setup code.
Tested on Mac Studio (worker1-k8s0) — reprovision + self-registration
confirmed working via curl | bash pipe.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The Asahi installer's urlcache.py fails with AssertionError on macOS
when streaming ZIP via HTTP Range requests from Fastify. Fix: download
the ZIP with curl first (reliable on macOS), then set REPO_BASE to the
local directory so the installer opens it as a local file.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
k3s host prep:
- Add iSCSI initiator install+enable (Fedora: iscsi-initiator-utils,
Ubuntu: open-iscsi) — required by Longhorn
- Add Longhorn disk label to k3s server+agent configs
- Add Longhorn disk annotation operation in post-install hardening
CLI:
- Add `labctl provision asahi` command with interactive install guide
- Change default SSH user from "michal" to "lab" in all commands
- Change admin user in bastion progress callback to "lab"
Asahi provisioning fixes:
- Download installer_data.json locally (installer reads it as file)
- Use REPO_BASE to serve upstream ZIP from bastion (LAN speed)
- Fix ZIP32 vs ZIP64: serve original upstream ZIP unmodified
(our repackaged ZIP used ZIP64 which breaks Asahi urlcache)
- Add /data/asahi-repo fallback path for k3s container PVC mount
- Deploy script syncs asahi-repo to bastion pod after deployment
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add scripts/build-asahi-rootfs.sh: downloads upstream Fedora Asahi
Remix Server, injects lab firstboot script + systemd service + SSH
keys, repackages with installer_data.json that adds LVM Data partition
- Bastion serves built artifacts at /asahi/repo/* via fastify-static
- installer_data.json prefers built config, falls back to minimal
- Fix __dirname crash in ESM module (use import.meta.url)
- Fix smoke test timeout (was crashing due to __dirname)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
VM-based end-to-end test using Fedora cloud image with two disks:
root (20GB) + data (200GB). Verifies the firstboot script creates
labvg with correct LV sizes, mounts volumes, migrates /home content,
sets hostname, creates admin user, and handles reprovision.
Fixes to firstboot script:
- Detect whole disks (not just partitions) for LVM PV
- Handle btrfs subvolume paths in root device detection
- Copy /home content before mounting LV (preserves SSH keys)
- Don't restart sshd (config takes effect on reboot)
- Make swapon and mount operations resilient to failures
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add bastion endpoints for provisioning Apple Silicon machines via the
Asahi Linux installer with custom LVM partitioning:
- GET /asahi — wrapper script (curl bastion:8080/asahi | sh)
- GET /asahi/installer_data.json — custom partition layout (60GB root + LVM data)
- GET /asahi/firstboot.sh — first-boot LVM setup matching kickstart layout
- GET /asahi/firstboot.service — systemd oneshot unit
The firstboot script creates labvg with role-specific LVs (var, varlog,
home, srv, rancher, longhorn) and handles reprovision by detecting
existing VGs. Includes 19 new tests.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Remove hardcoded devices/directRoutingDevice from Cilium install (let
Cilium auto-detect per node — needed for heterogeneous NICs like eno1 vs enP7s7)
- Set k8sServiceHost=127.0.0.1 k8sServicePort=6444 so Cilium init
containers can reach the API via k3s agent's local LB proxy
- Add node-role.kubernetes.io/worker label to agent config
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add `labctl provision register` to re-add machines to installed state
without reprovisioning (e.g. after bastion state loss). Full stack:
protocol type, bastion API + WS handler, labd route, CLI command.
Add `labctl app k3s kubeconfig <target>` to fetch kubeconfig from a
k3s node via SSH, rewrite server URL, and merge into ~/.kube/config.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add UserKnownHostsFile=/dev/null to SSH in debug and reprovision commands
- Track install state in log follower so it doesn't exit prematurely on "installed"
- Reorder bastion status check to prioritize active queue over stale installed state
- Update .gitignore with task file entries
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
