87 Commits

Author SHA1 Message Date
Michal
7bf3f42e19 vyos: WAN follows VRRP mastership, rehearsed in labsim
Some checks failed
CI/CD / typecheck (push) Failing after 10s
CI/CD / test (push) Failing after 9s
CI/CD / lint (push) Failing after 36s
CI/CD / build (push) Has been skipped
CI/CD / publish-rpm (push) Has been skipped
CI/CD / publish-deb (push) Has been skipped
The ISP is consumer with one static IP, so "both routers hold WAN" is not
available: the 10 gig lease is anchored to a cloned MAC and the PPPoE line to a
single credential. The WAN therefore has to move with mastership.

Proven end to end in the sim: the secondary was promoted, took the WAN, got a
lease, installed a default route, and a LAN VM reached the internet through it
(3/3, 9ms). Demoting released it -- link down, no address. Rebooting the master
converged correctly too: it came back BACKUP with the WAN disabled while the
peer kept it.

The rehearsal earned its keep four times over, and none of these were visible
from reading the docs:

  - `transition-script` alone is NOT safe to hang internet on. VyOS delivers it
    through keepalived-fifo.py, and on one promotion that helper logged NOTHING
    while Keepalived_vrrp logged all six instances entering MASTER and the
    built-in notify_master for conntrack-sync ran normally. The result was a
    router holding every VIP with no WAN -- the 2026-09-02 outage, recreated by
    the mechanism meant to prevent it. Hence vrrp-wan-reconcile on a 30s timer:
    the scripts give speed, the timer gives correctness.

  - The health check must ask REALITY, not a marker. Keying "am I master" on a
    /run file written by the transition script meant that when the script did
    not run, the router believed it was backup, passed the check, and kept the
    VIPs it could not serve. It now asks whether the VIP is actually on the box.

  - The old address-based check DEADLOCKED this design: may-I-be-master required
    already having WAN, and only the master gets WAN. That is why vyos002 sat in
    FAULT for ever -- the safety check had silently removed the redundancy it
    existed to protect.

  - script-template must be the FIRST thing a script does. Sourced after an if,
    an exec and a mkdir it terminated the script inside the source, rc=0, no
    output: the reconciler reported success having done nothing. Hence the split
    into vrrp-wan-apply, matching the shape /config/vyos-known-good already uses.

Two hazards found and handled rather than discovered in production:

  - A `configure` session whose process dies leaks a unionfs mount under
    /opt/vyatta/config/tmp, and one of those holds the commit lock -- after
    which every commit fails, including the manual one you try to fix it with.
    A 30s job that can leak one per failure wedges the box on its own, so the
    reconciler reaps dead sessions before it starts. It cleared 8 on the sim.

  - Any `save` while a box is master persists the enabled WAN into config.boot,
    so a reboot would claim the shared MAC regardless of VRRP state. Observed:
    an ordinary console-apply did exactly this. config.boot must keep `disable`
    on BOTH routers; the model asserts it and vyos:verify reports it as drift.

NOT yet applied to production, and it should not be until the remaining item is
settled: a clean, deliberately-triggered failover has been seen via reboot, but
`restart vrrp` twice failed to move mastership at all, so the trigger for a
planned failover is still unproven.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-09-02 18:02:58 +01:00
Michal
09ede73b67 migration: the watcher that caught vyos002, and what it found
vyos002-catch.sh is armed before power-on and strips the eth2 address the moment
SSH answers -- 6 seconds, where a human watching a console loses that race more
often than not. It commits eth2 on its own before doing anything else, because
every extra command in that commit is extra exposure.

Caught at 15:35:22 on the LoT leg. eth2 ended with no address and the sync-group
health-check installed; it exits 1 (WAN disabled on this box), so all six VRRP
groups sit in FAULT and it holds no VIPs at all. That is the protection that was
missing on 2026-09-02, working as intended rather than as a theory.

The bounded-risk note in the header is the part worth keeping: it comes up BACKUP
behind a healthy vyos001, so it never holds 192.168.8.1 and the GATEWAY cannot be
poisoned during the window. The unbounded case is it becoming MASTER with eth2
present, which the health-check now makes impossible.

It also surfaced that vyos002 had booted from a saved config predating the day's
work -- stale reservations including the two that hand the routers' own eth2 NICs
192.168.8.143/.144. Synced to vyos001 and imported; see kubernetes-deployment
7f92974.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-09-02 15:51:47 +01:00
Michal
a973b51b9c migration: vyos001 converted in production, and the config vyos002 needs first
UniFi does offer Native VLAN = None (7.5.10), and its dropdown lists Management
as VLAN 1 -- the API reports vlan:null, which is what made this look like a
blocker. The UI was right.

Applied to USW Aggregation ports 1+2, then the router over ssh vyos@10.0.1.252.
Confirmed by kea's own log: "the interface bond0 has no usable IPv4 addresses
configured" -- it opens no socket on the parent, so there is no wrong pool left
to answer from. VRRP held mastership with no transition; the VIPs never moved.

Proven end to end by the thing that was broken: vyos002's JetKVM console, which
had been sitting on a VLAN 3 port holding Management lease 192.168.1.28, was
restarted and took 192.168.3.14 -- one lease, right pool, right reservation.
That console is what unblocks vyos002.

commit-confirm cannot be driven non-interactively: `vbash -s` hangs on its
prompt. It failed safely (candidate discarded, nothing committed) but the
recovery card's commit-confirm advice only works typed by hand.

vyos002-return.conf carries the two defects that must not survive its next boot.
eth2 is US24 port 16, native VLAN 2 -- disable that port before powering the box
on and the ARP hazard is gone before it can happen.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-09-02 15:26:21 +01:00
Michal
d727a50ca0 migration: offline recovery card, and stop the leak test trusting a silent router
The change itself takes Management down, so the session that performs it has no
internet and no Claude. RECOVERY-CARD-vlan1-move.md is what is actually needed at
that point, on one page: the recovery path, the order, the commands, and what to
do when locked out.

The recovery path is `ssh vyos@10.0.1.252`, and the reason it is trustworthy is
that it is not routed -- the workstation is 10.0.0.210/23 and the router's LoT
leg is 10.0.1.252/23, so `ip route get` returns dev lanbr0 with no `via`. It
therefore survives Management, VRRP, the VIPs, DNS and the switch trunk all being
wrong at once. bond0.10 is untouched by the change and stays in the LAN group.

Order is switch-first, which is the opposite of what seems natural. The UniFi
controller is 192.168.1.5, on Management, reached from LoT *through vyos001*: do
the router first and you lose the controller you still need for the switch.

Two corrections to the leak test, both because it reported a router fault that
was its own:

  - it counted `pgrep -f 'tcpdump -i bond0'`, so a stray tcpdump from an earlier
    run satisfied the >=2 guard with none of this run's captures alive. A capture
    that records nothing reads as "the router sent no reply at all".
  - it believed a single silent run. Kea can be is-active and answering nothing
    for tens of seconds after a restart, so the first VLANs of a loop failed and
    the last passed. That produced two OPPOSITE and equally wrong conclusions
    about `listen-interface` before a retry showed the pattern.

On `listen-interface`: it is a real second branch in kea-dhcp4.conf.j2 that keeps
dhcp-socket-type raw, unlike listen-address which forces udp and took DHCP down
when it was tried live. But naming the sub-interfaces explicitly left Management
DHCP dead in the sim, reproducibly, against a clean control with interfaces:["*"].
Not adopted, and not needed -- the address move is the proven fix.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-09-02 14:30:35 +01:00
Michal
0481c38e09 labsim: prove the tagged-Management fix for kea's wrong-pool offers
Kea #1117: with dhcp-socket-type raw, a frame tagged for a sub-interface is
also delivered to the parent's AF_PACKET socket, and if the parent serves a
subnet kea answers from it too. Management being the native VLAN on bond0 is
what gives the parent that subnet. One DISCOVER on VLAN 3 produced two OFFERs,
and in the captures here the WRONG one arrives first as often as not -- which
is why this looked device-dependent rather than like a server bug.

labsim-vlan-leak-test.sh reproduces it and scores the SERVER's offers, not the
client's choice; a client picking correctly is how this hid. Fails on the old
shape, passes on the new one across all six LAN VLANs.

Three things the rehearsal caught that reasoning had not:

  - kea keeps its old raw socket. VyOS does not restart it for an interface
    address change, so the first post-fix test failed and looked exactly like
    the fix not working.
  - interface-group LAN names the bare bond0. Moving the address without
    moving the group drops every management session under default-deny.
  - there is no make-before-break. A port always egresses its native VLAN
    untagged, so while VLAN 1 is native the router can send tagged VLAN 1 but
    never receive it -- verified, the ARP landed on bond0 untagged.

What makes the cutover safe anyway is that tagged and untagged Management
coexist, so the firewalls convert one at a time: 0s of VIP downtime, versus
5m30s if both routers go before the switch does. In that state the healthy
BACKUP does NOT take over -- the sync group holds native BACKUP because the
other VLANs still hear the master.

Also fixes two ways the sim was lying. ovs_bond_router compared only the trunk
VLAN list on re-runs, so a VM restart left the bond holding taps that no longer
existed while the real ones sat in the bridge unbonded -- labsim-vyos2 had no
LACP at all. And the tap count included the primary's libvirt-NAT scaffold NIC,
so the primary's bond was skipped outright.

Runbook: migration/MANAGEMENT-VLAN-TAGGED.md

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-09-02 14:03:44 +01:00
Michal
23783b8486 vyos: VRRP health check so the WAN and the gateway VIP cannot separate
Outage of 2026-09-02: vyos002 held every floating gateway IP while vyos001 held
the only working WAN. The LAN had a gateway that could not reach the internet,
and it stayed that way until vyos002 was powered off by hand.

Three causes, none of them bad luck. VRRP had no health check of any kind, so
mastership was decided purely on whether the peer was still advertising and
never on whether this router could route. vyos002 structurally cannot route --
bond0.53 is `disable`d because the 10 gig lease is bound to a cloned MAC that
only one box may hold, and pppoe0 is not up. And `no-preempt` is set on every
group, so once vyos002 took master it kept it even with a healthy priority-200
peer sitting next to it.

Reproduced exactly in labsim: stopping keepalived on the primary moved every VIP
to the WAN-less secondary and LAN internet went from 9ms to 100% loss; restarting
the healthy primary left it BACKUP and the outage in place. Applying this check
self-healed it -- secondary to FAULT, primary to MASTER, internet back.

The check asks "do I have an address on a WAN interface", deliberately not "can
I reach the internet" and not "do I have a default route". During a real ISP
outage the default route disappears on BOTH routers; keying on that would put
both in FAULT, nobody would hold the VIPs, and an internet outage would become a
total one.

Config goes on the SYNC GROUP, not per group: VyOS refuses a per-group check
while the group is in a sync group ("Only sync group health check will be
used"), and sync-group scope is what we want anyway so all VIPs move together.

Known cost, measured: with the primary genuinely dead the secondary stays FAULT
and NOTHING holds the gateway, so inter-VLAN routing stops too. That is the
honest consequence of a backup that cannot route. The fix for it is to make the
WAN follow mastership so the backup CAN route -- next, and rehearsed separately,
since it is the one change that can lose the DHCP lease.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-09-02 11:39:44 +01:00
Michal
11344eab92 labsim: the IPAM switch needs no pod recycle when the CIDR sources agree
Re-ran the 3-node rehearsal, this time from a state that matches production
rather than one I had accidentally skewed.

The earlier "two nodes swapped CIDRs" result was an artefact of my own setup:
labsim had been running cluster-pool with per-node CIDRs that differed from
node.spec, I flipped it to ipam=kubernetes (which resyncs CiliumNode from
node.spec), and flipping back therefore looked like a renumber. Production has
only ever run ipam=kubernetes, and all five nodes were checked: CiliumNode and
node.spec agree everywhere.

From that matching state the switch is close to a non-event: per-node CIDRs
unchanged, every pod still inside its node's range, nothing stranded, no recycle
required. The only disruption is the cilium DaemonSet restarting itself -- one
agent sat in Init:0/6 and one node briefly took the agent-not-ready taint, both
of which cleared on their own. Cross-node connectivity verified after.

So the recycle is CONDITIONAL, not a fixed step, and `verify` is what decides.
Documented both cases in the script header, because the dangerous one is silent:
stranded pods report Running and Ready while being unreachable.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-08-24 23:20:16 +01:00
Michal
e8679f45b5 labsim: rehearse the IPAM switch on 3 nodes, and catch the trap in it
Rehearsed kubernetes -> cluster-pool on the 3-node labsim cluster, which is the
transition production faces. The switch itself is undramatic: agents stayed up,
the operator adopted the pool, and the agent-not-ready taint deadlock did NOT
occur. That deadlock is specific to ADDING IPv6 -- the agent blocks on an IPv6
pod CIDR that does not exist yet. A v4-only mode switch does not hit it.

The real hazard is quieter. The operator does not preserve which node held which
/24: two nodes swapped CIDRs. Their existing pods kept their old addresses,
which now fall outside the node's range, so every other node routes that prefix
to the wrong node. Cross-node ping to those pods dropped 100% while every pod
stayed Running and every node stayed Ready. Nothing in `kubectl get pods` shows
it.

So "pods kept the same address" is the FAILURE signal here, not the reassurance
it looks like. cilium-ipam-switch.sh verify now flags pods sitting outside their
node's CIDR, which is the check that decides whether a recycle is optional
(it is not) or mandatory (it is).

Recycling every deploy/ds/sts restored it: all pods back inside their node CIDR,
cross-node ping 0% loss. Sequence proven end to end:
  preflight -> apply -> restart operator then agents -> unstick if needed ->
  recycle all workloads -> verify

Also fixed the recycle hint the script printed: `kubectl rollout restart
deploy,ds,sts -A` is not valid (`unknown shorthand flag: 'A'`), so anyone
following it under pressure would have got an error instead of a recycle.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-08-24 23:13:20 +01:00
Michal
527e0798ae bastion/k3s: bootstrap new clusters with cluster-pool IPAM
Every cluster this bastion builds was pinned to ipam=kubernetes, which makes it
permanently single-stack: Cilium reads node.spec.podCIDRs, the controller-manager
writes that once at node join and never revises it, so adding IPv6 later fails
with `required IPv6 PodCIDR not available` and needs a rebuild.

cluster-pool puts allocation in the CiliumNode CRD, where the operator can add
an address family to a running node. Proven in labsim (842408c). It is also
Cilium's own default -- ipam=kubernetes was the deviation.

Pool matches the k3s cluster-cidr (10.42.0.0/16, /24 per node), so a node gets
the same CIDR shape it would have had. IPv4-only for now: the IPv6 pool wants an
apiserver carrying an IPv6 service CIDR, which is a separate change.

Mirrors @michal/cilium-values ciliumBaseValues (cilium-values@11ee509), which
these paths track but do not import.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-08-24 22:59:55 +01:00
Michal
842408c0d9 labsim: prove k3s CAN be converted to dual-stack in place
k3s documents that dual-stack "cannot be enabled on an existing cluster". Rather
than accept that for a 145-day-old production cluster, build both shapes and
diff them. dualstack-lab.sh builds a single-node IPv4 cluster and a native
dual-stack one, takes a reflink copy of the IPv4 disk so a failed conversion
costs 90 seconds to undo, converts in place, and diffs the results.

Result: the conversion works. etcd data is never touched.

The documented blocker is real but narrower than stated. Cilium reports it
exactly -- `required IPv6 PodCIDR not available` -- because node.spec.podCIDRs
is assigned at join and is immutable, and the Kubernetes IPAM controller will
not add a second family later. That objection only holds while Cilium runs
ipam=kubernetes and therefore reads that field. Switching to cluster-pool IPAM
moves pod CIDR allocation into the CiliumNode CRD, where the operator hands out
both families on a cluster that was born IPv4-only.

Sequence that works, in order:
  1. k3s unit gains --cluster-cidr/--service-cidr/--node-ip with both families.
     k3s validates the two CIDRs together and refuses to start on a mismatch
     ("must share the same IP version"), so a partial edit crash-loops rather
     than coming up half-configured. That is the safe failure mode.
  2. The ServiceCIDR object picks up the IPv6 range on restart -- this is
     upstream's supported "single-to-dual-stack preserving the primary
     ServiceCIDR" path, and existing Services keep their IPv4 addresses.
  3. Cilium to ipam=cluster-pool with an IPv6 pool, then DELETE the CiliumNode
     so the operator reallocates; it will not add a family to an existing one.
  4. Expect a deadlock here: the agent will not go ready without a pod CIDR, so
     the node keeps the node.cilium.io/agent-not-ready taint, so the new
     operator that would assign the CIDR cannot schedule. Remove the taint by
     hand once to break it.

Verified on the converted cluster: pod with 10.42.0.125 AND fd00:42::4843, and
a PreferDualStack Service holding 10.43.122.115 AND fd00:43::72a3.

The only field that still differs from a native build is node.spec.podCIDRs,
which stays IPv4-only -- immutable, and unused once Cilium owns IPAM. CiliumNode
podCIDRs and ServiceCIDR are identical to the native cluster.

Not yet answered: this is one node. Whether a 3-server etcd cluster converts as
cleanly, and what a rejoining agent does, is the next experiment.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-08-24 22:46:27 +01:00
Michal
ad6eb7a9a6 labsim: default-deny firewall policy, proven in the sim
Some checks failed
CI/CD / lint (push) Failing after 10s
CI/CD / test (push) Failing after 10s
CI/CD / typecheck (push) Failing after 24s
CI/CD / build (push) Has been skipped
CI/CD / publish-rpm (push) Has been skipped
CI/CD / publish-deb (push) Has been skipped
Policy: internal VLANs reach each other and the internet; the internet
initiates nothing inward. That was already the effect of the IPv4 ruleset, but
built as a blacklist -- default-action accept plus explicit drops per WAN
interface. Identical behaviour right up until a WAN is added, at which point it
is open and nothing looks wrong. This expresses it as a whitelist.

Two findings from the sim, both of which would have been outages in production:

`set` on a rule number is ADDITIVE. The sim already had a rule 10 carrying
inbound/outbound interface constraints; `set ... rule 10 state established`
ANDed onto it, producing a stateful-accept that applied to one interface pair
only. Return traffic from the internet then matched no rule and hit the default
drop, so LAN hosts could reach nothing outbound. The generator now deletes each
filter before rebuilding it, so the code owns the subtree. It is one commit, so
nftables is rebuilt atomically -- there is no window without a firewall.

DHCP lease renewal is unicast UDP to port 68 and conntrack does not reliably
cover it. Without an explicit rule the WAN keeps working until the lease
expires and then dies -- a delayed failure that looks nothing like a firewall
change. Also added a loopback accept for both families, absent from the v6
policy since it went default-deny.

Verified in labsim: inter-VLAN ok, LAN-to-internet ok, internet-to-router
dropped, and internet-to-LAN dropped with the drop counter incrementing by
exactly the packets sent, after routing the test through the router rather than
around it via the hypervisor.

Also extends the drift check to the firewall subtree, which it did not cover --
so it had been reporting "in sync" while that subtree was uncaptured.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-08-22 22:25:55 +01:00
Michal
7f551081ad labsim: capture BGP, dual WAN and both ISP VMs as code
The sim's routing config existed only as running state on the VMs. It was
applied by hand over SSH, so rebuilding a VM lost the rehearsal and nothing
recorded why any of it was shaped the way it was. The two ISP VMs were not
referenced anywhere in the repo at all.

sim-net-config.py generates all four roles; sim-net-apply.sh applies them over
the serial console, or diffs them against the running VMs. Verified reproducing
live state exactly before committing: primary 40/40 commands, secondary 16/16,
isp-dhcp 19/19, isp-pppoe 21/21.

Carries the reasoning that was previously nowhere: RFC 8212 needing policy in
both directions or the session carries zero prefixes; probe targets that must
not double as system name-servers; default-route-distance 210 rather than
no-default-route, which blanks new_routers and hands the default route to the
backup line; and the WI-8 bootstrap bug that pinned /32s fix.

Dropped a stale `pppoe-server interface eth0` on isp-pppoe (a NIC that does not
exist there) so a green drift check stays meaningful.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-08-22 16:26:36 +01:00
Michal
f41ffdd039 feat(vyos): reconciler that keeps the HE 6in4 tunnel on the live WAN
Some checks failed
CI/CD / lint (push) Failing after 9s
CI/CD / test (push) Failing after 13s
CI/CD / typecheck (push) Failing after 25s
CI/CD / build (push) Has been skipped
CI/CD / publish-rpm (push) Has been skipped
CI/CD / publish-deb (push) Has been skipped
Kernel-level (`ip tunnel change`), not VyOS config: no commit churn on a
flapping line, no drift against the Pulumi model, and a reboot restores
config.boot — which pins the 10 gig — so a wrong source cannot survive a
restart.

Sends `myip` explicitly, because mid-failover the update request may egress
either line and letting HE infer the address would point the tunnel at the WAN
we just left. Requires two consecutive agreeing runs before acting, since HE
rate-limits updates and a flapping WAN would hammer the API precisely when it
matters.

MTU moves with the WAN: 1480 on the 10 gig (1500-20), 1472 on PPPoE (1492-20).
Fixed at 1480, the backup path gives the signature people lose a day to — small
packets fine, large transfers hang.

Inert without /config/he-secrets, and a no-op when already in sync.
2026-08-21 02:04:43 +01:00
Michal
a187703a3a feat(vyos): pin a known-good config and restore it with one command
Some checks failed
CI/CD / lint (push) Failing after 9s
CI/CD / test (push) Failing after 10s
CI/CD / typecheck (push) Failing after 24s
CI/CD / build (push) Has been skipped
CI/CD / publish-rpm (push) Has been skipped
CI/CD / publish-deb (push) Has been skipped
VyOS already has rollback, but `rollback 1` returns you to the *previous*
revision, which may itself be broken — you can end up walking backwards through
several bad commits hunting for the one that worked, at exactly the moment you
have no network to look things up with. This pins a state a human has actually
used and found working, so recovery is one step and needs no memory of how many
changes ago things were fine.

    /config/vyos-known-good save      pin the running config
    /config/vyos-known-good status    when it was taken, how running differs
    /config/vyos-known-good diff      what a restore would change
    /config/vyos-known-good restore   go back to it

Deliberately not automatic. A config is only known-good once someone has used
the network; a snapshot taken after every commit would faithfully preserve the
broken one.

The restore is itself commit-confirmed, so even the recovery path is protected:
if the snapshot is somehow wrong, or access is still broken and nothing can be
confirmed, the router undoes the restore rather than leaving you worse off.
Silence reverts.

`save` refuses when there are uncommitted changes — a snapshot that did not
match what is actually running would look like a safety net without being one.

Two things found while building it, both of which made the script silently
useless rather than fail loudly:

  - Sourcing `script-template` **resets the positional parameters**, so `$1` was
    empty by the time the case statement ran and every invocation fell through
    to the usage message. Arguments are captured before the source.
  - `0600` made the snapshot unreadable to the `vyos` user, so `status` and
    `diff` — the two commands you run while deciding whether to restore — showed
    nothing. Now 0660 root:vyattacfg, matching /config/config.boot.

Installed on both routers with the current, verified-working config pinned
(vyos001 1029 lines, vyos002 1019).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-08-21 01:46:43 +01:00
Michal
86c2a36f00 feat(labsim): a real Kubernetes cluster for rehearsing Cilium <-> VyOS BGP
Some checks failed
CI/CD / lint (push) Failing after 9s
CI/CD / typecheck (push) Failing after 8s
CI/CD / test (push) Failing after 9s
CI/CD / build (push) Has been skipped
CI/CD / publish-rpm (push) Has been skipped
CI/CD / publish-deb (push) Has been skipped
BGP is about to be added to a network that currently works, where a bad
advertisement blackholes the house. That needs somewhere to fail first.

Three Debian nodes (4 GB / 2 vCPU) on the OVS `vlan2` access ports running k3s
with flannel disabled, so Cilium is the CNI under test. Three rather than two
because ECMP is only meaningfully tested if a node can be drained and more than
one path survives.

VMs rather than k3d: the thing under test is eBGP between Cilium and VyOS across
the switch fabric, nodes peering with the router's bond0.2 leg, directly
connected. k3d would put the nodes on a container bridge -- a different L2 path,
proving something else. (It also wants Docker; this host has podman.) The
existing micro VMs are Alpine with 256 MB and 1 vCPU, which is not close to
enough.

Debian rather than the sim's Alpine base: glibc, a stock kernel that Cilium's
eBPF probes are tested against, and cloud-init that actually applies
network-config -- the Alpine base notably does not.

One trap worth recording. An earlier draft called `selected_vlans "$K8S_VLAN"`
before `ovs_up`, and since `ovs_up` re-defines the libvirt network from
SELECTED, that silently deleted the portgroups for every other VLAN. Running
VMs kept working -- their taps were already attached -- so nothing complained
until the ISP VMs needed vlan51 and vlan53 and could not be attached. It now
selects every VLAN.

The generated kubeconfig is gitignored: it carries cluster-admin credentials and
is one `git add -A` away from being committed. Regenerate with
`k8s-up.sh --kubeconfig`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-08-19 13:15:45 +01:00
Michal
27a343bc75 Merge branch 'feat/unifi-export-and-vyos-dhcp': USG to VyOS migration
Some checks failed
CI/CD / typecheck (push) Failing after 12s
CI/CD / test (push) Failing after 10s
CI/CD / lint (push) Failing after 25s
CI/CD / build (push) Has been skipped
CI/CD / publish-rpm (push) Has been skipped
CI/CD / publish-deb (push) Has been skipped
Carries the UniFi export tooling, the generated VyOS DHCP/DNS config, the
reversible cutover switch, the health-checked WAN failover, and the labctl
side of applying a Pulumi-rendered bundle at install time.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-08-18 12:19:48 +01:00
Michal
672b89ce38 feat(labctl): install VyOS from a Pulumi-rendered bundle, and enable its API
Two halves of the same problem: a router should come up running the config
that is declared for it, and should be manageable the moment it does.

--vyos-bundle applies a bundle rendered by kubernetes-deployment verbatim,
replacing the derived --vyos-bond/--vlan/... path rather than merging with it.
Deriving a second opinion alongside a bundle is exactly the drift the bundle
exists to prevent: Pulumi and labctl would each believe they knew the
router's config and the box would end up with whichever ran last. Passing
both is rejected rather than silently resolved.

Secret-valued nodes arrive as @secret: sentinels and are dropped, with a
warning naming each one. Writing the sentinel text into config.boot would
look configured while being wrong, which is worse than being absent -- the
router comes up without its PPPoE credential and the first `pulumi up`
supplies it. A bundle committed to git has to stay safe to read.

The hostname is forced to the one the install was asked for. A bundle is
exported from one router and reused for its peer, and taking the hostname
from it would put two vyos001s on the network.

--vyos-api-key enables the HTTP API at install, on both the bundle and the
derived path, so every VyOS this bastion provisions is manageable from first
boot. vyos001 and vyos002 predate this and had to be enabled by hand on a
live firewall after their cutover -- which is the gap this closes. It is
deliberately not part of the Pulumi model: a provider able to rewrite its own
transport can revoke its own access.

listen-address is always set, and the API is NOT enabled when no address is
known -- under DHCP there is none at build time, and binding to every
interface would publish a config-write endpoint on the WAN. It warns and
leaves the router SSH-only instead.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-08-18 12:18:53 +01:00
Michal
f4984e3962 fix(vyos): health-check the 10 gig primary so failover actually fires
The 10 gig line was primary by route distance alone, which only fails over
when bond0.53 loses carrier or its DHCP lease. An ISP that keeps the link up
while dropping traffic -- the common failure -- would black-hole everything,
because a DHCP-installed route has nothing to withdraw it.

`protocols failover` now owns the live default route and pings two targets
bound to the interface, so the backup can never be validated through the
primary's path. Rehearsed on the labsim router: failover and failback both
inside 5s with the router's own interface still UP.

The vif keeps default-route-distance rather than no-default-route, demoted
below Vodafone. vyos-failover resolves a dhcp-interface gateway by reading
new_routers out of /run/dhclient/dhclient_<if>.lease, and no-default-route
leaves that field EMPTY -- the daemon then finds no next hop and installs
nothing. Observed on vyos001: the default route fell through to Vodafone.
Preference is now failover's kernel route (distance 0) > pppoe (10) >
DHCP (210), so the demoted route can never re-create the black hole it
exists to avoid.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-08-18 12:18:41 +01:00
Michal
7b5331ddcd fix(migration): the backup has no WAN by design; stop failing it for that
The cutover succeeded on vyos001 -- gateway live, bond0.53 holding
87.192.101.48 with the cloned MAC, kea serving, clients routing out through NAT.
vyos002 then ran the same script and was judged unhealthy, because the mandatory
checks are "default route exists / internet reachable / DNS resolves" and the
backup deliberately holds its WAN interfaces DOWN. Its config was correct; the
check did not apply to it. Confirmed by hand before the timer could revert a
good config.

This is the third instance of one mistake: asserting a condition that is not
true of the box being checked. First requiring every WAN when one suffices, now
requiring a WAN on the box that is configured not to have one.

A delta containing `interfaces ... disable` for the WAN now identifies the
backup, and the WAN-dependent checks are skipped with a note. kea and the DNS
forwarder remain mandatory on both -- those are what the backup must actually
be able to do.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-08-18 01:48:48 +01:00
Michal
ce6911c196 fix(migration): require a working WAN, not every WAN
This reverted a cutover that had actually succeeded.

The evidence, from the revert tearing it down:

  dhclient: DHCPRELEASE of 87.192.101.48 on bond0.53 to 185.232.119.244
  vtysh:    "no ip route 0.0.0.0/0 87.192.96.1 bond0.53 tag 210 1"
  netlinkd: RTM_NEWLINK -> bond0.53, mac=f0:9f:c2:12:9b:4f

bond0.53 came up with the cloned MAC and was handed 87.192.101.48 -- the exact
public address the USG holds -- with a default route via the real ISP gateway.
kea was serving live LAN clients at the same moment (10.0.0.12, 10.0.0.13,
192.168.8.28). The gateway was working.

The only failure was pppoe0: ppp@pppoe0.service exited 5/NOTINSTALLED. That is
the Vodafone FAILOVER line, and the health check listed "pppoe0 has an address"
as mandatory, so a working gateway was torn down because its backup WAN was
down. The check encoded "every WAN must work" when the requirement is "the box
must reach the internet".

Now: default route, reachability and DNS are mandatory; each WAN interface is
reported individually but fatal on neither. A failover line being down is worth
seeing, not worth reverting for.

This also incidentally settles the last genuine unknown in the migration, which
could not be tested any other way: the ISP does hand the same lease to the
cloned MAC. That was the one thing I had said was unknowable until the USG let
go of it.

Note the earlier polling fix (54b21fa) addressed a real weakness but not this
failure -- no amount of waiting would have satisfied a check that required a
line which was never going to come up.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-08-18 01:37:14 +01:00
Michal
54b21fa9ff fix(migration): poll for WAN health instead of sampling once at 25s
A real cutover attempt reported failure and reverted a configuration that may
well have been fine. The health check waited a fixed 25 seconds and then judged:

  [switch] committed. Waiting 25s for PPPoE and services to settle...
  [switch]   FAIL  pppoe0 has an address

25s is far too short for a WAN. PPPoE alone is PADI/PADO/PADR/PADS followed by
LCP, authentication and IPCP -- routinely 15-30s on its own. Both lines had also
just been released by the USG seconds earlier, and ISPs commonly hold the
previous session and MAC binding for minutes before leasing to the "same" CPE
again, which is exactly what a cloned MAC looks like from their side. The one
thing the design could not tolerate was being impatient, and it was.

Now polls every 15s up to HEALTH_BUDGET (default 180s), reporting progress, and
stops early the moment everything is healthy. The budget deliberately finishes
long before commit-confirm fires -- 180s against a 10 minute timer leaves 420s
of margin -- so the decision to confirm or revert stays ours rather than being
made by the timer.

Also recorded while chasing this: the earlier claim that VLANs 51/53 are not
trunked to the firewalls was WRONG, and the UniFi port settings disprove it --
those LAG ports are Native VLAN Management (1) with Tagged VLAN Management set
to Allow All. My evidence never supported the claim: a passive RX count cannot
distinguish an absent VLAN from a quiet one, because switches do not flood
unicast, and the active DHCP probe used a random MAC that an ISP binding to its
registered CPE would ignore regardless. Both observations fit a perfectly
healthy trunk.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-08-18 01:10:50 +01:00
Michal
ff86a421f4 fix(labsim): console-apply must handle both VyOS prompts, not just $
Two failures from one strict expect, both hit while building the sim ISPs.

A run that dies mid-config leaves the console parked in configuration mode. The
next run then waits for the operational `$ ` prompt against a perfectly healthy
VM and hangs until timeout, with nothing in the output to say why -- the box was
sitting at `vyos@isp-dhcp#` the whole time. Login now accepts `# ` as well and
discards the stale candidate rather than committing something nobody has seen.

The same mistake at the exit step: insisting on `$ ` after `save` hung, AND left
the console in config mode, which is what created the first failure for the
following run. Now accepts either prompt.

Known-bad, not fixed: the tool reports "committed and saved" when the set
commands have not applied. Verified against the clone -- prompt showed the
host-name change had landed while `grep -c dhcp-server` returned 0. The failure
detection only inspects c.before for a few strings and evidently misses the real
failure mode, so success is being reported without evidence. That needs fixing
before this tool is trusted for anything; it is currently only safe to use with
an independent check afterwards, which is how the gap was found.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-08-18 01:00:19 +01:00
Michal
ee070371a8 feat(labsim): add WAN transport VLANs so the sim can host fake ISPs
A cutover attempt failed on the WAN and nothing had tested it. The reason the
sim could not have caught it: labsim modelled every LAN VLAN faithfully and
omitted the WAN entirely -- vlans.conf had 1, 2, 3, 9, 10 and 200, never 51 or
53. Worse, the switch script's WAN health checks are conditional on the delta
configuring PPPoE, so in the sim they printed "this delta configures no WAN --
skipping all WAN health checks" and passed. The sim proved the delta commits; it
never proved the WAN works, and could not have.

Adds VLANs 51 (Vodafone/PPPoE) and 53 (10gig/DHCP) to the fabric so a fake ISP
can live on each and those checks actually execute. VyOS has service
pppoe-server (accel-ppp) natively -- authentication local-users, client-ip-pool,
gateway-address -- so a VyOS VM can play the concentrator, and dhcp-server can
play the other ISP.

vlans.conf gains host_octet 0, meaning "no host leg". A host address on a WAN
transport VLAN would misrepresent the segment: the point is that VyOS reaches an
ISP, not the host.

Also fixes a real gap in ovs_bond_router: it returned early when the bond
already existed, so adding a VLAN to vlans.conf never reached an existing bond.
Re-runs now reconcile the trunk and say so. That gap is the same SHAPE as the
production failure -- interface present, VLAN missing from the trunk, frames
silently dropped -- which is precisely the class of bug the sim needs to be able
to reproduce rather than embody.

Both bonds updated: [2,3,9,10,200] -> [2,3,9,10,51,53,200].

Note on the production diagnosis, which is NOT settled: a passive RX test showed
zero frames on 51/53 at the firewall, and an active DHCP DISCOVER (verified to
have transmitted, tx +2) drew no reply. That is consistent with the VLANs not
being trunked, but equally with the ISP only answering its registered CPE MAC --
which is exactly why the delta clones f0:9f:c2:12:9b:4f, and why it cannot be
settled from production while the USG holds that MAC.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-08-18 00:44:11 +01:00
Michal
41b5448f56 feat(pulumi-vyos): prototype VyOS subtrees as Pulumi resources with commit-confirm
Goal: change VyOS and Kubernetes in one codebase and one plan -- so a BGP change
touches both sides in a single `pulumi preview`.

First, the worry about per-command pushes turned out to be unfounded for the
community providers. Read foltik/vyos and its client library: a
`vyos_config_block_tree` flattens the whole subtree into a single payload array
and sends ONE POST to /configure, so one resource is one commit. Good.

What they do not do is send `confirm_time`. Their payload is only
op/path/value, so every change is an unprotected commit -- on a router you reach
through the router, that is the difference between a mistake and an outage. The
VyOS API itself supports commit-confirm; the providers simply do not use it.

So this is a ~180-line Pulumi dynamic provider that does. Verified end to end on
labsim: create and update each land in ~6s as one commit-confirmed transaction,
update reports [diff: ~commands], destroy removes the subtree, and an
unconfirmed commit was observed reverting the router on its own.

Three API details found the hard way, all now encoded and commented:

  - confirm_time is ONLY read when the body parses as ConfigureListModel, i.e.
    {"commands": [...], "confirm_time": N}. A bare array is accepted and
    committed with NO timer armed, and the response looks like success. This
    silently discards the entire safety net, so the resource now checks the
    response actually says "commit-confirm" and refuses to proceed otherwise.
  - There is no /confirm endpoint; confirm is an op on /configure.
  - Confirm requires a `path` field even though it ignores it -- the Union
    resolves to ConfigureModel, which mandates path. Without it: "missing 'path'
    field", and the timer keeps running.

Apply is `delete <path>` followed by the sets, in one request, so the result is
the declared state rather than a merge -- otherwise `pulumi up` accumulates
instead of converging.

Known gaps, in the README rather than hidden: no read/refresh so out-of-band
drift is not detected, and the API runs with a self-signed certificate and
verification disabled. Both need addressing before production. The cutover
itself should still use vyos-unifi-switch, which the API cannot replace.

Sim left as found: test resource destroyed, dns forwarding restored to 15 lines.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-08-17 01:09:05 +01:00
Michal
63061e6e7e feat(migration): peer link cabled and verified; conntrack-sync enabled in deltas
eth3 <-> eth3 direct cable is in. Both ends negotiated 2500Mb full duplex --
these are 2.5 GbE ports, not the 1G I had assumed.

Carrier alone proves nothing, so the link was tested end to end with temporary
kernel-level addresses (never committed to VyOS config, removed afterwards):
3/3 packets, 0% loss, 0.371ms average. A cable can show carrier and still not
pass traffic; now it is known to.

Deltas regenerated with --conntrack-link and installed on both boxes:

  vyos001  nat=21 fw=58 conntrack=9  eth3=10.255.255.1/30  disable=0
  vyos002  nat=21 fw=58 conntrack=9  eth3=10.255.255.2/30  disable=2

Identical apart from the peer /30, the VRRP/DHCP-HA roles, and the two disable
lines holding vyos002's WAN down. Both still report mode=unifi and nothing about
their behaviour has changed -- eth3 carries no address in the running config,
and conntrack-sync appears only in the delta, which is applied at cutover.

Not verified: multicast on the peer link. `ping -I eth3 224.0.0.1` drew no
responders, but that is the all-hosts group which VyOS need not answer, so it
proves nothing either way. conntrack-sync's own multicast (225.0.0.50) was
proven working in labsim over bond0.10, and this is a point-to-point link, so
the risk is low -- but it is untested on this specific cable and worth watching
at cutover.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-08-17 00:37:16 +01:00
Michal
ccdd1e7e49 fix(migration): both boxes carry WAN and NAT; the backup just holds it down
"No NAT? How are we supposed to get internet?" -- a fair question that exposed a
worse design than I had admitted. Internet did work, but only via vyos001: NAT
and the entire WAN were gated behind --with-wan, so vyos002 would have held the
LAN VIPs and routed between VLANs with no path to the outside at all. Failover
would have preserved addressing and lost the internet.

The fix rests on a checked fact rather than an assumption: VyOS WARNS but still
commits when a NAT rule names an interface that does not exist
("Interface bond0.53 for source NAT rule 900 does not exist!"). Verified on a
real VyOS before relying on it.

So both boxes now get the identical WAN, NAT, port-forward and firewall config,
and the backup's two WAN interfaces are simply set `disable`. The cloned WAN MAC
is therefore never live on two boxes at once, while everything needed to route
and masquerade is already in place. The two deltas are now byte-identical apart
from VRRP priority, own/peer addresses, DHCP HA role, the conntrack /30 -- and
the two disable lines.

Taking over the internet path becomes deleting two lines rather than
reconstructing NAT under pressure:

    delete interfaces bonding bond0 vif 53 disable
    delete interfaces pppoe pppoe0 disable

Both boxes now: 21 NAT rules, 58 firewall rules, full PPPoE. Backup delta
validated against a real VyOS config with the disable lines present -- commits
clean. Runbook updated with the takeover procedure and the warning that it must
only be done when vyos001 is genuinely down.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-08-17 00:30:23 +01:00
Michal
64e748ea94 test(labsim): conntrack-sync verified, and it exposed a delta defect
conntrack-sync proven working on the sim pair -- bidirectional replication with
zero errors:

  MASTER  internal 34  external(from peer) 52   62 pkts sent / 109 recv  0 err
  BACKUP  internal 76  external(from peer) 36  142 pkts sent /  73 recv  0 err

Getting there required learning something that changes the production config:
**VyOS only engages conntrack when a firewall or NAT is configured.** With
neither present, both routers reported zero conntrack entries and conntrack-sync
had nothing to replicate. Adding a single state-matching forward rule turned
tracking on and replication began immediately.

That is a defect in the delta, not just a test artifact. NAT and the firewall
were both gated behind --with-wan, so the BACKUP would have had neither -- it
would not have tracked connections at all, and replicated entries are useless to
a box whose conntrack is not engaged. Exactly the failure that only shows up
during a failover, when it is too late to notice.

Fixed: a stateful forward rule (accept established/related, default-action
accept) is now emitted on BOTH boxes, outside the WAN gate. Only NAT and the
WAN-scoped rules remain master-only. Verified: vyos002 now carries stateful
tracking and conntrack-sync but zero NAT lines. Master delta re-validated
against a real VyOS config -- no errors.

Also incidentally confirmed no-preempt: router1 rebooted and came back as
BACKUP rather than seizing the VIP, which is the opposite of what the production
pair did this afternoon (still on default preempt until cutover).

Two traps recorded while doing this:

  - The detached `setsid nohup` config-apply pattern can strand a VyOS config
    session. An orphaned session (dirs under /opt/vyatta/config/tmp/, PID long
    dead) blocked every subsequent `set` on that box with a bare "Set failed",
    and the dirs are overlay mounts so they cannot simply be deleted. Rebooting
    cleared it. This pattern is used to survive losing SSH mid-change, so it is
    worth knowing it has a failure mode of its own.
  - Only VLAN 10 passes traffic between the two sim routers; every other VLAN
    fails ARP despite identical vlan_mode/tag/trunks on both OVS bonds and
    distinct MACs. VRRP forms on all six groups regardless. The sync link had to
    be bond0.10 as a result. OVS-specific, absent in production, but it means
    the sim proves mechanism rather than topology.

Production deltas regenerated with --conntrack-link: eth3 at 10.255.255.1/30 and
.2/30 awaiting the cable, which is not yet plugged (carrier=0 on both).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-08-17 00:17:21 +01:00
Michal
bb654d83f8 test(labsim): second VyOS router proves DHCP active-passive HA
Answers the question a single router could not, and that would otherwise only
have been discovered at cutover: with kea high-availability active-passive, does
exactly ONE box answer a DHCP request?

Yes. Probing sim VLAN 10 with broadcast-dhcp-discover returns offers from a
single distinct Server Identifier -- 172.31.10.252, the primary. The secondary
runs kea but stays silent. Without this the delta would have put 6 subnets and
84 static-mappings on both boxes with nothing to arbitrate them, and two kea
instances would have raced on every broadcast domain.

Worth noting the raw response count is misleading: nmap reports "Response 1 of
2" because it sends several discovers, and both replies carry the same server
identifier. Counting responses says "2 servers"; counting distinct server
identifiers says "1". The second number is the true one.

labsim now runs a real pair, mirroring production:

    router1  172.31.<v>.252  priority 200  DHCP HA primary
    router2  172.31.<v>.253  priority 100  DHCP HA secondary
    VIP      172.31.<v>.1    floating, held by the master

That required converting router1, which held .1 directly, to .252 plus a
floating VIP -- otherwise it is two routers, not a pair. All six VRRP groups
show MASTER on router1 and BACKUP on router2.

New tooling:

  - sim-ha-config.py generates each role's config, reusing unifi-to-vyos.py
    --mode sim for the DHCP half so what is proven here and what production
    gets share a code path. VLAN 10 correctly carries /23.
  - console-apply.py applies config over the serial console, which is necessary
    because a freshly installed VyOS holds the same addresses as its peer and
    cannot safely be reached over the network at all until reconfigured.

Known sim-only quirk, deliberately not chased: router1 cannot ARP router2 on
the untagged VLAN 1 while every tagged VLAN works, and VRRP forms correctly on
all six groups regardless. Both OVS bonds carry identical vlan_mode/tag/trunks
and the bond MACs differ, so this is OVS bond behaviour on the native VLAN with
two bonds on one bridge -- not a VyOS config problem, and not present in
production, which uses a real switch.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-08-16 23:29:12 +01:00
Michal
952f5c66e3 feat(migration): complete the VyOS HA stack per the official docs
Prompted by "I thought we tested HA on libvirt" -- checking rather than
recalling showed the sim has ONE VyOS router with zero high-availability
config. VRRP was configured and running on the real pair, but it is only one of
four parts of what VyOS considers an HA pair.

Against docs.vyos.io (highavailability, conntrack-sync, dhcp-server, and the HA
walkthrough), three gaps are now closed in the delta:

  - VRRP was multicast-only with default preemption. Added unicast
    hello-source-address/peer-address per group, as the walkthrough does, plus
    no-preempt. Without no-preempt a recovered box reclaims the VIP before
    conntrack state has synced and drops every established connection; the docs
    are explicit that preempt-delay must otherwise be >= purge-timeout.
    The per-VLAN node addresses are a table, not derived: VLAN 3 is .4/.5 while
    every other VLAN is .252/.253.
  - DHCP high-availability, which fixes a real defect rather than adding a
    feature. Both boxes carried the full 6 subnets and 84 static-mappings, so
    after cutover two kea instances would have raced on the same broadcast
    domains. Now active-passive with primary/secondary and swapped
    source/remote, syncing over TCP 647 on the LoT addresses. Each subnet
    already carries the unique subnet-id kea HA requires, and the peer name
    deliberately differs from both host-names.
  - conntrack-sync over a dedicated eth3 <-> eth3 link, gated behind
    --conntrack-link because it needs a cable that is not plugged in yet. This
    is what the peer cable is actually for -- VRRP does not want one, since its
    hellos must travel on the segment they protect.

VRRP failover exercised on the production pair, which is free to break today
because nothing uses the .254 VIPs: keepalived stopped on vyos001, all six VIPs
moved to vyos002 within 12s, and returned on restart (preemption still default
on the live boxes). Both boxes clean afterwards, no config drift.

Master delta validated against vyos001's real running config on the sim router
before installing. Installed on both: 6 no-preempt, 6 unicast pairs, DHCP HA
primary/secondary respectively.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-08-16 22:48:39 +01:00
Michal
f81c94af43 feat(migration): dual WAN, cloned MAC, and the new 10.8.0.0/23 Private VLAN
Two corrections from reading the live USG instead of trusting UniFi's fields,
which report wan_type=dhcp for both WANs and are simply wrong:

  - There are TWO WANs, not one. WAN2 is the 10 gig ISP on VLAN 53, plain DHCP
    with a PUBLIC address (87.192.101.48/21, gw 87.192.96.1) on the USG's eth2 --
    and it is what actually carries traffic. WAN1 is Vodafone PPPoE on VLAN 51,
    the failover. The delta had PPPoE as the only WAN, which would have left the
    primary line unconfigured.
  - The DHCP lease is bound to MAC, so bond0.53 now clones the USG's WAN2 MAC
    (f0:9f:c2:12:9b:4f). That is how VyOS keeps the existing public lease rather
    than negotiating a new one -- or getting none, if the ISP allows one per
    line. Distances: 10 gig at 1, Vodafone at 10.

Only ONE box may hold the cloned MAC, so --with-wan gates the entire WAN, NAT
and firewall section. vyos001 gets it (320 set lines); vyos002 gets none (234,
zero WAN/NAT/firewall) and routes the LAN only. Pretending both could hold it
would have meant a duplicate MAC on VLAN 53 and a flapping switch table.

Private was rebuilt at 10.8.0.0/23 (VLAN 9) after the old 10.0.8.0/23 was
deleted. bond0.9 and the VRRP group were moved to 10.8.0.252/.253 with VIP
10.8.0.254 on both boxes, and the delta now targets 10.8.0.1.

Creating that network first required breaking a deadlock in UniFi: every LAN
write was rejected with api.err.WanIpOverlapped / 0.0.0.0/0, because WAN1 was
set to DHCP on a line that only speaks PPPoE, so it sat at 0.0.0.0 forever and
the validator treated that as a subnet overlapping everything. Verified
server-side, not a UI bug -- the API rejected it identically. Setting
wan_type=pppoe let it dial (90.241.226.213, MTU 1492), which cleared the phantom
overlap and incidentally PROVED the Vodafone credentials and line work, which
had been listed as untestable before cutover.

dhcp-options no-default-route-dns does not exist; the valid set is client-id,
default-route-distance, host-name, mtu, no-default-route, reject, user-class,
vendor-class-id. Caught by validating the delta against vyos001's real config on
the labsim router before installing.

After adding the network, the gateway's dhcpd.conf was checked with
`dhcpd3 -t -cf` (valid) and confirmed to contain the new subnet only after a
force-provision -- controller state is not device state.

Both boxes: mode unifi, VRRP unchanged, unifi.boot re-captured (232 lines,
carrying the new VLAN 9), no config drift.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-08-16 18:16:25 +01:00
Michal
febe4b72bc chore(migration): all DNS through VyOS to Google, NAS out of the path
The NAS is legacy for ad.itaz.eu and those records now live in Cloudflare, so
the zone resolves publicly -- verified: nas001.ad.itaz.eu and
kvm-macstudio1.ad.itaz.eu both answer from 8.8.8.8. That removes the reason for
a conditional forward and lets the NAS leave the DNS path entirely.

Two changes:

  - `service dns forwarding name-server` is now 8.8.8.8 and 8.8.4.4, the same
    pair the USG used on its WAN, instead of 10.0.0.194.
  - Every VLAN is handed the gateway as its resolver. UniFi set an explicit
    resolver on LoT only (the NAS); carrying that over would have kept the NAS
    in the path for one VLAN and not the other five, which is the sort of
    asymmetry nobody remembers a year later.

The NAS is still referenced 9 times, all legitimate and checked: 4 NAT
destination rules, the 4 matching firewall accepts for those port forwards, and
its own DHCP reservation. No DNS references remain.

Validated by loading vyos001's real running config on the labsim router and
applying the full delta -- all 318 commands accepted, no errors. Installed on
both boxes and verified in place: priority 200/100, upstream 8.8.8.8 + 8.8.4.4,
six client resolvers, 377 lines each.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-08-16 17:16:17 +01:00
Michal
3768657b91 chore(migration): firewalls resolve via 8.8.8.8/8.8.4.4
Matches the DNS the USG used on its WAN (wan_dns1/wan_dns2), replacing the
10.0.0.194 I had set earlier. Applied to both boxes and saved; VRRP unchanged
(MASTER/BACKUP), NTP still synced, no config drift.

/config/modes/unifi.boot was RE-CAPTURED on both afterwards. It had been taken
before this change, so the escape hatch would have quietly reverted the
resolver on any rollback -- a snapshot is only an escape hatch for the state it
was taken from.

Two things recorded in the runbook:

  - The boxes' name resolution now depends on the internet, so between
    unplugging the USG and PPPoE establishing they have no DNS. Harmless:
    nothing in the switch resolves a name, and the health checks use DNS
    precisely to prove the WAN came up.
  - Internal ad.itaz.eu names still resolve via Google, because that zone is
    published publicly with private addresses in it (nas001 -> 10.0.0.194,
    kvm-macstudio1 -> 192.168.3.8). So no conditional forward was needed --
    though it is worth knowing the internal topology is public.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-08-16 16:49:27 +01:00
Michal
2a8fcb3bd3 fix(migration): the runbook pointed at addresses that die with the USG
The access table led with 192.168.8.143/.144 and offered the LoT addresses as a
fallback ("if unreachable, try"). That is backwards and would have stranded the
operator at the worst moment: the workstation sits on LoT, and reaching
192.168.8.x routes *through the USG*, so those addresses are guaranteed dead the
instant it is unplugged. Measured:

  ip route get 192.168.8.143  ->  via 10.0.0.1   (the USG)
  ip route get 10.0.1.252     ->  dev lanbr0     (same L2, no gateway)

10.0.1.252 and .253 are on the LoT VLAN, same broadcast domain as the
workstation, and both answer SSH. They are now the only addresses the runbook
gives, with the k8s ones struck through.

Also recorded: the switch cannot be run before unplugging the USG (two devices
on every gateway address; the guard refuses), so the order is forced. And
during the gap between unplugging and completing the switch there is no
inter-VLAN routing at all -- which means the JetKVMs (Management and kvm) and
Tailscale are NOT fallbacks in that window. LoT SSH is the only remote path;
below it is physical console. Added a step 0: open both SSH sessions and leave
them open before touching anything.

Both boxes are now installed and pass the pre-flight gate: mode unifi,
unifi.boot 231 lines including the reload action, delta at the right priority
(200/100), wan-secrets 0600, script executable, no config drift, VRRP still
MASTER/BACKUP. `vyos-unifi-switch vyos` refuses on both -- all six gateway
addresses detected answering ARP -- and neither box has gained dhcp-server, dns
or nat, so nothing about their behaviour has changed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-08-16 16:25:04 +01:00
Michal
fc31013ceb fix(migration): apply the reviewed reservation plan, not a recomputed one
This caused a real outage. unifi-reserve-all.py recomputed its plan at --apply
time by re-reading stat/sta, so a client that renewed between the dry run and
the apply was pinned to whatever transient address it happened to hold at that
instant. worker1-k8s0 was reviewed at 192.168.8.13 and written as
192.168.8.242. On its next reboot it could not get an address at all, taking a
k8s node down.

A plan that gets reviewed and a plan that gets applied must be the same object.
The dry run now WRITES the plan to a file and --apply READS it and applies
exactly that, reporting any client whose current address has since drifted
rather than silently preferring the new value.

1 of 51 diverged; the rest were verified against the reviewed list and were
correct. worker1 has been restored to .13 and confirmed: DHCPOFFER for its own
MAC returns 192.168.8.13, and the node is up with a full lease and working
internet.

The second half of the outage was drift between controller and device: the USG
was still running config from ~16h before these changes, so the controller
looked perfectly correct while the gateway handed out something else. Writing
the controller is only half the job, so the script now says so explicitly and
gives the force-provision and DHCP-probe commands to verify with. `nmap
--script broadcast-dhcp-discover --script-args broadcast-dhcp-discover.mac=...`
is the way to prove a specific reservation is live without disturbing the
client -- it elicits an OFFER without ever sending a REQUEST.

_unifi.py gains post() for device commands.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-08-16 14:31:30 +01:00
Michal
b37cd79432 fix(migration): refuse an unprobeable delta instead of warning past it
The ARP guard against two devices holding the same gateway address is the one
check that prevents this script's worst outcome. It reads the addresses to
probe out of the delta -- so a delta with no VIP lines made the guard inert,
and it previously warned and carried on. That was a testing convenience (the
lab delta has no VIPs) weakening a production safety check, which is backwards.

It now refuses by default. ALLOW_NO_VIP_DELTA=1 is the explicit lab override.

The guard's probing path had never actually executed before this: every sim
run took the no-VIPs branch. Verified against the live USG from vyos001:
arping is present on VyOS, the regex extracts all six gateway addresses from
the real delta (192.168.1.1, 192.168.8.1, 192.168.3.1, 10.0.9.0, 10.0.0.1,
192.168.2.1), and every one of them answers ARP right now -- so on the real
boxes, with the USG connected, the guard fires.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-08-16 00:13:19 +01:00
Michal
7e464a2828 docs(migration): record what the rehearsal proved and what it could not
Ran vyos001's real running config plus the real production delta on the labsim
router -- same VyOS version, isolated OVS bridge with no physical NIC, so the
sim briefly holding vyos001's actual addresses could not reach the real LAN.

Result: all 317 commands accepted and the whole delta commits (COMMIT OK), the
revert is byte-exact, and auto-revert fires without rebooting (uptime and
boot-id unchanged across it).

Also written down are the two things this did NOT establish, because a runbook
that overstates its own coverage is worse than one that admits the gap: PPPoE
cannot be tried while the USG holds the single available session, and the
rehearsal ran with vyos001's eth2/eth3 stanzas stripped because the sim VM has
two NICs rather than four.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-08-16 00:08:13 +01:00
Michal
01a923352f fix(migration): do not emit a NAT translation port for a port list
`set nat destination rule N translation port '16881,6881'` is rejected --
"16881,6881 is not a valid service name" -- because mapping a list of ports
onto a list is ambiguous. `destination port` accepts the same list happily,
which is why this only shows up on the translation side.

All four UniFi port forwards map a port to itself, so translation port was
redundant anyway: omitting it makes VyOS preserve the original port, which is
exactly the intent. It is now emitted only when the forwarded port genuinely
differs, and generation fails loudly rather than producing a config that will
not commit if a differing port LIST ever appears.

Found by loading vyos001's real running config onto the labsim router and
applying the full delta to the candidate config without committing. Worth
noting the delta had already passed a read-through: this one only surfaced by
running it against a real VyOS of the same version.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-08-16 00:02:21 +01:00
Michal
7f5d3517a3 fix(migration): create the WAN vif before PPPoE references it
`set interfaces pppoe pppoe0 source-interface bond0.51` refers to an interface
that must already exist, and neither firewall has vif 51 -- only 2, 3, 9, 10
and 200 are configured. The commit would have failed, and since the whole delta
commits as one unit, that failure would have taken the entire switch with it at
the worst possible moment.

No address on the vif: PPPoE rides the VLAN and needs no L3 of its own.

Found by checking the running config against the generated delta rather than by
running it. The prod delta has still never been applied to any VyOS, which is
the remaining gap.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-08-15 23:51:13 +01:00
Michal
d56bbf6db0 feat(migration): reversible USG->VyOS switch, proven on the sim
The cutover is a switch, not a migration: unplug the USG, run one command, and
if anything is wrong run the other one and plug it back in. The operator will
have no internet during this and therefore no assistant, so the machinery has
to live on the boxes and the failure paths have to be proven in advance.

vyos-mode-delta.py generates the delta that turns the passive pair into the
gateway. Only one artifact is authored: gateway mode is always derived from
`load unifi.boot` + delta, so there is no inverse to maintain and no drift
between two hand-kept configs. It reuses unifi-to-vyos.py rather than
duplicating it, so what labsim proved and what production gets are one code
path. The PPPoE password is never written into the delta -- it carries a
placeholder the switch substitutes at apply time from /config/wan-secrets --
and generation fails if the real password appears in the output.

Two things the delta covers that the plan had underweighted:

  - VyOS defaults to ACCEPT while the USG has an implicit WAN drop. Migrating
    the port forwards alone would have left the router's own services and the
    whole LAN reachable from the WAN. Added a stateful baseline scoped to the
    WAN interface rather than a global default-action drop, so a mistake there
    cannot lock anyone out over the LAN -- the only way back during a cutover.
  - The old VIPs are NOT at network+254 on the /23 networks; they are
    192.168.9.254, 10.0.9.254 and 10.0.1.254, in the upper half. A delete
    naming a computed address fails quietly and leaves the group holding two
    VIPs. The delta deletes the whole address node instead of guessing.

vyos-unifi-switch runs on the box from /config, which survives image upgrades,
so it works from a local terminal or the JetKVM with no workstation.

Proven on labsim, not assumed:

  - unifi mode restores the previous config BYTE-EXACT (138 lines, diff clean).
  - Auto-revert fires when the commit is not confirmed: 85 static-mappings ->
    0, kea stopped, hostname restored, and uptime plus boot-id UNCHANGED, so
    it reloaded rather than rebooted. That distinction is the whole reason
    `commit-confirm action reload` is a prerequisite.
  - Health-check failure triggers an immediate revert_soft rather than waiting
    out the timer.

Four bugs found while doing it, each of which produced a wrong answer rather
than an error:

  - commit-confirm is TWO steps. `config-mgmt commit_confirm` only arms the
    revert timer; a normal `commit` still has to follow. Arming alone committed
    nothing while reporting success.
  - `sudo sg vyattacfg "config-mgmt ..."` loses the config-session environment,
    so it reported "No configuration changes to commit" against a candidate
    that plainly had 446 added lines.
  - `... | grep -q` under `set -o pipefail` reports FAILURE on a match: grep
    exits early, the producer takes SIGPIPE. Whether it triggers depends on
    output size, so `status` misreported the mode intermittently.
  - `show configuration commands` quotes values, so a fixed-string match for
    `action reload` never matched `action 'reload'`.

CUTOVER.md is the printable runbook: both reachable addresses per box, the
escape hatch first, and the note that PPPoE is the one thing that could not be
tested beforehand.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-08-15 23:30:28 +01:00
Michal
6c4318d3ae feat(migration): reserve every active client at its current address
kea does not inherit UniFi's lease database. At cutover it starts with an empty
view of who holds what, so it can hand an address that is currently in use to a
different device. Reservations are what carry "this device has this address"
across the switch, because they live in config rather than in lease state.

unifi-reserve-all.py creates one per active client, dry run by default. 51
written, 51/51 verified live by reading the records back; the controller now
holds 85 reservations and the generator emits all 85 with unique, valid
hostnames and no duplicate addresses. Active clients with no reservation went
from 48 to 4.

Three guards, each of which caught something real in the dry run:

  - VRRP virtual addresses are excluded. UniFi reports them as ordinary client
    addresses because the firewalls' bond MACs answer for them, and their
    apparent IP flips between the real interface address and the VIP. Without
    this, 192.168.1.254 -- the gateway VIP itself -- would have been given a
    DHCP reservation.
  - The firewalls' own interface MACs are excluded; those are statically
    configured routers, not DHCP clients.
  - Any address claimed by more than one MAC is dropped rather than guessed
    at. This is how the VIPs surfaced in the first place.

Also skipped: addresses already reserved to another MAC, network gateways, and
anything on a network that does not serve DHCP (which excludes the WAN transit
VLANs automatically).

labsim-dhcp-test.sh gained a lease-database flush, and it is not tidiness. Two
findings, both of which first appeared as a PASSING test:

  - Re-running against stale leases, kea gave dynamic addresses to three
    devices that have reservations. The reservations were present and correct
    in kea's own config throughout. Kea saw the reserved address as leased to
    "another client" -- same MAC, different client-id from the earlier boot --
    and allocated elsewhere. Cutover starts with an empty lease database so
    this is a testing artifact, but a reservation is evidently not
    unconditional once leases exist.
  - Removing only dhcp4-leases.csv does nothing: kea's memfile backend keeps
    lease-file-cleanup rotations (.csv.2) and restores from them on start.

The verdict logic no longer takes the first matching lease row. Doing so
reported an hours-old lease as the current answer and scored three failures as
passes, including one where the device had plainly been given a dynamic
address. A MAC with more than one lease is now an explicit failure rather than
a guess.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-08-15 23:07:07 +01:00
Michal
f36ff4c6e3 feat(migration): pin firewall management NICs and reserve them in UniFi
Prerequisite for the cutover. eth2 on both firewalls was DHCP-served by the USG,
and both addresses (192.168.8.143/.144) sit inside the pool VyOS will serve --
with no reservation for either MAC. After the switch they would renew from kea,
get no mapping, and the management addresses could move. That is the worst
possible moment for the address you SSH to to change, because losing the USG
also means losing internet and any outside help.

On both boxes, driven over the LoT path (bond0.10) so the interface being
changed was never the one carrying the session:

  - eth2 pinned static at its current address, so it no longer depends on DHCP
  - `system name-server eth2` replaced with 10.0.0.194. That setting inherited
    resolvers from the DHCP lease, i.e. the boxes were resolving via the USG and
    would have lost DNS with it. 10.0.0.194 is reachable directly over bond0.10
    and is authoritative for ad.itaz.eu, so internal names now resolve on the
    firewalls -- they did not before.
  - static default route via 192.168.8.1, replacing the one the lease provided.
    Superseded by PPPoE in vyos mode; this keeps unifi mode as it was.

Verified after each: SSH on the pinned address, external and internal DNS, NTP
still synced, VRRP unchanged (vyos001 MASTER, vyos002 BACKUP).

migration/unifi-reserve.py adds the matching UniFi reservations so the
controller cannot lease those addresses to anything else, keeping the
management address identical in both modes. It reads the record back after
writing, because a controller accepting a PUT is not proof it stored what was
asked for, and it is idempotent.

Also noted while doing this: VyOS `commit-confirm` REBOOTS the box if not
confirmed -- "Minutes until reboot, unless 'confirm'" -- it does not roll the
config back in place. For a gateway that means a real outage window, which
changes how the switch script must use it. `config-mgmt commit_confirm -y`
executes without the interactive prompt.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-08-15 22:12:42 +01:00
Michal
44dbd5188c feat(migration): export UniFi config and generate VyOS DHCP+DNS from it
Groundwork for replacing the USG with the VyOS pair without anything on the
network noticing. Three pieces:

migration/unifi-export.py pulls 13 endpoints off the classic controller into
timestamped JSON plus a normalised inventory: 11 networks, 31 DHCP
reservations, 4 port forwards, 2 firewall rules, 0 static routes. Two things
this turned up that a naive export would have lost:

  - 23 of the 31 reservations carry no network_id at all -- UniFi simply does
    not store the binding -- so they are resolved by subnet containment
    instead. Without that, three quarters of the reservations have no subnet
    to be placed in.
  - 30 of the 31 sit INSIDE the DHCP pool, which UniFi's dhcpd tolerates and
    which is flagged as a warning rather than discovered at cutover.

migration/unifi-to-vyos.py turns that inventory into VyOS `set` commands for
DHCP and DNS only -- the services the USG owns that VyOS must reproduce. Not a
general converter. --prod and --sim come from one code path so the config
proven in the sim and the config applied to the firewalls cannot drift. Prod
mode hard-fails if any reservation is missing, since a silent drop is the
failure mode that matters.

DNS is included because the USG resolves for 5 of 6 VLANs today: UniFi hands
out the gateway's own address whenever dhcpd_dns is empty, verified by
labmaster resolving against 192.168.8.1. Replacing the USG without a forwarder
would take DNS away from those VLANs entirely.

labsim/labsim-dhcp-test.sh proves it by booting throwaway VMs with real
production MACs -- the one piece of production config that transplants
verbatim. Safe because ovs-labsim has no physical NIC, so those MACs cannot
reach the real LAN.

Result on VyOS 2026.08 (kea), 4/4: printer1 got 172.31.10.46 from inside the
pool, sonoff-matter got 172.31.11.67 across the /23 boundary, Hubitat got its
out-of-pool .2, and an unreserved MAC got an unreserved address. kea honours
in-pool host reservations -- the open question blocking the cutover.

Supporting changes to labsim:

  - VLAN 10 widened to /23. Every reservation is in LoT and LoT spans 10.0.0.x
    and 10.0.1.x, which a /24 cannot represent.
  - LoT's host leg moved to .3, because 10.0.0.2 is a real reservation
    (Hubitat) that maps onto the host's own address.
  - vlans.conf gained optional masklen and host_octet fields, defaulting to
    24 and 2 so the other five VLANs are untouched.
  - Fixed /etc/network/interfaces hardcoding 255.255.255.0. That file is what
    actually takes effect on these Alpine guests -- cloud-init's
    network-config is ignored -- so any non-/24 VLAN was silently wrong.

Two generator bugs found by VyOS rejecting the output: static-mapping names
are validated as hostnames, so underscores fail; and two devices named
"espressif" plus two named "thebeast" collided into single names, which would
have overwritten one reservation with another's address.

The raw export holds WiFi passphrases and the WAN PPPoE credentials and is
gitignored.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-08-15 01:33:00 +01:00
Michal
b0b68f2edd Merge feat/vyos-unattended-install: VyOS HA install + DiskPressure incident fixes
Some checks failed
CI/CD / lint (push) Failing after 12s
CI/CD / test (push) Failing after 9s
CI/CD / typecheck (push) Failing after 25s
CI/CD / build (push) Has been skipped
CI/CD / publish-rpm (push) Has been skipped
CI/CD / publish-deb (push) Has been skipped
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017f6jyeeDqP4ufyeL3UER9w
2026-08-14 23:22:21 +01:00
Michal
72c54edce2 feat(k3s): enable swap and grow the rancher LV during host-prep
Some checks failed
CI/CD / lint (pull_request) Failing after 10s
CI/CD / test (pull_request) Failing after 10s
CI/CD / typecheck (pull_request) Failing after 22s
CI/CD / build (pull_request) Has been skipped
CI/CD / publish-rpm (pull_request) Has been skipped
CI/CD / publish-deb (pull_request) Has been skipped
Replace the CIS-style disableSwap op with enableSwap: activate the
labvg-swap LV with an fstab entry (kubelet runs failSwapOn=false; zram
stays the fast tier, the LV is overflow before OOM kill). Add
growRancherLv: extend labvg/rancher to 120G when the VG has free space,
covering nodes installed before the kickstart sizing change and vanilla
nodes converted to k8s; skips with a clear message when the VG is full.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017f6jyeeDqP4ufyeL3UER9w
2026-08-14 23:22:14 +01:00
Michal
33be713d0c feat(bastion): size the rancher LV at 120G for k8s roles in kickstart
The 20G /var/lib/rancher LV (k3s imageFs) idled at 85% used from
steady-state images alone; one ~5G image pull tripped imagefs eviction
and evicted unrelated pods (2026-08-14 DiskPressure incident). Create
the LV for both worker and infra roles at 120G — it must be sized here
because longhorn's --grow consumes all remaining VG space, making
post-install lvextend impossible.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017f6jyeeDqP4ufyeL3UER9w
2026-08-14 23:22:14 +01:00
Michal
a5b36678ed feat(labsim): live topology view with per-path latency
Some checks failed
CI/CD / lint (pull_request) Failing after 9s
CI/CD / test (pull_request) Failing after 9s
CI/CD / typecheck (pull_request) Failing after 24s
CI/CD / build (pull_request) Has been skipped
CI/CD / publish-rpm (pull_request) Has been skipped
CI/CD / publish-deb (pull_request) Has been skipped
The Grafana heatmap of 1s and 0s said almost nothing, and the state timeline
was an unreadable pile of overlapping series labels. Replaced as the primary
view with a purpose-built page served by the exporter itself.

- Probe now captures ICMP RTT, exposed as labsim_rtt_ms{src,dst}. A path that
  is up but slow is a different problem from one that is down, and a pass/fail
  grid cannot show it.
- Exporter serves / (topology), /api/matrix (JSON) and /metrics.
- topology.html: node per VLAN in a ring, VyOS router in the centre because
  every inter-VLAN packet really does traverse it, one line per pair coloured
  green/red with the RTT on it. Hovering gives per-direction state. A node ring
  goes red if anything to or from it is blocked. Side panels list blocked paths
  and the slowest links. Refreshes every 5s, no dependencies.

Grafana stays for what it is actually good at — history of when a path flipped.

Label placement is deliberate: RTT captions sit ~32% along each edge with a
perpendicular nudge, because every diagonal of a 6-node mesh crosses the centre
and midpoint labels stack on the router node.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-08-13 00:56:06 +01:00
Michal
c91e44f796 feat(labsim): libvirt replica of the lab network with LACP + VyOS routing
Some checks failed
CI/CD / lint (pull_request) Failing after 11s
CI/CD / typecheck (pull_request) Failing after 10s
CI/CD / test (pull_request) Failing after 9s
CI/CD / build (pull_request) Has been skipped
CI/CD / publish-rpm (pull_request) Has been skipped
CI/CD / publish-deb (pull_request) Has been skipped
A throwaway copy of the production VLAN topology so routing and firewall
changes can be tested before they touch the real network. Same VLAN IDs and
roles as UniFi, deliberately different ranges (172.31.<vlan>.0/24) so nothing
here can be mistaken for production.

- OVS fabric: real 802.1Q. Access port per micro VM, host leg per VLAN (.2,
  for SSH only — NOT the VMs' default route, so inter-VLAN tests exercise the
  router rather than the host's routing table), and a trunk portgroup with
  VLAN 1 declared nativeMode='untagged'.
- Six Alpine micro VMs (256MB, copy-on-write overlays on one 176MB image),
  SSH + a hello-world HTTP page naming the VLAN.
- VyOS router installed to disk unattended over the console, with the SAME
  config shape as the VP2440s: two NICs in an LACP bond carrying the trunk,
  VLAN 1 native, bond0.<vlan> holding the .1 gateway on each.
- labsim-matrix.py: full-mesh ICMP/TCP22/TCP80 probe, ~0.2s, --watch
  highlights cells that changed since the last sweep. Guest-side probe is
  python3 (already present via cloud-init) so nothing is installed on VMs
  that have no internet.
- Prometheus + Grafana (anonymous auth, no login) with a provisioned
  dashboard: heatmap plus a state timeline showing exactly when a path
  flipped. Verified end to end: one VyOS rule took sum(labsim_reachable)
  from 90 to 84, blocking precisely kvm<->k8s across all three protocols.

Traps found building this, all now encoded in the scripts:
- virtio-net breaks 802.3ad: the guest's bonding driver reports slaves
  "MII Status: down" despite carrier=1 and never sends an LACPDU, so the bond
  sits in AD_STATE_DEFAULTED. e1000e fixes it with no other change. Matches
  the netdev thread "bonding (IEEE 802.3ad) not working with qemu/virtio".
- OVS defaults bonds to active-backup, which does not speak LACP at all —
  bond_mode=balance-tcp is required.
- LACP deadlock: OVS holds members disabled until negotiation while the
  partner needs carrier before it will send LACPDUs. lacp-fallback-ab breaks it.
- LACPDUs are untagged, so a trunk with no native VLAN has nowhere to put them.
- --boot cdrom,hd re-runs the ISO on every restart, so every commit+save went
  to a live system that evaporated. Install now switches the VM to boot hd.
- cloud-init on Alpine: users stay locked without lock_passwd:false, one
  failing runcmd aborts the rest, busybox here has no httpd applet, and
  start-stop-daemon --exec /usr/bin/python3 matches cloud-init's own python3.
- The user-data heredoc is unquoted, so backticks in a COMMENT were executed
  by the host shell and their output corrupted the YAML. build_seed now
  validates with yaml.safe_load before building the ISO.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-08-13 00:42:39 +01:00
Michal
df2dfc5d71 fix(bastion): pin the VyOS boot NIC by MAC, and detect pre-installer stalls
Some checks failed
CI/CD / typecheck (pull_request) Failing after 10s
CI/CD / test (pull_request) Failing after 10s
CI/CD / lint (pull_request) Failing after 24s
CI/CD / build (pull_request) Has been skipped
CI/CD / publish-rpm (pull_request) Has been skipped
CI/CD / publish-deb (pull_request) Has been skipped
Both Protectli VP2440s failed to install on real hardware: they fetched
kernel+initrd and then went silent. The console showed why —

  Looking for a connected Ethernet interface ... e2 ? e3 ? e4 ? e5 ?
  Connected e4 found
  Connected e5 found
  [4.595647] igc 0000:02:00.0 e2: NIC Link is Up
  IP-Config: e4 ... no response after 15 secs - giving up
  Unable to find a live file system on the network

live-boot picks the first *connected* interface. The i40e SFP+ pair links
before the igc copper port (up at 4.6s), so it chose the fiber ports, which
have no DHCP, and never tried the NIC that actually PXE booted.

Fix: pass BOOTIF=01-<mac> on the kernel cmdline. live-boot's
Device_from_bootif() (verified present in this image) matches it against
/sys/class/net and sets DEVICE directly. The MAC comes from the dispatch
key — i.e. exactly the NIC that PXE booted — which is more reliable than
iPXE's ${net0} on a box where the booting NIC may not be net0.

Why the integration test missed it: the VM had ONE NIC, so "first connected
interface" was trivially correct, and virtio links instantly so there was no
negotiation race. createPxeVm now takes decoyNics, attaching extra NICs
ahead of the PXE NIC on a network with no route to the bastion; the VyOS
test uses 2. Without BOOTIF that reproduces the hardware failure. getVmMac
is network-aware so it still returns the booting NIC.

Also: the bastion had every clue and said nothing — it logged INSTALL
STARTED, served kernel+initrd, then nothing for 7 minutes. dispatch now
stamps dispatched_at, and /api/logs/:mac returns stalled_for_s / stalled
(8 min threshold, sized for the ~600MB squashfs fetch), so a machine wedged
before the installer environment comes up is diagnosable without a console.

Verified on hardware: both firewalls installed, bond0 802.3ad + VLANs
2/3/9/10/200 + VRRP (priority 200/100, VIP .254 per VLAN) applied, and
/config/lab-provisioned written.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-08-12 12:35:09 +01:00
Michal
e36a7a193c chore(cli): regenerate shell completions for VyOS flags
Some checks failed
CI/CD / lint (pull_request) Failing after 23s
CI/CD / typecheck (pull_request) Failing after 23s
CI/CD / test (pull_request) Failing after 23s
CI/CD / build (pull_request) Has been skipped
CI/CD / publish-rpm (pull_request) Has been skipped
CI/CD / publish-deb (pull_request) Has been skipped
pnpm completions:check was failing: labctl.fish/bash were stale. The
generated --os choices still listed only fedora-43 and ubuntu-26.04
(missing vyos-rolling since the OsId union gained it), and none of the
--vyos-*/--vlan flags were present.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-08-11 11:25:20 +01:00
Michal
5d00c42f5a feat(bastion): bring VyOS provisioning to Fedora-grade quality
Some checks failed
CI/CD / typecheck (pull_request) Failing after 10s
CI/CD / test (pull_request) Failing after 9s
CI/CD / lint (pull_request) Failing after 24s
CI/CD / build (pull_request) Has been skipped
CI/CD / publish-rpm (pull_request) Has been skipped
CI/CD / publish-deb (pull_request) Has been skipped
Ports the Fedora provisioning features that matter for a router onto the
VyOS path, and adds the libvirt integration test that proves them.

- Live install logs: the driver streams the installer pty (ANSI-stripped,
  batched, best-effort) to POST /api/log, so `labctl provision logs -f`
  works during a VyOS install the way Anaconda's syslog does for Fedora.
- installed.ip: report "ready at <ip>" -- the exact detail format
  routes/api.ts parses -- using the static mgmt address when known, else
  the live DHCP address. Without it VyOS machines landed with an empty IP,
  breaking provision list, logs-by-IP, recheck and reprovision.
  api.ts also guards the complete handler: VyOS boxes get the "vyos" SSH
  hint and never trigger the k3s post-provision.
- EFI network-first boot order: port of the Fedora %post efibootmgr step,
  run from the live env after install (NVRAM, not disk). Best-effort.
- Reinstall semantics: VyOS's installer already carries the previous
  config and SSH host keys forward -- the analog of Fedora's LV
  preservation -- so that stays the default. New --vyos-fresh-config
  overwrites the installed config.boot with the generated one instead,
  via a post-install target mount that also writes /config/lab-provisioned
  (mirrors Fedora's /etc/lab-provisioned, survives image upgrades).
- reprovision/recheck default to the "vyos" SSH user for VyOS machines.

Two hangs found by the VM test and fixed:
- On reinstall the installer asks "Would you like to copy data to the new
  image?" (search_previous_installation). Unanswered, the driver blocked
  on stdin until its stall timeout -- a silent 15-minute hang.
- The RAID regex missed "Would you like to choose two disks for RAID-1
  mirroring?", which would wedge any multi-disk box. Both prompts default
  to yes, so a miss also risks an unwanted mirror.

Both are now covered by a unit test asserting all 17 installer prompts
match exactly one rule -- verified to fail against the unfixed code, so
this class of bug is caught in a second instead of a 45-minute VM run.

tests/integration/vyos-provision.test.ts: fresh install, reinstall
preserves config + /config data, and freshConfig override. All 8 pass
against the real nightly ISO (EXIT=0). 273 unit tests pass; no new lint
errors in touched files.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-08-11 11:16:25 +01:00
Michal
cb9d99dd69 feat(bastion): unattended VyOS network install with HA (bond + VRRP)
Some checks failed
CI/CD / lint (pull_request) Failing after 10s
CI/CD / test (pull_request) Failing after 10s
CI/CD / typecheck (pull_request) Failing after 23s
CI/CD / build (pull_request) Has been skipped
CI/CD / publish-rpm (pull_request) Has been skipped
CI/CD / publish-deb (pull_request) Has been skipped
VyOS ships no unattended installer (install_image() is unconditionally
interactive; --no-prompt is wired only to 'add'), so the automation is
injected through live-config's hooks component: iPXE boots the live
kernel with fetch= and live-config.hooks=, the hook fetches a generated
per-MAC Python driver, and the driver builds config.boot, stages the
rootfs, and drives the interactive installer over a pty.

Bastion:
- vyos-boot.ipxe template (no 'nonetworking' — breaks the hook fetch;
  no console=ttyS0 — 30s/systemd-phase on UART-less boards)
- /vyos/autoinstall.sh + /vyos/install.py routes (per-MAC driver with
  the config spec baked in as base64)
- vyos-config-spec: bond0 (802.3ad) + tagged VLANs + VRRP groups
  (vrid = VLAN id) + sync group + hw-id pinning by MAC + SSH keys;
  config built from the image's own config.boot.default via
  vyos.configtree, version footer reattached via component_version
- prepareVyosArtifacts: extract kernel/initrd/squashfs from the nightly
  ISO with xorriso; initrd picked by size from regular files only;
  ISO URL "latest" resolves the newest vyos-nightly-build GH release
  (downloads.vyos.io no longer serves direct ISOs)

Verified end-to-end in a libvirt VM against the real nightly ISO —
installed system boots with bond/VRRP/hw-id config applied and no
migrations. Fixes found by the VM run, encoded in code comments:
config.boot.default lives at /usr/share/vyos at hook time; fetch= boot
has no medium so the rootfs is symlinked to the installer's expected
path; reboot must be --force (the hook is a child of the still-starting
live-config unit); installer disk answers are full /dev paths; zram0
passes the 2GB min-disk filter so the disk is always pinned.

CLI/labd: vyos spec threaded through provision install (--vyos-* and
--vlan/--vlan-vip flags with guards), labd install route, protocol
command-install, and the bastion's direct /api/install.

268 unit tests pass; no new lint errors in touched files.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-08-10 21:45:22 +01:00
Michal
4f9a6f64e4 k3s/networking: codify Multus + vlan-setup as lab operations
Some checks failed
CI/CD / lint (push) Failing after 11s
CI/CD / typecheck (push) Failing after 9s
CI/CD / test (push) Failing after 9s
CI/CD / build (push) Has been skipped
CI/CD / publish-rpm (push) Has been skipped
CI/CD / publish-deb (push) Has been skipped
The macvlan/VLAN-10 foundation HA depends on (Multus meta-CNI + a lan10
sub-interface + reference CNI plugins) was applied by hand during the HA
migration. Codify both as idempotent lab operations in the networking group,
after installCilium (which already sets cni.exclusive=false + bpf.vlanBypass={10}).
A fresh cluster now reproduces the full macvlan stack.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-07 15:37:20 +01:00
Michal
12fa954a05 k3s/cilium: install with cni.exclusive=false + bpf.vlanBypass={10} (Multus + VLAN-10 macvlan mDNS)
Some checks failed
CI/CD / lint (push) Failing after 14s
CI/CD / typecheck (push) Failing after 9s
CI/CD / test (push) Failing after 9s
CI/CD / build (push) Has been skipped
CI/CD / publish-rpm (push) Has been skipped
CI/CD / publish-deb (push) Has been skipped
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-06 21:06:24 +01:00
7181a61cec Merge pull request 'fix(labd): wire v2.0 Phase 1 routes + smoke tests' (#15) from fix/v2-wire-and-smoke-test into main
Some checks failed
CI/CD / lint (push) Failing after 10s
CI/CD / test (push) Failing after 9s
CI/CD / typecheck (push) Failing after 22s
CI/CD / build (push) Has been skipped
CI/CD / publish-rpm (push) Has been skipped
CI/CD / publish-deb (push) Has been skipped
2026-05-05 21:18:42 +00:00
Michal
cdf3b5c045 fix(labd): wire v2.0 Phase 1 routes into createApp + smoke tests
Some checks failed
CI/CD / typecheck (pull_request) Failing after 11s
CI/CD / test (pull_request) Failing after 9s
CI/CD / lint (pull_request) Failing after 22s
CI/CD / build (pull_request) Has been skipped
CI/CD / publish-rpm (pull_request) Has been skipped
CI/CD / publish-deb (pull_request) Has been skipped
The v2.0 Phase 1 commit (04faa07) added AuthService, RbacService,
ResourceStore, AuditService, the bearer auth middleware, and the
v2-auth/environments/resources route files, but createApp() never
registered any of them. They sat in the codebase as dead code: a
running labd would 404 on /api/auth/login, /api/resources, /api/events,
etc.

Wiring (server.ts)
- Instantiate AuthService, RbacService, ResourceStore, AuditService at
  app creation. Cast DbClient to PrismaClient (the runtime db is a real
  PrismaClient; DbClient is a structural shim).
- Start AuditService timer, register an onClose hook to stop it on
  shutdown so we never lose the last batch.
- Register v2 routes inside a Fastify scope with the bearer-auth
  middleware as preHandler. v1 routes (registered on the root scope)
  are unaffected so existing labd clients keep working.

AuditService (audit.ts)
- Expose flushPending() so tests can deterministically observe events
  without leaning on the 5-second flush interval. Implementation
  delegates to the existing private flush().

Smoke tests (v2-smoke.test.ts, 11 cases)
- Bootstrap: first POST /api/auth/login with empty users creates the
  admin (role=ADMIN, hashed password), returns a 64-hex token, marks
  isBootstrap=true, emits an auth_bootstrap audit event. Second login
  uses the normal flow. Wrong password returns 401 and audits failure.
  Missing credentials returns 400.
- RBAC: missing/empty/invalid bearer tokens return 401. ADMIN role
  bypasses RBAC. A non-admin with no role bindings gets 403 with
  "no matching role binding". A user with an env-A binding is denied
  for env-B resources.
- Audit: bootstrap event is queryable via /api/events?correlation=...
  Explicit parent/child chain (shared correlationId, parentEventId)
  is preserved across emits.

All 246 workspace tests pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 22:18:18 +01:00
f3c50f71ef Merge pull request 'feat: v2.0 Phase 1 foundation + bastion-restart identity fix + Dockerfile + BASTION_DIR' (#14) from feat/v2-phase1-foundation into main
Some checks failed
CI/CD / lint (push) Failing after 22s
CI/CD / typecheck (push) Failing after 21s
CI/CD / test (push) Failing after 22s
CI/CD / build (push) Has been skipped
CI/CD / publish-rpm (push) Has been skipped
CI/CD / publish-deb (push) Has been skipped
2026-05-05 21:10:25 +00:00
Michal
98b0ccc6c9 feat(cli): honor BASTION_DIR env var as default for --dir
Some checks failed
CI/CD / typecheck (pull_request) Failing after 21s
CI/CD / test (pull_request) Failing after 22s
CI/CD / lint (pull_request) Failing after 7m2s
CI/CD / build (pull_request) Has been skipped
CI/CD / publish-rpm (pull_request) Has been skipped
CI/CD / publish-deb (pull_request) Has been skipped
bastion serve/stop default for --dir was hardcoded to /tmp/lab-bastion.
Now reads BASTION_DIR from env if set, so a deployed bastion daemon
can run from a persistent directory without callers having to pass
--dir on every invocation.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 22:09:24 +01:00
Michal
37a3b51e57 build(labd): include @lab/core in the Dockerfile build chain
The v2.0 Phase 1 commit (04faa07) introduced the @lab/core package but
the labd Dockerfile still only copied @lab/shared and @lab/labd, so the
container build would fail to resolve @lab/core imports.

Both stages updated:
- Builder: copy @lab/core package.json/tsconfig + src, add it to the
  build order between @lab/shared and @lab/labd.
- Runtime: copy @lab/core dist and package.json into the final image.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 22:09:24 +01:00
Michal
d6e1f3c74d fix(labd): preserve machine identity across bastion restarts
The worker0-k8s0 bug: when labd restarts, the in-memory installed map
is lost. The next DHCP/PXE re-discovery for that MAC ran an upsert that
wrote status="discovered", silently downgrading the DB record from
"online" or "offline" and erasing the machine's known hostname/role
identity from the CLI view.

- server.ts: drop status="discovered" from the upsert update branch so
  re-discovery cannot downgrade an installed record.
- routes/bastions.ts (/api/machines): when the DB knows a real
  hostname+role for a MAC currently only in live.discovered, promote
  it back to live.installed so the CLI sees the right state. Also
  reordered the live-vs-DB fallback so DB online/offline maps to
  live.installed and the discovered branch is the else.
- tests: 3 new vitest cases covering promotion, fresh-discovery, and
  unknown-MAC fallback.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 22:09:24 +01:00
Michal
52e831b8c1 Merge branch 'main' into feat/v2-phase1-foundation 2026-05-05 22:06:34 +01:00
f5af24699a Merge pull request 'fix(k3s): audit logs via journald + etcd recovery' (#13) from fix/k3s-audit-via-journald into main
Some checks failed
CI/CD / typecheck (push) Failing after 11s
CI/CD / test (push) Failing after 9s
CI/CD / lint (push) Failing after 21s
CI/CD / build (push) Has been skipped
CI/CD / publish-rpm (push) Has been skipped
CI/CD / publish-deb (push) Has been skipped
2026-05-05 20:29:51 +00:00
Michal
dd92147341 fix(k3s): route audit logs through journald, codify etcd member recovery
Some checks failed
CI/CD / typecheck (pull_request) Failing after 13s
CI/CD / lint (pull_request) Failing after 23s
CI/CD / test (pull_request) Failing after 10s
CI/CD / build (pull_request) Has been skipped
CI/CD / publish-rpm (pull_request) Has been skipped
CI/CD / publish-deb (pull_request) Has been skipped
Two changes prompted by today's etcd raft panic on worker1-k8s0
(tocommit out of range, lost-write on follower) and the cascading
disk pressure that surfaced underneath it.

Audit logs to journald
- kube-apiserver now uses audit-log-path=- so audit events flow to
  k3s.service stdout and into journald instead of growing files in
  /var/log/kubernetes. The previous setup combined apiserver's
  internal rotation with a logrotate *.log glob that double-rotated
  the rotated files into permanent orphans (observed: 7+ GB).
- New journald-limits operation writes a SystemMaxUse=2G drop-in so
  audit volume cannot fill /var/log even under bursty load.
- log-rotation operation repurposed to decommission the obsolete
  logrotate rule and reap leftover audit files. Idempotent: no-op
  on fresh installs.

Etcd member recovery
- New recoverEtcdMember(broken, peer, hostname) codifies the
  documented k3s recovery: stop k3s, etcdctl member remove, wipe
  /var/lib/rancher/k3s/server/{db,tls,cred}, restart, poll for
  rejoin. Refuses to operate when cluster size < 3 to preserve
  quorum.

Tests
- 7 new unit tests covering both decommission paths and the
  recovery procedure (54 total, all green).
- install.test.ts asserts the file-based audit args are gone.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 21:29:16 +01:00
Michal
04faa079e2 feat: v2.0 Phase 1 foundation — @lab/core, auth, RBAC, audit, resource store
New packages:
- @lab/core: Resource types, Output<T> (Pulumi), audit event types,
  auth types, environment/account types, resource kind registry

New Prisma schema (mcpctl pattern):
- User (email/password/bcrypt), Session (bearer tokens), Group, GroupMember
- ServiceAccount, RbacDefinition (JSON subjects + roleBindings)
- AuditEvent (correlation IDs, causal chains, fire-and-forget batching)
- Environment, Account (driver config, Infisical secret path), Binding
- Resource (generic, kind/name/env unique, origin/managedBy tracking)
- Secret, Fleet, FleetMember, GitSource
- Keeps v1.0 models: Server, Agent, Bastion, Cluster, JoinToken

New services:
- AuthService: bearer token login, bootstrap (first login creates admin),
  session management with 30-day expiry
- RbacService: environment-scoped permission checks, group membership,
  role hierarchy (admin > edit > view)
- AuditService: fire-and-forget event collection, batch 50 / flush 5s,
  correlation IDs for causal chains
- ResourceStore: CRUD with origin/managedBy, RBAC-enforced routes

New routes:
- POST /api/auth/login, POST /api/auth/logout (bearer token auth)
- GET/POST/PUT/DELETE /api/resources (RBAC-enforced CRUD)
- GET/POST /api/environments, GET/POST /api/accounts
- POST /api/accounts/bind, GET /api/bindings
- GET /api/events (audit query with --last, --kind, --env, --correlation)

New middleware:
- Bearer token auth (validates Authorization header, resolves user identity)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-02 01:42:28 +01:00
95c99cb4d5 Merge pull request 'docs: CLAUDE.md routing rules + TODOS.md from v2.0 review' (#12) from feat/recheck-and-fixes into main
Some checks failed
CI/CD / lint (push) Failing after 12s
CI/CD / typecheck (push) Failing after 22s
CI/CD / test (push) Failing after 12s
CI/CD / build (push) Has been skipped
CI/CD / publish-rpm (push) Has been skipped
CI/CD / publish-deb (push) Has been skipped
Reviewed-on: #12
2026-04-02 00:31:44 +00:00
Michal
2eda926d4c docs: add TODOS.md from v2.0 CEO review
Some checks failed
CI/CD / typecheck (pull_request) Failing after 12s
CI/CD / lint (pull_request) Failing after 21s
CI/CD / test (pull_request) Failing after 11s
CI/CD / build (pull_request) Has been skipped
CI/CD / publish-rpm (pull_request) Has been skipped
CI/CD / publish-deb (pull_request) Has been skipped
Project tracking for labctl v2.0 platform design. Includes P1 (arch doc update),
P2 (SSH emergency mode, Prometheus metrics), and P3 (graph viz, import, secrets rotation)
items from the CEO and eng review sessions.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-02 01:29:30 +01:00
Michal
70258a0cc3 Merge remote-tracking branch 'origin/main' into feat/recheck-and-fixes 2026-04-02 01:27:45 +01:00
Michal
e9944c5413 chore: add gstack skill routing rules to CLAUDE.md 2026-04-01 23:56:47 +01:00
22e2946e95 Merge pull request 'feat: provision recheck, hardware info preservation, ISO boot fixes' (#11) from feat/recheck-and-fixes into main
Some checks failed
CI/CD / typecheck (push) Failing after 11s
CI/CD / lint (push) Failing after 22s
CI/CD / test (push) Failing after 11s
CI/CD / build (push) Has been skipped
CI/CD / publish-rpm (push) Has been skipped
CI/CD / publish-deb (push) Has been skipped
Reviewed-on: #11
2026-04-01 17:11:33 +00:00
Michal
9ddab24931 feat: provision recheck, hardware info preservation, ISO boot fixes
Some checks failed
CI/CD / lint (pull_request) Failing after 1m26s
CI/CD / typecheck (pull_request) Failing after 11s
CI/CD / test (pull_request) Failing after 11s
CI/CD / build (pull_request) Has been skipped
CI/CD / publish-rpm (pull_request) Has been skipped
CI/CD / publish-deb (pull_request) Has been skipped
- Add `labctl provision recheck` to refresh hardware info via SSH
- Preserve hardware info in InstalledInfo when install completes
- Fix /ks-auto: run nested %pre scripts from included kickstarts
- Add command-discover WebSocket routing for hw info updates
- Fix k3s join: clean stale TLS/cred when joining existing cluster
- Add --tls-verify=false for internal HTTP registry pushes
- Add fix-ssh-root.sh script for root SSH access on all nodes

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-01 17:59:39 +01:00
Michal
ae91f2895e feat: dynamic /ks-auto kickstart for ISO boot (R1 ARM support)
Some checks failed
CI/CD / lint (push) Failing after 11s
CI/CD / typecheck (push) Failing after 22s
CI/CD / test (push) Failing after 7m5s
CI/CD / build (push) Has been skipped
CI/CD / publish-rpm (push) Has been skipped
CI/CD / publish-deb (push) Has been skipped
Add state-aware kickstart dispatch for machines that boot from ISO
(no PXE/network at UEFI level). Replaces hardcoded discover.ks.

- /ks-auto: %pre detects MAC, queries /api/machine-state/<mac>,
  writes discover or install kickstart to /tmp/dynamic.ks,
  main body %include's it
- /api/machine-state/<mac>: simple state endpoint returning
  unknown|discovered|queued|installing|installed|debug
- ISO kernel cmdline updated: discover.ks → ks-auto
- Handles: discovery (first boot), install (queued), debug modes

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-01 16:17:08 +01:00
Michal
06fc40a857 fix: k3s install automation — skip Cilium on join, Longhorn via server, default root user
Some checks failed
CI/CD / typecheck (push) Failing after 10s
CI/CD / test (push) Failing after 9s
CI/CD / lint (push) Failing after 22s
CI/CD / build (push) Has been skipped
CI/CD / publish-rpm (push) Has been skipped
CI/CD / publish-deb (push) Has been skipped
- Skip Cilium install for joining servers (already in cluster via daemonset)
- Longhorn annotation for workers: SSH to server node from CLI to apply
  kubectl annotation (workers don't have kubectl access)
- Default SSH user for k3s/app commands changed to 'root' (operations
  need root privileges, using 'lab' user broke installs)
- k3s server config: cluster-init for initial server, server+token for joins

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-01 16:02:19 +01:00
Michal
a68d6d617e feat: k3s cluster-init for etcd HA, fix Cilium duplicate install
Some checks failed
CI/CD / lint (push) Failing after 11s
CI/CD / test (push) Failing after 10s
CI/CD / typecheck (push) Failing after 22s
CI/CD / build (push) Has been skipped
CI/CD / publish-rpm (push) Has been skipped
CI/CD / publish-deb (push) Has been skipped
- Server config now uses cluster-init: true for initial server (enables
  embedded etcd). Joining servers get server: + token: in config.
- Cilium install already checks for existing installation, so joining
  servers skip it gracefully (the "release name in use" error is non-fatal)

Cluster rebuilt as etcd HA:
  worker0-k8s0  control-plane,etcd  (initial server, cluster-init)
  worker1-k8s0  control-plane,etcd  (joined server, Mac Studio aarch64)
  spark-2935    worker              (DGX Spark, aarch64)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-01 15:53:18 +01:00
Michal
c49a650888 fix: firstboot fstab handling — no duplicates, compatible with Asahi sed
Some checks failed
CI/CD / typecheck (push) Failing after 10s
CI/CD / test (push) Failing after 11s
CI/CD / lint (push) Failing after 23s
CI/CD / build (push) Has been skipped
CI/CD / publish-rpm (push) Has been skipped
CI/CD / publish-deb (push) Has been skipped
- Replace sed with grep -v / awk for fstab manipulation (Asahi Fedora's
  sed doesn't support \| delimiter or \? quantifier)
- Use idempotent write_lab_fstab function: removes all old entries first,
  comments out conflicting btrfs subvol entries, adds fresh LVM entries
- Fix sed for SSH hardening: use #* instead of \? (POSIX compatible)
- Tested on Mac Studio: no duplicate fstab entries after multiple runs

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-01 15:40:29 +01:00
Michal
87e09af941 fix: default admin user to 'lab', case-insensitive OS detection for iSCSI
Some checks failed
CI/CD / typecheck (push) Failing after 10s
CI/CD / test (push) Failing after 10s
CI/CD / lint (push) Failing after 22s
CI/CD / build (push) Has been skipped
CI/CD / publish-rpm (push) Has been skipped
CI/CD / publish-deb (push) Has been skipped
- Firstboot script defaults admin user to 'lab' instead of bastion's
  config.adminUser (which was 'michal' from host system)
- iSCSI OS detection uses case-insensitive match for 'fedora'

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-01 15:13:53 +01:00
Michal
6f13e284fd fix: firstboot script auto-detects hostname and MAC, no query params needed
Some checks failed
CI/CD / typecheck (push) Failing after 10s
CI/CD / test (push) Failing after 10s
CI/CD / lint (push) Failing after 23s
CI/CD / build (push) Has been skipped
CI/CD / publish-rpm (push) Has been skipped
CI/CD / publish-deb (push) Has been skipped
The firstboot script now auto-detects hostname (from hostnamectl) and
MAC address (from first UP interface) at runtime. No URL query parameters
required — just `curl bastion/asahi/firstboot.sh | sudo bash`.

Fixes the shell escaping issue where `&` in query params broke curl piping.
Updated labctl provision asahi instructions accordingly.

Tested on Mac Studio (worker1-k8s0): hostname, MAC, and bastion
registration all auto-detected correctly.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-01 15:05:25 +01:00
Michal
6c963a15bd fix: firstboot reprovision path now runs hostname, user, and registration
Some checks failed
CI/CD / lint (push) Failing after 12s
CI/CD / test (push) Failing after 10s
CI/CD / typecheck (push) Failing after 29s
CI/CD / build (push) Has been skipped
CI/CD / publish-rpm (push) Has been skipped
CI/CD / publish-deb (push) Has been skipped
Previously the reprovision path exited early after re-mounting LVs,
skipping hostname setup, admin user creation, metadata, and bastion
registration. Now both paths fall through to the common post-setup code.

Tested on Mac Studio (worker1-k8s0) — reprovision + self-registration
confirmed working via curl | bash pipe.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-01 09:59:02 +01:00
8c737d163d Merge pull request 'feat: Asahi Linux provisioning for Apple Silicon' (#10) from feat/asahi-provisioning into main
Some checks failed
CI/CD / lint (push) Failing after 11s
CI/CD / typecheck (push) Failing after 22s
CI/CD / test (push) Failing after 7m7s
CI/CD / build (push) Has been skipped
CI/CD / publish-rpm (push) Has been skipped
CI/CD / publish-deb (push) Has been skipped
2026-03-31 23:30:41 +00:00
Michal
17bae7ddbf fix: pre-download rootfs ZIP to avoid macOS Python HTTP streaming issues
Some checks failed
CI/CD / lint (pull_request) Failing after 11s
CI/CD / test (pull_request) Failing after 10s
CI/CD / typecheck (pull_request) Failing after 22s
CI/CD / build (pull_request) Has been skipped
CI/CD / publish-rpm (pull_request) Has been skipped
CI/CD / publish-deb (pull_request) Has been skipped
The Asahi installer's urlcache.py fails with AssertionError on macOS
when streaming ZIP via HTTP Range requests from Fastify. Fix: download
the ZIP with curl first (reliable on macOS), then set REPO_BASE to the
local directory so the installer opens it as a local file.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-01 00:30:29 +01:00
Michal
bb8f37ef7d feat: iSCSI, Longhorn disk labels, labctl asahi command, ZIP32 fix
Some checks failed
CI/CD / typecheck (pull_request) Failing after 12s
CI/CD / lint (pull_request) Failing after 22s
CI/CD / test (pull_request) Failing after 10s
CI/CD / build (pull_request) Has been skipped
CI/CD / publish-rpm (pull_request) Has been skipped
CI/CD / publish-deb (pull_request) Has been skipped
k3s host prep:
- Add iSCSI initiator install+enable (Fedora: iscsi-initiator-utils,
  Ubuntu: open-iscsi) — required by Longhorn
- Add Longhorn disk label to k3s server+agent configs
- Add Longhorn disk annotation operation in post-install hardening

CLI:
- Add `labctl provision asahi` command with interactive install guide
- Change default SSH user from "michal" to "lab" in all commands
- Change admin user in bastion progress callback to "lab"

Asahi provisioning fixes:
- Download installer_data.json locally (installer reads it as file)
- Use REPO_BASE to serve upstream ZIP from bastion (LAN speed)
- Fix ZIP32 vs ZIP64: serve original upstream ZIP unmodified
  (our repackaged ZIP used ZIP64 which breaks Asahi urlcache)
- Add /data/asahi-repo fallback path for k3s container PVC mount
- Deploy script syncs asahi-repo to bastion pod after deployment

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-31 23:32:38 +01:00
Michal
a8dc79bc5a feat: Asahi validation tests, rootfs build fixes, shellcheck-clean scripts
Some checks failed
CI/CD / lint (pull_request) Failing after 12s
CI/CD / test (pull_request) Failing after 10s
CI/CD / typecheck (pull_request) Failing after 22s
CI/CD / build (pull_request) Has been skipped
CI/CD / publish-rpm (pull_request) Has been skipped
CI/CD / publish-deb (pull_request) Has been skipped
- Add 16 validation tests: shellcheck (3 roles), installer_data.json
  schema (8), Python parser validation, ZIP structure (3), rootfs mount
- Fix empty SSH keys generating invalid bash (SC1073)
- Fix __dirname crash in ESM modules (use import.meta.url)
- Fix rootfs build: mkdir -p before writing, correct binary paths
- Add .gitignore for large build artifacts (.asahi-cache, *.zip)
- Bump smoke test timeout for additional static plugin registration

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-31 13:22:24 +01:00
Michal
ad76c74020 fix: rootfs build script — mkdir before write, fix package path checks
Some checks failed
CI/CD / typecheck (pull_request) Failing after 10s
CI/CD / lint (pull_request) Failing after 21s
CI/CD / test (pull_request) Failing after 11s
CI/CD / build (pull_request) Has been skipped
CI/CD / publish-rpm (pull_request) Has been skipped
CI/CD / publish-deb (pull_request) Has been skipped
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-31 03:26:26 +01:00
Michal
6807632d46 feat: Asahi rootfs build pipeline + serve from bastion
Some checks failed
CI/CD / lint (pull_request) Failing after 10s
CI/CD / test (pull_request) Failing after 10s
CI/CD / typecheck (pull_request) Failing after 22s
CI/CD / build (pull_request) Has been skipped
CI/CD / publish-rpm (pull_request) Has been skipped
CI/CD / publish-deb (pull_request) Has been skipped
- Add scripts/build-asahi-rootfs.sh: downloads upstream Fedora Asahi
  Remix Server, injects lab firstboot script + systemd service + SSH
  keys, repackages with installer_data.json that adds LVM Data partition
- Bastion serves built artifacts at /asahi/repo/* via fastify-static
- installer_data.json prefers built config, falls back to minimal
- Fix __dirname crash in ESM module (use import.meta.url)
- Fix smoke test timeout (was crashing due to __dirname)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-31 03:20:12 +01:00
Michal
53265bb18c test: integration test for Asahi firstboot LVM setup
Some checks failed
CI/CD / lint (pull_request) Failing after 21s
CI/CD / typecheck (pull_request) Failing after 22s
CI/CD / test (pull_request) Failing after 22s
CI/CD / build (pull_request) Has been skipped
CI/CD / publish-rpm (pull_request) Has been skipped
CI/CD / publish-deb (pull_request) Has been skipped
VM-based end-to-end test using Fedora cloud image with two disks:
root (20GB) + data (200GB). Verifies the firstboot script creates
labvg with correct LV sizes, mounts volumes, migrates /home content,
sets hostname, creates admin user, and handles reprovision.

Fixes to firstboot script:
- Detect whole disks (not just partitions) for LVM PV
- Handle btrfs subvolume paths in root device detection
- Copy /home content before mounting LV (preserves SSH keys)
- Don't restart sshd (config takes effect on reboot)
- Make swapon and mount operations resilient to failures

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-31 03:07:38 +01:00
Michal
863c7f2b83 feat: Asahi Linux provisioning for Apple Silicon (Mac Studio)
Some checks failed
CI/CD / typecheck (pull_request) Failing after 11s
CI/CD / lint (pull_request) Failing after 22s
CI/CD / test (pull_request) Failing after 11s
CI/CD / build (pull_request) Has been skipped
CI/CD / publish-rpm (pull_request) Has been skipped
CI/CD / publish-deb (pull_request) Has been skipped
Add bastion endpoints for provisioning Apple Silicon machines via the
Asahi Linux installer with custom LVM partitioning:

- GET /asahi — wrapper script (curl bastion:8080/asahi | sh)
- GET /asahi/installer_data.json — custom partition layout (60GB root + LVM data)
- GET /asahi/firstboot.sh — first-boot LVM setup matching kickstart layout
- GET /asahi/firstboot.service — systemd oneshot unit

The firstboot script creates labvg with role-specific LVs (var, varlog,
home, srv, rancher, longhorn) and handles reprovision by detecting
existing VGs. Includes 19 new tests.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-31 02:46:27 +01:00
906f93f6f2 Merge pull request 'fix: Cilium multi-node support' (#9) from fix/cilium-multi-node into main
Some checks failed
CI/CD / lint (push) Failing after 22s
CI/CD / typecheck (push) Failing after 21s
CI/CD / test (push) Failing after 22s
CI/CD / build (push) Has been skipped
CI/CD / publish-rpm (push) Has been skipped
CI/CD / publish-deb (push) Has been skipped
2026-03-31 00:36:17 +00:00
Michal
aea28b5a0f fix: Cilium multi-node support — auto-detect NIC, k3s agent API port, worker label
Some checks failed
CI/CD / typecheck (pull_request) Failing after 10s
CI/CD / lint (pull_request) Failing after 22s
CI/CD / test (pull_request) Failing after 7m8s
CI/CD / build (pull_request) Has been skipped
CI/CD / publish-rpm (pull_request) Has been skipped
CI/CD / publish-deb (pull_request) Has been skipped
- Remove hardcoded devices/directRoutingDevice from Cilium install (let
  Cilium auto-detect per node — needed for heterogeneous NICs like eno1 vs enP7s7)
- Set k8sServiceHost=127.0.0.1 k8sServicePort=6444 so Cilium init
  containers can reach the API via k3s agent's local LB proxy
- Add node-role.kubernetes.io/worker label to agent config

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-31 01:35:51 +01:00
f3f0ea48e7 Merge pull request 'feat: provision register + k3s kubeconfig' (#8) from feat/register-and-kubeconfig into main
Some checks failed
CI/CD / lint (push) Failing after 10s
CI/CD / test (push) Failing after 10s
CI/CD / typecheck (push) Failing after 21s
CI/CD / build (push) Has been skipped
CI/CD / publish-rpm (push) Has been skipped
CI/CD / publish-deb (push) Has been skipped
2026-03-31 00:16:06 +00:00
171 changed files with 19205 additions and 254 deletions

10
.gitignore vendored
View File

@@ -27,3 +27,13 @@ node_modules/
# Task files
# tasks.json
# tasks/
# Asahi build artifacts (large)
bastion/.asahi-cache/
bastion/asahi-repo/*.zip
# Regenerated by labsim/dualstack-lab.sh; derived state, not source.
labsim/dualstack-evidence/
# Runtime snapshots from labsim/cilium-ipam-switch.sh
labsim/.ipam-switch-state/

19
CLAUDE.md Normal file
View File

@@ -0,0 +1,19 @@
## Skill routing
When the user's request matches an available skill, ALWAYS invoke it using the Skill
tool as your FIRST action. Do NOT answer directly, do NOT use other tools first.
The skill has specialized workflows that produce better results than ad-hoc answers.
Key routing rules:
- Product ideas, "is this worth building", brainstorming → invoke gstack-office-hours
- Bugs, errors, "why is this broken", 500 errors → invoke gstack-investigate
- Ship, deploy, push, create PR → invoke gstack-ship
- QA, test the site, find bugs → invoke gstack-qa
- Code review, check my diff → invoke gstack-review
- Update docs after shipping → invoke gstack-document-release
- Weekly retro → invoke gstack-retro
- Design system, brand → invoke gstack-design-consultation
- Visual audit, design polish → invoke gstack-design-review
- Architecture review → invoke gstack-plan-eng-review
- Save progress, checkpoint, resume → invoke gstack-checkpoint
- Code quality, health check → invoke gstack-health

47
TODOS.md Normal file
View File

@@ -0,0 +1,47 @@
# TODOS
## P1 — Ship with Phase 1
### v2.0 Architecture Document Update
Update `bastion/docs/ARCHITECTURE.md` to cover v2.0: driver model, fleet system,
Pulumi integration, Vault secrets, Deno evaluator, new CLI grammar. The existing
doc covers v1.0 comprehensively (432 lines). v2.0 adds 5+ major subsystems.
**Effort:** M (human: 1 week / CC: 1-2 days)
**Depends on:** Phase 1 complete
**Source:** CEO review 2026-04-01
## P2 — Post-v2.0 Core
### SSH Emergency Mode (scoped)
SSH-based operations limited to: (1) earliest necessary box provisioning before agent
is installed, and (2) emergency debugging/fixing operations that can't be done via agent.
NOT a general-purpose DeploymentTarget alternative. The v1.0 `recheck` and `fix-ssh-root.sh`
patterns are the model. Agent stays the primary management path.
**Effort:** S (human: 1 week / CC: 1 day)
**Depends on:** Phase 2 complete (DeploymentTarget interface exists)
**Source:** CEO review 2026-04-01
### Prometheus Metrics Endpoint
Add `/metrics` endpoint to labd: resource counts by status, apply duration histograms,
driver operation latency, fleet pipeline completion rates. Standard Prometheus scraping
for Grafana dashboards and alerting.
**Effort:** S (human: 2-3 days / CC: 2-3 hours)
**Depends on:** Phase 1 (labd exists with resource store)
**Source:** CEO review 2026-04-01 (observability gap)
## P3 — Future Enhancements
### Infrastructure Graph Visualization
Visual representation of resource dependencies, environment topology, fleet status.
Could be a web UI or terminal-based (like `kubectl tree`).
**Source:** CEO review 2026-04-01
### `labctl import` for Existing Cloud Resources
Discover and import existing AWS/GCP resources into the state store.
Pulumi's import functionality could be leveraged.
**Source:** CEO review 2026-04-01
### Built-in Secrets Rotation
Automatic rotation of managed secrets (database passwords, API keys).
Vault handles rotation but a labctl-native workflow could simplify.
**Source:** CEO review 2026-04-01

View File

@@ -11,6 +11,7 @@ WORKDIR /app
# Copy workspace config and package manifests first (layer cache)
COPY pnpm-workspace.yaml pnpm-lock.yaml package.json tsconfig.base.json tsconfig.json ./
COPY src/shared/package.json src/shared/tsconfig.json src/shared/
COPY src/core/package.json src/core/tsconfig.json src/core/
COPY src/labd/package.json src/labd/tsconfig.json src/labd/
# Install all dependencies (dev included -- needed for build)
@@ -22,10 +23,13 @@ RUN pnpm --filter @lab/labd exec prisma generate
# Copy source code
COPY src/shared/src/ src/shared/src/
COPY src/core/src/ src/core/src/
COPY src/labd/src/ src/labd/src/
# Build TypeScript (shared first via project references)
RUN pnpm --filter @lab/shared build && pnpm --filter @lab/labd build
# Build TypeScript (shared + core before labd via project references)
RUN pnpm --filter @lab/shared build \
&& pnpm --filter @lab/core build \
&& pnpm --filter @lab/labd build
# Hoist the generated Prisma client so stage 2 can COPY it from a stable path
RUN mkdir -p /app/_prisma && \
@@ -41,6 +45,7 @@ WORKDIR /app
# Copy workspace config and package manifests
COPY pnpm-workspace.yaml pnpm-lock.yaml package.json ./
COPY src/shared/package.json src/shared/
COPY src/core/package.json src/core/
COPY src/labd/package.json src/labd/
# Install production dependencies only
@@ -48,6 +53,7 @@ RUN pnpm install --frozen-lockfile --prod 2>/dev/null || pnpm install --prod
# Copy built output from builder
COPY --from=builder /app/src/shared/dist/ src/shared/dist/
COPY --from=builder /app/src/core/dist/ src/core/dist/
COPY --from=builder /app/src/labd/dist/ src/labd/dist/
# Copy Prisma schema + generated client into pnpm store location

View File

@@ -0,0 +1,47 @@
{
"os_list": [
{
"name": "Fedora Asahi Lab (infra)",
"default_os_name": "Fedora Linux Lab",
"boot_object": "m1n1.bin",
"next_object": "m1n1/boot.bin",
"package": "fedora-asahi-lab.zip",
"supported_fw": [
"12.3",
"12.3.1",
"13.5"
],
"partitions": [
{
"name": "EFI",
"type": "EFI",
"size": "524288000B",
"format": "fat",
"volume_id": "0x804be8a6",
"copy_firmware": true,
"copy_installer_data": true,
"source": "esp"
},
{
"name": "Boot",
"type": "Linux",
"size": "1073741824B",
"image": "boot.img"
},
{
"name": "Root",
"type": "Linux",
"size": "4626296832B",
"expand": false,
"image": "root.img"
},
{
"name": "Data",
"type": "Linux",
"size": "1073741824B",
"expand": true
}
]
}
]
}

4
bastion/bastion/.gitignore vendored Normal file
View File

@@ -0,0 +1,4 @@
# Asahi build artifacts (large)
.asahi-cache/
asahi-repo/*.zip

View File

@@ -59,10 +59,10 @@ _labctl() {
COMPREPLY=($(compgen -W "-h --help" -- "$cur"))
return ;;
"provision install")
COMPREPLY=($(compgen -W "--role --os --disk -h --help" -- "$cur"))
COMPREPLY=($(compgen -W "--role --os --disk --vyos-mgmt --vyos-mgmt-address --vyos-bond --vyos-bond-address --vyos-bond-vrrp --vlan-vip --vyos-vrrp-priority --vyos-mgmt-vlan --vlan --vyos-password --vyos-hwid --vyos-fresh-config -h --help" -- "$cur"))
return ;;
"provision reprovision")
COMPREPLY=($(compgen -W "--role --os --disk -h --help" -- "$cur"))
COMPREPLY=($(compgen -W "--role --os --disk --user -h --help" -- "$cur"))
return ;;
"provision debug")
COMPREPLY=($(compgen -W "--pxe-boot -h --help" -- "$cur"))
@@ -73,12 +73,18 @@ _labctl() {
"provision register")
COMPREPLY=($(compgen -W "--role --ip -h --help" -- "$cur"))
return ;;
"provision asahi")
COMPREPLY=($(compgen -W "-h --help" -- "$cur"))
return ;;
"provision logs")
COMPREPLY=($(compgen -W "-f --follow -h --help" -- "$cur"))
return ;;
"provision makeiso")
COMPREPLY=($(compgen -W "--arch --local --out -h --help" -- "$cur"))
return ;;
"provision recheck")
COMPREPLY=($(compgen -W "--user --target -h --help" -- "$cur"))
return ;;
"config list")
COMPREPLY=($(compgen -W "-h --help" -- "$cur"))
return ;;
@@ -104,7 +110,7 @@ _labctl() {
COMPREPLY=($(compgen -W "bastion -h --help" -- "$cur"))
return ;;
"provision")
COMPREPLY=($(compgen -W "list install reprovision debug forget register logs makeiso -h --help" -- "$cur"))
COMPREPLY=($(compgen -W "list install reprovision debug forget register asahi logs makeiso recheck -h --help" -- "$cur"))
return ;;
"config")
COMPREPLY=($(compgen -W "list get set path -h --help" -- "$cur"))

View File

@@ -125,18 +125,33 @@ complete -c labctl -n "__labctl_using_cmd provision" -a reprovision -d 'Queue in
complete -c labctl -n "__labctl_using_cmd provision" -a debug -d 'PXE boot into Fedora rescue mode for debugging (target: hostname, MAC, or IP)'
complete -c labctl -n "__labctl_using_cmd provision" -a forget -d 'Remove a machine from bastion state'
complete -c labctl -n "__labctl_using_cmd provision" -a register -d 'Register an already-installed machine (e.g. after state loss)'
complete -c labctl -n "__labctl_using_cmd provision" -a asahi -d 'Show instructions to provision an Apple Silicon Mac with Asahi Linux'
complete -c labctl -n "__labctl_using_cmd provision" -a logs -d 'Show provisioning logs for a machine (hostname, MAC, or IP)'
complete -c labctl -n "__labctl_using_cmd provision" -a makeiso -d 'Generate a UEFI-bootable iPXE ISO for network provisioning'
complete -c labctl -n "__labctl_using_cmd provision" -a recheck -d 'Refresh hardware info for all installed machines via SSH'
# provision install options
complete -c labctl -n "__labctl_in_cmd provision install" -l role -d 'Machine role (see below)' -xa 'vanilla worker infra labcontroller'
complete -c labctl -n "__labctl_in_cmd provision install" -l os -d 'Operating system' -xa 'fedora-43 ubuntu-26.04'
complete -c labctl -n "__labctl_in_cmd provision install" -l os -d 'Operating system' -xa 'fedora-43 ubuntu-26.04 vyos-rolling'
complete -c labctl -n "__labctl_in_cmd provision install" -l disk -d 'Target disk device (auto-detect if omitted)' -x
complete -c labctl -n "__labctl_in_cmd provision install" -l vyos-mgmt -d 'VyOS: untagged interface the machine PXE boots from (default eth0)' -x
complete -c labctl -n "__labctl_in_cmd provision install" -l vyos-mgmt-address -d 'VyOS: CIDR for the management interface, or \'dhcp\' (default dhcp)' -x
complete -c labctl -n "__labctl_in_cmd provision install" -l vyos-bond -d 'VyOS: comma-separated LACP bond members (must exclude the PXE NIC)' -x
complete -c labctl -n "__labctl_in_cmd provision install" -l vyos-bond-address -d 'VyOS: address on the untagged bond (trunk native VLAN)' -x
complete -c labctl -n "__labctl_in_cmd provision install" -l vyos-bond-vrrp -d 'VyOS: VRRP VIP floated on the untagged bond' -x
complete -c labctl -n "__labctl_in_cmd provision install" -l vlan-vip -d 'VyOS: VRRP VIP for a --vlan entry (repeatable)' -x
complete -c labctl -n "__labctl_in_cmd provision install" -l vyos-vrrp-priority -d 'VyOS: VRRP priority for all groups on this box (higher = master)' -x
complete -c labctl -n "__labctl_in_cmd provision install" -l vyos-mgmt-vlan -d 'VyOS: tagged management VLAN on the PXE port' -x
complete -c labctl -n "__labctl_in_cmd provision install" -l vlan -d 'VyOS: tagged VLAN sub-interface on the bond (repeatable)' -x
complete -c labctl -n "__labctl_in_cmd provision install" -l vyos-password -d 'VyOS: password for the \'vyos\' user' -x
complete -c labctl -n "__labctl_in_cmd provision install" -l vyos-hwid -d 'VyOS: pin an interface name to a MAC via hw-id (repeatable)' -x
complete -c labctl -n "__labctl_in_cmd provision install" -l vyos-fresh-config -d 'VyOS: on reinstall, overwrite the preserved config with the generated one'
# provision reprovision options
complete -c labctl -n "__labctl_in_cmd provision reprovision" -l role -d 'Machine role (see below)' -xa 'vanilla worker infra labcontroller'
complete -c labctl -n "__labctl_in_cmd provision reprovision" -l os -d 'Operating system' -xa 'fedora-43 ubuntu-26.04'
complete -c labctl -n "__labctl_in_cmd provision reprovision" -l os -d 'Operating system' -xa 'fedora-43 ubuntu-26.04 vyos-rolling'
complete -c labctl -n "__labctl_in_cmd provision reprovision" -l disk -d 'Target disk device (auto-detect if omitted)' -x
complete -c labctl -n "__labctl_in_cmd provision reprovision" -l user -d 'SSH user for the reboot (default: vyos for VyOS machines, else current user)' -x
# provision debug options
complete -c labctl -n "__labctl_in_cmd provision debug" -l pxe-boot -d 'Boot installed system via PXE (kernel+initrd from network, root from NVMe)'
@@ -153,6 +168,10 @@ complete -c labctl -n "__labctl_in_cmd provision makeiso" -l arch -d 'Target arc
complete -c labctl -n "__labctl_in_cmd provision makeiso" -l local -d 'Build ISO locally instead of using bastion-hosted URL'
complete -c labctl -n "__labctl_in_cmd provision makeiso" -l out -d 'Output path for local ISO build' -x
# provision recheck options
complete -c labctl -n "__labctl_in_cmd provision recheck" -l user -d 'SSH user' -x
complete -c labctl -n "__labctl_in_cmd provision recheck" -l target -d 'Only recheck a specific machine (by hostname or MAC)' -x
# config subcommands
complete -c labctl -n "__labctl_using_cmd config" -a list -d 'Show all configuration values'
complete -c labctl -n "__labctl_using_cmd config" -a get -d 'Get a configuration value'

View File

@@ -21,8 +21,14 @@
"test:integration:pxe:host": "sudo -E $(which npx) vitest run -c tests/integration/vitest.config.ts -t 'PXE boot'",
"test:integration:iso": "vitest run -c tests/integration/vitest.config.ts -t 'ISO boot'",
"test:integration:iso:host": "sudo -E $(which npx) vitest run -c tests/integration/vitest.config.ts -t 'ISO boot'",
"test:integration:vyos": "vitest run -c tests/integration/vitest.config.ts -t 'VyOS provisioning'",
"test:integration:vyos:host": "sudo -E $(which npx) vitest run -c tests/integration/vitest.config.ts -t 'VyOS provisioning'",
"test:integration:arm-iso": "vitest run -c tests/integration/vitest.config.ts -t 'ARM ISO'",
"test:integration:arm-iso:host": "sudo -E $(which npx) vitest run -c tests/integration/vitest.config.ts -t 'ARM ISO'"
"test:integration:arm-iso:host": "sudo -E $(which npx) vitest run -c tests/integration/vitest.config.ts -t 'ARM ISO'",
"test:integration:asahi": "vitest run -c tests/integration/vitest.config.ts -t 'asahi firstboot'",
"test:integration:asahi:host": "sudo -E $(which npx) vitest run -c tests/integration/vitest.config.ts -t 'asahi firstboot'",
"test:integration:asahi-validate": "vitest run -c tests/integration/vitest.config.ts -t 'asahi.*validation'",
"test:integration:asahi-validate:host": "sudo -E $(which npx) vitest run -c tests/integration/vitest.config.ts -t 'asahi.*validation'"
},
"engines": {
"node": ">=20.0.0",

1847
bastion/pnpm-lock.yaml generated

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,302 @@
#!/bin/bash
# Build a custom Fedora Asahi Remix rootfs with lab firstboot LVM setup.
#
# Downloads the upstream Fedora Asahi Remix Server package, injects our
# firstboot script + systemd service, and repackages it for the bastion.
#
# Requirements: root, curl, unzip, mount (loop), zip
# Output: bastion/asahi-repo/ directory with package + installer_data.json
#
# Usage: sudo ./scripts/build-asahi-rootfs.sh [--bastion-ip IP] [--http-port PORT]
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
PROJECT_DIR="$(cd "$SCRIPT_DIR/.." && pwd)"
ASAHI_DIR="$PROJECT_DIR/asahi-repo"
CACHE_DIR="$PROJECT_DIR/.asahi-cache"
WORK_DIR=""
# Defaults
BASTION_IP="${BASTION_IP:-192.168.8.23}"
HTTP_PORT="${HTTP_PORT:-8080}"
ROLE="${ROLE:-infra}"
HOSTNAME="${HOSTNAME:-mac-studio}"
MAC="${MAC:-00:00:00:00:00:00}"
ADMIN_USER="${ADMIN_USER:-michal}"
# Parse args
while [[ $# -gt 0 ]]; do
case "$1" in
--bastion-ip) BASTION_IP="$2"; shift 2 ;;
--http-port) HTTP_PORT="$2"; shift 2 ;;
--role) ROLE="$2"; shift 2 ;;
--hostname) HOSTNAME="$2"; shift 2 ;;
--mac) MAC="$2"; shift 2 ;;
--admin-user) ADMIN_USER="$2"; shift 2 ;;
*) echo "Unknown option: $1"; exit 1 ;;
esac
done
# ── Resolve upstream package URL ─────────────────────────────────
echo "==> Fetching Asahi installer data..."
INSTALLER_DATA=$(curl -sfL "https://cdn.asahilinux.org/installer/installer_data.json")
# Find the Server variant package URL
SERVER_URL=$(echo "$INSTALLER_DATA" | python3 -c "
import sys, json
data = json.load(sys.stdin)
for os in data.get('os_list', []):
name = os.get('name', '').lower()
if 'server' in name and 'uefi' not in name and not os.get('expert'):
print(os['package'])
break
" 2>/dev/null)
if [ -z "$SERVER_URL" ]; then
echo "ERROR: Could not find Fedora Asahi Remix Server in installer data."
echo "Available variants:"
echo "$INSTALLER_DATA" | python3 -c "
import sys, json
data = json.load(sys.stdin)
for os in data.get('os_list', []):
print(f\" - {os.get('name', '?')}\")" 2>/dev/null
exit 1
fi
PACKAGE_NAME=$(basename "$SERVER_URL")
echo " Variant: Fedora Asahi Remix Server"
echo " Package: $PACKAGE_NAME"
# Also extract the partition layout and supported_fw from upstream
UPSTREAM_CONFIG=$(echo "$INSTALLER_DATA" | python3 -c "
import sys, json
data = json.load(sys.stdin)
for os in data.get('os_list', []):
name = os.get('name', '').lower()
if 'server' in name and 'uefi' not in name and not os.get('expert'):
json.dump(os, sys.stdout)
break
")
# ── Download upstream package ────────────────────────────────────
mkdir -p "$CACHE_DIR" "$ASAHI_DIR"
CACHED_PKG="$CACHE_DIR/$PACKAGE_NAME"
if [ -f "$CACHED_PKG" ]; then
echo "==> Using cached package: $CACHED_PKG"
else
echo "==> Downloading $SERVER_URL..."
curl -# -L -o "$CACHED_PKG" "$SERVER_URL"
fi
# ── Extract and modify rootfs ────────────────────────────────────
WORK_DIR=$(mktemp -d)
trap 'echo "==> Cleaning up..."; umount "$WORK_DIR/rootfs" 2>/dev/null || true; rm -rf "$WORK_DIR"' EXIT
echo "==> Extracting package..."
unzip -q -o "$CACHED_PKG" -d "$WORK_DIR/pkg"
# List contents
echo " Package contents:"
ls -lh "$WORK_DIR/pkg/" | grep -v ^total | while read -r line; do echo " $line"; done
# Find root.img
ROOT_IMG=$(find "$WORK_DIR/pkg" -name "root.img" -type f | head -1)
if [ -z "$ROOT_IMG" ]; then
echo "ERROR: root.img not found in package."
echo "Contents: $(ls "$WORK_DIR/pkg/")"
exit 1
fi
echo "==> Mounting root.img..."
mkdir -p "$WORK_DIR/rootfs"
mount -o loop "$ROOT_IMG" "$WORK_DIR/rootfs"
# ── Read SSH keys from the system ────────────────────────────────
SSH_KEYS=""
REAL_USER="${SUDO_USER:-$USER}"
REAL_HOME=$(eval echo "~$REAL_USER")
for keyfile in "$REAL_HOME/.ssh/id_ed25519.pub" "$REAL_HOME/.ssh/id_ecdsa.pub" "$REAL_HOME/.ssh/id_rsa.pub"; do
if [ -f "$keyfile" ]; then
SSH_KEYS=$(cat "$keyfile")
echo " SSH key: $keyfile"
break
fi
done
if [ -z "$SSH_KEYS" ]; then
echo "WARNING: No SSH public key found. You'll need to add keys manually."
fi
# ── Generate firstboot script from bastion ───────────────────────
echo "==> Generating firstboot script..."
# Try to get the script from a running bastion, fall back to local generation
FIRSTBOOT_SCRIPT=""
FIRSTBOOT_URL="http://$BASTION_IP:$HTTP_PORT/asahi/firstboot.sh?hostname=$HOSTNAME&role=$ROLE&mac=$MAC&user=$ADMIN_USER"
FIRSTBOOT_SCRIPT=$(curl -sf "$FIRSTBOOT_URL" 2>/dev/null || echo "")
if [ -z "$FIRSTBOOT_SCRIPT" ]; then
echo " Bastion not reachable, generating script locally..."
# Generate a basic firstboot script inline
FIRSTBOOT_SCRIPT=$(cd "$PROJECT_DIR" && node -e "
const { renderFirstbootScript } = require('./src/bastion/dist/templates/asahi-firstboot.sh.js');
process.stdout.write(renderFirstbootScript({
hostname: '$HOSTNAME',
role: '$ROLE',
serverIp: '$BASTION_IP',
httpPort: $HTTP_PORT,
sshKeys: $([ -n "$SSH_KEYS" ] && echo "[\"$SSH_KEYS\"]" || echo "[]"),
adminUser: '$ADMIN_USER',
mac: '$MAC',
}));
" 2>/dev/null) || {
echo " ERROR: Could not generate firstboot script. Build the project first: npm run build"
exit 1
}
fi
# ── Inject files into rootfs ─────────────────────────────────────
echo "==> Injecting lab configuration into rootfs..."
# Firstboot script
mkdir -p "$WORK_DIR/rootfs/usr/local/bin"
echo "$FIRSTBOOT_SCRIPT" > "$WORK_DIR/rootfs/usr/local/bin/lab-firstboot.sh"
chmod 755 "$WORK_DIR/rootfs/usr/local/bin/lab-firstboot.sh"
echo " Installed: /usr/local/bin/lab-firstboot.sh"
# Systemd service
mkdir -p "$WORK_DIR/rootfs/etc/systemd/system"
cat > "$WORK_DIR/rootfs/etc/systemd/system/lab-firstboot.service" << 'UNIT'
[Unit]
Description=Lab first-boot LVM setup
After=local-fs.target network-online.target
Wants=network-online.target
ConditionPathExists=!/etc/lab-lvm-setup-done
[Service]
Type=oneshot
ExecStart=/usr/local/bin/lab-firstboot.sh
RemainAfterExit=yes
StandardOutput=journal+console
StandardError=journal+console
[Install]
WantedBy=multi-user.target
UNIT
echo " Installed: /etc/systemd/system/lab-firstboot.service"
# Enable the service
mkdir -p "$WORK_DIR/rootfs/etc/systemd/system/multi-user.target.wants"
ln -sf /etc/systemd/system/lab-firstboot.service \
"$WORK_DIR/rootfs/etc/systemd/system/multi-user.target.wants/lab-firstboot.service"
echo " Enabled: lab-firstboot.service"
# SSH authorized keys for root (for initial access before firstboot runs user creation)
if [ -n "$SSH_KEYS" ]; then
mkdir -p "$WORK_DIR/rootfs/root/.ssh"
chmod 700 "$WORK_DIR/rootfs/root/.ssh"
echo "$SSH_KEYS" > "$WORK_DIR/rootfs/root/.ssh/authorized_keys"
chmod 600 "$WORK_DIR/rootfs/root/.ssh/authorized_keys"
echo " Installed: /root/.ssh/authorized_keys"
fi
# Ensure lvm2 and xfsprogs are installed (should be in server image already)
echo " Checking required packages..."
if [ -f "$WORK_DIR/rootfs/usr/sbin/pvcreate" ] || [ -f "$WORK_DIR/rootfs/usr/bin/pvcreate" ]; then
echo " lvm2: present"
else
echo " WARNING: lvm2 not found in rootfs. LVM setup may fail."
fi
if [ -f "$WORK_DIR/rootfs/usr/sbin/mkfs.xfs" ] || [ -f "$WORK_DIR/rootfs/usr/bin/mkfs.xfs" ]; then
echo " xfsprogs: present"
else
echo " WARNING: xfsprogs not found in rootfs. LVM setup may fail."
fi
# ── Unmount and repackage ────────────────────────────────────────
echo "==> Unmounting rootfs..."
umount "$WORK_DIR/rootfs"
echo "==> Repackaging..."
OUTPUT_PKG="$ASAHI_DIR/fedora-asahi-lab.zip"
rm -f "$OUTPUT_PKG"
(cd "$WORK_DIR/pkg" && zip -q "$OUTPUT_PKG" *)
echo " Output: $OUTPUT_PKG ($(du -sh "$OUTPUT_PKG" | cut -f1))"
# ── Generate installer_data.json ─────────────────────────────────
echo "==> Generating installer_data.json..."
# Parse upstream config to get supported_fw, boot_object, next_object, and partition details
python3 << PYEOF > "$ASAHI_DIR/installer_data.json"
import json, sys
upstream = json.loads('''$UPSTREAM_CONFIG''')
# Build our custom installer data based on upstream
# Keep EFI and Boot partitions identical, modify Root to not expand,
# add Data partition that expands for LVM.
partitions = []
for p in upstream.get('partitions', []):
if p.get('type') == 'EFI':
partitions.append(p)
elif p.get('name') == 'Boot':
partitions.append(p)
elif p.get('name') == 'Root':
# Fixed size root, no expand
root_p = dict(p)
root_p['expand'] = False
# Keep the original size (it's the minimum needed for the rootfs)
partitions.append(root_p)
# Add Data partition for LVM
partitions.append({
"name": "Data",
"type": "Linux",
"size": "1073741824B", # 1GB minimum, will expand
"expand": True
})
data = {
"os_list": [{
"name": "Fedora Asahi Lab (${ROLE})",
"default_os_name": "Fedora Linux Lab",
"boot_object": upstream.get("boot_object", "m1n1.bin"),
"next_object": upstream.get("next_object", "m1n1/boot.bin"),
"package": "fedora-asahi-lab.zip",
"supported_fw": upstream.get("supported_fw", ["13.5"]),
"partitions": partitions,
}]
}
json.dump(data, sys.stdout, indent=2)
print()
PYEOF
echo " Generated: $ASAHI_DIR/installer_data.json"
# Pretty-print the partition layout
echo ""
echo " Partition layout:"
python3 -c "
import json
with open('$ASAHI_DIR/installer_data.json') as f:
data = json.load(f)
for p in data['os_list'][0]['partitions']:
size = p.get('size', '?')
expand = ' (expand)' if p.get('expand') else ''
image = f\" [{p['image']}]\" if 'image' in p else ''
print(f\" {p['name']:8s} {p['type']:8s} {size:>16s}{expand}{image}\")
"
echo ""
echo "==> Build complete!"
echo ""
echo " Package: $ASAHI_DIR/fedora-asahi-lab.zip"
echo " Config: $ASAHI_DIR/installer_data.json"
echo ""
echo " To serve from bastion, copy to the bastion's HTTP directory"
echo " or configure REPO_BASE to point here."
echo ""
echo " To install on Mac Studio:"
echo " curl http://$BASTION_IP:$HTTP_PORT/asahi | sh"

View File

@@ -99,16 +99,22 @@ if [ "$PUSH" = true ]; then
fi
fi
# Use --tls-verify=false for plain HTTP registries (e.g. 10.0.0.194:3012)
TLS_FLAG=""
if [[ "$REGISTRY" =~ ^[0-9] ]] || [[ "$REGISTRY" =~ ^localhost ]]; then
TLS_FLAG="--tls-verify=false"
fi
echo "==> Logging in to $REGISTRY..."
podman login -u michal -p "$GITEA_TOKEN" "$REGISTRY"
podman login $TLS_FLAG -u michal -p "$GITEA_TOKEN" "$REGISTRY"
echo "==> Pushing $FULL_IMAGE:$TAG..."
podman manifest push --all "$MANIFEST" "docker://$FULL_IMAGE:$TAG"
podman manifest push --all $TLS_FLAG "$MANIFEST" "docker://$FULL_IMAGE:$TAG"
# Also tag as :latest if not already
if [ "$TAG" != "latest" ]; then
echo "==> Also pushing as :latest..."
podman manifest push --all "$MANIFEST" "docker://$FULL_IMAGE:latest"
podman manifest push --all $TLS_FLAG "$MANIFEST" "docker://$FULL_IMAGE:latest"
fi
# Link package to repository if script exists

View File

@@ -92,15 +92,21 @@ if [ "$PUSH" = true ]; then
fi
fi
# Use --tls-verify=false for plain HTTP registries (e.g. 10.0.0.194:3012)
TLS_FLAG=""
if [[ "$REGISTRY" =~ ^[0-9] ]] || [[ "$REGISTRY" =~ ^localhost ]]; then
TLS_FLAG="--tls-verify=false"
fi
echo "==> Logging in to $REGISTRY..."
podman login -u michal -p "$GITEA_TOKEN" "$REGISTRY"
podman login $TLS_FLAG -u michal -p "$GITEA_TOKEN" "$REGISTRY"
echo "==> Pushing $FULL_IMAGE:$TAG..."
podman manifest push --all "$MANIFEST" "docker://$FULL_IMAGE:$TAG"
podman manifest push --all $TLS_FLAG "$MANIFEST" "docker://$FULL_IMAGE:$TAG"
if [ "$TAG" != "latest" ]; then
echo "==> Also pushing as :latest..."
podman manifest push --all "$MANIFEST" "docker://$FULL_IMAGE:latest"
podman manifest push --all $TLS_FLAG "$MANIFEST" "docker://$FULL_IMAGE:latest"
fi
if [ -f "$SCRIPT_DIR/link-package.sh" ]; then

View File

@@ -24,6 +24,21 @@ deploy_bastion() {
kubectl rollout restart deployment/bastion -n lab-infra
kubectl rollout status deployment/bastion -n lab-infra --timeout=180s
echo "✓ Bastion deployed"
# Sync Asahi rootfs package to bastion pod's persistent volume
if [ -d "$PROJECT_DIR/asahi-repo" ] && [ -f "$PROJECT_DIR/asahi-repo/fedora-asahi-lab.zip" ]; then
echo ""
echo "=== Syncing Asahi rootfs to bastion pod ==="
BASTION_POD=$(kubectl get pods -n lab-infra -l app=bastion -o jsonpath='{.items[0].metadata.name}' 2>/dev/null)
if [ -n "$BASTION_POD" ]; then
kubectl exec -n lab-infra "$BASTION_POD" -- mkdir -p /data/asahi-repo
kubectl cp "$PROJECT_DIR/asahi-repo/installer_data.json" "lab-infra/$BASTION_POD:/data/asahi-repo/installer_data.json"
kubectl cp "$PROJECT_DIR/asahi-repo/fedora-asahi-lab.zip" "lab-infra/$BASTION_POD:/data/asahi-repo/fedora-asahi-lab.zip"
echo "✓ Asahi rootfs synced ($(du -sh "$PROJECT_DIR/asahi-repo/fedora-asahi-lab.zip" | cut -f1))"
else
echo "WARNING: Could not find bastion pod — Asahi rootfs not synced"
fi
fi
}
deploy_labd() {

View File

@@ -0,0 +1,131 @@
#!/bin/bash
# Fix root SSH access on all provisioned machines.
# Tries root, lab, michal users to find one that works,
# then ensures root has the SSH key and PermitRootLogin is enabled.
set -euo pipefail
SSH_KEY="ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMJ3FkUGbG174eoO5RjZd2eNV680FM5pgp0AgpW/QwlJExK3qxMk0DJSr4ICmzGUx4yujAXcrqU1otcOMPzzFzwc5heWpSmlNHU3TIW6NHEt0sF9ZTAbGLw2zSw3si5UouqFkCcENA40mePFJqY+Q9R8N1uvLgu4m/do+Zrn/mk5Ewc1V7OCRE5Acrnaec4T7LTB0BuVXcjPUfAmZ0q5fI+bKPR1q2Kc3+IeGhVkBuZ9OJVeXXhnpedm0uEbLeriK/jUYKYw/1QhsNDM8Tyty+UIGr9QVnWwzCMHB+wuQcDYC9mPGTqg0fYwX8Mp8xMi1PPxdsh1G7bj/cpWMAF43KswWORF2ul8ICGbaE1zEgIYXO790SuBjpBHhaC6Iegqi58hmCuP+a9893q/EU9HyrWTJHCZXC5E4kP1MsM57KrhEpszM6I3sW9f9zMTPd5QsCXFi4si4OMwX4kYNVu3fQGQPpseDPlTTSrT6uUdqj4Irm0c1m9cYTmK0vYgsM3ss= michal@fedora"
SSH_OPTS="-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=ERROR -o ConnectTimeout=5"
USERS_TO_TRY=(root lab michal)
# Machines: hostname ip
MACHINES=(
"labmaster 192.168.8.11"
"worker0-k8s0 192.168.8.23"
"worker1-k8s0 192.168.8.13"
"worker2-k8s0 192.168.8.25"
"spark-2935 192.168.8.12"
)
BOLD="\033[1m"
GREEN="\033[0;32m"
RED="\033[0;31m"
DIM="\033[2m"
RESET="\033[0m"
# Script to run on each machine (via sudo if needed)
read -r -d '' FIX_SCRIPT << 'FIXEOF' || true
#!/bin/bash
set -e
KEY="$1"
# 1. Ensure root .ssh dir exists
mkdir -p /root/.ssh
chmod 700 /root/.ssh
touch /root/.ssh/authorized_keys
chmod 600 /root/.ssh/authorized_keys
# 2. Add key if not present
if ! grep -qF "$KEY" /root/.ssh/authorized_keys 2>/dev/null; then
echo "$KEY" >> /root/.ssh/authorized_keys
echo "KEY_ADDED"
else
echo "KEY_EXISTS"
fi
# 3. Fix sshd_config for root login with keys
SSHD_CONF="/etc/ssh/sshd_config"
CHANGED=0
# Ensure PermitRootLogin allows key auth
CURRENT=$(grep -E "^PermitRootLogin" "$SSHD_CONF" 2>/dev/null | tail -1 || true)
if [ "$CURRENT" = "PermitRootLogin prohibit-password" ] || [ "$CURRENT" = "PermitRootLogin without-password" ]; then
echo "SSHD_OK"
elif [ "$CURRENT" = "PermitRootLogin yes" ]; then
echo "SSHD_OK"
else
# Remove any existing PermitRootLogin lines
sed -i '/^#*PermitRootLogin/d' "$SSHD_CONF"
echo "PermitRootLogin prohibit-password" >> "$SSHD_CONF"
CHANGED=1
echo "SSHD_FIXED"
fi
# Ensure PubkeyAuthentication is enabled
if grep -qE "^PubkeyAuthentication no" "$SSHD_CONF" 2>/dev/null; then
sed -i 's/^PubkeyAuthentication no/PubkeyAuthentication yes/' "$SSHD_CONF"
CHANGED=1
echo "PUBKEY_FIXED"
else
echo "PUBKEY_OK"
fi
# Restart sshd if changed
if [ "$CHANGED" -eq 1 ]; then
systemctl restart sshd 2>/dev/null || systemctl restart ssh 2>/dev/null || true
echo "SSHD_RESTARTED"
fi
# 4. Verify root can be reached
echo "DONE"
FIXEOF
echo ""
echo -e "${BOLD}Fixing root SSH access on all machines...${RESET}"
echo ""
for entry in "${MACHINES[@]}"; do
read -r hostname ip <<< "$entry"
printf " %-24s ${DIM}(%s)${RESET} " "$hostname" "$ip"
# Try each user until one works
WORKING_USER=""
for user in "${USERS_TO_TRY[@]}"; do
if ssh $SSH_OPTS "$user@$ip" "true" 2>/dev/null; then
WORKING_USER="$user"
break
fi
done
if [ -z "$WORKING_USER" ]; then
echo -e "${RED}UNREACHABLE${RESET} (tried: ${USERS_TO_TRY[*]})"
continue
fi
# Run fix script (with sudo if not root)
if [ "$WORKING_USER" = "root" ]; then
RESULT=$(ssh $SSH_OPTS "root@$ip" "bash -s -- '$SSH_KEY'" <<< "$FIX_SCRIPT" 2>&1)
else
RESULT=$(ssh $SSH_OPTS "$WORKING_USER@$ip" "sudo bash -s -- '$SSH_KEY'" <<< "$FIX_SCRIPT" 2>&1)
fi
# Parse result
DETAILS=""
if echo "$RESULT" | grep -q "KEY_ADDED"; then DETAILS="key added"; fi
if echo "$RESULT" | grep -q "KEY_EXISTS"; then DETAILS="key ok"; fi
if echo "$RESULT" | grep -q "SSHD_FIXED"; then DETAILS="$DETAILS, sshd fixed"; fi
if echo "$RESULT" | grep -q "SSHD_OK"; then DETAILS="$DETAILS, sshd ok"; fi
if echo "$RESULT" | grep -q "SSHD_RESTARTED"; then DETAILS="$DETAILS, restarted"; fi
# Verify root works now
if ssh $SSH_OPTS "root@$ip" "true" 2>/dev/null; then
echo -e "${GREEN}OK${RESET} ${DIM}(via $WORKING_USER: $DETAILS)${RESET}"
else
echo -e "${RED}PARTIAL${RESET} ${DIM}(via $WORKING_USER: $DETAILS -- root still blocked)${RESET}"
fi
done
echo ""
echo -e "${BOLD}Done.${RESET} Verify: labctl provision recheck --user root"
echo ""

View File

@@ -20,6 +20,15 @@ export function loadConfig(overrides: Partial<BastionConfig> = {}): BastionConfi
const ubuntuMirror = overrides.ubuntuMirror ?? process.env["UBUNTU_MIRROR"]
?? `https://releases.ubuntu.com/${ubuntuVersion}`;
// "latest" resolves the newest nightly ISO from the vyos-nightly-build GitHub
// releases at startup. downloads.vyos.io no longer serves direct rolling ISOs
// (it returns the vyos.io site, and nightly builds sit behind a signup form);
// GitHub releases are the remaining free, unauthenticated direct source.
// LTS ISOs are subscription-only. Set VYOS_ISO_URL to pin a specific build.
const vyosIsoUrl = overrides.vyosIsoUrl ?? process.env["VYOS_ISO_URL"] ?? "latest";
const vyosDefaultPassword = overrides.vyosDefaultPassword
?? process.env["VYOS_DEFAULT_PASSWORD"] ?? "vyos";
const fedoraMirror = `https://download.fedoraproject.org/pub/fedora/linux/releases/${fedoraVersion}/Everything/${arch}/os`;
const tftpDir = `${bastionDir}/tftp`;
const httpDir = `${bastionDir}/http`;
@@ -38,6 +47,8 @@ export function loadConfig(overrides: Partial<BastionConfig> = {}): BastionConfi
dhcpRangeEnd,
ubuntuVersion,
ubuntuMirror,
vyosIsoUrl,
vyosDefaultPassword,
// These are populated at runtime by the network service
iface: overrides.iface ?? "",
serverIp: overrides.serverIp ?? "",

View File

@@ -40,6 +40,125 @@ function download(url: string, dest: string, label: string): void {
}
}
/**
* Pick the largest regular-file initrd from an `xorriso -lsl` listing.
*
* /live carries decoys: a 0-byte initrd.img placeholder on some images, or an
* initrd.img SYMLINK to the real version-suffixed file on others. Parsing is
* field-based (ls -l layout: perms links uid gid size month day time 'name')
* and considers only lines whose mode string marks a regular file — symlinks
* report their link size, not the target's, and must not win.
*/
export function pickLargestInitrd(
listing: string,
): { name: string; size: number } | undefined {
let best: { name: string; size: number } | undefined;
for (const line of listing.split("\n")) {
if (!line.startsWith("-")) continue; // regular files only
const quoted = /'([^']+)'/.exec(line);
const fields = line.trim().split(/\s+/);
const size = parseInt(fields[4] ?? "", 10);
const name = quoted?.[1] ?? "";
if (!name.startsWith("initrd")) continue;
if (!Number.isFinite(size) || size <= 0) continue;
if (best === undefined || size > best.size) {
best = { name, size };
}
}
return best;
}
const VYOS_NIGHTLY_RELEASES =
"https://api.github.com/repos/vyos/vyos-nightly-build/releases/latest";
/**
* Resolve the configured VyOS ISO URL, expanding the "latest" sentinel.
*
* The nightly asset filename embeds a build date, so there is no stable
* "latest.iso" path to hardcode — the newest release has to be looked up.
* Any other value is used verbatim, which is how VYOS_ISO_URL pins a build
* or points at a locally mirrored copy.
*/
function resolveVyosIsoUrl(configured: string): string {
if (configured !== "latest") return configured;
const body = execSync(`curl -sSfL "${VYOS_NIGHTLY_RELEASES}"`, {
encoding: "utf-8",
stdio: ["pipe", "pipe", "pipe"],
});
const release = JSON.parse(body) as {
tag_name?: string;
assets?: Array<{ name: string; browser_download_url: string }>;
};
const asset = (release.assets ?? []).find((a) =>
/generic-amd64\.iso$/.test(a.name),
);
if (!asset) {
throw new Error(
`No generic-amd64 ISO asset in VyOS nightly release ${release.tag_name ?? "?"}`,
);
}
logger.info(` VyOS ISO resolved to ${asset.name} (${release.tag_name ?? "?"})`);
return asset.browser_download_url;
}
/**
* Extract VyOS netboot artifacts from the release ISO.
*
* VyOS publishes no netboot bundle, so kernel/initrd/squashfs have to come out
* of the ISO. xorriso is already in the bastion image (used for boot.iso) and
* extracts without root or a loop mount.
*
* The initrd needs care: /live contains an empty initrd.img placeholder
* alongside the real one, which carries a version-suffixed name. Booting the
* 0-byte file fails with no useful diagnostic, so pick the largest initrd*.
*/
export function prepareVyosArtifacts(config: BastionConfig): void {
const kernel = `${config.httpDir}/vyos-vmlinuz`;
const initrd = `${config.httpDir}/vyos-initrd`;
const squashfs = `${config.httpDir}/vyos-filesystem.squashfs`;
if (existsSync(kernel) && existsSync(initrd) && existsSync(squashfs)) {
logger.info(" VyOS netboot artifacts -- cached");
return;
}
const iso = `${config.bastionDir}/vyos.iso`;
download(resolveVyosIsoUrl(config.vyosIsoUrl), iso, "VyOS ISO");
const extract = (isoPath: string, dest: string, label: string): void => {
execSync(
`xorriso -osirrox on -indev "${iso}" -extract "${isoPath}" "${dest}"`,
{ stdio: "pipe" },
);
logger.info(` ${label} -- extracted from ${isoPath}`);
};
extract("/live/vmlinuz", kernel, "VyOS kernel");
extract("/live/filesystem.squashfs", squashfs, "VyOS squashfs");
// Pick the real initrd by size from the ISO's own directory listing.
const listing = execSync(`xorriso -indev "${iso}" -lsl /live/ --`, {
encoding: "utf-8",
stdio: ["pipe", "pipe", "pipe"],
});
const best = pickLargestInitrd(listing);
if (best === undefined) {
throw new Error("No non-empty initrd found in /live on the VyOS ISO");
}
extract(`/live/${best.name}`, initrd, `VyOS initrd (${best.name}, ${best.size} bytes)`);
// The ISO is only needed to produce the three artifacts above.
try {
unlinkSync(iso);
} catch {
// Non-fatal: leaving it costs disk but nothing else.
}
}
function symlinkSafe(target: string, linkPath: string): void {
try {
symlinkSync(target, linkPath);
@@ -182,6 +301,17 @@ export async function startBastion(overrides: Partial<BastionConfig> = {}): Prom
logger.warn(`Ubuntu ${config.ubuntuVersion} artifacts not available -- Ubuntu provisioning disabled`);
}
// VyOS netboot artifacts (non-fatal — same policy as Ubuntu)
try {
logger.info("Preparing VyOS netboot artifacts...");
prepareVyosArtifacts(config);
} catch (err) {
logger.warn(
`VyOS artifacts not available -- VyOS provisioning disabled ` +
`(${err instanceof Error ? err.message : String(err)})`,
);
}
// Symlink iPXE binaries into HTTP dir for UEFI HTTP Boot
for (const name of ["ipxe.efi", "ipxe-arm64.efi"]) {
const src = `${config.tftpDir}/${name}`;
@@ -261,6 +391,7 @@ export async function startBastion(overrides: Partial<BastionConfig> = {}): Prom
role: msg.role as import("@lab/shared").Role,
os: msg.os as import("@lab/shared").OsId,
queued_at: new Date().toISOString(),
...(msg.vyos ? { vyos: msg.vyos } : {}),
};
});
return { status: "ok", data: { mac: msg.mac, hostname: msg.hostname } };
@@ -309,6 +440,32 @@ export async function startBastion(overrides: Partial<BastionConfig> = {}): Prom
return { status: "ok", data: { mac, hostname: msg.hostname } };
});
labdConn.onCommand("command-discover", async (msg) => {
if (msg.type !== "command-discover") throw new Error("unexpected");
const mac = (msg.mac as string).toLowerCase();
const now = new Date().toISOString();
const existing = state.load().discovered[mac];
state.update((s) => {
s.discovered[mac] = {
mac,
product: (msg.product as string) ?? "unknown",
board: (msg.board as string) ?? "unknown",
serial: (msg.serial as string) ?? "unknown",
manufacturer: (msg.manufacturer as string) ?? "unknown",
cpu_model: (msg.cpu_model as string) ?? "unknown",
cpu_cores: (msg.cpu_cores as number) ?? 0,
memory_gb: (msg.memory_gb as number) ?? 0,
arch: (msg.arch as string) ?? "unknown",
disks: (msg.disks as Array<{ name: string; size_gb: number; model: string }>) ?? [],
nics: (msg.nics as Array<{ name: string; mac: string; state: string }>) ?? [],
first_seen: existing?.first_seen ?? now,
last_seen: now,
};
});
logger.info(`HARDWARE UPDATED: ${mac} -- ${msg.manufacturer ?? "?"} ${msg.product ?? "?"} (${msg.cpu_model ?? "?"}, ${msg.cpu_cores ?? "?"} cores, ${msg.memory_gb ?? "?"}GB RAM)`);
return { status: "ok", data: { mac } };
});
labdConn.onCommand("command-role-update", async (msg) => {
if (msg.type !== "command-role-update") throw new Error("unexpected");
const mac = msg.mac.toLowerCase();

View File

@@ -5,8 +5,8 @@
// /api/discover - receive hardware discovery reports from PXE-booted machines
import type { FastifyInstance } from "fastify";
import type { HardwareInfo, InstalledInfo, Role } from "@lab/shared";
import { isValidOsId, SUPPORTED_ROLES } from "@lab/shared";
import type { HardwareInfo, InstalledInfo, Role, VyosInstallSpec } from "@lab/shared";
import { isValidOsId, SUPPORTED_ROLES, SUPPORTED_OS } from "@lab/shared";
import type { StateManager } from "../services/state.js";
import { logger } from "../services/logger.js";
import { triggerPostProvisionK3s } from "../services/post-provision.js";
@@ -15,6 +15,13 @@ import type { ProgressEvent } from "../services/progress-events.js";
import type { InstallLogBuffer } from "../services/install-log.js";
import type { SyslogListener } from "../services/syslog-listener.js";
/**
* Seconds after dispatch with zero progress before a machine is called stalled.
* Generous: the slowest legitimate gap is fetching a ~600MB VyOS squashfs over
* HTTP before the hook can report anything.
*/
const STALL_THRESHOLD_S = 8 * 60;
export function registerApiRoutes(
app: FastifyInstance,
state: StateManager,
@@ -34,9 +41,10 @@ export function registerApiRoutes(
disk?: string;
role?: string;
os?: string;
vyos?: VyosInstallSpec;
};
}>("/api/install", async (request, reply) => {
const { mac: rawMac, hostname, disk, role, os } = request.body ?? {};
const { mac: rawMac, hostname, disk, role, os, vyos } = request.body ?? {};
const mac = (rawMac ?? "").toLowerCase().replace(/-/g, ":");
if (mac === "") {
@@ -50,7 +58,7 @@ export function registerApiRoutes(
const osId = os ?? "fedora-43";
if (!isValidOsId(osId)) {
return reply.status(400).send({ error: `invalid os: '${osId}'. Supported: fedora-43, ubuntu-26.04` });
return reply.status(400).send({ error: `invalid os: '${osId}'. Supported: ${SUPPORTED_OS.join(", ")}` });
}
state.update((s) => {
@@ -60,6 +68,7 @@ export function registerApiRoutes(
role: validRole as Role,
os: osId,
queued_at: new Date().toISOString(),
...(vyos ? { vyos } : {}),
};
});
@@ -139,20 +148,36 @@ export function registerApiRoutes(
? detailStr.replace("ready at ", "").trim()
: "";
const hw = s.discovered[mac];
const installedInfo: InstalledInfo = {
hostname: cfg?.hostname ?? "?",
role: cfg?.role ?? "?",
...(cfg?.os !== undefined ? { os: cfg.os } : {}),
ip,
installed_at: new Date().toISOString(),
// Preserve hardware info from discovery
...(hw ? {
product: hw.product,
manufacturer: hw.manufacturer,
cpu_model: hw.cpu_model,
cpu_cores: hw.cpu_cores,
memory_gb: hw.memory_gb,
arch: hw.arch,
} : {}),
};
s.installed[mac] = installedInfo;
const admin = installedInfo.role !== "vanilla" && installedInfo.role !== "" ? "michal" : "root";
// VyOS: the only login user is "vyos", and a router never runs k3s —
// without this guard a non-vanilla role + recorded IP would trigger
// the k3s post-provision against a VyOS box.
const isVyos = (installedInfo.os ?? "").startsWith("vyos");
const admin = isVyos
? "vyos"
: installedInfo.role !== "vanilla" && installedInfo.role !== "" ? "lab" : "root";
console.log(`\n \x1b[0;32m\x1b[1m ssh ${admin}@${ip}\x1b[0m\n`); // eslint-disable-line no-console
// Auto-install k3s for non-vanilla roles
if (installedInfo.role !== "vanilla" && ip !== "") {
if (!isVyos && installedInfo.role !== "vanilla" && ip !== "") {
void triggerPostProvisionK3s(installedInfo.hostname, ip, installedInfo.role, admin, mac);
}
}
@@ -359,6 +384,23 @@ export function registerApiRoutes(
});
});
// Simple machine state query (used by ks-auto for ISO boot dispatch)
app.get<{
Params: { mac: string };
}>("/api/machine-state/:mac", async (request, reply) => {
const mac = request.params.mac.toLowerCase().replace(/-/g, ":");
const currentState = state.load();
if (currentState.debug[mac]) return reply.send("debug");
if (currentState.install_queue[mac]) {
const progress = currentState.install_queue[mac].progress;
return reply.send(progress ? "installing" : "queued");
}
if (currentState.installed[mac]) return reply.send("installed");
if (currentState.discovered[mac]) return reply.send("discovered");
return reply.send("unknown");
});
// Update a machine's role (e.g. promote infra -> labcontroller)
app.post<{
Body: {
@@ -407,6 +449,15 @@ export function registerApiRoutes(
const installedEntry = currentState.installed[mac];
if (queueEntry) {
// A machine that was handed an install script but has reported nothing
// since is wedged BEFORE the installer environment came up — a bad
// kernel/initrd, no network in the initramfs, or the wrong NIC picked.
// Surfacing it here is what makes that diagnosable without a console.
const since = queueEntry.progress_at ?? queueEntry.dispatched_at;
const stalledForS = since !== undefined && queueEntry.progress === undefined
? Math.floor((Date.now() - new Date(since).getTime()) / 1000)
: 0;
return reply.send({
mac,
hostname: queueEntry.hostname,
@@ -414,6 +465,9 @@ export function registerApiRoutes(
progress: queueEntry.progress ?? "queued",
progress_detail: queueEntry.progress_detail ?? "",
progress_at: queueEntry.progress_at ?? queueEntry.queued_at,
dispatched_at: queueEntry.dispatched_at,
stalled_for_s: stalledForS,
stalled: stalledForS > STALL_THRESHOLD_S,
role: queueEntry.role,
os: queueEntry.os,
stages: queueEntry.log ?? [],

View File

@@ -0,0 +1,176 @@
// Routes for Asahi Linux provisioning.
// GET /asahi — wrapper script (curl bastion:8080/asahi | sh)
// GET /asahi/installer_data.json — custom installer config (built or fallback)
// GET /asahi/repo/* — serves built rootfs package (fedora-asahi-lab.zip)
// GET /asahi/firstboot.sh — first-boot LVM setup script (for manual use)
import type { FastifyInstance } from "fastify";
import fastifyStatic from "@fastify/static";
import { existsSync, readFileSync } from "node:fs";
import { join, dirname } from "node:path";
import { fileURLToPath } from "node:url";
import type { BastionConfig } from "@lab/shared";
import { renderFirstbootScript, renderFirstbootUnit } from "../templates/asahi-firstboot.sh.js";
import type { Role } from "@lab/shared";
/** Find the asahi-repo directory (built by scripts/build-asahi-rootfs.sh). */
function findAsahiRepo(config: BastionConfig): string | null {
// Check relative to bastionDir (container deploy)
const inBastionDir = join(config.bastionDir, "asahi-repo");
if (existsSync(inBastionDir)) return inBastionDir;
// Check /data/asahi-repo (PVC mount in k3s container)
if (existsSync("/data/asahi-repo")) return "/data/asahi-repo";
// Check relative to project root (dev mode)
try {
const thisDir = dirname(fileURLToPath(import.meta.url));
const projectRoot = join(thisDir, "..", "..", "..", "..");
const inProjectRoot = join(projectRoot, "asahi-repo");
if (existsSync(inProjectRoot)) return inProjectRoot;
} catch { /* import.meta.url not available in tests */ }
return null;
}
export function registerAsahiRoutes(app: FastifyInstance, config: BastionConfig): void {
const repoDir = findAsahiRepo(config);
// Serve built rootfs package files (fedora-asahi-lab.zip, etc.)
if (repoDir) {
app.register(fastifyStatic, {
root: repoDir,
prefix: "/asahi/repo/",
decorateReply: false,
});
}
// Wrapper script — user runs: curl http://bastion:8080/asahi | sh
app.get("/asahi", async (_request, reply) => {
const script = `#!/bin/bash
# Lab Asahi provisioner — sets up Apple Silicon machines with lab LVM layout.
# This wraps the standard Asahi installer with custom installer_data.json
# that creates a separate LVM data partition.
set -euo pipefail
BASTION="http://${config.serverIp}:${config.httpPort}"
echo ""
echo " ╔══════════════════════════════════════════════╗"
echo " ║ Lab Asahi Provisioner ║"
echo " ║ Bastion: \${BASTION} ║"
echo " ╚══════════════════════════════════════════════╝"
echo ""
# Check we're on macOS
if [ "$(uname)" != "Darwin" ]; then
echo "ERROR: This script must be run from macOS on the target Mac."
echo " It uses the Asahi Linux installer to set up Apple Silicon boot."
exit 1
fi
# Download the standard Asahi installer
echo "Downloading Asahi Linux installer..."
WORKDIR=$(mktemp -d)
cd "$WORKDIR"
INSTALLER_BASE="https://cdn.asahilinux.org/installer"
PKG_VER=$(curl -s "\${INSTALLER_BASE}/latest")
echo " Version: \${PKG_VER}"
curl -# -L -o "installer-\${PKG_VER}.tar.gz" "\${INSTALLER_BASE}/installer-\${PKG_VER}.tar.gz"
echo " Extracting..."
tar xf "installer-\${PKG_VER}.tar.gz"
# Download our custom installer_data.json (installer reads it as a local file)
echo " Downloading custom installer data from bastion..."
curl -sfL -o installer_data.json "\${BASTION}/asahi/installer_data.json"
# Pre-download the rootfs package (avoids Python HTTP streaming issues on macOS)
echo " Downloading rootfs package from bastion..."
mkdir -p os
curl -# -L -o os/fedora-asahi-lab.zip "\${BASTION}/asahi/repo/fedora-asahi-lab.zip"
# Point installer to local directory (REPO_BASE + /os/ + package name)
export REPO_BASE="\${PWD}"
echo ""
echo " Using custom partition layout + rootfs from bastion."
echo " This will create:"
echo " - Standard Asahi boot infrastructure (m1n1 + U-Boot)"
echo " - Fedora Asahi Remix root partition"
echo " - LVM data partition (remaining space)"
echo ""
echo " After first boot, SSH in and set up LVM:"
echo " ssh lab@<ip> 'curl -sf \${BASTION}/asahi/firstboot.sh | sudo bash'"
echo ""
# Run the installer
if [ "$USER" != "root" ]; then
echo "The installer needs root. Enter your sudo password if prompted."
exec caffeinate -dis sudo -E ./install.sh "$@"
else
exec caffeinate -dis ./install.sh "$@"
fi
`;
return reply.type("text/x-shellscript").send(script);
});
// Custom installer_data.json — serves built config or fallback
app.get("/asahi/installer_data.json", async (_request, reply) => {
// Prefer the built installer_data.json (from build-asahi-rootfs.sh)
if (repoDir) {
const builtConfig = join(repoDir, "installer_data.json");
if (existsSync(builtConfig)) {
const data = JSON.parse(readFileSync(builtConfig, "utf-8"));
return reply.type("application/json").send(data);
}
}
// Fallback: minimal config (won't have boot.img, for testing only)
return reply.type("application/json").send({
os_list: [{
name: "Fedora Asahi Lab",
default_os_name: "Fedora Linux with Lab LVM",
boot_object: "m1n1.bin",
next_object: "m1n1/boot.bin",
package: "fedora-asahi-lab.zip",
supported_fw: ["13.5"],
partitions: [
{ name: "EFI", type: "EFI", size: "524288000B", format: "fat",
copy_firmware: true, copy_installer_data: true, source: "esp" },
{ name: "Root", type: "Linux", size: "5368709120B", image: "root.img", expand: false },
{ name: "Data", type: "Linux", size: "1073741824B", expand: true },
],
}],
});
});
// First-boot script — for manual download or embedding in rootfs
app.get<{
Querystring: { hostname?: string; role?: string; mac?: string; user?: string };
}>("/asahi/firstboot.sh", async (request, reply) => {
const hostname = request.query.hostname ?? "unknown";
const role = (request.query.role ?? "infra") as Role;
const mac = request.query.mac ?? "unknown";
const user = request.query.user ?? "lab";
const script = renderFirstbootScript({
hostname,
role,
serverIp: config.serverIp,
httpPort: config.httpPort,
sshKeys: config.sshKeys ?? [],
adminUser: user,
mac,
});
return reply.type("text/x-shellscript").send(script);
});
// Systemd unit file for first-boot service
app.get("/asahi/firstboot.service", async (_request, reply) => {
return reply.type("text/plain").send(renderFirstbootUnit());
});
}

View File

@@ -137,7 +137,7 @@ function generateIso(config: BastionConfig, outputPath: string): void {
"# Map iPXE arch names to Fedora mirror paths (arm64 -> aarch64)",
"set fedarch ${buildarch}",
"iseq ${buildarch} arm64 && set fedarch aarch64 ||",
`kernel file:/vmlinuz-\${buildarch} inst.ks=${bastionUrl}/discover.ks inst.repo=${FEDORA_MIRROR_BASE}/${config.fedoraVersion}/Everything/\${fedarch}/os inst.text || goto no_kernel`,
`kernel file:/vmlinuz-\${buildarch} inst.ks=${bastionUrl}/ks-auto inst.repo=${FEDORA_MIRROR_BASE}/${config.fedoraVersion}/Everything/\${fedarch}/os inst.text || goto no_kernel`,
`initrd file:/initrd-\${buildarch} || goto no_kernel`,
"boot || shell",
"",

View File

@@ -15,6 +15,7 @@ import {
renderLocalBootIpxe,
} from "../templates/boot.ipxe.js";
import { renderUbuntuInstallIpxe } from "../templates/ubuntu-boot.ipxe.js";
import { renderVyosInstallIpxe } from "../templates/vyos-boot.ipxe.js";
import { renderDebugKickstart } from "../templates/debug.ks.js";
import { logger } from "../services/logger.js";
@@ -99,8 +100,22 @@ echo "==============================="
const os = queueEntry.os ?? "fedora-43";
logger.info(`INSTALL STARTED: ${mac} -> ${hostname} (${os})`);
// Stamp the handoff so a machine that boots the installer but never
// reports can be spotted without a console.
state.update((s) => {
const entry = s.install_queue[mac];
if (entry) entry.dispatched_at = new Date().toISOString();
});
let script: string;
if (os.startsWith("ubuntu")) {
if (os.startsWith("vyos")) {
script = renderVyosInstallIpxe({
mac,
hostname,
serverIp: config.serverIp,
httpPort: config.httpPort,
});
} else if (os.startsWith("ubuntu")) {
script = renderUbuntuInstallIpxe({
mac,
hostname,

View File

@@ -41,6 +41,150 @@ export function registerKickstartRoutes(
return reply.type("text/plain").send(ks);
});
// Auto-detecting kickstart for ISO boot (no-network machines like R1 ARM).
// %pre detects MAC, queries bastion state, writes dynamic kickstart to /tmp.
// Main body %include's it — so Anaconda gets either discover or install content.
app.get("/ks-auto", async (_request, reply) => {
const bastionUrl = `http://${config.serverIp}:${config.httpPort}`;
const ks = `# Lab Bastion -- Auto-detect kickstart (ISO boot)
# %pre detects MAC, queries bastion state, writes /tmp/dynamic.ks.
# Main body %include's it to get either discovery reboot or full install.
%pre --erroronfail --log=/tmp/ks-auto.log
#!/bin/bash
set -x
# -- Detect MAC address --
MAC=$(ip link show | awk '/ether/ && !/00:00:00:00/ {print $2; exit}')
echo "Detected MAC: $MAC"
# -- Wait for network (Linux drivers may take a moment) --
for i in $(seq 1 30); do
if curl -sf "${bastionUrl}/healthz" >/dev/null 2>&1; then
echo "Bastion reachable at ${bastionUrl}"
break
fi
echo "Waiting for network... ($i/30)"
sleep 2
done
# -- Query bastion for machine state --
STATE=$(curl -sf "${bastionUrl}/api/machine-state/$MAC" 2>/dev/null || echo "unknown")
echo "Machine state: $STATE"
case "$STATE" in
queued|installing)
echo "=== Machine queued for install. Fetching install kickstart... ==="
curl -sf "${bastionUrl}/ks?mac=$MAC" > /tmp/dynamic.ks
if [ -s /tmp/dynamic.ks ]; then
echo "Install kickstart downloaded ($(wc -l < /tmp/dynamic.ks) lines)"
else
echo "ERROR: Failed to download install kickstart"
exit 1
fi
# Run any %pre scripts from the downloaded kickstart.
# Anaconda only runs %pre from the top-level file, not from %include'd files.
python3 -c "
import re, subprocess
content = open('/tmp/dynamic.ks').read()
blocks = re.findall(r'%pre[^\\n]*\\n(.*?)%end', content, re.DOTALL)
for i, script in enumerate(blocks):
path = f'/tmp/inner-pre-{i}.sh'
with open(path, 'w') as f:
f.write(script)
print(f'Running inner %pre script {i} ({len(script.splitlines())} lines)')
subprocess.run(['bash', path], check=False)
"
;;
debug)
echo "=== Debug mode ==="
curl -sf "${bastionUrl}/debug.ks?mac=$MAC" > /tmp/dynamic.ks 2>/dev/null
if [ ! -s /tmp/dynamic.ks ]; then
echo "rescue" > /tmp/dynamic.ks
fi
;;
*)
echo "=== Running hardware discovery ==="
# Collect hardware info
PRODUCT=$(cat /sys/class/dmi/id/product_name 2>/dev/null || echo "unknown")
BOARD=$(cat /sys/class/dmi/id/board_name 2>/dev/null || echo "unknown")
SERIAL=$(cat /sys/class/dmi/id/product_serial 2>/dev/null || echo "unknown")
MANUFACTURER=$(cat /sys/class/dmi/id/sys_vendor 2>/dev/null || echo "unknown")
CPUMODEL=$(grep -m1 'model name' /proc/cpuinfo | cut -d: -f2 | sed 's/^ //')
CPUCORES=$(grep -c '^processor' /proc/cpuinfo)
MEMGB=$(awk '/MemTotal/ {printf "%d", $2/1024/1024}' /proc/meminfo)
ARCHTYPE=$(uname -m)
DISKS_JSON=$(lsblk -Jb -o NAME,SIZE,TYPE,MODEL 2>/dev/null | python3 -c "
import sys, json
data = json.load(sys.stdin)
disks = [d for d in data.get('blockdevices', []) if d.get('type') == 'disk']
result = []
for d in disks:
size_gb = round(int(d.get('size', 0)) / 1073741824, 1)
result.append({'name': d.get('name', '?'), 'size_gb': size_gb, 'model': (d.get('model') or 'unknown').strip()})
print(json.dumps(result))
" 2>/dev/null || echo '[]')
NICS_JSON=$(ip -j link show 2>/dev/null | python3 -c "
import sys, json
nics = json.load(sys.stdin)
result = []
for n in nics:
if n.get('link_type') == 'loopback': continue
result.append({'name': n.get('ifname', '?'), 'mac': n.get('address', '?'), 'state': n.get('operstate', '?')})
print(json.dumps(result))
" 2>/dev/null || echo '[]')
PAYLOAD=$(python3 -c "
import json
print(json.dumps({
'mac': '$MAC', 'product': '$PRODUCT', 'board': '$BOARD', 'serial': '$SERIAL',
'manufacturer': '$MANUFACTURER', 'cpu_model': '$CPUMODEL',
'cpu_cores': int('$CPUCORES' or 0), 'memory_gb': int('$MEMGB' or 0),
'arch': '$ARCHTYPE', 'disks': $DISKS_JSON, 'nics': $NICS_JSON
}))
")
curl -sf -X POST "${bastionUrl}/api/discover" \\
-H "Content-Type: application/json" \\
-d "$PAYLOAD" || true
echo ""
echo "=== Discovery complete ==="
echo "Machine MAC: $MAC"
echo "Queue for install: labctl provision install $MAC <hostname> --role infra"
echo "Then reboot to start installation."
echo ""
# Write a minimal kickstart that just reboots
cat > /tmp/dynamic.ks << 'DISCOVER_KS'
# Discovery mode -- reboot to allow install queue
reboot
DISCOVER_KS
# Force reboot now (don't wait for Anaconda)
sleep 3
echo 1 > /proc/sys/kernel/sysrq
echo b > /proc/sysrq-trigger
sleep 5
reboot -f
;;
esac
%end
# Include the dynamically chosen kickstart
%include /tmp/dynamic.ks
`;
return reply.type("text/plain").send(ks);
});
// Ubuntu autoinstall user-data (cloud-init)
app.get<{ Params: { mac: string } }>("/autoinstall/:mac/user-data", async (request, reply) => {
const mac = request.params.mac.toLowerCase().replace(/-/g, ":");

View File

@@ -0,0 +1,71 @@
// VyOS network install routes.
//
// VyOS has no unattended installer, so the automation is injected via
// live-config's `hooks` component: the iPXE script passes
// live-config.hooks=<.../vyos/autoinstall.sh>, live-config wgets it and runs it
// as root, and that script fetches and executes the generated install driver.
import type { FastifyInstance } from "fastify";
import type { BastionConfig } from "@lab/shared";
import type { StateManager } from "../services/state.js";
import { buildVyosConfigSpec } from "../templates/vyos-config-spec.js";
import { renderVyosInstallPy } from "../templates/vyos-install.py.js";
import { logger } from "../services/logger.js";
function normalizeMac(value: string | undefined): string {
return (value ?? "").toLowerCase().replace(/-/g, ":");
}
export function registerVyosRoutes(
app: FastifyInstance,
config: BastionConfig,
state: StateManager,
): void {
// live-config hook. Kept minimal: everything version-specific lives in the
// generated Python. wget is guaranteed present -- live-config used it to
// fetch this very script.
app.get<{ Querystring: { mac?: string } }>("/vyos/autoinstall.sh", async (request, reply) => {
const mac = normalizeMac(request.query.mac);
const base = `http://${config.serverIp}:${config.httpPort}`;
logger.info(`VYOS AUTOINSTALL HOOK served to ${mac || "unknown MAC"}`);
const script = `#!/bin/sh
# Lab PXE Bastion -- VyOS unattended install hook (run by live-config as root)
set -eu
wget -q "${base}/vyos/install.py?mac=${mac}" -O /tmp/vyos-install.py
exec python3 /tmp/vyos-install.py
`;
return reply.type("text/plain").send(script);
});
// Per-MAC install driver, with the machine's config spec baked in.
app.get<{ Querystring: { mac?: string } }>("/vyos/install.py", async (request, reply) => {
const mac = normalizeMac(request.query.mac);
const queueEntry = state.load().install_queue[mac];
const spec = buildVyosConfigSpec({
hostname: queueEntry?.hostname ?? "vyos",
spec: queueEntry?.vyos,
defaultPassword: config.vyosDefaultPassword,
sshKeys: config.sshKeys,
disk: queueEntry?.disk,
});
logger.info(
`VYOS INSTALL DRIVER served to ${mac} (${spec.hostname}, ` +
`${spec.sets.length} config ops, disk="${spec.disk || "auto"}")`,
);
const script = renderVyosInstallPy({
spec,
mac,
serverIp: config.serverIp,
httpPort: config.httpPort,
role: queueEntry?.role ?? "vanilla",
});
return reply.type("text/plain").send(script);
});
}

View File

@@ -11,6 +11,8 @@ import { logger } from "./services/logger.js";
import { registerDispatchRoutes } from "./routes/dispatch.js";
import { registerKickstartRoutes } from "./routes/kickstart.js";
import { registerApiRoutes } from "./routes/api.js";
import { registerAsahiRoutes } from "./routes/asahi.js";
import { registerVyosRoutes } from "./routes/vyos.js";
export function createApp(config: BastionConfig): { app: ReturnType<typeof Fastify>; state: StateManager; installLog: InstallLogBuffer; syslog: SyslogListener } {
@@ -45,6 +47,8 @@ export function createApp(config: BastionConfig): { app: ReturnType<typeof Fasti
registerDispatchRoutes(app, config, state);
registerKickstartRoutes(app, config, state, syslog);
registerApiRoutes(app, state, installLog, syslog);
registerAsahiRoutes(app, config);
registerVyosRoutes(app, config, state);
// boot.iso is generated at startup and served as a static file from httpDir
// (static serving supports HTTP Range requests, required by JetKVM streaming)

View File

@@ -166,6 +166,7 @@ export class BastionConnection {
case "command-role-update":
case "command-debug":
case "command-register":
case "command-discover":
void this.handleCommand(msg);
break;
}

View File

@@ -0,0 +1,311 @@
// First-boot LVM setup script for Asahi-provisioned machines.
// Embedded in the custom rootfs as a systemd service that runs once on first boot.
// Creates the standard lab LVM layout on the data partition, matching install.ks.ts.
import type { Role } from "@lab/shared";
export interface AsahiFirstbootParams {
hostname: string;
role: Role;
serverIp: string;
httpPort: number;
sshKeys: string[];
adminUser: string;
mac: string;
}
export function renderFirstbootScript(params: AsahiFirstbootParams): string {
const { hostname, role, serverIp, httpPort, sshKeys, adminUser, mac } = params;
const isWorker = role === "worker";
const isInfra = role === "infra" || role === "labcontroller";
// Role-specific LV creation commands
const roleLvLines: string[] = [];
const roleFormatLines: string[] = [];
const roleMountLines: string[] = [];
const roleFstabLines: string[] = [];
if (isInfra) {
roleLvLines.push('lvcreate -L 20480M -n rancher labvg -y');
roleFormatLines.push('mkfs.xfs /dev/labvg/rancher');
roleMountLines.push('mount_lv rancher /var/lib/rancher');
roleFstabLines.push('echo "/dev/labvg/rancher /var/lib/rancher xfs defaults 0 0" >> /etc/fstab');
}
if (isWorker || isInfra) {
roleLvLines.push('lvcreate -l 100%FREE -n longhorn labvg -y');
roleFormatLines.push('mkfs.xfs /dev/labvg/longhorn');
roleMountLines.push('mount_lv longhorn /var/lib/longhorn');
roleFstabLines.push('echo "/dev/labvg/longhorn /var/lib/longhorn xfs defaults 0 0" >> /etc/fstab');
}
// SSH key injection block (empty if no keys)
const sshKeyBlock = sshKeys.length > 0
? sshKeys.map(k => `echo '${k}' >> "$ADMIN_SSH/authorized_keys"`).join('\n')
: 'true # no SSH keys configured';
const rootSshKeyBlock = sshKeys.length > 0
? sshKeys.map(k => `echo '${k}' >> /root/.ssh/authorized_keys`).join('\n')
: 'true # no SSH keys configured';
// NOTE: All bash $ references use $VAR not \${VAR} to avoid TS template conflicts.
// Where ${} is needed in bash, we use \\${...} to escape.
return `#!/bin/bash
# Lab first-boot LVM setup — generated by bastion
# This script runs once on first boot via systemd, then disables itself.
set -euo pipefail
MARKER="/etc/lab-lvm-setup-done"
LOG="/var/log/lab-firstboot.log"
exec > >(tee -a "$LOG") 2>&1
echo "=== Lab first-boot LVM setup ==="
date
# Already done?
if [ -f "$MARKER" ]; then
echo "LVM setup already completed, skipping."
exit 0
fi
# ── Find the data partition ──────────────────────────────────────
# The data partition/disk is a large block device that is NOT the root filesystem.
# Handles: NVMe partitions, SCSI partitions, whole unpartitioned disks.
ROOT_DEV=$(findmnt -n -o SOURCE / | sed 's/\\[.*\\]//') # strip btrfs subvol
ROOT_DISK=$(lsblk -n -o PKNAME "$ROOT_DEV" 2>/dev/null | head -1)
echo "Root device: $ROOT_DEV (disk: $ROOT_DISK)"
DATA_PART=""
# Scan partitions first, then whole disks
for part in /dev/nvme*n*p* /dev/sd*[0-9] /dev/vd*[0-9] /dev/nvme*n* /dev/sd[b-z] /dev/vd[b-z]; do
[ -b "$part" ] || continue
# Skip root device and root disk
[ "$part" = "$ROOT_DEV" ] && continue
PART_DISK=$(basename "$part" | sed 's/p[0-9]*$//' | sed 's/[0-9]*$//')
[ "$PART_DISK" = "$ROOT_DISK" ] && continue
# Skip small devices (<50GB) — EFI, boot, APFS stubs
SIZE_BYTES=$(blockdev --getsize64 "$part" 2>/dev/null || echo 0)
SIZE_GB=$((SIZE_BYTES / 1073741824))
[ "$SIZE_GB" -lt 50 ] && continue
# Use if unformatted or already LVM
FSTYPE=$(blkid -o value -s TYPE "$part" 2>/dev/null || echo "")
if [ -z "$FSTYPE" ] || [ "$FSTYPE" = "LVM2_member" ]; then
DATA_PART="$part"
echo "Found data device: $DATA_PART ($SIZE_GB GB)"
break
fi
done
if [ -z "$DATA_PART" ]; then
echo "ERROR: No suitable data partition found for LVM."
echo "Expected a large (>50GB) unformatted partition."
exit 1
fi
# ── Helper function ──────────────────────────────────────────────
mount_lv() {
local lv="$1" mp="$2"
if lvs "labvg/$lv" &>/dev/null; then
mkdir -p "$mp"
mount "/dev/labvg/$lv" "$mp" 2>/dev/null || true
echo " Mounted $lv -> $mp"
fi
}
# ── Write fstab function (idempotent) ────────────────────────────
write_lab_fstab() {
# Remove any previous lab LVM entries (clean slate)
sed -i '/# lab-lvm:/d' /etc/fstab
sed -i '/# Lab LVM volumes/d' /etc/fstab
grep -v "/dev/labvg/" /etc/fstab > /etc/fstab.tmp && mv /etc/fstab.tmp /etc/fstab
# Comment out non-LVM entries for mount points we manage
for mp in "/var " "/var/log " "/home " "/srv "; do
if grep -q "$mp" /etc/fstab; then
awk -v m="$mp" '{if($0 !~ /^#/ && index($0,m)) print "# lab-lvm: " $0; else print}' /etc/fstab > /etc/fstab.tmp
mv /etc/fstab.tmp /etc/fstab
fi
done
# Add fresh LVM entries
echo "# Lab LVM volumes" >> /etc/fstab
echo "/dev/labvg/swap none swap defaults 0 0" >> /etc/fstab
echo "/dev/labvg/var /var xfs defaults 0 0" >> /etc/fstab
echo "/dev/labvg/varlog /var/log xfs defaults 0 0" >> /etc/fstab
echo "/dev/labvg/home /home xfs defaults 0 0" >> /etc/fstab
echo "/dev/labvg/srv /srv xfs defaults 0 0" >> /etc/fstab
${roleFstabLines.join('\n ')}
}
# ── Check for existing VG ────────────────────────────────────────
if vgs labvg &>/dev/null; then
echo "Volume group 'labvg' already exists — reprovision detected."
echo "Activating existing volumes..."
vgchange -ay labvg
mount_lv var /var
mount_lv varlog /var/log
mount_lv home /home
mount_lv srv /srv
${roleMountLines.map(l => ` ${l}`).join('\n')}
# Enable swap
if lvs labvg/swap &>/dev/null; then
swapon /dev/labvg/swap 2>/dev/null || true
echo " Enabled swap"
fi
# Ensure fstab entries exist — comment out conflicting btrfs subvol entries
write_lab_fstab
echo "Existing LVM volumes re-mounted."
else
# ── Fresh install: create LVM ────────────────────────────────────
echo "Creating LVM on $DATA_PART..."
pvcreate "$DATA_PART"
vgcreate labvg "$DATA_PART"
# Create LVs — sizes match install.ks.ts (in MiB)
echo "Creating logical volumes..."
lvcreate -L 27648M -n swap labvg -y # 27GB swap
lvcreate -L 102400M -n var labvg -y # 100GB /var
lvcreate -L 10240M -n varlog labvg -y # 10GB /var/log
lvcreate -L 10240M -n home labvg -y # 10GB /home
lvcreate -L 20480M -n srv labvg -y # 20GB /srv
${roleLvLines.join('\n')}
# Format
echo "Formatting volumes..."
mkswap /dev/labvg/swap
mkfs.xfs /dev/labvg/var
mkfs.xfs /dev/labvg/varlog
mkfs.xfs /dev/labvg/home
mkfs.xfs /dev/labvg/srv
${roleFormatLines.join('\n')}
# Migrate and mount volumes that can be switched live.
# Copy existing content first so we don't shadow files (e.g. /home/user/.ssh).
for LV_MOUNT in "home /home" "srv /srv"; do
LV_NAME=$(echo "$LV_MOUNT" | awk '{print $1}')
MOUNT_PT=$(echo "$LV_MOUNT" | awk '{print $2}')
STAGING="/mnt/labvg-$LV_NAME-staging"
mkdir -p "$STAGING"
mount "/dev/labvg/$LV_NAME" "$STAGING"
cp -a "$MOUNT_PT"/. "$STAGING/" 2>/dev/null || true
umount "$STAGING"
rmdir "$STAGING"
mount_lv "$LV_NAME" "$MOUNT_PT"
done
# Mount role-specific volumes (empty, no content to preserve)
set +e
${roleMountLines.join('\n')}
set -e
# Copy existing /var content into the LV for next boot
echo "Preparing /var LV for next boot..."
TMPVAR="/mnt/labvg-var-staging"
mkdir -p "$TMPVAR"
mount /dev/labvg/var "$TMPVAR"
cp -a /var/. "$TMPVAR/" 2>/dev/null || true
umount "$TMPVAR"
rmdir "$TMPVAR"
# Same for /var/log
TMPVARLOG="/mnt/labvg-varlog-staging"
mkdir -p "$TMPVARLOG"
mount /dev/labvg/varlog "$TMPVARLOG"
cp -a /var/log/. "$TMPVARLOG/" 2>/dev/null || true
umount "$TMPVARLOG"
rmdir "$TMPVARLOG"
echo "NOTE: /var and /var/log will switch to LVM on next reboot."
# Enable swap
swapon /dev/labvg/swap 2>/dev/null || true
write_lab_fstab
echo "LVM setup complete."
lvs labvg
fi # end if/else for reprovision vs fresh install
# ── Set hostname (use configured value, or keep existing) ────────
CONF_HOSTNAME="${hostname}"
if [ "$CONF_HOSTNAME" != "unknown" ] && [ -n "$CONF_HOSTNAME" ]; then
hostnamectl set-hostname "$CONF_HOSTNAME"
fi
ACTUAL_HOSTNAME=$(hostname)
# ── Detect MAC address ───────────────────────────────────────────
CONF_MAC="${mac}"
if [ "$CONF_MAC" = "unknown" ] || [ -z "$CONF_MAC" ]; then
CONF_MAC=$(ip -o link show | grep -v "lo:" | grep "state UP" | head -1 | grep -oP 'link/ether \\K[^ ]+' || echo "unknown")
fi
# ── Configure admin user ─────────────────────────────────────────
ADMIN="${adminUser}"
if ! id "$ADMIN" &>/dev/null; then
useradd -m -G wheel "$ADMIN"
echo "$ADMIN ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/$ADMIN
chmod 440 /etc/sudoers.d/$ADMIN
fi
ADMIN_SSH="/home/$ADMIN/.ssh"
mkdir -p "$ADMIN_SSH"
chmod 700 "$ADMIN_SSH"
${sshKeyBlock}
chmod 600 "$ADMIN_SSH/authorized_keys"
chown -R $ADMIN:$ADMIN "$ADMIN_SSH"
# Also authorize root
mkdir -p /root/.ssh
chmod 700 /root/.ssh
${rootSshKeyBlock}
chmod 600 /root/.ssh/authorized_keys
# ── Harden SSH (takes effect on next sshd restart/reboot) ────────
sed -i 's/^#*PermitRootLogin.*/PermitRootLogin prohibit-password/' /etc/ssh/sshd_config
sed -i 's/^#*PasswordAuthentication.*/PasswordAuthentication no/' /etc/ssh/sshd_config
# ── Write provisioning metadata ──────────────────────────────────
cat > /etc/lab-provisioned << LABMETA
hostname=$ACTUAL_HOSTNAME
role=${role}
mac=$CONF_MAC
provisioned_at=$(date -Iseconds)
method=asahi-firstboot
LABMETA
# ── Register with bastion ─────────────────────────────────────────
IP=$(hostname -I | awk '{print $1}')
echo "Registering with bastion at ${serverIp}:${httpPort}..."
curl -sf -X POST "http://${serverIp}:${httpPort}/api/register" \\
-H "Content-Type: application/json" \\
-d "{\\"mac\\":\\"$CONF_MAC\\",\\"hostname\\":\\"$ACTUAL_HOSTNAME\\",\\"role\\":\\"${role}\\",\\"ip\\":\\"$IP\\"}" \\
2>/dev/null && echo " Registered as $ACTUAL_HOSTNAME ($IP)" \\
|| echo " WARNING: Could not reach bastion — register manually with: labctl provision register $CONF_MAC $ACTUAL_HOSTNAME --role ${role} --ip $IP"
# ── Mark done ────────────────────────────────────────────────────
touch "$MARKER"
echo "=== First-boot setup complete ==="
`;
}
/** Systemd unit file for the first-boot service */
export function renderFirstbootUnit(): string {
return `[Unit]
Description=Lab first-boot LVM setup
After=local-fs.target network-online.target
Wants=network-online.target
ConditionPathExists=!/etc/lab-lvm-setup-done
[Service]
Type=oneshot
ExecStart=/usr/local/bin/lab-firstboot.sh
RemainAfterExit=yes
StandardOutput=journal+console
StandardError=journal+console
[Install]
WantedBy=multi-user.target
`;
}

View File

@@ -40,6 +40,11 @@ export function renderInstallKickstart(params: InstallKickstartParams): string {
const now = new Date().toISOString();
const hasLonghorn = role === "worker";
const hasRancher = role === "infra";
// k8s roles get a dedicated 120G image-store LV. 2026-08 incident: the old
// 20G LV idled at 85% used, so a single ~5G image pull tripped imagefs
// eviction. Must be sized here — longhorn's --grow consumes all remaining
// VG space, making post-install lvextend impossible on worker nodes.
const hasRancherLv = role === "infra" || role === "worker";
const isVanilla = role === "vanilla";
// -- Auth section --
@@ -113,9 +118,9 @@ done
? `logvol /var/lib/longhorn --vgname=${vg} --name=longhorn --fstype=xfs --grow --size=1`
: "";
// -- Rancher LV for fresh install (infra role) --
const rancherFreshLine = hasRancher
? `logvol /var/lib/rancher --vgname=${vg} --name=rancher --fstype=xfs --size=20480`
// -- Rancher LV for fresh install (k8s roles: worker + infra) --
const rancherFreshLine = hasRancherLv
? `logvol /var/lib/rancher --vgname=${vg} --name=rancher --fstype=xfs --size=122880`
: "";
return `# Lab Bastion -- Fedora ${fedoraVersion} server install

View File

@@ -0,0 +1,53 @@
// iPXE boot script template for VyOS network install.
//
// VyOS ships no unattended installer: `install image` is unconditionally
// interactive (image_installer.py's install action takes no arguments, and
// --no-prompt is wired only to `add`). So PXE boots the *live* system and the
// automation is injected through live-config's `hooks` component, which fetches
// a script over HTTP and runs it as root late in live boot.
//
// Unlike the Fedora/Ubuntu paths this boots a live image rather than an
// installer, so there is no kickstart/autoinstall equivalent — see
// routes/vyos.ts for the hook that actually drives the install.
export function renderVyosInstallIpxe(params: {
mac: string;
hostname: string;
serverIp: string;
httpPort: number;
}): string {
const base = `http://${params.serverIp}:${params.httpPort}`;
// Pin the boot NIC by MAC. live-boot otherwise scans for the first
// *connected* interface, and on a multi-NIC box that race is lost by
// whichever port negotiates slowest: on the Protectli VP2440 the SFP+
// pair links first, so live-boot picked the fiber ports (which have no
// DHCP), burned 15s per port, and gave up with "Unable to find a live
// file system on the network" -- while the copper port that actually PXE
// booted came up at 4.6s and was never tried.
//
// live-boot's Device_from_bootif() strips the "01-" and matches the MAC
// against /sys/class/net/*. params.mac is the dispatch key, i.e. exactly
// the NIC that PXE booted -- more reliable than iPXE's ${net0} on a box
// where the booting NIC may not be net0.
const bootif = `01-${params.mac.toLowerCase().replace(/:/g, "-")}`;
// Deliberately NOT passing `nonetworking` (present in VyOS's own PXE docs):
// live-config's hook component needs networking up to fetch the hook over
// HTTP. Also no `console=ttyS0` — on hardware without a physical UART that
// costs 30s at every systemd boot phase.
return `#!ipxe
echo
echo =============================================
echo Lab PXE Bastion - INSTALLING VyOS
echo Target: ${params.hostname}
echo MAC: ${params.mac}
echo =============================================
echo
kernel ${base}/vyos-vmlinuz boot=live nopersistence noautologin BOOTIF=${bootif} fetch=${base}/vyos-filesystem.squashfs live-config.hooks=${base}/vyos/autoinstall.sh?mac=${params.mac}
initrd ${base}/vyos-initrd
boot
`;
}

View File

@@ -0,0 +1,352 @@
// Builds the VyOS configuration spec applied by the autoinstall hook.
//
// We deliberately do NOT emit a config.boot file as text. A config.boot carries a
// `vyos-config-version` trailer; without a trailer matching the running image,
// VyOS runs its migration scripts from version 0 on first boot. Instead the hook
// loads the image's own /opt/vyatta/etc/config.boot.default through vyos.configtree
// and applies these set operations on top, so syntax and version trailer always
// match the exact image being installed.
import type { VyosInstallSpec } from "@lab/shared";
export interface VyosSetOp {
path: string[];
value?: string;
/** false appends to a multi-value node (e.g. bond members) instead of replacing. */
replace?: boolean;
}
export interface VyosConfigSpec {
hostname: string;
/** "" means accept the installer default (the running image's version string). */
imageName: string;
password: string;
console: "K" | "S";
/** Target disk name (e.g. "nvme0n1"); "" accepts the installer's first-disk default. */
disk: string;
/**
* IP the driver should report in the "complete" callback ("ready at <ip>" —
* the exact format routes/api.ts parses installed.ip from). The mgmt
* address when static; "" means detect the live DHCP address at runtime.
*/
reportAddress: string;
/** Whether to accept RAID-1 when the installer finds more than one disk. */
raid: boolean;
/** Overwrite the installed config.boot with the generated one on reinstall. */
freshConfig: boolean;
sets: VyosSetOp[];
/** Paths that are VyOS tag nodes — must be marked as such in the ConfigTree. */
tags: string[][];
}
/**
* Normalise a target disk to the form the installer expects.
*
* find_disks() enumerates via `lsblk -Jbp` (-p = full paths), so its valid
* responses are "/dev/mmcblk0"-style. A bare "mmcblk0" is rejected by
* ask_input()'s valid_responses check and re-prompts forever.
*/
function normalizeDiskPath(value: string | undefined): string {
const raw = (value ?? "").trim();
if (raw === "") return "";
return raw.startsWith("/dev/") ? raw : `/dev/${raw}`;
}
/** Sentinel marking a value that lives in Pulumi config, not in the bundle. */
const SECRET_PREFIX = "@secret:";
/**
* Enable the VyOS HTTP API so the router is manageable the moment it boots.
*
* This belongs at install time rather than in the Pulumi model: the model is
* applied THROUGH this API, so a router that lacks it cannot be brought under
* management without a hand-run change on a live firewall. It is also why the
* model excludes `service https` outright -- a provider able to rewrite its own
* transport can lock itself out permanently.
*
* `listen-address` is always set. Leaving it unbound would expose a
* config-write endpoint on every segment the router touches, the WAN included.
*/
function apiSets(apiKey: string, listenAddress: string): VyosSetOp[] {
const sets: VyosSetOp[] = [
{ path: ["service", "https", "api", "keys", "id", "pulumi", "key"], value: apiKey },
{ path: ["service", "https", "api", "rest"] },
];
if (listenAddress !== "") {
sets.push({ path: ["service", "https", "listen-address"], value: listenAddress });
}
return sets;
}
/** Tag nodes introduced by the API config, needed by the installer's ConfigTree. */
const API_TAGS: string[][] = [["service", "https", "api", "keys", "id"]];
/**
* The address to bind the API to: an explicit choice, else the management
* address with its prefix length stripped. Under DHCP there is no address to
* bind at build time, so the caller must pass one or the listener stays unbound
* and the API is not enabled at all.
*/
function apiListenAddress(spec: VyosInstallSpec, mgmtAddress: string): string {
if (spec.apiListenAddress !== undefined && spec.apiListenAddress !== "") {
return spec.apiListenAddress;
}
return mgmtAddress.includes("/") ? (mgmtAddress.split("/")[0] ?? "") : "";
}
/**
* Use a Pulumi-rendered bundle as the router's config verbatim.
*
* Secret-valued nodes are dropped rather than installed with their sentinel
* text: writing `@secret:pppoePassword` into config.boot would look configured
* while being wrong, which is worse than being absent. The router comes up
* without those values and the first `pulumi up` fills them in.
*
* `system host-name` is forced to the hostname the install was asked for. The
* bundle carries the name of whichever router it was exported from, and
* installing vyos001's hostname onto vyos002 would collide on the network.
*/
function buildFromBundle(
params: { hostname: string; defaultPassword: string; disk?: string | undefined },
spec: VyosInstallSpec,
bundle: NonNullable<VyosInstallSpec["bundle"]>,
mgmtAddress: string,
): VyosConfigSpec {
const sets: VyosSetOp[] = [];
const dropped: string[] = [];
for (const op of bundle.sets) {
if (op.value !== undefined && op.value.startsWith(SECRET_PREFIX)) {
dropped.push(op.path.join(" "));
continue;
}
if (op.path.length === 2 && op.path[0] === "system" && op.path[1] === "host-name") {
continue;
}
sets.push({
path: op.path,
...(op.value === undefined ? {} : { value: op.value }),
...(op.replace === undefined ? {} : { replace: op.replace }),
});
}
sets.unshift({ path: ["system", "host-name"], value: params.hostname });
if (dropped.length > 0) {
console.warn(
`vyos ${params.hostname}: ${dropped.length} secret-valued node(s) left unset by the ` +
`bundle; run \`pulumi up\` to supply them: ${dropped.join(", ")}`,
);
}
const tags = [...bundle.tags];
const api = enableApi(spec, params.hostname, mgmtAddress);
if (api.length > 0) {
sets.push(...api);
tags.push(...API_TAGS);
}
return {
hostname: params.hostname,
imageName: "",
password: spec.password ?? params.defaultPassword,
console: "K",
disk: normalizeDiskPath(params.disk),
reportAddress: mgmtAddress.includes("/") ? (mgmtAddress.split("/")[0] ?? "") : "",
raid: false,
freshConfig: spec.freshConfig ?? false,
sets,
tags,
};
}
/**
* The API config for this install, or nothing when it cannot be enabled safely.
*
* Refusing to enable it unbound is deliberate. Under DHCP there is no address
* known at build time, and the alternative -- binding to every interface --
* would publish a config-write endpoint on the WAN. Better to leave the router
* SSH-only and say so than to open it everywhere.
*/
function enableApi(spec: VyosInstallSpec, hostname: string, mgmtAddress: string): VyosSetOp[] {
if (spec.apiKey === undefined || spec.apiKey === "") return [];
const listen = apiListenAddress(spec, mgmtAddress);
if (listen === "") {
console.warn(
`vyos ${hostname}: --vyos-api-key given but no address to bind to ` +
`(management is "${mgmtAddress}"). Pass --vyos-api-listen <addr>; the HTTP API ` +
`has NOT been enabled, so Pulumi cannot manage this router yet.`,
);
return [];
}
return apiSets(spec.apiKey, listen);
}
export function buildVyosConfigSpec(params: {
hostname: string;
spec?: VyosInstallSpec | undefined;
defaultPassword: string;
sshKeys?: string[] | undefined;
disk?: string | undefined;
}): VyosConfigSpec {
const spec = params.spec ?? {};
const mgmt = spec.mgmtInterface ?? "eth0";
const mgmtAddress = spec.mgmtAddress ?? "dhcp";
// A rendered bundle replaces the derived config entirely. Deriving a second
// opinion alongside it is the drift the bundle exists to prevent: Pulumi and
// labctl would each believe they knew the router's config, and the box would
// end up with whichever ran last.
if (spec.bundle !== undefined) {
return buildFromBundle(params, spec, spec.bundle, mgmtAddress);
}
const bondMembers = spec.bondMembers ?? [];
const vlans = spec.vlans ?? [];
const sets: VyosSetOp[] = [];
const tags: string[][] = [
["interfaces", "ethernet"],
["system", "login", "user"],
];
const hwIds = spec.hwIds ?? {};
const pinHwId = (iface: string): void => {
const mac = hwIds[iface];
if (mac !== undefined && mac !== "") {
sets.push({ path: ["interfaces", "ethernet", iface, "hw-id"], value: mac });
}
};
sets.push({ path: ["system", "host-name"], value: params.hostname });
// Management interface — the NIC that PXE booted, left untagged and unbonded.
sets.push({ path: ["interfaces", "ethernet", mgmt, "address"], value: mgmtAddress });
pinHwId(mgmt);
// Tagged management VLAN on the PXE port. Emitted regardless of bonding, so
// the box stays reachable on the management VLAN while still booting untagged
// on whichever VLAN the bastion's proxy DHCP serves.
const mgmtVlan = spec.mgmtVlan;
if (mgmtVlan !== undefined) {
tags.push(["interfaces", "ethernet", mgmt, "vif"]);
const vif = ["interfaces", "ethernet", mgmt, "vif", String(mgmtVlan.id)];
sets.push({ path: [...vif, "address"], value: mgmtVlan.address });
if (mgmtVlan.description !== undefined && mgmtVlan.description !== "") {
sets.push({ path: [...vif, "description"], value: mgmtVlan.description });
}
}
// LACP bond. Members must exclude the PXE NIC; firmware PXE cannot run over LACP.
const bonded = bondMembers.length > 0;
if (bonded) {
tags.push(["interfaces", "bonding"]);
sets.push({ path: ["interfaces", "bonding", "bond0", "mode"], value: "802.3ad" });
sets.push({ path: ["interfaces", "bonding", "bond0", "hash-policy"], value: "layer2+3" });
for (const member of bondMembers) {
sets.push({
path: ["interfaces", "bonding", "bond0", "member", "interface"],
value: member,
replace: false,
});
pinHwId(member);
}
// Address on the trunk's native/untagged VLAN.
if (spec.bondAddress !== undefined && spec.bondAddress !== "") {
sets.push({ path: ["interfaces", "bonding", "bond0", "address"], value: spec.bondAddress });
}
}
// VRRP groups accumulate here; emitted (plus a sync group) after the VLANs.
// interface accepts dotted vifs (constraint regex `[0-9]+(.\d+)?`), address
// is a tag node (the VIP is the tag value itself), vrid range is 1-255.
const vrrpGroups: Array<{ name: string; iface: string; vrid: number; vip: string }> = [];
if (bonded && spec.bondVrrp !== undefined && spec.bondVrrp !== "") {
// vrid 1 for the untagged group: the native VLAN is never a vif, so this
// cannot collide with a vlan-id-derived vrid.
vrrpGroups.push({ name: "native", iface: "bond0", vrid: 1, vip: spec.bondVrrp });
}
// Tagged VLAN sub-interfaces hang off the bond when there is one, else off mgmt.
const parent = bonded
? ["interfaces", "bonding", "bond0"]
: ["interfaces", "ethernet", mgmt];
if (vlans.length > 0) {
tags.push([...parent, "vif"]);
const parentName = bonded ? "bond0" : mgmt;
for (const vlan of vlans) {
const vif = [...parent, "vif", String(vlan.id)];
sets.push({ path: [...vif, "address"], value: vlan.address });
if (vlan.description !== undefined && vlan.description !== "") {
sets.push({ path: [...vif, "description"], value: vlan.description });
}
if (vlan.vrrp !== undefined && vlan.vrrp !== "") {
vrrpGroups.push({
name: `vlan${vlan.id}`,
iface: `${parentName}.${vlan.id}`,
vrid: vlan.id,
vip: vlan.vrrp,
});
}
}
}
// Emit VRRP groups plus one sync group so all VLANs fail over together —
// without it a single-link event could split mastership across the pair.
if (vrrpGroups.length > 0) {
tags.push(["high-availability", "vrrp", "group"]);
tags.push(["high-availability", "vrrp", "sync-group"]);
const priority = String(spec.vrrpPriority ?? 100);
for (const g of vrrpGroups) {
const base = ["high-availability", "vrrp", "group", g.name];
sets.push({ path: [...base, "interface"], value: g.iface });
sets.push({ path: [...base, "vrid"], value: String(g.vrid) });
sets.push({ path: [...base, "priority"], value: priority });
// address is a tag node: the VIP is the path's final segment, no value.
sets.push({ path: [...base, "address", g.vip] });
tags.push([...base, "address"]);
sets.push({
path: ["high-availability", "vrrp", "sync-group", "MAIN", "member"],
value: g.name,
replace: false,
});
}
}
sets.push({ path: ["service", "ssh", "port"], value: "22" });
const sshKeys = params.sshKeys ?? [];
if (sshKeys.length > 0) {
tags.push(["system", "login", "user", "vyos", "authentication", "public-keys"]);
sshKeys.forEach((entry, index) => {
const parts = entry.trim().split(/\s+/);
const type = parts[0] ?? "";
const key = parts[1] ?? "";
if (!type.startsWith("ssh-") && !type.startsWith("ecdsa-")) return;
if (!key) return;
const name = parts[2] ?? `lab-key-${index}`;
const base = ["system", "login", "user", "vyos", "authentication", "public-keys", name];
sets.push({ path: [...base, "type"], value: type });
sets.push({ path: [...base, "key"], value: key });
});
}
// Enabled here too, not just for bundle installs: every VyOS this bastion
// provisions should be manageable from first boot.
const api = enableApi(spec, params.hostname, mgmtAddress);
if (api.length > 0) {
sets.push(...api);
tags.push(...API_TAGS);
}
return {
hostname: params.hostname,
imageName: "",
password: spec.password ?? params.defaultPassword,
console: "K",
disk: normalizeDiskPath(params.disk),
// Static mgmt address wins; under DHCP the driver detects the live IP.
reportAddress: mgmtAddress.includes("/") ? (mgmtAddress.split("/")[0] ?? "") : "",
raid: false,
freshConfig: spec.freshConfig ?? false,
sets,
tags,
};
}

View File

@@ -0,0 +1,514 @@
// Renders the Python program that performs the unattended VyOS install.
//
// It runs as root inside the live system, fetched and executed by live-config's
// `hooks` component (see vyos-boot.ipxe.ts). It does three things:
// 1. builds config.boot from the image's own default via vyos.configtree
// 2. drives the interactive `install image` through a pty
// 3. reports progress back to the bastion, then reboots
//
// A pty is used rather than piping stdin because the installer reads the
// password through getpass(), which opens /dev/tty directly and would ignore a
// pipe. Prompts are matched by text rather than replayed positionally: the
// installer skips the boot-config question when it finds a previous
// installation, so a fixed answer sequence desyncs on reinstall.
import type { VyosConfigSpec } from "./vyos-config-spec.js";
export function renderVyosInstallPy(params: {
spec: VyosConfigSpec;
mac: string;
serverIp: string;
httpPort: number;
role: string;
}): string {
// Base64 so arbitrary values (passwords, descriptions, SSH keys) can never
// terminate the Python string literal that carries them.
const specB64 = Buffer.from(JSON.stringify(params.spec), "utf-8").toString("base64");
return `#!/usr/bin/env python3
"""Unattended VyOS install driver -- generated by the lab PXE bastion."""
import base64
import json
import os
import pty
import re
import select
import subprocess
import sys
import time
import urllib.request
SPEC = json.loads(base64.b64decode("${specB64}").decode("utf-8"))
BASTION = "http://${params.serverIp}:${params.httpPort}"
MAC = "${params.mac}"
ROLE = ${JSON.stringify(params.role ?? "vanilla")}
INSTALLER = "/usr/libexec/vyos/op_mode/image_installer.py"
CONFIG_DIR = "/opt/vyatta/etc/config"
# The installer copies the rootfs from the boot MEDIUM path -- which only a
# CD/USB boot provides. With fetch= (HTTP netboot) nothing is mounted there
# (verified in VM: Errno 2), so the squashfs must be linked or re-fetched into
# place before 'install image' runs.
ROOTFS_EXPECTED = "/usr/lib/live/mount/medium/live/filesystem.squashfs"
SQUASHFS_URL = "http://${params.serverIp}:${params.httpPort}/vyos-filesystem.squashfs"
# The live-config hook runs BEFORE vyos-router creates the /opt/vyatta compat
# path, so the squashfs's own location must be tried too (verified in VM: only
# /usr/share/vyos/config.boot.default exists at hook time).
DEFAULT_CONFIG_CANDIDATES = [
"/opt/vyatta/etc/config.boot.default",
"/usr/share/vyos/config.boot.default",
]
STALL_TIMEOUT = 900 # seconds without installer output before giving up
def detect_ip():
"""Best-effort local IP as seen on the route toward the bastion.
Matches Fedora's semantics (IP captured during install): under DHCP the
installed system will renew on the same NIC/subnet the live env used.
"""
import socket
try:
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.connect(("${params.serverIp}", ${params.httpPort}))
ip = s.getsockname()[0]
s.close()
return ip
except Exception:
return ""
class LogStreamer:
"""Stream install output to the bastion's /api/log so 'labctl provision
logs -f' works live for VyOS, like Anaconda's syslog does for Fedora.
Strictly best-effort: a failed POST drops the batch and must never stall
the pty read loop or fail the install.
"""
ANSI = re.compile(rb"\\x1b\\[[0-9;?]*[a-zA-Z]|\\x1b[=>]|\\r")
def __init__(self):
self.partial = b""
self.pending = []
self.last_flush = time.time()
def feed(self, chunk):
"""Raw pty bytes: split into lines, strip ANSI noise, queue."""
self.partial += chunk
while b"\\n" in self.partial:
raw, self.partial = self.partial.split(b"\\n", 1)
text = self.ANSI.sub(b"", raw).decode("utf-8", "replace").rstrip()
if text:
self.pending.append(text)
self.maybe_flush()
def line(self, text):
"""A driver-originated message (already a clean string)."""
self.pending.append(text)
self.maybe_flush()
def maybe_flush(self):
if len(self.pending) >= 20 or (self.pending and time.time() - self.last_flush >= 2):
self.flush()
def flush(self):
if not self.pending:
return
batch, self.pending = self.pending[:200], self.pending[200:]
self.last_flush = time.time()
try:
body = json.dumps({"mac": MAC, "lines": batch}).encode()
req = urllib.request.Request(
BASTION + "/api/log",
data=body,
headers={"Content-Type": "application/json"},
)
urllib.request.urlopen(req, timeout=5).read()
except Exception:
pass
STREAM = LogStreamer()
def say(msg):
"""Print locally and stream to the bastion log buffer."""
print(msg)
STREAM.line(str(msg))
def report(stage, detail=""):
"""Best-effort progress callback; never fatal."""
STREAM.flush()
try:
body = json.dumps({"mac": MAC, "stage": stage, "detail": detail}).encode()
req = urllib.request.Request(
BASTION + "/api/progress",
data=body,
headers={"Content-Type": "application/json"},
)
urllib.request.urlopen(req, timeout=5).read()
except Exception:
pass
def build_config():
"""Apply our set operations onto the image's own default config.
Using config.boot.default as the base keeps the vyos-config-version trailer
consistent with the running image, so first boot does not run migrations.
"""
from vyos.configtree import ConfigTree
default_config = next(
(p for p in DEFAULT_CONFIG_CANDIDATES if os.path.exists(p)), None)
if default_config is None:
raise FileNotFoundError(
"no config.boot.default found (tried %s)" % ", ".join(DEFAULT_CONFIG_CANDIDATES))
say("base config: %s" % default_config)
with open(default_config) as handle:
config = ConfigTree(handle.read())
for op in SPEC["sets"]:
replace = op.get("replace", True)
if "value" in op and op["value"] is not None:
config.set(op["path"], value=op["value"], replace=replace)
else:
config.set(op["path"])
# Tag nodes must be marked after the nodes exist, as the installer itself does.
for tag in SPEC["tags"]:
try:
config.set_tag(tag)
except Exception as err:
say("warning: set_tag %s failed: %s" % (tag, err))
os.makedirs(CONFIG_DIR, exist_ok=True)
target = os.path.join(CONFIG_DIR, "config.boot")
# Re-attach the vyos-config-version footer: ConfigTree.to_string() emits
# only the config body, and a config without the footer is treated as
# ancient -- the boot migrator then runs every migration over it and (as
# observed in the VM test) crashes in system/31-to-32. Building the footer
# from the running system pins it to the exact image being installed.
body = config.to_string()
try:
from vyos.component_version import version_info_from_system
info = version_info_from_system()
info.update_config_body(body)
info.write(target)
say("wrote %s (footer: %s)" % (target, info.release))
except Exception as err:
say("warning: version footer failed (%s); writing bare config" % err)
with open(target, "w") as handle:
handle.write(body)
return target
def find_live_squashfs():
"""Locate the squashfs live-boot fetched, without walking into the mounted
rootfs or overlay (each would mean traversing the entire OS tree)."""
explicit = [
"/run/live/medium/live/filesystem.squashfs",
"/lib/live/mount/medium/live/filesystem.squashfs",
]
for path in explicit:
if os.path.isfile(path) and os.path.getsize(path) > 0:
return path
for root in ("/run/live", "/lib/live/mount", "/usr/lib/live/mount"):
for dirpath, dirs, files in os.walk(root):
depth = dirpath.count(os.sep) - root.count(os.sep)
dirs[:] = [d for d in dirs
if d not in ("rootfs", "overlay")
and not d.endswith(".squashfs")
and depth < 3]
if "filesystem.squashfs" in files:
path = os.path.join(dirpath, "filesystem.squashfs")
if os.path.isfile(path) and os.path.getsize(path) > 0:
return path
return None
def ensure_rootfs():
"""Make FILE_ROOTFS_SRC exist so the installer can copy the system image."""
if os.path.isfile(ROOTFS_EXPECTED) and os.path.getsize(ROOTFS_EXPECTED) > 0:
return
src = find_live_squashfs()
if src is None:
say("squashfs not in live mounts; re-fetching %s" % SQUASHFS_URL)
src = "/tmp/filesystem.squashfs"
urllib.request.urlretrieve(SQUASHFS_URL, src)
os.makedirs(os.path.dirname(ROOTFS_EXPECTED), exist_ok=True)
if os.path.lexists(ROOTFS_EXPECTED):
os.remove(ROOTFS_EXPECTED)
os.symlink(src, ROOTFS_EXPECTED)
say("rootfs source: %s -> %s" % (ROOTFS_EXPECTED, src))
def build_rules():
"""Prompt -> response table for the interactive installer."""
password = SPEC["password"].encode() + b"\\n"
image_name = SPEC["imageName"].encode() + b"\\n"
disk = SPEC["disk"].encode() + b"\\n"
console = SPEC["console"].encode() + b"\\n"
raid = (b"yes\\n" if SPEC["raid"] else b"no\\n")
return [
(re.compile(rb"Would you like to continue\\?"), b"yes\\n"),
(re.compile(rb"What would you like to name this image\\?"), image_name),
(re.compile(rb"Please confirm password for the .vyos. user:"), password),
(re.compile(rb"Please enter a password for the .vyos. user:"), password),
(re.compile(rb"What console should be used by default"), console),
# Three RAID variants: "configure RAID-1 mirroring?", "...on them?",
# and "choose two disks for RAID-1 mirroring?" -- all default to YES,
# so a missed one both hangs the install and risks an unwanted mirror.
(re.compile(rb"Would you like to [^?]*RAID-1 mirroring"), raid),
(re.compile(rb"Installation will delete all data on (?:the drive|both drives)\\. Continue\\?"), b"yes\\n"),
(re.compile(rb"Which one should be used for installation\\?"), disk),
(re.compile(rb"Would you like to use all the free space on the drive\\?"), b"yes\\n"),
(re.compile(rb"Which file would you like as boot config\\?"), b"1\\n"),
# Reinstall path only (search_previous_installation): carrying the old
# /config and SSH host keys forward is VyOS's "reinstall without losing
# data". Always yes -- freshConfig replaces config.boot afterwards, so
# answering no here would also discard non-config data under /config.
(re.compile(rb"Would you like to copy data to the new image\\?"), b"yes\\n"),
(re.compile(rb"Would you like to copy the encrypted config to the new image\\?"), b"yes\\n"),
# More than one previous image found -- take the first offered.
(re.compile(rb"From which image would you like to save config information\\?"), b"1\\n"),
(re.compile(rb"From which image would you like to copy the encrypted config\\?"), b"1\\n"),
]
def run_installer():
"""Drive image_installer.py over a pty, answering prompts as they appear."""
rules = build_rules()
master, slave = pty.openpty()
proc = subprocess.Popen(
[INSTALLER, "--action", "install"],
stdin=slave,
stdout=slave,
stderr=slave,
close_fds=True,
preexec_fn=os.setsid,
)
os.close(slave)
buf = b""
transcript = b"" # rolling tail of everything the installer printed
last_output = time.time()
while True:
ready, _, _ = select.select([master], [], [], 1.0)
if ready:
try:
chunk = os.read(master, 4096)
except OSError:
break
if not chunk:
break
sys.stdout.buffer.write(chunk)
sys.stdout.buffer.flush()
buf += chunk
transcript = (transcript + chunk)[-8000:]
STREAM.feed(chunk)
last_output = time.time()
# Answer every prompt currently in the buffer, earliest first, so
# ordering is preserved even when the installer skips questions --
# and so a single chunk carrying two prompts gets both answers.
while True:
best = None
for pattern, response in rules:
found = pattern.search(buf)
if found and (best is None or found.start() < best[0].start()):
best = (found, response)
if best is None:
break
found, response = best
os.write(master, response)
transcript = (transcript + b"\\n>>> answered: " + response)[-8000:]
STREAM.line(">>> answered: " + response.decode("utf-8", "replace").strip())
buf = buf[found.end():]
# Bound memory if the installer emits a lot without prompting.
if len(buf) > 65536:
buf = buf[-8192:]
elif proc.poll() is not None:
break
STREAM.maybe_flush()
if time.time() - last_output > STALL_TIMEOUT:
proc.kill()
raise SystemExit("installer produced no output for %ds" % STALL_TIMEOUT)
os.close(master)
return proc.wait(), transcript.decode("utf-8", "replace")
def ensure_network_boot_first():
"""Keep network boot first so the bastion intercepts every reboot.
Port of the Fedora kickstart's %post efibootmgr step (install.ks.ts) --
what makes reprovision-by-reboot work. Best-effort: skipped on BIOS boots
or when efibootmgr is absent. Runs from the live env after the installer;
efibootmgr edits NVRAM, not the disk, so installer cleanup is irrelevant.
"""
import shutil
if not os.path.isdir("/sys/firmware/efi") or shutil.which("efibootmgr") is None:
say("boot order: skipped (BIOS boot or efibootmgr missing)")
return
try:
out = subprocess.run(["efibootmgr"], capture_output=True, text=True, timeout=30).stdout
order = []
network_entry = None
for line in out.splitlines():
m = re.match(r"^BootOrder:\\s*(.*)$", line)
if m:
order = [x.strip() for x in m.group(1).split(",") if x.strip()]
continue
m = re.match(r"^Boot([0-9A-Fa-f]{4})\\*?\\s+(.*)$", line)
if m and network_entry is None:
if re.search(r"network|pxe|ipv4|ipv6|http", m.group(2), re.IGNORECASE):
network_entry = m.group(1).upper()
if network_entry is None or not order:
say("boot order: no network boot entry found; leaving as is")
return
new_order = [network_entry] + [x for x in order if x.upper() != network_entry]
if [x.upper() for x in order] == [x.upper() for x in new_order]:
say("boot order: network entry Boot%s already first" % network_entry)
return
subprocess.run(["efibootmgr", "-o", ",".join(new_order)],
capture_output=True, timeout=30)
say("boot order: moved network entry Boot%s first" % network_entry)
except Exception as err:
say("warning: boot order adjustment failed: %s" % err)
def with_target_mounted(fn):
"""Mount the installed root partition, call fn(rw_dir), always unmount.
The installer has unmounted and cleaned the target by the time this runs,
so the block device is free. The partition holding boot/<image>/rw is the
VyOS root; the glob also yields the installed image's rw dir directly.
"""
import glob
disk = SPEC["disk"]
if not disk:
# No pinned disk (installer picked the default) -- enumerate all disks.
candidates = ["/dev/" + b for b in os.listdir("/sys/block")
if not b.startswith(("loop", "ram", "zram", "sr"))]
else:
candidates = [disk]
mnt = "/mnt/lab-target"
os.makedirs(mnt, exist_ok=True)
for dev in candidates:
name = os.path.basename(dev)
parts = sorted(p for p in os.listdir("/sys/block/%s" % name)
if p.startswith(name)) if os.path.isdir("/sys/block/%s" % name) else []
for part in parts:
pdev = "/dev/" + part
if subprocess.run(["mount", pdev, mnt], capture_output=True).returncode != 0:
continue
try:
rw_dirs = glob.glob(os.path.join(mnt, "boot", "*", "rw"))
if rw_dirs:
fn(rw_dirs[0])
return True
finally:
subprocess.run(["umount", mnt], capture_output=True)
return False
def post_install_target_steps():
"""Metadata + optional fresh-config overwrite inside the installed image."""
def apply(rw_dir):
config_dir = os.path.join(rw_dir, "opt/vyatta/etc/config")
os.makedirs(config_dir, exist_ok=True)
# /config/lab-provisioned -- survives VyOS image upgrades. Mirrors the
# Fedora kickstart's /etc/lab-provisioned.
try:
with open(os.path.join(config_dir, "lab-provisioned"), "w") as handle:
handle.write("hostname=%s\\n" % SPEC["hostname"])
handle.write("role=%s\\n" % ROLE)
handle.write("provisioned=%s\\n" % time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()))
handle.write("bastion=%s\\n" % BASTION)
say("wrote /config/lab-provisioned")
except Exception as err:
say("warning: lab-provisioned metadata failed: %s" % err)
# freshConfig: make the bastion-generated config win over the previous
# installation's carried-forward config. Explicit intent -- failure is
# fatal (raised out of with_target_mounted).
if SPEC.get("freshConfig"):
import shutil
shutil.copyfile(os.path.join(CONFIG_DIR, "config.boot"),
os.path.join(config_dir, "config.boot"))
say("freshConfig: replaced installed config.boot with generated config")
mounted = with_target_mounted(apply)
if not mounted:
if SPEC.get("freshConfig"):
raise RuntimeError("freshConfig requested but installed root partition not found")
say("warning: installed root partition not found; skipping metadata")
def main():
report("vyos-install", "building config.boot")
try:
build_config()
except Exception as err:
report("error", "config generation failed: %s" % err)
raise
report("vyos-install", "staging rootfs for installer")
try:
ensure_rootfs()
except Exception as err:
report("error", "rootfs staging failed: %s" % err)
raise
report("vyos-install", "running install image")
code, transcript = run_installer()
if code != 0:
# Surface the installer's last words in bastion progress -- the console
# they were printed on is usually invisible during unattended installs.
report("error", "install image exited %d | tail: %s" % (code, transcript[-4000:]))
raise SystemExit(code)
report("post-install", "boot order + metadata")
ensure_network_boot_first()
try:
post_install_target_steps()
except Exception as err:
report("error", "post-install target steps failed: %s" % err)
raise
# "complete" is the stage the bastion uses to move a machine out of the
# install queue into installed state, and "ready at <ip>" is the exact
# detail format it parses installed.ip from -- see routes/api.ts.
ip = SPEC.get("reportAddress") or detect_ip()
report("complete", "ready at %s" % ip if ip else "VyOS installed, rebooting")
os.system("sync")
# --force: this driver is a child of live-config.service, whose start job is
# still running -- a normal reboot deadlocks waiting for it (verified in VM:
# shutdown blocked >1min on "start job is running for live-config"). The
# installer has already unmounted and cleaned the target, so an immediate
# reboot is safe.
os.system("systemctl reboot --force")
if __name__ == "__main__":
main()
`;
}

View File

@@ -0,0 +1,225 @@
import { describe, it, expect, beforeEach, afterEach } from "vitest";
import { mkdirSync, rmSync } from "node:fs";
import { join } from "node:path";
import { tmpdir } from "node:os";
import type { BastionConfig } from "@lab/shared";
import { createApp } from "../src/server.js";
import type { FastifyInstance } from "fastify";
import { renderFirstbootScript, renderFirstbootUnit } from "../src/templates/asahi-firstboot.sh.js";
function createTestConfig(testDir: string): BastionConfig {
return {
fedoraVersion: "43",
arch: "x86_64",
httpPort: 0,
timezone: "Europe/London",
locale: "en_GB.UTF-8",
bastionDir: testDir,
domain: "test.local",
dhcpMode: "proxy",
dhcpRangeStart: "",
dhcpRangeEnd: "",
ubuntuVersion: "26.04",
ubuntuMirror: "https://releases.ubuntu.com/26.04",
iface: "eth0",
serverIp: "192.168.8.1",
network: "192.168.8.0",
gateway: "192.168.8.1",
sshKeys: ["ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAITEST test@lab"],
adminUser: "michal",
syslogPort: 15514,
skipDnsmasq: true,
skipArtifacts: true,
fedoraMirror: "https://download.fedoraproject.org/pub/fedora/linux/releases/43/Everything/x86_64/os",
tftpDir: join(testDir, "tftp"),
httpDir: join(testDir, "http"),
stateFile: join(testDir, "state.json"),
};
}
describe("asahi routes", () => {
let testDir: string;
let app: FastifyInstance;
beforeEach(() => {
testDir = join(tmpdir(), `bastion-asahi-test-${Date.now()}-${Math.random().toString(36).slice(2)}`);
mkdirSync(testDir, { recursive: true });
mkdirSync(join(testDir, "http"), { recursive: true });
mkdirSync(join(testDir, "tftp"), { recursive: true });
const config = createTestConfig(testDir);
const result = createApp(config);
app = result.app;
});
afterEach(async () => {
await app.close();
rmSync(testDir, { recursive: true, force: true });
});
it("GET /asahi returns wrapper shell script", async () => {
const resp = await app.inject({ method: "GET", url: "/asahi" });
expect(resp.statusCode).toBe(200);
expect(resp.headers["content-type"]).toContain("text/x-shellscript");
expect(resp.body).toContain("#!/bin/bash");
expect(resp.body).toContain("installer_data.json");
expect(resp.body).toContain("192.168.8.1");
expect(resp.body).toContain("install.sh");
});
it("GET /asahi/installer_data.json returns valid config", async () => {
const resp = await app.inject({ method: "GET", url: "/asahi/installer_data.json" });
expect(resp.statusCode).toBe(200);
const data = JSON.parse(resp.body);
expect(data.os_list).toHaveLength(1);
const os = data.os_list[0];
expect(os.name).toContain("Fedora Asahi Lab");
// 3 partitions (fallback) or 4 (built: EFI + Boot + Root + Data)
expect(os.partitions.length).toBeGreaterThanOrEqual(3);
expect(os.partitions[0].type).toBe("EFI");
// Last partition should be the expanding Data partition
const lastPart = os.partitions[os.partitions.length - 1];
expect(lastPart.type).toBe("Linux");
expect(lastPart.expand).toBe(true);
// Root partition (second-to-last) should NOT expand
const rootPart = os.partitions[os.partitions.length - 2];
expect(rootPart.expand).toBe(false);
expect(rootPart.image).toBe("root.img");
});
it("GET /asahi/firstboot.sh returns parameterized script", async () => {
const resp = await app.inject({
method: "GET",
url: "/asahi/firstboot.sh?hostname=mac-studio&role=infra&mac=00:11:22:33:44:55",
});
expect(resp.statusCode).toBe(200);
expect(resp.body).toContain("#!/bin/bash");
expect(resp.body).toContain("mac-studio");
expect(resp.body).toContain("labvg");
expect(resp.body).toContain("rancher"); // infra gets rancher LV
expect(resp.body).toContain("longhorn"); // infra also gets longhorn
expect(resp.body).toContain("ssh-ed25519"); // SSH key injected
});
it("GET /asahi/firstboot.service returns systemd unit", async () => {
const resp = await app.inject({ method: "GET", url: "/asahi/firstboot.service" });
expect(resp.statusCode).toBe(200);
expect(resp.body).toContain("[Unit]");
expect(resp.body).toContain("lab-firstboot.sh");
expect(resp.body).toContain("ConditionPathExists=!/etc/lab-lvm-setup-done");
});
});
describe("renderFirstbootScript", () => {
const baseParams = {
hostname: "test-node",
serverIp: "10.0.0.1",
httpPort: 8080,
sshKeys: ["ssh-ed25519 AAAA... user@host"],
adminUser: "testadmin",
mac: "aa:bb:cc:dd:ee:ff",
};
it("generates valid bash with shebang", () => {
const script = renderFirstbootScript({ ...baseParams, role: "worker" });
expect(script.startsWith("#!/bin/bash")).toBe(true);
});
it("includes LVM creation commands", () => {
const script = renderFirstbootScript({ ...baseParams, role: "infra" });
expect(script).toContain("pvcreate");
expect(script).toContain("vgcreate labvg");
expect(script).toContain("lvcreate");
});
it("uses correct LV sizes from kickstart layout", () => {
const script = renderFirstbootScript({ ...baseParams, role: "infra" });
expect(script).toContain("27648M"); // swap
expect(script).toContain("102400M"); // /var
expect(script).toContain("10240M"); // /var/log and /home
expect(script).toContain("20480M"); // /srv and /rancher
});
it("includes rancher LV for infra role", () => {
const script = renderFirstbootScript({ ...baseParams, role: "infra" });
expect(script).toContain("rancher");
expect(script).toContain("/var/lib/rancher");
});
it("includes longhorn for worker role", () => {
const script = renderFirstbootScript({ ...baseParams, role: "worker" });
expect(script).toContain("longhorn");
expect(script).toContain("/var/lib/longhorn");
// Worker should NOT have rancher
expect(script).not.toContain("rancher");
});
it("includes longhorn for infra role", () => {
const script = renderFirstbootScript({ ...baseParams, role: "infra" });
expect(script).toContain("longhorn");
expect(script).toContain("/var/lib/longhorn");
});
it("vanilla role gets no role-specific LVs", () => {
const script = renderFirstbootScript({ ...baseParams, role: "vanilla" });
expect(script).not.toContain("rancher");
expect(script).not.toContain("longhorn");
});
it("handles reprovision (existing labvg)", () => {
const script = renderFirstbootScript({ ...baseParams, role: "infra" });
expect(script).toContain("reprovision detected");
expect(script).toContain("vgchange -ay labvg");
expect(script).toContain("mount_lv var /var");
});
it("injects SSH keys for admin user and root", () => {
const script = renderFirstbootScript({ ...baseParams, role: "worker" });
expect(script).toContain("ssh-ed25519 AAAA...");
expect(script).toContain("testadmin");
expect(script).toContain("/root/.ssh/authorized_keys");
});
it("sets hostname", () => {
const script = renderFirstbootScript({ ...baseParams, role: "worker" });
expect(script).toContain('CONF_HOSTNAME="test-node"');
expect(script).toContain("hostnamectl set-hostname");
});
it("includes bastion self-registration", () => {
const script = renderFirstbootScript({ ...baseParams, role: "worker" });
expect(script).toContain("/api/register");
expect(script).toContain("aa:bb:cc:dd:ee:ff");
expect(script).toContain("test-node");
});
it("writes provisioning metadata", () => {
const script = renderFirstbootScript({ ...baseParams, role: "infra" });
expect(script).toContain("/etc/lab-provisioned");
expect(script).toContain("method=asahi-firstboot");
});
it("creates marker file to prevent re-run", () => {
const script = renderFirstbootScript({ ...baseParams, role: "worker" });
expect(script).toContain("/etc/lab-lvm-setup-done");
expect(script).toContain('touch "$MARKER"');
});
});
describe("renderFirstbootUnit", () => {
it("generates valid systemd unit", () => {
const unit = renderFirstbootUnit();
expect(unit).toContain("[Unit]");
expect(unit).toContain("[Service]");
expect(unit).toContain("[Install]");
expect(unit).toContain("Type=oneshot");
expect(unit).toContain("WantedBy=multi-user.target");
});
it("only runs when marker is missing", () => {
const unit = renderFirstbootUnit();
expect(unit).toContain("ConditionPathExists=!/etc/lab-lvm-setup-done");
});
});

View File

@@ -22,6 +22,8 @@ function createTestConfig(testDir: string): BastionConfig {
dhcpRangeEnd: "",
ubuntuVersion: "26.04",
ubuntuMirror: "https://releases.ubuntu.com/26.04",
vyosIsoUrl: "https://downloads.vyos.io/rolling/current/generic/vyos-rolling-latest.iso",
vyosDefaultPassword: "vyos",
iface: "eth0",
serverIp: "10.0.0.1",
network: "10.0.0.0",

View File

@@ -96,9 +96,9 @@ describe("renderInstallKickstart", () => {
expect(ks).toContain("/api/progress");
});
it("infra role has /var/lib/rancher partition", () => {
it("infra role has 120G /var/lib/rancher partition", () => {
const ks = renderInstallKickstart(baseParams({ role: "infra" }));
expect(ks).toContain("logvol /var/lib/rancher --vgname=labvg --name=rancher --fstype=xfs --size=20480");
expect(ks).toContain("logvol /var/lib/rancher --vgname=labvg --name=rancher --fstype=xfs --size=122880");
});
it("infra role has k3s install", () => {
@@ -106,10 +106,14 @@ describe("renderInstallKickstart", () => {
expect(ks).toContain("curl -sfL https://get.k3s.io | INSTALL_K3S_SKIP_START=true sh -");
});
it("worker role does NOT have /var/lib/rancher partition in fresh install", () => {
it("worker role has 120G /var/lib/rancher partition (imageFs must be sized before longhorn --grow)", () => {
const ks = renderInstallKickstart(baseParams({ role: "worker" }));
// Worker should not have the fresh-install rancher partition line
expect(ks).not.toContain("logvol /var/lib/rancher --vgname=labvg --name=rancher --fstype=xfs --size=20480");
expect(ks).toContain("logvol /var/lib/rancher --vgname=labvg --name=rancher --fstype=xfs --size=122880");
});
it("vanilla role does NOT have /var/lib/rancher partition in fresh install", () => {
const ks = renderInstallKickstart(baseParams({ role: "vanilla" }));
expect(ks).not.toContain("--name=rancher --fstype=xfs");
});
it("worker role does NOT have k3s install", () => {

View File

@@ -0,0 +1,155 @@
import { describe, it, expect, vi } from "vitest";
import type { VyosBundle } from "@lab/shared";
import { buildVyosConfigSpec } from "../src/templates/vyos-config-spec.js";
/**
* A bundle is what makes "one config, two apply paths" true rather than
* aspirational: `pulumi up` POSTs the subtree model to a running router, labctl
* writes the same model into config.boot during a PXE install. These tests pin
* the properties that keep the two honest.
*/
const bundle: VyosBundle = {
sets: [
{ path: ["system", "host-name"], value: "vyos001" },
{ path: ["interfaces", "bonding", "bond0", "address"], value: "192.168.1.252/24" },
{ path: ["interfaces", "bonding", "bond0", "member", "interface"], value: "eth1", replace: false },
{ path: ["interfaces", "bonding", "bond0", "vif", "53", "disable"] },
{ path: ["interfaces", "pppoe", "pppoe0", "authentication", "password"], value: "@secret:pppoePassword" },
{ path: ["interfaces", "pppoe", "pppoe0", "mtu"], value: "1492" },
],
tags: [["interfaces", "bonding", "bond0"], ["interfaces", "ethernet"]],
};
const build = (hostname: string, extra: Record<string, unknown> = {}) =>
buildVyosConfigSpec({
hostname,
spec: { bundle, ...extra },
defaultPassword: "changeme",
});
describe("vyos config spec from a Pulumi bundle", () => {
it("applies non-secret nodes verbatim, preserving valuelessness and replace:false", () => {
const spec = build("vyos001");
expect(spec.sets).toContainEqual({
path: ["interfaces", "bonding", "bond0", "address"],
value: "192.168.1.252/24",
});
// A multi-value node must keep replace:false or the second bond member
// overwrites the first.
expect(spec.sets).toContainEqual({
path: ["interfaces", "bonding", "bond0", "member", "interface"],
value: "eth1",
replace: false,
});
// A valueless node must not acquire a value on the way through.
expect(spec.sets).toContainEqual({
path: ["interfaces", "bonding", "bond0", "vif", "53", "disable"],
});
expect(spec.tags).toEqual(bundle.tags);
});
it("drops secret-valued nodes instead of installing the sentinel text", () => {
const warn = vi.spyOn(console, "warn").mockImplementation(() => {});
const spec = build("vyos001");
const values = spec.sets.map((s) => s.value ?? "");
expect(values.some((v) => v.startsWith("@secret:"))).toBe(false);
expect(spec.sets.some((s) => s.path.includes("authentication"))).toBe(false);
// Silently dropping the WAN credential would leave someone debugging a dead
// PPPoE link, so it has to be said out loud.
expect(warn).toHaveBeenCalledWith(expect.stringContaining("pulumi up"));
warn.mockRestore();
});
it("forces the hostname the install was asked for, not the bundle's", () => {
// The bundle is exported from one router and reused for its peer; taking the
// hostname from it would put two vyos001s on the network.
const spec = build("vyos002");
const hostnames = spec.sets.filter(
(s) => s.path.length === 2 && s.path[0] === "system" && s.path[1] === "host-name",
);
expect(hostnames).toEqual([{ path: ["system", "host-name"], value: "vyos002" }]);
});
it("still honours installer inputs, which are not router config", () => {
const spec = buildVyosConfigSpec({
hostname: "vyos001",
spec: { bundle, password: "s3cret", freshConfig: true },
defaultPassword: "changeme",
disk: "nvme0n1",
});
expect(spec.password).toBe("s3cret");
expect(spec.freshConfig).toBe(true);
expect(spec.disk).toBe("/dev/nvme0n1");
});
it("enables the HTTP API at install so Pulumi can manage the router from first boot", () => {
const warn = vi.spyOn(console, "warn").mockImplementation(() => {});
const spec = buildVyosConfigSpec({
hostname: "vyos001",
spec: { bundle, apiKey: "k3y", apiListenAddress: "10.0.1.252" },
defaultPassword: "changeme",
});
expect(spec.sets).toContainEqual({
path: ["service", "https", "api", "keys", "id", "pulumi", "key"],
value: "k3y",
});
expect(spec.sets).toContainEqual({ path: ["service", "https", "api", "rest"] });
expect(spec.sets).toContainEqual({
path: ["service", "https", "listen-address"],
value: "10.0.1.252",
});
// The key id is a tag node; without this the installer's ConfigTree rejects it.
expect(spec.tags).toContainEqual(["service", "https", "api", "keys", "id"]);
warn.mockRestore();
});
it("binds the API to the static management address when none is given", () => {
const spec = buildVyosConfigSpec({
hostname: "vyos001",
spec: { apiKey: "k3y", mgmtAddress: "192.168.1.252/24" },
defaultPassword: "changeme",
});
expect(spec.sets).toContainEqual({
path: ["service", "https", "listen-address"],
value: "192.168.1.252",
});
});
it("refuses to enable the API unbound rather than exposing it on the WAN", () => {
const warn = vi.spyOn(console, "warn").mockImplementation(() => {});
// Management is DHCP, so there is no address to bind at build time. Binding
// to everything would put a config-write endpoint on the WAN.
const spec = buildVyosConfigSpec({
hostname: "vyos001",
spec: { apiKey: "k3y", mgmtAddress: "dhcp" },
defaultPassword: "changeme",
});
expect(spec.sets.some((s) => s.path[0] === "service" && s.path[1] === "https")).toBe(false);
expect(warn).toHaveBeenCalledWith(expect.stringContaining("has NOT been enabled"));
warn.mockRestore();
});
it("does not enable the API when no key is supplied", () => {
const spec = buildVyosConfigSpec({
hostname: "vyos001",
spec: { mgmtAddress: "192.168.1.252/24" },
defaultPassword: "changeme",
});
expect(spec.sets.some((s) => s.path[0] === "service" && s.path[1] === "https")).toBe(false);
});
it("ignores the derived path entirely when a bundle is present", () => {
// Belt and braces: even if topology flags reach this far (the CLI rejects
// them), the bundle must win rather than merge.
const spec = buildVyosConfigSpec({
hostname: "vyos001",
spec: { bundle, bondMembers: ["eth2", "eth3"], vlans: [{ id: 99, address: "10.9.9.1/24" }] },
defaultPassword: "changeme",
});
expect(spec.sets.some((s) => s.path.includes("99"))).toBe(false);
expect(spec.sets.filter((s) => s.value === "eth2" || s.value === "eth3")).toEqual([]);
});
});

View File

@@ -0,0 +1,548 @@
import { describe, it, expect, beforeEach, afterEach } from "vitest";
import { mkdirSync, rmSync } from "node:fs";
import { join } from "node:path";
import { tmpdir } from "node:os";
import type { BastionConfig } from "@lab/shared";
import type { FastifyInstance } from "fastify";
import { createApp } from "../src/server.js";
import type { StateManager } from "../src/services/state.js";
import { buildVyosConfigSpec } from "../src/templates/vyos-config-spec.js";
import { renderVyosInstallPy } from "../src/templates/vyos-install.py.js";
function createTestConfig(testDir: string): BastionConfig {
return {
fedoraVersion: "43",
arch: "x86_64",
httpPort: 0,
timezone: "Europe/London",
locale: "en_GB.UTF-8",
bastionDir: testDir,
domain: "test.local",
dhcpMode: "proxy",
dhcpRangeStart: "",
dhcpRangeEnd: "",
ubuntuVersion: "26.04",
ubuntuMirror: "https://releases.ubuntu.com/26.04",
vyosIsoUrl: "https://example.invalid/vyos.iso",
vyosDefaultPassword: "test-pw",
iface: "eth0",
serverIp: "10.0.0.1",
network: "10.0.0.0",
gateway: "10.0.0.1",
sshKeys: ["ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAITEST lab@test"],
adminUser: "testadmin",
syslogPort: 15515,
skipDnsmasq: true,
skipArtifacts: true,
fedoraMirror: "https://example.invalid/fedora",
tftpDir: join(testDir, "tftp"),
httpDir: join(testDir, "http"),
stateFile: join(testDir, "state.json"),
};
}
/** Pull the base64 spec back out of the generated Python driver. */
function decodeSpecFrom(python: string): Record<string, unknown> {
const match = /base64\.b64decode\("([^"]+)"\)/.exec(python);
if (!match?.[1]) throw new Error("no base64 spec found in generated driver");
return JSON.parse(Buffer.from(match[1], "base64").toString("utf-8"));
}
describe("vyos config spec", () => {
it("puts VLANs on the bond when members are given", () => {
const spec = buildVyosConfigSpec({
hostname: "fw1",
defaultPassword: "pw",
spec: {
mgmtInterface: "eth0",
mgmtAddress: "10.0.8.2/24",
bondMembers: ["eth2", "eth3"],
vlans: [{ id: 10, address: "10.0.10.1/24", description: "k8s" }],
},
});
const paths = spec.sets.map((s) => s.path.join(" "));
expect(paths).toContain("interfaces bonding bond0 mode");
expect(paths).toContain("interfaces bonding bond0 vif 10 address");
// VLANs must hang off the bond, not the management NIC.
expect(paths).not.toContain("interfaces ethernet eth0 vif 10 address");
// Bond members are a multi-value node — appending, not replacing, is what
// keeps the second member from overwriting the first.
const members = spec.sets.filter(
(s) => s.path.join(" ") === "interfaces bonding bond0 member interface",
);
expect(members.map((m) => m.value)).toEqual(["eth2", "eth3"]);
expect(members.every((m) => m.replace === false)).toBe(true);
});
it("falls back to VLANs on the management NIC when unbonded", () => {
const spec = buildVyosConfigSpec({
hostname: "fw2",
defaultPassword: "pw",
spec: { mgmtInterface: "eth1", vlans: [{ id: 20, address: "10.0.20.1/24" }] },
});
const paths = spec.sets.map((s) => s.path.join(" "));
expect(paths).toContain("interfaces ethernet eth1 vif 20 address");
});
it("normalises the target disk to a full /dev path", () => {
// find_disks() enumerates with `lsblk -Jbp`, so valid responses are full
// paths; a bare name fails valid_responses and re-prompts forever.
expect(buildVyosConfigSpec({ hostname: "fw3", defaultPassword: "pw", disk: "/dev/mmcblk0" }).disk)
.toBe("/dev/mmcblk0");
expect(buildVyosConfigSpec({ hostname: "fw3", defaultPassword: "pw", disk: "mmcblk0" }).disk)
.toBe("/dev/mmcblk0");
expect(buildVyosConfigSpec({ hostname: "fw3", defaultPassword: "pw" }).disk).toBe("");
});
it("defaults to dhcp on eth0 and never opts into RAID", () => {
const spec = buildVyosConfigSpec({ hostname: "fw4", defaultPassword: "pw" });
const address = spec.sets.find(
(s) => s.path.join(" ") === "interfaces ethernet eth0 address",
);
expect(address?.value).toBe("dhcp");
// The installer's RAID prompt defaults to yes; a second disk must not
// silently produce a mirror.
expect(spec.raid).toBe(false);
});
});
describe("vyos routes", () => {
let testDir: string;
let app: FastifyInstance;
let state: StateManager;
const mac = "aa:bb:cc:11:22:33";
beforeEach(() => {
testDir = join(tmpdir(), `bastion-vyos-test-${Date.now()}-${Math.random().toString(36).slice(2)}`);
mkdirSync(join(testDir, "http"), { recursive: true });
mkdirSync(join(testDir, "tftp"), { recursive: true });
const result = createApp(createTestConfig(testDir));
app = result.app;
state = result.state;
});
afterEach(async () => {
await app.close();
rmSync(testDir, { recursive: true, force: true });
});
it("dispatches a queued vyos machine to the live-boot script", async () => {
state.update((s) => {
s.install_queue[mac] = {
hostname: "fw1",
disk: "/dev/nvme0n1",
role: "worker",
os: "vyos-rolling",
queued_at: new Date().toISOString(),
};
});
const response = await app.inject({ method: "GET", url: `/dispatch?mac=${mac}` });
expect(response.statusCode).toBe(200);
expect(response.body).toContain("/vyos-vmlinuz");
expect(response.body).toContain("fetch=http://10.0.0.1:0/vyos-filesystem.squashfs");
expect(response.body).toContain(`live-config.hooks=http://10.0.0.1:0/vyos/autoinstall.sh?mac=${mac}`);
// `nonetworking` appears in VyOS's own PXE docs but breaks the hook fetch,
// and console=ttyS0 costs 30s per systemd phase on boards with no UART.
expect(response.body).not.toContain("nonetworking");
expect(response.body).not.toContain("console=ttyS0");
});
it("serves a hook that fetches and executes the install driver", async () => {
const response = await app.inject({ method: "GET", url: `/vyos/autoinstall.sh?mac=${mac}` });
expect(response.statusCode).toBe(200);
expect(response.body).toContain(`/vyos/install.py?mac=${mac}`);
expect(response.body).toContain("python3 /tmp/vyos-install.py");
});
it("bakes the machine's config into the generated install driver", async () => {
state.update((s) => {
s.install_queue[mac] = {
hostname: "fw1",
disk: "/dev/nvme0n1",
role: "worker",
os: "vyos-rolling",
queued_at: new Date().toISOString(),
vyos: {
mgmtInterface: "eth0",
mgmtAddress: "10.0.8.2/24",
bondMembers: ["eth2", "eth3"],
vlans: [{ id: 10, address: "10.0.10.1/24" }],
password: "s3cret",
},
};
});
const response = await app.inject({ method: "GET", url: `/vyos/install.py?mac=${mac}` });
expect(response.statusCode).toBe(200);
// Builds config from the image's own default so the vyos-config-version
// trailer matches and first boot skips migrations.
expect(response.body).toContain("/opt/vyatta/etc/config.boot.default");
expect(response.body).toContain("/usr/libexec/vyos/op_mode/image_installer.py");
// "complete" is what moves the machine out of the install queue.
expect(response.body).toContain('report("complete"');
const spec = decodeSpecFrom(response.body);
expect(spec["hostname"]).toBe("fw1");
expect(spec["password"]).toBe("s3cret");
expect(spec["disk"]).toBe("/dev/nvme0n1");
const paths = (spec["sets"] as Array<{ path: string[] }>).map((s) => s.path.join(" "));
expect(paths).toContain("interfaces bonding bond0 vif 10 address");
expect(paths).toContain("system host-name");
});
it("falls back to the bastion default password when none is set", async () => {
state.update((s) => {
s.install_queue[mac] = {
hostname: "fw9",
disk: "",
role: "worker",
os: "vyos-rolling",
queued_at: new Date().toISOString(),
};
});
const response = await app.inject({ method: "GET", url: `/vyos/install.py?mac=${mac}` });
const spec = decodeSpecFrom(response.body);
expect(spec["password"]).toBe("test-pw");
// Empty disk means "accept the installer's first-disk default".
expect(spec["disk"]).toBe("");
});
});
describe("vyos hw-id pinning", () => {
it("emits hw-id for the mgmt interface and each bond member", () => {
// Discovery sees enp2s0/enp1s0f0np0 under Fedora, but VyOS enumerates its
// own eth<N>. Pinning by MAC is what makes the mapping deterministic.
const spec = buildVyosConfigSpec({
hostname: "fw1",
defaultPassword: "pw",
spec: {
mgmtInterface: "eth2",
bondMembers: ["eth0", "eth1"],
hwIds: {
eth2: "64:62:66:25:96:47",
eth0: "64:62:66:25:96:45",
eth1: "64:62:66:25:96:46",
},
},
});
const hw = spec.sets.filter((s) => s.path[s.path.length - 1] === "hw-id");
expect(hw.map((s) => [s.path[2], s.value])).toEqual([
["eth2", "64:62:66:25:96:47"],
["eth0", "64:62:66:25:96:45"],
["eth1", "64:62:66:25:96:46"],
]);
});
it("omits hw-id entirely when no mapping is given", () => {
const spec = buildVyosConfigSpec({ hostname: "fw1", defaultPassword: "pw" });
expect(spec.sets.some((s) => s.path.includes("hw-id"))).toBe(false);
});
});
describe("vyos management VLAN", () => {
it("puts the mgmt VLAN on the PXE port while the bond carries routed VLANs", () => {
// Trunked PXE port: boots untagged on the VLAN the bastion serves, stays
// reachable on the tagged management VLAN.
const spec = buildVyosConfigSpec({
hostname: "vyos001",
defaultPassword: "pw",
spec: {
mgmtInterface: "eth2",
mgmtAddress: "dhcp",
mgmtVlan: { id: 3, address: "192.168.3.4/24", description: "kvm" },
bondMembers: ["eth0", "eth1"],
vlans: [{ id: 2, address: "192.168.8.2/23" }],
},
});
const paths = spec.sets.map((s) => s.path.join(" "));
expect(paths).toContain("interfaces ethernet eth2 vif 3 address");
expect(paths).toContain("interfaces bonding bond0 vif 2 address");
// The mgmt VLAN must not land on the bond.
expect(paths).not.toContain("interfaces bonding bond0 vif 3 address");
expect(spec.tags.map((t) => t.join(" "))).toContain("interfaces ethernet eth2 vif");
});
});
describe("vyos VRRP HA", () => {
const haSpec = {
mgmtInterface: "eth2",
mgmtAddress: "dhcp",
bondMembers: ["eth0", "eth1"],
bondAddress: "192.168.1.252/24",
bondVrrp: "192.168.1.254/24",
vrrpPriority: 200,
vlans: [
{ id: 3, address: "192.168.3.4/24", vrrp: "192.168.3.254/24" },
{ id: 200, address: "192.168.2.252/24" }, // no VIP on this one
],
};
it("emits a vrrp group per VIP with vrid = VLAN id and dotted vif interface", () => {
const spec = buildVyosConfigSpec({ hostname: "fw1", defaultPassword: "pw", spec: haSpec });
const paths = spec.sets.map((s) => `${s.path.join(" ")}${s.value !== undefined ? "=" + s.value : ""}`);
expect(paths).toContain("interfaces bonding bond0 address=192.168.1.252/24");
// untagged bond group: vrid 1, interface bond0 itself
expect(paths).toContain("high-availability vrrp group native interface=bond0");
expect(paths).toContain("high-availability vrrp group native vrid=1");
// address is a tag node -- VIP is the final path segment, no value
expect(paths).toContain("high-availability vrrp group native address 192.168.1.254/24");
// VLAN group: vrid = VLAN id, dotted vif
expect(paths).toContain("high-availability vrrp group vlan3 interface=bond0.3");
expect(paths).toContain("high-availability vrrp group vlan3 vrid=3");
expect(paths).toContain("high-availability vrrp group vlan3 address 192.168.3.254/24");
// VLAN without a VIP gets no group
expect(paths.some((p) => p.includes("group vlan200"))).toBe(false);
});
it("applies the box-wide priority and one sync group over all groups", () => {
const spec = buildVyosConfigSpec({ hostname: "fw1", defaultPassword: "pw", spec: haSpec });
const prio = spec.sets.filter((s) => s.path[s.path.length - 1] === "priority"
&& s.path[0] === "high-availability");
expect(prio).toHaveLength(2);
expect(prio.every((s) => s.value === "200")).toBe(true);
// sync group binds the pair: all groups fail over together
const members = spec.sets.filter(
(s) => s.path.join(" ") === "high-availability vrrp sync-group MAIN member",
);
expect(members.map((m) => m.value)).toEqual(["native", "vlan3"]);
expect(members.every((m) => m.replace === false)).toBe(true);
});
it("emits no high-availability nodes when no VIPs are given", () => {
const spec = buildVyosConfigSpec({
hostname: "fw1",
defaultPassword: "pw",
spec: { bondMembers: ["eth0", "eth1"], vlans: [{ id: 3, address: "192.168.3.4/24" }] },
});
expect(spec.sets.some((s) => s.path[0] === "high-availability")).toBe(false);
});
});
describe("pickLargestInitrd", async () => {
const { pickLargestInitrd } = await import("../src/main.js");
// Verbatim from `xorriso -lsl /live/` on vyos-2026.08.05-0033-rolling.
const realListing = `total 8
-r--r--r-- 1 0 0 22255 Aug 5 01:33 'filesystem.packages'
-r--r--r-- 1 0 0 6 Aug 5 01:33 'filesystem.packages-remove'
-r--r--r-- 1 0 0 541192192 Aug 5 01:33 'filesystem.squashfs'
-r--r--r-- 1 0 0 50352547 Aug 5 01:33 'initrd.img'
-r--r--r-- 1 0 0 50352547 Aug 5 01:33 'initrd.img-6.18.41-vyos'
-r--r--r-- 1 0 0 20 Aug 5 01:33 'packages.txt'
-r--r--r-- 1 0 0 9135104 Aug 2 19:54 'vmlinuz'
-r--r--r-- 1 0 0 9135104 Aug 2 19:54 'vmlinuz-6.18.41-vyos'
`;
it("picks a full-size initrd from a real nightly listing", () => {
expect(pickLargestInitrd(realListing)).toEqual({ name: "initrd.img", size: 50352547 });
});
it("ignores 0-byte decoys and symlinks (which report link size, not target size)", () => {
const listing = `total 8
-r--r--r-- 1 0 0 0 Aug 5 01:33 'initrd.img'
lrwxrwxrwx 1 0 0 24 Aug 5 01:33 'initrd.img-link' -> 'initrd.img-6.18.41-vyos'
-r--r--r-- 1 0 0 50352547 Aug 5 01:33 'initrd.img-6.18.41-vyos'
`;
expect(pickLargestInitrd(listing)).toEqual({ name: "initrd.img-6.18.41-vyos", size: 50352547 });
});
it("returns undefined when only decoys exist", () => {
expect(pickLargestInitrd("-r--r--r-- 1 0 0 0 Aug 5 01:33 'initrd.img'\n")).toBeUndefined();
});
});
describe("vyos fedora-parity features", () => {
it("computes reportAddress from a static mgmt address, empty for dhcp", () => {
const staticSpec = buildVyosConfigSpec({
hostname: "fw1", defaultPassword: "pw",
spec: { mgmtAddress: "192.168.8.2/23" },
});
expect(staticSpec.reportAddress).toBe("192.168.8.2");
const dhcpSpec = buildVyosConfigSpec({ hostname: "fw1", defaultPassword: "pw" });
expect(dhcpSpec.reportAddress).toBe("");
});
it("defaults freshConfig off (reinstall preserves the on-disk config)", () => {
expect(buildVyosConfigSpec({ hostname: "fw1", defaultPassword: "pw" }).freshConfig).toBe(false);
expect(buildVyosConfigSpec({
hostname: "fw1", defaultPassword: "pw", spec: { freshConfig: true },
}).freshConfig).toBe(true);
});
it("driver streams logs to /api/log and reports 'ready at' on completion", async () => {
const testDir = join(tmpdir(), `bastion-vyos-parity-${Date.now()}`);
mkdirSync(join(testDir, "http"), { recursive: true });
mkdirSync(join(testDir, "tftp"), { recursive: true });
const { app: parityApp, state: parityState } = createApp(createTestConfig(testDir));
try {
parityState.update((s) => {
s.install_queue["aa:bb:cc:44:55:66"] = {
hostname: "fw9", disk: "/dev/vda", role: "vanilla",
os: "vyos-rolling", queued_at: new Date().toISOString(),
};
});
const response = await parityApp.inject({
method: "GET", url: "/vyos/install.py?mac=aa:bb:cc:44:55:66",
});
expect(response.body).toContain("/api/log");
expect(response.body).toContain('"lines": batch');
expect(response.body).toContain('report("complete", "ready at %s"');
expect(response.body).toContain("ensure_network_boot_first");
expect(response.body).toContain("lab-provisioned");
expect(response.body).toContain('ROLE = "vanilla"');
} finally {
await parityApp.close();
rmSync(testDir, { recursive: true, force: true });
}
});
it("complete with 'ready at' records installed.ip for a vyos machine", async () => {
const testDir = join(tmpdir(), `bastion-vyos-complete-${Date.now()}`);
mkdirSync(join(testDir, "http"), { recursive: true });
mkdirSync(join(testDir, "tftp"), { recursive: true });
const { app: cApp, state: cState } = createApp(createTestConfig(testDir));
try {
const mac2 = "aa:bb:cc:77:88:99";
cState.update((s) => {
s.install_queue[mac2] = {
hostname: "fw1", disk: "/dev/vda", role: "vanilla",
os: "vyos-rolling", queued_at: new Date().toISOString(),
};
});
const response = await cApp.inject({
method: "POST", url: "/api/progress",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ mac: mac2, stage: "complete", detail: "ready at 192.168.8.2" }),
});
expect(response.statusCode).toBe(200);
const installed = cState.load().installed[mac2];
expect(installed?.ip).toBe("192.168.8.2");
expect(installed?.os).toBe("vyos-rolling");
} finally {
await cApp.close();
rmSync(testDir, { recursive: true, force: true });
}
});
});
describe("vyos installer prompt coverage", () => {
// Every interactive prompt image_installer.py can emit, copied verbatim from
// the MSG_* constants (including the reinstall-only search_previous_installation
// ones). An unanswered prompt does not fail loudly -- the installer simply
// blocks on stdin until the driver's stall timeout, which is how the reinstall
// path silently hung for 15 minutes in the VM test.
const PROMPTS: Record<string, string> = {
continue: "Would you like to continue? [y/N] ",
imageName: "What would you like to name this image? (Default: 1.5-rolling) ",
password: 'Please enter a password for the "vyos" user: ',
passwordConfirm: 'Please confirm password for the "vyos" user: ',
console: "What console should be used by default? (K: KVM, S: Serial)? (Default: K) ",
raidConfigure: "Would you like to configure RAID-1 mirroring? [Y/n] ",
raidFoundDisks: "Would you like to configure RAID-1 mirroring on them? [Y/n] ",
raidChooseDisks: "Would you like to choose two disks for RAID-1 mirroring? [Y/n] ",
diskSelect: "Which one should be used for installation? (Default: /dev/vda) ",
diskConfirm: "Installation will delete all data on the drive. Continue? [y/N] ",
raidConfirm: "Installation will delete all data on both drives. Continue? [y/N] ",
rootSizeAll: "Would you like to use all the free space on the drive? [Y/n] ",
bootConfig: "Which file would you like as boot config? ",
copyData: "Would you like to copy data to the new image? [Y/n] ",
chooseCopyData: "From which image would you like to save config information? ",
copyEncData: "Would you like to copy the encrypted config to the new image? [Y/n] ",
chooseCopyEncData: "From which image would you like to copy the encrypted config? ",
};
it("answers every installer prompt exactly once", () => {
const { execFileSync } = require("node:child_process") as typeof import("node:child_process");
const { writeFileSync, unlinkSync, mkdtempSync } = require("node:fs") as typeof import("node:fs");
// Skip cleanly where python3 is unavailable (same spirit as the
// ksvalidator-backed kickstart test).
try {
execFileSync("python3", ["--version"], { stdio: "pipe" });
} catch {
return;
}
const spec = buildVyosConfigSpec({
hostname: "fw1", defaultPassword: "pw", disk: "/dev/vda",
});
const driver = renderVyosInstallPy({
spec, mac: "aa:bb:cc:11:22:33", serverIp: "10.0.0.1", httpPort: 8080, role: "vanilla",
});
const dir = mkdtempSync(join(tmpdir(), "vyos-rules-"));
const driverPath = join(dir, "driver.py");
const checkPath = join(dir, "check.py");
writeFileSync(driverPath, driver);
writeFileSync(checkPath, `
import importlib.util, json, sys
spec = importlib.util.spec_from_file_location("drv", ${JSON.stringify(driverPath)})
drv = importlib.util.module_from_spec(spec); spec.loader.exec_module(drv)
rules = drv.build_rules()
prompts = json.loads(sys.argv[1])
out = {}
for label, text in prompts.items():
out[label] = len([r for p, r in rules if p.search(text.encode())])
print(json.dumps(out))
`);
try {
const stdout = execFileSync("python3", [checkPath, JSON.stringify(PROMPTS)], {
encoding: "utf-8", stdio: ["pipe", "pipe", "pipe"],
});
const counts = JSON.parse(stdout) as Record<string, number>;
const unanswered = Object.entries(counts).filter(([, n]) => n !== 1);
expect(unanswered).toEqual([]);
} finally {
try { unlinkSync(driverPath); unlinkSync(checkPath); } catch { /* best effort */ }
try { rmSync(dir, { recursive: true, force: true }); } catch { /* best effort */ }
}
});
});
describe("vyos boot NIC pinning", () => {
it("pins the boot interface by MAC via BOOTIF", async () => {
// Without this, live-boot picks the first *connected* NIC. On the VP2440
// the SFP+ pair links before the copper PXE port, so live-boot tried the
// fiber ports (no DHCP), timed out 15s each, and failed with "Unable to
// find a live file system on the network".
const testDir = join(tmpdir(), `bastion-vyos-bootif-${Date.now()}`);
mkdirSync(join(testDir, "http"), { recursive: true });
mkdirSync(join(testDir, "tftp"), { recursive: true });
const { app: a, state: st } = createApp(createTestConfig(testDir));
try {
const m = "64:62:66:25:96:47";
st.update((s) => {
s.install_queue[m] = {
hostname: "vyos001", disk: "/dev/mmcblk0", role: "vanilla",
os: "vyos-rolling", queued_at: new Date().toISOString(),
};
});
const res = await a.inject({ method: "GET", url: `/dispatch?mac=${m}` });
// live-boot's Device_from_bootif() expects 01-<mac with dashes>
expect(res.body).toContain("BOOTIF=01-64-62-66-25-96-47");
// and it must be on the kernel line, before fetch= is attempted
const kernelLine = res.body.split("\n").find((l) => l.startsWith("kernel "));
expect(kernelLine).toContain("BOOTIF=01-64-62-66-25-96-47");
expect(kernelLine).toContain("fetch=");
} finally {
await a.close();
rmSync(testDir, { recursive: true, force: true });
}
});
});

View File

@@ -90,6 +90,7 @@ export class LabdClient {
async installMachine(opts: {
mac: string; hostname: string; disk?: string; role?: string; os?: string;
vyos?: import("@lab/shared").VyosInstallSpec;
}): Promise<{ status: string; data?: unknown; error?: string }> {
return this.request("POST", "/api/machines/install", { body: opts });
}
@@ -104,6 +105,16 @@ export class LabdClient {
return this.request("POST", "/api/machines/debug", { body: { mac, pxeBoot: opts?.pxeBoot } });
}
async discoverMachine(data: {
mac: string; product?: string; board?: string; serial?: string;
manufacturer?: string; cpu_model?: string; cpu_cores?: number;
memory_gb?: number; arch?: string;
disks?: Array<{ name: string; size_gb: number; model: string }>;
nics?: Array<{ name: string; mac: string; state: string }>;
}): Promise<{ status: string; error?: string }> {
return this.request("POST", "/api/machines/discover", { body: data });
}
async forgetMachine(mac: string): Promise<{ status: string }> {
return this.request("DELETE", `/api/machines/${encodeURIComponent(mac)}`);
}

View File

@@ -70,7 +70,7 @@ export function registerAppCommand(program: Command): void {
.command("install <target>")
.description("Install k3s on a target machine (hostname, IP, or MAC)")
.option("--role <role>", "k3s role: infra (server) or worker (agent)", "infra")
.option("--user <user>", "SSH user", "michal")
.option("--user <user>", "SSH user", "root")
.option("--k3s-server <url>", "k3s server URL (required for worker role)")
.option("--k3s-token <token>", "k3s join token (required for worker role)")
.action(async (target: string, opts: {
@@ -164,7 +164,7 @@ export function registerAppCommand(program: Command): void {
k3sCmd
.command("health [target]")
.description("Check k3s health (all hosts if no target given)")
.option("--user <user>", "SSH user", "michal")
.option("--user <user>", "SSH user", "root")
.action(async (target: string | undefined, opts: { user: string }) => {
const sshKey = findSshKey();
@@ -304,7 +304,7 @@ export function registerAppCommand(program: Command): void {
k3sCmd
.command("list")
.description("List installed machines and their k3s status")
.option("--user <user>", "SSH user", "michal")
.option("--user <user>", "SSH user", "root")
.action(async (opts: { user: string }) => {
let state: BastionState;
try {

View File

@@ -0,0 +1,69 @@
// CLI command: provision asahi
// Prints the curl command to run on the Mac Studio (macOS) to install
// Fedora Asahi Remix with lab LVM layout.
import type { Command } from "commander";
import { getLabdClient } from "../api/config.js";
export function registerAsahiCommand(parent: Command): void {
parent
.command("asahi")
.description("Show instructions to provision an Apple Silicon Mac with Asahi Linux")
.action(async () => {
// Try to get bastion info to determine the correct URL
let bastionUrl = "";
try {
const bastions = await getLabdClient().getBastions();
const online = bastions.find(b => b.status === "online");
if (online) {
bastionUrl = `http://${online.serverIp}:8080`;
}
} catch { /* labd not reachable */ }
if (!bastionUrl) {
// Fall back to config
const { loadConfig } = await import("../config/index.js");
const config = loadConfig();
bastionUrl = config.labdUrl ?? "http://<bastion-ip>:8080";
// Convert labd URL to bastion URL (labd is on different port/host)
bastionUrl = bastionUrl.replace(/:\d+$/, ":8080");
}
const BOLD = "\x1b[1m";
const CYAN = "\x1b[36m";
const DIM = "\x1b[2m";
const RESET = "\x1b[0m";
console.log("");
console.log(`${BOLD} Asahi Linux Provisioning${RESET}`);
console.log(`${DIM} For Apple Silicon Macs (Mac Studio, MacBook, etc.)${RESET}`);
console.log("");
console.log(` Run this command ${BOLD}on the Mac${RESET} (from macOS Terminal):`);
console.log("");
console.log(` ${CYAN}${BOLD}curl ${bastionUrl}/asahi | sh${RESET}`);
console.log("");
console.log(` The installer will ask a few interactive questions:`);
console.log(` ${BOLD}1.${RESET} Action: press ${BOLD}r${RESET} to resize macOS`);
console.log(` ${BOLD}2.${RESET} How much space for Linux: choose maximum`);
console.log(` ${BOLD}3.${RESET} Confirm the resize operation`);
console.log(` ${BOLD}4.${RESET} macOS password for firmware authentication`);
console.log("");
console.log(` After that, everything is automatic:`);
console.log(` - Asahi boot infrastructure (m1n1 + U-Boot)`);
console.log(` - Fedora Asahi Remix root partition`);
console.log(` - LVM data partition (remaining space)`);
console.log("");
console.log(` On first boot, LVM volumes are created automatically:`);
console.log(` ${DIM}labvg/swap (27GB), labvg/var (100GB), labvg/varlog (10GB),`);
console.log(` labvg/home (10GB), labvg/srv (20GB), labvg/rancher (20GB),`);
console.log(` labvg/longhorn (remaining space)${RESET}`);
console.log("");
console.log(` After first boot, SSH in and run the firstboot script:`);
console.log(` ${BOLD}ssh root@<ip> 'curl -sf ${bastionUrl}/asahi/firstboot.sh | bash'${RESET}`);
console.log("");
console.log(` This sets up LVM, detects hostname/MAC, and self-registers.`);
console.log(` Then install k3s:`);
console.log(` ${BOLD}labctl app k3s install <hostname> --role infra${RESET}`);
console.log("");
});
}

View File

@@ -1,10 +1,61 @@
// CLI command: provision install
// Queue a discovered machine for OS installation via labd.
import { Command, Option } from "commander";
import { readFileSync } from "node:fs";
import { Command, Option, InvalidArgumentError } from "commander";
import { isValidOsId, SUPPORTED_OS, SUPPORTED_ROLES, ROLE_REGISTRY } from "@lab/shared";
import type { VyosBundle, VyosInstallSpec, VyosVlanSpec } from "@lab/shared";
import { getLabdClient } from "../api/config.js";
/**
* Load one router's config out of a Pulumi-rendered bundle.
*
* The bundle is produced by `kubernetes-deployment` (npm run vyos:bundle) and
* holds every router it manages, keyed by name. Selecting by hostname here is
* what keeps bring-up and `pulumi up` describing the same box: labctl replays
* the declared config rather than deriving its own.
*/
export function loadVyosBundle(path: string, hostname: string): VyosBundle {
let parsed: { version?: number; routers?: Record<string, VyosBundle> };
try {
parsed = JSON.parse(readFileSync(path, "utf8"));
} catch (e) {
throw new InvalidArgumentError(`Cannot read VyOS bundle ${path}: ${(e as Error).message}`);
}
if (parsed.version !== 1) {
throw new InvalidArgumentError(
`VyOS bundle ${path} has version ${parsed.version ?? "<none>"}; this labctl understands 1`,
);
}
const router = parsed.routers?.[hostname];
if (router === undefined) {
const known = Object.keys(parsed.routers ?? {}).join(", ") || "<none>";
throw new InvalidArgumentError(
`VyOS bundle ${path} has no entry for "${hostname}" (has: ${known})`,
);
}
return router;
}
/** Parse a repeated --vlan flag: "<id>:<cidr>[:<description>]". */
export function parseVlan(value: string, previous: VyosVlanSpec[] = []): VyosVlanSpec[] {
const parts = value.split(":");
const id = Number(parts[0]);
const address = parts[1] ?? "";
// InvalidArgumentError makes commander print a clean message instead of
// dumping a stack trace at the operator.
if (!Number.isInteger(id) || id < 1 || id > 4094) {
throw new InvalidArgumentError(`Invalid VLAN id in "${value}" (expected 1-4094)`);
}
if (!address.includes("/")) {
throw new InvalidArgumentError(
`Invalid VLAN address in "${value}" (expected CIDR, e.g. 10.0.10.1/24)`,
);
}
const description = parts.slice(2).join(":");
return [...previous, { id, address, ...(description ? { description } : {}) }];
}
function roleTable(): string {
const lines: string[] = ["", "Available roles:"];
for (const r of ROLE_REGISTRY) {
@@ -15,6 +66,38 @@ function roleTable(): string {
return lines.join("\n");
}
/** Parse a repeated --vlan-vip flag: "<id>:<cidr>" — VRRP VIP for a --vlan entry. */
export function parseVlanVip(
value: string,
previous: Record<number, string> = {},
): Record<number, string> {
const index = value.indexOf(":");
const id = Number(index === -1 ? Number.NaN : value.slice(0, index));
const cidr = index === -1 ? "" : value.slice(index + 1).trim();
if (!Number.isInteger(id) || id < 1 || id > 4094 || !cidr.includes("/")) {
throw new InvalidArgumentError(
`Invalid VLAN VIP "${value}" (expected <id>:<cidr>, e.g. 3:192.168.3.254/24)`,
);
}
return { ...previous, [id]: cidr };
}
/** Parse a repeated --vyos-hwid flag: "<iface>=<mac>". */
export function parseHwId(
value: string,
previous: Record<string, string> = {},
): Record<string, string> {
const index = value.indexOf("=");
const iface = index === -1 ? "" : value.slice(0, index).trim();
const mac = index === -1 ? "" : value.slice(index + 1).trim().toLowerCase();
if (iface === "" || !/^([0-9a-f]{2}:){5}[0-9a-f]{2}$/.test(mac)) {
throw new InvalidArgumentError(
`Invalid hw-id "${value}" (expected <iface>=<mac>, e.g. eth2=64:62:66:25:96:47)`,
);
}
return { ...previous, [iface]: mac };
}
export function registerInstallCommand(parent: Command): void {
parent
.command("install <mac> <hostname>")
@@ -24,10 +107,51 @@ export function registerInstallCommand(parent: Command): void {
.addOption(new Option("--role <role>", "Machine role (see below)").choices([...SUPPORTED_ROLES]).default("worker"))
.addOption(new Option("--os <os>", "Operating system").choices([...SUPPORTED_OS]).default("fedora-43"))
.option("--disk <device>", "Target disk device (auto-detect if omitted)")
.option("--vyos-mgmt <iface>", "VyOS: untagged interface the machine PXE boots from (default eth0)")
.option("--vyos-mgmt-address <addr>", "VyOS: CIDR for the management interface, or 'dhcp' (default dhcp)")
.option("--vyos-bond <ifaces>", "VyOS: comma-separated LACP bond members (must exclude the PXE NIC)")
.option("--vyos-bond-address <cidr>", "VyOS: address on the untagged bond (trunk native VLAN)")
.option("--vyos-bond-vrrp <cidr>", "VyOS: VRRP VIP floated on the untagged bond")
.option("--vlan-vip <id:cidr>", "VyOS: VRRP VIP for a --vlan entry (repeatable)", parseVlanVip)
.option("--vyos-vrrp-priority <n>", "VyOS: VRRP priority for all groups on this box (higher = master)")
.option("--vyos-mgmt-vlan <id:cidr[:desc]>", "VyOS: tagged management VLAN on the PXE port")
.option("--vlan <id:cidr[:desc]>", "VyOS: tagged VLAN sub-interface on the bond (repeatable)", parseVlan)
.option("--vyos-password <password>", "VyOS: password for the 'vyos' user")
.option("--vyos-hwid <iface=mac>", "VyOS: pin an interface name to a MAC via hw-id (repeatable)", parseHwId)
.option("--vyos-fresh-config", "VyOS: on reinstall, overwrite the preserved config with the generated one")
.option(
"--vyos-bundle <path>",
"VyOS: apply a Pulumi-rendered bundle verbatim (kubernetes-deployment/infra/vyos/vyos-bundle.json). " +
"Replaces the derived --vyos-bond/--vlan/... config; secret values are left unset for `pulumi up`.",
)
.option(
"--vyos-api-key <key>",
"VyOS: enable the HTTP API with this key so Pulumi can manage the router from first boot",
)
.option(
"--vyos-api-listen <addr>",
"VyOS: address the HTTP API binds to (default: the static management address). " +
"Required when management is DHCP; the API is never bound to all interfaces.",
)
.action(async (mac: string, hostname: string, opts: {
role: string;
os: string;
disk?: string;
vyosMgmt?: string;
vyosMgmtAddress?: string;
vyosBond?: string;
vyosBondAddress?: string;
vyosBondVrrp?: string;
vlan?: VyosVlanSpec[];
vlanVip?: Record<number, string>;
vyosVrrpPriority?: string;
vyosMgmtVlan?: string;
vyosPassword?: string;
vyosHwid?: Record<string, string>;
vyosFreshConfig?: boolean;
vyosBundle?: string;
vyosApiKey?: string;
vyosApiListen?: string;
}) => {
if (!isValidOsId(opts.os)) {
console.error(`Unknown OS: ${opts.os}. Supported: ${SUPPORTED_OS.join(", ")}`);
@@ -39,6 +163,89 @@ export function registerInstallCommand(parent: Command): void {
process.exit(1);
}
const bondMembers = opts.vyosBond !== undefined && opts.vyosBond !== ""
? opts.vyosBond.split(",").map((s) => s.trim()).filter((s) => s.length > 0)
: [];
// Attach --vlan-vip entries to their --vlan definitions. A VIP for a VLAN
// that was never defined is a typo that would otherwise vanish silently.
const vips = opts.vlanVip ?? {};
const vlans = (opts.vlan ?? []).map((v) =>
vips[v.id] !== undefined ? { ...v, vrrp: vips[v.id] as string } : v,
);
for (const id of Object.keys(vips)) {
if (!vlans.some((v) => String(v.id) === id)) {
console.error(`--vlan-vip ${id}:... has no matching --vlan ${id}:... entry`);
process.exit(1);
}
}
const vrrpPriority = opts.vyosVrrpPriority !== undefined && opts.vyosVrrpPriority !== ""
? Number(opts.vyosVrrpPriority)
: undefined;
if (vrrpPriority !== undefined
&& (!Number.isInteger(vrrpPriority) || vrrpPriority < 1 || vrrpPriority > 255)) {
console.error(`--vyos-vrrp-priority must be an integer 1-255 (got ${opts.vyosVrrpPriority})`);
process.exit(1);
}
const vyos: VyosInstallSpec = {
...(opts.vyosMgmt !== undefined && opts.vyosMgmt !== ""
? { mgmtInterface: opts.vyosMgmt } : {}),
...(opts.vyosMgmtAddress !== undefined && opts.vyosMgmtAddress !== ""
? { mgmtAddress: opts.vyosMgmtAddress } : {}),
...(bondMembers.length > 0 ? { bondMembers } : {}),
...(opts.vyosBondAddress !== undefined && opts.vyosBondAddress !== ""
? { bondAddress: opts.vyosBondAddress } : {}),
...(opts.vyosBondVrrp !== undefined && opts.vyosBondVrrp !== ""
? { bondVrrp: opts.vyosBondVrrp } : {}),
...(vrrpPriority !== undefined ? { vrrpPriority } : {}),
...(vlans.length > 0 ? { vlans } : {}),
...(opts.vyosPassword !== undefined && opts.vyosPassword !== ""
? { password: opts.vyosPassword } : {}),
...(opts.vyosHwid !== undefined && Object.keys(opts.vyosHwid).length > 0
? { hwIds: opts.vyosHwid } : {}),
...(opts.vyosMgmtVlan !== undefined && opts.vyosMgmtVlan !== ""
? { mgmtVlan: parseVlan(opts.vyosMgmtVlan)[0] as VyosVlanSpec } : {}),
...(opts.vyosFreshConfig === true ? { freshConfig: true } : {}),
...(opts.vyosBundle !== undefined && opts.vyosBundle !== ""
? { bundle: loadVyosBundle(opts.vyosBundle, hostname) } : {}),
...(opts.vyosApiKey !== undefined && opts.vyosApiKey !== ""
? { apiKey: opts.vyosApiKey } : {}),
...(opts.vyosApiListen !== undefined && opts.vyosApiListen !== ""
? { apiListenAddress: opts.vyosApiListen } : {}),
};
const hasVyosOptions = Object.keys(vyos).length > 0;
// A bundle already describes the whole router. Accepting derived topology
// flags alongside it would silently discard them (the bundle wins in
// buildVyosConfigSpec), so say so rather than appear to honour both.
if (vyos.bundle !== undefined) {
const derived = ["mgmtInterface", "mgmtAddress", "bondMembers", "bondAddress",
"bondVrrp", "vrrpPriority", "vlans", "mgmtVlan"] as const;
const conflicting = derived.filter((k) => vyos[k] !== undefined);
if (conflicting.length > 0) {
console.error(
`--vyos-bundle describes the whole router; these would be ignored: ${conflicting.join(", ")}`,
);
console.error("Remove them, or change the bundle in kubernetes-deployment and re-render.");
process.exit(1);
}
}
if (hasVyosOptions && !opts.os.startsWith("vyos")) {
console.error(`VyOS options require --os vyos-rolling (got --os ${opts.os})`);
process.exit(1);
}
// Firmware PXE cannot run over LACP, so the NIC that boots the installer
// must stay out of the bond — otherwise the next reinstall has no path in.
const mgmt = vyos.mgmtInterface ?? "eth0";
if (bondMembers.includes(mgmt)) {
console.error(`--vyos-bond must not include the PXE/management interface "${mgmt}"`);
console.error("PXE cannot boot over an LACP bond; keep that NIC unbonded.");
process.exit(1);
}
try {
const result = await getLabdClient().installMachine({
mac,
@@ -46,11 +253,14 @@ export function registerInstallCommand(parent: Command): void {
role: opts.role,
os: opts.os,
...(opts.disk ? { disk: opts.disk } : {}),
...(hasVyosOptions ? { vyos } : {}),
});
console.log(JSON.stringify(result, null, 2));
console.log("");
const osLabel = opts.os.startsWith("ubuntu") ? "Ubuntu" : "Fedora";
const osLabel = opts.os.startsWith("ubuntu")
? "Ubuntu"
: opts.os.startsWith("vyos") ? "VyOS" : "Fedora";
console.log(`Power on the machine to start ${osLabel} installation.`);
const roleInfo = ROLE_REGISTRY.find(r => r.name === opts.role);

View File

@@ -38,7 +38,7 @@ export function registerLabcontrollerCommands(appCmd: Command): void {
lcCmd
.command("deploy <target>")
.description("Deploy labcontroller stack to a k3s node")
.option("--user <user>", "SSH user", "michal")
.option("--user <user>", "SSH user", "root")
.option("--crdb-replicas <n>", "CockroachDB replicas", "1")
.action(async (target: string, opts: {
user: string;
@@ -193,7 +193,7 @@ export function registerLabcontrollerCommands(appCmd: Command): void {
lcCmd
.command("status [target]")
.description("Check labcontroller deployment status (all hosts if no target)")
.option("--user <user>", "SSH user", "michal")
.option("--user <user>", "SSH user", "root")
.action(async (target: string | undefined, opts: { user: string }) => {
const sshKey = findSshKey();
const sshOpts = sshKey ? { keyPath: sshKey } : {};

View File

@@ -69,10 +69,10 @@ export function registerListCommand(parent: Command): void {
const hostname = inst?.hostname ?? queued?.hostname ?? "-";
const role = inst?.role ?? queued?.role ?? "-";
const ip = inst?.ip ?? "-";
const cpu = hw?.cpu_model ?? "-";
const cores = hw?.cpu_cores != null ? String(hw.cpu_cores) : "-";
const ram = hw?.memory_gb != null ? `${hw.memory_gb}GB` : "-";
const product = hw?.product ?? "-";
const cpu = hw?.cpu_model ?? inst?.cpu_model ?? "-";
const cores = (hw?.cpu_cores ?? inst?.cpu_cores) != null ? String(hw?.cpu_cores ?? inst?.cpu_cores) : "-";
const ram = (hw?.memory_gb ?? inst?.memory_gb) != null ? `${hw?.memory_gb ?? inst?.memory_gb}GB` : "-";
const product = hw?.product ?? inst?.product ?? "-";
const color = statusColor(status);

View File

@@ -0,0 +1,97 @@
// CLI command: provision recheck
// SSH into all installed machines, collect hardware info, update bastion state.
import type { Command } from "commander";
import { sshExec } from "@lab/modules";
import { getLabdClient } from "../api/config.js";
const BOLD = "\x1b[1m";
const GREEN = "\x1b[0;32m";
const RED = "\x1b[0;31m";
const DIM = "\x1b[2m";
const RESET = "\x1b[0m";
const SSH_OPTS = { timeoutMs: 30_000 };
// Shell script that collects hardware info as JSON.
// Kept simple — no Python, pure shell + awk.
const HW_COLLECT_SCRIPT = [
'P=$(cat /sys/class/dmi/id/product_name 2>/dev/null || echo unknown)',
'B=$(cat /sys/class/dmi/id/board_name 2>/dev/null || echo unknown)',
'S=$(cat /sys/class/dmi/id/product_serial 2>/dev/null || echo unknown)',
'M=$(cat /sys/class/dmi/id/sys_vendor 2>/dev/null || echo unknown)',
'C=$(grep -m1 "model name" /proc/cpuinfo 2>/dev/null | cut -d: -f2 | sed "s/^ //" || grep -m1 Model /proc/cpuinfo 2>/dev/null | cut -d: -f2 | sed "s/^ //" || echo unknown)',
'N=$(grep -c "^processor" /proc/cpuinfo 2>/dev/null || echo 0)',
'R=$(awk "/MemTotal/ {printf \\"%d\\", \\$2/1024/1024}" /proc/meminfo 2>/dev/null || echo 0)',
'A=$(uname -m)',
'printf \'{"product":"%s","board":"%s","serial":"%s","manufacturer":"%s","cpu_model":"%s","cpu_cores":%s,"memory_gb":%s,"arch":"%s"}\\n\' "$P" "$B" "$S" "$M" "$C" "$N" "$R" "$A"',
].join("; ");
export function registerRecheckCommand(parent: Command): void {
parent
.command("recheck")
.description("Refresh hardware info for all installed machines via SSH")
.option("--user <user>", "SSH user", "root")
.option("--target <hostname>", "Only recheck a specific machine (by hostname or MAC)")
.action(async (opts: { user: string; target?: string }) => {
const client = getLabdClient();
let state;
try {
state = await client.getMachines();
} catch (err) {
console.error(`Cannot reach labd: ${err instanceof Error ? err.message : String(err)}`);
process.exit(1);
}
// Build list of machines to check
const targets: Array<{ mac: string; hostname: string; ip: string; sshUser: string }> = [];
const userIsDefault = opts.user === "root";
for (const [mac, info] of Object.entries(state.installed)) {
if (!info.ip) continue;
if (opts.target && info.hostname !== opts.target && mac !== opts.target) continue;
// VyOS boxes only have the "vyos" login; honor an explicit --user.
const sshUser = userIsDefault && (info.os ?? "").startsWith("vyos") ? "vyos" : opts.user;
targets.push({ mac, hostname: info.hostname, ip: info.ip, sshUser });
}
if (targets.length === 0) {
console.log("No installed machines with IPs to check.");
return;
}
console.log(`\n${BOLD}Rechecking ${targets.length} machine(s)...${RESET}\n`);
let updated = 0;
let failed = 0;
for (const { mac, hostname, ip, sshUser } of targets) {
process.stdout.write(` ${hostname.padEnd(24)} ${DIM}(${ip})${RESET} `);
try {
const t0 = Date.now();
const result = await sshExec(ip, sshUser, HW_COLLECT_SCRIPT, SSH_OPTS);
const elapsed = Date.now() - t0;
if (result.exitCode !== 0) {
console.log(`${RED}SSH failed (exit ${result.exitCode}, ${elapsed}ms)${RESET}`);
if (result.stderr) console.log(` ${DIM}${result.stderr.substring(0, 200)}${RESET}`);
console.log(`${RED}SSH failed (exit ${result.exitCode})${RESET}`);
failed++;
continue;
}
const hwData = JSON.parse(result.stdout.trim());
await client.discoverMachine({ mac, ...hwData });
const cpu = hwData.cpu_model || "?";
const cores = hwData.cpu_cores || "?";
const mem = hwData.memory_gb || "?";
console.log(`${GREEN}OK${RESET} ${DIM}${cpu}, ${cores} cores, ${mem}GB${RESET}`);
updated++;
} catch (err) {
console.log(`${RED}FAIL${RESET} ${DIM}${err instanceof Error ? err.message : String(err)}${RESET}`);
failed++;
}
}
console.log(`\n${BOLD}Done:${RESET} ${updated} updated, ${failed} failed\n`);
});
}

View File

@@ -24,12 +24,12 @@ function roleTable(): string {
function resolveTarget(
target: string,
state: BastionState,
): { mac: string; hostname: string; ip: string } | null {
): { mac: string; hostname: string; ip: string; os?: string } | null {
const normalized = target.toLowerCase().replace(/-/g, ":");
if (state.installed[normalized]) {
const info = state.installed[normalized];
return { mac: normalized, hostname: info.hostname, ip: info.ip };
return { mac: normalized, hostname: info.hostname, ip: info.ip, ...(info.os !== undefined ? { os: info.os } : {}) };
}
if (state.discovered[normalized]) {
@@ -38,13 +38,13 @@ function resolveTarget(
for (const [mac, info] of Object.entries(state.installed)) {
if (info.hostname === target || info.hostname.startsWith(target + ".")) {
return { mac, hostname: info.hostname, ip: info.ip };
return { mac, hostname: info.hostname, ip: info.ip, ...(info.os !== undefined ? { os: info.os } : {}) };
}
}
for (const [mac, info] of Object.entries(state.installed)) {
if (info.ip === target) {
return { mac, hostname: info.hostname, ip: info.ip };
return { mac, hostname: info.hostname, ip: info.ip, ...(info.os !== undefined ? { os: info.os } : {}) };
}
}
@@ -60,10 +60,12 @@ export function registerReprovisionCommand(parent: Command): void {
.addOption(new Option("--role <role>", "Machine role (see below)").choices([...SUPPORTED_ROLES]).default("worker"))
.addOption(new Option("--os <os>", "Operating system").choices([...SUPPORTED_OS]).default("fedora-43"))
.option("--disk <device>", "Target disk device (auto-detect if omitted)")
.option("--user <user>", "SSH user for the reboot (default: vyos for VyOS machines, else current user)")
.action(async (target: string, hostnameOverride: string | undefined, opts: {
role: string;
os: string;
disk?: string;
user?: string;
}) => {
if (!isValidOsId(opts.os)) {
console.error(`Unknown OS: ${opts.os}. Supported: ${SUPPORTED_OS.join(", ")}`);
@@ -123,7 +125,11 @@ export function registerReprovisionCommand(parent: Command): void {
return;
}
const adminUser = process.env["SUDO_USER"] ?? process.env["USER"] ?? "";
// SSH user: explicit flag > the machine's current OS (VyOS boxes only
// have the "vyos" login) > the invoking user.
const currentOsIsVyos = (resolved.os ?? "").startsWith("vyos");
const adminUser = opts.user
?? (currentOsIsVyos ? "vyos" : (process.env["SUDO_USER"] ?? process.env["USER"] ?? ""));
const effectiveUser = adminUser === "root" ? "" : adminUser;
if (effectiveUser === "") {

View File

@@ -11,7 +11,7 @@ export function registerStartCommand(parent: Command): void {
.command("start")
.description("Start the bastion server (HTTP + dnsmasq PXE)")
.option("--port <port>", "HTTP port", "8080")
.option("--dir <dir>", "Bastion data directory", "/tmp/lab-bastion")
.option("--dir <dir>", "Bastion data directory", process.env["BASTION_DIR"] ?? "/tmp/lab-bastion")
.option("--domain <domain>", "Internal domain for hostnames", "ad.itaz.eu")
.option("--dhcp-mode <mode>", "DHCP mode: proxy or full", "proxy")
.option("--fedora <version>", "Fedora version", "43")

View File

@@ -8,7 +8,7 @@ export function registerStopCommand(parent: Command): void {
parent
.command("stop")
.description("Stop a running bastion server")
.option("--dir <dir>", "Bastion data directory", "/tmp/lab-bastion")
.option("--dir <dir>", "Bastion data directory", process.env["BASTION_DIR"] ?? "/tmp/lab-bastion")
.action((opts: { dir: string }) => {
const pidFile = `${opts.dir}/bastion.pid`;

View File

@@ -17,8 +17,10 @@ import { registerReprovisionCommand } from "./commands/reprovision.js";
import { registerDebugCommand } from "./commands/debug.js";
import { registerForgetCommand } from "./commands/forget.js";
import { registerRegisterCommand } from "./commands/register.js";
import { registerAsahiCommand } from "./commands/asahi.js";
import { registerLogsCommand } from "./commands/logs.js";
import { registerMakeIsoCommand } from "./commands/makeiso.js";
import { registerRecheckCommand } from "./commands/recheck.js";
import { registerConfigCommand } from "./commands/config.js";
import { registerLoginCommand } from "./commands/login.js";
import { registerDoctorCommand } from "./commands/doctor.js";
@@ -100,8 +102,10 @@ export function createProgram(): Command {
registerDebugCommand(provisionCmd);
registerForgetCommand(provisionCmd);
registerRegisterCommand(provisionCmd);
registerAsahiCommand(provisionCmd);
registerLogsCommand(provisionCmd);
registerMakeIsoCommand(provisionCmd);
registerRecheckCommand(provisionCmd);
// config list/get/set/path
registerConfigCommand(program);

View File

@@ -0,0 +1,35 @@
// Tests for VyOS install option parsing.
import { describe, it, expect } from "vitest";
import { parseVlan } from "../src/commands/install.js";
describe("parseVlan", () => {
it("parses id and CIDR", () => {
expect(parseVlan("10:10.0.10.1/24")).toEqual([{ id: 10, address: "10.0.10.1/24" }]);
});
it("accumulates across repeated flags", () => {
const first = parseVlan("10:10.0.10.1/24");
const both = parseVlan("20:10.0.20.1/24", first);
expect(both).toHaveLength(2);
expect(both[1]).toEqual({ id: 20, address: "10.0.20.1/24" });
});
it("keeps a description, including one containing colons", () => {
expect(parseVlan("30:10.0.30.1/24:mgmt:secondary")).toEqual([
{ id: 30, address: "10.0.30.1/24", description: "mgmt:secondary" },
]);
});
it("rejects an address that is not CIDR", () => {
// A bare address would produce a VyOS config that fails to commit on first
// boot, long after the operator has stopped watching.
expect(() => parseVlan("10:10.0.10.1")).toThrow(/CIDR/);
});
it("rejects out-of-range and non-numeric VLAN ids", () => {
expect(() => parseVlan("0:10.0.10.1/24")).toThrow(/1-4094/);
expect(() => parseVlan("4095:10.0.10.1/24")).toThrow(/1-4094/);
expect(() => parseVlan("abc:10.0.10.1/24")).toThrow(/1-4094/);
});
});

View File

@@ -137,7 +137,7 @@ describe("bastion smoke tests", () => {
// Wait for the server to start (look for the banner)
const startedAt = Date.now();
const maxWait = 10_000;
const maxWait = 15_000;
while (Date.now() - startedAt < maxWait) {
if (stdout.includes("Waiting for PXE boot requests")) break;
await sleep(200);

View File

@@ -0,0 +1,23 @@
{
"name": "@lab/core",
"version": "0.1.0",
"private": true,
"type": "module",
"main": "./dist/index.js",
"types": "./dist/index.d.ts",
"exports": {
".": {
"import": "./dist/index.js",
"types": "./dist/index.d.ts"
}
},
"scripts": {
"build": "tsc --build",
"clean": "rimraf dist",
"test": "vitest",
"test:run": "vitest run"
},
"dependencies": {
"@pulumi/pulumi": "^3.0.0"
}
}

View File

@@ -0,0 +1,75 @@
// Audit event types for the labctl platform.
// Every mutation is tracked with correlation IDs for causal chains.
export type AuditEventKind =
| "resource_created"
| "resource_updated"
| "resource_deleted"
| "resource_state_change"
| "plan_generated"
| "apply_started"
| "apply_step"
| "apply_completed"
| "driver_translate"
| "driver_execute"
| "driver_error"
| "fleet_discovery"
| "fleet_classification"
| "fleet_approval"
| "fleet_auto_approve"
| "pipeline_started"
| "pipeline_step_started"
| "pipeline_step_completed"
| "pipeline_completed"
| "deploy_started"
| "deploy_completed"
| "deploy_failed"
| "drift_detected"
| "drift_corrected"
| "sync_triggered"
| "sync_completed"
| "auth_login"
| "auth_logout"
| "auth_bootstrap"
| "rbac_decision"
| "impersonation"
| "server_started"
| "controller_started"
| "agent_connected"
| "agent_disconnected"
| "bastion_registered";
export type AuditSource =
| "cli"
| "labd"
| "agent"
| "driver"
| "fleet-controller"
| "sync-controller";
export type AuditResult = "success" | "failure" | "denied" | "skipped";
export interface AuditEvent {
id: string;
timestamp: Date;
eventKind: AuditEventKind;
source: AuditSource;
verified: boolean;
userId?: string;
userName?: string;
sessionId?: string;
environmentName?: string;
accountName?: string;
resourceKind?: string;
resourceName?: string;
correlationId: string;
parentEventId?: string;
details: Record<string, unknown>;
result: AuditResult;
error?: string;
durationMs?: number;
}

View File

@@ -0,0 +1,50 @@
// Auth types for the labctl platform.
// Bearer token auth for CLI/SDK. mTLS stays for agent/bastion.
export type UserRole = "USER" | "ADMIN";
export interface User {
id: string;
email: string;
name?: string;
role: UserRole;
createdAt: Date;
}
export interface Session {
id: string;
userId: string;
token: string;
expiresAt: Date;
createdAt: Date;
}
export interface Group {
id: string;
name: string;
description?: string;
}
export type SubjectKind = "User" | "Group" | "ServiceAccount";
export interface RoleBinding {
role: "view" | "edit" | "create" | "delete" | "run" | "admin";
resource: string;
name?: string;
environment?: string;
action?: string;
}
export interface RbacSubject {
kind: SubjectKind;
name: string;
}
export interface RbacDefinition {
id: string;
name: string;
subjects: RbacSubject[];
roleBindings: RoleBinding[];
createdAt: Date;
updatedAt: Date;
}

View File

@@ -0,0 +1,24 @@
// Environment and Account types.
// An Environment is a logical boundary (production, staging, dev).
// An Account is a configured driver instance with credentials.
export interface Environment {
id: string;
name: string;
status: "active" | "archived";
createdAt: Date;
}
export interface Account {
id: string;
name: string;
driver: string;
config: Record<string, unknown>;
createdAt: Date;
}
export interface Binding {
id: string;
environmentId: string;
accountId: string;
}

View File

@@ -0,0 +1,9 @@
// @lab/core — foundation types for the labctl platform.
// Phase 1 stub: resource types, auth types, audit types, Output<T>.
// Phase 5 adds: CompositeResource, evaluator integration, full SDK.
export * from "./resource.js";
export * from "./environment.js";
export * from "./audit.js";
export * from "./auth.js";
export { Output, output, all, interpolate, secret } from "./output.js";

View File

@@ -0,0 +1,5 @@
// Re-export Pulumi's Output<T> type for use across the platform.
// Cloud drivers use this for future values (endpoints, IPs, kubeconfigs).
// Phase 1: type re-export only. Phase 5 adds full evaluator integration.
export { Output, output, all, interpolate, secret } from "@pulumi/pulumi";

View File

@@ -0,0 +1,83 @@
// Core resource types for the labctl platform.
// Every managed thing (Server, Database, App, Cluster) is a Resource.
export type ResourceOrigin = "file" | "cli" | "fleet" | "imported";
export type ResourceManagedBy = "gitops" | "manual" | "auto";
export type ResourceStatus =
| "pending"
| "creating"
| "ready"
| "updating"
| "deleting"
| "error"
| "unknown";
export interface ResourceMetadata {
kind: string;
name: string;
environmentId: string;
accountId: string;
origin: ResourceOrigin;
managedBy: ResourceManagedBy;
sourceRef?: string;
}
export interface ResourceState {
status: ResourceStatus;
message?: string;
lastReconciled?: Date;
platformRef?: string;
}
export interface Resource<TSpec = Record<string, unknown>> {
id: string;
metadata: ResourceMetadata;
desiredSpec: TSpec;
actualSpec?: TSpec;
state: ResourceState;
createdAt: Date;
updatedAt: Date;
}
// Well-known resource kinds. Drivers register additional kinds.
export const RESOURCE_KINDS = {
SERVER: "server",
DATABASE: "database",
CACHE: "cache",
CLUSTER: "cluster",
APP: "app",
SERVICE: "service",
CRONJOB: "cronjob",
NETWORK: "network",
LOADBALANCER: "loadbalancer",
DNSZONE: "dnszone",
CERTIFICATE: "certificate",
OBJECTSTORE: "objectstore",
QUEUE: "queue",
SECRET: "secret",
FLEET: "fleet",
} as const;
export type ResourceKind = (typeof RESOURCE_KINDS)[keyof typeof RESOURCE_KINDS];
// Resource aliases for CLI (kubectl-style shortnames)
export const RESOURCE_ALIASES: Record<string, string> = {
srv: "server",
db: "database",
cl: "cluster",
svc: "service",
cj: "cronjob",
lb: "loadbalancer",
dns: "dnszone",
cert: "certificate",
os: "objectstore",
mq: "queue",
sec: "secret",
fl: "fleet",
};
export function resolveResourceKind(input: string): string {
const lower = input.toLowerCase();
return RESOURCE_ALIASES[lower] ?? lower;
}

View File

@@ -0,0 +1,8 @@
{
"extends": "../../tsconfig.base.json",
"compilerOptions": {
"rootDir": "src",
"outDir": "dist"
},
"include": ["src/**/*.ts"]
}

View File

@@ -26,8 +26,10 @@
"dependencies": {
"@fastify/rate-limit": "^10.3.0",
"@fastify/websocket": "^11.0.2",
"@lab/core": "workspace:^",
"@lab/shared": "workspace:*",
"@prisma/client": "^6.9.0",
"bcryptjs": "^3.0.3",
"fastify": "^5.3.3",
"winston": "^3.17.0",
"ws": "^8.19.0",
@@ -37,6 +39,7 @@
"seed": "tsx prisma/seed.ts"
},
"devDependencies": {
"@types/bcryptjs": "^3.0.0",
"@types/node": "^22.14.1",
"@types/ws": "^8.18.1",
"prisma": "^6.9.0",

View File

@@ -7,6 +7,225 @@ datasource db {
url = env("DATABASE_URL")
}
// ── Auth (mcpctl pattern: email/password + bearer token sessions) ──
model User {
id String @id @default(cuid())
email String @unique
password String // bcrypt
name String?
role UserRole @default(USER)
createdAt DateTime @default(now())
updatedAt DateTime @updatedAt
sessions Session[]
auditLogs AuditEvent[]
groups GroupMember[]
}
enum UserRole {
USER
ADMIN
}
model Session {
id String @id @default(cuid())
userId String
user User @relation(fields: [userId], references: [id], onDelete: Cascade)
token String @unique
expiresAt DateTime
createdAt DateTime @default(now())
@@index([userId])
@@index([token])
}
model Group {
id String @id @default(cuid())
name String @unique
description String?
createdAt DateTime @default(now())
members GroupMember[]
}
model GroupMember {
id String @id @default(cuid())
groupId String
group Group @relation(fields: [groupId], references: [id], onDelete: Cascade)
userId String
user User @relation(fields: [userId], references: [id], onDelete: Cascade)
@@unique([groupId, userId])
}
model ServiceAccount {
id String @id @default(cuid())
name String @unique
token String @unique
createdAt DateTime @default(now())
}
// ── RBAC (mcpctl pattern: named definitions with JSON subjects/bindings) ──
model RbacDefinition {
id String @id @default(cuid())
name String @unique
subjects Json // [{kind: "User"|"Group"|"ServiceAccount", name: string}]
roleBindings Json // [{role, resource, name?, environment?, action?}]
createdAt DateTime @default(now())
updatedAt DateTime @updatedAt
}
// ── Audit (mcpctl pattern: fire-and-forget with correlation IDs) ──
model AuditEvent {
id String @id @default(cuid())
timestamp DateTime @default(now())
eventKind String
source String // cli | labd | agent | driver | fleet-controller | sync-controller
verified Boolean @default(false)
userId String?
user User? @relation(fields: [userId], references: [id])
userName String?
sessionId String?
environmentName String?
accountName String?
resourceKind String?
resourceName String?
correlationId String
parentEventId String?
details Json @default("{}")
result String // success | failure | denied | skipped
error String?
durationMs Int?
@@index([correlationId])
@@index([eventKind, timestamp])
@@index([environmentName, timestamp])
@@index([resourceKind, resourceName])
@@index([userId, timestamp])
}
// ── Core infrastructure ──
model Environment {
id String @id @default(cuid())
name String @unique
status String @default("active") // active | archived
createdAt DateTime @default(now())
updatedAt DateTime @updatedAt
bindings Binding[]
resources Resource[]
}
model Account {
id String @id @default(cuid())
name String @unique
driver String // baremetal-pxe | aws | gcp | kubernetes | ovh
config Json @default("{}")
// Credentials stored in Infisical, referenced by secretPath
secretPath String?
createdAt DateTime @default(now())
updatedAt DateTime @updatedAt
bindings Binding[]
resources Resource[]
}
model Binding {
id String @id @default(cuid())
environmentId String
environment Environment @relation(fields: [environmentId], references: [id], onDelete: Cascade)
accountId String
account Account @relation(fields: [accountId], references: [id], onDelete: Cascade)
@@unique([environmentId, accountId])
}
model Resource {
id String @id @default(cuid())
kind String
name String
environmentId String
environment Environment @relation(fields: [environmentId], references: [id])
accountId String
account Account @relation(fields: [accountId], references: [id])
origin String @default("cli") // file | cli | fleet | imported
managedBy String @default("manual") // gitops | manual | auto
sourceRef String?
desiredSpec Json @default("{}")
actualSpec Json?
platformRef String?
status String @default("pending") // pending | creating | ready | updating | deleting | error
statusMessage String?
lastReconciled DateTime?
createdAt DateTime @default(now())
updatedAt DateTime @updatedAt
@@unique([kind, name, environmentId])
@@index([environmentId])
@@index([accountId])
@@index([kind, status])
}
model Secret {
id String @id @default(cuid())
name String @unique
// Encrypted data — application-layer encryption as fallback if Infisical unavailable
data Json @default("{}")
version Int @default(1)
createdAt DateTime @default(now())
updatedAt DateTime @updatedAt
}
// ── Fleet ──
model Fleet {
id String @id @default(cuid())
name String
environmentId String
accountId String
selector Json // fact-matching rules
onboardPipeline Json // step definitions
offboardPipeline Json?
approvalConfig Json?
status String @default("active")
createdAt DateTime @default(now())
updatedAt DateTime @updatedAt
members FleetMember[]
}
model FleetMember {
id String @id @default(cuid())
fleetId String
fleet Fleet @relation(fields: [fleetId], references: [id], onDelete: Cascade)
serverId String
status String // discovered | pending | onboarding | active | offboarding | removed
joinedAt DateTime @default(now())
@@index([fleetId])
}
// ── Git sources (for sync controller) ──
model GitSource {
id String @id @default(cuid())
name String @unique
repo String
branch String @default("main")
path String @default("environments/")
lastSync DateTime?
createdAt DateTime @default(now())
}
// ── Existing v1.0 models (kept for bastion/agent compatibility) ──
model Server {
id String @id @default(uuid())
hostname String @unique
@@ -17,13 +236,12 @@ model Server {
labels Json @default("{}")
ip String?
agentVersion String?
status String @default("unknown") // unknown, online, offline, provisioning
status String @default("unknown")
lastHeartbeat DateTime?
createdAt DateTime @default(now())
updatedAt DateTime @updatedAt
agent Agent?
auditLogs AuditLog[]
}
model Agent {
@@ -33,112 +251,29 @@ model Agent {
certificatePem String?
enrolledAt DateTime @default(now())
lastSeen DateTime?
facts Json? // hardware facts reported by agent
@@index([serverId])
}
model User {
id String @id @default(uuid())
username String @unique
displayName String?
certFingerprint String? @unique
createdAt DateTime @default(now())
updatedAt DateTime @updatedAt
roleBindings UserRole[]
auditLogs AuditLog[]
}
model Role {
id String @id @default(uuid())
name String @unique
description String?
createdAt DateTime @default(now())
permissions Permission[]
userBindings UserRole[]
}
model Permission {
id String @id @default(uuid())
roleId String
role Role @relation(fields: [roleId], references: [id], onDelete: Cascade)
type String @default("allow") // allow or deny
action String // read, exec, apply, destroy, manage, admin, kubectl, *
cloud String @default("*")
environment String @default("*")
server String @default("*")
@@index([roleId])
}
model UserRole {
id String @id @default(uuid())
userId String
user User @relation(fields: [userId], references: [id], onDelete: Cascade)
roleId String
role Role @relation(fields: [roleId], references: [id], onDelete: Cascade)
@@unique([userId, roleId])
@@index([userId])
@@index([roleId])
}
model JoinToken {
id String @id @default(uuid())
token String @unique
type String @default("one-time") // one-time or reusable
type String @default("one-time")
label String?
usedBy String? // server hostname that used it
usedBy String?
usedAt DateTime?
revokedAt DateTime?
createdAt DateTime @default(now())
expiresAt DateTime?
}
model AuditLog {
id String @id @default(uuid())
userId String?
user User? @relation(fields: [userId], references: [id])
serverId String?
server Server? @relation(fields: [serverId], references: [id])
sessionId String?
action String // exec, kubectl, apply, login, rbac-denied, etc.
resourceType String? // server, cluster, role, app, etc.
resourceName String?
args String? // sanitized command args
result String @default("success") // success, denied, error
durationMs Int?
sourceIp String?
timestamp DateTime @default(now())
@@index([userId])
@@index([serverId])
@@index([sessionId])
@@index([timestamp])
@@index([action])
}
model PulumiRun {
id String @id @default(uuid())
userId String
stackName String
action String // up, preview, destroy
status String @default("pending") // pending, running, succeeded, failed
output String?
startedAt DateTime @default(now())
completedAt DateTime?
@@index([userId])
@@index([stackName])
}
model Bastion {
id String @id @default(uuid())
hostname String @unique
network String
serverIp String
status String @default("offline") // online, offline
status String @default("offline")
lastHeartbeat DateTime?
createdAt DateTime @default(now())
updatedAt DateTime @updatedAt
@@ -149,7 +284,7 @@ model Cluster {
name String @unique
cloud String @default("baremetal")
environment String @default("default")
kubeconfigEnc String? // encrypted kubeconfig
kubeconfigEnc String?
labels Json @default("{}")
createdAt DateTime @default(now())
updatedAt DateTime @updatedAt

View File

@@ -0,0 +1,65 @@
// Bearer token auth middleware for Fastify.
// Validates Authorization header, resolves user identity, attaches to request.
import type { FastifyRequest, FastifyReply } from "fastify";
import type { AuthService } from "../services/auth.js";
declare module "fastify" {
interface FastifyRequest {
userId?: string;
userEmail?: string;
userRole?: string;
}
}
// Paths that don't require authentication
const PUBLIC_PATHS = new Set([
"/health",
"/api/auth/login",
"/ws/bastion",
"/ws/agent",
"/api/auth/enroll",
]);
export function createBearerAuthMiddleware(authService: AuthService) {
return async function bearerAuth(
request: FastifyRequest,
reply: FastifyReply,
): Promise<void> {
// Skip auth for public paths
if (PUBLIC_PATHS.has(request.url.split("?")[0] ?? "")) {
return;
}
// Skip auth for WebSocket upgrade requests (handled by their own auth)
if (request.headers.upgrade === "websocket") {
return;
}
const authHeader = request.headers.authorization;
if (!authHeader) {
void reply.code(401).send({ error: "Authorization header required" });
return;
}
if (!authHeader.startsWith("Bearer ")) {
void reply.code(401).send({ error: "Invalid authorization format, expected: Bearer <token>" });
return;
}
const token = authHeader.slice(7);
if (token.length === 0) {
void reply.code(401).send({ error: "Empty bearer token" });
return;
}
try {
const identity = await authService.validateToken(token);
request.userId = identity.userId;
request.userEmail = identity.email;
request.userRole = identity.role;
} catch {
void reply.code(401).send({ error: "Invalid or expired token. Run: labctl login" });
}
};
}

View File

@@ -10,6 +10,7 @@ import type { FastifyInstance } from "fastify";
import type { DbClient } from "../server.js";
import { bastionRegistry } from "../services/bastion-registry.js";
import { generateRequestId } from "@lab/shared";
import type { VyosInstallSpec } from "@lab/shared";
const COMMAND_TIMEOUT_MS = 15_000;
@@ -84,7 +85,6 @@ export function registerBastionRoutes(app: FastifyInstance, db: DbClient): void
app.get("/api/machines", async () => {
const live = bastionRegistry.getAggregatedState();
// Merge DB records for machines not currently in any bastion's live state
try {
const dbServers = (await db.server.findMany({})) as Array<{
mac: string | null; hostname: string; role: string; ip: string | null;
@@ -93,9 +93,49 @@ export function registerBastionRoutes(app: FastifyInstance, db: DbClient): void
for (const s of dbServers) {
if (!s.mac) continue;
const mac = s.mac.toLowerCase();
// Only add from DB if not already in live state
// DB knows this machine has been installed at some point if it has a real
// hostname+role (not just product-name-as-hostname and role="unknown").
// Status alone is unreliable: a rediscovery can re-set it without erasing the
// install identity. If the bastion restarted and lost its installed map, the
// machine will only show up in live.discovered — promote it here so the CLI
// still sees hostname/role/IP.
const dbKnowsInstalled =
s.role !== "unknown" && s.role !== "" &&
s.hostname !== "" && s.hostname !== s.mac;
if (dbKnowsInstalled && !(mac in live.installed) && !(mac in live.install_queue)) {
const hw = live.discovered[mac];
live.installed[mac] = {
hostname: s.hostname,
role: s.role,
ip: s.ip ?? "",
installed_at: "",
bastionId: hw?.bastionId ?? "db",
...(hw ? {
product: hw.product,
manufacturer: hw.manufacturer,
cpu_model: hw.cpu_model,
cpu_cores: hw.cpu_cores,
memory_gb: hw.memory_gb,
arch: hw.arch,
} : {}),
};
delete live.discovered[mac];
continue;
}
// Unknown-to-live MAC: fall back to whatever the DB says.
if (!(mac in live.discovered) && !(mac in live.install_queue) && !(mac in live.installed)) {
if (s.status === "discovered") {
if (s.status === "online" || s.status === "offline") {
live.installed[mac] = {
hostname: s.hostname,
role: s.role,
ip: s.ip ?? "",
installed_at: "",
bastionId: "db",
};
} else {
live.discovered[mac] = {
mac,
product: String(s.labels?.product ?? "unknown"),
@@ -112,14 +152,6 @@ export function registerBastionRoutes(app: FastifyInstance, db: DbClient): void
last_seen: "",
bastionId: "db",
};
} else if (s.status === "online" || s.status === "offline") {
live.installed[mac] = {
hostname: s.hostname,
role: s.role,
ip: s.ip ?? "",
installed_at: "",
bastionId: "db",
};
}
}
}
@@ -132,9 +164,9 @@ export function registerBastionRoutes(app: FastifyInstance, db: DbClient): void
// Queue install — route to correct bastion by MAC
app.post<{
Body: { mac?: string; hostname?: string; disk?: string; role?: string; os?: string };
Body: { mac?: string; hostname?: string; disk?: string; role?: string; os?: string; vyos?: VyosInstallSpec };
}>("/api/machines/install", async (request, reply) => {
const { mac, hostname, disk, role, os } = request.body ?? {};
const { mac, hostname, disk, role, os, vyos } = request.body ?? {};
if (!mac || !hostname) {
return reply.code(400).send({ error: "mac and hostname are required" });
}
@@ -152,6 +184,7 @@ export function registerBastionRoutes(app: FastifyInstance, db: DbClient): void
const result = await sendCommand(all[0]!.bastionId, {
type: "command-install",
mac, hostname, disk: disk ?? "", role: role ?? "infra", os: os ?? "fedora-43",
...(vyos ? { vyos } : {}),
});
return reply.code(result.status === "ok" ? 200 : 500).send(result);
} catch (err) {
@@ -165,6 +198,7 @@ export function registerBastionRoutes(app: FastifyInstance, db: DbClient): void
const result = await sendCommand(bastion.bastionId, {
type: "command-install",
mac, hostname, disk: disk ?? "", role: role ?? "infra", os: os ?? "fedora-43",
...(vyos ? { vyos } : {}),
});
return reply.code(result.status === "ok" ? 200 : 500).send(result);
} catch (err) {
@@ -260,6 +294,37 @@ export function registerBastionRoutes(app: FastifyInstance, db: DbClient): void
}
});
// Update hardware info (discovery data) for a machine
app.post<{
Body: {
mac?: string; product?: string; board?: string; serial?: string;
manufacturer?: string; cpu_model?: string; cpu_cores?: number;
memory_gb?: number; arch?: string;
disks?: Array<{ name: string; size_gb: number; model: string }>;
nics?: Array<{ name: string; mac: string; state: string }>;
};
}>("/api/machines/discover", async (request, reply) => {
const data = request.body ?? {};
const mac = (data.mac ?? "").toLowerCase().replace(/-/g, ":");
if (!mac) {
return reply.code(400).send({ error: "mac is required" });
}
const bastion = bastionRegistry.findBastionByMac(mac);
const target = bastion ?? (bastionRegistry.getAll().length === 1 ? bastionRegistry.getAll()[0] : null);
if (!target) {
return reply.code(503).send({ error: "No bastion found for this MAC" });
}
try {
const result = await sendCommand(target.bastionId, { type: "command-discover", ...data, mac });
return reply.code(result.status === "ok" ? 200 : 500).send(result);
} catch (err) {
return reply.code(500).send({ error: err instanceof Error ? err.message : String(err) });
}
});
// Update role
app.post<{
Body: { mac?: string; role?: string };

View File

@@ -0,0 +1,191 @@
// Environment and Account management routes.
// GET/POST /api/environments — list/create environments
// GET/POST /api/accounts — list/create accounts
// POST /api/accounts/bind — bind account to environment
// GET /api/bindings — list bindings
import type { FastifyInstance } from "fastify";
import type { PrismaClient, Prisma } from "@prisma/client";
import type { RbacService } from "../services/rbac.js";
import type { AuditService } from "../services/audit.js";
export function registerEnvironmentRoutes(
app: FastifyInstance,
db: PrismaClient,
rbacService: RbacService,
auditService: AuditService,
): void {
// List environments
app.get("/api/environments", async (_request, reply) => {
const envs = await db.environment.findMany({ orderBy: { name: "asc" } });
return reply.send(envs);
});
// Create environment
app.post<{
Body: { name?: string };
}>("/api/environments", async (request, reply) => {
const { name } = request.body ?? {};
if (!name) {
return reply.code(400).send({ error: "name is required" });
}
const rbac = await rbacService.check({
userId: request.userId!,
userEmail: request.userEmail!,
userRole: request.userRole!,
action: "admin",
resource: "environments",
});
if (!rbac.allowed) {
return reply.code(403).send({ error: rbac.reason });
}
try {
const env = await db.environment.create({ data: { name } });
auditService.emit({
eventKind: "resource_created",
source: "labd",
verified: true,
userId: request.userId ?? null,
resourceKind: "environment",
resourceName: name,
result: "success",
});
return reply.code(201).send(env);
} catch (err) {
if (err instanceof Error && err.message.includes("Unique constraint")) {
return reply.code(409).send({ error: `Environment '${name}' already exists` });
}
throw err;
}
});
// List accounts
app.get("/api/accounts", async (_request, reply) => {
const accounts = await db.account.findMany({
orderBy: { name: "asc" },
select: { id: true, name: true, driver: true, config: true, createdAt: true, updatedAt: true },
});
return reply.send(accounts);
});
// Create account
app.post<{
Body: { name?: string; driver?: string; config?: Record<string, unknown> };
}>("/api/accounts", async (request, reply) => {
const { name, driver, config } = request.body ?? {};
if (!name || !driver) {
return reply.code(400).send({ error: "name and driver are required" });
}
const rbac = await rbacService.check({
userId: request.userId!,
userEmail: request.userEmail!,
userRole: request.userRole!,
action: "admin",
resource: "accounts",
});
if (!rbac.allowed) {
return reply.code(403).send({ error: rbac.reason });
}
try {
const account = await db.account.create({
data: { name, driver, config: (config ?? {}) as Prisma.InputJsonValue },
});
auditService.emit({
eventKind: "resource_created",
source: "labd",
verified: true,
userId: request.userId ?? null,
resourceKind: "account",
resourceName: name,
result: "success",
details: { driver },
});
return reply.code(201).send(account);
} catch (err) {
if (err instanceof Error && err.message.includes("Unique constraint")) {
return reply.code(409).send({ error: `Account '${name}' already exists` });
}
throw err;
}
});
// Bind account to environment
app.post<{
Body: { environmentId?: string; accountId?: string };
}>("/api/accounts/bind", async (request, reply) => {
const { environmentId, accountId } = request.body ?? {};
if (!environmentId || !accountId) {
return reply.code(400).send({ error: "environmentId and accountId are required" });
}
const rbac = await rbacService.check({
userId: request.userId!,
userEmail: request.userEmail!,
userRole: request.userRole!,
action: "admin",
resource: "accounts",
});
if (!rbac.allowed) {
return reply.code(403).send({ error: rbac.reason });
}
try {
const binding = await db.binding.create({
data: { environmentId, accountId },
});
return reply.code(201).send(binding);
} catch (err) {
if (err instanceof Error && err.message.includes("Unique constraint")) {
return reply.code(409).send({ error: "This account is already bound to this environment" });
}
throw err;
}
});
// List bindings
app.get("/api/bindings", async (_request, reply) => {
const bindings = await db.binding.findMany({
include: { environment: true, account: true },
});
return reply.send(bindings);
});
// Audit event query
app.get<{
Querystring: {
last?: string;
kind?: string;
env?: string;
correlation?: string;
limit?: string;
};
}>("/api/events", async (request, reply) => {
const { last, kind, env, correlation, limit } = request.query as { last?: string; kind?: string; env?: string; correlation?: string; limit?: string };
const where: Record<string, unknown> = {};
if (last) {
const match = last.match(/^(\d+)(h|d|m)$/);
if (match) {
const [, num, unit] = match;
const ms = { h: 3_600_000, d: 86_400_000, m: 60_000 }[unit!]!;
where.timestamp = { gte: new Date(Date.now() - parseInt(num!) * ms) };
}
}
if (kind) where.eventKind = kind;
if (env) where.environmentName = env;
if (correlation) where.correlationId = correlation;
const events = await db.auditEvent.findMany({
where,
orderBy: { timestamp: "desc" },
take: Math.min(parseInt(limit ?? "100"), 500),
});
return reply.send(events);
});
}

View File

@@ -0,0 +1,196 @@
// Resource CRUD routes with RBAC enforcement.
// GET /api/resources — list (filtered by RBAC scope)
// GET /api/resources/:id — get
// POST /api/resources — create
// PUT /api/resources/:id — update
// DELETE /api/resources/:id — delete (marks as deleting)
import type { FastifyInstance } from "fastify";
import type { ResourceStore, CreateResourceInput } from "../services/resource-store.js";
import type { RbacService } from "../services/rbac.js";
import type { AuditService } from "../services/audit.js";
import { resolveResourceKind } from "@lab/core";
export function registerResourceRoutes(
app: FastifyInstance,
resourceStore: ResourceStore,
rbacService: RbacService,
auditService: AuditService,
): void {
// List resources (filtered by kind, environment, status)
app.get<{
Querystring: { kind?: string; environment?: string; status?: string };
}>("/api/resources", async (request, reply) => {
const rbac = await rbacService.check({
userId: request.userId!,
userEmail: request.userEmail!,
userRole: request.userRole!,
action: "view",
resource: request.query.kind ? resolveResourceKind(request.query.kind) : undefined,
});
if (!rbac.allowed) {
return reply.code(403).send({ error: rbac.reason });
}
const resources = await resourceStore.list({
kind: request.query.kind ? resolveResourceKind(request.query.kind) : undefined,
environmentId: request.query.environment,
status: request.query.status,
});
return reply.send(resources);
});
// Get single resource
app.get<{
Params: { id: string };
}>("/api/resources/:id", async (request, reply) => {
const resource = await resourceStore.get(request.params.id);
if (!resource) {
return reply.code(404).send({ error: "Resource not found" });
}
const rbac = await rbacService.check({
userId: request.userId!,
userEmail: request.userEmail!,
userRole: request.userRole!,
action: "view",
resource: resource.kind,
name: resource.name,
});
if (!rbac.allowed) {
return reply.code(403).send({ error: rbac.reason });
}
return reply.send(resource);
});
// Create resource
app.post<{
Body: CreateResourceInput;
}>("/api/resources", async (request, reply) => {
const input = request.body;
if (!input?.kind || !input?.name || !input?.environmentId || !input?.accountId) {
return reply.code(400).send({ error: "kind, name, environmentId, and accountId are required" });
}
const kind = resolveResourceKind(input.kind);
const rbac = await rbacService.check({
userId: request.userId!,
userEmail: request.userEmail!,
userRole: request.userRole!,
action: "create",
resource: kind,
});
if (!rbac.allowed) {
return reply.code(403).send({ error: rbac.reason });
}
const correlationId = auditService.createCorrelation();
try {
const resource = await resourceStore.create({ ...input, kind });
auditService.emit({
eventKind: "resource_created",
source: "labd",
verified: true,
userId: request.userId ?? null,
userName: request.userEmail ?? null,
resourceKind: kind,
resourceName: input.name,
correlationId,
result: "success",
});
return reply.code(201).send(resource);
} catch (err) {
// Prisma unique constraint violation
if (err instanceof Error && err.message.includes("Unique constraint")) {
return reply.code(409).send({ error: `Resource ${kind}/${input.name} already exists in this environment` });
}
throw err;
}
});
// Update resource
app.put<{
Params: { id: string };
Body: { desiredSpec?: Record<string, unknown>; status?: string };
}>("/api/resources/:id", async (request, reply) => {
const resource = await resourceStore.get(request.params.id);
if (!resource) {
return reply.code(404).send({ error: "Resource not found" });
}
const rbac = await rbacService.check({
userId: request.userId!,
userEmail: request.userEmail!,
userRole: request.userRole!,
action: "edit",
resource: resource.kind,
name: resource.name,
});
if (!rbac.allowed) {
return reply.code(403).send({ error: rbac.reason });
}
const updated = await resourceStore.update(request.params.id, request.body);
auditService.emit({
eventKind: "resource_updated",
source: "labd",
verified: true,
userId: request.userId ?? null,
userName: request.userEmail ?? null,
resourceKind: resource.kind,
resourceName: resource.name,
result: "success",
});
return reply.send(updated);
});
// Delete resource (marks as deleting)
app.delete<{
Params: { id: string };
}>("/api/resources/:id", async (request, reply) => {
const resource = await resourceStore.get(request.params.id);
if (!resource) {
return reply.code(404).send({ error: "Resource not found" });
}
const rbac = await rbacService.check({
userId: request.userId!,
userEmail: request.userEmail!,
userRole: request.userRole!,
action: "delete",
resource: resource.kind,
name: resource.name,
});
if (!rbac.allowed) {
return reply.code(403).send({ error: rbac.reason });
}
await resourceStore.delete(request.params.id);
auditService.emit({
eventKind: "resource_deleted",
source: "labd",
verified: true,
userId: request.userId ?? null,
userName: request.userEmail ?? null,
resourceKind: resource.kind,
resourceName: resource.name,
result: "success",
});
return reply.send({ status: "deleting", id: request.params.id });
});
}

View File

@@ -0,0 +1,81 @@
// v2 Auth routes: bearer token login/logout.
// POST /api/auth/login — email + password → session token
// POST /api/auth/logout — revoke session
import type { FastifyInstance } from "fastify";
import type { AuthService } from "../services/auth.js";
import type { AuditService } from "../services/audit.js";
import { AuthError } from "../services/auth.js";
export function registerV2AuthRoutes(
app: FastifyInstance,
authService: AuthService,
auditService: AuditService,
): void {
app.post<{
Body: { email?: string; password?: string };
}>("/api/auth/login", async (request, reply) => {
const { email, password } = request.body ?? {};
if (!email || !password) {
return reply.code(400).send({ error: "email and password are required" });
}
try {
const result = await authService.login(email, password);
auditService.emit({
eventKind: result.isBootstrap ? "auth_bootstrap" : "auth_login",
source: "labd",
verified: true,
userId: result.userId,
userName: email,
result: "success",
details: { isBootstrap: result.isBootstrap },
});
return reply.send({
token: result.token,
expiresAt: result.expiresAt.toISOString(),
isBootstrap: result.isBootstrap,
});
} catch (err) {
if (err instanceof AuthError) {
auditService.emit({
eventKind: "auth_login",
source: "labd",
verified: true,
userName: email,
result: "failure",
error: err.message,
});
return reply.code(401).send({ error: err.message });
}
return reply.code(500).send({ error: "Login failed" });
}
});
app.post("/api/auth/logout", async (request, reply) => {
const token = request.headers.authorization?.slice(7);
if (!token) {
return reply.code(400).send({ error: "Authorization header required" });
}
try {
await authService.logout(token);
auditService.emit({
eventKind: "auth_logout",
source: "labd",
verified: true,
userId: request.userId ?? null,
result: "success",
});
return reply.send({ status: "logged_out" });
} catch (err) {
if (err instanceof AuthError) {
return reply.code(400).send({ error: err.message });
}
return reply.code(500).send({ error: "Logout failed" });
}
});
}

View File

@@ -2,6 +2,7 @@
import Fastify from "fastify";
import websocket from "@fastify/websocket";
import type { PrismaClient } from "@prisma/client";
import type { LabdConfig } from "./config.js";
import { logger } from "./services/logger.js";
import { registerHealthRoutes } from "./routes/health.js";
@@ -9,8 +10,16 @@ import { registerServerRoutes } from "./routes/servers.js";
import { registerAuthRoutes } from "./routes/auth.js";
import { registerAgentRoutes } from "./routes/agents.js";
import { registerBastionRoutes } from "./routes/bastions.js";
import { registerV2AuthRoutes } from "./routes/v2-auth.js";
import { registerEnvironmentRoutes } from "./routes/environments.js";
import { registerResourceRoutes } from "./routes/resources.js";
import { setupRateLimiting } from "./middleware/rate-limit.js";
import { createBearerAuthMiddleware } from "./middleware/bearer-auth.js";
import { bastionRegistry } from "./services/bastion-registry.js";
import { AuthService } from "./services/auth.js";
import { RbacService } from "./services/rbac.js";
import { ResourceStore } from "./services/resource-store.js";
import { AuditService } from "./services/audit.js";
import { isBastionMessage } from "@lab/shared";
export interface DbClient {
@@ -37,6 +46,7 @@ export interface DbClient {
export async function createApp(_config: LabdConfig, db: DbClient): Promise<{
app: ReturnType<typeof Fastify>;
auditService: AuditService;
}> {
const app = Fastify({
logger: false, // We use winston instead
@@ -48,13 +58,39 @@ export async function createApp(_config: LabdConfig, db: DbClient): Promise<{
// Register WebSocket support
void app.register(websocket);
// Register route handlers
// v2 services. The structural DbClient is a subset of the real PrismaClient;
// at runtime db IS the PrismaClient instance, so the cast is safe. Tests that
// exercise v2 routes provide a PrismaClient-shaped mock (see auth-bootstrap,
// rbac-deny, audit-correlation tests).
const prisma = db as unknown as PrismaClient;
const authService = new AuthService(prisma);
const rbacService = new RbacService(prisma);
const resourceStore = new ResourceStore(prisma);
const auditService = new AuditService(prisma);
auditService.start();
// Register v1 (legacy) route handlers
registerHealthRoutes(app, db);
registerServerRoutes(app, db);
registerAuthRoutes(app, db);
registerAgentRoutes(app);
registerBastionRoutes(app, db);
// v2 routes live in a scope with bearer-auth as preHandler. Public paths
// (login, /health, websockets) are skipped inside the middleware itself.
// v1 routes above are unaffected — they're registered on the root scope.
await app.register(async (scope) => {
scope.addHook("preHandler", createBearerAuthMiddleware(authService));
registerV2AuthRoutes(scope, authService, auditService);
registerEnvironmentRoutes(scope, prisma, rbacService, auditService);
registerResourceRoutes(scope, resourceStore, rbacService, auditService);
});
// Flush pending audit events on shutdown so we never lose the last batch.
app.addHook("onClose", async () => {
auditService.stop();
});
// WebSocket handler for agent connections
app.register(async (fastify) => {
fastify.get("/ws/agent", { websocket: true }, (socket, _request) => {
@@ -192,7 +228,9 @@ export async function createApp(_config: LabdConfig, db: DbClient): Promise<{
labels: { cpu: hw.cpu_model, cores: hw.cpu_cores, memory_gb: hw.memory_gb, arch: hw.arch, product: hw.product, manufacturer: hw.manufacturer },
},
update: {
status: "discovered",
// Leave status alone — a previously "online"/"offline" record
// must not be downgraded to "discovered" just because the bastion
// restarted and re-discovered the MAC via DHCP/PXE.
lastHeartbeat: new Date(),
labels: { cpu: hw.cpu_model, cores: hw.cpu_cores, memory_gb: hw.memory_gb, arch: hw.arch, product: hw.product, manufacturer: hw.manufacturer },
},
@@ -265,5 +303,5 @@ export async function createApp(_config: LabdConfig, db: DbClient): Promise<{
logger.info(`HTTP: ${request.ip} ${request.method} ${request.url}`);
});
return { app };
return { app, auditService };
}

View File

@@ -0,0 +1,106 @@
// Audit service: fire-and-forget event collection with batching.
// Batches 50 events or flushes every 5 seconds, whichever comes first.
// Failures never block the operation being audited.
import { randomBytes } from "node:crypto";
import type { PrismaClient, Prisma } from "@prisma/client";
import { logger } from "./logger.js";
const BATCH_SIZE = 50;
const FLUSH_INTERVAL_MS = 5_000;
export interface AuditEventInput {
eventKind: string;
source: string;
verified?: boolean;
userId?: string | null;
userName?: string | null;
sessionId?: string | null;
environmentName?: string | null;
accountName?: string | null;
resourceKind?: string | null;
resourceName?: string | null;
correlationId?: string | null;
parentEventId?: string | null;
details?: Record<string, unknown>;
result: string;
error?: string | null;
durationMs?: number | null;
}
export class AuditService {
private batch: AuditEventInput[] = [];
private timer: ReturnType<typeof setInterval> | null = null;
constructor(private readonly db: PrismaClient) {}
start(): void {
this.timer = setInterval(() => {
void this.flush();
}, FLUSH_INTERVAL_MS);
}
stop(): void {
if (this.timer) {
clearInterval(this.timer);
this.timer = null;
}
void this.flush();
}
emit(event: AuditEventInput): void {
// Generate correlation ID if not provided
if (!event.correlationId) {
event.correlationId = `corr_${randomBytes(8).toString("hex")}`;
}
this.batch.push(event);
if (this.batch.length >= BATCH_SIZE) {
void this.flush();
}
}
/** Create a correlation context for a chain of related events. */
createCorrelation(): string {
return `corr_${randomBytes(8).toString("hex")}`;
}
/** Flush all pending events synchronously. Tests await this; production
* relies on the interval timer or stop() during shutdown. */
async flushPending(): Promise<void> {
await this.flush();
}
private async flush(): Promise<void> {
if (this.batch.length === 0) return;
const events = this.batch.splice(0);
try {
await this.db.auditEvent.createMany({
data: events.map((e) => ({
eventKind: e.eventKind,
source: e.source,
verified: e.verified ?? false,
userId: e.userId ?? null,
userName: e.userName ?? null,
sessionId: e.sessionId ?? null,
environmentName: e.environmentName ?? null,
accountName: e.accountName ?? null,
resourceKind: e.resourceKind ?? null,
resourceName: e.resourceName ?? null,
correlationId: e.correlationId ?? `corr_${randomBytes(8).toString("hex")}`,
parentEventId: e.parentEventId ?? null,
details: (e.details ?? {}) as Prisma.InputJsonValue,
result: e.result,
error: e.error ?? null,
durationMs: e.durationMs ?? null,
})),
});
logger.info(`AUDIT: flushed ${events.length} events`);
} catch (err) {
// Fire-and-forget: audit failures never block operations
logger.warn(`AUDIT: failed to flush ${events.length} events: ${err instanceof Error ? err.message : String(err)}`);
}
}
}

View File

@@ -0,0 +1,119 @@
// Auth service: bearer token authentication with bootstrap flow.
// First login creates the admin user. Subsequent logins return session tokens.
import { randomBytes } from "node:crypto";
import bcrypt from "bcryptjs";
import type { PrismaClient } from "@prisma/client";
import { logger } from "./logger.js";
const SESSION_EXPIRY_DAYS = 30;
const BCRYPT_ROUNDS = 12;
export interface LoginResult {
token: string;
expiresAt: Date;
userId: string;
isBootstrap: boolean;
}
export class AuthService {
constructor(private readonly db: PrismaClient) {}
async login(email: string, password: string): Promise<LoginResult> {
const userCount = await this.db.user.count();
// Bootstrap: first login creates admin user
if (userCount === 0) {
return this.bootstrap(email, password);
}
const user = await this.db.user.findUnique({ where: { email } });
if (!user) {
// Same error for unknown user and wrong password (no enumeration)
throw new AuthError("Invalid email or password");
}
const valid = await bcrypt.compare(password, user.password);
if (!valid) {
throw new AuthError("Invalid email or password");
}
const session = await this.createSession(user.id);
logger.info(`AUTH LOGIN: ${email} (${user.id.slice(0, 8)}...)`);
return {
token: session.token,
expiresAt: session.expiresAt,
userId: user.id,
isBootstrap: false,
};
}
async logout(token: string): Promise<void> {
const session = await this.db.session.findUnique({ where: { token } });
if (!session) {
throw new AuthError("Invalid session");
}
await this.db.session.delete({ where: { id: session.id } });
logger.info(`AUTH LOGOUT: session ${session.id.slice(0, 8)}...`);
}
async validateToken(token: string): Promise<{ userId: string; email: string; role: string }> {
const session = await this.db.session.findUnique({
where: { token },
include: { user: true },
});
if (!session) {
throw new AuthError("Invalid token");
}
if (session.expiresAt < new Date()) {
await this.db.session.delete({ where: { id: session.id } });
throw new AuthError("Token expired");
}
return {
userId: session.user.id,
email: session.user.email,
role: session.user.role,
};
}
private async bootstrap(email: string, password: string): Promise<LoginResult> {
const hashed = await bcrypt.hash(password, BCRYPT_ROUNDS);
const user = await this.db.user.create({
data: {
email,
password: hashed,
role: "ADMIN",
name: email.split("@")[0] ?? null,
},
});
const session = await this.createSession(user.id);
logger.info(`AUTH BOOTSTRAP: created admin user ${email} (${user.id.slice(0, 8)}...)`);
return {
token: session.token,
expiresAt: session.expiresAt,
userId: user.id,
isBootstrap: true,
};
}
private async createSession(userId: string) {
const token = randomBytes(32).toString("hex");
const expiresAt = new Date(Date.now() + SESSION_EXPIRY_DAYS * 24 * 60 * 60 * 1000);
return this.db.session.create({
data: { userId, token, expiresAt },
});
}
}
export class AuthError extends Error {
constructor(message: string) {
super(message);
this.name = "AuthError";
}
}

View File

@@ -0,0 +1,123 @@
// RBAC service: environment-scoped permission checks.
// Uses named RbacDefinition records with JSON subjects and roleBindings.
//
// Resolution flow:
// 1. Find all RbacDefinitions where subjects match the current user/groups
// 2. Collect all roleBindings from matching definitions
// 3. Check if any binding grants the requested action on the requested resource
import type { PrismaClient } from "@prisma/client";
import { logger } from "./logger.js";
export interface RbacCheck {
userId: string;
userEmail: string;
userRole: string;
action: string; // "view" | "edit" | "create" | "delete" | "run" | "admin"
resource?: string | undefined; // "servers" | "databases" | "clusters" | "*"
name?: string | undefined; // specific resource name
environment?: string | undefined; // specific environment name
}
export interface RbacResult {
allowed: boolean;
reason: string;
matchedDefinition?: string;
}
interface StoredSubject {
kind: string;
name: string;
}
interface StoredBinding {
role: string;
resource?: string;
name?: string;
environment?: string;
action?: string;
}
export class RbacService {
constructor(private readonly db: PrismaClient) {}
async check(req: RbacCheck): Promise<RbacResult> {
// Admin users bypass RBAC
if (req.userRole === "ADMIN") {
return { allowed: true, reason: "admin role" };
}
// Collect user's group memberships
const memberships = await this.db.groupMember.findMany({
where: { userId: req.userId },
include: { group: true },
});
const groupNames = memberships.map((m) => m.group.name);
// Find all RBAC definitions
const definitions = await this.db.rbacDefinition.findMany();
for (const def of definitions) {
const subjects = def.subjects as unknown as StoredSubject[];
const bindings = def.roleBindings as unknown as StoredBinding[];
// Check if this definition's subjects match the user
const subjectMatch = subjects.some((s) => {
if (s.kind === "User" && s.name === req.userEmail) return true;
if (s.kind === "Group" && groupNames.includes(s.name)) return true;
return false;
});
if (!subjectMatch) continue;
// Check if any binding grants the requested permission
for (const binding of bindings) {
if (this.bindingMatches(binding, req)) {
logger.info(`RBAC ALLOW: ${req.userEmail} ${req.action} ${req.resource ?? "*"}${req.name ? `/${req.name}` : ""} via ${def.name}`);
return {
allowed: true,
reason: `granted by ${def.name}`,
matchedDefinition: def.name,
};
}
}
}
logger.info(`RBAC DENY: ${req.userEmail} ${req.action} ${req.resource ?? "*"}${req.name ? `/${req.name}` : ""}`);
return {
allowed: false,
reason: `no matching role binding for ${req.action} on ${req.resource ?? "*"}`,
};
}
private bindingMatches(binding: StoredBinding, req: RbacCheck): boolean {
// Check role grants the action
if (!this.roleGrantsAction(binding.role, req.action)) return false;
// Check resource scope
if (binding.resource && binding.resource !== "*" && binding.resource !== req.resource) return false;
// Check name scope
if (binding.name && binding.name !== req.name) return false;
// Check environment scope
if (binding.environment && binding.environment !== req.environment) return false;
// Check operation scope (for "run" role with specific actions)
if (binding.action && binding.action !== "*" && binding.action !== req.action) return false;
return true;
}
private roleGrantsAction(role: string, action: string): boolean {
const grants: Record<string, string[]> = {
admin: ["view", "edit", "create", "delete", "run", "admin"],
edit: ["view", "edit", "create", "delete"],
create: ["create"],
delete: ["delete"],
view: ["view"],
run: ["run"],
};
return grants[role]?.includes(action) ?? false;
}
}

View File

@@ -0,0 +1,108 @@
// Resource store: CRUD for generic resources with origin/managedBy tracking.
// All mutations go through this service so RBAC and audit are applied consistently.
import type { PrismaClient, Resource as PrismaResource, Prisma } from "@prisma/client";
import { logger } from "./logger.js";
export interface CreateResourceInput {
kind: string;
name: string;
environmentId: string;
accountId: string;
origin?: string;
managedBy?: string;
sourceRef?: string;
desiredSpec: Record<string, unknown>;
}
export interface UpdateResourceInput {
desiredSpec?: Record<string, unknown>;
status?: string;
statusMessage?: string;
actualSpec?: Record<string, unknown>;
platformRef?: string;
}
export interface ListResourcesFilter {
kind?: string | undefined;
environmentId?: string | undefined;
accountId?: string | undefined;
status?: string | undefined;
}
export class ResourceStore {
constructor(private readonly db: PrismaClient) {}
async create(input: CreateResourceInput): Promise<PrismaResource> {
const resource = await this.db.resource.create({
data: {
kind: input.kind,
name: input.name,
environmentId: input.environmentId,
accountId: input.accountId,
origin: input.origin ?? "cli",
managedBy: input.managedBy ?? "manual",
sourceRef: input.sourceRef ?? null,
desiredSpec: input.desiredSpec as Prisma.InputJsonValue,
status: "pending",
},
});
logger.info(`RESOURCE CREATED: ${input.kind}/${input.name} in env ${input.environmentId.slice(0, 8)}...`);
return resource;
}
async get(id: string): Promise<PrismaResource | null> {
return this.db.resource.findUnique({ where: { id } });
}
async getByKindNameEnv(kind: string, name: string, environmentId: string): Promise<PrismaResource | null> {
return this.db.resource.findUnique({
where: { kind_name_environmentId: { kind, name, environmentId } },
});
}
async list(filter: ListResourcesFilter = {}): Promise<PrismaResource[]> {
return this.db.resource.findMany({
where: {
...(filter.kind ? { kind: filter.kind } : {}),
...(filter.environmentId ? { environmentId: filter.environmentId } : {}),
...(filter.accountId ? { accountId: filter.accountId } : {}),
...(filter.status ? { status: filter.status } : {}),
},
orderBy: { createdAt: "desc" },
});
}
async update(id: string, input: UpdateResourceInput): Promise<PrismaResource> {
const data: Prisma.ResourceUpdateInput = {};
if (input.desiredSpec !== undefined) data.desiredSpec = input.desiredSpec as Prisma.InputJsonValue;
if (input.status !== undefined) data.status = input.status;
if (input.statusMessage !== undefined) data.statusMessage = input.statusMessage;
if (input.actualSpec !== undefined) data.actualSpec = input.actualSpec as Prisma.InputJsonValue;
if (input.platformRef !== undefined) data.platformRef = input.platformRef;
if (input.status === "ready") data.lastReconciled = new Date();
const resource = await this.db.resource.update({ where: { id }, data });
logger.info(`RESOURCE UPDATED: ${resource.kind}/${resource.name} -> ${input.status ?? "spec change"}`);
return resource;
}
async delete(id: string): Promise<void> {
const resource = await this.db.resource.findUnique({ where: { id } });
if (!resource) return;
// Mark as deleting first (driver handles actual deletion)
await this.db.resource.update({
where: { id },
data: { status: "deleting" },
});
logger.info(`RESOURCE DELETING: ${resource.kind}/${resource.name}`);
}
async hardDelete(id: string): Promise<void> {
await this.db.resource.delete({ where: { id } });
}
}

View File

@@ -0,0 +1,144 @@
import { describe, it, expect, vi, beforeEach } from "vitest";
import Fastify from "fastify";
import { registerBastionRoutes } from "../src/routes/bastions.js";
import { bastionRegistry } from "../src/services/bastion-registry.js";
import type { DbClient } from "../src/server.js";
import type { BastionState } from "@lab/shared";
function createMockDb(servers: unknown[] = []): DbClient {
return {
$queryRaw: vi.fn().mockResolvedValue([{ "?column?": 1 }]),
server: {
findMany: vi.fn().mockResolvedValue(servers),
findUnique: vi.fn().mockResolvedValue(null),
upsert: vi.fn().mockResolvedValue({}),
},
joinToken: {
findUnique: vi.fn().mockResolvedValue(null),
findMany: vi.fn().mockResolvedValue([]),
create: vi.fn().mockResolvedValue({ id: "t" }),
update: vi.fn().mockResolvedValue({}),
},
bastion: {
upsert: vi.fn().mockResolvedValue({}),
findMany: vi.fn().mockResolvedValue([]),
findUnique: vi.fn().mockResolvedValue(null),
update: vi.fn().mockResolvedValue({}),
},
};
}
function registerFakeBastion(bastionId: string, state: BastionState): void {
bastionRegistry.register({
bastionId,
hostname: "fake",
network: "192.168.8.0/24",
serverIp: "192.168.8.11",
// socket is referenced only on commands, not during aggregation
socket: { on: () => undefined, off: () => undefined, send: () => undefined, close: () => undefined } as never,
connectedAt: new Date(),
lastHeartbeat: new Date(),
state,
});
}
describe("GET /api/machines aggregation", () => {
beforeEach(() => {
for (const b of bastionRegistry.getAll()) bastionRegistry.unregister(b.bastionId);
});
it("promotes a live-discovered MAC to installed when the DB has a real hostname+role for it", async () => {
// Simulates the worker0-k8s0 bug: bastion restarted, lost its installed map,
// rediscovered the machine via DHCP/PXE. DB still has hostname=worker0-k8s0,
// role=infra, ip=192.168.8.23. Without the fix, the CLI sees a "discovered"
// row with no hostname/role/IP. With the fix, the row is promoted to
// "installed" with full identity preserved.
const mac = "78:55:36:08:28:fb";
registerFakeBastion("b1", {
discovered: {
[mac]: {
mac, product: "SER", board: "SER", serial: "x", manufacturer: "AZW",
cpu_model: "AMD Ryzen 7 255", cpu_cores: 16, memory_gb: 58, arch: "x86_64",
disks: [], nics: [], first_seen: "", last_seen: "",
},
},
install_queue: {},
installed: {},
debug: {},
});
const app = Fastify({ logger: false });
const db = createMockDb([
{ mac, hostname: "worker0-k8s0", role: "infra", ip: "192.168.8.23", status: "discovered", labels: {} },
]);
registerBastionRoutes(app, db);
const res = await app.inject({ method: "GET", url: "/api/machines" });
expect(res.statusCode).toBe(200);
const body = JSON.parse(res.body);
expect(body.discovered[mac]).toBeUndefined();
expect(body.installed[mac]).toMatchObject({
hostname: "worker0-k8s0",
role: "infra",
ip: "192.168.8.23",
cpu_model: "AMD Ryzen 7 255",
cpu_cores: 16,
memory_gb: 58,
});
await app.close();
});
it("leaves a fresh-discovery MAC in discovered when DB only has a discovery-shaped record", async () => {
const mac = "aa:bb:cc:dd:ee:ff";
registerFakeBastion("b1", {
discovered: {
[mac]: {
mac, product: "SER", board: "SER", serial: "x", manufacturer: "AZW",
cpu_model: "AMD Ryzen 7", cpu_cores: 8, memory_gb: 32, arch: "x86_64",
disks: [], nics: [], first_seen: "", last_seen: "",
},
},
install_queue: {},
installed: {},
debug: {},
});
const app = Fastify({ logger: false });
// Matches what labd writes on first discovery: hostname=product, role="unknown"
const db = createMockDb([
{ mac, hostname: "SER", role: "unknown", ip: null, status: "discovered", labels: {} },
]);
registerBastionRoutes(app, db);
const res = await app.inject({ method: "GET", url: "/api/machines" });
const body = JSON.parse(res.body);
expect(body.discovered[mac]).toBeDefined();
expect(body.installed[mac]).toBeUndefined();
await app.close();
});
it("falls back to DB for MACs not in any live bucket", async () => {
const mac = "11:22:33:44:55:66";
// No bastions connected
const app = Fastify({ logger: false });
const db = createMockDb([
{ mac, hostname: "worker1-k8s0", role: "infra", ip: "192.168.8.13", status: "online", labels: {} },
]);
registerBastionRoutes(app, db);
const res = await app.inject({ method: "GET", url: "/api/machines" });
const body = JSON.parse(res.body);
expect(body.installed[mac]).toMatchObject({
hostname: "worker1-k8s0",
role: "infra",
ip: "192.168.8.13",
});
await app.close();
});
});

View File

@@ -0,0 +1,425 @@
// End-to-end smoke tests for the v2.0 Phase 1 surface (auth bootstrap, RBAC,
// audit correlation). These exercise the wiring in createApp(): the bearer
// auth middleware, the v2 routes scope, and the AuditService lifecycle.
//
// We don't spin up CockroachDB. Instead we provide a PrismaClient-shaped
// in-memory mock that matches the surface the v2 services actually touch.
// Tests follow the project convention of using mock DBs + Fastify.inject().
import { describe, it, expect, beforeEach, afterEach, vi } from "vitest";
import bcrypt from "bcryptjs";
import { createApp } from "../src/server.js";
import type { DbClient } from "../src/server.js";
import type { AuditService } from "../src/services/audit.js";
const TEST_CONFIG = { port: 0, host: "127.0.0.1", databaseUrl: "", caDir: "/tmp", logLevel: "silent" };
interface UserRow { id: string; email: string; password: string; role: string; name: string | null; }
interface SessionRow { id: string; userId: string; token: string; expiresAt: Date; user?: UserRow; }
interface RbacDefRow { id: string; name: string; subjects: unknown; roleBindings: unknown; }
interface AuditEventRow {
id: string;
eventKind: string;
source: string;
verified: boolean;
userId: string | null;
userName: string | null;
environmentName: string | null;
resourceKind: string | null;
correlationId: string | null;
parentEventId: string | null;
details: unknown;
result: string;
error: string | null;
durationMs: number | null;
timestamp: Date;
}
interface Stores {
users: Map<string, UserRow>;
sessions: Map<string, SessionRow>;
groupMembers: Array<{ userId: string; group: { name: string } }>;
rbacDefs: RbacDefRow[];
auditEvents: AuditEventRow[];
resources: Array<Record<string, unknown>>;
}
function makeStores(): Stores {
return {
users: new Map(),
sessions: new Map(),
groupMembers: [],
rbacDefs: [],
auditEvents: [],
resources: [],
};
}
function makeMockDb(s: Stores): DbClient {
let idCounter = 0;
const newId = (prefix: string): string => `${prefix}-${++idCounter}`;
return {
$queryRaw: vi.fn(async () => [{ "?column?": 1 }]),
server: { findMany: vi.fn(async () => []), findUnique: vi.fn(), upsert: vi.fn() },
joinToken: { findUnique: vi.fn(), findMany: vi.fn(), create: vi.fn(), update: vi.fn() },
bastion: { upsert: vi.fn(), findMany: vi.fn(), findUnique: vi.fn(), update: vi.fn() },
user: {
count: vi.fn(async () => s.users.size),
findUnique: vi.fn(async (args: { where: { email?: string; id?: string } }) => {
if (args.where.email) {
for (const u of s.users.values()) if (u.email === args.where.email) return u;
}
if (args.where.id) return s.users.get(args.where.id) ?? null;
return null;
}),
create: vi.fn(async (args: { data: Omit<UserRow, "id"> }) => {
const id = newId("user");
const row: UserRow = { id, ...args.data };
s.users.set(id, row);
return row;
}),
},
session: {
findUnique: vi.fn(async (args: { where: { token?: string; id?: string }; include?: { user?: boolean } }) => {
let session: SessionRow | undefined;
if (args.where.token) {
for (const sess of s.sessions.values()) if (sess.token === args.where.token) { session = sess; break; }
} else if (args.where.id) {
session = s.sessions.get(args.where.id);
}
if (!session) return null;
if (args.include?.user) {
return { ...session, user: s.users.get(session.userId)! };
}
return session;
}),
create: vi.fn(async (args: { data: { userId: string; token: string; expiresAt: Date } }) => {
const id = newId("sess");
const row: SessionRow = { id, ...args.data };
s.sessions.set(id, row);
return row;
}),
delete: vi.fn(async (args: { where: { id: string } }) => {
s.sessions.delete(args.where.id);
return null;
}),
},
groupMember: {
findMany: vi.fn(async (args: { where: { userId: string } }) =>
s.groupMembers.filter((m) => m.userId === args.where.userId),
),
},
rbacDefinition: {
findMany: vi.fn(async () => s.rbacDefs),
},
auditEvent: {
createMany: vi.fn(async (args: { data: Array<Omit<AuditEventRow, "id" | "timestamp">> }) => {
const ts = new Date();
for (const e of args.data) {
s.auditEvents.push({ id: newId("evt"), timestamp: ts, ...e });
}
return { count: args.data.length };
}),
findMany: vi.fn(async (args: { where?: Record<string, unknown>; orderBy?: unknown; take?: number }) => {
const where = args.where ?? {};
const filtered = s.auditEvents.filter((e) => {
if (where["eventKind"] && e.eventKind !== where["eventKind"]) return false;
if (where["correlationId"] && e.correlationId !== where["correlationId"]) return false;
if (where["environmentName"] && e.environmentName !== where["environmentName"]) return false;
return true;
});
return filtered.slice(0, args.take ?? 100);
}),
},
resource: {
findMany: vi.fn(async () => s.resources),
findUnique: vi.fn(),
create: vi.fn(),
update: vi.fn(),
delete: vi.fn(),
},
environment: { findMany: vi.fn(async () => []), findUnique: vi.fn(), create: vi.fn() },
account: { findMany: vi.fn(async () => []), findUnique: vi.fn(), create: vi.fn() },
binding: { findMany: vi.fn(async () => []), create: vi.fn() },
} as unknown as DbClient;
}
async function buildApp(s: Stores) {
const db = makeMockDb(s);
const result = await createApp(TEST_CONFIG, db);
await result.app.ready();
return result;
}
describe("v2 auth: bootstrap flow", () => {
let stores: Stores;
let app: Awaited<ReturnType<typeof buildApp>>["app"];
let auditService: AuditService;
beforeEach(async () => {
stores = makeStores();
const built = await buildApp(stores);
app = built.app;
auditService = built.auditService;
});
afterEach(async () => {
await app.close(); // triggers auditService.stop()
});
it("first login with no users seeds the admin and returns a session token", async () => {
expect(stores.users.size).toBe(0);
const resp = await app.inject({
method: "POST",
url: "/api/auth/login",
payload: { email: "admin@itaz.eu", password: "s3cret-pw" },
});
expect(resp.statusCode).toBe(200);
const body = resp.json();
expect(body.isBootstrap).toBe(true);
expect(body.token).toMatch(/^[a-f0-9]{64}$/);
expect(typeof body.expiresAt).toBe("string");
expect(stores.users.size).toBe(1);
const created = [...stores.users.values()][0]!;
expect(created.email).toBe("admin@itaz.eu");
expect(created.role).toBe("ADMIN");
// Password is hashed, not stored plaintext.
expect(created.password).not.toBe("s3cret-pw");
expect(await bcrypt.compare("s3cret-pw", created.password)).toBe(true);
// Bootstrap emits an audit event.
await auditService.flushPending();
const bootstrapEvents = stores.auditEvents.filter((e) => e.eventKind === "auth_bootstrap");
expect(bootstrapEvents).toHaveLength(1);
expect(bootstrapEvents[0]!.result).toBe("success");
expect(bootstrapEvents[0]!.userName).toBe("admin@itaz.eu");
});
it("returns 400 for missing credentials", async () => {
const resp = await app.inject({ method: "POST", url: "/api/auth/login", payload: {} });
expect(resp.statusCode).toBe(400);
});
it("second login uses normal flow (no isBootstrap)", async () => {
// Bootstrap once
await app.inject({
method: "POST",
url: "/api/auth/login",
payload: { email: "admin@itaz.eu", password: "s3cret-pw" },
});
expect(stores.users.size).toBe(1);
// Login again
const resp = await app.inject({
method: "POST",
url: "/api/auth/login",
payload: { email: "admin@itaz.eu", password: "s3cret-pw" },
});
expect(resp.statusCode).toBe(200);
expect(resp.json().isBootstrap).toBe(false);
expect(stores.users.size).toBe(1); // no new user
});
it("rejects wrong password with 401", async () => {
// Seed admin
await app.inject({
method: "POST",
url: "/api/auth/login",
payload: { email: "admin@itaz.eu", password: "s3cret-pw" },
});
const resp = await app.inject({
method: "POST",
url: "/api/auth/login",
payload: { email: "admin@itaz.eu", password: "wrong" },
});
expect(resp.statusCode).toBe(401);
// Failed login is also audited.
await auditService.flushPending();
const fails = stores.auditEvents.filter((e) => e.eventKind === "auth_login" && e.result === "failure");
expect(fails).toHaveLength(1);
});
});
describe("v2 RBAC: env-scoped denial", () => {
let stores: Stores;
let app: Awaited<ReturnType<typeof buildApp>>["app"];
async function seedSession(role: string): Promise<string> {
stores.users.set("u-1", {
id: "u-1",
email: `${role.toLowerCase()}@itaz.eu`,
password: "x",
role,
name: null,
});
const token = "test-token-" + role;
stores.sessions.set("s-1", {
id: "s-1",
userId: "u-1",
token,
expiresAt: new Date(Date.now() + 86_400_000),
});
return token;
}
beforeEach(async () => {
stores = makeStores();
app = (await buildApp(stores)).app;
});
afterEach(async () => {
await app.close();
});
it("non-admin user with no role bindings gets 403 on /api/resources", async () => {
const token = await seedSession("EDITOR"); // not admin, no bindings
const resp = await app.inject({
method: "GET",
url: "/api/resources",
headers: { authorization: `Bearer ${token}` },
});
expect(resp.statusCode).toBe(403);
expect(resp.json().error).toMatch(/no matching role binding/);
});
it("missing/empty bearer token gets 401 (auth, not RBAC)", async () => {
const r1 = await app.inject({ method: "GET", url: "/api/resources" });
expect(r1.statusCode).toBe(401);
const r2 = await app.inject({
method: "GET",
url: "/api/resources",
headers: { authorization: "Bearer " },
});
expect(r2.statusCode).toBe(401);
});
it("invalid bearer token gets 401", async () => {
const resp = await app.inject({
method: "GET",
url: "/api/resources",
headers: { authorization: "Bearer not-a-real-token" },
});
expect(resp.statusCode).toBe(401);
});
it("admin role bypasses RBAC", async () => {
const token = await seedSession("ADMIN");
const resp = await app.inject({
method: "GET",
url: "/api/resources",
headers: { authorization: `Bearer ${token}` },
});
expect(resp.statusCode).toBe(200);
expect(resp.json()).toEqual([]);
});
it("user with binding for env A is denied for resources in env B", async () => {
const token = await seedSession("EDITOR");
stores.groupMembers.push({ userId: "u-1", group: { name: "team-a" } });
stores.rbacDefs.push({
id: "rbac-1",
name: "team-a-edit-on-env-a",
subjects: [{ kind: "Group", name: "team-a" }],
roleBindings: [{ role: "edit", environment: "env-a" }],
});
// List in env-a → should pass RBAC (no env query so it's global view, but
// the binding scope is environment-specific → for global list the binding
// doesn't apply when an environment scope is set on the binding).
// Smoke test the targeted denial: trying to create in env-b is rejected.
const respB = await app.inject({
method: "POST",
url: "/api/resources",
headers: { authorization: `Bearer ${token}` },
payload: { kind: "database", name: "x", environmentId: "env-b", accountId: "acc-1" },
});
expect(respB.statusCode).toBe(403);
expect(respB.json().error).toMatch(/no matching role binding/);
});
});
describe("v2 audit: correlation chain visible via /api/events", () => {
let stores: Stores;
let app: Awaited<ReturnType<typeof buildApp>>["app"];
let auditService: AuditService;
beforeEach(async () => {
stores = makeStores();
const built = await buildApp(stores);
app = built.app;
auditService = built.auditService;
});
afterEach(async () => {
await app.close();
});
it("emitted audit events are queryable by correlation id", async () => {
// Seed admin so /api/events is accessible (it sits behind bearer auth)
const loginResp = await app.inject({
method: "POST",
url: "/api/auth/login",
payload: { email: "admin@itaz.eu", password: "pw" },
});
const token = loginResp.json().token;
// Force flush so the bootstrap event is in the DB
await auditService.flushPending();
expect(stores.auditEvents.length).toBeGreaterThan(0);
const bootstrap = stores.auditEvents.find((e) => e.eventKind === "auth_bootstrap")!;
expect(bootstrap.correlationId).toMatch(/^corr_[a-f0-9]{16}$/);
// Query /api/events filtered by correlation id
const queryResp = await app.inject({
method: "GET",
url: `/api/events?correlation=${bootstrap.correlationId}`,
headers: { authorization: `Bearer ${token}` },
});
expect(queryResp.statusCode).toBe(200);
const events = queryResp.json() as Array<{ correlationId: string; eventKind: string }>;
expect(events.length).toBe(1);
expect(events[0]!.eventKind).toBe("auth_bootstrap");
expect(events[0]!.correlationId).toBe(bootstrap.correlationId);
});
it("explicit parent/child correlation chain is preserved across emits", async () => {
const correlationId = auditService.createCorrelation();
auditService.emit({
eventKind: "test_parent",
source: "test",
result: "success",
correlationId,
});
auditService.emit({
eventKind: "test_child",
source: "test",
result: "success",
correlationId,
parentEventId: "evt-1",
});
await auditService.flushPending();
const chain = stores.auditEvents.filter((e) => e.correlationId === correlationId);
expect(chain).toHaveLength(2);
expect(chain.map((e) => e.eventKind).sort()).toEqual(["test_child", "test_parent"]);
expect(chain.find((e) => e.eventKind === "test_child")!.parentEventId).toBe("evt-1");
});
});

View File

@@ -1,18 +1,22 @@
// Hardening: Pod Security Standards, certificate check, log rotation.
// Hardening: Pod Security Standards, certificate check, journald cap, storage.
import type { OperationContext, OperationResult, OperationGroup } from "../types.js";
import { runSequential } from "../utils.js";
import { applyPodSecurityStandards } from "../operations/pod-security.js";
import { checkCertExpiry } from "../operations/cert-check.js";
import { configureLogRotation } from "../operations/log-rotation.js";
import { configureJournaldLimits } from "../operations/journald-limits.js";
import { configureLonghornDisk } from "../operations/longhorn-disk.js";
export const hardeningGroup: OperationGroup = {
name: "hardening",
description: "Pod security, certificate check, log rotation",
description: "Pod security, certificate check, journald cap, storage",
operations: [
{ name: "Apply Pod Security Standards", fn: applyPodSecurityStandards },
{ name: "Check certificate expiry", fn: checkCertExpiry },
{ name: "Configure log rotation", fn: configureLogRotation },
{ name: "Decommission file-based audit logs", fn: configureLogRotation },
{ name: "Configure journald disk cap", fn: configureJournaldLimits },
{ name: "Configure Longhorn disk", fn: configureLonghornDisk },
],
};

View File

@@ -1,22 +1,26 @@
// Host preparation: kernel modules, sysctl, swap, firewall, SELinux.
// Host preparation: kernel modules, sysctl, swap, storage, firewall, SELinux.
import type { OperationContext, OperationResult, OperationGroup } from "../types.js";
import { runSequential } from "../utils.js";
import { loadKernelModules } from "../operations/kernel-modules.js";
import { applyCisHardening } from "../operations/sysctl.js";
import { disableSwap } from "../operations/swap.js";
import { enableSwap } from "../operations/swap.js";
import { growRancherLv } from "../operations/rancher-storage.js";
import { disableFirewall } from "../operations/firewall.js";
import { setSelinuxPermissive } from "../operations/selinux.js";
import { enableIscsi } from "../operations/iscsi.js";
export const hostPrepGroup: OperationGroup = {
name: "host-prep",
description: "Prepare host for k3s: kernel modules, sysctl, swap, firewall, SELinux",
description: "Prepare host for k3s: kernel modules, sysctl, swap, imageFs sizing, firewall, SELinux, iSCSI",
operations: [
{ name: "Load kernel modules", fn: loadKernelModules },
{ name: "Apply CIS sysctl", fn: applyCisHardening },
{ name: "Disable swap", fn: disableSwap },
{ name: "Enable swap", fn: enableSwap },
{ name: "Grow rancher LV", fn: growRancherLv },
{ name: "Disable firewall", fn: disableFirewall },
{ name: "Set SELinux permissive", fn: setSelinuxPermissive },
{ name: "Enable iSCSI", fn: enableIscsi },
],
};

View File

@@ -3,6 +3,8 @@
import type { OperationContext, OperationResult, OperationGroup } from "../types.js";
import { runSequential } from "../utils.js";
import { installCilium } from "../operations/cilium.js";
import { installMultus } from "../operations/multus.js";
import { installVlanSetup } from "../operations/vlan-setup.js";
import { fixCoreDnsUpstream } from "../operations/dns-fix.js";
import { applyDefaultNetworkPolicies } from "../operations/network-policy.js";
@@ -11,6 +13,11 @@ export const networkingGroup: OperationGroup = {
description: "Install Cilium CNI, fix DNS, apply network policies",
operations: [
{ name: "Install Cilium CNI", fn: installCilium },
// Multus + vlan-setup: give pods a second interface on VLAN 10 (macvlan)
// for LAN device discovery (Matter/HomeKit mDNS). Must follow Cilium
// (needs cni.exclusive=false + bpf.vlanBypass={10} from installCilium).
{ name: "Install Multus CNI", fn: installMultus },
{ name: "Install vlan-setup (lan10 + CNI plugins)", fn: installVlanSetup },
{ name: "Fix CoreDNS upstream", fn: fixCoreDnsUpstream },
{ name: "Apply network policies", fn: applyDefaultNetworkPolicies },
],

View File

@@ -76,7 +76,6 @@ sed -i 's/^SELINUX=enforcing/SELINUX=permissive/' /etc/selinux/config 2>/dev/nul
# ── 5b. Create k3s config directory ──
echo "[5/10] Writing k3s server configuration..."
mkdir -p /etc/rancher/k3s
mkdir -p /var/log/kubernetes
cat > /etc/rancher/k3s/config.yaml << 'K3S_CONFIG'
# k3s server configuration — CIS hardened
@@ -91,13 +90,10 @@ disable:
- servicelb
- traefik
# API server hardening
# API server hardening (audit-log-path=- routes audit to journald via stdout)
kube-apiserver-arg:
- "anonymous-auth=false"
- "audit-log-path=/var/log/kubernetes/audit.log"
- "audit-log-maxage=30"
- "audit-log-maxbackup=10"
- "audit-log-maxsize=100"
- "audit-log-path=-"
- "audit-policy-file=/etc/rancher/k3s/audit-policy.yaml"
- "enable-admission-plugins=NodeRestriction,PodSecurity"
- "request-timeout=300s"
@@ -219,7 +215,9 @@ echo " Using network device: \$DEFAULT_DEV"
KUBECONFIG=/etc/rancher/k3s/k3s.yaml cilium install \\
--set kubeProxyReplacement=true \\
--set ipam.mode=kubernetes \\
--set ipam.mode=cluster-pool \\
--set ipam.operator.clusterPoolIPv4PodCIDRList='{10.42.0.0/16}' \\
--set ipam.operator.clusterPoolIPv4MaskSize=24 \\
--set devices="\$DEFAULT_DEV" \\
--set nodePort.directRoutingDevice="\$DEFAULT_DEV"

View File

@@ -78,9 +78,10 @@ export class K3sModule implements Module {
return toModuleResult("install", [...prepResults, ...k3sResults], start);
}
// Phase 3: Networking (server only — agents don't install Cilium)
// Phase 3: Networking (initial server only — joining servers get Cilium via daemonset)
let netResults: OperationResult[] = [];
if (isServer) {
const isJoiningServer = isServer && !!opCtx.config.k3sServerUrl;
if (isServer && !isJoiningServer) {
netResults = await runNetworking(opCtx);
}

View File

@@ -35,21 +35,25 @@ export const installCilium: Operation = async (ctx): Promise<OperationResult> =>
}
details.push(`Installed cilium CLI ${version} (${cliArch})`);
// Detect default network device (avoid tailscale/wireguard)
const devResult = await ctx.ssh.exec(
"ip -4 route show default | awk '{print $5}' | head -1",
sshOpts(ctx),
);
const defaultDev = devResult.stdout.trim();
details.push(`Network device: ${defaultDev}`);
// Install Cilium
// - No hardcoded devices: Cilium auto-detects per node (heterogeneous NICs like eno1 vs enP7s7)
// - k8sServiceHost/Port: k3s agents proxy the API on 127.0.0.1:6444 (not 6443)
// - cni.exclusive=false: required so Multus can install its CNI config alongside
// Cilium (Cilium otherwise deletes any non-Cilium CNI conf).
// - bpf.vlanBypass={10}: allow VLAN 10 (LoT) tagged traffic through the eBPF
// host VLAN filter, so pods on a macvlan/VLAN-10 interface receive multicast
// (Matter/mDNS ff02::fb + 224.0.0.251). Without this Cilium drops it
// ("VLAN traffic disallowed by VLAN filter", bpf_host.c).
const installResult = await ctx.ssh.exec(
`KUBECONFIG=/etc/rancher/k3s/k3s.yaml cilium install \
--set kubeProxyReplacement=true \
--set ipam.mode=kubernetes \
--set devices="${defaultDev}" \
--set nodePort.directRoutingDevice="${defaultDev}"`,
--set ipam.mode=cluster-pool \
--set ipam.operator.clusterPoolIPv4PodCIDRList='{10.42.0.0/16}' \
--set ipam.operator.clusterPoolIPv4MaskSize=24 \
--set k8sServiceHost=127.0.0.1 \
--set k8sServicePort=6444 \
--set cni.exclusive=false \
--set bpf.vlanBypass="{10}"`,
{ timeoutMs: 300_000 },
);
if (installResult.exitCode !== 0) {

View File

@@ -0,0 +1,194 @@
// Recover a broken etcd member by removing it from the cluster, wiping its
// local state, and restarting k3s so it rejoins as a fresh member.
//
// Use case: a node panics on startup with
// "tocommit(N+1) is out of range [lastIndex(N)]. Was the raft log corrupted,
// truncated, or lost?"
// This means the local raft WAL is missing the last entry the leader thinks
// the follower acknowledged (lost write, unclean shutdown, etc). The fix is
// always the same and well-documented; this codifies it so we don't fumble
// the procedure under pressure.
//
// Preconditions:
// - At least one healthy peer is reachable so the cluster has quorum after
// we remove the broken member. (For a 3-node cluster: 2 healthy. For a
// 5-node: 3 healthy.) If quorum would be lost, this function refuses.
// - SSH access to both the broken node and a healthy peer.
// - etcdctl available on the healthy peer (k3s does not bundle it; the
// procedure installs it on demand on Fedora).
import type { SshClient } from "../types.js";
const ETCD_TLS = {
ca: "/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt",
cert: "/var/lib/rancher/k3s/server/tls/etcd/server-client.crt",
key: "/var/lib/rancher/k3s/server/tls/etcd/server-client.key",
} as const;
const SSH_TIMEOUT = 60_000;
export interface RecoverEtcdMemberOptions {
/** SSH client for the broken node (the one panicking). */
broken: SshClient;
/** SSH client for any healthy server peer in the same cluster. */
peer: SshClient;
/** Hostname (k8s node name) of the broken node. Used to find its etcd member id. */
brokenHostname: string;
/** Logger for progress output. */
log?: (msg: string) => void;
}
export interface RecoverEtcdMemberResult {
success: boolean;
changed: boolean;
message: string;
/** New etcd member id assigned after rejoin (when known). */
newMemberId?: string;
/** Old etcd member id that was removed. */
removedMemberId?: string;
error?: string;
}
function etcdctl(subcmd: string): string {
return [
"ETCDCTL_API=3 etcdctl",
`--cacert=${ETCD_TLS.ca}`,
`--cert=${ETCD_TLS.cert}`,
`--key=${ETCD_TLS.key}`,
"--endpoints=https://127.0.0.1:2379",
"--command-timeout=10s",
subcmd,
].join(" ");
}
async function ensureEtcdctl(peer: SshClient): Promise<void> {
const probe = await peer.exec("command -v etcdctl 2>/dev/null", { timeoutMs: 5_000 });
if (probe.exitCode === 0 && probe.stdout.trim()) return;
// Best-effort install on Fedora. If the host isn't dnf-based, surface the
// error to the caller via the next etcdctl invocation.
await peer.exec("dnf install -y etcd 2>&1", { timeoutMs: 120_000 });
}
async function getMemberList(peer: SshClient): Promise<Array<{ id: string; name: string }>> {
const result = await peer.exec(etcdctl("member list"), { timeoutMs: SSH_TIMEOUT });
if (result.exitCode !== 0) {
throw new Error(`etcdctl member list failed: ${result.stderr || result.stdout}`);
}
// Format: <hex-id>, started, <name>, <peer-urls>, <client-urls>, <isLearner>
return result.stdout
.split("\n")
.map((line) => line.trim())
.filter(Boolean)
.map((line) => {
const [id, , name] = line.split(",").map((p) => p.trim());
return { id: id ?? "", name: name ?? "" };
})
.filter((m) => m.id);
}
export async function recoverEtcdMember(
opts: RecoverEtcdMemberOptions,
): Promise<RecoverEtcdMemberResult> {
const log = opts.log ?? (() => {});
try {
log(`Looking up etcd member id for ${opts.brokenHostname} via peer...`);
await ensureEtcdctl(opts.peer);
const members = await getMemberList(opts.peer);
if (members.length < 3) {
return {
success: false,
changed: false,
message: "Refusing to remove a member from a cluster with <3 members (quorum would be lost)",
error: `member count = ${members.length}`,
};
}
// Member names are <hostname>-<random-suffix>; match by hostname prefix.
const broken = members.find((m) => m.name.startsWith(opts.brokenHostname));
if (!broken) {
return {
success: false,
changed: false,
message: `No etcd member found matching hostname ${opts.brokenHostname}`,
error: `members: ${members.map((m) => m.name).join(", ")}`,
};
}
log(`Broken member: ${broken.id} (${broken.name})`);
log("Step 1/4: stopping k3s on broken node");
await opts.broken.exec("systemctl stop k3s 2>&1", { timeoutMs: SSH_TIMEOUT });
log("Step 2/4: removing broken etcd member from cluster");
const remove = await opts.peer.exec(
etcdctl(`member remove ${broken.id}`),
{ timeoutMs: SSH_TIMEOUT },
);
if (remove.exitCode !== 0) {
return {
success: false,
changed: false,
message: "etcdctl member remove failed",
error: remove.stderr || remove.stdout,
removedMemberId: broken.id,
};
}
log("Step 3/4: archiving corrupt etcd state and stale TLS/cred dirs on broken node");
const ts = Math.floor(Date.now() / 1000);
await opts.broken.exec(
[
`mv /var/lib/rancher/k3s/server/db /var/lib/rancher/k3s/server/db.corrupt-${ts} 2>/dev/null || true`,
"rm -rf /var/lib/rancher/k3s/server/tls /var/lib/rancher/k3s/server/cred",
].join(" && "),
{ timeoutMs: SSH_TIMEOUT },
);
log("Step 4/4: starting k3s on broken node — it will rejoin");
await opts.broken.exec("systemctl start k3s 2>&1", { timeoutMs: SSH_TIMEOUT });
// Poll for rejoin. The new member-id is what the cluster assigns on join.
let newMemberId: string | undefined;
for (let i = 0; i < 60; i++) {
await new Promise((r) => setTimeout(r, 5_000));
try {
const after = await getMemberList(opts.peer);
const rejoined = after.find(
(m) => m.name.startsWith(opts.brokenHostname) && m.id !== broken.id,
);
if (rejoined) {
newMemberId = rejoined.id;
break;
}
} catch {
// peer may briefly be unreachable mid-rejoin — keep polling
}
}
if (!newMemberId) {
return {
success: false,
changed: true,
message: "k3s started but new member did not appear in cluster within 5 minutes",
removedMemberId: broken.id,
};
}
log(`Rejoined as ${newMemberId}`);
return {
success: true,
changed: true,
message: `Recovered: removed ${broken.id}, rejoined as ${newMemberId}`,
removedMemberId: broken.id,
newMemberId,
};
} catch (err) {
return {
success: false,
changed: false,
message: "Recovery failed",
error: err instanceof Error ? err.message : String(err),
};
}
}

View File

@@ -1,6 +1,8 @@
export { loadKernelModules } from "./kernel-modules.js";
export { applyCisHardening } from "./sysctl.js";
export { disableSwap } from "./swap.js";
export { enableSwap } from "./swap.js";
export { growRancherLv } from "./rancher-storage.js";
export { enableIscsi } from "./iscsi.js";
export { disableFirewall } from "./firewall.js";
export { setSelinuxPermissive } from "./selinux.js";
export { writeK3sConfig } from "./k3s-config.js";
@@ -8,8 +10,17 @@ export { writeAuditPolicy } from "./audit-policy.js";
export { cleanupStaleCni } from "./cni-cleanup.js";
export { installK3sBinary } from "./k3s-install.js";
export { installCilium } from "./cilium.js";
export { installMultus } from "./multus.js";
export { installVlanSetup } from "./vlan-setup.js";
export { fixCoreDnsUpstream } from "./dns-fix.js";
export { configureLogRotation } from "./log-rotation.js";
export { configureJournaldLimits } from "./journald-limits.js";
export { applyDefaultNetworkPolicies } from "./network-policy.js";
export { applyPodSecurityStandards } from "./pod-security.js";
export { checkCertExpiry } from "./cert-check.js";
export { configureLonghornDisk } from "./longhorn-disk.js";
export { recoverEtcdMember } from "./etcd-recover.js";
export type {
RecoverEtcdMemberOptions,
RecoverEtcdMemberResult,
} from "./etcd-recover.js";

View File

@@ -0,0 +1,31 @@
// Install and enable iSCSI initiator (required by Longhorn storage).
// Fedora: iscsi-initiator-utils, Ubuntu: open-iscsi
import type { Operation, OperationResult } from "../types.js";
import { sshOpts } from "../utils.js";
export const enableIscsi: Operation = async (ctx): Promise<OperationResult> => {
// Check if iscsid is already running
const check = await ctx.ssh.exec("systemctl is-active iscsid 2>/dev/null", sshOpts(ctx));
if (check.stdout.trim() === "active") {
return { success: true, changed: false, message: "iSCSI already active" };
}
// Install the package (detect distro)
const osRelease = await ctx.ssh.exec("cat /etc/os-release", sshOpts(ctx));
const osLower = osRelease.stdout.toLowerCase();
const isFedora = osLower.includes("fedora") || osLower.includes("rhel") || osLower.includes("centos");
const pkg = isFedora ? "iscsi-initiator-utils" : "open-iscsi";
const installCmd = isFedora ? `sudo dnf install -y ${pkg}` : `sudo apt-get install -y ${pkg}`;
const install = await ctx.ssh.exec(installCmd, { timeoutMs: 120_000 });
if (install.exitCode !== 0) {
return { success: false, changed: false, message: `Failed to install ${pkg}`, error: install.stderr.trim() };
}
// Enable and start
await ctx.ssh.exec("sudo systemctl enable --now iscsid", sshOpts(ctx));
return { success: true, changed: true, message: `Installed ${pkg} and enabled iscsid` };
};

View File

@@ -0,0 +1,33 @@
// Cap journald disk usage so audit logs (which now flow through journald via
// kube-apiserver's stdout) cannot fill /var/log. Default journald uses up to
// 10% of the filesystem, capped at 4 GB. In a /var/log of ~10 GB shared with
// other services, that's still room for audit volume to evict useful logs.
// 2 GB / 200 MB-per-file is a comfortable middle.
import type { Operation, OperationResult } from "../types.js";
import { sshOpts, writeRemoteFile } from "../utils.js";
const DROPIN_CONTENT = `[Journal]
SystemMaxUse=2G
SystemKeepFree=1G
SystemMaxFileSize=200M
`;
const DROPIN_PATH = "/etc/systemd/journald.conf.d/10-k3s-audit-cap.conf";
export const configureJournaldLimits: Operation = async (ctx): Promise<OperationResult> => {
const changed = await writeRemoteFile(ctx, DROPIN_PATH, DROPIN_CONTENT);
if (changed) {
// Reload journald so the new limit applies without a reboot.
await ctx.ssh.exec(
"systemctl kill --signal=SIGUSR2 systemd-journald 2>/dev/null; " +
"systemctl restart systemd-journald 2>&1 || true",
sshOpts(ctx),
);
}
return {
success: true,
changed,
message: changed ? "journald limits configured (2 GB cap)" : "journald limits already configured",
};
};

View File

@@ -9,7 +9,18 @@ function isServerRole(role: string): boolean {
function generateServerConfig(config: K3sConfig): string {
const tlsSans = [config.hostname, config.ip, ...(config.tlsSans ?? [])];
return `# k3s server configuration — CIS hardened
const isJoining = !!config.k3sServerUrl;
const clusterLines = isJoining
? `server: "${config.k3sServerUrl}"\ntoken: "${config.k3sToken}"`
: "cluster-init: true";
// audit-log-path=- routes audit events to k3s.service's stdout, which systemd
// forwards to journald. journald enforces its own size caps (see
// configureJournaldLimits) so audit volume cannot fill the disk. File-based
// audit logs led to /var/log/kubernetes growing to 7+ GB because apiserver's
// own rotation produced files that any logrotate glob would double-rotate
// and never expire.
return `# k3s server configuration — CIS hardened, etcd HA
${clusterLines}
protect-kernel-defaults: true
secrets-encryption: true
write-kubeconfig-mode: "0640"
@@ -20,12 +31,12 @@ disable:
- servicelb
- traefik
node-label:
- "node.longhorn.io/create-default-disk=config"
kube-apiserver-arg:
- "anonymous-auth=false"
- "audit-log-path=/var/log/kubernetes/audit.log"
- "audit-log-maxage=30"
- "audit-log-maxbackup=10"
- "audit-log-maxsize=100"
- "audit-log-path=-"
- "audit-policy-file=/etc/rancher/k3s/audit-policy.yaml"
- "enable-admission-plugins=NodeRestriction,PodSecurity"
- "request-timeout=300s"
@@ -42,6 +53,9 @@ ${tlsSans.map((s) => ` - "${s}"`).join("\n")}
function generateAgentConfig(): string {
return `protect-kernel-defaults: true
node-label:
- "node-role.kubernetes.io/worker=true"
- "node.longhorn.io/create-default-disk=config"
kubelet-arg:
- "protect-kernel-defaults=true"
- "streaming-connection-idle-timeout=5m"
@@ -50,7 +64,7 @@ kubelet-arg:
}
export const writeK3sConfig: Operation = async (ctx): Promise<OperationResult> => {
await ctx.ssh.exec("mkdir -p /etc/rancher/k3s /var/log/kubernetes", sshOpts(ctx));
await ctx.ssh.exec("mkdir -p /etc/rancher/k3s", sshOpts(ctx));
const content = isServerRole(ctx.config.role)
? generateServerConfig(ctx.config)

View File

@@ -15,8 +15,21 @@ export const installK3sBinary: Operation = async (ctx): Promise<OperationResult>
const alreadyInstalled = version.exitCode === 0;
if (isServer) {
// Clean stale server state when joining an existing cluster
// (TLS certs from a previous run cause "newer than datastore" fatal error)
if (ctx.config.k3sServerUrl && ctx.config.k3sToken) {
await ctx.ssh.exec(
"rm -rf /var/lib/rancher/k3s/server/tls /var/lib/rancher/k3s/server/cred /var/lib/rancher/k3s/server/db",
sshOpts(ctx),
);
}
// If joining an existing cluster, pass K3S_URL and K3S_TOKEN
const joinEnv = ctx.config.k3sServerUrl && ctx.config.k3sToken
? `K3S_URL="${ctx.config.k3sServerUrl}" K3S_TOKEN="${ctx.config.k3sToken}"`
: "";
const result = await ctx.ssh.exec(
'curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="server" INSTALL_K3S_SKIP_SELINUX_RPM=true sh -',
`curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="server" INSTALL_K3S_SKIP_SELINUX_RPM=true ${joinEnv} sh -`,
{ timeoutMs: 300_000 },
);
if (result.exitCode !== 0) {

View File

@@ -1,25 +1,44 @@
// Configure log rotation for k3s.
// Decommission file-based k8s audit logging in favor of journald.
//
// Earlier versions wrote audit events to /var/log/kubernetes/audit.log and
// rotated them with a logrotate rule. Two failure modes followed: kube-apiserver
// rotated internally (audit-{ts}.log), the *.log glob in logrotate
// double-rotated those (-{date}), and the resulting filename matched no
// retention policy, so the directory grew unbounded (we observed 7+ GB).
//
// k3s now sets audit-log-path=- so audit goes to stdout → journald, which
// enforces SystemMaxUse caps. This operation removes the obsolete logrotate
// rule and reaps any audit files left behind by the old setup. Idempotent: on
// fresh installs everything is already absent and the operation is a no-op.
import type { Operation, OperationResult } from "../types.js";
import { writeRemoteFile } from "../utils.js";
import { sshOpts } from "../utils.js";
const LOGROTATE_CONFIG = `/var/log/kubernetes/*.log {
daily
rotate 14
compress
delaycompress
missingok
notifempty
copytruncate
maxsize 100M
}`;
const REMOVE_LOGROTATE = "rm -f /etc/logrotate.d/k3s";
// Bounded by a max-depth and explicit name pattern so we never reach outside
// the deprecated audit-log directory.
const REAP_OLD_AUDIT_FILES =
"find /var/log/kubernetes -maxdepth 1 -type f " +
"\\( -name 'audit*.log*' -o -name 'audit-*.log' \\) " +
"-delete 2>/dev/null; " +
"rmdir /var/log/kubernetes 2>/dev/null; true";
export const configureLogRotation: Operation = async (ctx): Promise<OperationResult> => {
const changed = await writeRemoteFile(ctx, "/etc/logrotate.d/k3s", LOGROTATE_CONFIG);
const before = await ctx.ssh.exec(
"test -e /etc/logrotate.d/k3s -o -d /var/log/kubernetes && echo present || echo absent",
sshOpts(ctx),
);
const wasPresent = before.stdout.trim() === "present";
await ctx.ssh.exec(REMOVE_LOGROTATE, sshOpts(ctx));
await ctx.ssh.exec(REAP_OLD_AUDIT_FILES, sshOpts(ctx));
return {
success: true,
changed,
message: changed ? "Log rotation configured" : "Log rotation already configured",
changed: wasPresent,
message: wasPresent
? "Removed legacy file-based audit logging (now via journald)"
: "No legacy audit log artifacts present",
};
};

View File

@@ -0,0 +1,50 @@
// Annotate nodes with Longhorn default disk config when /var/lib/longhorn exists.
// The label is set in k3s config (node-label), but the annotation must be applied via kubectl.
import type { Operation, OperationResult } from "../types.js";
import { sshOpts } from "../utils.js";
import { sshExec as remoteSshExec } from "../../../../src/ssh.js";
export const configureLonghornDisk: Operation = async (ctx): Promise<OperationResult> => {
// Check if /var/lib/longhorn exists on this node
const check = await ctx.ssh.exec("test -d /var/lib/longhorn && echo yes || echo no", sshOpts(ctx));
if (check.stdout.trim() !== "yes") {
return { success: true, changed: false, message: "No /var/lib/longhorn directory — skipping Longhorn disk config" };
}
// Find the node name (hostname as registered in k3s)
const nodeNameResult = await ctx.ssh.exec("hostname -f 2>/dev/null || hostname", sshOpts(ctx));
const nodeName = nodeNameResult.stdout.trim();
const annotation = JSON.stringify([{ path: "/var/lib/longhorn", allowScheduling: true }]);
// Try kubectl locally first (works on server nodes)
const result = await ctx.ssh.exec(
`k3s kubectl annotate node "${nodeName}" "node.longhorn.io/default-disks-config=${annotation}" --overwrite 2>&1 || true`,
sshOpts(ctx),
);
if (result.stdout.includes("annotated") || result.stdout.includes("unchanged")) {
return { success: true, changed: true, message: `Longhorn disk annotation applied to ${nodeName}` };
}
// For worker/agent nodes without local kubectl: apply via the server
if (ctx.config.k3sServerUrl) {
// The CLI has SSH access to the server — use sshExec from there
const serverHost = new URL(ctx.config.k3sServerUrl).hostname;
try {
const remoteResult = await remoteSshExec(
serverHost, "root",
`k3s kubectl annotate node "${nodeName}" "node.longhorn.io/default-disks-config=${annotation}" --overwrite`,
{ ...(ctx.ssh.keyPath ? { keyPath: ctx.ssh.keyPath } : {}), timeoutMs: 15_000 },
);
if (remoteResult.stdout.includes("annotated") || remoteResult.stdout.includes("unchanged")) {
return { success: true, changed: true, message: `Longhorn disk annotation applied to ${nodeName} (via server)` };
}
} catch {
// Fall through to manual instruction
}
}
return { success: true, changed: false, message: "Longhorn disk label set (annotation requires server kubectl)" };
};

View File

@@ -0,0 +1,34 @@
// Install Multus CNI (thick plugin) — the meta-CNI that lets pods attach an
// extra interface (macvlan on VLAN 10) alongside Cilium, via a
// NetworkAttachmentDefinition. Required for Home Assistant's LAN presence
// (Matter/HomeKit mDNS discovery). Cilium must be installed with
// cni.exclusive=false first (see cilium.ts) or it deletes Multus's CNI conf.
import type { Operation, OperationResult } from "../types.js";
import { sshOpts } from "../utils.js";
const MULTUS_VERSION = "v4.1.4";
const MULTUS_MANIFEST = `https://raw.githubusercontent.com/k8snetworkplumbingwg/multus-cni/${MULTUS_VERSION}/deployments/multus-daemonset-thick.yml`;
export const installMultus: Operation = async (ctx): Promise<OperationResult> => {
const K = "KUBECONFIG=/etc/rancher/k3s/k3s.yaml";
// Idempotent: skip if the Multus DaemonSet is already present.
const check = await ctx.ssh.exec(
`${K} kubectl -n kube-system get ds kube-multus-ds -o name 2>/dev/null`,
sshOpts(ctx),
);
if (check.exitCode === 0 && check.stdout.includes("kube-multus-ds")) {
return { success: true, changed: false, message: `Multus already installed (${MULTUS_VERSION})` };
}
const apply = await ctx.ssh.exec(
`${K} kubectl apply -f ${MULTUS_MANIFEST}`,
{ ...sshOpts(ctx), timeoutMs: 120_000 },
);
if (apply.exitCode !== 0) {
return { success: false, changed: false, message: "Failed to apply Multus manifest", error: apply.stderr };
}
return { success: true, changed: true, message: `Installed Multus ${MULTUS_VERSION} (thick)` };
};

View File

@@ -0,0 +1,48 @@
// Grow the labvg/rancher LV (k3s image store / imageFs) to 120G.
// 2026-08 incident: the original 20G LV sat at 85% used from steady-state
// images alone, so one ~5G image pull tripped imagefs eviction and evicted
// unrelated pods. Fresh installs are sized at 120G by the kickstart; this op
// covers nodes installed before that change and vanilla nodes converted to
// k8s later. Never removes or shrinks anything — if the VG lacks free space
// (e.g. a longhorn --grow LV consumed it), it reports and moves on.
import type { Operation, OperationResult } from "../types.js";
import { sshOpts } from "../utils.js";
const RANCHER_LV = "labvg/rancher";
const TARGET_MIB = 122880; // 120G
export const growRancherLv: Operation = async (ctx): Promise<OperationResult> => {
const lv = await ctx.ssh.exec(
`lvs --noheadings --units m --nosuffix -o lv_size ${RANCHER_LV} 2>/dev/null || true`,
sshOpts(ctx),
);
const sizeMib = Number.parseFloat(lv.stdout.trim());
if (Number.isNaN(sizeMib)) {
return { success: true, changed: false, message: "No labvg/rancher LV — imageFs shares /var, skipping" };
}
if (sizeMib >= TARGET_MIB) {
return { success: true, changed: false, message: `rancher LV already ${Math.round(sizeMib / 1024)}G` };
}
const vg = await ctx.ssh.exec(`vgs --noheadings --units m --nosuffix -o vg_free labvg`, sshOpts(ctx));
const freeMib = Number.parseFloat(vg.stdout.trim());
const neededMib = TARGET_MIB - sizeMib;
if (Number.isNaN(freeMib) || freeMib < neededMib) {
return {
success: true,
changed: false,
message: `VG labvg has ${Math.floor((Number.isNaN(freeMib) ? 0 : freeMib) / 1024)}G free — ` +
`need ${Math.ceil(neededMib / 1024)}G to grow rancher LV to 120G (manual LV rebuild required)`,
};
}
await ctx.ssh.exec(`lvextend -L ${TARGET_MIB}m /dev/${RANCHER_LV}`, sshOpts(ctx));
await ctx.ssh.exec(`xfs_growfs /var/lib/rancher`, sshOpts(ctx));
return {
success: true,
changed: true,
message: `rancher LV grown ${Math.round(sizeMib / 1024)}G → 120G`,
};
};

View File

@@ -1,22 +1,40 @@
// Disable swap (CIS requirement for k3s).
// Enable swap so memory pressure spills to disk instead of OOM-killing.
// kubelet runs with failSwapOn=false (k3s default); zram stays the fast tier,
// the labvg-swap LV is the overflow tier. Replaces the old CIS-style
// disableSwap op — a kernel OOM kill of a node daemon is worse than slow swap.
import type { Operation, OperationResult } from "../types.js";
import { sshOpts } from "../utils.js";
export const disableSwap: Operation = async (ctx): Promise<OperationResult> => {
const check = await ctx.ssh.exec("swapon --show --noheadings", sshOpts(ctx));
const active = check.stdout.trim().length > 0;
const SWAP_DEV = "/dev/mapper/labvg-swap";
if (active) {
await ctx.ssh.exec("swapoff -a", sshOpts(ctx));
export const enableSwap: Operation = async (ctx): Promise<OperationResult> => {
const lv = await ctx.ssh.exec(`test -b ${SWAP_DEV} && echo yes || echo no`, sshOpts(ctx));
if (lv.stdout.trim() !== "yes") {
return { success: true, changed: false, message: "No labvg-swap LV — skipping swap enable" };
}
// Remove swap entries from fstab permanently
await ctx.ssh.exec("sed -i '/\\sswap\\s/d' /etc/fstab", sshOpts(ctx));
const active = await ctx.ssh.exec(
`grep -q "^$(readlink -f ${SWAP_DEV}) " /proc/swaps && echo on || echo off`,
sshOpts(ctx),
);
const wasOff = active.stdout.trim() !== "on";
if (wasOff) {
// Format if the LV was never (or wrongly) initialised, then activate
await ctx.ssh.exec(`blkid ${SWAP_DEV} | grep -q 'TYPE="swap"' || mkswap ${SWAP_DEV}`, sshOpts(ctx));
await ctx.ssh.exec(`swapon ${SWAP_DEV}`, sshOpts(ctx));
}
// Persist across reboots (idempotent)
await ctx.ssh.exec(
`grep -q "labvg-swap" /etc/fstab || echo "${SWAP_DEV} none swap defaults 0 0" >> /etc/fstab`,
sshOpts(ctx),
);
return {
success: true,
changed: active,
message: active ? "Swap disabled" : "Swap already disabled",
changed: wasOff,
message: wasOff ? "LV swap enabled" : "LV swap already active",
};
};

View File

@@ -0,0 +1,96 @@
// vlan-setup DaemonSet — the node-level half of the macvlan/VLAN-10 story.
// On every node it (1) installs the reference CNI plugins (macvlan/ipvlan/
// static/host-local/vlan/tuning) into /opt/cni/bin if missing, and (2) creates
// a `lan10` VLAN-10 sub-interface on the primary NIC that macvlan
// NetworkAttachmentDefinitions use as their master. Idempotent + self-healing
// (re-creates lan10 if it disappears). Paired with Multus (multus.ts) + Cilium
// bpf.vlanBypass={10} (cilium.ts).
import type { Operation, OperationResult } from "../types.js";
import { sshOpts } from "../utils.js";
const MANIFEST = `apiVersion: v1
kind: Namespace
metadata:
name: macvlan-sys
labels:
pod-security.kubernetes.io/enforce: privileged
pod-security.kubernetes.io/audit: privileged
pod-security.kubernetes.io/warn: privileged
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: vlan-setup
namespace: macvlan-sys
spec:
selector:
matchLabels: { app: vlan-setup }
template:
metadata:
labels: { app: vlan-setup }
spec:
hostNetwork: true
tolerations:
- operator: Exists
containers:
- name: vlan
image: nicolaka/netshoot
securityContext:
privileged: true
command:
- sh
- -c
- |
set -x
# install reference CNI plugins (macvlan/ipvlan/static/host-local) if missing
if [ ! -f /host/opt/cni/bin/macvlan ] || [ ! -f /host/opt/cni/bin/ipvlan ]; then
case "$(uname -m)" in x86_64) A=amd64;; aarch64) A=arm64;; *) A=amd64;; esac
curl -sSL "https://github.com/containernetworking/plugins/releases/download/v1.5.1/cni-plugins-linux-$A-v1.5.1.tgz" -o /tmp/cni.tgz
tar -xzf /tmp/cni.tgz -C /host/opt/cni/bin ./macvlan ./ipvlan ./static ./host-local ./vlan ./tuning
fi
# detect the primary NIC (default route dev, else the one holding 192.168.8.x)
NIC="$(ip -o -4 route show default 2>/dev/null | awk '{print $5; exit}')"
[ -z "$NIC" ] && NIC="$(ip -o -4 addr show 2>/dev/null | awk '/192\\.168\\.8\\./{print $2; exit}')"
echo "primary NIC = $NIC"
while true; do
if [ -n "$NIC" ]; then
ip link show lan10 >/dev/null 2>&1 || ip link add link "$NIC" name lan10 type vlan id 10
ip link set lan10 up
# NIC-driver workarounds for VLAN multicast RX
ip link set "$NIC" allmulticast on 2>/dev/null
ethtool -K "$NIC" rxvlan off rx-vlan-filter off 2>/dev/null
fi
sleep 30
done
volumeMounts:
- name: cnibin
mountPath: /host/opt/cni/bin
volumes:
- name: cnibin
hostPath:
path: /opt/cni/bin
`;
export const installVlanSetup: Operation = async (ctx): Promise<OperationResult> => {
const K = "KUBECONFIG=/etc/rancher/k3s/k3s.yaml";
const check = await ctx.ssh.exec(
`${K} kubectl -n macvlan-sys get ds vlan-setup -o name 2>/dev/null`,
sshOpts(ctx),
);
if (check.exitCode === 0 && check.stdout.includes("vlan-setup")) {
return { success: true, changed: false, message: "vlan-setup DaemonSet already installed" };
}
const b64 = Buffer.from(MANIFEST).toString("base64");
const apply = await ctx.ssh.exec(
`echo ${b64} | base64 -d | ${K} kubectl apply -f -`,
{ ...sshOpts(ctx), timeoutMs: 60_000 },
);
if (apply.exitCode !== 0) {
return { success: false, changed: false, message: "Failed to apply vlan-setup DaemonSet", error: apply.stderr };
}
return { success: true, changed: true, message: "Installed vlan-setup DaemonSet (lan10 + CNI plugins)" };
};

View File

@@ -71,9 +71,14 @@ describe("k3s install script — server role", () => {
expect(script).toContain("enable-admission-plugins=NodeRestriction,PodSecurity");
});
it("configures audit logging", () => {
expect(script).toContain("audit-log-path=/var/log/kubernetes/audit.log");
expect(script).toContain("audit-log-maxage=30");
it("configures audit logging via journald (stdout)", () => {
expect(script).toContain("audit-log-path=-");
// file-based fields and the now-obsolete log directory must be gone
expect(script).not.toContain("/var/log/kubernetes/audit.log");
expect(script).not.toContain("audit-log-maxage");
expect(script).not.toContain("audit-log-maxbackup");
expect(script).not.toContain("audit-log-maxsize");
expect(script).not.toContain("mkdir -p /var/log/kubernetes");
});
it("cleans stale flannel vxlan before Cilium install", () => {

View File

@@ -72,31 +72,97 @@ describe("applyCisHardening", () => {
// --- Swap ---
import { disableSwap } from "../src/operations/swap.js";
import { enableSwap } from "../src/operations/swap.js";
describe("disableSwap", () => {
it("disables active swap", async () => {
describe("enableSwap", () => {
it("activates LV swap when present but off", async () => {
const ctx = mockCtx();
ctx.ssh.exec
.mockResolvedValueOnce(stdout("/dev/sda2 partition 2G")) // swap active
.mockResolvedValueOnce(OK) // swapoff
.mockResolvedValueOnce(OK); // sed fstab
.mockResolvedValueOnce(stdout("yes")) // LV exists
.mockResolvedValueOnce(stdout("off")) // not in /proc/swaps
.mockResolvedValueOnce(OK) // blkid || mkswap
.mockResolvedValueOnce(OK) // swapon
.mockResolvedValueOnce(OK); // fstab entry
const result = await disableSwap(ctx);
const result = await enableSwap(ctx);
expect(result.success).toBe(true);
expect(result.changed).toBe(true);
expectCommand(ctx.ssh, "swapoff -a");
expectCommand(ctx.ssh, "swapon /dev/mapper/labvg-swap");
});
it("is idempotent when swap already off", async () => {
it("is idempotent when LV swap already active", async () => {
const ctx = mockCtx();
ctx.ssh.exec
.mockResolvedValueOnce(stdout("")) // no swap
.mockResolvedValueOnce(OK); // sed fstab (always runs)
.mockResolvedValueOnce(stdout("yes")) // LV exists
.mockResolvedValueOnce(stdout("on")) // already in /proc/swaps
.mockResolvedValueOnce(OK); // fstab entry (always ensured)
const result = await disableSwap(ctx);
const result = await enableSwap(ctx);
expect(result.changed).toBe(false);
expectNoCommand(ctx.ssh, "swapoff");
expectNoCommand(ctx.ssh, "swapon /dev/mapper/labvg-swap");
});
it("skips when no labvg-swap LV exists", async () => {
const ctx = mockCtx();
ctx.ssh.exec.mockResolvedValueOnce(stdout("no")); // LV missing
const result = await enableSwap(ctx);
expect(result.success).toBe(true);
expect(result.changed).toBe(false);
expectNoCommand(ctx.ssh, "swapon");
});
});
// --- Rancher LV (imageFs sizing) ---
import { growRancherLv } from "../src/operations/rancher-storage.js";
describe("growRancherLv", () => {
it("grows a 20G LV to 120G when the VG has space", async () => {
const ctx = mockCtx();
ctx.ssh.exec
.mockResolvedValueOnce(stdout(" 20480.00")) // lv_size
.mockResolvedValueOnce(stdout(" 747807.00")) // vg_free
.mockResolvedValueOnce(OK) // lvextend
.mockResolvedValueOnce(OK); // xfs_growfs
const result = await growRancherLv(ctx);
expect(result.success).toBe(true);
expect(result.changed).toBe(true);
expectCommand(ctx.ssh, "lvextend -L 122880m /dev/labvg/rancher");
expectCommand(ctx.ssh, "xfs_growfs /var/lib/rancher");
});
it("is idempotent when the LV is already 120G", async () => {
const ctx = mockCtx();
ctx.ssh.exec.mockResolvedValueOnce(stdout(" 122880.00")); // lv_size
const result = await growRancherLv(ctx);
expect(result.changed).toBe(false);
expectNoCommand(ctx.ssh, "lvextend");
});
it("reports without failing when the VG has no free space", async () => {
const ctx = mockCtx();
ctx.ssh.exec
.mockResolvedValueOnce(stdout(" 20480.00")) // lv_size
.mockResolvedValueOnce(stdout(" 0.00")); // vg_free
const result = await growRancherLv(ctx);
expect(result.success).toBe(true);
expect(result.changed).toBe(false);
expect(result.message).toContain("free");
expectNoCommand(ctx.ssh, "lvextend");
});
it("skips when there is no rancher LV", async () => {
const ctx = mockCtx();
ctx.ssh.exec.mockResolvedValueOnce(stdout("")); // lvs empty
const result = await growRancherLv(ctx);
expect(result.success).toBe(true);
expect(result.changed).toBe(false);
expectNoCommand(ctx.ssh, "lvextend");
});
});
@@ -348,3 +414,143 @@ describe("applyPodSecurityStandards", () => {
expectCommand(ctx.ssh, "pod-security.kubernetes.io/audit=restricted");
});
});
// --- Audit Logging Decommission (file-based → journald) ---
import { configureLogRotation } from "../src/operations/log-rotation.js";
import { configureJournaldLimits } from "../src/operations/journald-limits.js";
describe("configureLogRotation (decommission file-based audit logs)", () => {
it("removes the legacy logrotate rule and reaps obsolete audit files", async () => {
const ctx = mockCtx();
ctx.ssh.exec.mockResolvedValueOnce(stdout("present")); // probe: legacy artifacts exist
ctx.ssh.exec.mockResolvedValue(OK);
const result = await configureLogRotation(ctx);
expect(result.success).toBe(true);
expect(result.changed).toBe(true);
expectCommand(ctx.ssh, "rm -f /etc/logrotate.d/k3s");
expectCommand(ctx.ssh, /find \/var\/log\/kubernetes.*audit.*-delete/);
expectCommand(ctx.ssh, "rmdir /var/log/kubernetes");
});
it("is a no-op when nothing legacy is present", async () => {
const ctx = mockCtx();
ctx.ssh.exec.mockResolvedValueOnce(stdout("absent"));
ctx.ssh.exec.mockResolvedValue(OK);
const result = await configureLogRotation(ctx);
expect(result.success).toBe(true);
expect(result.changed).toBe(false);
});
});
describe("configureJournaldLimits", () => {
it("writes a 2 GB SystemMaxUse drop-in and reloads journald when changed", async () => {
const ctx = mockCtx();
ctx.ssh.exec.mockResolvedValueOnce(stdout("__LABCTL_NOT_FOUND__")); // no existing drop-in
ctx.ssh.exec.mockResolvedValue(OK);
const result = await configureJournaldLimits(ctx);
expect(result.success).toBe(true);
expect(result.changed).toBe(true);
const writeCall = ctx.ssh.exec.mock.calls.find((c) => {
const cmd = c[0] as string;
return cmd.includes("10-k3s-audit-cap.conf") && cmd.includes("LABCTL_EOF");
});
expect(writeCall).toBeTruthy();
const written = writeCall?.[0] as string;
expect(written).toContain("SystemMaxUse=2G");
expect(written).toContain("SystemKeepFree=1G");
expectCommand(ctx.ssh, "systemctl restart systemd-journald");
});
it("does not restart journald when the drop-in is already correct", async () => {
const ctx = mockCtx();
const existing =
"[Journal]\nSystemMaxUse=2G\nSystemKeepFree=1G\nSystemMaxFileSize=200M\n";
ctx.ssh.exec.mockResolvedValueOnce(stdout(existing));
ctx.ssh.exec.mockResolvedValue(OK);
const result = await configureJournaldLimits(ctx);
expect(result.success).toBe(true);
expect(result.changed).toBe(false);
expectNoCommand(ctx.ssh, "systemctl restart systemd-journald");
});
});
// --- Etcd Recovery ---
import { recoverEtcdMember } from "../src/operations/etcd-recover.js";
import { mockSsh } from "./helpers.js";
describe("recoverEtcdMember", () => {
it("refuses to operate when cluster is below 3 members (quorum risk)", async () => {
const broken = mockSsh();
const peer = mockSsh();
peer.exec.mockResolvedValueOnce(stdout("/usr/bin/etcdctl")); // etcdctl present
peer.exec.mockResolvedValueOnce(stdout(
"111, started, host-a-aaa, https://10.0.0.1:2380, https://10.0.0.1:2379, false\n" +
"222, started, host-b-bbb, https://10.0.0.2:2380, https://10.0.0.2:2379, false",
));
const result = await recoverEtcdMember({ broken, peer, brokenHostname: "host-b" });
expect(result.success).toBe(false);
expect(result.message).toMatch(/quorum/i);
// Critically: must NOT have stopped k3s or removed anything
expect(broken.exec).not.toHaveBeenCalledWith(expect.stringContaining("systemctl stop k3s"), expect.anything());
});
it("performs full procedure when quorum is preserved", async () => {
const broken = mockSsh();
const peer = mockSsh();
// ensureEtcdctl: present
peer.exec.mockResolvedValueOnce(stdout("/usr/bin/etcdctl"));
// member list (3 members, target = host-b)
peer.exec.mockResolvedValueOnce(stdout(
"111, started, host-a-aaa, https://10.0.0.1:2380, https://10.0.0.1:2379, false\n" +
"222, started, host-b-bbb, https://10.0.0.2:2380, https://10.0.0.2:2379, false\n" +
"333, started, host-c-ccc, https://10.0.0.3:2380, https://10.0.0.3:2379, false",
));
// member remove
peer.exec.mockResolvedValueOnce(stdout("Member 222 removed"));
// post-rejoin member list — new id 444 for host-b
peer.exec.mockResolvedValueOnce(stdout(
"111, started, host-a-aaa, https://10.0.0.1:2380, https://10.0.0.1:2379, false\n" +
"333, started, host-c-ccc, https://10.0.0.3:2380, https://10.0.0.3:2379, false\n" +
"444, started, host-b-zzz, https://10.0.0.2:2380, https://10.0.0.2:2379, false",
));
const result = await recoverEtcdMember({ broken, peer, brokenHostname: "host-b" });
expect(result.success).toBe(true);
expect(result.removedMemberId).toBe("222");
expect(result.newMemberId).toBe("444");
expectCommand(broken,"systemctl stop k3s");
expectCommand(peer,"member remove 222");
expectCommand(broken,/db\.corrupt-/);
expectCommand(broken,/rm -rf .*\/server\/tls/);
expectCommand(broken,"systemctl start k3s");
});
it("fails clearly when no member matches the broken hostname", async () => {
const broken = mockSsh();
const peer = mockSsh();
peer.exec.mockResolvedValueOnce(stdout("/usr/bin/etcdctl"));
peer.exec.mockResolvedValueOnce(stdout(
"111, started, host-a-aaa, https://10.0.0.1:2380, https://10.0.0.1:2379, false\n" +
"222, started, host-b-bbb, https://10.0.0.2:2380, https://10.0.0.2:2379, false\n" +
"333, started, host-c-ccc, https://10.0.0.3:2380, https://10.0.0.3:2379, false",
));
const result = await recoverEtcdMember({ broken, peer, brokenHostname: "host-d" });
expect(result.success).toBe(false);
expect(result.message).toMatch(/No etcd member found/);
expect(broken.exec).not.toHaveBeenCalledWith(expect.stringContaining("systemctl stop k3s"), expect.anything());
});
});

View File

@@ -16,7 +16,7 @@ describe("smoke: full server install pipeline", () => {
const pipeline: NamedOperation[] = [
{ name: "Kernel modules", fn: ops.loadKernelModules },
{ name: "Sysctl hardening", fn: ops.applyCisHardening },
{ name: "Disable swap", fn: ops.disableSwap },
{ name: "Enable swap", fn: ops.enableSwap },
{ name: "Disable firewall", fn: ops.disableFirewall },
{ name: "SELinux permissive", fn: ops.setSelinuxPermissive },
{ name: "Write k3s config", fn: ops.writeK3sConfig },
@@ -73,7 +73,7 @@ describe("smoke: pipeline stops on failure", () => {
};
const results = await runSequential(ctx, [
{ name: "OK op", fn: ops.disableSwap },
{ name: "OK op", fn: ops.enableSwap },
{ name: "Failing op", fn: failingOp },
{ name: "Never called", fn: neverCalled },
]);
@@ -98,11 +98,12 @@ describe("smoke: agent install rejects missing config", () => {
});
describe("smoke: all operations are exported", () => {
it("exports all 15 operations", () => {
it("exports all 16 operations", () => {
const exported = [
ops.loadKernelModules,
ops.applyCisHardening,
ops.disableSwap,
ops.enableSwap,
ops.growRancherLv,
ops.disableFirewall,
ops.setSelinuxPermissive,
ops.writeK3sConfig,
@@ -117,7 +118,7 @@ describe("smoke: all operations are exported", () => {
ops.checkCertExpiry,
];
expect(exported).toHaveLength(15);
expect(exported).toHaveLength(16);
for (const op of exported) {
expect(typeof op).toBe("function");
}

View File

@@ -8,6 +8,10 @@ export type {
DebugConfig,
BastionState,
BastionConfig,
VyosVlanSpec,
VyosInstallSpec,
VyosBundle,
VyosBundleSetOp,
} from "./types/index.js";
export { SUPPORTED_OS, SUPPORTED_ROLES, ROLE_REGISTRY, isValidOsId } from "./types/index.js";

View File

@@ -1,6 +1,7 @@
// Protocol types for agent-labd WebSocket communication.
import { randomUUID } from "node:crypto";
import type { VyosInstallSpec } from "../types/state.js";
// --- Agent -> labd messages ---
@@ -108,11 +109,12 @@ export type BastionMessage =
export type LabdBastionMessage =
| { type: "bastion-enrolled"; bastionId: string }
| { type: "bastion-heartbeat-ack"; serverTime: string }
| { type: "command-install"; requestId: string; mac: string; hostname: string; disk?: string; role: string; os: string }
| { type: "command-install"; requestId: string; mac: string; hostname: string; disk?: string; role: string; os: string; vyos?: VyosInstallSpec }
| { type: "command-forget"; requestId: string; mac: string }
| { type: "command-role-update"; requestId: string; mac: string; role: string }
| { type: "command-debug"; requestId: string; mac: string; pxeBoot?: boolean }
| { type: "command-register"; requestId: string; mac: string; hostname: string; role: string; ip: string }
| { type: "command-discover"; requestId: string; mac: string; product?: string; board?: string; serial?: string; manufacturer?: string; cpu_model?: string; cpu_cores?: number; memory_gb?: number; arch?: string; disks?: Array<{ name: string; size_gb: number; model: string }>; nics?: Array<{ name: string; mac: string; state: string }> }
| { type: "server-shutdown"; reconnectAfter: number };
export type BastionMessageType = BastionMessage["type"];
@@ -127,7 +129,7 @@ const BASTION_MESSAGE_TYPES = new Set<string>([
const LABD_BASTION_MESSAGE_TYPES = new Set<string>([
"bastion-enrolled", "bastion-heartbeat-ack", "command-install",
"command-forget", "command-role-update", "command-debug", "command-register", "server-shutdown",
"command-forget", "command-role-update", "command-debug", "command-register", "command-discover", "server-shutdown",
]);
export function isBastionMessage(msg: unknown): msg is BastionMessage {

View File

@@ -14,6 +14,10 @@ export interface BastionConfig {
// Ubuntu support
ubuntuVersion: string;
ubuntuMirror: string;
// VyOS support — netboot artifacts are extracted from the ISO at startup.
// LTS ISOs are subscription-only, so this defaults to a rolling release.
vyosIsoUrl: string;
vyosDefaultPassword: string;
// Syslog listener for install logs (Anaconda logging --host)
syslogPort: number;
// Flags

View File

@@ -7,6 +7,10 @@ export type {
InstalledInfo,
DebugConfig,
BastionState,
VyosVlanSpec,
VyosInstallSpec,
VyosBundle,
VyosBundleSetOp,
} from "./state.js";
export { SUPPORTED_OS, SUPPORTED_ROLES, ROLE_REGISTRY, isValidOsId } from "./state.js";

View File

@@ -2,10 +2,10 @@
export type ProvisionStackType = "dhcpproxy" | "iso" | "cloud-init";
export type OsId = "fedora-43" | "ubuntu-26.04";
export type OsId = "fedora-43" | "ubuntu-26.04" | "vyos-rolling";
export type Arch = "x86_64" | "aarch64";
export const SUPPORTED_OS: readonly OsId[] = ["fedora-43", "ubuntu-26.04"] as const;
export const SUPPORTED_OS: readonly OsId[] = ["fedora-43", "ubuntu-26.04", "vyos-rolling"] as const;
export function isValidOsId(value: string): value is OsId {
return (SUPPORTED_OS as readonly string[]).includes(value);
@@ -75,13 +75,141 @@ export interface ProgressLogEntry {
timestamp: string;
}
/** A tagged VLAN sub-interface on the bond (or on the mgmt NIC when unbonded). */
export interface VyosVlanSpec {
id: number;
address: string; // CIDR, e.g. "10.0.10.1/24"
description?: string;
/**
* VRRP virtual address (CIDR) floated on this VLAN. Emitted as a
* high-availability vrrp group with vrid = VLAN id, so the same spec on both
* HA peers (with different priorities) produces a matching group pair.
*/
vrrp?: string;
}
/** One config node in a rendered bundle. Mirrors VyosSetOp on the bastion side. */
export interface VyosBundleSetOp {
path: string[];
value?: string;
/** false appends to a multi-value node (e.g. bond members) instead of replacing. */
replace?: boolean;
}
/**
* A router's complete desired config, rendered from the Pulumi model.
*
* Produced by `kubernetes-deployment/scripts/vyos-render-bundle.ts` from the
* same subtree model `pulumi up` applies. The point is that labctl never
* authors VyOS config: bring-up replays what Pulumi already declares, so a
* freshly installed router and a `pulumi up` cannot disagree.
*
* Secret values arrive as `@secret:<key>` sentinels and are DROPPED at install
* time -- the bundle is committed to git and must stay safe to read. The router
* comes up on the LAN without its PPPoE credential; the first `pulumi up`
* supplies it. That handoff is deliberate.
*/
export interface VyosBundle {
sets: VyosBundleSetOp[];
/** Paths that are VyOS tag nodes — the installer's ConfigTree needs them marked. */
tags: string[][];
}
/**
* VyOS-specific install parameters. Rendered into the config.boot that the
* installer adopts, so the router comes up already configured.
*
* NOTE: bondMembers must NOT include the interface PXE booted from. Firmware
* PXE cannot run over LACP, so the install-time NIC has to stay unbonded.
*/
export interface VyosInstallSpec {
/**
* A complete rendered config for this router. When present it REPLACES the
* derived interface/VLAN/VRRP config below -- the bundle already describes
* all of it, and deriving a second opinion is exactly the drift this exists
* to prevent. The remaining install parameters (password, disk, console) are
* still honoured because they are installer inputs, not router config.
*/
bundle?: VyosBundle;
/**
* Key for the VyOS HTTP API, enabled at install so the router is manageable
* from the moment it boots.
*
* Without this the box comes up reachable only over SSH, and enabling the API
* later is a hand-run config change on a live firewall -- which is exactly the
* gap that left vyos001/vyos002 unmanageable by Pulumi after their cutover.
* The API is deliberately NOT part of the Pulumi model: a provider that
* manages its own transport can revoke its own access.
*/
apiKey?: string;
/**
* Address the API listens on. Defaults to the management address when static.
* Never left unbound: an unrestricted listener puts a config-write endpoint on
* every segment the router touches, including the WAN.
*/
apiListenAddress?: string;
/** Interfaces aggregated into bond0 with LACP (802.3ad). Omit for no bond. */
bondMembers?: string[];
/** CIDR address on bond0 itself — the switch trunk's native/untagged VLAN. */
bondAddress?: string;
/** VRRP virtual address (CIDR) floated on the untagged bond (vrid 1). */
bondVrrp?: string;
/**
* VRRP priority for every group on this box. Higher wins mastership.
* The HA pair differs ONLY here (e.g. 200 on the primary, 100 on the
* standby) — addresses differ per box, VIPs and vrids match.
*/
vrrpPriority?: number;
/** Tagged VLAN sub-interfaces, created on bond0 when bonded, else on mgmtInterface. */
vlans?: VyosVlanSpec[];
/** Untagged interface the machine PXE booted from. Defaults to "eth0". */
mgmtInterface?: string;
/** CIDR address for mgmtInterface, or "dhcp". Defaults to "dhcp". */
mgmtAddress?: string;
/**
* Tagged management VLAN on mgmtInterface, separate from the routed VLANs
* carried by the bond.
*
* Needed when the PXE port is a trunk: it boots untagged on the VLAN the
* bastion's proxy DHCP serves, and carries the management VLAN tagged so the
* router stays reachable there without giving up reinstallability.
*/
mgmtVlan?: VyosVlanSpec;
/** Password for the "vyos" user. Falls back to the bastion default. */
password?: string;
/**
* On reinstall the VyOS installer carries the previous on-disk config (and
* SSH host keys) forward -- the "reinstall without losing data" default.
* Set true to make the bastion-generated config win instead: after install
* the driver overwrites the installed image's config.boot.
*/
freshConfig?: boolean;
/**
* VyOS interface name -> MAC, emitted as `hw-id` so names bind deterministically.
*
* Discovery runs under Fedora and reports predictable names (enp2s0,
* enp1s0f0np0), but VyOS enumerates its own eth<N> names, so a name observed
* during discovery cannot be used directly. Pinning by MAC removes the guess
* about which physical port a given eth<N> is.
*/
hwIds?: Record<string, string>;
}
export interface InstallConfig {
hostname: string;
disk: string;
role: Role;
os?: OsId; // defaults to "fedora-43" for backward compat
vyos?: VyosInstallSpec; // only consulted when os is "vyos-rolling"
arch?: Arch; // detected from HardwareInfo or overridden
queued_at: string;
/**
* When dispatch last served this machine an install boot script. Progress
* callbacks only start once the installer environment is up, so a machine
* dispatched long ago with no progress is wedged before that point (bad
* kernel/initrd, no network in the initramfs, wrong NIC picked...).
*/
dispatched_at?: string;
progress?: string;
progress_at?: string;
progress_detail?: string;
@@ -96,6 +224,13 @@ export interface InstalledInfo {
ip: string;
installed_at: string;
bastionId?: string; // set when aggregated through labd
// Hardware info (copied from discovered on install completion)
product?: string;
manufacturer?: string;
cpu_model?: string;
cpu_cores?: number;
memory_gb?: number;
arch?: string;
}
export interface DebugConfig {

View File

@@ -0,0 +1,355 @@
// Integration test: Asahi first-boot LVM setup.
//
// Tests the first-boot script that creates the standard lab LVM layout
// on a separate data disk — simulating the Asahi provisioning flow where
// the root partition is pre-installed and a data partition is left for LVM.
//
// Uses a Fedora cloud VM with two disks:
// disk0: 20GB root (Fedora cloud image)
// disk1: 200GB empty (simulates the Asahi "Data" partition)
//
// The firstboot script should detect disk1, create labvg + LVs, mount them.
// Then we test reprovision: wipe marker, re-run, verify existing VG reused.
//
// Prerequisites: libvirt, virsh, virt-install, qemu, sudo access, lvm2
// Run: sudo pnpm run test:integration:asahi
import { describe, it, expect, beforeAll, afterAll } from "vitest";
import { readFileSync, existsSync } from "node:fs";
import { execSync } from "node:child_process";
import { join } from "node:path";
import { homedir } from "node:os";
import { destroyVm, waitForVmIp, waitForSsh, log, ensureCloudImage, createCloudInitIso } from "./helpers/libvirt.js";
import { ensureTestNetwork, TEST_NETWORK_NAME } from "./helpers/network.js";
import { sshExec, sshRun } from "./helpers/ssh.js";
import { renderFirstbootScript } from "../../src/bastion/src/templates/asahi-firstboot.sh.js";
const VM_NAME = "lab-asahi-firstboot-test";
const VM_MEMORY = 4096;
const VM_VCPUS = 2;
const VM_ROOT_DISK_GB = 20;
const VM_DATA_DISK_GB = 200; // Simulates the Asahi "Data" partition
const SSH_USER = "fedora";
const IMAGE_DIR = "/var/lib/libvirt/images";
const IS_ROOT = process.getuid?.() === 0;
const FEDORA_CLOUD_IMAGE = "https://download.fedoraproject.org/pub/fedora/linux/releases/43/Cloud/x86_64/images/Fedora-Cloud-Base-Generic-43-1.6.x86_64.qcow2";
function run(cmd: string, opts?: { timeout?: number }): string {
const full = IS_ROOT ? cmd : `sudo ${cmd}`;
return execSync(full, { encoding: "utf-8", stdio: "pipe", timeout: opts?.timeout ?? 60_000 });
}
function findSshKey(): { pubKey: string; keyPath: string } {
const homes = [homedir()];
const sudoUser = process.env["SUDO_USER"];
if (sudoUser) homes.push(join("/home", sudoUser));
if (process.env["SSH_KEY_PATH"]) {
const keyPath = process.env["SSH_KEY_PATH"];
const pubPath = `${keyPath}.pub`;
if (existsSync(keyPath) && existsSync(pubPath)) {
return { pubKey: readFileSync(pubPath, "utf-8").trim(), keyPath };
}
}
for (const home of homes) {
for (const name of ["id_ed25519", "id_ecdsa", "id_rsa"]) {
const keyPath = join(home, ".ssh", name);
const pubPath = `${keyPath}.pub`;
if (existsSync(keyPath) && existsSync(pubPath)) {
return { pubKey: readFileSync(pubPath, "utf-8").trim(), keyPath };
}
}
}
throw new Error("No SSH key found");
}
/** Create a VM with two disks: root (cloud image) + empty data disk. */
function createTwoDiskVm(config: {
name: string;
memory: number;
vcpus: number;
rootDiskGb: number;
dataDiskGb: number;
network: string;
cloudImageUrl: string;
sshPubKey: string;
}): void {
destroyVm(config.name);
log(`Creating two-disk VM: ${config.name} (root=${config.rootDiskGb}GB, data=${config.dataDiskGb}GB)`);
const baseImage = ensureCloudImage(config.cloudImageUrl, `${config.name}-base`);
const rootDiskPath = join(IMAGE_DIR, `${config.name}.qcow2`);
const dataDiskPath = join(IMAGE_DIR, `${config.name}-data.qcow2`);
// Root disk from cloud image
run(`cp "${baseImage}" "${rootDiskPath}"`);
run(`qemu-img resize "${rootDiskPath}" ${config.rootDiskGb}G`);
// Empty data disk
run(`qemu-img create -f qcow2 "${dataDiskPath}" ${config.dataDiskGb}G`);
// Cloud-init with LVM tools
const cloudInitIso = createCloudInitIso(config.name, {
name: config.name,
memory: config.memory,
vcpus: config.vcpus,
diskSize: config.rootDiskGb,
network: config.network,
cloudImageUrl: config.cloudImageUrl,
sshPubKey: config.sshPubKey,
userData: `#cloud-config
hostname: ${config.name}
manage_etc_hosts: true
users:
- default
- name: fedora
sudo: ALL=(ALL) NOPASSWD:ALL
shell: /bin/bash
ssh_authorized_keys:
- ${config.sshPubKey}
ssh_pwauth: false
package_update: false
packages:
- lvm2
- xfsprogs
`,
});
const virtInstallArgs = [
"virt-install",
`--name=${config.name}`,
`--memory=${config.memory}`,
`--vcpus=${config.vcpus}`,
`--disk=path=${rootDiskPath},format=qcow2`,
`--disk=path=${dataDiskPath},format=qcow2`, // Second disk for LVM
`--disk=path=${cloudInitIso},device=cdrom`,
`--network=network=${config.network},model=virtio`,
"--os-variant=generic",
"--import",
"--noautoconsole",
"--wait=0",
];
run(virtInstallArgs.join(" "));
log(`Two-disk VM ${config.name} created`);
}
describe("asahi firstboot LVM integration", () => {
let vmIp: string;
let sshKeyPath: string;
let sshPubKey: string;
beforeAll(async () => {
const keys = findSshKey();
sshKeyPath = keys.keyPath;
sshPubKey = keys.pubKey;
log("Setting up test network...");
ensureTestNetwork();
log("Creating two-disk VM...");
createTwoDiskVm({
name: VM_NAME,
memory: VM_MEMORY,
vcpus: VM_VCPUS,
rootDiskGb: VM_ROOT_DISK_GB,
dataDiskGb: VM_DATA_DISK_GB,
network: TEST_NETWORK_NAME,
cloudImageUrl: FEDORA_CLOUD_IMAGE,
sshPubKey,
});
log("Waiting for VM IP...");
vmIp = await waitForVmIp(VM_NAME, 120_000);
log("Waiting for SSH...");
await waitForSsh(vmIp, SSH_USER, 180_000, sshKeyPath);
log("Waiting for cloud-init to finish...");
await sshRun(vmIp, SSH_USER, "sudo cloud-init status --wait 2>/dev/null || sleep 30", "cloud-init", { keyPath: sshKeyPath });
// Verify second disk exists
const disks = sshExec(vmIp, SSH_USER, "lsblk -d -n -o NAME,SIZE", { keyPath: sshKeyPath });
log(`Disks:\n${disks.stdout}`);
}, 300_000);
afterAll(async () => {
log("Cleaning up VM...");
destroyVm(VM_NAME);
// Also remove data disk
try { run(`rm -f "${join(IMAGE_DIR, `${VM_NAME}-data.qcow2`)}"`); } catch { /* ignore */ }
});
it("second disk is visible and unformatted", () => {
const result = sshExec(vmIp, SSH_USER, "lsblk -d -n -o NAME,SIZE,TYPE | grep disk", { keyPath: sshKeyPath });
const disks = result.stdout.trim().split("\n");
expect(disks.length).toBeGreaterThanOrEqual(2);
// Second disk (vdb) should exist
const vdb = sshExec(vmIp, SSH_USER, "sudo blkid /dev/vdb 2>/dev/null; echo exit=$?", { keyPath: sshKeyPath });
// Should have no filesystem (blkid returns nothing or non-zero)
expect(vdb.stdout).toContain("exit=2");
});
it("firstboot script creates LVM on data disk", async () => {
// Generate the firstboot script
const script = renderFirstbootScript({
hostname: "asahi-test",
role: "infra",
serverIp: "10.0.0.1",
httpPort: 8080,
sshKeys: [sshPubKey],
adminUser: "testadmin",
mac: "52:54:00:aa:bb:cc",
});
// Upload and run
log("Uploading firstboot script...");
await sshRun(vmIp, SSH_USER,
`cat > /tmp/firstboot.sh << 'SCRIPT_EOF'\n${script}\nSCRIPT_EOF\nchmod +x /tmp/firstboot.sh`,
"upload script", { keyPath: sshKeyPath });
log("Running firstboot script...");
const result = await sshRun(vmIp, SSH_USER,
"sudo /tmp/firstboot.sh 2>&1",
"firstboot", { keyPath: sshKeyPath, timeout: 120_000 });
expect(result).toBe(0);
}, 180_000);
it("SSH still works after firstboot script", () => {
const result = sshExec(vmIp, SSH_USER, "echo hello", { keyPath: sshKeyPath });
if (result.stdout.trim() !== "hello") {
log(`SSH debug: exitCode=${result.exitCode} stdout='${result.stdout}' stderr='${result.stderr}'`);
}
expect(result.stdout.trim()).toBe("hello");
});
it("volume group labvg exists", () => {
const result = sshExec(vmIp, SSH_USER, "sudo vgs labvg --noheadings -o vg_name", { keyPath: sshKeyPath });
expect(result.stdout.trim()).toBe("labvg");
});
it("all expected logical volumes exist", () => {
const result = sshExec(vmIp, SSH_USER,
"sudo lvs labvg --noheadings -o lv_name --sort lv_name",
{ keyPath: sshKeyPath });
const lvs = result.stdout.trim().split("\n").map(l => l.trim()).sort();
expect(lvs).toContain("home");
expect(lvs).toContain("longhorn");
expect(lvs).toContain("rancher"); // infra role
expect(lvs).toContain("srv");
expect(lvs).toContain("swap");
expect(lvs).toContain("var");
expect(lvs).toContain("varlog");
});
it("LV sizes match kickstart layout", () => {
const result = sshExec(vmIp, SSH_USER,
"sudo lvs labvg --noheadings -o lv_name,lv_size --units m --nosuffix",
{ keyPath: sshKeyPath });
const lvMap = new Map<string, number>();
for (const line of result.stdout.trim().split("\n")) {
const [name, size] = line.trim().split(/\s+/);
if (name && size) lvMap.set(name, Math.round(parseFloat(size)));
}
expect(lvMap.get("swap")).toBe(27648);
expect(lvMap.get("var")).toBe(102400);
expect(lvMap.get("varlog")).toBe(10240);
expect(lvMap.get("home")).toBe(10240);
expect(lvMap.get("srv")).toBe(20480);
expect(lvMap.get("rancher")).toBe(20480);
// longhorn gets remaining — should be at least 5GB (200GB disk - ~191GB used)
expect(lvMap.get("longhorn")).toBeGreaterThan(5000);
});
it("non-var volumes are mounted with XFS", () => {
const mounts = sshExec(vmIp, SSH_USER, "mount | grep labvg", { keyPath: sshKeyPath });
// /var and /var/log deferred to next reboot (can't migrate live)
expect(mounts.stdout).toContain("/home ");
expect(mounts.stdout).toContain("/srv ");
expect(mounts.stdout).toContain("/var/lib/rancher ");
expect(mounts.stdout).toContain("/var/lib/longhorn ");
expect(mounts.stdout).toContain("xfs");
});
it("swap is active", () => {
const result = sshExec(vmIp, SSH_USER, "swapon --show --noheadings", { keyPath: sshKeyPath });
// swapon may show /dev/dm-X or /dev/labvg/swap
expect(result.stdout.length).toBeGreaterThan(0);
});
it("fstab has LVM entries", () => {
const result = sshExec(vmIp, SSH_USER, "grep labvg /etc/fstab", { keyPath: sshKeyPath });
const lines = result.stdout.trim().split("\n");
expect(lines.length).toBeGreaterThanOrEqual(7); // swap + var + varlog + home + srv + rancher + longhorn
});
it("hostname was set", () => {
const result = sshExec(vmIp, SSH_USER, "hostname", { keyPath: sshKeyPath });
expect(result.stdout.trim()).toBe("asahi-test");
});
it("admin user was created with sudo", () => {
const result = sshExec(vmIp, SSH_USER, "sudo id testadmin", { keyPath: sshKeyPath });
expect(result.stdout).toContain("testadmin");
expect(result.stdout).toContain("wheel");
});
it("provisioning metadata file exists", () => {
const result = sshExec(vmIp, SSH_USER, "cat /etc/lab-provisioned", { keyPath: sshKeyPath });
expect(result.stdout).toContain("hostname=asahi-test");
expect(result.stdout).toContain("role=infra");
expect(result.stdout).toContain("method=asahi-firstboot");
});
it("marker file prevents re-run", () => {
const result = sshExec(vmIp, SSH_USER, "test -f /etc/lab-lvm-setup-done && echo yes", { keyPath: sshKeyPath });
expect(result.stdout.trim()).toBe("yes");
});
// ── Reprovision test ──────────────────────────────────────────────
it("reprovision: detects existing labvg and re-mounts", async () => {
// Write a test file to a preserved LV
await sshRun(vmIp, SSH_USER,
"echo 'precious-data' | sudo tee /var/lib/rancher/test-preserve.txt",
"write test data", { keyPath: sshKeyPath });
// Remove marker to simulate fresh boot after reinstall
await sshRun(vmIp, SSH_USER, "sudo rm /etc/lab-lvm-setup-done", "remove marker", { keyPath: sshKeyPath });
// Unmount everything (simulate reinstall wiping root)
await sshRun(vmIp, SSH_USER, `
sudo umount /var/lib/longhorn 2>/dev/null || true
sudo umount /var/lib/rancher 2>/dev/null || true
sudo umount /srv 2>/dev/null || true
sudo umount /home 2>/dev/null || true
sudo umount /var/log 2>/dev/null || true
# Don't unmount /var — it's in use
sudo swapoff /dev/labvg/swap 2>/dev/null || true
sudo sed -i '/labvg/d' /etc/fstab
`, "unmount LVs", { keyPath: sshKeyPath });
// Re-run firstboot script — should detect existing VG
log("Re-running firstboot (reprovision)...");
const result = await sshRun(vmIp, SSH_USER,
"sudo /tmp/firstboot.sh 2>&1",
"firstboot reprovision", { keyPath: sshKeyPath });
expect(result).toBe(0);
// Verify data was preserved
const data = sshExec(vmIp, SSH_USER, "cat /var/lib/rancher/test-preserve.txt", { keyPath: sshKeyPath });
expect(data.stdout.trim()).toBe("precious-data");
// Verify marker was re-created
const marker = sshExec(vmIp, SSH_USER, "test -f /etc/lab-lvm-setup-done && echo yes", { keyPath: sshKeyPath });
expect(marker.stdout.trim()).toBe("yes");
// Verify fstab was re-populated
const fstab = sshExec(vmIp, SSH_USER, "grep labvg /etc/fstab", { keyPath: sshKeyPath });
expect(fstab.stdout).toContain("/var/lib/rancher");
}, 60_000);
});

View File

@@ -0,0 +1,353 @@
// Validation tests for Asahi provisioning artifacts.
//
// Tests that can run WITHOUT Apple Silicon hardware:
// 1. Shellcheck the generated firstboot script
// 2. Verify the built rootfs ZIP structure
// 3. Mount the rootfs and verify injected files
// 4. Validate installer_data.json against the Asahi installer's Python parser
// 5. Verify partition layout arithmetic
//
// Prerequisites:
// - Run scripts/build-asahi-rootfs.sh first (creates asahi-repo/)
// - shellcheck installed (dnf install ShellCheck)
// - python3 installed
// - root for loop mount (sudo)
//
// Run: sudo pnpm run test:integration:asahi-validate
import { describe, it, expect, beforeAll, afterAll } from "vitest";
import { existsSync, lstatSync, readFileSync, writeFileSync, mkdirSync, rmSync } from "node:fs";
import { execSync, spawnSync } from "node:child_process";
import { join } from "node:path";
import { tmpdir } from "node:os";
import { renderFirstbootScript } from "../../src/bastion/src/templates/asahi-firstboot.sh.js";
const PROJECT_ROOT = join(import.meta.dirname, "..", "..");
const ASAHI_REPO = join(PROJECT_ROOT, "asahi-repo");
const ASAHI_CACHE = join(PROJECT_ROOT, ".asahi-cache");
const IS_ROOT = process.getuid?.() === 0;
function run(cmd: string, opts?: { timeout?: number }): string {
const full = IS_ROOT ? cmd : `sudo ${cmd}`;
return execSync(full, { encoding: "utf-8", stdio: "pipe", timeout: opts?.timeout ?? 60_000 });
}
function hasBuiltArtifacts(): boolean {
return existsSync(join(ASAHI_REPO, "fedora-asahi-lab.zip")) &&
existsSync(join(ASAHI_REPO, "installer_data.json"));
}
describe("asahi script validation", () => {
it("firstboot script passes shellcheck", () => {
const script = renderFirstbootScript({
hostname: "test-node",
role: "infra",
serverIp: "10.0.0.1",
httpPort: 8080,
sshKeys: ["ssh-ed25519 AAAA... user@host"],
adminUser: "testadmin",
mac: "aa:bb:cc:dd:ee:ff",
});
const tmpFile = join(tmpdir(), `asahi-shellcheck-${Date.now()}.sh`);
writeFileSync(tmpFile, script);
try {
const result = spawnSync("shellcheck", [
"-s", "bash",
"-e", "SC2086,SC2164", // allow unquoted variables (intentional in some LVM commands)
tmpFile,
], { encoding: "utf-8", stdio: "pipe", timeout: 30_000 });
if (result.status !== 0) {
console.log("Shellcheck warnings/errors:");
console.log(result.stdout);
}
// Allow warnings (exit 1 for warnings), fail on errors (exit 2+)
expect(result.status).toBeLessThan(2);
} finally {
try { rmSync(tmpFile); } catch { /* ignore */ }
}
});
it("firstboot script for worker role passes shellcheck", () => {
const script = renderFirstbootScript({
hostname: "worker-node",
role: "worker",
serverIp: "10.0.0.1",
httpPort: 8080,
sshKeys: [],
adminUser: "michal",
mac: "00:11:22:33:44:55",
});
const tmpFile = join(tmpdir(), `asahi-shellcheck-worker-${Date.now()}.sh`);
writeFileSync(tmpFile, script);
try {
const result = spawnSync("shellcheck", ["-s", "bash", "-e", "SC2086,SC2164", tmpFile],
{ encoding: "utf-8", stdio: "pipe", timeout: 30_000 });
if (result.status !== 0) console.log(result.stdout);
expect(result.status).toBeLessThan(2);
} finally {
try { rmSync(tmpFile); } catch { /* ignore */ }
}
});
it("firstboot script for vanilla role passes shellcheck", () => {
const script = renderFirstbootScript({
hostname: "vanilla-node",
role: "vanilla",
serverIp: "10.0.0.1",
httpPort: 8080,
sshKeys: ["ssh-rsa AAAA... user@host"],
adminUser: "admin",
mac: "ff:ee:dd:cc:bb:aa",
});
const tmpFile = join(tmpdir(), `asahi-shellcheck-vanilla-${Date.now()}.sh`);
writeFileSync(tmpFile, script);
try {
const result = spawnSync("shellcheck", ["-s", "bash", "-e", "SC2086,SC2164", tmpFile],
{ encoding: "utf-8", stdio: "pipe", timeout: 30_000 });
if (result.status !== 0) console.log(result.stdout);
expect(result.status).toBeLessThan(2);
} finally {
try { rmSync(tmpFile); } catch { /* ignore */ }
}
});
});
describe("asahi installer_data.json validation", () => {
let installerData: Record<string, unknown>;
beforeAll(() => {
if (!hasBuiltArtifacts()) {
throw new Error("Run scripts/build-asahi-rootfs.sh first to generate artifacts");
}
installerData = JSON.parse(readFileSync(join(ASAHI_REPO, "installer_data.json"), "utf-8"));
});
it("has os_list with one entry", () => {
const osList = installerData["os_list"] as unknown[];
expect(osList).toBeInstanceOf(Array);
expect(osList.length).toBe(1);
});
it("has required top-level fields", () => {
const os = (installerData["os_list"] as Record<string, unknown>[])[0]!;
expect(os["name"]).toBeDefined();
expect(os["default_os_name"]).toBeDefined();
expect(os["boot_object"]).toBeDefined();
expect(os["next_object"]).toBeDefined();
expect(os["package"]).toBe("fedora-asahi-lab.zip");
expect(os["supported_fw"]).toBeInstanceOf(Array);
expect((os["supported_fw"] as string[]).length).toBeGreaterThan(0);
});
it("has 4 partitions (EFI + Boot + Root + Data)", () => {
const os = (installerData["os_list"] as Record<string, unknown>[])[0]!;
const partitions = os["partitions"] as Record<string, unknown>[];
expect(partitions).toHaveLength(4);
expect(partitions[0]!["name"]).toBe("EFI");
expect(partitions[1]!["name"]).toBe("Boot");
expect(partitions[2]!["name"]).toBe("Root");
expect(partitions[3]!["name"]).toBe("Data");
});
it("EFI partition has correct format", () => {
const os = (installerData["os_list"] as Record<string, unknown>[])[0]!;
const efi = (os["partitions"] as Record<string, unknown>[])[0]!;
expect(efi["type"]).toBe("EFI");
expect(efi["format"]).toBe("fat");
expect(efi["copy_firmware"]).toBe(true);
// Size should be ~500MB in bytes
const size = parseInt(String(efi["size"]).replace("B", ""), 10);
expect(size).toBeGreaterThanOrEqual(500 * 1024 * 1024);
});
it("Boot partition references boot.img", () => {
const os = (installerData["os_list"] as Record<string, unknown>[])[0]!;
const boot = (os["partitions"] as Record<string, unknown>[])[1]!;
expect(boot["type"]).toBe("Linux");
expect(boot["image"]).toBe("boot.img");
});
it("Root partition does NOT expand", () => {
const os = (installerData["os_list"] as Record<string, unknown>[])[0]!;
const root = (os["partitions"] as Record<string, unknown>[])[2]!;
expect(root["type"]).toBe("Linux");
expect(root["image"]).toBe("root.img");
expect(root["expand"]).toBe(false);
});
it("Data partition expands for LVM", () => {
const os = (installerData["os_list"] as Record<string, unknown>[])[0]!;
const data = (os["partitions"] as Record<string, unknown>[])[3]!;
expect(data["type"]).toBe("Linux");
expect(data["expand"]).toBe(true);
expect(data["image"]).toBeUndefined(); // No image — empty partition for LVM
});
it("partition sizes use bytes format (NB suffix)", () => {
const os = (installerData["os_list"] as Record<string, unknown>[])[0]!;
const partitions = os["partitions"] as Record<string, unknown>[];
for (const p of partitions) {
const size = String(p["size"]);
expect(size).toMatch(/^\d+B$/);
}
});
it("validates against Asahi installer Python parser", () => {
// Download the Asahi installer and run its validation logic on our config
const validation = spawnSync("python3", ["-c", `
import json, sys
with open("${join(ASAHI_REPO, "installer_data.json")}") as f:
data = json.load(f)
errors = []
os_list = data.get("os_list", [])
if not os_list:
errors.append("Empty os_list")
for os_entry in os_list:
required = ["name", "default_os_name", "boot_object", "next_object", "package", "supported_fw", "partitions"]
for field in required:
if field not in os_entry:
errors.append(f"Missing field: {field}")
partitions = os_entry.get("partitions", [])
if not partitions:
errors.append("No partitions defined")
has_efi = False
has_root_image = False
expand_count = 0
for p in partitions:
if "name" not in p or "type" not in p or "size" not in p:
errors.append(f"Partition missing name/type/size: {p}")
if p.get("type") == "EFI":
has_efi = True
if p.get("format") != "fat":
errors.append("EFI partition must be FAT format")
if p.get("image"):
has_root_image = True
if p.get("expand"):
expand_count += 1
# Validate size format
size_str = str(p.get("size", ""))
if not size_str.endswith("B") or not size_str[:-1].isdigit():
errors.append(f"Invalid size format: {size_str} (expected NB)")
if not has_efi:
errors.append("No EFI partition found")
if not has_root_image:
errors.append("No partition with root image found")
if expand_count > 1:
errors.append(f"Multiple expanding partitions ({expand_count}) — only one should expand")
# Verify supported_fw is a list of strings
fw = os_entry.get("supported_fw", [])
if not isinstance(fw, list) or not all(isinstance(v, str) for v in fw):
errors.append("supported_fw must be a list of strings")
if errors:
print("ERRORS:")
for e in errors:
print(f" - {e}")
sys.exit(1)
else:
print("OK: installer_data.json is valid")
`], { encoding: "utf-8", stdio: "pipe", timeout: 10_000 });
if (validation.status !== 0) {
console.log(validation.stdout);
console.log(validation.stderr);
}
expect(validation.stdout).toContain("OK");
expect(validation.status).toBe(0);
});
});
describe("asahi rootfs ZIP validation", () => {
beforeAll(() => {
if (!hasBuiltArtifacts()) {
throw new Error("Run scripts/build-asahi-rootfs.sh first to generate artifacts");
}
});
it("ZIP contains required files", () => {
const result = spawnSync("unzip", ["-l", join(ASAHI_REPO, "fedora-asahi-lab.zip")],
{ encoding: "utf-8", stdio: "pipe", timeout: 10_000 });
expect(result.stdout).toContain("boot.img");
expect(result.stdout).toContain("root.img");
expect(result.stdout).toContain("esp/");
});
it("boot.img is ~1GB", () => {
const result = spawnSync("unzip", ["-l", join(ASAHI_REPO, "fedora-asahi-lab.zip")],
{ encoding: "utf-8", stdio: "pipe", timeout: 10_000 });
const bootLine = result.stdout.split("\n").find(l => l.includes("boot.img") && !l.includes("/"));
expect(bootLine).toBeDefined();
const size = parseInt(bootLine!.trim().split(/\s+/)[0]!, 10);
expect(size).toBeGreaterThan(500 * 1024 * 1024); // > 500MB
expect(size).toBeLessThan(2 * 1024 * 1024 * 1024); // < 2GB
});
it("root.img is > 3GB", () => {
const result = spawnSync("unzip", ["-l", join(ASAHI_REPO, "fedora-asahi-lab.zip")],
{ encoding: "utf-8", stdio: "pipe", timeout: 10_000 });
const rootLine = result.stdout.split("\n").find(l => l.includes("root.img"));
expect(rootLine).toBeDefined();
const size = parseInt(rootLine!.trim().split(/\s+/)[0]!, 10);
expect(size).toBeGreaterThan(3 * 1024 * 1024 * 1024); // > 3GB
});
it("rootfs contains lab-firstboot.sh", () => {
const mountDir = join(tmpdir(), `asahi-rootfs-check-${Date.now()}`);
const extractDir = join(tmpdir(), `asahi-rootfs-extract-${Date.now()}`);
mkdirSync(mountDir);
mkdirSync(extractDir);
try {
// Extract root.img from ZIP
run(`unzip -o -j "${join(ASAHI_REPO, "fedora-asahi-lab.zip")}" root.img -d "${extractDir}"`);
// Mount and check
run(`mount -o loop,ro "${join(extractDir, "root.img")}" "${mountDir}"`);
// Verify firstboot script
expect(existsSync(join(mountDir, "usr/local/bin/lab-firstboot.sh"))).toBe(true);
const script = readFileSync(join(mountDir, "usr/local/bin/lab-firstboot.sh"), "utf-8");
expect(script).toContain("#!/bin/bash");
expect(script).toContain("labvg");
expect(script).toContain("pvcreate");
// Verify systemd service
expect(existsSync(join(mountDir, "etc/systemd/system/lab-firstboot.service"))).toBe(true);
const service = readFileSync(join(mountDir, "etc/systemd/system/lab-firstboot.service"), "utf-8");
expect(service).toContain("lab-firstboot.sh");
// Verify service is enabled (symlink exists)
const symlinkPath = join(mountDir, "etc/systemd/system/multi-user.target.wants/lab-firstboot.service");
let symlinkExists = false;
try { lstatSync(symlinkPath); symlinkExists = true; } catch { /* not found */ }
expect(symlinkExists).toBe(true);
// Verify SSH keys
expect(existsSync(join(mountDir, "root/.ssh/authorized_keys"))).toBe(true);
// Verify lvm2 + xfsprogs are in the image
const hasLvm = existsSync(join(mountDir, "usr/bin/pvcreate")) || existsSync(join(mountDir, "usr/sbin/pvcreate"));
const hasXfs = existsSync(join(mountDir, "usr/bin/mkfs.xfs")) || existsSync(join(mountDir, "usr/sbin/mkfs.xfs"));
expect(hasLvm).toBe(true);
expect(hasXfs).toBe(true);
} finally {
run(`umount "${mountDir}" 2>/dev/null || true`);
rmSync(mountDir, { recursive: true, force: true });
rmSync(extractDir, { recursive: true, force: true });
}
}, 120_000);
});

View File

@@ -29,6 +29,17 @@ export interface PxeVmConfig {
diskSize: number; // GB
network: string; // libvirt network name
arch?: "x86_64" | "aarch64";
/**
* Extra NICs enumerated BEFORE the PXE NIC, on a network with no route to
* the bastion (defaults to libvirt's "default").
*
* Real multi-NIC boxes expose a class of bug a single-NIC VM cannot: an
* initramfs that picks "the first connected interface" grabs one of these
* instead of the NIC that PXE booted, and then cannot reach the bastion.
* Defaults to 0 (single NIC).
*/
decoyNics?: number;
decoyNetwork?: string;
}
/** Create a blank UEFI VM that PXE boots from the network. */
@@ -61,6 +72,10 @@ export function createPxeVm(config: PxeVmConfig): void {
`--memory=${config.memory}`,
`--vcpus=${config.vcpus}`,
`--disk=path=${diskPath},format=qcow2,bus=virtio`,
// Decoys first so they enumerate ahead of the PXE NIC. They are up and
// carry a lease, but have no route to the bastion.
...Array.from({ length: config.decoyNics ?? 0 }, () =>
`--network=network=${config.decoyNetwork ?? "default"},model=virtio`),
`--network=network=${config.network},model=virtio`,
// UEFI firmware — required for PXE boot in modern mode
`--boot=uefi,network,hd`,
@@ -95,12 +110,21 @@ export function destroyPxeVm(name: string): void {
}
/** Get the MAC address of a VM's first NIC. */
export function getVmMac(name: string): string | null {
export function getVmMac(name: string, network?: string): string | null {
const result = virsh("domiflist", name);
if (result.status !== 0) return null;
// Output format: Interface Type Source Model MAC
const match = result.stdout.match(/([0-9a-f]{2}:[0-9a-f]{2}:[0-9a-f]{2}:[0-9a-f]{2}:[0-9a-f]{2}:[0-9a-f]{2})/i);
return match ? match[1].toLowerCase() : null;
// With decoy NICs present, match the line for the PXE network so we return
// the NIC that actually boots rather than whichever is listed first.
const lines = result.stdout.split("\n");
const candidates = network === undefined
? lines
: lines.filter((l) => l.split(/\s+/).includes(network));
for (const line of candidates.length > 0 ? candidates : lines) {
const m = line.match(/([0-9a-f]{2}:[0-9a-f]{2}:[0-9a-f]{2}:[0-9a-f]{2}:[0-9a-f]{2}:[0-9a-f]{2})/i);
if (m) return m[1].toLowerCase();
}
return null;
}
/** Reboot a VM (force off + start). */

View File

@@ -161,7 +161,7 @@ EOF'
CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
curl -L --fail --silent "https://github.com/cilium/cilium-cli/releases/download/\${CILIUM_CLI_VERSION}/cilium-linux-amd64.tar.gz" | sudo tar xz -C /usr/local/bin
DEFAULT_DEV=$(ip -4 route show default | awk '{print $5}' | head -1)
sudo KUBECONFIG=/etc/rancher/k3s/k3s.yaml cilium install --set kubeProxyReplacement=true --set ipam.mode=kubernetes --set devices=$DEFAULT_DEV --set nodePort.directRoutingDevice=$DEFAULT_DEV
sudo KUBECONFIG=/etc/rancher/k3s/k3s.yaml cilium install --set kubeProxyReplacement=true --set ipam.mode=cluster-pool --set ipam.operator.clusterPoolIPv4PodCIDRList='{10.42.0.0/16}' --set ipam.operator.clusterPoolIPv4MaskSize=24 --set devices=$DEFAULT_DEV --set nodePort.directRoutingDevice=$DEFAULT_DEV
`.trim(), "cilium install", { keyPath: sshKeyPath, timeout: 120_000 });
log("Waiting for Cilium to be ready...");

Some files were not shown because too many files have changed in this diff Show More