eth3 <-> eth3 direct cable is in. Both ends negotiated 2500Mb full duplex --
these are 2.5 GbE ports, not the 1G I had assumed.
Carrier alone proves nothing, so the link was tested end to end with temporary
kernel-level addresses (never committed to VyOS config, removed afterwards):
3/3 packets, 0% loss, 0.371ms average. A cable can show carrier and still not
pass traffic; now it is known to.
Deltas regenerated with --conntrack-link and installed on both boxes:
vyos001 nat=21 fw=58 conntrack=9 eth3=10.255.255.1/30 disable=0
vyos002 nat=21 fw=58 conntrack=9 eth3=10.255.255.2/30 disable=2
Identical apart from the peer /30, the VRRP/DHCP-HA roles, and the two disable
lines holding vyos002's WAN down. Both still report mode=unifi and nothing about
their behaviour has changed -- eth3 carries no address in the running config,
and conntrack-sync appears only in the delta, which is applied at cutover.
Not verified: multicast on the peer link. `ping -I eth3 224.0.0.1` drew no
responders, but that is the all-hosts group which VyOS need not answer, so it
proves nothing either way. conntrack-sync's own multicast (225.0.0.50) was
proven working in labsim over bond0.10, and this is a point-to-point link, so
the risk is low -- but it is untested on this specific cable and worth watching
at cutover.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
"No NAT? How are we supposed to get internet?" -- a fair question that exposed a
worse design than I had admitted. Internet did work, but only via vyos001: NAT
and the entire WAN were gated behind --with-wan, so vyos002 would have held the
LAN VIPs and routed between VLANs with no path to the outside at all. Failover
would have preserved addressing and lost the internet.
The fix rests on a checked fact rather than an assumption: VyOS WARNS but still
commits when a NAT rule names an interface that does not exist
("Interface bond0.53 for source NAT rule 900 does not exist!"). Verified on a
real VyOS before relying on it.
So both boxes now get the identical WAN, NAT, port-forward and firewall config,
and the backup's two WAN interfaces are simply set `disable`. The cloned WAN MAC
is therefore never live on two boxes at once, while everything needed to route
and masquerade is already in place. The two deltas are now byte-identical apart
from VRRP priority, own/peer addresses, DHCP HA role, the conntrack /30 -- and
the two disable lines.
Taking over the internet path becomes deleting two lines rather than
reconstructing NAT under pressure:
delete interfaces bonding bond0 vif 53 disable
delete interfaces pppoe pppoe0 disable
Both boxes now: 21 NAT rules, 58 firewall rules, full PPPoE. Backup delta
validated against a real VyOS config with the disable lines present -- commits
clean. Runbook updated with the takeover procedure and the warning that it must
only be done when vyos001 is genuinely down.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
conntrack-sync proven working on the sim pair -- bidirectional replication with
zero errors:
MASTER internal 34 external(from peer) 52 62 pkts sent / 109 recv 0 err
BACKUP internal 76 external(from peer) 36 142 pkts sent / 73 recv 0 err
Getting there required learning something that changes the production config:
**VyOS only engages conntrack when a firewall or NAT is configured.** With
neither present, both routers reported zero conntrack entries and conntrack-sync
had nothing to replicate. Adding a single state-matching forward rule turned
tracking on and replication began immediately.
That is a defect in the delta, not just a test artifact. NAT and the firewall
were both gated behind --with-wan, so the BACKUP would have had neither -- it
would not have tracked connections at all, and replicated entries are useless to
a box whose conntrack is not engaged. Exactly the failure that only shows up
during a failover, when it is too late to notice.
Fixed: a stateful forward rule (accept established/related, default-action
accept) is now emitted on BOTH boxes, outside the WAN gate. Only NAT and the
WAN-scoped rules remain master-only. Verified: vyos002 now carries stateful
tracking and conntrack-sync but zero NAT lines. Master delta re-validated
against a real VyOS config -- no errors.
Also incidentally confirmed no-preempt: router1 rebooted and came back as
BACKUP rather than seizing the VIP, which is the opposite of what the production
pair did this afternoon (still on default preempt until cutover).
Two traps recorded while doing this:
- The detached `setsid nohup` config-apply pattern can strand a VyOS config
session. An orphaned session (dirs under /opt/vyatta/config/tmp/, PID long
dead) blocked every subsequent `set` on that box with a bare "Set failed",
and the dirs are overlay mounts so they cannot simply be deleted. Rebooting
cleared it. This pattern is used to survive losing SSH mid-change, so it is
worth knowing it has a failure mode of its own.
- Only VLAN 10 passes traffic between the two sim routers; every other VLAN
fails ARP despite identical vlan_mode/tag/trunks on both OVS bonds and
distinct MACs. VRRP forms on all six groups regardless. The sync link had to
be bond0.10 as a result. OVS-specific, absent in production, but it means
the sim proves mechanism rather than topology.
Production deltas regenerated with --conntrack-link: eth3 at 10.255.255.1/30 and
.2/30 awaiting the cable, which is not yet plugged (carrier=0 on both).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
Answers the question a single router could not, and that would otherwise only
have been discovered at cutover: with kea high-availability active-passive, does
exactly ONE box answer a DHCP request?
Yes. Probing sim VLAN 10 with broadcast-dhcp-discover returns offers from a
single distinct Server Identifier -- 172.31.10.252, the primary. The secondary
runs kea but stays silent. Without this the delta would have put 6 subnets and
84 static-mappings on both boxes with nothing to arbitrate them, and two kea
instances would have raced on every broadcast domain.
Worth noting the raw response count is misleading: nmap reports "Response 1 of
2" because it sends several discovers, and both replies carry the same server
identifier. Counting responses says "2 servers"; counting distinct server
identifiers says "1". The second number is the true one.
labsim now runs a real pair, mirroring production:
router1 172.31.<v>.252 priority 200 DHCP HA primary
router2 172.31.<v>.253 priority 100 DHCP HA secondary
VIP 172.31.<v>.1 floating, held by the master
That required converting router1, which held .1 directly, to .252 plus a
floating VIP -- otherwise it is two routers, not a pair. All six VRRP groups
show MASTER on router1 and BACKUP on router2.
New tooling:
- sim-ha-config.py generates each role's config, reusing unifi-to-vyos.py
--mode sim for the DHCP half so what is proven here and what production
gets share a code path. VLAN 10 correctly carries /23.
- console-apply.py applies config over the serial console, which is necessary
because a freshly installed VyOS holds the same addresses as its peer and
cannot safely be reached over the network at all until reconfigured.
Known sim-only quirk, deliberately not chased: router1 cannot ARP router2 on
the untagged VLAN 1 while every tagged VLAN works, and VRRP forms correctly on
all six groups regardless. Both OVS bonds carry identical vlan_mode/tag/trunks
and the bond MACs differ, so this is OVS bond behaviour on the native VLAN with
two bonds on one bridge -- not a VyOS config problem, and not present in
production, which uses a real switch.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
Prompted by "I thought we tested HA on libvirt" -- checking rather than
recalling showed the sim has ONE VyOS router with zero high-availability
config. VRRP was configured and running on the real pair, but it is only one of
four parts of what VyOS considers an HA pair.
Against docs.vyos.io (highavailability, conntrack-sync, dhcp-server, and the HA
walkthrough), three gaps are now closed in the delta:
- VRRP was multicast-only with default preemption. Added unicast
hello-source-address/peer-address per group, as the walkthrough does, plus
no-preempt. Without no-preempt a recovered box reclaims the VIP before
conntrack state has synced and drops every established connection; the docs
are explicit that preempt-delay must otherwise be >= purge-timeout.
The per-VLAN node addresses are a table, not derived: VLAN 3 is .4/.5 while
every other VLAN is .252/.253.
- DHCP high-availability, which fixes a real defect rather than adding a
feature. Both boxes carried the full 6 subnets and 84 static-mappings, so
after cutover two kea instances would have raced on the same broadcast
domains. Now active-passive with primary/secondary and swapped
source/remote, syncing over TCP 647 on the LoT addresses. Each subnet
already carries the unique subnet-id kea HA requires, and the peer name
deliberately differs from both host-names.
- conntrack-sync over a dedicated eth3 <-> eth3 link, gated behind
--conntrack-link because it needs a cable that is not plugged in yet. This
is what the peer cable is actually for -- VRRP does not want one, since its
hellos must travel on the segment they protect.
VRRP failover exercised on the production pair, which is free to break today
because nothing uses the .254 VIPs: keepalived stopped on vyos001, all six VIPs
moved to vyos002 within 12s, and returned on restart (preemption still default
on the live boxes). Both boxes clean afterwards, no config drift.
Master delta validated against vyos001's real running config on the sim router
before installing. Installed on both: 6 no-preempt, 6 unicast pairs, DHCP HA
primary/secondary respectively.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
Two corrections from reading the live USG instead of trusting UniFi's fields,
which report wan_type=dhcp for both WANs and are simply wrong:
- There are TWO WANs, not one. WAN2 is the 10 gig ISP on VLAN 53, plain DHCP
with a PUBLIC address (87.192.101.48/21, gw 87.192.96.1) on the USG's eth2 --
and it is what actually carries traffic. WAN1 is Vodafone PPPoE on VLAN 51,
the failover. The delta had PPPoE as the only WAN, which would have left the
primary line unconfigured.
- The DHCP lease is bound to MAC, so bond0.53 now clones the USG's WAN2 MAC
(f0:9f:c2:12:9b:4f). That is how VyOS keeps the existing public lease rather
than negotiating a new one -- or getting none, if the ISP allows one per
line. Distances: 10 gig at 1, Vodafone at 10.
Only ONE box may hold the cloned MAC, so --with-wan gates the entire WAN, NAT
and firewall section. vyos001 gets it (320 set lines); vyos002 gets none (234,
zero WAN/NAT/firewall) and routes the LAN only. Pretending both could hold it
would have meant a duplicate MAC on VLAN 53 and a flapping switch table.
Private was rebuilt at 10.8.0.0/23 (VLAN 9) after the old 10.0.8.0/23 was
deleted. bond0.9 and the VRRP group were moved to 10.8.0.252/.253 with VIP
10.8.0.254 on both boxes, and the delta now targets 10.8.0.1.
Creating that network first required breaking a deadlock in UniFi: every LAN
write was rejected with api.err.WanIpOverlapped / 0.0.0.0/0, because WAN1 was
set to DHCP on a line that only speaks PPPoE, so it sat at 0.0.0.0 forever and
the validator treated that as a subnet overlapping everything. Verified
server-side, not a UI bug -- the API rejected it identically. Setting
wan_type=pppoe let it dial (90.241.226.213, MTU 1492), which cleared the phantom
overlap and incidentally PROVED the Vodafone credentials and line work, which
had been listed as untestable before cutover.
dhcp-options no-default-route-dns does not exist; the valid set is client-id,
default-route-distance, host-name, mtu, no-default-route, reject, user-class,
vendor-class-id. Caught by validating the delta against vyos001's real config on
the labsim router before installing.
After adding the network, the gateway's dhcpd.conf was checked with
`dhcpd3 -t -cf` (valid) and confirmed to contain the new subnet only after a
force-provision -- controller state is not device state.
Both boxes: mode unifi, VRRP unchanged, unifi.boot re-captured (232 lines,
carrying the new VLAN 9), no config drift.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
The NAS is legacy for ad.itaz.eu and those records now live in Cloudflare, so
the zone resolves publicly -- verified: nas001.ad.itaz.eu and
kvm-macstudio1.ad.itaz.eu both answer from 8.8.8.8. That removes the reason for
a conditional forward and lets the NAS leave the DNS path entirely.
Two changes:
- `service dns forwarding name-server` is now 8.8.8.8 and 8.8.4.4, the same
pair the USG used on its WAN, instead of 10.0.0.194.
- Every VLAN is handed the gateway as its resolver. UniFi set an explicit
resolver on LoT only (the NAS); carrying that over would have kept the NAS
in the path for one VLAN and not the other five, which is the sort of
asymmetry nobody remembers a year later.
The NAS is still referenced 9 times, all legitimate and checked: 4 NAT
destination rules, the 4 matching firewall accepts for those port forwards, and
its own DHCP reservation. No DNS references remain.
Validated by loading vyos001's real running config on the labsim router and
applying the full delta -- all 318 commands accepted, no errors. Installed on
both boxes and verified in place: priority 200/100, upstream 8.8.8.8 + 8.8.4.4,
six client resolvers, 377 lines each.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
Matches the DNS the USG used on its WAN (wan_dns1/wan_dns2), replacing the
10.0.0.194 I had set earlier. Applied to both boxes and saved; VRRP unchanged
(MASTER/BACKUP), NTP still synced, no config drift.
/config/modes/unifi.boot was RE-CAPTURED on both afterwards. It had been taken
before this change, so the escape hatch would have quietly reverted the
resolver on any rollback -- a snapshot is only an escape hatch for the state it
was taken from.
Two things recorded in the runbook:
- The boxes' name resolution now depends on the internet, so between
unplugging the USG and PPPoE establishing they have no DNS. Harmless:
nothing in the switch resolves a name, and the health checks use DNS
precisely to prove the WAN came up.
- Internal ad.itaz.eu names still resolve via Google, because that zone is
published publicly with private addresses in it (nas001 -> 10.0.0.194,
kvm-macstudio1 -> 192.168.3.8). So no conditional forward was needed --
though it is worth knowing the internal topology is public.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
The access table led with 192.168.8.143/.144 and offered the LoT addresses as a
fallback ("if unreachable, try"). That is backwards and would have stranded the
operator at the worst moment: the workstation sits on LoT, and reaching
192.168.8.x routes *through the USG*, so those addresses are guaranteed dead the
instant it is unplugged. Measured:
ip route get 192.168.8.143 -> via 10.0.0.1 (the USG)
ip route get 10.0.1.252 -> dev lanbr0 (same L2, no gateway)
10.0.1.252 and .253 are on the LoT VLAN, same broadcast domain as the
workstation, and both answer SSH. They are now the only addresses the runbook
gives, with the k8s ones struck through.
Also recorded: the switch cannot be run before unplugging the USG (two devices
on every gateway address; the guard refuses), so the order is forced. And
during the gap between unplugging and completing the switch there is no
inter-VLAN routing at all -- which means the JetKVMs (Management and kvm) and
Tailscale are NOT fallbacks in that window. LoT SSH is the only remote path;
below it is physical console. Added a step 0: open both SSH sessions and leave
them open before touching anything.
Both boxes are now installed and pass the pre-flight gate: mode unifi,
unifi.boot 231 lines including the reload action, delta at the right priority
(200/100), wan-secrets 0600, script executable, no config drift, VRRP still
MASTER/BACKUP. `vyos-unifi-switch vyos` refuses on both -- all six gateway
addresses detected answering ARP -- and neither box has gained dhcp-server, dns
or nat, so nothing about their behaviour has changed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
This caused a real outage. unifi-reserve-all.py recomputed its plan at --apply
time by re-reading stat/sta, so a client that renewed between the dry run and
the apply was pinned to whatever transient address it happened to hold at that
instant. worker1-k8s0 was reviewed at 192.168.8.13 and written as
192.168.8.242. On its next reboot it could not get an address at all, taking a
k8s node down.
A plan that gets reviewed and a plan that gets applied must be the same object.
The dry run now WRITES the plan to a file and --apply READS it and applies
exactly that, reporting any client whose current address has since drifted
rather than silently preferring the new value.
1 of 51 diverged; the rest were verified against the reviewed list and were
correct. worker1 has been restored to .13 and confirmed: DHCPOFFER for its own
MAC returns 192.168.8.13, and the node is up with a full lease and working
internet.
The second half of the outage was drift between controller and device: the USG
was still running config from ~16h before these changes, so the controller
looked perfectly correct while the gateway handed out something else. Writing
the controller is only half the job, so the script now says so explicitly and
gives the force-provision and DHCP-probe commands to verify with. `nmap
--script broadcast-dhcp-discover --script-args broadcast-dhcp-discover.mac=...`
is the way to prove a specific reservation is live without disturbing the
client -- it elicits an OFFER without ever sending a REQUEST.
_unifi.py gains post() for device commands.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
The ARP guard against two devices holding the same gateway address is the one
check that prevents this script's worst outcome. It reads the addresses to
probe out of the delta -- so a delta with no VIP lines made the guard inert,
and it previously warned and carried on. That was a testing convenience (the
lab delta has no VIPs) weakening a production safety check, which is backwards.
It now refuses by default. ALLOW_NO_VIP_DELTA=1 is the explicit lab override.
The guard's probing path had never actually executed before this: every sim
run took the no-VIPs branch. Verified against the live USG from vyos001:
arping is present on VyOS, the regex extracts all six gateway addresses from
the real delta (192.168.1.1, 192.168.8.1, 192.168.3.1, 10.0.9.0, 10.0.0.1,
192.168.2.1), and every one of them answers ARP right now -- so on the real
boxes, with the USG connected, the guard fires.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
Ran vyos001's real running config plus the real production delta on the labsim
router -- same VyOS version, isolated OVS bridge with no physical NIC, so the
sim briefly holding vyos001's actual addresses could not reach the real LAN.
Result: all 317 commands accepted and the whole delta commits (COMMIT OK), the
revert is byte-exact, and auto-revert fires without rebooting (uptime and
boot-id unchanged across it).
Also written down are the two things this did NOT establish, because a runbook
that overstates its own coverage is worse than one that admits the gap: PPPoE
cannot be tried while the USG holds the single available session, and the
rehearsal ran with vyos001's eth2/eth3 stanzas stripped because the sim VM has
two NICs rather than four.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
`set nat destination rule N translation port '16881,6881'` is rejected --
"16881,6881 is not a valid service name" -- because mapping a list of ports
onto a list is ambiguous. `destination port` accepts the same list happily,
which is why this only shows up on the translation side.
All four UniFi port forwards map a port to itself, so translation port was
redundant anyway: omitting it makes VyOS preserve the original port, which is
exactly the intent. It is now emitted only when the forwarded port genuinely
differs, and generation fails loudly rather than producing a config that will
not commit if a differing port LIST ever appears.
Found by loading vyos001's real running config onto the labsim router and
applying the full delta to the candidate config without committing. Worth
noting the delta had already passed a read-through: this one only surfaced by
running it against a real VyOS of the same version.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
`set interfaces pppoe pppoe0 source-interface bond0.51` refers to an interface
that must already exist, and neither firewall has vif 51 -- only 2, 3, 9, 10
and 200 are configured. The commit would have failed, and since the whole delta
commits as one unit, that failure would have taken the entire switch with it at
the worst possible moment.
No address on the vif: PPPoE rides the VLAN and needs no L3 of its own.
Found by checking the running config against the generated delta rather than by
running it. The prod delta has still never been applied to any VyOS, which is
the remaining gap.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
The cutover is a switch, not a migration: unplug the USG, run one command, and
if anything is wrong run the other one and plug it back in. The operator will
have no internet during this and therefore no assistant, so the machinery has
to live on the boxes and the failure paths have to be proven in advance.
vyos-mode-delta.py generates the delta that turns the passive pair into the
gateway. Only one artifact is authored: gateway mode is always derived from
`load unifi.boot` + delta, so there is no inverse to maintain and no drift
between two hand-kept configs. It reuses unifi-to-vyos.py rather than
duplicating it, so what labsim proved and what production gets are one code
path. The PPPoE password is never written into the delta -- it carries a
placeholder the switch substitutes at apply time from /config/wan-secrets --
and generation fails if the real password appears in the output.
Two things the delta covers that the plan had underweighted:
- VyOS defaults to ACCEPT while the USG has an implicit WAN drop. Migrating
the port forwards alone would have left the router's own services and the
whole LAN reachable from the WAN. Added a stateful baseline scoped to the
WAN interface rather than a global default-action drop, so a mistake there
cannot lock anyone out over the LAN -- the only way back during a cutover.
- The old VIPs are NOT at network+254 on the /23 networks; they are
192.168.9.254, 10.0.9.254 and 10.0.1.254, in the upper half. A delete
naming a computed address fails quietly and leaves the group holding two
VIPs. The delta deletes the whole address node instead of guessing.
vyos-unifi-switch runs on the box from /config, which survives image upgrades,
so it works from a local terminal or the JetKVM with no workstation.
Proven on labsim, not assumed:
- unifi mode restores the previous config BYTE-EXACT (138 lines, diff clean).
- Auto-revert fires when the commit is not confirmed: 85 static-mappings ->
0, kea stopped, hostname restored, and uptime plus boot-id UNCHANGED, so
it reloaded rather than rebooted. That distinction is the whole reason
`commit-confirm action reload` is a prerequisite.
- Health-check failure triggers an immediate revert_soft rather than waiting
out the timer.
Four bugs found while doing it, each of which produced a wrong answer rather
than an error:
- commit-confirm is TWO steps. `config-mgmt commit_confirm` only arms the
revert timer; a normal `commit` still has to follow. Arming alone committed
nothing while reporting success.
- `sudo sg vyattacfg "config-mgmt ..."` loses the config-session environment,
so it reported "No configuration changes to commit" against a candidate
that plainly had 446 added lines.
- `... | grep -q` under `set -o pipefail` reports FAILURE on a match: grep
exits early, the producer takes SIGPIPE. Whether it triggers depends on
output size, so `status` misreported the mode intermittently.
- `show configuration commands` quotes values, so a fixed-string match for
`action reload` never matched `action 'reload'`.
CUTOVER.md is the printable runbook: both reachable addresses per box, the
escape hatch first, and the note that PPPoE is the one thing that could not be
tested beforehand.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
kea does not inherit UniFi's lease database. At cutover it starts with an empty
view of who holds what, so it can hand an address that is currently in use to a
different device. Reservations are what carry "this device has this address"
across the switch, because they live in config rather than in lease state.
unifi-reserve-all.py creates one per active client, dry run by default. 51
written, 51/51 verified live by reading the records back; the controller now
holds 85 reservations and the generator emits all 85 with unique, valid
hostnames and no duplicate addresses. Active clients with no reservation went
from 48 to 4.
Three guards, each of which caught something real in the dry run:
- VRRP virtual addresses are excluded. UniFi reports them as ordinary client
addresses because the firewalls' bond MACs answer for them, and their
apparent IP flips between the real interface address and the VIP. Without
this, 192.168.1.254 -- the gateway VIP itself -- would have been given a
DHCP reservation.
- The firewalls' own interface MACs are excluded; those are statically
configured routers, not DHCP clients.
- Any address claimed by more than one MAC is dropped rather than guessed
at. This is how the VIPs surfaced in the first place.
Also skipped: addresses already reserved to another MAC, network gateways, and
anything on a network that does not serve DHCP (which excludes the WAN transit
VLANs automatically).
labsim-dhcp-test.sh gained a lease-database flush, and it is not tidiness. Two
findings, both of which first appeared as a PASSING test:
- Re-running against stale leases, kea gave dynamic addresses to three
devices that have reservations. The reservations were present and correct
in kea's own config throughout. Kea saw the reserved address as leased to
"another client" -- same MAC, different client-id from the earlier boot --
and allocated elsewhere. Cutover starts with an empty lease database so
this is a testing artifact, but a reservation is evidently not
unconditional once leases exist.
- Removing only dhcp4-leases.csv does nothing: kea's memfile backend keeps
lease-file-cleanup rotations (.csv.2) and restores from them on start.
The verdict logic no longer takes the first matching lease row. Doing so
reported an hours-old lease as the current answer and scored three failures as
passes, including one where the device had plainly been given a dynamic
address. A MAC with more than one lease is now an explicit failure rather than
a guess.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
Prerequisite for the cutover. eth2 on both firewalls was DHCP-served by the USG,
and both addresses (192.168.8.143/.144) sit inside the pool VyOS will serve --
with no reservation for either MAC. After the switch they would renew from kea,
get no mapping, and the management addresses could move. That is the worst
possible moment for the address you SSH to to change, because losing the USG
also means losing internet and any outside help.
On both boxes, driven over the LoT path (bond0.10) so the interface being
changed was never the one carrying the session:
- eth2 pinned static at its current address, so it no longer depends on DHCP
- `system name-server eth2` replaced with 10.0.0.194. That setting inherited
resolvers from the DHCP lease, i.e. the boxes were resolving via the USG and
would have lost DNS with it. 10.0.0.194 is reachable directly over bond0.10
and is authoritative for ad.itaz.eu, so internal names now resolve on the
firewalls -- they did not before.
- static default route via 192.168.8.1, replacing the one the lease provided.
Superseded by PPPoE in vyos mode; this keeps unifi mode as it was.
Verified after each: SSH on the pinned address, external and internal DNS, NTP
still synced, VRRP unchanged (vyos001 MASTER, vyos002 BACKUP).
migration/unifi-reserve.py adds the matching UniFi reservations so the
controller cannot lease those addresses to anything else, keeping the
management address identical in both modes. It reads the record back after
writing, because a controller accepting a PUT is not proof it stored what was
asked for, and it is idempotent.
Also noted while doing this: VyOS `commit-confirm` REBOOTS the box if not
confirmed -- "Minutes until reboot, unless 'confirm'" -- it does not roll the
config back in place. For a gateway that means a real outage window, which
changes how the switch script must use it. `config-mgmt commit_confirm -y`
executes without the interactive prompt.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
Groundwork for replacing the USG with the VyOS pair without anything on the
network noticing. Three pieces:
migration/unifi-export.py pulls 13 endpoints off the classic controller into
timestamped JSON plus a normalised inventory: 11 networks, 31 DHCP
reservations, 4 port forwards, 2 firewall rules, 0 static routes. Two things
this turned up that a naive export would have lost:
- 23 of the 31 reservations carry no network_id at all -- UniFi simply does
not store the binding -- so they are resolved by subnet containment
instead. Without that, three quarters of the reservations have no subnet
to be placed in.
- 30 of the 31 sit INSIDE the DHCP pool, which UniFi's dhcpd tolerates and
which is flagged as a warning rather than discovered at cutover.
migration/unifi-to-vyos.py turns that inventory into VyOS `set` commands for
DHCP and DNS only -- the services the USG owns that VyOS must reproduce. Not a
general converter. --prod and --sim come from one code path so the config
proven in the sim and the config applied to the firewalls cannot drift. Prod
mode hard-fails if any reservation is missing, since a silent drop is the
failure mode that matters.
DNS is included because the USG resolves for 5 of 6 VLANs today: UniFi hands
out the gateway's own address whenever dhcpd_dns is empty, verified by
labmaster resolving against 192.168.8.1. Replacing the USG without a forwarder
would take DNS away from those VLANs entirely.
labsim/labsim-dhcp-test.sh proves it by booting throwaway VMs with real
production MACs -- the one piece of production config that transplants
verbatim. Safe because ovs-labsim has no physical NIC, so those MACs cannot
reach the real LAN.
Result on VyOS 2026.08 (kea), 4/4: printer1 got 172.31.10.46 from inside the
pool, sonoff-matter got 172.31.11.67 across the /23 boundary, Hubitat got its
out-of-pool .2, and an unreserved MAC got an unreserved address. kea honours
in-pool host reservations -- the open question blocking the cutover.
Supporting changes to labsim:
- VLAN 10 widened to /23. Every reservation is in LoT and LoT spans 10.0.0.x
and 10.0.1.x, which a /24 cannot represent.
- LoT's host leg moved to .3, because 10.0.0.2 is a real reservation
(Hubitat) that maps onto the host's own address.
- vlans.conf gained optional masklen and host_octet fields, defaulting to
24 and 2 so the other five VLANs are untouched.
- Fixed /etc/network/interfaces hardcoding 255.255.255.0. That file is what
actually takes effect on these Alpine guests -- cloud-init's
network-config is ignored -- so any non-/24 VLAN was silently wrong.
Two generator bugs found by VyOS rejecting the output: static-mapping names
are validated as hostnames, so underscores fail; and two devices named
"espressif" plus two named "thebeast" collided into single names, which would
have overwritten one reservation with another's address.
The raw export holds WiFi passphrases and the WAN PPPoE credentials and is
gitignored.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
Replace the CIS-style disableSwap op with enableSwap: activate the
labvg-swap LV with an fstab entry (kubelet runs failSwapOn=false; zram
stays the fast tier, the LV is overflow before OOM kill). Add
growRancherLv: extend labvg/rancher to 120G when the VG has free space,
covering nodes installed before the kickstart sizing change and vanilla
nodes converted to k8s; skips with a clear message when the VG is full.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017f6jyeeDqP4ufyeL3UER9w
The 20G /var/lib/rancher LV (k3s imageFs) idled at 85% used from
steady-state images alone; one ~5G image pull tripped imagefs eviction
and evicted unrelated pods (2026-08-14 DiskPressure incident). Create
the LV for both worker and infra roles at 120G — it must be sized here
because longhorn's --grow consumes all remaining VG space, making
post-install lvextend impossible.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017f6jyeeDqP4ufyeL3UER9w
The Grafana heatmap of 1s and 0s said almost nothing, and the state timeline
was an unreadable pile of overlapping series labels. Replaced as the primary
view with a purpose-built page served by the exporter itself.
- Probe now captures ICMP RTT, exposed as labsim_rtt_ms{src,dst}. A path that
is up but slow is a different problem from one that is down, and a pass/fail
grid cannot show it.
- Exporter serves / (topology), /api/matrix (JSON) and /metrics.
- topology.html: node per VLAN in a ring, VyOS router in the centre because
every inter-VLAN packet really does traverse it, one line per pair coloured
green/red with the RTT on it. Hovering gives per-direction state. A node ring
goes red if anything to or from it is blocked. Side panels list blocked paths
and the slowest links. Refreshes every 5s, no dependencies.
Grafana stays for what it is actually good at — history of when a path flipped.
Label placement is deliberate: RTT captions sit ~32% along each edge with a
perpendicular nudge, because every diagonal of a 6-node mesh crosses the centre
and midpoint labels stack on the router node.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
A throwaway copy of the production VLAN topology so routing and firewall
changes can be tested before they touch the real network. Same VLAN IDs and
roles as UniFi, deliberately different ranges (172.31.<vlan>.0/24) so nothing
here can be mistaken for production.
- OVS fabric: real 802.1Q. Access port per micro VM, host leg per VLAN (.2,
for SSH only — NOT the VMs' default route, so inter-VLAN tests exercise the
router rather than the host's routing table), and a trunk portgroup with
VLAN 1 declared nativeMode='untagged'.
- Six Alpine micro VMs (256MB, copy-on-write overlays on one 176MB image),
SSH + a hello-world HTTP page naming the VLAN.
- VyOS router installed to disk unattended over the console, with the SAME
config shape as the VP2440s: two NICs in an LACP bond carrying the trunk,
VLAN 1 native, bond0.<vlan> holding the .1 gateway on each.
- labsim-matrix.py: full-mesh ICMP/TCP22/TCP80 probe, ~0.2s, --watch
highlights cells that changed since the last sweep. Guest-side probe is
python3 (already present via cloud-init) so nothing is installed on VMs
that have no internet.
- Prometheus + Grafana (anonymous auth, no login) with a provisioned
dashboard: heatmap plus a state timeline showing exactly when a path
flipped. Verified end to end: one VyOS rule took sum(labsim_reachable)
from 90 to 84, blocking precisely kvm<->k8s across all three protocols.
Traps found building this, all now encoded in the scripts:
- virtio-net breaks 802.3ad: the guest's bonding driver reports slaves
"MII Status: down" despite carrier=1 and never sends an LACPDU, so the bond
sits in AD_STATE_DEFAULTED. e1000e fixes it with no other change. Matches
the netdev thread "bonding (IEEE 802.3ad) not working with qemu/virtio".
- OVS defaults bonds to active-backup, which does not speak LACP at all —
bond_mode=balance-tcp is required.
- LACP deadlock: OVS holds members disabled until negotiation while the
partner needs carrier before it will send LACPDUs. lacp-fallback-ab breaks it.
- LACPDUs are untagged, so a trunk with no native VLAN has nowhere to put them.
- --boot cdrom,hd re-runs the ISO on every restart, so every commit+save went
to a live system that evaporated. Install now switches the VM to boot hd.
- cloud-init on Alpine: users stay locked without lock_passwd:false, one
failing runcmd aborts the rest, busybox here has no httpd applet, and
start-stop-daemon --exec /usr/bin/python3 matches cloud-init's own python3.
- The user-data heredoc is unquoted, so backticks in a COMMENT were executed
by the host shell and their output corrupted the YAML. build_seed now
validates with yaml.safe_load before building the ISO.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
Both Protectli VP2440s failed to install on real hardware: they fetched
kernel+initrd and then went silent. The console showed why —
Looking for a connected Ethernet interface ... e2 ? e3 ? e4 ? e5 ?
Connected e4 found
Connected e5 found
[4.595647] igc 0000:02:00.0 e2: NIC Link is Up
IP-Config: e4 ... no response after 15 secs - giving up
Unable to find a live file system on the network
live-boot picks the first *connected* interface. The i40e SFP+ pair links
before the igc copper port (up at 4.6s), so it chose the fiber ports, which
have no DHCP, and never tried the NIC that actually PXE booted.
Fix: pass BOOTIF=01-<mac> on the kernel cmdline. live-boot's
Device_from_bootif() (verified present in this image) matches it against
/sys/class/net and sets DEVICE directly. The MAC comes from the dispatch
key — i.e. exactly the NIC that PXE booted — which is more reliable than
iPXE's ${net0} on a box where the booting NIC may not be net0.
Why the integration test missed it: the VM had ONE NIC, so "first connected
interface" was trivially correct, and virtio links instantly so there was no
negotiation race. createPxeVm now takes decoyNics, attaching extra NICs
ahead of the PXE NIC on a network with no route to the bastion; the VyOS
test uses 2. Without BOOTIF that reproduces the hardware failure. getVmMac
is network-aware so it still returns the booting NIC.
Also: the bastion had every clue and said nothing — it logged INSTALL
STARTED, served kernel+initrd, then nothing for 7 minutes. dispatch now
stamps dispatched_at, and /api/logs/:mac returns stalled_for_s / stalled
(8 min threshold, sized for the ~600MB squashfs fetch), so a machine wedged
before the installer environment comes up is diagnosable without a console.
Verified on hardware: both firewalls installed, bond0 802.3ad + VLANs
2/3/9/10/200 + VRRP (priority 200/100, VIP .254 per VLAN) applied, and
/config/lab-provisioned written.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
pnpm completions:check was failing: labctl.fish/bash were stale. The
generated --os choices still listed only fedora-43 and ubuntu-26.04
(missing vyos-rolling since the OsId union gained it), and none of the
--vyos-*/--vlan flags were present.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
Ports the Fedora provisioning features that matter for a router onto the
VyOS path, and adds the libvirt integration test that proves them.
- Live install logs: the driver streams the installer pty (ANSI-stripped,
batched, best-effort) to POST /api/log, so `labctl provision logs -f`
works during a VyOS install the way Anaconda's syslog does for Fedora.
- installed.ip: report "ready at <ip>" -- the exact detail format
routes/api.ts parses -- using the static mgmt address when known, else
the live DHCP address. Without it VyOS machines landed with an empty IP,
breaking provision list, logs-by-IP, recheck and reprovision.
api.ts also guards the complete handler: VyOS boxes get the "vyos" SSH
hint and never trigger the k3s post-provision.
- EFI network-first boot order: port of the Fedora %post efibootmgr step,
run from the live env after install (NVRAM, not disk). Best-effort.
- Reinstall semantics: VyOS's installer already carries the previous
config and SSH host keys forward -- the analog of Fedora's LV
preservation -- so that stays the default. New --vyos-fresh-config
overwrites the installed config.boot with the generated one instead,
via a post-install target mount that also writes /config/lab-provisioned
(mirrors Fedora's /etc/lab-provisioned, survives image upgrades).
- reprovision/recheck default to the "vyos" SSH user for VyOS machines.
Two hangs found by the VM test and fixed:
- On reinstall the installer asks "Would you like to copy data to the new
image?" (search_previous_installation). Unanswered, the driver blocked
on stdin until its stall timeout -- a silent 15-minute hang.
- The RAID regex missed "Would you like to choose two disks for RAID-1
mirroring?", which would wedge any multi-disk box. Both prompts default
to yes, so a miss also risks an unwanted mirror.
Both are now covered by a unit test asserting all 17 installer prompts
match exactly one rule -- verified to fail against the unfixed code, so
this class of bug is caught in a second instead of a 45-minute VM run.
tests/integration/vyos-provision.test.ts: fresh install, reinstall
preserves config + /config data, and freshConfig override. All 8 pass
against the real nightly ISO (EXIT=0). 273 unit tests pass; no new lint
errors in touched files.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
VyOS ships no unattended installer (install_image() is unconditionally
interactive; --no-prompt is wired only to 'add'), so the automation is
injected through live-config's hooks component: iPXE boots the live
kernel with fetch= and live-config.hooks=, the hook fetches a generated
per-MAC Python driver, and the driver builds config.boot, stages the
rootfs, and drives the interactive installer over a pty.
Bastion:
- vyos-boot.ipxe template (no 'nonetworking' — breaks the hook fetch;
no console=ttyS0 — 30s/systemd-phase on UART-less boards)
- /vyos/autoinstall.sh + /vyos/install.py routes (per-MAC driver with
the config spec baked in as base64)
- vyos-config-spec: bond0 (802.3ad) + tagged VLANs + VRRP groups
(vrid = VLAN id) + sync group + hw-id pinning by MAC + SSH keys;
config built from the image's own config.boot.default via
vyos.configtree, version footer reattached via component_version
- prepareVyosArtifacts: extract kernel/initrd/squashfs from the nightly
ISO with xorriso; initrd picked by size from regular files only;
ISO URL "latest" resolves the newest vyos-nightly-build GH release
(downloads.vyos.io no longer serves direct ISOs)
Verified end-to-end in a libvirt VM against the real nightly ISO —
installed system boots with bond/VRRP/hw-id config applied and no
migrations. Fixes found by the VM run, encoded in code comments:
config.boot.default lives at /usr/share/vyos at hook time; fetch= boot
has no medium so the rootfs is symlinked to the installer's expected
path; reboot must be --force (the hook is a child of the still-starting
live-config unit); installer disk answers are full /dev paths; zram0
passes the 2GB min-disk filter so the disk is always pinned.
CLI/labd: vyos spec threaded through provision install (--vyos-* and
--vlan/--vlan-vip flags with guards), labd install route, protocol
command-install, and the bastion's direct /api/install.
268 unit tests pass; no new lint errors in touched files.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
The macvlan/VLAN-10 foundation HA depends on (Multus meta-CNI + a lan10
sub-interface + reference CNI plugins) was applied by hand during the HA
migration. Codify both as idempotent lab operations in the networking group,
after installCilium (which already sets cni.exclusive=false + bpf.vlanBypass={10}).
A fresh cluster now reproduces the full macvlan stack.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The v2.0 Phase 1 commit (04faa07) added AuthService, RbacService,
ResourceStore, AuditService, the bearer auth middleware, and the
v2-auth/environments/resources route files, but createApp() never
registered any of them. They sat in the codebase as dead code: a
running labd would 404 on /api/auth/login, /api/resources, /api/events,
etc.
Wiring (server.ts)
- Instantiate AuthService, RbacService, ResourceStore, AuditService at
app creation. Cast DbClient to PrismaClient (the runtime db is a real
PrismaClient; DbClient is a structural shim).
- Start AuditService timer, register an onClose hook to stop it on
shutdown so we never lose the last batch.
- Register v2 routes inside a Fastify scope with the bearer-auth
middleware as preHandler. v1 routes (registered on the root scope)
are unaffected so existing labd clients keep working.
AuditService (audit.ts)
- Expose flushPending() so tests can deterministically observe events
without leaning on the 5-second flush interval. Implementation
delegates to the existing private flush().
Smoke tests (v2-smoke.test.ts, 11 cases)
- Bootstrap: first POST /api/auth/login with empty users creates the
admin (role=ADMIN, hashed password), returns a 64-hex token, marks
isBootstrap=true, emits an auth_bootstrap audit event. Second login
uses the normal flow. Wrong password returns 401 and audits failure.
Missing credentials returns 400.
- RBAC: missing/empty/invalid bearer tokens return 401. ADMIN role
bypasses RBAC. A non-admin with no role bindings gets 403 with
"no matching role binding". A user with an env-A binding is denied
for env-B resources.
- Audit: bootstrap event is queryable via /api/events?correlation=...
Explicit parent/child chain (shared correlationId, parentEventId)
is preserved across emits.
All 246 workspace tests pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
bastion serve/stop default for --dir was hardcoded to /tmp/lab-bastion.
Now reads BASTION_DIR from env if set, so a deployed bastion daemon
can run from a persistent directory without callers having to pass
--dir on every invocation.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The v2.0 Phase 1 commit (04faa07) introduced the @lab/core package but
the labd Dockerfile still only copied @lab/shared and @lab/labd, so the
container build would fail to resolve @lab/core imports.
Both stages updated:
- Builder: copy @lab/core package.json/tsconfig + src, add it to the
build order between @lab/shared and @lab/labd.
- Runtime: copy @lab/core dist and package.json into the final image.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The worker0-k8s0 bug: when labd restarts, the in-memory installed map
is lost. The next DHCP/PXE re-discovery for that MAC ran an upsert that
wrote status="discovered", silently downgrading the DB record from
"online" or "offline" and erasing the machine's known hostname/role
identity from the CLI view.
- server.ts: drop status="discovered" from the upsert update branch so
re-discovery cannot downgrade an installed record.
- routes/bastions.ts (/api/machines): when the DB knows a real
hostname+role for a MAC currently only in live.discovered, promote
it back to live.installed so the CLI sees the right state. Also
reordered the live-vs-DB fallback so DB online/offline maps to
live.installed and the discovered branch is the else.
- tests: 3 new vitest cases covering promotion, fresh-discovery, and
unknown-MAC fallback.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two changes prompted by today's etcd raft panic on worker1-k8s0
(tocommit out of range, lost-write on follower) and the cascading
disk pressure that surfaced underneath it.
Audit logs to journald
- kube-apiserver now uses audit-log-path=- so audit events flow to
k3s.service stdout and into journald instead of growing files in
/var/log/kubernetes. The previous setup combined apiserver's
internal rotation with a logrotate *.log glob that double-rotated
the rotated files into permanent orphans (observed: 7+ GB).
- New journald-limits operation writes a SystemMaxUse=2G drop-in so
audit volume cannot fill /var/log even under bursty load.
- log-rotation operation repurposed to decommission the obsolete
logrotate rule and reap leftover audit files. Idempotent: no-op
on fresh installs.
Etcd member recovery
- New recoverEtcdMember(broken, peer, hostname) codifies the
documented k3s recovery: stop k3s, etcdctl member remove, wipe
/var/lib/rancher/k3s/server/{db,tls,cred}, restart, poll for
rejoin. Refuses to operate when cluster size < 3 to preserve
quorum.
Tests
- 7 new unit tests covering both decommission paths and the
recovery procedure (54 total, all green).
- install.test.ts asserts the file-based audit args are gone.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Project tracking for labctl v2.0 platform design. Includes P1 (arch doc update),
P2 (SSH emergency mode, Prometheus metrics), and P3 (graph viz, import, secrets rotation)
items from the CEO and eng review sessions.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add `labctl provision recheck` to refresh hardware info via SSH
- Preserve hardware info in InstalledInfo when install completes
- Fix /ks-auto: run nested %pre scripts from included kickstarts
- Add command-discover WebSocket routing for hw info updates
- Fix k3s join: clean stale TLS/cred when joining existing cluster
- Add --tls-verify=false for internal HTTP registry pushes
- Add fix-ssh-root.sh script for root SSH access on all nodes
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add state-aware kickstart dispatch for machines that boot from ISO
(no PXE/network at UEFI level). Replaces hardcoded discover.ks.
- /ks-auto: %pre detects MAC, queries /api/machine-state/<mac>,
writes discover or install kickstart to /tmp/dynamic.ks,
main body %include's it
- /api/machine-state/<mac>: simple state endpoint returning
unknown|discovered|queued|installing|installed|debug
- ISO kernel cmdline updated: discover.ks → ks-auto
- Handles: discovery (first boot), install (queued), debug modes
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Skip Cilium install for joining servers (already in cluster via daemonset)
- Longhorn annotation for workers: SSH to server node from CLI to apply
kubectl annotation (workers don't have kubectl access)
- Default SSH user for k3s/app commands changed to 'root' (operations
need root privileges, using 'lab' user broke installs)
- k3s server config: cluster-init for initial server, server+token for joins
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Server config now uses cluster-init: true for initial server (enables
embedded etcd). Joining servers get server: + token: in config.
- Cilium install already checks for existing installation, so joining
servers skip it gracefully (the "release name in use" error is non-fatal)
Cluster rebuilt as etcd HA:
worker0-k8s0 control-plane,etcd (initial server, cluster-init)
worker1-k8s0 control-plane,etcd (joined server, Mac Studio aarch64)
spark-2935 worker (DGX Spark, aarch64)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Replace sed with grep -v / awk for fstab manipulation (Asahi Fedora's
sed doesn't support \| delimiter or \? quantifier)
- Use idempotent write_lab_fstab function: removes all old entries first,
comments out conflicting btrfs subvol entries, adds fresh LVM entries
- Fix sed for SSH hardening: use #* instead of \? (POSIX compatible)
- Tested on Mac Studio: no duplicate fstab entries after multiple runs
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Firstboot script defaults admin user to 'lab' instead of bastion's
config.adminUser (which was 'michal' from host system)
- iSCSI OS detection uses case-insensitive match for 'fedora'
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
