ce6911c196d0843c552f5b4d029c01b78fe6e053
129 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Michal
|
ce6911c196 |
fix(migration): require a working WAN, not every WAN
This reverted a cutover that had actually succeeded.
The evidence, from the revert tearing it down:
dhclient: DHCPRELEASE of 87.192.101.48 on bond0.53 to 185.232.119.244
vtysh: "no ip route 0.0.0.0/0 87.192.96.1 bond0.53 tag 210 1"
netlinkd: RTM_NEWLINK -> bond0.53, mac=f0:9f:c2:12:9b:4f
bond0.53 came up with the cloned MAC and was handed 87.192.101.48 -- the exact
public address the USG holds -- with a default route via the real ISP gateway.
kea was serving live LAN clients at the same moment (10.0.0.12, 10.0.0.13,
192.168.8.28). The gateway was working.
The only failure was pppoe0: ppp@pppoe0.service exited 5/NOTINSTALLED. That is
the Vodafone FAILOVER line, and the health check listed "pppoe0 has an address"
as mandatory, so a working gateway was torn down because its backup WAN was
down. The check encoded "every WAN must work" when the requirement is "the box
must reach the internet".
Now: default route, reachability and DNS are mandatory; each WAN interface is
reported individually but fatal on neither. A failover line being down is worth
seeing, not worth reverting for.
This also incidentally settles the last genuine unknown in the migration, which
could not be tested any other way: the ISP does hand the same lease to the
cloned MAC. That was the one thing I had said was unknowable until the USG let
go of it.
Note the earlier polling fix (
|
||
|
Michal
|
54b21fa9ff |
fix(migration): poll for WAN health instead of sampling once at 25s
A real cutover attempt reported failure and reverted a configuration that may well have been fine. The health check waited a fixed 25 seconds and then judged: [switch] committed. Waiting 25s for PPPoE and services to settle... [switch] FAIL pppoe0 has an address 25s is far too short for a WAN. PPPoE alone is PADI/PADO/PADR/PADS followed by LCP, authentication and IPCP -- routinely 15-30s on its own. Both lines had also just been released by the USG seconds earlier, and ISPs commonly hold the previous session and MAC binding for minutes before leasing to the "same" CPE again, which is exactly what a cloned MAC looks like from their side. The one thing the design could not tolerate was being impatient, and it was. Now polls every 15s up to HEALTH_BUDGET (default 180s), reporting progress, and stops early the moment everything is healthy. The budget deliberately finishes long before commit-confirm fires -- 180s against a 10 minute timer leaves 420s of margin -- so the decision to confirm or revert stays ours rather than being made by the timer. Also recorded while chasing this: the earlier claim that VLANs 51/53 are not trunked to the firewalls was WRONG, and the UniFi port settings disprove it -- those LAG ports are Native VLAN Management (1) with Tagged VLAN Management set to Allow All. My evidence never supported the claim: a passive RX count cannot distinguish an absent VLAN from a quiet one, because switches do not flood unicast, and the active DHCP probe used a random MAC that an ISP binding to its registered CPE would ignore regardless. Both observations fit a perfectly healthy trunk. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH |
||
|
Michal
|
ff86a421f4 |
fix(labsim): console-apply must handle both VyOS prompts, not just $
Two failures from one strict expect, both hit while building the sim ISPs. A run that dies mid-config leaves the console parked in configuration mode. The next run then waits for the operational `$ ` prompt against a perfectly healthy VM and hangs until timeout, with nothing in the output to say why -- the box was sitting at `vyos@isp-dhcp#` the whole time. Login now accepts `# ` as well and discards the stale candidate rather than committing something nobody has seen. The same mistake at the exit step: insisting on `$ ` after `save` hung, AND left the console in config mode, which is what created the first failure for the following run. Now accepts either prompt. Known-bad, not fixed: the tool reports "committed and saved" when the set commands have not applied. Verified against the clone -- prompt showed the host-name change had landed while `grep -c dhcp-server` returned 0. The failure detection only inspects c.before for a few strings and evidently misses the real failure mode, so success is being reported without evidence. That needs fixing before this tool is trusted for anything; it is currently only safe to use with an independent check afterwards, which is how the gap was found. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH |
||
|
Michal
|
ee070371a8 |
feat(labsim): add WAN transport VLANs so the sim can host fake ISPs
A cutover attempt failed on the WAN and nothing had tested it. The reason the sim could not have caught it: labsim modelled every LAN VLAN faithfully and omitted the WAN entirely -- vlans.conf had 1, 2, 3, 9, 10 and 200, never 51 or 53. Worse, the switch script's WAN health checks are conditional on the delta configuring PPPoE, so in the sim they printed "this delta configures no WAN -- skipping all WAN health checks" and passed. The sim proved the delta commits; it never proved the WAN works, and could not have. Adds VLANs 51 (Vodafone/PPPoE) and 53 (10gig/DHCP) to the fabric so a fake ISP can live on each and those checks actually execute. VyOS has service pppoe-server (accel-ppp) natively -- authentication local-users, client-ip-pool, gateway-address -- so a VyOS VM can play the concentrator, and dhcp-server can play the other ISP. vlans.conf gains host_octet 0, meaning "no host leg". A host address on a WAN transport VLAN would misrepresent the segment: the point is that VyOS reaches an ISP, not the host. Also fixes a real gap in ovs_bond_router: it returned early when the bond already existed, so adding a VLAN to vlans.conf never reached an existing bond. Re-runs now reconcile the trunk and say so. That gap is the same SHAPE as the production failure -- interface present, VLAN missing from the trunk, frames silently dropped -- which is precisely the class of bug the sim needs to be able to reproduce rather than embody. Both bonds updated: [2,3,9,10,200] -> [2,3,9,10,51,53,200]. Note on the production diagnosis, which is NOT settled: a passive RX test showed zero frames on 51/53 at the firewall, and an active DHCP DISCOVER (verified to have transmitted, tx +2) drew no reply. That is consistent with the VLANs not being trunked, but equally with the ISP only answering its registered CPE MAC -- which is exactly why the delta clones f0:9f:c2:12:9b:4f, and why it cannot be settled from production while the USG holds that MAC. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH |
||
|
Michal
|
41b5448f56 |
feat(pulumi-vyos): prototype VyOS subtrees as Pulumi resources with commit-confirm
Goal: change VyOS and Kubernetes in one codebase and one plan -- so a BGP change
touches both sides in a single `pulumi preview`.
First, the worry about per-command pushes turned out to be unfounded for the
community providers. Read foltik/vyos and its client library: a
`vyos_config_block_tree` flattens the whole subtree into a single payload array
and sends ONE POST to /configure, so one resource is one commit. Good.
What they do not do is send `confirm_time`. Their payload is only
op/path/value, so every change is an unprotected commit -- on a router you reach
through the router, that is the difference between a mistake and an outage. The
VyOS API itself supports commit-confirm; the providers simply do not use it.
So this is a ~180-line Pulumi dynamic provider that does. Verified end to end on
labsim: create and update each land in ~6s as one commit-confirmed transaction,
update reports [diff: ~commands], destroy removes the subtree, and an
unconfirmed commit was observed reverting the router on its own.
Three API details found the hard way, all now encoded and commented:
- confirm_time is ONLY read when the body parses as ConfigureListModel, i.e.
{"commands": [...], "confirm_time": N}. A bare array is accepted and
committed with NO timer armed, and the response looks like success. This
silently discards the entire safety net, so the resource now checks the
response actually says "commit-confirm" and refuses to proceed otherwise.
- There is no /confirm endpoint; confirm is an op on /configure.
- Confirm requires a `path` field even though it ignores it -- the Union
resolves to ConfigureModel, which mandates path. Without it: "missing 'path'
field", and the timer keeps running.
Apply is `delete <path>` followed by the sets, in one request, so the result is
the declared state rather than a merge -- otherwise `pulumi up` accumulates
instead of converging.
Known gaps, in the README rather than hidden: no read/refresh so out-of-band
drift is not detected, and the API runs with a self-signed certificate and
verification disabled. Both need addressing before production. The cutover
itself should still use vyos-unifi-switch, which the API cannot replace.
Sim left as found: test resource destroyed, dns forwarding restored to 15 lines.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
|
||
|
Michal
|
63061e6e7e |
feat(migration): peer link cabled and verified; conntrack-sync enabled in deltas
eth3 <-> eth3 direct cable is in. Both ends negotiated 2500Mb full duplex -- these are 2.5 GbE ports, not the 1G I had assumed. Carrier alone proves nothing, so the link was tested end to end with temporary kernel-level addresses (never committed to VyOS config, removed afterwards): 3/3 packets, 0% loss, 0.371ms average. A cable can show carrier and still not pass traffic; now it is known to. Deltas regenerated with --conntrack-link and installed on both boxes: vyos001 nat=21 fw=58 conntrack=9 eth3=10.255.255.1/30 disable=0 vyos002 nat=21 fw=58 conntrack=9 eth3=10.255.255.2/30 disable=2 Identical apart from the peer /30, the VRRP/DHCP-HA roles, and the two disable lines holding vyos002's WAN down. Both still report mode=unifi and nothing about their behaviour has changed -- eth3 carries no address in the running config, and conntrack-sync appears only in the delta, which is applied at cutover. Not verified: multicast on the peer link. `ping -I eth3 224.0.0.1` drew no responders, but that is the all-hosts group which VyOS need not answer, so it proves nothing either way. conntrack-sync's own multicast (225.0.0.50) was proven working in labsim over bond0.10, and this is a point-to-point link, so the risk is low -- but it is untested on this specific cable and worth watching at cutover. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH |
||
|
Michal
|
ccdd1e7e49 |
fix(migration): both boxes carry WAN and NAT; the backup just holds it down
"No NAT? How are we supposed to get internet?" -- a fair question that exposed a
worse design than I had admitted. Internet did work, but only via vyos001: NAT
and the entire WAN were gated behind --with-wan, so vyos002 would have held the
LAN VIPs and routed between VLANs with no path to the outside at all. Failover
would have preserved addressing and lost the internet.
The fix rests on a checked fact rather than an assumption: VyOS WARNS but still
commits when a NAT rule names an interface that does not exist
("Interface bond0.53 for source NAT rule 900 does not exist!"). Verified on a
real VyOS before relying on it.
So both boxes now get the identical WAN, NAT, port-forward and firewall config,
and the backup's two WAN interfaces are simply set `disable`. The cloned WAN MAC
is therefore never live on two boxes at once, while everything needed to route
and masquerade is already in place. The two deltas are now byte-identical apart
from VRRP priority, own/peer addresses, DHCP HA role, the conntrack /30 -- and
the two disable lines.
Taking over the internet path becomes deleting two lines rather than
reconstructing NAT under pressure:
delete interfaces bonding bond0 vif 53 disable
delete interfaces pppoe pppoe0 disable
Both boxes now: 21 NAT rules, 58 firewall rules, full PPPoE. Backup delta
validated against a real VyOS config with the disable lines present -- commits
clean. Runbook updated with the takeover procedure and the warning that it must
only be done when vyos001 is genuinely down.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
|
||
|
Michal
|
64e748ea94 |
test(labsim): conntrack-sync verified, and it exposed a delta defect
conntrack-sync proven working on the sim pair -- bidirectional replication with
zero errors:
MASTER internal 34 external(from peer) 52 62 pkts sent / 109 recv 0 err
BACKUP internal 76 external(from peer) 36 142 pkts sent / 73 recv 0 err
Getting there required learning something that changes the production config:
**VyOS only engages conntrack when a firewall or NAT is configured.** With
neither present, both routers reported zero conntrack entries and conntrack-sync
had nothing to replicate. Adding a single state-matching forward rule turned
tracking on and replication began immediately.
That is a defect in the delta, not just a test artifact. NAT and the firewall
were both gated behind --with-wan, so the BACKUP would have had neither -- it
would not have tracked connections at all, and replicated entries are useless to
a box whose conntrack is not engaged. Exactly the failure that only shows up
during a failover, when it is too late to notice.
Fixed: a stateful forward rule (accept established/related, default-action
accept) is now emitted on BOTH boxes, outside the WAN gate. Only NAT and the
WAN-scoped rules remain master-only. Verified: vyos002 now carries stateful
tracking and conntrack-sync but zero NAT lines. Master delta re-validated
against a real VyOS config -- no errors.
Also incidentally confirmed no-preempt: router1 rebooted and came back as
BACKUP rather than seizing the VIP, which is the opposite of what the production
pair did this afternoon (still on default preempt until cutover).
Two traps recorded while doing this:
- The detached `setsid nohup` config-apply pattern can strand a VyOS config
session. An orphaned session (dirs under /opt/vyatta/config/tmp/, PID long
dead) blocked every subsequent `set` on that box with a bare "Set failed",
and the dirs are overlay mounts so they cannot simply be deleted. Rebooting
cleared it. This pattern is used to survive losing SSH mid-change, so it is
worth knowing it has a failure mode of its own.
- Only VLAN 10 passes traffic between the two sim routers; every other VLAN
fails ARP despite identical vlan_mode/tag/trunks on both OVS bonds and
distinct MACs. VRRP forms on all six groups regardless. The sync link had to
be bond0.10 as a result. OVS-specific, absent in production, but it means
the sim proves mechanism rather than topology.
Production deltas regenerated with --conntrack-link: eth3 at 10.255.255.1/30 and
.2/30 awaiting the cable, which is not yet plugged (carrier=0 on both).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
|
||
|
Michal
|
bb654d83f8 |
test(labsim): second VyOS router proves DHCP active-passive HA
Answers the question a single router could not, and that would otherwise only
have been discovered at cutover: with kea high-availability active-passive, does
exactly ONE box answer a DHCP request?
Yes. Probing sim VLAN 10 with broadcast-dhcp-discover returns offers from a
single distinct Server Identifier -- 172.31.10.252, the primary. The secondary
runs kea but stays silent. Without this the delta would have put 6 subnets and
84 static-mappings on both boxes with nothing to arbitrate them, and two kea
instances would have raced on every broadcast domain.
Worth noting the raw response count is misleading: nmap reports "Response 1 of
2" because it sends several discovers, and both replies carry the same server
identifier. Counting responses says "2 servers"; counting distinct server
identifiers says "1". The second number is the true one.
labsim now runs a real pair, mirroring production:
router1 172.31.<v>.252 priority 200 DHCP HA primary
router2 172.31.<v>.253 priority 100 DHCP HA secondary
VIP 172.31.<v>.1 floating, held by the master
That required converting router1, which held .1 directly, to .252 plus a
floating VIP -- otherwise it is two routers, not a pair. All six VRRP groups
show MASTER on router1 and BACKUP on router2.
New tooling:
- sim-ha-config.py generates each role's config, reusing unifi-to-vyos.py
--mode sim for the DHCP half so what is proven here and what production
gets share a code path. VLAN 10 correctly carries /23.
- console-apply.py applies config over the serial console, which is necessary
because a freshly installed VyOS holds the same addresses as its peer and
cannot safely be reached over the network at all until reconfigured.
Known sim-only quirk, deliberately not chased: router1 cannot ARP router2 on
the untagged VLAN 1 while every tagged VLAN works, and VRRP forms correctly on
all six groups regardless. Both OVS bonds carry identical vlan_mode/tag/trunks
and the bond MACs differ, so this is OVS bond behaviour on the native VLAN with
two bonds on one bridge -- not a VyOS config problem, and not present in
production, which uses a real switch.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
|
||
|
Michal
|
952f5c66e3 |
feat(migration): complete the VyOS HA stack per the official docs
Prompted by "I thought we tested HA on libvirt" -- checking rather than
recalling showed the sim has ONE VyOS router with zero high-availability
config. VRRP was configured and running on the real pair, but it is only one of
four parts of what VyOS considers an HA pair.
Against docs.vyos.io (highavailability, conntrack-sync, dhcp-server, and the HA
walkthrough), three gaps are now closed in the delta:
- VRRP was multicast-only with default preemption. Added unicast
hello-source-address/peer-address per group, as the walkthrough does, plus
no-preempt. Without no-preempt a recovered box reclaims the VIP before
conntrack state has synced and drops every established connection; the docs
are explicit that preempt-delay must otherwise be >= purge-timeout.
The per-VLAN node addresses are a table, not derived: VLAN 3 is .4/.5 while
every other VLAN is .252/.253.
- DHCP high-availability, which fixes a real defect rather than adding a
feature. Both boxes carried the full 6 subnets and 84 static-mappings, so
after cutover two kea instances would have raced on the same broadcast
domains. Now active-passive with primary/secondary and swapped
source/remote, syncing over TCP 647 on the LoT addresses. Each subnet
already carries the unique subnet-id kea HA requires, and the peer name
deliberately differs from both host-names.
- conntrack-sync over a dedicated eth3 <-> eth3 link, gated behind
--conntrack-link because it needs a cable that is not plugged in yet. This
is what the peer cable is actually for -- VRRP does not want one, since its
hellos must travel on the segment they protect.
VRRP failover exercised on the production pair, which is free to break today
because nothing uses the .254 VIPs: keepalived stopped on vyos001, all six VIPs
moved to vyos002 within 12s, and returned on restart (preemption still default
on the live boxes). Both boxes clean afterwards, no config drift.
Master delta validated against vyos001's real running config on the sim router
before installing. Installed on both: 6 no-preempt, 6 unicast pairs, DHCP HA
primary/secondary respectively.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
|
||
|
Michal
|
f81c94af43 |
feat(migration): dual WAN, cloned MAC, and the new 10.8.0.0/23 Private VLAN
Two corrections from reading the live USG instead of trusting UniFi's fields,
which report wan_type=dhcp for both WANs and are simply wrong:
- There are TWO WANs, not one. WAN2 is the 10 gig ISP on VLAN 53, plain DHCP
with a PUBLIC address (87.192.101.48/21, gw 87.192.96.1) on the USG's eth2 --
and it is what actually carries traffic. WAN1 is Vodafone PPPoE on VLAN 51,
the failover. The delta had PPPoE as the only WAN, which would have left the
primary line unconfigured.
- The DHCP lease is bound to MAC, so bond0.53 now clones the USG's WAN2 MAC
(f0:9f:c2:12:9b:4f). That is how VyOS keeps the existing public lease rather
than negotiating a new one -- or getting none, if the ISP allows one per
line. Distances: 10 gig at 1, Vodafone at 10.
Only ONE box may hold the cloned MAC, so --with-wan gates the entire WAN, NAT
and firewall section. vyos001 gets it (320 set lines); vyos002 gets none (234,
zero WAN/NAT/firewall) and routes the LAN only. Pretending both could hold it
would have meant a duplicate MAC on VLAN 53 and a flapping switch table.
Private was rebuilt at 10.8.0.0/23 (VLAN 9) after the old 10.0.8.0/23 was
deleted. bond0.9 and the VRRP group were moved to 10.8.0.252/.253 with VIP
10.8.0.254 on both boxes, and the delta now targets 10.8.0.1.
Creating that network first required breaking a deadlock in UniFi: every LAN
write was rejected with api.err.WanIpOverlapped / 0.0.0.0/0, because WAN1 was
set to DHCP on a line that only speaks PPPoE, so it sat at 0.0.0.0 forever and
the validator treated that as a subnet overlapping everything. Verified
server-side, not a UI bug -- the API rejected it identically. Setting
wan_type=pppoe let it dial (90.241.226.213, MTU 1492), which cleared the phantom
overlap and incidentally PROVED the Vodafone credentials and line work, which
had been listed as untestable before cutover.
dhcp-options no-default-route-dns does not exist; the valid set is client-id,
default-route-distance, host-name, mtu, no-default-route, reject, user-class,
vendor-class-id. Caught by validating the delta against vyos001's real config on
the labsim router before installing.
After adding the network, the gateway's dhcpd.conf was checked with
`dhcpd3 -t -cf` (valid) and confirmed to contain the new subnet only after a
force-provision -- controller state is not device state.
Both boxes: mode unifi, VRRP unchanged, unifi.boot re-captured (232 lines,
carrying the new VLAN 9), no config drift.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
|
||
|
Michal
|
febe4b72bc |
chore(migration): all DNS through VyOS to Google, NAS out of the path
The NAS is legacy for ad.itaz.eu and those records now live in Cloudflare, so
the zone resolves publicly -- verified: nas001.ad.itaz.eu and
kvm-macstudio1.ad.itaz.eu both answer from 8.8.8.8. That removes the reason for
a conditional forward and lets the NAS leave the DNS path entirely.
Two changes:
- `service dns forwarding name-server` is now 8.8.8.8 and 8.8.4.4, the same
pair the USG used on its WAN, instead of 10.0.0.194.
- Every VLAN is handed the gateway as its resolver. UniFi set an explicit
resolver on LoT only (the NAS); carrying that over would have kept the NAS
in the path for one VLAN and not the other five, which is the sort of
asymmetry nobody remembers a year later.
The NAS is still referenced 9 times, all legitimate and checked: 4 NAT
destination rules, the 4 matching firewall accepts for those port forwards, and
its own DHCP reservation. No DNS references remain.
Validated by loading vyos001's real running config on the labsim router and
applying the full delta -- all 318 commands accepted, no errors. Installed on
both boxes and verified in place: priority 200/100, upstream 8.8.8.8 + 8.8.4.4,
six client resolvers, 377 lines each.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
|
||
|
Michal
|
3768657b91 |
chore(migration): firewalls resolve via 8.8.8.8/8.8.4.4
Matches the DNS the USG used on its WAN (wan_dns1/wan_dns2), replacing the
10.0.0.194 I had set earlier. Applied to both boxes and saved; VRRP unchanged
(MASTER/BACKUP), NTP still synced, no config drift.
/config/modes/unifi.boot was RE-CAPTURED on both afterwards. It had been taken
before this change, so the escape hatch would have quietly reverted the
resolver on any rollback -- a snapshot is only an escape hatch for the state it
was taken from.
Two things recorded in the runbook:
- The boxes' name resolution now depends on the internet, so between
unplugging the USG and PPPoE establishing they have no DNS. Harmless:
nothing in the switch resolves a name, and the health checks use DNS
precisely to prove the WAN came up.
- Internal ad.itaz.eu names still resolve via Google, because that zone is
published publicly with private addresses in it (nas001 -> 10.0.0.194,
kvm-macstudio1 -> 192.168.3.8). So no conditional forward was needed --
though it is worth knowing the internal topology is public.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
|
||
|
Michal
|
2a8fcb3bd3 |
fix(migration): the runbook pointed at addresses that die with the USG
The access table led with 192.168.8.143/.144 and offered the LoT addresses as a
fallback ("if unreachable, try"). That is backwards and would have stranded the
operator at the worst moment: the workstation sits on LoT, and reaching
192.168.8.x routes *through the USG*, so those addresses are guaranteed dead the
instant it is unplugged. Measured:
ip route get 192.168.8.143 -> via 10.0.0.1 (the USG)
ip route get 10.0.1.252 -> dev lanbr0 (same L2, no gateway)
10.0.1.252 and .253 are on the LoT VLAN, same broadcast domain as the
workstation, and both answer SSH. They are now the only addresses the runbook
gives, with the k8s ones struck through.
Also recorded: the switch cannot be run before unplugging the USG (two devices
on every gateway address; the guard refuses), so the order is forced. And
during the gap between unplugging and completing the switch there is no
inter-VLAN routing at all -- which means the JetKVMs (Management and kvm) and
Tailscale are NOT fallbacks in that window. LoT SSH is the only remote path;
below it is physical console. Added a step 0: open both SSH sessions and leave
them open before touching anything.
Both boxes are now installed and pass the pre-flight gate: mode unifi,
unifi.boot 231 lines including the reload action, delta at the right priority
(200/100), wan-secrets 0600, script executable, no config drift, VRRP still
MASTER/BACKUP. `vyos-unifi-switch vyos` refuses on both -- all six gateway
addresses detected answering ARP -- and neither box has gained dhcp-server, dns
or nat, so nothing about their behaviour has changed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
|
||
|
Michal
|
fc31013ceb |
fix(migration): apply the reviewed reservation plan, not a recomputed one
This caused a real outage. unifi-reserve-all.py recomputed its plan at --apply time by re-reading stat/sta, so a client that renewed between the dry run and the apply was pinned to whatever transient address it happened to hold at that instant. worker1-k8s0 was reviewed at 192.168.8.13 and written as 192.168.8.242. On its next reboot it could not get an address at all, taking a k8s node down. A plan that gets reviewed and a plan that gets applied must be the same object. The dry run now WRITES the plan to a file and --apply READS it and applies exactly that, reporting any client whose current address has since drifted rather than silently preferring the new value. 1 of 51 diverged; the rest were verified against the reviewed list and were correct. worker1 has been restored to .13 and confirmed: DHCPOFFER for its own MAC returns 192.168.8.13, and the node is up with a full lease and working internet. The second half of the outage was drift between controller and device: the USG was still running config from ~16h before these changes, so the controller looked perfectly correct while the gateway handed out something else. Writing the controller is only half the job, so the script now says so explicitly and gives the force-provision and DHCP-probe commands to verify with. `nmap --script broadcast-dhcp-discover --script-args broadcast-dhcp-discover.mac=...` is the way to prove a specific reservation is live without disturbing the client -- it elicits an OFFER without ever sending a REQUEST. _unifi.py gains post() for device commands. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH |
||
|
Michal
|
b37cd79432 |
fix(migration): refuse an unprobeable delta instead of warning past it
The ARP guard against two devices holding the same gateway address is the one check that prevents this script's worst outcome. It reads the addresses to probe out of the delta -- so a delta with no VIP lines made the guard inert, and it previously warned and carried on. That was a testing convenience (the lab delta has no VIPs) weakening a production safety check, which is backwards. It now refuses by default. ALLOW_NO_VIP_DELTA=1 is the explicit lab override. The guard's probing path had never actually executed before this: every sim run took the no-VIPs branch. Verified against the live USG from vyos001: arping is present on VyOS, the regex extracts all six gateway addresses from the real delta (192.168.1.1, 192.168.8.1, 192.168.3.1, 10.0.9.0, 10.0.0.1, 192.168.2.1), and every one of them answers ARP right now -- so on the real boxes, with the USG connected, the guard fires. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH |
||
|
Michal
|
7e464a2828 |
docs(migration): record what the rehearsal proved and what it could not
Ran vyos001's real running config plus the real production delta on the labsim router -- same VyOS version, isolated OVS bridge with no physical NIC, so the sim briefly holding vyos001's actual addresses could not reach the real LAN. Result: all 317 commands accepted and the whole delta commits (COMMIT OK), the revert is byte-exact, and auto-revert fires without rebooting (uptime and boot-id unchanged across it). Also written down are the two things this did NOT establish, because a runbook that overstates its own coverage is worse than one that admits the gap: PPPoE cannot be tried while the USG holds the single available session, and the rehearsal ran with vyos001's eth2/eth3 stanzas stripped because the sim VM has two NICs rather than four. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH |
||
|
Michal
|
01a923352f |
fix(migration): do not emit a NAT translation port for a port list
`set nat destination rule N translation port '16881,6881'` is rejected -- "16881,6881 is not a valid service name" -- because mapping a list of ports onto a list is ambiguous. `destination port` accepts the same list happily, which is why this only shows up on the translation side. All four UniFi port forwards map a port to itself, so translation port was redundant anyway: omitting it makes VyOS preserve the original port, which is exactly the intent. It is now emitted only when the forwarded port genuinely differs, and generation fails loudly rather than producing a config that will not commit if a differing port LIST ever appears. Found by loading vyos001's real running config onto the labsim router and applying the full delta to the candidate config without committing. Worth noting the delta had already passed a read-through: this one only surfaced by running it against a real VyOS of the same version. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH |
||
|
Michal
|
7f5d3517a3 |
fix(migration): create the WAN vif before PPPoE references it
`set interfaces pppoe pppoe0 source-interface bond0.51` refers to an interface that must already exist, and neither firewall has vif 51 -- only 2, 3, 9, 10 and 200 are configured. The commit would have failed, and since the whole delta commits as one unit, that failure would have taken the entire switch with it at the worst possible moment. No address on the vif: PPPoE rides the VLAN and needs no L3 of its own. Found by checking the running config against the generated delta rather than by running it. The prod delta has still never been applied to any VyOS, which is the remaining gap. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH |
||
|
Michal
|
d56bbf6db0 |
feat(migration): reversible USG->VyOS switch, proven on the sim
The cutover is a switch, not a migration: unplug the USG, run one command, and
if anything is wrong run the other one and plug it back in. The operator will
have no internet during this and therefore no assistant, so the machinery has
to live on the boxes and the failure paths have to be proven in advance.
vyos-mode-delta.py generates the delta that turns the passive pair into the
gateway. Only one artifact is authored: gateway mode is always derived from
`load unifi.boot` + delta, so there is no inverse to maintain and no drift
between two hand-kept configs. It reuses unifi-to-vyos.py rather than
duplicating it, so what labsim proved and what production gets are one code
path. The PPPoE password is never written into the delta -- it carries a
placeholder the switch substitutes at apply time from /config/wan-secrets --
and generation fails if the real password appears in the output.
Two things the delta covers that the plan had underweighted:
- VyOS defaults to ACCEPT while the USG has an implicit WAN drop. Migrating
the port forwards alone would have left the router's own services and the
whole LAN reachable from the WAN. Added a stateful baseline scoped to the
WAN interface rather than a global default-action drop, so a mistake there
cannot lock anyone out over the LAN -- the only way back during a cutover.
- The old VIPs are NOT at network+254 on the /23 networks; they are
192.168.9.254, 10.0.9.254 and 10.0.1.254, in the upper half. A delete
naming a computed address fails quietly and leaves the group holding two
VIPs. The delta deletes the whole address node instead of guessing.
vyos-unifi-switch runs on the box from /config, which survives image upgrades,
so it works from a local terminal or the JetKVM with no workstation.
Proven on labsim, not assumed:
- unifi mode restores the previous config BYTE-EXACT (138 lines, diff clean).
- Auto-revert fires when the commit is not confirmed: 85 static-mappings ->
0, kea stopped, hostname restored, and uptime plus boot-id UNCHANGED, so
it reloaded rather than rebooted. That distinction is the whole reason
`commit-confirm action reload` is a prerequisite.
- Health-check failure triggers an immediate revert_soft rather than waiting
out the timer.
Four bugs found while doing it, each of which produced a wrong answer rather
than an error:
- commit-confirm is TWO steps. `config-mgmt commit_confirm` only arms the
revert timer; a normal `commit` still has to follow. Arming alone committed
nothing while reporting success.
- `sudo sg vyattacfg "config-mgmt ..."` loses the config-session environment,
so it reported "No configuration changes to commit" against a candidate
that plainly had 446 added lines.
- `... | grep -q` under `set -o pipefail` reports FAILURE on a match: grep
exits early, the producer takes SIGPIPE. Whether it triggers depends on
output size, so `status` misreported the mode intermittently.
- `show configuration commands` quotes values, so a fixed-string match for
`action reload` never matched `action 'reload'`.
CUTOVER.md is the printable runbook: both reachable addresses per box, the
escape hatch first, and the note that PPPoE is the one thing that could not be
tested beforehand.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
|
||
|
Michal
|
6c4318d3ae |
feat(migration): reserve every active client at its current address
kea does not inherit UniFi's lease database. At cutover it starts with an empty
view of who holds what, so it can hand an address that is currently in use to a
different device. Reservations are what carry "this device has this address"
across the switch, because they live in config rather than in lease state.
unifi-reserve-all.py creates one per active client, dry run by default. 51
written, 51/51 verified live by reading the records back; the controller now
holds 85 reservations and the generator emits all 85 with unique, valid
hostnames and no duplicate addresses. Active clients with no reservation went
from 48 to 4.
Three guards, each of which caught something real in the dry run:
- VRRP virtual addresses are excluded. UniFi reports them as ordinary client
addresses because the firewalls' bond MACs answer for them, and their
apparent IP flips between the real interface address and the VIP. Without
this, 192.168.1.254 -- the gateway VIP itself -- would have been given a
DHCP reservation.
- The firewalls' own interface MACs are excluded; those are statically
configured routers, not DHCP clients.
- Any address claimed by more than one MAC is dropped rather than guessed
at. This is how the VIPs surfaced in the first place.
Also skipped: addresses already reserved to another MAC, network gateways, and
anything on a network that does not serve DHCP (which excludes the WAN transit
VLANs automatically).
labsim-dhcp-test.sh gained a lease-database flush, and it is not tidiness. Two
findings, both of which first appeared as a PASSING test:
- Re-running against stale leases, kea gave dynamic addresses to three
devices that have reservations. The reservations were present and correct
in kea's own config throughout. Kea saw the reserved address as leased to
"another client" -- same MAC, different client-id from the earlier boot --
and allocated elsewhere. Cutover starts with an empty lease database so
this is a testing artifact, but a reservation is evidently not
unconditional once leases exist.
- Removing only dhcp4-leases.csv does nothing: kea's memfile backend keeps
lease-file-cleanup rotations (.csv.2) and restores from them on start.
The verdict logic no longer takes the first matching lease row. Doing so
reported an hours-old lease as the current answer and scored three failures as
passes, including one where the device had plainly been given a dynamic
address. A MAC with more than one lease is now an explicit failure rather than
a guess.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
|
||
|
Michal
|
f36ff4c6e3 |
feat(migration): pin firewall management NICs and reserve them in UniFi
Prerequisite for the cutover. eth2 on both firewalls was DHCP-served by the USG,
and both addresses (192.168.8.143/.144) sit inside the pool VyOS will serve --
with no reservation for either MAC. After the switch they would renew from kea,
get no mapping, and the management addresses could move. That is the worst
possible moment for the address you SSH to to change, because losing the USG
also means losing internet and any outside help.
On both boxes, driven over the LoT path (bond0.10) so the interface being
changed was never the one carrying the session:
- eth2 pinned static at its current address, so it no longer depends on DHCP
- `system name-server eth2` replaced with 10.0.0.194. That setting inherited
resolvers from the DHCP lease, i.e. the boxes were resolving via the USG and
would have lost DNS with it. 10.0.0.194 is reachable directly over bond0.10
and is authoritative for ad.itaz.eu, so internal names now resolve on the
firewalls -- they did not before.
- static default route via 192.168.8.1, replacing the one the lease provided.
Superseded by PPPoE in vyos mode; this keeps unifi mode as it was.
Verified after each: SSH on the pinned address, external and internal DNS, NTP
still synced, VRRP unchanged (vyos001 MASTER, vyos002 BACKUP).
migration/unifi-reserve.py adds the matching UniFi reservations so the
controller cannot lease those addresses to anything else, keeping the
management address identical in both modes. It reads the record back after
writing, because a controller accepting a PUT is not proof it stored what was
asked for, and it is idempotent.
Also noted while doing this: VyOS `commit-confirm` REBOOTS the box if not
confirmed -- "Minutes until reboot, unless 'confirm'" -- it does not roll the
config back in place. For a gateway that means a real outage window, which
changes how the switch script must use it. `config-mgmt commit_confirm -y`
executes without the interactive prompt.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
|
||
|
Michal
|
44dbd5188c |
feat(migration): export UniFi config and generate VyOS DHCP+DNS from it
Groundwork for replacing the USG with the VyOS pair without anything on the
network noticing. Three pieces:
migration/unifi-export.py pulls 13 endpoints off the classic controller into
timestamped JSON plus a normalised inventory: 11 networks, 31 DHCP
reservations, 4 port forwards, 2 firewall rules, 0 static routes. Two things
this turned up that a naive export would have lost:
- 23 of the 31 reservations carry no network_id at all -- UniFi simply does
not store the binding -- so they are resolved by subnet containment
instead. Without that, three quarters of the reservations have no subnet
to be placed in.
- 30 of the 31 sit INSIDE the DHCP pool, which UniFi's dhcpd tolerates and
which is flagged as a warning rather than discovered at cutover.
migration/unifi-to-vyos.py turns that inventory into VyOS `set` commands for
DHCP and DNS only -- the services the USG owns that VyOS must reproduce. Not a
general converter. --prod and --sim come from one code path so the config
proven in the sim and the config applied to the firewalls cannot drift. Prod
mode hard-fails if any reservation is missing, since a silent drop is the
failure mode that matters.
DNS is included because the USG resolves for 5 of 6 VLANs today: UniFi hands
out the gateway's own address whenever dhcpd_dns is empty, verified by
labmaster resolving against 192.168.8.1. Replacing the USG without a forwarder
would take DNS away from those VLANs entirely.
labsim/labsim-dhcp-test.sh proves it by booting throwaway VMs with real
production MACs -- the one piece of production config that transplants
verbatim. Safe because ovs-labsim has no physical NIC, so those MACs cannot
reach the real LAN.
Result on VyOS 2026.08 (kea), 4/4: printer1 got 172.31.10.46 from inside the
pool, sonoff-matter got 172.31.11.67 across the /23 boundary, Hubitat got its
out-of-pool .2, and an unreserved MAC got an unreserved address. kea honours
in-pool host reservations -- the open question blocking the cutover.
Supporting changes to labsim:
- VLAN 10 widened to /23. Every reservation is in LoT and LoT spans 10.0.0.x
and 10.0.1.x, which a /24 cannot represent.
- LoT's host leg moved to .3, because 10.0.0.2 is a real reservation
(Hubitat) that maps onto the host's own address.
- vlans.conf gained optional masklen and host_octet fields, defaulting to
24 and 2 so the other five VLANs are untouched.
- Fixed /etc/network/interfaces hardcoding 255.255.255.0. That file is what
actually takes effect on these Alpine guests -- cloud-init's
network-config is ignored -- so any non-/24 VLAN was silently wrong.
Two generator bugs found by VyOS rejecting the output: static-mapping names
are validated as hostnames, so underscores fail; and two devices named
"espressif" plus two named "thebeast" collided into single names, which would
have overwritten one reservation with another's address.
The raw export holds WiFi passphrases and the WAN PPPoE credentials and is
gitignored.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
|
||
|
Michal
|
b0b68f2edd |
Merge feat/vyos-unattended-install: VyOS HA install + DiskPressure incident fixes
Some checks failed
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017f6jyeeDqP4ufyeL3UER9w |
||
|
Michal
|
72c54edce2 |
feat(k3s): enable swap and grow the rancher LV during host-prep
Some checks failed
CI/CD / lint (pull_request) Failing after 10s
CI/CD / test (pull_request) Failing after 10s
CI/CD / typecheck (pull_request) Failing after 22s
CI/CD / build (pull_request) Has been skipped
CI/CD / publish-rpm (pull_request) Has been skipped
CI/CD / publish-deb (pull_request) Has been skipped
Replace the CIS-style disableSwap op with enableSwap: activate the labvg-swap LV with an fstab entry (kubelet runs failSwapOn=false; zram stays the fast tier, the LV is overflow before OOM kill). Add growRancherLv: extend labvg/rancher to 120G when the VG has free space, covering nodes installed before the kickstart sizing change and vanilla nodes converted to k8s; skips with a clear message when the VG is full. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017f6jyeeDqP4ufyeL3UER9w |
||
|
Michal
|
33be713d0c |
feat(bastion): size the rancher LV at 120G for k8s roles in kickstart
The 20G /var/lib/rancher LV (k3s imageFs) idled at 85% used from steady-state images alone; one ~5G image pull tripped imagefs eviction and evicted unrelated pods (2026-08-14 DiskPressure incident). Create the LV for both worker and infra roles at 120G — it must be sized here because longhorn's --grow consumes all remaining VG space, making post-install lvextend impossible. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017f6jyeeDqP4ufyeL3UER9w |
||
|
Michal
|
a5b36678ed |
feat(labsim): live topology view with per-path latency
Some checks failed
CI/CD / lint (pull_request) Failing after 9s
CI/CD / test (pull_request) Failing after 9s
CI/CD / typecheck (pull_request) Failing after 24s
CI/CD / build (pull_request) Has been skipped
CI/CD / publish-rpm (pull_request) Has been skipped
CI/CD / publish-deb (pull_request) Has been skipped
The Grafana heatmap of 1s and 0s said almost nothing, and the state timeline
was an unreadable pile of overlapping series labels. Replaced as the primary
view with a purpose-built page served by the exporter itself.
- Probe now captures ICMP RTT, exposed as labsim_rtt_ms{src,dst}. A path that
is up but slow is a different problem from one that is down, and a pass/fail
grid cannot show it.
- Exporter serves / (topology), /api/matrix (JSON) and /metrics.
- topology.html: node per VLAN in a ring, VyOS router in the centre because
every inter-VLAN packet really does traverse it, one line per pair coloured
green/red with the RTT on it. Hovering gives per-direction state. A node ring
goes red if anything to or from it is blocked. Side panels list blocked paths
and the slowest links. Refreshes every 5s, no dependencies.
Grafana stays for what it is actually good at — history of when a path flipped.
Label placement is deliberate: RTT captions sit ~32% along each edge with a
perpendicular nudge, because every diagonal of a 6-node mesh crosses the centre
and midpoint labels stack on the router node.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
|
||
|
Michal
|
c91e44f796 |
feat(labsim): libvirt replica of the lab network with LACP + VyOS routing
Some checks failed
CI/CD / lint (pull_request) Failing after 11s
CI/CD / typecheck (pull_request) Failing after 10s
CI/CD / test (pull_request) Failing after 9s
CI/CD / build (pull_request) Has been skipped
CI/CD / publish-rpm (pull_request) Has been skipped
CI/CD / publish-deb (pull_request) Has been skipped
A throwaway copy of the production VLAN topology so routing and firewall changes can be tested before they touch the real network. Same VLAN IDs and roles as UniFi, deliberately different ranges (172.31.<vlan>.0/24) so nothing here can be mistaken for production. - OVS fabric: real 802.1Q. Access port per micro VM, host leg per VLAN (.2, for SSH only — NOT the VMs' default route, so inter-VLAN tests exercise the router rather than the host's routing table), and a trunk portgroup with VLAN 1 declared nativeMode='untagged'. - Six Alpine micro VMs (256MB, copy-on-write overlays on one 176MB image), SSH + a hello-world HTTP page naming the VLAN. - VyOS router installed to disk unattended over the console, with the SAME config shape as the VP2440s: two NICs in an LACP bond carrying the trunk, VLAN 1 native, bond0.<vlan> holding the .1 gateway on each. - labsim-matrix.py: full-mesh ICMP/TCP22/TCP80 probe, ~0.2s, --watch highlights cells that changed since the last sweep. Guest-side probe is python3 (already present via cloud-init) so nothing is installed on VMs that have no internet. - Prometheus + Grafana (anonymous auth, no login) with a provisioned dashboard: heatmap plus a state timeline showing exactly when a path flipped. Verified end to end: one VyOS rule took sum(labsim_reachable) from 90 to 84, blocking precisely kvm<->k8s across all three protocols. Traps found building this, all now encoded in the scripts: - virtio-net breaks 802.3ad: the guest's bonding driver reports slaves "MII Status: down" despite carrier=1 and never sends an LACPDU, so the bond sits in AD_STATE_DEFAULTED. e1000e fixes it with no other change. Matches the netdev thread "bonding (IEEE 802.3ad) not working with qemu/virtio". - OVS defaults bonds to active-backup, which does not speak LACP at all — bond_mode=balance-tcp is required. - LACP deadlock: OVS holds members disabled until negotiation while the partner needs carrier before it will send LACPDUs. lacp-fallback-ab breaks it. - LACPDUs are untagged, so a trunk with no native VLAN has nowhere to put them. - --boot cdrom,hd re-runs the ISO on every restart, so every commit+save went to a live system that evaporated. Install now switches the VM to boot hd. - cloud-init on Alpine: users stay locked without lock_passwd:false, one failing runcmd aborts the rest, busybox here has no httpd applet, and start-stop-daemon --exec /usr/bin/python3 matches cloud-init's own python3. - The user-data heredoc is unquoted, so backticks in a COMMENT were executed by the host shell and their output corrupted the YAML. build_seed now validates with yaml.safe_load before building the ISO. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH |
||
|
Michal
|
df2dfc5d71 |
fix(bastion): pin the VyOS boot NIC by MAC, and detect pre-installer stalls
Some checks failed
CI/CD / typecheck (pull_request) Failing after 10s
CI/CD / test (pull_request) Failing after 10s
CI/CD / lint (pull_request) Failing after 24s
CI/CD / build (pull_request) Has been skipped
CI/CD / publish-rpm (pull_request) Has been skipped
CI/CD / publish-deb (pull_request) Has been skipped
Both Protectli VP2440s failed to install on real hardware: they fetched
kernel+initrd and then went silent. The console showed why —
Looking for a connected Ethernet interface ... e2 ? e3 ? e4 ? e5 ?
Connected e4 found
Connected e5 found
[4.595647] igc 0000:02:00.0 e2: NIC Link is Up
IP-Config: e4 ... no response after 15 secs - giving up
Unable to find a live file system on the network
live-boot picks the first *connected* interface. The i40e SFP+ pair links
before the igc copper port (up at 4.6s), so it chose the fiber ports, which
have no DHCP, and never tried the NIC that actually PXE booted.
Fix: pass BOOTIF=01-<mac> on the kernel cmdline. live-boot's
Device_from_bootif() (verified present in this image) matches it against
/sys/class/net and sets DEVICE directly. The MAC comes from the dispatch
key — i.e. exactly the NIC that PXE booted — which is more reliable than
iPXE's ${net0} on a box where the booting NIC may not be net0.
Why the integration test missed it: the VM had ONE NIC, so "first connected
interface" was trivially correct, and virtio links instantly so there was no
negotiation race. createPxeVm now takes decoyNics, attaching extra NICs
ahead of the PXE NIC on a network with no route to the bastion; the VyOS
test uses 2. Without BOOTIF that reproduces the hardware failure. getVmMac
is network-aware so it still returns the booting NIC.
Also: the bastion had every clue and said nothing — it logged INSTALL
STARTED, served kernel+initrd, then nothing for 7 minutes. dispatch now
stamps dispatched_at, and /api/logs/:mac returns stalled_for_s / stalled
(8 min threshold, sized for the ~600MB squashfs fetch), so a machine wedged
before the installer environment comes up is diagnosable without a console.
Verified on hardware: both firewalls installed, bond0 802.3ad + VLANs
2/3/9/10/200 + VRRP (priority 200/100, VIP .254 per VLAN) applied, and
/config/lab-provisioned written.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
|
||
|
Michal
|
e36a7a193c |
chore(cli): regenerate shell completions for VyOS flags
Some checks failed
CI/CD / lint (pull_request) Failing after 23s
CI/CD / typecheck (pull_request) Failing after 23s
CI/CD / test (pull_request) Failing after 23s
CI/CD / build (pull_request) Has been skipped
CI/CD / publish-rpm (pull_request) Has been skipped
CI/CD / publish-deb (pull_request) Has been skipped
pnpm completions:check was failing: labctl.fish/bash were stale. The generated --os choices still listed only fedora-43 and ubuntu-26.04 (missing vyos-rolling since the OsId union gained it), and none of the --vyos-*/--vlan flags were present. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH |
||
|
Michal
|
5d00c42f5a |
feat(bastion): bring VyOS provisioning to Fedora-grade quality
Some checks failed
CI/CD / typecheck (pull_request) Failing after 10s
CI/CD / test (pull_request) Failing after 9s
CI/CD / lint (pull_request) Failing after 24s
CI/CD / build (pull_request) Has been skipped
CI/CD / publish-rpm (pull_request) Has been skipped
CI/CD / publish-deb (pull_request) Has been skipped
Ports the Fedora provisioning features that matter for a router onto the VyOS path, and adds the libvirt integration test that proves them. - Live install logs: the driver streams the installer pty (ANSI-stripped, batched, best-effort) to POST /api/log, so `labctl provision logs -f` works during a VyOS install the way Anaconda's syslog does for Fedora. - installed.ip: report "ready at <ip>" -- the exact detail format routes/api.ts parses -- using the static mgmt address when known, else the live DHCP address. Without it VyOS machines landed with an empty IP, breaking provision list, logs-by-IP, recheck and reprovision. api.ts also guards the complete handler: VyOS boxes get the "vyos" SSH hint and never trigger the k3s post-provision. - EFI network-first boot order: port of the Fedora %post efibootmgr step, run from the live env after install (NVRAM, not disk). Best-effort. - Reinstall semantics: VyOS's installer already carries the previous config and SSH host keys forward -- the analog of Fedora's LV preservation -- so that stays the default. New --vyos-fresh-config overwrites the installed config.boot with the generated one instead, via a post-install target mount that also writes /config/lab-provisioned (mirrors Fedora's /etc/lab-provisioned, survives image upgrades). - reprovision/recheck default to the "vyos" SSH user for VyOS machines. Two hangs found by the VM test and fixed: - On reinstall the installer asks "Would you like to copy data to the new image?" (search_previous_installation). Unanswered, the driver blocked on stdin until its stall timeout -- a silent 15-minute hang. - The RAID regex missed "Would you like to choose two disks for RAID-1 mirroring?", which would wedge any multi-disk box. Both prompts default to yes, so a miss also risks an unwanted mirror. Both are now covered by a unit test asserting all 17 installer prompts match exactly one rule -- verified to fail against the unfixed code, so this class of bug is caught in a second instead of a 45-minute VM run. tests/integration/vyos-provision.test.ts: fresh install, reinstall preserves config + /config data, and freshConfig override. All 8 pass against the real nightly ISO (EXIT=0). 273 unit tests pass; no new lint errors in touched files. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH |
||
|
Michal
|
cb9d99dd69 |
feat(bastion): unattended VyOS network install with HA (bond + VRRP)
Some checks failed
CI/CD / lint (pull_request) Failing after 10s
CI/CD / test (pull_request) Failing after 10s
CI/CD / typecheck (pull_request) Failing after 23s
CI/CD / build (pull_request) Has been skipped
CI/CD / publish-rpm (pull_request) Has been skipped
CI/CD / publish-deb (pull_request) Has been skipped
VyOS ships no unattended installer (install_image() is unconditionally interactive; --no-prompt is wired only to 'add'), so the automation is injected through live-config's hooks component: iPXE boots the live kernel with fetch= and live-config.hooks=, the hook fetches a generated per-MAC Python driver, and the driver builds config.boot, stages the rootfs, and drives the interactive installer over a pty. Bastion: - vyos-boot.ipxe template (no 'nonetworking' — breaks the hook fetch; no console=ttyS0 — 30s/systemd-phase on UART-less boards) - /vyos/autoinstall.sh + /vyos/install.py routes (per-MAC driver with the config spec baked in as base64) - vyos-config-spec: bond0 (802.3ad) + tagged VLANs + VRRP groups (vrid = VLAN id) + sync group + hw-id pinning by MAC + SSH keys; config built from the image's own config.boot.default via vyos.configtree, version footer reattached via component_version - prepareVyosArtifacts: extract kernel/initrd/squashfs from the nightly ISO with xorriso; initrd picked by size from regular files only; ISO URL "latest" resolves the newest vyos-nightly-build GH release (downloads.vyos.io no longer serves direct ISOs) Verified end-to-end in a libvirt VM against the real nightly ISO — installed system boots with bond/VRRP/hw-id config applied and no migrations. Fixes found by the VM run, encoded in code comments: config.boot.default lives at /usr/share/vyos at hook time; fetch= boot has no medium so the rootfs is symlinked to the installer's expected path; reboot must be --force (the hook is a child of the still-starting live-config unit); installer disk answers are full /dev paths; zram0 passes the 2GB min-disk filter so the disk is always pinned. CLI/labd: vyos spec threaded through provision install (--vyos-* and --vlan/--vlan-vip flags with guards), labd install route, protocol command-install, and the bastion's direct /api/install. 268 unit tests pass; no new lint errors in touched files. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH |
||
|
Michal
|
4f9a6f64e4 |
k3s/networking: codify Multus + vlan-setup as lab operations
Some checks failed
The macvlan/VLAN-10 foundation HA depends on (Multus meta-CNI + a lan10
sub-interface + reference CNI plugins) was applied by hand during the HA
migration. Codify both as idempotent lab operations in the networking group,
after installCilium (which already sets cni.exclusive=false + bpf.vlanBypass={10}).
A fresh cluster now reproduces the full macvlan stack.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
Michal
|
12fa954a05 |
k3s/cilium: install with cni.exclusive=false + bpf.vlanBypass={10} (Multus + VLAN-10 macvlan mDNS)
Some checks failed
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
| 7181a61cec |
Merge pull request 'fix(labd): wire v2.0 Phase 1 routes + smoke tests' (#15) from fix/v2-wire-and-smoke-test into main
Some checks failed
|
|||
|
Michal
|
cdf3b5c045 |
fix(labd): wire v2.0 Phase 1 routes into createApp + smoke tests
Some checks failed
CI/CD / typecheck (pull_request) Failing after 11s
CI/CD / test (pull_request) Failing after 9s
CI/CD / lint (pull_request) Failing after 22s
CI/CD / build (pull_request) Has been skipped
CI/CD / publish-rpm (pull_request) Has been skipped
CI/CD / publish-deb (pull_request) Has been skipped
The v2.0 Phase 1 commit (
|
||
| f3c50f71ef |
Merge pull request 'feat: v2.0 Phase 1 foundation + bastion-restart identity fix + Dockerfile + BASTION_DIR' (#14) from feat/v2-phase1-foundation into main
Some checks failed
|
|||
|
Michal
|
98b0ccc6c9 |
feat(cli): honor BASTION_DIR env var as default for --dir
Some checks failed
CI/CD / typecheck (pull_request) Failing after 21s
CI/CD / test (pull_request) Failing after 22s
CI/CD / lint (pull_request) Failing after 7m2s
CI/CD / build (pull_request) Has been skipped
CI/CD / publish-rpm (pull_request) Has been skipped
CI/CD / publish-deb (pull_request) Has been skipped
bastion serve/stop default for --dir was hardcoded to /tmp/lab-bastion. Now reads BASTION_DIR from env if set, so a deployed bastion daemon can run from a persistent directory without callers having to pass --dir on every invocation. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Michal
|
37a3b51e57 |
build(labd): include @lab/core in the Dockerfile build chain
The v2.0 Phase 1 commit (
|
||
|
Michal
|
d6e1f3c74d |
fix(labd): preserve machine identity across bastion restarts
The worker0-k8s0 bug: when labd restarts, the in-memory installed map is lost. The next DHCP/PXE re-discovery for that MAC ran an upsert that wrote status="discovered", silently downgrading the DB record from "online" or "offline" and erasing the machine's known hostname/role identity from the CLI view. - server.ts: drop status="discovered" from the upsert update branch so re-discovery cannot downgrade an installed record. - routes/bastions.ts (/api/machines): when the DB knows a real hostname+role for a MAC currently only in live.discovered, promote it back to live.installed so the CLI sees the right state. Also reordered the live-vs-DB fallback so DB online/offline maps to live.installed and the discovered branch is the else. - tests: 3 new vitest cases covering promotion, fresh-discovery, and unknown-MAC fallback. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Michal
|
52e831b8c1 | Merge branch 'main' into feat/v2-phase1-foundation | ||
| f5af24699a |
Merge pull request 'fix(k3s): audit logs via journald + etcd recovery' (#13) from fix/k3s-audit-via-journald into main
Some checks failed
|
|||
|
Michal
|
dd92147341 |
fix(k3s): route audit logs through journald, codify etcd member recovery
Some checks failed
CI/CD / typecheck (pull_request) Failing after 13s
CI/CD / lint (pull_request) Failing after 23s
CI/CD / test (pull_request) Failing after 10s
CI/CD / build (pull_request) Has been skipped
CI/CD / publish-rpm (pull_request) Has been skipped
CI/CD / publish-deb (pull_request) Has been skipped
Two changes prompted by today's etcd raft panic on worker1-k8s0
(tocommit out of range, lost-write on follower) and the cascading
disk pressure that surfaced underneath it.
Audit logs to journald
- kube-apiserver now uses audit-log-path=- so audit events flow to
k3s.service stdout and into journald instead of growing files in
/var/log/kubernetes. The previous setup combined apiserver's
internal rotation with a logrotate *.log glob that double-rotated
the rotated files into permanent orphans (observed: 7+ GB).
- New journald-limits operation writes a SystemMaxUse=2G drop-in so
audit volume cannot fill /var/log even under bursty load.
- log-rotation operation repurposed to decommission the obsolete
logrotate rule and reap leftover audit files. Idempotent: no-op
on fresh installs.
Etcd member recovery
- New recoverEtcdMember(broken, peer, hostname) codifies the
documented k3s recovery: stop k3s, etcdctl member remove, wipe
/var/lib/rancher/k3s/server/{db,tls,cred}, restart, poll for
rejoin. Refuses to operate when cluster size < 3 to preserve
quorum.
Tests
- 7 new unit tests covering both decommission paths and the
recovery procedure (54 total, all green).
- install.test.ts asserts the file-based audit args are gone.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Michal
|
04faa079e2 |
feat: v2.0 Phase 1 foundation — @lab/core, auth, RBAC, audit, resource store
New packages: - @lab/core: Resource types, Output<T> (Pulumi), audit event types, auth types, environment/account types, resource kind registry New Prisma schema (mcpctl pattern): - User (email/password/bcrypt), Session (bearer tokens), Group, GroupMember - ServiceAccount, RbacDefinition (JSON subjects + roleBindings) - AuditEvent (correlation IDs, causal chains, fire-and-forget batching) - Environment, Account (driver config, Infisical secret path), Binding - Resource (generic, kind/name/env unique, origin/managedBy tracking) - Secret, Fleet, FleetMember, GitSource - Keeps v1.0 models: Server, Agent, Bastion, Cluster, JoinToken New services: - AuthService: bearer token login, bootstrap (first login creates admin), session management with 30-day expiry - RbacService: environment-scoped permission checks, group membership, role hierarchy (admin > edit > view) - AuditService: fire-and-forget event collection, batch 50 / flush 5s, correlation IDs for causal chains - ResourceStore: CRUD with origin/managedBy, RBAC-enforced routes New routes: - POST /api/auth/login, POST /api/auth/logout (bearer token auth) - GET/POST/PUT/DELETE /api/resources (RBAC-enforced CRUD) - GET/POST /api/environments, GET/POST /api/accounts - POST /api/accounts/bind, GET /api/bindings - GET /api/events (audit query with --last, --kind, --env, --correlation) New middleware: - Bearer token auth (validates Authorization header, resolves user identity) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |
||
| 95c99cb4d5 |
Merge pull request 'docs: CLAUDE.md routing rules + TODOS.md from v2.0 review' (#12) from feat/recheck-and-fixes into main
Some checks failed
Reviewed-on: #12 |
|||
|
Michal
|
2eda926d4c |
docs: add TODOS.md from v2.0 CEO review
Some checks failed
CI/CD / typecheck (pull_request) Failing after 12s
CI/CD / lint (pull_request) Failing after 21s
CI/CD / test (pull_request) Failing after 11s
CI/CD / build (pull_request) Has been skipped
CI/CD / publish-rpm (pull_request) Has been skipped
CI/CD / publish-deb (pull_request) Has been skipped
Project tracking for labctl v2.0 platform design. Includes P1 (arch doc update), P2 (SSH emergency mode, Prometheus metrics), and P3 (graph viz, import, secrets rotation) items from the CEO and eng review sessions. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Michal
|
70258a0cc3 | Merge remote-tracking branch 'origin/main' into feat/recheck-and-fixes | ||
|
Michal
|
e9944c5413 | chore: add gstack skill routing rules to CLAUDE.md | ||
| 22e2946e95 |
Merge pull request 'feat: provision recheck, hardware info preservation, ISO boot fixes' (#11) from feat/recheck-and-fixes into main
Some checks failed
Reviewed-on: #11 |
|||
|
Michal
|
9ddab24931 |
feat: provision recheck, hardware info preservation, ISO boot fixes
Some checks failed
CI/CD / lint (pull_request) Failing after 1m26s
CI/CD / typecheck (pull_request) Failing after 11s
CI/CD / test (pull_request) Failing after 11s
CI/CD / build (pull_request) Has been skipped
CI/CD / publish-rpm (pull_request) Has been skipped
CI/CD / publish-deb (pull_request) Has been skipped
- Add `labctl provision recheck` to refresh hardware info via SSH - Preserve hardware info in InstalledInfo when install completes - Fix /ks-auto: run nested %pre scripts from included kickstarts - Add command-discover WebSocket routing for hw info updates - Fix k3s join: clean stale TLS/cred when joining existing cluster - Add --tls-verify=false for internal HTTP registry pushes - Add fix-ssh-root.sh script for root SSH access on all nodes Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
