141 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
Michal	e8679f45b5	labsim: rehearse the IPAM switch on 3 nodes, and catch the trap in it ... Rehearsed kubernetes -> cluster-pool on the 3-node labsim cluster, which is the transition production faces. The switch itself is undramatic: agents stayed up, the operator adopted the pool, and the agent-not-ready taint deadlock did NOT occur. That deadlock is specific to ADDING IPv6 -- the agent blocks on an IPv6 pod CIDR that does not exist yet. A v4-only mode switch does not hit it. The real hazard is quieter. The operator does not preserve which node held which /24: two nodes swapped CIDRs. Their existing pods kept their old addresses, which now fall outside the node's range, so every other node routes that prefix to the wrong node. Cross-node ping to those pods dropped 100% while every pod stayed Running and every node stayed Ready. Nothing in `kubectl get pods` shows it. So "pods kept the same address" is the FAILURE signal here, not the reassurance it looks like. cilium-ipam-switch.sh verify now flags pods sitting outside their node's CIDR, which is the check that decides whether a recycle is optional (it is not) or mandatory (it is). Recycling every deploy/ds/sts restored it: all pods back inside their node CIDR, cross-node ping 0% loss. Sequence proven end to end: preflight -> apply -> restart operator then agents -> unstick if needed -> recycle all workloads -> verify Also fixed the recycle hint the script printed: `kubectl rollout restart deploy,ds,sts -A` is not valid (`unknown shorthand flag: 'A'`), so anyone following it under pressure would have got an error instead of a recycle. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH	2026-08-24 23:13:20 +01:00
Michal	527e0798ae	bastion/k3s: bootstrap new clusters with cluster-pool IPAM ... Every cluster this bastion builds was pinned to ipam=kubernetes, which makes it permanently single-stack: Cilium reads node.spec.podCIDRs, the controller-manager writes that once at node join and never revises it, so adding IPv6 later fails with `required IPv6 PodCIDR not available` and needs a rebuild. cluster-pool puts allocation in the CiliumNode CRD, where the operator can add an address family to a running node. Proven in labsim (842408c). It is also Cilium's own default -- ipam=kubernetes was the deviation. Pool matches the k3s cluster-cidr (10.42.0.0/16, /24 per node), so a node gets the same CIDR shape it would have had. IPv4-only for now: the IPv6 pool wants an apiserver carrying an IPv6 service CIDR, which is a separate change. Mirrors @michal/cilium-values ciliumBaseValues (cilium-values@11ee509), which these paths track but do not import. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH	2026-08-24 22:59:55 +01:00
Michal	842408c0d9	labsim: prove k3s CAN be converted to dual-stack in place ... k3s documents that dual-stack "cannot be enabled on an existing cluster". Rather than accept that for a 145-day-old production cluster, build both shapes and diff them. dualstack-lab.sh builds a single-node IPv4 cluster and a native dual-stack one, takes a reflink copy of the IPv4 disk so a failed conversion costs 90 seconds to undo, converts in place, and diffs the results. Result: the conversion works. etcd data is never touched. The documented blocker is real but narrower than stated. Cilium reports it exactly -- `required IPv6 PodCIDR not available` -- because node.spec.podCIDRs is assigned at join and is immutable, and the Kubernetes IPAM controller will not add a second family later. That objection only holds while Cilium runs ipam=kubernetes and therefore reads that field. Switching to cluster-pool IPAM moves pod CIDR allocation into the CiliumNode CRD, where the operator hands out both families on a cluster that was born IPv4-only. Sequence that works, in order: 1. k3s unit gains --cluster-cidr/--service-cidr/--node-ip with both families. k3s validates the two CIDRs together and refuses to start on a mismatch ("must share the same IP version"), so a partial edit crash-loops rather than coming up half-configured. That is the safe failure mode. 2. The ServiceCIDR object picks up the IPv6 range on restart -- this is upstream's supported "single-to-dual-stack preserving the primary ServiceCIDR" path, and existing Services keep their IPv4 addresses. 3. Cilium to ipam=cluster-pool with an IPv6 pool, then DELETE the CiliumNode so the operator reallocates; it will not add a family to an existing one. 4. Expect a deadlock here: the agent will not go ready without a pod CIDR, so the node keeps the node.cilium.io/agent-not-ready taint, so the new operator that would assign the CIDR cannot schedule. Remove the taint by hand once to break it. Verified on the converted cluster: pod with 10.42.0.125 AND fd00:42::4843, and a PreferDualStack Service holding 10.43.122.115 AND fd00:43::72a3. The only field that still differs from a native build is node.spec.podCIDRs, which stays IPv4-only -- immutable, and unused once Cilium owns IPAM. CiliumNode podCIDRs and ServiceCIDR are identical to the native cluster. Not yet answered: this is one node. Whether a 3-server etcd cluster converts as cleanly, and what a rejoining agent does, is the next experiment. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH	2026-08-24 22:46:27 +01:00
Michal	ad6eb7a9a6	labsim: default-deny firewall policy, proven in the sim ... Policy: internal VLANs reach each other and the internet; the internet initiates nothing inward. That was already the effect of the IPv4 ruleset, but built as a blacklist -- default-action accept plus explicit drops per WAN interface. Identical behaviour right up until a WAN is added, at which point it is open and nothing looks wrong. This expresses it as a whitelist. Two findings from the sim, both of which would have been outages in production: `set` on a rule number is ADDITIVE. The sim already had a rule 10 carrying inbound/outbound interface constraints; `set ... rule 10 state established` ANDed onto it, producing a stateful-accept that applied to one interface pair only. Return traffic from the internet then matched no rule and hit the default drop, so LAN hosts could reach nothing outbound. The generator now deletes each filter before rebuilding it, so the code owns the subtree. It is one commit, so nftables is rebuilt atomically -- there is no window without a firewall. DHCP lease renewal is unicast UDP to port 68 and conntrack does not reliably cover it. Without an explicit rule the WAN keeps working until the lease expires and then dies -- a delayed failure that looks nothing like a firewall change. Also added a loopback accept for both families, absent from the v6 policy since it went default-deny. Verified in labsim: inter-VLAN ok, LAN-to-internet ok, internet-to-router dropped, and internet-to-LAN dropped with the drop counter incrementing by exactly the packets sent, after routing the test through the router rather than around it via the hypervisor. Also extends the drift check to the firewall subtree, which it did not cover -- so it had been reporting "in sync" while that subtree was uncaptured. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH	2026-08-22 22:25:55 +01:00
Michal	7f551081ad	labsim: capture BGP, dual WAN and both ISP VMs as code ... The sim's routing config existed only as running state on the VMs. It was applied by hand over SSH, so rebuilding a VM lost the rehearsal and nothing recorded why any of it was shaped the way it was. The two ISP VMs were not referenced anywhere in the repo at all. sim-net-config.py generates all four roles; sim-net-apply.sh applies them over the serial console, or diffs them against the running VMs. Verified reproducing live state exactly before committing: primary 40/40 commands, secondary 16/16, isp-dhcp 19/19, isp-pppoe 21/21. Carries the reasoning that was previously nowhere: RFC 8212 needing policy in both directions or the session carries zero prefixes; probe targets that must not double as system name-servers; default-route-distance 210 rather than no-default-route, which blanks new_routers and hands the default route to the backup line; and the WI-8 bootstrap bug that pinned /32s fix. Dropped a stale `pppoe-server interface eth0` on isp-pppoe (a NIC that does not exist there) so a green drift check stays meaningful. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH	2026-08-22 16:26:36 +01:00
Michal	f41ffdd039	feat(vyos): reconciler that keeps the HE 6in4 tunnel on the live WAN ... Kernel-level (`ip tunnel change`), not VyOS config: no commit churn on a flapping line, no drift against the Pulumi model, and a reboot restores config.boot — which pins the 10 gig — so a wrong source cannot survive a restart. Sends `myip` explicitly, because mid-failover the update request may egress either line and letting HE infer the address would point the tunnel at the WAN we just left. Requires two consecutive agreeing runs before acting, since HE rate-limits updates and a flapping WAN would hammer the API precisely when it matters. MTU moves with the WAN: 1480 on the 10 gig (1500-20), 1472 on PPPoE (1492-20). Fixed at 1480, the backup path gives the signature people lose a day to — small packets fine, large transfers hang. Inert without /config/he-secrets, and a no-op when already in sync.	2026-08-21 02:04:43 +01:00
Michal	a187703a3a	feat(vyos): pin a known-good config and restore it with one command ... VyOS already has rollback, but `rollback 1` returns you to the *previous* revision, which may itself be broken — you can end up walking backwards through several bad commits hunting for the one that worked, at exactly the moment you have no network to look things up with. This pins a state a human has actually used and found working, so recovery is one step and needs no memory of how many changes ago things were fine. /config/vyos-known-good save pin the running config /config/vyos-known-good status when it was taken, how running differs /config/vyos-known-good diff what a restore would change /config/vyos-known-good restore go back to it Deliberately not automatic. A config is only known-good once someone has used the network; a snapshot taken after every commit would faithfully preserve the broken one. The restore is itself commit-confirmed, so even the recovery path is protected: if the snapshot is somehow wrong, or access is still broken and nothing can be confirmed, the router undoes the restore rather than leaving you worse off. Silence reverts. `save` refuses when there are uncommitted changes — a snapshot that did not match what is actually running would look like a safety net without being one. Two things found while building it, both of which made the script silently useless rather than fail loudly: - Sourcing `script-template` **resets the positional parameters**, so `$1` was empty by the time the case statement ran and every invocation fell through to the usage message. Arguments are captured before the source. - `0600` made the snapshot unreadable to the `vyos` user, so `status` and `diff` — the two commands you run while deciding whether to restore — showed nothing. Now 0660 root:vyattacfg, matching /config/config.boot. Installed on both routers with the current, verified-working config pinned (vyos001 1029 lines, vyos002 1019). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH	2026-08-21 01:46:43 +01:00
Michal	86c2a36f00	feat(labsim): a real Kubernetes cluster for rehearsing Cilium <-> VyOS BGP ... BGP is about to be added to a network that currently works, where a bad advertisement blackholes the house. That needs somewhere to fail first. Three Debian nodes (4 GB / 2 vCPU) on the OVS `vlan2` access ports running k3s with flannel disabled, so Cilium is the CNI under test. Three rather than two because ECMP is only meaningfully tested if a node can be drained and more than one path survives. VMs rather than k3d: the thing under test is eBGP between Cilium and VyOS across the switch fabric, nodes peering with the router's bond0.2 leg, directly connected. k3d would put the nodes on a container bridge -- a different L2 path, proving something else. (It also wants Docker; this host has podman.) The existing micro VMs are Alpine with 256 MB and 1 vCPU, which is not close to enough. Debian rather than the sim's Alpine base: glibc, a stock kernel that Cilium's eBPF probes are tested against, and cloud-init that actually applies network-config -- the Alpine base notably does not. One trap worth recording. An earlier draft called `selected_vlans "$K8S_VLAN"` before `ovs_up`, and since `ovs_up` re-defines the libvirt network from SELECTED, that silently deleted the portgroups for every other VLAN. Running VMs kept working -- their taps were already attached -- so nothing complained until the ISP VMs needed vlan51 and vlan53 and could not be attached. It now selects every VLAN. The generated kubeconfig is gitignored: it carries cluster-admin credentials and is one `git add -A` away from being committed. Regenerate with `k8s-up.sh --kubeconfig`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH	2026-08-19 13:15:45 +01:00
Michal	27a343bc75	Merge branch 'feat/unifi-export-and-vyos-dhcp': USG to VyOS migration ... Carries the UniFi export tooling, the generated VyOS DHCP/DNS config, the reversible cutover switch, the health-checked WAN failover, and the labctl side of applying a Pulumi-rendered bundle at install time. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH	2026-08-18 12:19:48 +01:00
Michal	672b89ce38	feat(labctl): install VyOS from a Pulumi-rendered bundle, and enable its API ... Two halves of the same problem: a router should come up running the config that is declared for it, and should be manageable the moment it does. --vyos-bundle applies a bundle rendered by kubernetes-deployment verbatim, replacing the derived --vyos-bond/--vlan/... path rather than merging with it. Deriving a second opinion alongside a bundle is exactly the drift the bundle exists to prevent: Pulumi and labctl would each believe they knew the router's config and the box would end up with whichever ran last. Passing both is rejected rather than silently resolved. Secret-valued nodes arrive as @secret: sentinels and are dropped, with a warning naming each one. Writing the sentinel text into config.boot would look configured while being wrong, which is worse than being absent -- the router comes up without its PPPoE credential and the first `pulumi up` supplies it. A bundle committed to git has to stay safe to read. The hostname is forced to the one the install was asked for. A bundle is exported from one router and reused for its peer, and taking the hostname from it would put two vyos001s on the network. --vyos-api-key enables the HTTP API at install, on both the bundle and the derived path, so every VyOS this bastion provisions is manageable from first boot. vyos001 and vyos002 predate this and had to be enabled by hand on a live firewall after their cutover -- which is the gap this closes. It is deliberately not part of the Pulumi model: a provider able to rewrite its own transport can revoke its own access. listen-address is always set, and the API is NOT enabled when no address is known -- under DHCP there is none at build time, and binding to every interface would publish a config-write endpoint on the WAN. It warns and leaves the router SSH-only instead. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH	2026-08-18 12:18:53 +01:00
Michal	f4984e3962	fix(vyos): health-check the 10 gig primary so failover actually fires ... The 10 gig line was primary by route distance alone, which only fails over when bond0.53 loses carrier or its DHCP lease. An ISP that keeps the link up while dropping traffic -- the common failure -- would black-hole everything, because a DHCP-installed route has nothing to withdraw it. `protocols failover` now owns the live default route and pings two targets bound to the interface, so the backup can never be validated through the primary's path. Rehearsed on the labsim router: failover and failback both inside 5s with the router's own interface still UP. The vif keeps default-route-distance rather than no-default-route, demoted below Vodafone. vyos-failover resolves a dhcp-interface gateway by reading new_routers out of /run/dhclient/dhclient_<if>.lease, and no-default-route leaves that field EMPTY -- the daemon then finds no next hop and installs nothing. Observed on vyos001: the default route fell through to Vodafone. Preference is now failover's kernel route (distance 0) > pppoe (10) > DHCP (210), so the demoted route can never re-create the black hole it exists to avoid. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH	2026-08-18 12:18:41 +01:00
Michal	7b5331ddcd	fix(migration): the backup has no WAN by design; stop failing it for that ... The cutover succeeded on vyos001 -- gateway live, bond0.53 holding 87.192.101.48 with the cloned MAC, kea serving, clients routing out through NAT. vyos002 then ran the same script and was judged unhealthy, because the mandatory checks are "default route exists / internet reachable / DNS resolves" and the backup deliberately holds its WAN interfaces DOWN. Its config was correct; the check did not apply to it. Confirmed by hand before the timer could revert a good config. This is the third instance of one mistake: asserting a condition that is not true of the box being checked. First requiring every WAN when one suffices, now requiring a WAN on the box that is configured not to have one. A delta containing `interfaces ... disable` for the WAN now identifies the backup, and the WAN-dependent checks are skipped with a note. kea and the DNS forwarder remain mandatory on both -- those are what the backup must actually be able to do. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH	2026-08-18 01:48:48 +01:00
Michal	ce6911c196	fix(migration): require a working WAN, not every WAN ... This reverted a cutover that had actually succeeded. The evidence, from the revert tearing it down: dhclient: DHCPRELEASE of 87.192.101.48 on bond0.53 to 185.232.119.244 vtysh: "no ip route 0.0.0.0/0 87.192.96.1 bond0.53 tag 210 1" netlinkd: RTM_NEWLINK -> bond0.53, mac=f0:9f:c2:12:9b:4f bond0.53 came up with the cloned MAC and was handed 87.192.101.48 -- the exact public address the USG holds -- with a default route via the real ISP gateway. kea was serving live LAN clients at the same moment (10.0.0.12, 10.0.0.13, 192.168.8.28). The gateway was working. The only failure was pppoe0: ppp@pppoe0.service exited 5/NOTINSTALLED. That is the Vodafone FAILOVER line, and the health check listed "pppoe0 has an address" as mandatory, so a working gateway was torn down because its backup WAN was down. The check encoded "every WAN must work" when the requirement is "the box must reach the internet". Now: default route, reachability and DNS are mandatory; each WAN interface is reported individually but fatal on neither. A failover line being down is worth seeing, not worth reverting for. This also incidentally settles the last genuine unknown in the migration, which could not be tested any other way: the ISP does hand the same lease to the cloned MAC. That was the one thing I had said was unknowable until the USG let go of it. Note the earlier polling fix (54b21fa) addressed a real weakness but not this failure -- no amount of waiting would have satisfied a check that required a line which was never going to come up. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH	2026-08-18 01:37:14 +01:00
Michal	54b21fa9ff	fix(migration): poll for WAN health instead of sampling once at 25s ... A real cutover attempt reported failure and reverted a configuration that may well have been fine. The health check waited a fixed 25 seconds and then judged: [switch] committed. Waiting 25s for PPPoE and services to settle... [switch] FAIL pppoe0 has an address 25s is far too short for a WAN. PPPoE alone is PADI/PADO/PADR/PADS followed by LCP, authentication and IPCP -- routinely 15-30s on its own. Both lines had also just been released by the USG seconds earlier, and ISPs commonly hold the previous session and MAC binding for minutes before leasing to the "same" CPE again, which is exactly what a cloned MAC looks like from their side. The one thing the design could not tolerate was being impatient, and it was. Now polls every 15s up to HEALTH_BUDGET (default 180s), reporting progress, and stops early the moment everything is healthy. The budget deliberately finishes long before commit-confirm fires -- 180s against a 10 minute timer leaves 420s of margin -- so the decision to confirm or revert stays ours rather than being made by the timer. Also recorded while chasing this: the earlier claim that VLANs 51/53 are not trunked to the firewalls was WRONG, and the UniFi port settings disprove it -- those LAG ports are Native VLAN Management (1) with Tagged VLAN Management set to Allow All. My evidence never supported the claim: a passive RX count cannot distinguish an absent VLAN from a quiet one, because switches do not flood unicast, and the active DHCP probe used a random MAC that an ISP binding to its registered CPE would ignore regardless. Both observations fit a perfectly healthy trunk. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH	2026-08-18 01:10:50 +01:00
Michal	ff86a421f4	fix(labsim): console-apply must handle both VyOS prompts, not just $ ... Two failures from one strict expect, both hit while building the sim ISPs. A run that dies mid-config leaves the console parked in configuration mode. The next run then waits for the operational `$ ` prompt against a perfectly healthy VM and hangs until timeout, with nothing in the output to say why -- the box was sitting at `vyos@isp-dhcp#` the whole time. Login now accepts `# ` as well and discards the stale candidate rather than committing something nobody has seen. The same mistake at the exit step: insisting on `$ ` after `save` hung, AND left the console in config mode, which is what created the first failure for the following run. Now accepts either prompt. Known-bad, not fixed: the tool reports "committed and saved" when the set commands have not applied. Verified against the clone -- prompt showed the host-name change had landed while `grep -c dhcp-server` returned 0. The failure detection only inspects c.before for a few strings and evidently misses the real failure mode, so success is being reported without evidence. That needs fixing before this tool is trusted for anything; it is currently only safe to use with an independent check afterwards, which is how the gap was found. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH	2026-08-18 01:00:19 +01:00
Michal	ee070371a8	feat(labsim): add WAN transport VLANs so the sim can host fake ISPs ... A cutover attempt failed on the WAN and nothing had tested it. The reason the sim could not have caught it: labsim modelled every LAN VLAN faithfully and omitted the WAN entirely -- vlans.conf had 1, 2, 3, 9, 10 and 200, never 51 or 53. Worse, the switch script's WAN health checks are conditional on the delta configuring PPPoE, so in the sim they printed "this delta configures no WAN -- skipping all WAN health checks" and passed. The sim proved the delta commits; it never proved the WAN works, and could not have. Adds VLANs 51 (Vodafone/PPPoE) and 53 (10gig/DHCP) to the fabric so a fake ISP can live on each and those checks actually execute. VyOS has service pppoe-server (accel-ppp) natively -- authentication local-users, client-ip-pool, gateway-address -- so a VyOS VM can play the concentrator, and dhcp-server can play the other ISP. vlans.conf gains host_octet 0, meaning "no host leg". A host address on a WAN transport VLAN would misrepresent the segment: the point is that VyOS reaches an ISP, not the host. Also fixes a real gap in ovs_bond_router: it returned early when the bond already existed, so adding a VLAN to vlans.conf never reached an existing bond. Re-runs now reconcile the trunk and say so. That gap is the same SHAPE as the production failure -- interface present, VLAN missing from the trunk, frames silently dropped -- which is precisely the class of bug the sim needs to be able to reproduce rather than embody. Both bonds updated: [2,3,9,10,200] -> [2,3,9,10,51,53,200]. Note on the production diagnosis, which is NOT settled: a passive RX test showed zero frames on 51/53 at the firewall, and an active DHCP DISCOVER (verified to have transmitted, tx +2) drew no reply. That is consistent with the VLANs not being trunked, but equally with the ISP only answering its registered CPE MAC -- which is exactly why the delta clones f0:9f:c2:12:9b:4f, and why it cannot be settled from production while the USG holds that MAC. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH	2026-08-18 00:44:11 +01:00
Michal	41b5448f56	feat(pulumi-vyos): prototype VyOS subtrees as Pulumi resources with commit-confirm ... Goal: change VyOS and Kubernetes in one codebase and one plan -- so a BGP change touches both sides in a single `pulumi preview`. First, the worry about per-command pushes turned out to be unfounded for the community providers. Read foltik/vyos and its client library: a `vyos_config_block_tree` flattens the whole subtree into a single payload array and sends ONE POST to /configure, so one resource is one commit. Good. What they do not do is send `confirm_time`. Their payload is only op/path/value, so every change is an unprotected commit -- on a router you reach through the router, that is the difference between a mistake and an outage. The VyOS API itself supports commit-confirm; the providers simply do not use it. So this is a ~180-line Pulumi dynamic provider that does. Verified end to end on labsim: create and update each land in ~6s as one commit-confirmed transaction, update reports [diff: ~commands], destroy removes the subtree, and an unconfirmed commit was observed reverting the router on its own. Three API details found the hard way, all now encoded and commented: - confirm_time is ONLY read when the body parses as ConfigureListModel, i.e. {"commands": [...], "confirm_time": N}. A bare array is accepted and committed with NO timer armed, and the response looks like success. This silently discards the entire safety net, so the resource now checks the response actually says "commit-confirm" and refuses to proceed otherwise. - There is no /confirm endpoint; confirm is an op on /configure. - Confirm requires a `path` field even though it ignores it -- the Union resolves to ConfigureModel, which mandates path. Without it: "missing 'path' field", and the timer keeps running. Apply is `delete <path>` followed by the sets, in one request, so the result is the declared state rather than a merge -- otherwise `pulumi up` accumulates instead of converging. Known gaps, in the README rather than hidden: no read/refresh so out-of-band drift is not detected, and the API runs with a self-signed certificate and verification disabled. Both need addressing before production. The cutover itself should still use vyos-unifi-switch, which the API cannot replace. Sim left as found: test resource destroyed, dns forwarding restored to 15 lines. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH	2026-08-17 01:09:05 +01:00
Michal	63061e6e7e	feat(migration): peer link cabled and verified; conntrack-sync enabled in deltas ... eth3 <-> eth3 direct cable is in. Both ends negotiated 2500Mb full duplex -- these are 2.5 GbE ports, not the 1G I had assumed. Carrier alone proves nothing, so the link was tested end to end with temporary kernel-level addresses (never committed to VyOS config, removed afterwards): 3/3 packets, 0% loss, 0.371ms average. A cable can show carrier and still not pass traffic; now it is known to. Deltas regenerated with --conntrack-link and installed on both boxes: vyos001 nat=21 fw=58 conntrack=9 eth3=10.255.255.1/30 disable=0 vyos002 nat=21 fw=58 conntrack=9 eth3=10.255.255.2/30 disable=2 Identical apart from the peer /30, the VRRP/DHCP-HA roles, and the two disable lines holding vyos002's WAN down. Both still report mode=unifi and nothing about their behaviour has changed -- eth3 carries no address in the running config, and conntrack-sync appears only in the delta, which is applied at cutover. Not verified: multicast on the peer link. `ping -I eth3 224.0.0.1` drew no responders, but that is the all-hosts group which VyOS need not answer, so it proves nothing either way. conntrack-sync's own multicast (225.0.0.50) was proven working in labsim over bond0.10, and this is a point-to-point link, so the risk is low -- but it is untested on this specific cable and worth watching at cutover. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH	2026-08-17 00:37:16 +01:00
Michal	ccdd1e7e49	fix(migration): both boxes carry WAN and NAT; the backup just holds it down ... "No NAT? How are we supposed to get internet?" -- a fair question that exposed a worse design than I had admitted. Internet did work, but only via vyos001: NAT and the entire WAN were gated behind --with-wan, so vyos002 would have held the LAN VIPs and routed between VLANs with no path to the outside at all. Failover would have preserved addressing and lost the internet. The fix rests on a checked fact rather than an assumption: VyOS WARNS but still commits when a NAT rule names an interface that does not exist ("Interface bond0.53 for source NAT rule 900 does not exist!"). Verified on a real VyOS before relying on it. So both boxes now get the identical WAN, NAT, port-forward and firewall config, and the backup's two WAN interfaces are simply set `disable`. The cloned WAN MAC is therefore never live on two boxes at once, while everything needed to route and masquerade is already in place. The two deltas are now byte-identical apart from VRRP priority, own/peer addresses, DHCP HA role, the conntrack /30 -- and the two disable lines. Taking over the internet path becomes deleting two lines rather than reconstructing NAT under pressure: delete interfaces bonding bond0 vif 53 disable delete interfaces pppoe pppoe0 disable Both boxes now: 21 NAT rules, 58 firewall rules, full PPPoE. Backup delta validated against a real VyOS config with the disable lines present -- commits clean. Runbook updated with the takeover procedure and the warning that it must only be done when vyos001 is genuinely down. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH	2026-08-17 00:30:23 +01:00
Michal	64e748ea94	test(labsim): conntrack-sync verified, and it exposed a delta defect ... conntrack-sync proven working on the sim pair -- bidirectional replication with zero errors: MASTER internal 34 external(from peer) 52 62 pkts sent / 109 recv 0 err BACKUP internal 76 external(from peer) 36 142 pkts sent / 73 recv 0 err Getting there required learning something that changes the production config: **VyOS only engages conntrack when a firewall or NAT is configured.** With neither present, both routers reported zero conntrack entries and conntrack-sync had nothing to replicate. Adding a single state-matching forward rule turned tracking on and replication began immediately. That is a defect in the delta, not just a test artifact. NAT and the firewall were both gated behind --with-wan, so the BACKUP would have had neither -- it would not have tracked connections at all, and replicated entries are useless to a box whose conntrack is not engaged. Exactly the failure that only shows up during a failover, when it is too late to notice. Fixed: a stateful forward rule (accept established/related, default-action accept) is now emitted on BOTH boxes, outside the WAN gate. Only NAT and the WAN-scoped rules remain master-only. Verified: vyos002 now carries stateful tracking and conntrack-sync but zero NAT lines. Master delta re-validated against a real VyOS config -- no errors. Also incidentally confirmed no-preempt: router1 rebooted and came back as BACKUP rather than seizing the VIP, which is the opposite of what the production pair did this afternoon (still on default preempt until cutover). Two traps recorded while doing this: - The detached `setsid nohup` config-apply pattern can strand a VyOS config session. An orphaned session (dirs under /opt/vyatta/config/tmp/, PID long dead) blocked every subsequent `set` on that box with a bare "Set failed", and the dirs are overlay mounts so they cannot simply be deleted. Rebooting cleared it. This pattern is used to survive losing SSH mid-change, so it is worth knowing it has a failure mode of its own. - Only VLAN 10 passes traffic between the two sim routers; every other VLAN fails ARP despite identical vlan_mode/tag/trunks on both OVS bonds and distinct MACs. VRRP forms on all six groups regardless. The sync link had to be bond0.10 as a result. OVS-specific, absent in production, but it means the sim proves mechanism rather than topology. Production deltas regenerated with --conntrack-link: eth3 at 10.255.255.1/30 and .2/30 awaiting the cable, which is not yet plugged (carrier=0 on both). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH	2026-08-17 00:17:21 +01:00
Michal	bb654d83f8	test(labsim): second VyOS router proves DHCP active-passive HA ... Answers the question a single router could not, and that would otherwise only have been discovered at cutover: with kea high-availability active-passive, does exactly ONE box answer a DHCP request? Yes. Probing sim VLAN 10 with broadcast-dhcp-discover returns offers from a single distinct Server Identifier -- 172.31.10.252, the primary. The secondary runs kea but stays silent. Without this the delta would have put 6 subnets and 84 static-mappings on both boxes with nothing to arbitrate them, and two kea instances would have raced on every broadcast domain. Worth noting the raw response count is misleading: nmap reports "Response 1 of 2" because it sends several discovers, and both replies carry the same server identifier. Counting responses says "2 servers"; counting distinct server identifiers says "1". The second number is the true one. labsim now runs a real pair, mirroring production: router1 172.31.<v>.252 priority 200 DHCP HA primary router2 172.31.<v>.253 priority 100 DHCP HA secondary VIP 172.31.<v>.1 floating, held by the master That required converting router1, which held .1 directly, to .252 plus a floating VIP -- otherwise it is two routers, not a pair. All six VRRP groups show MASTER on router1 and BACKUP on router2. New tooling: - sim-ha-config.py generates each role's config, reusing unifi-to-vyos.py --mode sim for the DHCP half so what is proven here and what production gets share a code path. VLAN 10 correctly carries /23. - console-apply.py applies config over the serial console, which is necessary because a freshly installed VyOS holds the same addresses as its peer and cannot safely be reached over the network at all until reconfigured. Known sim-only quirk, deliberately not chased: router1 cannot ARP router2 on the untagged VLAN 1 while every tagged VLAN works, and VRRP forms correctly on all six groups regardless. Both OVS bonds carry identical vlan_mode/tag/trunks and the bond MACs differ, so this is OVS bond behaviour on the native VLAN with two bonds on one bridge -- not a VyOS config problem, and not present in production, which uses a real switch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH	2026-08-16 23:29:12 +01:00
Michal	952f5c66e3	feat(migration): complete the VyOS HA stack per the official docs ... Prompted by "I thought we tested HA on libvirt" -- checking rather than recalling showed the sim has ONE VyOS router with zero high-availability config. VRRP was configured and running on the real pair, but it is only one of four parts of what VyOS considers an HA pair. Against docs.vyos.io (highavailability, conntrack-sync, dhcp-server, and the HA walkthrough), three gaps are now closed in the delta: - VRRP was multicast-only with default preemption. Added unicast hello-source-address/peer-address per group, as the walkthrough does, plus no-preempt. Without no-preempt a recovered box reclaims the VIP before conntrack state has synced and drops every established connection; the docs are explicit that preempt-delay must otherwise be >= purge-timeout. The per-VLAN node addresses are a table, not derived: VLAN 3 is .4/.5 while every other VLAN is .252/.253. - DHCP high-availability, which fixes a real defect rather than adding a feature. Both boxes carried the full 6 subnets and 84 static-mappings, so after cutover two kea instances would have raced on the same broadcast domains. Now active-passive with primary/secondary and swapped source/remote, syncing over TCP 647 on the LoT addresses. Each subnet already carries the unique subnet-id kea HA requires, and the peer name deliberately differs from both host-names. - conntrack-sync over a dedicated eth3 <-> eth3 link, gated behind --conntrack-link because it needs a cable that is not plugged in yet. This is what the peer cable is actually for -- VRRP does not want one, since its hellos must travel on the segment they protect. VRRP failover exercised on the production pair, which is free to break today because nothing uses the .254 VIPs: keepalived stopped on vyos001, all six VIPs moved to vyos002 within 12s, and returned on restart (preemption still default on the live boxes). Both boxes clean afterwards, no config drift. Master delta validated against vyos001's real running config on the sim router before installing. Installed on both: 6 no-preempt, 6 unicast pairs, DHCP HA primary/secondary respectively. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH	2026-08-16 22:48:39 +01:00
Michal	f81c94af43	feat(migration): dual WAN, cloned MAC, and the new 10.8.0.0/23 Private VLAN ... Two corrections from reading the live USG instead of trusting UniFi's fields, which report wan_type=dhcp for both WANs and are simply wrong: - There are TWO WANs, not one. WAN2 is the 10 gig ISP on VLAN 53, plain DHCP with a PUBLIC address (87.192.101.48/21, gw 87.192.96.1) on the USG's eth2 -- and it is what actually carries traffic. WAN1 is Vodafone PPPoE on VLAN 51, the failover. The delta had PPPoE as the only WAN, which would have left the primary line unconfigured. - The DHCP lease is bound to MAC, so bond0.53 now clones the USG's WAN2 MAC (f0:9f:c2:12:9b:4f). That is how VyOS keeps the existing public lease rather than negotiating a new one -- or getting none, if the ISP allows one per line. Distances: 10 gig at 1, Vodafone at 10. Only ONE box may hold the cloned MAC, so --with-wan gates the entire WAN, NAT and firewall section. vyos001 gets it (320 set lines); vyos002 gets none (234, zero WAN/NAT/firewall) and routes the LAN only. Pretending both could hold it would have meant a duplicate MAC on VLAN 53 and a flapping switch table. Private was rebuilt at 10.8.0.0/23 (VLAN 9) after the old 10.0.8.0/23 was deleted. bond0.9 and the VRRP group were moved to 10.8.0.252/.253 with VIP 10.8.0.254 on both boxes, and the delta now targets 10.8.0.1. Creating that network first required breaking a deadlock in UniFi: every LAN write was rejected with api.err.WanIpOverlapped / 0.0.0.0/0, because WAN1 was set to DHCP on a line that only speaks PPPoE, so it sat at 0.0.0.0 forever and the validator treated that as a subnet overlapping everything. Verified server-side, not a UI bug -- the API rejected it identically. Setting wan_type=pppoe let it dial (90.241.226.213, MTU 1492), which cleared the phantom overlap and incidentally PROVED the Vodafone credentials and line work, which had been listed as untestable before cutover. dhcp-options no-default-route-dns does not exist; the valid set is client-id, default-route-distance, host-name, mtu, no-default-route, reject, user-class, vendor-class-id. Caught by validating the delta against vyos001's real config on the labsim router before installing. After adding the network, the gateway's dhcpd.conf was checked with `dhcpd3 -t -cf` (valid) and confirmed to contain the new subnet only after a force-provision -- controller state is not device state. Both boxes: mode unifi, VRRP unchanged, unifi.boot re-captured (232 lines, carrying the new VLAN 9), no config drift. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH	2026-08-16 18:16:25 +01:00
Michal	febe4b72bc	chore(migration): all DNS through VyOS to Google, NAS out of the path ... The NAS is legacy for ad.itaz.eu and those records now live in Cloudflare, so the zone resolves publicly -- verified: nas001.ad.itaz.eu and kvm-macstudio1.ad.itaz.eu both answer from 8.8.8.8. That removes the reason for a conditional forward and lets the NAS leave the DNS path entirely. Two changes: - `service dns forwarding name-server` is now 8.8.8.8 and 8.8.4.4, the same pair the USG used on its WAN, instead of 10.0.0.194. - Every VLAN is handed the gateway as its resolver. UniFi set an explicit resolver on LoT only (the NAS); carrying that over would have kept the NAS in the path for one VLAN and not the other five, which is the sort of asymmetry nobody remembers a year later. The NAS is still referenced 9 times, all legitimate and checked: 4 NAT destination rules, the 4 matching firewall accepts for those port forwards, and its own DHCP reservation. No DNS references remain. Validated by loading vyos001's real running config on the labsim router and applying the full delta -- all 318 commands accepted, no errors. Installed on both boxes and verified in place: priority 200/100, upstream 8.8.8.8 + 8.8.4.4, six client resolvers, 377 lines each. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH	2026-08-16 17:16:17 +01:00
Michal	3768657b91	chore(migration): firewalls resolve via 8.8.8.8/8.8.4.4 ... Matches the DNS the USG used on its WAN (wan_dns1/wan_dns2), replacing the 10.0.0.194 I had set earlier. Applied to both boxes and saved; VRRP unchanged (MASTER/BACKUP), NTP still synced, no config drift. /config/modes/unifi.boot was RE-CAPTURED on both afterwards. It had been taken before this change, so the escape hatch would have quietly reverted the resolver on any rollback -- a snapshot is only an escape hatch for the state it was taken from. Two things recorded in the runbook: - The boxes' name resolution now depends on the internet, so between unplugging the USG and PPPoE establishing they have no DNS. Harmless: nothing in the switch resolves a name, and the health checks use DNS precisely to prove the WAN came up. - Internal ad.itaz.eu names still resolve via Google, because that zone is published publicly with private addresses in it (nas001 -> 10.0.0.194, kvm-macstudio1 -> 192.168.3.8). So no conditional forward was needed -- though it is worth knowing the internal topology is public. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH	2026-08-16 16:49:27 +01:00
Michal	2a8fcb3bd3	fix(migration): the runbook pointed at addresses that die with the USG ... The access table led with 192.168.8.143/.144 and offered the LoT addresses as a fallback ("if unreachable, try"). That is backwards and would have stranded the operator at the worst moment: the workstation sits on LoT, and reaching 192.168.8.x routes *through the USG*, so those addresses are guaranteed dead the instant it is unplugged. Measured: ip route get 192.168.8.143 -> via 10.0.0.1 (the USG) ip route get 10.0.1.252 -> dev lanbr0 (same L2, no gateway) 10.0.1.252 and .253 are on the LoT VLAN, same broadcast domain as the workstation, and both answer SSH. They are now the only addresses the runbook gives, with the k8s ones struck through. Also recorded: the switch cannot be run before unplugging the USG (two devices on every gateway address; the guard refuses), so the order is forced. And during the gap between unplugging and completing the switch there is no inter-VLAN routing at all -- which means the JetKVMs (Management and kvm) and Tailscale are NOT fallbacks in that window. LoT SSH is the only remote path; below it is physical console. Added a step 0: open both SSH sessions and leave them open before touching anything. Both boxes are now installed and pass the pre-flight gate: mode unifi, unifi.boot 231 lines including the reload action, delta at the right priority (200/100), wan-secrets 0600, script executable, no config drift, VRRP still MASTER/BACKUP. `vyos-unifi-switch vyos` refuses on both -- all six gateway addresses detected answering ARP -- and neither box has gained dhcp-server, dns or nat, so nothing about their behaviour has changed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH	2026-08-16 16:25:04 +01:00
Michal	fc31013ceb	fix(migration): apply the reviewed reservation plan, not a recomputed one ... This caused a real outage. unifi-reserve-all.py recomputed its plan at --apply time by re-reading stat/sta, so a client that renewed between the dry run and the apply was pinned to whatever transient address it happened to hold at that instant. worker1-k8s0 was reviewed at 192.168.8.13 and written as 192.168.8.242. On its next reboot it could not get an address at all, taking a k8s node down. A plan that gets reviewed and a plan that gets applied must be the same object. The dry run now WRITES the plan to a file and --apply READS it and applies exactly that, reporting any client whose current address has since drifted rather than silently preferring the new value. 1 of 51 diverged; the rest were verified against the reviewed list and were correct. worker1 has been restored to .13 and confirmed: DHCPOFFER for its own MAC returns 192.168.8.13, and the node is up with a full lease and working internet. The second half of the outage was drift between controller and device: the USG was still running config from ~16h before these changes, so the controller looked perfectly correct while the gateway handed out something else. Writing the controller is only half the job, so the script now says so explicitly and gives the force-provision and DHCP-probe commands to verify with. `nmap --script broadcast-dhcp-discover --script-args broadcast-dhcp-discover.mac=...` is the way to prove a specific reservation is live without disturbing the client -- it elicits an OFFER without ever sending a REQUEST. _unifi.py gains post() for device commands. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH	2026-08-16 14:31:30 +01:00
Michal	b37cd79432	fix(migration): refuse an unprobeable delta instead of warning past it ... The ARP guard against two devices holding the same gateway address is the one check that prevents this script's worst outcome. It reads the addresses to probe out of the delta -- so a delta with no VIP lines made the guard inert, and it previously warned and carried on. That was a testing convenience (the lab delta has no VIPs) weakening a production safety check, which is backwards. It now refuses by default. ALLOW_NO_VIP_DELTA=1 is the explicit lab override. The guard's probing path had never actually executed before this: every sim run took the no-VIPs branch. Verified against the live USG from vyos001: arping is present on VyOS, the regex extracts all six gateway addresses from the real delta (192.168.1.1, 192.168.8.1, 192.168.3.1, 10.0.9.0, 10.0.0.1, 192.168.2.1), and every one of them answers ARP right now -- so on the real boxes, with the USG connected, the guard fires. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH	2026-08-16 00:13:19 +01:00
Michal	7e464a2828	docs(migration): record what the rehearsal proved and what it could not ... Ran vyos001's real running config plus the real production delta on the labsim router -- same VyOS version, isolated OVS bridge with no physical NIC, so the sim briefly holding vyos001's actual addresses could not reach the real LAN. Result: all 317 commands accepted and the whole delta commits (COMMIT OK), the revert is byte-exact, and auto-revert fires without rebooting (uptime and boot-id unchanged across it). Also written down are the two things this did NOT establish, because a runbook that overstates its own coverage is worse than one that admits the gap: PPPoE cannot be tried while the USG holds the single available session, and the rehearsal ran with vyos001's eth2/eth3 stanzas stripped because the sim VM has two NICs rather than four. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH	2026-08-16 00:08:13 +01:00
Michal	01a923352f	fix(migration): do not emit a NAT translation port for a port list ... `set nat destination rule N translation port '16881,6881'` is rejected -- "16881,6881 is not a valid service name" -- because mapping a list of ports onto a list is ambiguous. `destination port` accepts the same list happily, which is why this only shows up on the translation side. All four UniFi port forwards map a port to itself, so translation port was redundant anyway: omitting it makes VyOS preserve the original port, which is exactly the intent. It is now emitted only when the forwarded port genuinely differs, and generation fails loudly rather than producing a config that will not commit if a differing port LIST ever appears. Found by loading vyos001's real running config onto the labsim router and applying the full delta to the candidate config without committing. Worth noting the delta had already passed a read-through: this one only surfaced by running it against a real VyOS of the same version. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH	2026-08-16 00:02:21 +01:00
Michal	7f5d3517a3	fix(migration): create the WAN vif before PPPoE references it ... `set interfaces pppoe pppoe0 source-interface bond0.51` refers to an interface that must already exist, and neither firewall has vif 51 -- only 2, 3, 9, 10 and 200 are configured. The commit would have failed, and since the whole delta commits as one unit, that failure would have taken the entire switch with it at the worst possible moment. No address on the vif: PPPoE rides the VLAN and needs no L3 of its own. Found by checking the running config against the generated delta rather than by running it. The prod delta has still never been applied to any VyOS, which is the remaining gap. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH	2026-08-15 23:51:13 +01:00
Michal	d56bbf6db0	feat(migration): reversible USG->VyOS switch, proven on the sim ... The cutover is a switch, not a migration: unplug the USG, run one command, and if anything is wrong run the other one and plug it back in. The operator will have no internet during this and therefore no assistant, so the machinery has to live on the boxes and the failure paths have to be proven in advance. vyos-mode-delta.py generates the delta that turns the passive pair into the gateway. Only one artifact is authored: gateway mode is always derived from `load unifi.boot` + delta, so there is no inverse to maintain and no drift between two hand-kept configs. It reuses unifi-to-vyos.py rather than duplicating it, so what labsim proved and what production gets are one code path. The PPPoE password is never written into the delta -- it carries a placeholder the switch substitutes at apply time from /config/wan-secrets -- and generation fails if the real password appears in the output. Two things the delta covers that the plan had underweighted: - VyOS defaults to ACCEPT while the USG has an implicit WAN drop. Migrating the port forwards alone would have left the router's own services and the whole LAN reachable from the WAN. Added a stateful baseline scoped to the WAN interface rather than a global default-action drop, so a mistake there cannot lock anyone out over the LAN -- the only way back during a cutover. - The old VIPs are NOT at network+254 on the /23 networks; they are 192.168.9.254, 10.0.9.254 and 10.0.1.254, in the upper half. A delete naming a computed address fails quietly and leaves the group holding two VIPs. The delta deletes the whole address node instead of guessing. vyos-unifi-switch runs on the box from /config, which survives image upgrades, so it works from a local terminal or the JetKVM with no workstation. Proven on labsim, not assumed: - unifi mode restores the previous config BYTE-EXACT (138 lines, diff clean). - Auto-revert fires when the commit is not confirmed: 85 static-mappings -> 0, kea stopped, hostname restored, and uptime plus boot-id UNCHANGED, so it reloaded rather than rebooted. That distinction is the whole reason `commit-confirm action reload` is a prerequisite. - Health-check failure triggers an immediate revert_soft rather than waiting out the timer. Four bugs found while doing it, each of which produced a wrong answer rather than an error: - commit-confirm is TWO steps. `config-mgmt commit_confirm` only arms the revert timer; a normal `commit` still has to follow. Arming alone committed nothing while reporting success. - `sudo sg vyattacfg "config-mgmt ..."` loses the config-session environment, so it reported "No configuration changes to commit" against a candidate that plainly had 446 added lines. - `... | grep -q` under `set -o pipefail` reports FAILURE on a match: grep exits early, the producer takes SIGPIPE. Whether it triggers depends on output size, so `status` misreported the mode intermittently. - `show configuration commands` quotes values, so a fixed-string match for `action reload` never matched `action 'reload'`. CUTOVER.md is the printable runbook: both reachable addresses per box, the escape hatch first, and the note that PPPoE is the one thing that could not be tested beforehand. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH	2026-08-15 23:30:28 +01:00
Michal	6c4318d3ae	feat(migration): reserve every active client at its current address ... kea does not inherit UniFi's lease database. At cutover it starts with an empty view of who holds what, so it can hand an address that is currently in use to a different device. Reservations are what carry "this device has this address" across the switch, because they live in config rather than in lease state. unifi-reserve-all.py creates one per active client, dry run by default. 51 written, 51/51 verified live by reading the records back; the controller now holds 85 reservations and the generator emits all 85 with unique, valid hostnames and no duplicate addresses. Active clients with no reservation went from 48 to 4. Three guards, each of which caught something real in the dry run: - VRRP virtual addresses are excluded. UniFi reports them as ordinary client addresses because the firewalls' bond MACs answer for them, and their apparent IP flips between the real interface address and the VIP. Without this, 192.168.1.254 -- the gateway VIP itself -- would have been given a DHCP reservation. - The firewalls' own interface MACs are excluded; those are statically configured routers, not DHCP clients. - Any address claimed by more than one MAC is dropped rather than guessed at. This is how the VIPs surfaced in the first place. Also skipped: addresses already reserved to another MAC, network gateways, and anything on a network that does not serve DHCP (which excludes the WAN transit VLANs automatically). labsim-dhcp-test.sh gained a lease-database flush, and it is not tidiness. Two findings, both of which first appeared as a PASSING test: - Re-running against stale leases, kea gave dynamic addresses to three devices that have reservations. The reservations were present and correct in kea's own config throughout. Kea saw the reserved address as leased to "another client" -- same MAC, different client-id from the earlier boot -- and allocated elsewhere. Cutover starts with an empty lease database so this is a testing artifact, but a reservation is evidently not unconditional once leases exist. - Removing only dhcp4-leases.csv does nothing: kea's memfile backend keeps lease-file-cleanup rotations (.csv.2) and restores from them on start. The verdict logic no longer takes the first matching lease row. Doing so reported an hours-old lease as the current answer and scored three failures as passes, including one where the device had plainly been given a dynamic address. A MAC with more than one lease is now an explicit failure rather than a guess. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH	2026-08-15 23:07:07 +01:00
Michal	f36ff4c6e3	feat(migration): pin firewall management NICs and reserve them in UniFi ... Prerequisite for the cutover. eth2 on both firewalls was DHCP-served by the USG, and both addresses (192.168.8.143/.144) sit inside the pool VyOS will serve -- with no reservation for either MAC. After the switch they would renew from kea, get no mapping, and the management addresses could move. That is the worst possible moment for the address you SSH to to change, because losing the USG also means losing internet and any outside help. On both boxes, driven over the LoT path (bond0.10) so the interface being changed was never the one carrying the session: - eth2 pinned static at its current address, so it no longer depends on DHCP - `system name-server eth2` replaced with 10.0.0.194. That setting inherited resolvers from the DHCP lease, i.e. the boxes were resolving via the USG and would have lost DNS with it. 10.0.0.194 is reachable directly over bond0.10 and is authoritative for ad.itaz.eu, so internal names now resolve on the firewalls -- they did not before. - static default route via 192.168.8.1, replacing the one the lease provided. Superseded by PPPoE in vyos mode; this keeps unifi mode as it was. Verified after each: SSH on the pinned address, external and internal DNS, NTP still synced, VRRP unchanged (vyos001 MASTER, vyos002 BACKUP). migration/unifi-reserve.py adds the matching UniFi reservations so the controller cannot lease those addresses to anything else, keeping the management address identical in both modes. It reads the record back after writing, because a controller accepting a PUT is not proof it stored what was asked for, and it is idempotent. Also noted while doing this: VyOS `commit-confirm` REBOOTS the box if not confirmed -- "Minutes until reboot, unless 'confirm'" -- it does not roll the config back in place. For a gateway that means a real outage window, which changes how the switch script must use it. `config-mgmt commit_confirm -y` executes without the interactive prompt. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH	2026-08-15 22:12:42 +01:00
Michal	44dbd5188c	feat(migration): export UniFi config and generate VyOS DHCP+DNS from it ... Groundwork for replacing the USG with the VyOS pair without anything on the network noticing. Three pieces: migration/unifi-export.py pulls 13 endpoints off the classic controller into timestamped JSON plus a normalised inventory: 11 networks, 31 DHCP reservations, 4 port forwards, 2 firewall rules, 0 static routes. Two things this turned up that a naive export would have lost: - 23 of the 31 reservations carry no network_id at all -- UniFi simply does not store the binding -- so they are resolved by subnet containment instead. Without that, three quarters of the reservations have no subnet to be placed in. - 30 of the 31 sit INSIDE the DHCP pool, which UniFi's dhcpd tolerates and which is flagged as a warning rather than discovered at cutover. migration/unifi-to-vyos.py turns that inventory into VyOS `set` commands for DHCP and DNS only -- the services the USG owns that VyOS must reproduce. Not a general converter. --prod and --sim come from one code path so the config proven in the sim and the config applied to the firewalls cannot drift. Prod mode hard-fails if any reservation is missing, since a silent drop is the failure mode that matters. DNS is included because the USG resolves for 5 of 6 VLANs today: UniFi hands out the gateway's own address whenever dhcpd_dns is empty, verified by labmaster resolving against 192.168.8.1. Replacing the USG without a forwarder would take DNS away from those VLANs entirely. labsim/labsim-dhcp-test.sh proves it by booting throwaway VMs with real production MACs -- the one piece of production config that transplants verbatim. Safe because ovs-labsim has no physical NIC, so those MACs cannot reach the real LAN. Result on VyOS 2026.08 (kea), 4/4: printer1 got 172.31.10.46 from inside the pool, sonoff-matter got 172.31.11.67 across the /23 boundary, Hubitat got its out-of-pool .2, and an unreserved MAC got an unreserved address. kea honours in-pool host reservations -- the open question blocking the cutover. Supporting changes to labsim: - VLAN 10 widened to /23. Every reservation is in LoT and LoT spans 10.0.0.x and 10.0.1.x, which a /24 cannot represent. - LoT's host leg moved to .3, because 10.0.0.2 is a real reservation (Hubitat) that maps onto the host's own address. - vlans.conf gained optional masklen and host_octet fields, defaulting to 24 and 2 so the other five VLANs are untouched. - Fixed /etc/network/interfaces hardcoding 255.255.255.0. That file is what actually takes effect on these Alpine guests -- cloud-init's network-config is ignored -- so any non-/24 VLAN was silently wrong. Two generator bugs found by VyOS rejecting the output: static-mapping names are validated as hostnames, so underscores fail; and two devices named "espressif" plus two named "thebeast" collided into single names, which would have overwritten one reservation with another's address. The raw export holds WiFi passphrases and the WAN PPPoE credentials and is gitignored. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH	2026-08-15 01:33:00 +01:00
Michal	b0b68f2edd	Merge feat/vyos-unattended-install: VyOS HA install + DiskPressure incident fixes ... Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017f6jyeeDqP4ufyeL3UER9w	2026-08-14 23:22:21 +01:00
Michal	72c54edce2	feat(k3s): enable swap and grow the rancher LV during host-prep ... Replace the CIS-style disableSwap op with enableSwap: activate the labvg-swap LV with an fstab entry (kubelet runs failSwapOn=false; zram stays the fast tier, the LV is overflow before OOM kill). Add growRancherLv: extend labvg/rancher to 120G when the VG has free space, covering nodes installed before the kickstart sizing change and vanilla nodes converted to k8s; skips with a clear message when the VG is full. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017f6jyeeDqP4ufyeL3UER9w	2026-08-14 23:22:14 +01:00
Michal	33be713d0c	feat(bastion): size the rancher LV at 120G for k8s roles in kickstart ... The 20G /var/lib/rancher LV (k3s imageFs) idled at 85% used from steady-state images alone; one ~5G image pull tripped imagefs eviction and evicted unrelated pods (2026-08-14 DiskPressure incident). Create the LV for both worker and infra roles at 120G — it must be sized here because longhorn's --grow consumes all remaining VG space, making post-install lvextend impossible. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017f6jyeeDqP4ufyeL3UER9w	2026-08-14 23:22:14 +01:00
Michal	a5b36678ed	feat(labsim): live topology view with per-path latency ... The Grafana heatmap of 1s and 0s said almost nothing, and the state timeline was an unreadable pile of overlapping series labels. Replaced as the primary view with a purpose-built page served by the exporter itself. - Probe now captures ICMP RTT, exposed as labsim_rtt_ms{src,dst}. A path that is up but slow is a different problem from one that is down, and a pass/fail grid cannot show it. - Exporter serves / (topology), /api/matrix (JSON) and /metrics. - topology.html: node per VLAN in a ring, VyOS router in the centre because every inter-VLAN packet really does traverse it, one line per pair coloured green/red with the RTT on it. Hovering gives per-direction state. A node ring goes red if anything to or from it is blocked. Side panels list blocked paths and the slowest links. Refreshes every 5s, no dependencies. Grafana stays for what it is actually good at — history of when a path flipped. Label placement is deliberate: RTT captions sit ~32% along each edge with a perpendicular nudge, because every diagonal of a 6-node mesh crosses the centre and midpoint labels stack on the router node. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH	2026-08-13 00:56:06 +01:00
Michal	c91e44f796	feat(labsim): libvirt replica of the lab network with LACP + VyOS routing ... A throwaway copy of the production VLAN topology so routing and firewall changes can be tested before they touch the real network. Same VLAN IDs and roles as UniFi, deliberately different ranges (172.31.<vlan>.0/24) so nothing here can be mistaken for production. - OVS fabric: real 802.1Q. Access port per micro VM, host leg per VLAN (.2, for SSH only — NOT the VMs' default route, so inter-VLAN tests exercise the router rather than the host's routing table), and a trunk portgroup with VLAN 1 declared nativeMode='untagged'. - Six Alpine micro VMs (256MB, copy-on-write overlays on one 176MB image), SSH + a hello-world HTTP page naming the VLAN. - VyOS router installed to disk unattended over the console, with the SAME config shape as the VP2440s: two NICs in an LACP bond carrying the trunk, VLAN 1 native, bond0.<vlan> holding the .1 gateway on each. - labsim-matrix.py: full-mesh ICMP/TCP22/TCP80 probe, ~0.2s, --watch highlights cells that changed since the last sweep. Guest-side probe is python3 (already present via cloud-init) so nothing is installed on VMs that have no internet. - Prometheus + Grafana (anonymous auth, no login) with a provisioned dashboard: heatmap plus a state timeline showing exactly when a path flipped. Verified end to end: one VyOS rule took sum(labsim_reachable) from 90 to 84, blocking precisely kvm<->k8s across all three protocols. Traps found building this, all now encoded in the scripts: - virtio-net breaks 802.3ad: the guest's bonding driver reports slaves "MII Status: down" despite carrier=1 and never sends an LACPDU, so the bond sits in AD_STATE_DEFAULTED. e1000e fixes it with no other change. Matches the netdev thread "bonding (IEEE 802.3ad) not working with qemu/virtio". - OVS defaults bonds to active-backup, which does not speak LACP at all — bond_mode=balance-tcp is required. - LACP deadlock: OVS holds members disabled until negotiation while the partner needs carrier before it will send LACPDUs. lacp-fallback-ab breaks it. - LACPDUs are untagged, so a trunk with no native VLAN has nowhere to put them. - --boot cdrom,hd re-runs the ISO on every restart, so every commit+save went to a live system that evaporated. Install now switches the VM to boot hd. - cloud-init on Alpine: users stay locked without lock_passwd:false, one failing runcmd aborts the rest, busybox here has no httpd applet, and start-stop-daemon --exec /usr/bin/python3 matches cloud-init's own python3. - The user-data heredoc is unquoted, so backticks in a COMMENT were executed by the host shell and their output corrupted the YAML. build_seed now validates with yaml.safe_load before building the ISO. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH	2026-08-13 00:42:39 +01:00
Michal	df2dfc5d71	fix(bastion): pin the VyOS boot NIC by MAC, and detect pre-installer stalls ... Both Protectli VP2440s failed to install on real hardware: they fetched kernel+initrd and then went silent. The console showed why — Looking for a connected Ethernet interface ... e2 ? e3 ? e4 ? e5 ? Connected e4 found Connected e5 found [4.595647] igc 0000:02:00.0 e2: NIC Link is Up IP-Config: e4 ... no response after 15 secs - giving up Unable to find a live file system on the network live-boot picks the first *connected* interface. The i40e SFP+ pair links before the igc copper port (up at 4.6s), so it chose the fiber ports, which have no DHCP, and never tried the NIC that actually PXE booted. Fix: pass BOOTIF=01-<mac> on the kernel cmdline. live-boot's Device_from_bootif() (verified present in this image) matches it against /sys/class/net and sets DEVICE directly. The MAC comes from the dispatch key — i.e. exactly the NIC that PXE booted — which is more reliable than iPXE's ${net0} on a box where the booting NIC may not be net0. Why the integration test missed it: the VM had ONE NIC, so "first connected interface" was trivially correct, and virtio links instantly so there was no negotiation race. createPxeVm now takes decoyNics, attaching extra NICs ahead of the PXE NIC on a network with no route to the bastion; the VyOS test uses 2. Without BOOTIF that reproduces the hardware failure. getVmMac is network-aware so it still returns the booting NIC. Also: the bastion had every clue and said nothing — it logged INSTALL STARTED, served kernel+initrd, then nothing for 7 minutes. dispatch now stamps dispatched_at, and /api/logs/:mac returns stalled_for_s / stalled (8 min threshold, sized for the ~600MB squashfs fetch), so a machine wedged before the installer environment comes up is diagnosable without a console. Verified on hardware: both firewalls installed, bond0 802.3ad + VLANs 2/3/9/10/200 + VRRP (priority 200/100, VIP .254 per VLAN) applied, and /config/lab-provisioned written. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH	2026-08-12 12:35:09 +01:00
Michal	e36a7a193c	chore(cli): regenerate shell completions for VyOS flags ... pnpm completions:check was failing: labctl.fish/bash were stale. The generated --os choices still listed only fedora-43 and ubuntu-26.04 (missing vyos-rolling since the OsId union gained it), and none of the --vyos-*/--vlan flags were present. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH	2026-08-11 11:25:20 +01:00
Michal	5d00c42f5a	feat(bastion): bring VyOS provisioning to Fedora-grade quality ... Ports the Fedora provisioning features that matter for a router onto the VyOS path, and adds the libvirt integration test that proves them. - Live install logs: the driver streams the installer pty (ANSI-stripped, batched, best-effort) to POST /api/log, so `labctl provision logs -f` works during a VyOS install the way Anaconda's syslog does for Fedora. - installed.ip: report "ready at <ip>" -- the exact detail format routes/api.ts parses -- using the static mgmt address when known, else the live DHCP address. Without it VyOS machines landed with an empty IP, breaking provision list, logs-by-IP, recheck and reprovision. api.ts also guards the complete handler: VyOS boxes get the "vyos" SSH hint and never trigger the k3s post-provision. - EFI network-first boot order: port of the Fedora %post efibootmgr step, run from the live env after install (NVRAM, not disk). Best-effort. - Reinstall semantics: VyOS's installer already carries the previous config and SSH host keys forward -- the analog of Fedora's LV preservation -- so that stays the default. New --vyos-fresh-config overwrites the installed config.boot with the generated one instead, via a post-install target mount that also writes /config/lab-provisioned (mirrors Fedora's /etc/lab-provisioned, survives image upgrades). - reprovision/recheck default to the "vyos" SSH user for VyOS machines. Two hangs found by the VM test and fixed: - On reinstall the installer asks "Would you like to copy data to the new image?" (search_previous_installation). Unanswered, the driver blocked on stdin until its stall timeout -- a silent 15-minute hang. - The RAID regex missed "Would you like to choose two disks for RAID-1 mirroring?", which would wedge any multi-disk box. Both prompts default to yes, so a miss also risks an unwanted mirror. Both are now covered by a unit test asserting all 17 installer prompts match exactly one rule -- verified to fail against the unfixed code, so this class of bug is caught in a second instead of a 45-minute VM run. tests/integration/vyos-provision.test.ts: fresh install, reinstall preserves config + /config data, and freshConfig override. All 8 pass against the real nightly ISO (EXIT=0). 273 unit tests pass; no new lint errors in touched files. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH	2026-08-11 11:16:25 +01:00
Michal	cb9d99dd69	feat(bastion): unattended VyOS network install with HA (bond + VRRP) ... VyOS ships no unattended installer (install_image() is unconditionally interactive; --no-prompt is wired only to 'add'), so the automation is injected through live-config's hooks component: iPXE boots the live kernel with fetch= and live-config.hooks=, the hook fetches a generated per-MAC Python driver, and the driver builds config.boot, stages the rootfs, and drives the interactive installer over a pty. Bastion: - vyos-boot.ipxe template (no 'nonetworking' — breaks the hook fetch; no console=ttyS0 — 30s/systemd-phase on UART-less boards) - /vyos/autoinstall.sh + /vyos/install.py routes (per-MAC driver with the config spec baked in as base64) - vyos-config-spec: bond0 (802.3ad) + tagged VLANs + VRRP groups (vrid = VLAN id) + sync group + hw-id pinning by MAC + SSH keys; config built from the image's own config.boot.default via vyos.configtree, version footer reattached via component_version - prepareVyosArtifacts: extract kernel/initrd/squashfs from the nightly ISO with xorriso; initrd picked by size from regular files only; ISO URL "latest" resolves the newest vyos-nightly-build GH release (downloads.vyos.io no longer serves direct ISOs) Verified end-to-end in a libvirt VM against the real nightly ISO — installed system boots with bond/VRRP/hw-id config applied and no migrations. Fixes found by the VM run, encoded in code comments: config.boot.default lives at /usr/share/vyos at hook time; fetch= boot has no medium so the rootfs is symlinked to the installer's expected path; reboot must be --force (the hook is a child of the still-starting live-config unit); installer disk answers are full /dev paths; zram0 passes the 2GB min-disk filter so the disk is always pinned. CLI/labd: vyos spec threaded through provision install (--vyos-* and --vlan/--vlan-vip flags with guards), labd install route, protocol command-install, and the bastion's direct /api/install. 268 unit tests pass; no new lint errors in touched files. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH	2026-08-10 21:45:22 +01:00
Michal	4f9a6f64e4	k3s/networking: codify Multus + vlan-setup as lab operations ... The macvlan/VLAN-10 foundation HA depends on (Multus meta-CNI + a lan10 sub-interface + reference CNI plugins) was applied by hand during the HA migration. Codify both as idempotent lab operations in the networking group, after installCilium (which already sets cni.exclusive=false + bpf.vlanBypass={10}). A fresh cluster now reproduces the full macvlan stack. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-07 15:37:20 +01:00
Michal	12fa954a05	k3s/cilium: install with cni.exclusive=false + bpf.vlanBypass={10} (Multus + VLAN-10 macvlan mDNS) ... Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-06 21:06:24 +01:00
michal	7181a61cec	Merge pull request 'fix(labd): wire v2.0 Phase 1 routes + smoke tests' (#15) from fix/v2-wire-and-smoke-test into main	2026-05-05 21:18:42 +00:00
Michal	cdf3b5c045	fix(labd): wire v2.0 Phase 1 routes into createApp + smoke tests ... The v2.0 Phase 1 commit (04faa07) added AuthService, RbacService, ResourceStore, AuditService, the bearer auth middleware, and the v2-auth/environments/resources route files, but createApp() never registered any of them. They sat in the codebase as dead code: a running labd would 404 on /api/auth/login, /api/resources, /api/events, etc. Wiring (server.ts) - Instantiate AuthService, RbacService, ResourceStore, AuditService at app creation. Cast DbClient to PrismaClient (the runtime db is a real PrismaClient; DbClient is a structural shim). - Start AuditService timer, register an onClose hook to stop it on shutdown so we never lose the last batch. - Register v2 routes inside a Fastify scope with the bearer-auth middleware as preHandler. v1 routes (registered on the root scope) are unaffected so existing labd clients keep working. AuditService (audit.ts) - Expose flushPending() so tests can deterministically observe events without leaning on the 5-second flush interval. Implementation delegates to the existing private flush(). Smoke tests (v2-smoke.test.ts, 11 cases) - Bootstrap: first POST /api/auth/login with empty users creates the admin (role=ADMIN, hashed password), returns a 64-hex token, marks isBootstrap=true, emits an auth_bootstrap audit event. Second login uses the normal flow. Wrong password returns 401 and audits failure. Missing credentials returns 400. - RBAC: missing/empty/invalid bearer tokens return 401. ADMIN role bypasses RBAC. A non-admin with no role bindings gets 403 with "no matching role binding". A user with an env-A binding is denied for env-B resources. - Audit: bootstrap event is queryable via /api/events?correlation=... Explicit parent/child chain (shared correlationId, parentEventId) is preserved across emits. All 246 workspace tests pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-05 22:18:18 +01:00
michal	f3c50f71ef	Merge pull request 'feat: v2.0 Phase 1 foundation + bastion-restart identity fix + Dockerfile + BASTION_DIR' (#14) from feat/v2-phase1-foundation into main	2026-05-05 21:10:25 +00:00
Michal	98b0ccc6c9	feat(cli): honor BASTION_DIR env var as default for --dir ... bastion serve/stop default for --dir was hardcoded to /tmp/lab-bastion. Now reads BASTION_DIR from env if set, so a deployed bastion daemon can run from a persistent directory without callers having to pass --dir on every invocation. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-05 22:09:24 +01:00