e8679f45b50c13bbd9a0f90647f931b549092524
12 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Michal
|
e8679f45b5 |
labsim: rehearse the IPAM switch on 3 nodes, and catch the trap in it
Rehearsed kubernetes -> cluster-pool on the 3-node labsim cluster, which is the transition production faces. The switch itself is undramatic: agents stayed up, the operator adopted the pool, and the agent-not-ready taint deadlock did NOT occur. That deadlock is specific to ADDING IPv6 -- the agent blocks on an IPv6 pod CIDR that does not exist yet. A v4-only mode switch does not hit it. The real hazard is quieter. The operator does not preserve which node held which /24: two nodes swapped CIDRs. Their existing pods kept their old addresses, which now fall outside the node's range, so every other node routes that prefix to the wrong node. Cross-node ping to those pods dropped 100% while every pod stayed Running and every node stayed Ready. Nothing in `kubectl get pods` shows it. So "pods kept the same address" is the FAILURE signal here, not the reassurance it looks like. cilium-ipam-switch.sh verify now flags pods sitting outside their node's CIDR, which is the check that decides whether a recycle is optional (it is not) or mandatory (it is). Recycling every deploy/ds/sts restored it: all pods back inside their node CIDR, cross-node ping 0% loss. Sequence proven end to end: preflight -> apply -> restart operator then agents -> unstick if needed -> recycle all workloads -> verify Also fixed the recycle hint the script printed: `kubectl rollout restart deploy,ds,sts -A` is not valid (`unknown shorthand flag: 'A'`), so anyone following it under pressure would have got an error instead of a recycle. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH |
||
|
Michal
|
842408c0d9 |
labsim: prove k3s CAN be converted to dual-stack in place
k3s documents that dual-stack "cannot be enabled on an existing cluster". Rather
than accept that for a 145-day-old production cluster, build both shapes and
diff them. dualstack-lab.sh builds a single-node IPv4 cluster and a native
dual-stack one, takes a reflink copy of the IPv4 disk so a failed conversion
costs 90 seconds to undo, converts in place, and diffs the results.
Result: the conversion works. etcd data is never touched.
The documented blocker is real but narrower than stated. Cilium reports it
exactly -- `required IPv6 PodCIDR not available` -- because node.spec.podCIDRs
is assigned at join and is immutable, and the Kubernetes IPAM controller will
not add a second family later. That objection only holds while Cilium runs
ipam=kubernetes and therefore reads that field. Switching to cluster-pool IPAM
moves pod CIDR allocation into the CiliumNode CRD, where the operator hands out
both families on a cluster that was born IPv4-only.
Sequence that works, in order:
1. k3s unit gains --cluster-cidr/--service-cidr/--node-ip with both families.
k3s validates the two CIDRs together and refuses to start on a mismatch
("must share the same IP version"), so a partial edit crash-loops rather
than coming up half-configured. That is the safe failure mode.
2. The ServiceCIDR object picks up the IPv6 range on restart -- this is
upstream's supported "single-to-dual-stack preserving the primary
ServiceCIDR" path, and existing Services keep their IPv4 addresses.
3. Cilium to ipam=cluster-pool with an IPv6 pool, then DELETE the CiliumNode
so the operator reallocates; it will not add a family to an existing one.
4. Expect a deadlock here: the agent will not go ready without a pod CIDR, so
the node keeps the node.cilium.io/agent-not-ready taint, so the new
operator that would assign the CIDR cannot schedule. Remove the taint by
hand once to break it.
Verified on the converted cluster: pod with 10.42.0.125 AND fd00:42::4843, and
a PreferDualStack Service holding 10.43.122.115 AND fd00:43::72a3.
The only field that still differs from a native build is node.spec.podCIDRs,
which stays IPv4-only -- immutable, and unused once Cilium owns IPAM. CiliumNode
podCIDRs and ServiceCIDR are identical to the native cluster.
Not yet answered: this is one node. Whether a 3-server etcd cluster converts as
cleanly, and what a rejoining agent does, is the next experiment.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
|
||
|
Michal
|
ad6eb7a9a6 |
labsim: default-deny firewall policy, proven in the sim
Some checks failed
Policy: internal VLANs reach each other and the internet; the internet initiates nothing inward. That was already the effect of the IPv4 ruleset, but built as a blacklist -- default-action accept plus explicit drops per WAN interface. Identical behaviour right up until a WAN is added, at which point it is open and nothing looks wrong. This expresses it as a whitelist. Two findings from the sim, both of which would have been outages in production: `set` on a rule number is ADDITIVE. The sim already had a rule 10 carrying inbound/outbound interface constraints; `set ... rule 10 state established` ANDed onto it, producing a stateful-accept that applied to one interface pair only. Return traffic from the internet then matched no rule and hit the default drop, so LAN hosts could reach nothing outbound. The generator now deletes each filter before rebuilding it, so the code owns the subtree. It is one commit, so nftables is rebuilt atomically -- there is no window without a firewall. DHCP lease renewal is unicast UDP to port 68 and conntrack does not reliably cover it. Without an explicit rule the WAN keeps working until the lease expires and then dies -- a delayed failure that looks nothing like a firewall change. Also added a loopback accept for both families, absent from the v6 policy since it went default-deny. Verified in labsim: inter-VLAN ok, LAN-to-internet ok, internet-to-router dropped, and internet-to-LAN dropped with the drop counter incrementing by exactly the packets sent, after routing the test through the router rather than around it via the hypervisor. Also extends the drift check to the firewall subtree, which it did not cover -- so it had been reporting "in sync" while that subtree was uncaptured. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH |
||
|
Michal
|
7f551081ad |
labsim: capture BGP, dual WAN and both ISP VMs as code
The sim's routing config existed only as running state on the VMs. It was applied by hand over SSH, so rebuilding a VM lost the rehearsal and nothing recorded why any of it was shaped the way it was. The two ISP VMs were not referenced anywhere in the repo at all. sim-net-config.py generates all four roles; sim-net-apply.sh applies them over the serial console, or diffs them against the running VMs. Verified reproducing live state exactly before committing: primary 40/40 commands, secondary 16/16, isp-dhcp 19/19, isp-pppoe 21/21. Carries the reasoning that was previously nowhere: RFC 8212 needing policy in both directions or the session carries zero prefixes; probe targets that must not double as system name-servers; default-route-distance 210 rather than no-default-route, which blanks new_routers and hands the default route to the backup line; and the WI-8 bootstrap bug that pinned /32s fix. Dropped a stale `pppoe-server interface eth0` on isp-pppoe (a NIC that does not exist there) so a green drift check stays meaningful. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH |
||
|
Michal
|
86c2a36f00 |
feat(labsim): a real Kubernetes cluster for rehearsing Cilium <-> VyOS BGP
Some checks failed
BGP is about to be added to a network that currently works, where a bad advertisement blackholes the house. That needs somewhere to fail first. Three Debian nodes (4 GB / 2 vCPU) on the OVS `vlan2` access ports running k3s with flannel disabled, so Cilium is the CNI under test. Three rather than two because ECMP is only meaningfully tested if a node can be drained and more than one path survives. VMs rather than k3d: the thing under test is eBGP between Cilium and VyOS across the switch fabric, nodes peering with the router's bond0.2 leg, directly connected. k3d would put the nodes on a container bridge -- a different L2 path, proving something else. (It also wants Docker; this host has podman.) The existing micro VMs are Alpine with 256 MB and 1 vCPU, which is not close to enough. Debian rather than the sim's Alpine base: glibc, a stock kernel that Cilium's eBPF probes are tested against, and cloud-init that actually applies network-config -- the Alpine base notably does not. One trap worth recording. An earlier draft called `selected_vlans "$K8S_VLAN"` before `ovs_up`, and since `ovs_up` re-defines the libvirt network from SELECTED, that silently deleted the portgroups for every other VLAN. Running VMs kept working -- their taps were already attached -- so nothing complained until the ISP VMs needed vlan51 and vlan53 and could not be attached. It now selects every VLAN. The generated kubeconfig is gitignored: it carries cluster-admin credentials and is one `git add -A` away from being committed. Regenerate with `k8s-up.sh --kubeconfig`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH |
||
|
Michal
|
ff86a421f4 |
fix(labsim): console-apply must handle both VyOS prompts, not just $
Two failures from one strict expect, both hit while building the sim ISPs. A run that dies mid-config leaves the console parked in configuration mode. The next run then waits for the operational `$ ` prompt against a perfectly healthy VM and hangs until timeout, with nothing in the output to say why -- the box was sitting at `vyos@isp-dhcp#` the whole time. Login now accepts `# ` as well and discards the stale candidate rather than committing something nobody has seen. The same mistake at the exit step: insisting on `$ ` after `save` hung, AND left the console in config mode, which is what created the first failure for the following run. Now accepts either prompt. Known-bad, not fixed: the tool reports "committed and saved" when the set commands have not applied. Verified against the clone -- prompt showed the host-name change had landed while `grep -c dhcp-server` returned 0. The failure detection only inspects c.before for a few strings and evidently misses the real failure mode, so success is being reported without evidence. That needs fixing before this tool is trusted for anything; it is currently only safe to use with an independent check afterwards, which is how the gap was found. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH |
||
|
Michal
|
ee070371a8 |
feat(labsim): add WAN transport VLANs so the sim can host fake ISPs
A cutover attempt failed on the WAN and nothing had tested it. The reason the sim could not have caught it: labsim modelled every LAN VLAN faithfully and omitted the WAN entirely -- vlans.conf had 1, 2, 3, 9, 10 and 200, never 51 or 53. Worse, the switch script's WAN health checks are conditional on the delta configuring PPPoE, so in the sim they printed "this delta configures no WAN -- skipping all WAN health checks" and passed. The sim proved the delta commits; it never proved the WAN works, and could not have. Adds VLANs 51 (Vodafone/PPPoE) and 53 (10gig/DHCP) to the fabric so a fake ISP can live on each and those checks actually execute. VyOS has service pppoe-server (accel-ppp) natively -- authentication local-users, client-ip-pool, gateway-address -- so a VyOS VM can play the concentrator, and dhcp-server can play the other ISP. vlans.conf gains host_octet 0, meaning "no host leg". A host address on a WAN transport VLAN would misrepresent the segment: the point is that VyOS reaches an ISP, not the host. Also fixes a real gap in ovs_bond_router: it returned early when the bond already existed, so adding a VLAN to vlans.conf never reached an existing bond. Re-runs now reconcile the trunk and say so. That gap is the same SHAPE as the production failure -- interface present, VLAN missing from the trunk, frames silently dropped -- which is precisely the class of bug the sim needs to be able to reproduce rather than embody. Both bonds updated: [2,3,9,10,200] -> [2,3,9,10,51,53,200]. Note on the production diagnosis, which is NOT settled: a passive RX test showed zero frames on 51/53 at the firewall, and an active DHCP DISCOVER (verified to have transmitted, tx +2) drew no reply. That is consistent with the VLANs not being trunked, but equally with the ISP only answering its registered CPE MAC -- which is exactly why the delta clones f0:9f:c2:12:9b:4f, and why it cannot be settled from production while the USG holds that MAC. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH |
||
|
Michal
|
bb654d83f8 |
test(labsim): second VyOS router proves DHCP active-passive HA
Answers the question a single router could not, and that would otherwise only
have been discovered at cutover: with kea high-availability active-passive, does
exactly ONE box answer a DHCP request?
Yes. Probing sim VLAN 10 with broadcast-dhcp-discover returns offers from a
single distinct Server Identifier -- 172.31.10.252, the primary. The secondary
runs kea but stays silent. Without this the delta would have put 6 subnets and
84 static-mappings on both boxes with nothing to arbitrate them, and two kea
instances would have raced on every broadcast domain.
Worth noting the raw response count is misleading: nmap reports "Response 1 of
2" because it sends several discovers, and both replies carry the same server
identifier. Counting responses says "2 servers"; counting distinct server
identifiers says "1". The second number is the true one.
labsim now runs a real pair, mirroring production:
router1 172.31.<v>.252 priority 200 DHCP HA primary
router2 172.31.<v>.253 priority 100 DHCP HA secondary
VIP 172.31.<v>.1 floating, held by the master
That required converting router1, which held .1 directly, to .252 plus a
floating VIP -- otherwise it is two routers, not a pair. All six VRRP groups
show MASTER on router1 and BACKUP on router2.
New tooling:
- sim-ha-config.py generates each role's config, reusing unifi-to-vyos.py
--mode sim for the DHCP half so what is proven here and what production
gets share a code path. VLAN 10 correctly carries /23.
- console-apply.py applies config over the serial console, which is necessary
because a freshly installed VyOS holds the same addresses as its peer and
cannot safely be reached over the network at all until reconfigured.
Known sim-only quirk, deliberately not chased: router1 cannot ARP router2 on
the untagged VLAN 1 while every tagged VLAN works, and VRRP forms correctly on
all six groups regardless. Both OVS bonds carry identical vlan_mode/tag/trunks
and the bond MACs differ, so this is OVS bond behaviour on the native VLAN with
two bonds on one bridge -- not a VyOS config problem, and not present in
production, which uses a real switch.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
|
||
|
Michal
|
6c4318d3ae |
feat(migration): reserve every active client at its current address
kea does not inherit UniFi's lease database. At cutover it starts with an empty
view of who holds what, so it can hand an address that is currently in use to a
different device. Reservations are what carry "this device has this address"
across the switch, because they live in config rather than in lease state.
unifi-reserve-all.py creates one per active client, dry run by default. 51
written, 51/51 verified live by reading the records back; the controller now
holds 85 reservations and the generator emits all 85 with unique, valid
hostnames and no duplicate addresses. Active clients with no reservation went
from 48 to 4.
Three guards, each of which caught something real in the dry run:
- VRRP virtual addresses are excluded. UniFi reports them as ordinary client
addresses because the firewalls' bond MACs answer for them, and their
apparent IP flips between the real interface address and the VIP. Without
this, 192.168.1.254 -- the gateway VIP itself -- would have been given a
DHCP reservation.
- The firewalls' own interface MACs are excluded; those are statically
configured routers, not DHCP clients.
- Any address claimed by more than one MAC is dropped rather than guessed
at. This is how the VIPs surfaced in the first place.
Also skipped: addresses already reserved to another MAC, network gateways, and
anything on a network that does not serve DHCP (which excludes the WAN transit
VLANs automatically).
labsim-dhcp-test.sh gained a lease-database flush, and it is not tidiness. Two
findings, both of which first appeared as a PASSING test:
- Re-running against stale leases, kea gave dynamic addresses to three
devices that have reservations. The reservations were present and correct
in kea's own config throughout. Kea saw the reserved address as leased to
"another client" -- same MAC, different client-id from the earlier boot --
and allocated elsewhere. Cutover starts with an empty lease database so
this is a testing artifact, but a reservation is evidently not
unconditional once leases exist.
- Removing only dhcp4-leases.csv does nothing: kea's memfile backend keeps
lease-file-cleanup rotations (.csv.2) and restores from them on start.
The verdict logic no longer takes the first matching lease row. Doing so
reported an hours-old lease as the current answer and scored three failures as
passes, including one where the device had plainly been given a dynamic
address. A MAC with more than one lease is now an explicit failure rather than
a guess.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
|
||
|
Michal
|
44dbd5188c |
feat(migration): export UniFi config and generate VyOS DHCP+DNS from it
Groundwork for replacing the USG with the VyOS pair without anything on the
network noticing. Three pieces:
migration/unifi-export.py pulls 13 endpoints off the classic controller into
timestamped JSON plus a normalised inventory: 11 networks, 31 DHCP
reservations, 4 port forwards, 2 firewall rules, 0 static routes. Two things
this turned up that a naive export would have lost:
- 23 of the 31 reservations carry no network_id at all -- UniFi simply does
not store the binding -- so they are resolved by subnet containment
instead. Without that, three quarters of the reservations have no subnet
to be placed in.
- 30 of the 31 sit INSIDE the DHCP pool, which UniFi's dhcpd tolerates and
which is flagged as a warning rather than discovered at cutover.
migration/unifi-to-vyos.py turns that inventory into VyOS `set` commands for
DHCP and DNS only -- the services the USG owns that VyOS must reproduce. Not a
general converter. --prod and --sim come from one code path so the config
proven in the sim and the config applied to the firewalls cannot drift. Prod
mode hard-fails if any reservation is missing, since a silent drop is the
failure mode that matters.
DNS is included because the USG resolves for 5 of 6 VLANs today: UniFi hands
out the gateway's own address whenever dhcpd_dns is empty, verified by
labmaster resolving against 192.168.8.1. Replacing the USG without a forwarder
would take DNS away from those VLANs entirely.
labsim/labsim-dhcp-test.sh proves it by booting throwaway VMs with real
production MACs -- the one piece of production config that transplants
verbatim. Safe because ovs-labsim has no physical NIC, so those MACs cannot
reach the real LAN.
Result on VyOS 2026.08 (kea), 4/4: printer1 got 172.31.10.46 from inside the
pool, sonoff-matter got 172.31.11.67 across the /23 boundary, Hubitat got its
out-of-pool .2, and an unreserved MAC got an unreserved address. kea honours
in-pool host reservations -- the open question blocking the cutover.
Supporting changes to labsim:
- VLAN 10 widened to /23. Every reservation is in LoT and LoT spans 10.0.0.x
and 10.0.1.x, which a /24 cannot represent.
- LoT's host leg moved to .3, because 10.0.0.2 is a real reservation
(Hubitat) that maps onto the host's own address.
- vlans.conf gained optional masklen and host_octet fields, defaulting to
24 and 2 so the other five VLANs are untouched.
- Fixed /etc/network/interfaces hardcoding 255.255.255.0. That file is what
actually takes effect on these Alpine guests -- cloud-init's
network-config is ignored -- so any non-/24 VLAN was silently wrong.
Two generator bugs found by VyOS rejecting the output: static-mapping names
are validated as hostnames, so underscores fail; and two devices named
"espressif" plus two named "thebeast" collided into single names, which would
have overwritten one reservation with another's address.
The raw export holds WiFi passphrases and the WAN PPPoE credentials and is
gitignored.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
|
||
|
Michal
|
a5b36678ed |
feat(labsim): live topology view with per-path latency
Some checks failed
CI/CD / lint (pull_request) Failing after 9s
CI/CD / test (pull_request) Failing after 9s
CI/CD / typecheck (pull_request) Failing after 24s
CI/CD / build (pull_request) Has been skipped
CI/CD / publish-rpm (pull_request) Has been skipped
CI/CD / publish-deb (pull_request) Has been skipped
The Grafana heatmap of 1s and 0s said almost nothing, and the state timeline
was an unreadable pile of overlapping series labels. Replaced as the primary
view with a purpose-built page served by the exporter itself.
- Probe now captures ICMP RTT, exposed as labsim_rtt_ms{src,dst}. A path that
is up but slow is a different problem from one that is down, and a pass/fail
grid cannot show it.
- Exporter serves / (topology), /api/matrix (JSON) and /metrics.
- topology.html: node per VLAN in a ring, VyOS router in the centre because
every inter-VLAN packet really does traverse it, one line per pair coloured
green/red with the RTT on it. Hovering gives per-direction state. A node ring
goes red if anything to or from it is blocked. Side panels list blocked paths
and the slowest links. Refreshes every 5s, no dependencies.
Grafana stays for what it is actually good at — history of when a path flipped.
Label placement is deliberate: RTT captions sit ~32% along each edge with a
perpendicular nudge, because every diagonal of a 6-node mesh crosses the centre
and midpoint labels stack on the router node.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
|
||
|
Michal
|
c91e44f796 |
feat(labsim): libvirt replica of the lab network with LACP + VyOS routing
Some checks failed
CI/CD / lint (pull_request) Failing after 11s
CI/CD / typecheck (pull_request) Failing after 10s
CI/CD / test (pull_request) Failing after 9s
CI/CD / build (pull_request) Has been skipped
CI/CD / publish-rpm (pull_request) Has been skipped
CI/CD / publish-deb (pull_request) Has been skipped
A throwaway copy of the production VLAN topology so routing and firewall changes can be tested before they touch the real network. Same VLAN IDs and roles as UniFi, deliberately different ranges (172.31.<vlan>.0/24) so nothing here can be mistaken for production. - OVS fabric: real 802.1Q. Access port per micro VM, host leg per VLAN (.2, for SSH only — NOT the VMs' default route, so inter-VLAN tests exercise the router rather than the host's routing table), and a trunk portgroup with VLAN 1 declared nativeMode='untagged'. - Six Alpine micro VMs (256MB, copy-on-write overlays on one 176MB image), SSH + a hello-world HTTP page naming the VLAN. - VyOS router installed to disk unattended over the console, with the SAME config shape as the VP2440s: two NICs in an LACP bond carrying the trunk, VLAN 1 native, bond0.<vlan> holding the .1 gateway on each. - labsim-matrix.py: full-mesh ICMP/TCP22/TCP80 probe, ~0.2s, --watch highlights cells that changed since the last sweep. Guest-side probe is python3 (already present via cloud-init) so nothing is installed on VMs that have no internet. - Prometheus + Grafana (anonymous auth, no login) with a provisioned dashboard: heatmap plus a state timeline showing exactly when a path flipped. Verified end to end: one VyOS rule took sum(labsim_reachable) from 90 to 84, blocking precisely kvm<->k8s across all three protocols. Traps found building this, all now encoded in the scripts: - virtio-net breaks 802.3ad: the guest's bonding driver reports slaves "MII Status: down" despite carrier=1 and never sends an LACPDU, so the bond sits in AD_STATE_DEFAULTED. e1000e fixes it with no other change. Matches the netdev thread "bonding (IEEE 802.3ad) not working with qemu/virtio". - OVS defaults bonds to active-backup, which does not speak LACP at all — bond_mode=balance-tcp is required. - LACP deadlock: OVS holds members disabled until negotiation while the partner needs carrier before it will send LACPDUs. lacp-fallback-ab breaks it. - LACPDUs are untagged, so a trunk with no native VLAN has nowhere to put them. - --boot cdrom,hd re-runs the ISO on every restart, so every commit+save went to a live system that evaporated. Install now switches the VM to boot hd. - cloud-init on Alpine: users stay locked without lock_passwd:false, one failing runcmd aborts the rest, busybox here has no httpd applet, and start-stop-daemon --exec /usr/bin/python3 matches cloud-init's own python3. - The user-data heredoc is unquoted, so backticks in a COMMENT were executed by the host shell and their output corrupted the YAML. build_seed now validates with yaml.safe_load before building the ISO. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH |