25 Commits

Author	SHA1	Message	Date
Michal	395577850c	IPv6 was never HA, and the WAN becoming HA is what exposed it ... Reviewed the parked IPv6 task against the PPPoE-HA work of 2026-09-05/06. The gate that parked it ("WI-8 before IPv6") is cleared, but the same work invalidated the assumption the IPv6 design rested on. Verified on the live routers: vyos002 has no tun0, no he-tunnel-follow, no he-secrets, no VLAN 9 prefix and no route6 ::/0 -- only the pre-staged default-deny v6 firewall, which is correctly on both. Failover is now automatic and drill-proven, so every failover takes the whole v6 estate down for as long as vyos002 holds the VIP. Four things that came out of checking rather than reading: - PPPOE-HA.md's "tun0 survived untouched and IPv6 stayed up at 15.5ms" does not follow from its own premise and is corrected in place. The endpoint address is stable, but it MOVES to vyos002, which has nothing to decapsulate protocol 41. wan-drill had no IPv6 check at all, which is why nobody caught it. - A 22-second near-miss: vif53-pin-boot-disable bounced the 10 gig, he-tunnel- follow ticked once and saw the PPPoE address, and vyos-failover restored the route 22s before the second tick would have pointed HE at an address Vodafone reissues on every dial. - VyOS does NOT leave a tunnel down when its source-address is absent (the override's stated reason for leaving IPv6 single-homed). Measured in labsim: it commits rc=0 and brings the link UP -- a blackhole that attracts the v6 default route. The runtime gate is load-bearing, like the PPPoE gate. - The RA link-mtu was pinned at 1480 while the tunnel correctly drops to 1472 on the PPPoE path. Mechanism, mirroring PPPoE HA -- identical config on both, gated at runtime, no commit in the failover path: - vrrp-wan-reconcile: a v6 kernel plane. tun0 and radvd follow the VIP; radvd is stopped BEFORE the WAN goes so its farewell RA (router-lifetime 0) still has a path out. The WAN early-exits became if-blocks so the plane runs every tick. It deliberately does NOT call he-tunnel-follow: that would halve the hysteresis the near-miss above showed we depend on. - he-tunnel-follow: a master guard reading the same vrrp-wan.conf VIP, so the backup copy cannot point HE at its own idle PPPoE line, plus a stubbable HE_UPDATE_URL. - vrrp-wan-install carries both, so --check and the upgrade runbook cover IPv6. - wan-drill measures IPv6 in both timing loops and asserts zero HE API calls across a router failover. labsim finally has an HE endpoint, closing the gap the override itself cited as why this was never rehearsed. Both ISP islands already share the libvirt network, so that becomes the backbone and HE lives behind it on one address reachable over either WAN. Proven in the sim: backup tun=DOWN radvd=inactive, master tun=UP radvd=active, hysteresis then HE call then MTU 1480->1472, and VLAN 9 hosts autoconfiguring from the RA. The end-to-end v6 datapath is NOT yet proven -- inter-island transit crosses libvirt NAT and the return path is lost. Recorded as a KNOWN SIM GAP rather than papered over. The model change is staged, not merged: another agent runs pulumi up on that repo, and the gate must exist on vyos002 before the tunnel does. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH	2026-09-06 14:30:44 +01:00
Michal	5ed0e4888a	labsim: both matrices green end to end, numbers reproduced ... Final confirming run with GRACE=300 and every fix installed: --all ALL PASS (T0 T3 T5 T8 T11 T12) --hard ALL PASS (replace / deny / disable, each policy verified) Second independent measurement of the hard failover, which is what makes the figures trustworthy rather than anecdotal: run 1 run 2 replace 26s 26s deny 148s 141s disable 21s 20s `deny` sits at ~141-148s across both, so it is the AC's dead-peer behaviour and not a one-off; GRACE=300 keeps roughly 2x margin. Corrected an overclaim in PPPOE-HA.md while confirming it: the invariant row read "AC never showed two simdsl sessions", and under session-control=disable it did -- one live, one orphaned from the destroyed router. Only ever one LIVE router dialled, which is the invariant that matters. Said so plainly rather than leaving a table that reads better than the evidence.	2026-09-06 00:42:01 +01:00
Michal	97efb5abb2	labsim: record the failover numbers and how they were nearly wrong ... The three T4 measurements (replace 26s, deny 148s, disable 21s), why deny is the only one that sizes GRACE, and why 148s is a floor rather than a worst case -- idle VMs, and an AC that shares an OVS bridge so `virsh destroy` lets it see the peer vanish in a way a real BRAS never would. Also writes down the broken setter that made the previous numbers fiction, so the next person to touch the matrix knows to check the policy actually landed.	2026-09-06 00:27:38 +01:00
Michal	6987b324f1	vrrp-wan: size GRACE from the measured hostile failover, not the theory ... With the matrix actually setting session-control, T4 timed a destroyed master's takeover at: replace 26s deny 148s <-- sizing case disable 21s `deny` is the case GRACE exists for: the AC refuses the survivor until its own dead-peer timer frees the dead session. The session poller caught it happening -- the destroyed router's session stayed in the table while the survivor's dials appeared and were rejected, twice, before one took at 148s. 148s is well past the lcp-echo-interval(30) x failure(3) = 90s budget that 180 was sized against, leaving 32s of margin. GRACE=300 is ~2x the worst observed. Treat 148s as a floor, not a worst case: these are idle 2-vCPU VMs, and the AC shares an OVS bridge with the routers, so `virsh destroy` removes the port and accel-ppp sees the peer vanish. A real BRAS over DSL never learns our router died and waits out longer timers of its own. The cost is stated in the conf: GRACE is also how long an alive-but-unroutable master holds every VIP before yielding. bond0.53 covers most of that -- a DHCP lease satisfies the check in seconds -- so GRACE only dominates when PPPoE is the last path. Kept the health check's fallback in step, since keepalived runs it with no environment and that number decides mastership if the conf is ever missing. check_invariant no longer fails blind on the AC's session count. That count is only a proxy for "two of our routers dialled", and only while the AC enforces single-session; under `disable` it does not, so a destroyed router's session lingers and the count reads 2 with exactly one live router dialled. The real invariant -- at most one router holds pppoe0 -- is now the failing one, and the stale session is reported as a WARN rather than silenced, because it still occupies the slot at a real ISP and is precisely what made `deny` take 148s.	2026-09-06 00:25:00 +01:00
Michal	9221c71ff0	labsim: the session-control matrix was never setting session-control ... T4 prints "--- session-control=deny ---" and then measures whatever policy the AC already had. The setter was isp "vbash -c 'source script-template; configure; set ...; commit; save'" and that form does not start a config session at all -- commit dies with "Invalid command: [commit]" on stderr, which isp() discards. `show configuration commands | grep session-control` on the ISP VM returned nothing after a full matrix run: all three iterations had run against the accel-ppp default. The labels were fiction, and a harness that reports coverage it does not have is worse than one that reports a failure. Driving it from a real script FILE works. isp_session_control() does that, reads the value back, and fails the iteration if it disagrees rather than measuring the wrong policy. `session-control` is a valid node here (checked the template dir on VyOS 2026.08.12-0831-rolling), so this was purely the invocation. Staged the two Pulumi overrides in migration/ rather than adding them to kubernetes-deployment: another agent runs `pulumi up` on that repo, so merging `remove: pppoe0 disable` before the gate exists on vyos002 would let it dial and take the single Vodafone session off the live master. Ordering is written at the top of the file.	2026-09-06 00:15:45 +01:00
Michal	4d47b609a2	PPPOE-HA: record the two failure modes found by running the thing ... Both were invisible to inspection and only appeared under the hard matrix: the flap damper tearing down an established session via the lease it stopped renewing, and a missing peers file being completely silent. The second has a production edge worth spelling out -- an unsaved commit reverts on reboot and takes pppoe0 with it, leaving a standby that can never take over while looking perfectly healthy.	2026-09-06 00:11:32 +01:00
Michal	b659e0d47e	vrrp-wan: a flap holdoff must not tear down a live WAN session ... ppp_dial() checked the flap holdoff and returned BEFORE renewing /run/vrrp-wan/may-dial. That lease is what vrrp-wan-guard expires after LEASE_TTL, so tripping the damper stopped the renew and the guard hung up pppoe0 on the MASTER ~80s later. A damper meant to suppress repeated DIALS was tearing down a working WAN instead. Observed in labsim, end to end: DIAL FLAP: >=6 attempts in 600s -- holding off 900s GUARD: lease stale (81s > 75s) -- hanging up pppoe0 An established session now outranks every check below it: ppp_active renews the lease and returns first. Everything after it only decides whether to start a NEW session. Two supporting fixes for how that storm started. The dial attempts were all no-ops because /etc/ppp/peers/pppoe0 was missing, and nothing said so -- systemd logs "skipped because of an unmet condition check" exactly once and the gate looks identical to a healthy backup. ppp_dial() now reports it, and distinguishes "configured but not rendered" (re-commit the subtree) from "no pppoe0 in config at all", which is what a reboot leaves behind when a commit was never saved. That is precisely how the sim secondary lost its WAN. Also `cat | wc -l` rather than `wc -l < file`: redirections are applied left to right, so the missing-file error escapes the 2>/dev/null on every first-ever dial. Harness: T11 copied-then-removed instead of mv, and verifies the restore -- losing that file strands a router permanently, which cost a debugging session. preflight now refuses to run if either router lacks the peers file or the pppoe0 config, since every failover result would otherwise be a false negative blamed on the ISP. New T12 forges a 900s holdoff against a live session and asserts it survives.	2026-09-06 00:04:41 +01:00
Michal	d626750010	labsim: hard failover and reboot safety hold; runbook for the production apply ... Hard failover proven by destroying the master outright (virsh destroy -- no PADT, the case a graceful stop cannot cover): the survivor took the VIPs, dialled, and the access concentrator showed exactly ONE simdsl session from its MAC for the whole seven minutes. When the destroyed router came back it did NOT dial -- ConditionResult=no, ActiveState=inactive, NRestarts=0, pppoe0 absent, may_dial=no -- and the AC session count stayed at 1. That is the gate doing the one job it exists for, on the path that previously had no protection at all. Two more harness bugs of the same family as the last three, both of which reported a working system as broken: - ppp_on() returned an EMPTY string for an unreachable router, and `[ "" = 0 ]` is false, so the hard-failover wait sat for its full timeout waiting for a DESTROYED box to report zero -- long after the survivor had taken over correctly. Absent now means 0. - a VM restart recreates its taps under new names and the OVS bond keeps the old ones: lacp dies, VLAN 1 goes with it, and the box returns reachable on some VLANs and not others. That looked exactly like a failed failover. It is the same stale-membership fault ovs_bond_router already detects, but nothing ran it after a restart; the harness now does. migration/PPPOE-HA.md is the runbook: what the design is, why `disable` cannot work, the measured numbers, the deploy order (vyos002 first, on its own commit, verified on the wire with tcpdump rather than from state), and the one-line rollback. It also records a model hazard found while writing it. The imported baseline captures the RUNNING state, not the safe one -- vyos001 has no `vif 53 disable` because it happens to be master, vyos002 does. An apply performed while vyos002 held the VIP would therefore enable vyos001's WAN too, putting the cloned MAC on both boxes. An override must assert `vif 53 disable` on BOTH, accepting that an apply then briefly disables the current master's 10 gig until the reconciler restores it. Still not applied to production. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH	2026-09-05 19:21:13 +01:00
Michal	93fed7826b	labsim: PPPoE HA passes the matrix, and the health check had a real flap bug ... Full run green: baseline (one AC session, held by the VIP holder), clean failover (pppoe0 moves in 26s, old master releases), 10 gig down (route falls to pppoe0 and the LAN is back online in 5s), lease expiry (the guard hangs up), and a missing peers file (NRestarts=0, no loop). Evidence in labsim/wan-failover-evidence/. The 10 gig test found a genuine bug in vrrp-wan-health, not in the sim. GRACE was measured from PROMOTION, so an established master had no grace at all -- after hours of uptime `now - since` far exceeds any window. The first moment bond0.53 went down while pppoe0 was mid-redial, the master failed its own check, shed every VIP, and the peer inherited the same WAN outage and did the same. A brief WAN blip would have flapped the production pair. The stamp is now refreshed on every healthy tick, so grace measures time since the box last demonstrably HAD a WAN -- survivable wherever the gap happens, not only just after a promotion. Three harness bugs, all the same shape, all of which produced a confident wrong answer before being caught: - waiting for "exactly one pppoe0 holder" returns INSTANTLY during a handover, because it was already true. The useful question is who holds it. - judging connectivity on a single ping 20s after a link drop reported an outage that had already healed. Poll, do not sample. - `-o PreferredAuthentications=password` suits the routers but not the Alpine LAN VMs, whose sshd offers keyboard-interactive: ssh exited 255 before running anything and the test read that as "the LAN lost the internet". A tcpdump on the router showed the pings leaving pppoe0 NATed to 198.51.100.117 and the replies coming back the whole time. An exit code that can mean "the network is broken" or "I could not log in" is not a connectivity test, so the check now asserts on what the guest reported. That last one is why the harness asks the routers and the access concentrator rather than a client, and why it refuses to run at all while either router still has a default route via eth2 -- the libvirt-NAT scaffold answers connectivity checks the WAN under test would have failed. Still to run: hard failover (destroy the master), and the session-control replace/deny/disable axis that brackets Vodafone's unknown behaviour and sets the final GRACE. Nothing applied to production. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH	2026-09-05 19:11:07 +01:00
Michal	4efd70c987	vyos: move PPPoE off the config plane onto a gated systemd unit ... PPPoE HA could not work as written, and the reason is structural rather than a bug: `set interfaces pppoe pppoe0 disable` and `delete` are handled identically by interfaces_pppoe.py -- both UNLINK /etc/ppp/peers/pppoe0. That path is pppd's own options file, so the resting state destroyed exactly what the promotion path needed, and `ppp@pppoe0` restart-looped against it (observed: 47 restarts, zero sessions at the access concentrator). It also made op-mode `connect interface pppoe0` unusable, and put every failover behind a priority-322 commit where one unrelated invalid node fails the whole thing -- which has already taken the 10 gig down once. pppoe0 is now configured identically and ENABLED on both routers, so the peers file always exists, and dialling is gated by a drop-in on the unit: ConditionPathExists=/run/vrrp-wan/may-dial ConditionPathExists=/etc/ppp/peers/pppoe0 /run is tmpfs, so the gate is shut at boot and neither box can dial before VRRP has decided. That matters more than it first appears: with the node enabled, interfaces_pppoe.py restarts ppp on EVERY commit touching the pppoe subtree when the daemon is not running -- so the backup actively tries to dial whenever anything commits. The gate is the only thing making that a no-op, which is why vrrp-wan-reconcile now refuses to bless a box whose drop-in is missing: /etc is per-image, and a VyOS upgrade would otherwise silently remove the protection. may-dial is a LEASE, not a flag. ConditionPathExists is evaluated at start only -- it can prevent a dial, never revoke one -- so a reconciler that stops running while its box is demoted would keep the one ISP session for ever. The reconciler renews the lease; a new 5s vrrp-wan-guard revokes it, and only ever revokes. It fired correctly first time: "GUARD: lease stale (81s > 75s)". Also: remove-then-stop on release (the file's absence blocks a NEW start that a concurrent commit would trigger); a flap damper, because two routers that both believe they hold the VIP will both dial and each dial kills the other's session -- against a real ISP that is how an account gets rate-limited; and a guard on `cfg` returning empty under commit-lock contention, which had already produced one spurious "releasing" on a box that needed nothing. GRACE 90 -> 180. accel-ppp's dead-peer budget is lcp-echo-interval(30) x failure(3) = 90s, so the old value sat exactly on the boundary: a hard failover into an AC that does not replace the stale session would fail its own check, shed the VIPs, and leave both routers in FAULT. The sim could not have tested any of this. Both routers now get the identical WAN -- the secondary had none "because two PPPoE clients sharing one credential is a different failure mode than anything production has", which is backwards: that IS production. It also left the pair incomparable, ten NAT rules against none. Safety now comes from resting state, not asymmetry. Three more things the sim was hiding: - the drift check's secondary regex omitted interfaces pppoe/bonding, nat source and protocols failover, so it reported "in sync" for a box with no WAN at all; - the VRRP health-check and transition-script hooks existed on both live VMs and in NEITHER generator -- the mechanism under test was pure undetected drift; - labsim-vyos's only default route was the libvirt-NAT scaffold, so every "the LAN still has internet" verdict on it was answered by eth2 rather than the WAN. --drop-scaffold applied; the earlier DHCP-failover proof is being re-run because of it. vrrp-wan-install ends the other half of that: the sim's previous proof came from scripts hand-`sed`-ed in place, so the tested behaviour was not the committed behaviour. `--check` now makes that a hard failure. First green run: master holds both WANs, backup released, and the AC reports exactly ONE session. sim-net-apply.sh check: all four in sync. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH	2026-09-05 18:50:16 +01:00
Michal	d727a50ca0	migration: offline recovery card, and stop the leak test trusting a silent router ... The change itself takes Management down, so the session that performs it has no internet and no Claude. RECOVERY-CARD-vlan1-move.md is what is actually needed at that point, on one page: the recovery path, the order, the commands, and what to do when locked out. The recovery path is `ssh vyos@10.0.1.252`, and the reason it is trustworthy is that it is not routed -- the workstation is 10.0.0.210/23 and the router's LoT leg is 10.0.1.252/23, so `ip route get` returns dev lanbr0 with no `via`. It therefore survives Management, VRRP, the VIPs, DNS and the switch trunk all being wrong at once. bond0.10 is untouched by the change and stays in the LAN group. Order is switch-first, which is the opposite of what seems natural. The UniFi controller is 192.168.1.5, on Management, reached from LoT *through vyos001*: do the router first and you lose the controller you still need for the switch. Two corrections to the leak test, both because it reported a router fault that was its own: - it counted `pgrep -f 'tcpdump -i bond0'`, so a stray tcpdump from an earlier run satisfied the >=2 guard with none of this run's captures alive. A capture that records nothing reads as "the router sent no reply at all". - it believed a single silent run. Kea can be is-active and answering nothing for tens of seconds after a restart, so the first VLANs of a loop failed and the last passed. That produced two OPPOSITE and equally wrong conclusions about `listen-interface` before a retry showed the pattern. On `listen-interface`: it is a real second branch in kea-dhcp4.conf.j2 that keeps dhcp-socket-type raw, unlike listen-address which forces udp and took DHCP down when it was tried live. But naming the sub-interfaces explicitly left Management DHCP dead in the sim, reproducibly, against a clean control with interfaces:["*"]. Not adopted, and not needed -- the address move is the proven fix. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH	2026-09-02 14:30:35 +01:00
Michal	0481c38e09	labsim: prove the tagged-Management fix for kea's wrong-pool offers ... Kea #1117: with dhcp-socket-type raw, a frame tagged for a sub-interface is also delivered to the parent's AF_PACKET socket, and if the parent serves a subnet kea answers from it too. Management being the native VLAN on bond0 is what gives the parent that subnet. One DISCOVER on VLAN 3 produced two OFFERs, and in the captures here the WRONG one arrives first as often as not -- which is why this looked device-dependent rather than like a server bug. labsim-vlan-leak-test.sh reproduces it and scores the SERVER's offers, not the client's choice; a client picking correctly is how this hid. Fails on the old shape, passes on the new one across all six LAN VLANs. Three things the rehearsal caught that reasoning had not: - kea keeps its old raw socket. VyOS does not restart it for an interface address change, so the first post-fix test failed and looked exactly like the fix not working. - interface-group LAN names the bare bond0. Moving the address without moving the group drops every management session under default-deny. - there is no make-before-break. A port always egresses its native VLAN untagged, so while VLAN 1 is native the router can send tagged VLAN 1 but never receive it -- verified, the ARP landed on bond0 untagged. What makes the cutover safe anyway is that tagged and untagged Management coexist, so the firewalls convert one at a time: 0s of VIP downtime, versus 5m30s if both routers go before the switch does. In that state the healthy BACKUP does NOT take over -- the sync group holds native BACKUP because the other VLANs still hear the master. Also fixes two ways the sim was lying. ovs_bond_router compared only the trunk VLAN list on re-runs, so a VM restart left the bond holding taps that no longer existed while the real ones sat in the bridge unbonded -- labsim-vyos2 had no LACP at all. And the tap count included the primary's libvirt-NAT scaffold NIC, so the primary's bond was skipped outright. Runbook: migration/MANAGEMENT-VLAN-TAGGED.md Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH	2026-09-02 14:03:44 +01:00
Michal	11344eab92	labsim: the IPAM switch needs no pod recycle when the CIDR sources agree ... Re-ran the 3-node rehearsal, this time from a state that matches production rather than one I had accidentally skewed. The earlier "two nodes swapped CIDRs" result was an artefact of my own setup: labsim had been running cluster-pool with per-node CIDRs that differed from node.spec, I flipped it to ipam=kubernetes (which resyncs CiliumNode from node.spec), and flipping back therefore looked like a renumber. Production has only ever run ipam=kubernetes, and all five nodes were checked: CiliumNode and node.spec agree everywhere. From that matching state the switch is close to a non-event: per-node CIDRs unchanged, every pod still inside its node's range, nothing stranded, no recycle required. The only disruption is the cilium DaemonSet restarting itself -- one agent sat in Init:0/6 and one node briefly took the agent-not-ready taint, both of which cleared on their own. Cross-node connectivity verified after. So the recycle is CONDITIONAL, not a fixed step, and `verify` is what decides. Documented both cases in the script header, because the dangerous one is silent: stranded pods report Running and Ready while being unreachable. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH	2026-08-24 23:20:16 +01:00
Michal	e8679f45b5	labsim: rehearse the IPAM switch on 3 nodes, and catch the trap in it ... Rehearsed kubernetes -> cluster-pool on the 3-node labsim cluster, which is the transition production faces. The switch itself is undramatic: agents stayed up, the operator adopted the pool, and the agent-not-ready taint deadlock did NOT occur. That deadlock is specific to ADDING IPv6 -- the agent blocks on an IPv6 pod CIDR that does not exist yet. A v4-only mode switch does not hit it. The real hazard is quieter. The operator does not preserve which node held which /24: two nodes swapped CIDRs. Their existing pods kept their old addresses, which now fall outside the node's range, so every other node routes that prefix to the wrong node. Cross-node ping to those pods dropped 100% while every pod stayed Running and every node stayed Ready. Nothing in `kubectl get pods` shows it. So "pods kept the same address" is the FAILURE signal here, not the reassurance it looks like. cilium-ipam-switch.sh verify now flags pods sitting outside their node's CIDR, which is the check that decides whether a recycle is optional (it is not) or mandatory (it is). Recycling every deploy/ds/sts restored it: all pods back inside their node CIDR, cross-node ping 0% loss. Sequence proven end to end: preflight -> apply -> restart operator then agents -> unstick if needed -> recycle all workloads -> verify Also fixed the recycle hint the script printed: `kubectl rollout restart deploy,ds,sts -A` is not valid (`unknown shorthand flag: 'A'`), so anyone following it under pressure would have got an error instead of a recycle. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH	2026-08-24 23:13:20 +01:00
Michal	842408c0d9	labsim: prove k3s CAN be converted to dual-stack in place ... k3s documents that dual-stack "cannot be enabled on an existing cluster". Rather than accept that for a 145-day-old production cluster, build both shapes and diff them. dualstack-lab.sh builds a single-node IPv4 cluster and a native dual-stack one, takes a reflink copy of the IPv4 disk so a failed conversion costs 90 seconds to undo, converts in place, and diffs the results. Result: the conversion works. etcd data is never touched. The documented blocker is real but narrower than stated. Cilium reports it exactly -- `required IPv6 PodCIDR not available` -- because node.spec.podCIDRs is assigned at join and is immutable, and the Kubernetes IPAM controller will not add a second family later. That objection only holds while Cilium runs ipam=kubernetes and therefore reads that field. Switching to cluster-pool IPAM moves pod CIDR allocation into the CiliumNode CRD, where the operator hands out both families on a cluster that was born IPv4-only. Sequence that works, in order: 1. k3s unit gains --cluster-cidr/--service-cidr/--node-ip with both families. k3s validates the two CIDRs together and refuses to start on a mismatch ("must share the same IP version"), so a partial edit crash-loops rather than coming up half-configured. That is the safe failure mode. 2. The ServiceCIDR object picks up the IPv6 range on restart -- this is upstream's supported "single-to-dual-stack preserving the primary ServiceCIDR" path, and existing Services keep their IPv4 addresses. 3. Cilium to ipam=cluster-pool with an IPv6 pool, then DELETE the CiliumNode so the operator reallocates; it will not add a family to an existing one. 4. Expect a deadlock here: the agent will not go ready without a pod CIDR, so the node keeps the node.cilium.io/agent-not-ready taint, so the new operator that would assign the CIDR cannot schedule. Remove the taint by hand once to break it. Verified on the converted cluster: pod with 10.42.0.125 AND fd00:42::4843, and a PreferDualStack Service holding 10.43.122.115 AND fd00:43::72a3. The only field that still differs from a native build is node.spec.podCIDRs, which stays IPv4-only -- immutable, and unused once Cilium owns IPAM. CiliumNode podCIDRs and ServiceCIDR are identical to the native cluster. Not yet answered: this is one node. Whether a 3-server etcd cluster converts as cleanly, and what a rejoining agent does, is the next experiment. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH	2026-08-24 22:46:27 +01:00
Michal	ad6eb7a9a6	labsim: default-deny firewall policy, proven in the sim ... Policy: internal VLANs reach each other and the internet; the internet initiates nothing inward. That was already the effect of the IPv4 ruleset, but built as a blacklist -- default-action accept plus explicit drops per WAN interface. Identical behaviour right up until a WAN is added, at which point it is open and nothing looks wrong. This expresses it as a whitelist. Two findings from the sim, both of which would have been outages in production: `set` on a rule number is ADDITIVE. The sim already had a rule 10 carrying inbound/outbound interface constraints; `set ... rule 10 state established` ANDed onto it, producing a stateful-accept that applied to one interface pair only. Return traffic from the internet then matched no rule and hit the default drop, so LAN hosts could reach nothing outbound. The generator now deletes each filter before rebuilding it, so the code owns the subtree. It is one commit, so nftables is rebuilt atomically -- there is no window without a firewall. DHCP lease renewal is unicast UDP to port 68 and conntrack does not reliably cover it. Without an explicit rule the WAN keeps working until the lease expires and then dies -- a delayed failure that looks nothing like a firewall change. Also added a loopback accept for both families, absent from the v6 policy since it went default-deny. Verified in labsim: inter-VLAN ok, LAN-to-internet ok, internet-to-router dropped, and internet-to-LAN dropped with the drop counter incrementing by exactly the packets sent, after routing the test through the router rather than around it via the hypervisor. Also extends the drift check to the firewall subtree, which it did not cover -- so it had been reporting "in sync" while that subtree was uncaptured. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH	2026-08-22 22:25:55 +01:00
Michal	7f551081ad	labsim: capture BGP, dual WAN and both ISP VMs as code ... The sim's routing config existed only as running state on the VMs. It was applied by hand over SSH, so rebuilding a VM lost the rehearsal and nothing recorded why any of it was shaped the way it was. The two ISP VMs were not referenced anywhere in the repo at all. sim-net-config.py generates all four roles; sim-net-apply.sh applies them over the serial console, or diffs them against the running VMs. Verified reproducing live state exactly before committing: primary 40/40 commands, secondary 16/16, isp-dhcp 19/19, isp-pppoe 21/21. Carries the reasoning that was previously nowhere: RFC 8212 needing policy in both directions or the session carries zero prefixes; probe targets that must not double as system name-servers; default-route-distance 210 rather than no-default-route, which blanks new_routers and hands the default route to the backup line; and the WI-8 bootstrap bug that pinned /32s fix. Dropped a stale `pppoe-server interface eth0` on isp-pppoe (a NIC that does not exist there) so a green drift check stays meaningful. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH	2026-08-22 16:26:36 +01:00
Michal	86c2a36f00	feat(labsim): a real Kubernetes cluster for rehearsing Cilium <-> VyOS BGP ... BGP is about to be added to a network that currently works, where a bad advertisement blackholes the house. That needs somewhere to fail first. Three Debian nodes (4 GB / 2 vCPU) on the OVS `vlan2` access ports running k3s with flannel disabled, so Cilium is the CNI under test. Three rather than two because ECMP is only meaningfully tested if a node can be drained and more than one path survives. VMs rather than k3d: the thing under test is eBGP between Cilium and VyOS across the switch fabric, nodes peering with the router's bond0.2 leg, directly connected. k3d would put the nodes on a container bridge -- a different L2 path, proving something else. (It also wants Docker; this host has podman.) The existing micro VMs are Alpine with 256 MB and 1 vCPU, which is not close to enough. Debian rather than the sim's Alpine base: glibc, a stock kernel that Cilium's eBPF probes are tested against, and cloud-init that actually applies network-config -- the Alpine base notably does not. One trap worth recording. An earlier draft called `selected_vlans "$K8S_VLAN"` before `ovs_up`, and since `ovs_up` re-defines the libvirt network from SELECTED, that silently deleted the portgroups for every other VLAN. Running VMs kept working -- their taps were already attached -- so nothing complained until the ISP VMs needed vlan51 and vlan53 and could not be attached. It now selects every VLAN. The generated kubeconfig is gitignored: it carries cluster-admin credentials and is one `git add -A` away from being committed. Regenerate with `k8s-up.sh --kubeconfig`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH	2026-08-19 13:15:45 +01:00
Michal	ff86a421f4	fix(labsim): console-apply must handle both VyOS prompts, not just $ ... Two failures from one strict expect, both hit while building the sim ISPs. A run that dies mid-config leaves the console parked in configuration mode. The next run then waits for the operational `$ ` prompt against a perfectly healthy VM and hangs until timeout, with nothing in the output to say why -- the box was sitting at `vyos@isp-dhcp#` the whole time. Login now accepts `# ` as well and discards the stale candidate rather than committing something nobody has seen. The same mistake at the exit step: insisting on `$ ` after `save` hung, AND left the console in config mode, which is what created the first failure for the following run. Now accepts either prompt. Known-bad, not fixed: the tool reports "committed and saved" when the set commands have not applied. Verified against the clone -- prompt showed the host-name change had landed while `grep -c dhcp-server` returned 0. The failure detection only inspects c.before for a few strings and evidently misses the real failure mode, so success is being reported without evidence. That needs fixing before this tool is trusted for anything; it is currently only safe to use with an independent check afterwards, which is how the gap was found. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH	2026-08-18 01:00:19 +01:00
Michal	ee070371a8	feat(labsim): add WAN transport VLANs so the sim can host fake ISPs ... A cutover attempt failed on the WAN and nothing had tested it. The reason the sim could not have caught it: labsim modelled every LAN VLAN faithfully and omitted the WAN entirely -- vlans.conf had 1, 2, 3, 9, 10 and 200, never 51 or 53. Worse, the switch script's WAN health checks are conditional on the delta configuring PPPoE, so in the sim they printed "this delta configures no WAN -- skipping all WAN health checks" and passed. The sim proved the delta commits; it never proved the WAN works, and could not have. Adds VLANs 51 (Vodafone/PPPoE) and 53 (10gig/DHCP) to the fabric so a fake ISP can live on each and those checks actually execute. VyOS has service pppoe-server (accel-ppp) natively -- authentication local-users, client-ip-pool, gateway-address -- so a VyOS VM can play the concentrator, and dhcp-server can play the other ISP. vlans.conf gains host_octet 0, meaning "no host leg". A host address on a WAN transport VLAN would misrepresent the segment: the point is that VyOS reaches an ISP, not the host. Also fixes a real gap in ovs_bond_router: it returned early when the bond already existed, so adding a VLAN to vlans.conf never reached an existing bond. Re-runs now reconcile the trunk and say so. That gap is the same SHAPE as the production failure -- interface present, VLAN missing from the trunk, frames silently dropped -- which is precisely the class of bug the sim needs to be able to reproduce rather than embody. Both bonds updated: [2,3,9,10,200] -> [2,3,9,10,51,53,200]. Note on the production diagnosis, which is NOT settled: a passive RX test showed zero frames on 51/53 at the firewall, and an active DHCP DISCOVER (verified to have transmitted, tx +2) drew no reply. That is consistent with the VLANs not being trunked, but equally with the ISP only answering its registered CPE MAC -- which is exactly why the delta clones f0:9f:c2:12:9b:4f, and why it cannot be settled from production while the USG holds that MAC. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH	2026-08-18 00:44:11 +01:00
Michal	bb654d83f8	test(labsim): second VyOS router proves DHCP active-passive HA ... Answers the question a single router could not, and that would otherwise only have been discovered at cutover: with kea high-availability active-passive, does exactly ONE box answer a DHCP request? Yes. Probing sim VLAN 10 with broadcast-dhcp-discover returns offers from a single distinct Server Identifier -- 172.31.10.252, the primary. The secondary runs kea but stays silent. Without this the delta would have put 6 subnets and 84 static-mappings on both boxes with nothing to arbitrate them, and two kea instances would have raced on every broadcast domain. Worth noting the raw response count is misleading: nmap reports "Response 1 of 2" because it sends several discovers, and both replies carry the same server identifier. Counting responses says "2 servers"; counting distinct server identifiers says "1". The second number is the true one. labsim now runs a real pair, mirroring production: router1 172.31.<v>.252 priority 200 DHCP HA primary router2 172.31.<v>.253 priority 100 DHCP HA secondary VIP 172.31.<v>.1 floating, held by the master That required converting router1, which held .1 directly, to .252 plus a floating VIP -- otherwise it is two routers, not a pair. All six VRRP groups show MASTER on router1 and BACKUP on router2. New tooling: - sim-ha-config.py generates each role's config, reusing unifi-to-vyos.py --mode sim for the DHCP half so what is proven here and what production gets share a code path. VLAN 10 correctly carries /23. - console-apply.py applies config over the serial console, which is necessary because a freshly installed VyOS holds the same addresses as its peer and cannot safely be reached over the network at all until reconfigured. Known sim-only quirk, deliberately not chased: router1 cannot ARP router2 on the untagged VLAN 1 while every tagged VLAN works, and VRRP forms correctly on all six groups regardless. Both OVS bonds carry identical vlan_mode/tag/trunks and the bond MACs differ, so this is OVS bond behaviour on the native VLAN with two bonds on one bridge -- not a VyOS config problem, and not present in production, which uses a real switch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH	2026-08-16 23:29:12 +01:00
Michal	6c4318d3ae	feat(migration): reserve every active client at its current address ... kea does not inherit UniFi's lease database. At cutover it starts with an empty view of who holds what, so it can hand an address that is currently in use to a different device. Reservations are what carry "this device has this address" across the switch, because they live in config rather than in lease state. unifi-reserve-all.py creates one per active client, dry run by default. 51 written, 51/51 verified live by reading the records back; the controller now holds 85 reservations and the generator emits all 85 with unique, valid hostnames and no duplicate addresses. Active clients with no reservation went from 48 to 4. Three guards, each of which caught something real in the dry run: - VRRP virtual addresses are excluded. UniFi reports them as ordinary client addresses because the firewalls' bond MACs answer for them, and their apparent IP flips between the real interface address and the VIP. Without this, 192.168.1.254 -- the gateway VIP itself -- would have been given a DHCP reservation. - The firewalls' own interface MACs are excluded; those are statically configured routers, not DHCP clients. - Any address claimed by more than one MAC is dropped rather than guessed at. This is how the VIPs surfaced in the first place. Also skipped: addresses already reserved to another MAC, network gateways, and anything on a network that does not serve DHCP (which excludes the WAN transit VLANs automatically). labsim-dhcp-test.sh gained a lease-database flush, and it is not tidiness. Two findings, both of which first appeared as a PASSING test: - Re-running against stale leases, kea gave dynamic addresses to three devices that have reservations. The reservations were present and correct in kea's own config throughout. Kea saw the reserved address as leased to "another client" -- same MAC, different client-id from the earlier boot -- and allocated elsewhere. Cutover starts with an empty lease database so this is a testing artifact, but a reservation is evidently not unconditional once leases exist. - Removing only dhcp4-leases.csv does nothing: kea's memfile backend keeps lease-file-cleanup rotations (.csv.2) and restores from them on start. The verdict logic no longer takes the first matching lease row. Doing so reported an hours-old lease as the current answer and scored three failures as passes, including one where the device had plainly been given a dynamic address. A MAC with more than one lease is now an explicit failure rather than a guess. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH	2026-08-15 23:07:07 +01:00
Michal	44dbd5188c	feat(migration): export UniFi config and generate VyOS DHCP+DNS from it ... Groundwork for replacing the USG with the VyOS pair without anything on the network noticing. Three pieces: migration/unifi-export.py pulls 13 endpoints off the classic controller into timestamped JSON plus a normalised inventory: 11 networks, 31 DHCP reservations, 4 port forwards, 2 firewall rules, 0 static routes. Two things this turned up that a naive export would have lost: - 23 of the 31 reservations carry no network_id at all -- UniFi simply does not store the binding -- so they are resolved by subnet containment instead. Without that, three quarters of the reservations have no subnet to be placed in. - 30 of the 31 sit INSIDE the DHCP pool, which UniFi's dhcpd tolerates and which is flagged as a warning rather than discovered at cutover. migration/unifi-to-vyos.py turns that inventory into VyOS `set` commands for DHCP and DNS only -- the services the USG owns that VyOS must reproduce. Not a general converter. --prod and --sim come from one code path so the config proven in the sim and the config applied to the firewalls cannot drift. Prod mode hard-fails if any reservation is missing, since a silent drop is the failure mode that matters. DNS is included because the USG resolves for 5 of 6 VLANs today: UniFi hands out the gateway's own address whenever dhcpd_dns is empty, verified by labmaster resolving against 192.168.8.1. Replacing the USG without a forwarder would take DNS away from those VLANs entirely. labsim/labsim-dhcp-test.sh proves it by booting throwaway VMs with real production MACs -- the one piece of production config that transplants verbatim. Safe because ovs-labsim has no physical NIC, so those MACs cannot reach the real LAN. Result on VyOS 2026.08 (kea), 4/4: printer1 got 172.31.10.46 from inside the pool, sonoff-matter got 172.31.11.67 across the /23 boundary, Hubitat got its out-of-pool .2, and an unreserved MAC got an unreserved address. kea honours in-pool host reservations -- the open question blocking the cutover. Supporting changes to labsim: - VLAN 10 widened to /23. Every reservation is in LoT and LoT spans 10.0.0.x and 10.0.1.x, which a /24 cannot represent. - LoT's host leg moved to .3, because 10.0.0.2 is a real reservation (Hubitat) that maps onto the host's own address. - vlans.conf gained optional masklen and host_octet fields, defaulting to 24 and 2 so the other five VLANs are untouched. - Fixed /etc/network/interfaces hardcoding 255.255.255.0. That file is what actually takes effect on these Alpine guests -- cloud-init's network-config is ignored -- so any non-/24 VLAN was silently wrong. Two generator bugs found by VyOS rejecting the output: static-mapping names are validated as hostnames, so underscores fail; and two devices named "espressif" plus two named "thebeast" collided into single names, which would have overwritten one reservation with another's address. The raw export holds WiFi passphrases and the WAN PPPoE credentials and is gitignored. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH	2026-08-15 01:33:00 +01:00
Michal	a5b36678ed	feat(labsim): live topology view with per-path latency ... The Grafana heatmap of 1s and 0s said almost nothing, and the state timeline was an unreadable pile of overlapping series labels. Replaced as the primary view with a purpose-built page served by the exporter itself. - Probe now captures ICMP RTT, exposed as labsim_rtt_ms{src,dst}. A path that is up but slow is a different problem from one that is down, and a pass/fail grid cannot show it. - Exporter serves / (topology), /api/matrix (JSON) and /metrics. - topology.html: node per VLAN in a ring, VyOS router in the centre because every inter-VLAN packet really does traverse it, one line per pair coloured green/red with the RTT on it. Hovering gives per-direction state. A node ring goes red if anything to or from it is blocked. Side panels list blocked paths and the slowest links. Refreshes every 5s, no dependencies. Grafana stays for what it is actually good at — history of when a path flipped. Label placement is deliberate: RTT captions sit ~32% along each edge with a perpendicular nudge, because every diagonal of a 6-node mesh crosses the centre and midpoint labels stack on the router node. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH	2026-08-13 00:56:06 +01:00
Michal	c91e44f796	feat(labsim): libvirt replica of the lab network with LACP + VyOS routing ... A throwaway copy of the production VLAN topology so routing and firewall changes can be tested before they touch the real network. Same VLAN IDs and roles as UniFi, deliberately different ranges (172.31.<vlan>.0/24) so nothing here can be mistaken for production. - OVS fabric: real 802.1Q. Access port per micro VM, host leg per VLAN (.2, for SSH only — NOT the VMs' default route, so inter-VLAN tests exercise the router rather than the host's routing table), and a trunk portgroup with VLAN 1 declared nativeMode='untagged'. - Six Alpine micro VMs (256MB, copy-on-write overlays on one 176MB image), SSH + a hello-world HTTP page naming the VLAN. - VyOS router installed to disk unattended over the console, with the SAME config shape as the VP2440s: two NICs in an LACP bond carrying the trunk, VLAN 1 native, bond0.<vlan> holding the .1 gateway on each. - labsim-matrix.py: full-mesh ICMP/TCP22/TCP80 probe, ~0.2s, --watch highlights cells that changed since the last sweep. Guest-side probe is python3 (already present via cloud-init) so nothing is installed on VMs that have no internet. - Prometheus + Grafana (anonymous auth, no login) with a provisioned dashboard: heatmap plus a state timeline showing exactly when a path flipped. Verified end to end: one VyOS rule took sum(labsim_reachable) from 90 to 84, blocking precisely kvm<->k8s across all three protocols. Traps found building this, all now encoded in the scripts: - virtio-net breaks 802.3ad: the guest's bonding driver reports slaves "MII Status: down" despite carrier=1 and never sends an LACPDU, so the bond sits in AD_STATE_DEFAULTED. e1000e fixes it with no other change. Matches the netdev thread "bonding (IEEE 802.3ad) not working with qemu/virtio". - OVS defaults bonds to active-backup, which does not speak LACP at all — bond_mode=balance-tcp is required. - LACP deadlock: OVS holds members disabled until negotiation while the partner needs carrier before it will send LACPDUs. lacp-fallback-ab breaks it. - LACPDUs are untagged, so a trunk with no native VLAN has nowhere to put them. - --boot cdrom,hd re-runs the ISO on every restart, so every commit+save went to a live system that evaporated. Install now switches the VM to boot hd. - cloud-init on Alpine: users stay locked without lock_passwd:false, one failing runcmd aborts the rest, busybox here has no httpd applet, and start-stop-daemon --exec /usr/bin/python3 matches cloud-init's own python3. - The user-data heredoc is unquoted, so backticks in a COMMENT were executed by the host shell and their output corrupted the YAML. build_seed now validates with yaml.safe_load before building the ISO. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH	2026-08-13 00:42:39 +01:00