3c933b96e0f2f698cdc36f9715b7f5a6c54910ef
177 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Michal
|
3c933b96e0 |
VLAN 2 IPv6 applied to production: RA proven, MAC reservations do NOT match
Some checks failed
The unattended window's W1/W2. Both routers now carry the VLAN 2 IPv6 config (addresses ::1/::2, RA with managed-flag + no-autonomous-flag + link-mtu 1472, DHCPv6 reservations for all five nodes), applied via migration/vlan2-v6-apply with a vlan2-v6-watchdog armed on both routers throughout. IPv4 untouched: internet up, 6 MASTER / 6 BACKUP, 5/5 nodes Ready, watchdogs quiet. default-lifetime 0 on purpose -- addressing without egress. Turning on v6 egress moves image pulls onto a tunnel of unmeasured throughput, and the person who would notice is away. One line to flip when attended. THE WINDOW'S QUESTION IS ANSWERED, HALF YES, HALF NO: YES: NetworkManager follows the managed flag. worker0 logged dhcp6 (eno1): activation: beginning transaction minutes after the RA appeared. "ipv6.method=auto will do DHCPv6" is now evidence, not inference. NO: the reservations never match. Every packet kea logs is [no hwaddr info] -- the clients identify with DUID-UUID (and one DUID-LLT), kea derives no MAC from any of them, so hw-address reservations cannot match and no node got its reserved address. VyOS accepting `static-mapping mac` renders valid kea config that simply never matches these clients. The "one source of truth with IPv4" addressing scheme does not survive contact with DHCPv6; options (DUID keys, kea mac-sources, dynamic range + discovery, or SLAAC) are written up in migration/window-evidence/2026-09-06-dhcpv6.txt for an attended decision. FOUND LIVE AND FIXED IN THE SAME WINDOW: `service dhcpv6-server` with no listen-interface renders kea6 with interfaces: ["*"] -- a DHCPv6 server on EVERY VLAN. kea was answering an unrelated device on bond0.10 within seconds of the first apply. Same family as the kea IPv4 cross-VLAN bug (ISC #1117). Now pinned to bond0.2 on both routers. Also in this commit, three self-inflicted script bugs found by their own failures: log() wrote progress lines into the captured config stream (VyOS rejected each as "Invalid command", leaving the two routers correct but NOT identical); "Invalid command" was missing from the failure patterns so that run reported success; and the MAC lookup matched its own freshly-created v6 reservations on the second run, returning doubled MACs. All three fixed, both routers converged and diffed identical. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH |
||
|
Michal
|
5c9f004759 |
labsim: rehearse VLAN 2 IPv6, RA and DHCPv6 reservations before production sees it
Some checks failed
Phase 1.1 of cluster dual-stack -- the same VyOS config the production change
will make, applied to the sim pair first. It found four things.
VyOS specifics this version wants, none of them guessable:
* `subnet-id` is mandatory per DHCPv6 subnet ("Unique subnet ID not specified")
* `managed-flag` is VALUELESS -- `managed-flag true` is rejected
* the prefix option is `no-autonomous-flag`, not `autonomous-flag false`
* static-mapping accepts `mac`, so the reservation keys on the same MAC the
IPv4 one does rather than on a client-generated DUID
Two findings that matter more than the syntax:
managed-flag is NECESSARY BUT NOT SUFFICIENT. It only tells a host to use
DHCPv6; the kernel's accept_ra implements SLAAC and nothing else. The sim's
Debian nodes have no DHCPv6 client managing the interface, so they took a kernel
SLAAC address (EUI-64 from the MAC) and never asked for their reserved ::11.
Production's Fedora nodes and the DGX Sparks run NetworkManager, which does
start a DHCPv6 client on the managed flag -- but that has to be verified per
node, not assumed. The k3s module preflight catches the consequence; the fix is
node-side.
TURNING AUTONOMOUS OFF DOES NOT RETRACT ADDRESSES ALREADY FORMED. An earlier
partial apply advertised the prefix while autonomous was still on, the nodes
autoconfigured, and adding no-autonomous-flag afterwards left those addresses
with a 30-day lifetime. In production VLAN 2 has no IPv6 at all yet, so
no-autonomous-flag MUST be in the same commit that first advertises the prefix.
Also worth knowing for the production apply: A FAILED COMMIT DOES NOT MEAN
NOTHING CHANGED. VyOS commits node groups independently -- one run here left the
interface address and router-advert applied while `[[service dhcpv6-server]]
failed`. Re-read the config after any failure instead of assuming rollback.
The script's first version printed "up" after both routers had failed to commit.
vbash exits 0 even when the commit fails, so it now reads the output and dies.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
|
||
|
Michal
|
914135c47d |
labctl: refuse to write a k3s config naming an IPv6 the node does not have
Some checks failed
Most of this estate is SSH-onboard, not PXE-provisioned, and that changes where the dual-stack guarantee has to come from. os-install-research.md classifies the two paths: Asahi cannot PXE at all, and the DGX Sparks (spark-2935 and aitopatom-3a1c, both NVIDIA_DGX_Spark on NVIDIA's own OS) must never be reinstalled -- see project_dgx_spark_kernel_recovery. Of the five nodes, at most worker0 and worker2 ever run our install templates. So for the majority the templates govern nothing, this module is the only labctl touchpoint, and nothing upstream can promise the vendor OS took its DHCPv6 lease. Writing node-ip for a missing address does not fail here; it fails later when k3s will not start, and it reads as a bind error rather than as a missing address. One ssh round trip turns that into a sentence naming the address and telling you to check the kea reservation. This is what makes "out of the box" true for heterogeneous hardware rather than just for the nodes we image ourselves. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH |
||
|
Michal
|
a9ff182bd6 |
Ubuntu autoinstall emitted invalid YAML for every role -- and now asks for IPv6
Some checks failed
Two separate things, found because adding IPv6 to the provisioning templates
meant parsing the output for the first time.
PRE-EXISTING, AND THE BIGGER NEWS: renderUbuntuAutoinstall produced a document
that does not parse, for vanilla, worker and infra alike. There was no test on
this template, and `toContain` assertions would never have caught it. Three
faults:
* late-commands were serialised as `- "${c}"` with no escaping at all.
Embedded double quotes ended the scalar early -- `echo "tmpfs /tmp ..." >>
/etc/fstab` gives "expected <block end>, but found '<scalar>'".
* the same naive quoting put REAL newlines inside a double-quoted scalar, and
YAML folds those into spaces. So even where it parsed, each heredoc reached
the target as one long line and wrote a file with no line breaks -- the k3s
modules-load and sysctl files among them.
* the longhorn/rancher LVM entries were indented 8 spaces where their siblings
in storage.config sit at 6, so the storage list became a nested map.
Fixed by serialising with JSON.stringify (JSON is a subset of YAML, so escaping
comes for free and \n survives the round trip) and dedenting the LVM blocks.
Verified by parsing the rendered document: the modules heredoc now arrives with
its four lines intact and the fstab command keeps its quotes.
Guarded by a new test that parses the output with a real YAML parser for every
role, mirroring how kickstart.test.ts shells out to ksvalidator. python3's yaml
rather than a new npm dependency.
DUAL-STACK: both templates now request an IPv6 lease. install.ks.ts uses
--ipv6=auto rather than =dhcp so the RA's managed-flag steers the node to DHCPv6
while a VLAN with no DHCPv6 yet still installs -- failing an OS install because
IPv6 was not ready is the worse trade. ubuntu-autoinstall.ts gets a netplan
block with dhcp4+dhcp6 and optional: true, for the same reason. The address
comes from a kea DHCPv6 reservation keyed on MAC, the same source of truth as
the v4 address, so neither template needs to know it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
|
||
|
Michal
|
51bf300474 |
labctl: k3s config can carry both address families
Some checks failed
Phase 2d of cluster dual-stack. This is the file that makes a NEW node correct by construction: it already owns /etc/rancher/k3s/config.yaml for every node, so once it emits dual values, a joining node is dual-stack with nothing else to do. Before this there was nowhere for a node's IPv6 to come from -- K3sConfig had no v6 field and generateAgentConfig() took no arguments at all. The property that matters most is the negative one: with no ipv6 and no CIDRs, addressFamilyLines() returns "" and the output is BYTE-IDENTICAL to what the five existing nodes already have. Verified by diffing the generator's output for worker0's parameters against worker0's live config.yaml -- identical. So this change is inert until dual-stack config is supplied, rather than a flag day that rewrites five healthy nodes and restarts the cluster to tell it what it knew. node-ip is written only once there is a second family to name. k3s auto-detects a sensible IPv4 by itself, and emitting it unconditionally would be that same pointless rewrite. IPv4 stays first: the supported single-to-dual-stack conversion preserves the primary family, which is what lets existing Services keep their ClusterIP. Agents get their own node-ip and no CIDRs. An agent without one joins a dual-stack cluster as IPv4-only, gets no IPv6 pod CIDR, and fails later as pods unreachable over v6 while the node itself reads Ready. The IPv6 address is also added as a TLS SAN. Without it anything reaching the apiserver over v6 -- a peer server joining, kubectl against the v6 address -- fails verification with an error naming the certificate rather than the missing SAN. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH |
||
|
Michal
|
cb03987c33 |
The drill evidence was silently gitignored by *.log
Some checks failed
PPPOE-HA.md cited migration/drill-evidence/wan-drill-2026-09-06-ipv6.log as the record behind the takeover/failback/HE-call figures, and the repo reported clean -- because the root .gitignore has `*.log`, so the file was never added. The citation was dangling and nobody would have found out until they went looking for the numbers. Renamed to .txt, which is also what this repo already does for evidence: labsim/wan-failover-evidence/ and labsim/vlan-leak-evidence/ store state.txt. The ignore rule is fine; the filename was wrong. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH |
||
|
Michal
|
8a0ef52909 |
The drill measured IPv6 through a failover: zero HE calls, and it followed
Some checks failed
Second drill of the day, after IPv6-follows-master went in. First time the IPv6 behaviour of a failover is a measurement rather than an assertion -- the thing this whole review started from was a 15.5ms figure read outside the window. TAKEOVER OK: 37s IPv6 followed in 37s (v4 37s, gap 0s) FAILBACK OK: 32s IPv6 back in 44s tun0 src : 87.192.101.48 -> 87.192.101.48 unchanged HE updates from vyos001: 0 HE updates from vyos002: 0 Zero HE API calls across a full takeover and failback is the invariant the design rests on, and it now has evidence: the 10 gig lease follows the cloned MAC, so the tunnel endpoint is the same address on whichever router holds the WAN and there is nothing to tell Hurricane Electric. This also exercised the proto-41 accept rule added to vyos002 earlier today. Without it the drill would have shown IPv6 failing to return while tun0 was up and radvd running -- which is precisely how the original gap hid. The two directions are NOT symmetric and the doc says so: IPv6 arrived in the same 5s sample on takeover but trailed by 12s on failback, because the reconciler enables bond0.53 first and v6_take only raises the tunnel once the source address exists. Bounded by one 30s tick. Also noted: the drill samples every 5s, so "gap 0s" means within the same sample, not simultaneous -- and the 37s vs the morning's 52s is a different run of the same IPv4 mechanism, not an improvement. Vodafone handed out a new address again across the drill (90.251.142.103 -> 90.251.152.236), corroborating that nothing may be pinned to the PPPoE address. Production returned to normal: vyos001 MASTER on all six with both WANs, vyos002 BACKUP with tun0 down and radvd stopped, force-fault clear. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH |
||
|
Michal
|
da86b60dce |
IPv6 model merged: mark the staging record, and what the merge caught
Some checks failed
kubernetes-deployment@6d4e080 on main, vyos:verify 533/534 zero drift. The merge was not a formality. firewall-accept-he-6in4 was scoped vyos001-only because "the tunnel is anchored to its WAN address" -- untrue once the WAN became HA. vyos002 had no proto-41 accept under its IPv4 input default-deny, so a failover would have brought tun0 up and started radvd on the new master while its own firewall dropped the inbound 6in4. Every other piece would have looked right. Rule 150 added to vyos002. Also recorded: main lives in the .worktrees/grafana-token worktree. The first merge attempt was made on fix/openbao-preview-blockers, 15 commits behind main and predating the PPPoE HA overrides -- it would have reverted them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH |
||
|
Michal
|
d0b733f831 |
IPv6 follows master, deployed: vyos002 has a tunnel and a gate that holds it shut
Some checks failed
Applied step 0 to production, backup first, IPv4 untouched throughout. vyos001 held the VIP and the internet the whole way; verified before and after every commit. Order, which is the safety property: he-secrets onto vyos002 -> vrrp-wan-install on BOTH (so the gate exists before the tunnel) -> tunnel + task-scheduler on vyos002 -> route6 + bond0.9 ::2 + router-advert on vyos002 -> link-mtu 1472 on vyos001. The blackhole is confirmed behaviour, not theory. The moment the tunnel committed, vyos002 brought tun0 UP with source 87.192.101.48 -- an address it does not own -- and the reconciler closed it on the next tick: vrrp-wan: not MASTER: bringing tun0 down vrrp-wan: not MASTER: stopping radvd (deprecates the v6 gateway) Same for radvd: VyOS started it on commit, the reconciler stopped it. Both have stayed shut. That is the override's old "it would simply stay down" assumption failing in production exactly as it failed in the sim. wan-drill --dry now reports the same tunnel source on both routers, where it previously read "<no tunnel -- IPv6 cannot survive a failover>". Two things found by deploying rather than reading: - commit-confirm hangs non-interactively here, exactly as PPPOE-HA.md records. The ssh timed out leaving an orphaned config-mgmt commit_confirm holding the config lock, with nothing committed. Killed it and used plain commit + save -- safe on the backup, which holds no VIPs, no WAN, and whose LoT path is untouched. Worth knowing the failure is clean: no partial config landed. - he-tunnel-follow's status path died with "WAN_MTU: bad array subscript" on the backup. An empty array subscript is a hard bash error, not an empty expansion, so the :- default never applies -- and a backup has no default route, so it broke on precisely the box whose state you most need to read. Fixed and redeployed to both. This is now live DRIFT against the Pulumi model. Until migration/pulumi-override-he-tunnel-both.json is merged into overrides.json, a pulumi up can strip the tunnel, route6, bond0.9 addresses, router-advert and task-scheduler entry. Recorded there and in the backlog's drift section. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH |
||
|
Michal
|
395577850c |
IPv6 was never HA, and the WAN becoming HA is what exposed it
Some checks failed
Reviewed the parked IPv6 task against the PPPoE-HA work of 2026-09-05/06. The
gate that parked it ("WI-8 before IPv6") is cleared, but the same work
invalidated the assumption the IPv6 design rested on.
Verified on the live routers: vyos002 has no tun0, no he-tunnel-follow, no
he-secrets, no VLAN 9 prefix and no route6 ::/0 -- only the pre-staged
default-deny v6 firewall, which is correctly on both. Failover is now automatic
and drill-proven, so every failover takes the whole v6 estate down for as long
as vyos002 holds the VIP.
Four things that came out of checking rather than reading:
- PPPOE-HA.md's "tun0 survived untouched and IPv6 stayed up at 15.5ms" does not
follow from its own premise and is corrected in place. The endpoint address is
stable, but it MOVES to vyos002, which has nothing to decapsulate protocol 41.
wan-drill had no IPv6 check at all, which is why nobody caught it.
- A 22-second near-miss: vif53-pin-boot-disable bounced the 10 gig, he-tunnel-
follow ticked once and saw the PPPoE address, and vyos-failover restored the
route 22s before the second tick would have pointed HE at an address Vodafone
reissues on every dial.
- VyOS does NOT leave a tunnel down when its source-address is absent (the
override's stated reason for leaving IPv6 single-homed). Measured in labsim:
it commits rc=0 and brings the link UP -- a blackhole that attracts the v6
default route. The runtime gate is load-bearing, like the PPPoE gate.
- The RA link-mtu was pinned at 1480 while the tunnel correctly drops to 1472 on
the PPPoE path.
Mechanism, mirroring PPPoE HA -- identical config on both, gated at runtime, no
commit in the failover path:
- vrrp-wan-reconcile: a v6 kernel plane. tun0 and radvd follow the VIP; radvd is
stopped BEFORE the WAN goes so its farewell RA (router-lifetime 0) still has a
path out. The WAN early-exits became if-blocks so the plane runs every tick.
It deliberately does NOT call he-tunnel-follow: that would halve the
hysteresis the near-miss above showed we depend on.
- he-tunnel-follow: a master guard reading the same vrrp-wan.conf VIP, so the
backup copy cannot point HE at its own idle PPPoE line, plus a stubbable
HE_UPDATE_URL.
- vrrp-wan-install carries both, so --check and the upgrade runbook cover IPv6.
- wan-drill measures IPv6 in both timing loops and asserts zero HE API calls
across a router failover.
labsim finally has an HE endpoint, closing the gap the override itself cited as
why this was never rehearsed. Both ISP islands already share the libvirt network,
so that becomes the backbone and HE lives behind it on one address reachable over
either WAN. Proven in the sim: backup tun=DOWN radvd=inactive, master tun=UP
radvd=active, hysteresis then HE call then MTU 1480->1472, and VLAN 9 hosts
autoconfiguring from the RA. The end-to-end v6 datapath is NOT yet proven --
inter-island transit crosses libvirt NAT and the return path is lost. Recorded as
a KNOWN SIM GAP rather than papered over.
The model change is staged, not merged: another agent runs pulumi up on that
repo, and the gate must exist on vyos002 before the tunnel does.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
|
||
|
Michal
|
061b9e3d7e |
Close the last two items: config.boot pinned, and an upgrade runbook
Some checks failed
vif53-pin-boot-disable ran on vyos001. Both config.boots now pin `vif 53 disable` while vyos001's running config keeps the WAN, so a reboot in any order comes up unable to claim the cloned MAC. Cost: a 32s window (10s in the sim -- production commits under kea/BGP/conntrack are slower), bond0.53 re-leased 87.192.101.48 in 5s, and vyos-failover restored the primary route about a minute later. The house rode pppoe0 in between rather than losing the internet. Recorded the shape of that recovery, because I misread it myself: for ~60s after the bounce the default route really is on pppoe0, since vyos-failover only re-adds the bond0.53 route once its probes pass. A fresh `ip route show` in that window looks like a regression and is not one. VYOS-IMAGE-UPGRADE.md is step 9. An image upgrade keeps /config and REPLACES /etc, which silently removes the ppp@pppoe0 gate drop-in -- the only thing stopping the backup from dialling into a single-session account. The reconciler fails closed, so the symptom is "PPPoE never comes up" rather than "both routers dialled", but an upgraded box has no PPPoE until the gate is back. One router at a time, backup first. Deleted the "model hazard to fix before applying" section rather than leaving it: it advised asserting `vif 53 disable` on BOTH routers via an override, which is exactly wrong. That is runtime state owned by vrrp-wan-reconcile, and pinning it would fight the reconciler on every apply and briefly disable the live master's 10 gig each time. Replaced with what is actually done -- follow reality at runtime, hardcode safe at boot and at install time -- and said plainly not to reintroduce it. |
||
|
Michal
|
bda5854563 |
vif53-pin-boot-disable: get disable into config.boot without losing the WAN
Some checks failed
Closes the last open item. The convention is that BOTH routers' config.boot carry `vif 53 disable`, so a reboot in any order comes up unable to claim the cloned MAC and the reconciler enables it on whoever holds the VIP. vyos001's config.boot predates this work and does not. The clean fix does not exist: `save` writes the RUNNING config, not the candidate. Tested in labsim -- set the node, save, discard, and config.boot came back WITHOUT `disable` and the WAN untouched. So the node has to be genuinely disabled, saved, and re-enabled. That costs a real bounce of the 10 gig, but not the internet: pppoe0 is up on the master and the failover route falls to it, which is the T5 path. Measured window in labsim: 10s, with a default route present in every 0.5s sample. Found and fixed a race while testing. The first run collided with vrrp-wan-reconcile's own commit -- "Configuration system temporarily locked due to another commit in progress" -- and the `save` landed while the RE-ENABLE did not, leaving the master with its 10 gig down. The script now takes the reconciler's own /run/vrrp-wan.lock, which the reconciler skips a tick rather than block on, with 9>&- so the config session's unionfs child cannot inherit and hold it. Worth recording that the bad run still ended correctly: the reconciler logged "MASTER with bond0.53 disabled -> enabling" and repaired it in 4s. The failure mode is bounded by design. The script no longer relies on that, but it is why a half-completed run is survivable, and it verifies the re-enable and shouts rather than reporting a success it did not achieve. |
||
|
Michal
|
d08f68e28b |
PPPOE-HA: step 8 done -- model merged, zero drift
Some checks failed
Both overrides are on kubernetes-deployment main (45033dd) and the transition-scripts were applied to the boxes directly rather than left for an unattended `pulumi up` to find. vyos:verify reports both routers in sync. Also records the drill result in the header, since "deployed" and "proven to fail over" are different claims and only the second one is worth much. |
||
|
Michal
|
e1c571d004 |
PPPOE-HA: the drill closed the two biggest unknowns
Some checks failed
Controlled failover in production: takeover 52s, failback 36s, ~88s of interruption across two deliberate transitions. The cloned-MAC lease TRANSFERS. That was named in this file as "the largest untested item in any failover" -- whether the 10 gig ISP would re-issue 87.192.101.48 to f0:9f:c2:12:9b:4f arriving on a different switch port. It did, same address, inside the takeover window. Moved out of "what the sim cannot prove" rather than left there contradicting the evidence. VyOS dialling Vodafone also worked from both routers, which had never been done -- PPPoE was only ever proven on the USG. Vodafone did not refuse either re-dial, so its session-control behaves like `replace`, not the hostile `deny`. GRACE stays at 300 anyway: one drill on one evening is not the ISP's policy under all conditions. Worth knowing for anything added later: Vodafone hands out a DIFFERENT IPv4 on every dial (83.106.5.72 -> 90.251.153.180 -> 90.251.142.103). Nothing may be pinned to the pppoe0 address. Checked the HE IPv6 tunnel specifically, since it carries a hardcoded source-address -- it is pinned to 87.192.101.48, which is the 10 gig and stable across failover, so tun0 was untouched and IPv6 stayed up at 15.5ms. |
||
|
Michal
|
47ce0c1aea |
wan-drill: run the failover drill unattended, because the operator goes offline
Some checks failed
The drill takes the household's internet down, which means anything driving it step-by-step stops being able to act at exactly the moment it matters -- an agent needs the internet to think, so it would freeze mid-failover with the levers half-thrown. This script needs only the LAN, and its cleanup runs from an EXIT trap so both force-fault levers are cleared on any exit path, including being killed. Belt and braces around it: the watchdog on vyos002 stands the box down by itself after 150s holding the VIP with no WAN, so even if the script dies the internet comes back. Refuses to start if the internet is already down or if vyos001 is not the current holder. |
||
|
Michal
|
85819e40c1 |
wan-drill-watchdog: bound the blast radius of a failover drill
Some checks failed
The drill takes the internet down for as long as the new master needs a WAN. If it never gets one, whoever is running the drill is offline too -- and an agent simply stops responding mid-incident. The abort therefore cannot depend on anyone being present. Armed on the router expected to take over: if it holds the VIP and has had no WAN for HOLD consecutive seconds, it sets force-fault on itself, sheds every VIP, and the healthy non-preempting peer takes them straight back. setsid so it outlives the ssh session that armed it, which is the entire point. 150s by default, deliberately shorter than GRACE=300. GRACE is sized for a real hostile-ISP takeover that is still making progress; this is sized for "the drill failed, give the house its internet back". Both paths proven on vyos002 in production: it stayed silent for 35s while a BACKUP (a misfire here would itself cause an outage), and fired within 20s when pointed at an address the box does hold with no WAN, logging "ABORT: held the VIP with no WAN for 20s -- standing down". The first firing test was my own bug, worth noting: sed'ing the default VIP= line does nothing, because vrrp-wan.conf is sourced afterwards and VRRP_WAN_VIP puts 192.168.1.1 straight back. The watchdog was watching an address the box does not hold and correctly stayed quiet -- a test that proved nothing while looking like it proved the feature was broken. |
||
|
Michal
|
13dcdff1ef |
wan-panic: a one-command revert that works with no internet and no Claude
Some checks failed
The WAN now follows VRRP mastership, so a bad failover takes the house offline -- and whoever is debugging it is offline too. The recovery path therefore has to be already on the box, not in a chat log. `/config/wan-panic` works out which router it is running on and does the right thing: on vyos002 it stands down (force-fault), which is what lets vyos001 take the VIPs back; on vyos001 it clears force-fault and says plainly that no-preempt means you must run it on the OTHER box to actually move anything. `status` shows who holds the VIP, which WANs are up and the default route. `undo` stops both timers so nothing moves the WAN again, leaving whatever is currently up exactly as it is. It uses the force-fault lever rather than restarting keepalived because failing the health check is the supported way to shed mastership -- `restart vrrp` is not dependable, since with advert_int 1 the peer declares the master dead in ~3.6s and the restart usually finishes inside that window. Proven in production on vyos002, which carries no traffic: BACKUP -> FAULT in ~20s and back to BACKUP on clearing, with vyos001 untouched and the internet steady at 7.7-8.1ms throughout. The card is installed to /config/RECOVERY-CARD.md on both routers and copied to ~/WAN-RECOVERY-CARD.md, because a card you can only read with working internet is not a card. It leads with how to reach the routers over the LoT leg, which is L2-direct and survives Management/VRRP/routing being broken. |
||
|
Michal
|
8aa3d0ebaa |
PPPOE-HA: record that vyos001's config.boot lacks vif 53 disable
Some checks failed
Runbook step 7 says confirm it on both; it holds on vyos002 and not on vyos001, whose config.boot dates from 2026-09-02 and predates this work. The exposure is new even though the condition is not: if vyos001 reboots while vyos002 is master holding the 10 gig, the cloned MAC is briefly live on both. That could not happen yesterday, because vyos002 was stuck in FAULT and could never be master. It self-heals in <=30s -- vyos001 comes up BACKUP under no-preempt and the reconciler commits `disable` on the next tick -- so this is a bounded window, not a split. Left as an outstanding item rather than fixed, because writing config.boot means `save`ing the running state, so the fix takes the live 10 gig down briefly and belongs in a maintenance window. Hand-editing config.boot would avoid the blip at the risk of an unbootable router, which is worse. |
||
|
Michal
|
8fce03e705 |
PPPOE-HA: deployed to production
Some checks failed
Mechanism installed on both routers (--check clean against git on both), `interfaces pppoe pppoe0 disable` removed from vyos002, and vyos002 is out of FAULT and holding BACKUP on all six groups for the first time in 3d10h -- the lab has a real standby again. Verified on the wire rather than from state: with a tcpdump running on vyos002's bond0.51 across the commit, ZERO PADI/PADR. The peers file rendered (so it can dial the instant it is promoted) while ConditionResult stayed `no` and NRestarts 0. vyos001's live session was untouched throughout -- same MainPID 2931, same 83.106.5.72, internet 7.6-8.4ms at 0% loss. Not done, deliberately: the controlled failover drill, which interrupts the household's internet, and merging the staged Pulumi override, which lands via another agent's `pulumi up` on a branch they have checked out. |
||
|
Michal
|
5ed0e4888a |
labsim: both matrices green end to end, numbers reproduced
Some checks failed
Final confirming run with GRACE=300 and every fix installed:
--all ALL PASS (T0 T3 T5 T8 T11 T12)
--hard ALL PASS (replace / deny / disable, each policy verified)
Second independent measurement of the hard failover, which is what makes the
figures trustworthy rather than anecdotal:
run 1 run 2
replace 26s 26s
deny 148s 141s
disable 21s 20s
`deny` sits at ~141-148s across both, so it is the AC's dead-peer behaviour
and not a one-off; GRACE=300 keeps roughly 2x margin.
Corrected an overclaim in PPPOE-HA.md while confirming it: the invariant row
read "AC never showed two simdsl sessions", and under session-control=disable
it did -- one live, one orphaned from the destroyed router. Only ever one LIVE
router dialled, which is the invariant that matters. Said so plainly rather
than leaving a table that reads better than the evidence.
|
||
|
Michal
|
97efb5abb2 |
labsim: record the failover numbers and how they were nearly wrong
Some checks failed
The three T4 measurements (replace 26s, deny 148s, disable 21s), why deny is the only one that sizes GRACE, and why 148s is a floor rather than a worst case -- idle VMs, and an AC that shares an OVS bridge so `virsh destroy` lets it see the peer vanish in a way a real BRAS never would. Also writes down the broken setter that made the previous numbers fiction, so the next person to touch the matrix knows to check the policy actually landed. |
||
|
Michal
|
6987b324f1 |
vrrp-wan: size GRACE from the measured hostile failover, not the theory
Some checks failed
With the matrix actually setting session-control, T4 timed a destroyed
master's takeover at:
replace 26s
deny 148s <-- sizing case
disable 21s
`deny` is the case GRACE exists for: the AC refuses the survivor until its own
dead-peer timer frees the dead session. The session poller caught it happening
-- the destroyed router's session stayed in the table while the survivor's
dials appeared and were rejected, twice, before one took at 148s.
148s is well past the lcp-echo-interval(30) x failure(3) = 90s budget that 180
was sized against, leaving 32s of margin. GRACE=300 is ~2x the worst observed.
Treat 148s as a floor, not a worst case: these are idle 2-vCPU VMs, and the AC
shares an OVS bridge with the routers, so `virsh destroy` removes the port and
accel-ppp sees the peer vanish. A real BRAS over DSL never learns our router
died and waits out longer timers of its own.
The cost is stated in the conf: GRACE is also how long an alive-but-unroutable
master holds every VIP before yielding. bond0.53 covers most of that -- a DHCP
lease satisfies the check in seconds -- so GRACE only dominates when PPPoE is
the last path. Kept the health check's fallback in step, since keepalived runs
it with no environment and that number decides mastership if the conf is ever
missing.
check_invariant no longer fails blind on the AC's session count. That count is
only a proxy for "two of our routers dialled", and only while the AC enforces
single-session; under `disable` it does not, so a destroyed router's session
lingers and the count reads 2 with exactly one live router dialled. The real
invariant -- at most one router holds pppoe0 -- is now the failing one, and the
stale session is reported as a WARN rather than silenced, because it still
occupies the slot at a real ISP and is precisely what made `deny` take 148s.
|
||
|
Michal
|
9221c71ff0 |
labsim: the session-control matrix was never setting session-control
Some checks failed
T4 prints "--- session-control=deny ---" and then measures whatever policy the AC already had. The setter was isp "vbash -c 'source script-template; configure; set ...; commit; save'" and that form does not start a config session at all -- commit dies with "Invalid command: [commit]" on stderr, which isp() discards. `show configuration commands | grep session-control` on the ISP VM returned nothing after a full matrix run: all three iterations had run against the accel-ppp default. The labels were fiction, and a harness that reports coverage it does not have is worse than one that reports a failure. Driving it from a real script FILE works. isp_session_control() does that, reads the value back, and fails the iteration if it disagrees rather than measuring the wrong policy. `session-control` is a valid node here (checked the template dir on VyOS 2026.08.12-0831-rolling), so this was purely the invocation. Staged the two Pulumi overrides in migration/ rather than adding them to kubernetes-deployment: another agent runs `pulumi up` on that repo, so merging `remove: pppoe0 disable` before the gate exists on vyos002 would let it dial and take the single Vodafone session off the live master. Ordering is written at the top of the file. |
||
|
Michal
|
4d47b609a2 |
PPPOE-HA: record the two failure modes found by running the thing
Some checks failed
Both were invisible to inspection and only appeared under the hard matrix: the flap damper tearing down an established session via the lease it stopped renewing, and a missing peers file being completely silent. The second has a production edge worth spelling out -- an unsaved commit reverts on reboot and takes pppoe0 with it, leaving a standby that can never take over while looking perfectly healthy. |
||
|
Michal
|
b659e0d47e |
vrrp-wan: a flap holdoff must not tear down a live WAN session
Some checks failed
ppp_dial() checked the flap holdoff and returned BEFORE renewing /run/vrrp-wan/may-dial. That lease is what vrrp-wan-guard expires after LEASE_TTL, so tripping the damper stopped the renew and the guard hung up pppoe0 on the MASTER ~80s later. A damper meant to suppress repeated DIALS was tearing down a working WAN instead. Observed in labsim, end to end: DIAL FLAP: >=6 attempts in 600s -- holding off 900s GUARD: lease stale (81s > 75s) -- hanging up pppoe0 An established session now outranks every check below it: ppp_active renews the lease and returns first. Everything after it only decides whether to start a NEW session. Two supporting fixes for how that storm started. The dial attempts were all no-ops because /etc/ppp/peers/pppoe0 was missing, and nothing said so -- systemd logs "skipped because of an unmet condition check" exactly once and the gate looks identical to a healthy backup. ppp_dial() now reports it, and distinguishes "configured but not rendered" (re-commit the subtree) from "no pppoe0 in config at all", which is what a reboot leaves behind when a commit was never saved. That is precisely how the sim secondary lost its WAN. Also `cat | wc -l` rather than `wc -l < file`: redirections are applied left to right, so the missing-file error escapes the 2>/dev/null on every first-ever dial. Harness: T11 copied-then-removed instead of mv, and verifies the restore -- losing that file strands a router permanently, which cost a debugging session. preflight now refuses to run if either router lacks the peers file or the pppoe0 config, since every failover result would otherwise be a false negative blamed on the ISP. New T12 forges a 900s holdoff against a live session and asserts it survives. |
||
|
Michal
|
d626750010 |
labsim: hard failover and reboot safety hold; runbook for the production apply
Some checks failed
Hard failover proven by destroying the master outright (virsh destroy -- no PADT,
the case a graceful stop cannot cover): the survivor took the VIPs, dialled, and
the access concentrator showed exactly ONE simdsl session from its MAC for the
whole seven minutes. When the destroyed router came back it did NOT dial --
ConditionResult=no, ActiveState=inactive, NRestarts=0, pppoe0 absent, may_dial=no
-- and the AC session count stayed at 1. That is the gate doing the one job it
exists for, on the path that previously had no protection at all.
Two more harness bugs of the same family as the last three, both of which
reported a working system as broken:
- ppp_on() returned an EMPTY string for an unreachable router, and `[ "" = 0 ]`
is false, so the hard-failover wait sat for its full timeout waiting for a
DESTROYED box to report zero -- long after the survivor had taken over
correctly. Absent now means 0.
- a VM restart recreates its taps under new names and the OVS bond keeps the
old ones: lacp dies, VLAN 1 goes with it, and the box returns reachable on
some VLANs and not others. That looked exactly like a failed failover. It is
the same stale-membership fault ovs_bond_router already detects, but nothing
ran it after a restart; the harness now does.
migration/PPPOE-HA.md is the runbook: what the design is, why `disable` cannot
work, the measured numbers, the deploy order (vyos002 first, on its own commit,
verified on the wire with tcpdump rather than from state), and the one-line
rollback.
It also records a model hazard found while writing it. The imported baseline
captures the RUNNING state, not the safe one -- vyos001 has no `vif 53 disable`
because it happens to be master, vyos002 does. An apply performed while vyos002
held the VIP would therefore enable vyos001's WAN too, putting the cloned MAC on
both boxes. An override must assert `vif 53 disable` on BOTH, accepting that an
apply then briefly disables the current master's 10 gig until the reconciler
restores it.
Still not applied to production.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
|
||
|
Michal
|
93fed7826b |
labsim: PPPoE HA passes the matrix, and the health check had a real flap bug
Some checks failed
Full run green: baseline (one AC session, held by the VIP holder), clean
failover (pppoe0 moves in 26s, old master releases), 10 gig down (route falls to
pppoe0 and the LAN is back online in 5s), lease expiry (the guard hangs up), and
a missing peers file (NRestarts=0, no loop). Evidence in
labsim/wan-failover-evidence/.
The 10 gig test found a genuine bug in vrrp-wan-health, not in the sim. GRACE
was measured from PROMOTION, so an established master had no grace at all --
after hours of uptime `now - since` far exceeds any window. The first moment
bond0.53 went down while pppoe0 was mid-redial, the master failed its own check,
shed every VIP, and the peer inherited the same WAN outage and did the same. A
brief WAN blip would have flapped the production pair. The stamp is now
refreshed on every healthy tick, so grace measures time since the box last
demonstrably HAD a WAN -- survivable wherever the gap happens, not only just
after a promotion.
Three harness bugs, all the same shape, all of which produced a confident wrong
answer before being caught:
- waiting for "exactly one pppoe0 holder" returns INSTANTLY during a handover,
because it was already true. The useful question is who holds it.
- judging connectivity on a single ping 20s after a link drop reported an
outage that had already healed. Poll, do not sample.
- `-o PreferredAuthentications=password` suits the routers but not the Alpine
LAN VMs, whose sshd offers keyboard-interactive: ssh exited 255 before
running anything and the test read that as "the LAN lost the internet". A
tcpdump on the router showed the pings leaving pppoe0 NATed to
198.51.100.117 and the replies coming back the whole time. An exit code that
can mean "the network is broken" or "I could not log in" is not a
connectivity test, so the check now asserts on what the guest reported.
That last one is why the harness asks the routers and the access concentrator
rather than a client, and why it refuses to run at all while either router still
has a default route via eth2 -- the libvirt-NAT scaffold answers connectivity
checks the WAN under test would have failed.
Still to run: hard failover (destroy the master), and the session-control
replace/deny/disable axis that brackets Vodafone's unknown behaviour and sets
the final GRACE. Nothing applied to production.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
|
||
|
Michal
|
4efd70c987 |
vyos: move PPPoE off the config plane onto a gated systemd unit
Some checks failed
PPPoE HA could not work as written, and the reason is structural rather than a
bug: `set interfaces pppoe pppoe0 disable` and `delete` are handled identically
by interfaces_pppoe.py -- both UNLINK /etc/ppp/peers/pppoe0. That path is pppd's
own options file, so the resting state destroyed exactly what the promotion path
needed, and `ppp@pppoe0` restart-looped against it (observed: 47 restarts, zero
sessions at the access concentrator). It also made op-mode `connect interface
pppoe0` unusable, and put every failover behind a priority-322 commit where one
unrelated invalid node fails the whole thing -- which has already taken the
10 gig down once.
pppoe0 is now configured identically and ENABLED on both routers, so the peers
file always exists, and dialling is gated by a drop-in on the unit:
ConditionPathExists=/run/vrrp-wan/may-dial
ConditionPathExists=/etc/ppp/peers/pppoe0
/run is tmpfs, so the gate is shut at boot and neither box can dial before VRRP
has decided. That matters more than it first appears: with the node enabled,
interfaces_pppoe.py restarts ppp on EVERY commit touching the pppoe subtree when
the daemon is not running -- so the backup actively tries to dial whenever
anything commits. The gate is the only thing making that a no-op, which is why
vrrp-wan-reconcile now refuses to bless a box whose drop-in is missing: /etc is
per-image, and a VyOS upgrade would otherwise silently remove the protection.
may-dial is a LEASE, not a flag. ConditionPathExists is evaluated at start only
-- it can prevent a dial, never revoke one -- so a reconciler that stops running
while its box is demoted would keep the one ISP session for ever. The reconciler
renews the lease; a new 5s vrrp-wan-guard revokes it, and only ever revokes. It
fired correctly first time: "GUARD: lease stale (81s > 75s)".
Also: remove-then-stop on release (the file's absence blocks a NEW start that a
concurrent commit would trigger); a flap damper, because two routers that both
believe they hold the VIP will both dial and each dial kills the other's session
-- against a real ISP that is how an account gets rate-limited; and a guard on
`cfg` returning empty under commit-lock contention, which had already produced
one spurious "releasing" on a box that needed nothing.
GRACE 90 -> 180. accel-ppp's dead-peer budget is lcp-echo-interval(30) x
failure(3) = 90s, so the old value sat exactly on the boundary: a hard failover
into an AC that does not replace the stale session would fail its own check,
shed the VIPs, and leave both routers in FAULT.
The sim could not have tested any of this. Both routers now get the identical
WAN -- the secondary had none "because two PPPoE clients sharing one credential
is a different failure mode than anything production has", which is backwards:
that IS production. It also left the pair incomparable, ten NAT rules against
none. Safety now comes from resting state, not asymmetry.
Three more things the sim was hiding:
- the drift check's secondary regex omitted interfaces pppoe/bonding, nat
source and protocols failover, so it reported "in sync" for a box with no
WAN at all;
- the VRRP health-check and transition-script hooks existed on both live VMs
and in NEITHER generator -- the mechanism under test was pure undetected
drift;
- labsim-vyos's only default route was the libvirt-NAT scaffold, so every
"the LAN still has internet" verdict on it was answered by eth2 rather than
the WAN. --drop-scaffold applied; the earlier DHCP-failover proof is being
re-run because of it.
vrrp-wan-install ends the other half of that: the sim's previous proof came from
scripts hand-`sed`-ed in place, so the tested behaviour was not the committed
behaviour. `--check` now makes that a hard failure.
First green run: master holds both WANs, backup released, and the AC reports
exactly ONE session. sim-net-apply.sh check: all four in sync.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
|
||
|
Michal
|
2e828b8af2 |
vyos: a failover you can actually trigger, and four bugs found triggering it
Some checks failed
A planned failover had no reliable lever. VyOS offers only `restart vrrp`, and
neither that nor `systemctl restart keepalived` is dependable: with advert_int 1
the peer declares the master dead after ~3.6s and a restart usually finishes
inside that window. Measured -- the same command moved mastership on one run and
not on the next three. A fail-back step you cannot trigger on purpose is not a
procedure, and the recovery card depended on one.
The lever is now `touch /run/vrrp-wan/force-fault`: the health check fails, the
sync group sheds every VIP, and the peer takes over. It exercises the same path
a real WAN loss takes rather than a special case, and it lives in /run so a
reboot cannot leave a router permanently ineligible.
Proven end to end in labsim: lever -> mastership moves -> WAN follows -> the old
master releases -> a LAN VM has internet -> the faulted router returns to BACKUP
and is eligible again.
Getting there exposed four real bugs, two of which would have broken a GENUINE
failover, not just the drill:
- The grace stamp was written only by the 30s reconciler, so a freshly
promoted master reached the 5s health check with no stamp, scored grace = 0,
failed instantly and went FAULT. With the peer already faulted that left
BOTH routers in FAULT and the LAN with no gateway at all -- worse than the
outage the check exists to prevent. The check now stamps on promotion.
- And it inherited STALE stamps from an earlier mastership, failing ~5s after
passing. The stamp is now cleared on the way down, by the health check
itself, not only by the reconciler.
- The lock fd leaked into VyOS's config session: `exec 9>` is inherited by the
long-lived unionfs-fuse the session spawns, which never closes it. From the
first config change on, every later reconciler run lost the flock and exited
0 having done nothing -- healthy-looking journal, silently stopped
reconciling. That is how a demoted router kept the WAN. Children now get 9>&-.
- vrrp-wan-apply touched pppoe0 unconditionally. On a box where pppoe0 has no
source-interface VyOS rejects the whole commit ("Physical source-interface
required"), taking the bond0.53 change down with it -- and the script still
returned 0, so the reconciler logged a release that never happened. pppoe0 is
now guarded on existence and the commit's verdict is propagated.
Still NOT applied to production. The pair is single-homed on WAN until it is.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
|
||
|
Michal
|
7bf3f42e19 |
vyos: WAN follows VRRP mastership, rehearsed in labsim
Some checks failed
The ISP is consumer with one static IP, so "both routers hold WAN" is not
available: the 10 gig lease is anchored to a cloned MAC and the PPPoE line to a
single credential. The WAN therefore has to move with mastership.
Proven end to end in the sim: the secondary was promoted, took the WAN, got a
lease, installed a default route, and a LAN VM reached the internet through it
(3/3, 9ms). Demoting released it -- link down, no address. Rebooting the master
converged correctly too: it came back BACKUP with the WAN disabled while the
peer kept it.
The rehearsal earned its keep four times over, and none of these were visible
from reading the docs:
- `transition-script` alone is NOT safe to hang internet on. VyOS delivers it
through keepalived-fifo.py, and on one promotion that helper logged NOTHING
while Keepalived_vrrp logged all six instances entering MASTER and the
built-in notify_master for conntrack-sync ran normally. The result was a
router holding every VIP with no WAN -- the 2026-09-02 outage, recreated by
the mechanism meant to prevent it. Hence vrrp-wan-reconcile on a 30s timer:
the scripts give speed, the timer gives correctness.
- The health check must ask REALITY, not a marker. Keying "am I master" on a
/run file written by the transition script meant that when the script did
not run, the router believed it was backup, passed the check, and kept the
VIPs it could not serve. It now asks whether the VIP is actually on the box.
- The old address-based check DEADLOCKED this design: may-I-be-master required
already having WAN, and only the master gets WAN. That is why vyos002 sat in
FAULT for ever -- the safety check had silently removed the redundancy it
existed to protect.
- script-template must be the FIRST thing a script does. Sourced after an if,
an exec and a mkdir it terminated the script inside the source, rc=0, no
output: the reconciler reported success having done nothing. Hence the split
into vrrp-wan-apply, matching the shape /config/vyos-known-good already uses.
Two hazards found and handled rather than discovered in production:
- A `configure` session whose process dies leaks a unionfs mount under
/opt/vyatta/config/tmp, and one of those holds the commit lock -- after
which every commit fails, including the manual one you try to fix it with.
A 30s job that can leak one per failure wedges the box on its own, so the
reconciler reaps dead sessions before it starts. It cleared 8 on the sim.
- Any `save` while a box is master persists the enabled WAN into config.boot,
so a reboot would claim the shared MAC regardless of VRRP state. Observed:
an ordinary console-apply did exactly this. config.boot must keep `disable`
on BOTH routers; the model asserts it and vyos:verify reports it as drift.
NOT yet applied to production, and it should not be until the remaining item is
settled: a clean, deliberately-triggered failover has been seen via reboot, but
`restart vrrp` twice failed to move mastership at all, so the trigger for a
planned failover is still unproven.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
|
||
|
Michal
|
09ede73b67 |
migration: the watcher that caught vyos002, and what it found
vyos002-catch.sh is armed before power-on and strips the eth2 address the moment SSH answers -- 6 seconds, where a human watching a console loses that race more often than not. It commits eth2 on its own before doing anything else, because every extra command in that commit is extra exposure. Caught at 15:35:22 on the LoT leg. eth2 ended with no address and the sync-group health-check installed; it exits 1 (WAN disabled on this box), so all six VRRP groups sit in FAULT and it holds no VIPs at all. That is the protection that was missing on 2026-09-02, working as intended rather than as a theory. The bounded-risk note in the header is the part worth keeping: it comes up BACKUP behind a healthy vyos001, so it never holds 192.168.8.1 and the GATEWAY cannot be poisoned during the window. The unbounded case is it becoming MASTER with eth2 present, which the health-check now makes impossible. It also surfaced that vyos002 had booted from a saved config predating the day's work -- stale reservations including the two that hand the routers' own eth2 NICs 192.168.8.143/.144. Synced to vyos001 and imported; see kubernetes-deployment 7f92974. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH |
||
|
Michal
|
a973b51b9c |
migration: vyos001 converted in production, and the config vyos002 needs first
UniFi does offer Native VLAN = None (7.5.10), and its dropdown lists Management as VLAN 1 -- the API reports vlan:null, which is what made this look like a blocker. The UI was right. Applied to USW Aggregation ports 1+2, then the router over ssh vyos@10.0.1.252. Confirmed by kea's own log: "the interface bond0 has no usable IPv4 addresses configured" -- it opens no socket on the parent, so there is no wrong pool left to answer from. VRRP held mastership with no transition; the VIPs never moved. Proven end to end by the thing that was broken: vyos002's JetKVM console, which had been sitting on a VLAN 3 port holding Management lease 192.168.1.28, was restarted and took 192.168.3.14 -- one lease, right pool, right reservation. That console is what unblocks vyos002. commit-confirm cannot be driven non-interactively: `vbash -s` hangs on its prompt. It failed safely (candidate discarded, nothing committed) but the recovery card's commit-confirm advice only works typed by hand. vyos002-return.conf carries the two defects that must not survive its next boot. eth2 is US24 port 16, native VLAN 2 -- disable that port before powering the box on and the ARP hazard is gone before it can happen. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH |
||
|
Michal
|
d727a50ca0 |
migration: offline recovery card, and stop the leak test trusting a silent router
The change itself takes Management down, so the session that performs it has no
internet and no Claude. RECOVERY-CARD-vlan1-move.md is what is actually needed at
that point, on one page: the recovery path, the order, the commands, and what to
do when locked out.
The recovery path is `ssh vyos@10.0.1.252`, and the reason it is trustworthy is
that it is not routed -- the workstation is 10.0.0.210/23 and the router's LoT
leg is 10.0.1.252/23, so `ip route get` returns dev lanbr0 with no `via`. It
therefore survives Management, VRRP, the VIPs, DNS and the switch trunk all being
wrong at once. bond0.10 is untouched by the change and stays in the LAN group.
Order is switch-first, which is the opposite of what seems natural. The UniFi
controller is 192.168.1.5, on Management, reached from LoT *through vyos001*: do
the router first and you lose the controller you still need for the switch.
Two corrections to the leak test, both because it reported a router fault that
was its own:
- it counted `pgrep -f 'tcpdump -i bond0'`, so a stray tcpdump from an earlier
run satisfied the >=2 guard with none of this run's captures alive. A capture
that records nothing reads as "the router sent no reply at all".
- it believed a single silent run. Kea can be is-active and answering nothing
for tens of seconds after a restart, so the first VLANs of a loop failed and
the last passed. That produced two OPPOSITE and equally wrong conclusions
about `listen-interface` before a retry showed the pattern.
On `listen-interface`: it is a real second branch in kea-dhcp4.conf.j2 that keeps
dhcp-socket-type raw, unlike listen-address which forces udp and took DHCP down
when it was tried live. But naming the sub-interfaces explicitly left Management
DHCP dead in the sim, reproducibly, against a clean control with interfaces:["*"].
Not adopted, and not needed -- the address move is the proven fix.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
|
||
|
Michal
|
0481c38e09 |
labsim: prove the tagged-Management fix for kea's wrong-pool offers
Kea #1117: with dhcp-socket-type raw, a frame tagged for a sub-interface is also delivered to the parent's AF_PACKET socket, and if the parent serves a subnet kea answers from it too. Management being the native VLAN on bond0 is what gives the parent that subnet. One DISCOVER on VLAN 3 produced two OFFERs, and in the captures here the WRONG one arrives first as often as not -- which is why this looked device-dependent rather than like a server bug. labsim-vlan-leak-test.sh reproduces it and scores the SERVER's offers, not the client's choice; a client picking correctly is how this hid. Fails on the old shape, passes on the new one across all six LAN VLANs. Three things the rehearsal caught that reasoning had not: - kea keeps its old raw socket. VyOS does not restart it for an interface address change, so the first post-fix test failed and looked exactly like the fix not working. - interface-group LAN names the bare bond0. Moving the address without moving the group drops every management session under default-deny. - there is no make-before-break. A port always egresses its native VLAN untagged, so while VLAN 1 is native the router can send tagged VLAN 1 but never receive it -- verified, the ARP landed on bond0 untagged. What makes the cutover safe anyway is that tagged and untagged Management coexist, so the firewalls convert one at a time: 0s of VIP downtime, versus 5m30s if both routers go before the switch does. In that state the healthy BACKUP does NOT take over -- the sync group holds native BACKUP because the other VLANs still hear the master. Also fixes two ways the sim was lying. ovs_bond_router compared only the trunk VLAN list on re-runs, so a VM restart left the bond holding taps that no longer existed while the real ones sat in the bridge unbonded -- labsim-vyos2 had no LACP at all. And the tap count included the primary's libvirt-NAT scaffold NIC, so the primary's bond was skipped outright. Runbook: migration/MANAGEMENT-VLAN-TAGGED.md Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH |
||
|
Michal
|
23783b8486 |
vyos: VRRP health check so the WAN and the gateway VIP cannot separate
Outage of 2026-09-02: vyos002 held every floating gateway IP while vyos001 held
the only working WAN. The LAN had a gateway that could not reach the internet,
and it stayed that way until vyos002 was powered off by hand.
Three causes, none of them bad luck. VRRP had no health check of any kind, so
mastership was decided purely on whether the peer was still advertising and
never on whether this router could route. vyos002 structurally cannot route --
bond0.53 is `disable`d because the 10 gig lease is bound to a cloned MAC that
only one box may hold, and pppoe0 is not up. And `no-preempt` is set on every
group, so once vyos002 took master it kept it even with a healthy priority-200
peer sitting next to it.
Reproduced exactly in labsim: stopping keepalived on the primary moved every VIP
to the WAN-less secondary and LAN internet went from 9ms to 100% loss; restarting
the healthy primary left it BACKUP and the outage in place. Applying this check
self-healed it -- secondary to FAULT, primary to MASTER, internet back.
The check asks "do I have an address on a WAN interface", deliberately not "can
I reach the internet" and not "do I have a default route". During a real ISP
outage the default route disappears on BOTH routers; keying on that would put
both in FAULT, nobody would hold the VIPs, and an internet outage would become a
total one.
Config goes on the SYNC GROUP, not per group: VyOS refuses a per-group check
while the group is in a sync group ("Only sync group health check will be
used"), and sync-group scope is what we want anyway so all VIPs move together.
Known cost, measured: with the primary genuinely dead the secondary stays FAULT
and NOTHING holds the gateway, so inter-VLAN routing stops too. That is the
honest consequence of a backup that cannot route. The fix for it is to make the
WAN follow mastership so the backup CAN route -- next, and rehearsed separately,
since it is the one change that can lose the DHCP lease.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
|
||
|
Michal
|
11344eab92 |
labsim: the IPAM switch needs no pod recycle when the CIDR sources agree
Re-ran the 3-node rehearsal, this time from a state that matches production rather than one I had accidentally skewed. The earlier "two nodes swapped CIDRs" result was an artefact of my own setup: labsim had been running cluster-pool with per-node CIDRs that differed from node.spec, I flipped it to ipam=kubernetes (which resyncs CiliumNode from node.spec), and flipping back therefore looked like a renumber. Production has only ever run ipam=kubernetes, and all five nodes were checked: CiliumNode and node.spec agree everywhere. From that matching state the switch is close to a non-event: per-node CIDRs unchanged, every pod still inside its node's range, nothing stranded, no recycle required. The only disruption is the cilium DaemonSet restarting itself -- one agent sat in Init:0/6 and one node briefly took the agent-not-ready taint, both of which cleared on their own. Cross-node connectivity verified after. So the recycle is CONDITIONAL, not a fixed step, and `verify` is what decides. Documented both cases in the script header, because the dangerous one is silent: stranded pods report Running and Ready while being unreachable. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH |
||
|
Michal
|
e8679f45b5 |
labsim: rehearse the IPAM switch on 3 nodes, and catch the trap in it
Rehearsed kubernetes -> cluster-pool on the 3-node labsim cluster, which is the transition production faces. The switch itself is undramatic: agents stayed up, the operator adopted the pool, and the agent-not-ready taint deadlock did NOT occur. That deadlock is specific to ADDING IPv6 -- the agent blocks on an IPv6 pod CIDR that does not exist yet. A v4-only mode switch does not hit it. The real hazard is quieter. The operator does not preserve which node held which /24: two nodes swapped CIDRs. Their existing pods kept their old addresses, which now fall outside the node's range, so every other node routes that prefix to the wrong node. Cross-node ping to those pods dropped 100% while every pod stayed Running and every node stayed Ready. Nothing in `kubectl get pods` shows it. So "pods kept the same address" is the FAILURE signal here, not the reassurance it looks like. cilium-ipam-switch.sh verify now flags pods sitting outside their node's CIDR, which is the check that decides whether a recycle is optional (it is not) or mandatory (it is). Recycling every deploy/ds/sts restored it: all pods back inside their node CIDR, cross-node ping 0% loss. Sequence proven end to end: preflight -> apply -> restart operator then agents -> unstick if needed -> recycle all workloads -> verify Also fixed the recycle hint the script printed: `kubectl rollout restart deploy,ds,sts -A` is not valid (`unknown shorthand flag: 'A'`), so anyone following it under pressure would have got an error instead of a recycle. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH |
||
|
Michal
|
527e0798ae |
bastion/k3s: bootstrap new clusters with cluster-pool IPAM
Every cluster this bastion builds was pinned to ipam=kubernetes, which makes it
permanently single-stack: Cilium reads node.spec.podCIDRs, the controller-manager
writes that once at node join and never revises it, so adding IPv6 later fails
with `required IPv6 PodCIDR not available` and needs a rebuild.
cluster-pool puts allocation in the CiliumNode CRD, where the operator can add
an address family to a running node. Proven in labsim (
|
||
|
Michal
|
842408c0d9 |
labsim: prove k3s CAN be converted to dual-stack in place
k3s documents that dual-stack "cannot be enabled on an existing cluster". Rather
than accept that for a 145-day-old production cluster, build both shapes and
diff them. dualstack-lab.sh builds a single-node IPv4 cluster and a native
dual-stack one, takes a reflink copy of the IPv4 disk so a failed conversion
costs 90 seconds to undo, converts in place, and diffs the results.
Result: the conversion works. etcd data is never touched.
The documented blocker is real but narrower than stated. Cilium reports it
exactly -- `required IPv6 PodCIDR not available` -- because node.spec.podCIDRs
is assigned at join and is immutable, and the Kubernetes IPAM controller will
not add a second family later. That objection only holds while Cilium runs
ipam=kubernetes and therefore reads that field. Switching to cluster-pool IPAM
moves pod CIDR allocation into the CiliumNode CRD, where the operator hands out
both families on a cluster that was born IPv4-only.
Sequence that works, in order:
1. k3s unit gains --cluster-cidr/--service-cidr/--node-ip with both families.
k3s validates the two CIDRs together and refuses to start on a mismatch
("must share the same IP version"), so a partial edit crash-loops rather
than coming up half-configured. That is the safe failure mode.
2. The ServiceCIDR object picks up the IPv6 range on restart -- this is
upstream's supported "single-to-dual-stack preserving the primary
ServiceCIDR" path, and existing Services keep their IPv4 addresses.
3. Cilium to ipam=cluster-pool with an IPv6 pool, then DELETE the CiliumNode
so the operator reallocates; it will not add a family to an existing one.
4. Expect a deadlock here: the agent will not go ready without a pod CIDR, so
the node keeps the node.cilium.io/agent-not-ready taint, so the new
operator that would assign the CIDR cannot schedule. Remove the taint by
hand once to break it.
Verified on the converted cluster: pod with 10.42.0.125 AND fd00:42::4843, and
a PreferDualStack Service holding 10.43.122.115 AND fd00:43::72a3.
The only field that still differs from a native build is node.spec.podCIDRs,
which stays IPv4-only -- immutable, and unused once Cilium owns IPAM. CiliumNode
podCIDRs and ServiceCIDR are identical to the native cluster.
Not yet answered: this is one node. Whether a 3-server etcd cluster converts as
cleanly, and what a rejoining agent does, is the next experiment.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
|
||
|
Michal
|
ad6eb7a9a6 |
labsim: default-deny firewall policy, proven in the sim
Some checks failed
Policy: internal VLANs reach each other and the internet; the internet initiates nothing inward. That was already the effect of the IPv4 ruleset, but built as a blacklist -- default-action accept plus explicit drops per WAN interface. Identical behaviour right up until a WAN is added, at which point it is open and nothing looks wrong. This expresses it as a whitelist. Two findings from the sim, both of which would have been outages in production: `set` on a rule number is ADDITIVE. The sim already had a rule 10 carrying inbound/outbound interface constraints; `set ... rule 10 state established` ANDed onto it, producing a stateful-accept that applied to one interface pair only. Return traffic from the internet then matched no rule and hit the default drop, so LAN hosts could reach nothing outbound. The generator now deletes each filter before rebuilding it, so the code owns the subtree. It is one commit, so nftables is rebuilt atomically -- there is no window without a firewall. DHCP lease renewal is unicast UDP to port 68 and conntrack does not reliably cover it. Without an explicit rule the WAN keeps working until the lease expires and then dies -- a delayed failure that looks nothing like a firewall change. Also added a loopback accept for both families, absent from the v6 policy since it went default-deny. Verified in labsim: inter-VLAN ok, LAN-to-internet ok, internet-to-router dropped, and internet-to-LAN dropped with the drop counter incrementing by exactly the packets sent, after routing the test through the router rather than around it via the hypervisor. Also extends the drift check to the firewall subtree, which it did not cover -- so it had been reporting "in sync" while that subtree was uncaptured. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH |
||
|
Michal
|
7f551081ad |
labsim: capture BGP, dual WAN and both ISP VMs as code
The sim's routing config existed only as running state on the VMs. It was applied by hand over SSH, so rebuilding a VM lost the rehearsal and nothing recorded why any of it was shaped the way it was. The two ISP VMs were not referenced anywhere in the repo at all. sim-net-config.py generates all four roles; sim-net-apply.sh applies them over the serial console, or diffs them against the running VMs. Verified reproducing live state exactly before committing: primary 40/40 commands, secondary 16/16, isp-dhcp 19/19, isp-pppoe 21/21. Carries the reasoning that was previously nowhere: RFC 8212 needing policy in both directions or the session carries zero prefixes; probe targets that must not double as system name-servers; default-route-distance 210 rather than no-default-route, which blanks new_routers and hands the default route to the backup line; and the WI-8 bootstrap bug that pinned /32s fix. Dropped a stale `pppoe-server interface eth0` on isp-pppoe (a NIC that does not exist there) so a green drift check stays meaningful. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH |
||
|
Michal
|
f41ffdd039 |
feat(vyos): reconciler that keeps the HE 6in4 tunnel on the live WAN
Some checks failed
Kernel-level (`ip tunnel change`), not VyOS config: no commit churn on a flapping line, no drift against the Pulumi model, and a reboot restores config.boot — which pins the 10 gig — so a wrong source cannot survive a restart. Sends `myip` explicitly, because mid-failover the update request may egress either line and letting HE infer the address would point the tunnel at the WAN we just left. Requires two consecutive agreeing runs before acting, since HE rate-limits updates and a flapping WAN would hammer the API precisely when it matters. MTU moves with the WAN: 1480 on the 10 gig (1500-20), 1472 on PPPoE (1492-20). Fixed at 1480, the backup path gives the signature people lose a day to — small packets fine, large transfers hang. Inert without /config/he-secrets, and a no-op when already in sync. |
||
|
Michal
|
a187703a3a |
feat(vyos): pin a known-good config and restore it with one command
Some checks failed
VyOS already has rollback, but `rollback 1` returns you to the *previous*
revision, which may itself be broken — you can end up walking backwards through
several bad commits hunting for the one that worked, at exactly the moment you
have no network to look things up with. This pins a state a human has actually
used and found working, so recovery is one step and needs no memory of how many
changes ago things were fine.
/config/vyos-known-good save pin the running config
/config/vyos-known-good status when it was taken, how running differs
/config/vyos-known-good diff what a restore would change
/config/vyos-known-good restore go back to it
Deliberately not automatic. A config is only known-good once someone has used
the network; a snapshot taken after every commit would faithfully preserve the
broken one.
The restore is itself commit-confirmed, so even the recovery path is protected:
if the snapshot is somehow wrong, or access is still broken and nothing can be
confirmed, the router undoes the restore rather than leaving you worse off.
Silence reverts.
`save` refuses when there are uncommitted changes — a snapshot that did not
match what is actually running would look like a safety net without being one.
Two things found while building it, both of which made the script silently
useless rather than fail loudly:
- Sourcing `script-template` **resets the positional parameters**, so `$1` was
empty by the time the case statement ran and every invocation fell through
to the usage message. Arguments are captured before the source.
- `0600` made the snapshot unreadable to the `vyos` user, so `status` and
`diff` — the two commands you run while deciding whether to restore — showed
nothing. Now 0660 root:vyattacfg, matching /config/config.boot.
Installed on both routers with the current, verified-working config pinned
(vyos001 1029 lines, vyos002 1019).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
|
||
|
Michal
|
86c2a36f00 |
feat(labsim): a real Kubernetes cluster for rehearsing Cilium <-> VyOS BGP
Some checks failed
BGP is about to be added to a network that currently works, where a bad advertisement blackholes the house. That needs somewhere to fail first. Three Debian nodes (4 GB / 2 vCPU) on the OVS `vlan2` access ports running k3s with flannel disabled, so Cilium is the CNI under test. Three rather than two because ECMP is only meaningfully tested if a node can be drained and more than one path survives. VMs rather than k3d: the thing under test is eBGP between Cilium and VyOS across the switch fabric, nodes peering with the router's bond0.2 leg, directly connected. k3d would put the nodes on a container bridge -- a different L2 path, proving something else. (It also wants Docker; this host has podman.) The existing micro VMs are Alpine with 256 MB and 1 vCPU, which is not close to enough. Debian rather than the sim's Alpine base: glibc, a stock kernel that Cilium's eBPF probes are tested against, and cloud-init that actually applies network-config -- the Alpine base notably does not. One trap worth recording. An earlier draft called `selected_vlans "$K8S_VLAN"` before `ovs_up`, and since `ovs_up` re-defines the libvirt network from SELECTED, that silently deleted the portgroups for every other VLAN. Running VMs kept working -- their taps were already attached -- so nothing complained until the ISP VMs needed vlan51 and vlan53 and could not be attached. It now selects every VLAN. The generated kubeconfig is gitignored: it carries cluster-admin credentials and is one `git add -A` away from being committed. Regenerate with `k8s-up.sh --kubeconfig`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH |
||
|
Michal
|
27a343bc75 |
Merge branch 'feat/unifi-export-and-vyos-dhcp': USG to VyOS migration
Some checks failed
Carries the UniFi export tooling, the generated VyOS DHCP/DNS config, the reversible cutover switch, the health-checked WAN failover, and the labctl side of applying a Pulumi-rendered bundle at install time. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH |
||
|
Michal
|
672b89ce38 |
feat(labctl): install VyOS from a Pulumi-rendered bundle, and enable its API
Two halves of the same problem: a router should come up running the config that is declared for it, and should be manageable the moment it does. --vyos-bundle applies a bundle rendered by kubernetes-deployment verbatim, replacing the derived --vyos-bond/--vlan/... path rather than merging with it. Deriving a second opinion alongside a bundle is exactly the drift the bundle exists to prevent: Pulumi and labctl would each believe they knew the router's config and the box would end up with whichever ran last. Passing both is rejected rather than silently resolved. Secret-valued nodes arrive as @secret: sentinels and are dropped, with a warning naming each one. Writing the sentinel text into config.boot would look configured while being wrong, which is worse than being absent -- the router comes up without its PPPoE credential and the first `pulumi up` supplies it. A bundle committed to git has to stay safe to read. The hostname is forced to the one the install was asked for. A bundle is exported from one router and reused for its peer, and taking the hostname from it would put two vyos001s on the network. --vyos-api-key enables the HTTP API at install, on both the bundle and the derived path, so every VyOS this bastion provisions is manageable from first boot. vyos001 and vyos002 predate this and had to be enabled by hand on a live firewall after their cutover -- which is the gap this closes. It is deliberately not part of the Pulumi model: a provider able to rewrite its own transport can revoke its own access. listen-address is always set, and the API is NOT enabled when no address is known -- under DHCP there is none at build time, and binding to every interface would publish a config-write endpoint on the WAN. It warns and leaves the router SSH-only instead. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH |
||
|
Michal
|
f4984e3962 |
fix(vyos): health-check the 10 gig primary so failover actually fires
The 10 gig line was primary by route distance alone, which only fails over when bond0.53 loses carrier or its DHCP lease. An ISP that keeps the link up while dropping traffic -- the common failure -- would black-hole everything, because a DHCP-installed route has nothing to withdraw it. `protocols failover` now owns the live default route and pings two targets bound to the interface, so the backup can never be validated through the primary's path. Rehearsed on the labsim router: failover and failback both inside 5s with the router's own interface still UP. The vif keeps default-route-distance rather than no-default-route, demoted below Vodafone. vyos-failover resolves a dhcp-interface gateway by reading new_routers out of /run/dhclient/dhclient_<if>.lease, and no-default-route leaves that field EMPTY -- the daemon then finds no next hop and installs nothing. Observed on vyos001: the default route fell through to Vodafone. Preference is now failover's kernel route (distance 0) > pppoe (10) > DHCP (210), so the demoted route can never re-create the black hole it exists to avoid. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH |
||
|
Michal
|
7b5331ddcd |
fix(migration): the backup has no WAN by design; stop failing it for that
The cutover succeeded on vyos001 -- gateway live, bond0.53 holding 87.192.101.48 with the cloned MAC, kea serving, clients routing out through NAT. vyos002 then ran the same script and was judged unhealthy, because the mandatory checks are "default route exists / internet reachable / DNS resolves" and the backup deliberately holds its WAN interfaces DOWN. Its config was correct; the check did not apply to it. Confirmed by hand before the timer could revert a good config. This is the third instance of one mistake: asserting a condition that is not true of the box being checked. First requiring every WAN when one suffices, now requiring a WAN on the box that is configured not to have one. A delta containing `interfaces ... disable` for the WAN now identifies the backup, and the WAN-dependent checks are skipped with a note. kea and the DNS forwarder remain mandatory on both -- those are what the backup must actually be able to do. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH |
||
|
Michal
|
ce6911c196 |
fix(migration): require a working WAN, not every WAN
This reverted a cutover that had actually succeeded.
The evidence, from the revert tearing it down:
dhclient: DHCPRELEASE of 87.192.101.48 on bond0.53 to 185.232.119.244
vtysh: "no ip route 0.0.0.0/0 87.192.96.1 bond0.53 tag 210 1"
netlinkd: RTM_NEWLINK -> bond0.53, mac=f0:9f:c2:12:9b:4f
bond0.53 came up with the cloned MAC and was handed 87.192.101.48 -- the exact
public address the USG holds -- with a default route via the real ISP gateway.
kea was serving live LAN clients at the same moment (10.0.0.12, 10.0.0.13,
192.168.8.28). The gateway was working.
The only failure was pppoe0: ppp@pppoe0.service exited 5/NOTINSTALLED. That is
the Vodafone FAILOVER line, and the health check listed "pppoe0 has an address"
as mandatory, so a working gateway was torn down because its backup WAN was
down. The check encoded "every WAN must work" when the requirement is "the box
must reach the internet".
Now: default route, reachability and DNS are mandatory; each WAN interface is
reported individually but fatal on neither. A failover line being down is worth
seeing, not worth reverting for.
This also incidentally settles the last genuine unknown in the migration, which
could not be tested any other way: the ISP does hand the same lease to the
cloned MAC. That was the one thing I had said was unknowable until the USG let
go of it.
Note the earlier polling fix (
|
||
|
Michal
|
54b21fa9ff |
fix(migration): poll for WAN health instead of sampling once at 25s
A real cutover attempt reported failure and reverted a configuration that may well have been fine. The health check waited a fixed 25 seconds and then judged: [switch] committed. Waiting 25s for PPPoE and services to settle... [switch] FAIL pppoe0 has an address 25s is far too short for a WAN. PPPoE alone is PADI/PADO/PADR/PADS followed by LCP, authentication and IPCP -- routinely 15-30s on its own. Both lines had also just been released by the USG seconds earlier, and ISPs commonly hold the previous session and MAC binding for minutes before leasing to the "same" CPE again, which is exactly what a cloned MAC looks like from their side. The one thing the design could not tolerate was being impatient, and it was. Now polls every 15s up to HEALTH_BUDGET (default 180s), reporting progress, and stops early the moment everything is healthy. The budget deliberately finishes long before commit-confirm fires -- 180s against a 10 minute timer leaves 420s of margin -- so the decision to confirm or revert stays ours rather than being made by the timer. Also recorded while chasing this: the earlier claim that VLANs 51/53 are not trunked to the firewalls was WRONG, and the UniFi port settings disprove it -- those LAG ports are Native VLAN Management (1) with Tagged VLAN Management set to Allow All. My evidence never supported the claim: a passive RX count cannot distinguish an absent VLAN from a quiet one, because switches do not flood unicast, and the active DHCP probe used a random MAC that an ISP binding to its registered CPE would ignore regardless. Both observations fit a perfectly healthy trunk. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH |