main
58 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Michal
|
ac8653f519 |
IPv6 addressing phase closed: all five nodes bound, aitopatom included
Some checks failed
aitopatom-3a1c (the DGX that refused my key) reached via michal@ with passwordless sudo, took its reserved ::27 immediately on the eui64 flip. All five cluster nodes now hold their exact VLAN 2 reservations, 5/5 Ready. Node addressing is done and enforced (provisioning default + fleet drop-in). What remains for IPv6 is the egress flip and the cluster conversion, both attended. Docmost IPv6 page updated to match. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH |
||
|
Michal
|
7c2cbfaf31 |
ROOT CAUSE found: EUI-64 vs stable-privacy link-local, not arch or DUID
Some checks failed
Michal pushed back on "arm64 nodes behave differently" -- correctly. worker2 is aarch64 and binds fine, so architecture was a coincidence. Chased it to the real cause, proven by the kea ALLOC_ENGINE log. The subnet is reservations-only, no dynamic pool. Every node sends DUID-UUID (no MAC), so kea can only match the hw-address reservation by deriving the MAC from the source link-local -- which works ONLY for EUI-64 link-locals. worker0/worker2 have ipv6.addr-gen-mode=eui64; worker1/spark use the default (stable-privacy, RFC 7217), whose link-local embeds no MAC. kea derives nothing, no reservation matches, no pool exists to fall back to: ALLOC_ENGINE_V6_ALLOC_FAIL_NO_POOLS: no pools were available So the "MAC reservation" scheme is really a link-local-EUI-64 scheme, and only works where every node uses EUI-64 link-locals -- not the modern NM default. Fix options (attended) written up in the evidence file: enforce addr-gen-mode=eui64 fleet-wide (smallest, keeps one source of truth), DUID keys, or dynamic pool + discovery. This is the keying decision the plan flagged, now with a precise cause behind it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH |
||
|
Michal
|
3e43385639 |
window 2026-09-06: final state + handoff note
Some checks failed
VLAN 2 IPv6 live on both routers and in the model. IPv4 untouched (6 MASTER / 6 BACKUP, default route on bond0.53, watchdog 0 reverts, 5/5 Ready). worker0 and worker2 hold their reserved ::23/::25; the two arm64 nodes are the open item. Watchdog disarmed. Full handoff at the top of window-evidence/2026-09-06-final.txt. vyos002 showing internet DOWN is the correct resting state of the gated backup, not a fault. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH |
||
|
Michal
|
d9f74aa294 |
CORRECTION: MAC reservations DO work -- I measured too early and said otherwise
Some checks failed
Commit
|
||
|
Michal
|
c392bb9233 |
window: option (b) mac-sources has no VyOS knob; noted for the attended decision
Some checks failed
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH |
||
|
Michal
|
3c933b96e0 |
VLAN 2 IPv6 applied to production: RA proven, MAC reservations do NOT match
Some checks failed
The unattended window's W1/W2. Both routers now carry the VLAN 2 IPv6 config (addresses ::1/::2, RA with managed-flag + no-autonomous-flag + link-mtu 1472, DHCPv6 reservations for all five nodes), applied via migration/vlan2-v6-apply with a vlan2-v6-watchdog armed on both routers throughout. IPv4 untouched: internet up, 6 MASTER / 6 BACKUP, 5/5 nodes Ready, watchdogs quiet. default-lifetime 0 on purpose -- addressing without egress. Turning on v6 egress moves image pulls onto a tunnel of unmeasured throughput, and the person who would notice is away. One line to flip when attended. THE WINDOW'S QUESTION IS ANSWERED, HALF YES, HALF NO: YES: NetworkManager follows the managed flag. worker0 logged dhcp6 (eno1): activation: beginning transaction minutes after the RA appeared. "ipv6.method=auto will do DHCPv6" is now evidence, not inference. NO: the reservations never match. Every packet kea logs is [no hwaddr info] -- the clients identify with DUID-UUID (and one DUID-LLT), kea derives no MAC from any of them, so hw-address reservations cannot match and no node got its reserved address. VyOS accepting `static-mapping mac` renders valid kea config that simply never matches these clients. The "one source of truth with IPv4" addressing scheme does not survive contact with DHCPv6; options (DUID keys, kea mac-sources, dynamic range + discovery, or SLAAC) are written up in migration/window-evidence/2026-09-06-dhcpv6.txt for an attended decision. FOUND LIVE AND FIXED IN THE SAME WINDOW: `service dhcpv6-server` with no listen-interface renders kea6 with interfaces: ["*"] -- a DHCPv6 server on EVERY VLAN. kea was answering an unrelated device on bond0.10 within seconds of the first apply. Same family as the kea IPv4 cross-VLAN bug (ISC #1117). Now pinned to bond0.2 on both routers. Also in this commit, three self-inflicted script bugs found by their own failures: log() wrote progress lines into the captured config stream (VyOS rejected each as "Invalid command", leaving the two routers correct but NOT identical); "Invalid command" was missing from the failure patterns so that run reported success; and the MAC lookup matched its own freshly-created v6 reservations on the second run, returning doubled MACs. All three fixed, both routers converged and diffed identical. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH |
||
|
Michal
|
cb03987c33 |
The drill evidence was silently gitignored by *.log
Some checks failed
PPPOE-HA.md cited migration/drill-evidence/wan-drill-2026-09-06-ipv6.log as the record behind the takeover/failback/HE-call figures, and the repo reported clean -- because the root .gitignore has `*.log`, so the file was never added. The citation was dangling and nobody would have found out until they went looking for the numbers. Renamed to .txt, which is also what this repo already does for evidence: labsim/wan-failover-evidence/ and labsim/vlan-leak-evidence/ store state.txt. The ignore rule is fine; the filename was wrong. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH |
||
|
Michal
|
8a0ef52909 |
The drill measured IPv6 through a failover: zero HE calls, and it followed
Some checks failed
Second drill of the day, after IPv6-follows-master went in. First time the IPv6 behaviour of a failover is a measurement rather than an assertion -- the thing this whole review started from was a 15.5ms figure read outside the window. TAKEOVER OK: 37s IPv6 followed in 37s (v4 37s, gap 0s) FAILBACK OK: 32s IPv6 back in 44s tun0 src : 87.192.101.48 -> 87.192.101.48 unchanged HE updates from vyos001: 0 HE updates from vyos002: 0 Zero HE API calls across a full takeover and failback is the invariant the design rests on, and it now has evidence: the 10 gig lease follows the cloned MAC, so the tunnel endpoint is the same address on whichever router holds the WAN and there is nothing to tell Hurricane Electric. This also exercised the proto-41 accept rule added to vyos002 earlier today. Without it the drill would have shown IPv6 failing to return while tun0 was up and radvd running -- which is precisely how the original gap hid. The two directions are NOT symmetric and the doc says so: IPv6 arrived in the same 5s sample on takeover but trailed by 12s on failback, because the reconciler enables bond0.53 first and v6_take only raises the tunnel once the source address exists. Bounded by one 30s tick. Also noted: the drill samples every 5s, so "gap 0s" means within the same sample, not simultaneous -- and the 37s vs the morning's 52s is a different run of the same IPv4 mechanism, not an improvement. Vodafone handed out a new address again across the drill (90.251.142.103 -> 90.251.152.236), corroborating that nothing may be pinned to the PPPoE address. Production returned to normal: vyos001 MASTER on all six with both WANs, vyos002 BACKUP with tun0 down and radvd stopped, force-fault clear. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH |
||
|
Michal
|
da86b60dce |
IPv6 model merged: mark the staging record, and what the merge caught
Some checks failed
kubernetes-deployment@6d4e080 on main, vyos:verify 533/534 zero drift. The merge was not a formality. firewall-accept-he-6in4 was scoped vyos001-only because "the tunnel is anchored to its WAN address" -- untrue once the WAN became HA. vyos002 had no proto-41 accept under its IPv4 input default-deny, so a failover would have brought tun0 up and started radvd on the new master while its own firewall dropped the inbound 6in4. Every other piece would have looked right. Rule 150 added to vyos002. Also recorded: main lives in the .worktrees/grafana-token worktree. The first merge attempt was made on fix/openbao-preview-blockers, 15 commits behind main and predating the PPPoE HA overrides -- it would have reverted them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH |
||
|
Michal
|
d0b733f831 |
IPv6 follows master, deployed: vyos002 has a tunnel and a gate that holds it shut
Some checks failed
Applied step 0 to production, backup first, IPv4 untouched throughout. vyos001 held the VIP and the internet the whole way; verified before and after every commit. Order, which is the safety property: he-secrets onto vyos002 -> vrrp-wan-install on BOTH (so the gate exists before the tunnel) -> tunnel + task-scheduler on vyos002 -> route6 + bond0.9 ::2 + router-advert on vyos002 -> link-mtu 1472 on vyos001. The blackhole is confirmed behaviour, not theory. The moment the tunnel committed, vyos002 brought tun0 UP with source 87.192.101.48 -- an address it does not own -- and the reconciler closed it on the next tick: vrrp-wan: not MASTER: bringing tun0 down vrrp-wan: not MASTER: stopping radvd (deprecates the v6 gateway) Same for radvd: VyOS started it on commit, the reconciler stopped it. Both have stayed shut. That is the override's old "it would simply stay down" assumption failing in production exactly as it failed in the sim. wan-drill --dry now reports the same tunnel source on both routers, where it previously read "<no tunnel -- IPv6 cannot survive a failover>". Two things found by deploying rather than reading: - commit-confirm hangs non-interactively here, exactly as PPPOE-HA.md records. The ssh timed out leaving an orphaned config-mgmt commit_confirm holding the config lock, with nothing committed. Killed it and used plain commit + save -- safe on the backup, which holds no VIPs, no WAN, and whose LoT path is untouched. Worth knowing the failure is clean: no partial config landed. - he-tunnel-follow's status path died with "WAN_MTU: bad array subscript" on the backup. An empty array subscript is a hard bash error, not an empty expansion, so the :- default never applies -- and a backup has no default route, so it broke on precisely the box whose state you most need to read. Fixed and redeployed to both. This is now live DRIFT against the Pulumi model. Until migration/pulumi-override-he-tunnel-both.json is merged into overrides.json, a pulumi up can strip the tunnel, route6, bond0.9 addresses, router-advert and task-scheduler entry. Recorded there and in the backlog's drift section. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH |
||
|
Michal
|
395577850c |
IPv6 was never HA, and the WAN becoming HA is what exposed it
Some checks failed
Reviewed the parked IPv6 task against the PPPoE-HA work of 2026-09-05/06. The
gate that parked it ("WI-8 before IPv6") is cleared, but the same work
invalidated the assumption the IPv6 design rested on.
Verified on the live routers: vyos002 has no tun0, no he-tunnel-follow, no
he-secrets, no VLAN 9 prefix and no route6 ::/0 -- only the pre-staged
default-deny v6 firewall, which is correctly on both. Failover is now automatic
and drill-proven, so every failover takes the whole v6 estate down for as long
as vyos002 holds the VIP.
Four things that came out of checking rather than reading:
- PPPOE-HA.md's "tun0 survived untouched and IPv6 stayed up at 15.5ms" does not
follow from its own premise and is corrected in place. The endpoint address is
stable, but it MOVES to vyos002, which has nothing to decapsulate protocol 41.
wan-drill had no IPv6 check at all, which is why nobody caught it.
- A 22-second near-miss: vif53-pin-boot-disable bounced the 10 gig, he-tunnel-
follow ticked once and saw the PPPoE address, and vyos-failover restored the
route 22s before the second tick would have pointed HE at an address Vodafone
reissues on every dial.
- VyOS does NOT leave a tunnel down when its source-address is absent (the
override's stated reason for leaving IPv6 single-homed). Measured in labsim:
it commits rc=0 and brings the link UP -- a blackhole that attracts the v6
default route. The runtime gate is load-bearing, like the PPPoE gate.
- The RA link-mtu was pinned at 1480 while the tunnel correctly drops to 1472 on
the PPPoE path.
Mechanism, mirroring PPPoE HA -- identical config on both, gated at runtime, no
commit in the failover path:
- vrrp-wan-reconcile: a v6 kernel plane. tun0 and radvd follow the VIP; radvd is
stopped BEFORE the WAN goes so its farewell RA (router-lifetime 0) still has a
path out. The WAN early-exits became if-blocks so the plane runs every tick.
It deliberately does NOT call he-tunnel-follow: that would halve the
hysteresis the near-miss above showed we depend on.
- he-tunnel-follow: a master guard reading the same vrrp-wan.conf VIP, so the
backup copy cannot point HE at its own idle PPPoE line, plus a stubbable
HE_UPDATE_URL.
- vrrp-wan-install carries both, so --check and the upgrade runbook cover IPv6.
- wan-drill measures IPv6 in both timing loops and asserts zero HE API calls
across a router failover.
labsim finally has an HE endpoint, closing the gap the override itself cited as
why this was never rehearsed. Both ISP islands already share the libvirt network,
so that becomes the backbone and HE lives behind it on one address reachable over
either WAN. Proven in the sim: backup tun=DOWN radvd=inactive, master tun=UP
radvd=active, hysteresis then HE call then MTU 1480->1472, and VLAN 9 hosts
autoconfiguring from the RA. The end-to-end v6 datapath is NOT yet proven --
inter-island transit crosses libvirt NAT and the return path is lost. Recorded as
a KNOWN SIM GAP rather than papered over.
The model change is staged, not merged: another agent runs pulumi up on that
repo, and the gate must exist on vyos002 before the tunnel does.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
|
||
|
Michal
|
061b9e3d7e |
Close the last two items: config.boot pinned, and an upgrade runbook
Some checks failed
vif53-pin-boot-disable ran on vyos001. Both config.boots now pin `vif 53 disable` while vyos001's running config keeps the WAN, so a reboot in any order comes up unable to claim the cloned MAC. Cost: a 32s window (10s in the sim -- production commits under kea/BGP/conntrack are slower), bond0.53 re-leased 87.192.101.48 in 5s, and vyos-failover restored the primary route about a minute later. The house rode pppoe0 in between rather than losing the internet. Recorded the shape of that recovery, because I misread it myself: for ~60s after the bounce the default route really is on pppoe0, since vyos-failover only re-adds the bond0.53 route once its probes pass. A fresh `ip route show` in that window looks like a regression and is not one. VYOS-IMAGE-UPGRADE.md is step 9. An image upgrade keeps /config and REPLACES /etc, which silently removes the ppp@pppoe0 gate drop-in -- the only thing stopping the backup from dialling into a single-session account. The reconciler fails closed, so the symptom is "PPPoE never comes up" rather than "both routers dialled", but an upgraded box has no PPPoE until the gate is back. One router at a time, backup first. Deleted the "model hazard to fix before applying" section rather than leaving it: it advised asserting `vif 53 disable` on BOTH routers via an override, which is exactly wrong. That is runtime state owned by vrrp-wan-reconcile, and pinning it would fight the reconciler on every apply and briefly disable the live master's 10 gig each time. Replaced with what is actually done -- follow reality at runtime, hardcode safe at boot and at install time -- and said plainly not to reintroduce it. |
||
|
Michal
|
bda5854563 |
vif53-pin-boot-disable: get disable into config.boot without losing the WAN
Some checks failed
Closes the last open item. The convention is that BOTH routers' config.boot carry `vif 53 disable`, so a reboot in any order comes up unable to claim the cloned MAC and the reconciler enables it on whoever holds the VIP. vyos001's config.boot predates this work and does not. The clean fix does not exist: `save` writes the RUNNING config, not the candidate. Tested in labsim -- set the node, save, discard, and config.boot came back WITHOUT `disable` and the WAN untouched. So the node has to be genuinely disabled, saved, and re-enabled. That costs a real bounce of the 10 gig, but not the internet: pppoe0 is up on the master and the failover route falls to it, which is the T5 path. Measured window in labsim: 10s, with a default route present in every 0.5s sample. Found and fixed a race while testing. The first run collided with vrrp-wan-reconcile's own commit -- "Configuration system temporarily locked due to another commit in progress" -- and the `save` landed while the RE-ENABLE did not, leaving the master with its 10 gig down. The script now takes the reconciler's own /run/vrrp-wan.lock, which the reconciler skips a tick rather than block on, with 9>&- so the config session's unionfs child cannot inherit and hold it. Worth recording that the bad run still ended correctly: the reconciler logged "MASTER with bond0.53 disabled -> enabling" and repaired it in 4s. The failure mode is bounded by design. The script no longer relies on that, but it is why a half-completed run is survivable, and it verifies the re-enable and shouts rather than reporting a success it did not achieve. |
||
|
Michal
|
d08f68e28b |
PPPOE-HA: step 8 done -- model merged, zero drift
Some checks failed
Both overrides are on kubernetes-deployment main (45033dd) and the transition-scripts were applied to the boxes directly rather than left for an unattended `pulumi up` to find. vyos:verify reports both routers in sync. Also records the drill result in the header, since "deployed" and "proven to fail over" are different claims and only the second one is worth much. |
||
|
Michal
|
e1c571d004 |
PPPOE-HA: the drill closed the two biggest unknowns
Some checks failed
Controlled failover in production: takeover 52s, failback 36s, ~88s of interruption across two deliberate transitions. The cloned-MAC lease TRANSFERS. That was named in this file as "the largest untested item in any failover" -- whether the 10 gig ISP would re-issue 87.192.101.48 to f0:9f:c2:12:9b:4f arriving on a different switch port. It did, same address, inside the takeover window. Moved out of "what the sim cannot prove" rather than left there contradicting the evidence. VyOS dialling Vodafone also worked from both routers, which had never been done -- PPPoE was only ever proven on the USG. Vodafone did not refuse either re-dial, so its session-control behaves like `replace`, not the hostile `deny`. GRACE stays at 300 anyway: one drill on one evening is not the ISP's policy under all conditions. Worth knowing for anything added later: Vodafone hands out a DIFFERENT IPv4 on every dial (83.106.5.72 -> 90.251.153.180 -> 90.251.142.103). Nothing may be pinned to the pppoe0 address. Checked the HE IPv6 tunnel specifically, since it carries a hardcoded source-address -- it is pinned to 87.192.101.48, which is the 10 gig and stable across failover, so tun0 was untouched and IPv6 stayed up at 15.5ms. |
||
|
Michal
|
47ce0c1aea |
wan-drill: run the failover drill unattended, because the operator goes offline
Some checks failed
The drill takes the household's internet down, which means anything driving it step-by-step stops being able to act at exactly the moment it matters -- an agent needs the internet to think, so it would freeze mid-failover with the levers half-thrown. This script needs only the LAN, and its cleanup runs from an EXIT trap so both force-fault levers are cleared on any exit path, including being killed. Belt and braces around it: the watchdog on vyos002 stands the box down by itself after 150s holding the VIP with no WAN, so even if the script dies the internet comes back. Refuses to start if the internet is already down or if vyos001 is not the current holder. |
||
|
Michal
|
85819e40c1 |
wan-drill-watchdog: bound the blast radius of a failover drill
Some checks failed
The drill takes the internet down for as long as the new master needs a WAN. If it never gets one, whoever is running the drill is offline too -- and an agent simply stops responding mid-incident. The abort therefore cannot depend on anyone being present. Armed on the router expected to take over: if it holds the VIP and has had no WAN for HOLD consecutive seconds, it sets force-fault on itself, sheds every VIP, and the healthy non-preempting peer takes them straight back. setsid so it outlives the ssh session that armed it, which is the entire point. 150s by default, deliberately shorter than GRACE=300. GRACE is sized for a real hostile-ISP takeover that is still making progress; this is sized for "the drill failed, give the house its internet back". Both paths proven on vyos002 in production: it stayed silent for 35s while a BACKUP (a misfire here would itself cause an outage), and fired within 20s when pointed at an address the box does hold with no WAN, logging "ABORT: held the VIP with no WAN for 20s -- standing down". The first firing test was my own bug, worth noting: sed'ing the default VIP= line does nothing, because vrrp-wan.conf is sourced afterwards and VRRP_WAN_VIP puts 192.168.1.1 straight back. The watchdog was watching an address the box does not hold and correctly stayed quiet -- a test that proved nothing while looking like it proved the feature was broken. |
||
|
Michal
|
13dcdff1ef |
wan-panic: a one-command revert that works with no internet and no Claude
Some checks failed
The WAN now follows VRRP mastership, so a bad failover takes the house offline -- and whoever is debugging it is offline too. The recovery path therefore has to be already on the box, not in a chat log. `/config/wan-panic` works out which router it is running on and does the right thing: on vyos002 it stands down (force-fault), which is what lets vyos001 take the VIPs back; on vyos001 it clears force-fault and says plainly that no-preempt means you must run it on the OTHER box to actually move anything. `status` shows who holds the VIP, which WANs are up and the default route. `undo` stops both timers so nothing moves the WAN again, leaving whatever is currently up exactly as it is. It uses the force-fault lever rather than restarting keepalived because failing the health check is the supported way to shed mastership -- `restart vrrp` is not dependable, since with advert_int 1 the peer declares the master dead in ~3.6s and the restart usually finishes inside that window. Proven in production on vyos002, which carries no traffic: BACKUP -> FAULT in ~20s and back to BACKUP on clearing, with vyos001 untouched and the internet steady at 7.7-8.1ms throughout. The card is installed to /config/RECOVERY-CARD.md on both routers and copied to ~/WAN-RECOVERY-CARD.md, because a card you can only read with working internet is not a card. It leads with how to reach the routers over the LoT leg, which is L2-direct and survives Management/VRRP/routing being broken. |
||
|
Michal
|
8aa3d0ebaa |
PPPOE-HA: record that vyos001's config.boot lacks vif 53 disable
Some checks failed
Runbook step 7 says confirm it on both; it holds on vyos002 and not on vyos001, whose config.boot dates from 2026-09-02 and predates this work. The exposure is new even though the condition is not: if vyos001 reboots while vyos002 is master holding the 10 gig, the cloned MAC is briefly live on both. That could not happen yesterday, because vyos002 was stuck in FAULT and could never be master. It self-heals in <=30s -- vyos001 comes up BACKUP under no-preempt and the reconciler commits `disable` on the next tick -- so this is a bounded window, not a split. Left as an outstanding item rather than fixed, because writing config.boot means `save`ing the running state, so the fix takes the live 10 gig down briefly and belongs in a maintenance window. Hand-editing config.boot would avoid the blip at the risk of an unbootable router, which is worse. |
||
|
Michal
|
8fce03e705 |
PPPOE-HA: deployed to production
Some checks failed
Mechanism installed on both routers (--check clean against git on both), `interfaces pppoe pppoe0 disable` removed from vyos002, and vyos002 is out of FAULT and holding BACKUP on all six groups for the first time in 3d10h -- the lab has a real standby again. Verified on the wire rather than from state: with a tcpdump running on vyos002's bond0.51 across the commit, ZERO PADI/PADR. The peers file rendered (so it can dial the instant it is promoted) while ConditionResult stayed `no` and NRestarts 0. vyos001's live session was untouched throughout -- same MainPID 2931, same 83.106.5.72, internet 7.6-8.4ms at 0% loss. Not done, deliberately: the controlled failover drill, which interrupts the household's internet, and merging the staged Pulumi override, which lands via another agent's `pulumi up` on a branch they have checked out. |
||
|
Michal
|
5ed0e4888a |
labsim: both matrices green end to end, numbers reproduced
Some checks failed
Final confirming run with GRACE=300 and every fix installed:
--all ALL PASS (T0 T3 T5 T8 T11 T12)
--hard ALL PASS (replace / deny / disable, each policy verified)
Second independent measurement of the hard failover, which is what makes the
figures trustworthy rather than anecdotal:
run 1 run 2
replace 26s 26s
deny 148s 141s
disable 21s 20s
`deny` sits at ~141-148s across both, so it is the AC's dead-peer behaviour
and not a one-off; GRACE=300 keeps roughly 2x margin.
Corrected an overclaim in PPPOE-HA.md while confirming it: the invariant row
read "AC never showed two simdsl sessions", and under session-control=disable
it did -- one live, one orphaned from the destroyed router. Only ever one LIVE
router dialled, which is the invariant that matters. Said so plainly rather
than leaving a table that reads better than the evidence.
|
||
|
Michal
|
6987b324f1 |
vrrp-wan: size GRACE from the measured hostile failover, not the theory
Some checks failed
With the matrix actually setting session-control, T4 timed a destroyed
master's takeover at:
replace 26s
deny 148s <-- sizing case
disable 21s
`deny` is the case GRACE exists for: the AC refuses the survivor until its own
dead-peer timer frees the dead session. The session poller caught it happening
-- the destroyed router's session stayed in the table while the survivor's
dials appeared and were rejected, twice, before one took at 148s.
148s is well past the lcp-echo-interval(30) x failure(3) = 90s budget that 180
was sized against, leaving 32s of margin. GRACE=300 is ~2x the worst observed.
Treat 148s as a floor, not a worst case: these are idle 2-vCPU VMs, and the AC
shares an OVS bridge with the routers, so `virsh destroy` removes the port and
accel-ppp sees the peer vanish. A real BRAS over DSL never learns our router
died and waits out longer timers of its own.
The cost is stated in the conf: GRACE is also how long an alive-but-unroutable
master holds every VIP before yielding. bond0.53 covers most of that -- a DHCP
lease satisfies the check in seconds -- so GRACE only dominates when PPPoE is
the last path. Kept the health check's fallback in step, since keepalived runs
it with no environment and that number decides mastership if the conf is ever
missing.
check_invariant no longer fails blind on the AC's session count. That count is
only a proxy for "two of our routers dialled", and only while the AC enforces
single-session; under `disable` it does not, so a destroyed router's session
lingers and the count reads 2 with exactly one live router dialled. The real
invariant -- at most one router holds pppoe0 -- is now the failing one, and the
stale session is reported as a WARN rather than silenced, because it still
occupies the slot at a real ISP and is precisely what made `deny` take 148s.
|
||
|
Michal
|
9221c71ff0 |
labsim: the session-control matrix was never setting session-control
Some checks failed
T4 prints "--- session-control=deny ---" and then measures whatever policy the AC already had. The setter was isp "vbash -c 'source script-template; configure; set ...; commit; save'" and that form does not start a config session at all -- commit dies with "Invalid command: [commit]" on stderr, which isp() discards. `show configuration commands | grep session-control` on the ISP VM returned nothing after a full matrix run: all three iterations had run against the accel-ppp default. The labels were fiction, and a harness that reports coverage it does not have is worse than one that reports a failure. Driving it from a real script FILE works. isp_session_control() does that, reads the value back, and fails the iteration if it disagrees rather than measuring the wrong policy. `session-control` is a valid node here (checked the template dir on VyOS 2026.08.12-0831-rolling), so this was purely the invocation. Staged the two Pulumi overrides in migration/ rather than adding them to kubernetes-deployment: another agent runs `pulumi up` on that repo, so merging `remove: pppoe0 disable` before the gate exists on vyos002 would let it dial and take the single Vodafone session off the live master. Ordering is written at the top of the file. |
||
|
Michal
|
4d47b609a2 |
PPPOE-HA: record the two failure modes found by running the thing
Some checks failed
Both were invisible to inspection and only appeared under the hard matrix: the flap damper tearing down an established session via the lease it stopped renewing, and a missing peers file being completely silent. The second has a production edge worth spelling out -- an unsaved commit reverts on reboot and takes pppoe0 with it, leaving a standby that can never take over while looking perfectly healthy. |
||
|
Michal
|
b659e0d47e |
vrrp-wan: a flap holdoff must not tear down a live WAN session
Some checks failed
ppp_dial() checked the flap holdoff and returned BEFORE renewing /run/vrrp-wan/may-dial. That lease is what vrrp-wan-guard expires after LEASE_TTL, so tripping the damper stopped the renew and the guard hung up pppoe0 on the MASTER ~80s later. A damper meant to suppress repeated DIALS was tearing down a working WAN instead. Observed in labsim, end to end: DIAL FLAP: >=6 attempts in 600s -- holding off 900s GUARD: lease stale (81s > 75s) -- hanging up pppoe0 An established session now outranks every check below it: ppp_active renews the lease and returns first. Everything after it only decides whether to start a NEW session. Two supporting fixes for how that storm started. The dial attempts were all no-ops because /etc/ppp/peers/pppoe0 was missing, and nothing said so -- systemd logs "skipped because of an unmet condition check" exactly once and the gate looks identical to a healthy backup. ppp_dial() now reports it, and distinguishes "configured but not rendered" (re-commit the subtree) from "no pppoe0 in config at all", which is what a reboot leaves behind when a commit was never saved. That is precisely how the sim secondary lost its WAN. Also `cat | wc -l` rather than `wc -l < file`: redirections are applied left to right, so the missing-file error escapes the 2>/dev/null on every first-ever dial. Harness: T11 copied-then-removed instead of mv, and verifies the restore -- losing that file strands a router permanently, which cost a debugging session. preflight now refuses to run if either router lacks the peers file or the pppoe0 config, since every failover result would otherwise be a false negative blamed on the ISP. New T12 forges a 900s holdoff against a live session and asserts it survives. |
||
|
Michal
|
d626750010 |
labsim: hard failover and reboot safety hold; runbook for the production apply
Some checks failed
Hard failover proven by destroying the master outright (virsh destroy -- no PADT,
the case a graceful stop cannot cover): the survivor took the VIPs, dialled, and
the access concentrator showed exactly ONE simdsl session from its MAC for the
whole seven minutes. When the destroyed router came back it did NOT dial --
ConditionResult=no, ActiveState=inactive, NRestarts=0, pppoe0 absent, may_dial=no
-- and the AC session count stayed at 1. That is the gate doing the one job it
exists for, on the path that previously had no protection at all.
Two more harness bugs of the same family as the last three, both of which
reported a working system as broken:
- ppp_on() returned an EMPTY string for an unreachable router, and `[ "" = 0 ]`
is false, so the hard-failover wait sat for its full timeout waiting for a
DESTROYED box to report zero -- long after the survivor had taken over
correctly. Absent now means 0.
- a VM restart recreates its taps under new names and the OVS bond keeps the
old ones: lacp dies, VLAN 1 goes with it, and the box returns reachable on
some VLANs and not others. That looked exactly like a failed failover. It is
the same stale-membership fault ovs_bond_router already detects, but nothing
ran it after a restart; the harness now does.
migration/PPPOE-HA.md is the runbook: what the design is, why `disable` cannot
work, the measured numbers, the deploy order (vyos002 first, on its own commit,
verified on the wire with tcpdump rather than from state), and the one-line
rollback.
It also records a model hazard found while writing it. The imported baseline
captures the RUNNING state, not the safe one -- vyos001 has no `vif 53 disable`
because it happens to be master, vyos002 does. An apply performed while vyos002
held the VIP would therefore enable vyos001's WAN too, putting the cloned MAC on
both boxes. An override must assert `vif 53 disable` on BOTH, accepting that an
apply then briefly disables the current master's 10 gig until the reconciler
restores it.
Still not applied to production.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
|
||
|
Michal
|
93fed7826b |
labsim: PPPoE HA passes the matrix, and the health check had a real flap bug
Some checks failed
Full run green: baseline (one AC session, held by the VIP holder), clean
failover (pppoe0 moves in 26s, old master releases), 10 gig down (route falls to
pppoe0 and the LAN is back online in 5s), lease expiry (the guard hangs up), and
a missing peers file (NRestarts=0, no loop). Evidence in
labsim/wan-failover-evidence/.
The 10 gig test found a genuine bug in vrrp-wan-health, not in the sim. GRACE
was measured from PROMOTION, so an established master had no grace at all --
after hours of uptime `now - since` far exceeds any window. The first moment
bond0.53 went down while pppoe0 was mid-redial, the master failed its own check,
shed every VIP, and the peer inherited the same WAN outage and did the same. A
brief WAN blip would have flapped the production pair. The stamp is now
refreshed on every healthy tick, so grace measures time since the box last
demonstrably HAD a WAN -- survivable wherever the gap happens, not only just
after a promotion.
Three harness bugs, all the same shape, all of which produced a confident wrong
answer before being caught:
- waiting for "exactly one pppoe0 holder" returns INSTANTLY during a handover,
because it was already true. The useful question is who holds it.
- judging connectivity on a single ping 20s after a link drop reported an
outage that had already healed. Poll, do not sample.
- `-o PreferredAuthentications=password` suits the routers but not the Alpine
LAN VMs, whose sshd offers keyboard-interactive: ssh exited 255 before
running anything and the test read that as "the LAN lost the internet". A
tcpdump on the router showed the pings leaving pppoe0 NATed to
198.51.100.117 and the replies coming back the whole time. An exit code that
can mean "the network is broken" or "I could not log in" is not a
connectivity test, so the check now asserts on what the guest reported.
That last one is why the harness asks the routers and the access concentrator
rather than a client, and why it refuses to run at all while either router still
has a default route via eth2 -- the libvirt-NAT scaffold answers connectivity
checks the WAN under test would have failed.
Still to run: hard failover (destroy the master), and the session-control
replace/deny/disable axis that brackets Vodafone's unknown behaviour and sets
the final GRACE. Nothing applied to production.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
|
||
|
Michal
|
4efd70c987 |
vyos: move PPPoE off the config plane onto a gated systemd unit
Some checks failed
PPPoE HA could not work as written, and the reason is structural rather than a
bug: `set interfaces pppoe pppoe0 disable` and `delete` are handled identically
by interfaces_pppoe.py -- both UNLINK /etc/ppp/peers/pppoe0. That path is pppd's
own options file, so the resting state destroyed exactly what the promotion path
needed, and `ppp@pppoe0` restart-looped against it (observed: 47 restarts, zero
sessions at the access concentrator). It also made op-mode `connect interface
pppoe0` unusable, and put every failover behind a priority-322 commit where one
unrelated invalid node fails the whole thing -- which has already taken the
10 gig down once.
pppoe0 is now configured identically and ENABLED on both routers, so the peers
file always exists, and dialling is gated by a drop-in on the unit:
ConditionPathExists=/run/vrrp-wan/may-dial
ConditionPathExists=/etc/ppp/peers/pppoe0
/run is tmpfs, so the gate is shut at boot and neither box can dial before VRRP
has decided. That matters more than it first appears: with the node enabled,
interfaces_pppoe.py restarts ppp on EVERY commit touching the pppoe subtree when
the daemon is not running -- so the backup actively tries to dial whenever
anything commits. The gate is the only thing making that a no-op, which is why
vrrp-wan-reconcile now refuses to bless a box whose drop-in is missing: /etc is
per-image, and a VyOS upgrade would otherwise silently remove the protection.
may-dial is a LEASE, not a flag. ConditionPathExists is evaluated at start only
-- it can prevent a dial, never revoke one -- so a reconciler that stops running
while its box is demoted would keep the one ISP session for ever. The reconciler
renews the lease; a new 5s vrrp-wan-guard revokes it, and only ever revokes. It
fired correctly first time: "GUARD: lease stale (81s > 75s)".
Also: remove-then-stop on release (the file's absence blocks a NEW start that a
concurrent commit would trigger); a flap damper, because two routers that both
believe they hold the VIP will both dial and each dial kills the other's session
-- against a real ISP that is how an account gets rate-limited; and a guard on
`cfg` returning empty under commit-lock contention, which had already produced
one spurious "releasing" on a box that needed nothing.
GRACE 90 -> 180. accel-ppp's dead-peer budget is lcp-echo-interval(30) x
failure(3) = 90s, so the old value sat exactly on the boundary: a hard failover
into an AC that does not replace the stale session would fail its own check,
shed the VIPs, and leave both routers in FAULT.
The sim could not have tested any of this. Both routers now get the identical
WAN -- the secondary had none "because two PPPoE clients sharing one credential
is a different failure mode than anything production has", which is backwards:
that IS production. It also left the pair incomparable, ten NAT rules against
none. Safety now comes from resting state, not asymmetry.
Three more things the sim was hiding:
- the drift check's secondary regex omitted interfaces pppoe/bonding, nat
source and protocols failover, so it reported "in sync" for a box with no
WAN at all;
- the VRRP health-check and transition-script hooks existed on both live VMs
and in NEITHER generator -- the mechanism under test was pure undetected
drift;
- labsim-vyos's only default route was the libvirt-NAT scaffold, so every
"the LAN still has internet" verdict on it was answered by eth2 rather than
the WAN. --drop-scaffold applied; the earlier DHCP-failover proof is being
re-run because of it.
vrrp-wan-install ends the other half of that: the sim's previous proof came from
scripts hand-`sed`-ed in place, so the tested behaviour was not the committed
behaviour. `--check` now makes that a hard failure.
First green run: master holds both WANs, backup released, and the AC reports
exactly ONE session. sim-net-apply.sh check: all four in sync.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
|
||
|
Michal
|
2e828b8af2 |
vyos: a failover you can actually trigger, and four bugs found triggering it
Some checks failed
A planned failover had no reliable lever. VyOS offers only `restart vrrp`, and
neither that nor `systemctl restart keepalived` is dependable: with advert_int 1
the peer declares the master dead after ~3.6s and a restart usually finishes
inside that window. Measured -- the same command moved mastership on one run and
not on the next three. A fail-back step you cannot trigger on purpose is not a
procedure, and the recovery card depended on one.
The lever is now `touch /run/vrrp-wan/force-fault`: the health check fails, the
sync group sheds every VIP, and the peer takes over. It exercises the same path
a real WAN loss takes rather than a special case, and it lives in /run so a
reboot cannot leave a router permanently ineligible.
Proven end to end in labsim: lever -> mastership moves -> WAN follows -> the old
master releases -> a LAN VM has internet -> the faulted router returns to BACKUP
and is eligible again.
Getting there exposed four real bugs, two of which would have broken a GENUINE
failover, not just the drill:
- The grace stamp was written only by the 30s reconciler, so a freshly
promoted master reached the 5s health check with no stamp, scored grace = 0,
failed instantly and went FAULT. With the peer already faulted that left
BOTH routers in FAULT and the LAN with no gateway at all -- worse than the
outage the check exists to prevent. The check now stamps on promotion.
- And it inherited STALE stamps from an earlier mastership, failing ~5s after
passing. The stamp is now cleared on the way down, by the health check
itself, not only by the reconciler.
- The lock fd leaked into VyOS's config session: `exec 9>` is inherited by the
long-lived unionfs-fuse the session spawns, which never closes it. From the
first config change on, every later reconciler run lost the flock and exited
0 having done nothing -- healthy-looking journal, silently stopped
reconciling. That is how a demoted router kept the WAN. Children now get 9>&-.
- vrrp-wan-apply touched pppoe0 unconditionally. On a box where pppoe0 has no
source-interface VyOS rejects the whole commit ("Physical source-interface
required"), taking the bond0.53 change down with it -- and the script still
returned 0, so the reconciler logged a release that never happened. pppoe0 is
now guarded on existence and the commit's verdict is propagated.
Still NOT applied to production. The pair is single-homed on WAN until it is.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
|
||
|
Michal
|
7bf3f42e19 |
vyos: WAN follows VRRP mastership, rehearsed in labsim
Some checks failed
The ISP is consumer with one static IP, so "both routers hold WAN" is not
available: the 10 gig lease is anchored to a cloned MAC and the PPPoE line to a
single credential. The WAN therefore has to move with mastership.
Proven end to end in the sim: the secondary was promoted, took the WAN, got a
lease, installed a default route, and a LAN VM reached the internet through it
(3/3, 9ms). Demoting released it -- link down, no address. Rebooting the master
converged correctly too: it came back BACKUP with the WAN disabled while the
peer kept it.
The rehearsal earned its keep four times over, and none of these were visible
from reading the docs:
- `transition-script` alone is NOT safe to hang internet on. VyOS delivers it
through keepalived-fifo.py, and on one promotion that helper logged NOTHING
while Keepalived_vrrp logged all six instances entering MASTER and the
built-in notify_master for conntrack-sync ran normally. The result was a
router holding every VIP with no WAN -- the 2026-09-02 outage, recreated by
the mechanism meant to prevent it. Hence vrrp-wan-reconcile on a 30s timer:
the scripts give speed, the timer gives correctness.
- The health check must ask REALITY, not a marker. Keying "am I master" on a
/run file written by the transition script meant that when the script did
not run, the router believed it was backup, passed the check, and kept the
VIPs it could not serve. It now asks whether the VIP is actually on the box.
- The old address-based check DEADLOCKED this design: may-I-be-master required
already having WAN, and only the master gets WAN. That is why vyos002 sat in
FAULT for ever -- the safety check had silently removed the redundancy it
existed to protect.
- script-template must be the FIRST thing a script does. Sourced after an if,
an exec and a mkdir it terminated the script inside the source, rc=0, no
output: the reconciler reported success having done nothing. Hence the split
into vrrp-wan-apply, matching the shape /config/vyos-known-good already uses.
Two hazards found and handled rather than discovered in production:
- A `configure` session whose process dies leaks a unionfs mount under
/opt/vyatta/config/tmp, and one of those holds the commit lock -- after
which every commit fails, including the manual one you try to fix it with.
A 30s job that can leak one per failure wedges the box on its own, so the
reconciler reaps dead sessions before it starts. It cleared 8 on the sim.
- Any `save` while a box is master persists the enabled WAN into config.boot,
so a reboot would claim the shared MAC regardless of VRRP state. Observed:
an ordinary console-apply did exactly this. config.boot must keep `disable`
on BOTH routers; the model asserts it and vyos:verify reports it as drift.
NOT yet applied to production, and it should not be until the remaining item is
settled: a clean, deliberately-triggered failover has been seen via reboot, but
`restart vrrp` twice failed to move mastership at all, so the trigger for a
planned failover is still unproven.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
|
||
|
Michal
|
09ede73b67 |
migration: the watcher that caught vyos002, and what it found
vyos002-catch.sh is armed before power-on and strips the eth2 address the moment SSH answers -- 6 seconds, where a human watching a console loses that race more often than not. It commits eth2 on its own before doing anything else, because every extra command in that commit is extra exposure. Caught at 15:35:22 on the LoT leg. eth2 ended with no address and the sync-group health-check installed; it exits 1 (WAN disabled on this box), so all six VRRP groups sit in FAULT and it holds no VIPs at all. That is the protection that was missing on 2026-09-02, working as intended rather than as a theory. The bounded-risk note in the header is the part worth keeping: it comes up BACKUP behind a healthy vyos001, so it never holds 192.168.8.1 and the GATEWAY cannot be poisoned during the window. The unbounded case is it becoming MASTER with eth2 present, which the health-check now makes impossible. It also surfaced that vyos002 had booted from a saved config predating the day's work -- stale reservations including the two that hand the routers' own eth2 NICs 192.168.8.143/.144. Synced to vyos001 and imported; see kubernetes-deployment 7f92974. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH |
||
|
Michal
|
a973b51b9c |
migration: vyos001 converted in production, and the config vyos002 needs first
UniFi does offer Native VLAN = None (7.5.10), and its dropdown lists Management as VLAN 1 -- the API reports vlan:null, which is what made this look like a blocker. The UI was right. Applied to USW Aggregation ports 1+2, then the router over ssh vyos@10.0.1.252. Confirmed by kea's own log: "the interface bond0 has no usable IPv4 addresses configured" -- it opens no socket on the parent, so there is no wrong pool left to answer from. VRRP held mastership with no transition; the VIPs never moved. Proven end to end by the thing that was broken: vyos002's JetKVM console, which had been sitting on a VLAN 3 port holding Management lease 192.168.1.28, was restarted and took 192.168.3.14 -- one lease, right pool, right reservation. That console is what unblocks vyos002. commit-confirm cannot be driven non-interactively: `vbash -s` hangs on its prompt. It failed safely (candidate discarded, nothing committed) but the recovery card's commit-confirm advice only works typed by hand. vyos002-return.conf carries the two defects that must not survive its next boot. eth2 is US24 port 16, native VLAN 2 -- disable that port before powering the box on and the ARP hazard is gone before it can happen. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH |
||
|
Michal
|
d727a50ca0 |
migration: offline recovery card, and stop the leak test trusting a silent router
The change itself takes Management down, so the session that performs it has no
internet and no Claude. RECOVERY-CARD-vlan1-move.md is what is actually needed at
that point, on one page: the recovery path, the order, the commands, and what to
do when locked out.
The recovery path is `ssh vyos@10.0.1.252`, and the reason it is trustworthy is
that it is not routed -- the workstation is 10.0.0.210/23 and the router's LoT
leg is 10.0.1.252/23, so `ip route get` returns dev lanbr0 with no `via`. It
therefore survives Management, VRRP, the VIPs, DNS and the switch trunk all being
wrong at once. bond0.10 is untouched by the change and stays in the LAN group.
Order is switch-first, which is the opposite of what seems natural. The UniFi
controller is 192.168.1.5, on Management, reached from LoT *through vyos001*: do
the router first and you lose the controller you still need for the switch.
Two corrections to the leak test, both because it reported a router fault that
was its own:
- it counted `pgrep -f 'tcpdump -i bond0'`, so a stray tcpdump from an earlier
run satisfied the >=2 guard with none of this run's captures alive. A capture
that records nothing reads as "the router sent no reply at all".
- it believed a single silent run. Kea can be is-active and answering nothing
for tens of seconds after a restart, so the first VLANs of a loop failed and
the last passed. That produced two OPPOSITE and equally wrong conclusions
about `listen-interface` before a retry showed the pattern.
On `listen-interface`: it is a real second branch in kea-dhcp4.conf.j2 that keeps
dhcp-socket-type raw, unlike listen-address which forces udp and took DHCP down
when it was tried live. But naming the sub-interfaces explicitly left Management
DHCP dead in the sim, reproducibly, against a clean control with interfaces:["*"].
Not adopted, and not needed -- the address move is the proven fix.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
|
||
|
Michal
|
0481c38e09 |
labsim: prove the tagged-Management fix for kea's wrong-pool offers
Kea #1117: with dhcp-socket-type raw, a frame tagged for a sub-interface is also delivered to the parent's AF_PACKET socket, and if the parent serves a subnet kea answers from it too. Management being the native VLAN on bond0 is what gives the parent that subnet. One DISCOVER on VLAN 3 produced two OFFERs, and in the captures here the WRONG one arrives first as often as not -- which is why this looked device-dependent rather than like a server bug. labsim-vlan-leak-test.sh reproduces it and scores the SERVER's offers, not the client's choice; a client picking correctly is how this hid. Fails on the old shape, passes on the new one across all six LAN VLANs. Three things the rehearsal caught that reasoning had not: - kea keeps its old raw socket. VyOS does not restart it for an interface address change, so the first post-fix test failed and looked exactly like the fix not working. - interface-group LAN names the bare bond0. Moving the address without moving the group drops every management session under default-deny. - there is no make-before-break. A port always egresses its native VLAN untagged, so while VLAN 1 is native the router can send tagged VLAN 1 but never receive it -- verified, the ARP landed on bond0 untagged. What makes the cutover safe anyway is that tagged and untagged Management coexist, so the firewalls convert one at a time: 0s of VIP downtime, versus 5m30s if both routers go before the switch does. In that state the healthy BACKUP does NOT take over -- the sync group holds native BACKUP because the other VLANs still hear the master. Also fixes two ways the sim was lying. ovs_bond_router compared only the trunk VLAN list on re-runs, so a VM restart left the bond holding taps that no longer existed while the real ones sat in the bridge unbonded -- labsim-vyos2 had no LACP at all. And the tap count included the primary's libvirt-NAT scaffold NIC, so the primary's bond was skipped outright. Runbook: migration/MANAGEMENT-VLAN-TAGGED.md Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH |
||
|
Michal
|
23783b8486 |
vyos: VRRP health check so the WAN and the gateway VIP cannot separate
Outage of 2026-09-02: vyos002 held every floating gateway IP while vyos001 held
the only working WAN. The LAN had a gateway that could not reach the internet,
and it stayed that way until vyos002 was powered off by hand.
Three causes, none of them bad luck. VRRP had no health check of any kind, so
mastership was decided purely on whether the peer was still advertising and
never on whether this router could route. vyos002 structurally cannot route --
bond0.53 is `disable`d because the 10 gig lease is bound to a cloned MAC that
only one box may hold, and pppoe0 is not up. And `no-preempt` is set on every
group, so once vyos002 took master it kept it even with a healthy priority-200
peer sitting next to it.
Reproduced exactly in labsim: stopping keepalived on the primary moved every VIP
to the WAN-less secondary and LAN internet went from 9ms to 100% loss; restarting
the healthy primary left it BACKUP and the outage in place. Applying this check
self-healed it -- secondary to FAULT, primary to MASTER, internet back.
The check asks "do I have an address on a WAN interface", deliberately not "can
I reach the internet" and not "do I have a default route". During a real ISP
outage the default route disappears on BOTH routers; keying on that would put
both in FAULT, nobody would hold the VIPs, and an internet outage would become a
total one.
Config goes on the SYNC GROUP, not per group: VyOS refuses a per-group check
while the group is in a sync group ("Only sync group health check will be
used"), and sync-group scope is what we want anyway so all VIPs move together.
Known cost, measured: with the primary genuinely dead the secondary stays FAULT
and NOTHING holds the gateway, so inter-VLAN routing stops too. That is the
honest consequence of a backup that cannot route. The fix for it is to make the
WAN follow mastership so the backup CAN route -- next, and rehearsed separately,
since it is the one change that can lose the DHCP lease.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
|
||
|
Michal
|
f41ffdd039 |
feat(vyos): reconciler that keeps the HE 6in4 tunnel on the live WAN
Some checks failed
Kernel-level (`ip tunnel change`), not VyOS config: no commit churn on a flapping line, no drift against the Pulumi model, and a reboot restores config.boot — which pins the 10 gig — so a wrong source cannot survive a restart. Sends `myip` explicitly, because mid-failover the update request may egress either line and letting HE infer the address would point the tunnel at the WAN we just left. Requires two consecutive agreeing runs before acting, since HE rate-limits updates and a flapping WAN would hammer the API precisely when it matters. MTU moves with the WAN: 1480 on the 10 gig (1500-20), 1472 on PPPoE (1492-20). Fixed at 1480, the backup path gives the signature people lose a day to — small packets fine, large transfers hang. Inert without /config/he-secrets, and a no-op when already in sync. |
||
|
Michal
|
a187703a3a |
feat(vyos): pin a known-good config and restore it with one command
Some checks failed
VyOS already has rollback, but `rollback 1` returns you to the *previous*
revision, which may itself be broken — you can end up walking backwards through
several bad commits hunting for the one that worked, at exactly the moment you
have no network to look things up with. This pins a state a human has actually
used and found working, so recovery is one step and needs no memory of how many
changes ago things were fine.
/config/vyos-known-good save pin the running config
/config/vyos-known-good status when it was taken, how running differs
/config/vyos-known-good diff what a restore would change
/config/vyos-known-good restore go back to it
Deliberately not automatic. A config is only known-good once someone has used
the network; a snapshot taken after every commit would faithfully preserve the
broken one.
The restore is itself commit-confirmed, so even the recovery path is protected:
if the snapshot is somehow wrong, or access is still broken and nothing can be
confirmed, the router undoes the restore rather than leaving you worse off.
Silence reverts.
`save` refuses when there are uncommitted changes — a snapshot that did not
match what is actually running would look like a safety net without being one.
Two things found while building it, both of which made the script silently
useless rather than fail loudly:
- Sourcing `script-template` **resets the positional parameters**, so `$1` was
empty by the time the case statement ran and every invocation fell through
to the usage message. Arguments are captured before the source.
- `0600` made the snapshot unreadable to the `vyos` user, so `status` and
`diff` — the two commands you run while deciding whether to restore — showed
nothing. Now 0660 root:vyattacfg, matching /config/config.boot.
Installed on both routers with the current, verified-working config pinned
(vyos001 1029 lines, vyos002 1019).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
|
||
|
Michal
|
f4984e3962 |
fix(vyos): health-check the 10 gig primary so failover actually fires
The 10 gig line was primary by route distance alone, which only fails over when bond0.53 loses carrier or its DHCP lease. An ISP that keeps the link up while dropping traffic -- the common failure -- would black-hole everything, because a DHCP-installed route has nothing to withdraw it. `protocols failover` now owns the live default route and pings two targets bound to the interface, so the backup can never be validated through the primary's path. Rehearsed on the labsim router: failover and failback both inside 5s with the router's own interface still UP. The vif keeps default-route-distance rather than no-default-route, demoted below Vodafone. vyos-failover resolves a dhcp-interface gateway by reading new_routers out of /run/dhclient/dhclient_<if>.lease, and no-default-route leaves that field EMPTY -- the daemon then finds no next hop and installs nothing. Observed on vyos001: the default route fell through to Vodafone. Preference is now failover's kernel route (distance 0) > pppoe (10) > DHCP (210), so the demoted route can never re-create the black hole it exists to avoid. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH |
||
|
Michal
|
7b5331ddcd |
fix(migration): the backup has no WAN by design; stop failing it for that
The cutover succeeded on vyos001 -- gateway live, bond0.53 holding 87.192.101.48 with the cloned MAC, kea serving, clients routing out through NAT. vyos002 then ran the same script and was judged unhealthy, because the mandatory checks are "default route exists / internet reachable / DNS resolves" and the backup deliberately holds its WAN interfaces DOWN. Its config was correct; the check did not apply to it. Confirmed by hand before the timer could revert a good config. This is the third instance of one mistake: asserting a condition that is not true of the box being checked. First requiring every WAN when one suffices, now requiring a WAN on the box that is configured not to have one. A delta containing `interfaces ... disable` for the WAN now identifies the backup, and the WAN-dependent checks are skipped with a note. kea and the DNS forwarder remain mandatory on both -- those are what the backup must actually be able to do. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH |
||
|
Michal
|
ce6911c196 |
fix(migration): require a working WAN, not every WAN
This reverted a cutover that had actually succeeded.
The evidence, from the revert tearing it down:
dhclient: DHCPRELEASE of 87.192.101.48 on bond0.53 to 185.232.119.244
vtysh: "no ip route 0.0.0.0/0 87.192.96.1 bond0.53 tag 210 1"
netlinkd: RTM_NEWLINK -> bond0.53, mac=f0:9f:c2:12:9b:4f
bond0.53 came up with the cloned MAC and was handed 87.192.101.48 -- the exact
public address the USG holds -- with a default route via the real ISP gateway.
kea was serving live LAN clients at the same moment (10.0.0.12, 10.0.0.13,
192.168.8.28). The gateway was working.
The only failure was pppoe0: ppp@pppoe0.service exited 5/NOTINSTALLED. That is
the Vodafone FAILOVER line, and the health check listed "pppoe0 has an address"
as mandatory, so a working gateway was torn down because its backup WAN was
down. The check encoded "every WAN must work" when the requirement is "the box
must reach the internet".
Now: default route, reachability and DNS are mandatory; each WAN interface is
reported individually but fatal on neither. A failover line being down is worth
seeing, not worth reverting for.
This also incidentally settles the last genuine unknown in the migration, which
could not be tested any other way: the ISP does hand the same lease to the
cloned MAC. That was the one thing I had said was unknowable until the USG let
go of it.
Note the earlier polling fix (
|
||
|
Michal
|
54b21fa9ff |
fix(migration): poll for WAN health instead of sampling once at 25s
A real cutover attempt reported failure and reverted a configuration that may well have been fine. The health check waited a fixed 25 seconds and then judged: [switch] committed. Waiting 25s for PPPoE and services to settle... [switch] FAIL pppoe0 has an address 25s is far too short for a WAN. PPPoE alone is PADI/PADO/PADR/PADS followed by LCP, authentication and IPCP -- routinely 15-30s on its own. Both lines had also just been released by the USG seconds earlier, and ISPs commonly hold the previous session and MAC binding for minutes before leasing to the "same" CPE again, which is exactly what a cloned MAC looks like from their side. The one thing the design could not tolerate was being impatient, and it was. Now polls every 15s up to HEALTH_BUDGET (default 180s), reporting progress, and stops early the moment everything is healthy. The budget deliberately finishes long before commit-confirm fires -- 180s against a 10 minute timer leaves 420s of margin -- so the decision to confirm or revert stays ours rather than being made by the timer. Also recorded while chasing this: the earlier claim that VLANs 51/53 are not trunked to the firewalls was WRONG, and the UniFi port settings disprove it -- those LAG ports are Native VLAN Management (1) with Tagged VLAN Management set to Allow All. My evidence never supported the claim: a passive RX count cannot distinguish an absent VLAN from a quiet one, because switches do not flood unicast, and the active DHCP probe used a random MAC that an ISP binding to its registered CPE would ignore regardless. Both observations fit a perfectly healthy trunk. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH |
||
|
Michal
|
63061e6e7e |
feat(migration): peer link cabled and verified; conntrack-sync enabled in deltas
eth3 <-> eth3 direct cable is in. Both ends negotiated 2500Mb full duplex -- these are 2.5 GbE ports, not the 1G I had assumed. Carrier alone proves nothing, so the link was tested end to end with temporary kernel-level addresses (never committed to VyOS config, removed afterwards): 3/3 packets, 0% loss, 0.371ms average. A cable can show carrier and still not pass traffic; now it is known to. Deltas regenerated with --conntrack-link and installed on both boxes: vyos001 nat=21 fw=58 conntrack=9 eth3=10.255.255.1/30 disable=0 vyos002 nat=21 fw=58 conntrack=9 eth3=10.255.255.2/30 disable=2 Identical apart from the peer /30, the VRRP/DHCP-HA roles, and the two disable lines holding vyos002's WAN down. Both still report mode=unifi and nothing about their behaviour has changed -- eth3 carries no address in the running config, and conntrack-sync appears only in the delta, which is applied at cutover. Not verified: multicast on the peer link. `ping -I eth3 224.0.0.1` drew no responders, but that is the all-hosts group which VyOS need not answer, so it proves nothing either way. conntrack-sync's own multicast (225.0.0.50) was proven working in labsim over bond0.10, and this is a point-to-point link, so the risk is low -- but it is untested on this specific cable and worth watching at cutover. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH |
||
|
Michal
|
ccdd1e7e49 |
fix(migration): both boxes carry WAN and NAT; the backup just holds it down
"No NAT? How are we supposed to get internet?" -- a fair question that exposed a
worse design than I had admitted. Internet did work, but only via vyos001: NAT
and the entire WAN were gated behind --with-wan, so vyos002 would have held the
LAN VIPs and routed between VLANs with no path to the outside at all. Failover
would have preserved addressing and lost the internet.
The fix rests on a checked fact rather than an assumption: VyOS WARNS but still
commits when a NAT rule names an interface that does not exist
("Interface bond0.53 for source NAT rule 900 does not exist!"). Verified on a
real VyOS before relying on it.
So both boxes now get the identical WAN, NAT, port-forward and firewall config,
and the backup's two WAN interfaces are simply set `disable`. The cloned WAN MAC
is therefore never live on two boxes at once, while everything needed to route
and masquerade is already in place. The two deltas are now byte-identical apart
from VRRP priority, own/peer addresses, DHCP HA role, the conntrack /30 -- and
the two disable lines.
Taking over the internet path becomes deleting two lines rather than
reconstructing NAT under pressure:
delete interfaces bonding bond0 vif 53 disable
delete interfaces pppoe pppoe0 disable
Both boxes now: 21 NAT rules, 58 firewall rules, full PPPoE. Backup delta
validated against a real VyOS config with the disable lines present -- commits
clean. Runbook updated with the takeover procedure and the warning that it must
only be done when vyos001 is genuinely down.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
|
||
|
Michal
|
64e748ea94 |
test(labsim): conntrack-sync verified, and it exposed a delta defect
conntrack-sync proven working on the sim pair -- bidirectional replication with
zero errors:
MASTER internal 34 external(from peer) 52 62 pkts sent / 109 recv 0 err
BACKUP internal 76 external(from peer) 36 142 pkts sent / 73 recv 0 err
Getting there required learning something that changes the production config:
**VyOS only engages conntrack when a firewall or NAT is configured.** With
neither present, both routers reported zero conntrack entries and conntrack-sync
had nothing to replicate. Adding a single state-matching forward rule turned
tracking on and replication began immediately.
That is a defect in the delta, not just a test artifact. NAT and the firewall
were both gated behind --with-wan, so the BACKUP would have had neither -- it
would not have tracked connections at all, and replicated entries are useless to
a box whose conntrack is not engaged. Exactly the failure that only shows up
during a failover, when it is too late to notice.
Fixed: a stateful forward rule (accept established/related, default-action
accept) is now emitted on BOTH boxes, outside the WAN gate. Only NAT and the
WAN-scoped rules remain master-only. Verified: vyos002 now carries stateful
tracking and conntrack-sync but zero NAT lines. Master delta re-validated
against a real VyOS config -- no errors.
Also incidentally confirmed no-preempt: router1 rebooted and came back as
BACKUP rather than seizing the VIP, which is the opposite of what the production
pair did this afternoon (still on default preempt until cutover).
Two traps recorded while doing this:
- The detached `setsid nohup` config-apply pattern can strand a VyOS config
session. An orphaned session (dirs under /opt/vyatta/config/tmp/, PID long
dead) blocked every subsequent `set` on that box with a bare "Set failed",
and the dirs are overlay mounts so they cannot simply be deleted. Rebooting
cleared it. This pattern is used to survive losing SSH mid-change, so it is
worth knowing it has a failure mode of its own.
- Only VLAN 10 passes traffic between the two sim routers; every other VLAN
fails ARP despite identical vlan_mode/tag/trunks on both OVS bonds and
distinct MACs. VRRP forms on all six groups regardless. The sync link had to
be bond0.10 as a result. OVS-specific, absent in production, but it means
the sim proves mechanism rather than topology.
Production deltas regenerated with --conntrack-link: eth3 at 10.255.255.1/30 and
.2/30 awaiting the cable, which is not yet plugged (carrier=0 on both).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
|
||
|
Michal
|
952f5c66e3 |
feat(migration): complete the VyOS HA stack per the official docs
Prompted by "I thought we tested HA on libvirt" -- checking rather than
recalling showed the sim has ONE VyOS router with zero high-availability
config. VRRP was configured and running on the real pair, but it is only one of
four parts of what VyOS considers an HA pair.
Against docs.vyos.io (highavailability, conntrack-sync, dhcp-server, and the HA
walkthrough), three gaps are now closed in the delta:
- VRRP was multicast-only with default preemption. Added unicast
hello-source-address/peer-address per group, as the walkthrough does, plus
no-preempt. Without no-preempt a recovered box reclaims the VIP before
conntrack state has synced and drops every established connection; the docs
are explicit that preempt-delay must otherwise be >= purge-timeout.
The per-VLAN node addresses are a table, not derived: VLAN 3 is .4/.5 while
every other VLAN is .252/.253.
- DHCP high-availability, which fixes a real defect rather than adding a
feature. Both boxes carried the full 6 subnets and 84 static-mappings, so
after cutover two kea instances would have raced on the same broadcast
domains. Now active-passive with primary/secondary and swapped
source/remote, syncing over TCP 647 on the LoT addresses. Each subnet
already carries the unique subnet-id kea HA requires, and the peer name
deliberately differs from both host-names.
- conntrack-sync over a dedicated eth3 <-> eth3 link, gated behind
--conntrack-link because it needs a cable that is not plugged in yet. This
is what the peer cable is actually for -- VRRP does not want one, since its
hellos must travel on the segment they protect.
VRRP failover exercised on the production pair, which is free to break today
because nothing uses the .254 VIPs: keepalived stopped on vyos001, all six VIPs
moved to vyos002 within 12s, and returned on restart (preemption still default
on the live boxes). Both boxes clean afterwards, no config drift.
Master delta validated against vyos001's real running config on the sim router
before installing. Installed on both: 6 no-preempt, 6 unicast pairs, DHCP HA
primary/secondary respectively.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
|
||
|
Michal
|
f81c94af43 |
feat(migration): dual WAN, cloned MAC, and the new 10.8.0.0/23 Private VLAN
Two corrections from reading the live USG instead of trusting UniFi's fields,
which report wan_type=dhcp for both WANs and are simply wrong:
- There are TWO WANs, not one. WAN2 is the 10 gig ISP on VLAN 53, plain DHCP
with a PUBLIC address (87.192.101.48/21, gw 87.192.96.1) on the USG's eth2 --
and it is what actually carries traffic. WAN1 is Vodafone PPPoE on VLAN 51,
the failover. The delta had PPPoE as the only WAN, which would have left the
primary line unconfigured.
- The DHCP lease is bound to MAC, so bond0.53 now clones the USG's WAN2 MAC
(f0:9f:c2:12:9b:4f). That is how VyOS keeps the existing public lease rather
than negotiating a new one -- or getting none, if the ISP allows one per
line. Distances: 10 gig at 1, Vodafone at 10.
Only ONE box may hold the cloned MAC, so --with-wan gates the entire WAN, NAT
and firewall section. vyos001 gets it (320 set lines); vyos002 gets none (234,
zero WAN/NAT/firewall) and routes the LAN only. Pretending both could hold it
would have meant a duplicate MAC on VLAN 53 and a flapping switch table.
Private was rebuilt at 10.8.0.0/23 (VLAN 9) after the old 10.0.8.0/23 was
deleted. bond0.9 and the VRRP group were moved to 10.8.0.252/.253 with VIP
10.8.0.254 on both boxes, and the delta now targets 10.8.0.1.
Creating that network first required breaking a deadlock in UniFi: every LAN
write was rejected with api.err.WanIpOverlapped / 0.0.0.0/0, because WAN1 was
set to DHCP on a line that only speaks PPPoE, so it sat at 0.0.0.0 forever and
the validator treated that as a subnet overlapping everything. Verified
server-side, not a UI bug -- the API rejected it identically. Setting
wan_type=pppoe let it dial (90.241.226.213, MTU 1492), which cleared the phantom
overlap and incidentally PROVED the Vodafone credentials and line work, which
had been listed as untestable before cutover.
dhcp-options no-default-route-dns does not exist; the valid set is client-id,
default-route-distance, host-name, mtu, no-default-route, reject, user-class,
vendor-class-id. Caught by validating the delta against vyos001's real config on
the labsim router before installing.
After adding the network, the gateway's dhcpd.conf was checked with
`dhcpd3 -t -cf` (valid) and confirmed to contain the new subnet only after a
force-provision -- controller state is not device state.
Both boxes: mode unifi, VRRP unchanged, unifi.boot re-captured (232 lines,
carrying the new VLAN 9), no config drift.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
|
||
|
Michal
|
febe4b72bc |
chore(migration): all DNS through VyOS to Google, NAS out of the path
The NAS is legacy for ad.itaz.eu and those records now live in Cloudflare, so
the zone resolves publicly -- verified: nas001.ad.itaz.eu and
kvm-macstudio1.ad.itaz.eu both answer from 8.8.8.8. That removes the reason for
a conditional forward and lets the NAS leave the DNS path entirely.
Two changes:
- `service dns forwarding name-server` is now 8.8.8.8 and 8.8.4.4, the same
pair the USG used on its WAN, instead of 10.0.0.194.
- Every VLAN is handed the gateway as its resolver. UniFi set an explicit
resolver on LoT only (the NAS); carrying that over would have kept the NAS
in the path for one VLAN and not the other five, which is the sort of
asymmetry nobody remembers a year later.
The NAS is still referenced 9 times, all legitimate and checked: 4 NAT
destination rules, the 4 matching firewall accepts for those port forwards, and
its own DHCP reservation. No DNS references remain.
Validated by loading vyos001's real running config on the labsim router and
applying the full delta -- all 318 commands accepted, no errors. Installed on
both boxes and verified in place: priority 200/100, upstream 8.8.8.8 + 8.8.4.4,
six client resolvers, 377 lines each.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
|
||
|
Michal
|
3768657b91 |
chore(migration): firewalls resolve via 8.8.8.8/8.8.4.4
Matches the DNS the USG used on its WAN (wan_dns1/wan_dns2), replacing the
10.0.0.194 I had set earlier. Applied to both boxes and saved; VRRP unchanged
(MASTER/BACKUP), NTP still synced, no config drift.
/config/modes/unifi.boot was RE-CAPTURED on both afterwards. It had been taken
before this change, so the escape hatch would have quietly reverted the
resolver on any rollback -- a snapshot is only an escape hatch for the state it
was taken from.
Two things recorded in the runbook:
- The boxes' name resolution now depends on the internet, so between
unplugging the USG and PPPoE establishing they have no DNS. Harmless:
nothing in the switch resolves a name, and the health checks use DNS
precisely to prove the WAN came up.
- Internal ad.itaz.eu names still resolve via Google, because that zone is
published publicly with private addresses in it (nas001 -> 10.0.0.194,
kvm-macstudio1 -> 192.168.3.8). So no conditional forward was needed --
though it is worth knowing the internal topology is public.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
|
||
|
Michal
|
2a8fcb3bd3 |
fix(migration): the runbook pointed at addresses that die with the USG
The access table led with 192.168.8.143/.144 and offered the LoT addresses as a
fallback ("if unreachable, try"). That is backwards and would have stranded the
operator at the worst moment: the workstation sits on LoT, and reaching
192.168.8.x routes *through the USG*, so those addresses are guaranteed dead the
instant it is unplugged. Measured:
ip route get 192.168.8.143 -> via 10.0.0.1 (the USG)
ip route get 10.0.1.252 -> dev lanbr0 (same L2, no gateway)
10.0.1.252 and .253 are on the LoT VLAN, same broadcast domain as the
workstation, and both answer SSH. They are now the only addresses the runbook
gives, with the k8s ones struck through.
Also recorded: the switch cannot be run before unplugging the USG (two devices
on every gateway address; the guard refuses), so the order is forced. And
during the gap between unplugging and completing the switch there is no
inter-VLAN routing at all -- which means the JetKVMs (Management and kvm) and
Tailscale are NOT fallbacks in that window. LoT SSH is the only remote path;
below it is physical console. Added a step 0: open both SSH sessions and leave
them open before touching anything.
Both boxes are now installed and pass the pre-flight gate: mode unifi,
unifi.boot 231 lines including the reload action, delta at the right priority
(200/100), wan-secrets 0600, script executable, no config drift, VRRP still
MASTER/BACKUP. `vyos-unifi-switch vyos` refuses on both -- all six gateway
addresses detected answering ARP -- and neither box has gained dhcp-server, dns
or nat, so nothing about their behaviour has changed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
|
||
|
Michal
|
fc31013ceb |
fix(migration): apply the reviewed reservation plan, not a recomputed one
This caused a real outage. unifi-reserve-all.py recomputed its plan at --apply time by re-reading stat/sta, so a client that renewed between the dry run and the apply was pinned to whatever transient address it happened to hold at that instant. worker1-k8s0 was reviewed at 192.168.8.13 and written as 192.168.8.242. On its next reboot it could not get an address at all, taking a k8s node down. A plan that gets reviewed and a plan that gets applied must be the same object. The dry run now WRITES the plan to a file and --apply READS it and applies exactly that, reporting any client whose current address has since drifted rather than silently preferring the new value. 1 of 51 diverged; the rest were verified against the reviewed list and were correct. worker1 has been restored to .13 and confirmed: DHCPOFFER for its own MAC returns 192.168.8.13, and the node is up with a full lease and working internet. The second half of the outage was drift between controller and device: the USG was still running config from ~16h before these changes, so the controller looked perfectly correct while the gateway handed out something else. Writing the controller is only half the job, so the script now says so explicitly and gives the force-provision and DHCP-probe commands to verify with. `nmap --script broadcast-dhcp-discover --script-args broadcast-dhcp-discover.mac=...` is the way to prove a specific reservation is live without disturbing the client -- it elicits an OFFER without ever sending a REQUEST. _unifi.py gains post() for device commands. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH |