Files
lab/migration/pulumi-override-he-tunnel-both.json

188 lines
8.2 KiB
JSON
Raw Normal View History

IPv6 was never HA, and the WAN becoming HA is what exposed it Reviewed the parked IPv6 task against the PPPoE-HA work of 2026-09-05/06. The gate that parked it ("WI-8 before IPv6") is cleared, but the same work invalidated the assumption the IPv6 design rested on. Verified on the live routers: vyos002 has no tun0, no he-tunnel-follow, no he-secrets, no VLAN 9 prefix and no route6 ::/0 -- only the pre-staged default-deny v6 firewall, which is correctly on both. Failover is now automatic and drill-proven, so every failover takes the whole v6 estate down for as long as vyos002 holds the VIP. Four things that came out of checking rather than reading: - PPPOE-HA.md's "tun0 survived untouched and IPv6 stayed up at 15.5ms" does not follow from its own premise and is corrected in place. The endpoint address is stable, but it MOVES to vyos002, which has nothing to decapsulate protocol 41. wan-drill had no IPv6 check at all, which is why nobody caught it. - A 22-second near-miss: vif53-pin-boot-disable bounced the 10 gig, he-tunnel- follow ticked once and saw the PPPoE address, and vyos-failover restored the route 22s before the second tick would have pointed HE at an address Vodafone reissues on every dial. - VyOS does NOT leave a tunnel down when its source-address is absent (the override's stated reason for leaving IPv6 single-homed). Measured in labsim: it commits rc=0 and brings the link UP -- a blackhole that attracts the v6 default route. The runtime gate is load-bearing, like the PPPoE gate. - The RA link-mtu was pinned at 1480 while the tunnel correctly drops to 1472 on the PPPoE path. Mechanism, mirroring PPPoE HA -- identical config on both, gated at runtime, no commit in the failover path: - vrrp-wan-reconcile: a v6 kernel plane. tun0 and radvd follow the VIP; radvd is stopped BEFORE the WAN goes so its farewell RA (router-lifetime 0) still has a path out. The WAN early-exits became if-blocks so the plane runs every tick. It deliberately does NOT call he-tunnel-follow: that would halve the hysteresis the near-miss above showed we depend on. - he-tunnel-follow: a master guard reading the same vrrp-wan.conf VIP, so the backup copy cannot point HE at its own idle PPPoE line, plus a stubbable HE_UPDATE_URL. - vrrp-wan-install carries both, so --check and the upgrade runbook cover IPv6. - wan-drill measures IPv6 in both timing loops and asserts zero HE API calls across a router failover. labsim finally has an HE endpoint, closing the gap the override itself cited as why this was never rehearsed. Both ISP islands already share the libvirt network, so that becomes the backbone and HE lives behind it on one address reachable over either WAN. Proven in the sim: backup tun=DOWN radvd=inactive, master tun=UP radvd=active, hysteresis then HE call then MTU 1480->1472, and VLAN 9 hosts autoconfiguring from the RA. The end-to-end v6 datapath is NOT yet proven -- inter-island transit crosses libvirt NAT and the return path is lost. Recorded as a KNOWN SIM GAP rather than papered over. The model change is staged, not merged: another agent runs pulumi up on that repo, and the gate must exist on vyos002 before the tunnel does. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-09-06 14:30:44 +01:00
{
"id": "he-ipv6-tunnel",
"reason": "6in4 tunnel to Hurricane Electric, bringing 2001:470:187e::/48 in, on BOTH routers. SUPERSEDES he-ipv6-tunnel-vyos001, whose reason said 'vyos002 carries bond0.53 disabled, so the source address does not exist there and the tunnel would simply stay down -- adding it there is for a later takeover story, not now.' Both halves of that expired on 2026-09-06. `vif 53 disable` is no longer a property of vyos002: it is RUNTIME state owned by vrrp-wan-reconcile, keyed on who holds the management VIP, and deliberately absent from this model (see pppoe-gated-not-config-disabled and PPPOE-HA.md). And the takeover story shipped -- a drill moved the WAN and back, 52s and 36s. IPv6 did not follow it, so every failover took the whole v6 estate down for as long as vyos002 held the VIP. WHY THIS IS CHEAP: 87.192.101.48 is the 10 gig lease bound to the cloned MAC, and the drill proved the ISP re-issues THE SAME address to that MAC on the other router's port. The tunnel endpoint is therefore stable across the pair, so a router-level failover needs no HE API call at all -- only the tunnel present on both boxes and live on exactly one. HE updates remain solely for the within-box fall back to PPPoE, which /config/he-tunnel-follow already handles. WHY THE RUNTIME GATE IS LOAD-BEARING, not a nicety: measured in labsim 2026-09-06, VyOS ACCEPTS a tunnel whose source-address does not exist on the box (commit rc=0) and brings the link UP anyway. It is a blackhole that will happily attract the v6 default route -- not the inert node the old reason assumed. vrrp-wan-reconcile holds tun0 down on the backup and brings it up on the master, at kernel level, with no commit in the failover path. Verified in the sim: backup tun=DOWN radvd=inactive, master tun=UP radvd=active. MTU 1480, not 1500: 6in4 adds a 20-byte outer IPv4 header. Leave it at 1500 and IPv6 appears to work while large transfers hang. The RA link-mtu is 1472 -- the PPPoE figure -- deliberately, on BOTH routers: it cannot be reconciled at runtime because it needs a commit, so advertise the lower of the two paths and be correct on either WAN. Production previously pinned 1480 and was silently wrong whenever the WAN fell back. bond0.9 takes ::1 on vyos001 and ::2 on vyos002 -- NOT the same address: SLAAC hosts take their gateway from the advertising router's link-local, so the global address need not move, and duplicating it would only produce a DAD conflict. default-preference is high on vyos001 and low on vyos002 so that if both ever advertise at once -- radvd's config is rendered into /run and a booting backup starts it before VRRP has decided -- hosts prefer the normal master, while a genuinely dead vyos001 still leaves vyos002 as the only router on the link. No firewall change is needed: the IPv6 firewall accepts only from interface-group LAN, so tun0 is untrusted by default. It is already default-deny on BOTH routers -- that ordering held. REHEARSAL: labsim/labsim-he-endpoint.sh now builds a fake HE endpoint and stub tunnelbroker API, closing the 'the sim has no public IPv4 and no HE endpoint' gap the old reason cited as why this was never tested. labsim/labsim-ipv6-ha-test.sh is the matrix. Mechanism proven there; the end-to-end v6 datapath is not yet, because the sim's inter-island transit crosses libvirt NAT -- see that file's KNOWN SIM GAP header.",
"set": [
{
"path": [
"interfaces",
"tunnel",
"tun0",
"encapsulation"
],
"value": "sit"
},
{
"path": [
"interfaces",
"tunnel",
"tun0",
"source-address"
],
"value": "87.192.101.48"
},
{
"path": [
"interfaces",
"tunnel",
"tun0",
"remote"
],
"value": "216.66.88.98"
},
{
"path": [
"interfaces",
"tunnel",
"tun0",
"address"
],
"value": "2001:470:1f1c:f6::2/64"
},
{
"path": [
"interfaces",
"tunnel",
"tun0",
"mtu"
],
"value": "1480"
},
{
"path": [
"interfaces",
"tunnel",
"tun0",
"description"
],
"value": "HE 6in4 tunnel - 2001:470:187e::/48"
},
{
"path": [
"protocols",
"static",
"route6",
"::/0",
"next-hop",
"2001:470:1f1c:f6::1"
],
"value": {}
},
{
"path": [
"system",
"task-scheduler",
"task",
"he-tunnel-follow",
"executable",
"path"
],
"value": "/config/he-tunnel-follow"
},
{
"path": [
"system",
"task-scheduler",
"task",
"he-tunnel-follow",
"executable",
"arguments"
],
"value": "run"
},
{
"path": [
"system",
"task-scheduler",
"task",
"he-tunnel-follow",
"interval"
],
"value": "1m"
}
],
"perRouter": {
"vyos001": [
{
"path": [
"interfaces",
"bonding",
"bond0",
"vif",
"9",
"address"
],
"value": "2001:470:187e:9::1/64"
},
{
"path": [
"service",
"router-advert",
"interface",
"bond0.9",
"default-preference"
],
"value": "high"
}
],
"vyos002": [
{
"path": [
"interfaces",
"bonding",
"bond0",
"vif",
"9",
"address"
],
"value": "2001:470:187e:9::2/64"
},
{
"path": [
"service",
"router-advert",
"interface",
"bond0.9",
"default-preference"
],
"value": "low"
}
]
},
"sharedRouterAdvert": [
{
"path": [
"service",
"router-advert",
"interface",
"bond0.9",
"link-mtu"
],
"value": "1472"
},
{
"path": [
"service",
"router-advert",
"interface",
"bond0.9",
"prefix",
"2001:470:187e:9::/64",
"preferred-lifetime"
],
"value": "604800"
},
{
"path": [
"service",
"router-advert",
"interface",
"bond0.9",
"prefix",
"2001:470:187e:9::/64",
"valid-lifetime"
],
"value": "2592000"
}
],
IPv6 follows master, deployed: vyos002 has a tunnel and a gate that holds it shut Applied step 0 to production, backup first, IPv4 untouched throughout. vyos001 held the VIP and the internet the whole way; verified before and after every commit. Order, which is the safety property: he-secrets onto vyos002 -> vrrp-wan-install on BOTH (so the gate exists before the tunnel) -> tunnel + task-scheduler on vyos002 -> route6 + bond0.9 ::2 + router-advert on vyos002 -> link-mtu 1472 on vyos001. The blackhole is confirmed behaviour, not theory. The moment the tunnel committed, vyos002 brought tun0 UP with source 87.192.101.48 -- an address it does not own -- and the reconciler closed it on the next tick: vrrp-wan: not MASTER: bringing tun0 down vrrp-wan: not MASTER: stopping radvd (deprecates the v6 gateway) Same for radvd: VyOS started it on commit, the reconciler stopped it. Both have stayed shut. That is the override's old "it would simply stay down" assumption failing in production exactly as it failed in the sim. wan-drill --dry now reports the same tunnel source on both routers, where it previously read "<no tunnel -- IPv6 cannot survive a failover>". Two things found by deploying rather than reading: - commit-confirm hangs non-interactively here, exactly as PPPOE-HA.md records. The ssh timed out leaving an orphaned config-mgmt commit_confirm holding the config lock, with nothing committed. Killed it and used plain commit + save -- safe on the backup, which holds no VIPs, no WAN, and whose LoT path is untouched. Worth knowing the failure is clean: no partial config landed. - he-tunnel-follow's status path died with "WAN_MTU: bad array subscript" on the backup. An empty array subscript is a hard bash error, not an empty expansion, so the :- default never applies -- and a backup has no default route, so it broke on precisely the box whose state you most need to read. Fixed and redeployed to both. This is now live DRIFT against the Pulumi model. Until migration/pulumi-override-he-tunnel-both.json is merged into overrides.json, a pulumi up can strip the tunnel, route6, bond0.9 addresses, router-advert and task-scheduler entry. Recorded there and in the backlog's drift section. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-09-06 14:46:44 +01:00
"_staging_note": "APPLIED TO THE BOXES 2026-09-06, NOT YET IN THE MODEL -- this file is now a record of live DRIFT, not a proposal. Both routers were configured by hand in the order the note below prescribes, and the mechanism was verified working (vyos002: tun0 DOWN + radvd inactive, both closed by vrrp-wan-reconcile; vyos001: unchanged, internet and IPv6 up). What remains is folding this into infra/vyos/subtrees/overrides.json so `pulumi up` stops being able to revert it. UNTIL THAT MERGE LANDS, a `pulumi up` on kubernetes-deployment may strip the tunnel, the route6, the bond0.9 v6 addresses, the router-advert block and the task-scheduler entry -- taking IPv6 down and, worse, leaving vyos002 half-configured. Run `npm run vyos:export && npm run vyos:render` before any vyos apply. ORDER USED, and it is the safe one: (1) /config/he-secrets onto vyos002; (2) migration/vrrp-wan-install on BOTH, so the runtime gate that holds tun0 down exists BEFORE the tunnel does; (3) tunnel + task-scheduler on vyos002; (4) route6 + bond0.9 ::2 + router-advert on vyos002; (5) vyos001 link-mtu 1480 -> 1472. Reversed at step 2/3, vyos002 gets an UP blackhole that attracts the v6 default route -- confirmed behaviour, not theory: the commit at step 3 DID bring tun0 up with a source address vyos002 does not own, and the reconciler closed it 30s later. NOTE ON commit-confirm: it hangs non-interactively here exactly as PPPOE-HA.md records -- the ssh timed out with an orphaned `config-mgmt commit_confirm` holding the config lock and NOTHING committed. Plain `commit` + `save` was used instead, which is safe on the backup: no VIPs, no WAN, and the LoT path (bond0.10) is untouched by any of this. The perRouter/sharedRouterAdvert shape here is descriptive -- fold it into whatever form overrides.json actually supports for per-router values when merging; the vyos001-only precedent is the `routers` key."
IPv6 was never HA, and the WAN becoming HA is what exposed it Reviewed the parked IPv6 task against the PPPoE-HA work of 2026-09-05/06. The gate that parked it ("WI-8 before IPv6") is cleared, but the same work invalidated the assumption the IPv6 design rested on. Verified on the live routers: vyos002 has no tun0, no he-tunnel-follow, no he-secrets, no VLAN 9 prefix and no route6 ::/0 -- only the pre-staged default-deny v6 firewall, which is correctly on both. Failover is now automatic and drill-proven, so every failover takes the whole v6 estate down for as long as vyos002 holds the VIP. Four things that came out of checking rather than reading: - PPPOE-HA.md's "tun0 survived untouched and IPv6 stayed up at 15.5ms" does not follow from its own premise and is corrected in place. The endpoint address is stable, but it MOVES to vyos002, which has nothing to decapsulate protocol 41. wan-drill had no IPv6 check at all, which is why nobody caught it. - A 22-second near-miss: vif53-pin-boot-disable bounced the 10 gig, he-tunnel- follow ticked once and saw the PPPoE address, and vyos-failover restored the route 22s before the second tick would have pointed HE at an address Vodafone reissues on every dial. - VyOS does NOT leave a tunnel down when its source-address is absent (the override's stated reason for leaving IPv6 single-homed). Measured in labsim: it commits rc=0 and brings the link UP -- a blackhole that attracts the v6 default route. The runtime gate is load-bearing, like the PPPoE gate. - The RA link-mtu was pinned at 1480 while the tunnel correctly drops to 1472 on the PPPoE path. Mechanism, mirroring PPPoE HA -- identical config on both, gated at runtime, no commit in the failover path: - vrrp-wan-reconcile: a v6 kernel plane. tun0 and radvd follow the VIP; radvd is stopped BEFORE the WAN goes so its farewell RA (router-lifetime 0) still has a path out. The WAN early-exits became if-blocks so the plane runs every tick. It deliberately does NOT call he-tunnel-follow: that would halve the hysteresis the near-miss above showed we depend on. - he-tunnel-follow: a master guard reading the same vrrp-wan.conf VIP, so the backup copy cannot point HE at its own idle PPPoE line, plus a stubbable HE_UPDATE_URL. - vrrp-wan-install carries both, so --check and the upgrade runbook cover IPv6. - wan-drill measures IPv6 in both timing loops and asserts zero HE API calls across a router failover. labsim finally has an HE endpoint, closing the gap the override itself cited as why this was never rehearsed. Both ISP islands already share the libvirt network, so that becomes the backbone and HE lives behind it on one address reachable over either WAN. Proven in the sim: backup tun=DOWN radvd=inactive, master tun=UP radvd=active, hysteresis then HE call then MTU 1480->1472, and VLAN 9 hosts autoconfiguring from the RA. The end-to-end v6 datapath is NOT yet proven -- inter-island transit crosses libvirt NAT and the return path is lost. Recorded as a KNOWN SIM GAP rather than papered over. The model change is staged, not merged: another agent runs pulumi up on that repo, and the gate must exist on vyos002 before the tunnel does. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
2026-09-06 14:30:44 +01:00
}