From d0b733f8315b16bb3f391c5d5a90463e9526ada1 Mon Sep 17 00:00:00 2001 From: Michal Date: Sun, 6 Sep 2026 14:46:44 +0100 Subject: [PATCH] IPv6 follows master, deployed: vyos002 has a tunnel and a gate that holds it shut Applied step 0 to production, backup first, IPv4 untouched throughout. vyos001 held the VIP and the internet the whole way; verified before and after every commit. Order, which is the safety property: he-secrets onto vyos002 -> vrrp-wan-install on BOTH (so the gate exists before the tunnel) -> tunnel + task-scheduler on vyos002 -> route6 + bond0.9 ::2 + router-advert on vyos002 -> link-mtu 1472 on vyos001. The blackhole is confirmed behaviour, not theory. The moment the tunnel committed, vyos002 brought tun0 UP with source 87.192.101.48 -- an address it does not own -- and the reconciler closed it on the next tick: vrrp-wan: not MASTER: bringing tun0 down vrrp-wan: not MASTER: stopping radvd (deprecates the v6 gateway) Same for radvd: VyOS started it on commit, the reconciler stopped it. Both have stayed shut. That is the override's old "it would simply stay down" assumption failing in production exactly as it failed in the sim. wan-drill --dry now reports the same tunnel source on both routers, where it previously read "". Two things found by deploying rather than reading: - commit-confirm hangs non-interactively here, exactly as PPPOE-HA.md records. The ssh timed out leaving an orphaned config-mgmt commit_confirm holding the config lock, with nothing committed. Killed it and used plain commit + save -- safe on the backup, which holds no VIPs, no WAN, and whose LoT path is untouched. Worth knowing the failure is clean: no partial config landed. - he-tunnel-follow's status path died with "WAN_MTU: bad array subscript" on the backup. An empty array subscript is a hard bash error, not an empty expansion, so the :- default never applies -- and a backup has no default route, so it broke on precisely the box whose state you most need to read. Fixed and redeployed to both. This is now live DRIFT against the Pulumi model. Until migration/pulumi-override-he-tunnel-both.json is merged into overrides.json, a pulumi up can strip the tunnel, route6, bond0.9 addresses, router-advert and task-scheduler entry. Recorded there and in the backlog's drift section. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH --- migration/he-tunnel-follow | 8 +++++++- migration/pulumi-override-he-tunnel-both.json | 2 +- 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/migration/he-tunnel-follow b/migration/he-tunnel-follow index 73f8385..a76d33d 100755 --- a/migration/he-tunnel-follow +++ b/migration/he-tunnel-follow @@ -166,7 +166,13 @@ case "${1:-status}" in printf ' wan addr : %s\n' "$(addr_of "${wan:-lo}")" printf ' tunnel : %s\n' "$(ip -br link show "$TUNNEL" 2>/dev/null | awk '{print $2}' || echo '')" printf ' tunnel src : %s\n' "$(tunnel_src)" - printf ' tunnel mtu : %s (want %s)\n' "$(tunnel_mtu)" "${WAN_MTU[${wan:-}]:-?}" + # `${WAN_MTU[$wan]}` with an EMPTY subscript is a hard bash error -- + # "bad array subscript" -- not an empty expansion, and the :- default never + # gets a chance to apply. A backup router has no default route, so `wan` is + # empty there and `status` printed an error line on exactly the box whose + # state you most need to read. Only index the array once there is a key. + printf ' tunnel mtu : %s (want %s)\n' "$(tunnel_mtu)" \ + "$([ -n "${wan:-}" ] && echo "${WAN_MTU[$wan]:-?}" || echo '- (no WAN; this box is not master)')" printf ' he endpoint: %s\n' "$HE_UPDATE_URL" [ -r "$SECRETS" ] && printf ' credentials: present\n' || printf ' credentials: MISSING (%s)\n' "$SECRETS" ;; diff --git a/migration/pulumi-override-he-tunnel-both.json b/migration/pulumi-override-he-tunnel-both.json index 4ce06ba..f55d77b 100644 --- a/migration/pulumi-override-he-tunnel-both.json +++ b/migration/pulumi-override-he-tunnel-both.json @@ -184,5 +184,5 @@ "value": "2592000" } ], - "_staging_note": "STAGED, NOT MERGED -- on purpose, exactly like migration/pulumi-override-pppoe-gated.json was. Another agent runs `pulumi up` on kubernetes-deployment, so merging this IS a production change made by someone else at a time you do not choose. Order of operations, and it matters: (1) place /config/he-secrets on vyos002 and run migration/vrrp-wan-install on BOTH routers, so the runtime gate that holds tun0 down exists there BEFORE the tunnel does; (2) run `npm run vyos:export && npm run vyos:render` immediately before merging, so the model follows whichever router actually holds the WAN; (3) merge. Reversed, vyos002 gets a tunnel with no gate and an UP blackhole interface that attracts the v6 default route. The perRouter/sharedRouterAdvert shape here is descriptive -- fold it into whatever form overrides.json actually supports for per-router values when merging; the vyos001-only precedent is the `routers` key." + "_staging_note": "APPLIED TO THE BOXES 2026-09-06, NOT YET IN THE MODEL -- this file is now a record of live DRIFT, not a proposal. Both routers were configured by hand in the order the note below prescribes, and the mechanism was verified working (vyos002: tun0 DOWN + radvd inactive, both closed by vrrp-wan-reconcile; vyos001: unchanged, internet and IPv6 up). What remains is folding this into infra/vyos/subtrees/overrides.json so `pulumi up` stops being able to revert it. UNTIL THAT MERGE LANDS, a `pulumi up` on kubernetes-deployment may strip the tunnel, the route6, the bond0.9 v6 addresses, the router-advert block and the task-scheduler entry -- taking IPv6 down and, worse, leaving vyos002 half-configured. Run `npm run vyos:export && npm run vyos:render` before any vyos apply. ORDER USED, and it is the safe one: (1) /config/he-secrets onto vyos002; (2) migration/vrrp-wan-install on BOTH, so the runtime gate that holds tun0 down exists BEFORE the tunnel does; (3) tunnel + task-scheduler on vyos002; (4) route6 + bond0.9 ::2 + router-advert on vyos002; (5) vyos001 link-mtu 1480 -> 1472. Reversed at step 2/3, vyos002 gets an UP blackhole that attracts the v6 default route -- confirmed behaviour, not theory: the commit at step 3 DID bring tun0 up with a source address vyos002 does not own, and the reconciler closed it 30s later. NOTE ON commit-confirm: it hangs non-interactively here exactly as PPPOE-HA.md records -- the ssh timed out with an orphaned `config-mgmt commit_confirm` holding the config lock and NOTHING committed. Plain `commit` + `save` was used instead, which is safe on the backup: no VIPs, no WAN, and the LoT path (bond0.10) is untouched by any of this. The perRouter/sharedRouterAdvert shape here is descriptive -- fold it into whatever form overrides.json actually supports for per-router values when merging; the vyos001-only precedent is the `routers` key." } \ No newline at end of file