IPv6 follows master, deployed: vyos002 has a tunnel and a gate that holds it shut
Some checks failed
CI/CD / lint (push) Failing after 9s
CI/CD / test (push) Failing after 9s
CI/CD / typecheck (push) Failing after 25s
CI/CD / build (push) Has been skipped
CI/CD / publish-rpm (push) Has been skipped
CI/CD / publish-deb (push) Has been skipped

Applied step 0 to production, backup first, IPv4 untouched throughout. vyos001
held the VIP and the internet the whole way; verified before and after every
commit.

Order, which is the safety property: he-secrets onto vyos002 -> vrrp-wan-install
on BOTH (so the gate exists before the tunnel) -> tunnel + task-scheduler on
vyos002 -> route6 + bond0.9 ::2 + router-advert on vyos002 -> link-mtu 1472 on
vyos001.

The blackhole is confirmed behaviour, not theory. The moment the tunnel
committed, vyos002 brought tun0 UP with source 87.192.101.48 -- an address it
does not own -- and the reconciler closed it on the next tick:

  vrrp-wan: not MASTER: bringing tun0 down
  vrrp-wan: not MASTER: stopping radvd (deprecates the v6 gateway)

Same for radvd: VyOS started it on commit, the reconciler stopped it. Both have
stayed shut. That is the override's old "it would simply stay down" assumption
failing in production exactly as it failed in the sim.

wan-drill --dry now reports the same tunnel source on both routers, where it
previously read "<no tunnel -- IPv6 cannot survive a failover>".

Two things found by deploying rather than reading:

- commit-confirm hangs non-interactively here, exactly as PPPOE-HA.md records.
  The ssh timed out leaving an orphaned config-mgmt commit_confirm holding the
  config lock, with nothing committed. Killed it and used plain commit + save --
  safe on the backup, which holds no VIPs, no WAN, and whose LoT path is
  untouched. Worth knowing the failure is clean: no partial config landed.
- he-tunnel-follow's status path died with "WAN_MTU: bad array subscript" on the
  backup. An empty array subscript is a hard bash error, not an empty expansion,
  so the :- default never applies -- and a backup has no default route, so it
  broke on precisely the box whose state you most need to read. Fixed and
  redeployed to both.

This is now live DRIFT against the Pulumi model. Until
migration/pulumi-override-he-tunnel-both.json is merged into overrides.json, a
pulumi up can strip the tunnel, route6, bond0.9 addresses, router-advert and
task-scheduler entry. Recorded there and in the backlog's drift section.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH
This commit is contained in:
Michal
2026-09-06 14:46:44 +01:00
parent 395577850c
commit d0b733f831
2 changed files with 8 additions and 2 deletions

View File

@@ -166,7 +166,13 @@ case "${1:-status}" in
printf ' wan addr : %s\n' "$(addr_of "${wan:-lo}")"
printf ' tunnel : %s\n' "$(ip -br link show "$TUNNEL" 2>/dev/null | awk '{print $2}' || echo '<absent>')"
printf ' tunnel src : %s\n' "$(tunnel_src)"
printf ' tunnel mtu : %s (want %s)\n' "$(tunnel_mtu)" "${WAN_MTU[${wan:-}]:-?}"
# `${WAN_MTU[$wan]}` with an EMPTY subscript is a hard bash error --
# "bad array subscript" -- not an empty expansion, and the :- default never
# gets a chance to apply. A backup router has no default route, so `wan` is
# empty there and `status` printed an error line on exactly the box whose
# state you most need to read. Only index the array once there is a key.
printf ' tunnel mtu : %s (want %s)\n' "$(tunnel_mtu)" \
"$([ -n "${wan:-}" ] && echo "${WAN_MTU[$wan]:-?}" || echo '- (no WAN; this box is not master)')"
printf ' he endpoint: %s\n' "$HE_UPDATE_URL"
[ -r "$SECRETS" ] && printf ' credentials: present\n' || printf ' credentials: MISSING (%s)\n' "$SECRETS"
;;

View File

@@ -184,5 +184,5 @@
"value": "2592000"
}
],
"_staging_note": "STAGED, NOT MERGED -- on purpose, exactly like migration/pulumi-override-pppoe-gated.json was. Another agent runs `pulumi up` on kubernetes-deployment, so merging this IS a production change made by someone else at a time you do not choose. Order of operations, and it matters: (1) place /config/he-secrets on vyos002 and run migration/vrrp-wan-install on BOTH routers, so the runtime gate that holds tun0 down exists there BEFORE the tunnel does; (2) run `npm run vyos:export && npm run vyos:render` immediately before merging, so the model follows whichever router actually holds the WAN; (3) merge. Reversed, vyos002 gets a tunnel with no gate and an UP blackhole interface that attracts the v6 default route. The perRouter/sharedRouterAdvert shape here is descriptive -- fold it into whatever form overrides.json actually supports for per-router values when merging; the vyos001-only precedent is the `routers` key."
"_staging_note": "APPLIED TO THE BOXES 2026-09-06, NOT YET IN THE MODEL -- this file is now a record of live DRIFT, not a proposal. Both routers were configured by hand in the order the note below prescribes, and the mechanism was verified working (vyos002: tun0 DOWN + radvd inactive, both closed by vrrp-wan-reconcile; vyos001: unchanged, internet and IPv6 up). What remains is folding this into infra/vyos/subtrees/overrides.json so `pulumi up` stops being able to revert it. UNTIL THAT MERGE LANDS, a `pulumi up` on kubernetes-deployment may strip the tunnel, the route6, the bond0.9 v6 addresses, the router-advert block and the task-scheduler entry -- taking IPv6 down and, worse, leaving vyos002 half-configured. Run `npm run vyos:export && npm run vyos:render` before any vyos apply. ORDER USED, and it is the safe one: (1) /config/he-secrets onto vyos002; (2) migration/vrrp-wan-install on BOTH, so the runtime gate that holds tun0 down exists BEFORE the tunnel does; (3) tunnel + task-scheduler on vyos002; (4) route6 + bond0.9 ::2 + router-advert on vyos002; (5) vyos001 link-mtu 1480 -> 1472. Reversed at step 2/3, vyos002 gets an UP blackhole that attracts the v6 default route -- confirmed behaviour, not theory: the commit at step 3 DID bring tun0 up with a source address vyos002 does not own, and the reconciler closed it 30s later. NOTE ON commit-confirm: it hangs non-interactively here exactly as PPPOE-HA.md records -- the ssh timed out with an orphaned `config-mgmt commit_confirm` holding the config lock and NOTHING committed. Plain `commit` + `save` was used instead, which is safe on the backup: no VIPs, no WAN, and the LoT path (bond0.10) is untouched by any of this. The perRouter/sharedRouterAdvert shape here is descriptive -- fold it into whatever form overrides.json actually supports for per-router values when merging; the vyos001-only precedent is the `routers` key."
}