lab/migration/pulumi-override-he-tunnel-both.json

{
  "id": "he-ipv6-tunnel",
  "reason": "6in4 tunnel to Hurricane Electric, bringing 2001:470:187e::/48 in, on BOTH routers. SUPERSEDES he-ipv6-tunnel-vyos001, whose reason said 'vyos002 carries bond0.53 disabled, so the source address does not exist there and the tunnel would simply stay down -- adding it there is for a later takeover story, not now.' Both halves of that expired on 2026-09-06. `vif 53 disable` is no longer a property of vyos002: it is RUNTIME state owned by vrrp-wan-reconcile, keyed on who holds the management VIP, and deliberately absent from this model (see pppoe-gated-not-config-disabled and PPPOE-HA.md). And the takeover story shipped -- a drill moved the WAN and back, 52s and 36s. IPv6 did not follow it, so every failover took the whole v6 estate down for as long as vyos002 held the VIP. WHY THIS IS CHEAP: 87.192.101.48 is the 10 gig lease bound to the cloned MAC, and the drill proved the ISP re-issues THE SAME address to that MAC on the other router's port. The tunnel endpoint is therefore stable across the pair, so a router-level failover needs no HE API call at all -- only the tunnel present on both boxes and live on exactly one. HE updates remain solely for the within-box fall back to PPPoE, which /config/he-tunnel-follow already handles. WHY THE RUNTIME GATE IS LOAD-BEARING, not a nicety: measured in labsim 2026-09-06, VyOS ACCEPTS a tunnel whose source-address does not exist on the box (commit rc=0) and brings the link UP anyway. It is a blackhole that will happily attract the v6 default route -- not the inert node the old reason assumed. vrrp-wan-reconcile holds tun0 down on the backup and brings it up on the master, at kernel level, with no commit in the failover path. Verified in the sim: backup tun=DOWN radvd=inactive, master tun=UP radvd=active. MTU 1480, not 1500: 6in4 adds a 20-byte outer IPv4 header. Leave it at 1500 and IPv6 appears to work while large transfers hang. The RA link-mtu is 1472 -- the PPPoE figure -- deliberately, on BOTH routers: it cannot be reconciled at runtime because it needs a commit, so advertise the lower of the two paths and be correct on either WAN. Production previously pinned 1480 and was silently wrong whenever the WAN fell back. bond0.9 takes ::1 on vyos001 and ::2 on vyos002 -- NOT the same address: SLAAC hosts take their gateway from the advertising router's link-local, so the global address need not move, and duplicating it would only produce a DAD conflict. default-preference is high on vyos001 and low on vyos002 so that if both ever advertise at once -- radvd's config is rendered into /run and a booting backup starts it before VRRP has decided -- hosts prefer the normal master, while a genuinely dead vyos001 still leaves vyos002 as the only router on the link. No firewall change is needed: the IPv6 firewall accepts only from interface-group LAN, so tun0 is untrusted by default. It is already default-deny on BOTH routers -- that ordering held. REHEARSAL: labsim/labsim-he-endpoint.sh now builds a fake HE endpoint and stub tunnelbroker API, closing the 'the sim has no public IPv4 and no HE endpoint' gap the old reason cited as why this was never tested. labsim/labsim-ipv6-ha-test.sh is the matrix. Mechanism proven there; the end-to-end v6 datapath is not yet, because the sim's inter-island transit crosses libvirt NAT -- see that file's KNOWN SIM GAP header.",
  "set": [
    {
      "path": [
        "interfaces",
        "tunnel",
        "tun0",
        "encapsulation"
      ],
      "value": "sit"
    },
    {
      "path": [
        "interfaces",
        "tunnel",
        "tun0",
        "source-address"
      ],
      "value": "87.192.101.48"
    },
    {
      "path": [
        "interfaces",
        "tunnel",
        "tun0",
        "remote"
      ],
      "value": "216.66.88.98"
    },
    {
      "path": [
        "interfaces",
        "tunnel",
        "tun0",
        "address"
      ],
      "value": "2001:470:1f1c:f6::2/64"
    },
    {
      "path": [
        "interfaces",
        "tunnel",
        "tun0",
        "mtu"
      ],
      "value": "1480"
    },
    {
      "path": [
        "interfaces",
        "tunnel",
        "tun0",
        "description"
      ],
      "value": "HE 6in4 tunnel - 2001:470:187e::/48"
    },
    {
      "path": [
        "protocols",
        "static",
        "route6",
        "::/0",
        "next-hop",
        "2001:470:1f1c:f6::1"
      ],
      "value": {}
    },
    {
      "path": [
        "system",
        "task-scheduler",
        "task",
        "he-tunnel-follow",
        "executable",
        "path"
      ],
      "value": "/config/he-tunnel-follow"
    },
    {
      "path": [
        "system",
        "task-scheduler",
        "task",
        "he-tunnel-follow",
        "executable",
        "arguments"
      ],
      "value": "run"
    },
    {
      "path": [
        "system",
        "task-scheduler",
        "task",
        "he-tunnel-follow",
        "interval"
      ],
      "value": "1m"
    }
  ],
  "perRouter": {
    "vyos001": [
      {
        "path": [
          "interfaces",
          "bonding",
          "bond0",
          "vif",
          "9",
          "address"
        ],
        "value": "2001:470:187e:9::1/64"
      },
      {
        "path": [
          "service",
          "router-advert",
          "interface",
          "bond0.9",
          "default-preference"
        ],
        "value": "high"
      }
    ],
    "vyos002": [
      {
        "path": [
          "interfaces",
          "bonding",
          "bond0",
          "vif",
          "9",
          "address"
        ],
        "value": "2001:470:187e:9::2/64"
      },
      {
        "path": [
          "service",
          "router-advert",
          "interface",
          "bond0.9",
          "default-preference"
        ],
        "value": "low"
      }
    ]
  },
  "sharedRouterAdvert": [
    {
      "path": [
        "service",
        "router-advert",
        "interface",
        "bond0.9",
        "link-mtu"
      ],
      "value": "1472"
    },
    {
      "path": [
        "service",
        "router-advert",
        "interface",
        "bond0.9",
        "prefix",
        "2001:470:187e:9::/64",
        "preferred-lifetime"
      ],
      "value": "604800"
    },
    {
      "path": [
        "service",
        "router-advert",
        "interface",
        "bond0.9",
        "prefix",
        "2001:470:187e:9::/64",
        "valid-lifetime"
      ],
      "value": "2592000"
    }
  ],
  "_staging_note": "MERGED 2026-09-06 as kubernetes-deployment@6d4e080 on main. This file is now only a historical record; the live model is infra/vyos/subtrees/overrides.json, where the four IPv6 overrides (he-ipv6-tunnel, he-tunnel-follow-scheduler, firewall-accept-he-6in4, ipv6-vlan9-private + ipv6-vlan9-addr-vyos00[12]) are scoped to both routers. vyos:verify reports 533/534 nodes, zero drift. TWO THINGS THE MERGE ITSELF CAUGHT, worth carrying: (1) firewall-accept-he-6in4 was vyos001-only, so vyos002 had no proto-41 accept under its IPv4 input default-deny -- a failover would have left tun0 up and radvd running on the new master while HE's encapsulated traffic was dropped by its own firewall. The tunnel deployment alone was NOT sufficient. (2) The per-router bond0.9 override must list the IPv4 address next to the IPv6 one, because an override set declares the COMPLETE set for its path. Also: do not stage vyos model changes on a feature branch. The first attempt at this merge was made on fix/openbao-preview-blockers, which is 15 commits behind main and predates the PPPoE HA overrides -- committing it would have reverted pppoe-gated-not-config-disabled and vrrp-transition-scripts-wan-follows-master. main lives in the .worktrees/grafana-token worktree."
}