fix(vyos): health-check the 10 gig primary so failover actually fires

The 10 gig line was primary by route distance alone, which only fails over
when bond0.53 loses carrier or its DHCP lease. An ISP that keeps the link up
while dropping traffic -- the common failure -- would black-hole everything,
because a DHCP-installed route has nothing to withdraw it.

`protocols failover` now owns the live default route and pings two targets
bound to the interface, so the backup can never be validated through the
primary's path. Rehearsed on the labsim router: failover and failback both
inside 5s with the router's own interface still UP.

The vif keeps default-route-distance rather than no-default-route, demoted
below Vodafone. vyos-failover resolves a dhcp-interface gateway by reading
new_routers out of /run/dhclient/dhclient_<if>.lease, and no-default-route
leaves that field EMPTY -- the daemon then finds no next hop and installs
nothing. Observed on vyos001: the default route fell through to Vodafone.
Preference is now failover's kernel route (distance 0) > pppoe (10) >
DHCP (210), so the demoted route can never re-create the black hole it
exists to avoid.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH

migration/vyos-mode-delta.py

@@ -54,7 +54,30 @@ WAN_DHCP_VIF = "bond0.53"       # 10 gig ISP
 WAN_DHCP_MAC = "f0:9f:c2:12:9b:4f"
 
 # Route distances: the 10 gig line wins, Vodafone is failover.
-DIST_DHCP, DIST_PPPOE = 1, 10
+#
+# The live 10 gig default route is owned by `protocols failover`, so that losing
+# the ISP *without* losing carrier withdraws it instead of black-holing every
+# packet -- a DHCP-installed route never withdraws on a dead upstream.
+#
+# The vif still needs default-route-distance rather than no-default-route:
+# vyos-failover resolves a dhcp-interface gateway by reading new_routers out of
+# /run/dhclient/dhclient_<if>.lease, and no-default-route leaves that field
+# EMPTY, so the daemon finds no next hop and installs nothing. Verified on
+# vyos001: with no-default-route the default route fell through to Vodafone.
+#
+# So DHCP keeps a route, deliberately demoted BELOW Vodafone. Order of
+# preference: failover's kernel route (distance 0) > pppoe (10) > DHCP (210).
+# The demoted route is never selected while pppoe is up, so it cannot re-create
+# the black-hole it exists to avoid.
+DIST_PPPOE = 10
+DIST_DHCP_FALLBACK = 210
+
+# Health-checked primary. Two targets, any-available, so one resolver having a
+# bad day is not read as "the line is down". Verified on the sim: failover and
+# failback both inside 5s with the router's own interface still UP.
+FAILOVER_METRIC = 1
+FAILOVER_TARGETS = ["8.8.8.8", "1.1.1.1"]
+FAILOVER_TIMEOUT = 5
 
 PLACEHOLDER = "@@WAN_PASSWORD@@"
 
@@ -172,7 +195,10 @@ def build_delta(inv: dict, priority: int, wan_user: str, with_wan: bool,
             "# a new one. Only the box carrying the WAN may set this.",
             f"set interfaces bonding bond0 vif {WAN_DHCP_VIF.split('.')[1]} mac '{WAN_DHCP_MAC}'",
             f"set interfaces bonding bond0 vif {WAN_DHCP_VIF.split('.')[1]} address dhcp",
-            f"set interfaces bonding bond0 vif {WAN_DHCP_VIF.split('.')[1]} dhcp-options default-route-distance {DIST_DHCP}",
+            # Demoted below Vodafone; `protocols failover` owns the live route.
+            # NOT no-default-route -- that blanks new_routers in the lease and
+            # leaves the failover daemon with no gateway to install.
+            f"set interfaces bonding bond0 vif {WAN_DHCP_VIF.split('.')[1]} dhcp-options default-route-distance {DIST_DHCP_FALLBACK}",
             "",
             "# WAN1, Vodafone -- failover at a higher distance. Verified working",
             "# on the USG: pppoe0 came up with a public address, MTU 1492.",
@@ -188,6 +214,25 @@ def build_delta(inv: dict, priority: int, wan_user: str, with_wan: bool,
             "# USG is the next hop. Both WANs supply one here.",
             "delete protocols static route 0.0.0.0/0",
             "",
+            "# --- Health-checked primary ----------------------------",
+            "# Without this, failover only fires when bond0.53 loses carrier",
+            "# or its lease. An ISP that keeps the link up while dropping",
+            "# traffic -- the common failure -- would black-hole everything,",
+            "# because a DHCP-installed route has nothing to withdraw it.",
+            "#",
+            "# vyos-failover pings each target bound to the interface",
+            "# (`ping -I bond0.53`), so the backup can never be validated",
+            "# through the primary's path and vice versa. On withdrawal the",
+            "# kernel falls through to Vodafone's distance-10 route.",
+            f"set protocols failover route 0.0.0.0/0 dhcp-interface {WAN_DHCP_VIF} check type icmp",
+            f"set protocols failover route 0.0.0.0/0 dhcp-interface {WAN_DHCP_VIF} check policy any-available",
+            f"set protocols failover route 0.0.0.0/0 dhcp-interface {WAN_DHCP_VIF} check timeout {FAILOVER_TIMEOUT}",
+            f"set protocols failover route 0.0.0.0/0 dhcp-interface {WAN_DHCP_VIF} metric {FAILOVER_METRIC}",
+            *[
+                f"set protocols failover route 0.0.0.0/0 dhcp-interface {WAN_DHCP_VIF} check target {t}"
+                for t in FAILOVER_TARGETS
+            ],
+            "",
             "# --- NAT -----------------------------------------------",
             f"set nat source rule 100 outbound-interface name {WAN_DHCP_VIF}",
             "set nat source rule 100 translation address masquerade",