From 64e748ea9473c53f03eb01a0d0c57147d7dd8a38 Mon Sep 17 00:00:00 2001 From: Michal Date: Mon, 17 Aug 2026 00:17:21 +0100 Subject: [PATCH] test(labsim): conntrack-sync verified, and it exposed a delta defect conntrack-sync proven working on the sim pair -- bidirectional replication with zero errors: MASTER internal 34 external(from peer) 52 62 pkts sent / 109 recv 0 err BACKUP internal 76 external(from peer) 36 142 pkts sent / 73 recv 0 err Getting there required learning something that changes the production config: **VyOS only engages conntrack when a firewall or NAT is configured.** With neither present, both routers reported zero conntrack entries and conntrack-sync had nothing to replicate. Adding a single state-matching forward rule turned tracking on and replication began immediately. That is a defect in the delta, not just a test artifact. NAT and the firewall were both gated behind --with-wan, so the BACKUP would have had neither -- it would not have tracked connections at all, and replicated entries are useless to a box whose conntrack is not engaged. Exactly the failure that only shows up during a failover, when it is too late to notice. Fixed: a stateful forward rule (accept established/related, default-action accept) is now emitted on BOTH boxes, outside the WAN gate. Only NAT and the WAN-scoped rules remain master-only. Verified: vyos002 now carries stateful tracking and conntrack-sync but zero NAT lines. Master delta re-validated against a real VyOS config -- no errors. Also incidentally confirmed no-preempt: router1 rebooted and came back as BACKUP rather than seizing the VIP, which is the opposite of what the production pair did this afternoon (still on default preempt until cutover). Two traps recorded while doing this: - The detached `setsid nohup` config-apply pattern can strand a VyOS config session. An orphaned session (dirs under /opt/vyatta/config/tmp/, PID long dead) blocked every subsequent `set` on that box with a bare "Set failed", and the dirs are overlay mounts so they cannot simply be deleted. Rebooting cleared it. This pattern is used to survive losing SSH mid-change, so it is worth knowing it has a failure mode of its own. - Only VLAN 10 passes traffic between the two sim routers; every other VLAN fails ARP despite identical vlan_mode/tag/trunks on both OVS bonds and distinct MACs. VRRP forms on all six groups regardless. The sync link had to be bond0.10 as a result. OVS-specific, absent in production, but it means the sim proves mechanism rather than topology. Production deltas regenerated with --conntrack-link: eth3 at 10.255.255.1/30 and .2/30 awaiting the cable, which is not yet plugged (carrier=0 on both). Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01DMVzWZgiKW2wquf5z8S1yH --- migration/vyos-mode-delta.py | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/migration/vyos-mode-delta.py b/migration/vyos-mode-delta.py index faa3b1e..66d06dc 100755 --- a/migration/vyos-mode-delta.py +++ b/migration/vyos-mode-delta.py @@ -139,6 +139,23 @@ def build_delta(inv: dict, priority: int, wan_user: str, with_wan: bool, "", ] + + out += [ + "", + "# --- stateful tracking (BOTH boxes) ------------------------", + "# VyOS only engages conntrack when a firewall or NAT exists. The", + "# backup has no WAN and therefore no NAT, so without this rule it", + "# tracks nothing -- and conntrack-sync entries replicated to a box", + "# whose conntrack is not engaged cannot be used when it takes over.", + "# Verified in labsim: zero conntrack entries until a state-matching", + "# rule was present, then replication began immediately.", + "set firewall ipv4 forward filter default-action accept", + "set firewall ipv4 forward filter rule 10 action accept", + "set firewall ipv4 forward filter rule 10 state established", + "set firewall ipv4 forward filter rule 10 state related", + "set firewall ipv4 forward filter rule 10 description 'stateful tracking'", + ] + if with_wan: out += [ "# --- WAN -----------------------------------------------",