mlx5_core ovs/ovn hardware offload - broken network connectivity between instances

Observation:
With everything Groovy and ct offload enabled the problem is not there. If I disable port security the problem IS there, so perhaps incorrect flows are created or the card is interpreting them incorrectly, the offloaded flows do indeed look quite different when pivoting port security on/off.

The non-offloaded instances still work dandy in either case.

Example:

$ ping 10.5.2.128
PING 10.5.2.128 (10.5.2.128) 56(84) bytes of data.
64 bytes from 10.5.2.128: icmp_seq=1 ttl=64 time=12.9 ms
^C
--- 10.5.2.128 ping statistics ---
2 packets transmitted, 1 received, 50% packet loss, time 1001ms
rtt min/avg/max/mdev = 12.938/12.938/12.938/0.000 ms

Digging a bit more - I see that the instances with broken network is on the same hypervisor as an instance on the same project network:

ps5-ra4-n4.maas

all other instances ended up on different hypervisors so maybe its a conflicting offloaded flow.

If I delete one of the machines on the hypervisor then rebuild the other (forcing a rebuild of some sort) connectivity is restored:

$ ssh 10.5.0.247 "ping -c 2 10.5.2.128"
Warning: Permanently added '10.5.0.247' (ECDSA) to the list of known hosts.
PING 10.5.2.128 (10.5.2.128) 56(84) bytes of data.
64 bytes from 10.5.2.128: icmp_seq=1 ttl=64 time=0.149 ms
64 bytes from 10.5.2.128: icmp_seq=2 ttl=64 time=0.197 ms

As mentioned in #1 everything appear to work with port security enabled and a 5.8 kernel which has support for offload of conntrack flows. But as we know that is a bit too bleeding edge for production use at this point in time.

To try an alternative approach I took the OVN VXLAN support patch [0] and backported it to the 20.03 package in OVN.

Testing so far suggests that solves the issue, so that might indicate that the root of the issue is in NIC firmware and/or driver for the Geneve offload support. So perhaps we need to go with VXLAN for now and move back to Geneve once the 5.8 kernel becomes stable enough to be consumable in production?

0: https://github.com/ovn-org/ovn/commit/b07f1bc3d068e231e40840b0d977bd57158987fd