Ussuri deployment returns RA responding to RS, but Wallaby/Xena doesn't

Subscribing ~field-high.

I was deploying multiple releases but will deploy the testbed with focal-xena so I should be able to offer any info to be requested.

This is what I see from a VM on top of Xena deployment - no RA until the periodic interval:

10:50:07.933169 fa:16:3e:d9:ca:78 > 33:33:00:00:00:02, ethertype IPv6 (0x86dd), length 70: (flowlabel 0x8f5ed, hlim 255, next-header ICMPv6 (58) payload length: 16) fe80::f816:3eff:fed9:ca78 > ff02::2: [icmp6 sum ok] ICMP6, router solicitation, length 16
          source link-address option (1), length 8 (1): fa:16:3e:d9:ca:78
10:50:12.282979 fa:16:3e:d9:ca:78 > 33:33:00:00:00:02, ethertype IPv6 (0x86dd), length 70: (flowlabel 0x8f5ed, hlim 255, next-header ICMPv6 (58) payload length: 16) fe80::f816:3eff:fed9:ca78 > ff02::2: [icmp6 sum ok] ICMP6, router solicitation, length 16
          source link-address option (1), length 8 (1): fa:16:3e:d9:ca:78
10:50:20.680418 fa:16:3e:d9:ca:78 > 33:33:00:00:00:02, ethertype IPv6 (0x86dd), length 70: (flowlabel 0x8f5ed, hlim 255, next-header ICMPv6 (58) payload length: 16) fe80::f816:3eff:fed9:ca78 > ff02::2: [icmp6 sum ok] ICMP6, router solicitation, length 16
          source link-address option (1), length 8 (1): fa:16:3e:d9:ca:78
10:50:36.651279 fa:16:3e:d9:ca:78 > 33:33:00:00:00:02, ethertype IPv6 (0x86dd), length 70: (flowlabel 0x8f5ed, hlim 255, next-header ICMPv6 (58) payload length: 16) fe80::f816:3eff:fed9:ca78 > ff02::2: [icmp6 sum ok] ICMP6, router solicitation, length 16
          source link-address option (1), length 8 (1): fa:16:3e:d9:ca:78
10:51:08.183176 fa:16:3e:d9:ca:78 > 33:33:00:00:00:02, ethertype IPv6 (0x86dd), length 70: (flowlabel 0x8f5ed, hlim 255, next-header ICMPv6 (58) payload length: 16) fe80::f816:3eff:fed9:ca78 > ff02::2: [icmp6 sum ok] ICMP6, router solicitation, length 16
          source link-address option (1), length 8 (1): fa:16:3e:d9:ca:78
10:52:08.250195 fa:16:3e:d9:ca:78 > 33:33:00:00:00:02, ethertype IPv6 (0x86dd), length 70: (flowlabel 0x8f5ed, hlim 255, next-header ICMPv6 (58) payload length: 16) fe80::f816:3eff:fed9:ca78 > ff02::2: [icmp6 sum ok] ICMP6, router solicitation, length 16
          source link-address option (1), length 8 (1): fa:16:3e:d9:ca:78
10:52:50.716674 fa:16:3e:bc:47:96 > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length 118: (hlim 255, next-header ICMPv6 (58) payload length: 64) fe80::f816:3eff:febc:4796 > ff02::1: [icmp6 sum ok] ICMP6, router advertisement, length 64
        hop limit 255, Flags [none], pref medium, router lifetime 65535s, reachable time 0ms, retrans timer 0ms
          source link-address option (1), length 8 (1): fa:16:3e:bc:47:96
          mtu option (5), length 8 (1): 1442
          prefix info option (3), length 32 (4): fc00:aa:aa:aa::/64, Flags [onlink, auto], valid time infinity, pref. time infinity
10:52:50.801190 fa:16:3e:d9:ca:78 > 33:33:ff:d9:ca:78, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) :: > ff02::1:ffd9:ca78: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has fc00:aa:aa:aa:f816:3eff:fed9:ca78
          unknown option (14), length 8 (1):
          0x0000: 8243 99fc 248e
10:59:45.754605 fe:16:3e:d9:ca:78 > 33:33:00:00:00:02, ethertype IPv6 (0x86dd), length 70: (hlim 255, next-header ICMPv6 (58) payload length: 16) fe80::fc16:3eff:fed9:ca78 > ff02::2:...

Just checked on focal/xena

openstack subnet create --network net-ipv6 \
  --ip-version 6 \
  --subnet-range 2620:128:e082:4008::/64 \
  --ipv6-address-mode slaac \
  --ipv6-ra-mode slaac \
  subnet1

and

openstack router add subnet router subnet1

I turned on tcpdump for router advertisement inside the VM (which already had ipv4 address on the ens3 interface) this way:
sudo tcpdump -vvvv -ttt -i any icmp6 and 'ip6[40] = 134'

then after hot-plug of the interface with subnet1 to the VM and

sudo ip link set ens8 up

almost instantly I get router advertisement and IP on subnet1 on ens8:

00:01:34.253736 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 64) _gateway > test62: [icmp6 sum ok] ICMP6, router advertisement, length 64
        hop limit 255, Flags [none], pref medium, router lifetime 65535s, reachable time 0ms, retrans timer 0ms
          source link-address option (1), length 8 (1): fa:16:3e:1e:d5:f2
            0x0000: fa16 3e1e d5f2
          mtu option (5), length 8 (1): 8942
            0x0000: 0000 0000 22ee
          prefix info option (3), length 32 (4): 2620:128:e082:4008::/64, Flags [onlink, auto], valid time infinity, pref. time infinity
            0x0000: 40c0 ffff ffff ffff ffff 0000 0000 2620
            0x0010: 0128 e082 4008 0000 0000 0000 0000

Also when I do

sudo radvdump
sudo ip link set ens8 down
sudo ip link set ens8 up

I receive the advertisement in just 2-3-4 seconds

same with
sudo radvdump
sudo ip link set ens8 down
sudo systemctl restart systemd-networkd.service

interface ens8
{
        AdvSendAdvert on;
        # Note: {Min,Max}RtrAdvInterval cannot be obtained with radvdump
        AdvManagedFlag off;
        AdvOtherConfigFlag off;
        AdvReachableTime 0;
        AdvRetransTimer 0;
        AdvCurHopLimit 255;
        AdvDefaultLifetime 65535;
        AdvHomeAgentFlag off;
        AdvDefaultPreference medium;
        AdvSourceLLAddress on;
        AdvLinkMTU 8942;

        prefix 2620:128:e082:4008::/64
        {
                AdvValidLifetime infinity; # (0xffffffff)
                AdvPreferredLifetime infinity; # (0xffffffff)
                AdvOnLink on;
                AdvAutonomous on;
                AdvRouterAddr off;
        }; # End of prefix definition

}; # End of interface definition
#

Unsubscribing ~field-high for now since the env doesn't have the issue after tweaking other set up such as network configs. The issue with the testbed is real but not a field issue for now.

Thanks for testing.

OVN should periodically send RAs based on the options specified in the northbound DB (send_periodic, max_interval, min_interval).

https://github.com/ovn-org/ovn/commit/c04c004efaf64b77460de5338092e642385e582d
https://github.com/ovn-org/ovn/commit/ec5bcc68b34e73b8541f3143dd1f69a8e884cbef

https://github.com/ovn-org/ovn/blob/ed9bb4d59f78d14300d27d68fd2c4ec4621f2256/controller/pinctrl.c#L3818-L3893
https://github.com/ovn-org/ovn/blob/ed9bb4d59f78d14300d27d68fd2c4ec4621f2256/controller/pinctrl.c#L3404

The Logical_Router_Port table in the ovn-nb may contain the following based on the docs:

https://man7.org/linux/man-pages/man5/ovn-nb.5.html#Logical_Router_Port_TABLE
         ipv6_ra_configs : send_periodic
                                     optional string
         ipv6_ra_configs : max_interval
                                     optional string
         ipv6_ra_configs : min_interval
                                     optional string

On an ovn-central unit the following should show LRPs:

ovn-nbctl list Logical_Router_Port

====

As far as Neutron is concerned, `send_periodic` is applied only to internal networks where as intervals are not explicitly set:

https://github.com/openstack/neutron/blob/dd55f1acd36dee4ab4ec806a694b5efcc8e53cb8/neutron/plugins/ml2/drivers/ovn/mech_driver/ovsdb/ovn_client.py#L1169-L1175
                ipv6_ra_configs['send_periodic'] = 'true'
                if is_gw_port and utils.is_provider_network(net):
                    ipv6_ra_configs['send_periodic'] = 'false'

https://github.com/openstack/neutron/blob/dd55f1acd36dee4ab4ec806a694b5efcc8e53cb8/neutron/common/ovn/utils.py#L526-L527
def is_provider_network(network):
    return network.get(external_net.EXTERNAL, False)

Intervals for RAs in OVN are both set per RFC 4861 section 6.2.1 (https://datatracker.ietf.org/doc/html/rfc4861#section-6.2.1):

https://github.com/ovn-org/ovn/blob/ed9bb4d59f78d14300d27d68fd2c4ec4621f2256/controller/pinctrl.c#L3588-L3591
    config->max_interval = smap_get_int(&pb->options, "ipv6_ra_max_interval",
            ND_RA_MAX_INTERVAL_DEFAULT);
    config->min_interval = smap_get_int(&pb->options, "ipv6_ra_min_interval",
            nd_ra_min_interval_default(config->max_interval));

https://github.com/openvswitch/ovs/blob/31b467a751892df2dd938338d91d39e995a6a18c/lib/packets.h#L1083
#define ND_RA_MAX_INTERVAL_DEFAULT 600

https://github.com/openvswitch/ovs/blob/31b467a751892df2dd938338d91d39e995a6a18c/lib/packets.h#L1085-L1089
static inline int
nd_ra_min_interval_default(int max)
{
    return max >= 9 ? max / 3 : max * 3 / 4;
}

So it should be min_interval = 600 / 3 seconds = 200 seconds

To be clear, I'm not talking about the periodic RA packets per se. But no RA response to RS or those interaction is dropped somewhere in between.