Backup HA router sending traffic, traffic from switch interrupted

On what version of Neutron have this behavior been seen?

Hi Ann,

[root@overcloud-controller-2 ~]# rpm -qa | grep neut
openstack-neutron-lbaas-9.1.0-1.el7ost.noarch
openstack-neutron-ml2-9.1.0-8.el7ost.noarch
python-neutronclient-6.0.0-2.el7ost.noarch
python-neutron-lib-0.4.0-1.el7ost.noarch
openstack-neutron-bigswitch-lldp-9.40.0-1.1.el7ost.noarch
openstack-neutron-metering-agent-9.1.0-8.el7ost.noarch
puppet-neutron-9.4.2-1.el7ost.noarch
python-neutron-9.1.0-8.el7ost.noarch
python-neutron-tests-9.1.0-8.el7ost.noarch
python-neutron-lbaas-9.1.0-1.el7ost.noarch
openstack-neutron-openvswitch-9.1.0-8.el7ost.noarch
openstack-neutron-bigswitch-agent-9.40.0-1.1.el7ost.noarch
openstack-neutron-sriov-nic-agent-9.1.0-8.el7ost.noarch
openstack-neutron-common-9.1.0-8.el7ost.noarch
openstack-neutron-9.1.0-8.el7ost.noarch

Aaron

On Wed, Mar 1, 2017 at 5:25 PM, Ann Taraday <email address hidden>
wrote:

> On what version of Neutron have this behavior been seen?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1667756
>
> Title:
> Backup HA router sending traffic, traffic from switch interrupted
>
> Status in neutron:
> New
>
> Bug description:
> As outlined in https://review.openstack.org/#/c/142843/, backup HA
> routers should not send any traffic. Any traffic will cause the
> connected switch to learn a new port for the associated src mac
> address since the mac address will be in use on the primary HA router.
>
> We are observing backup routers sending IPv6 RA and RS messages
> probably in response to incoming IPv6 RA messages. The subnets
> associated with the HA routers are not intended for IPv6 traffic.
>
> A typical traffic sequence is:
>
> Packet from external switch...
> 08:81:f4:a6:dc:01 > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length
> 110: (hlim 255, next-header ICMPv6 (58) payload length: 56)
> fe80:52:0:136c::fe > ff02::1: [icmp6 sum ok] ICMP6, router advertisement,
> length 56
>
> Immediately followed by a packet from the backup HA router...
> fa:16:3e:a7:ae:63 > 33:33:ff:a7:ae:63, ethertype IPv6 (0x86dd), length
> 86: (hlim 1, next-header Options (0) payload length: 32) :: >
> ff02::1:ffa7:ae63: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6,
> multicast listener reportmax resp delay: 0 addr: ff02::1:ffa7:ae63
>
> Another pkt...
> fa:16:3e:a7:ae:63 > 33:33:ff:a7:ae:63, ethertype IPv6 (0x86dd), length
> 78: (hlim 255, next-header ICMPv6 (58) payload length: 24) :: >
> ff02::1:ffa7:ae63: [icmp6 sum ok] ICMP6, neighbor solicitation, length 24,
> who has 2620:52:0:136c:f816:3eff:fea7:ae63
>
> Another Pkt...
> fa:16:3e:a7:ae:63 > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length
> 86: (hlim 255, next-header ICMPv6 (58) payload length: 32)
>
> At this point, the switch has updated its mac table and traffic to the
> fa:16:3e:a7:ae:63 address has been redirected to the backup host.
> SSH/ping traffic resumes at a later time when the primary router node
> sends traffic with the fa:16:3e:a7:ae:63 source address.
>
> This problem is reproducible in our environment as follows:
>
> 1. Deploy OSP10
> 2. Create external network
> 3. Create external subnet (IPv4)
> 4...

[root@overcloud-controller-2 ~]# rpm -qa | grep neut
openstack-neutron-lbaas-9.1.0-1.el7ost.noarch
openstack-neutron-ml2-9.1.0-8.el7ost.noarch
python-neutronclient-6.0.0-2.el7ost.noarch
python-neutron-lib-0.4.0-1.el7ost.noarch
openstack-neutron-bigswitch-lldp-9.40.0-1.1.el7ost.noarch
openstack-neutron-metering-agent-9.1.0-8.el7ost.noarch
puppet-neutron-9.4.2-1.el7ost.noarch
python-neutron-9.1.0-8.el7ost.noarch
python-neutron-tests-9.1.0-8.el7ost.noarch
python-neutron-lbaas-9.1.0-1.el7ost.noarch
openstack-neutron-openvswitch-9.1.0-8.el7ost.noarch
openstack-neutron-bigswitch-agent-9.40.0-1.1.el7ost.noarch
openstack-neutron-sriov-nic-agent-9.1.0-8.el7ost.noarch
openstack-neutron-common-9.1.0-8.el7ost.noarch
openstack-neutron-9.1.0-8.el7ost.noarch

Hi here is a packet capture of an occurrence..
This packet capture is in the netns of the qrouter....

[root@overcloud-controller-1 ~]# ip netns ls
qdhcp-8897a5a6-dd5e-44d3-852f-c0dcd870b228
qdhcp-9b61e335-1b6c-498f-82b4-f660180e6876
qdhcp-ec0bad75-3466-43fb-b240-2b7d53a1d2c7
qdhcp-f76fcc81-0b57-4258-9e1a-04aa0c786428
qdhcp-b918b2ce-ef92-444d-80a0-9bf1f93406f3
qdhcp-3e184953-6df2-45c0-a0d4-7ca837110740
qdhcp-06c553c8-0adb-4b95-b2c4-4dc163f95a4f
qdhcp-184592ea-da08-4180-a9cb-a660125e5d1a
qrouter-b65e3bdc-390e-48b7-9cd9-fca429dafb45
[root@overcloud-controller-1 ~]# ip netns exec qrouter-b65e3bdc-390e-48b7-9cd9-fca429dafb45 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
31: ha-bce4b526-c4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1446 qdisc noqueue state UNKNOWN qlen 1000
    link/ether fa:16:3e:2d:1c:2c brd ff:ff:ff:ff:ff:ff
    inet 169.254.192.2/18 brd 169.254.255.255 scope global ha-bce4b526-c4
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe2d:1c2c/64 scope link
       valid_lft forever preferred_lft forever
32: qr-1656d4d1-2c: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1446 qdisc noqueue state UNKNOWN qlen 1000
    link/ether fa:16:3e:54:77:0b brd ff:ff:ff:ff:ff:ff
33: qg-3f327e61-8b: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1496 qdisc noqueue state UNKNOWN qlen 1000
    link/ether fa:16:3e:a7:ae:63 brd ff:ff:ff:ff:ff:ff
[root@overcloud-controller-1 ~]# ip netns exec qrouter-b65e3bdc-390e-48b7-9cd9-fca429dafb45 bash
[root@overcloud-controller-1 ~]# tcpdump -nnvv -e -i qg-3f327e61-8b
tcpdump: WARNING: qg-3f327e61-8b: no IPv4 address assigned
tcpdump: listening on qg-3f327e61-8b, link-type EN10MB (Ethernet), capture size 65535 bytes
14:24:09.814889 08:81:f4:a6:dc:01 > 01:00:5e:00:00:01, ethertype IPv4 (0x0800), length 60: (tos 0xc0, ttl 1, id 30720, offset 0, flags [none], proto IGMP (2), length 32, options (RA))
    10.19.108.254 > 224.0.0.1: igmp query v2
14:24:13.799395 08:81:f4:a6:dc:01 > 33:33:00:00:00:0d, ethertype IPv6 (0x86dd), length 110: (class 0xc0, hlim 1, next-header PIM (103) payload length: 56) fe80:52:0:136c::fe > ff02::d: PIMv2, length 56
 Hello, cksum 0x018f (correct)
   Hold Time Option (1), length 2, Value: 1m45s
     0x0000: 0069
   LAN Prune Delay Option (2), length 4, Value:
     T-bit=1, LAN delay 500ms, Override interval 2000ms
     0x0000: 81f4 07d0
   DR Priority Option (19), length 4, Value: 1
     0x0000: 0000 0001
   Address List Option (24), length 18, Value:
     2620:52:0:136c::fe
     0x0000: 0200 2620 0052 0000 136c 0000 0000 0000
     0x0010: 00fe
   Generation ID Option (20), length 4, Value: 0x00ec032c
     0x0000: 00ec 032c
14:24:21.481044 08:81:f4:a6:dc:01 > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length 86: (hlim 1, next-header Options (0) payload length: 32) fe80:52:0:136c::fe > ff02::1: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener querymax resp delay: 10000 addr: ::
14:24:28.241400 fa:16:3e:ba:6f:6f >...

My apologies, I copied the wrong controller window....

In the capture below, you will see several packets come in from
08:81:f4:a6:dc:01. Eventually, fa:16:3e:a7:ae:63 replies and ICMP
requests are redirected to this backup controller.

root@overcloud-controller-0 ~]# ip netns exec qrouter-b65e3bdc-390e-48b7-9cd9-fca429dafb45 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
28: ha-94c36543-95: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1446 qdisc noqueue state UNKNOWN qlen 1000
    link/ether fa:16:3e:52:98:a4 brd ff:ff:ff:ff:ff:ff
    inet 169.254.192.8/18 brd 169.254.255.255 scope global ha-94c36543-95
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe52:98a4/64 scope link
       valid_lft forever preferred_lft forever
29: qr-1656d4d1-2c: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1446 qdisc noqueue state UNKNOWN qlen 1000
    link/ether fa:16:3e:54:77:0b brd ff:ff:ff:ff:ff:ff
30: qg-3f327e61-8b: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1496 qdisc noqueue state UNKNOWN qlen 1000
    link/ether fa:16:3e:a7:ae:63 brd ff:ff:ff:ff:ff:ff
[root@overcloud-controller-0 ~]# ip netns exec qrouter-b65e3bdc-390e-48b7-9cd9-fca429dafb45 bash
[root@overcloud-controller-0 ~]# tcpdump -nnvv -e -i qg-3f327e61-8b
tcpdump: WARNING: qg-3f327e61-8b: no IPv4 address assigned
tcpdump: listening on qg-3f327e61-8b, link-type EN10MB (Ethernet), capture size 65535 bytes
15:11:03.232415 08:81:f4:a6:dc:01 > 01:00:5e:00:00:0d, ethertype IPv4 (0x0800), length 68: (tos 0xc0, ttl 1, id 30322, offset 0, flags [none], proto PIM (103), length 54)
    10.19.108.254 > 224.0.0.13: PIMv2, length 34
 Hello, cksum 0x4fa8 (correct)
   Hold Time Option (1), length 2, Value: 1m45s
     0x0000: 0069
   LAN Prune Delay Option (2), length 4, Value:
     T-bit=1, LAN delay 500ms, Override interval 2000ms
     0x0000: 81f4 07d0
   DR Priority Option (19), length 4, Value: 1
     0x0000: 0000 0001
   Generation ID Option (20), length 4, Value: 0x7c75897b
     0x0000: 7c75 897b
15:11:17.743272 08:81:f4:a6:dc:01 > 33:33:00:00:00:0d, ethertype IPv6 (0x86dd), length 110: (class 0xc0, hlim 1, next-header PIM (103) payload length: 56) fe80:52:0:136c::fe > ff02::d: PIMv2, length 56
 Hello, cksum 0x018f (correct)
   Hold Time Option (1), length 2, Value: 1m45s
     0x0000: 0069
   LAN Prune Delay Option (2), length 4, Value:
     T-bit=1, LAN delay 500ms, Override interval 2000ms
     0x0000: 81f4 07d0
   DR Priority Option (19), length 4, Value: 1
     0x0000: 0000 0001
   Address List Option (24), length 18, Value:
     2620:52:0:136c::fe
     0x0000: 0200 2620 0052 0000 136c 0000 0000 0000
     0x0010: 00fe
   Generation ID Option (20), length 4, Value: 0x00ec032c
     0x0000: 00ec 032c
15:11:31.219937 08:81:f4:a6:dc:01 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Ethernet (len 6), IPv4 (len 4), Request who-has 10.19.108.155 tell 10.19.108.254, length 46
15:11:31.646955 08:81:f4:a6:dc:01 > 01:00:5e:00:00:0d, et...

Here is the accept_ra setting on the back controllers...

[root@overcloud-controller-1 ~]# ip netns exec qrouter-b65e3bdc-390e-48b7-9cd9-fca429dafb45 bash
[root@overcloud-controller-1 ~]# cat /proc/sys/net/ipv6/
anycast_src_echo_reply conf/ ip6frag_high_thresh ip6frag_time neigh/ xfrm6_gc_thresh
bindv6only icmp/ ip6frag_low_thresh ip_nonlocal_bind route/
[root@overcloud-controller-1 ~]# cat /proc/sys/net/ipv6/conf/
all/ default/ ha-bce4b526-c4/ lo/ qg-3f327e61-8b/ qr-1656d4d1-2c/
[root@overcloud-controller-1 ~]# cat /proc/sys/net/ipv6/conf/qg-3f327e61-8b/accept_ra
2

This bug is related to this other one [0].

When namespaces are created, ipv6 forwarding is enabled by default on them [1].
Setting the IPv6 procfs forwarding entry value causes the specified network interface to join or leave 3 multicast address groups (link-local all-routers multicast group (ff02::2), interface-local all routers multicast group (ff01::2), and site-local all routers multicast group (ff05::2)). So, when enabled, we'll see traffic such as Multicast Listener query/response packets:

==QUERY==
16:45:41.010437 08:81:f4:a6:dc:01 > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length 86: (hlim 1, next-header Options (0) payload length: 32) fe80:52:0:136c::fe > ff02::1: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener querymax resp delay: 10000 addr: ::
==RESPONSE==
16:45:41.843696 fa:16:3e:a7:ae:63 > 33:33:00:00:00:02, ethertype IPv6 (0x86dd), length 86: (hlim 1, next-header Options (0) payload length: 32) :: > ff02::2: HBH (rtalert: 0x0000) (padn) [icmp6 sum ok] ICMP6, multicast listener reportmax resp delay: 0 addr: ff02::2

So the solution to avoid this behavior on backup instances is to disable the forwarding entry in the procfs for the gateway interface. However, this is not enough to stop seeing traffic as it may still send neighbor solicitations/advertisements:

09:29:09.669712 fa:16:3e:a7:ae:63 > 33:33:ff:a7:ae:63, ethertype IPv6 (0x86dd), length 78: (hlim 255, next-header ICMPv6 (58) payload length: 24) :: > ff02::1:ffa7:ae63: [icmp6 sum ok] ICMP6, neighbor solicitation, length 24, who has 2620:52:0:136c:f816:3eff:fea7:ae63

09:29:09.670352 fa:16:3e:a7:ae:63 > 33:33:00:00:00:01, ethertype IPv6 (0x86dd), length 86: (hlim 255, next-header ICMPv6 (58) payload length: 32) 2620:52:0:136c:f816:3eff:fea7:ae63 > ff02::1: [icmp6 sum ok] ICMP6, neighbor advertisement, length 32, tgt is 2620:52:0:136c:f816:3eff:fea7:ae63, Flags [router, override]
   destination link-address option (2), length 8 (1): fa:16:3e:a7:ae:63
     0x0000: fa16 3ea7 ae63

For this, accept_ra procfs entry has to be set to 0. Neutron code only sets it to '2' right now but it never sets it to 0 when the router transitions to standby.

[0] https://bugs.launchpad.net/neutron/+bug/1669765
[1] https://github.com/openstack/neutron/blob/master/neutron/agent/l3/namespaces.py#L97

Fix proposed to branch: master
Review: https://review.openstack.org/442518

Reviewed: https://review.openstack.org/442518
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=676a3ebe2f5b62f0ce7a3f7f434526931d5504a5
Submitter: Jenkins
Branch: master

commit 676a3ebe2f5b62f0ce7a3f7f434526931d5504a5
Author: Daniel Alvarez <email address hidden>
Date: Sun Jan 22 13:33:07 2017 +0000

    Disable RA and IPv6 forwarding on backup HA routers

    Neutron does not disable ipv6 forwarding for HA routers and it's
    enabled by default in all router namespaces. For ipv6, this means
    that it will automatically join the following groups:

    * link-local all-routers multicast group (ff02::2)
    * interface-local all routers multicast group (ff01::2)
    * site-local all routers multicast group (ff05::2))

    As a side effect it will answer to multicast listener queries, thus
    causing external switch to learn its MAC address and disrupting traffic
    to the master instance.

    This patch will enable ipv6 forwarding on the gateway interface only
    for master instances and disable it otherwise to fix the issue.

    Also, the accept_ra procfs entry was enabled under certain
    circumstances but it wasn't disabled otherwise. This patch, will
    disable RA on the gateway interface for non master instances.

    Closes-Bug: #1667756

    Change-Id: I9bc890b43f750cad68fc67f4c79f1426c3506863

This issue was fixed in the openstack/neutron 11.0.0.0b1 development milestone.

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/460918

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/460924

Related fix proposed to branch: master
Review: https://review.openstack.org/461073

Reviewed: https://review.openstack.org/461073
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=507cdde679613e85b54e3d86c6ae50a73e28e494
Submitter: Jenkins
Branch: master

commit 507cdde679613e85b54e3d86c6ae50a73e28e494
Author: Daniel Alvarez <email address hidden>
Date: Fri Apr 28 16:35:57 2017 +0200

    Fix: set IPv6 forwarding when there's an IPv6 gw

    Currently when there is an IPv6 gateway set, IPv6 forwarding
    configuration is skipped. This patch fixes it and also adds test
    coverage to check the values of `accept_ra` and `forwarding` with
    and without an IPv6 gateway.

    Related-bug: #1667756

    Change-Id: I0297aa6d0afeb56a8c69be41424d4b70509377d2

Related fix proposed to branch: stable/ocata
Review: https://review.openstack.org/464512

Reviewed: https://review.openstack.org/460918
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=4529630e2746e2ed8d3c7cf96f589b8b5236463e
Submitter: Jenkins
Branch: stable/ocata

commit 4529630e2746e2ed8d3c7cf96f589b8b5236463e
Author: Daniel Alvarez <email address hidden>
Date: Sun Jan 22 13:33:07 2017 +0000

    Disable RA and IPv6 forwarding on backup HA routers

    Neutron does not disable ipv6 forwarding for HA routers and it's
    enabled by default in all router namespaces. For ipv6, this means
    that it will automatically join the following groups:

    * link-local all-routers multicast group (ff02::2)
    * interface-local all routers multicast group (ff01::2)
    * site-local all routers multicast group (ff05::2))

    As a side effect it will answer to multicast listener queries, thus
    causing external switch to learn its MAC address and disrupting traffic
    to the master instance.

    This patch will enable ipv6 forwarding on the gateway interface only
    for master instances and disable it otherwise to fix the issue.

    Also, the accept_ra procfs entry was enabled under certain
    circumstances but it wasn't disabled otherwise. This patch, will
    disable RA on the gateway interface for non master instances.

    Closes-Bug: #1667756

    Change-Id: I9bc890b43f750cad68fc67f4c79f1426c3506863
    (cherry picked from commit 676a3ebe2f5b62f0ce7a3f7f434526931d5504a5)

Reviewed: https://review.openstack.org/464512
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=a0cf9c285970cc7e419db4dda1e070cb2d23ab36
Submitter: Jenkins
Branch: stable/ocata

commit a0cf9c285970cc7e419db4dda1e070cb2d23ab36
Author: Daniel Alvarez <email address hidden>
Date: Fri Apr 28 16:35:57 2017 +0200

    Fix: set IPv6 forwarding when there's an IPv6 gw

    Currently when there is an IPv6 gateway set, IPv6 forwarding
    configuration is skipped. This patch fixes it and also adds test
    coverage to check the values of `accept_ra` and `forwarding` with
    and without an IPv6 gateway.

    Related-bug: #1667756

    Change-Id: I0297aa6d0afeb56a8c69be41424d4b70509377d2
    (cherry picked from commit 507cdde679613e85b54e3d86c6ae50a73e28e494)

Related fix proposed to branch: stable/newton
Review: https://review.openstack.org/482847

Reviewed: https://review.openstack.org/460924
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=1495453768c3d37057575167d1be634cdee206d8
Submitter: Jenkins
Branch: stable/newton

commit 1495453768c3d37057575167d1be634cdee206d8
Author: Daniel Alvarez <email address hidden>
Date: Sun Jan 22 13:33:07 2017 +0000

    Disable RA and IPv6 forwarding on backup HA routers

    Neutron does not disable ipv6 forwarding for HA routers and it's
    enabled by default in all router namespaces. For ipv6, this means
    that it will automatically join the following groups:

    * link-local all-routers multicast group (ff02::2)
    * interface-local all routers multicast group (ff01::2)
    * site-local all routers multicast group (ff05::2))

    As a side effect it will answer to multicast listener queries, thus
    causing external switch to learn its MAC address and disrupting traffic
    to the master instance.

    This patch will enable ipv6 forwarding on the gateway interface only
    for master instances and disable it otherwise to fix the issue.

    Also, the accept_ra procfs entry was enabled under certain
    circumstances but it wasn't disabled otherwise. This patch, will
    disable RA on the gateway interface for non master instances.

    Conflicts:
     neutron/tests/functional/agent/l3/test_ha_router.py

    Closes-Bug: #1667756

    Change-Id: I9bc890b43f750cad68fc67f4c79f1426c3506863
    (cherry picked from commit 676a3ebe2f5b62f0ce7a3f7f434526931d5504a5)
    (cherry picked from commit 9360fb90ba73dcda58a255d7f1c9ae2a6b95451f)

Reviewed: https://review.openstack.org/482847
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=8f866dea74774801175d739dc9c2d1a7c245b18e
Submitter: Jenkins
Branch: stable/newton

commit 8f866dea74774801175d739dc9c2d1a7c245b18e
Author: Daniel Alvarez <email address hidden>
Date: Fri Apr 28 16:35:57 2017 +0200

    Fix: set IPv6 forwarding when there's an IPv6 gw

    Currently when there is an IPv6 gateway set, IPv6 forwarding
    configuration is skipped. This patch fixes it and also adds test
    coverage to check the values of `accept_ra` and `forwarding` with
    and without an IPv6 gateway.

    Related-bug: #1667756

    Conflicts:
     neutron/tests/functional/agent/l3/test_ha_router.py

    Change-Id: I0297aa6d0afeb56a8c69be41424d4b70509377d2
    (cherry picked from commit a0cf9c285970cc7e419db4dda1e070cb2d23ab36)

This issue was fixed in the openstack/neutron 9.4.1 release.

This issue was fixed in the openstack/neutron 10.0.3 release.