neutron

l3 agent fails to process router breaking floating ips when updating them on the router qg port via pyroute2

Bug #1991501 reported by sean mooney on 2022-10-03

This bug affects 2 people

Affects		Status	Importance	Assigned to	Milestone
	neutron	Fix Released	Undecided	Unassigned

Bug Description

2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent [-] Failed to process compatible router: edf1cc99-879a-4fe5-a7b2-d19acb8fdcbf: neutron.privileged.agent.linux.ip_lib.InterfaceOperationNotSupported: Operation not supported on interface qg-4bb9d20b-a0, namespace qrouter-edf1cc99-879a-4fe5-a7b2-d19acb8fdcbf.
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent Traceback (most recent call last):
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/l3/agent.py", line 848, in _process_routers_if_compatible
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent self._process_router_if_compatible(router)
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/l3/agent.py", line 639, in _process_router_if_compatible
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent self._process_updated_router(router)
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/l3/agent.py", line 719, in _process_updated_router
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent ri.process()
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/common/utils.py", line 177, in call
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent self.logger(e)
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python3.9/site-packages/oslo_utils/excutils.py", line 227, in __exit__
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent self.force_reraise()
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python3.9/site-packages/oslo_utils/excutils.py", line 200, in force_reraise
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent raise self.value
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/common/utils.py", line 174, in call
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent return func(*args, **kwargs)
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/l3/router_info.py", line 1312, in process
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent self.process_external()
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python3.9/site-packages/decorator.py", line 232, in fun
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent return caller(func, *(extras + args), **kw)
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/common/coordination.py", line 78, in _synchronized
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent return f(*a, **k)
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/l3/router_info.py", line 1046, in process_external
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent self._process_external_gateway(ex_gw_port)
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/l3/router_info.py", line 931, in _process_external_gateway
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent self.external_gateway_updated(ex_gw_port, interface_name)
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/l3/router_info.py", line 884, in external_gateway_updated
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent self._external_gateway_added(
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/l3/router_info.py", line 822, in _external_gateway_added
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent self._external_gateway_settings(ex_gw_port, interface_name,
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/l3/router_info.py", line 835, in _external_gateway_settings
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent self.driver.init_router_port(
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/linux/interface.py", line 171, in init_router_port
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent self.init_l3(device_name=device_name,
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/linux/interface.py", line 146, in init_l3
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent device.delete_addr_and_conntrack_state(ip_cidr)
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/linux/ip_lib.py", line 361, in delete_addr_and_conntrack_state
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent self.addr.delete(cidr)
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/linux/ip_lib.py", line 540, in delete
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent delete_ip_address(cidr, self.name, self._parent.namespace)
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/linux/ip_lib.py", line 834, in delete_ip_address
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent privileged.delete_ip_address(
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python3.9/site-packages/oslo_privsep/priv_context.py", line 271, in _wrap
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent return self.channel.remote_call(name, args, kwargs,
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent File "/var/lib/kolla/venv/lib/python3.9/site-packages/oslo_privsep/daemon.py", line 215, in remote_call
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent raise exc_type(*result[2])
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent neutron.privileged.agent.linux.ip_lib.InterfaceOperationNotSupported: Operation not supported on interface qg-4bb9d20b-a0, namespace qrouter-edf1cc99-879a-4fe5-a7b2-d19acb8fdcbf.
2022-10-03 03:23:33.868 21 ERROR neutron.agent.l3.agent
2022-10-03 03:23:33.869 21 WARNING neutron.agent.l3.agent [-] Hit retry limit with router update for edf1cc99-879a-4fe5-a7b2-d19acb8fdcbf, action 3

version info
yoga deoployed with kolla-ansible using debian ubuntu or centos image and popos 22.04 host

popos 22.04 is based on ubuntu 22.04 and uses nftables intead of legacy ip tables

in the contianer i noticed that iptable -L is slow but iptable -S is fast
it is not clare if this is a race or some other issue but intially the router is fine and hte floating ips work and restarting the l3 agent fixes the issue for a time then it breaks with this message.

Tags:

sean mooney (sean-k-mooney) on 2022-10-03

tags:

added: l3 ovs

Revision history for this message

sean mooney (sean-k-mooney) wrote on 2022-10-03 (last edit on 2022-10-03):

Download full text (3.7 KiB)

just adding this for context if i manually exec into the container as root(so that i can spawn the privsep damon process) and invoke the failing command via python it fails in the same way

sean@cloud:~$ docker exec -it -u root neutron_l3_agent bash
(neutron-l3-agent)[root@cloud /]# python3
Python 3.9.2 (default, Feb 28 2021, 17:03:44)
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from neutron.agent.linux import ip_lib
>>> ip_lib.delete_ip_address("172.20.0.79/32", "qg-4bb9d20b-a0", namespace="qrouter-edf1cc99-879a-4fe5-a7b2-d19acb8fdcbf")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/linux/ip_lib.py", line 834, in delete_ip_address
    privileged.delete_ip_address(
  File "/var/lib/kolla/venv/lib/python3.9/site-packages/oslo_privsep/priv_context.py", line 271, in _wrap
    return self.channel.remote_call(name, args, kwargs,
  File "/var/lib/kolla/venv/lib/python3.9/site-packages/oslo_privsep/daemon.py", line 215, in remote_call
    raise exc_type(*result[2])
neutron.privileged.agent.linux.ip_lib.InterfaceOperationNotSupported: Operation not supported on interface qg-4bb9d20b-a0, namespace qrouter-edf1cc99-879a-4fe5-a7b2-d19acb8fdcbf.

however i can add and remove the IP via the commandline
sudo ip netns exec qrouter-edf1cc99-879a-4fe5-a7b2-d19acb8fdcbf bash

root@cloud:/home/sean# ip a show qg-4bb9d20b-a0
365: qg-4bb9d20b-a0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether fa:16:3e:16:8f:4d brd ff:ff:ff:ff:ff:ff
    inet 172.20.0.114/24 brd 172.20.0.255 scope global qg-4bb9d20b-a0
       valid_lft forever preferred_lft forever
    inet 172.20.0.142/32 brd 172.20.0.142 scope global qg-4bb9d20b-a0
       valid_lft forever preferred_lft forever
    inet 172.20.0.79/32 brd 172.20.0.79 scope global qg-4bb9d20b-a0
       valid_lft forever preferred_lft forever
    inet 172.20.0.143/32 brd 172.20.0.143 scope global qg-4bb9d20b-a0
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe16:8f4d/64 scope link
       valid_lft forever preferred_lft forever
root@cloud:/home/sean# ip a del 172.20.0.79/32 dev qg-4bb9d20b-a0
root@cloud:/home/sean# ip a show qg-4bb9d20b-a0
365: qg-4bb9d20b-a0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether fa:16:3e:16:8f:4d brd ff:ff:ff:ff:ff:ff
    inet 172.20.0.114/24 brd 172.20.0.255 scope global qg-4bb9d20b-a0
       valid_lft forever preferred_lft forever
    inet 172.20.0.142/32 brd 172.20.0.142 scope global qg-4bb9d20b-a0
       valid_lft forever preferred_lft forever
    inet 172.20.0.143/32 brd 172.20.0.143 scope global qg-4bb9d20b-a0
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe16:8f4d/64 scope link
       valid_lft forever preferred_lft forever
root@cloud:/home/sean# ip a add 172.20.0.79/32 dev qg-4bb9d20b-a0
root@cloud:/home/sean# ip a show qg-4bb9d20b-a0
365: qg-4bb9d20b-a0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/e...

just adding this for context if i manually exec into the container as root(so that i can spawn the privsep damon process) and invoke the failing command via python it fails in the same way

sean@cloud:~$ docker exec -it -u root neutron_l3_agent bash
(neutron-l3-agent)[root@cloud /]# python3
Python 3.9.2 (default, Feb 28 2021, 17:03:44) 
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from neutron.agent.linux import ip_lib
>>> ip_lib.delete_ip_address("172.20.0.79/32", "qg-4bb9d20b-a0", namespace="qrouter-edf1cc99-879a-4fe5-a7b2-d19acb8fdcbf")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/var/lib/kolla/venv/lib/python3.9/site-packages/neutron/agent/linux/ip_lib.py", line 834, in delete_ip_address
    privileged.delete_ip_address(
  File "/var/lib/kolla/venv/lib/python3.9/site-packages/oslo_privsep/priv_context.py", line 271, in _wrap
    return self.channel.remote_call(name, args, kwargs,
  File "/var/lib/kolla/venv/lib/python3.9/site-packages/oslo_privsep/daemon.py", line 215, in remote_call
    raise exc_type(*result[2])
neutron.privileged.agent.linux.ip_lib.InterfaceOperationNotSupported: Operation not supported on interface qg-4bb9d20b-a0, namespace qrouter-edf1cc99-879a-4fe5-a7b2-d19acb8fdcbf.

however i can add and remove the IP via the commandline 
sudo ip netns exec qrouter-edf1cc99-879a-4fe5-a7b2-d19acb8fdcbf bash

root@cloud:/home/sean# ip a show  qg-4bb9d20b-a0
365: qg-4bb9d20b-a0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether fa:16:3e:16:8f:4d brd ff:ff:ff:ff:ff:ff
    inet 172.20.0.114/24 brd 172.20.0.255 scope global qg-4bb9d20b-a0
       valid_lft forever preferred_lft forever
    inet 172.20.0.142/32 brd 172.20.0.142 scope global qg-4bb9d20b-a0
       valid_lft forever preferred_lft forever
    inet 172.20.0.79/32 brd 172.20.0.79 scope global qg-4bb9d20b-a0
       valid_lft forever preferred_lft forever
    inet 172.20.0.143/32 brd 172.20.0.143 scope global qg-4bb9d20b-a0
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe16:8f4d/64 scope link 
       valid_lft forever preferred_lft forever
root@cloud:/home/sean# ip a del 172.20.0.79/32 dev qg-4bb9d20b-a0
root@cloud:/home/sean# ip a show  qg-4bb9d20b-a0
365: qg-4bb9d20b-a0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether fa:16:3e:16:8f:4d brd ff:ff:ff:ff:ff:ff
    inet 172.20.0.114/24 brd 172.20.0.255 scope global qg-4bb9d20b-a0
       valid_lft forever preferred_lft forever
    inet 172.20.0.142/32 brd 172.20.0.142 scope global qg-4bb9d20b-a0
       valid_lft forever preferred_lft forever
    inet 172.20.0.143/32 brd 172.20.0.143 scope global qg-4bb9d20b-a0
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe16:8f4d/64 scope link 
       valid_lft forever preferred_lft forever
root@cloud:/home/sean# ip a add 172.20.0.79/32 dev qg-4bb9d20b-a0
root@cloud:/home/sean# ip a show  qg-4bb9d20b-a0
365: qg-4bb9d20b-a0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether fa:16:3e:16:8f:4d brd ff:ff:ff:ff:ff:ff
    inet 172.20.0.114/24 brd 172.20.0.255 scope global qg-4bb9d20b-a0
       valid_lft forever preferred_lft forever
    inet 172.20.0.142/32 brd 172.20.0.142 scope global qg-4bb9d20b-a0
       valid_lft forever preferred_lft forever
    inet 172.20.0.143/32 brd 172.20.0.143 scope global qg-4bb9d20b-a0
       valid_lft forever preferred_lft forever
    inet 172.20.0.79/32 scope global qg-4bb9d20b-a0
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fe16:8f4d/64 scope link 
       valid_lft forever preferred_lft forever

Revision history for this message

sean mooney (sean-k-mooney) wrote on 2022-10-03:

i think this might be

https://bugzilla.redhat.com/show_bug.cgi?id=2094986
which is fixed in https://github.com/svinota/pyroute2/commit/1eb08312de30a083bcfddfaa9c1d5e124b6368df

which is released in pyroute2 0.6.10
I'm using 0.6.6 which is the current cap in yoga
https://github.com/openstack/requirements/blob/stable/yoga/upper-constraints.txt#L165

sean mooney (sean-k-mooney) on 2022-10-03

summary:	- l3 agent fails to init router breaking floating ips + l3 agent fails to process router breaking floating ips when updating + them on the router qg port via iproute
summary:	l3 agent fails to process router breaking floating ips when updating - them on the router qg port via iproute + them on the router qg port via pyroute2