neutron

n-d-r: Peering failing on mixed IPv4 + IPv6 updates

Bug #2023632 reported by Dr. Jens Harbott on 2023-06-13

This bug affects 2 people

Affects		Status	Importance	Assigned to	Milestone
	neutron	In Progress	High	Dr. Jens Harbott

Bug Description

Scenario: Having BGP peers defined for IPv6 to advertise tenant networks, like for a standard deployment with public IPv6 connectivity. When a router has both IPv4 and IPv6 subnets attached, updates like adding a new IPv6 subnet also create updates for the IPv4 prefixes, but these crash the peering:

Jun 13 08:06:03 vm2 neutron-bgp-dragent[3280003]: netaddr.core.AddrFormatError: base address '2001:db8::308' is not IPv4
Jun 13 08:06:03 vm2 neutron-bgp-dragent[3280003]: During handling of the above exception, another exception occurred:
Jun 13 08:06:03 vm2 neutron-bgp-dragent[3280003]: Traceback (most recent call last):
Jun 13 08:06:03 vm2 neutron-bgp-dragent[3280003]: File "/usr/local/lib/python3.10/dist-packages/os_ken/lib/hub.py", line 69, in _launch
Jun 13 08:06:03 vm2 neutron-bgp-dragent[3280003]: return func(*args, **kwargs)
Jun 13 08:06:03 vm2 neutron-bgp-dragent[3280003]: File "/usr/local/lib/python3.10/dist-packages/os_ken/services/protocols/bgp/base.py", line 253, in start
Jun 13 08:06:03 vm2 neutron-bgp-dragent[3280003]: self._run(*args, **kwargs)
Jun 13 08:06:03 vm2 neutron-bgp-dragent[3280003]: File "/usr/local/lib/python3.10/dist-packages/os_ken/services/protocols/bgp/peer.py", line 683, in _run
Jun 13 08:06:03 vm2 neutron-bgp-dragent[3280003]: self._process_outgoing_msg_list()
Jun 13 08:06:03 vm2 neutron-bgp-dragent[3280003]: File "/usr/local/lib/python3.10/dist-packages/os_ken/services/protocols/bgp/peer.py", line 769, in _process_outgoing_msg_list
Jun 13 08:06:03 vm2 neutron-bgp-dragent[3280003]: self._send_outgoing_route(outgoing_msg)
Jun 13 08:06:03 vm2 neutron-bgp-dragent[3280003]: File "/usr/local/lib/python3.10/dist-packages/os_ken/services/protocols/bgp/peer.py", line 729, in _send_outgoing_route
Jun 13 08:06:03 vm2 neutron-bgp-dragent[3280003]: self._protocol.send(update_msg)
Jun 13 08:06:03 vm2 neutron-bgp-dragent[3280003]: File "/usr/local/lib/python3.10/dist-packages/os_ken/services/protocols/bgp/speaker.py", line 395, in send
Jun 13 08:06:03 vm2 neutron-bgp-dragent[3280003]: self._send_with_lock(msg)
Jun 13 08:06:03 vm2 neutron-bgp-dragent[3280003]: File "/usr/local/lib/python3.10/dist-packages/os_ken/services/protocols/bgp/speaker.py", line 384, in _send_with_lock
Jun 13 08:06:03 vm2 neutron-bgp-dragent[3280003]: self._socket.sendall(msg.serialize())
Jun 13 08:06:03 vm2 neutron-bgp-dragent[3280003]: File "/usr/local/lib/python3.10/dist-packages/os_ken/lib/packet/bgp.py", line 5245, in serialize
Jun 13 08:06:03 vm2 neutron-bgp-dragent[3280003]: tail = self.serialize_tail()
Jun 13 08:06:03 vm2 neutron-bgp-dragent[3280003]: File "/usr/local/lib/python3.10/dist-packages/os_ken/lib/packet/bgp.py", line 5465, in serialize_tail
Jun 13 08:06:03 vm2 neutron-bgp-dragent[3280003]: binpathattrs += pa.serialize()
Jun 13 08:06:03 vm2 neutron-bgp-dragent[3280003]: File "/usr/local/lib/python3.10/dist-packages/os_ken/lib/packet/bgp.py", line 3661, in serialize
Jun 13 08:06:03 vm2 neutron-bgp-dragent[3280003]: value = self.serialize_value()
Jun 13 08:06:03 vm2 neutron-bgp-dragent[3280003]: File "/usr/local/lib/python3.10/dist-packages/os_ken/lib/packet/bgp.py", line 3871, in serialize_value
Jun 13 08:06:03 vm2 neutron-bgp-dragent[3280003]: addrconv.ipv4.text_to_bin(self.value))
Jun 13 08:06:03 vm2 neutron-bgp-dragent[3280003]: File "/usr/local/lib/python3.10/dist-packages/os_ken/lib/addrconv.py", line 36, in text_to_bin
Jun 13 08:06:03 vm2 neutron-bgp-dragent[3280003]: ip = self._fallback(text, **self._addr_kwargs)
Jun 13 08:06:03 vm2 neutron-bgp-dragent[3280003]: File "/usr/local/lib/python3.10/dist-packages/netaddr/ip/__init__.py", line 930, in __init__
Jun 13 08:06:03 vm2 neutron-bgp-dragent[3280003]: value, prefixlen = parse_ip_network(_ipv4, addr,
Jun 13 08:06:03 vm2 neutron-bgp-dragent[3280003]: File "/usr/local/lib/python3.10/dist-packages/netaddr/ip/__init__.py", line 803, in parse_ip_network
Jun 13 08:06:03 vm2 neutron-bgp-dragent[3280003]: expanded_addr = _ipv4.expand_partial_address(val1)
Jun 13 08:06:03 vm2 neutron-bgp-dragent[3280003]: File "/usr/local/lib/python3.10/dist-packages/netaddr/strategy/ipv4.py", line 259, in expand_partial_address
Jun 13 08:06:03 vm2 neutron-bgp-dragent[3280003]: raise error
Jun 13 08:06:03 vm2 neutron-bgp-dragent[3280003]: netaddr.core.AddrFormatError: invalid partial IPv4 address: '2001:db8::308'!
Jun 13 08:06:03 vm2 neutron-bgp-dragent[3280003]: : netaddr.core.AddrFormatError: invalid partial IPv4 address: '2001:db8::308'!

Full logs attached.

Tags:

Revision history for this message

Dr. Jens Harbott (j-harbott) wrote on 2023-06-13:

svc.log Edit (1.7 KiB, text/plain)

tags:

added: l3-bgp

Revision history for this message

Dr. Jens Harbott (j-harbott) wrote on 2023-06-13:

dr-agent.log Edit (20.0 KiB, text/plain)

Changed in neutron:
assignee:	nobody → Dr. Jens Harbott (j-harbott)

Lajos Katona (lajos-katona) on 2023-06-13

tags:	added: os-ken
Changed in neutron:
importance:	Undecided → High

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-06-13: Fix proposed to neutron-dynamic-routing (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron-dynamic-routing/+/885993

Changed in neutron:
status:	New → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2023-06-13: Fix proposed to os-ken (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/os-ken/+/885998

Revision history for this message

Roberto Bartzen Acosta (rbartzen) wrote on 2023-07-17:

Download full text (3.2 KiB)

I also see this issue in my local deployment.

Steps to reproduce:

Step 1 - default n-d-r config
----------------------------

# Provider network
openstack network create provider --external --provider-physical-network \
provider --provider-network-type flat

# Address scope
openstack address scope create --share --ip-version 6 test-address-scope-ipv6

# Provider IPv6 Subnet pool
openstack subnet pool create --address-scope test-address-scope-ipv6 --pool-prefix 2001:db8:1:1::/64 --default-prefix-length 64 test-pool-ipv6-provider

# Provider IPv6 subnet
openstack subnet create --ip-version 6 --subnet-pool test-pool-ipv6-provider --network provider public-subnet-v6 --ipv6-address-mode dhcpv6-stateful --ipv6-ra-mode dhcpv6-stateful --allocation-pool start=2001:db8:1:1::4-2001:db8:1:1:ffff:ffff:ffff:ffff --gateway 2001:db8:1:1::1

# Provider IPv4 subnet - no address scope required...
openstack subnet create --ip-version 4 --network provider --dhcp --host-route destination=172.16.0.0/24,gateway=172.16.0.1 --subnet-range 172.16.0.0/24 public-subnet-v4

# self-service Network

openstack network create self-service

# Provider IPv6 Self-service Network
openstack subnet pool create --address-scope test-address-scope-ipv6 --share --default --pool-prefix 2001:db8:1234:8000::/50 --default-prefix-length 64 --max-prefix-length 64 test-default-pool-ipv6

openstack subnet create --ip-version 6 --subnet-pool test-default-pool-ipv6 --network self-service --ipv6-address-mode dhcpv6-stateful --ipv6-ra-mode dhcpv6-stateful self-service-v6

openstack subnet create --ip-version 4 --network self-service --dhcp --host-route destination=192.168.0.0/24,gateway=192.168.0.1 --subnet-range 192.168.0.0/24 self-service-v4

# create router
openstack router create router1

# add self-service subnet as an interface on the router
openstack router add subnet router1 self-service-v4
openstack router add subnet router1 self-service-v6

# Add the provider network as a gateway on each router.
openstack router set --external-gateway provider router1

# create bgp speaker
openstack bgp speaker create --ip-version 6 --local-as 65000 bgpspeaker
openstack bgp speaker add network bgpspeaker provider

# create a vm on the self-service network
openstack server create --image cirros --flavor 1vcpu --network=self-service --security-group cf2e7d53-0db7-4873-82ab-cf67eceda937 vm1

*** works without issues

----------------------------

Step 2 - aditional config to reproduce the problem

# create some random address scope for IPv4
openstack address scope create --ip-version 4 test-reproducer

# create the subnet linked with the IPv4 address scope
openstack subnet pool create --address-scope test-reproducer --pool-prefix 10.173.0.0/16 --default-prefix-length 16 self-service-v4-reproducer

# create an additional router
openstack router create router2

# add self-service subnet as an interface on the router ***** the self-service-v4-reproducer triggers the problem
openstack router add subnet router2 self-service-v4-reproducer
openstack router add subnet router2 self-service-v6

# Add the provider network as a gateway on each router.
openstack router set --external-gateway provid...

I also see this issue in my local deployment.

Steps to reproduce:

Step 1 - default n-d-r config 
----------------------------

# Provider network
openstack network create provider --external --provider-physical-network \
  provider --provider-network-type flat

# Address scope
openstack address scope create --share --ip-version 6 test-address-scope-ipv6

# Provider IPv6 Subnet pool 
openstack subnet pool create --address-scope test-address-scope-ipv6 --pool-prefix 2001:db8:1:1::/64 --default-prefix-length 64 test-pool-ipv6-provider

# Provider IPv6 subnet 
openstack subnet create --ip-version 6 --subnet-pool test-pool-ipv6-provider --network provider public-subnet-v6 --ipv6-address-mode dhcpv6-stateful  --ipv6-ra-mode dhcpv6-stateful --allocation-pool start=2001:db8:1:1::4-2001:db8:1:1:ffff:ffff:ffff:ffff --gateway 2001:db8:1:1::1

# Provider IPv4 subnet - no address scope required... 
openstack subnet create --ip-version 4 --network provider --dhcp --host-route  destination=172.16.0.0/24,gateway=172.16.0.1 --subnet-range 172.16.0.0/24 public-subnet-v4

# self-service Network

openstack network create self-service

openstack subnet create --ip-version 6 --subnet-pool test-default-pool-ipv6 --network self-service --ipv6-address-mode dhcpv6-stateful --ipv6-ra-mode dhcpv6-stateful self-service-v6

openstack subnet create --ip-version 4 --network self-service  --dhcp --host-route  destination=192.168.0.0/24,gateway=192.168.0.1 --subnet-range 192.168.0.0/24 self-service-v4

# create router
openstack router create router1

# add self-service subnet as an interface on the router
openstack router add subnet router1 self-service-v4
openstack router add subnet router1 self-service-v6

# Add the provider network as a gateway on each router.
openstack router set --external-gateway provider router1

# create bgp speaker
openstack bgp speaker create --ip-version 6 --local-as 65000 bgpspeaker
openstack bgp speaker add network bgpspeaker provider

# create a vm on the self-service network
openstack server create --image cirros --flavor 1vcpu --network=self-service --security-group cf2e7d53-0db7-4873-82ab-cf67eceda937 vm1

*** works without issues

----------------------------

Step 2 - aditional config to reproduce the problem

# create some random address scope for IPv4
openstack address scope create --ip-version 4 test-reproducer

# create the subnet linked with the IPv4 address scope
openstack subnet pool create --address-scope test-reproducer --pool-prefix 10.173.0.0/16 --default-prefix-length 16 self-service-v4-reproducer

# create an additional router
openstack router create router2

# Add the provider network as a gateway on each router.
openstack router set --external-gateway provider router2

# the problem occurs! logs of exception in os_ken.

Revision history for this message

Roberto Bartzen Acosta (rbartzen) wrote on 2023-07-18:

Hi Frickler,

I fixed the problem with the mixed routes on n-d-r, can I submit a patch on this LP or should I open a new LP BUG?

Regards,
Roberto

Revision history for this message

Roberto Bartzen Acosta (rbartzen) wrote on 2023-07-20:

Proposed Patch: https://review.opendev.org/c/openstack/neutron-dynamic-routing/+/888787

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Bug attachments

Add attachment

Remote bug watches

Bug watches keep track of this bug in other bug trackers.