IP address isn't assigned to a subnet gateway interface in some cases.
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
neutron |
Confirmed
|
High
|
Unassigned |
Bug Description
Hello everyone,
Seems I caught a race condition bug in neutron-l3-agent.
We have automated tests. One of the test performs the following scenario. Creates different resources such as network, subnet and so on. Then the test connects the subnet to a router and perform other things that are not related to this bug. The test is performed in a cycle with different parameters but we use the same parameters for the Neutron resources. I mean the test always creates subnet with the same CIDR 192.168.0.0/24 and the subnet gateway interface gets 192.168.0.1 IP address. The bug happens in the moment when the subnet is connecting to the router. I would like to note that is not a permanent bug, sometimes it happens but sometimes not.
So bug looks like you don't access to the instances(VMs) using floating IPs. It's not possible to ping them. I did some debug, it turned out the subnet gateway interface didn't get an IP sometimes. For example, when the bug happens the interface looks so
root@network-
...
389: qr-7dc17e0a-
link/ether fa:16:3e:1e:47:78 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet6 fe80::f816:
valid_lft forever preferred_lft forever
For a success case it looks so.
root@network-
...
393: qr-cccf794e-
link/ether fa:16:3e:54:0c:11 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 192.168.0.1/24 scope global qr-cccf794e-86
valid_lft forever preferred_lft forever
inet6 fe80::f816:
valid_lft forever preferred_lft forever
We are using Juju to deploy OpenStack. The version of the neutron-gateway 9.4.1, the version of the charm is 244.
tags: | added: l3-ha |
Changed in neutron: | |
status: | New → Confirmed |
importance: | Undecided → High |
More debug. I've slightly modified a couple of functions in neutron/ agent/linux/ keepalived. py. Just for debug purpose.
def add_vip(self, ip_cidr, interface_name, scope): dress(ip_ cidr, interface_name, scope)
self.vips. append( vip)
LOG.debug( 'VIP %s was added', vip)
LOG.debug( 'VIP %s already present in %s', vip, self.vips)
vip = KeepalivedVipAd
if vip not in self.vips:
else:
def remove_ vips_vroutes_ by_interface( self, interface_name): debug(' remove_ vips_vroutes_ by_interface: %s', interface_name)
LOG.
self.vips = [vip for vip in self.vips
if vip.interface_name != interface_name]
self. virtual_ routes. remove_ routes_ on_interface( interface_ name)
def remove_ vip_by_ ip_address( self, ip_address): debug(' remove_ vip_by_ ip_address: %s', ip_address)
LOG.
self.vips = [vip for vip in self.vips
if vip.ip_address != ip_address]
The original code can be found here https:/ /github. com/openstack/ neutron/ blob/master/ neutron/ agent/l3/ ha_router. py#L213
Got the following logs. The unimportant part was delete for readability. Also comments were added for every line.
# Create a gateway interface for the subnet. agent.linux. keepalived [req-f3c3106a- d72b-4eb4- b9ba-3ba6ba3b35 fd - 0d042481da874d2 29f542e35ef7ac5 89 - - -] VIP [fe80:: f816:3eff: fe8c:7ef8/ 64, qr-8c8b4c01-40, link] was added add_vip /usr/lib/ python2. 7/dist- packages/ neutron/ agent/linux/ keepalived. py:206 agent.linux. keepalived [req-f3c3106a- d72b-4eb4- b9ba-3ba6ba3b35 fd - 0d042481da874d2 29f542e35ef7ac5 89 - - -] VIP [192.168.0.1/24, qr-8c8b4c01-40, None] was added add_vip /usr/lib/ python2. 7/dist- packages/ neutron/ agent/linux/ keepalived. py:206 agent.linux. keepalived [req-f3c3106a- d72b-4eb4- b9ba-3ba6ba3b35 fd - 0d042481da874d2 29f542e35ef7ac5 89 - - -] Keepalived spawned with config /var/lib/ neutron/ ha_confs/ a35f384b- 549e-41c6- 8076-2283be384e 1b/keepalived. conf spawn /usr/lib/ python2. 7/dist- packages/ neutron/ agent/linux/ keepalived. py:447 agent.linux. keepalived [req-7ae4a777- fa22-42e1- b9a4-726d75ff36 20 - 0d042481da874d2 29f542e35ef7ac5 89 - - -] VIP [fe80:: f816:3eff: fe1e:4778/ 64, qr-7dc17e0a-97, link] was added add_vip /usr/lib/ python2. 7/dist- packages/ neutron/ agent/linux/ keepalived. py:206 agent.linux. keepalived [req-7ae4a777- fa22-42e1- b9a4-726d75ff36 20 - 0d042481da874d2 29f542e35ef7ac5 89 - - -] VIP [192.168.0.1/24, qr-7dc17e0a-97, None] already present in [<neutron. agent.linux. keepalived. KeepalivedVipAd dress object at 0x7f6d5750cd90>, <neutron. agent.linux. keepalived. KeepalivedVipAd dress object at 0x7f6d571ce7d0>, <neutron. agent.linux. keepalived. KeepalivedVipAd dress object at 0x7f6d5722dc50>, <neutron. agent.linux. keepalived. KeepalivedVipAd dress object at 0x7f6d57235490>, <neutron. agent.linux. ke...
2018-02-16 10:04:37.152 14476 DEBUG neutron.
# Assign an IP address to the interface
2018-02-16 10:04:37.153 14476 DEBUG neutron.
# Update keepalived configuration
2018-02-16 10:04:40.970 14476 DEBUG neutron.
...
# Create a new gateway interface for the subnet.
2018-02-16 10:06:11.728 14476 DEBUG neutron.
# Tries to assign an address but this operation fails as the address (192.168.0.1/24) is already in use.
2018-02-16 10:06:11.729 14476 DEBUG neutron.