Fuel for OpenStack

teardown_unused_network_gateway parameter is False by default

Bug #1401870 reported by Oleg S. Gelbukh on 2014-12-12

This bug affects 2 people

	Status	Importance	Assigned to	Milestone
Fuel for OpenStack	Fix Released	Medium	Bartłomiej Piotrowski	Fuel for OpenStack 6.1
5.1.x	Fix Committed	Medium	Bartłomiej Piotrowski	Fuel for OpenStack 5.1.1-updates
6.0.x	Fix Committed	Medium	Bartłomiej Piotrowski	Fuel for OpenStack 6.0-updates
6.1.x	Fix Released	Medium	Bartłomiej Piotrowski	Fuel for OpenStack 6.1

Bug Description

This parameter is used by nova-network to determine if a bridge and gateway ip address of a network must be deleted from Compute node when the network is deleted in API. By default, it is set to 'False' which means that bridge, ip address and dnsmasq process listening on that address are left behind even after the network is deleted.

This leads to problems when another network with the same parameters (i.e. cidr and vlan id) created on that host: nova-network fails to start dnsmasq for that network.

2014-12-12 10:49:58.062 8450 ERROR oslo.messaging.rpc.dispatcher [req-ba613287-419b-4ec3-8e06-b0eb30b93362 ] Exception during message handling: Unexpected error while running command.
Command: sudo nova-rootwrap /etc/nova/rootwrap.conf env CONFIG_FILE=["/usr/share/nova/nova-dist.conf", "/etc/nova/nova.conf", "/etc/nova/nova.conf"] NETWORK_ID=84 dnsmasq --strict-order --bind-interfaces --conf-fi
le= --pid-file=/var/lib/nova/networks/nova-br203.pid --listen-address=10.42.1.3 --except-interface=lo --dhcp-range=set:pumphouse--network-664099187,10.42.1.3,static,255.255.255.0,120s --dhcp-lease-max=256 --dhcp-h
ostsfile=/var/lib/nova/networks/nova-br203.conf --dhcp-script=/usr/bin/nova-dhcpbridge --leasefile-ro --domain=novalocal --no-hosts --addn-hosts=/var/lib/nova/networks/nova-br203.hosts
Exit code: 2
Stdout: u''
Stderr: u'\ndnsmasq: failed to create listening socket for 10.42.1.3: Address already in use\n'
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher incoming.message))
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args)
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/network/floating_ips.py", line 119, in allocate_for_instance
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher **kwargs)
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 515, in allocate_for_instance
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher requested_networks=requested_networks)
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 216, in _allocate_fixed_ips
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher vpn=vpn, address=address)
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 1778, in allocate_fixed_ip
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher self._setup_network_on_host(context, network)
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/openstack/common/lockutils.py", line 249, in inner
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher return f(*args, **kwargs)
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/network/manager.py", line 1892, in _setup_network_on_host
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher self.driver.update_dhcp(elevated, dev, network)
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/network/linux_net.py", line 1004, in update_dhcp
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher restart_dhcp(context, dev, network_ref)
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/openstack/common/lockutils.py", line 249, in inner
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher return f(*args, **kwargs)
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/network/linux_net.py", line 1118, in restart_dhcp
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher _execute(*cmd, run_as_root=True)
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/network/linux_net.py", line 1211, in _execute
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher return utils.execute(*cmd, **kwargs)
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/utils.py", line 165, in execute
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher return processutils.execute(*cmd, **kwargs)
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.6/site-packages/nova/openstack/common/processutils.py", line 195, in execute
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher cmd=sanitized_cmd)
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher ProcessExecutionError: Unexpected error while running command.
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher Command: sudo nova-rootwrap /etc/nova/rootwrap.conf env CONFIG_FILE=["/usr/share/nova/nova-dist.conf", "/etc/nova/nova.conf", "/etc/nova/nova.conf"]
NETWORK_ID=84 dnsmasq --strict-order --bind-interfaces --conf-file= --pid-file=/var/lib/nova/networks/nova-br203.pid --listen-address=10.42.1.3 --except-interface=lo --dhcp-range=set:pumphouse--network-664099187,
10.42.1.3,static,255.255.255.0,120s --dhcp-lease-max=256 --dhcp-hostsfile=/var/lib/nova/networks/nova-br203.conf --dhcp-script=/usr/bin/nova-dhcpbridge --leasefile-ro --domain=novalocal --no-hosts --addn-hosts=/va
r/lib/nova/networks/nova-br203.hosts
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher Exit code: 2
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher Stdout: u''
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher Stderr: u'\ndnsmasq: failed to create listening socket for 10.42.1.3: Address already in use\n'
2014-12-12 10:49:58.062 8450 TRACE oslo.messaging.rpc.dispatcher

See original description

Tags:

Oleg S. Gelbukh (gelbuhos) on 2014-12-12

description:

updated

Stanislav Makar (smakar) on 2014-12-12

Changed in fuel:
importance:	Undecided → Medium
assignee:	nobody → Fuel Library Team (fuel-library)
status:	New → Triaged
tags:	added: low-hanging-fruit

Vladimir Kuklin (vkuklin) on 2014-12-12

Changed in fuel:
milestone:	none → 6.1

Bartłomiej Piotrowski (bpiotrowski) on 2014-12-16

Changed in fuel:
assignee:	Fuel Library Team (fuel-library) → Bartlomiej Piotrowski (bpiotrowski)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-01-29: Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/151196

Changed in fuel:
status:	Triaged → In Progress

Revision history for this message

Bogdan Dobrelya (bogdando) wrote on 2015-02-05:

It looks like we should backport this medium bug for 6.0 and 5.1 as well

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-02-06: Fix proposed to fuel-library (stable/6.0)

Fix proposed to branch: stable/6.0
Review: https://review.openstack.org/153493

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-02-06: Fix proposed to fuel-library (stable/5.1)

Fix proposed to branch: stable/5.1
Review: https://review.openstack.org/153494

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-02-16: Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/151196
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=6aa926d89540f4af1af1d58e55fe6eddacba6e58
Submitter: Jenkins
Branch: master

commit 6aa926d89540f4af1af1d58e55fe6eddacba6e58
Author: Bartłomiej Piotrowski <email address hidden>
Date: Mon Feb 2 11:00:55 2015 +0100

Enable teardown_unused_network_gateway for nova-network deployments

    The parameter is used by nova-network to determine if a bridge and
    gateway IP address of a network must be deleted from the compute node
    when the network is deleted in API. By default, it is set to 'False'
    which means that bridge, IP address and dnsmasq process listening on
    that address are left behind even after the network is deleted.

Change-Id: I4c94bb03501f4238ead2378cf504485b7d67b236
Closes-Bug: 1401870

Changed in fuel:
status:	In Progress → Fix Committed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-02-16: Fix merged to fuel-library (stable/5.1)

Reviewed: https://review.openstack.org/153494
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=59a2149566b07e11c4520fa9f885732ea1041e46
Submitter: Jenkins
Branch: stable/5.1

commit 59a2149566b07e11c4520fa9f885732ea1041e46
Author: Bartłomiej Piotrowski <email address hidden>
Date: Thu Jan 29 12:56:57 2015 +0100

Enable teardown_unused_network_gateway for nova-network deployments

Change-Id: I4c94bb03501f4238ead2378cf504485b7d67b236
Closes-Bug: 1401870

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-02-16: Fix merged to fuel-library (stable/6.0)

Reviewed: https://review.openstack.org/153493
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=7b9cbc30d457600649fdf57c25990493a075069f
Submitter: Jenkins
Branch: stable/6.0

commit 7b9cbc30d457600649fdf57c25990493a075069f
Author: Bartłomiej Piotrowski <email address hidden>
Date: Thu Jan 29 12:56:57 2015 +0100

Enable teardown_unused_network_gateway for nova-network deployments

Change-Id: I4c94bb03501f4238ead2378cf504485b7d67b236
Closes-Bug: 1401870

Revision history for this message

Bartłomiej Piotrowski (bpiotrowski) wrote on 2015-05-27:

Verified on 6.1 #471.

[root@node-1 ~]# grep teardown_unused_network_gateway /etc/nova/nova.conf
#teardown_unused_network_gateway=false
teardown_unused_network_gateway=True

Revision history for this message

apporc (appleorchard2000) wrote on 2015-06-02:

I found this option is set just on primary controller, i don't know whether this is right. And i don't use nova-network, so can't test it.

Revision history for this message

Bartłomiej Piotrowski (bpiotrowski) wrote on 2015-06-02:

#10

apporc: It is supposed to be false only on nova-network environments, so everything is fine IMHO.

Revision history for this message

David A. Desrosiers (setuid) wrote on 2018-08-27:

#11

Years later, but adding a comment that this option doesn't appear to work in 100% of the cases, leaving empty bridges on remote hypervisors where no interfaces are attached.

Combined with another ebtables bug that appears to allow traffic to traverse the local bridge and br received by the remote, now empty bridge, it causes DNS and DHCP failures inside instances on the local bridge.

There appears to be a race condition between the time 'fip.host = None' is checked and when that API call commit the change to the db is flushed, where fip.host is checked too soon, and fails to remove the empty bridge.

The result, is an empty, remote bridge receiving traffic destined for a local bridge on another hypervisor.

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.