OpenStack Compute (nova)

deallocate_fixed_ip attempts to update an already deleted fixed_ip

Bug #1017633 reported by John Tran
28
This bug affects 5 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Medium
John Tran
OpenStack Compute (nova) 2012.2 "folsom"
Essex
Fix Released
Medium
Vish Ishaya
OpenStack Compute (nova) 2012.1.3
nova (Ubuntu)
Fix Released
Undecided
Unassigned
Nominated for Precise by Yolanda Robla

Bug Description

When deleting a vm , the nova network manager attempts to deallocate the fixed ip but the context has read_deleted set to 'yes' therefore attempts to update an already deleted fixed_ip (if exists).

Use case to replicate the problem, via devstack environment:

1. create a network , delete it, recreate it ; So that you'll have a set of deleted fixed_ips in the db.

$ nova-manage network create --fixed_range_v4 192.168.100.0/24 --label mynet
$ nova-manage network delete 192.168.100.0/24
$ nova-manage network create --fixed_range_v4 192.168.100.0/24 --label mynet

2. Verify that the network id is no longer '1', and that there are fixed_ips w/ 'deleted' column set to '1':

$ mysql -u root -p nova -e 'select id from networks;'
Enter password:
+----+
| id |
+----+
| 2 |
+----+

$ mysql -u root -p nova -e 'select id, address, deleted from fixed_ips limit 10;'
Enter password:
+----+---------------+---------+
| id | address | deleted |
+----+---------------+---------+
| 1 | 192.168.100.0 | 1 |
| 2 | 192.168.100.1 | 1 |
| 3 | 192.168.100.2 | 1 |
| 4 | 192.168.100.3 | 1 |
| 5 | 192.168.100.4 | 1 |
| 6 | 192.168.100.5 | 1 |
| 7 | 192.168.100.6 | 1 |
| 8 | 192.168.100.7 | 1 |
| 9 | 192.168.100.8 | 1 |
| 10 | 192.168.100.9 | 1 |
+----+---------------+---------+

3. Create a new instance:

$ nova boot --flavor m1.tiny --image cirros-0.3.0-x86_64-uec test1

4. Delete the instance and verify the trace in network log/console

$ nova delete test1

from network log, trace shows it is looking at the wrong network_id "1" instead of the new network_id "2":

2012-06-25 11:40:24 DEBUG nova.network.manager [req-ad72fa8a-713a-477a-8a3b-0e4eb0f50f28 45bb07bd602b4a029a606927e1599b00 0f2bd5d9b82e4e45af76e1579070c313] network deallocation for instance |3| from (pid=4155) deallocate_for_instance /opt/stack/nova/nova/network/manager.py:984
2012-06-25 11:40:24 ERROR nova.openstack.common.rpc.amqp [-] Exception during message handling
2012-06-25 11:40:24 TRACE nova.openstack.common.rpc.amqp Traceback (most recent call last):
2012-06-25 11:40:24 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/amqp.py", line 273, in _process_data
2012-06-25 11:40:24 TRACE nova.openstack.common.rpc.amqp rval = self.proxy.dispatch(ctxt, version, method, **args)
2012-06-25 11:40:24 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/openstack/common/rpc/dispatcher.py", line 103, in dispatch
2012-06-25 11:40:24 TRACE nova.openstack.common.rpc.amqp return getattr(proxyobj, method)(ctxt, **kwargs)
2012-06-25 11:40:24 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/network/manager.py", line 263, in wrapped
2012-06-25 11:40:24 TRACE nova.openstack.common.rpc.amqp return func(self, context, *args, **kwargs)
2012-06-25 11:40:24 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/network/manager.py", line 384, in deallocate_for_instance
2012-06-25 11:40:24 TRACE nova.openstack.common.rpc.amqp super(FloatingIP, self).deallocate_for_instance(context, **kwargs)
2012-06-25 11:40:24 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/network/manager.py", line 263, in wrapped
2012-06-25 11:40:24 TRACE nova.openstack.common.rpc.amqp return func(self, context, *args, **kwargs)
2012-06-25 11:40:24 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/network/manager.py", line 988, in deallocate_for_instance
2012-06-25 11:40:24 TRACE nova.openstack.common.rpc.amqp **kwargs)
2012-06-25 11:40:24 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/network/manager.py", line 236, in deallocate_fixed_ip
2012-06-25 11:40:24 TRACE nova.openstack.common.rpc.amqp network = self._get_network_by_id(context, fixed_ip['network_id'])
2012-06-25 11:40:24 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/network/manager.py", line 1794, in _get_network_by_id
2012-06-25 11:40:24 TRACE nova.openstack.common.rpc.amqp network_id)
2012-06-25 11:40:24 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/network/manager.py", line 1604, in _get_network_by_id
2012-06-25 11:40:24 TRACE nova.openstack.common.rpc.amqp return self.db.network_get(context, network_id)
2012-06-25 11:40:24 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/db/api.py", line 795, in network_get
2012-06-25 11:40:24 TRACE nova.openstack.common.rpc.amqp return IMPL.network_get(context, network_id)
2012-06-25 11:40:24 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 132, in wrapper
2012-06-25 11:40:24 TRACE nova.openstack.common.rpc.amqp return f(*args, **kwargs)
2012-06-25 11:40:24 TRACE nova.openstack.common.rpc.amqp File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 2041, in network_get
2012-06-25 11:40:24 TRACE nova.openstack.common.rpc.amqp raise exception.NetworkNotFound(network_id=network_id)
2012-06-25 11:40:24 TRACE nova.openstack.common.rpc.amqp NetworkNotFound: Network 1 could not be found.
2012-06-25 11:40:24 TRACE nova.openstack.common.rpc.amqp

John Tran (jtran)
Changed in nova:
assignee: nobody → John Tran (jtran)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/8971

Changed in nova:
status: New → In Progress
Vish Ishaya (vishvananda)
Changed in nova:
importance: Undecided → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/8971
Committed: http://github.com/openstack/nova/commit/61ab72d15b3ac61b245e0bdd4a7bee5f3a673f75
Submitter: Jenkins
Branch: master

commit 61ab72d15b3ac61b245e0bdd4a7bee5f3a673f75
Author: John H. Tran <email address hidden>
Date: Fri Jun 22 19:14:06 2012 -0700

    deallocate_fixed_ip attempts to update deleted ip

    Fixes bug 1017633. When deleting a vm, the nova
    network manager looks to deallocate wrong fixed
    ip when context has read_deleted set to 'yes',
    in case when a network had been deleted and re-
    created , it attempts to update already deleted
    fixed_ips and therefore looks to teardown from
    wrong network_id as well.

    Change-Id: I574a20273220ef81498403da80f489732ae81eb1

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
milestone: none → folsom-2
status: Fix Committed → Fix Released
Revision history for this message
Roman Sokolkov (rsokolkov) wrote :
Download full text (4.8 KiB)

Folks, could we port FIX to STABLE/ESSEX ?

Also we need to port this one:

https://github.com/openstack/nova/commit/50b9c032fdc520c1461ff4651b60b4fc4b8f8e19

Because I've also plunged in this traceback, if I've tried to launch an instance:

compute.log

2012-08-07 22:25:03 DEBUG nova.rpc.amqp [req-fd243ca3-feaf-4071-926d-2a83ed6467ba t1a t1] Making asynchronous cast on network... from (pid=31512) cast /usr/lib/python2.6/site-packages/nova/rpc/amqp.py:351
2012-08-07 22:25:03 ERROR nova.rpc.amqp [req-fd243ca3-feaf-4071-926d-2a83ed6467ba t1a t1] Exception during message handling
2012-08-07 22:25:03 TRACE nova.rpc.amqp Traceback (most recent call last):
2012-08-07 22:25:03 TRACE nova.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/rpc/amqp.py", line 253, in _process_data
2012-08-07 22:25:03 TRACE nova.rpc.amqp rval = node_func(context=ctxt, **node_args)
2012-08-07 22:25:03 TRACE nova.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/exception.py", line 114, in wrapped
2012-08-07 22:25:03 TRACE nova.rpc.amqp return f(*args, **kw)
2012-08-07 22:25:03 TRACE nova.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 183, in decorated_function
2012-08-07 22:25:03 TRACE nova.rpc.amqp sys.exc_info())
2012-08-07 22:25:03 TRACE nova.rpc.amqp File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2012-08-07 22:25:03 TRACE nova.rpc.amqp self.gen.next()
2012-08-07 22:25:03 TRACE nova.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 177, in decorated_function
2012-08-07 22:25:03 TRACE nova.rpc.amqp return function(self, context, instance_uuid, *args, **kwargs)
2012-08-07 22:25:03 TRACE nova.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 657, in run_instance
2012-08-07 22:25:03 TRACE nova.rpc.amqp do_run_instance()
2012-08-07 22:25:03 TRACE nova.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/utils.py", line 945, in inner
2012-08-07 22:25:03 TRACE nova.rpc.amqp retval = f(*args, **kwargs)
2012-08-07 22:25:03 TRACE nova.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 656, in do_run_instance
2012-08-07 22:25:03 TRACE nova.rpc.amqp self._run_instance(context, instance_uuid, **kwargs)
2012-08-07 22:25:03 TRACE nova.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 457, in _run_instance
2012-08-07 22:25:03 TRACE nova.rpc.amqp self._set_instance_error_state(context, instance_uuid)
2012-08-07 22:25:03 TRACE nova.rpc.amqp File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2012-08-07 22:25:03 TRACE nova.rpc.amqp self.gen.next()
2012-08-07 22:25:03 TRACE nova.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 438, in _run_instance
2012-08-07 22:25:03 TRACE nova.rpc.amqp self._deallocate_network(context, instance)
2012-08-07 22:25:03 TRACE nova.rpc.amqp File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
2012-08-07 22:25:03 TRACE nova.rpc.amqp self.gen.next()
2012-08-07 22:25:03 TRACE nova.rpc.amqp File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 435, in _run...

Read more...

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (stable/essex)

Fix proposed to branch: stable/essex
Review: https://review.openstack.org/13678

Thierry Carrez (ttx)
Changed in nova:
milestone: folsom-2 → 2012.2
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (stable/essex)

Reviewed: https://review.openstack.org/13678
Committed: http://github.com/openstack/nova/commit/ae9c5f41b2a21183e80e6df19bdd7c56380c84a8
Submitter: Jenkins
Branch: stable/essex

commit ae9c5f41b2a21183e80e6df19bdd7c56380c84a8
Author: John H. Tran <email address hidden>
Date: Fri Jun 22 19:14:06 2012 -0700

    deallocate_fixed_ip attempts to update deleted ip

    Fixes bug 1017633. When deleting a vm, the nova
    network manager looks to deallocate wrong fixed
    ip when context has read_deleted set to 'yes',
    in case when a network had been deleted and re-
    created , it attempts to update already deleted
    fixed_ips and therefore looks to teardown from
    wrong network_id as well.

    Change-Id: I574a20273220ef81498403da80f489732ae81eb1
    (cherry picked from commit 61ab72d15b3ac61b245e0bdd4a7bee5f3a673f75)

Yolanda Robla (yolanda.robla)
Changed in nova (Ubuntu):
status: New → Fix Released
Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello John, or anyone else affected,

Accepted nova into precise-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/nova/2012.1.3+stable-20130423-e52e6912-0ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

tags: added: verification-needed
Revision history for this message
Yolanda Robla (yolanda.robla) wrote : Verification report.

Please find the attached test log from the Ubuntu Server Team's CI infrastructure. As part of the verification process for this bug, Nova has been deployed and configured across multiple nodes using precise-proposed as an installation source. After successful bring-up and configuration of the cluster, a number of exercises and smoke tests have be invoked to ensure the updated package did not introduce any regressions. A number of test iterations were carried out to catch any possible transient errors.

Please Note the list of installed packages at the top and bottom of the report.

For records of upstream test coverage of this update, please see the Jenkins links in the comments of the relevant upstream code-review(s):

Trunk review: https://review.openstack.org/8971
Stable review: https://review.openstack.org/13678

As per the provisional Micro Release Exception granted to this package by the Technical Board, we hope this contributes toward verification of this update.

Revision history for this message
Yolanda Robla (yolanda.robla) wrote :

Test coverage log.

Revision history for this message
Yolanda Robla (yolanda.robla) wrote :

Please find the attached test log from the Ubuntu Server Team's CI infrastructure. As part of the verification process for this bug, Nova has been deployed and configured across multiple nodes using precise-proposed as an installation source. After successful bring-up and configuration of the cluster, a number of exercises and smoke tests have be invoked to ensure the updated package did not introduce any regressions. A number of test iterations were carried out to catch any possible transient errors.

Please Note the list of installed packages at the top and bottom of the report.

For records of upstream test coverage of this update, please see the Jenkins links in the comments of the relevant upstream code-review(s):

Trunk review: https://review.openstack.org/8971
Stable review: https://review.openstack.org/13678

As per the provisional Micro Release Exception granted to this package by the Technical Board, we hope this contributes toward verification of this update.

Revision history for this message
Yolanda Robla (yolanda.robla) wrote :

Test coverage log.

Yolanda Robla (yolanda.robla)
tags: added: verification-done
removed: verification-needed
Revision history for this message
Scott Kitterman (kitterman) wrote : Update Released

The verification of this Stable Release Update has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regresssions.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.