ML2 delete_network can loop infinitely if called in transaction

Bug #1551958 reported by Kevin Benton
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
High
Kevin Benton

Bug Description

Both delete_network and delete_subnet in ML2 use a while True to continuously attempt to delete a network or subnet. This logic is fine assuming its DB lookups are fresh on each iteration to get the latest state. However, it fails when these functions are called inside of another transaction because its lookups will get stale data and continue to loop over that data forever.

Thanks to Armando who spotted it: http://paste.openstack.org/show/488834/

Changed in neutron:
assignee: nobody → Kevin Benton (kevinbenton)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/286843

Changed in neutron:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/286920

Revision history for this message
Armando Migliaccio (armando-migliaccio) wrote :

You're welcome.

Changed in neutron:
milestone: none → mitaka-rc1
importance: Undecided → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.openstack.org/286920
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=d1c501cd1d31ed3f859975b40c01cb7aaa06d26d
Submitter: Jenkins
Branch: master

commit d1c501cd1d31ed3f859975b40c01cb7aaa06d26d
Author: Kevin Benton <email address hidden>
Date: Tue Mar 1 13:49:48 2016 -0800

    L3HA: Do not wrap create/delete in transaction

    This is unsafe when calling ML2 because ML2 assumes that its
    functions will not be called inside of a transaction. This is
    not only an issue for drivers that try to do DB lookups using
    a different session in the post commit operation, but it's a
    big issue for the delete methods.

    The delete subnet and network methods in ML2 have 'while True'
    loops that catch concurrency errors and retry the operation after
    looking up info. If these are called inside a transaction, the
    lookups will contain stale information and it can lead to the
    while True loop never terminating!

    Closes-Bug: #1551958
    Change-Id: I33dc084ed15e5491fdda19da712a746ca87fbc8c

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (master)

Reviewed: https://review.openstack.org/286843
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=afe1a834000d33900b8646d308fa26fa807a2ca0
Submitter: Jenkins
Branch: master

commit afe1a834000d33900b8646d308fa26fa807a2ca0
Author: Kevin Benton <email address hidden>
Date: Tue Mar 1 10:47:00 2016 -0800

    Block delete_(network|subnet) transactioned calls

    These functions assume that they won't be called inside of a
    transaction for two reasons. The first is an infinite loop that
    will only terminate if it can actually retrieve the latest
    information from the database. The second is that ML2's post-commit
    operations are expected to occur after data was actually committed
    so mechanism drivers that do DB lookups using a different session
    may break.

    This adds an explicit guard that prevents these two functions from
    being called within an active DB session. The rest of the ML2
    functions should eventually be converted to this style as well
    (see the partial bug).

    Related-Bug: #1551958
    Partial-Bug: #1540844
    Change-Id: I5c00b186585369ef6c8e2b9cb5a43b8bba0e5a7c

Revision history for this message
Thierry Carrez (ttx) wrote : Fix included in openstack/neutron 8.0.0.0rc1

This issue was fixed in the openstack/neutron 8.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/liberty)

Related fix proposed to branch: stable/liberty
Review: https://review.openstack.org/314022

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (stable/liberty)

Change abandoned by John Schwarz (<email address hidden>) on branch: stable/liberty
Review: https://review.openstack.org/314022

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/liberty)

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/321633

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/liberty)

Reviewed: https://review.openstack.org/321633
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=7130d4c5013e6cc37f3b0861424c7766ad5a66a4
Submitter: Jenkins
Branch: stable/liberty

commit 7130d4c5013e6cc37f3b0861424c7766ad5a66a4
Author: Kevin Benton <email address hidden>
Date: Tue Mar 1 13:49:48 2016 -0800

    L3HA: Do not wrap create/delete in transaction

    This is unsafe when calling ML2 because ML2 assumes that its
    functions will not be called inside of a transaction. This is
    not only an issue for drivers that try to do DB lookups using
    a different session in the post commit operation, but it's a
    big issue for the delete methods.

    The delete subnet and network methods in ML2 have 'while True'
    loops that catch concurrency errors and retry the operation after
    looking up info. If these are called inside a transaction, the
    lookups will contain stale information and it can lead to the
    while True loop never terminating!

    Conflicts:
     doc/source/devref/effective_neutron.rst
     neutron/db/l3_hamode_db.py

    Closes-Bug: #1551958
    Change-Id: I33dc084ed15e5491fdda19da712a746ca87fbc8c
    (cherry picked from commit d1c501cd1d31ed3f859975b40c01cb7aaa06d26d)

tags: added: in-stable-liberty
Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/neutron 7.1.2

This issue was fixed in the openstack/neutron 7.1.2 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.