Deleting network associated with L2P results in infinite loop

Bug #1470646 reported by Sumit Naiksatam
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Group Based Policy
Triaged
High
Robert Kukura

Bug Description

Steps to reproduce:

stack@ubuntu:~/devstack$ gbp l2policy-create test-l2p
Created a new l2_policy:
+----------------------+--------------------------------------+
| Field | Value |
+----------------------+--------------------------------------+
| description | |
| id | 48befa7b-c4e5-474a-a070-8323a1259867 |
| l3_policy_id | ae382baa-a2d8-45fc-affd-b5b2131bded8 |
| name | test-l2p |
| network_id | 94a36d8b-b5a3-4a2a-90dd-e55edc521eab |
| policy_target_groups | |
| shared | False |
| tenant_id | 804cf28eaa634ceb8e9cbbe8d7f4b6d1 |
+----------------------+--------------------------------------+
stack@ubuntu:~/devstack$ neutron net-delete 94a36d8b-b5a3-4a2a-90dd-e55edc521eab

** hangs at this point **

In the Neutron log you will see an infinite loop of the following:

2015-07-01 14:11:50.311 DEBUG neutron.plugins.ml2.plugin [req-8fec1f1f-c848-4b03-977a-924e61ddd4cb admin admin] Deleting network 94a36d8b-b5a3-4a2a-90dd-e55edc521eab delete_network /opt/stack/neutron/neutron/plugins/ml2/plugin.py:711
2015-07-01 14:11:50.313 DEBUG oslo_concurrency.lockutils [req-8fec1f1f-c848-4b03-977a-924e61ddd4cb admin admin] Acquired semaphore "db-access" lock /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:391
2015-07-01 14:11:50.316 DEBUG neutron.plugins.ml2.plugin [req-8fec1f1f-c848-4b03-977a-924e61ddd4cb admin admin] Ports to auto-delete: [] delete_network /opt/stack/neutron/neutron/plugins/ml2/plugin.py:734
2015-07-01 14:11:50.317 DEBUG neutron.plugins.ml2.plugin [req-8fec1f1f-c848-4b03-977a-924e61ddd4cb admin admin] Subnets to auto-delete: [] delete_network /opt/stack/neutron/neutron/plugins/ml2/plugin.py:747
2015-07-01 14:11:50.330 DEBUG neutron.plugins.ml2.drivers.type_tunnel [req-8fec1f1f-c848-4b03-977a-924e61ddd4cb admin admin] Releasing vxlan tunnel 1071 to pool release_segment /opt/stack/neutron/neutron/plugins/ml2/drivers/type_tunnel.py:174
2015-07-01 14:11:50.339 DEBUG neutron.plugins.ml2.plugin [req-8fec1f1f-c848-4b03-977a-924e61ddd4cb admin admin] Deleting network record <neutron.db.models_v2.Network[object at 7ff88d8ae850] {tenant_id=u'804cf28eaa634ceb8e9cbbe8d7f4b6d1', id=u'94a36d8b-b5a3-4a2a-90dd-e55edc521eab', name=u'l2p_test-l2p', status=u'ACTIVE', admin_state_up=True, shared=False, mtu=0L, vlan_transparent=None}> delete_network /opt/stack/neutron/neutron/plugins/ml2/plugin.py:759
2015-07-01 14:11:50.339 DEBUG neutron.plugins.ml2.plugin [req-8fec1f1f-c848-4b03-977a-924e61ddd4cb admin admin] Committing transaction delete_network /opt/stack/neutron/neutron/plugins/ml2/plugin.py:764
2015-07-01 14:11:50.354 ERROR oslo_db.sqlalchemy.exc_filters [req-8fec1f1f-c848-4b03-977a-924e61ddd4cb admin admin] DBAPIError exception wrapped from (IntegrityError) (1451, 'Cannot delete or update a parent row: a foreign key constraint fails (`neutron`.`gp_l2_policies`, CONSTRAINT `gp_l2_policies_ibfk_2` FOREIGN KEY (`network_id`) REFERENCES `networks` (`id`))') 'DELETE FROM networks WHERE networks.id = %s' ('94a36d8b-b5a3-4a2a-90dd-e55edc521eab',)
2015-07-01 14:11:50.354 63383 TRACE oslo_db.sqlalchemy.exc_filters Traceback (most recent call last):
2015-07-01 14:11:50.354 63383 TRACE oslo_db.sqlalchemy.exc_filters File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1063, in _execute_context
2015-07-01 14:11:50.354 63383 TRACE oslo_db.sqlalchemy.exc_filters context)
2015-07-01 14:11:50.354 63383 TRACE oslo_db.sqlalchemy.exc_filters File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 442, in do_execute
2015-07-01 14:11:50.354 63383 TRACE oslo_db.sqlalchemy.exc_filters cursor.execute(statement, parameters)
2015-07-01 14:11:50.354 63383 TRACE oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 174, in execute
2015-07-01 14:11:50.354 63383 TRACE oslo_db.sqlalchemy.exc_filters self.errorhandler(self, exc, value)
2015-07-01 14:11:50.354 63383 TRACE oslo_db.sqlalchemy.exc_filters File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
2015-07-01 14:11:50.354 63383 TRACE oslo_db.sqlalchemy.exc_filters raise errorclass, errorvalue
2015-07-01 14:11:50.354 63383 TRACE oslo_db.sqlalchemy.exc_filters IntegrityError: (1451, 'Cannot delete or update a parent row: a foreign key constraint fails (`neutron`.`gp_l2_policies`, CONSTRAINT `gp_l2_policies_ibfk_2` FOREIGN KEY (`network_id`) REFERENCES `networks` (`id`))')
2015-07-01 14:11:50.354 63383 TRACE oslo_db.sqlalchemy.exc_filters
2015-07-01 14:11:50.372 DEBUG oslo_concurrency.lockutils [req-8fec1f1f-c848-4b03-977a-924e61ddd4cb admin admin] Releasing semaphore "db-access" lock /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:404

This happens because of this code block in the ML2 plugin:
https://github.com/openstack/neutron/blob/master/neutron/plugins/ml2/plugin.py#L721-L780

Workaround:
Restart the neutron server

Changed in group-based-policy:
milestone: kilo-gbp-4 → next
Changed in group-based-policy:
importance: High → Medium
importance: Medium → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to group-based-policy (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/243334

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to group-based-policy (master)

Reviewed: https://review.openstack.org/243334
Committed: https://git.openstack.org/cgit/openstack/group-based-policy/commit/?id=86b4c6d42828ab5d4bc6d8b14d0e915d613fb2c7
Submitter: Jenkins
Branch: master

commit 86b4c6d42828ab5d4bc6d8b14d0e915d613fb2c7
Author: Robert Kukura <email address hidden>
Date: Mon Nov 9 17:16:37 2015 -0500

    Limit ML2 delete_network/subnet retries

    Monkey-patch ML2's delete_network() and delete_subnet() methods to
    limit the number of times they retry to avoid potential infinite
    loops. Also add some logging to help determine when/why the
    delete_network() loops occur. This does not resolve the actual bug -
    it just mitigates the damage when it occurs.

    Partial-bug: 1510327
    Related-bug: 1470646

    Change-Id: I193d56b0ed16bcc69f434a87d11a355e9177eb1e

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to group-based-policy (stable/kilo)

Related fix proposed to branch: stable/kilo
Review: https://review.openstack.org/245350

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to group-based-policy (stable/kilo)

Reviewed: https://review.openstack.org/245350
Committed: https://git.openstack.org/cgit/openstack/group-based-policy/commit/?id=095216e11d1c8b92694f50d7ed8bf11df5a68744
Submitter: Jenkins
Branch: stable/kilo

commit 095216e11d1c8b92694f50d7ed8bf11df5a68744
Author: Robert Kukura <email address hidden>
Date: Mon Nov 9 17:16:37 2015 -0500

    Limit ML2 delete_network/subnet retries

    Monkey-patch ML2's delete_network() and delete_subnet() methods to
    limit the number of times they retry to avoid potential infinite
    loops. Also add some logging to help determine when/why the
    delete_network() loops occur. This does not resolve the actual bug -
    it just mitigates the damage when it occurs.

    Partial-bug: 1510327
    Related-bug: 1470646

    Change-Id: I193d56b0ed16bcc69f434a87d11a355e9177eb1e
    (cherry picked from commit 86b4c6d42828ab5d4bc6d8b14d0e915d613fb2c7)

tags: added: in-stable-kilo
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to group-based-policy (stable/juno)

Related fix proposed to branch: stable/juno
Review: https://review.openstack.org/245435

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to group-based-policy (stable/juno)

Reviewed: https://review.openstack.org/245435
Committed: https://git.openstack.org/cgit/openstack/group-based-policy/commit/?id=bea20393eacde996245c7f07f76d13fb585b96c8
Submitter: Jenkins
Branch: stable/juno

commit bea20393eacde996245c7f07f76d13fb585b96c8
Author: Robert Kukura <email address hidden>
Date: Mon Nov 9 17:16:37 2015 -0500

    Limit ML2 delete_network/subnet retries

    Monkey-patch ML2's delete_network() and delete_subnet() methods to
    limit the number of times they retry to avoid potential infinite
    loops. Also add some logging to help determine when/why the
    delete_network() loops occur. This does not resolve the actual bug -
    it just mitigates the damage when it occurs.

    Partial-bug: 1510327
    Related-bug: 1470646

    Conflicts:
     gbpservice/neutron/extensions/patch_ml2.py

    Change-Id: I193d56b0ed16bcc69f434a87d11a355e9177eb1e
    (cherry picked from commit 86b4c6d42828ab5d4bc6d8b14d0e915d613fb2c7)
    (cherry picked from commit 095216e11d1c8b92694f50d7ed8bf11df5a68744)

tags: added: in-stable-juno
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.