neutron

DBReferenceError is raised when updating dvr port binding

  1. Series kilo
  2. Bug #1501515
Bug #1501515 reported by Adolfo Duarte
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
neutron
Fix Released
Medium
Kevin Benton
neutron mitaka-rc1 "RC1"
Kilo
New
Undecided
Unassigned

Bug Description

several tests upstream are failing with a DBReferenceError exception when calls to
File "/opt/stack/new/neutron/neutron/plugins/ml2/db.py", line 208, in ensure_dvr_port_binding

This appears to be caused by accessing database resource without locking it (race condition).

Here is an excerpt of the error:
2015-09-29 18:39:00.822 7813 ERROR oslo_messaging.rpc.dispatcher DBReferenceError: (pymysql.err.IntegrityError) (1452, u'Cannot add or update a child row: a foreign key constraint fails (`neutron`.`ml2_dvr_port_bindings`, CONSTRAINT `ml2_dvr_port_bindings_ibfk_1` FOREIGN KEY (`port_id`) REFERENCES `ports` (`id`) ON DELETE CASCADE)') [SQL: u'INSERT INTO ml2_dvr_port_bindings (port_id, host, router_id, vif_type, vif_details, vnic_type, profile, status) VALUES (%s, %s, %s, %s, %s, %s, %s, %s)'] [parameters: (u'851c0627-5133-43e2-b7a3-da9c29afd4ea', u'devstack-trusty-hpcloud-b2-5150973', u'973254dc-d1aa-4177-b952-2ac648bad4b5', 'unbound', '', 'normal', '', 'DOWN')]

An example of the failure can be found here:
http://logs.openstack.org/17/227517/3/check/gate-tempest-dsvm-neutron-dvr/fc1efa2/logs/screen-q-svc.txt.gz?level=ERROR

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (master)

Related fix proposed to branch: master
Review: https://review.openstack.org/229662

Revision history for this message
Adolfo Duarte (adolfo-duarte) wrote : Re: mysql exception DBReferenceError is raised when calling update_dvr_port_binding

proposed following fix to see if it fixes problem:
https://review.openstack.org/#/c/229662/

Revision history for this message
Armando Migliaccio (armando-migliaccio) wrote :

I have started collecting some notes on how to improve the way we do bug reporting and management. Please make yourself familiar with them. Thanks

https://review.openstack.org/#/c/228733/

Changed in neutron:
status: New → Confirmed
tags: added: l3-dvr-backlog
Changed in neutron:
importance: Undecided → Medium
summary: - mysql exception DBReferenceError is raised when calling
- update_dvr_port_binding
+ DBReferenceError is raised when updating dvr port binding
Revision history for this message
Armando Migliaccio (armando-migliaccio) wrote :

This seems to be log noise, as we have both failures and successes:

http://logstash.openstack.org/#eyJzZWFyY2giOiJtZXNzYWdlOlwiREJSZWZlcmVuY2VFcnJvcjogKHB5bXlzcWwuZXJyLkludGVncml0eUVycm9yKSAoMTQ1MiwgdSdDYW5ub3QgYWRkIG9yIHVwZGF0ZSBhIGNoaWxkIHJvdzogYSBmb3JlaWduIGtleSBjb25zdHJhaW50IGZhaWxzIChgbmV1dHJvbmAuYG1sMl9kdnJfcG9ydF9iaW5kaW5nc2BcIiBBTkQgZmlsZW5hbWU6XCJsb2dzL3NjcmVlbi1xLXN2Yy50eHRcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTQ0MzY3MTc2NDkxNSwibW9kZSI6IiIsImFuYWx5emVfZmllbGQiOiIifQ==

message:"DBReferenceError: (pymysql.err.IntegrityError) (1452, u'Cannot add or update a child row: a foreign key constraint fails (`neutron`.`ml2_dvr_port_bindings`" AND filename:"logs/screen-q-svc.txt"

Changed in neutron:
importance: Medium → Low
assignee: nobody → Adolfo Duarte (adolfo-duarte)
Swaminathan Vasudevan (swaminathan-vasudevan)
Changed in neutron:
status: Confirmed → In Progress
Kevin Benton (kevinbenton)
Changed in neutron:
importance: Low → Medium
milestone: none → mitaka-rc1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/290120

Changed in neutron:
assignee: Adolfo Duarte (adolfo-duarte) → Kevin Benton (kevinbenton)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (master)

Change abandoned by Kevin Benton (<email address hidden>) on branch: master
Review: https://review.openstack.org/229662
Reason: See Ieddcb10bb019fe6dd7079255124e48305f406614

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (master)

Reviewed: https://review.openstack.org/290120
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=06569179aefcc569e7a3cc1fb828f0340d38d058
Submitter: Jenkins
Branch: master

commit 06569179aefcc569e7a3cc1fb828f0340d38d058
Author: Kevin Benton <email address hidden>
Date: Tue Mar 8 12:28:41 2016 -0800

    Catch DB reference errors in binding DVR ports

    A port can be deleted right after get_port is called but before
    the binding is created. So we need to catch both PortNotFound
    from the get_port and DBReferenceError from the binding creation.

    Co-Authored-By: Adolfo Duarte <email address hidden>

    Closes-Bug: #1501515
    Change-Id: Ieddcb10bb019fe6dd7079255124e48305f406614

Changed in neutron:
status: In Progress → Fix Released
Revision history for this message
Thierry Carrez (ttx) wrote : Fix included in openstack/neutron 8.0.0.0rc1

This issue was fixed in the openstack/neutron 8.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/liberty)

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/295987

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to neutron (stable/kilo)

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/296536

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/liberty)

Reviewed: https://review.openstack.org/295987
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=362896a7cf3f31f1deaa4699cdf4cc302d7125d6
Submitter: Jenkins
Branch: stable/liberty

commit 362896a7cf3f31f1deaa4699cdf4cc302d7125d6
Author: Kevin Benton <email address hidden>
Date: Tue Mar 8 12:28:41 2016 -0800

    Catch DB reference errors in binding DVR ports

    A port can be deleted right after get_port is called but before
    the binding is created. So we need to catch both PortNotFound
    from the get_port and DBReferenceError from the binding creation.

    Co-Authored-By: Adolfo Duarte <email address hidden>

    Closes-Bug: #1501515
    Change-Id: Ieddcb10bb019fe6dd7079255124e48305f406614
    (cherry picked from commit 06569179aefcc569e7a3cc1fb828f0340d38d058)

tags: added: in-stable-liberty
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to neutron (stable/kilo)

Reviewed: https://review.openstack.org/296536
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=b403b84d0cff53515464da4a21f52c0e3ab3420b
Submitter: Jenkins
Branch: stable/kilo

commit b403b84d0cff53515464da4a21f52c0e3ab3420b
Author: Kevin Benton <email address hidden>
Date: Tue Mar 8 12:28:41 2016 -0800

    Catch DB reference errors in binding DVR ports

    A port can be deleted right after get_port is called but before
    the binding is created. So we need to catch both PortNotFound
    from the get_port and DBReferenceError from the binding creation.

    Co-Authored-By: Adolfo Duarte <email address hidden>

    Closes-Bug: #1501515
    Change-Id: Ieddcb10bb019fe6dd7079255124e48305f406614
    (cherry picked from commit 06569179aefcc569e7a3cc1fb828f0340d38d058)

tags: added: in-stable-kilo
Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/neutron 2015.1.4

This issue was fixed in the openstack/neutron 2015.1.4 release.

Revision history for this message
Thierry Carrez (ttx) wrote : Fix included in openstack/neutron 7.1.0

This issue was fixed in the openstack/neutron 7.1.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/neutron 2015.1.4

This issue was fixed in the openstack/neutron 2015.1.4 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.