neutron

ipam driver not using retryrequest correctly

Bug #1555387 reported by Kevin Benton on 2016-03-10

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	neutron	Fix Released	High	Kevin Benton	neutron mitaka-rc1 "RC1"

Bug Description

The IPAM driver is not passing a constructed exception into RetryRequest so the reraise on retry exhaustion fails:

2016-03-09 16:05:41.989 98258 ERROR neutron.api.v2.resource [req-6e0e721f-12df-4132-81ef-f1b96875cdd7 c_rally_3c9bcbbd_m5PryrFI -] create failed
2016-03-09 16:05:41.989 98258 ERROR neutron.api.v2.resource Traceback (most recent call last):
2016-03-09 16:05:41.989 98258 ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/api/v2/resource.py", line 84, in resource
2016-03-09 16:05:41.989 98258 ERROR neutron.api.v2.resource result = method(request=request, **args)
2016-03-09 16:05:41.989 98258 ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/api/v2/base.py", line 410, in create
2016-03-09 16:05:41.989 98258 ERROR neutron.api.v2.resource return self._create(request, body, **kwargs)
2016-03-09 16:05:41.989 98258 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 148, in wrapper
2016-03-09 16:05:41.989 98258 ERROR neutron.api.v2.resource ectxt.value = e.inner_exc
2016-03-09 16:05:41.989 98258 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-03-09 16:05:41.989 98258 ERROR neutron.api.v2.resource self.force_reraise()
2016-03-09 16:05:41.989 98258 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-03-09 16:05:41.989 98258 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
2016-03-09 16:05:41.989 98258 ERROR neutron.api.v2.resource File "<string>", line 2, in reraise
2016-03-09 16:05:41.989 98258 ERROR neutron.api.v2.resource TypeError: exceptions must be old-style classes or derived from BaseException, not type
2016-03-09 16:05:41.989 98258 ERROR neutron.api.v2.resource
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api [req-d357eb93-048c-425b-b6de-f860be1d767a c_rally_3c9bcbbd_IA8UUZUc -] DB exceeded retry limit.
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api Traceback (most recent call last):
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 138, in wrapper
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api return f(*args, **kwargs)
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api File "/opt/stack/neutron/neutron/api/v2/base.py", line 521, in _create
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api obj = do_create(body)
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api File "/opt/stack/neutron/neutron/api/v2/base.py", line 503, in do_create
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api request.context, reservation.reservation_id)
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api self.force_reraise()
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api six.reraise(self.type_, self.value, self.tb)
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api File "/opt/stack/neutron/neutron/api/v2/base.py", line 496, in do_create
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api return obj_creator(request.context, **kwargs)
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api File "/opt/stack/neutron/neutron/db/l3_hamode_db.py", line 463, in create_router
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api self.delete_router(context, router_dict['id'])
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api self.force_reraise()
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api six.reraise(self.type_, self.value, self.tb)
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api File "/opt/stack/neutron/neutron/db/l3_hamode_db.py", line 457, in create_router
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api context, router_db)[1]
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api File "/opt/stack/neutron/neutron/db/l3_hamode_db.py", line 443, in _create_ha_interfaces_and_ensure_network
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api creator, dep_getter, dep_creator, dep_id_attr)
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api File "/opt/stack/neutron/neutron/common/utils.py", line 707, in create_object_with_dependency
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api ctx.reraise = False
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api self.force_reraise()
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api six.reraise(self.type_, self.value, self.tb)
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api File "/opt/stack/neutron/neutron/common/utils.py", line 696, in create_object_with_dependency
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api result = creator(dependency)
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api File "/opt/stack/neutron/neutron/db/l3_hamode_db.py", line 394, in _create_ha_interfaces
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api l3_port_check=False)
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api self.force_reraise()
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api six.reraise(self.type_, self.value, self.tb)
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api File "/opt/stack/neutron/neutron/db/l3_hamode_db.py", line 388, in _create_ha_interfaces
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api router.tenant_id)
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api File "/opt/stack/neutron/neutron/db/l3_hamode_db.py", line 375, in add_ha_port
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api deletion, content)
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api File "/opt/stack/neutron/neutron/db/common_db_mixin.py", line 55, in safe_creation
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api obj = create_fn()
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api File "/opt/stack/neutron/neutron/plugins/common/utils.py", line 161, in create_port
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api return core_plugin.create_port(context, {'port': port_data})
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api File "/opt/stack/neutron/neutron/plugins/ml2/plugin.py", line 1123, in create_port
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api result, mech_context = self._create_port_db(context, port)
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api File "/opt/stack/neutron/neutron/plugins/ml2/plugin.py", line 1094, in _create_port_db
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api result = super(Ml2Plugin, self).create_port(context, port)
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api File "/opt/stack/neutron/neutron/db/db_base_plugin_v2.py", line 1237, in create_port
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api port_id)
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api File "/opt/stack/neutron/neutron/db/ipam_pluggable_backend.py", line 179, in allocate_ips_for_port_and_store
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api revert_on_fail=False)
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api self.force_reraise()
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api six.reraise(self.type_, self.value, self.tb)
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api File "/opt/stack/neutron/neutron/db/ipam_pluggable_backend.py", line 163, in allocate_ips_for_port_and_store
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api ips = self._allocate_ips_for_port(context, port_copy)
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api File "/opt/stack/neutron/neutron/db/ipam_pluggable_backend.py", line 235, in _allocate_ips_for_port
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api return self._ipam_allocate_ips(context, ipam_driver, p, ips)
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api File "/opt/stack/neutron/neutron/db/ipam_pluggable_backend.py", line 135, in _ipam_allocate_ips
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api "external system for %s"), addresses)
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api self.force_reraise()
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api six.reraise(self.type_, self.value, self.tb)
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api File "/opt/stack/neutron/neutron/db/ipam_pluggable_backend.py", line 120, in _ipam_allocate_ips
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api context, ipam_driver, port, ip_list)
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api File "/opt/stack/neutron/neutron/db/ipam_pluggable_backend.py", line 91, in _ipam_allocate_single_ip
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api port, subnet),
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api File "/opt/stack/neutron/neutron/db/ipam_pluggable_backend.py", line 80, in _ipam_try_allocate_ip
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api return ipam_subnet.allocate(ip_request)
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api File "/opt/stack/neutron/neutron/ipam/drivers/neutrondb_ipam/driver.py", line 350, in allocate
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api auto_generated)
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api File "/opt/stack/neutron/neutron/ipam/drivers/neutrondb_ipam/driver.py", line 219, in _allocate_specific_ip
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api session, db_range, first_ip=first_ip, last_ip=last_ip)
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api File "/opt/stack/neutron/neutron/ipam/drivers/neutrondb_ipam/db_api.py", line 167, in update_range
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api raise db_exc.RetryRequest(ipam_exc.IPAllocationFailed)
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api RetryRequest
2016-03-09 16:06:10.210 98260 ERROR oslo_db.api
2016-03-09 16:06:10.219 98260 ERROR neutron.api.v2.resource [req-d357eb93-048c-425b-b6de-f860be1d767a c_rally_3c9bcbbd_IA8UUZUc -] create failed
2016-03-09 16:06:10.219 98260 ERROR neutron.api.v2.resource Traceback (most recent call last):
2016-03-09 16:06:10.219 98260 ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/api/v2/resource.py", line 84, in resource
2016-03-09 16:06:10.219 98260 ERROR neutron.api.v2.resource result = method(request=request, **args)
2016-03-09 16:06:10.219 98260 ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/api/v2/base.py", line 410, in create
2016-03-09 16:06:10.219 98260 ERROR neutron.api.v2.resource return self._create(request, body, **kwargs)
2016-03-09 16:06:10.219 98260 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_db/api.py", line 148, in wrapper
2016-03-09 16:06:10.219 98260 ERROR neutron.api.v2.resource ectxt.value = e.inner_exc
2016-03-09 16:06:10.219 98260 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-03-09 16:06:10.219 98260 ERROR neutron.api.v2.resource self.force_reraise()
2016-03-09 16:06:10.219 98260 ERROR neutron.api.v2.resource File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-03-09 16:06:10.219 98260 ERROR neutron.api.v2.resource six.reraise(self.type_, self.value, self.tb)
2016-03-09 16:06:10.219 98260 ERROR neutron.api.v2.resource File "<string>", line 2, in reraise
2016-03-09 16:06:10.219 98260 ERROR neutron.api.v2.resource TypeError: exceptions must be old-style classes or derived from BaseException, not type
2016-03-09 16:06:10.219 98260 ERROR neutron.api.v2.resource

Tags:

Kevin Benton (kevinbenton) on 2016-03-10

Changed in neutron:
milestone:	none → mitaka-rc1
importance:	Undecided → High
assignee:	nobody → Kevin Benton (kevinbenton)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-03-10: Fix proposed to neutron (master)

Fix proposed to branch: master
Review: https://review.openstack.org/290922

Changed in neutron:
status:	New → In Progress

Ihar Hrachyshka (ihar-hrachyshka) on 2016-03-10

tags:

added: l3-ipam-dhcp

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-03-11: Fix merged to neutron (master)

Reviewed: https://review.openstack.org/290922
Committed: https://git.openstack.org/cgit/openstack/neutron/commit/?id=2dc9d67f6e0fbec8272dc1dfdb594d05378536d5
Submitter: Jenkins
Branch: master

commit 2dc9d67f6e0fbec8272dc1dfdb594d05378536d5
Author: Kevin Benton <email address hidden>
Date: Wed Mar 9 16:08:06 2016 -0800

Construct exceptions before passing to retryrequest

    The IP allocation failure constructions weren't being constructed
    before being passed to RetryRequest so they would fail to be reraised
    after the retries were exhausted.

This patch just constructs them and adjusts a few of the unit tests
to execute the full retry life-cycle.

Closes-Bug: #1555387
Change-Id: I1c1ea488886ee3f4790eaa53911ea5845f327334

Changed in neutron:
status:	In Progress → Fix Released

Revision history for this message

Thierry Carrez (ttx) wrote on 2016-03-16: Fix included in openstack/neutron 8.0.0.0rc1

This issue was fixed in the openstack/neutron 8.0.0.0rc1 release candidate.

Revision history for this message

Nate Johnston (nate-johnston) wrote on 2016-03-30:

This was listed in https://etherpad.openstack.org/p/stable-bug-candidates-from-master but I don't think this is a backport candidate. It looks like liberty doesn't do a retry at all - compare the liberty version (no retry) at https://github.com/openstack/neutron/blob/stable/liberty/neutron/ipam/drivers/neutrondb_ipam/driver.py#L194-L197 to the mitaka version (with retries) at https://github.com/openstack/neutron/blob/stable/mitaka/neutron/ipam/drivers/neutrondb_ipam/driver.py#L197-L202

So unless we want to backport the entire retry functionality (which seems like a functional change) I think this should be removed from https://etherpad.openstack.org/p/stable-bug-candidates-from-master

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.