OpenStack Compute (nova)

Error during scheduler Manager expire reservations

Bug #1040942 reported by Naveen Joy on 2012-08-24

This bug affects 2 people

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Compute (nova)	Fix Released	High	Mark McLoughlin	OpenStack Compute (nova) 2012.2 "folsom"

Bug Description

I am periodically seeing this error in my nova.log file. Not sure what's causing this.

2012-08-23 17:12:55 INFO nova.compute.manager [-] Updating host status
2012-08-23 17:13:00 ERROR nova.manager [-] Error during SchedulerManager._expire_reservations: 'Reservation' object has no attribute 'usage'
2012-08-23 17:13:00 TRACE nova.manager Traceback (most recent call last):
2012-08-23 17:13:00 TRACE nova.manager File "/usr/local/lib/python2.7/dist-packages/nova-2012.2-py2.7.egg/nova/manager.py", line 173, in periodic_tasks
2012-08-23 17:13:00 TRACE nova.manager task(self, context)
2012-08-23 17:13:00 TRACE nova.manager File "/usr/local/lib/python2.7/dist-packages/nova-2012.2-py2.7.egg/nova/scheduler/manager.py", line 288, in _expire_reservations
2012-08-23 17:13:00 TRACE nova.manager QUOTAS.expire(context)
2012-08-23 17:13:00 TRACE nova.manager File "/usr/local/lib/python2.7/dist-packages/nova-2012.2-py2.7.egg/nova/quota.py", line 757, in expire
2012-08-23 17:13:00 TRACE nova.manager self._driver.expire(context)
2012-08-23 17:13:00 TRACE nova.manager File "/usr/local/lib/python2.7/dist-packages/nova-2012.2-py2.7.egg/nova/quota.py", line 380, in expire
2012-08-23 17:13:00 TRACE nova.manager db.reservation_expire(context)
2012-08-23 17:13:00 TRACE nova.manager File "/usr/local/lib/python2.7/dist-packages/nova-2012.2-py2.7.egg/nova/db/api.py", line 1066, in reservation_expire
2012-08-23 17:13:00 TRACE nova.manager return IMPL.reservation_expire(context)
2012-08-23 17:13:00 TRACE nova.manager File "/usr/local/lib/python2.7/dist-packages/nova-2012.2-py2.7.egg/nova/db/sqlalchemy/api.py", line 112, in wrapper
2012-08-23 17:13:00 TRACE nova.manager return f(*args, **kwargs)
2012-08-23 17:13:00 TRACE nova.manager File "/usr/local/lib/python2.7/dist-packages/nova-2012.2-py2.7.egg/nova/db/sqlalchemy/api.py", line 2864, in reservation_expire
2012-08-23 17:13:00 TRACE nova.manager reservation.usage.reserved -= reservation.delta
2012-08-23 17:13:00 TRACE nova.manager AttributeError: 'Reservation' object has no attribute 'usage'
2012-08-23 17:13:00 TRACE nova.manager

Rongze Zhu (zrzhit) on 2012-08-24

Changed in nova:
assignee:	nobody → Rongze Zhu (zrzhit)

Rongze Zhu (zrzhit) on 2012-08-24

Changed in nova:
assignee:	Rongze Zhu (zrzhit) → nobody

Revision history for this message

Mark McLoughlin (markmc) wrote on 2012-08-30:

Thanks for the report!

I've just reproduced this by:

1) Set reservation_expire=10 and periodic_interval=20 in nova.conf

2) Do 'nova secgroup-create foo bar' twice (see #1043765)

3) Confirm that we have a lingering reservation from that failure by running 'select * from reservations where deleted=0;' against the nova DB

Wait a while and, sure enough, I see the same exception you do.

By the way, whatever caused the quota reservation to leak is a bug too. It would be good to track that bug down. A good start is to look at the 'resource' column of the leaked reservations.

Changed in nova:
status:	New → Confirmed
importance:	Undecided → High
assignee:	nobody → Mark McLoughlin (markmc)
milestone:	none → folsom-rc1

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2012-08-31: Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/12263

Changed in nova:
status:	Confirmed → In Progress

OpenStack Infra (hudson-openstack) on 2012-08-31

Changed in nova:
assignee:	Mark McLoughlin (markmc) → Vish Ishaya (vishvananda)

OpenStack Infra (hudson-openstack) on 2012-09-01

Changed in nova:
assignee:	Vish Ishaya (vishvananda) → Mark McLoughlin (markmc)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2012-09-03: Fix merged to nova (master)

Reviewed: https://review.openstack.org/12263
Committed: http://github.com/openstack/nova/commit/22f0e324f3d3172b563aa67e513fe4d9318de2e5
Submitter: Jenkins
Branch: master

commit 22f0e324f3d3172b563aa67e513fe4d9318de2e5
Author: Mark McLoughlin <email address hidden>
Date: Sat Sep 1 09:37:15 2012 +0100

Fix quota reservation expiration

Fixes bug #1040942

The db.reservation_expire() function assumes a 'usage' attribute on
Reservation objects, but we don't actually define that relationship.

    The end result is that reservation_expire() currently traceback if
    it actually needs to expire any reservations. This happens pretty
    rarely since reservations should only need expiring if they are
    leaked because of another bug.

Also define a test case to actually excercise the expiration code
path.

    Add a missing chain-up to tearDown in test_limits which was causing
    the get_project_quotas() stub not to be unset and, in turn, the
    reservation expiration test to fail.

Change-Id: Ib61dbf9fd5dfb5badaf05f20c423a69925d83754

Changed in nova:
status:	In Progress → Fix Committed

Thierry Carrez (ttx) on 2012-09-19

Changed in nova:
status:	Fix Committed → Fix Released

Thierry Carrez (ttx) on 2012-09-27

Changed in nova:
milestone:	folsom-rc1 → 2012.2

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.