rbd delete_image does not catch ImageNotFound when deleting snap

Bug #1233097 reported by Edward Hope-Morley
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Glance
Fix Released
High
Edward Hope-Morley

Bug Description

the store.rbd._delete_image() method does not catch rbd.ImageNotFound and return exception.NotFound when trying to delete a snapshot. The behaviour should be the same as when deleting the image itself.

This produces errors like this:

2013-09-30 14:30:10.139 442 ERROR glance.api.v2.image_data [0be083f3-4d9f-4d88-9264-b69ca9bbd2c2 18ab8e3ef26b499a8c581b8f18f2d33f 133f3a73a0294e98aaee00e8139bb922] Failed to upload image data due to internal error
2013-09-30 14:30:10.139 442 TRACE glance.api.v2.image_data Traceback (most recent call last):
2013-09-30 14:30:10.139 442 TRACE glance.api.v2.image_data File "/mnt/glance/glance/api/v2/image_data.py", line 55, in upload
2013-09-30 14:30:10.139 442 TRACE glance.api.v2.image_data image.set_data(data, size)
2013-09-30 14:30:10.139 442 TRACE glance.api.v2.image_data File "/mnt/glance/glance/domain/proxy.py", line 126, in set_data
2013-09-30 14:30:10.139 442 TRACE glance.api.v2.image_data self.base.set_data(data, size)
2013-09-30 14:30:10.139 442 TRACE glance.api.v2.image_data File "/mnt/glance/glance/notifier/__init__.py", line 202, in set_data
2013-09-30 14:30:10.139 442 TRACE glance.api.v2.image_data self.image.set_data(data, size)
2013-09-30 14:30:10.139 442 TRACE glance.api.v2.image_data File "/mnt/glance/glance/domain/proxy.py", line 126, in set_data
2013-09-30 14:30:10.139 442 TRACE glance.api.v2.image_data self.base.set_data(data, size)
2013-09-30 14:30:10.139 442 TRACE glance.api.v2.image_data File "/mnt/glance/glance/quota/__init__.py", line 140, in set_data
2013-09-30 14:30:10.139 442 TRACE glance.api.v2.image_data self.image.set_data(data, size=size)
2013-09-30 14:30:10.139 442 TRACE glance.api.v2.image_data File "/mnt/glance/glance/store/__init__.py", line 644, in set_data
2013-09-30 14:30:10.139 442 TRACE glance.api.v2.image_data self.image.image_id, utils.CooperativeReader(data), size)
2013-09-30 14:30:10.139 442 TRACE glance.api.v2.image_data File "/mnt/glance/glance/store/__init__.py", line 355, in add_to_backend
2013-09-30 14:30:10.139 442 TRACE glance.api.v2.image_data return store_add_to_backend(image_id, data, size, store)
2013-09-30 14:30:10.139 442 TRACE glance.api.v2.image_data File "/mnt/glance/glance/store/__init__.py", line 333, in store_add_to_backend
2013-09-30 14:30:10.139 442 TRACE glance.api.v2.image_data (location, size, checksum, metadata) = store.add(image_id, data, size)
2013-09-30 14:30:10.139 442 TRACE glance.api.v2.image_data File "/mnt/glance/glance/store/rbd.py", line 330, in add
2013-09-30 14:30:10.139 442 TRACE glance.api.v2.image_data self._delete_image(loc.image, loc.snapshot)
2013-09-30 14:30:10.139 442 TRACE glance.api.v2.image_data File "/mnt/glance/glance/store/rbd.py", line 267, in _delete_image
2013-09-30 14:30:10.139 442 TRACE glance.api.v2.image_data image.unprotect_snap(snapshot_name)
2013-09-30 14:30:10.139 442 TRACE glance.api.v2.image_data File "/usr/lib/python2.7/dist-packages/rbd.py", line 578, in unprotect_snap
2013-09-30 14:30:10.139 442 TRACE glance.api.v2.image_data raise make_ex(ret, 'error unprotecting snapshot %s@%s' % (self.name, name))
2013-09-30 14:30:10.139 442 TRACE glance.api.v2.image_data ImageNotFound: error unprotecting snapshot bccb3478-e7cf-4661-86ba-d405cffe0912@snap

where we are doing a cleanup but the image may not exist in which case we would want to ignore the error.

Changed in glance:
assignee: nobody → Edward Hope-Morley (hopem)
status: New → In Progress
Revision history for this message
Edward Hope-Morley (hopem) wrote :

this bug appears to have been introduced by https://review.openstack.org/#/c/42896/

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to glance (master)

Fix proposed to branch: master
Review: https://review.openstack.org/48943

description: updated
Changed in glance:
milestone: none → icehouse-1
importance: Undecided → High
tags: added: havana-rc-potential
Revision history for this message
Edward Hope-Morley (hopem) wrote :

I am now hitting this issue a lot for example when uploads (e.g. from cinder) fail and the underlying image is not deleted in my backend store. Currently the only way to delete these images is not manually remove them from the backend store.

Thierry Carrez (ttx)
Changed in glance:
milestone: icehouse-1 → havana-rc2
tags: removed: havana-rc-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to glance (master)

Reviewed: https://review.openstack.org/48943
Committed: http://github.com/openstack/glance/commit/6d5fd4836475d0f3787b669cab5ef2d5266f474f
Submitter: Jenkins
Branch: master

commit 6d5fd4836475d0f3787b669cab5ef2d5266f474f
Author: Edward Hope-Morley <email address hidden>
Date: Mon Sep 30 12:22:30 2013 +0100

    Fixes rbd _delete_image snapshot with missing image

    Also ignore if _delete_image returns NotFound when cleaning up
    following failed attempt to create a new image with add().

    Also added unit tests.

    Change-Id: Id66866b4260385a6324cc277c5ac665f81493c89
    Fixes: bug 1233097

Changed in glance:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to glance (milestone-proposed)

Fix proposed to branch: milestone-proposed
Review: https://review.openstack.org/51045

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to glance (milestone-proposed)

Reviewed: https://review.openstack.org/51045
Committed: http://github.com/openstack/glance/commit/1fd64662fe66eda5fee404a37586f30cae2bf785
Submitter: Jenkins
Branch: milestone-proposed

commit 1fd64662fe66eda5fee404a37586f30cae2bf785
Author: Edward Hope-Morley <email address hidden>
Date: Mon Sep 30 12:22:30 2013 +0100

    Fixes rbd _delete_image snapshot with missing image

    Also ignore if _delete_image returns NotFound when cleaning up
    following failed attempt to create a new image with add().

    Also added unit tests.

    Change-Id: Id66866b4260385a6324cc277c5ac665f81493c89
    Fixes: bug 1233097

Changed in glance:
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in glance:
milestone: havana-rc2 → 2013.2
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.