Glance

While deleting an image, Glance Swift store fails to delete remaining chunks upon erroring on one of the chunks

Bug #1256364 reported by Hemanth Makkapati
24
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Glance
Fix Released
Medium
Hemanth Makkapati
Glance 2014.2 "juno"

Bug Description

While attempting to delete an image, glance swift store:
1. checks if the image is stored as chunks.
2. if stored as chunks:
   2.a. fetch the list of chunks
   2.b. iterates over chunks and deletes each one
   2.c. deletes the manifest
3. if not stored as chunks, deletes the object

In step 2.b, when iterating over the list of chunks and deleting each one, if glance swift store receives an error on one of the chunks, an exception is raised and the delete process halts leaving the remaining chunks orphaned. This may cause orphaned chunks to be leftover in swift.

Hemanth Makkapati (hemanth-makkapati)
Changed in glance:
assignee: nobody → Hemanth Makkapati (hemanth-makkapati)
Iccha Sethi (iccha-sethi)
Changed in glance:
importance: Undecided → Medium
status: New → Triaged
Revision history for this message
Zhi Yan Liu (lzy-dev) wrote :

Hi Hemanth, do you still working on this? any update? thanks.

Revision history for this message
Hemanth Makkapati (hemanth-makkapati) wrote :

I'll be fixing this tomorrow as a part of glance bug day II.
Thanks for following up though :)

Revision history for this message
Erno Kuvaja (jokke) wrote :

This bug seems to be duplicate of subset of https://bugs.launchpad.net/glance/+bug/1177564 . Hemanth, could you flag that bug as well if you get this fixed.

Revision history for this message
Jin Liu (liujinbj) wrote :

Hi Hemanth, when do you plan to raise fixing proposal? I wanna know if this fix will cover all store back-ends case, thanks.

Revision history for this message
Hemanth Makkapati (hemanth-makkapati) wrote :

Erno, sure, I'll flag the other one as well.

Revision history for this message
Hemanth Makkapati (hemanth-makkapati) wrote :

Jin, I should have probably mentioned it explicitly, I was specifically talking about swift store. I'm not sure if this bug exists in the stores as well to begin with. But, you bring up a good point, it's worth checking other stores as well.

Btw, I have a fix ready, just writing tests. Expect to see a review up early next week.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to glance (master)

Fix proposed to branch: master
Review: https://review.openstack.org/115128

Changed in glance:
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/115332

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on glance (master)

Change abandoned by Hemanth Makkapati (<email address hidden>) on branch: master
Review: https://review.openstack.org/115128

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to glance_store (master)

Fix proposed to branch: master
Review: https://review.openstack.org/123584

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on glance (master)

Change abandoned by Hemanth Makkapati (<email address hidden>) on branch: master
Review: https://review.openstack.org/115332
Reason: With glance_store refactoring, this is not valid anymore. Glance store review here https://review.openstack.org/#/c/123584/

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to glance_store (master)

Reviewed: https://review.openstack.org/123584
Committed: https://git.openstack.org/cgit/openstack/glance_store/commit/?id=8f99cce9b12a419d4627a4d15624b9b9df4cf099
Submitter: Jenkins
Branch: master

commit 8f99cce9b12a419d4627a4d15624b9b9df4cf099
Author: Hemanth Makkapati <email address hidden>
Date: Tue Sep 23 16:43:16 2014 -0400

    Recover from errors while deleting image segments

    When Glance attempts to delete an image, which is stored as a
    segmented object, if an error occurs while deleting one of the
    segments, delete operation fails while leaving behind rest of
    the segments.
    This patch enables Glance to recover from one or more errors during
    segment deletion and proceed with deleting other segments. However,
    this patch doesn't do anything with the failed segments themselves.
    With this patch it'd be possible that even after an image is
    deleted, certain chunks are left behind in the store.

    Change-Id: Ib3df99b13abb6aa1f579c1dce9e2b64d6e72da91
    Closes-Bug: 1256364

Changed in glance:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in glance:
milestone: none → juno-rc1
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in glance:
milestone: juno-rc1 → 2014.2
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.