image download process won't terminate after deleting image from dashboard

I could reproduce this in a DevStack deployment on Ubuntu 12.04 LTS.

## Glance images folder

vkmc@havana:/opt/stack/data/glance/images$ ls -la
total 33052
drwxrwxr-x 2 vkmc vkmc 4096 Jul 3 12:22 .
drwxrwxr-x 4 vkmc vkmc 4096 Jul 3 12:09 ..
-rw-r----- 1 vkmc vkmc 0 Jul 3 12:22 1aa74a9a-67d0-4005-b78e-0936be8770aa <-
-rw-r----- 1 vkmc vkmc 4955792 Jul 3 12:11 2fb167b6-b0c4-4d78-8667-cefd077433a6
-rw-r----- 1 vkmc vkmc 25165824 Jul 3 12:12 b0b77bb0-36c5-465b-a102-9e282e908b8f
-rw-r----- 1 vkmc vkmc 3714968 Jul 3 12:11 be71ae20-15c5-4dd0-b1d8-44de8afe044d
vkmc@havana:/opt/stack/data/glance/images$ ls -la
total 42000
drwxrwxr-x 2 vkmc vkmc 4096 Jul 3 12:22 .
drwxrwxr-x 4 vkmc vkmc 4096 Jul 3 12:09 ..
-rw-r----- 1 vkmc vkmc 9159168 Jul 3 12:22 1aa74a9a-67d0-4005-b78e-0936be8770aa <-
-rw-r----- 1 vkmc vkmc 4955792 Jul 3 12:11 2fb167b6-b0c4-4d78-8667-cefd077433a6
-rw-r----- 1 vkmc vkmc 25165824 Jul 3 12:12 b0b77bb0-36c5-465b-a102-9e282e908b8f
-rw-r----- 1 vkmc vkmc 3714968 Jul 3 12:11 be71ae20-15c5-4dd0-b1d8-44de8afe044d

## image_locations table after bug reproduction

It's present in the Glance image_locations table

mysql> select * from image_locations;
+----+--------------------------------------+---------------------------------------------------------------------------+---------------------+---------------------+------------+---------+
| id | image_id | value | created_at | updated_at | deleted_at | deleted |
+----+--------------------------------------+---------------------------------------------------------------------------+---------------------+---------------------+------------+---------+
| 1 | 2fb167b6-b0c4-4d78-8667-cefd077433a6 | file:///opt/stack/data/glance/images/2fb167b6-b0c4-4d78-8667-cefd077433a6 | 2013-07-03 15:11:58 | 2013-07-03 15:11:58 | NULL | 0 |
| 2 | be71ae20-15c5-4dd0-b1d8-44de8afe044d | file:///opt/stack/data/glance/images/be71ae20-15c5-4dd0-b1d8-44de8afe044d | 2013-07-03 15:11:59 | 2013-07-03 15:11:59 | NULL | 0 |
| 3 | b0b77bb0-36c5-465b-a102-9e282e908b8f | file:///opt/stack/data/glance/images/b0b77bb0-36c5-465b-a102-9e282e908b8f | 2013-07-03 15:12:01 | 2013-07-03 15:12:01 | NULL | 0 |
| 4 | 1aa74a9a-67d0-4005-b78e-0936be8770aa | file:///opt/stack/data/glance/images/1aa74a9a-67d0-4005-b78e-0936be8770aa | 2013-07-03 15:22:39 | 2013-07-03 15:22:39 | NULL | 0 | <-
+----+--------------------------------------+---------------------------------------------------------------------------+---------------------+---------------------+------------+---------+
4 rows in set (0.00 sec)

## image-list after bug reproduction

But, it's not in the image-list

vkmc@havana:/opt/stack/data/glance/images$ glance image-list
+--------------------------------------+---------------------------------+-------------+------------------+----------+--------+
| ID | Name | Disk Format | Container Format | Size | Status |
+--------------------------------------+--------------------...

From the description I'm 95% certain this is an issue on the glance end of things. If I'm wrong, please re-open it for Horizon.

I can create this issue based on latest devstack. And after reviewed the database record, there is no image created yet. So seems after cancel the image creation, there is no good way to tell glance to remove the image data.

typo. I mean I can recreate it.

Fix proposed to branch: master
Review: https://review.openstack.org/121639

Glance side update:

(To be) abandoned: https://review.openstack.org/121639

Complete fix proposed: https://review.openstack.org/#/c/144464/

https://review.openstack.org/#/c/144464/ resolves the issue on Glance side. However, the download process may still be running from dashboard in some cases.

We've a race condition between when the download has started and when the upload operation raises a Duplicate exception.

One suggestion is to poll for Image status and delete the download process on the client (here Dashboard) side.

Change abandoned by Mike Fedosin (<email address hidden>) on branch: master
Review: https://review.openstack.org/121639

Can someone please "nominate" this fix for Juno and Icehouse ?

And the backport is not trivial, tests seems to require a tiny bit of rework, can someone also submit https://review.openstack.org/#/c/144464/ for stable/Juno and stable/Icehouse branch ?

Reviewed: https://review.openstack.org/144464
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=0dc8fbb3479a53c5bba8475d14f4c7206904c5ea
Submitter: Jenkins
Branch: master

commit 0dc8fbb3479a53c5bba8475d14f4c7206904c5ea
Author: Zhi Yan Liu <email address hidden>
Date: Tue Dec 30 22:25:50 2014 +0800

    Cleanup chunks for deleted image that was 'saving'

    Currently image data cannot be removed synchronously for an image that
    is in saving state. And when, the upload operation for such an image is
    completed the operator configured quota can be exceeded.

    This patch fixes the issue of left over chunks for an image which was
    deleted from saving status. However, by the limitation of the design we
    cannot enforce a global quota check for the image in saving status.

    This change introduces a inconsonance between http response codes of
    v1 and v2 APIs. The status codes which we will now see after the upload
    process completes on an image which was deleted mid way are:

    v1: 412 Precondition Failed
    v2: 410 Gone

    SecurityImpact
    UpgradeImpact
    APIImpact

    Closes-Bug: 1383973
    Closes-Bug: 1398830
    Closes-Bug: 1188532

    Change-Id: I47229b366c25367ec1bd48aec684e0880f3dfe60
    Signed-off-by: Zhi Yan Liu <email address hidden>

Tristan, this has been nominated. If someone doesn't beat me, I'll take a crack at backporting this.

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/149387

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/149397

Reviewed: https://review.openstack.org/149387
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=7d5d8657fd70b20518610b3c6f8e41e16c72fa31
Submitter: Jenkins
Branch: stable/juno

commit 7d5d8657fd70b20518610b3c6f8e41e16c72fa31
Author: Zhi Yan Liu <email address hidden>
Date: Tue Dec 30 22:25:50 2014 +0800

    Cleanup chunks for deleted image that was 'saving'

    Currently image data cannot be removed synchronously for an image that
    is in saving state. And when, the upload operation for such an image is
    completed the operator configured quota can be exceeded.

    This patch fixes the issue of left over chunks for an image which was
    deleted from saving status. However, by the limitation of the design we
    cannot enforce a global quota check for the image in saving status.

    This change introduces a inconsonance between http response codes of
    v1 and v2 APIs. The status codes which we will now see after the upload
    process completes on an image which was deleted mid way are:

    v1: 412 Precondition Failed
    v2: 410 Gone

    SecurityImpact
    UpgradeImpact
    APIImpact

    Closes-Bug: 1383973
    Closes-Bug: 1398830
    Closes-Bug: 1188532

    Conflicts:
     glance/api/v1/upload_utils.py
     glance/api/v2/image_data.py
     glance/tests/unit/test_domain_proxy.py
     glance/tests/unit/v1/test_api.py

    Change-Id: I47229b366c25367ec1bd48aec684e0880f3dfe60
    Signed-off-by: Zhi Yan Liu <email address hidden>
    (cherry picked from commit 0dc8fbb3479a53c5bba8475d14f4c7206904c5ea)

Fix proposed to branch: stable/icehouse
Review: https://review.openstack.org/149646

Change abandoned by nikhil komawar (<email address hidden>) on branch: stable/icehouse
Review: https://review.openstack.org/149397
Reason: https://review.openstack.org/#/c/149646

Reviewed: https://review.openstack.org/149646
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=f1260cc771ee068651aa62b972bef49d9af81eb0
Submitter: Jenkins
Branch: stable/icehouse

commit f1260cc771ee068651aa62b972bef49d9af81eb0
Author: Zhi Yan Liu <email address hidden>
Date: Tue Dec 30 22:25:50 2014 +0800

    Cleanup chunks for deleted image that was 'saving'

    Currently image data cannot be removed synchronously for an image that
    is in saving state. And when, the upload operation for such an image is
    completed the operator configured quota can be exceeded.
    This patch fixes the issue of left over chunks for an image which was
    deleted from saving status. However, by the limitation of the design we
    cannot enforce a global quota check for the image in saving status.
    This change introduces a inconsonance between http response codes of
    v1 and v2 APIs. The status codes which we will now see after the upload
    process completes on an image which was deleted mid way are:

    v1: 412 Precondition Failed
    v2: 410 Gone

    SecurityImpact
    UpgradeImpact
    APIImpact

    Closes-Bug: 1383973
    Closes-Bug: 1398830
    Closes-Bug: 1188532

    Conflicts:
            glance/api/v1/upload_utils.py
            glance/api/v2/image_data.py
            glance/quota/__init__.py
            glance/tests/unit/test_domain_proxy.py
            glance/tests/unit/test_quota.py
            glance/tests/unit/v1/test_api.py

    Signed-off-by: Zhi Yan Liu <email address hidden>
    (cherry picked from commit 0dc8fbb3479a53c5bba8475d14f4c7206904c5ea)

    Change-Id: I47229b366c25367ec1bd48aec684e0880f3dfe60