v2 upload returns '500' when quota exceeded on local filesystem storage

Bug #1265446 reported by David Koo
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Glance
Fix Released
Medium
David Koo

Bug Description

Steps to reproduce:
1) Configure glance to use local filesystem backend, make sure the quota is explicitly configured.

2) glance --os-auth-token "$(cat ~/user-token)" \
    --os-image-url http://127.0.0.1:9292 \
    --os-image-api-version 2 \
    image-create --container-format bare \
        --disk-format qcow2 --name localfs-image

3) Upload image data. Make sure the image data size > quota

 glance --os-auth-token "$(cat ~/user-token)" \
    --os-image-url http://127.0.0.1:9292 \
    --os-image-api-version 2 \
    image-upload --file $HOME/testImg.qcow2 $img_id

Request returned failure status.
HTTPInternalServerError (HTTP 500)

Same thing happens if you use curl:

1) Configure glance to use local filesystem backend

2) glance --os-auth-token "$(cat ~/user-token)" \
    --os-image-url http://127.0.0.1:9292 \
    --os-image-api-version 2 \
    image-create --container-format bare \
        --disk-format qcow2 --name localfs-image

3) curl -i -X PUT -H 'Transfer-Encoding: chunked' \
    -H 'X-Auth-Token: $(cat ~/user-token)' \
    -H 'Content-Type: application/octet-stream' \
    -H 'User-Agent: python-glanceclient' \
    --data-binary @$HOME/testImg.qcow2 http://127.0.0.1:9292/v2/images/$img_id/file

HTTP/1.1 100 Continue

HTTP/1.1 500 Internal Server Error
Content-Type: text/plain
Content-Length: 0
Date: Thu, 02 Jan 2014 07:27:26 GMT
Connection: close

David Koo (kpublicmail)
Changed in glance:
assignee: nobody → David Koo (kpublicmail)
status: New → In Progress
description: updated
description: updated
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to glance (master)

Fix proposed to branch: master
Review: https://review.openstack.org/64840

Revision history for this message
David Koo (kpublicmail) wrote :

For the record: this happens because the store.safe_delete_from_backend() is called with an incorrect parameter sequence by
function the quota.ImageProxy.set_data() (the 'location' and 'context' parameters are switched).

The interesting thing is the way this bug is triggered ...

When the image data file is uploaded using curl or the glance client the data is transferred using chunked transfer. As a result the 'Content-Length' header is not set. As a result of that quota.ImageProxy.set_data() cannot directly check whether or not the file has exceeded quota and so lets the filesystem backend first download the entire file and then check the file size to see if the quota has been exceeded.

Since the quota has been exceeded it tries to delete the downloaded file by calling store.safe_delete_from_backend() where this bug is hit.

The quota tests (in tests/functional/v2/test_images.py) don't trigger this bug because they don't use chunked transfer. So the 'requests' module always calculates the right Content-Length and so quota.ImageProxy.set_data() is always able to perform the quota check without downloading the image data. This bug is triggered only when using chunked transfer!

Feilong Wang (flwang)
Changed in glance:
importance: Undecided → Medium
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to glance (master)

Reviewed: https://review.openstack.org/64840
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=d37907af5b71024024fb70d9454e4b3e05875105
Submitter: Jenkins
Branch: master

commit d37907af5b71024024fb70d9454e4b3e05875105
Author: David Koo <email address hidden>
Date: Fri Jan 3 22:51:28 2014 +0800

    Fix call to store.safe_delete_from_backend

    When uploading data of a newly created image to a local filesystem
    store, if the image data size exceeds the quota allocated to the user
    then the operation fails with a "500 Internal Server Error".

    This bug is triggered only when using chunked transfers to upload the
    image data (curl and glanceclient both use chunked transfers).

    When using chunked transfers the 'Content-Length' header is not set and
    so the quota checking code in start of quota.ImageProxy.set_data() is
    skipped. Furthermore, because of the missing size info, the store
    downloads the entire image data without checking for the quota (the
    quota is checked only after the image data has been downloaded and the
    size determined).

    If after the image data has been downloaded it is found that the quota
    has been exceeded, then the StorageQuotaFull exception is raised and a
    call to store.safe_delete_from_backend() made to clean up the image
    data. But the 'location' and 'context' parameters to this call are given
    in the wrong order and so this operation raises an (uncaught) exception,
    giving rise to this bug.

    This bug was not caught by existing tests because existing quota tests
    don't use chunked transfer.

    This commit fixes the order of the parameters and adds quota tests using
    chunked transfers.

    Change-Id: Ib8ab11f2115e3aead98af70788eaa45c2c33219c
    Closes-bug: #1265446

Changed in glance:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in glance:
milestone: none → icehouse-2
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in glance:
milestone: icehouse-2 → 2014.1
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.