Glance fail to upload images to radosgw with Ceph Nautilus

Bug #1824533 reported by Alfredo Moralejo
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
glance_store
Fix Released
Undecided
Unassigned
Queens
New
Undecided
Unassigned
Rocky
New
Undecided
Unassigned
Stein
Fix Released
Undecided
Unassigned

Bug Description

Configuring Ceph RadosGW Nautilus as swift backend for glance, it fails to upload images with following error:

# openstack image create --file hosts test-nuevo
HTTP 500 Internal Server Error: The server has either erred or is incapable of performing the requested operation.

Logs in glance api:

2019-04-12 13:15:25.477 789960 INFO swiftclient [req-ccd5ce54-5af5-47df-898b-2049a8b89951 be945c20abf24e5783216b35408eb33d f5a52119c6b340c88652afb7d1a798ad - default default] REQ: curl -i http://[::1]:8080/swift
/v1/glance/1aea2157-a2be-4337-be03-3faaa458018b -X PUT -H "Content-Length: 0" -H "ETag: d41d8cd98f00b204e9800998ecf8427e" -H "X-Object-Manifest: glance/1aea2157-a2be-4337-be03-3faaa458018b-" -H "X-Auth-Token: gA
AAAABcsI9qJZ_N..."
2019-04-12 13:15:25.477 789960 INFO swiftclient [req-ccd5ce54-5af5-47df-898b-2049a8b89951 be945c20abf24e5783216b35408eb33d f5a52119c6b340c88652afb7d1a798ad - default default] RESP STATUS: 422 Unprocessable Entit
y
2019-04-12 13:15:25.478 789960 INFO swiftclient [req-ccd5ce54-5af5-47df-898b-2049a8b89951 be945c20abf24e5783216b35408eb33d f5a52119c6b340c88652afb7d1a798ad - default default] RESP HEADERS: {u'Content-Length': u'
19', u'Accept-Ranges': u'bytes', u'Last-Modified': u'Thu, 01 Jan 1970 00:00:00 GMT', u'etag': u'b0e641c998cc3eae6fa2f8726d98cddd', u'X-Trans-Id': u'tx000000000000000000009-005cb08f6c-b491-default', u'Date': u'Fr
i, 12 Apr 2019 13:15:25 GMT', u'Content-Type': u'text/plain; charset=utf-8', u'X-Openstack-Request-Id': u'tx000000000000000000009-005cb08f6c-b491-default'}
2019-04-12 13:15:25.478 789960 INFO swiftclient [req-ccd5ce54-5af5-47df-898b-2049a8b89951 be945c20abf24e5783216b35408eb33d f5a52119c6b340c88652afb7d1a798ad - default default] RESP BODY: UnprocessableEntity
2019-04-12 13:15:25.478 789960 ERROR glance_store._drivers.swift.store [req-ccd5ce54-5af5-47df-898b-2049a8b89951 be945c20abf24e5783216b35408eb33d f5a52119c6b340c88652afb7d1a798ad - default default] Failed to add
 object to Swift.
Got error from Swift: Object PUT failed: http://::1:8080/swift/v1/glance/1aea2157-a2be-4337-be03-3faaa458018b 422 Unprocessable Entity UnprocessableEntity.: ClientException: Object PUT failed: http://::1:8080/
swift/v1/glance/1aea2157-a2be-4337-be03-3faaa458018b 422 Unprocessable Entity UnprocessableEntity
2019-04-12 13:15:25.481 789960 ERROR glance.api.v2.image_data [req-ccd5ce54-5af5-47df-898b-2049a8b89951 be945c20abf24e5783216b35408eb33d f5a52119c6b340c88652afb7d1a798ad - default default] Failed to upload image
 data due to internal error: BackendException: Failed to add object to Swift.
2019-04-12 13:15:25.509 789960 ERROR glance.common.wsgi [req-ccd5ce54-5af5-47df-898b-2049a8b89951 be945c20abf24e5783216b35408eb33d f5a52119c6b340c88652afb7d1a798ad - default default] Caught error: Failed to add
object to Swift.
Got error from Swift: Object PUT failed: http://::1:8080/swift/v1/glance/1aea2157-a2be-4337-be03-3faaa458018b 422 Unprocessable Entity UnprocessableEntity.: BackendException: Failed to add object to Swift.
Got error from Swift: Object PUT failed: http://::1:8080/swift/v1/glance/1aea2157-a2be-4337-be03-3faaa458018b 422 Unprocessable Entity UnprocessableEntity.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to glance_store (master)

Reviewed: https://review.openstack.org/651337
Committed: https://git.openstack.org/cgit/openstack/glance_store/commit/?id=ed356fc6b46e133899e1bf1c54df239536ea78b9
Submitter: Zuul
Branch: master

commit ed356fc6b46e133899e1bf1c54df239536ea78b9
Author: Alfredo Moralejo <email address hidden>
Date: Tue Apr 9 19:27:29 2019 +0200

    Do not include ETag when puting manifest in chunked uploads

    While testing glance with Ceph Rados Gateway using latest Ceph release
    (Nautilus), i've found that glance fails to upload the manifest using
    dynamic large objects mode because of the value used in ETag request.

    This issue has been reported to Ceph as it seems related to some recent
    change in radosgw code [1].

    However, checking at the upload workflow used by glance and comparing
    to Swift documentation [2], I wonder if adding the etag is actually
    providing any value. In the Swift the ETag header is used to validate
    integrity when uploading chunks, not the manifest while glance is doing
    exactly the oposite, not sending the etag in the chunks (I guess to
    avoid checksuming big images, which makes sense to me) and sending it
    when puting the manifest.

    This patch is removing the etag header when sending the PUT request for
    the manifest in chunked uploads.

    [1] https://tracker.ceph.com/issues/39160
    [2] https://docs.openstack.org/swift/latest/api/large_objects.html#dynamic-large-objects

    Closes-bug: #1824533
    Change-Id: I0b563dfcdc30026669fb089c82db8c3df7edc808

Changed in glance-store:
status: New → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to glance_store (stable/stein)

Fix proposed to branch: stable/stein
Review: https://review.openstack.org/653486

Revision history for this message
Donny Davis (donny-g) wrote :

I can confirm the proposed patch works with Stein/Nautilus RGW in my deployment

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to glance_store (stable/stein)

Reviewed: https://review.opendev.org/653486
Committed: https://git.openstack.org/cgit/openstack/glance_store/commit/?id=053614be148a5b085b186c82ba630bf6ce2aa067
Submitter: Zuul
Branch: stable/stein

commit 053614be148a5b085b186c82ba630bf6ce2aa067
Author: Alfredo Moralejo <email address hidden>
Date: Tue Apr 9 19:27:29 2019 +0200

    Do not include ETag when puting manifest in chunked uploads

    While testing glance with Ceph Rados Gateway using latest Ceph release
    (Nautilus), i've found that glance fails to upload the manifest using
    dynamic large objects mode because of the value used in ETag request.

    This issue has been reported to Ceph as it seems related to some recent
    change in radosgw code [1].

    However, checking at the upload workflow used by glance and comparing
    to Swift documentation [2], I wonder if adding the etag is actually
    providing any value. In the Swift the ETag header is used to validate
    integrity when uploading chunks, not the manifest while glance is doing
    exactly the oposite, not sending the etag in the chunks (I guess to
    avoid checksuming big images, which makes sense to me) and sending it
    when puting the manifest.

    This patch is removing the etag header when sending the PUT request for
    the manifest in chunked uploads.

    [1] https://tracker.ceph.com/issues/39160
    [2] https://docs.openstack.org/swift/latest/api/large_objects.html#dynamic-large-objects

    Closes-bug: #1824533
    Change-Id: I0b563dfcdc30026669fb089c82db8c3df7edc808
    (cherry picked from commit ed356fc6b46e133899e1bf1c54df239536ea78b9)

tags: added: in-stable-stein
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/glance_store 0.29.0

This issue was fixed in the openstack/glance_store 0.29.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.