Glance failed to upload image to swift storage

Please share your glance-api.conf file...

my glance-api.conf:
http://paste.openstack.org/show/479697/

The swift driver is meant to be used to talk to Swift. If there's a problem in RadosGW swift API, we should notify the RadosGW maintainers.

Looks like problem in Glance API v2, with v1 works as expected.

I've debugged this issue a bit further and it's not related to RadosGW. The swift driver deletes that last 0-byte chunk when the image is being uploaded and that's causing this conflict.

---
                    if bytes_read == 0:
                        # Delete the last chunk, because it's of zero size.
                        # This will happen if size == 0.
                        LOG.debug("Deleting final zero-length chunk")
                        connection.delete_object(location.container,
                                                 chunk_name)
                        break

                    chunk_id += 1
                    combined_chunks_size += bytes_read
---

The above was added here: https://review.openstack.org/#/c/2728/

I think Flavio is right here - RadosGW claimed to be 100% compatible with Swift API, but here we see inconsistencies.

As a workaround we can use glance v1, which works fine and used by default in Nova in Liberty. But it must be fixed in Mitaka, because we can break many deployments after that.

One of the things triggering this misbehavior is that glanceclient is not sending the request's content-lenght and therefore, Glance doesn't know the image/request size when the call to the store is made.

Unfortunately, there's a bit more to this issue since this doesn't happen deterministically. There seems to be a race in the swift driver that's causing the first conflict error, which then makes the subsequent requests fail.

For some reason this does not show up, but I submitted https://review.openstack.org/#/c/254873/ to fix the issue.

It took some amount of time to debug this issue for Glance.
It turned out that both Glance and Swift support chunked requests. Unfortunately, chunked uploading is not supported mod_fastcgi that is used by RadosGW. It is very typical for Apache to response with 411 error because some cgi (or wsgi) frameworks always require Content-length to be specified.
Request with transfer-encoding=chunked is a part of Http spec so glance_store prepares correct request here.
So I would recommend to deploy RadosGW under different CGI (for example mod_proxy_fcgi) that supports chunked requests.
I will mark this as Invalid for Glance, please re-open the bug if you don't agree.

Kairat,

I reopened the bug because its reproducible with Swift not just with RadosGW. Please, review a previous comments, especially #5, #8 and solution proposed in comment #9.

> As a workaround we can use glance v1, which works fine and used by default in Nova in Liberty.

Is this still the case? Or can we reproduce this on v1 also?

> One of the things triggering this misbehavior is that glanceclient is not sending the request's content-lenght and therefore, Glance doesn't know the image/request size when the call to the store is made.

Not sending the image size should be common behaviour. (I think it may be the case with Nova snapshots, but I'd need to double check.) If that was problematic I'd have thought this issue would have been flushed out by now.

What are the steps to reproduce?

Thanks.

Stuart, Kairat,

This bug is hard to reproduce, TBH. The race happens in a combination of Glance's v2 behavior and swift's driver. This is not *entirely* Glance's fault, though. The Glance "weird" part is not being able to recognize "0-sized" chunks before submitting them. However, this is being refactored a bit by one of our current swift specs (need to find the link).

The swift issue is that the DELETE is hitting the swift node before the last chunk is written/available. This could be caused by unsynchronized clocks or just a race condition.

I've managed to reproduce it in one of our installers CI systems with some frequency but I have yet to find a deterministic way to do so. Meanwhile, I can confirm this issue exists.

While it's true that part of this code is being refactored by the "buffered chunk writer" spec, I still think we should fix it as it's eligible for backport.

Hope the above helps clarifying a bit the real problem

I can confirm that image upload via v2 to radosgw works with mod_proxy_fcgi.

@Flavio

Thanks for the extra info.

> The race happens in a combination of Glance's v2 behavior and swift's driver.

Ok, can someone smarter than me explain why we don't see this with v1? :-)

(I updated the code review with an alternative fix.)

This issue was fixed in the openstack/glance_store 0.11.0 release.

This issue was fixed in the openstack/glance_store 0.9.2 release.

This issue was fixed in the openstack/glance_store 0.9.2 release.