Radosgw does not handle Transfer-Encoding: chunked

High due to customer-found

To fix this the following is required:
- use Apache2 2.4.9+ (I have tested with 2.4.17)
- enable mod_proxy and mod_proxy_fcgi, optionally disable mod_fastcgi since it is not needed anymore
- edit /etc/apache2/sites-available/rgw.conf:
--- remove/comment all fcgi stuff
--- disable s3gw RewriteRule, use generic:
RewriteRule .* - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization},L]
--- add ProxyPassMatch with syntax like below:
ProxyPassPatch "^.*$" "unix:/tmp/radosgw.sock|fcgi://localhost/var/www/radosgw/"

Side note - this may break S3 API (Swift API will work and chunked upload will work), so this is not a complete solution - RewriteRule needs to be customized somehow (or ProxyPassMatch parameters)

Customer found on 6.1 - need fix.

This bug https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1411030 has a patch. Can you include it in an upcoming update?

Through debugging the issue with Alexander Lemeshko, it has been observed
that the failed HTTP request isn't even delivered to radosgw instance - Apache
simply rejects it before passing through FastCGI.

radosgw does support chunked upload [1].

[1] https://github.com/ceph/ceph/blob/master/src/rgw/rgw_rest_swift.cc#L532

We don't deliver apache2 in MOS6.1+. It comes from upstream's repos for ubuntu.

Guys, did I understand correct, that you asked to have our own appach package?

@dipatov, @dlapshin, @dmeltsaykin: waiting for your answer

related bug: https://bugs.launchpad.net/glance/+bug/1518431
can works via mod_proxy_fcgi and radosgw as cgi server.

New apache2 package 2.4.10-9ubuntu1.1 from vivid has dependency on newer version of packages.
We can't update version apache2 without updating other packages.

proposed fix https://review.openstack.org/#/c/258521/

Fuel w/a https://review.openstack.org/#/c/258521

Reviewed: https://review.openstack.org/258521
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=e8567ee17b6f013d8d0f1460415523af084adcc7
Submitter: Jenkins
Branch: master

commit e8567ee17b6f013d8d0f1460415523af084adcc7
Author: Andrey Shestakov <email address hidden>
Date: Wed Dec 16 16:05:18 2015 +0200

    Change apache2 CGI module for RadosGW

    In apache-2.4.7 fastcgi module is not support chunked transfer.
    It breaks uploads of objects to RadosGW.
    Fix available since apache-2.4.10 (vivid) and upgrade apache in MOS is not
    possible, so this workaround proposed.
    RadosGW runs with native CGI server on 127.0.0.1:9000 and apache uses
    mod_proxy_fcgi to proxy user requests to rados.

    Closes-bug: #1521623
    Closes-bug: #1505273

    Change-Id: Ia3959c4009c6c673074a9b7536b1cc87834c301d

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/261353

Reviewed: https://review.openstack.org/261353
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=b092234605defaf6e41b15dc949472bd4b2d04ee
Submitter: Jenkins
Branch: stable/8.0

commit b092234605defaf6e41b15dc949472bd4b2d04ee
Author: Andrey Shestakov <email address hidden>
Date: Wed Dec 16 16:05:18 2015 +0200

    Change apache2 CGI module for RadosGW

    In apache-2.4.7 fastcgi module is not support chunked transfer.
    It breaks uploads of objects to RadosGW.
    Fix available since apache-2.4.10 (vivid) and upgrade apache in MOS is not
    possible, so this workaround proposed.
    RadosGW runs with native CGI server on 127.0.0.1:9000 and apache uses
    mod_proxy_fcgi to proxy user requests to rados.

    Closes-bug: #1521623
    Closes-bug: #1505273

    Change-Id: Ia3959c4009c6c673074a9b7536b1cc87834c301d
    (cherry picked from commit e8567ee17b6f013d8d0f1460415523af084adcc7)

Related fix proposed to branch: master
Review: https://review.openstack.org/265966

Reviewed: https://review.openstack.org/265966
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=a2d391a76082c7b00740b11a36e698006f185808
Submitter: Jenkins
Branch: master

commit a2d391a76082c7b00740b11a36e698006f185808
Author: Alex Schultz <email address hidden>
Date: Mon Jan 11 11:12:01 2016 -0700

    Add apache mod_proxy for ceph radosgw

    With the switch to fcgi for radosgw we need to include mod_proxy as
    mod_proxy_fcgi requires it. This change pulls in the proxy module to
    resolve apache2 startup failures when ceph is enabled.

    Change-Id: I7044b58066e70b11ed9358579a2f369ce761ee06
    Related-Bug: #1521623
    Related-Bug: #1505273
    Closes-Bug: #1532860

Related fix proposed to branch: stable/8.0
Review: https://review.openstack.org/266370

Reviewed: https://review.openstack.org/266370
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=ac654f2a5bac67ee8bd8330cb4b8443a84b3ea28
Submitter: Jenkins
Branch: stable/8.0

commit ac654f2a5bac67ee8bd8330cb4b8443a84b3ea28
Author: Alex Schultz <email address hidden>
Date: Mon Jan 11 11:12:01 2016 -0700

    Add apache mod_proxy for ceph radosgw

    With the switch to fcgi for radosgw we need to include mod_proxy as
    mod_proxy_fcgi requires it. This change pulls in the proxy module to
    resolve apache2 startup failures when ceph is enabled.

    Change-Id: I7044b58066e70b11ed9358579a2f369ce761ee06
    Related-Bug: #1521623
    Related-Bug: #1505273
    Closes-Bug: #1532860
    (cherry picked from commit a2d391a76082c7b00740b11a36e698006f185808)

The fixes for the master/9.0 have been committed and merged
https://review.openstack.org/#/c/258521/6
https://review.openstack.org/#/c/265966/

Due this, the status for master, MOS 9.0 was changed to fix_committed.

Verified as fixed in 8.0-570

[root@nailgun ~]# curl -s -d '{"auth": {"tenantName": "admin", "passwordCredentials": {"username": "admin", "password": "admin"}}}' -H 'Content-type: application/json' http://10.109.25.2:5000/v2.0/tokens
{"access": {"token": {"issued_at": "2016-02-19T12:21:41.798504", "expires": "2016-02-20T12:21:41Z", "id": "8beb2c1f6b2c4745ac78e2296416f179", "tenant": {"description": "", "enabled": true, "id": "13382b34fa8e4e7ab1f6695d80c2d595", "name": "admin"}, "audit_ids": ["7TDFCu8cQt2bLeQdXTNlWQ"]}, "serviceCatalog": [{"endpoints": [{"adminURL": "http://10.109.25.2:8000/ostf", "region": "RegionOne", "internalURL": "http://10.109.25.2:8000/ostf", "id": "dacfb57b4ec74aa1899d2e51213c78eb", "publicURL": "http://10.109.25.2:8000/ostf"}], "endpoints_links": [], "type": "ostf", "name": "ostf"}, {"endpoints": [{"adminURL": "http://10.109.25.2:8000/api", "region": "RegionOne", "internalURL": "http://10.109.25.2:8000/api", "id": "f68271098315479d8e2152a303bf426c", "publicURL": "http://10.109.25.2:8000/api"}], "endpoints_links": [], "type": "fuel", "name": "nailgun"}, {"endpoints": [{"adminURL": "http://10.109.25.2:35357/v2.0/v2.0", "region": "RegionOne", "internalURL": "http://10.109.25.2:5000/v2.0/v2.0", "id": "e43b006835be4ec588f1367cce2c7c14", "publicURL": "http://10.109.25.2:5000/v2.0/v2.0"}], "endpoints_links": [], "type": "identity", "name": "keystone"}], "user": {"username": "admin", "roles_links": [], "id": "7be419bf1d554f34a877f564ab132489", "roles": [{"name": "admin"}], "name": "admin"}, "metadata": {"is_admin": 0, "roles": ["b999d49c009d4e6493f3aa65a3968fbe"]}}}

[root@nailgun ~]# TOKEN=8beb2c1f6b2c4745ac78e2296416f179

[root@nailgun ~]# curl -k -i http://10.109.25.2:8080/swift/v1/test/test -X PUT -H "X-Auth-Token: $TOKEN" -H "Transfer-Encoding: chunked" -H "Content-Length: 5274" HTTP/1.1 405 Not Allowed
Server: nginx
Date: Fri, 19 Feb 2016 12:21:59 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 166
Connection: keep-alive

<html>
<head><title>405 Not Allowed</title></head>
<body bgcolor="white">
<center><h1>405 Not Allowed</h1></center>
<hr><center>nginx</center>
</body>
</html>

Release notes:

RadosGW now properly handles requests with chunked encoding. Previously it used to fail with "411 Length Required"

Fix proposed to branch: 6.1
Change author: Denis Puchkin <email address hidden>
Review: https://review.fuel-infra.org/19379

Verified on MOS 9.0 (build 248) - works!
Proof: http://pastebin.com/kYcYUV3q

Fix for 6.1 needs additional work - moved out of 6.1-mu-6 scope

We are not going to update ceph in 6.1 and 7.0 MUs - Won't Fix.