OpenStack Object Storage (swift)

User Operations Fail in case if Keystone's endpoint list is larger than 8 KB

Bug #1657390 reported by Sanchit Malhotra
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Object Storage (swift)
Confirmed
Low
Unassigned

Bug Description

If Swift Server is associated with Keystone Server which has endpoints of all openstack key services distributed across various regions, then while authenticating a user requests, the whole list is appended in request header.

For example:
'X-Service-Catalog': '[{"endpoints": [{"adminURL": "http://192.168.136.199:35357/", "region": "RegionOne", "internalURL": "http://192.168.136.199:35357/", "publicURL": "http://192.168.136.199:5000"}], "type": "identity", "name": "keystone"}, {"endpoints": [{"adminURL": "http://192.168.136.198:8777", "region": "regionA"}, {"adminURL": "http://192.168.136.198:8777", "region": "regionB"}, {"adminURL": "http://192.168.136.198:8777", "region": "RegionC"}, {"adminURL": "http://192.168.136.198:8777", "region": "RegionD"}, {"adminURL": "http://192.168.136.198:8777", "region": "RegionE"}, {"adminURL": "http://192.168.136.198:8777", "region": "RegionF"}, {"adminURL": "http://192.168.136.198:8777", ..............

If suppose the size of list of endpoints is larger than 8 KB, then the request would fail at WSGI as it accepts header Line of a max size of 8 KB.

File: eventlet/wsgi.py
{code}
MAX_HEADER_LINE = 8192

class FileObjectForHeaders(object):

    def __init__(self, fp):
        self.fp = fp
        self.total_header_size = 0

    def readline(self, size=-1):
        sz = size
        if size < 0:
            sz = MAX_HEADER_LINE
        rv = self.fp.readline(sz)
        if size < 0 and len(rv) >= MAX_HEADER_LINE:
            raise HeaderLineTooLong()
        self.total_header_size += len(rv)
        if self.total_header_size > MAX_TOTAL_HEADER_SIZE:
            raise HeadersTooLarge()
        return rv
{code}

Following is the error:

root@swift:~# swift upload abad .bashrc --debug
DEBUG:keystoneclient.auth.identity.v2:Making authentication request to http://192.168.136.199:5000/v2.0/tokens
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): 192.168.136.199
DEBUG:requests.packages.urllib3.connectionpool:"POST /v2.0/tokens HTTP/1.1" 200 21389
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): 192.168.136.192
DEBUG:requests.packages.urllib3.connectionpool:"PUT /v1/AUTH_6b70d083cb684111bc20e084d37704b6/abad HTTP/1.1" 400 40
INFO:swiftclient:REQ: curl -i http://192.168.136.192:8080/v1/AUTH_6b70d083cb684111bc20e084d37704b6/abad -X PUT -H "Content-Length: 0" -H "X-Auth-Token: 3260f6aa815b4767a1022e168766e035"
INFO:swiftclient:RESP STATUS: 400 Bad Request
INFO:swiftclient:RESP HEADERS: {u'Date': u'Wed, 18 Jan 2017 11:51:13 GMT', u'Content-Length': u'40', u'Content-Type': u'text/plain', u'X-Trans-Id': u'tx75aae9d07c4848e9bf168-00587f56b1'}
INFO:swiftclient:RESP BODY: Header value too long: X-Service-Catalog
ERROR:swiftclient:Container PUT failed: http://192.168.136.192:8080/v1/AUTH_6b70d083cb684111bc20e084d37704b6/abad 400 Bad Request Header value too long: X-Service-Catalog
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1565, in _retry
    service_token=self.service_token, **kwargs)
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 976, in put_container
    http_response_content=body)
ClientException: Container PUT failed: http://192.168.136.192:8080/v1/AUTH_6b70d083cb684111bc20e084d37704b6/abad 400 Bad Request Header value too long: X-Service-Catalog
ERROR:swiftclient.service:Container PUT failed: http://192.168.136.192:8080/v1/AUTH_6b70d083cb684111bc20e084d37704b6/abad 400 Bad Request Header value too long: X-Service-Catalog
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/swiftclient/service.py", line 1554, in _create_container_job
    container, headers, response_dict=create_response
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1643, in put_container
    response_dict=response_dict)
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1565, in _retry
    service_token=self.service_token, **kwargs)
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 976, in put_container
    http_response_content=body)
ClientException: Container PUT failed: http://192.168.136.192:8080/v1/AUTH_6b70d083cb684111bc20e084d37704b6/abad 400 Bad Request Header value too long: X-Service-Catalog
Warning: failed to create container 'abad': 400 Bad Request: Header value too long: X-Service-Catalog
DEBUG:keystoneclient.auth.identity.v2:Making authentication request to http://192.168.136.199:5000/v2.0/tokens
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): 192.168.136.199
DEBUG:requests.packages.urllib3.connectionpool:"POST /v2.0/tokens HTTP/1.1" 200 21389
INFO:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): 192.168.136.192
DEBUG:requests.packages.urllib3.connectionpool:"HEAD /v1/AUTH_6b70d083cb684111bc20e084d37704b6/abad/.bashrc HTTP/1.1" 400 0
INFO:swiftclient:REQ: curl -i http://192.168.136.192:8080/v1/AUTH_6b70d083cb684111bc20e084d37704b6/abad/.bashrc -I -H "X-Auth-Token: 3ea0c82a1b26469c82dac9452e7f13de"
INFO:swiftclient:RESP STATUS: 400 Header Line Too Long
INFO:swiftclient:RESP HEADERS: {u'Date': u'Wed, 18 Jan 2017 11:51:13 GMT', u'Content-Length': u'0', u'Content-Type': u'text/html; charset=UTF-8', u'X-Trans-Id': u'tx8bb737da355b48a481e17-00587f56b1'}
ERROR:swiftclient:Object HEAD failed: http://192.168.136.192:8080/v1/AUTH_6b70d083cb684111bc20e084d37704b6/abad/.bashrc 400 Header Line Too Long
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1565, in _retry
    service_token=self.service_token, **kwargs)
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1166, in head_object
    http_response_content=body)
ClientException: Object HEAD failed: http://192.168.136.192:8080/v1/AUTH_6b70d083cb684111bc20e084d37704b6/abad/.bashrc 400 Header Line Too Long
ERROR:swiftclient.service:Object HEAD failed: http://192.168.136.192:8080/v1/AUTH_6b70d083cb684111bc20e084d37704b6/abad/.bashrc 400 Header Line Too Long
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/swiftclient/service.py", line 1783, in _upload_object_job
    headers = conn.head_object(container, obj)
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1657, in head_object
    return self._retry(None, head_object, container, obj, headers=headers)
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1565, in _retry
    service_token=self.service_token, **kwargs)
  File "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1166, in head_object
    http_response_content=body)
ClientException: Object HEAD failed: http://192.168.136.192:8080/v1/AUTH_6b70d083cb684111bc20e084d37704b6/abad/.bashrc 400 Header Line Too Long
Object HEAD failed: http://192.168.136.192:8080/v1/AUTH_6b70d083cb684111bc20e084d37704b6/abad/.bashrc 400 Header Line Too Long

Revision history for this message
Donagh McCabe (donagh-mccabe) wrote :

What is your setting for include_service_catalog?

By default, the value is "false". This was originally done for two reasons:

- saves space in the environment [since Swift does not use/need the catalog]
- prevented the above issue

[Having said this, I had thought the header-too-long problem had been fixed elsewhere, but maybe I miss-remember and the only solution used was to set include_service_catalog=false]

Note: Swift itself does need X-Service-Catalog. The Keystone audit middleware *does* use it, but unless you have good reason to use audit middleware, you can set include_service_catalog=false.

Revision history for this message
Donagh McCabe (donagh-mccabe) wrote :

FYI: https://review.openstack.org/#/c/59371

Revision history for this message
Sanchit Malhotra (isanchitm) wrote :

>What is your setting for include_service_catalog?

Unfortunately, my proxy configuration file does not include the configuration parameter "include_service_catalog" under auth_token filter.

But I suppose this setting is just for applicable for "X-Service-Catalog" request header.

Revision history for this message
Tim Burke (1-tim-z) wrote :

Our sample config makes it seem like that defaults to False [1], but it definitely defaults to True [2] and has for a very long time [3].

Seems like we should clarify what the default is and add a note about why you may want to change the option to False, similar to what we do for delay_auth_decision.

[1] https://github.com/openstack/swift/blob/2.15.1/etc/proxy-server.conf-sample#L385
[2] https://github.com/openstack/keystonemiddleware/blob/4.17.0/keystonemiddleware/auth_token/_opts.py#L149
[3] https://github.com/openstack/python-keystoneclient/commit/a97b293

Changed in swift:
status: New → Confirmed
importance: Undecided → Low
tags: added: documentation low-hanging-fruit
Revision history for this message
Tim Burke (1-tim-z) wrote :

*Maybe* we could also get away with having swift's keystoneauth middleware drop any x-service-catalog headers it finds? I'm not sure who uses it or where they would be in the pipeline, though.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.