Glance

Token auth fails when token is larger than 8k

Series havana
Bug #1190149

Bug #1190149 reported by Steve Baker on 2013-06-12

This bug affects 7 people

	Status	Importance	Assigned to	Milestone
Cinder	Fix Released	Undecided	LiShaokai	Cinder 2014.1 "icehouse"
Havana	Fix Released	Undecided	ling-yun
Glance	Fix Released	Undecided	Unassigned	Glance 2014.1 "icehouse"
Havana	Fix Committed	Undecided	wanghong	Glance 2013.2.3
Murano	Fix Released	High	Kirill Zaitsev	Murano 1.0.1
OpenStack Compute (nova)	Fix Released	Medium	Steve Baker	OpenStack Compute (nova) 2013.2 "havana"
OpenStack DBaaS (Trove)	Fix Released	Undecided	lvdongbing	OpenStack DBaaS (Trove) 2014.1 "icehouse"
OpenStack Heat	Fix Released	Critical	Steve Baker	OpenStack Heat 2014.1 "icehouse"
OpenStack Identity (keystone)	Fix Released	Medium	Chmouel Boudjnah	OpenStack Identity (keystone) 2013.2 "havana"
OpenStack Object Storage (swift)	Fix Released	Undecided	Unassigned	OpenStack Object Storage (swift) 1.12.0
Sahara	Fix Released	High	Deliang Fan	Sahara 2015.1.0 "kilo"
neutron	Fix Released	Medium	Hua Zhang	neutron 2014.1 "icehouse"

Bug Description

The following tests fail when there are 8 or more endpoints registered with keystone
tempest.api.compute.test_auth_token.AuthTokenTestJSON.test_v3_token
tempest.api.compute.test_auth_token.AuthTokenTestXML.test_v3_token

Steps to reproduce:
- run devstack with the following services (the heat h-* apis push the endpoint count over the threshold
  ENABLED_SERVICES=g-api,g-reg,key,n-api,n-crt,n-obj,n-cpu,n-sch,horizon,mysql,rabbit,sysstat,tempest,s-proxy,s-account,s-container,s-object,cinder,c-api,c-vol,c-sch,n-cond,heat,h-api,h-api-cfn,h-api-cw,h-eng,n-net
- run the failing tempest tests, eg
  testr run test_v3_token
- results in the following errors:
ERROR: tempest.api.compute.test_auth_token.AuthTokenTestJSON.test_v3_token
tags: worker-0
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tempest/api/compute/test_auth_token.py", line 48, in test_v3_token
    self.servers_v3.list_servers()
  File "tempest/services/compute/json/servers_client.py", line 138, in list_servers
    resp, body = self.get(url)
  File "tempest/common/rest_client.py", line 269, in get
    return self.request('GET', url, headers)
  File "tempest/common/rest_client.py", line 394, in request
    resp, resp_body)
  File "tempest/common/rest_client.py", line 443, in _error_checker
    resp_body = self._parse_resp(resp_body)
  File "tempest/common/rest_client.py", line 327, in _parse_resp
    return json.loads(body)
  File "/usr/lib64/python2.7/json/__init__.py", line 326, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python2.7/json/decoder.py", line 384, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
======================================================================
ERROR: tempest.api.compute.test_auth_token.AuthTokenTestXML.test_v3_token
tags: worker-0
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tempest/api/compute/test_auth_token.py", line 48, in test_v3_token
    self.servers_v3.list_servers()
  File "tempest/services/compute/xml/servers_client.py", line 181, in list_servers
    resp, body = self.get(url, self.headers)
  File "tempest/common/rest_client.py", line 269, in get
    return self.request('GET', url, headers)
  File "tempest/common/rest_client.py", line 394, in request
    resp, resp_body)
  File "tempest/common/rest_client.py", line 443, in _error_checker
    resp_body = self._parse_resp(resp_body)
  File "tempest/common/rest_client.py", line 519, in _parse_resp
    return xml_to_json(etree.fromstring(body))
  File "lxml.etree.pyx", line 2993, in lxml.etree.fromstring (src/lxml/lxml.etree.c:63285)
  File "parser.pxi", line 1617, in lxml.etree._parseMemoryDocument (src/lxml/lxml.etree.c:93571)
  File "parser.pxi", line 1495, in lxml.etree._parseDoc (src/lxml/lxml.etree.c:92370)
  File "parser.pxi", line 1011, in lxml.etree._BaseParser._parseDoc (src/lxml/lxml.etree.c:89010)
  File "parser.pxi", line 577, in lxml.etree._ParserContext._handleParseResultDoc (src/lxml/lxml.etree.c:84711)
  File "parser.pxi", line 676, in lxml.etree._handleParseResult (src/lxml/lxml.etree.c:85816)
  File "parser.pxi", line 627, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:85308)
XMLSyntaxError: None
Ran 2 tests in 2.497s (+0.278s)
FAILED (id=214, failures=2)

- run keystone endpoint-delete on endpoints until there is 7 endpoints
- failing tests should now pass

Tags:

Revision history for this message

Steve Baker (steve-stevebaker) wrote on 2013-06-12:

This bug is currently blocking heat from being enabled in tempest gating https://review.openstack.org/#/c/32002

Revision history for this message

Dolph Mathews (dolph) wrote on 2013-06-12:

What is the JSON/XML response returned from keystone in either case? I'm guessing there's something wrong with a specific endpoint, rather than an issue with the specific number of endpoints.

Changed in keystone:
status:	New → Incomplete
importance:	Undecided → Medium

Revision history for this message

Steve Baker (steve-stevebaker) wrote on 2013-06-12:

I've done some testing and I really do thing the working endpoint limit is 8.

Steps to reproduce:
- keystone endpoint-delete until there are 6 endpoints
- confirm test_v3_token works with 6 endpoints
- create a new endpoint
  keystone service-create --name ep1 --type test1
  keystone endpoint-create --region RegionOne --service-id <serviceid> --publicurl http://example.com --adminurl http://example.com --internalurl http://example.com
- confirm test_v3_token works with 7 endpoints
- create a new endpoint
  keystone service-create --name ep2 --type test2
  keystone endpoint-create --region RegionOne --service-id <serviceid> --publicurl http://example.com --adminurl http://example.com --internalurl http://example.com
- confirm test_v3_token fails with 8 endpoints
- delete endpoint ep1
- confirm test_v3_token works with 7 endpoints

Is there some simple curl foo I can use to make keystone v3 requests?

Revision history for this message

Steve Baker (steve-stevebaker) wrote on 2013-06-12:

In the testing mentioned above, the token for 8 endpoints is 8448 bytes. For 7 endpoints its 7640.

I'd bet a beer in Hong Kong that something is truncating the token header to 8k somewhere.

Revision history for this message

Steve Baker (steve-stevebaker) wrote on 2013-06-12:

The v2 catalog format is more compact, hence smaller tokens. I created enough endpoints to make a v2 token over 8k (20 endpoints total)

This results in many calls to keystone, then test_auth_token.py not running any tests at all

summary:

- Using v3 tokens fails when there is 8 or more endpoints defined
+ Token auth fails when token is larger than 8k

Revision history for this message

Dolph Mathews (dolph) wrote on 2013-06-12:

This would be resolved by https://blueprints.launchpad.net/keystone/+spec/catalog-optional

Changed in keystone:
status:	Incomplete → Confirmed

Revision history for this message

Steve Baker (steve-stevebaker) wrote on 2013-06-12:

What could we do in the short term so that heat tests can run on gating?

Revision history for this message

Dolph Mathews (dolph) wrote on 2013-06-13:

Fixing this would provide a small improvement here https://bugs.launchpad.net/keystone/+bug/1164458

Revision history for this message

Steve Baker (steve-stevebaker) wrote on 2013-06-14:

I've found a fix, but it would have to be applied to every project that uses eventlet.wsgi:
http://lists.openstack.org/pipermail/openstack-dev/2013-June/010366.html

Revision history for this message

Steve Baker (steve-stevebaker) wrote on 2013-06-19:

#10

Fixed in nova with https://review.openstack.org/#/c/33362/

Changed in nova:
assignee:	nobody → Steve Baker (steve-stevebaker)
status:	New → Fix Committed

Thierry Carrez (ttx) on 2013-07-17

Changed in nova:
milestone:	none → havana-2
status:	Fix Committed → Fix Released

Chmouel Boudjnah (chmouel) on 2013-08-02

Changed in keystone:
assignee:	nobody → Chmouel Boudjnah (chmouel)

Revision history for this message

Chmouel Boudjnah (chmouel) wrote on 2013-08-02:

#11

https://review.openstack.org/#/c/39882/

Chmouel Boudjnah (chmouel) on 2013-08-06

Changed in keystone:
status:	Confirmed → Fix Committed

Thierry Carrez (ttx) on 2013-09-05

Changed in keystone:
milestone:	none → havana-3
status:	Fix Committed → Fix Released

Thierry Carrez (ttx) on 2013-10-17

Changed in nova:
milestone:	havana-2 → 2013.2

Thierry Carrez (ttx) on 2013-10-17

Changed in keystone:
milestone:	havana-3 → 2013.2

LiShaokai (lshaokai) on 2014-01-10

Changed in cinder:
status:	New → Confirmed
assignee:	nobody → LiShaokai (lshaokai)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-01-10: Fix merged to cinder (master)

#12

Reviewed: https://review.openstack.org/64641
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=aa1ba1f0176af286506e4188628e5acbb8f7f0e6
Submitter: Jenkins
Branch: master

commit aa1ba1f0176af286506e4188628e5acbb8f7f0e6
Author: Shao Kai Li <email address hidden>
Date: Thu Jan 2 00:51:41 2014 -0500

Raise max header size to accommodate large tokens.

    The max header is exceeded in the following scenario
    - Auth tokens built with a keystone v3 API catalog
    - A catalog with approximately 8 or more endpoints defined

An equivalent fix will be required for all projects which use
eventlet.wsgi.

Copied from https://review.openstack.org/#/c/33362/

Closes-Bug: 1190149

Change-Id: I3fcb196b25a7a477b799c8dfa2ff5115977ca06d

Changed in cinder:
status:	Confirmed → Fix Committed

Florent Flament (florentflament) on 2014-01-13

Changed in glance:
assignee:	nobody → Florent Flament (florent-flament-ext)
Changed in neutron:
assignee:	nobody → Florent Flament (florent-flament-ext)
Changed in swift:
assignee:	nobody → Florent Flament (florent-flament-ext)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-01-13: Fix proposed to glance (master)

#13

Fix proposed to branch: master
Review: https://review.openstack.org/66332

Changed in glance:
status:	New → In Progress

Mark Washenberger (markwash) on 2014-01-14

Changed in glance:
milestone:	none → icehouse-2

Revision history for this message

Florent Flament (florentflament) wrote on 2014-01-14:

#14

Neutron fix has been previsouly proposed by Zhang Hua here:
https://review.openstack.org/#/c/56511/

Trying and reactivating this abandonned patch.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-01-14: Fix proposed to swift (master)

#15

Fix proposed to branch: master
Review: https://review.openstack.org/66615

Changed in swift:
status:	New → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-01-15: Fix merged to glance (master)

#16

Reviewed: https://review.openstack.org/66332
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=df8be62e58d8eb36cef43a7f3b594ff0a0e1ccb6
Submitter: Jenkins
Branch: master

commit df8be62e58d8eb36cef43a7f3b594ff0a0e1ccb6
Author: Florent Flament <email address hidden>
Date: Mon Jan 13 13:43:59 2014 +0000

Allow users to customize max header size

Also increases the default value from 8192 to 16384

    The former max header (8192) is exceeded in the following scenario
    - Auth tokens built with a keystone v3 API catalog
    - A catalog with approximately 8 or more endpoints defined

Equivalent fixes have been proposed in other projects

Change-Id: Ic714166440e2aa08b619ab1e5e5e0cf3cea3e309
Closes-Bug: 1190149

Changed in glance:
status:	In Progress → Fix Committed

Revision history for this message

Florent Flament (florentflament) wrote on 2014-01-17:

#17

Zhang Hua's Neutron fix has been reactivated:

https://review.openstack.org/#/c/56511/

Changed in neutron:
assignee:	Florent Flament (florent-flament-ext) → nobody
status:	New → In Progress

Revision history for this message

Florent Flament (florentflament) wrote on 2014-01-17:

#18

An interesting comment from Mark McClain about redefining constants:
https://review.openstack.org/#/c/56511/2/neutron/wsgi.py

I agree with him that redefining constants should be avoided when possible, since it can possibly create unexpected behaviors if the constant is redefined to different values in different parts of the code.

However, in the case of the eventlet.wsgi.MAX_HEADER_LINE constant, I don't see any better solution until eventlet's code is updated:
https://bitbucket.org/eventlet/eventlet/src/66cc547edabfb7b45afaead69f029cae7f51f161/eventlet/wsgi.py#cl-189

We should be careful about not redefining this constant several times (grep -r MAX_HEADER_LINE .)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-01-17: Related fix proposed to cinder (master)

#19

Related fix proposed to branch: master
Review: https://review.openstack.org/67601

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-01-19: Fix merged to swift (master)

#20

Reviewed: https://review.openstack.org/66615
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=865243c16751e18e28b60ef03ecaada44c22cd13
Submitter: Jenkins
Branch: master

commit 865243c16751e18e28b60ef03ecaada44c22cd13
Author: Florent Flament <email address hidden>
Date: Tue Jan 14 15:58:38 2014 +0000

Fix Error 400 Header Line Too Long

Fix Error 400 Header Line Too Long when using Identity v3 PKI Tokens

Uses swift.conf max_header_size option to set wsgi.MAX_HEADER_LINE,
allowing the operator to customize this parameter.

    The default value has been let to 8192 to avoid unexpected
    configuration change on deployed platforms. The max_header_size option
    has to be increased (for example to 16384), to accomodate for large
    Identity v3 PKI tokens, including more than 7 catalog entries.

    The default max header line size of 8192 is exceeded in the following
    scenario:
    - Auth tokens generated by Keystone v3 API include the catalog.
    - Keystone's catalog contains more than 7 services.

Similar fixes have been merged in other projects.

Change-Id: Ia838b18331f57dfd02b9f71d4523d4059f38e600
Closes-Bug: 1190149

Changed in swift:
status:	In Progress → Fix Committed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-01-20: Related fix proposed to nova (master)

#21

Related fix proposed to branch: master
Review: https://review.openstack.org/67800

Thierry Carrez (ttx) on 2014-01-22

Changed in glance:
status:	Fix Committed → Fix Released

Thierry Carrez (ttx) on 2014-01-22

Changed in swift:
milestone:	none → 1.12.0
status:	Fix Committed → Fix Released

Thierry Carrez (ttx) on 2014-01-22

Changed in cinder:
milestone:	none → icehouse-2
status:	Fix Committed → Fix Released

Steve Baker (steve-stevebaker) on 2014-01-31

Changed in heat:
status:	New → Triaged
importance:	Undecided → Medium
assignee:	nobody → Steve Baker (steve-stevebaker)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-01-31: Fix proposed to heat (master)

#22

Fix proposed to branch: master
Review: https://review.openstack.org/70289

Changed in heat:
status:	Triaged → In Progress

Revision history for this message

Steve Baker (steve-stevebaker) wrote on 2014-01-31:

#23

Actually I'm going to mark this as critical for heat, since I think our switch to keystone v3 (and its bigger tokens) will trigger this in the common case.

Changed in heat:
importance:	Medium → Critical
milestone:	none → icehouse-3

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-01-31: Fix proposed to swift (feature/ec)

#24

Fix proposed to branch: feature/ec
Review: https://review.openstack.org/70442

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-02-11: Related fix merged to nova (master)

#25

Reviewed: https://review.openstack.org/67800
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=933f1dca3662b0667c90a3fc143c5245fe6b3f91
Submitter: Jenkins
Branch: master

commit 933f1dca3662b0667c90a3fc143c5245fe6b3f91
Author: Florent Flament <email address hidden>
Date: Mon Jan 20 10:07:53 2014 +0000

Allow operators to customize max header size

    HTTP messages max header line size has been increased from 8K to 16K
    by default to allow using tokens including up to 14-15 catalog
    entries. The same issue (https://bugs.launchpad.net/swift/+bug/119014)
    may occur again in the future, if Keystone's catalog grows further.

    Allowing operators to customize the max header size, will allow them
    to have Nova working whatever the size of the catalog (if the option
    is properly set).

Related-Bug: #1190149
DocImpact

Change-Id: I96694ac9bc242267cfc9f4d0c18b2b7fe0000460

Revision history for this message

Kai Qiang Wu(Kennan) (wkqwu) wrote on 2014-02-11:

#26

If my understand correctly, the new change make max header line as configurable field in conf file, keystone need follow that.

Seems keystone hard code that.

Revision history for this message

Steve Baker (steve-stevebaker) wrote on 2014-02-11:

#27

It looks like we're converging on doing this as a conf options rather than hardcoded.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-02-13: Related fix merged to cinder (master)

#28

Reviewed: https://review.openstack.org/67601
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=be163cf1028aff50413a89060bb8badfa7d51ad5
Submitter: Jenkins
Branch: master

commit be163cf1028aff50413a89060bb8badfa7d51ad5
Author: Florent Flament <email address hidden>
Date: Fri Jan 17 10:54:27 2014 +0000

Allow operators to customize max header size

    HTTP messages max header line size has been increased from 8K to 16K
    by default to allow using tokens including up to 14-15 catalog
    entries. The same issue
    (https://bugs.launchpad.net/swift/+bug/1190149) may occur again in
    the future, if keystone's catalog grows further.

    Allowing operators to customize the max header size, will allow them
    to have Cinder working whatever the size of the catalog (if the option
    is properly set).

    Change-Id: Iac269d96fadc96c1e9383dc03422f2d87196dd9d
    Related-Bug: #1190149
    DocImpact

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-02-17: Fix merged to heat (master)

#29

Reviewed: https://review.openstack.org/70289
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=0b02feb20d4485d0c6d486c5a72b814ce3bdf9e5
Submitter: Jenkins
Branch: master

commit 0b02feb20d4485d0c6d486c5a72b814ce3bdf9e5
Author: Steve Baker <email address hidden>
Date: Mon Feb 3 08:45:14 2014 +1300

Raise the default max header to accommodate large tokens

PKI tokens hit the default limit if there is enough
services defined in the keystone catalog.

Also the v3 catalog is larger than the v2 catalog which would explain
why this bug is being hit just now.

This change adds the configuration option max_header_line to each of the
API confurations which has a default of 16384.

Closes-Bug: #1190149
Change-Id: I5da09aa08a1242c5e356bd8bf532baa9347ce075

Changed in heat:
status:	In Progress → Fix Committed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-02-24: Related fix proposed to cinder (stable/havana)

#30

Related fix proposed to branch: stable/havana
Review: https://review.openstack.org/75732

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-02-24: Related fix proposed to nova (stable/havana)

#31

Related fix proposed to branch: stable/havana
Review: https://review.openstack.org/75747

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-02-24: Fix proposed to heat (stable/havana)

#32

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/75784

lvdongbing (dbcocle) on 2014-02-24

Changed in trove:
assignee:	nobody → lvdongbing (dbcocle)

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-02-24: Fix proposed to trove (master)

#33

Fix proposed to branch: master
Review: https://review.openstack.org/75786

Changed in trove:
status:	New → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-02-28: Fix proposed to glance (stable/havana)

#34

Fix proposed to branch: stable/havana
Review: https://review.openstack.org/77108

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-02-28: Fix merged to heat (stable/havana)

#35

Reviewed: https://review.openstack.org/75784
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=defcf235c158e93938c93f556383939515852c88
Submitter: Jenkins
Branch: stable/havana

commit defcf235c158e93938c93f556383939515852c88
Author: Steve Baker <email address hidden>
Date: Mon Feb 3 08:45:14 2014 +1300

Raise the default max header to accommodate large tokens

PKI tokens hit the default limit if there is enough
services defined in the keystone catalog.

Also the v3 catalog is larger than the v2 catalog which would explain
why this bug is being hit just now.

This change adds the configuration option max_header_line to each of the
API confurations which has a default of 16384.

    Closes-Bug: #1190149
    Change-Id: I5da09aa08a1242c5e356bd8bf532baa9347ce075
    (cherry picked from commit 0b02feb20d4485d0c6d486c5a72b814ce3bdf9e5)

tags:

added: in-stable-havana

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-03-03: Fix merged to trove (master)

#36

Reviewed: https://review.openstack.org/75786
Committed: https://git.openstack.org/cgit/openstack/trove/commit/?id=4df858db82a8684f19bdc25bf1747bdafd4bd627
Submitter: Jenkins
Branch: master

commit 4df858db82a8684f19bdc25bf1747bdafd4bd627
Author: lvdongbing <email address hidden>
Date: Mon Feb 24 15:57:20 2014 +0800

Raise max header size to accommodate large tokens

    The max header is exceeded in the following scenario
    - Auth tokens built with a keystone v3 API catalog
    - A catalog with approximately 8 or more endpoints defined

An equivalent fix will be required for all projects which use
eventlet.wsgi.

Closes-Bug: #1190149

Change-Id: Ia0882e43395f1025137173b51aab147433b54255

Changed in trove:
status:	In Progress → Fix Committed

Auston McReynolds (amcrn) on 2014-03-03

Changed in trove:
milestone:	none → icehouse-3

Thierry Carrez (ttx) on 2014-03-05

Changed in trove:
milestone:	icehouse-3 → none

Thierry Carrez (ttx) on 2014-03-05

Changed in trove:
milestone:	none → icehouse-3
status:	Fix Committed → Fix Released

Alan Pevec (apevec) on 2014-03-05

Changed in nova:
importance:	Undecided → Medium

Thierry Carrez (ttx) on 2014-03-05

Changed in heat:
status:	Fix Committed → Fix Released

Alan Pevec (apevec) on 2014-03-05

tags:

removed: in-stable-havana

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-03-10: Fix merged to glance (stable/havana)

#37

Reviewed: https://review.openstack.org/77108
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=1690a739cd94efe6cb2af8e794a4acdf59de54cd
Submitter: Jenkins
Branch: stable/havana

commit 1690a739cd94efe6cb2af8e794a4acdf59de54cd
Author: Florent Flament <email address hidden>
Date: Mon Jan 13 13:43:59 2014 +0000

Raise max header size to accommodate large tokens

    The former max header (8192) is exceeded in the following scenario
    - Auth tokens built with a keystone v3 API catalog
    - A catalog with approximately 8 or more endpoints defined

Equivalent fixes have been proposed in other projects

Change-Id: Ic714166440e2aa08b619ab1e5e5e0cf3cea3e309
Closes-Bug: 1190149

Revision history for this message

Alan Pevec (apevec) wrote on 2014-03-16:

#38

Proposed to Cinder stable/havana https://review.openstack.org/78513

Alan Pevec (apevec) on 2014-03-16

Changed in neutron:
assignee:	nobody → Hua Zhang (zhhuabj)
importance:	Undecided → Medium
status:	In Progress → Fix Committed

Akihiro Motoki (amotoki) on 2014-03-22

Changed in neutron:
milestone:	none → icehouse-rc1

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-03-24: Fix merged to cinder (stable/havana)

#39

Reviewed: https://review.openstack.org/78513
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=5b9fc5c132b0ace37c3c99f2f9a32bc88450403c
Submitter: Jenkins
Branch: stable/havana

commit 5b9fc5c132b0ace37c3c99f2f9a32bc88450403c
Author: Shao Kai Li <email address hidden>
Date: Thu Jan 2 00:51:41 2014 -0500

Raise max header size to accommodate large tokens

    The max header is exceeded in the following scenario
    - Auth tokens built with a keystone v3 API catalog
    - A catalog with approximately 8 or more endpoints defined

An equivalent fix will be required for all projects which use
eventlet.wsgi.

Copied from https://review.openstack.org/#/c/33362/

Closes-Bug: 1190149

Change-Id: I3fcb196b25a7a477b799c8dfa2ff5115977ca06d
(cherry picked from commit aa1ba1f0176af286506e4188628e5acbb8f7f0e6)

Thierry Carrez (ttx) on 2014-04-01

Changed in neutron:
status:	Fix Committed → Fix Released

Revision history for this message

Adam Gandelman (gandelman-a) wrote on 2014-04-07:

#40

Note this has been Fix Released in the 2013.2.3 releases of Cinder, Glance and Heat. I can't seem to update the status on this bug without launchpad error.

Revision history for this message

Adam Young (ayoung) wrote on 2014-04-08:

#41

Note that a recent commit to Master mitigates the effect of this: it removes two values from each endpoint, significantly shrinking the size of the service catalog.

http://git.openstack.org/cgit/openstack/keystone/commit/?id=6f4cade9ed980b7c87abf5f7aa17bf56874252fa

A future enhancement will compress the tokens, but it is a little too invasive to get working for Icehouse.

Thierry Carrez (ttx) on 2014-04-17

Changed in trove:
milestone:	icehouse-3 → 2014.1

Thierry Carrez (ttx) on 2014-04-17

Changed in cinder:
milestone:	icehouse-2 → 2014.1

Thierry Carrez (ttx) on 2014-04-17

Changed in heat:
milestone:	icehouse-3 → 2014.1

Thierry Carrez (ttx) on 2014-04-17

Changed in glance:
milestone:	icehouse-2 → 2014.1

Thierry Carrez (ttx) on 2014-04-17

Changed in neutron:
milestone:	icehouse-rc1 → 2014.1

Valeriy Ponomaryov (vponomaryov) on 2014-04-24

Changed in manila:
assignee:	nobody → Valeriy Ponomaryov (vponomaryov)

Revision history for this message

Openstack Gerrit (openstack-gerrit) wrote on 2014-04-24: Related fix proposed to manila (master)

#42

Related fix proposed to branch: master
Review: https://review.openstack.org/90130

Revision history for this message

Openstack Gerrit (openstack-gerrit) wrote on 2014-04-25: Related fix merged to manila (master)

#43

Reviewed: https://review.openstack.org/90130
Committed: https://git.openstack.org/cgit/stackforge/manila/commit/?id=979c2c25409a33247955c199cbe8f897477e3a07
Submitter: Jenkins
Branch: master

commit 979c2c25409a33247955c199cbe8f897477e3a07
Author: vponomaryov <email address hidden>
Date: Thu Apr 24 19:03:11 2014 +0300

Raise max header size to accommodate large tokens

Related-Bug: #1190149

Change-Id: I14a64c538b8d35549747ae74c2aea192fccbed9b

Valeriy Ponomaryov (vponomaryov) on 2014-04-29

Changed in manila:
status:	New → Fix Committed

Revision history for this message

Valeriy Ponomaryov (vponomaryov) wrote on 2014-05-19:

#44

Who has permissions to set status "fix released" please set it to manila's project.
Thanks in advance.

Valeriy Ponomaryov (vponomaryov) on 2014-06-16

no longer affects:

manila

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-11-12: Related fix proposed to swift (master)

#45

Related fix proposed to branch: master
Review: https://review.openstack.org/133967

Revision history for this message

Amrith Kumar (amrith) wrote on 2014-11-12:

#46

I don't believe this is a problem in Trove which already defaults to 16k for tokens. Now if 16k isn't sufficient, there's a configuration parameter.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2014-11-21: Change abandoned on swift (master)

#47

Change abandoned by Thomas Bechtold (<email address hidden>) on branch: master
Review: https://review.openstack.org/133967
Reason: consensus is to leave he default.

Deliang Fan (vanderliang) on 2015-03-20

Changed in sahara:
assignee:	nobody → Deliang Fan (vanderliang)

Deliang Fan (vanderliang) on 2015-03-20

Changed in sahara:
status:	New → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-03-20: Fix proposed to sahara (master)

#48

Fix proposed to branch: master
Review: https://review.openstack.org/166083

Sergey Reshetnyak (sreshetniak) on 2015-03-20

Changed in sahara:
importance:	Undecided → High
milestone:	none → kilo-rc1

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-03-20: Fix merged to sahara (master)

#49

Reviewed: https://review.openstack.org/166083
Committed: https://git.openstack.org/cgit/openstack/sahara/commit/?id=07dc4fd895d883d3ca9339d6725d0ac46a7d7e94
Submitter: Jenkins
Branch: master

commit 07dc4fd895d883d3ca9339d6725d0ac46a7d7e94
Author: Deliang Fan <email address hidden>
Date: Thu Mar 19 19:48:39 2015 -0700

Raise the default max header to accommodate large tokens

PKI tokens hit the default limit if there is enough
services defined in the keystone catalog.

So sahara should allow users to customize max header size and
also increase the default value from 8192 to 16384.

Change-Id: If3daff1ba18f7fcd4cf3b7d9b4152b551d8ad277
Closes-Bug: 1190149

Changed in sahara:
status:	In Progress → Fix Committed

Thierry Carrez (ttx) on 2015-04-09

Changed in sahara:
status:	Fix Committed → Fix Released

Thierry Carrez (ttx) on 2015-04-30

Changed in sahara:
milestone:	kilo-rc1 → 2015.1.0

Serg Melikyan (smelikyan) on 2015-07-17

Changed in murano:
importance:	Undecided → High
milestone:	none → liberty-3
status:	New → Confirmed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-07-17: Fix proposed to murano (master)

#51

Fix proposed to branch: master
Review: https://review.openstack.org/203286

Changed in murano:
assignee:	nobody → Kirill Zaitsev (kzaitsev)
status:	Confirmed → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-07-17: Fix proposed to murano (stable/kilo)

#52

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/203288

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-07-17: Fix proposed to murano (stable/juno)

#53

Fix proposed to branch: stable/juno
Review: https://review.openstack.org/203289

Florent Flament (florentflament) on 2015-07-18

Changed in glance:
assignee:	Florent Flament (florentflament) → nobody
Changed in swift:
assignee:	Florent Flament (florentflament) → nobody

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-07-22: Fix merged to murano (stable/juno)

#54

Reviewed: https://review.openstack.org/203289
Committed: https://git.openstack.org/cgit/openstack/murano/commit/?id=7a6faa63b898861ce2af2295c283a036bffcce87
Submitter: Jenkins
Branch: stable/juno

commit 7a6faa63b898861ce2af2295c283a036bffcce87
Author: Kirill Zaitsev <email address hidden>
Date: Sat Jul 18 02:24:51 2015 +0300

Allow setting eventlet.wsgi.MAX_HEADER_LINE

    This commit allows setting wsgi MAX_HEADER_LINE. This is done to allow
    large keystone tokens. Also removes seeimngly unused run_server
    function, to not cause confustion.

Change-Id: I2848c08d0758f39cb988079a8fabf36d6d2a2c98
Closes-Bug: #1190149

tags:

added: in-stable-juno

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-07-22: Fix merged to murano (master)

#55

Reviewed: https://review.openstack.org/203286
Committed: https://git.openstack.org/cgit/openstack/murano/commit/?id=20756379cd07e1194543e7b09444efb3412f3db8
Submitter: Jenkins
Branch: master

commit 20756379cd07e1194543e7b09444efb3412f3db8
Author: Kirill Zaitsev <email address hidden>
Date: Sat Jul 18 02:24:51 2015 +0300

Allow setting eventlet.wsgi.MAX_HEADER_LINE

    This commit allows setting wsgi MAX_HEADER_LINE. This is done to allow
    large keystone tokens. Also removes seeimngly unused run_server
    function, to not cause confustion.

Change-Id: I2848c08d0758f39cb988079a8fabf36d6d2a2c98
Closes-Bug: #1190149

Changed in murano:
status:	In Progress → Fix Committed
tags:	added: in-stable-kilo

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2015-07-22: Fix merged to murano (stable/kilo)

#56

Reviewed: https://review.openstack.org/203288
Committed: https://git.openstack.org/cgit/openstack/murano/commit/?id=99a727e346172f73da6cc6a23edee87348b8a100
Submitter: Jenkins
Branch: stable/kilo

commit 99a727e346172f73da6cc6a23edee87348b8a100
Author: Kirill Zaitsev <email address hidden>
Date: Sat Jul 18 02:24:51 2015 +0300

Allow setting eventlet.wsgi.MAX_HEADER_LINE

    This commit allows setting wsgi MAX_HEADER_LINE. This is done to allow
    large keystone tokens. Also removes seeimngly unused run_server
    function, to not cause confustion.

Change-Id: I2848c08d0758f39cb988079a8fabf36d6d2a2c98
Closes-Bug: #1190149

Serg Melikyan (smelikyan) on 2015-08-03

Changed in murano:
milestone:	liberty-3 → liberty-2
status:	Fix Committed → Fix Released

Serg Melikyan (smelikyan) on 2015-10-15

Changed in murano:
milestone:	liberty-2 → 1.0.0

Kirill Zaitsev (kzaitsev) on 2015-11-06

Changed in murano:
milestone:	1.0.0 → 1.0.1

Thomas Herve (therve) on 2016-05-25

no longer affects:

heat/havana

Revision history for this message

Doug Hellmann (doug-hellmann) wrote on 2016-06-08: Fix included in openstack/murano 2015.1.1

#57

This issue was fixed in the openstack/murano 2015.1.1 release.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2016-11-10:

#58

This issue was fixed in the openstack/murano 2015.1.1 release.

Report a bug

This report contains Public information

Everyone can see this information.

Duplicates of this bug

Bug #1248842

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.