Error 413 (Request entity too large) while doing port-list on scale setup

Bug #1460675 reported by Vedamurthy Joshi
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Fix Committed
High
Prakash Bailkeri
R2.20
Fix Committed
High
Prakash Bailkeri

Bug Description

R2.20 BUild 30 Ubuntu 14.04 Juno multi-node setup

On the tor-scale setup with 128K vmis on 500 vns, port-list operations (for ex : admin overview/instances page in horizon) get error 413 (Request entity too large)

global HttpError = <class 'cfgm_common.exceptions.HttpError'>
status = 413
content = u'\n <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML...t to large</pre>\n </body>\n </html>\n'
<class 'cfgm_common.exceptions.HttpError'>: HTTP Status: 413 Content:
    <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
    <html>
        <head>
            <title>Error: 413 Request Entity Too Large</title>
            <style type="text/css">
              html {background-color: #eee; font-family: sans;}
              body {background-color: #fff; border: 1px solid #ddd;
                    padding: 15px; margin: 15px;}
              pre {background-color: #eee; border: 1px solid #ddd; padding: 5px;}
            </style>
        </head>
        <body>
            <h1>Error: 413 Request Entity Too Large</h1>
            <p>Sorry, the requested URL <tt>&#039;http://127.0.0.1:9100/list-bulk-collection&#039;</tt>
               caused an error:</p>
            <pre>Request to large</pre>
        </body>
    </html>

    __class__ = <class 'cfgm_common.exceptions.HttpError'>
    __delattr__ = <method-wrapper '__delattr__' of HttpError object>
    __dict__ = {'content': u'\n <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML...t to large</pre>\n </body>\n </html>\n', 'status_code': 413}
    __doc__ = None
    __format__ = <built-in method __format__ of HttpError object>
    __getattribute__ = <method-wrapper '__getattribute__' of HttpError object>
    __getitem__ = <method-wrapper '__getitem__' of HttpError object>
    __getslice__ = <method-wrapper '__getslice__' of HttpError object>
    __hash__ = <method-wrapper '__hash__' of HttpError object>
    __init__ = <bound method HttpError.__init__ of HttpError()>
    __module__ = 'cfgm_common.exceptions'
    __new__ = <built-in method __new__ of type object>
    __reduce__ = <built-in method __reduce__ of HttpError object>
    __reduce_ex__ = <built-in method __reduce_ex__ of HttpError object>
    __repr__ = <method-wrapper '__repr__' of HttpError object>
    __setattr__ = <method-wrapper '__setattr__' of HttpError object>
    __setstate__ = <built-in method __setstate__ of HttpError object>
    __repr__ = <method-wrapper '__repr__' of HttpError object>
    __setattr__ = <method-wrapper '__setattr__' of HttpError object>
    __setstate__ = <built-in method __setstate__ of HttpError object>
    __sizeof__ = <built-in method __sizeof__ of HttpError object>
    __str__ = <bound method HttpError.__str__ of HttpError()>
    __subclasshook__ = <built-in method __subclasshook__ of type object>
    __unicode__ = <built-in method __unicode__ of HttpError object>
    __weakref__ = None
    args = ()
    content = u'\n <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML...t to large</pre>\n </body>\n </html>\n'
    message = ''
    status_code = 413

The above is a description of an error in a Python program. Here is
the original traceback:

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/vnc_openstack/neutron_plugin_interface.py", line 436, in plugin_get_ports
    ports_info = cfgdb.port_list(context, filters)
  File "/usr/lib/python2.7/dist-packages/vnc_openstack/neutron_plugin_db.py", line 3633, in port_list
    is_admin=context['is_admin'])
  File "/usr/lib/python2.7/dist-packages/vnc_openstack/neutron_plugin_db.py", line 709, in _port_list_project
    return self._port_list(all_port_objs)
  File "/usr/lib/python2.7/dist-packages/vnc_openstack/neutron_plugin_db.py", line 647, in _port_list
    detail=True)
  File "/usr/lib/python2.7/dist-packages/vnc_openstack/neutron_plugin_db.py", line 335, in _virtual_network_list
    count=count)
  File "/usr/lib/python2.7/dist-packages/vnc_api/gen/vnc_api_client_gen.py", line 4596, in virtual_networks_list
    return self.resource_list('virtual-network', parent_id = parent_id, parent_fq_name = parent_fq_name, back_ref_id = back_ref_id, obj_uuids = obj_uuids, fields = fields, detail = detail, count = count)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 653, in resource_list
    uri, json_body)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 368, in _request_server
    retry_count=retry_count)
  File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 429, in _request
    raise HttpError(status, content)
HttpError: HTTP Status: 413 Content:
    <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
    <html>
        <head>
            <title>Error: 413 Request Entity Too Large</title>
            <style type="text/css">
              html {background-color: #eee; font-family: sans;}
              body {background-color: #fff; border: 1px solid #ddd;
                    padding: 15px; margin: 15px;}
              pre {background-color: #eee; border: 1px solid #ddd; padding: 5px;}
            </style>
        </head>
        <body>
            <h1>Error: 413 Request Entity Too Large</h1>
            <p>Sorry, the requested URL <tt>&#039;http://127.0.0.1:9100/list-bulk-collection&#039;</tt>
               caused an error:</p>
            <pre>Request to large</pre>
        </body>
    </html>

Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/bottle.py", line 856, in _handle
    return route.call(**args)
  File "/usr/lib/python2.7/dist-packages/bottle.py", line 1721, in wrapper
    rv = callback(*a, **ka)
  File "/usr/lib/python2.7/dist-packages/vnc_openstack/neutron_plugin_interface.py", line 474, in plugin_http_post_port
    return self.plugin_get_ports(context, port)
  File "/usr/lib/python2.7/dist-packages/vnc_openstack/neutron_plugin_interface.py", line 440, in plugin_get_ports
    raise e
HttpError: HTTP Status: 413 Content:
    <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
    <html>
        <head>
            <title>Error: 413 Request Entity Too Large</title>
            <style type="text/css">
              html {background-color: #eee; font-family: sans;}
              body {background-color: #fff; border: 1px solid #ddd;
                    padding: 15px; margin: 15px;}
              pre {background-color: #eee; border: 1px solid #ddd; padding: 5px;}
            </style>
        </head>
        <body>
            <h1>Error: 413 Request Entity Too Large</h1>
            <p>Sorry, the requested URL <tt>&#039;http://127.0.0.1:9100/list-bulk-collection&#039;</tt>
               caused an error:</p>
            <pre>Request to large</pre>
        </body>
    </html>

192.168.1.2 - - [2015-05-29 21:24:31] "POST /neutron/port HTTP/1.1" 500 156 165.221431

Tags: config scale
Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : R2.20

Review in progress for https://review.opencontrail.org/11164
Submitter: Prakash Bailkeri (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : master

Review in progress for https://review.opencontrail.org/11165
Submitter: Prakash Bailkeri (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/11165
Committed: http://github.org/Juniper/contrail-controller/commit/b78f3671f1326cfda4a6fd336feeb5b0ff8b9982
Submitter: Zuul
Branch: master

commit b78f3671f1326cfda4a6fd336feeb5b0ff8b9982
Author: Prakash Bailkeri <email address hidden>
Date: Mon Jun 1 22:18:47 2015 -0700

HTTP Request size error in list-bulk-collection POST request

In scaled setup with large number of VMIs, neutron_plugin_db calls "list-bulk-collection"
with large number of duplicate VNs. This causes the http request size to increase

Fix: Add logic to prune duplicate VNs to limit the http request size

Change-Id: I5eefdab2e9b5bf2830150a5b2e64d8e47c88f676
Closes-bug: #1460675

Changed in juniperopenstack:
status: New → Fix Committed
Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/11164
Committed: http://github.org/Juniper/contrail-controller/commit/ad315ce27c3e128a2db13e6ce3a7036ac195465c
Submitter: Zuul
Branch: R2.20

commit ad315ce27c3e128a2db13e6ce3a7036ac195465c
Author: Prakash Bailkeri <email address hidden>
Date: Mon Jun 1 22:18:47 2015 -0700

HTTP Request size error in list-bulk-collection POST request

In scaled setup with large number of VMIs, neutron_plugin_db calls "list-bulk-collection"
with large number of duplicate VNs. This causes the http request size to increase

Fix: Add logic to prune duplicate VNs to limit the http request size

Change-Id: I5eefdab2e9b5bf2830150a5b2e64d8e47c88f676
Closes-bug: #1460675

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.