Requests lib causes connectionpool full error

Bug #1295812 reported by Iccha Sethi
44
This bug affects 9 people
Affects Status Importance Assigned to Milestone
python-swiftclient
In Progress
Undecided
Jesse J. Cook

Bug Description

Since glance started using python-swiftclient 2.0.8 it has been seeing WARNING requests.packages.urllib3.connectionpool [-] HttpConnectionPool is full, discarding connection: errors. This might be related to the requests lib change in 2.0.

https://github.com/openstack/python-swiftclient/commit/19d7e1812a99d73785146667ae2f3a7156f06898

This gives some detail about changing this value: http://stackoverflow.com/questions/18466079/can-i-change-the-connection-pool-size-for-pythons-requests-module
might be a useful fix.

Revision history for this message
Jordan Pittier (jordan-pittier) wrote :

I am also seeing this a lot

Changed in python-swiftclient:
status: New → Confirmed
Changed in python-swiftclient:
assignee: nobody → Jesse J. Cook (jesse-j-cook)
Changed in python-swiftclient:
status: Confirmed → In Progress
Revision history for this message
Matt Riedemann (mriedem) wrote :

Bug 1341777 might be a duplicate of this.

Revision history for this message
Matt Riedemann (mriedem) wrote :

There was a recent fix in python-novaclient for connection pooling:

https://github.com/openstack/python-novaclient/commit/98934d7bf1464afe0f7fe98efd2a591d95ac9c41

Maybe helpful, seems to be doing the same thing as mentioned here:

http://stackoverflow.com/questions/18466079/can-i-change-the-connection-pool-size-for-pythons-requests-module

Revision history for this message
Matt Riedemann (mriedem) wrote :

As noted in bug 1341777, I also checked python-swiftclient and it's hard-coding stream=True for the request regardless of content-type, which can be a problem with requests and connections not getting closed if the response content is not fully read or the response isn't properly closed by the client.

See the proposed fix for glanceclient to get an idea: https://review.openstack.org/#/c/107080/

Revision history for this message
Jesse J. Cook (jesse-j-cook) wrote :

This log message is being generated by urllib3 when a connection is being put back into the pool for reuse. If the pool is already full (at pool_maxsize), then the connection will be discarded. Many of these messages would indicate that the pool is not optimally sized.

In order to see this message, pool_block needs to be false. Otherwise, connections will not be created when the pool is empty. Therefore, you will not end up with more connections then the pool_maxsize and returning a connection to the pool can't exceed this limit.

So far, this indicates to me that the pool is sized inefficiently and a fix would be a resource utilization optimization.

Revision history for this message
Matt Riedemann (mriedem) wrote :

pool_block=False is the default, and the default pool size is 10.

Revision history for this message
Jesse J. Cook (jesse-j-cook) wrote :

Open question: What is the ideal default pool size?

Revision history for this message
Matt Riedemann (mriedem) wrote :

See some of my comments/notes in bug 1341777 about the pool size, I think we should work around the db max_connections value, which for mysql is 151 and for postgresql we set it to 200 in the gate since the default of 100 wasn't working.

Revision history for this message
Jesse J. Cook (jesse-j-cook) wrote :

https://urllib3.readthedocs.org/en/latest/pools.html

"""
A connection pool is a container for a collection of connections to a specific host.

If you need to make requests to the same host repeatedly, then you should use a HTTPConnectionPool.

By default, the pool will cache just one connection. If you’re planning on using such a pool in a multithreaded environment, you should set the maxsize of the pool to a higher number, such as the number of threads. You can also control many other variables like timeout, blocking, and default headers.
"""

http://docs.python-requests.org/en/latest/api/

"""
pool_connections – The number of urllib3 connection pools to cache.
pool_maxsize – The maximum number of connections to save in the pool.
"""

Based on this information, I would image the default pool_connections is probably sufficient. However, the pool_maxsize should be increased to roughly the number of threads. I need to research further, but I would guess that connections are created when needed (dependent on number that have been created and value of the pool_block argument) and put into the poll when done. If this is true, having a high pool_maxsize won't result in unnecessary connections being created, it just won't start discarding until maxsize is exceeded.

Revision history for this message
Jesse J. Cook (jesse-j-cook) wrote :

https://github.com/shazow/urllib3/blob/master/urllib3/connectionpool.py#106-110

https://github.com/shazow/urllib3/blob/master/urllib3/connectionpool.py#L195-219

https://github.com/shazow/urllib3/blob/master/urllib3/connectionpool.py#L168-169

https://github.com/shazow/urllib3/blob/0998c2c6a58085a096cbe6f4a9a2e67ce0c12ce7/urllib3/connectionpool.py#L209

https://github.com/shazow/urllib3/blob/0998c2c6a58085a096cbe6f4a9a2e67ce0c12ce7/urllib3/connectionpool.py#L231

The pool is a LifoQueue (https://docs.python.org/2/library/queue.html) initialized to maxsize None elements. A connection is obtained from the LifoQueue, and if it is new, a new connection is created. Setting a high maxsize won't result in a bunch of unnecessary connections. It just means the connections aren't discarded when pushing back into queue until the queue is full.

Revision history for this message
Jesse J. Cook (jesse-j-cook) wrote :

s/if it is new/if it is None/

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to python-swiftclient (master)

Fix proposed to branch: master
Review: https://review.openstack.org/116065

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on python-swiftclient (master)

Change abandoned by Christian Schwede (<email address hidden>) on branch: master
Review: https://review.openstack.org/116065
Reason: There has been no change to this patch since ~ 4 months, thus abandoning this patch to clear the list of incoming reviews.

Please feel free to restore this patch in case you are still working on it. Thank you!

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.