Comment 5 for bug 771512

Wasn't there some idea that this was because nova-api doesn't monkeypatch all of the libraries?

Vish

On May 4, 2011, at 1:58 PM, Soren Hansen wrote:

> I'm still blaming eventlet for this. Looking at the logs, it looks like
> the sheer volume of requests simply makes it take more than 5 seconds
> before the event loop in eventlet gets back to the attempt to establish
> a connection. We can optimise the time-per-request, but that'll won't
> solve anything, it'll just hide the problem until we try 100 or 200
> requests at a time. Same if we increase the timeout.
>
> --
> You received this bug notification because you are a member of Nova Bug
> Team, which is subscribed to OpenStack Compute (nova).
> https://bugs.launchpad.net/bugs/771512
>
> Title:
> Timeout from API with 50 Simultaneous Builds
>
> Status in OpenStack Compute (Nova):
> Confirmed
>
> Bug description:
> Running cactus, I attempted a build of 50 VMs at once via the API and
> ran into a timeout where 12 of the 50 VMs timed out and are stuck in
> build. The remaining went active.
>
> for i in `seq 1 50`; do nova boot test --flavor 1 --image 10 & >
> /dev/null ; echo $i ; done
>
> API Limits were raised well past this in limits.py for stress testing.
>
> 2011-04-26 17:36:51,852 ERROR nova.api.openstack [-] Caught error: timed out
> (nova.api.openstack): TRACE: Traceback (most recent call last):
> (nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/nova/api/openstack/__init__.py", line 59, in __call__
> (nova.api.openstack): TRACE: return req.get_response(self.application)
> (nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/webob/request.py", line 919, in get_response
> (nova.api.openstack): TRACE: application, catch_exc_info=False)
> (nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/webob/request.py", line 887, in call_application
> (nova.api.openstack): TRACE: app_iter = application(self.environ, start_response)
> (nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 159, in __call__
> (nova.api.openstack): TRACE: return resp(environ, start_response)
> (nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 159, in __call__
> (nova.api.openstack): TRACE: return resp(environ, start_response)
> (nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 159, in __call__
> (nova.api.openstack): TRACE: return resp(environ, start_response)
> (nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/routes/middleware.py", line 131, in __call__
> (nova.api.openstack): TRACE: response = self.app(environ, start_response)
> (nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 159, in __call__
> (nova.api.openstack): TRACE: return resp(environ, start_response)
> (nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 147, in __call__
> (nova.api.openstack): TRACE: resp = self.call_func(req, *args, **self.kwargs)
> (nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/webob/dec.py", line 208, in call_func
> (nova.api.openstack): TRACE: return self.func(req, *args, **kwargs)
> (nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/nova/wsgi.py", line 356, in __call__
> (nova.api.openstack): TRACE: result = method(**arg_dict)
> (nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/nova/api/openstack/servers.py", line 171, in create
> (nova.api.openstack): TRACE: injected_files=injected_files)
> (nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/nova/compute/api.py", line 256, in create
> (nova.api.openstack): TRACE: "injected_files": injected_files}})
> (nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/nova/rpc.py", line 394, in cast
> (nova.api.openstack): TRACE: publisher = TopicPublisher(connection=conn, topic=topic)
> (nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/nova/rpc.py", line 241, in __init__
> (nova.api.openstack): TRACE: super(TopicPublisher, self).__init__(connection=connection)
> (nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/carrot/messaging.py", line 653, in __init__
> (nova.api.openstack): TRACE: self.declare()
> (nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/carrot/messaging.py", line 664, in declare
> (nova.api.openstack): TRACE: auto_delete=self.auto_delete)
> (nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/carrot/backends/pyamqplib.py", line 240, in exchange_declare
> (nova.api.openstack): TRACE: return self.channel.exchange_declare(exchange=exchange,
> (nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/carrot/backends/pyamqplib.py", line 179, in channel
> (nova.api.openstack): TRACE: self._channel_ref = weakref.ref(self.connection.get_channel())
> (nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/carrot/connection.py", line 150, in get_channel
> (nova.api.openstack): TRACE: return self.connection.channel()
> (nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/carrot/connection.py", line 120, in connection
> (nova.api.openstack): TRACE: self._connection = self._establish_connection()
> (nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/carrot/connection.py", line 133, in _establish_connection
> (nova.api.openstack): TRACE: return self.create_backend().establish_connection()
> (nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/carrot/backends/pyamqplib.py", line 195, in establish_connection
> (nova.api.openstack): TRACE: connect_timeout=conninfo.connect_timeout)
> (nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/amqplib/client_0_8/connection.py", line 125, in __init__
> (nova.api.openstack): TRACE: self.transport = create_transport(host, connect_timeout, ssl)
> (nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/amqplib/client_0_8/transport.py", line 220, in create_transport
> (nova.api.openstack): TRACE: return TCPTransport(host, connect_timeout)
> (nova.api.openstack): TRACE: File "/usr/lib/pymodules/python2.6/amqplib/client_0_8/transport.py", line 58, in __init__
> (nova.api.openstack): TRACE: self.sock.connect((host, port))
> (nova.api.openstack): TRACE: File "/usr/lib/python2.6/dist-packages/eventlet/greenio.py", line 178, in connect
> (nova.api.openstack): TRACE: timeout_exc=socket.timeout("timed out"))
> (nova.api.openstack): TRACE: File "/usr/lib/python2.6/dist-packages/eventlet/hubs/__init__.py", line 121, in trampoline
> (nova.api.openstack): TRACE: return hub.switch()
> (nova.api.openstack): TRACE: File "/usr/lib/python2.6/dist-packages/eventlet/hubs/hub.py", line 177, in switch
> (nova.api.openstack): TRACE: return self.greenlet.switch()
> (nova.api.openstack): TRACE: timeout: timed out
> (nova.api.openstack): TRACE: