SWIFT request hangs as "oslo_messaging.Notifier.info()" interface does not returns in some failure scenarios
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
ceilometermiddleware |
New
|
Undecided
|
Unassigned | ||
oslo.messaging |
Incomplete
|
Undecided
|
Unassigned |
Bug Description
When Ceilometer Middleware is integrated with SWIFT, some of user requests hangs at ceilometer level.
It is observed that with default max pool size of 30, in case if 31 requests are received at oslo_messaging, the hung behaviour (indefinite hung) is observed.
When we debugged firther, to identify where the requests are stuck, it is observed that the following call of oslo_messaging Notifier class does not return.
File: https:/
{code}
47 import oslo_messaging
....
121 self._notifier = oslo_messaging.
122 oslo_messaging.
123 publisher_
124 driver=
125 topic=conf.
126
....
283
284 self._notifier.
285 'objectstore.
286
{code}
the call "self._
It is confirmed that there is no problem with RabbitMQ server or ceilometer server as both of them are accessible from other SWIFT cluster at the time problem is observed on different setup.
When conformed from network packet dump, it is seen that ceilometer middleware does not sends any data(metering information) to rabbitMQ server.
It is thus confirmed that the request is stuck somewhere in oslo_messaging notifier API call.
Also, it's effect is seen in Proxy Service as no new user requests are handled once any of the request is stuck in such a manner.
On restart of Proxy Service, the problem seems to be resolved temporarily.
description: | updated |
description: | updated |
description: | updated |
description: | updated |
Changed in oslo.messaging: | |
status: | New → Incomplete |
Hi Sanchit,
Which version of oslo.messaging are you using?
Any chance you can get a traceback of the hung process? Like via pstack?
Can you get a dump of rabbitmq queue counters by running the following commands *after* the hang occurs?
$ sudo rabbitmqctl list_queues unacknowledged
$ sudo rabbitmqctl list_queues messages_
And lastly can you capture logs (debug level and up) from the hung service?
thanks