Api hangs if it doesn't receive an ack from conductor
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Magnum |
Confirmed
|
Medium
|
Surojit Pathak |
Bug Description
API hangs when waiting to receive ack from Conductor when unable to send message to RabbitMQ
Test infrastructure:
$ uname -a
Linux containers-test 3.13.0-62-generic #102-Ubuntu SMP Tue Aug 11
14:29:36 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
$ lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 40
On-line CPU(s) list: 0-39
Thread(s) per core: 2
Core(s) per socket: 10
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 62
Stepping: 4
CPU MHz: 2800.000
BogoMIPS: 5601.67
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 25600K
NUMA node0 CPU(s): 0-9,20-29
NUMA node1 CPU(s): 10-19,30-39
$ vmstat
procs -------
------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id
wa st
5 0 0 95732384 315932 19664548 0 0 11 10 0 0 7
0 93 0 0
Test setup:
[magnum]
image_id = fedora-21-atomic-3
nic_id = public
keypair_id = default
flavor_id = m1.small
NOTES:
I did notice open sockets leaking on this environment (RabbitMQ setup out of the box - mismanaged).
$ lsof -i | grep amqp | wc -l
1027
$ lsof -i | grep amqp | grep magnum | wc -l
652
I asked others to reproduce and those utilizing bare metal servers were
able to see similar behavior. Digging in further, it looks like
oslo-messaging opens up a new connection every time it attempts to send a
message to rabbit, even though rabbit might not be available to receive
the message (for whatever reason)
$ sudo rabbitmqctl status
Status of node 'rabbit@
[{pid,2011},
{running_
{file_descripto
{processes,
{run_queue,0},
{uptime,510653}]
As you can see here, sockets are not cleaned up but the run_queue is
empty.
Changed in magnum: | |
status: | New → Confirmed |
importance: | Undecided → Medium |
Changed in magnum: | |
milestone: | none → mitaka-1 |
Changed in magnum: | |
assignee: | nobody → Surojit Pathak (suro-patz) |
Fix proposed to branch: master /review. openstack. org/264333
Review: https:/