rabbitMQ locks all services for 10 secs after creating instance snapshot

Bug #1357414 reported by Diogo Monteiro
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Glance
New
Undecided
Unassigned

Bug Description

Steps to reproduce the problem.
#1 Create a private template from a running virtual instance
#2 Create a new virtual instance using the private template

Result:
All openstack services lock for about 10 secs -- api server stops responding to any requests.

The problem is always reproducible.

Error logs:

1. Create a private template from a running virtual instance
2. Create a new virtual instance using the private template

Observations:

RabbitMQ Logs:

=WARNING REPORT==== 15-Aug-2014::15:29:46 ===
exception on TCP connection <0.13976.3> from 127.0.0.1:52700
connection_closed_abruptly

=INFO REPORT==== 15-Aug-2014::15:29:46 ===
closing TCP connection <0.13976.3> from 127.0.0.1:52700

=WARNING REPORT==== 15-Aug-2014::15:29:46 ===
exception on TCP connection <0.13980.3> from 127.0.0.1:52701
connection_closed_abruptly

=INFO REPORT==== 15-Aug-2014::15:29:46 ===
closing TCP connection <0.13980.3> from 127.0.0.1:52701

In nova-conductor

2014-08-15 09:30:03.677 22592 WARNING nova.openstack.common.loopingcall [-] task run outlasted interval by 0.119076 sec
2014-08-15 09:30:14.506 22592 WARNING nova.openstack.common.loopingcall [-] task run outlasted interval by 0.823113 sec
2014-08-15 09:33:59.423 22592 WARNING nova.openstack.common.loopingcall [-] task run outlasted interval by 0.470631 sec
2014-08-15 14:01:12.217 30924 WARNING nova.openstack.common.loopingcall [-] task run outlasted interval by 2.697045 sec
2014-08-15 14:06:10.072 30924 WARNING nova.openstack.common.loopingcall [-] task run outlasted interval by 0.716281 sec
2014-08-15 14:33:08.902 51990 WARNING nova.openstack.common.loopingcall [-] task run outlasted interval by 2.419363 sec

Overview of the environment:
OS: 3.8.0-35-generic #50~precise1-Ubuntu SMP Wed Dec 4 17:25:51 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

Openstack version: havana

Services:
nova - 2.15.0
neutron - 2.3.0
glance - 0.11.0
cinder - 1.0.6
keystone - 0.3.2

RabbitMQ
 {running_applications,[{rabbit,"RabbitMQ","2.7.1"},
                        {os_mon,"CPO CXC 138 46","2.2.7"},
                        {sasl,"SASL CXC 138 11","2.1.10"},
                        {mnesia,"MNESIA CXC 138 12","4.5"},
                        {stdlib,"ERTS CXC 138 10","1.17.5"},
                        {kernel,"ERTS CXC 138 10","2.14.5"}]},
 {os,{unix,linux}},
 {erlang_version,"Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:8:8] [rq:8] [async-threads:30] [kernel-poll:true]\n"},
 {memory,[{total,30703760},
          {processes,14640352},
          {processes_used,14630712},
          {system,16063408},
          {atom,1124441},
          {atom_used,1120308},
          {binary,478496},
          {code,11134417},
          {ets,986440}]},
 {vm_memory_high_watermark,0.39999999997532165},
 {vm_memory_limit,3241710387}]

description: updated
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.