CI -- Jobs some times fail due to openstack-nova-compute 'dead'

Bug #1410992 reported by Ananth Suryanarayana
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
New
Undecided
Unassigned

Bug Description

e.g. https://jenkins.opencontrail.org/job/ci-contrail-server-manager-systest-ubuntu-precise-pangolin-havana/231/console

2015-01-13 18:40:01 /usr/local/jenkins/slave_scripts/contrail-systest-job.rb:508:in `upto'() /usr/local/jenkins/slave_scripts/contrail-systest-job.rb:508:in `verify_contrail'() ssh ci-oc-subslave-ubuntu-10-84-35-136-1.localdomain.com openstack-status:
2015-01-13 18:40:03 == Nova services ==
2015-01-13 18:40:04 openstack-nova-api: active
2015-01-13 18:40:05 openstack-nova-compute: dead
2015-01-13 18:40:05 openstack-nova-network: inactive (disabled on boot)
2015-01-13 18:40:05 openstack-nova-scheduler: active
2015-01-13 18:40:05 openstack-nova-volume: inactive (disabled on boot)
2015-01-13 18:40:06 openstack-nova-conductor: active
2015-01-13 18:40:06 == Glance services ==
2015-01-13 18:40:07 openstack-glance-api: active
2015-01-13 18:40:08 openstack-glance-registry: active
2015-01-13 18:40:08 == Keystone service ==
2015-01-13 18:40:09 openstack-keystone: active
2015-01-13 18:40:09 == Cinder services ==
2015-01-13 18:40:10 openstack-cinder-api: active
2015-01-13 18:40:10 openstack-cinder-scheduler: active
2015-01-13 18:40:10 openstack-cinder-volume: inactive (disabled on boot)
2015-01-13 18:40:10 == Support services ==
2015-01-13 18:40:10 mysql: inactive (disabled on boot)
2015-01-13 18:40:11 libvirt-bin: active
2015-01-13 18:40:11 rabbitmq-server: active
2015-01-13 18:40:11 memcached: inactive (disabled on boot)
2015-01-13 18:40:11 == Keystone users ==

VM logs at /ci-admin/failed_systest_logs/ci-contrail-server-manager-systest-ubuntu-precise-pangolin-havana/231/.

Problem is likely with the AMQP server. nova-compute cannot talk to nova-conductor, which cannot talk to AMQP Server

tail /build/ci-admin/failed_systest_logs/ci-contrail-server-manager-systest-ubuntu-precise-pangolin-havana/231/var/log/nova/nova-conductor.log
2015-01-13 18:27:13.865 27590 ERROR nova.openstack.common.rpc.common [-] AMQP server on 192.168.255.180:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 21 seconds.
2015-01-13 18:27:34.671 27591 INFO nova.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.255.180:5672
2015-01-13 18:27:34.685 27591 INFO nova.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.255.180:5672
2015-01-13 18:27:34.868 27590 INFO nova.openstack.common.rpc.common [-] Reconnecting to AMQP server on 192.168.255.180:5672
2015-01-13 18:27:34.884 27590 INFO nova.openstack.common.rpc.common [-] Connected to AMQP server on 192.168.255.180:5672
2015-01-13 18:27:43.600 27591 INFO nova.openstack.common.rpc.common [req-3f467fa6-f21c-4206-92e8-5e6941f10eea None None] Connected to AMQP server on 192.168.255.180:5672
2015-01-13 18:28:23.839 27591 WARNING nova.openstack.common.loopingcall [-] task run outlasted interval by 29.133822 sec
2015-01-13 18:28:23.841 27590 WARNING nova.openstack.common.loopingcall [-] task run outlasted interval by 28.937576 sec
2015-01-13 18:28:53.440 27591 WARNING nova.openstack.common.loopingcall [-] task run outlasted interval by 19.142471 sec
2015-01-13 18:29:00.134 27590 INFO nova.openstack.common.rpc.common [req-bcc70916-216d-4332-afb3-3d46ef2ca7c2 None None] Connected to AMQP server on 192.168.255.180:5672

Tags: ci openstack
information type: Proprietary → Public
tags: added: openstack vrouter
tags: removed: vrouter
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.