[ubuntu-havana-R1.06-40] contrail-database service failure

Bug #1330059 reported by Prashant Shetty
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Fix Released
Critical
Megh Bhatt

Bug Description

Team,

Contrail-database is in failure state on all 3 nodes with latest #40. Can someone look into?

Provision logs: http://anamika.englab.juniper.net:8080/job/ubuntu-12-04_havana_Multi_Node_Sanity/251/consoleFull

CFGM : [u'a6s32', u'a6s33']
Control Nodes : [u'a6s33', u'a6s38']
Compute Nodes : [u'a6s19', u'a6s22']
Collector : [u'a6s32']
WebUI : a6s32
OpenstackUI : a6s33
Logs:

2014-06-14 04:36:45,954 CRIT Supervisor running as root (no user in config file)
2014-06-14 04:36:45,954 INFO Increased RLIMIT_NOFILE limit to 10240
2014-06-14 04:36:45,971 INFO RPC interface 'supervisor' initialized
2014-06-14 04:36:45,972 CRIT Server 'inet_http_server' running without any HTTP authentication checking
2014-06-14 04:36:45,972 INFO RPC interface 'supervisor' initialized
2014-06-14 04:36:45,972 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2014-06-14 04:36:45,972 INFO supervisord started with pid 12636
2014-06-14 04:36:46,975 INFO spawned: 'contrail-database-nodemgr' with pid 12658
2014-06-14 04:36:46,976 INFO spawned: 'contrail-database' with pid 12659
2014-06-14 04:36:47,567 INFO exited: contrail-database (exit status 1; not expected)
2014-06-14 04:36:48,158 INFO success: contrail-database-nodemgr entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2014-06-14 04:36:49,160 INFO spawned: 'contrail-database' with pid 12720
2014-06-14 04:36:49,737 INFO exited: contrail-database (exit status 1; not expected)
2014-06-14 04:36:52,159 INFO spawned: 'contrail-database' with pid 12779
2014-06-14 04:36:52,742 INFO exited: contrail-database (exit status 1; not expected)
2014-06-14 04:36:56,168 INFO spawned: 'contrail-database' with pid 12835
2014-06-14 04:36:56,742 INFO exited: contrail-database (exit status 1; not expected)
2014-06-14 04:36:57,168 INFO gave up: contrail-database entered FATAL state, too many start retries too quickly

2014-06-14 Sat 05:21:07:056.910 PDT a6s32 [Thread 139732275201792, Pid 14629]: a6s32:Analytics:Collector:0: StatTableInsert: Addition of FieldNames, fields attrib Source into table StatsTableByStrStrTag FAILED
2014-06-14 Sat 05:21:07:056.942 PDT a6s32 [Thread 139732275201792, Pid 14629]: a6s32:Analytics:Collector:0: StatTableInsert: Addition of FieldNames, fields attrib name into table StatsTableByStrStrTag FAILED
2014-06-14 Sat 05:21:07:057.002 PDT a6s32 [Thread 139732275201792, Pid 14629]: a6s32:Analytics:Collector:0: ObjectTableInsert: Addition of a6s32:Analytics:Collector:0, message UUID 4b59c3a1-e50a-4d43-b05b-c4b350db62df into table ObjectGeneratorInfo FAILED
2014-06-14 Sat 05:21:07:175.249 PDT a6s32 [Thread 139732197570304, Pid 14629]: a6s32:Global: Initialize: Initializing..
2014-06-14 Sat 05:21:07:175.249 PDT a6s32 [Thread 139732275201792, Pid 14629]: a6s32:Analytics:Collector:0: Setup: Setup..
2014-06-14 Sat 05:21:07:175.586 PDT a6s32 [Thread 139732197570304, Pid 14629]: a6s32:Global: Db_Init:controller/src/gendb/cdb_if.cc:662: analytics::DbHandler:-1: TTransportException: connect() failed: Connection refused
2014-06-14 Sat 05:21:07:175.630 PDT a6s32 [Thread 139732197570304, Pid 14629]: a6s32:Global: Initialize: Connection to DB FAILED
2014-06-14 Sat 05:21:07:175.653 PDT a6s32 [Thread 139732197570304, Pid 14629]: Init DB Handler initialization failed
2014-06-14 Sat 05:21:07:175.598 PDT a6s32 [Thread 139732275201792, Pid 14629]: a6s32:Analytics:Collector:0: Db_Init:controller/src/gendb/cdb_if.cc:662: analytics::DbHandler:0: TTransportException: connect() failed: Connection refused
2014-06-14 Sat 05:21:07:175.981 PDT a6s32 [Thread 139732275201792, Pid 14629]: a6s32:Analytics:Collector:0: Setup: Connection to DB FAILED
2014-06-14 Sat 05:21:07:176.007 PDT a6s32 [Thread 139732275201792, Pid 14629]: a6s32:Analytics:Collector:0: Db_Connection_Init: : Database setup FAILED

root@a6s32:~# contrail-status
vRouter is NOT PRESENT
vRouter agent is NOT PRESENT
== Contrail Analytics ==
supervisor-analytics: active
contrail-analytics-nodemgr active
contrail-collector active
contrail-opserver active
contrail-qe active
redis-query active
redis-uve active
== Contrail Config ==
supervisor-config: active
contrail-api:0 active
contrail-config-nodemgr active
contrail-discovery:0 active
contrail-schema active
contrail-svc-monitor active
ifmap active
== Contrail Web UI ==
supervisor-webui: active
contrail-webui active
contrail-webui-middleware active
redis-webui active
== Contrail Database ==
supervisord-contrail-database:active
contrail-database failed
contrail-database-nodemgr active
root@a6s32:~#

Changed in juniperopenstack:
assignee: nobody → Raj Reddy (rajreddy)
Changed in juniperopenstack:
assignee: Raj Reddy (rajreddy) → Megh Bhatt (meghb)
tags: added: blocker
Revision history for this message
Prashant Shetty (shettyp) wrote :

  Branch: refs/heads/R1.06
  Home: https://github.com/Juniper/contrail-packaging
  Commit: bd2c467ab19066aa26287a8216e9737d3f4ac53c
      https://github.com/Juniper/contrail-packaging/commit/bd2c467ab19066aa26287a8216e9737d3f4ac53c
  Author: Raj Reddy <email address hidden>
  Date: 2014-06-13 (Fri, 13 Jun 2014)

  Changed paths:
    M common/debian/contrail-web-core/debian/rules

  Log Message:
  -----------
  use Juniper logo for Juniper installations

  Commit: 6ddea5aa0c6c8b40dd6b4566b629ed693c3c8c96
      https://github.com/Juniper/contrail-packaging/commit/6ddea5aa0c6c8b40dd6b4566b629ed693c3c8c96
  Author: Megh Bhatt <email address hidden>
  Date: 2014-06-14 (Sat, 14 Jun 2014)

  Changed paths:
    M build/libs/packager/common.py

  Log Message:
  -----------
  Merge branch 'R1.06' of github.com:Juniper/contrail-packaging into R1.06

  Commit: 45c5d416fc8f30c3ccba01ca7dc2c18db840365f
      https://github.com/Juniper/contrail-packaging/commit/45c5d416fc8f30c3ccba01ca7dc2c18db840365f
  Author: Megh Bhatt <email address hidden>
  Date: 2014-06-14 (Sat, 14 Jun 2014)

  Changed paths:
    M common/debian/contrail-openstack-database/debian/contrail-openstack-database.postinst

  Log Message:
  -----------
  Fix issue with previous commit to remove /var/lib/cassandra
and /etc/init.d/cassandra only on install and not on
upgrade. postinst script is called with configure option.

Changed in juniperopenstack:
status: New → Fix Committed
information type: Proprietary → Public
Raj Reddy (rajreddy)
Changed in juniperopenstack:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.