redis server not starting after provisoning Ubuntu14.04 With HA

Bug #1402466 reported by Vinod Nair
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R2.0
Fix Released
Critical
Raj Reddy
Trunk
Fix Released
Critical
Raj Reddy

Bug Description

Redis server not starting after provisioning on a HA cluster.

Image 2.0 Build 16 , Icehouse Ubuntu 14.04

From the setup_all log , see that the server is started , but when we check the host , it is not running

grep -r redis-server setup_all_2014_12_16_52_09_989976.log
2014-12-14 17:12:34:975378: [root@10.87.141.1] run: service redis-server stop
2014-12-14 17:12:34:975991: [root@10.87.141.1] out: Stopping redis-server: redis-server.
2014-12-14 17:12:36:077639: [root@10.87.141.1] run: service redis-server start
2014-12-14 17:12:36:136209: [root@10.87.141.1] out: Starting redis-server: redis-server.
2014-12-14 17:12:38:617522: [root@10.87.141.2] run: service redis-server stop
2014-12-14 17:12:38:617855: [root@10.87.141.2] out: Stopping redis-server: redis-server.
2014-12-14 17:12:39:771678: [root@10.87.141.2] run: service redis-server start
2014-12-14 17:12:39:826538: [root@10.87.141.2] out: Starting redis-server: redis-server.
2014-12-14 17:12:43:236943: [root@10.87.141.3] run: service redis-server stop
2014-12-14 17:12:43:237311: [root@10.87.141.3] out: Stopping redis-server: redis-server.
2014-12-14 17:12:44:422999: [root@10.87.141.3] run: service redis-server start
2014-12-14 17:12:44:474398: [root@10.87.141.3] out: Starting redis-server: redis-server.

root@cs-scale-1:/opt/contrail/utils# service redis-server status
redis-server is not running

root@cs-scale-1:/opt/contrail/utils# contrail-status
== Contrail Control ==
supervisor-control: active
contrail-control active
contrail-control-nodemgr active
contrail-dns active
contrail-named active

== Contrail Analytics ==
supervisor-analytics: active
contrail-analytics-api initializing (Collector connection down)
contrail-analytics-nodemgr active
contrail-collector initializing (Collector, Redis:From, Redis:To connection down)
contrail-query-engine initializing (Number of connections:2, Expected: 3)

== Contrail Config ==
supervisor-config: active
contrail-api:0 active
contrail-config-nodemgr active
contrail-discovery:0 active
contrail-schema initializing (Collector connection down)
contrail-svc-monitor active
ifmap active

== Contrail Web UI ==
supervisor-webui: active
contrail-webui failed
contrail-webui-middleware failed

== Contrail Database ==
supervisor-database: active
contrail-database active

== Contrail Support Services ==
supervisor-support-service: active
rabbitmq-server active

redis-server log shows that it is getting shutdown

31034] 14 Dec 16:45:33.448 # Server started, Redis version 2.6.13
[31034] 14 Dec 16:45:33.448 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
[31034] 14 Dec 16:45:33.448 * The server is now ready to accept connections on port 6379
[31034 | signal handler] (1418605955) Received SIGTERM, scheduling shutdown...
[31034] 14 Dec 17:12:35.047 # User requested shutdown...
[31034] 14 Dec 17:12:35.047 * Saving the final RDB snapshot before exiting.
[31034] 14 Dec 17:12:35.048 * DB saved on disk
[31034] 14 Dec 17:12:35.048 * Removing the pid file.
[31034] 14 Dec 17:12:35.048 # Redis is now ready to exit, bye bye...

Vinod Nair (vinodnair)
description: updated
information type: Proprietary → Public
tags: added: releasenote
Revision history for this message
Raj Reddy (rajreddy) wrote :

sometimes it seems 'service redis-server start' that's called immediately after 'stop' is not taking effect

workaround is to check 'service redis-server status' after provisioning and do
 'service redis-server start' as needed after provisioning is done

tags: added: analytics ha provisioning
Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/5774
Committed: http://github.org/Juniper/contrail-fabric-utils/commit/475b93865cbb4ff1afa78acd0f6ee83d8092ca3d
Submitter: Zuul
Branch: R2.0

commit 475b93865cbb4ff1afa78acd0f6ee83d8092ca3d
Author: Raj Reddy <email address hidden>
Date: Thu Dec 18 10:49:32 2014 -0800

we have noticed that sometimes on ubuntu 14.04, 'service redis-server start' is
not starting the redis service, so added a check and a restart in such cases

Change-Id: I6c0065704ddb1547b9fc383ef6de7d749ab9e448
Closes-Bug: #1402466

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/5796
Committed: http://github.org/Juniper/contrail-fabric-utils/commit/89d07dc8f301fdda3699290af3b6cc31190e20ef
Submitter: Zuul
Branch: master

commit 89d07dc8f301fdda3699290af3b6cc31190e20ef
Author: Raj Reddy <email address hidden>
Date: Thu Dec 18 10:49:32 2014 -0800

we have noticed that sometimes on ubuntu 14.04, 'service redis-server start' is
not starting the redis service, so added a check and a restart in such cases

Closes-Bug: #1402466

Conflicts:

 fabfile/tasks/provision.py

Change-Id: I6c0065704ddb1547b9fc383ef6de7d749ab9e448

Changed in juniperopenstack:
status: New → Fix Committed
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.