From: Sundaresan Rajangam
Sent: Wednesday, August 23, 2017 12:07 PM
To: Usha Rani Mani
Cc: Naga Kiran K Y S; Anish Mehta; Manoj Naik; Biswajit Mandal; Megh Bhatt
Subject: Re: Reg: https://bugs.launchpad.net/juniperopenstack/+bug/1711366
 
+Megh

From the fab log, I see that redis was started twice during provisioning time

setup_all_2017_08_22_17_04_57_009210.log:2017-08-22 17:11:44:340936: [root@10.204.217.81] sudo: service redis start
setup_all_2017_08_22_17_04_57_009210.log:2017-08-22 17:12:12:420994: [root@10.204.217.81] sudo: service redis start

The corresponding logs from redis.log:

[27217] 22 Aug 17:11:44.512 # Server started, Redis version 2.8.19
[28824] 22 Aug 17:12:12.483 # Server started, Redis version 2.8.19

However, the system was rebooted @ 17:17

[root@nodec41 fabric-utils]# last reboot
reboot   system boot  3.10.0-229.el7.x Tue Aug 22 17:17 - 10:45  (17:27)
reboot   system boot  3.10.0-229.el7.x Tue Aug 22 17:04 - 10:45  (17:40)
reboot   system boot  3.10.0-229.el7.x Tue Aug 22 16:49 - 10:45  (17:55)

From the /var/log/messages, it seems that redis started before the network was up and hence failed to start.
redis service was not started later after the network was up.

Aug 22 17:17:39 nodec41 kernel: IPv6: ADDRCONF(NETDEV_UP): p1p0p0: link is not ready
Aug 22 17:17:40 nodec41 network: Bringing up interface p1p0p0:  [  OK  ]
Aug 22 17:17:40 nodec41 network: Bringing up interface vhost0:  ERROR    : [/etc/sysconfig/network-scripts/ifup-eth] Device vhost0 does not seem to be present, delaying initialization.
Aug 22 17:17:41 nodec41 /etc/sysconfig/network-scripts/ifup-eth: Device vhost0 does not seem to be present, delaying initialization. <<<<<<<
Aug 22 17:17:41 nodec41 network: [FAILED]  <<<<<<<< 
Aug 22 17:17:41 nodec41 systemd-sysctl: Overwriting earlier assignment of net/ipv4/ip_local_reserved_ports in file '/usr/lib/sysctl.d/openstack-keystone.conf'.
Aug 22 17:17:41 nodec41 systemd-sysctl: Overwriting earlier assignment of net/ipv4/ip_local_reserved_ports in file '/etc/sysctl.conf'.
Aug 22 17:17:41 nodec41 systemd: network.service: control process exited, code=exited status=1    <<<<<<<<<<<<
Aug 22 17:17:41 nodec41 systemd: Failed to start LSB: Bring up/down networking.
Aug 22 17:17:41 nodec41 systemd: Starting Redis persistent key-value database…  <<<<<<<<<<<
Aug 22 17:17:41 nodec41 systemd: Started Redis persistent key-value database.   <<<<<
Aug 22 17:17:43 nodec41 redis-shutdown: Could not connect to Redis at 10.204.217.81:6379: Network is unreachable   <<<<<<<
Aug 22 17:17:43 nodec41 systemd: redis.service: control process exited, code=exited status=1
Aug 22 17:17:43 nodec41 systemd: Unit redis.service entered failed state.
...
Aug 22 17:18:06 nodec41 systemd-sysctl: Overwriting earlier assignment of net/ipv4/ip_local_reserved_ports in file '/etc/sysctl.conf'.
Aug 22 17:18:06 nodec41 NetworkManager[704]: <info>  (vhost0): carrier is OFF
Aug 22 17:18:06 nodec41 NetworkManager[704]: <info>  (vhost0): new Generic device (driver: 'vhost' ifindex: 7)
Aug 22 17:18:06 nodec41 NetworkManager[704]: <info>  (vhost0): exported as /org/freedesktop/NetworkManager/Devices/6
Aug 22 17:18:06 nodec41 NetworkManager[704]: <info>  (vhost0): link connected

Megh,

“Restart” option is not set in redis.service
Shouldn’t the Restart=on-failure be set in redis.service?

Thanks,
Sundar