[R4.1]: Analytics down on fresh provisioning due to zookeeper not coming up

Build : R4.1 - 48
CB Build

Setup:
K8s contrail-ansible setup
3 node setup

1 node having following roles: k8s master, controller, analytics, analyticsdb
2 nodes as k8s slaves and agent

Summary:
Zookeeper inside analyticsdb tries to bind to 2181 and gets an exception:
java.net.BindException: Address already in use

This is because the zookeeper running outside, also binds to the same port with value 2181
Thus, internal zookeeper binding fails.
Surprisingly, the zoo.conf is having port value as 2182 but still it tries to connect to 2181 during provisioning.
Seems like config file(zoo.conf) get updated correctly later during provisioning.

Workaround:
As zoo.conf is correct and tries to bind to 2182, restarting the zookeeper corrected all the problems.

Logs:
Zookeeper:
2017-11-15 14:18:12,262 - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:2181
2017-11-15 14:18:12,264 - ERROR [main:ZooKeeperServerMain@63] - Unexpected exception, exiting abnormally
java.net.BindException: Address already in use
        at sun.nio.ch.Net.bind0(Native Method)
        at sun.nio.ch.Net.bind(Net.java:433)
        at sun.nio.ch.Net.bind(Net.java:425)
        at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
        at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
        at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:67)
        at org.apache.zookeeper.server.NIOServerCnxnFactory.configure(NIOServerCnxnFactory.java:90)
        at org.apache.zookeeper.server.ZooKeeperServerMain.runFromConfig(ZooKeeperServerMain.java:111)
        at org.apache.zookeeper.server.ZooKeeperServerMain.initializeAndRun(ZooKeeperServerMain.java:86)
        at org.apache.zookeeper.server.ZooKeeperServerMain.main(ZooKeeperServerMain.java:52)
        at org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:116)
        at org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
2017-11-15 14:18:24,985 - INFO [main:QuorumPeerConfig@103] - Reading configuration from: /etc/zookeeper/conf/zoo.cfg

Status of Zookeeper:
root@testbed-1-vm1(analyticsdb):/var/log/zookeeper# service zookeeper status
● zookeeper.service - LSB: centralized coordination service
   Loaded: loaded (/etc/init.d/zookeeper; bad; vendor preset: enabled)
   Active: active (exited) since Wed 2017-11-15 14:18:24 IST; 1h 25min ago
     Docs: man:systemd-sysv-generator(8)

Nov 15 14:18:24 testbed-1-vm1 systemd[1]: Stopped LSB: centralized coordination service.
Nov 15 14:18:24 testbed-1-vm1 systemd[1]: Starting LSB: centralized coordination service...
Nov 15 14:18:24 testbed-1-vm1 systemd[1]: Started LSB: centralized coordination service.

Status of Kafka:
root@testbed-1-vm1(analyticsdb):/# service kafka status
● kafka.service - kafka
   Loaded: loaded (/lib/systemd/system/kafka.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2017-11-15 15:38:43 IST; 1s ago
  Process: 25047 ExecStop=/usr/share/kafka/bin/kafka-server-stop.sh /usr/share/kafka/config/server.properties (code=exited, status=0/SUCCESS)
 Main PID: 25129 (java)
   CGroup: /docker/61fd8808a1bc6d384a1674b0b5b3b466979f4485b1214d796320948f4dc7c206/system.slice/kafka.service
           └─25129 java -Xmx1G -Xms1G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+DisableExplicitGC -Djava.awt.headless=t
           ‣ 25129 java -Xmx1G -Xms1G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+DisableExplicitGC -Djava.awt.headless=t

Nov 15 15:38:45 testbed-1-vm1 kafka-server-start.sh[25129]: [2017-11-15 15:38:45,039] INFO Client environment:user.dir=/ (org.apache.zookeeper.ZooKeeper)
Nov 15 15:38:45 testbed-1-vm1 kafka-server-start.sh[25129]: [2017-11-15 15:38:45,040] INFO Initiating client connection, connectString=10.204.217.194:2182 sessionTim
Nov 15 15:38:45 testbed-1-vm1 kafka-server-start.sh[25129]: [2017-11-15 15:38:45,057] INFO Waiting for keeper state SyncConnected (org.I0Itec.zkclient.ZkClient)
Nov 15 15:38:45 testbed-1-vm1 kafka-server-start.sh[25129]: [2017-11-15 15:38:45,061] INFO Opening socket connection to server 10.204.217.194/10.204.217.194:2182. Wi
Nov 15 15:38:45 testbed-1-vm1 kafka-server-start.sh[25129]: [2017-11-15 15:38:45,069] WARN Session 0x0 for server null, unexpected error, closing socket connection a
Nov 15 15:38:45 testbed-1-vm1 kafka-server-start.sh[25129]: java.net.ConnectException: Connection refused
Nov 15 15:38:45 testbed-1-vm1 kafka-server-start.sh[25129]: at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
Nov 15 15:38:45 testbed-1-vm1 kafka-server-start.sh[25129]: at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
Nov 15 15:38:45 testbed-1-vm1 kafka-server-start.sh[25129]: at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
Nov 15 15:38:45 testbed-1-vm1 kafka-server-start.sh[25129]: at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
root@testbed-1-vm1(analyticsdb):/# vi /tmp/k^C

Rest of the logs can be find at following location:
bhushana@mayamruga
Path: /home/bhushana/Documents/technical/bugs/new_bug_zookeeper