Juniper Openstack

Vcenter-only: provisioning broken on latest build #3

Bug #1605500 reported by Sarath
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R3.1
Fix Committed
Critical
amudhar
Juniper Openstack r3.1.0.0-fcs
Trunk
Fix Committed
Critical
amudhar

Bug Description

Amudha debugged the setup evening and opening this bug for tracking and this issue seen both when
using "contrail-installer package part of bug fix#1599583 (or) usual contrail-install package

Per initial triaging, looks default vcenter-only mode not picked from testbed.py definitions.
Attached to bug, logs and also Fab cmd Logs

###########
Contrail-installer part of bug-fix #1599583
##########

2016-07-21 15:58:46:935257:
2016-07-21 15:58:46:947855: INFO:
2016-07-21 15:58:46:947855: Openstack and cfgm nodes are same, No need for contrail_internal_vip to be specified in testbed.py.
2016-07-21 15:58:46:947916: [root@10.87.26.197] Executing task 'setup_rabbitmq_cluster'
2016-07-21 15:58:46:948163: Provisioning rabbitq in cfgm nodes
2016-07-21 15:58:46:948283: [root@10.87.26.197] Executing task 'verify_cluster_status'
2016-07-21 15:58:46:948402: [root@10.87.26.197] sudo: service rabbitmq-server status
2016-07-21 15:58:46:948544: [root@10.87.26.197] out: rabbitmq-server RUNNING pid 1877, uptime 0:02:20
2016-07-21 15:58:47:179843: [root@10.87.26.197] out:

::
::

2016-07-21 16:13:55:988804: [root@10.87.26.199] out:
2016-07-21 16:14:26:011303: [root@10.87.26.199] out: DIAGNOSTICS
2016-07-21 16:14:26:011506: [root@10.87.26.199] out: ===========
2016-07-21 16:14:26:011578: [root@10.87.26.199] out:
2016-07-21 16:14:26:011650: [root@10.87.26.199] out: attempted to contact: ['rabbit@oblocknode04-ctrl']
2016-07-21 16:14:26:011711: [root@10.87.26.199] out:
2016-07-21 16:14:26:011790: [root@10.87.26.199] out: rabbit@oblocknode04-ctrl:
2016-07-21 16:14:26:011869: [root@10.87.26.199] out: * unable to connect to epmd (port 4369) on oblocknode04-ctrl: timeout (timed out)
2016-07-21 16:14:26:011954: [root@10.87.26.199] out:
2016-07-21 16:14:26:012023: [root@10.87.26.199] out:
2016-07-21 16:14:26:012095: [root@10.87.26.199] out: current node details:
2016-07-21 16:14:26:012168: [root@10.87.26.199] out: - node name: 'rabbitmq-cli-27908@oblocknode04'
2016-07-21 16:14:26:012229: [root@10.87.26.199] out: - home dir: /var/lib/rabbitmq
2016-07-21 16:14:26:012288: [root@10.87.26.199] out: - cookie hash: ak2LCDpyGddGyEdUhkGghw==
2016-07-21 16:14:26:012348: [root@10.87.26.199] out:
2016-07-21 16:14:26:012407: [root@10.87.26.199] out:
2016-07-21 16:14:26:012928:
2016-07-21 16:14:26:030521: Unable to setup RabbitMQ cluster in role[cfgm]....

###########
Contrail-install
##########

2016-07-21 23:23:13:186807: [root@10.87.26.197] out: Error {'succeeded': True, 'failed': False, 'return_code': 0, 'stderr': ''} in getting status of [contrail-api].
2016-07-21 23:23:13:250537: [root@10.87.26.197] out: [contrail-api and rabbitmq] not yet started by supervisor config, Retrying.
2016-07-21 23:23:13:250639: [root@10.87.26.197] out: [localhost] local: sudo service rabbitmq-server status
2016-07-21 23:23:15:252948: [root@10.87.26.197] out: [rabbitmq-server] started by supervisor config.
2016-07-21 23:23:15:353317: [root@10.87.26.197] out: [localhost] local: sudo service contrail-api status
2016-07-21 23:23:15:353450: [root@10.87.26.197] out: [contrail-api] started by supervisor config.
2016-07-21 23:23:15:417126: [root@10.87.26.197] out: [contrail-api and rabbitmq] started by supervisor config, continue to provision.
2016-07-21 23:23:15:417267: [root@10.87.26.197] out:
2016-07-21 23:23:15:417446:
2016-07-21 23:23:15:417804: [localhost] local: echo '10.87.26.205:172.16.80.110' >> /tmp/ESXiToVRouterIp-root@10.87.26.197
2016-07-21 23:23:15:418011: [localhost] local: echo '10.87.26.203:172.16.80.108' >> /tmp/ESXiToVRouterIp-root@10.87.26.197
2016-07-21 23:23:15:420256: [localhost] local: echo '10.87.26.201:172.16.80.106' >> /tmp/ESXiToVRouterIp-root@10.87.26.197
2016-07-21 23:23:15:421855: [localhost] local: echo '10.87.26.207:172.16.80.112' >> /tmp/ESXiToVRouterIp-root@10.87.26.197
2016-07-21 23:23:15:423371: [localhost] local: echo '10.87.26.200:172.16.80.105' >> /tmp/ESXiToVRouterIp-root@10.87.26.197
2016-07-21 23:23:15:424992: [root@10.87.26.197] put: /tmp/ESXiToVRouterIp-root@10.87.26.197 -> /etc/contrail/ESXiToVRouterIp.map
2016-07-21 23:23:15:433465: [localhost] local: rm /tmp/ESXiToVRouterIp-root@10.87.26.197
2016-07-21 23:23:15:454006: [root@10.87.26.197] sudo: setup-vcenter-plugin --vcenter_url 10.84.22.110 --vcenter_username <email address hidden> --vcenter_password Contrail123! --vcenter_datacenter datacenter --vcenter_dvswitch dvs2-mini4 --vcenter_ipfabricpg contrail-fab-pg --api_hostname 172.16.80.2 --api_port 8082 --zookeeper_serverlist 172.16.80.2:2181,172.16.80.13:2181,172.16.80.4:2181 --vcenter_mode vcenter-only
2016-07-21 23:23:15:456818: [root@10.87.26.197] out: [localhost] local: ln -sf /bin/true /sbin/chkconfig
2016-07-21 23:23:15:590154: [root@10.87.26.197] out: [localhost] local: sudo mv /tmp/tmpDyMbcj/contrail-vcenter-plugin.conf /etc/contrail/contrail-vcenter-plugin.conf
2016-07-21 23:23:15:593572: [root@10.87.26.197] out: [localhost] local: sudo vcenter-plugin-setup.sh
2016-07-21 23:23:15:601020: [root@10.87.26.197] out: contrail-vcenter-plugin: unrecognized service
2016-07-21 23:23:15:616550: [root@10.87.26.197] out:
2016-07-21 23:23:15:616661: [root@10.87.26.197] out: Fatal error: local() encountered an error (return code 1) while executing 'sudo vcenter-plugin-setup.sh'
2016-07-21 23:23:15:616726: [root@10.87.26.197] out:
2016-07-21 23:23:15:616816: [root@10.87.26.197] out: Aborting.
2016-07-21 23:23:15:616874: [root@10.87.26.197] out:
2016-07-21 23:23:15:618270:
2016-07-21 23:23:15:621729: Disconnecting from 10.87.26.208... done.
2016-07-21 23:23:15:627447: Disconnecting from 10.87.26.197... done.
2016-07-21 23:23:15:718619: Disconnecting from 10.87.26.199... done.
2016-07-21 23:23:15:796082:
2016-07-21 23:22:48:988795: Warning: sudo() received nonzero return code 2 while executing 'sudo ls /etc/contrail/haproxy.token'!
2016-07-21 23:22:48:988795:
2016-07-21 23:22:48:988795:
2016-07-21 23:22:51:061828: Warning: sudo() received nonzero return code 2 while executing 'sudo ls /etc/contrail/service.token'!
2016-07-21 23:22:51:061828:
2016-07-21 23:22:51:061828:
2016-07-21 23:23:15:621833: Fatal error: sudo() received nonzero return code 1 while executing!
2016-07-21 23:23:15:621833:
2016-07-21 23:23:15:621833: Requested: setup-vcenter-plugin --vcenter_url 10.84.22.110 --vcenter_username <email address hidden> --vcenter_password Contrail123! --vcenter_datacenter datacenter --vcenter_dvswitch dvs2-mini4 --vcenter_ipfabricpg contrail-fab-pg --api_hostname 172.16.80.2 --api_port 8082 --zookeeper_serverlist 172.16.80.2:2181,172.16.80.13:2181,172.16.80.4:2181 --vcenter_mode vcenter-only
2016-07-21 23:23:15:621833: Executed: sudo -S -p 'sudo password:' /bin/bash -l -c "cd /opt/contrail/bin && setup-vcenter-plugin --vcenter_url 10.84.22.110 --vcenter_username <email address hidden> --vcenter_password Contrail123! --vcenter_datacenter datacenter --vcenter_dvswitch dvs2-mini4 --vcenter_ipfabricpg contrail-fab-pg --api_hostname 172.16.80.2 --api_port 8082 --zookeeper_serverlist 172.16.80.2:2181,172.16.80.13:2181,172.16.80.4:2181 --vcenter_mode vcenter-only"
2016-07-21 23:23:15:621833:

See original description
Sarath (nsarath)
summary: - Vcenter-only: Fresh-install fails @ fab setup_all sees Openstack roles
- and default mode looks not ignored with latest build 3
+ Vcenter-only: Fresh-install fails @ fab setup_all seen Openstack roles
+ and default mode looks ignored with latest build 3
Revision history for this message
Sarath (nsarath) wrote : Re: Vcenter-only: Fresh-install fails @ fab setup_all seen Openstack roles and default mode looks ignored with latest build 3

-bash-4.1$ pwd
/cs-shared/bugs/1605500
-bash-4.1$
-bash-4.1$ ls -l
total 69496
-rwxrwxrwx 1 nsarath test 26009600 Jul 21 21:57 Ctrl-A-log.tar*
-rwxrwxrwx 1 nsarath test 4802560 Jul 21 22:04 Ctrl-A-Utils.tar*
-rwxrwxrwx 1 nsarath test 20121600 Jul 21 21:57 Ctrl-B-log.tar*
-rwxrwxrwx 1 nsarath test 19916800 Jul 21 21:57 Ctrl-C-log.tar*
drwxrwxrwx 4 nsarath test 8192 Jul 22 00:07 utils/

Sarath (nsarath)
tags: added: blocker
description: updated
Sarath (nsarath)
summary: - Vcenter-only: Fresh-install fails @ fab setup_all seen Openstack roles
- and default mode looks ignored with latest build 3
+ Vcenter-only: provisioning broken on latest build #3
Revision history for this message
amudhar (amudha) wrote :

With build 3, I do not see any provisioning failures for vcenter-only mode.
With build 5, setup_all failed due to #1606654.

Fix will be committed through #1606654, closing this bug.

Please reopen the bug if you again see the issue you have reported with build 3.

-Amudha

Revision history for this message
amudhar (amudha) wrote :

The issue reported on Sarath's setup with build 3 described in this bug is not due to openstack roles or vcenter-only mode ignored (as in the description).

The error seems to be due to missing setup-vcenter-plugin.
2016-07-21 23:23:15:454006: [root@10.87.26.197] sudo: setup-vcenter-plugin --vcenter_url 10.84.22.110 --vcenter_username <email address hidden> --vcenter_password Contrail123! --vcenter_datacenter datacenter --vcenter_dvswitch dvs2-mini4 --vcenter_ipfabricpg contrail-fab-pg --api_hostname 172.16.80.2 --api_port 8082 --zookeeper_serverlist 172.16.80.2:2181,172.16.80.13:2181,172.16.80.4:2181 --vcenter_mode vcenter-only
2016-07-21 23:23:15:456818: [root@10.87.26.197] out: [localhost] local: ln -sf /bin/true /sbin/chkconfig
2016-07-21 23:23:15:590154: [root@10.87.26.197] out: [localhost] local: sudo mv /tmp/tmpDyMbcj/contrail-vcenter-plugin.conf /etc/contrail/contrail-vcenter-plugin.conf
2016-07-21 23:23:15:593572: [root@10.87.26.197] out: [localhost] local: sudo vcenter-plugin-setup.sh
2016-07-21 23:23:15:601020: [root@10.87.26.197] out: contrail-vcenter-plugin: unrecognized service
2016-07-21 23:23:15:616550: [root@10.87.26.197] out:
2016-07-21 23:23:15:616661: [root@10.87.26.197] out: Fatal error: local() encountered an error (return code 1) while executing 'sudo vcenter-plugin-setup.sh'

This might be because of the vcenter-plugin not installed on 10.87.26.197. I was not able to see this issue on my single node setup with build 3, will triage if this is reported happening again.

-Amudha

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.