HA provisioning with fab fails if config nodes in testbed.py are not is sequential order ( The chosen master node is not the first node)

Bug #1455698 reported by Vivek
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R2.20
Fix Committed
Critical
Ranjeet R
Trunk
Fix Committed
Critical
Ranjeet R

Bug Description

testbed.py with failed configuration
-------------------------------------------------------------------------------------------------
from fabric.api import env
import os
#Management ip addresses of hosts in the cluster
host1 = 'root@10.87.140.27'
host2 = 'root@10.87.140.48'
host3 = 'root@10.87.129.168'
host4 = 'root@10.87.129.59'
host5 = 'root@10.87.140.49'
#External routers if any
ext_routers = [('montreal', '10.87.140.140')]

#Autonomous system number
router_asn = 64512
public_vn_rtgt = 10000
public_vn_subnet = "10.87.129.224/27"

#Host from which the fab commands are triggered to install and provision
host_build = 'root@10.87.140.27'

#Role definition of the hosts.
env.roledefs = {
    'all': [host1, host2, host3, host4, host5],
    'cfgm': [host1,host2,host3],
    'openstack': [host1,host2,host3],
    'control': [host1,host2,host3],
    'compute': [host4,host5],
    'collector': [host1,host2,host3],
    'webui': [host1,host2,host3],
    'database': [host1,host2,host3],
    'storage-master': [host1,host2,host3],
    'storage-compute': [ host4, host5],
    'build': [host_build],
}

env.hostnames = {
    'all': [ 'cmbu-vxa20-15','cmbu-vxa22-19', 'cmbu-vxa20-2','cmbu-vxa22-9','cmbu-vxa20-4' ]
}
--------------------------------------------------------------------------------------------

During fab setup_all, the IPs got sorted and master node was chosen to be host3 10.87.129.168 and the VIP were assigned to this node

This led to following error
2015-05-14 18:38:37:837959: [root@10.87.129.59] out: [localhost] local: chkconfig supervisor-vrouter on
2015-05-14 18:38:37:838152: [root@10.87.129.59] out: [localhost] local: python /opt/contrail/utils/provision_vrouter.py --host_name cmbu-vxa22-9 --host_ip 13.0.0.5 --api_server_ip 13.0.0.7 --oper add --admin_user admin --admin_password n1keenA --admin_tenant_name admin --openstack_ip 13.0.0.7
2015-05-14 18:38:37:838274: [root@10.87.129.59] out: Traceback (most recent call last):
2015-05-14 18:38:38:171540: [root@10.87.129.59] out: File "/opt/contrail/utils/provision_vrouter.py", line 186, in <module>
2015-05-14 18:38:38:171698: [root@10.87.129.59] out: main()
2015-05-14 18:38:38:171791: [root@10.87.129.59] out: File "/opt/contrail/utils/provision_vrouter.py", line 182, in main
2015-05-14 18:38:38:171881: [root@10.87.129.59] out: VrouterProvisioner(args_str)
2015-05-14 18:38:38:171968: [root@10.87.129.59] out: File "/opt/contrail/utils/provision_vrouter.py", line 32, in __init__
2015-05-14 18:38:38:172056: [root@10.87.129.59] out: auth_host=self._args.openstack_ip)
2015-05-14 18:38:38:172143: [root@10.87.129.59] out: File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 200, in __init__
2015-05-14 18:38:38:172264: [root@10.87.129.59] out: retry_on_error=False)
2015-05-14 18:38:38:172361: [root@10.87.129.59] out: File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 404, in _request
2015-05-14 18:38:38:172450: [root@10.87.129.59] out: content = self._request(op, url, data=data, retry_after_authn=True)
2015-05-14 18:38:38:172564: [root@10.87.129.59] out: File "/usr/lib/python2.7/dist-packages/vnc_api/vnc_api.py", line 429, in _request
2015-05-14 18:38:38:172653: [root@10.87.129.59] out: raise HttpError(status, content)
2015-05-14 18:38:38:172741: [root@10.87.129.59] out: cfgm_common.exceptions.HttpError: HTTP Status: 401 Content: Authentication required
2015-05-14 18:38:38:172826: [root@10.87.129.59] out:
2015-05-14 18:38:38:189702: [root@10.87.129.59] out: Fatal error: local() encountered an error (return code 1) while executing 'python /opt/contrail/utils/provision_vrouter.py --host_name cmbu-vxa22-9 --host_ip 13.0.0.5 --api_server_ip 13.0.0.7 --oper add --admin_user admin --admin_password n1keenA --admin_tenant_name admin --openstack_ip 13.0.0.7'
2015-05-14 18:38:38:189806: [root@10.87.129.59] out:
2015-05-14 18:38:38:189889: [root@10.87.129.59] out: Aborting.
2015-05-14 18:38:38:189954: [root@10.87.129.59] out:

Tags: ha
Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : R2.20

Review in progress for https://review.opencontrail.org/10618
Submitter: Ranjeet R (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : master

Review in progress for https://review.opencontrail.org/10619
Submitter: Ranjeet R (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/10618
Committed: http://github.org/Juniper/contrail-fabric-utils/commit/53c69f193cc2e85918d2f42fc616705ece26d8e2
Submitter: Zuul
Branch: R2.20

commit 53c69f193cc2e85918d2f42fc616705ece26d8e2
Author: Ranjeet R <email address hidden>
Date: Wed May 20 13:56:15 2015 -0700

Fixes: HA provisioning with fab fails if config nodes in testbed.py
are not is sequential order ( The chosen master node is not the first node)

While comparing the openstack and cfgm node sets, we are sorting the
nodes which changes the order in which they are mentioned.

Changing the compare to use sets instead of using sort.

Change-Id: I312664cb67ad51244cdcc5ae0eecdb739644a8df
ClosesBug: 1455698

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/10619
Committed: http://github.org/Juniper/contrail-fabric-utils/commit/3e1c9cd9ce29379d90dca655401cd5b4e4258283
Submitter: Zuul
Branch: master

commit 3e1c9cd9ce29379d90dca655401cd5b4e4258283
Author: Ranjeet R <email address hidden>
Date: Wed May 20 14:01:19 2015 -0700

Fixes: HA provisioning with fab fails if config nodes in testbed.py
are not is sequential order ( The chosen master node is not the first node)

While comparing the openstack and cfgm node sets, we are sorting the
nodes which changes the order in which they are mentioned.

Changing the compare to use sets instead of using sort.

Change-Id: Iaa1577202965eeec899c8226321e8bd946f81844
ClosesBug: 1455698

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.