Juniper Openstack

Fresh install 3.2 #29 provisioning fails with keystone issues

Bug #1673274 reported by Sarath on 2017-03-15

This bug affects 1 person

	Status	Importance	Assigned to	Milestone
Juniper Openstack	Status tracked in Trunk
R3.2	Fix Committed	Critical	amudhar	Juniper Openstack r3.2.2.0
Trunk	Fix Committed	Critical	amudhar	Juniper Openstack r4.0.0.0-fcs "r4.0.0.0"

Bug Description

This issue seen multiple times on Sanity automated runs of Vcenter-compute
Amudha debugged this and below findings,
---------------
keystone was already started and when supervisor-openstack tries to start it, it fails with port already in use error
v have pre and post install scripts in contrail-openstack that stop keystone and then start it from supervisor-openstack, and they all seemed to be in place on the failed node
that is why wanted to install manually and see whats happening
but seems to go through fine
----------------

>>> below failed outputs
2017-03-15 14:28:46:158766: [root@10.84.13.33] Executing task 'verify_openstack'
2017-03-15 14:28:46:158879: [root@10.84.13.33] sudo: python -c 'from platform import linux_distribution; print linux_distribution()'
2017-03-15 14:28:46:159015: [root@10.84.13.33] out: ('Ubuntu', '14.04', 'trusty')
2017-03-15 14:28:46:193296: [root@10.84.13.33] out:
2017-03-15 14:28:46:196639:
2017-03-15 14:28:46:225485: [root@10.84.13.33] sudo: service keystone status
2017-03-15 14:28:46:225733: [root@10.84.13.33] out: keystone STARTING
2017-03-15 14:28:46:398729: [root@10.84.13.33] out:
2017-03-15 14:28:46:398925:
2017-03-15 14:28:46:399047: [root@10.84.13.33] sudo: service keystone status
2017-03-15 14:29:06:419021: [root@10.84.13.33] out: keystone FATAL Exited too quickly (process log may have details)
2017-03-15 14:29:06:586650: [root@10.84.13.33] out:
2017-03-15 14:29:06:586863:
2017-03-15 14:29:06:587126: [root@10.84.13.33] sudo: service keystone status
2017-03-15 14:29:26:606640: [root@10.84.13.33] out: keystone FATAL Exited too quickly (process log may have details)
2017-03-15 14:29:26:709554: [root@10.84.13.33] out:
2017-03-15 14:29:26:710907:
2017-03-15 14:29:26:720505: [root@10.84.13.33] sudo: service keystone status
2017-03-15 14:29:46:731976: [root@10.84.13.33] out: keystone FATAL Exited too quickly (process log may have details)
2017-03-15 14:29:46:835432: [root@10.84.13.33] out:
2017-03-15 14:29:46:835635:
2017-03-15 14:29:46:836448: [root@10.84.13.33] sudo: service keystone status
2017-03-15 14:30:06:855177: [root@10.84.13.33] out: keystone FATAL Exited too quickly (process log may have details)
2017-03-15 14:30:07:022576: [root@10.84.13.33] out:
2017-03-15 14:30:07:022764:
2017-03-15 14:30:07:022995: [root@10.84.13.33] sudo: service keystone status
2017-03-15 14:30:27:042361: [root@10.84.13.33] out: keystone FATAL Exited too quickly (process log may have details)
2017-03-15 14:30:27:177038: [root@10.84.13.33] out:
2017-03-15 14:30:27:184424:
2017-03-15 14:30:27:209481: [root@10.84.13.33] sudo: service keystone status
2017-03-15 14:30:47:226381: [root@10.84.13.33] out: keystone FATAL Exited too quickly (process log may have details)
2017-03-15 14:30:47:329995: [root@10.84.13.33] out:
2017-03-15 14:30:47:330227:
2017-03-15 14:30:47:331251: [root@10.84.13.33] sudo: service keystone status
2017-03-15 14:31:07:338409: [root@10.84.13.33] out: keystone FATAL Exited too quickly (process log may have details)
2017-03-15 14:31:07:474891: [root@10.84.13.33] out:
2017-03-15 14:31:07:475068:
2017-03-15 14:31:07:475267: [root@10.84.13.33] sudo: service keystone status
2017-03-15 14:31:27:494378: [root@10.84.13.33] out: keystone FATAL Exited too quickly (process log may have details)
2017-03-15 14:31:27:598384: [root@10.84.13.33] out:
2017-03-15 14:31:27:598592:
2017-03-15 14:31:27:598839: [root@10.84.13.33] sudo: service keystone status
2017-03-15 14:31:47:618642: [root@10.84.13.33] out: keystone FATAL Exited too quickly (process log may have details)
2017-03-15 14:31:47:785450: [root@10.84.13.33] out:
2017-03-15 14:31:47:785603:
2017-03-15 14:31:47:785713: Disconnecting from 10.84.13.32... done.
2017-03-15 14:32:07:867950: Disconnecting from 10.84.13.2... done.
2017-03-15 14:32:07:931758: Disconnecting from 10.84.13.222... done.
2017-03-15 14:32:07:963468: Disconnecting from 10.84.13.223... done.
2017-03-15 14:32:08:027281: Disconnecting from 10.84.13.33... done.
2017-03-15 14:32:08:141189: Disconnecting from 10.84.13.38... done.
2017-03-15 14:32:08:211044: Service keystone not running.
2017-03-15 14:32:08:211159: root@a6s32:~/fabric-utils#
root@a6s32:~/fabric-utils#
root@a6s32:~/fabric-utils# contrail-status
== Contrail Control ==
supervisor-control: inactive (disabled on boot)
unix:///var/run/supervisord_control.sockno

== Contrail Analytics ==
supervisor-analytics: inactive (disabled on boot)
unix:///var/run/supervisord_analytics.sockno

== Contrail Config ==
supervisor-config: inactive (disabled on boot)
unix:///var/run/supervisord_config.sockno

== Contrail Web UI ==
supervisor-webui: inactive (disabled on boot)
unix:///var/run/supervisord_webui.sockno

== Contrail Database ==
contrail-database: active

== Contrail Supervisor Database ==
supervisor-database: active
contrail-database-nodemgr active
kafka active

== Contrail Support Services ==
supervisor-support-service: active
rabbitmq-server active

root@a6s32:~/fabric-utils#

Tags:

Revision history for this message

Sarath (nsarath) wrote on 2017-03-15:

nsarath@ubuntu-build04:/auto/cores/1673274$ ls -ltrd *
-rwxrwxrwx 1 nsarath test 8960000 Mar 15 15:28 Ctrl-A-Utils.tar
-rwxrwxrwx 1 nsarath test 427786240 Mar 15 15:28 Ctrl-A-log.tar
-rwxrwxrwx 1 nsarath test 422901760 Mar 15 15:28 Ctrl-B-log.tar
-rwxrwxrwx 1 nsarath test 429762560 Mar 15 15:28 Ctrl-C-log.tar
-rwxrwxrwx 1 nsarath test 9594880 Mar 15 15:36 utils-buildServer.tar

Revision history for this message

Jeba Paulaiyan (jebap) wrote on 2017-03-17:

Download full text (6.5 KiB)

Similar issue is seen in openstack cluster also:

2017-03-14 20:35:31:393150: [root@10.87.119.12] out: [localhost] local: setup-quantum-in-keystone --ks_server_ip 192.16.7.28 --quant_server_ip 192.16.7.27 --tenant admin --user admin --password c0ntrail123 --svc_password c0ntrail123 --svc_tenant_name service --root_password None --auth_protocol https --insecure --region_name RegionOne
2017-03-14 20:35:31:400578: [root@10.87.119.12] out: /usr/lib/python2.7/dist-packages/urllib3/util/ssl_.py:315: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#snimissingwarning.
2017-03-14 20:35:32:102125: [root@10.87.119.12] out: SNIMissingWarning
2017-03-14 20:35:32:102320: [root@10.87.119.12] out: /usr/lib/python2.7/dist-packages/urllib3/util/ssl_.py:120: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
2017-03-14 20:35:32:102364: [root@10.87.119.12] out: InsecurePlatformWarning
2017-03-14 20:35:32:102400: [root@10.87.119.12] out: /usr/lib/python2.7/dist-packages/urllib3/connectionpool.py:794: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
2017-03-14 20:35:32:102430: [root@10.87.119.12] out: InsecureRequestWarning)
2017-03-14 20:35:32:102465: [root@10.87.119.12] out: Authorization Failed: Bad Gateway (HTTP 502)
2017-03-14 20:35:32:467432: [root@10.87.119.12] out: Traceback (most recent call last):
2017-03-14 20:35:32:467605: [root@10.87.119.12] out: File "/usr/bin/setup-quantum-in-keystone", line 9, in <module>
2017-03-14 20:35:32:467662: [root@10.87.119.12] out: load_entry_point('ContrailProvisioning==0.1dev', 'console_scripts', 'setup-quantum-in-keystone')()
2017-03-14 20:35:32:467713: [root@10.87.119.12] out: File "/usr/local/lib/python2.7/dist-packages/contrail_provisioning/config/quantum_in_keystone_setup.py", line 306, in main
2017-03-14 20:35:32:467771: [root@10.87.119.12] out: quant_obj = QuantumSetup(args_str)
2017-03-14 20:35:32:467832: [root@10.87.119.12] out: File "/usr/local/lib/python2.7/dist-packages/contrail_provisioning/config/quantum_in_keystone_setup.py", line 72, in __init__
2017-03-14 20:35:32:467893: [root@10.87.119.12] out: raise e
2017-03-14 20:35:32:467966: [root@10.87.119.12] out: keystoneauth1.exceptions.auth.AuthorizationFailure: Authorization Failed: Bad Gateway (HTTP 502)
2017-03-14 20:35:32:468015: [root@10.87.119.12] out:
2017-03-14 20:35:32:471683: [root@10.87.119.12] out: Fatal error: local() encountered an error (return code 1) while executing 'setup-quantum-in-keystone --ks_server_ip 192.16.7.28 --quant_server_ip 192.16.7...

Similar issue is seen in openstack cluster also:

2017-03-14 20:35:31:393150: [root@10.87.119.12] out: [localhost] local: setup-quantum-in-keystone --ks_server_ip     192.16.7.28 --quant_server_ip  192.16.7.27 --tenant           admin --user             admin --password         c0ntrail123 --svc_password     c0ntrail123 --svc_tenant_name  service --root_password    None --auth_protocol    https --insecure --region_name RegionOne
2017-03-14 20:35:31:400578: [root@10.87.119.12] out: /usr/lib/python2.7/dist-packages/urllib3/util/ssl_.py:315: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#snimissingwarning.
2017-03-14 20:35:32:102125: [root@10.87.119.12] out:   SNIMissingWarning
2017-03-14 20:35:32:102320: [root@10.87.119.12] out: /usr/lib/python2.7/dist-packages/urllib3/util/ssl_.py:120: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
2017-03-14 20:35:32:102364: [root@10.87.119.12] out:   InsecurePlatformWarning
2017-03-14 20:35:32:102400: [root@10.87.119.12] out: /usr/lib/python2.7/dist-packages/urllib3/connectionpool.py:794: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.org/en/latest/security.html
2017-03-14 20:35:32:102430: [root@10.87.119.12] out:   InsecureRequestWarning)
2017-03-14 20:35:32:102465: [root@10.87.119.12] out: Authorization Failed: Bad Gateway (HTTP 502)
2017-03-14 20:35:32:467432: [root@10.87.119.12] out: Traceback (most recent call last):
2017-03-14 20:35:32:467605: [root@10.87.119.12] out:   File "/usr/bin/setup-quantum-in-keystone", line 9, in <module>
2017-03-14 20:35:32:467662: [root@10.87.119.12] out:     load_entry_point('ContrailProvisioning==0.1dev', 'console_scripts', 'setup-quantum-in-keystone')()
2017-03-14 20:35:32:467713: [root@10.87.119.12] out:   File "/usr/local/lib/python2.7/dist-packages/contrail_provisioning/config/quantum_in_keystone_setup.py", line 306, in main
2017-03-14 20:35:32:467771: [root@10.87.119.12] out:     quant_obj = QuantumSetup(args_str)
2017-03-14 20:35:32:467832: [root@10.87.119.12] out:   File "/usr/local/lib/python2.7/dist-packages/contrail_provisioning/config/quantum_in_keystone_setup.py", line 72, in __init__
2017-03-14 20:35:32:467893: [root@10.87.119.12] out:     raise e
2017-03-14 20:35:32:467966: [root@10.87.119.12] out: keystoneauth1.exceptions.auth.AuthorizationFailure: Authorization Failed: Bad Gateway (HTTP 502)
2017-03-14 20:35:32:468015: [root@10.87.119.12] out: 
2017-03-14 20:35:32:471683: [root@10.87.119.12] out: Fatal error: local() encountered an error (return code 1) while executing 'setup-quantum-in-keystone --ks_server_ip     192.16.7.28 --quant_server_ip  192.16.7.27 --tenant           admin --user             admin --password         c0ntrail123 --svc_password     c0ntrail123 --svc_tenant_name  service --root_password    None --auth_protocol    https --insecure --region_name RegionOne'
2017-03-14 20:35:32:471862: [root@10.87.119.12] out: 
2017-03-14 20:35:32:471933: [root@10.87.119.12] out: Aborting.
2017-03-14 20:35:32:471982: [root@10.87.119.12] out: 
2017-03-14 20:35:32:535963:

2017-03-14 20:35:32:537028: Fatal error: sudo() received nonzero return code 1 while executing!
2017-03-14 20:35:32:537028: 
2017-03-14 20:35:32:537028: Requested: setup-vnc-config --self_ip 192.16.7.2 --cfgm_index 1 --collector_ip 192.16.7.4 --cassandra_ip_list 192.16.7.2 192.16.7.6 192.16.7.10 --zookeeper_ip_list 192.16.7.2 192.16.7.6 192.16.7.10 --control_ip_list 192.16.7.3 192.16.7.7 192.16.7.11 --quantum_port 9697 --nworkers 1 --service_token 3f29ba50ce6d85ec71a4 --amqp_ip_list 192.16.7.2 192.16.7.6 192.16.7.10 --amqp_port 5672 --apiserver_insecure False --apiserver_certfile /etc/contrail/ssl/certs/contrail.pem --apiserver_keyfile /etc/contrail/ssl/private/contrail.key --apiserver_cafile /etc/contrail/ssl/certs/contrail_ca.pem --first_cfgm_ip 192.16.7.2 --orchestrator openstack --seed_list 192.16.7.2,192.16.7.6 --data_dir /var/lib/cassandra/data --manage_db --cloud_admin_role admin --aaa_mode cloud-admin --keystone_ip 192.16.7.28 --keystone_version v3 --keystone_admin_passwd c0ntrail123 --keystone_service_tenant_name service --neutron_password c0ntrail123 --keystone_auth_protocol https --keystone_auth_port 35357 --keystone_insecure False --keystone_certfile /etc/keystone/ssl/certs/keystone.pem --keystone_keyfile /etc/keystone/ssl/private/keystone.key --keystone_cafile /etc/keystone/ssl/certs/keystone_ca.pem --region_name RegionOne --internal_vip 192.16.7.28 --contrail_internal_vip 192.16.7.27 --cloud_admin_role admin
2017-03-14 20:35:32:537028: Executed: sudo -S -p 'sudo password:'  /bin/bash -l -c "cd /opt/contrail/bin && setup-vnc-config --self_ip 192.16.7.2 --cfgm_index 1 --collector_ip 192.16.7.4 --cassandra_ip_list 192.16.7.2 192.16.7.6 192.16.7.10 --zookeeper_ip_list 192.16.7.2 192.16.7.6 192.16.7.10 --control_ip_list 192.16.7.3 192.16.7.7 192.16.7.11 --quantum_port 9697 --nworkers 1 --service_token 3f29ba50ce6d85ec71a4 --amqp_ip_list 192.16.7.2 192.16.7.6 192.16.7.10 --amqp_port 5672 --apiserver_insecure False --apiserver_certfile /etc/contrail/ssl/certs/contrail.pem --apiserver_keyfile /etc/contrail/ssl/private/contrail.key --apiserver_cafile /etc/contrail/ssl/certs/contrail_ca.pem --first_cfgm_ip 192.16.7.2 --orchestrator openstack --seed_list 192.16.7.2,192.16.7.6 --data_dir /var/lib/cassandra/data --manage_db --cloud_admin_role admin --aaa_mode cloud-admin --keystone_ip 192.16.7.28 --keystone_version v3 --keystone_admin_passwd c0ntrail123 --keystone_service_tenant_name service --neutron_password c0ntrail123 --keystone_auth_protocol https --keystone_auth_port 35357 --keystone_insecure False --keystone_certfile /etc/keystone/ssl/certs/keystone.pem --keystone_keyfile /etc/keystone/ssl/private/keystone.key --keystone_cafile /etc/keystone/ssl/certs/keystone_ca.pem --region_name RegionOne --internal_vip 192.16.7.28 --contrail_internal_vip 192.16.7.27 --cloud_admin_role admin"
2017-03-14 20:35:32:537028: 
2017-03-14 20:35:32:537100: Aborting.
2017-03-14 20:35:32:537100: 2017-03-14 20:35:32:536779: Disconnecting from 10.87.119.21... done.

summary:

- Vcenter-as-compute: Fresh install 3.2 #29 provisioning fails
+ Fresh install 3.2 #29 provisioning fails with keystone issues

Sarath (nsarath) on 2017-03-17

tags:

added: sanity

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2017-03-17: [Review update] master

Review in progress for https://review.opencontrail.org/29667
Submitter: Ignatious Johnson Christopher (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2017-03-17: [Review update] R3.2

Review in progress for https://review.opencontrail.org/29668
Submitter: Ignatious Johnson Christopher (<email address hidden>)

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2017-03-17: A change has been merged

Reviewed: https://review.opencontrail.org/29668
Committed: http://github.org/Juniper/contrail-provisioning/commit/dcafa86868444c27306a1ae101044b34d327f797
Submitter: Zuul (<email address hidden>)
Branch: R3.2

commit dcafa86868444c27306a1ae101044b34d327f797
Author: Ignatious Johnson Christopher <email address hidden>
Date: Thu Mar 16 21:58:12 2017 -0700

Removing the unnecessary supervisor-openstack

restart during keystone-server-setup.sh, immediately
after supervisor-openstack start.
Becasue supervisor-openstack start followed by supervisor-openstack
restart without delay cause supervisor to leave the child orphaned.

Change-Id: Ib8b4ac24423b157f8d90f4bbffd0b898f9413ad0
Closes-Bug: 1673274

Revision history for this message

OpenContrail Admin (ci-admin-f) wrote on 2017-03-17:

Reviewed: https://review.opencontrail.org/29667
Committed: http://github.org/Juniper/contrail-provisioning/commit/8b4f1cdd27e73d97082a80a3dca00f7a32941342
Submitter: Zuul (<email address hidden>)
Branch: master

commit 8b4f1cdd27e73d97082a80a3dca00f7a32941342
Author: Ignatious Johnson Christopher <email address hidden>
Date: Thu Mar 16 21:58:12 2017 -0700

Removing the unnecessary supervisor-openstack

Change-Id: Ib8b4ac24423b157f8d90f4bbffd0b898f9413ad0
Closes-Bug: 1673274

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.