OSP10-4-01-Build30-controller deployment failed at step-2

Bug #1716989 reported by shajuvk
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R4.0
Invalid
Critical
shajuvk
Trunk
Invalid
Critical
shajuvk

Bug Description

I am seeing below failure in overcloud controller with Newton build 30. Could you please check my system ?
Note: Contrail repo created with (Ocata contrail-docker-images + newton contrail-install-packages). Since the failure in openstack node, this might not be the issue with our contrail repo.

Setup : 10.87.67.42 root/c0ntrail123

[root@overcloud-controller-0 ~]# grep Error /var/log/messages
Sep 13 02:29:53 localhost os-collect-config: [2017-09-13 02:29:53,945] (heat-config) [INFO] {"deploy_stdout": "/ /var/lib/heat-config/heat-config-script\n/var/lib/heat-config/heat-config-script\n", "deploy_stderr": "\ngzip: stdin: not in gzip format\ntar: Child returned status 1\ntar: Error is not recoverable: exiting now\n", "deploy_status_code": 0}
Sep 13 02:29:53 localhost os-collect-config: tar: Error is not recoverable: exiting now
Sep 13 02:57:07 localhost os-collect-config: [2017-09-13 02:57:07,601] (heat-config) [INFO] {"deploy_stdout": "/ /var/lib/heat-config/heat-config-script\n/var/lib/heat-config/heat-config-script\n", "deploy_stderr": "\ngzip: stdin: not in gzip format\ntar: Child returned status 1\ntar: Error is not recoverable: exiting now\n", "deploy_status_code": 0}
Sep 13 02:57:07 localhost os-collect-config: tar: Error is not recoverable: exiting now
Sep 13 03:07:33 localhost os-collect-config: ib::Db::Mysql::Host_access[glance_10.0.0.10]/Mysql_user[glance@10.0.0.10]/ensure: created\u001b[0m\n\u001b[mNotice: /Stage[main]/Glance::Db::Mysql/Openstacklib::Db::Mysql[glance]/Openstacklib::Db::Mysql::Host_access[glance_10.0.0.10]/Mysql_grant[glance@10.0.0.10/glance.*]/ensure: created\u001b[0m\n\u001b[mNotice: /Stage[main]/Glance::Deps/Anchor[glance::db::end]: Triggered 'refresh' from 1 events\u001b[0m\n\u001b[mNotice: /Stage[main]/Glance::Deps/Anchor[glance::dbsync::begin]: Triggered 'refresh' from 1 events\u001b[0m\n\u001b[mNotice: /Firewall[998 log all]: Dependency Rabbitmq_user[guest] has failures: true\u001b[0m\n\u001b[mNotice: /Firewall[999 drop all]: Dependency Rabbitmq_user[guest] has failures: true\u001b[0m\n\u001b[mNotice: Finished catalog run in 355.15 seconds\u001b[0m\n", "deploy_stderr": "exception: connect failed\n\u001b[1;31mWarning: Scope(Class[Mongodb::Server]): Replset specified, but no replset_members or replset_config provided.\u001b[0m\n\u001b[1;31mWarning: Scope(Haproxy::Config[haproxy]): haproxy: The $merge_options parameter will default to true in the next major release. Please review the documentation regarding the implications.\u001b[0m\n\u001b[1;31mError: Could not prefetch mysql_user provider 'mysql': Execution of '/usr/bin/mysql -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)\u001b[0m\n\u001b[1;31mError: /Stage[main]/Tripleo::Profile::Base::Rabbitmq/Rabbitmq_user[guest]: Could not evaluate: Command is still failing after 180 seconds expired!\u001b[0m\n\u001b[1;31mWarning: /Firewall[998 log all]: Skipping because of failed dependencies\u001b[0m\n\u001b[1;31mWarning: /Firewall[999 drop all]: Skipping because of failed dependencies\u001b[0m\n", "deploy_status_code": 6}
Sep 13 03:07:33 localhost os-collect-config: #033[1;31mError: Could not prefetch mysql_user provider 'mysql': Execution of '/usr/bin/mysql -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)#033[0m
Sep 13 03:07:33 localhost os-collect-config: #033[1;31mError: /Stage[main]/Tripleo::Profile::Base::Rabbitmq/Rabbitmq_user[guest]: Could not evaluate: Command is still failing after 180 seconds expired!#033[0m
Sep 13 03:07:33 localhost os-collect-config: [2017-09-13 07:07:33,929] (heat-config) [ERROR] Error running /var/lib/heat-config/heat-config-puppet/32c2e3a0-c3e8-47ea-9776-21274d72214e.pp. [6]
Sep 13 03:07:53 localhost lrmd[26203]: notice: rabbitmq_start_0:46901:stderr [ Error performing operation: No such device or address ]
Sep 13 03:07:53 localhost lrmd[26203]: notice: rabbitmq_start_0:46901:stderr [ Error: unable to connect to node 'rabbit@overcloud-controller-0': nodedown ]
Sep 13 03:07:53 localhost lrmd[26203]: notice: rabbitmq_start_0:46901:stderr [ Error: operation forget_cluster_node used with invalid parameter: [] ]
[root@overcloud-controller-0 ~]#

=============

[stack@5b9s38-undercloud ~]$ openstack stack failures list overcloud
overcloud.AllNodesDeploySteps.ControllerDeployment_Step2.1:
  resource_type: OS::Heat::StructuredDeployment
  physical_resource_id: 3ea63a30-cebf-4358-b3dc-5eb1ffbc10c2
  status: CREATE_FAILED
  status_reason: |
    Error: resources[1]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6
  deploy_stdout: |
    ...
    Notice: /Stage[main]/Swift/Swift_config[swift-constraints/max_header_size]/ensure: created
    Notice: /Stage[main]/Swift/Swift_config[swift-hash/swift_hash_path_suffix]/value: value changed '%SWIFT_HASH_PATH_SUFFIX%' to 'Zd8MhmZhhDVBrYqWEDad7sAAJ'
    Notice: /Stage[main]/Swift::Deps/Anchor[swift::config::end]: Triggered 'refresh' from 2 events
    Notice: /Stage[main]/Swift::Deps/Anchor[swift::service::begin]: Triggered 'refresh' from 2 events
    Notice: /File[/etc/localtime]/seltype: seltype changed 'locale_t' to 'etc_t'
    Notice: /Stage[main]/Pacemaker::Service/Service[pacemaker]/enable: enable changed 'false' to 'true'
    Notice: /Stage[main]/Pacemaker::Service/Service[corosync]/enable: enable changed 'false' to 'true'
    Notice: /Firewall[998 log all]: Dependency Rabbitmq_user[guest] has failures: true
    Notice: /Firewall[999 drop all]: Dependency Rabbitmq_user[guest] has failures: true
    Notice: Finished catalog run in 263.71 seconds
    (truncated, view all with --long)
  deploy_stderr: |
    exception: connect failed
    Warning: Scope(Class[Mongodb::Server]): Replset specified, but no replset_members or replset_config provided.
    Warning: Scope(Haproxy::Config[haproxy]): haproxy: The $merge_options parameter will default to true in the next major release. Please review the documentat
ion regarding the implications.
    Error: /Stage[main]/Tripleo::Profile::Base::Rabbitmq/Rabbitmq_user[guest]: Could not evaluate: Command is still failing after 180 seconds expired!
    Warning: /Firewall[998 log all]: Skipping because of failed dependencies
    Warning: /Firewall[999 drop all]: Skipping because of failed dependencies
overcloud.AllNodesDeploySteps.ControllerDeployment_Step2.0:
  resource_type: OS::Heat::StructuredDeployment
  physical_resource_id: b3381e5a-11cf-4bc1-978b-247237123870
  status: CREATE_FAILED
  status_reason: |
    Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6
  deploy_stdout: |
    ...
    Notice: /Stage[main]/Neutron::Db::Mysql/Openstacklib::Db::Mysql[neutron]/Openstacklib::Db::Mysql::Host_access[ovs_neutron_10.0.0.10]/Mysql_grant[neutron@10.
0.0.10/ovs_neutron.*]/ensure: created
    Notice: /Stage[main]/Neutron::Deps/Anchor[neutron::db::end]: Triggered 'refresh' from 1 events
    Notice: /Stage[main]/Neutron::Deps/Anchor[neutron::dbsync::begin]: Triggered 'refresh' from 1 events
    Notice: /Stage[main]/Glance::Db::Mysql/Openstacklib::Db::Mysql[glance]/Openstacklib::Db::Mysql::Host_access[glance_10.0.0.10]/Mysql_user[glance@10.0.0.10]/e
nsure: created
    Notice: /Stage[main]/Glance::Db::Mysql/Openstacklib::Db::Mysql[glance]/Openstacklib::Db::Mysql::Host_access[glance_10.0.0.10]/Mysql_grant[glance@10.0.0.10/g
lance.*]/ensure: created
    Notice: /Stage[main]/Glance::Deps/Anchor[glance::db::end]: Triggered 'refresh' from 1 events
    Notice: /Stage[main]/Glance::Deps/Anchor[glance::dbsync::begin]: Triggered 'refresh' from 1 events
    Notice: /Firewall[998 log all]: Dependency Rabbitmq_user[guest] has failures: true
    Notice: /Firewall[999 drop all]: Dependency Rabbitmq_user[guest] has failures: true
    Notice: Finished catalog run in 355.15 seconds
    (truncated, view all with --long)
  deploy_stderr: |
    exception: connect failed
    Warning: Scope(Class[Mongodb::Server]): Replset specified, but no replset_members or replset_config provided.
    Warning: Scope(Haproxy::Config[haproxy]): haproxy: The $merge_options parameter will default to true in the next major release. Please review the documentation regarding the implications.
    Error: Could not prefetch mysql_user provider 'mysql': Execution of '/usr/bin/mysql -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
    Error: /Stage[main]/Tripleo::Profile::Base::Rabbitmq/Rabbitmq_user[guest]: Could not evaluate: Command is still failing after 180 seconds expired!
    Warning: /Firewall[998 log all]: Skipping because of failed dependencies
    Warning: /Firewall[999 drop all]: Skipping because of failed dependencies
overcloud.AllNodesDeploySteps.ControllerDeployment_Step2.2:
  resource_type: OS::Heat::StructuredDeployment
  physical_resource_id: 70e924f4-3818-4dd2-bb74-899e4f2d6567
  status: CREATE_FAILED
  status_reason: |
    Error: resources[2]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6
  deploy_stdout: |
    ...
    Notice: /Stage[main]/Swift/Swift_config[swift-constraints/max_header_size]/ensure: created
    Notice: /Stage[main]/Swift/Swift_config[swift-hash/swift_hash_path_suffix]/value: value changed '%SWIFT_HASH_PATH_SUFFIX%' to 'Zd8MhmZhhDVBrYqWEDad7sAAJ'
    Notice: /Stage[main]/Swift::Deps/Anchor[swift::config::end]: Triggered 'refresh' from 2 events
    Notice: /Stage[main]/Swift::Deps/Anchor[swift::service::begin]: Triggered 'refresh' from 2 events
    Notice: /File[/etc/localtime]/seltype: seltype changed 'locale_t' to 'etc_t'
    Notice: /Stage[main]/Pacemaker::Service/Service[pacemaker]/enable: enable changed 'false' to 'true'
    Notice: /Stage[main]/Pacemaker::Service/Service[corosync]/enable: enable changed 'false' to 'true'
    Notice: /Firewall[998 log all]: Dependency Rabbitmq_user[guest] has failures: true
    Notice: /Firewall[999 drop all]: Dependency Rabbitmq_user[guest] has failures: true
    Notice: Finished catalog run in 274.63 seconds
    (truncated, view all with --long)
  deploy_stderr: |
    exception: connect failed
    Warning: Scope(Class[Mongodb::Server]): Replset specified, but no replset_members or replset_config provided.
    Warning: Scope(Haproxy::Config[haproxy]): haproxy: The $merge_options parameter will default to true in the next major release. Please review the documentation regarding the implications.
    Error: /Stage[main]/Tripleo::Profile::Base::Rabbitmq/Rabbitmq_user[guest]: Could not evaluate: Command is still failing after 180 seconds expired!
    Warning: /Firewall[998 log all]: Skipping because of failed dependencies
    Warning: /Firewall[999 drop all]: Skipping because of failed dependencies
[stack@5b9s38-undercloud ~]$

2017-09-13 07:04:22Z [overcloud-AllNodesDeploySteps-smyquny6jczf.ContrailDpdkDeployment_Step2.0]: CREATE_COMPLETE state changed
2017-09-13 07:04:22Z [overcloud-AllNodesDeploySteps-smyquny6jczf.ContrailDpdkDeployment_Step2]: CREATE_COMPLETE Stack CREATE completed successfully
2017-09-13 07:04:23Z [overcloud-AllNodesDeploySteps-smyquny6jczf.ContrailDpdkDeployment_Step2]: CREATE_COMPLETE state changed

2017-09-13 07:06:08Z [overcloud-AllNodesDeploySteps-smyquny6jczf.ControllerDeployment_Step2.1]: SIGNAL_IN_PROGRESS Signal: deployment 3ea63a30-cebf-4358-b3dc-5
eb1ffbc10c2 failed (6)
2017-09-13 07:06:08Z [overcloud-AllNodesDeploySteps-smyquny6jczf.ControllerDeployment_Step2.1]: CREATE_FAILED Error: resources[1]: Deployment to server failed:
deploy_status_code : Deployment exited with non-zero status code: 6
2017-09-13 07:06:34Z [overcloud-AllNodesDeploySteps-smyquny6jczf.ControllerDeployment_Step2.2]: SIGNAL_IN_PROGRESS Signal: deployment 70e924f4-3818-4dd2-bb74-8
99e4f2d6567 failed (6)
2017-09-13 07:06:36Z [overcloud-AllNodesDeploySteps-smyquny6jczf.ControllerDeployment_Step2.2]: CREATE_FAILED Error: resources[2]: Deployment to server failed:
deploy_status_code : Deployment exited with non-zero status code: 6
2017-09-13 07:07:36Z [overcloud-AllNodesDeploySteps-smyquny6jczf.ControllerDeployment_Step2.0]: SIGNAL_IN_PROGRESS Signal: deployment b3381e5a-11cf-4bc1-978b-2
47237123870 failed (6)
2017-09-13 07:07:36Z [overcloud-AllNodesDeploySteps-smyquny6jczf.ControllerDeployment_Step2.0]: CREATE_FAILED Error: resources[0]: Deployment to server failed:
deploy_status_code : Deployment exited with non-zero status code: 6
2017-09-13 07:07:37Z [overcloud-AllNodesDeploySteps-smyquny6jczf.ControllerDeployment_Step2]: CREATE_FAILED Resource CREATE failed: Error: resources[1]: Deploy
ment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6
2017-09-13 07:07:37Z [overcloud-AllNodesDeploySteps-smyquny6jczf.ControllerDeployment_Step2]: CREATE_FAILED Error: resources.ControllerDeployment_Step2.resourc
es[1]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 6
2017-09-13 07:07:37Z [overcloud-AllNodesDeploySteps-smyquny6jczf]: UPDATE_FAILED Error: resources.ControllerDeployment_Step2.resources[1]: Deployment to server
failed: deploy_status_code: Deployment exited with non-zero status code: 6
2017-09-13 07:07:38Z [AllNodesDeploySteps]: UPDATE_FAILED resources.AllNodesDeploySteps: Error: resources.ControllerDeployment_Step2.resources[1]: Deployment t
o server failed: deploy_status_code: Deployment exited with non-zero status code: 6
2017-09-13 07:07:38Z [overcloud]: UPDATE_FAILED resources.AllNodesDeploySteps: Error: resources.ControllerDeployment_Step2.resources[1]: Deployment to server f
ailed: deploy_status_code: Deployment exited with non-zero status code: 6

Stack overcloud UPDATE_FAILED

shajuvk (shajuvk)
description: updated
tags: added: ospd provisioning
information type: Proprietary → Public
Revision history for this message
shajuvk (shajuvk) wrote :
Download full text (6.5 KiB)

[stack@5b9s38-undercloud ~]$ openstack stack failures list overcloud
overcloud.AllNodesDeploySteps.ControllerDeployment_Step2.1:
  resource_type: OS::Heat::StructuredDeployment
  physical_resource_id: 3ea63a30-cebf-4358-b3dc-5eb1ffbc10c2
  status: CREATE_FAILED
  status_reason: |
    Error: resources[1]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6
  deploy_stdout: |
    ...
    Notice: /Stage[main]/Swift/Swift_config[swift-constraints/max_header_size]/ensure: created
    Notice: /Stage[main]/Swift/Swift_config[swift-hash/swift_hash_path_suffix]/value: value changed '%SWIFT_HASH_PATH_SUFFIX%' to 'Zd8MhmZhhDVBrYqWEDad7sAAJ'
    Notice: /Stage[main]/Swift::Deps/Anchor[swift::config::end]: Triggered 'refresh' from 2 events
    Notice: /Stage[main]/Swift::Deps/Anchor[swift::service::begin]: Triggered 'refresh' from 2 events
    Notice: /File[/etc/localtime]/seltype: seltype changed 'locale_t' to 'etc_t'
    Notice: /Stage[main]/Pacemaker::Service/Service[pacemaker]/enable: enable changed 'false' to 'true'
    Notice: /Stage[main]/Pacemaker::Service/Service[corosync]/enable: enable changed 'false' to 'true'
    Notice: /Firewall[998 log all]: Dependency Rabbitmq_user[guest] has failures: true
    Notice: /Firewall[999 drop all]: Dependency Rabbitmq_user[guest] has failures: true
    Notice: Finished catalog run in 263.71 seconds
    (truncated, view all with --long)
  deploy_stderr: |
    exception: connect failed
    Warning: Scope(Class[Mongodb::Server]): Replset specified, but no replset_members or replset_config provided.
    Warning: Scope(Haproxy::Config[haproxy]): haproxy: The $merge_options parameter will default to true in the next major release. Please review the documentat
ion regarding the implications.
    Error: /Stage[main]/Tripleo::Profile::Base::Rabbitmq/Rabbitmq_user[guest]: Could not evaluate: Command is still failing after 180 seconds expired!
    Warning: /Firewall[998 log all]: Skipping because of failed dependencies
    Warning: /Firewall[999 drop all]: Skipping because of failed dependencies
overcloud.AllNodesDeploySteps.ControllerDeployment_Step2.0:
  resource_type: OS::Heat::StructuredDeployment
  physical_resource_id: b3381e5a-11cf-4bc1-978b-247237123870
  status: CREATE_FAILED
  status_reason: |
    Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6
  deploy_stdout: |
    ...
    Notice: /Stage[main]/Neutron::Db::Mysql/Openstacklib::Db::Mysql[neutron]/Openstacklib::Db::Mysql::Host_access[ovs_neutron_10.0.0.10]/Mysql_grant[neutron@10.0.0.10/ovs_neutron.*]/ensure: created
    Notice: /Stage[main]/Neutron::Deps/Anchor[neutron::db::end]: Triggered 'refresh' from 1 events
    Notice: /Stage[main]/Neutron::Deps/Anchor[neutron::dbsync::begin]: Triggered 'refresh' from 1 events
    Notice: /Stage[main]/Glance::Db::Mysql/Openstacklib::Db::Mysql[glance]/Openstacklib::Db::Mysql::Host_access[glance_10.0.0.10]/Mysql_user[glance@10.0.0.10]/ensure: created
    Notice: /Stage[main]/Glance::Db::Mysql/Openstacklib::Db::Mysql[glance]/Openstacklib::Db::Mysql::Host_access[glance_10.0.0.10]/Mysql_grant[glance@10.0.0.10/glanc...

Read more...

tags: added: blocker
Revision history for this message
Michael Henkel (mhenkel-3) wrote :

is this a fresh deploy or a stack update? please add more information.

Revision history for this message
shajuvk (shajuvk) wrote :

This is the stack update, After adding review comment from bug : https://bugs.launchpad.net/juniperopenstack/+bug/1716526 I did the stack update.

Revision history for this message
alexey-mr (alexey-morlang) wrote :

Actually It looks that it is not related to Contrail itself...
The error is somewhere in rabbitmq configuration: mError: Could not prefetch mysql_user provider 'mysql': Execution of '/usr/bin/mysql -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock'

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.