I am seeing below failure in overcloud controller with Newton build 30. Could you please check my system ?
Note: Contrail repo created with (Ocata contrail-docker-images + newton contrail-install-packages). Since the failure in openstack node, this might not be the issue with our contrail repo.
Setup : 10.87.67.42 root/c0ntrail123
[root@overcloud-controller-0 ~]# grep Error /var/log/messages
Sep 13 02:29:53 localhost os-collect-config: [2017-09-13 02:29:53,945] (heat-config) [INFO] {"deploy_stdout": "/ /var/lib/heat-config/heat-config-script\n/var/lib/heat-config/heat-config-script\n", "deploy_stderr": "\ngzip: stdin: not in gzip format\ntar: Child returned status 1\ntar: Error is not recoverable: exiting now\n", "deploy_status_code": 0}
Sep 13 02:29:53 localhost os-collect-config: tar: Error is not recoverable: exiting now
Sep 13 02:57:07 localhost os-collect-config: [2017-09-13 02:57:07,601] (heat-config) [INFO] {"deploy_stdout": "/ /var/lib/heat-config/heat-config-script\n/var/lib/heat-config/heat-config-script\n", "deploy_stderr": "\ngzip: stdin: not in gzip format\ntar: Child returned status 1\ntar: Error is not recoverable: exiting now\n", "deploy_status_code": 0}
Sep 13 02:57:07 localhost os-collect-config: tar: Error is not recoverable: exiting now
Sep 13 03:07:33 localhost os-collect-config: ib::Db::Mysql::Host_access[glance_10.0.0.10]/Mysql_user[glance@10.0.0.10]/ensure: created\u001b[0m\n\u001b[mNotice: /Stage[main]/Glance::Db::Mysql/Openstacklib::Db::Mysql[glance]/Openstacklib::Db::Mysql::Host_access[glance_10.0.0.10]/Mysql_grant[glance@10.0.0.10/glance.*]/ensure: created\u001b[0m\n\u001b[mNotice: /Stage[main]/Glance::Deps/Anchor[glance::db::end]: Triggered 'refresh' from 1 events\u001b[0m\n\u001b[mNotice: /Stage[main]/Glance::Deps/Anchor[glance::dbsync::begin]: Triggered 'refresh' from 1 events\u001b[0m\n\u001b[mNotice: /Firewall[998 log all]: Dependency Rabbitmq_user[guest] has failures: true\u001b[0m\n\u001b[mNotice: /Firewall[999 drop all]: Dependency Rabbitmq_user[guest] has failures: true\u001b[0m\n\u001b[mNotice: Finished catalog run in 355.15 seconds\u001b[0m\n", "deploy_stderr": "exception: connect failed\n\u001b[1;31mWarning: Scope(Class[Mongodb::Server]): Replset specified, but no replset_members or replset_config provided.\u001b[0m\n\u001b[1;31mWarning: Scope(Haproxy::Config[haproxy]): haproxy: The $merge_options parameter will default to true in the next major release. Please review the documentation regarding the implications.\u001b[0m\n\u001b[1;31mError: Could not prefetch mysql_user provider 'mysql': Execution of '/usr/bin/mysql -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)\u001b[0m\n\u001b[1;31mError: /Stage[main]/Tripleo::Profile::Base::Rabbitmq/Rabbitmq_user[guest]: Could not evaluate: Command is still failing after 180 seconds expired!\u001b[0m\n\u001b[1;31mWarning: /Firewall[998 log all]: Skipping because of failed dependencies\u001b[0m\n\u001b[1;31mWarning: /Firewall[999 drop all]: Skipping because of failed dependencies\u001b[0m\n", "deploy_status_code": 6}
Sep 13 03:07:33 localhost os-collect-config: #033[1;31mError: Could not prefetch mysql_user provider 'mysql': Execution of '/usr/bin/mysql -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)#033[0m
Sep 13 03:07:33 localhost os-collect-config: #033[1;31mError: /Stage[main]/Tripleo::Profile::Base::Rabbitmq/Rabbitmq_user[guest]: Could not evaluate: Command is still failing after 180 seconds expired!#033[0m
Sep 13 03:07:33 localhost os-collect-config: [2017-09-13 07:07:33,929] (heat-config) [ERROR] Error running /var/lib/heat-config/heat-config-puppet/32c2e3a0-c3e8-47ea-9776-21274d72214e.pp. [6]
Sep 13 03:07:53 localhost lrmd[26203]: notice: rabbitmq_start_0:46901:stderr [ Error performing operation: No such device or address ]
Sep 13 03:07:53 localhost lrmd[26203]: notice: rabbitmq_start_0:46901:stderr [ Error: unable to connect to node 'rabbit@overcloud-controller-0': nodedown ]
Sep 13 03:07:53 localhost lrmd[26203]: notice: rabbitmq_start_0:46901:stderr [ Error: operation forget_cluster_node used with invalid parameter: [] ]
[root@overcloud-controller-0 ~]#
=============
[stack@5b9s38-undercloud ~]$ openstack stack failures list overcloud
overcloud.AllNodesDeploySteps.ControllerDeployment_Step2.1:
resource_type: OS::Heat::StructuredDeployment
physical_resource_id: 3ea63a30-cebf-4358-b3dc-5eb1ffbc10c2
status: CREATE_FAILED
status_reason: |
Error: resources[1]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6
deploy_stdout: |
...
Notice: /Stage[main]/Swift/Swift_config[swift-constraints/max_header_size]/ensure: created
Notice: /Stage[main]/Swift/Swift_config[swift-hash/swift_hash_path_suffix]/value: value changed '%SWIFT_HASH_PATH_SUFFIX%' to 'Zd8MhmZhhDVBrYqWEDad7sAAJ'
Notice: /Stage[main]/Swift::Deps/Anchor[swift::config::end]: Triggered 'refresh' from 2 events
Notice: /Stage[main]/Swift::Deps/Anchor[swift::service::begin]: Triggered 'refresh' from 2 events
Notice: /File[/etc/localtime]/seltype: seltype changed 'locale_t' to 'etc_t'
Notice: /Stage[main]/Pacemaker::Service/Service[pacemaker]/enable: enable changed 'false' to 'true'
Notice: /Stage[main]/Pacemaker::Service/Service[corosync]/enable: enable changed 'false' to 'true'
Notice: /Firewall[998 log all]: Dependency Rabbitmq_user[guest] has failures: true
Notice: /Firewall[999 drop all]: Dependency Rabbitmq_user[guest] has failures: true
Notice: Finished catalog run in 263.71 seconds
(truncated, view all with --long)
deploy_stderr: |
exception: connect failed
Warning: Scope(Class[Mongodb::Server]): Replset specified, but no replset_members or replset_config provided.
Warning: Scope(Haproxy::Config[haproxy]): haproxy: The $merge_options parameter will default to true in the next major release. Please review the documentat
ion regarding the implications.
Error: /Stage[main]/Tripleo::Profile::Base::Rabbitmq/Rabbitmq_user[guest]: Could not evaluate: Command is still failing after 180 seconds expired!
Warning: /Firewall[998 log all]: Skipping because of failed dependencies
Warning: /Firewall[999 drop all]: Skipping because of failed dependencies
overcloud.AllNodesDeploySteps.ControllerDeployment_Step2.0:
resource_type: OS::Heat::StructuredDeployment
physical_resource_id: b3381e5a-11cf-4bc1-978b-247237123870
status: CREATE_FAILED
status_reason: |
Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6
deploy_stdout: |
...
Notice: /Stage[main]/Neutron::Db::Mysql/Openstacklib::Db::Mysql[neutron]/Openstacklib::Db::Mysql::Host_access[ovs_neutron_10.0.0.10]/Mysql_grant[neutron@10.
0.0.10/ovs_neutron.*]/ensure: created
Notice: /Stage[main]/Neutron::Deps/Anchor[neutron::db::end]: Triggered 'refresh' from 1 events
Notice: /Stage[main]/Neutron::Deps/Anchor[neutron::dbsync::begin]: Triggered 'refresh' from 1 events
Notice: /Stage[main]/Glance::Db::Mysql/Openstacklib::Db::Mysql[glance]/Openstacklib::Db::Mysql::Host_access[glance_10.0.0.10]/Mysql_user[glance@10.0.0.10]/e
nsure: created
Notice: /Stage[main]/Glance::Db::Mysql/Openstacklib::Db::Mysql[glance]/Openstacklib::Db::Mysql::Host_access[glance_10.0.0.10]/Mysql_grant[glance@10.0.0.10/g
lance.*]/ensure: created
Notice: /Stage[main]/Glance::Deps/Anchor[glance::db::end]: Triggered 'refresh' from 1 events
Notice: /Stage[main]/Glance::Deps/Anchor[glance::dbsync::begin]: Triggered 'refresh' from 1 events
Notice: /Firewall[998 log all]: Dependency Rabbitmq_user[guest] has failures: true
Notice: /Firewall[999 drop all]: Dependency Rabbitmq_user[guest] has failures: true
Notice: Finished catalog run in 355.15 seconds
(truncated, view all with --long)
deploy_stderr: |
exception: connect failed
Warning: Scope(Class[Mongodb::Server]): Replset specified, but no replset_members or replset_config provided.
Warning: Scope(Haproxy::Config[haproxy]): haproxy: The $merge_options parameter will default to true in the next major release. Please review the documentation regarding the implications.
Error: Could not prefetch mysql_user provider 'mysql': Execution of '/usr/bin/mysql -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
Error: /Stage[main]/Tripleo::Profile::Base::Rabbitmq/Rabbitmq_user[guest]: Could not evaluate: Command is still failing after 180 seconds expired!
Warning: /Firewall[998 log all]: Skipping because of failed dependencies
Warning: /Firewall[999 drop all]: Skipping because of failed dependencies
overcloud.AllNodesDeploySteps.ControllerDeployment_Step2.2:
resource_type: OS::Heat::StructuredDeployment
physical_resource_id: 70e924f4-3818-4dd2-bb74-899e4f2d6567
status: CREATE_FAILED
status_reason: |
Error: resources[2]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6
deploy_stdout: |
...
Notice: /Stage[main]/Swift/Swift_config[swift-constraints/max_header_size]/ensure: created
Notice: /Stage[main]/Swift/Swift_config[swift-hash/swift_hash_path_suffix]/value: value changed '%SWIFT_HASH_PATH_SUFFIX%' to 'Zd8MhmZhhDVBrYqWEDad7sAAJ'
Notice: /Stage[main]/Swift::Deps/Anchor[swift::config::end]: Triggered 'refresh' from 2 events
Notice: /Stage[main]/Swift::Deps/Anchor[swift::service::begin]: Triggered 'refresh' from 2 events
Notice: /File[/etc/localtime]/seltype: seltype changed 'locale_t' to 'etc_t'
Notice: /Stage[main]/Pacemaker::Service/Service[pacemaker]/enable: enable changed 'false' to 'true'
Notice: /Stage[main]/Pacemaker::Service/Service[corosync]/enable: enable changed 'false' to 'true'
Notice: /Firewall[998 log all]: Dependency Rabbitmq_user[guest] has failures: true
Notice: /Firewall[999 drop all]: Dependency Rabbitmq_user[guest] has failures: true
Notice: Finished catalog run in 274.63 seconds
(truncated, view all with --long)
deploy_stderr: |
exception: connect failed
Warning: Scope(Class[Mongodb::Server]): Replset specified, but no replset_members or replset_config provided.
Warning: Scope(Haproxy::Config[haproxy]): haproxy: The $merge_options parameter will default to true in the next major release. Please review the documentation regarding the implications.
Error: /Stage[main]/Tripleo::Profile::Base::Rabbitmq/Rabbitmq_user[guest]: Could not evaluate: Command is still failing after 180 seconds expired!
Warning: /Firewall[998 log all]: Skipping because of failed dependencies
Warning: /Firewall[999 drop all]: Skipping because of failed dependencies
[stack@5b9s38-undercloud ~]$
2017-09-13 07:04:22Z [overcloud-AllNodesDeploySteps-smyquny6jczf.ContrailDpdkDeployment_Step2.0]: CREATE_COMPLETE state changed
2017-09-13 07:04:22Z [overcloud-AllNodesDeploySteps-smyquny6jczf.ContrailDpdkDeployment_Step2]: CREATE_COMPLETE Stack CREATE completed successfully
2017-09-13 07:04:23Z [overcloud-AllNodesDeploySteps-smyquny6jczf.ContrailDpdkDeployment_Step2]: CREATE_COMPLETE state changed
2017-09-13 07:06:08Z [overcloud-AllNodesDeploySteps-smyquny6jczf.ControllerDeployment_Step2.1]: SIGNAL_IN_PROGRESS Signal: deployment 3ea63a30-cebf-4358-b3dc-5
eb1ffbc10c2 failed (6)
2017-09-13 07:06:08Z [overcloud-AllNodesDeploySteps-smyquny6jczf.ControllerDeployment_Step2.1]: CREATE_FAILED Error: resources[1]: Deployment to server failed:
deploy_status_code : Deployment exited with non-zero status code: 6
2017-09-13 07:06:34Z [overcloud-AllNodesDeploySteps-smyquny6jczf.ControllerDeployment_Step2.2]: SIGNAL_IN_PROGRESS Signal: deployment 70e924f4-3818-4dd2-bb74-8
99e4f2d6567 failed (6)
2017-09-13 07:06:36Z [overcloud-AllNodesDeploySteps-smyquny6jczf.ControllerDeployment_Step2.2]: CREATE_FAILED Error: resources[2]: Deployment to server failed:
deploy_status_code : Deployment exited with non-zero status code: 6
2017-09-13 07:07:36Z [overcloud-AllNodesDeploySteps-smyquny6jczf.ControllerDeployment_Step2.0]: SIGNAL_IN_PROGRESS Signal: deployment b3381e5a-11cf-4bc1-978b-2
47237123870 failed (6)
2017-09-13 07:07:36Z [overcloud-AllNodesDeploySteps-smyquny6jczf.ControllerDeployment_Step2.0]: CREATE_FAILED Error: resources[0]: Deployment to server failed:
deploy_status_code : Deployment exited with non-zero status code: 6
2017-09-13 07:07:37Z [overcloud-AllNodesDeploySteps-smyquny6jczf.ControllerDeployment_Step2]: CREATE_FAILED Resource CREATE failed: Error: resources[1]: Deploy
ment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6
2017-09-13 07:07:37Z [overcloud-AllNodesDeploySteps-smyquny6jczf.ControllerDeployment_Step2]: CREATE_FAILED Error: resources.ControllerDeployment_Step2.resourc
es[1]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 6
2017-09-13 07:07:37Z [overcloud-AllNodesDeploySteps-smyquny6jczf]: UPDATE_FAILED Error: resources.ControllerDeployment_Step2.resources[1]: Deployment to server
failed: deploy_status_code: Deployment exited with non-zero status code: 6
2017-09-13 07:07:38Z [AllNodesDeploySteps]: UPDATE_FAILED resources.AllNodesDeploySteps: Error: resources.ControllerDeployment_Step2.resources[1]: Deployment t
o server failed: deploy_status_code: Deployment exited with non-zero status code: 6
2017-09-13 07:07:38Z [overcloud]: UPDATE_FAILED resources.AllNodesDeploySteps: Error: resources.ControllerDeployment_Step2.resources[1]: Deployment to server f
ailed: deploy_status_code: Deployment exited with non-zero status code: 6
Stack overcloud UPDATE_FAILED
[stack@ 5b9s38- undercloud ~]$ openstack stack failures list overcloud AllNodesDeployS teps.Controller Deployment_ Step2.1: :StructuredDepl oyment resource_ id: 3ea63a30- cebf-4358- b3dc-5eb1ffbc10 c2 main]/Swift/ Swift_config[ swift-constrain ts/max_ header_ size]/ensure: created main]/Swift/ Swift_config[ swift-hash/ swift_hash_ path_suffix] /value: value changed '%SWIFT_ HASH_PATH_ SUFFIX% ' to 'Zd8MhmZhhDVBrY qWEDad7sAAJ' main]/Swift: :Deps/Anchor[ swift:: config: :end]: Triggered 'refresh' from 2 events main]/Swift: :Deps/Anchor[ swift:: service: :begin] : Triggered 'refresh' from 2 events etc/localtime] /seltype: seltype changed 'locale_t' to 'etc_t' main]/Pacemaker ::Service/ Service[ pacemaker] /enable: enable changed 'false' to 'true' main]/Pacemaker ::Service/ Service[ corosync] /enable: enable changed 'false' to 'true' user[guest] has failures: true user[guest] has failures: true Mongodb: :Server] ): Replset specified, but no replset_members or replset_config provided. :Config[ haproxy] ): haproxy: The $merge_options parameter will default to true in the next major release. Please review the documentat main]/Tripleo: :Profile: :Base:: Rabbitmq/ Rabbitmq_ user[guest] : Could not evaluate: Command is still failing after 180 seconds expired! AllNodesDeployS teps.Controller Deployment_ Step2.0: :StructuredDepl oyment resource_ id: b3381e5a- 11cf-4bc1- 978b-2472371238 70 main]/Neutron: :Db::Mysql/ Openstacklib: :Db::Mysql[ neutron] /Openstacklib: :Db::Mysql: :Host_access[ ovs_neutron_ 10.0.0. 10]/Mysql_ grant[neutron@ 10.0.0. 10/ovs_ neutron. *]/ensure: created main]/Neutron: :Deps/Anchor[ neutron: :db::end] : Triggered 'refresh' from 1 events main]/Neutron: :Deps/Anchor[ neutron: :dbsync: :begin] : Triggered 'refresh' from 1 events main]/Glance: :Db::Mysql/ Openstacklib: :Db::Mysql[ glance] /Openstacklib: :Db::Mysql: :Host_access[ glance_ 10.0.0. 10]/Mysql_ user[glance@ 10.0.0. 10]/ensure: created main]/Glance: :Db::Mysql/ Openstacklib: :Db::Mysql[ glance] /Openstacklib: :Db::Mysql: :Host_access[ glance_ 10.0.0. 10]/Mysql_ grant[glance@ 10.0.0. 10/glanc. ..
overcloud.
resource_type: OS::Heat:
physical_
status: CREATE_FAILED
status_reason: |
Error: resources[1]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6
deploy_stdout: |
...
Notice: /Stage[
Notice: /Stage[
Notice: /Stage[
Notice: /Stage[
Notice: /File[/
Notice: /Stage[
Notice: /Stage[
Notice: /Firewall[998 log all]: Dependency Rabbitmq_
Notice: /Firewall[999 drop all]: Dependency Rabbitmq_
Notice: Finished catalog run in 263.71 seconds
(truncated, view all with --long)
deploy_stderr: |
exception: connect failed
Warning: Scope(Class[
Warning: Scope(Haproxy:
ion regarding the implications.
Error: /Stage[
Warning: /Firewall[998 log all]: Skipping because of failed dependencies
Warning: /Firewall[999 drop all]: Skipping because of failed dependencies
overcloud.
resource_type: OS::Heat:
physical_
status: CREATE_FAILED
status_reason: |
Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6
deploy_stdout: |
...
Notice: /Stage[
Notice: /Stage[
Notice: /Stage[
Notice: /Stage[
Notice: /Stage[