| 2017-09-13 17:51:18 |
shajuvk |
bug |
|
|
added bug |
| 2017-09-13 17:51:27 |
shajuvk |
nominated for series |
|
juniperopenstack/r4.0 |
|
| 2017-09-13 17:51:27 |
shajuvk |
bug task added |
|
juniperopenstack/r4.0 |
|
| 2017-09-13 17:51:27 |
shajuvk |
nominated for series |
|
juniperopenstack/trunk |
|
| 2017-09-13 17:51:27 |
shajuvk |
bug task added |
|
juniperopenstack/trunk |
|
| 2017-09-13 18:33:38 |
shajuvk |
juniperopenstack/r4.0: assignee |
|
alexey-mr (alexey-morlang) |
|
| 2017-09-13 18:33:42 |
shajuvk |
juniperopenstack/r4.0: milestone |
|
r4.0.1.0 |
|
| 2017-09-13 18:33:46 |
shajuvk |
juniperopenstack/trunk: milestone |
|
r4.1.0.0-fcs |
|
| 2017-09-13 18:33:53 |
shajuvk |
juniperopenstack/trunk: assignee |
|
alexey-mr (alexey-morlang) |
|
| 2017-09-13 18:33:58 |
shajuvk |
juniperopenstack/r4.0: importance |
Undecided |
High |
|
| 2017-09-13 18:33:59 |
shajuvk |
juniperopenstack/trunk: importance |
Undecided |
High |
|
| 2017-09-13 18:36:13 |
shajuvk |
description |
...
Notice: /Stage[main]/Swift/Swift_config[swift-constraints/max_header_size]/ensure: created
Notice: /Stage[main]/Swift/Swift_config[swift-hash/swift_hash_path_suffix]/value: value changed '%SWIFT_HASH_PATH_SUFFIX%' to 'Zd8MhmZhhDVBrYqWEDad7sAAJ'
Notice: /Stage[main]/Swift::Deps/Anchor[swift::config::end]: Triggered 'refresh' from 2 events
Notice: /Stage[main]/Swift::Deps/Anchor[swift::service::begin]: Triggered 'refresh' from 2 events
Notice: /File[/etc/localtime]/seltype: seltype changed 'locale_t' to 'etc_t'
Notice: /Stage[main]/Pacemaker::Service/Service[pacemaker]/enable: enable changed 'false' to 'true'
Notice: /Stage[main]/Pacemaker::Service/Service[corosync]/enable: enable changed 'false' to 'true'
Notice: /Firewall[998 log all]: Dependency Rabbitmq_user[guest] has failures: true
Notice: /Firewall[999 drop all]: Dependency Rabbitmq_user[guest] has failures: true
Notice: Finished catalog run in 274.63 seconds
(truncated, view all with --long)
deploy_stderr: |
exception: connect failed
Warning: Scope(Class[Mongodb::Server]): Replset specified, but no replset_members or replset_config provided.
Warning: Scope(Haproxy::Config[haproxy]): haproxy: The $merge_options parameter will default to true in the next major release. Please review the documentation regarding the implications.
Error: /Stage[main]/Tripleo::Profile::Base::Rabbitmq/Rabbitmq_user[guest]: Could not evaluate: Command is still failing after 180 seconds expired!
Warning: /Firewall[998 log all]: Skipping because of failed dependencies
Warning: /Firewall[999 drop all]: Skipping because of failed dependencies
[stack@5b9s38-undercloud ~]$
[stack@5b9s38-undercloud ~]$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 170G 48G 123G 28% /
devtmpfs 7.8G 0 7.8G 0% /dev
tmpfs 7.8G 4.0K 7.8G 1% /dev/shm
tmpfs 7.8G 604K 7.8G 1% /run
tmpfs 7.8G 0 7.8G 0% /sys/fs/cgroup
tmpfs 1.6G 0 1.6G 0% /run/user/0 |
I am seeing below failure in overcloud controller with Newton build 30. Could you please check my system ?
Note: Contrail repo created with (Ocata contrail-docker-images + newton contrail-install-packages). Since the failure in openstack node, this might not be the issue with our contrail repo.
Setup : 10.87.67.42 root/c0ntrail123
[root@overcloud-controller-0 ~]# grep Error /var/log/messages
Sep 13 02:29:53 localhost os-collect-config: [2017-09-13 02:29:53,945] (heat-config) [INFO] {"deploy_stdout": "/ /var/lib/heat-config/heat-config-script\n/var/lib/heat-config/heat-config-script\n", "deploy_stderr": "\ngzip: stdin: not in gzip format\ntar: Child returned status 1\ntar: Error is not recoverable: exiting now\n", "deploy_status_code": 0}
Sep 13 02:29:53 localhost os-collect-config: tar: Error is not recoverable: exiting now
Sep 13 02:57:07 localhost os-collect-config: [2017-09-13 02:57:07,601] (heat-config) [INFO] {"deploy_stdout": "/ /var/lib/heat-config/heat-config-script\n/var/lib/heat-config/heat-config-script\n", "deploy_stderr": "\ngzip: stdin: not in gzip format\ntar: Child returned status 1\ntar: Error is not recoverable: exiting now\n", "deploy_status_code": 0}
Sep 13 02:57:07 localhost os-collect-config: tar: Error is not recoverable: exiting now
Sep 13 03:07:33 localhost os-collect-config: ib::Db::Mysql::Host_access[glance_10.0.0.10]/Mysql_user[glance@10.0.0.10]/ensure: created\u001b[0m\n\u001b[mNotice: /Stage[main]/Glance::Db::Mysql/Openstacklib::Db::Mysql[glance]/Openstacklib::Db::Mysql::Host_access[glance_10.0.0.10]/Mysql_grant[glance@10.0.0.10/glance.*]/ensure: created\u001b[0m\n\u001b[mNotice: /Stage[main]/Glance::Deps/Anchor[glance::db::end]: Triggered 'refresh' from 1 events\u001b[0m\n\u001b[mNotice: /Stage[main]/Glance::Deps/Anchor[glance::dbsync::begin]: Triggered 'refresh' from 1 events\u001b[0m\n\u001b[mNotice: /Firewall[998 log all]: Dependency Rabbitmq_user[guest] has failures: true\u001b[0m\n\u001b[mNotice: /Firewall[999 drop all]: Dependency Rabbitmq_user[guest] has failures: true\u001b[0m\n\u001b[mNotice: Finished catalog run in 355.15 seconds\u001b[0m\n", "deploy_stderr": "exception: connect failed\n\u001b[1;31mWarning: Scope(Class[Mongodb::Server]): Replset specified, but no replset_members or replset_config provided.\u001b[0m\n\u001b[1;31mWarning: Scope(Haproxy::Config[haproxy]): haproxy: The $merge_options parameter will default to true in the next major release. Please review the documentation regarding the implications.\u001b[0m\n\u001b[1;31mError: Could not prefetch mysql_user provider 'mysql': Execution of '/usr/bin/mysql -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)\u001b[0m\n\u001b[1;31mError: /Stage[main]/Tripleo::Profile::Base::Rabbitmq/Rabbitmq_user[guest]: Could not evaluate: Command is still failing after 180 seconds expired!\u001b[0m\n\u001b[1;31mWarning: /Firewall[998 log all]: Skipping because of failed dependencies\u001b[0m\n\u001b[1;31mWarning: /Firewall[999 drop all]: Skipping because of failed dependencies\u001b[0m\n", "deploy_status_code": 6}
Sep 13 03:07:33 localhost os-collect-config: #033[1;31mError: Could not prefetch mysql_user provider 'mysql': Execution of '/usr/bin/mysql -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)#033[0m
Sep 13 03:07:33 localhost os-collect-config: #033[1;31mError: /Stage[main]/Tripleo::Profile::Base::Rabbitmq/Rabbitmq_user[guest]: Could not evaluate: Command is still failing after 180 seconds expired!#033[0m
Sep 13 03:07:33 localhost os-collect-config: [2017-09-13 07:07:33,929] (heat-config) [ERROR] Error running /var/lib/heat-config/heat-config-puppet/32c2e3a0-c3e8-47ea-9776-21274d72214e.pp. [6]
Sep 13 03:07:53 localhost lrmd[26203]: notice: rabbitmq_start_0:46901:stderr [ Error performing operation: No such device or address ]
Sep 13 03:07:53 localhost lrmd[26203]: notice: rabbitmq_start_0:46901:stderr [ Error: unable to connect to node 'rabbit@overcloud-controller-0': nodedown ]
Sep 13 03:07:53 localhost lrmd[26203]: notice: rabbitmq_start_0:46901:stderr [ Error: operation forget_cluster_node used with invalid parameter: [] ]
[root@overcloud-controller-0 ~]#
=============
[stack@5b9s38-undercloud ~]$ openstack stack failures list overcloud
overcloud.AllNodesDeploySteps.ControllerDeployment_Step2.1:
resource_type: OS::Heat::StructuredDeployment
physical_resource_id: 3ea63a30-cebf-4358-b3dc-5eb1ffbc10c2
status: CREATE_FAILED
status_reason: |
Error: resources[1]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6
deploy_stdout: |
...
Notice: /Stage[main]/Swift/Swift_config[swift-constraints/max_header_size]/ensure: created
Notice: /Stage[main]/Swift/Swift_config[swift-hash/swift_hash_path_suffix]/value: value changed '%SWIFT_HASH_PATH_SUFFIX%' to 'Zd8MhmZhhDVBrYqWEDad7sAAJ'
Notice: /Stage[main]/Swift::Deps/Anchor[swift::config::end]: Triggered 'refresh' from 2 events
Notice: /Stage[main]/Swift::Deps/Anchor[swift::service::begin]: Triggered 'refresh' from 2 events
Notice: /File[/etc/localtime]/seltype: seltype changed 'locale_t' to 'etc_t'
Notice: /Stage[main]/Pacemaker::Service/Service[pacemaker]/enable: enable changed 'false' to 'true'
Notice: /Stage[main]/Pacemaker::Service/Service[corosync]/enable: enable changed 'false' to 'true'
Notice: /Firewall[998 log all]: Dependency Rabbitmq_user[guest] has failures: true
Notice: /Firewall[999 drop all]: Dependency Rabbitmq_user[guest] has failures: true
Notice: Finished catalog run in 263.71 seconds
(truncated, view all with --long)
deploy_stderr: |
exception: connect failed
Warning: Scope(Class[Mongodb::Server]): Replset specified, but no replset_members or replset_config provided.
Warning: Scope(Haproxy::Config[haproxy]): haproxy: The $merge_options parameter will default to true in the next major release. Please review the documentat
ion regarding the implications.
Error: /Stage[main]/Tripleo::Profile::Base::Rabbitmq/Rabbitmq_user[guest]: Could not evaluate: Command is still failing after 180 seconds expired!
Warning: /Firewall[998 log all]: Skipping because of failed dependencies
Warning: /Firewall[999 drop all]: Skipping because of failed dependencies
overcloud.AllNodesDeploySteps.ControllerDeployment_Step2.0:
resource_type: OS::Heat::StructuredDeployment
physical_resource_id: b3381e5a-11cf-4bc1-978b-247237123870
status: CREATE_FAILED
status_reason: |
Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6
deploy_stdout: |
...
Notice: /Stage[main]/Neutron::Db::Mysql/Openstacklib::Db::Mysql[neutron]/Openstacklib::Db::Mysql::Host_access[ovs_neutron_10.0.0.10]/Mysql_grant[neutron@10.
0.0.10/ovs_neutron.*]/ensure: created
Notice: /Stage[main]/Neutron::Deps/Anchor[neutron::db::end]: Triggered 'refresh' from 1 events
Notice: /Stage[main]/Neutron::Deps/Anchor[neutron::dbsync::begin]: Triggered 'refresh' from 1 events
Notice: /Stage[main]/Glance::Db::Mysql/Openstacklib::Db::Mysql[glance]/Openstacklib::Db::Mysql::Host_access[glance_10.0.0.10]/Mysql_user[glance@10.0.0.10]/e
nsure: created
Notice: /Stage[main]/Glance::Db::Mysql/Openstacklib::Db::Mysql[glance]/Openstacklib::Db::Mysql::Host_access[glance_10.0.0.10]/Mysql_grant[glance@10.0.0.10/g
lance.*]/ensure: created
Notice: /Stage[main]/Glance::Deps/Anchor[glance::db::end]: Triggered 'refresh' from 1 events
Notice: /Stage[main]/Glance::Deps/Anchor[glance::dbsync::begin]: Triggered 'refresh' from 1 events
Notice: /Firewall[998 log all]: Dependency Rabbitmq_user[guest] has failures: true
Notice: /Firewall[999 drop all]: Dependency Rabbitmq_user[guest] has failures: true
Notice: Finished catalog run in 355.15 seconds
(truncated, view all with --long)
deploy_stderr: |
exception: connect failed
Warning: Scope(Class[Mongodb::Server]): Replset specified, but no replset_members or replset_config provided.
Warning: Scope(Haproxy::Config[haproxy]): haproxy: The $merge_options parameter will default to true in the next major release. Please review the documentation regarding the implications.
Error: Could not prefetch mysql_user provider 'mysql': Execution of '/usr/bin/mysql -NBe SELECT CONCAT(User, '@',Host) AS User FROM mysql.user' returned 1: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)
Error: /Stage[main]/Tripleo::Profile::Base::Rabbitmq/Rabbitmq_user[guest]: Could not evaluate: Command is still failing after 180 seconds expired!
Warning: /Firewall[998 log all]: Skipping because of failed dependencies
Warning: /Firewall[999 drop all]: Skipping because of failed dependencies
overcloud.AllNodesDeploySteps.ControllerDeployment_Step2.2:
resource_type: OS::Heat::StructuredDeployment
physical_resource_id: 70e924f4-3818-4dd2-bb74-899e4f2d6567
status: CREATE_FAILED
status_reason: |
Error: resources[2]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6
deploy_stdout: |
...
Notice: /Stage[main]/Swift/Swift_config[swift-constraints/max_header_size]/ensure: created
Notice: /Stage[main]/Swift/Swift_config[swift-hash/swift_hash_path_suffix]/value: value changed '%SWIFT_HASH_PATH_SUFFIX%' to 'Zd8MhmZhhDVBrYqWEDad7sAAJ'
Notice: /Stage[main]/Swift::Deps/Anchor[swift::config::end]: Triggered 'refresh' from 2 events
Notice: /Stage[main]/Swift::Deps/Anchor[swift::service::begin]: Triggered 'refresh' from 2 events
Notice: /File[/etc/localtime]/seltype: seltype changed 'locale_t' to 'etc_t'
Notice: /Stage[main]/Pacemaker::Service/Service[pacemaker]/enable: enable changed 'false' to 'true'
Notice: /Stage[main]/Pacemaker::Service/Service[corosync]/enable: enable changed 'false' to 'true'
Notice: /Firewall[998 log all]: Dependency Rabbitmq_user[guest] has failures: true
Notice: /Firewall[999 drop all]: Dependency Rabbitmq_user[guest] has failures: true
Notice: Finished catalog run in 274.63 seconds
(truncated, view all with --long)
deploy_stderr: |
exception: connect failed
Warning: Scope(Class[Mongodb::Server]): Replset specified, but no replset_members or replset_config provided.
Warning: Scope(Haproxy::Config[haproxy]): haproxy: The $merge_options parameter will default to true in the next major release. Please review the documentation regarding the implications.
Error: /Stage[main]/Tripleo::Profile::Base::Rabbitmq/Rabbitmq_user[guest]: Could not evaluate: Command is still failing after 180 seconds expired!
Warning: /Firewall[998 log all]: Skipping because of failed dependencies
Warning: /Firewall[999 drop all]: Skipping because of failed dependencies
[stack@5b9s38-undercloud ~]$
2017-09-13 07:04:22Z [overcloud-AllNodesDeploySteps-smyquny6jczf.ContrailDpdkDeployment_Step2.0]: CREATE_COMPLETE state changed
2017-09-13 07:04:22Z [overcloud-AllNodesDeploySteps-smyquny6jczf.ContrailDpdkDeployment_Step2]: CREATE_COMPLETE Stack CREATE completed successfully
2017-09-13 07:04:23Z [overcloud-AllNodesDeploySteps-smyquny6jczf.ContrailDpdkDeployment_Step2]: CREATE_COMPLETE state changed
2017-09-13 07:06:08Z [overcloud-AllNodesDeploySteps-smyquny6jczf.ControllerDeployment_Step2.1]: SIGNAL_IN_PROGRESS Signal: deployment 3ea63a30-cebf-4358-b3dc-5
eb1ffbc10c2 failed (6)
2017-09-13 07:06:08Z [overcloud-AllNodesDeploySteps-smyquny6jczf.ControllerDeployment_Step2.1]: CREATE_FAILED Error: resources[1]: Deployment to server failed:
deploy_status_code : Deployment exited with non-zero status code: 6
2017-09-13 07:06:34Z [overcloud-AllNodesDeploySteps-smyquny6jczf.ControllerDeployment_Step2.2]: SIGNAL_IN_PROGRESS Signal: deployment 70e924f4-3818-4dd2-bb74-8
99e4f2d6567 failed (6)
2017-09-13 07:06:36Z [overcloud-AllNodesDeploySteps-smyquny6jczf.ControllerDeployment_Step2.2]: CREATE_FAILED Error: resources[2]: Deployment to server failed:
deploy_status_code : Deployment exited with non-zero status code: 6
2017-09-13 07:07:36Z [overcloud-AllNodesDeploySteps-smyquny6jczf.ControllerDeployment_Step2.0]: SIGNAL_IN_PROGRESS Signal: deployment b3381e5a-11cf-4bc1-978b-2
47237123870 failed (6)
2017-09-13 07:07:36Z [overcloud-AllNodesDeploySteps-smyquny6jczf.ControllerDeployment_Step2.0]: CREATE_FAILED Error: resources[0]: Deployment to server failed:
deploy_status_code : Deployment exited with non-zero status code: 6
2017-09-13 07:07:37Z [overcloud-AllNodesDeploySteps-smyquny6jczf.ControllerDeployment_Step2]: CREATE_FAILED Resource CREATE failed: Error: resources[1]: Deploy
ment to server failed: deploy_status_code : Deployment exited with non-zero status code: 6
2017-09-13 07:07:37Z [overcloud-AllNodesDeploySteps-smyquny6jczf.ControllerDeployment_Step2]: CREATE_FAILED Error: resources.ControllerDeployment_Step2.resourc
es[1]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 6
2017-09-13 07:07:37Z [overcloud-AllNodesDeploySteps-smyquny6jczf]: UPDATE_FAILED Error: resources.ControllerDeployment_Step2.resources[1]: Deployment to server
failed: deploy_status_code: Deployment exited with non-zero status code: 6
2017-09-13 07:07:38Z [AllNodesDeploySteps]: UPDATE_FAILED resources.AllNodesDeploySteps: Error: resources.ControllerDeployment_Step2.resources[1]: Deployment t
o server failed: deploy_status_code: Deployment exited with non-zero status code: 6
2017-09-13 07:07:38Z [overcloud]: UPDATE_FAILED resources.AllNodesDeploySteps: Error: resources.ControllerDeployment_Step2.resources[1]: Deployment to server f
ailed: deploy_status_code: Deployment exited with non-zero status code: 6
Stack overcloud UPDATE_FAILED |
|
| 2017-09-13 18:36:36 |
shajuvk |
tags |
|
ospd provisioning |
|
| 2017-09-13 18:37:25 |
shajuvk |
information type |
Proprietary |
Public |
|
| 2017-09-13 19:08:34 |
shajuvk |
tags |
ospd provisioning |
blocker ospd provisioning |
|
| 2017-09-13 19:08:45 |
shajuvk |
juniperopenstack/r4.0: importance |
High |
Critical |
|
| 2017-09-13 19:08:48 |
shajuvk |
juniperopenstack/trunk: importance |
High |
Critical |
|
| 2017-09-14 05:08:32 |
Andrey Pavlov |
bug |
|
|
added subscriber alexey-mr |
| 2017-09-14 09:36:28 |
alexey-mr |
juniperopenstack/r4.0: assignee |
alexey-mr (alexey-morlang) |
shajuvk (shajuvk) |
|
| 2017-09-14 09:36:50 |
alexey-mr |
juniperopenstack/trunk: assignee |
alexey-mr (alexey-morlang) |
shajuvk (shajuvk) |
|
| 2017-09-14 23:26:20 |
Rudra Rugge |
juniperopenstack/r4.0: status |
New |
Invalid |
|
| 2017-09-14 23:26:24 |
Rudra Rugge |
juniperopenstack/trunk: status |
New |
Invalid |
|
