OPNFV

heat stack creation failed with keystone v3

Bug #1715465 reported by Narinder Gupta
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OPNFV
In Progress
High
Unassigned
OpenStack Heat Charm
Fix Released
High
James Page
OpenStack Heat Charm 17.11

Bug Description

After doing the deployment with latest heat charm running a simple test to create the resource under heat template fails on authentication failure.

cat autoscaling_policy.yaml.template
heat_template_version: 2017-02-24

resources:
  test_group:
    type: OS::Heat::AutoScalingGroup
    properties:
      desired_capacity: 0
      max_size: 0
      min_size: 0
      resource:
        type: OS::Heat::RandomString
  test_policy:
    type: OS::Heat::ScalingPolicy
    properties:
      adjustment_type: change_in_capacity
      auto_scaling_group_id: { get_resource: test_group }
      scaling_adjustment: 1

ubuntu@etsi-ubuntu-jh:~$

heat stack-create --template-file autoscaling_policy.yaml.template heat_st
WARNING (shell) "heat stack-create" is deprecated, please use "openstack stack create" instead
WARNING (shell) "heat stack-list" is deprecated, please use "openstack stack list" instead
+--------------------------------------+------------+--------------------+----------------------+--------------+
| id | stack_name | stack_status | creation_time | updated_time |
+--------------------------------------+------------+--------------------+----------------------+--------------+
| c0a3cb1a-176b-47a6-bef3-b3d146990b38 | heat_st | CREATE_IN_PROGRESS | 2017-09-06T18:35:58Z | None |
+--------------------------------------+------------+--------------------+----------------------+--------------+
ubuntu@etsi-ubuntu-jh:~$

 heat stack-show heat_st
WARNING (shell) "heat stack-show" is deprecated, please use "openstack stack show" instead
+-----------------------+------------------------------------------------------------------------------------------------------------------------+
| Property | Value |
+-----------------------+------------------------------------------------------------------------------------------------------------------------+
| capabilities | [] |
| creation_time | 2017-09-06T18:35:58Z |
| deletion_time | None |
| description | No description |
| disable_rollback | True |
| id | c0a3cb1a-176b-47a6-bef3-b3d146990b38 |
| links | http://192.0.5.194:8004/v1/546499c82a8249fdafb1982979a0baa1/stacks/heat_st/c0a3cb1a-176b-47a6-bef3-b3d146990b38 (self) |
| notification_topics | [] |
| outputs | [] |
| parameters | { |
| | "OS::project_id": "546499c82a8249fdafb1982979a0baa1", |
| | "OS::stack_id": "c0a3cb1a-176b-47a6-bef3-b3d146990b38", |
| | "OS::stack_name": "heat_st" |
| | } |
| parent | None |
| stack_name | heat_st |
| stack_owner | None |
| stack_status | CREATE_FAILED |
| stack_status_reason | Resource CREATE failed: AuthorizationFailure: |
| | resources.test_policy: Authorization failed. |
| stack_user_project_id | None |
| tags | null |
| template_description | No description |
| timeout_mins | None |
| updated_time | None |
+-----------------------+------------------------------------------------------------------------------------------------------------------------+

Revision history for this message
Narinder Gupta (narindergupta) wrote :

heat log sho the below error.
2017-09-06 10:56:49.663 27620 WARNING oslo_config.cfg [-] Option "rabbit_password" from group "oslo_messaging_rabbit" is deprecated for removal. Its value may be silently ignored in the future.
2017-09-06 17:01:16.177 27616 WARNING oslo_config.cfg [req-d4e1387c-9a1a-4957-9e19-cc66fe5fa44c - admin - - -] Option "auth_plugin" from group "trustee" is deprecated. Use option "auth_type" from group "trustee".
2017-09-06 17:01:16.925 27616 ERROR heat.engine.clients.keystoneclient [req-d4e1387c-9a1a-4957-9e19-cc66fe5fa44c - admin - - -] Domain admin client authentication failed
2017-09-06 17:01:17.673 27617 ERROR heat.engine.clients.keystoneclient [req-d4e1387c-9a1a-4957-9e19-cc66fe5fa44c - admin - - -] Domain admin client authentication failed
2017-09-06 17:01:19.341 27619 ERROR heat.engine.clients.keystoneclient [req-d4e1387c-9a1a-4957-9e19-cc66fe5fa44c - admin - - -] Domain admin client authentication failed
2017-09-06 17:03:21.484 27620 WARNING oslo_config.cfg [req-729f984d-ac3b-4cba-b679-22fa46711ece - admin - - -] Option "auth_plugin" from group "trustee" is deprecated. Use option "auth_type" from group "trustee".
2017-09-06 17:03:22.165 27620 ERROR heat.engine.clients.keystoneclient [req-729f984d-ac3b-4cba-b679-22fa46711ece - admin - - -] Domain admin client authentication failed
2017-09-06 17:03:22.892 27618 ERROR heat.engine.clients.keystoneclient [req-729f984d-ac3b-4cba-b679-22fa46711ece - admin - - -] Domain admin client authentication failed
2017-09-06 17:03:24.497 27618 ERROR heat.engine.clients.keystoneclient [req-729f984d-ac3b-4cba-b679-22fa46711ece - admin - - -] Domain admin client authentication failed
2017-09-06 18:32:27.573 27617 WARNING oslo_config.cfg [req-3b132976-9f5a-4e68-8719-9d1597be163e - admin - - -] Option "auth_plugin" from group "trustee" is deprecated. Use option "auth_type" from group "trustee".
2017-09-06 18:32:28.194 27617 ERROR heat.engine.clients.keystoneclient [req-3b132976-9f5a-4e68-8719-9d1597be163e - admin - - -] Domain admin client authentication failed
2017-09-06 18:32:28.948 27620 ERROR heat.engine.clients.keystoneclient [req-3b132976-9f5a-4e68-8719-9d1597be163e - admin - - -] Domain admin client authentication failed
2017-09-06 18:32:30.553 27620 ERROR heat.engine.clients.keystoneclient [req-3b132976-9f5a-4e68-8719-9d1597be163e - admin - - -] Domain admin client authentication failed
2017-09-06 18:35:58.046 27616 ERROR heat.engine.clients.keystoneclient [req-2ccc34cd-821a-47cd-be03-4350c1f880f6 - admin - - -] Domain admin client authentication failed
2017-09-06 18:35:58.759 27617 ERROR heat.engine.clients.keystoneclient [req-2ccc34cd-821a-47cd-be03-4350c1f880f6 - admin - - -] Domain admin client authentication failed
2017-09-06 18:36:00.306 27617 ERROR heat.engine.clients.keystoneclient [req-2ccc34cd-821a-47cd-be03-4350c1f880f6 - admin - - -] Domain admin client authentication failed

complete logs can be found at

sudo pastebinit < /var/log/heat/heat-engine.log
http://paste.ubuntu.com/25479762/

Revision history for this message
Narinder Gupta (narindergupta) wrote :

cat joid_config/admin-openrc
export OS_AUTH_URL=http://192.0.5.229:5000/v3
export OS_USER_DOMAIN_NAME=admin_domain
export OS_PROJECT_DOMAIN_NAME=admin_domain
export OS_USERNAME=admin
export OS_TENANT_NAME=admin
export OS_PROJECT_NAME=admin
export OS_PASSWORD=openstack
export OS_IDENTITY_API_VERSION=3
export OS_REGION_NAME=RegionOne

Revision history for this message
Jason Hobbs (jason-hobbs) wrote :

I tested this on our lab and was able to reproduce the issue, just as narinder reported above.

Revision history for this message
Nobuto Murata (nobuto) wrote :

Out of curiosity, did you execute the heat action below before trying to use Heat?

$ juju run-action heat/0 domain-setup

Revision history for this message
Narinder Gupta (narindergupta) wrote :

yes thats part of post deployment step we execute and I can confirm that was executed as we have CI in OPNFV community and we are able to reproduce there also.

Revision history for this message
huangsm (huangsm) wrote :

I test this on my lab, heat_template_version: 2016-04-08, not 2017-02-24.
But the issue don't be reproduced.
can you show your heat.conf

Revision history for this message
Narinder Gupta (narindergupta) wrote :

here is my heat.conf

http://paste.ubuntu.com/25488358/
 and openrc file

cat joid_config/admin-openrc
export OS_AUTH_URL=http://192.0.5.229:5000/v3
export OS_USER_DOMAIN_NAME=admin_domain
export OS_PROJECT_DOMAIN_NAME=admin_domain
export OS_USERNAME=admin
export OS_TENANT_NAME=admin
export OS_PROJECT_NAME=admin
export OS_PASSWORD=openstack
export OS_IDENTITY_API_VERSION=3
export OS_REGION_NAME=RegionOne

Revision history for this message
Narinder Gupta (narindergupta) wrote :

This is for OPNFV E release which is based on Ocata and current charms integrated in this is 17.08 bundle file is as follows.
http://paste.ubuntu.com/25532647/

Where charm has been downloaded locally first before deployment. Charm download script is available at:

https://gerrit.opnfv.org/gerrit/gitweb?p=joid.git;a=blob;f=ci/nosdn/fetch-charms.sh

Revision history for this message
James Page (james-page) wrote :

The role assignment for the heat_domain_admin was incorrectly scoped for v3 deployments; raising a fix now.

Changed in charm-heat:
status: New → Confirmed
importance: Undecided → High
assignee: nobody → James Page (james-page)
status: Confirmed → In Progress
Revision history for this message
James Page (james-page) wrote :

https://review.openstack.org/504219

Revision history for this message
Narinder Gupta (narindergupta) wrote :
Download full text (4.3 KiB)

I can confirm the patch posted by James fixes the problem here is the result after the fixes.

ubuntu@etsi-ubuntu-jh:~$ heat stack-show heat_st
WARNING (shell) "heat stack-show" is deprecated, please use "openstack stack show" instead
+-----------------------+------------------------------------------------------------------------------------------------------------------------+
| Property | Value |
 +-----------------------+------------------------------------------------------------------------------------------------------------------------+
| capabilities | [] |
| creation_time | 2017-09-14T21:27:09Z |
| deletion_time | None |
| description | No description |
| disable_rollback | True |
| id | 81694895-0fd0-49ec-a468-0e2f150ccc21 |
| links | http://192.0.5.159:8004/v1/2f236fb1e55c4db9841f64b7b386426c/stacks/heat_st/81694895-0fd0-49ec-a468-0e2f150ccc21 (self) |
| notification_topics | [] |
| outputs | [] |
| parameters | { |
| | "OS::project_id": "2f236fb1e55c4db9841f64b7b386426c", |
| | "OS::stack_id": "81694895-0fd0-49ec-a468-0e2f150ccc21", |
| | "OS::stack_name": "heat_st" |
| | } |
| parent | None |
| stack_name | heat_st |
| stack_owner | None |
| stack_status |...

Read more...

Narinder Gupta (narindergupta)
Changed in opnfv:
status: New → In Progress
importance: Undecided → High
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-heat (master)

Reviewed: https://review.openstack.org/504219
Committed: https://git.openstack.org/cgit/openstack/charm-heat/commit/?id=501e3415b346322769947ddfb5665e56490b3e3f
Submitter: Jenkins
Branch: master

commit 501e3415b346322769947ddfb5665e56490b3e3f
Author: James Page <email address hidden>
Date: Thu Sep 14 14:20:49 2017 -0600

    Correctly scope role assignment for heat_domain_admin

    The admin role assignment for the heat_domain_admin user
    needs to be scoped to the heat domain; update domain-setup
    action to scope the role assigment correctly.

    This change also stops ignoring the return code of the
    role assignment; re-assigning a role already granted is
    idemponent in openstackclient.

    Change-Id: Ia5ba409e2f566614e6db9bbf9540c563af8e4a82
    Closes-Bug: 1715465

Changed in charm-heat:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-heat (stable/17.08)

Fix proposed to branch: stable/17.08
Review: https://review.openstack.org/505614

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-heat (stable/17.08)

Reviewed: https://review.openstack.org/505614
Committed: https://git.openstack.org/cgit/openstack/charm-heat/commit/?id=f30723b839fb91f0f50873b16cc724c02845e324
Submitter: Jenkins
Branch: stable/17.08

commit f30723b839fb91f0f50873b16cc724c02845e324
Author: James Page <email address hidden>
Date: Thu Sep 14 14:20:49 2017 -0600

    Correctly scope role assignment for heat_domain_admin

    The admin role assignment for the heat_domain_admin user
    needs to be scoped to the heat domain; update domain-setup
    action to scope the role assigment correctly.

    This change also stops ignoring the return code of the
    role assignment; re-assigning a role already granted is
    idemponent in openstackclient.

    Change-Id: Ia5ba409e2f566614e6db9bbf9540c563af8e4a82
    Closes-Bug: 1715465
    (cherry picked from commit 501e3415b346322769947ddfb5665e56490b3e3f)

James Page (james-page)
Changed in charm-heat:
milestone: none → 17.11
James Page (james-page)
Changed in charm-heat:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.