Mirantis OpenStack

Stack validation (stack-update --dry-run) with --show-nested fails, while stack update itself works

Bug #1802977 reported by Dmitry Sutyagin on 2018-11-12

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Mirantis OpenStack	Fix Released	High	Oleksiy Molchanov	Mirantis OpenStack 9.2-mu-10

Bug Description

Issue was previously reported as https://bugs.launchpad.net/mos/+bug/1799006, however it is present in MOS 9.2 and can be reproduced.

Error is:

2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher [req-10e9d8e5-ac94-4bd6-bacc-03a5f740b37c - admin - default default] Exception during message handling: Property private_network_id not assigned
2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply
2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher incoming.message))
2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch
2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch
2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/osprofiler/profiler.py", line 117, in wrapper
2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher return f(*args, **kwargs)
2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/heat/common/context.py", line 329, in wrapped
2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher return func(self, ctx, *args, **kwargs)
2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/heat/engine/service.py", line 1085, in preview_update_stack
2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher nested_fmt_actions(current_stack, updated_stack, actions)
2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/heat/engine/service.py", line 1077, in nested_fmt_actions
2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher n_actions = update_task.preview()
2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/heat/engine/update.py", line 268, in preview
2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher None):
2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 1062, in preview_update
2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher before_props)
2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 566, in update_template_diff_properties
2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher changed_properties_set = set(k for k in after_props if prop_changed(k))
2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 566, in <genexpr>
2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher changed_properties_set = set(k for k in after_props if prop_changed(k))
2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 563, in prop_changed
2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher return before != after_props.get(key)
2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/_abcoll.py", line 363, in get
2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher return self[key]
2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/heat/engine/properties.py", line 453, in __getitem__
2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher return self._get_property_value(key)
2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/heat/engine/properties.py", line 450, in _get_property_value
2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher raise ValueError(_('Property %s not assigned') % key)
2018-11-12 20:07:18.053 32589 ERROR oslo_messaging.rpc.dispatcher ValueError: Property private_network_id not assigned

If "--show-nested" is not used, then validation works. Stack update also works, to the issue is only with the --dry-run method when --show-nested is used.

To Mirantis staff - I have a lab environment where I have reproduced this issue, can provide access, please contact me in Slack.

See original description

Tags:

Revision history for this message

Dmitry Sutyagin (dsutyagin) wrote on 2018-11-13:

I have identified that in:

File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 1062, in preview_update

The values for IDs which should be extracted from nested stacks are 'None' in the updated_stack structure:

http://paste.openstack.org/show/734732/

If debug log is enabled, I can see that Heat tries to "GET" these resources via going to respective APIs and passes None as ID, fails several times.

Subsequently in:

File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 563, in prop_changed
return before != after_props.get(key)

Heat raises this error because after_props does not contain some of the keys.

Right now it seems that updated_stack does not get populated with any resource IDs known to current_stack, and I'm not sure how this should work.

description:

updated

Revision history for this message

Dmitry Sutyagin (dsutyagin) wrote on 2018-11-13:

p.s. line numbers in tracebacks might be off a bit due to my debugging code inserts

Oleksiy Molchanov (omolchanov) on 2018-11-13

Changed in mos:
assignee:	MOS Maintenance (mos-maintenance) → Oleksiy Molchanov (omolchanov)

Revision history for this message

Oleksiy Molchanov (omolchanov) wrote on 2018-11-20:

After the investigation we have found out that it must be a bug in heat, so we have created a bugreport for heat-team

https://storyboard.openstack.org/#!/story/2004385

Denis Meltsaykin (dmeltsaykin) on 2018-11-27

Changed in mos:
milestone:	9.x-updates → 9.2-mu-10
importance:	Undecided → High
status:	New → Confirmed

Revision history for this message

Oleksiy Molchanov (omolchanov) wrote on 2018-12-03:

Waiting for upstream patch to backport it to mitaka.

Revision history for this message

Oleksiy Molchanov (omolchanov) wrote on 2018-12-04:

https://review.fuel-infra.org/#/c/39911/1

Changed in mos:
status:	Confirmed → In Progress

Oleksiy Molchanov (omolchanov) on 2018-12-05

Changed in mos:
status:	In Progress → Fix Committed

Revision history for this message

Mikhail Samoylov (msamoylov) wrote on 2019-01-25:

Download full text (4.1 KiB)

Steps for verify:
1. Create configs
repro.yaml

heat_template_version: 2015-04-30
resources:
  net:
    type: OS::Neutron::Net
    properties:
      name: repro
  grp:
    type: OS::Heat::ResourceGroup
    properties:
      count: 1
      resource_def:
        type: repro-nested.yaml
        properties:
          private_network_id: { get_resource: net }

repro-nested.yaml

heat_template_version: 2015-04-30
parameters:
  private_network_id:
    type: string
resources:
  unit:
    type: OS::Heat::TestResource
    properties:
      value: { get_param: private_network_id }

openstack stack create repro -t repro.yaml
openstack stack update repro -t repro.yaml --dry-run --show-nested

2. Run verify
heat stack-update repro +-----------+- | state | resource_name +-----------+- | unchanged | net | | | | | | | | | | | | | | | | updated | grp | | | | | | | | | | | | | | | | updated | 0 | | | | | | | | | | | | --template-file repro.yaml --dry-run --show-nested
/>--------------+--------------------------------+-----------------------------------------------------------+
| resource_type | resource_identity |
/>--------------+--------------------------------+-----------------------------------------------------------+
| OS::Neutron::Net | { |
| "stack_name": "repro", |
| "stack_id": "4442b7fe-bb2b-4766-a6dd-4947603a213b", |
| "tenant": "6fe6cd742c384eafa3f853dc7ecb995b", |
| "path": "/resources/net" |
| } |
| OS::Heat::ResourceGroup | { |
| "stack_name": "repro", |
| "stack_id": "4442b7fe-bb2b-4766-a6dd-4947603a213b", |
| "tenant": "6fe6cd742c384eafa3f853dc7ecb995b", |
| "path": "/resources/grp" |
| } |
| file:///root/repro-nested.yaml | { |
| "stack_name": "repro-grp-zpvljfhha3zg", |
| "stack_id": "e233c0dc-e355-4720-95f0-65ae68b30502", |
| "tenant": "6fe6cd742c384eafa3f853dc7ecb995b", |
| "path": "/resources/...

Steps for verify:
1. Create configs
repro.yaml

repro-nested.yaml

heat_template_version: 2015-04-30
parameters:
  private_network_id:
    type: string
resources:
  unit:
    type: OS::Heat::TestResource
    properties:
      value: { get_param: private_network_id }

openstack stack create repro -t repro.yaml
openstack stack update repro -t repro.yaml --dry-run --show-nested

2. Run verify
heat stack-update repro --template-file repro.yaml --dry-run --show-nested
+-----------+---------------+--------------------------------+-----------------------------------------------------------+
| state     | resource_name | resource_type                  | resource_identity                                         |
+-----------+---------------+--------------------------------+-----------------------------------------------------------+
| unchanged | net           | OS::Neutron::Net               | {                                                         |
|           |               |                                |   "stack_name": "repro",                                  |
|           |               |                                |   "stack_id": "4442b7fe-bb2b-4766-a6dd-4947603a213b",     |
|           |               |                                |   "tenant": "6fe6cd742c384eafa3f853dc7ecb995b",           |
|           |               |                                |   "path": "/resources/net"                                |
|           |               |                                | }                                                         |
| updated   | grp           | OS::Heat::ResourceGroup        | {                                                         |
|           |               |                                |   "stack_name": "repro",                                  |
|           |               |                                |   "stack_id": "4442b7fe-bb2b-4766-a6dd-4947603a213b",     |
|           |               |                                |   "tenant": "6fe6cd742c384eafa3f853dc7ecb995b",           |
|           |               |                                |   "path": "/resources/grp"                                |
|           |               |                                | }                                                         |
| updated   | 0             | file:///root/repro-nested.yaml | {                                                         |
|           |               |                                |   "stack_name": "repro-grp-zpvljfhha3zg",                 |
|           |               |                                |   "stack_id": "e233c0dc-e355-4720-95f0-65ae68b30502",     |
|           |               |                                |   "tenant": "6fe6cd742c384eafa3f853dc7ecb995b",           |
|           |               |                                |   "path": "/resources/0"                                  |
|           |               |                                | }                                                         |
| updated   | unit          | OS::Heat::TestResource         | {                                                         |
|           |               |                                |   "stack_name": "repro-grp-zpvljfhha3zg-0-3u5ylabj6wna",  |
|           |               |                                |   "stack_id": "cdf97e3f-fc17-4e17-9501-79ef3188af1c",     |
|           |               |                                |   "tenant": "6fe6cd742c384eafa3f853dc7ecb995b",           |
|           |               |                                |   "path": "/resources/unit"                               |
|           |               |                                | }                                                         |
+-----------+---------------+--------------------------------+-----------------------------------------------------------+

Changed in mos:
status:	Fix Committed → Fix Released

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.