Murano

Any component in murano environment hangs in state 'Creating VM for ...'

  1. Series kilo
  2. Bug #1500200
Bug #1500200 reported by Victor Ryzhenkin
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Murano
Fix Released
Critical
Stan Lagun
Murano mitaka-2 "m2"
Kilo
Fix Committed
Critical
Nikolay Starodubtsev
Murano 2015.1.1
Liberty
Fix Released
Critical
Serg Melikyan
Murano 1.0.1
Mitaka
Fix Released
Critical
Stan Lagun
Murano mitaka-2 "m2"
murano (Ubuntu)
Fix Released
Undecided
Unassigned

Bug Description

Using master branch for all components (affects liberty-rc1 too)

Steps:
1. Deploy Murano with devstack
2. Add any app to environment
3. Deploy environment.

Expected: Application successfully deployed

Actual:
Murano environment hangs in 'DEPLOYING' state.
Component status hangs with state 'Creating VM for Apache Server'

Instance is ready and active (metadata injected successfully, keypair on it, murano-agent armed, config of murano-agent is correct)
Heat stack state is 'Update Complete'

In murano logs we can see endless requests to heat, like this:

2015-09-27 00:11:38.826 19802 DEBUG heatclient.common.http [-]
HTTP/1.1 302 Found
content-length: 208
connection: keep-alive
location: http://192.168.111.10:8004/v1/d42dbd0145344e3fa4132d52e14fb2f9/stacks/murano-lkiutif1ozv3a2/bbff7349-3a4d-4f3e-a0cb-e10fe76b29e2
date: Sun, 27 Sep 2015 00:11:38 GMT
content-type: text/plain; charset=UTF-8
x-openstack-request-id: req-531aa109-6036-41d3-a601-c21ad304e349

302 Found

The resource was found at http://192.168.111.10:8004/v1/d42dbd0145344e3fa4132d52e14fb2f9/stacks/murano-lkiutif1ozv3a2/bbff7349-3a4d-4f3e-a0cb-e10fe76b29e2; you should be redirected automatically.
 log_http_response /usr/local/lib/python2.7/dist-packages/heatclient/common/http.py:142
2015-09-27 00:11:38.826 19802 DEBUG heatclient.common.http [-] curl -g -i -X GET -H 'User-Agent: python-heatclient' -H 'Content-Type: application/json' -H 'Accept: application/json' -H 'X-Auth-Token: {SHA1}7953d
b33f752d12f9df93cbd75e647839b8fa14a' http://192.168.111.10:8004/v1/d42dbd0145344e3fa4132d52e14fb2f9/stacks/murano-lkiutif1ozv3a2/bbff7349-3a4d-4f3e-a0cb-e10fe76b29e2 log_curl_request /usr/local/lib/python2.7/dis
t-packages/heatclient/common/http.py:129

Finally, after an ~hour we got a traceback which connected with auth:

2015-09-27 00:11:50.284 19802 DEBUG keystoneclient.session [-] Request returned failure status: 403 request /usr/local/lib/python2.7/dist-packages/keystoneclient/session.py:400
2015-09-27 00:11:50.377 19802 ERROR murano.common.engine [-]
  keystoneclient.exceptions.AuthorizationFailure: Authorization failed: User is not a trustee. (Disable debug mode to suppress these details.) (HTTP 403) (Request-ID: req-73cdef62-3412-4cd4-8866-98363379472b)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to murano (master)

Fix proposed to branch: master
Review: https://review.openstack.org/228255

Changed in murano:
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to murano (stable/liberty)

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/228287

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to murano (master)

Reviewed: https://review.openstack.org/228255
Committed: https://git.openstack.org/cgit/openstack/murano/commit/?id=8b7e07d8f01a83bc37b01b44888cdca113313286
Submitter: Jenkins
Branch: master

commit 8b7e07d8f01a83bc37b01b44888cdca113313286
Author: Stan Lagun <email address hidden>
Date: Sun Sep 27 18:22:40 2015 +0300

    Murano waited indefinitely for UPDATE_COMPLETE stack status

    On recent Heat version last modification timestamp changes on
    update start rather than finish. As a result when stack exits from
    IN_PROGRESS state timestamp doesn't change any longer.
    But Murano also waited for timestamp to change. As a result
    Murano waited indefinitely and deployment hung.

    With this change IN_PROGRESS status will no longer set
    last modification timestamps. So now we will be comparing
    timestamps between 2 terminal statuses rather than between
    last UPDATE_IN_PROGRESS and UPDATE_COMPLETE that
    now have the same timestamp.

    Change-Id: Idd933d453ef7715e409439a53a3baf9eaa202370
    Closes-Bug: #1500200

Changed in murano:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to murano (stable/liberty)

Reviewed: https://review.openstack.org/228287
Committed: https://git.openstack.org/cgit/openstack/murano/commit/?id=e130ed3c42ef6e837f5f42c7182f0806d4eba104
Submitter: Jenkins
Branch: stable/liberty

commit e130ed3c42ef6e837f5f42c7182f0806d4eba104
Author: Stan Lagun <email address hidden>
Date: Sun Sep 27 18:22:40 2015 +0300

    Murano waited indefinitely for UPDATE_COMPLETE stack status

    On recent Heat version last modification timestamp changes on
    update start rather than finish. As a result when stack exits from
    IN_PROGRESS state timestamp doesn't change any longer.
    But Murano also waited for timestamp to change. As a result
    Murano waited indefinitely and deployment hung.

    With this change IN_PROGRESS status will no longer set
    last modification timestamps. So now we will be comparing
    timestamps between 2 terminal statuses rather than between
    last UPDATE_IN_PROGRESS and UPDATE_COMPLETE that
    now have the same timestamp.

    Change-Id: Idd933d453ef7715e409439a53a3baf9eaa202370
    Closes-Bug: #1500200
    (cherry picked from commit 8b7e07d8f01a83bc37b01b44888cdca113313286)

Serg Melikyan (smelikyan)
Changed in murano:
milestone: liberty-rc2 → 1.0.0
Kirill Zaitsev (kzaitsev)
Changed in murano:
status: Fix Committed → Fix Released
milestone: mitaka-1 → mitaka-2
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to murano (stable/kilo)

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/267303

Revision history for this message
Nikolay Starodubtsev (starodubcevna) wrote :

need to backport this in kilo, as far as this bug affects gate jobs

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to murano (stable/kilo)

Reviewed: https://review.openstack.org/267303
Committed: https://git.openstack.org/cgit/openstack/murano/commit/?id=4a3fc9b07775313e9802c2e11e4f388d75731950
Submitter: Jenkins
Branch: stable/kilo

commit 4a3fc9b07775313e9802c2e11e4f388d75731950
Author: Stan Lagun <email address hidden>
Date: Sun Sep 27 18:22:40 2015 +0300

    Murano waited indefinitely for UPDATE_COMPLETE stack status

    On recent Heat version last modification timestamp changes on
    update start rather than finish. As a result when stack exits from
    IN_PROGRESS state timestamp doesn't change any longer.
    But Murano also waited for timestamp to change. As a result
    Murano waited indefinitely and deployment hung.

    With this change IN_PROGRESS status will no longer set
    last modification timestamps. So now we will be comparing
    timestamps between 2 terminal statuses rather than between
    last UPDATE_IN_PROGRESS and UPDATE_COMPLETE that
    now have the same timestamp.

    Change-Id: Idd933d453ef7715e409439a53a3baf9eaa202370
    Closes-Bug: #1500200

György Szombathelyi (gyurco)
affects: ubuntu → murano (Ubuntu)
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in murano (Ubuntu):
status: New → Confirmed
Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/murano 2015.1.1

This issue was fixed in the openstack/murano 2015.1.1 release.

Chuck Short (zulcss)
Changed in murano (Ubuntu):
status: Confirmed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

This issue was fixed in the openstack/murano 2015.1.1 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.