OpenStack Heat

Loading stack using stored context calls keystone

Bug #1570569 reported by Zane Bitter
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Heat
Fix Released
Medium
huangtianhua
OpenStack Heat newton-2 "n2"
Mitaka
Triaged
Medium
Unassigned
Newton
Fix Released
Medium
huangtianhua
OpenStack Heat newton-2 "n2"

Bug Description

When we load a stack from the database using the stored user context, we make a bunch of calls to Keystone to fetch the roles, user_domain_id and project_domain_id. These calls are fixes for bug 1529354 and bug 1531406.

One of the things we pass stored_context=True for is at startup for loading stacks that were left stuck IN_PROGRESS by some previous dead engine. We are unable to do this if Keystone is not available - which is a fairly likely scenario after e.g. a restart of the box.

If we could lazily load the information we need only when it is required, it would not only be more efficient but we could also be sure to always reset stuck stacks at startup regardless of the order in which services start.

Thomas Herve (therve)
Changed in heat:
assignee: nobody → Thomas Herve (therve)
milestone: none → newton-1
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (master)

Fix proposed to branch: master
Review: https://review.openstack.org/315772

Changed in heat:
assignee: Thomas Herve (therve) → Zane Bitter (zaneb)
status: Triaged → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/317604

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to heat (master)

Reviewed: https://review.openstack.org/315772
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=bcadd55a277b8b8170078f3eca7cfe471456e70b
Submitter: Jenkins
Branch: master

commit bcadd55a277b8b8170078f3eca7cfe471456e70b
Author: Zane Bitter <email address hidden>
Date: Thu May 12 16:20:20 2016 -0400

    Lazy-load context information requiring Keystone calls

    When we load a stored context, don't retrieve additional data from keystone
    unless and until it is needed. This makes stacks that don't require the
    full data more efficient, and may help prevent an inability to contact
    keystone at startup blocking the process of resetting stuck stacks.

    Change-Id: Ifa8f5f22686a2ee5c30b8cdb42d921163467b64a
    Co-Authored-By: Thomas Herve <email address hidden>
    Partial-Bug: #1570569

Changed in heat:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Reviewed: https://review.openstack.org/317604
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=026cc94eba48364b35a561fd21e33d6d482ab39d
Submitter: Jenkins
Branch: master

commit 026cc94eba48364b35a561fd21e33d6d482ab39d
Author: Zane Bitter <email address hidden>
Date: Tue May 17 11:08:03 2016 -0400

    Don't use stored context to reset stacks

    The resetting of in-progress stacks at startup is done using an admin
    context. No user actions (e.g. ReST API calls) are performed, so there is
    no need to load the user context that is stored with the stack. (Nor is the
    current context rewritten to the database in the process of resetting the
    state.)

    Since loading the user context from the DB may result in a call to the
    keystone API, eliminating this ensures that stacks can be reset
    successfully even if heat-engine starts before keystone is available.

    Change-Id: I8109c622b7661254a658d78d98f8dc8f756d8755
    Closes-Bug: #1570569

Revision history for this message
huangtianhua (huangtianhua) wrote :

I found a bug, seems introduced by this change, please see https://bugs.launchpad.net/heat/+bug/1584724, and I think we should revert this change first.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (master)

Fix proposed to branch: master
Review: https://review.openstack.org/320348

Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/heat 7.0.0.0b1

This issue was fixed in the openstack/heat 7.0.0.0b1 development milestone.

Revision history for this message
Zane Bitter (zaneb) wrote :

Reopening as the first patch was reverted.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to heat (master)

Reviewed: https://review.openstack.org/320348
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=9d239cc80e218dd75571e33e5171f6fee82023b9
Submitter: Jenkins
Branch: master

commit 9d239cc80e218dd75571e33e5171f6fee82023b9
Author: huangtianhua <email address hidden>
Date: Tue May 24 18:36:46 2016 +0800

    Don't use stored context to reset stacks

    The resetting of in-progress stacks at startup is done using an admin
    context. No user actions (e.g. ReST API calls) are performed, so there is
    no need to load the user context that is stored with the stack. (Nor is
    the
    current context rewritten to the database in the process of resetting the
    state.)

    Since loading the user context from the DB may result in a call to the
    keystone API, eliminating this ensures that stacks can be reset
    successfully even if heat-engine starts before keystone is available.

    Also there is no need to resolve outputs data while reset stacks, so set
    resolve_data=False when load stack.

    Change-Id: I9379ef6cc6f1d3bbc2569ff2150795bfaffee430
    Closes-Bug: #1570569

Changed in heat:
status: In Progress → Fix Released
Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/heat 7.0.0.0b2

This issue was fixed in the openstack/heat 7.0.0.0b2 development milestone.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.