Heat asg stuck in UPDATE_IN_PROGRESS

Bug #1559132 reported by Rakesh H S
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Heat
Fix Released
Medium
Rabi Mishra

Bug Description

sample template to reproduce this issue:
-------------------------------------------------------------------
heat_template_version: 2013-05-23

resources:
  test_group:
    type: OS::Heat::AutoScalingGroup
    properties:
      cooldown: 120
      desired_capacity: 1
      max_size: 5
      min_size: 1
      resource:
        type: OS::Heat::RandomString

  test_policy:
    type: OS::Heat::ScalingPolicy
    properties:
      adjustment_type: change_in_capacity
      auto_scaling_group_id: { get_resource: test_group }
      scaling_adjustment: 1
------------------------------------------------------------------------

Once the above template is created, change some properties, let's say 'scaling_adjustment: 2' and give an update.
The stack will be stuck in UPDATE_IN_PROGRESS.

This is because, once creation is done, we set the cooldown as 120.
Now when the next update was given, asg will think it is in cooldown time and the stack will be stuck in UPDATE_IN_PROGRESS.

This ^^ is clearly a bug, since the customer will only be able to update the stack after cooldown(120s) time.

Rakesh H S (rh-s)
Changed in heat:
assignee: nobody → Rakesh H S (rh-s)
Revision history for this message
Rabi Mishra (rabi) wrote :

Rakesh, I'm handling some of these issues in https://review.openstack.org/#/c/294023/ as part of another bug. You can give the patch a try and let me know.

Revision history for this message
Rabi Mishra (rabi) wrote :

This above mentioned issue happens as you can't update the stack when scaling is in progress and the stack get's stuck in UPDATE_IN_PROGRESS. I think the patch I mentioned above handles it. I'll use the bug in that patch.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (master)

Fix proposed to branch: master
Review: https://review.openstack.org/294944

Changed in heat:
assignee: Rakesh H S (rh-s) → Rabi Mishra (rabi)
status: New → In Progress
Rabi Mishra (rabi)
Changed in heat:
milestone: none → newton-1
importance: Undecided → Medium
tags: added: mitaka-backport-potential
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/298154

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to heat (master)

Reviewed: https://review.openstack.org/294944
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=6bd7352dc4838b8ef782f2345a4dfdf57ba3e356
Submitter: Jenkins
Branch: master

commit 6bd7352dc4838b8ef782f2345a4dfdf57ba3e356
Author: Rabi Mishra <email address hidden>
Date: Sat Mar 19 19:33:42 2016 +0530

    Fix update issues with autoscaling group

    - Fix update stuck in IN_PROGRESS if issued within
    the cooldown period after CREATE_COMPLETE.

    - Fix update stuck in IN_PROGRESS if issued when scaling
    is happening.

    Change-Id: I31074783f525ea811056e8df7ff46545e58075b5
    Closes-Bug: #1559132

Changed in heat:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (stable/liberty)

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/298583

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to heat (stable/mitaka)

Reviewed: https://review.openstack.org/298154
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=a2a6fa77118aa718fdd19ecc8483db8d4cb67d7e
Submitter: Jenkins
Branch: stable/mitaka

commit a2a6fa77118aa718fdd19ecc8483db8d4cb67d7e
Author: Rabi Mishra <email address hidden>
Date: Sat Mar 19 19:33:42 2016 +0530

    Fix update issues with autoscaling group

    - Fix update stuck in IN_PROGRESS if issued within
    the cooldown period after CREATE_COMPLETE.

    - Fix update stuck in IN_PROGRESS if issued when scaling
    is happening.

    Change-Id: I31074783f525ea811056e8df7ff46545e58075b5
    Closes-Bug: #1559132
    (cherry picked from commit 6bd7352dc4838b8ef782f2345a4dfdf57ba3e356)

tags: added: in-stable-mitaka
Revision history for this message
Thierry Carrez (ttx) wrote : Fix included in openstack/heat 6.0.0.0rc2

This issue was fixed in the openstack/heat 6.0.0.0rc2 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to heat (stable/liberty)

Reviewed: https://review.openstack.org/298583
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=db9a61819bc864f4810fc817a1c7a142ffc99d02
Submitter: Jenkins
Branch: stable/liberty

commit db9a61819bc864f4810fc817a1c7a142ffc99d02
Author: Rabi Mishra <email address hidden>
Date: Sat Mar 19 19:33:42 2016 +0530

    Fix update issues with autoscaling group

    - Fix update stuck in IN_PROGRESS if issued within
    the cooldown period after CREATE_COMPLETE.

    - Fix update stuck in IN_PROGRESS if issued when scaling
    is happening.

    Change-Id: I31074783f525ea811056e8df7ff46545e58075b5
    Closes-Bug: #1559132
    (cherry picked from commit 6bd7352dc4838b8ef782f2345a4dfdf57ba3e356)

tags: added: in-stable-liberty
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (master)

Fix proposed to branch: master
Review: https://review.openstack.org/302565

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to heat (master)
Download full text (13.0 KiB)

Reviewed: https://review.openstack.org/302565
Committed: https://git.openstack.org/cgit/openstack/heat/commit/?id=eeed79b7a2da4a8f7fcbe0c8a7c7a069f304ae11
Submitter: Jenkins
Branch: master

commit bea576f802f993204aaaaea82812820f2f51c668
Author: Pavlo Shchelokovskyy <email address hidden>
Date: Wed Mar 30 16:09:29 2016 +0300

    Sync integration tests requirements

    The patch that auto-generates integration tests requirements [1] missed
    Mitaka release.

    This patch manually syncs the integration tests requirements hopefully
    for the last time.

    [1] Ie79338cc10cc101fbf15b51c7923e3a7b8e4fbb4

    Change-Id: I3f001bcb7900f6d822055bf3567773b3a594227d

commit 950505d956e99bc9eed7d6fb0223800251d88b95
Author: Rabi Mishra <email address hidden>
Date: Wed Mar 30 09:34:36 2016 +0530

    Revert "Check RBAC policy for nested stacks"

    This is causing issues with tripleo failures
    with ValueError, when trying do stack-preview as
    part of policy enforcement for stack DELETE.

    This reverts commit a8accbba98b5fc15562864030f49d20570bf8412.

    Change-Id: I366316f27f24c650bd435e67dd17abd8676cedf4
    Closes-Bug: #1561172
    (cherry picked from commit 354f1bcef038340d95f7dc4bc29268c9efed642e)

commit 184b09a40672ccfe1964bbf4fc72234d4cd012b6
Author: OpenStack Proposal Bot <email address hidden>
Date: Wed Mar 30 06:08:07 2016 +0000

    Imported Translations from Zanata

    For more information about this automatic import see:
    https://wiki.openstack.org/wiki/Translations/Infrastructure

    Change-Id: I656cedd8162b5a7d1682a33ab9b836ba25d22de9

commit 0c407b8596499466b93d5b511e19b0de130b6f03
Author: Zane Bitter <email address hidden>
Date: Mon Mar 28 17:19:47 2016 -0400

    Add translation rule to delete ssh auth key from Magnum baymodel

    Add a translation rule to ensure that users with existing templates will
    not have them break when they update due to the 'ssh_authorized_key'
    property having been removed from Magnum and hidden in Heat.

    Change-Id: Id73bace93bfc2379e80777da7dc0c0bb059d6cd3
    Related-Bug: #1560284
    (cherry picked from commit 0ab68f891348eeafa1efa219070ef39f776cb6c6)

commit db06ee4514b0cc8037f037788c34631c272cfc6b
Author: Sergey Kraynev <email address hidden>
Date: Thu Mar 24 04:30:39 2016 -0400

    Increase timeout of WaitCondition due to ConcurrentTransactions

    Currently we have retry logic with random period of delay.
    There is a chance, when we have several re-try attemts with maximum
    timeout, so it can be a root cause of floating fails, when we have not
    enough time for getting all signals.

    Also decrease number of signals from 30 to 25, due to small risk, when
    one of signals goes to ConcurrentTransaction error.

    Note, that all these changes are made in corresponding functional test
    and don't affect main code.

    Change-Id: I9ca85c4378d0f62e316f4f40ff73597ffaaabadf
    Related-Bug: #1497274
    (cherry picked from commit 8f370d6174c7574c6ee611b5089eac8a4c37dbf2)

commit 145be139cc8a34c57f84e1991031cfc2595477ba
Author: OpenStack Proposal Bot <openstack-infra...

Revision history for this message
Doug Hellmann (doug-hellmann) wrote : Fix included in openstack/heat 7.0.0.0b1

This issue was fixed in the openstack/heat 7.0.0.0b1 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/heat 5.0.2

This issue was fixed in the openstack/heat 5.0.2 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.