Mirantis OpenStack

Heat stack stops scaling after "can't compare datetime.datetime to NoneType"

  1. Series 6.0.x
  2. Bug #1557013
Bug #1557013 reported by Dmitry Sutyagin
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Mirantis OpenStack
Status tracked in 10.0.x
10.0.x
Invalid
High
Alexander Nagovitsyn
Mirantis OpenStack 10.0
6.0.x
In Progress
High
MOS Maintenance
Mirantis OpenStack 6.0-updates
6.1.x
Fix Released
High
Alexey Stupnikov
Mirantis OpenStack 6.1-mu-6
7.0.x
Fix Released
High
Alexey Stupnikov
Mirantis OpenStack 7.0-mu-4
8.0.x
Invalid
High
Alexey Stupnikov
Mirantis OpenStack 8.0-updates
9.x
Invalid
High
Alexander Nagovitsyn
Mirantis OpenStack 9.0

Bug Description

Confirmed on MOS 6.0 MU7, Ubuntu. Heat package version - 2014.2-fuel6.0~mira10.
Reported by Heat team to be related to https://bugs.launchpad.net/mos/+bug/1474332

Issue summary:
Heat stack stops scaling after several failed attempts (due to quotas / other non-heat errors).
All subsequent alarm triggers result in the following log messages:

<134>Mar 3 16:43:26 node-13 heat-engine 2016-03-03 16:43:26.723 10851 INFO heat.engine.resources.openstack.scaling_policy [req-65d7f565-c8f7-474d-8eac-8d6a058d2d45 None] scale_up_policy Alarm, adjusting Group my_asg with id march3rd-my_asg-qyq2gbppzjck by 1
<134>Mar 3 16:43:26 node-13 heat-engine 2016-03-03 16:43:26.724 10851 INFO heat.engine.resources.autoscaling [req-65d7f565-c8f7-474d-8eac-8d6a058d2d45 None] my_asg NOT performing scaling adjustment, cooldown 60

This behavior starts after the following traceback:

<131>Mar 3 16:40:26 node-13 heat-engine 2016-03-03 16:40:26.659 10851 ERROR heat.engine.resource [req-23cf5695-68b9-4ae7-a15f-39414a9c666f None] signal AutoScalingPolicy "scale_up_policy" [d7186b9aa0d94f22bb6719c3829dd852] Stack "march3rd" [2443de4f-ca1f-4bb3-9102-bfa3f863f444] : can't compare datetime.datetime to NoneType
2016-03-03 16:40:26.659 10851 TRACE heat.engine.resource Traceback (most recent call last):
2016-03-03 16:40:26.659 10851 TRACE heat.engine.resource File "/usr/lib/python2.7/dist-packages/heat/engine/resource.py", line 1032, in signal
2016-03-03 16:40:26.659 10851 TRACE heat.engine.resource signal_result = self.handle_signal(details)
2016-03-03 16:40:26.659 10851 TRACE heat.engine.resource File "/usr/lib/python2.7/dist-packages/heat/engine/resources/openstack/scaling_policy.py", line 153, in handle_signal
2016-03-03 16:40:26.659 10851 TRACE heat.engine.resource adjustment_type)
2016-03-03 16:40:26.659 10851 TRACE heat.engine.resource File "/usr/lib/python2.7/dist-packages/heat/engine/resources/autoscaling.py", line 622, in adjust
2016-03-03 16:40:26.659 10851 TRACE heat.engine.resource capacity = len(self.get_instances())
2016-03-03 16:40:26.659 10851 TRACE heat.engine.resource File "/usr/lib/python2.7/dist-packages/heat/engine/resources/autoscaling.py", line 208, in get_instances
2016-03-03 16:40:26.659 10851 TRACE heat.engine.resource return sorted(resources, key=lambda r: (r.created_time, r.name))
2016-03-03 16:40:26.659 10851 TRACE heat.engine.resource TypeError: can't compare datetime.datetime to NoneType
2016-03-03 16:40:26.659 10851 TRACE heat.engine.resource

Attached are extracts from mysql heat database.

Revision history for this message
Dmitry Sutyagin (dsutyagin) wrote :
Revision history for this message
Dmitry Sutyagin (dsutyagin) wrote :

Logs (for Mirantis staff) https://drive.google.com/a/mirantis.com/folderview?id=0B2VydA-aQqI9M20xaU90UHhhVk0&usp=sharing

Revision history for this message
Dmitry Sutyagin (dsutyagin) wrote :

Probably related to https://bugs.launchpad.net/heat/+bug/1480183

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to openstack/heat (openstack-ci/fuel-6.0-updates/2014.2)

Fix proposed to branch: openstack-ci/fuel-6.0-updates/2014.2
Change author: huangtianhua <email address hidden>
Review: https://review.fuel-infra.org/18087

Revision history for this message
Sergey Kraynev (skraynev) wrote :

Looks like issue, which was fixed in Liberty.
There is a similar bug: https://bugs.launchpad.net/heat/+bug/1480183
With fix: https://review.openstack.org/#/c/208386/

Revision history for this message
Dmitry Sutyagin (dsutyagin) wrote :

Tested with https://review.fuel-infra.org/18087 - could not reproduce, so I think the fix is good.

Revision history for this message
Sergey Kraynev (skraynev) wrote :

Looks like Mitaka and Liberty contain necessary fixes, so we may just check, that it works correct without porting and applying any fixes.

Revision history for this message
Dina Belova (dbelova) wrote :

Sergey, once you'll verify needed fixes are presented in L/M please move to Invalid for appropriate series.

tags: added: area-heat
Revision history for this message
Alexey Stupnikov (astupnikov) wrote :

Steps to reproduce:
1. Modify project's quota, limit number of volumes allowed.

2. Install and configure heat CLI client

3. Copy attached files to your current dir

4. Start the stack:
heat stack-create simple -f simple.yaml -e environment.yaml

5. Wait for 20 minutes until the messages from bug description will come up in heat-all.log

Revision history for this message
Alexander Nagovitsyn (gluk12189) wrote :

On latest 9.0 - could not reproduce

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to openstack/heat (openstack-ci/fuel-7.0/2015.1.0)

Fix proposed to branch: openstack-ci/fuel-7.0/2015.1.0
Change author: huangtianhua <email address hidden>
Review: https://review.fuel-infra.org/20416

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix proposed to openstack/heat (openstack-ci/fuel-6.1/2014.2)

Fix proposed to branch: openstack-ci/fuel-6.1/2014.2
Change author: huangtianhua <email address hidden>
Review: https://review.fuel-infra.org/20417

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix merged to openstack/heat (openstack-ci/fuel-6.1/2014.2)

Reviewed: https://review.fuel-infra.org/20417
Submitter: Vitaly Sedelnik <email address hidden>
Branch: openstack-ci/fuel-6.1/2014.2

Commit: 231adfda976713d13957e34ff710d39f00a259ef
Author: huangtianhua <email address hidden>
Date: Tue May 10 07:54:10 2016

Fix incorrect resource's information while describing

We can't get the resource'infos from db once the resource
is deleted. All information of a deleted resource come
from the initialization. So this change will modify some
infos when resource init:
1. set the action of resource to stack'action
2. set the created_time to stack's created_time
3. set the updated_time to stack's updated_time

Then the information will be reset for normal resources.

Change-Id: Iafdc99c572c5dab06bf1843842ed47554d6de128
Closes-Bug: #1557013

Revision history for this message
Alexey Stupnikov (astupnikov) wrote :

Setting this bug's status to Invalid for MOS8, since the fix is already in MOS8 code.

Revision history for this message
Fuel Devops McRobotson (fuel-devops-robot) wrote : Fix merged to openstack/heat (openstack-ci/fuel-7.0/2015.1.0)

Reviewed: https://review.fuel-infra.org/20416
Submitter: Vitaly Sedelnik <email address hidden>
Branch: openstack-ci/fuel-7.0/2015.1.0

Commit: 0ad90080053d64007436e9edc8a4fbc19cc7d0a7
Author: huangtianhua <email address hidden>
Date: Tue May 10 08:11:32 2016

Fix incorrect resource's information while describing

We can't get the resource'infos from db once the resource
is deleted. All information of a deleted resource come
from the initialization. So this change will modify some
infos when resource init:
1. set the action of resource to stack'action
2. set the created_time to stack's created_time
3. set the updated_time to stack's updated_time

Then the information will be reset for normal resources.

Change-Id: Iafdc99c572c5dab06bf1843842ed47554d6de128
Closes-Bug: #1557013

Alexander Gromov (agromov)
tags: added: on-verification
Revision history for this message
Alexander Gromov (agromov) wrote :

Verified on MOS 6.1 + mu6 updates.

Steps to reproduce are described above.

Notes:
1) limit of number of volumes was set to 3.
2) log can appear later then 20 minutes

Without updates the trace from description was found on one controller.

After updates the trace wasn't found. Error "OverLimit: VolumeLimitExceeded" continued to appear.

tags: removed: on-verification
Alexander Gromov (agromov)
tags: added: on-verification
Revision history for this message
Alexander Gromov (agromov) wrote :

Verified on MOS 7.0 + mu4 updates.

Steps to reproduce are described above.

tags: removed: on-verification
Revision history for this message
Sergey Kraynev (skraynev) wrote :

It's invalid for any release after Liberty, because fix is already in upstream code.

Revision history for this message
Sergey Kraynev (skraynev) wrote :

Fix for 6.0 was proposed, so please just review it https://review.fuel-infra.org/#/c/18087/

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.