OpenStack Heat

delete huge number of resources cluster might make resource service hang

Bug #1715398 reported by Rico Lin
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Heat
In Progress
High
Rico Lin
OpenStack Heat rocky-2

Bug Description

We got some report from Sahara and Magnum team when deleting a huge number of Cinder Volumes within resource group, might encounter with Cinder API leak and makes action stuck in in progress.
Actually to request for huge number action always become a burden for any services.
We should make it possible to create a delay when resource cluster deletion

Tags: magnum sahara
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to heat (master)

Fix proposed to branch: master
Review: https://review.openstack.org/501276

Changed in heat:
status: Triaged → In Progress
Revision history for this message
Spyros Trigazis (strigazi) wrote : Re: delete huge resources cluster might make resource service hang

The problem in our deployment was in cinder. Even without heat we could make cinder fail by deleting 10 volumes at once. We have cinder since a while and the db connection wasn't configured to use pymyql which is the recommended one since a while.

Switching cinder to use pymysql solved the problem. Now we can delete without any problems 100s of volumes with or without heat.

Rico Lin (rico-lin)
summary: - delete huge resources cluster might make resource service hang
+ delete huge number of resources cluster might make resource service
+ hang
Rico Lin (rico-lin)
Changed in heat:
milestone: queens-1 → queens-2
Rico Lin (rico-lin)
Changed in heat:
milestone: queens-2 → queens-3
Zane Bitter (zaneb)
Changed in heat:
importance: Medium → High
Rico Lin (rico-lin)
Changed in heat:
milestone: queens-3 → queens-rc1
Revision history for this message
Zane Bitter (zaneb) wrote :

The proposed patch requires a resource schema change and we are past feature-freeze, so bumping this to Rocky.

Changed in heat:
milestone: queens-rc1 → rocky-1
Rico Lin (rico-lin)
Changed in heat:
milestone: rocky-1 → rocky-2
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.