Cinder hangs while deleting 500 volumes (MOS 9.0 build 150)
Affects | Status | Importance | Assigned to | Milestone | ||
---|---|---|---|---|---|---|
Mirantis OpenStack | Status tracked in 10.0.x | |||||
10.0.x |
Confirmed
|
High
|
MOS Cinder | |||
9.x |
In Progress
|
High
|
Ivan Kolodyazhny |
Bug Description
Environment:
MOS 9.0 build 150
1 Controller node, 2 Compute nodes, 3 Ceph nodes
Steps to reproduce:
Write python script to generate and then delete volumes using this code
def wait_for_
while True:
volume = cinder.
if status == volume.status:
break
else:
time.sleep(3)
def generate_
for i in range(0, 500):
volume = cinder.
self.
self.
def cleanup_
for volume in self.created_
cinder.
Expected result:
Cinder deleted all volumes after some time
Actual result:
Cinder deleted several volumes and then stopped do that
All volumes are in "deleting" state forever
From logs:
2016-04-29 07:35:39.320 9801 INFO cinder.
2016-04-29 07:35:39.475 9801 INFO cinder.
2016-04-29 07:35:39.875 9801 INFO cinder.
2016-04-29 07:35:40.269 9801 INFO cinder.
2016-04-29 07:36:31.519 9801 WARNING cinder.
2016-04-29 07:36:38.544 9801 INFO cinder.
2016-04-29 07:36:38.587 9801 INFO cinder.
2016-04-29 07:36:46.351 9801 INFO cinder.
After that nothing happens for 30 minutes.
See attached cinder and ceph logs collected from Controller node
affects: | cinder → mos |
Changed in mos: | |
assignee: | nobody → MOS Cinder (mos-cinder) |
milestone: | none → 9.0 |
summary: |
- Cinder hungs while deleting 500 volumes (MOS 9.0 build 150) + Cinder hangs while deleting 500 volumes (MOS 9.0 build 150) |
Changed in mos: | |
assignee: | MOS Cinder (mos-cinder) → Ivan Kolodyazhny (e0ne) |
status: | New → In Progress |
Changed in mos: | |
importance: | Undecided → High |
tags: | added: area-cinder |
Conroller node restart fixes this problem. "nova volume-list" shows empty list after restart.