Mirantis OpenStack

nova-compute can't detach volume from the instance

Bug #1550180 reported by Olga Klochkova
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Mirantis OpenStack
Status tracked in 10.0.x
10.0.x
Invalid
High
Timur Nurlygayanov
Mirantis OpenStack 10.0
8.0.x
Invalid
High
Timur Nurlygayanov
Mirantis OpenStack 8.0-updates
9.x
Confirmed
Medium
Timofey Durakov
Mirantis OpenStack 9.x-updates

Bug Description

Environment has ceph as a storage backend
Steps to reproduce:
1. create image:

glance --os-image-api-version 1 image-update <image-id> --location http://releases.ubuntu.com/14.04/ubuntu-14.04.3-server-amd64.iso --disk-format qcow2 --container-format bare
This is important step, as the issue is not reproducible with TestVM image

2. spawn a VM from that image
 nova boot TestVm --flavor m1.tiny --image <image-id> --nic net-id=<net-id>
3. create volume
cinder --os-volume-api-version 2 create --name TestVOL 1
4. attach volume to the VM
nova volume-attach <VM-id> <Volume-id>
5. detach volume from the VM
nova volume-detach <VM-id> <Volume-id>

Volume stays in 'detaching’ state for some time (~30 sec) and then gets back to 'in-use' state.

relevant nova-compute logs:
2016-02-26 07:37:11.002 21387 ERROR oslo.service.loopingcall [-] Cannot retry nova.virt.libvirt.guest._do_wait_and_retry_detach upon suggested exception since retry count (7) reached max retry count (7).
2016-02-26 07:37:11.002 21387 ERROR oslo.service.loopingcall [-] Dynamic interval looping call 'oslo_service.loopingcall._func' failed
2016-02-26 07:37:11.002 21387 ERROR oslo.service.loopingcall Traceback (most recent call last):
2016-02-26 07:37:11.002 21387 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/dist-packages/oslo_service/loopingcall.py", line 113, in _run_loop
2016-02-26 07:37:11.002 21387 ERROR oslo.service.loopingcall result = func(*self.args, **self.kw)
2016-02-26 07:37:11.002 21387 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/dist-packages/oslo_service/loopingcall.py", line 269, in _func
2016-02-26 07:37:11.002 21387 ERROR oslo.service.loopingcall return self._sleep_time
2016-02-26 07:37:11.002 21387 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 195, in __exit__
2016-02-26 07:37:11.002 21387 ERROR oslo.service.loopingcall six.reraise(self.type_, self.value, self.tb)
2016-02-26 07:37:11.002 21387 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/dist-packages/oslo_service/loopingcall.py", line 247, in _func
2016-02-26 07:37:11.002 21387 ERROR oslo.service.loopingcall result = f(*args, **kwargs)
2016-02-26 07:37:11.002 21387 ERROR oslo.service.loopingcall File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/guest.py", line 326, in _do_wait_and_retry_detach
2016-02-26 07:37:11.002 21387 ERROR oslo.service.loopingcall reason=reason)
2016-02-26 07:37:11.002 21387 ERROR oslo.service.loopingcall DeviceDetachFailed: Device detach failed for vdb: Unable to detach from guest transient domain.)
2016-02-26 07:37:11.002 21387 ERROR oslo.service.loopingcall
2016-02-26 07:37:11.002 21387 ERROR nova.compute.manager [req-7944b96d-8d80-442e-a8c6-bf2148a34309 5c52fdfdec6f46f2960fdc2108b78c63 29a3255438bc49e7895968ff67c14682 - - -] [instance: 117d58ad-642c-475e-a4e0-8a91a9c0e864] Failed to detach volume 5b07c574-656b-40a4-9c7b-6dd4701235ca from /dev/vdb

relevant libvirt logs:

2016-02-26 07:00:50.876+0000: 3927: error : qemuDomainDetachDeviceConfig:7045 : invalid argument: no target device vdb

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "569"
  build_id: "569"
  fuel-nailgun_sha: "558ca91a854cf29e395940c232911ffb851899c1"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "658be72c4b42d3e1436b86ac4567ab914bfb451b"
  fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "33634ec27be77ecfb0b56b7e07497ad86d1fdcd3"
  fuel-ostf_sha: "3bc76a63a9e7d195ff34eadc29552f4235fa6c52"
  fuel-mirror_sha: "fb45b80d7bee5899d931f926e5c9512e2b442749"
  fuelmenu_sha: "78ffc73065a9674b707c081d128cb7eea611474f"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "a43cf96cd9532f10794dce736350bf5bed350e9d"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "d605bcbabf315382d56d0ce8143458be67c53434"

See original description
Olga Klochkova (oklochkova)
Changed in mos:
milestone: none → 8.0-updates
Revision history for this message
Eugene Nikanorov (enikanorov) wrote :

This is important issue that is nearly critical. Very basic functionality doesn't work with commonly used cloud image.

tags: added: nova
Changed in mos:
status: New → Confirmed
importance: Undecided → High
assignee: nobody → Eugene Nikanorov (enikanorov)
assignee: Eugene Nikanorov (enikanorov) → MOS Nova (mos-nova)
Olga Klochkova (oklochkova)
description: updated
Roman Podoliaka (rpodolyaka)
tags: added: area-nova
removed: nova
Roman Podoliaka (rpodolyaka)
summary: - Nova-compute can't detach volume from the instance
+ nova-compute can't detach volume from the instance
Revision history for this message
Roman Podoliaka (rpodolyaka) wrote :

I gave it a try on the 9.0 environment with Ceph used for both ephemerals and volumes and *did not* manage to reproduce the issue:

http://paste.openstack.org/show/494666/

^ the volume is attached / detached properly

Packages versions: http://paste.openstack.org/show/494667/

Image: https://cloud-images.ubuntu.com/trusty/current/

Revision history for this message
Roman Podoliaka (rpodolyaka) wrote :

Moreover, the same goes (to my surprise) for a volume with a formatted and mounted FS:

http://paste.openstack.org/show/494669/

Revision history for this message
Roman Podoliaka (rpodolyaka) wrote :

Olga, could you please confirm you are still seeing this issue?

Revision history for this message
Timur Nurlygayanov (tnurlygayanov) wrote :

I'm going to check the issue.

Revision history for this message
Timur Nurlygayanov (tnurlygayanov) wrote :

I can't reproduce the issue:

root@node-1:~# nova volume-attach cf1fce9b-933f-455f-94af-42a369d1ba6c 3a5e7a11-1a7b-4ed6-9ad8-08e3099b4b0d
+----------+--------------------------------------+
| Property | Value |
 +----------+--------------------------------------+
| device | /dev/vdb |
| id | 3a5e7a11-1a7b-4ed6-9ad8-08e3099b4b0d |
| serverId | cf1fce9b-933f-455f-94af-42a369d1ba6c |
| volumeId | 3a5e7a11-1a7b-4ed6-9ad8-08e3099b4b0d |
 +----------+--------------------------------------+
root@node-1:~# nova volume-list
WARNING: Command volume-list is deprecated and will be removed after Nova 13.0.0 is released. Use python-cinderclient or openstackclient instead.
+--------------------------------------+-----------+--------------+------+-------------+-------------+
| ID | Status | Display Name | Size | Volume Type | Attached to |
 +--------------------------------------+-----------+--------------+------+-------------+-------------+
| 3a5e7a11-1a7b-4ed6-9ad8-08e3099b4b0d | attaching | | 1 | - | |
 +--------------------------------------+-----------+--------------+------+-------------+-------------+
root@node-1:~#
root@node-1:~#
root@node-1:~# nova volume-list
WARNING: Command volume-list is deprecated and will be removed after Nova 13.0.0 is released. Use python-cinderclient or openstackclient instead.
+--------------------------------------+--------+--------------+------+-------------+--------------------------------------+
| ID | Status | Display Name | Size | Volume Type | Attached to |
 +--------------------------------------+--------+--------------+------+-------------+--------------------------------------+
| 3a5e7a11-1a7b-4ed6-9ad8-08e3099b4b0d | in-use | | 1 | - | cf1fce9b-933f-455f-94af-42a369d1ba6c |
 +--------------------------------------+--------+--------------+------+-------------+--------------------------------------+
root@node-1:~#
root@node-1:~#
root@node-1:~# nova volume-detach cf1fce9b-933f-455f-94af-42a369d1ba6c 3a5e7a11-1a7b-4ed6-9ad8-08e3099b4b0d
root@node-1:~# nova volume-list
WARNING: Command volume-list is deprecated and will be removed after Nova 13.0.0 is released. Use python-cinderclient or openstackclient instead.
+--------------------------------------+-----------+--------------+------+-------------+-------------+
| ID | Status | Display Name | Size | Volume Type | Attached to |
 +--------------------------------------+-----------+--------------+------+-------------+-------------+
| 3a5e7a11-1a7b-4ed6-9ad8-08e3099b4b0d | available | | 1 | - | |
 +--------------------------------------+-----------+--------------+------+-------------+-------------+

Revision history for this message
Mikhail Chernik (mchernik) wrote :

Reproduced on scale lab for MOS 9.2 snapshot 688.

Rally log (scenario inside) and report: http://mos-scale.vm.mirantis.net/test_results/build_9.0.0-495/jenkins-10_env_run_rally_custom-297-ubuntu-MSK-2017-01-16-14:54:35-2017-01-16-15:34:35/

Diagnostic snapshot: http://mos-scale-share.mirantis.com/fuel-snapshot-env-10-custom-296-pymysql-2017-01-16_16-43-05/

Revision history for this message
chandra shekar (sekharvajjula) wrote :

We have seen this problem as well in Queens release. Our VMs write huge data to cinder Volumes (/dev/vdb) all the time.
At this point of time if we run "stack delete", it results exactly the same failure.

This problem subsides if we first shutdown the VMs before stack delete. Should this be somehow incorporated in the heat code itself?

(Or)
Should heat simply delete the libvirt VM domain without trying to detach the volume?

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.