OpenStack Compute (nova)

[Mitaka] Fails to resize the instance booted by creating a volume, recognizing it as an ephemeral disk when using ceph

Bug #1599067 reported by Dongwon Cho on 2016-07-05

This bug report is a duplicate of: Bug #1549194: resize a instance,then revert the resize,excute cold migrate,error using ceph. Edit Remove

This bug affects 4 people

Affects		Status	Importance	Assigned to	Milestone
	OpenStack Compute (nova)	New	Undecided	Unassigned

Bug Description

Description
===========
nova-compute fails to resize instance that was booted by creating a volume, recognizing it as an ephemeral disk.

Steps to reproduce
==================
1. Create an instance by selecting boot source as 'image' and creating a new volume
2. Resize the instance
3. Confirm the resizing

Expected result
===============
successful resize

Actual result
=============
It passes until the step of resize confirmation without any error but it fails after the resize confirmation is hit.

Environment
===========
1. Exact version of OpenStack you are running. See the following
Mitaka
dpkg -l | grep nova
ii nova-common 2:13.0.0-0ubuntu2 all OpenStack Compute - common files
ii nova-compute 2:13.0.0-0ubuntu2 all OpenStack Compute - compute node base
ii nova-compute-kvm 2:13.0.0-0ubuntu2 all OpenStack Compute - compute node (KVM)
ii nova-compute-libvirt 2:13.0.0-0ubuntu2 all OpenStack Compute - compute node libvirt support
ii python-nova 2:13.0.0-0ubuntu2 all OpenStack Compute Python libraries
ii python-novaclient 2:3.3.1-2 all client library for OpenStack Compute API - Python 2.7

2. Which hypervisor did you use?
Libvirt + KVM
dpkg -l | egrep 'libvirt|qemu|kvm'
ii ipxe-qemu 1.0.0+git-20150424.a25a16d-1ubuntu1 all PXE boot firmware - ROM images for qemu
ii libvirt-bin 1.3.1-1ubuntu10.1 amd64 programs for the libvirt library
ii libvirt0:amd64 1.3.1-1ubuntu10.1 amd64 library for interfacing with different virtualization systems
ii nova-compute-kvm 2:13.0.0-0ubuntu2 all OpenStack Compute - compute node (KVM)
ii nova-compute-libvirt 2:13.0.0-0ubuntu2 all OpenStack Compute - compute node libvirt support
ii python-libvirt 1.3.1-1ubuntu1 amd64 libvirt Python bindings
ii qemu-block-extra:amd64 1:2.5+dfsg-5ubuntu10.2 amd64 extra block backend modules for qemu-system and qemu-utils
ii qemu-slof 20151103+dfsg-1ubuntu1 all Slimline Open Firmware -- QEMU PowerPC version
ii qemu-system 1:2.5+dfsg-5ubuntu10.2 amd64 QEMU full system emulation binaries
ii qemu-system-arm 1:2.5+dfsg-5ubuntu10.2 amd64 QEMU full system emulation binaries (arm)
ii qemu-system-common 1:2.5+dfsg-5ubuntu10.2 amd64 QEMU full system emulation binaries (common files)
ii qemu-system-mips 1:2.5+dfsg-5ubuntu10.2 amd64 QEMU full system emulation binaries (mips)
ii qemu-system-misc 1:2.5+dfsg-5ubuntu10.2 amd64 QEMU full system emulation binaries (miscelaneous)
ii qemu-system-ppc 1:2.5+dfsg-5ubuntu10.2 amd64 QEMU full system emulation binaries (ppc)
ii qemu-system-sparc 1:2.5+dfsg-5ubuntu10.2 amd64 QEMU full system emulation binaries (sparc)
ii qemu-system-x86 1:2.5+dfsg-5ubuntu10.2 amd64 QEMU full system emulation binaries (x86)
ii qemu-utils 1:2.5+dfsg-5ubuntu10.2 amd64 QEMU utilities

2. Which storage type did you use?
Ceph
dpkg -l | egrep 'ceph'
ii ceph-common 10.2.2-1xenial amd64 common utilities to mount and interact with a ceph storage cluster
ii libcephfs1 10.2.2-1xenial amd64 Ceph distributed file system client library
ii python-cephfs 10.2.2-1xenial amd64 Python libraries for the Ceph libcephfs library

3. Which networking type did you use?
Neutron with OpenVSwitch
# On the api node
dpkg -l | grep neutron
ii neutron-common 2:8.1.0-0ubuntu0.16.04.2 all Neutron is a virtual network service for Openstack - common
ii neutron-plugin-ml2 2:8.1.0-0ubuntu0.16.04.2 all Neutron is a virtual network service for Openstack - ML2 plugin
ii neutron-server 2:8.1.0-0ubuntu0.16.04.2 all Neutron is a virtual network service for Openstack - server
ii python-neutron 2:8.1.0-0ubuntu0.16.04.2 all Neutron is a virtual network service for Openstack - Python library
ii python-neutron-fwaas 1:8.0.0-0ubuntu1 all Firewall-as-a-Service driver for OpenStack Neutron
ii python-neutron-lib 0.0.2-2 all Neutron shared routines and utilities - Python 2.7
ii python-neutronclient 1:4.1.1-2 all client API library for Neutron - Python 2.7

# On the network node
dpkg -l | grep neutron
ii neutron-common 2:8.1.0-0ubuntu0.16.04.2 all Neutron is a virtual network service for Openstack - common
ii neutron-dhcp-agent 2:8.1.0-0ubuntu0.16.04.2 all Neutron is a virtual network service for Openstack - DHCP agent
ii neutron-l3-agent 2:8.1.0-0ubuntu0.16.04.2 all Neutron is a virtual network service for Openstack - l3 agent
ii neutron-metadata-agent 2:8.1.0-0ubuntu0.16.04.2 all Neutron is a virtual network service for Openstack - metadata agent
ii neutron-openvswitch-agent 2:8.1.0-0ubuntu0.16.04.2 all Neutron is a virtual network service for Openstack - Open vSwitch plugin agent
ii python-neutron 2:8.1.0-0ubuntu0.16.04.2 all Neutron is a virtual network service for Openstack - Python library
ii python-neutron-fwaas 1:8.0.0-0ubuntu1 all Firewall-as-a-Service driver for OpenStack Neutron
ii python-neutron-lib 0.0.2-2 all Neutron shared routines and utilities - Python 2.7
ii python-neutronclient 1:4.1.1-2 all client API library for Neutron - Python 2.7

Logs & Configs
==============
2016-07-05 15:40:32.701 99291 DEBUG oslo_concurrency.lockutils [req-41fc2f7f-8c45-4c2d-ba7b-2144fcbfd757 8e06675285464030a9fd7badf4b0939cba85e018ab8a20ea8cb911222fef7771 04e7fe0cfb19418a9ec2eacfe1d334d5 - - -] Releasing semaphore "refresh_cache-9debce4f-ee92-4982-815f-769380728951" lock /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:228
2016-07-05 15:40:32.702 99291 DEBUG oslo_concurrency.processutils [req-41fc2f7f-8c45-4c2d-ba7b-2144fcbfd757 8e06675285464030a9fd7badf4b0939cba85e018ab8a20ea8cb911222fef7771 04e7fe0cfb19418a9ec2eacfe1d334d5 - - -] Running cmd (subprocess): rm -rf /var/lib/nova/instances/9debce4f-ee92-4982-815f-769380728951_resize execute /usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:326
2016-07-05 15:40:32.728 99291 DEBUG oslo_concurrency.processutils [req-41fc2f7f-8c45-4c2d-ba7b-2144fcbfd757 8e06675285464030a9fd7badf4b0939cba85e018ab8a20ea8cb911222fef7771 04e7fe0cfb19418a9ec2eacfe1d334d5 - - -] CMD "rm -rf /var/lib/nova/instances/9debce4f-ee92-4982-815f-769380728951_resize" returned: 0 in 0.026s execute /usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:356
2016-07-05 15:40:32.741 99291 DEBUG nova.virt.libvirt.storage.rbd_utils [req-41fc2f7f-8c45-4c2d-ba7b-2144fcbfd757 8e06675285464030a9fd7badf4b0939cba85e018ab8a20ea8cb911222fef7771 04e7fe0cfb19418a9ec2eacfe1d334d5 - - -] rbd image 9debce4f-ee92-4982-815f-769380728951_disk does not exist __init__ /usr/lib/python2.7/dist-packages/nova/virt/libvirt/storage/rbd_utils.py:64
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [req-41fc2f7f-8c45-4c2d-ba7b-2144fcbfd757 8e06675285464030a9fd7badf4b0939cba85e018ab8a20ea8cb911222fef7771 04e7fe0cfb19418a9ec2eacfe1d334d5 - - -] [instance: 9debce4f-ee92-4982-815f-769380728951] Setting instance vm_state to ERROR
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] Traceback (most recent call last):
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6588, in _error_out_instance_on_exception
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] yield
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3531, in _confirm_resize
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] network_info)
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 7484, in confirm_migration
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] self._cleanup_resize(instance, network_info)
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1258, in _cleanup_resize
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] ignore_errors=True)
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 941, in remove_snap
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] return self.driver.remove_snap(self.rbd_name, name, ignore_errors)
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/storage/rbd_utils.py", line 396, in remove_snap
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] with RBDVolumeProxy(self, str(volume), pool=pool) as vol:
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/storage/rbd_utils.py", line 65, in __init__
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] driver._disconnect_from_rados(client, ioctx)
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] self.force_reraise()
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] six.reraise(self.type_, self.value, self.tb)
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/storage/rbd_utils.py", line 61, in __init__
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] read_only=read_only)
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] File "rbd.pyx", line 1042, in rbd.Image.__init__ (/tmp/buildd/ceph-10.2.2/src/build/rbd.c:9862)
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] ImageNotFound: error opening image 9debce4f-ee92-4982-815f-769380728951_disk at snapshot None
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951]
2016-07-05 15:40:32.811 99291 DEBUG oslo_concurrency.lockutils [req-41fc2f7f-8c45-4c2d-ba7b-2144fcbfd757 8e06675285464030a9fd7badf4b0939cba85e018ab8a20ea8cb911222fef7771 04e7fe0cfb19418a9ec2eacfe1d334d5 - - -] Lock "9debce4f-ee92-4982-815f-769380728951" released by "nova.compute.manager.do_confirm_resize" :: held 0.507s inner /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:285
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher [req-41fc2f7f-8c45-4c2d-ba7b-2144fcbfd757 8e06675285464030a9fd7badf4b0939cba85e018ab8a20ea8cb911222fef7771 04e7fe0cfb19418a9ec2eacfe1d334d5 - - -] Exception during message handling: error opening image 9debce4f-ee92-4982-815f-769380728951_disk at snapshot None
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher incoming.message))
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 110, in wrapped
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher payload)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher self.force_reraise()
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 89, in wrapped
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher return f(self, context, *args, **kw)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 409, in decorated_function
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 387, in decorated_function
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info())
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher self.force_reraise()
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 375, in decorated_function
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3507, in confirm_resize
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher do_confirm_resize(context, instance, migration.id)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 274, in inner
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher return f(*args, **kwargs)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3505, in do_confirm_resize
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher migration=migration)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3531, in _confirm_resize
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher network_info)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 7484, in confirm_migration
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher self._cleanup_resize(instance, network_info)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1258, in _cleanup_resize
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher ignore_errors=True)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 941, in remove_snap
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher return self.driver.remove_snap(self.rbd_name, name, ignore_errors)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/storage/rbd_utils.py", line 396, in remove_snap
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher with RBDVolumeProxy(self, str(volume), pool=pool) as vol:
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/storage/rbd_utils.py", line 65, in __init__
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher driver._disconnect_from_rados(client, ioctx)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher self.force_reraise()
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/storage/rbd_utils.py", line 61, in __init__
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher read_only=read_only)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "rbd.pyx", line 1042, in rbd.Image.__init__ (/tmp/buildd/ceph-10.2.2/src/build/rbd.c:9862)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher ImageNotFound: error opening image 9debce4f-ee92-4982-815f-769380728951_disk at snapshot None
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher

openstack server show 9debce4f-ee92-4982-815f-769380728951
+--------------------------------------+------------------------------------------------------------------+
| Field | Value |
+--------------------------------------+------------------------------------------------------------------+
| OS-DCF:diskConfig | AUTO |
| OS-EXT-AZ:availability_zone | lab |
| OS-EXT-SRV-ATTR:host | h12 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | h12.example.com |
| OS-EXT-SRV-ATTR:instance_name | instance-00000150 |
| OS-EXT-STS:power_state | 4 |
| OS-EXT-STS:task_state | None |
| OS-EXT-STS:vm_state | stopped |
| OS-SRV-USG:launched_at | 2016-07-05T06:40:11.000000 |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | 1st-network=172.16.0.12 |
| config_drive | |
| created | 2016-07-05T03:32:37Z |
| flavor | m1.large (4) |
| hostId | 90f85ee8c380c68c55b169ff1dc9917729638d0e744020641ace2700 |
| id | 9debce4f-ee92-4982-815f-769380728951 |
| image | |
| key_name | desktop |
| name | e1 |
| os-extended-volumes:volumes_attached | [{u'id': u'90ea9f31-57b4-4162-8986-47065b23f407'}] |
| project_id | 04e7fe0cfb19418a9ec2eacfe1d334d5 |
| properties | |
| security_groups | [{u'name': u'default'}] |
| status | SHUTOFF |
| updated | 2016-07-05T07:04:11Z |
| user_id | 8e06675285464030a9fd7badf4b0939cba85e018ab8a20ea8cb911222fef7771 |
+--------------------------------------+------------------------------------------------------------------+

openstack volume show 90ea9f31-57b4-4162-8986-47065b23f407
+--------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+--------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| attachments | [{u'server_id': u'9debce4f-ee92-4982-815f-769380728951', u'attachment_id': u'd119c454-65f4-4c7b-8135-3238a3e7a424', u'attached_at': u'2016-07-05T03:33:37.127003', u'host_name': None, u'volume_id': u'90ea9f31-57b4-4162-8986-47065b23f407', u'device': u'/dev/vda', u'id': u'90ea9f31-57b4-4162-8986-47065b23f407'}] |
| availability_zone | nova |
| bootable | true |
| consistencygroup_id | None |
| created_at | 2016-07-05T03:32:38.353914 |
| description | |
| encrypted | False |
| id | 90ea9f31-57b4-4162-8986-47065b23f407 |
| migration_status | None |
| multiattach | False |
| name | |
| os-vol-host-attr:host | mitaka-cinder@ceph#ceph |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | 04e7fe0cfb19418a9ec2eacfe1d334d5 |
| properties | attached_mode='rw', readonly='False' |
| replication_status | disabled |
| size | 40 |
| snapshot_id | None |
| source_volid | None |
| status | in-use |
| type | ceph |
| updated_at | 2016-07-05T03:33:37.161179 |
| user_id | 8e06675285464030a9fd7badf4b0939cba85e018ab8a20ea8cb911222fef7771 |
| volume_image_metadata | {u'description': u'', u'os_require_quiesce': u'yes', u'checksum': u'cb3d84ccc990821e537fbe5227b3dbc3', u'hw_qemu_guest_agent': u'yes', u'disk_format': u'raw', u'image_name': u'os2-2016062303', u'hw_scsi_model': u'virtio-scsi', u'image_id': u'083aadf6-6a4c-45aa-aab6-66b6d2b2c2ba', u'min_ram': u'0', u'container_format': u'bare', u'min_disk': u'3', u'hw_disk_bus': u'scsi', u'size': u'2705326080'} |
+--------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

# On the Ceph cluster
⟫ sudo rbd ls openstack-volume | grep 9debce4f-ee92-4982-815f-769380728951_disk
1 ⟫ sudo rbd ls openstack-volume | grep 90ea9f31-57b4-4162-8986-47065b23f407
volume-90ea9f31-57b4-4162-8986-47065b23f407
⟫ sudo rbd info -p openstack-volume volume-90ea9f31-57b4-4162-8986-47065b23f407
rbd image 'volume-90ea9f31-57b4-4162-8986-47065b23f407':
        size 40960 MB in 10240 objects
        order 22 (4096 kB objects)
        block_name_prefix: rbd_data.b412b0238e1f29
        format: 2
        features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
        flags:
(The volume size has not been changed.)

As you can see above, nova-compute tries to do something on the 9debce4f-ee92-4982-815f-769380728951_disk which doesn't actually exist, recognizing it as an ephemeral disk.

See original description

Tags:

Dongwon Cho (dongwoncho) on 2016-07-05

description:	updated
description:	updated

Xuanzhou Perry Dong (oss-xzdong) on 2016-07-08

Changed in nova:
assignee:	nobody → Xuanzhou Perry Dong (oss-xzdong)

Revision history for this message

Xuanzhou Perry Dong (oss-xzdong) wrote on 2016-07-08:

Tested in devstack with lvm backend. Can't reproduce the error.

Changed in nova:
assignee:	Xuanzhou Perry Dong (oss-xzdong) → nobody

Revision history for this message

John Garbutt (johngarbutt) wrote on 2016-07-14:

Adding the ceph and libvirt tags, as it looks like the clean up of the old instance on resize confirm is failing.

We don't have the correct logs here, but it feels like if you setup ceph, it should be an easy ish repo.

tags:	added: ceph
summary:	[Mitaka] Fails to resize the instance booted by creating a volume, - recognizing it as an ephemeral disk + recognizing it as an ephemeral disk when using ceph
tags:	added: libvirt

Revision history for this message

Antonio Messina (arcimboldo) wrote on 2016-09-29:

It looks to me this is not the same as bug #1549194

Revision history for this message

Antonio Messina (arcimboldo) wrote on 2016-09-29:

I think the issue is in

https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L1059

where it's trying to remove a snapshot that was not created because of the 'if not booted_from_volume' in:

https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L3063

I don't understand why this snapshot is not created when booting from volume, so I don't know if the correct fix would be to create the snapshot also when booting from volume or to add a 'if not booted_from_volume' conditional in `_cleanup_resize()`

Report a bug

This report contains Public information

Everyone can see this information.

Duplicate of bug #1549194 Remove

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.