| 2016-07-05 08:32:13 |
Dongwon Cho |
bug |
|
|
added bug |
| 2016-07-05 08:44:06 |
Dongwon Cho |
description |
Description
===========
nova-compute fails to resize instance that was booted by creating a volume, recognizing it as an ephemeral disk.
Steps to reproduce
==================
1. Create an instance by selecting boot source as 'image' and creating a new volume
2. Resize the instance
3. Confirm the resizing
Expected result
===============
successful resize
Actual result
=============
It passes until resize confirmation without any error but it fails after the resize confirmation is hit.
Environment
===========
1. Exact version of OpenStack you are running. See the following
dpkg -l | grep nova
ii nova-common 2:13.0.0-0ubuntu2 all OpenStack Compute - common files
ii nova-compute 2:13.0.0-0ubuntu2 all OpenStack Compute - compute node base
ii nova-compute-kvm 2:13.0.0-0ubuntu2 all OpenStack Compute - compute node (KVM)
ii nova-compute-libvirt 2:13.0.0-0ubuntu2 all OpenStack Compute - compute node libvirt support
ii python-nova 2:13.0.0-0ubuntu2 all OpenStack Compute Python libraries
ii python-novaclient 2:3.3.1-2 all client library for OpenStack Compute API - Python 2.7
2. Which hypervisor did you use?
Libvirt + KVM
dpkg -l | egrep 'libvirt|qemu|kvm'
ii ipxe-qemu 1.0.0+git-20150424.a25a16d-1ubuntu1 all PXE boot firmware - ROM images for qemu
ii libvirt-bin 1.3.1-1ubuntu10.1 amd64 programs for the libvirt library
ii libvirt0:amd64 1.3.1-1ubuntu10.1 amd64 library for interfacing with different virtualization systems
ii nova-compute-kvm 2:13.0.0-0ubuntu2 all OpenStack Compute - compute node (KVM)
ii nova-compute-libvirt 2:13.0.0-0ubuntu2 all OpenStack Compute - compute node libvirt support
ii python-libvirt 1.3.1-1ubuntu1 amd64 libvirt Python bindings
ii qemu-block-extra:amd64 1:2.5+dfsg-5ubuntu10.2 amd64 extra block backend modules for qemu-system and qemu-utils
ii qemu-slof 20151103+dfsg-1ubuntu1 all Slimline Open Firmware -- QEMU PowerPC version
ii qemu-system 1:2.5+dfsg-5ubuntu10.2 amd64 QEMU full system emulation binaries
ii qemu-system-arm 1:2.5+dfsg-5ubuntu10.2 amd64 QEMU full system emulation binaries (arm)
ii qemu-system-common 1:2.5+dfsg-5ubuntu10.2 amd64 QEMU full system emulation binaries (common files)
ii qemu-system-mips 1:2.5+dfsg-5ubuntu10.2 amd64 QEMU full system emulation binaries (mips)
ii qemu-system-misc 1:2.5+dfsg-5ubuntu10.2 amd64 QEMU full system emulation binaries (miscelaneous)
ii qemu-system-ppc 1:2.5+dfsg-5ubuntu10.2 amd64 QEMU full system emulation binaries (ppc)
ii qemu-system-sparc 1:2.5+dfsg-5ubuntu10.2 amd64 QEMU full system emulation binaries (sparc)
ii qemu-system-x86 1:2.5+dfsg-5ubuntu10.2 amd64 QEMU full system emulation binaries (x86)
ii qemu-utils 1:2.5+dfsg-5ubuntu10.2 amd64 QEMU utilities
2. Which storage type did you use?
Ceph
dpkg -l | egrep 'ceph'
ii ceph-common 10.2.2-1xenial amd64 common utilities to mount and interact with a ceph storage cluster
ii libcephfs1 10.2.2-1xenial amd64 Ceph distributed file system client library
ii python-cephfs 10.2.2-1xenial amd64 Python libraries for the Ceph libcephfs library
3. Which networking type did you use?
Neutron with OpenVSwitch
# On the api node
dpkg -l | grep neutron
ii neutron-common 2:8.1.0-0ubuntu0.16.04.2 all Neutron is a virtual network service for Openstack - common
ii neutron-plugin-ml2 2:8.1.0-0ubuntu0.16.04.2 all Neutron is a virtual network service for Openstack - ML2 plugin
ii neutron-server 2:8.1.0-0ubuntu0.16.04.2 all Neutron is a virtual network service for Openstack - server
ii python-neutron 2:8.1.0-0ubuntu0.16.04.2 all Neutron is a virtual network service for Openstack - Python library
ii python-neutron-fwaas 1:8.0.0-0ubuntu1 all Firewall-as-a-Service driver for OpenStack Neutron
ii python-neutron-lib 0.0.2-2 all Neutron shared routines and utilities - Python 2.7
ii python-neutronclient 1:4.1.1-2 all client API library for Neutron - Python 2.7
# On the network node
dpkg -l | grep neutron
ii neutron-common 2:8.1.0-0ubuntu0.16.04.2 all Neutron is a virtual network service for Openstack - common
ii neutron-dhcp-agent 2:8.1.0-0ubuntu0.16.04.2 all Neutron is a virtual network service for Openstack - DHCP agent
ii neutron-l3-agent 2:8.1.0-0ubuntu0.16.04.2 all Neutron is a virtual network service for Openstack - l3 agent
ii neutron-metadata-agent 2:8.1.0-0ubuntu0.16.04.2 all Neutron is a virtual network service for Openstack - metadata agent
ii neutron-openvswitch-agent 2:8.1.0-0ubuntu0.16.04.2 all Neutron is a virtual network service for Openstack - Open vSwitch plugin agent
ii python-neutron 2:8.1.0-0ubuntu0.16.04.2 all Neutron is a virtual network service for Openstack - Python library
ii python-neutron-fwaas 1:8.0.0-0ubuntu1 all Firewall-as-a-Service driver for OpenStack Neutron
ii python-neutron-lib 0.0.2-2 all Neutron shared routines and utilities - Python 2.7
ii python-neutronclient 1:4.1.1-2 all client API library for Neutron - Python 2.7
Logs & Configs
==============
2016-07-05 15:40:32.701 99291 DEBUG oslo_concurrency.lockutils [req-41fc2f7f-8c45-4c2d-ba7b-2144fcbfd757 8e06675285464030a9fd7badf4b0939cba85e018ab8a20ea8cb911222fef7771 04e7fe0cfb19418a9ec2eacfe1d334d5 - - -] Releasing semaphore "refresh_cache-9debce4f-ee92-4982-815f-769380728951" lock /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:228
2016-07-05 15:40:32.702 99291 DEBUG oslo_concurrency.processutils [req-41fc2f7f-8c45-4c2d-ba7b-2144fcbfd757 8e06675285464030a9fd7badf4b0939cba85e018ab8a20ea8cb911222fef7771 04e7fe0cfb19418a9ec2eacfe1d334d5 - - -] Running cmd (subprocess): rm -rf /var/lib/nova/instances/9debce4f-ee92-4982-815f-769380728951_resize execute /usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:326
2016-07-05 15:40:32.728 99291 DEBUG oslo_concurrency.processutils [req-41fc2f7f-8c45-4c2d-ba7b-2144fcbfd757 8e06675285464030a9fd7badf4b0939cba85e018ab8a20ea8cb911222fef7771 04e7fe0cfb19418a9ec2eacfe1d334d5 - - -] CMD "rm -rf /var/lib/nova/instances/9debce4f-ee92-4982-815f-769380728951_resize" returned: 0 in 0.026s execute /usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:356
2016-07-05 15:40:32.741 99291 DEBUG nova.virt.libvirt.storage.rbd_utils [req-41fc2f7f-8c45-4c2d-ba7b-2144fcbfd757 8e06675285464030a9fd7badf4b0939cba85e018ab8a20ea8cb911222fef7771 04e7fe0cfb19418a9ec2eacfe1d334d5 - - -] rbd image 9debce4f-ee92-4982-815f-769380728951_disk does not exist __init__ /usr/lib/python2.7/dist-packages/nova/virt/libvirt/storage/rbd_utils.py:64
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [req-41fc2f7f-8c45-4c2d-ba7b-2144fcbfd757 8e06675285464030a9fd7badf4b0939cba85e018ab8a20ea8cb911222fef7771 04e7fe0cfb19418a9ec2eacfe1d334d5 - - -] [instance: 9debce4f-ee92-4982-815f-769380728951] Setting instance vm_state to ERROR
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] Traceback (most recent call last):
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6588, in _error_out_instance_on_exception
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] yield
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3531, in _confirm_resize
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] network_info)
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 7484, in confirm_migration
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] self._cleanup_resize(instance, network_info)
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1258, in _cleanup_resize
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] ignore_errors=True)
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 941, in remove_snap
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] return self.driver.remove_snap(self.rbd_name, name, ignore_errors)
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/storage/rbd_utils.py", line 396, in remove_snap
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] with RBDVolumeProxy(self, str(volume), pool=pool) as vol:
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/storage/rbd_utils.py", line 65, in __init__
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] driver._disconnect_from_rados(client, ioctx)
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] self.force_reraise()
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] six.reraise(self.type_, self.value, self.tb)
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/storage/rbd_utils.py", line 61, in __init__
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] read_only=read_only)
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] File "rbd.pyx", line 1042, in rbd.Image.__init__ (/tmp/buildd/ceph-10.2.2/src/build/rbd.c:9862)
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] ImageNotFound: error opening image 9debce4f-ee92-4982-815f-769380728951_disk at snapshot None
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951]
2016-07-05 15:40:32.811 99291 DEBUG oslo_concurrency.lockutils [req-41fc2f7f-8c45-4c2d-ba7b-2144fcbfd757 8e06675285464030a9fd7badf4b0939cba85e018ab8a20ea8cb911222fef7771 04e7fe0cfb19418a9ec2eacfe1d334d5 - - -] Lock "9debce4f-ee92-4982-815f-769380728951" released by "nova.compute.manager.do_confirm_resize" :: held 0.507s inner /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:285
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher [req-41fc2f7f-8c45-4c2d-ba7b-2144fcbfd757 8e06675285464030a9fd7badf4b0939cba85e018ab8a20ea8cb911222fef7771 04e7fe0cfb19418a9ec2eacfe1d334d5 - - -] Exception during message handling: error opening image 9debce4f-ee92-4982-815f-769380728951_disk at snapshot None
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher incoming.message))
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 110, in wrapped
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher payload)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher self.force_reraise()
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 89, in wrapped
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher return f(self, context, *args, **kw)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 409, in decorated_function
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 387, in decorated_function
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info())
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher self.force_reraise()
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 375, in decorated_function
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3507, in confirm_resize
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher do_confirm_resize(context, instance, migration.id)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 274, in inner
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher return f(*args, **kwargs)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3505, in do_confirm_resize
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher migration=migration)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3531, in _confirm_resize
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher network_info)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 7484, in confirm_migration
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher self._cleanup_resize(instance, network_info)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1258, in _cleanup_resize
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher ignore_errors=True)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 941, in remove_snap
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher return self.driver.remove_snap(self.rbd_name, name, ignore_errors)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/storage/rbd_utils.py", line 396, in remove_snap
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher with RBDVolumeProxy(self, str(volume), pool=pool) as vol:
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/storage/rbd_utils.py", line 65, in __init__
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher driver._disconnect_from_rados(client, ioctx)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher self.force_reraise()
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/storage/rbd_utils.py", line 61, in __init__
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher read_only=read_only)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "rbd.pyx", line 1042, in rbd.Image.__init__ (/tmp/buildd/ceph-10.2.2/src/build/rbd.c:9862)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher ImageNotFound: error opening image 9debce4f-ee92-4982-815f-769380728951_disk at snapshot None
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher
openstack server show 9debce4f-ee92-4982-815f-769380728951
+--------------------------------------+------------------------------------------------------------------+
| Field | Value |
+--------------------------------------+------------------------------------------------------------------+
| OS-DCF:diskConfig | AUTO |
| OS-EXT-AZ:availability_zone | lab |
| OS-EXT-SRV-ATTR:host | h12 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | h12.example.com |
| OS-EXT-SRV-ATTR:instance_name | instance-00000150 |
| OS-EXT-STS:power_state | 4 |
| OS-EXT-STS:task_state | None |
| OS-EXT-STS:vm_state | stopped |
| OS-SRV-USG:launched_at | 2016-07-05T06:40:11.000000 |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | 1st-network=172.16.0.12 |
| config_drive | |
| created | 2016-07-05T03:32:37Z |
| flavor | m1.large (4) |
| hostId | 90f85ee8c380c68c55b169ff1dc9917729638d0e744020641ace2700 |
| id | 9debce4f-ee92-4982-815f-769380728951 |
| image | |
| key_name | desktop |
| name | e1 |
| os-extended-volumes:volumes_attached | [{u'id': u'90ea9f31-57b4-4162-8986-47065b23f407'}] |
| project_id | 04e7fe0cfb19418a9ec2eacfe1d334d5 |
| properties | |
| security_groups | [{u'name': u'default'}] |
| status | SHUTOFF |
| updated | 2016-07-05T07:04:11Z |
| user_id | 8e06675285464030a9fd7badf4b0939cba85e018ab8a20ea8cb911222fef7771 |
+--------------------------------------+------------------------------------------------------------------+
openstack volume show 90ea9f31-57b4-4162-8986-47065b23f407
+--------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+--------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| attachments | [{u'server_id': u'9debce4f-ee92-4982-815f-769380728951', u'attachment_id': u'd119c454-65f4-4c7b-8135-3238a3e7a424', u'attached_at': u'2016-07-05T03:33:37.127003', u'host_name': None, u'volume_id': u'90ea9f31-57b4-4162-8986-47065b23f407', u'device': u'/dev/vda', u'id': u'90ea9f31-57b4-4162-8986-47065b23f407'}] |
| availability_zone | nova |
| bootable | true |
| consistencygroup_id | None |
| created_at | 2016-07-05T03:32:38.353914 |
| description | |
| encrypted | False |
| id | 90ea9f31-57b4-4162-8986-47065b23f407 |
| migration_status | None |
| multiattach | False |
| name | |
| os-vol-host-attr:host | mitaka-cinder@ceph#ceph |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | 04e7fe0cfb19418a9ec2eacfe1d334d5 |
| properties | attached_mode='rw', readonly='False' |
| replication_status | disabled |
| size | 40 |
| snapshot_id | None |
| source_volid | None |
| status | in-use |
| type | ceph |
| updated_at | 2016-07-05T03:33:37.161179 |
| user_id | 8e06675285464030a9fd7badf4b0939cba85e018ab8a20ea8cb911222fef7771 |
| volume_image_metadata | {u'description': u'', u'os_require_quiesce': u'yes', u'checksum': u'cb3d84ccc990821e537fbe5227b3dbc3', u'hw_qemu_guest_agent': u'yes', u'disk_format': u'raw', u'image_name': u'os2-2016062303', u'hw_scsi_model': u'virtio-scsi', u'image_id': u'083aadf6-6a4c-45aa-aab6-66b6d2b2c2ba', u'min_ram': u'0', u'container_format': u'bare', u'min_disk': u'3', u'hw_disk_bus': u'scsi', u'size': u'2705326080'} |
+--------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
# On the Ceph cluster
⟫ sudo rbd ls openstack-volume | grep 9debce4f-ee92-4982-815f-769380728951_disk
1 ⟫ sudo rbd ls openstack-volume | grep 90ea9f31-57b4-4162-8986-47065b23f407
volume-90ea9f31-57b4-4162-8986-47065b23f407
⟫ sudo rbd info -p openstack-volume volume-90ea9f31-57b4-4162-8986-47065b23f407
rbd image 'volume-90ea9f31-57b4-4162-8986-47065b23f407':
size 40960 MB in 10240 objects
order 22 (4096 kB objects)
block_name_prefix: rbd_data.b412b0238e1f29
format: 2
features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
flags:
(The volume size has not been changed.)
As you can see above, nova-compute tries to do something on the 9debce4f-ee92-4982-815f-769380728951_disk which doesn't actually exist, recognizing it as an ephemeral disk. |
Description
===========
nova-compute fails to resize instance that was booted by creating a volume, recognizing it as an ephemeral disk.
Steps to reproduce
==================
1. Create an instance by selecting boot source as 'image' and creating a new volume
2. Resize the instance
3. Confirm the resizing
Expected result
===============
successful resize
Actual result
=============
It passes until resize confirmation without any error but it fails after the resize confirmation is hit.
Environment
===========
1. Exact version of OpenStack you are running. See the following
Mitaka
dpkg -l | grep nova
ii nova-common 2:13.0.0-0ubuntu2 all OpenStack Compute - common files
ii nova-compute 2:13.0.0-0ubuntu2 all OpenStack Compute - compute node base
ii nova-compute-kvm 2:13.0.0-0ubuntu2 all OpenStack Compute - compute node (KVM)
ii nova-compute-libvirt 2:13.0.0-0ubuntu2 all OpenStack Compute - compute node libvirt support
ii python-nova 2:13.0.0-0ubuntu2 all OpenStack Compute Python libraries
ii python-novaclient 2:3.3.1-2 all client library for OpenStack Compute API - Python 2.7
2. Which hypervisor did you use?
Libvirt + KVM
dpkg -l | egrep 'libvirt|qemu|kvm'
ii ipxe-qemu 1.0.0+git-20150424.a25a16d-1ubuntu1 all PXE boot firmware - ROM images for qemu
ii libvirt-bin 1.3.1-1ubuntu10.1 amd64 programs for the libvirt library
ii libvirt0:amd64 1.3.1-1ubuntu10.1 amd64 library for interfacing with different virtualization systems
ii nova-compute-kvm 2:13.0.0-0ubuntu2 all OpenStack Compute - compute node (KVM)
ii nova-compute-libvirt 2:13.0.0-0ubuntu2 all OpenStack Compute - compute node libvirt support
ii python-libvirt 1.3.1-1ubuntu1 amd64 libvirt Python bindings
ii qemu-block-extra:amd64 1:2.5+dfsg-5ubuntu10.2 amd64 extra block backend modules for qemu-system and qemu-utils
ii qemu-slof 20151103+dfsg-1ubuntu1 all Slimline Open Firmware -- QEMU PowerPC version
ii qemu-system 1:2.5+dfsg-5ubuntu10.2 amd64 QEMU full system emulation binaries
ii qemu-system-arm 1:2.5+dfsg-5ubuntu10.2 amd64 QEMU full system emulation binaries (arm)
ii qemu-system-common 1:2.5+dfsg-5ubuntu10.2 amd64 QEMU full system emulation binaries (common files)
ii qemu-system-mips 1:2.5+dfsg-5ubuntu10.2 amd64 QEMU full system emulation binaries (mips)
ii qemu-system-misc 1:2.5+dfsg-5ubuntu10.2 amd64 QEMU full system emulation binaries (miscelaneous)
ii qemu-system-ppc 1:2.5+dfsg-5ubuntu10.2 amd64 QEMU full system emulation binaries (ppc)
ii qemu-system-sparc 1:2.5+dfsg-5ubuntu10.2 amd64 QEMU full system emulation binaries (sparc)
ii qemu-system-x86 1:2.5+dfsg-5ubuntu10.2 amd64 QEMU full system emulation binaries (x86)
ii qemu-utils 1:2.5+dfsg-5ubuntu10.2 amd64 QEMU utilities
2. Which storage type did you use?
Ceph
dpkg -l | egrep 'ceph'
ii ceph-common 10.2.2-1xenial amd64 common utilities to mount and interact with a ceph storage cluster
ii libcephfs1 10.2.2-1xenial amd64 Ceph distributed file system client library
ii python-cephfs 10.2.2-1xenial amd64 Python libraries for the Ceph libcephfs library
3. Which networking type did you use?
Neutron with OpenVSwitch
# On the api node
dpkg -l | grep neutron
ii neutron-common 2:8.1.0-0ubuntu0.16.04.2 all Neutron is a virtual network service for Openstack - common
ii neutron-plugin-ml2 2:8.1.0-0ubuntu0.16.04.2 all Neutron is a virtual network service for Openstack - ML2 plugin
ii neutron-server 2:8.1.0-0ubuntu0.16.04.2 all Neutron is a virtual network service for Openstack - server
ii python-neutron 2:8.1.0-0ubuntu0.16.04.2 all Neutron is a virtual network service for Openstack - Python library
ii python-neutron-fwaas 1:8.0.0-0ubuntu1 all Firewall-as-a-Service driver for OpenStack Neutron
ii python-neutron-lib 0.0.2-2 all Neutron shared routines and utilities - Python 2.7
ii python-neutronclient 1:4.1.1-2 all client API library for Neutron - Python 2.7
# On the network node
dpkg -l | grep neutron
ii neutron-common 2:8.1.0-0ubuntu0.16.04.2 all Neutron is a virtual network service for Openstack - common
ii neutron-dhcp-agent 2:8.1.0-0ubuntu0.16.04.2 all Neutron is a virtual network service for Openstack - DHCP agent
ii neutron-l3-agent 2:8.1.0-0ubuntu0.16.04.2 all Neutron is a virtual network service for Openstack - l3 agent
ii neutron-metadata-agent 2:8.1.0-0ubuntu0.16.04.2 all Neutron is a virtual network service for Openstack - metadata agent
ii neutron-openvswitch-agent 2:8.1.0-0ubuntu0.16.04.2 all Neutron is a virtual network service for Openstack - Open vSwitch plugin agent
ii python-neutron 2:8.1.0-0ubuntu0.16.04.2 all Neutron is a virtual network service for Openstack - Python library
ii python-neutron-fwaas 1:8.0.0-0ubuntu1 all Firewall-as-a-Service driver for OpenStack Neutron
ii python-neutron-lib 0.0.2-2 all Neutron shared routines and utilities - Python 2.7
ii python-neutronclient 1:4.1.1-2 all client API library for Neutron - Python 2.7
Logs & Configs
==============
2016-07-05 15:40:32.701 99291 DEBUG oslo_concurrency.lockutils [req-41fc2f7f-8c45-4c2d-ba7b-2144fcbfd757 8e06675285464030a9fd7badf4b0939cba85e018ab8a20ea8cb911222fef7771 04e7fe0cfb19418a9ec2eacfe1d334d5 - - -] Releasing semaphore "refresh_cache-9debce4f-ee92-4982-815f-769380728951" lock /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:228
2016-07-05 15:40:32.702 99291 DEBUG oslo_concurrency.processutils [req-41fc2f7f-8c45-4c2d-ba7b-2144fcbfd757 8e06675285464030a9fd7badf4b0939cba85e018ab8a20ea8cb911222fef7771 04e7fe0cfb19418a9ec2eacfe1d334d5 - - -] Running cmd (subprocess): rm -rf /var/lib/nova/instances/9debce4f-ee92-4982-815f-769380728951_resize execute /usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:326
2016-07-05 15:40:32.728 99291 DEBUG oslo_concurrency.processutils [req-41fc2f7f-8c45-4c2d-ba7b-2144fcbfd757 8e06675285464030a9fd7badf4b0939cba85e018ab8a20ea8cb911222fef7771 04e7fe0cfb19418a9ec2eacfe1d334d5 - - -] CMD "rm -rf /var/lib/nova/instances/9debce4f-ee92-4982-815f-769380728951_resize" returned: 0 in 0.026s execute /usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:356
2016-07-05 15:40:32.741 99291 DEBUG nova.virt.libvirt.storage.rbd_utils [req-41fc2f7f-8c45-4c2d-ba7b-2144fcbfd757 8e06675285464030a9fd7badf4b0939cba85e018ab8a20ea8cb911222fef7771 04e7fe0cfb19418a9ec2eacfe1d334d5 - - -] rbd image 9debce4f-ee92-4982-815f-769380728951_disk does not exist __init__ /usr/lib/python2.7/dist-packages/nova/virt/libvirt/storage/rbd_utils.py:64
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [req-41fc2f7f-8c45-4c2d-ba7b-2144fcbfd757 8e06675285464030a9fd7badf4b0939cba85e018ab8a20ea8cb911222fef7771 04e7fe0cfb19418a9ec2eacfe1d334d5 - - -] [instance: 9debce4f-ee92-4982-815f-769380728951] Setting instance vm_state to ERROR
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] Traceback (most recent call last):
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6588, in _error_out_instance_on_exception
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] yield
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3531, in _confirm_resize
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] network_info)
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 7484, in confirm_migration
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] self._cleanup_resize(instance, network_info)
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1258, in _cleanup_resize
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] ignore_errors=True)
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 941, in remove_snap
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] return self.driver.remove_snap(self.rbd_name, name, ignore_errors)
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/storage/rbd_utils.py", line 396, in remove_snap
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] with RBDVolumeProxy(self, str(volume), pool=pool) as vol:
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/storage/rbd_utils.py", line 65, in __init__
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] driver._disconnect_from_rados(client, ioctx)
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] self.force_reraise()
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] six.reraise(self.type_, self.value, self.tb)
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/storage/rbd_utils.py", line 61, in __init__
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] read_only=read_only)
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] File "rbd.pyx", line 1042, in rbd.Image.__init__ (/tmp/buildd/ceph-10.2.2/src/build/rbd.c:9862)
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] ImageNotFound: error opening image 9debce4f-ee92-4982-815f-769380728951_disk at snapshot None
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951]
2016-07-05 15:40:32.811 99291 DEBUG oslo_concurrency.lockutils [req-41fc2f7f-8c45-4c2d-ba7b-2144fcbfd757 8e06675285464030a9fd7badf4b0939cba85e018ab8a20ea8cb911222fef7771 04e7fe0cfb19418a9ec2eacfe1d334d5 - - -] Lock "9debce4f-ee92-4982-815f-769380728951" released by "nova.compute.manager.do_confirm_resize" :: held 0.507s inner /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:285
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher [req-41fc2f7f-8c45-4c2d-ba7b-2144fcbfd757 8e06675285464030a9fd7badf4b0939cba85e018ab8a20ea8cb911222fef7771 04e7fe0cfb19418a9ec2eacfe1d334d5 - - -] Exception during message handling: error opening image 9debce4f-ee92-4982-815f-769380728951_disk at snapshot None
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher incoming.message))
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 110, in wrapped
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher payload)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher self.force_reraise()
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 89, in wrapped
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher return f(self, context, *args, **kw)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 409, in decorated_function
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 387, in decorated_function
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info())
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher self.force_reraise()
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 375, in decorated_function
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3507, in confirm_resize
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher do_confirm_resize(context, instance, migration.id)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 274, in inner
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher return f(*args, **kwargs)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3505, in do_confirm_resize
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher migration=migration)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3531, in _confirm_resize
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher network_info)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 7484, in confirm_migration
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher self._cleanup_resize(instance, network_info)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1258, in _cleanup_resize
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher ignore_errors=True)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 941, in remove_snap
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher return self.driver.remove_snap(self.rbd_name, name, ignore_errors)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/storage/rbd_utils.py", line 396, in remove_snap
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher with RBDVolumeProxy(self, str(volume), pool=pool) as vol:
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/storage/rbd_utils.py", line 65, in __init__
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher driver._disconnect_from_rados(client, ioctx)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher self.force_reraise()
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/storage/rbd_utils.py", line 61, in __init__
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher read_only=read_only)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "rbd.pyx", line 1042, in rbd.Image.__init__ (/tmp/buildd/ceph-10.2.2/src/build/rbd.c:9862)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher ImageNotFound: error opening image 9debce4f-ee92-4982-815f-769380728951_disk at snapshot None
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher
openstack server show 9debce4f-ee92-4982-815f-769380728951
+--------------------------------------+------------------------------------------------------------------+
| Field | Value |
+--------------------------------------+------------------------------------------------------------------+
| OS-DCF:diskConfig | AUTO |
| OS-EXT-AZ:availability_zone | lab |
| OS-EXT-SRV-ATTR:host | h12 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | h12.example.com |
| OS-EXT-SRV-ATTR:instance_name | instance-00000150 |
| OS-EXT-STS:power_state | 4 |
| OS-EXT-STS:task_state | None |
| OS-EXT-STS:vm_state | stopped |
| OS-SRV-USG:launched_at | 2016-07-05T06:40:11.000000 |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | 1st-network=172.16.0.12 |
| config_drive | |
| created | 2016-07-05T03:32:37Z |
| flavor | m1.large (4) |
| hostId | 90f85ee8c380c68c55b169ff1dc9917729638d0e744020641ace2700 |
| id | 9debce4f-ee92-4982-815f-769380728951 |
| image | |
| key_name | desktop |
| name | e1 |
| os-extended-volumes:volumes_attached | [{u'id': u'90ea9f31-57b4-4162-8986-47065b23f407'}] |
| project_id | 04e7fe0cfb19418a9ec2eacfe1d334d5 |
| properties | |
| security_groups | [{u'name': u'default'}] |
| status | SHUTOFF |
| updated | 2016-07-05T07:04:11Z |
| user_id | 8e06675285464030a9fd7badf4b0939cba85e018ab8a20ea8cb911222fef7771 |
+--------------------------------------+------------------------------------------------------------------+
openstack volume show 90ea9f31-57b4-4162-8986-47065b23f407
+--------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+--------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| attachments | [{u'server_id': u'9debce4f-ee92-4982-815f-769380728951', u'attachment_id': u'd119c454-65f4-4c7b-8135-3238a3e7a424', u'attached_at': u'2016-07-05T03:33:37.127003', u'host_name': None, u'volume_id': u'90ea9f31-57b4-4162-8986-47065b23f407', u'device': u'/dev/vda', u'id': u'90ea9f31-57b4-4162-8986-47065b23f407'}] |
| availability_zone | nova |
| bootable | true |
| consistencygroup_id | None |
| created_at | 2016-07-05T03:32:38.353914 |
| description | |
| encrypted | False |
| id | 90ea9f31-57b4-4162-8986-47065b23f407 |
| migration_status | None |
| multiattach | False |
| name | |
| os-vol-host-attr:host | mitaka-cinder@ceph#ceph |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | 04e7fe0cfb19418a9ec2eacfe1d334d5 |
| properties | attached_mode='rw', readonly='False' |
| replication_status | disabled |
| size | 40 |
| snapshot_id | None |
| source_volid | None |
| status | in-use |
| type | ceph |
| updated_at | 2016-07-05T03:33:37.161179 |
| user_id | 8e06675285464030a9fd7badf4b0939cba85e018ab8a20ea8cb911222fef7771 |
| volume_image_metadata | {u'description': u'', u'os_require_quiesce': u'yes', u'checksum': u'cb3d84ccc990821e537fbe5227b3dbc3', u'hw_qemu_guest_agent': u'yes', u'disk_format': u'raw', u'image_name': u'os2-2016062303', u'hw_scsi_model': u'virtio-scsi', u'image_id': u'083aadf6-6a4c-45aa-aab6-66b6d2b2c2ba', u'min_ram': u'0', u'container_format': u'bare', u'min_disk': u'3', u'hw_disk_bus': u'scsi', u'size': u'2705326080'} |
+--------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
# On the Ceph cluster
⟫ sudo rbd ls openstack-volume | grep 9debce4f-ee92-4982-815f-769380728951_disk
1 ⟫ sudo rbd ls openstack-volume | grep 90ea9f31-57b4-4162-8986-47065b23f407
volume-90ea9f31-57b4-4162-8986-47065b23f407
⟫ sudo rbd info -p openstack-volume volume-90ea9f31-57b4-4162-8986-47065b23f407
rbd image 'volume-90ea9f31-57b4-4162-8986-47065b23f407':
size 40960 MB in 10240 objects
order 22 (4096 kB objects)
block_name_prefix: rbd_data.b412b0238e1f29
format: 2
features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
flags:
(The volume size has not been changed.)
As you can see above, nova-compute tries to do something on the 9debce4f-ee92-4982-815f-769380728951_disk which doesn't actually exist, recognizing it as an ephemeral disk. |
|
| 2016-07-05 08:46:09 |
Dongwon Cho |
description |
Description
===========
nova-compute fails to resize instance that was booted by creating a volume, recognizing it as an ephemeral disk.
Steps to reproduce
==================
1. Create an instance by selecting boot source as 'image' and creating a new volume
2. Resize the instance
3. Confirm the resizing
Expected result
===============
successful resize
Actual result
=============
It passes until resize confirmation without any error but it fails after the resize confirmation is hit.
Environment
===========
1. Exact version of OpenStack you are running. See the following
Mitaka
dpkg -l | grep nova
ii nova-common 2:13.0.0-0ubuntu2 all OpenStack Compute - common files
ii nova-compute 2:13.0.0-0ubuntu2 all OpenStack Compute - compute node base
ii nova-compute-kvm 2:13.0.0-0ubuntu2 all OpenStack Compute - compute node (KVM)
ii nova-compute-libvirt 2:13.0.0-0ubuntu2 all OpenStack Compute - compute node libvirt support
ii python-nova 2:13.0.0-0ubuntu2 all OpenStack Compute Python libraries
ii python-novaclient 2:3.3.1-2 all client library for OpenStack Compute API - Python 2.7
2. Which hypervisor did you use?
Libvirt + KVM
dpkg -l | egrep 'libvirt|qemu|kvm'
ii ipxe-qemu 1.0.0+git-20150424.a25a16d-1ubuntu1 all PXE boot firmware - ROM images for qemu
ii libvirt-bin 1.3.1-1ubuntu10.1 amd64 programs for the libvirt library
ii libvirt0:amd64 1.3.1-1ubuntu10.1 amd64 library for interfacing with different virtualization systems
ii nova-compute-kvm 2:13.0.0-0ubuntu2 all OpenStack Compute - compute node (KVM)
ii nova-compute-libvirt 2:13.0.0-0ubuntu2 all OpenStack Compute - compute node libvirt support
ii python-libvirt 1.3.1-1ubuntu1 amd64 libvirt Python bindings
ii qemu-block-extra:amd64 1:2.5+dfsg-5ubuntu10.2 amd64 extra block backend modules for qemu-system and qemu-utils
ii qemu-slof 20151103+dfsg-1ubuntu1 all Slimline Open Firmware -- QEMU PowerPC version
ii qemu-system 1:2.5+dfsg-5ubuntu10.2 amd64 QEMU full system emulation binaries
ii qemu-system-arm 1:2.5+dfsg-5ubuntu10.2 amd64 QEMU full system emulation binaries (arm)
ii qemu-system-common 1:2.5+dfsg-5ubuntu10.2 amd64 QEMU full system emulation binaries (common files)
ii qemu-system-mips 1:2.5+dfsg-5ubuntu10.2 amd64 QEMU full system emulation binaries (mips)
ii qemu-system-misc 1:2.5+dfsg-5ubuntu10.2 amd64 QEMU full system emulation binaries (miscelaneous)
ii qemu-system-ppc 1:2.5+dfsg-5ubuntu10.2 amd64 QEMU full system emulation binaries (ppc)
ii qemu-system-sparc 1:2.5+dfsg-5ubuntu10.2 amd64 QEMU full system emulation binaries (sparc)
ii qemu-system-x86 1:2.5+dfsg-5ubuntu10.2 amd64 QEMU full system emulation binaries (x86)
ii qemu-utils 1:2.5+dfsg-5ubuntu10.2 amd64 QEMU utilities
2. Which storage type did you use?
Ceph
dpkg -l | egrep 'ceph'
ii ceph-common 10.2.2-1xenial amd64 common utilities to mount and interact with a ceph storage cluster
ii libcephfs1 10.2.2-1xenial amd64 Ceph distributed file system client library
ii python-cephfs 10.2.2-1xenial amd64 Python libraries for the Ceph libcephfs library
3. Which networking type did you use?
Neutron with OpenVSwitch
# On the api node
dpkg -l | grep neutron
ii neutron-common 2:8.1.0-0ubuntu0.16.04.2 all Neutron is a virtual network service for Openstack - common
ii neutron-plugin-ml2 2:8.1.0-0ubuntu0.16.04.2 all Neutron is a virtual network service for Openstack - ML2 plugin
ii neutron-server 2:8.1.0-0ubuntu0.16.04.2 all Neutron is a virtual network service for Openstack - server
ii python-neutron 2:8.1.0-0ubuntu0.16.04.2 all Neutron is a virtual network service for Openstack - Python library
ii python-neutron-fwaas 1:8.0.0-0ubuntu1 all Firewall-as-a-Service driver for OpenStack Neutron
ii python-neutron-lib 0.0.2-2 all Neutron shared routines and utilities - Python 2.7
ii python-neutronclient 1:4.1.1-2 all client API library for Neutron - Python 2.7
# On the network node
dpkg -l | grep neutron
ii neutron-common 2:8.1.0-0ubuntu0.16.04.2 all Neutron is a virtual network service for Openstack - common
ii neutron-dhcp-agent 2:8.1.0-0ubuntu0.16.04.2 all Neutron is a virtual network service for Openstack - DHCP agent
ii neutron-l3-agent 2:8.1.0-0ubuntu0.16.04.2 all Neutron is a virtual network service for Openstack - l3 agent
ii neutron-metadata-agent 2:8.1.0-0ubuntu0.16.04.2 all Neutron is a virtual network service for Openstack - metadata agent
ii neutron-openvswitch-agent 2:8.1.0-0ubuntu0.16.04.2 all Neutron is a virtual network service for Openstack - Open vSwitch plugin agent
ii python-neutron 2:8.1.0-0ubuntu0.16.04.2 all Neutron is a virtual network service for Openstack - Python library
ii python-neutron-fwaas 1:8.0.0-0ubuntu1 all Firewall-as-a-Service driver for OpenStack Neutron
ii python-neutron-lib 0.0.2-2 all Neutron shared routines and utilities - Python 2.7
ii python-neutronclient 1:4.1.1-2 all client API library for Neutron - Python 2.7
Logs & Configs
==============
2016-07-05 15:40:32.701 99291 DEBUG oslo_concurrency.lockutils [req-41fc2f7f-8c45-4c2d-ba7b-2144fcbfd757 8e06675285464030a9fd7badf4b0939cba85e018ab8a20ea8cb911222fef7771 04e7fe0cfb19418a9ec2eacfe1d334d5 - - -] Releasing semaphore "refresh_cache-9debce4f-ee92-4982-815f-769380728951" lock /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:228
2016-07-05 15:40:32.702 99291 DEBUG oslo_concurrency.processutils [req-41fc2f7f-8c45-4c2d-ba7b-2144fcbfd757 8e06675285464030a9fd7badf4b0939cba85e018ab8a20ea8cb911222fef7771 04e7fe0cfb19418a9ec2eacfe1d334d5 - - -] Running cmd (subprocess): rm -rf /var/lib/nova/instances/9debce4f-ee92-4982-815f-769380728951_resize execute /usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:326
2016-07-05 15:40:32.728 99291 DEBUG oslo_concurrency.processutils [req-41fc2f7f-8c45-4c2d-ba7b-2144fcbfd757 8e06675285464030a9fd7badf4b0939cba85e018ab8a20ea8cb911222fef7771 04e7fe0cfb19418a9ec2eacfe1d334d5 - - -] CMD "rm -rf /var/lib/nova/instances/9debce4f-ee92-4982-815f-769380728951_resize" returned: 0 in 0.026s execute /usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:356
2016-07-05 15:40:32.741 99291 DEBUG nova.virt.libvirt.storage.rbd_utils [req-41fc2f7f-8c45-4c2d-ba7b-2144fcbfd757 8e06675285464030a9fd7badf4b0939cba85e018ab8a20ea8cb911222fef7771 04e7fe0cfb19418a9ec2eacfe1d334d5 - - -] rbd image 9debce4f-ee92-4982-815f-769380728951_disk does not exist __init__ /usr/lib/python2.7/dist-packages/nova/virt/libvirt/storage/rbd_utils.py:64
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [req-41fc2f7f-8c45-4c2d-ba7b-2144fcbfd757 8e06675285464030a9fd7badf4b0939cba85e018ab8a20ea8cb911222fef7771 04e7fe0cfb19418a9ec2eacfe1d334d5 - - -] [instance: 9debce4f-ee92-4982-815f-769380728951] Setting instance vm_state to ERROR
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] Traceback (most recent call last):
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6588, in _error_out_instance_on_exception
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] yield
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3531, in _confirm_resize
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] network_info)
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 7484, in confirm_migration
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] self._cleanup_resize(instance, network_info)
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1258, in _cleanup_resize
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] ignore_errors=True)
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 941, in remove_snap
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] return self.driver.remove_snap(self.rbd_name, name, ignore_errors)
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/storage/rbd_utils.py", line 396, in remove_snap
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] with RBDVolumeProxy(self, str(volume), pool=pool) as vol:
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/storage/rbd_utils.py", line 65, in __init__
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] driver._disconnect_from_rados(client, ioctx)
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] self.force_reraise()
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] six.reraise(self.type_, self.value, self.tb)
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/storage/rbd_utils.py", line 61, in __init__
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] read_only=read_only)
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] File "rbd.pyx", line 1042, in rbd.Image.__init__ (/tmp/buildd/ceph-10.2.2/src/build/rbd.c:9862)
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] ImageNotFound: error opening image 9debce4f-ee92-4982-815f-769380728951_disk at snapshot None
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951]
2016-07-05 15:40:32.811 99291 DEBUG oslo_concurrency.lockutils [req-41fc2f7f-8c45-4c2d-ba7b-2144fcbfd757 8e06675285464030a9fd7badf4b0939cba85e018ab8a20ea8cb911222fef7771 04e7fe0cfb19418a9ec2eacfe1d334d5 - - -] Lock "9debce4f-ee92-4982-815f-769380728951" released by "nova.compute.manager.do_confirm_resize" :: held 0.507s inner /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:285
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher [req-41fc2f7f-8c45-4c2d-ba7b-2144fcbfd757 8e06675285464030a9fd7badf4b0939cba85e018ab8a20ea8cb911222fef7771 04e7fe0cfb19418a9ec2eacfe1d334d5 - - -] Exception during message handling: error opening image 9debce4f-ee92-4982-815f-769380728951_disk at snapshot None
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher incoming.message))
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 110, in wrapped
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher payload)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher self.force_reraise()
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 89, in wrapped
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher return f(self, context, *args, **kw)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 409, in decorated_function
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 387, in decorated_function
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info())
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher self.force_reraise()
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 375, in decorated_function
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3507, in confirm_resize
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher do_confirm_resize(context, instance, migration.id)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 274, in inner
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher return f(*args, **kwargs)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3505, in do_confirm_resize
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher migration=migration)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3531, in _confirm_resize
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher network_info)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 7484, in confirm_migration
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher self._cleanup_resize(instance, network_info)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1258, in _cleanup_resize
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher ignore_errors=True)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 941, in remove_snap
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher return self.driver.remove_snap(self.rbd_name, name, ignore_errors)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/storage/rbd_utils.py", line 396, in remove_snap
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher with RBDVolumeProxy(self, str(volume), pool=pool) as vol:
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/storage/rbd_utils.py", line 65, in __init__
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher driver._disconnect_from_rados(client, ioctx)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher self.force_reraise()
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/storage/rbd_utils.py", line 61, in __init__
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher read_only=read_only)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "rbd.pyx", line 1042, in rbd.Image.__init__ (/tmp/buildd/ceph-10.2.2/src/build/rbd.c:9862)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher ImageNotFound: error opening image 9debce4f-ee92-4982-815f-769380728951_disk at snapshot None
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher
openstack server show 9debce4f-ee92-4982-815f-769380728951
+--------------------------------------+------------------------------------------------------------------+
| Field | Value |
+--------------------------------------+------------------------------------------------------------------+
| OS-DCF:diskConfig | AUTO |
| OS-EXT-AZ:availability_zone | lab |
| OS-EXT-SRV-ATTR:host | h12 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | h12.example.com |
| OS-EXT-SRV-ATTR:instance_name | instance-00000150 |
| OS-EXT-STS:power_state | 4 |
| OS-EXT-STS:task_state | None |
| OS-EXT-STS:vm_state | stopped |
| OS-SRV-USG:launched_at | 2016-07-05T06:40:11.000000 |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | 1st-network=172.16.0.12 |
| config_drive | |
| created | 2016-07-05T03:32:37Z |
| flavor | m1.large (4) |
| hostId | 90f85ee8c380c68c55b169ff1dc9917729638d0e744020641ace2700 |
| id | 9debce4f-ee92-4982-815f-769380728951 |
| image | |
| key_name | desktop |
| name | e1 |
| os-extended-volumes:volumes_attached | [{u'id': u'90ea9f31-57b4-4162-8986-47065b23f407'}] |
| project_id | 04e7fe0cfb19418a9ec2eacfe1d334d5 |
| properties | |
| security_groups | [{u'name': u'default'}] |
| status | SHUTOFF |
| updated | 2016-07-05T07:04:11Z |
| user_id | 8e06675285464030a9fd7badf4b0939cba85e018ab8a20ea8cb911222fef7771 |
+--------------------------------------+------------------------------------------------------------------+
openstack volume show 90ea9f31-57b4-4162-8986-47065b23f407
+--------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+--------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| attachments | [{u'server_id': u'9debce4f-ee92-4982-815f-769380728951', u'attachment_id': u'd119c454-65f4-4c7b-8135-3238a3e7a424', u'attached_at': u'2016-07-05T03:33:37.127003', u'host_name': None, u'volume_id': u'90ea9f31-57b4-4162-8986-47065b23f407', u'device': u'/dev/vda', u'id': u'90ea9f31-57b4-4162-8986-47065b23f407'}] |
| availability_zone | nova |
| bootable | true |
| consistencygroup_id | None |
| created_at | 2016-07-05T03:32:38.353914 |
| description | |
| encrypted | False |
| id | 90ea9f31-57b4-4162-8986-47065b23f407 |
| migration_status | None |
| multiattach | False |
| name | |
| os-vol-host-attr:host | mitaka-cinder@ceph#ceph |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | 04e7fe0cfb19418a9ec2eacfe1d334d5 |
| properties | attached_mode='rw', readonly='False' |
| replication_status | disabled |
| size | 40 |
| snapshot_id | None |
| source_volid | None |
| status | in-use |
| type | ceph |
| updated_at | 2016-07-05T03:33:37.161179 |
| user_id | 8e06675285464030a9fd7badf4b0939cba85e018ab8a20ea8cb911222fef7771 |
| volume_image_metadata | {u'description': u'', u'os_require_quiesce': u'yes', u'checksum': u'cb3d84ccc990821e537fbe5227b3dbc3', u'hw_qemu_guest_agent': u'yes', u'disk_format': u'raw', u'image_name': u'os2-2016062303', u'hw_scsi_model': u'virtio-scsi', u'image_id': u'083aadf6-6a4c-45aa-aab6-66b6d2b2c2ba', u'min_ram': u'0', u'container_format': u'bare', u'min_disk': u'3', u'hw_disk_bus': u'scsi', u'size': u'2705326080'} |
+--------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
# On the Ceph cluster
⟫ sudo rbd ls openstack-volume | grep 9debce4f-ee92-4982-815f-769380728951_disk
1 ⟫ sudo rbd ls openstack-volume | grep 90ea9f31-57b4-4162-8986-47065b23f407
volume-90ea9f31-57b4-4162-8986-47065b23f407
⟫ sudo rbd info -p openstack-volume volume-90ea9f31-57b4-4162-8986-47065b23f407
rbd image 'volume-90ea9f31-57b4-4162-8986-47065b23f407':
size 40960 MB in 10240 objects
order 22 (4096 kB objects)
block_name_prefix: rbd_data.b412b0238e1f29
format: 2
features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
flags:
(The volume size has not been changed.)
As you can see above, nova-compute tries to do something on the 9debce4f-ee92-4982-815f-769380728951_disk which doesn't actually exist, recognizing it as an ephemeral disk. |
Description
===========
nova-compute fails to resize instance that was booted by creating a volume, recognizing it as an ephemeral disk.
Steps to reproduce
==================
1. Create an instance by selecting boot source as 'image' and creating a new volume
2. Resize the instance
3. Confirm the resizing
Expected result
===============
successful resize
Actual result
=============
It passes until the step of resize confirmation without any error but it fails after the resize confirmation is hit.
Environment
===========
1. Exact version of OpenStack you are running. See the following
Mitaka
dpkg -l | grep nova
ii nova-common 2:13.0.0-0ubuntu2 all OpenStack Compute - common files
ii nova-compute 2:13.0.0-0ubuntu2 all OpenStack Compute - compute node base
ii nova-compute-kvm 2:13.0.0-0ubuntu2 all OpenStack Compute - compute node (KVM)
ii nova-compute-libvirt 2:13.0.0-0ubuntu2 all OpenStack Compute - compute node libvirt support
ii python-nova 2:13.0.0-0ubuntu2 all OpenStack Compute Python libraries
ii python-novaclient 2:3.3.1-2 all client library for OpenStack Compute API - Python 2.7
2. Which hypervisor did you use?
Libvirt + KVM
dpkg -l | egrep 'libvirt|qemu|kvm'
ii ipxe-qemu 1.0.0+git-20150424.a25a16d-1ubuntu1 all PXE boot firmware - ROM images for qemu
ii libvirt-bin 1.3.1-1ubuntu10.1 amd64 programs for the libvirt library
ii libvirt0:amd64 1.3.1-1ubuntu10.1 amd64 library for interfacing with different virtualization systems
ii nova-compute-kvm 2:13.0.0-0ubuntu2 all OpenStack Compute - compute node (KVM)
ii nova-compute-libvirt 2:13.0.0-0ubuntu2 all OpenStack Compute - compute node libvirt support
ii python-libvirt 1.3.1-1ubuntu1 amd64 libvirt Python bindings
ii qemu-block-extra:amd64 1:2.5+dfsg-5ubuntu10.2 amd64 extra block backend modules for qemu-system and qemu-utils
ii qemu-slof 20151103+dfsg-1ubuntu1 all Slimline Open Firmware -- QEMU PowerPC version
ii qemu-system 1:2.5+dfsg-5ubuntu10.2 amd64 QEMU full system emulation binaries
ii qemu-system-arm 1:2.5+dfsg-5ubuntu10.2 amd64 QEMU full system emulation binaries (arm)
ii qemu-system-common 1:2.5+dfsg-5ubuntu10.2 amd64 QEMU full system emulation binaries (common files)
ii qemu-system-mips 1:2.5+dfsg-5ubuntu10.2 amd64 QEMU full system emulation binaries (mips)
ii qemu-system-misc 1:2.5+dfsg-5ubuntu10.2 amd64 QEMU full system emulation binaries (miscelaneous)
ii qemu-system-ppc 1:2.5+dfsg-5ubuntu10.2 amd64 QEMU full system emulation binaries (ppc)
ii qemu-system-sparc 1:2.5+dfsg-5ubuntu10.2 amd64 QEMU full system emulation binaries (sparc)
ii qemu-system-x86 1:2.5+dfsg-5ubuntu10.2 amd64 QEMU full system emulation binaries (x86)
ii qemu-utils 1:2.5+dfsg-5ubuntu10.2 amd64 QEMU utilities
2. Which storage type did you use?
Ceph
dpkg -l | egrep 'ceph'
ii ceph-common 10.2.2-1xenial amd64 common utilities to mount and interact with a ceph storage cluster
ii libcephfs1 10.2.2-1xenial amd64 Ceph distributed file system client library
ii python-cephfs 10.2.2-1xenial amd64 Python libraries for the Ceph libcephfs library
3. Which networking type did you use?
Neutron with OpenVSwitch
# On the api node
dpkg -l | grep neutron
ii neutron-common 2:8.1.0-0ubuntu0.16.04.2 all Neutron is a virtual network service for Openstack - common
ii neutron-plugin-ml2 2:8.1.0-0ubuntu0.16.04.2 all Neutron is a virtual network service for Openstack - ML2 plugin
ii neutron-server 2:8.1.0-0ubuntu0.16.04.2 all Neutron is a virtual network service for Openstack - server
ii python-neutron 2:8.1.0-0ubuntu0.16.04.2 all Neutron is a virtual network service for Openstack - Python library
ii python-neutron-fwaas 1:8.0.0-0ubuntu1 all Firewall-as-a-Service driver for OpenStack Neutron
ii python-neutron-lib 0.0.2-2 all Neutron shared routines and utilities - Python 2.7
ii python-neutronclient 1:4.1.1-2 all client API library for Neutron - Python 2.7
# On the network node
dpkg -l | grep neutron
ii neutron-common 2:8.1.0-0ubuntu0.16.04.2 all Neutron is a virtual network service for Openstack - common
ii neutron-dhcp-agent 2:8.1.0-0ubuntu0.16.04.2 all Neutron is a virtual network service for Openstack - DHCP agent
ii neutron-l3-agent 2:8.1.0-0ubuntu0.16.04.2 all Neutron is a virtual network service for Openstack - l3 agent
ii neutron-metadata-agent 2:8.1.0-0ubuntu0.16.04.2 all Neutron is a virtual network service for Openstack - metadata agent
ii neutron-openvswitch-agent 2:8.1.0-0ubuntu0.16.04.2 all Neutron is a virtual network service for Openstack - Open vSwitch plugin agent
ii python-neutron 2:8.1.0-0ubuntu0.16.04.2 all Neutron is a virtual network service for Openstack - Python library
ii python-neutron-fwaas 1:8.0.0-0ubuntu1 all Firewall-as-a-Service driver for OpenStack Neutron
ii python-neutron-lib 0.0.2-2 all Neutron shared routines and utilities - Python 2.7
ii python-neutronclient 1:4.1.1-2 all client API library for Neutron - Python 2.7
Logs & Configs
==============
2016-07-05 15:40:32.701 99291 DEBUG oslo_concurrency.lockutils [req-41fc2f7f-8c45-4c2d-ba7b-2144fcbfd757 8e06675285464030a9fd7badf4b0939cba85e018ab8a20ea8cb911222fef7771 04e7fe0cfb19418a9ec2eacfe1d334d5 - - -] Releasing semaphore "refresh_cache-9debce4f-ee92-4982-815f-769380728951" lock /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:228
2016-07-05 15:40:32.702 99291 DEBUG oslo_concurrency.processutils [req-41fc2f7f-8c45-4c2d-ba7b-2144fcbfd757 8e06675285464030a9fd7badf4b0939cba85e018ab8a20ea8cb911222fef7771 04e7fe0cfb19418a9ec2eacfe1d334d5 - - -] Running cmd (subprocess): rm -rf /var/lib/nova/instances/9debce4f-ee92-4982-815f-769380728951_resize execute /usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:326
2016-07-05 15:40:32.728 99291 DEBUG oslo_concurrency.processutils [req-41fc2f7f-8c45-4c2d-ba7b-2144fcbfd757 8e06675285464030a9fd7badf4b0939cba85e018ab8a20ea8cb911222fef7771 04e7fe0cfb19418a9ec2eacfe1d334d5 - - -] CMD "rm -rf /var/lib/nova/instances/9debce4f-ee92-4982-815f-769380728951_resize" returned: 0 in 0.026s execute /usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:356
2016-07-05 15:40:32.741 99291 DEBUG nova.virt.libvirt.storage.rbd_utils [req-41fc2f7f-8c45-4c2d-ba7b-2144fcbfd757 8e06675285464030a9fd7badf4b0939cba85e018ab8a20ea8cb911222fef7771 04e7fe0cfb19418a9ec2eacfe1d334d5 - - -] rbd image 9debce4f-ee92-4982-815f-769380728951_disk does not exist __init__ /usr/lib/python2.7/dist-packages/nova/virt/libvirt/storage/rbd_utils.py:64
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [req-41fc2f7f-8c45-4c2d-ba7b-2144fcbfd757 8e06675285464030a9fd7badf4b0939cba85e018ab8a20ea8cb911222fef7771 04e7fe0cfb19418a9ec2eacfe1d334d5 - - -] [instance: 9debce4f-ee92-4982-815f-769380728951] Setting instance vm_state to ERROR
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] Traceback (most recent call last):
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6588, in _error_out_instance_on_exception
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] yield
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3531, in _confirm_resize
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] network_info)
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 7484, in confirm_migration
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] self._cleanup_resize(instance, network_info)
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1258, in _cleanup_resize
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] ignore_errors=True)
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 941, in remove_snap
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] return self.driver.remove_snap(self.rbd_name, name, ignore_errors)
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/storage/rbd_utils.py", line 396, in remove_snap
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] with RBDVolumeProxy(self, str(volume), pool=pool) as vol:
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/storage/rbd_utils.py", line 65, in __init__
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] driver._disconnect_from_rados(client, ioctx)
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] self.force_reraise()
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] six.reraise(self.type_, self.value, self.tb)
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/storage/rbd_utils.py", line 61, in __init__
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] read_only=read_only)
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] File "rbd.pyx", line 1042, in rbd.Image.__init__ (/tmp/buildd/ceph-10.2.2/src/build/rbd.c:9862)
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951] ImageNotFound: error opening image 9debce4f-ee92-4982-815f-769380728951_disk at snapshot None
2016-07-05 15:40:32.744 99291 ERROR nova.compute.manager [instance: 9debce4f-ee92-4982-815f-769380728951]
2016-07-05 15:40:32.811 99291 DEBUG oslo_concurrency.lockutils [req-41fc2f7f-8c45-4c2d-ba7b-2144fcbfd757 8e06675285464030a9fd7badf4b0939cba85e018ab8a20ea8cb911222fef7771 04e7fe0cfb19418a9ec2eacfe1d334d5 - - -] Lock "9debce4f-ee92-4982-815f-769380728951" released by "nova.compute.manager.do_confirm_resize" :: held 0.507s inner /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:285
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher [req-41fc2f7f-8c45-4c2d-ba7b-2144fcbfd757 8e06675285464030a9fd7badf4b0939cba85e018ab8a20ea8cb911222fef7771 04e7fe0cfb19418a9ec2eacfe1d334d5 - - -] Exception during message handling: error opening image 9debce4f-ee92-4982-815f-769380728951_disk at snapshot None
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher incoming.message))
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 110, in wrapped
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher payload)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher self.force_reraise()
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 89, in wrapped
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher return f(self, context, *args, **kw)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 409, in decorated_function
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 387, in decorated_function
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info())
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher self.force_reraise()
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 375, in decorated_function
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3507, in confirm_resize
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher do_confirm_resize(context, instance, migration.id)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 274, in inner
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher return f(*args, **kwargs)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3505, in do_confirm_resize
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher migration=migration)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 3531, in _confirm_resize
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher network_info)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 7484, in confirm_migration
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher self._cleanup_resize(instance, network_info)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 1258, in _cleanup_resize
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher ignore_errors=True)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/imagebackend.py", line 941, in remove_snap
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher return self.driver.remove_snap(self.rbd_name, name, ignore_errors)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/storage/rbd_utils.py", line 396, in remove_snap
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher with RBDVolumeProxy(self, str(volume), pool=pool) as vol:
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/storage/rbd_utils.py", line 65, in __init__
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher driver._disconnect_from_rados(client, ioctx)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher self.force_reraise()
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/storage/rbd_utils.py", line 61, in __init__
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher read_only=read_only)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher File "rbd.pyx", line 1042, in rbd.Image.__init__ (/tmp/buildd/ceph-10.2.2/src/build/rbd.c:9862)
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher ImageNotFound: error opening image 9debce4f-ee92-4982-815f-769380728951_disk at snapshot None
2016-07-05 15:40:32.839 99291 ERROR oslo_messaging.rpc.dispatcher
openstack server show 9debce4f-ee92-4982-815f-769380728951
+--------------------------------------+------------------------------------------------------------------+
| Field | Value |
+--------------------------------------+------------------------------------------------------------------+
| OS-DCF:diskConfig | AUTO |
| OS-EXT-AZ:availability_zone | lab |
| OS-EXT-SRV-ATTR:host | h12 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | h12.example.com |
| OS-EXT-SRV-ATTR:instance_name | instance-00000150 |
| OS-EXT-STS:power_state | 4 |
| OS-EXT-STS:task_state | None |
| OS-EXT-STS:vm_state | stopped |
| OS-SRV-USG:launched_at | 2016-07-05T06:40:11.000000 |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | 1st-network=172.16.0.12 |
| config_drive | |
| created | 2016-07-05T03:32:37Z |
| flavor | m1.large (4) |
| hostId | 90f85ee8c380c68c55b169ff1dc9917729638d0e744020641ace2700 |
| id | 9debce4f-ee92-4982-815f-769380728951 |
| image | |
| key_name | desktop |
| name | e1 |
| os-extended-volumes:volumes_attached | [{u'id': u'90ea9f31-57b4-4162-8986-47065b23f407'}] |
| project_id | 04e7fe0cfb19418a9ec2eacfe1d334d5 |
| properties | |
| security_groups | [{u'name': u'default'}] |
| status | SHUTOFF |
| updated | 2016-07-05T07:04:11Z |
| user_id | 8e06675285464030a9fd7badf4b0939cba85e018ab8a20ea8cb911222fef7771 |
+--------------------------------------+------------------------------------------------------------------+
openstack volume show 90ea9f31-57b4-4162-8986-47065b23f407
+--------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+--------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| attachments | [{u'server_id': u'9debce4f-ee92-4982-815f-769380728951', u'attachment_id': u'd119c454-65f4-4c7b-8135-3238a3e7a424', u'attached_at': u'2016-07-05T03:33:37.127003', u'host_name': None, u'volume_id': u'90ea9f31-57b4-4162-8986-47065b23f407', u'device': u'/dev/vda', u'id': u'90ea9f31-57b4-4162-8986-47065b23f407'}] |
| availability_zone | nova |
| bootable | true |
| consistencygroup_id | None |
| created_at | 2016-07-05T03:32:38.353914 |
| description | |
| encrypted | False |
| id | 90ea9f31-57b4-4162-8986-47065b23f407 |
| migration_status | None |
| multiattach | False |
| name | |
| os-vol-host-attr:host | mitaka-cinder@ceph#ceph |
| os-vol-mig-status-attr:migstat | None |
| os-vol-mig-status-attr:name_id | None |
| os-vol-tenant-attr:tenant_id | 04e7fe0cfb19418a9ec2eacfe1d334d5 |
| properties | attached_mode='rw', readonly='False' |
| replication_status | disabled |
| size | 40 |
| snapshot_id | None |
| source_volid | None |
| status | in-use |
| type | ceph |
| updated_at | 2016-07-05T03:33:37.161179 |
| user_id | 8e06675285464030a9fd7badf4b0939cba85e018ab8a20ea8cb911222fef7771 |
| volume_image_metadata | {u'description': u'', u'os_require_quiesce': u'yes', u'checksum': u'cb3d84ccc990821e537fbe5227b3dbc3', u'hw_qemu_guest_agent': u'yes', u'disk_format': u'raw', u'image_name': u'os2-2016062303', u'hw_scsi_model': u'virtio-scsi', u'image_id': u'083aadf6-6a4c-45aa-aab6-66b6d2b2c2ba', u'min_ram': u'0', u'container_format': u'bare', u'min_disk': u'3', u'hw_disk_bus': u'scsi', u'size': u'2705326080'} |
+--------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
# On the Ceph cluster
⟫ sudo rbd ls openstack-volume | grep 9debce4f-ee92-4982-815f-769380728951_disk
1 ⟫ sudo rbd ls openstack-volume | grep 90ea9f31-57b4-4162-8986-47065b23f407
volume-90ea9f31-57b4-4162-8986-47065b23f407
⟫ sudo rbd info -p openstack-volume volume-90ea9f31-57b4-4162-8986-47065b23f407
rbd image 'volume-90ea9f31-57b4-4162-8986-47065b23f407':
size 40960 MB in 10240 objects
order 22 (4096 kB objects)
block_name_prefix: rbd_data.b412b0238e1f29
format: 2
features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
flags:
(The volume size has not been changed.)
As you can see above, nova-compute tries to do something on the 9debce4f-ee92-4982-815f-769380728951_disk which doesn't actually exist, recognizing it as an ephemeral disk. |
|
| 2016-07-08 07:57:38 |
Xuanzhou Perry Dong |
nova: assignee |
|
Xuanzhou Perry Dong (oss-xzdong) |
|
| 2016-07-08 08:19:21 |
Xuanzhou Perry Dong |
nova: assignee |
Xuanzhou Perry Dong (oss-xzdong) |
|
|
| 2016-07-14 17:22:20 |
John Garbutt |
tags |
|
ceph |
|
| 2016-07-14 17:22:29 |
John Garbutt |
summary |
[Mitaka] Fails to resize the instance booted by creating a volume, recognizing it as an ephemeral disk |
[Mitaka] Fails to resize the instance booted by creating a volume, recognizing it as an ephemeral disk when using ceph |
|
| 2016-07-14 17:23:57 |
John Garbutt |
tags |
ceph |
ceph libvirt |
|
| 2016-07-15 14:54:09 |
John Garbutt |
marked as duplicate |
|
1549194 |
|
| 2016-09-29 15:11:07 |
Mattia Belluco |
bug |
|
|
added subscriber Mattia Belluco |
