OpenStack Nova Compute Charm

instances-path config does not work properly

Bug #1866861 reported by Rodrigo Barbieri
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
OpenStack Nova Compute Charm
Triaged
Low
Unassigned

Bug Description

I tried changing the config "instances-path" to set path /nova/my_instances instead of the default /var/lib/nova/instances. Upon trying to create an instance, it fails to build and the stack trace below is found on nova-compute logs.

I realized that I had to create the path "/nova/my_instances" myself, as the charm doesn't create and it produces a different error. Also, I tried to address the permission denied error using "Chown -R /nova/my_instances", but still had no success. The charm actually does run chown so this shouldn't be needed.

After the error, I tried comparing it with the permissions of my default /var/lib/nova/instances, and the permission bits seemed to be the same.

Although there is a slightly similar bug 1656254, in my testing I was not using apparmor nor encryption.

2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [req-202c3fb4-a4e7-4bb3-aca3-b8c3f4c3f81a e78c033f8d974a27aeee94536d6afb1f 867f0965fdd4432f8e11517121b2ea81 - 54bdddc7db4547e1bf5d6f8651cb86fc 54bdddc7db4547e1bf5d6f8651cb86fc] [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] Instance failed to spawn: libvirtError: internal error: process exited while connecting to monitor: 2020-03-08T18:03:21.198569Z qemu-system-x86_64: -drive file=/nova/my_instances/c0b5e6f5-5936-4de5-b10d-48ac409ff434/disk,format=qcow2,if=none,id=drive-virtio-disk0,cache=none,discard=unmap: Could not open backing file: Could not open '/nova/my_instances/_base/8281f19ac8157de59a7df0860a99858960e2f8f0': Permission denied
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] Traceback (most recent call last):
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2274, in _build_resources
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] yield resources
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2054, in _build_and_run_instance
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] block_device_info=block_device_info)
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 3147, in spawn
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] destroy_disks_on_failure=True)
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5651, in _create_domain_and_network
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] destroy_disks_on_failure)
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] self.force_reraise()
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] six.reraise(self.type_, self.value, self.tb)
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5620, in _create_domain_and_network
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] post_xml_callback=post_xml_callback)
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5555, in _create_domain
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] guest.launch(pause=pause)
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/guest.py", line 144, in launch
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] self._encoded_xml, errors='ignore')
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] self.force_reraise()
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] six.reraise(self.type_, self.value, self.tb)
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/guest.py", line 139, in launch
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] return self._domain.createWithFlags(flags)
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 186, in doit
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] result = proxy_call(self._autowrap, f, *args, **kwargs)
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 144, in proxy_call
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] rv = execute(f, *args, **kwargs)
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 125, in execute
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] six.reraise(c, e, tb)
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] File "/usr/lib/python2.7/dist-packages/eventlet/tpool.py", line 83, in tworker
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] rv = meth(*args, **kwargs)
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] File "/usr/lib/python2.7/dist-packages/libvirt.py", line 1092, in createWithFlags
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] libvirtError: internal error: process exited while connecting to monitor: 2020-03-08T18:03:21.198569Z qemu-system-x86_64: -drive file=/nova/my_instances/c0b5e6f5-5936-4de5-b10d-48ac409ff434/disk,format=qcow2,if=none,id=drive-virtio-disk0,cache=none,discard=unmap: Could not open backing file: Could not open '/nova/my_instances/_base/8281f19ac8157de59a7df0860a99858960e2f8f0': Permission denied
2020-03-08 18:03:21.924 18406 ERROR nova.compute.manager [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434]
2020-03-08 18:03:21.934 18406 INFO nova.compute.manager [req-202c3fb4-a4e7-4bb3-aca3-b8c3f4c3f81a e78c033f8d974a27aeee94536d6afb1f 867f0965fdd4432f8e11517121b2ea81 - 54bdddc7db4547e1bf5d6f8651cb86fc 54bdddc7db4547e1bf5d6f8651cb86fc] [instance: c0b5e6f5-5936-4de5-b10d-48ac409ff434] Terminating instance

Revision history for this message
Andrew McLeod (admcleod) wrote :

Thanks for your bug report - in order to replicate this we need to know which distro and if you used specific charm revisions? An example bundle may also be helpful.

Changed in charm-nova-compute:
status: New → Incomplete
Revision history for this message
Rodrigo Barbieri (rodrigo-barbieri2010) wrote :
Revision history for this message
Rodrigo Barbieri (rodrigo-barbieri2010) wrote :

Hello Andrew,

I attached the output of my juju export-bundle. Version used was bionic-queens.

Rodrigo Barbieri (rodrigo-barbieri2010)
Changed in charm-nova-compute:
status: Incomplete → New
Revision history for this message
Rodrigo Barbieri (rodrigo-barbieri2010) wrote :

I tested editing the libvirt apparmor profile /etc/apparmor.d/usr.lib.libvirt.virt-aa-helper and it did indeed address the issue. I started coding a way to have the charm update that file, however I haven't figured out yet a way for the charm to reliably maintain its values.

The file's default path is /var/lib/nova/instances. Suppose the file is edited and the value "/var/lib/nova/instances" is not longer there, but the new path. The next instances-path config editing will need to update that accordingly.

Even with Config.previous, we run the risk of the file and the config going offsync if the charm hook fails.

The file is also a bit fragile in the sense that package updates may overwrite and cause nova to stop working and a config-changed hook or some periodic validation would need to kick in to address the problem.

Another point of fragility is that since there is no tag to specific lines, if a later update changes those specific lines of the file, it will break the charm and require an update.

Aurelien Lourot (aurelien-lourot)
Changed in charm-nova-compute:
importance: Undecided → Low
status: New → Triaged
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.