nova-lxd

Failed to boot with ephemeral disk and zfs storage pool

Bug #1817934 reported by Malte Swart
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
nova-lxd
New
Undecided
Unassigned

Bug Description

I am trying to start an lxd container with both a root disk and an ephemeral disk. I use the `default` storage pool with a zfs storage driver.

Launching failed due to "LXDAPIException: Storage volumes cannot be specified as absolute paths".

I am running:
- Openstack Queens
- Ubuntu Bionic
- nova-lxd 17.0.1-0ubuntu1
- lxd 3.0.3-0ubuntu1~18.04.1
- python-pylxd 2.2.6-0ubuntu1.1

The full info log of nova-lxd (I added info log statements to flavor.to_profile to print config, device and block_info):
INFO os_vif [req-5f3143bf-21e0-4d37-985f-3d81898b62bc bccab7d86ea9436a81c3345bf2f73fc5 d33bca6fdab744348a44f231da1f81ad - default default] Successfully plugged vif VIFBridge(active=False,address=fa:16:3e:d2:e2:78,bridge_name='brqc68b2e19-64',has_traffic_filtering=True,id=800d658e-163d-42c9-bb9d-89d72ae4fb5b,network=Network(c68b2e19-6454-4b9b-8f6e-cb2c763cf2e5),plugin='linux_bridge',port_profile=<?>,preserve_on_delete=False,vif_name='tap800d658e-16')
INFO nova.virt.lxd.flavor [req-5f3143bf-21e0-4d37-985f-3d81898b62bc bccab7d86ea9436a81c3345bf2f73fc5 d33bca6fdab744348a44f231da1f81ad - default default] {'environment.product_name': 'OpenStack Nova',
 'limits.cpu': '1',
 'limits.memory': '2048MB',
 'raw.lxc': 'lxc.console.logfile=/var/log/lxd/d33bca6f-lxdb-003372/console.log\n'}
INFO nova.virt.lxd.flavor [req-5f3143bf-21e0-4d37-985f-3d81898b62bc bccab7d86ea9436a81c3345bf2f73fc5 d33bca6fdab744348a44f231da1f81ad - default default] {'ephemeral0': {'path': '/mnt',
                'pool': 'default',
                'source': u'/var/lib/nova/instances/d33bca6f-lxdb-003372/storage/ephemeral0',
                'type': 'disk'},
 'root': {'path': '/', 'pool': 'default', 'size': '5GB', 'type': 'disk'},
 u'tap800d658e-16': {'hwaddr': 'fa:16:3e:d2:e2:78',
                     'nictype': 'physical',
                     'parent': u'tin800d658e-16',
                     'type': 'nic'}}
INFO nova.virt.lxd.flavor [req-5f3143bf-21e0-4d37-985f-3d81898b62bc bccab7d86ea9436a81c3345bf2f73fc5 d33bca6fdab744348a44f231da1f81ad - default default] {'block_device_mapping': [],
 'ephemerals': [{'device_name': u'/dev/sdb',
                 'num': 0,
                 'size': 5,
                 'virtual_name': 'ephemeral0'}],
 'root_device_name': u'/dev/sda',
 'swap': None}

INFO os_vif [req-5f3143bf-21e0-4d37-985f-3d81898b62bc bccab7d86ea9436a81c3345bf2f73fc5 d33bca6fdab744348a44f231da1f81ad - default default] Successfully unplugged vif VIFBridge(active=False,address=fa:16:3e:d2:e2:78,bridge_name='brqc68b2e19-64',has_traffic_filtering=True,id=800d658e-163d-42c9-bb9d-89d72ae4fb5b,network=Network(c68b2e19-6454-4b9b-8f6e-cb2c763cf2e5),plugin='linux_bridge',port_profile=<?>,preserve_on_delete=False,vif_name='tap800d658e-16')
ERROR root [req-5f3143bf-21e0-4d37-985f-3d81898b62bc bccab7d86ea9436a81c3345bf2f73fc5 d33bca6fdab744348a44f231da1f81ad - default default] Original exception being dropped: ['Traceback (most recent call last):\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/lxd/driver.py", line 539, in spawn\n self.client, instance, network_info, block_device_info)\n', ' File "/usr/lib/python2.7/dist-packages/nova/virt/lxd/flavor.py", line 236, in to_profile\n return client.profiles.create(name, config, devices)\n', ' File "/usr/lib/python2.7/dist-packages/pylxd/models/profile.py", line 60, in create\n client.api.profiles.post(json=profile)\n', ' File "/usr/lib/python2.7/dist-packages/pylxd/client.py", line 134, in post\n self._assert_response(response, allowed_status_codes=(200, 201, 202))\n', ' File "/usr/lib/python2.7/dist-packages/pylxd/client.py", line 91, in _assert_response\n raise exceptions.LXDAPIException(response)\n', 'LXDAPIException: Storage volumes cannot be specified as absolute paths\n']: ProcessExecutionError: Unexpected error while running command.
ERROR nova.compute.manager [req-5f3143bf-21e0-4d37-985f-3d81898b62bc bccab7d86ea9436a81c3345bf2f73fc5 d33bca6fdab744348a44f231da1f81ad - default default] [instance: a6b37119-e3fd-4538-858a-75a0f8515a8e] Instance failed to spawn: ProcessExecutionError: Unexpected error while running command.
Command: sudo nova-rootwrap /etc/nova/rootwrap.conf zfs destroy default/d33bca6f-lxdb-003372-ephemeral
Exit code: 1
Stdout: u''
Stderr: u"cannot open 'default/d33bca6f-lxdb-003372-ephemeral': dataset does not exist\n"
ERROR nova.compute.manager [instance: a6b37119-e3fd-4538-858a-75a0f8515a8e] Traceback (most recent call last):
ERROR nova.compute.manager [instance: a6b37119-e3fd-4538-858a-75a0f8515a8e] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2239, in _build_resources
ERROR nova.compute.manager [instance: a6b37119-e3fd-4538-858a-75a0f8515a8e] yield resources
ERROR nova.compute.manager [instance: a6b37119-e3fd-4538-858a-75a0f8515a8e] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2019, in _build_and_run_instance
ERROR nova.compute.manager [instance: a6b37119-e3fd-4538-858a-75a0f8515a8e] block_device_info=block_device_info)
ERROR nova.compute.manager [instance: a6b37119-e3fd-4538-858a-75a0f8515a8e] File "/usr/lib/python2.7/dist-packages/nova/virt/lxd/driver.py", line 543, in spawn
ERROR nova.compute.manager [instance: a6b37119-e3fd-4538-858a-75a0f8515a8e] context, instance, network_info, block_device_info)
ERROR nova.compute.manager [instance: a6b37119-e3fd-4538-858a-75a0f8515a8e] File "/usr/lib/python2.7/dist-packages/nova/virt/lxd/driver.py", line 645, in cleanup
ERROR nova.compute.manager [instance: a6b37119-e3fd-4538-858a-75a0f8515a8e] instance)
ERROR nova.compute.manager [instance: a6b37119-e3fd-4538-858a-75a0f8515a8e] File "/usr/lib/python2.7/dist-packages/nova/virt/lxd/storage.py", line 128, in detach_ephemeral
ERROR nova.compute.manager [instance: a6b37119-e3fd-4538-858a-75a0f8515a8e] run_as_root=True)
ERROR nova.compute.manager [instance: a6b37119-e3fd-4538-858a-75a0f8515a8e] File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 230, in execute
ERROR nova.compute.manager [instance: a6b37119-e3fd-4538-858a-75a0f8515a8e] return RootwrapProcessHelper().execute(*cmd, **kwargs)
ERROR nova.compute.manager [instance: a6b37119-e3fd-4538-858a-75a0f8515a8e] File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 113, in execute
ERROR nova.compute.manager [instance: a6b37119-e3fd-4538-858a-75a0f8515a8e] return processutils.execute(*cmd, **kwargs)
ERROR nova.compute.manager [instance: a6b37119-e3fd-4538-858a-75a0f8515a8e] File "/usr/lib/python2.7/dist-packages/oslo_concurrency/processutils.py", line 424, in execute
ERROR nova.compute.manager [instance: a6b37119-e3fd-4538-858a-75a0f8515a8e] cmd=sanitized_cmd)
ERROR nova.compute.manager [instance: a6b37119-e3fd-4538-858a-75a0f8515a8e] ProcessExecutionError: Unexpected error while running command.
ERROR nova.compute.manager [instance: a6b37119-e3fd-4538-858a-75a0f8515a8e] Command: sudo nova-rootwrap /etc/nova/rootwrap.conf zfs destroy default/d33bca6f-lxdb-003372-ephemeral
ERROR nova.compute.manager [instance: a6b37119-e3fd-4538-858a-75a0f8515a8e] Exit code: 1
ERROR nova.compute.manager [instance: a6b37119-e3fd-4538-858a-75a0f8515a8e] Stdout: u''
ERROR nova.compute.manager [instance: a6b37119-e3fd-4538-858a-75a0f8515a8e] Stderr: u"cannot open 'default/d33bca6f-lxdb-003372-ephemeral': dataset does not exist\n"
ERROR nova.compute.manager [instance: a6b37119-e3fd-4538-858a-75a0f8515a8e]
INFO nova.compute.manager [req-5f3143bf-21e0-4d37-985f-3d81898b62bc bccab7d86ea9436a81c3345bf2f73fc5 d33bca6fdab744348a44f231da1f81ad - default default] [instance: a6b37119-e3fd-4538-858a-75a0f8515a8e] Terminating instance
WARNING nova.virt.lxd.driver [req-5f3143bf-21e0-4d37-985f-3d81898b62bc bccab7d86ea9436a81c3345bf2f73fc5 d33bca6fdab744348a44f231da1f81ad - default default] Failed to delete instance. Container does not exist for d33bca6f-lxdb-003372.: NotFound: not found
INFO os_vif [req-5f3143bf-21e0-4d37-985f-3d81898b62bc bccab7d86ea9436a81c3345bf2f73fc5 d33bca6fdab744348a44f231da1f81ad - default default] Successfully unplugged vif VIFBridge(active=False,address=fa:16:3e:d2:e2:78,bridge_name='brqc68b2e19-64',has_traffic_filtering=True,id=800d658e-163d-42c9-bb9d-89d72ae4fb5b,network=Network(c68b2e19-6454-4b9b-8f6e-cb2c763cf2e5),plugin='linux_bridge',port_profile=<?>,preserve_on_delete=False,vif_name='tap800d658e-16')
WARNING nova.compute.manager [req-5f3143bf-21e0-4d37-985f-3d81898b62bc bccab7d86ea9436a81c3345bf2f73fc5 d33bca6fdab744348a44f231da1f81ad - default default] Could not clean up failed build, not rescheduling. Error: Unexpected error while running command.
Command: sudo nova-rootwrap /etc/nova/rootwrap.conf zfs destroy default/d33bca6f-lxdb-003372-ephemeral
Exit code: 1
Stdout: u''
Stderr: u"cannot open 'default/d33bca6f-lxdb-003372-ephemeral': dataset does not exist\n": ProcessExecutionError: Unexpected error while running command.
ERROR nova.compute.manager [req-5f3143bf-21e0-4d37-985f-3d81898b62bc bccab7d86ea9436a81c3345bf2f73fc5 d33bca6fdab744348a44f231da1f81ad - default default] [instance: a6b37119-e3fd-4538-858a-75a0f8515a8e] Build of instance a6b37119-e3fd-4538-858a-75a0f8515a8e aborted: Unexpected error while running command.
Command: sudo nova-rootwrap /etc/nova/rootwrap.conf zfs destroy default/d33bca6f-lxdb-003372-ephemeral
Exit code: 1
Stdout: u''
Stderr: u"cannot open 'default/d33bca6f-lxdb-003372-ephemeral': dataset does not exist\n": BuildAbortException: Build of instance a6b37119-e3fd-4538-858a-75a0f8515a8e aborted: Unexpected error while running command.
Command: sudo nova-rootwrap /etc/nova/rootwrap.conf zfs destroy default/d33bca6f-lxdb-003372-ephemeral
Exit code: 1
Stdout: u''
Stderr: u"cannot open 'default/d33bca6f-lxdb-003372-ephemeral': dataset does not exist\n"
ERROR nova.compute.manager [instance: a6b37119-e3fd-4538-858a-75a0f8515a8e] Traceback (most recent call last):
ERROR nova.compute.manager [instance: a6b37119-e3fd-4538-858a-75a0f8515a8e] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1828, in _do_build_and_run_instance
ERROR nova.compute.manager [instance: a6b37119-e3fd-4538-858a-75a0f8515a8e] filter_properties, request_spec)
ERROR nova.compute.manager [instance: a6b37119-e3fd-4538-858a-75a0f8515a8e] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2050, in _build_and_run_instance
ERROR nova.compute.manager [instance: a6b37119-e3fd-4538-858a-75a0f8515a8e] bdms=block_device_mapping)
ERROR nova.compute.manager [instance: a6b37119-e3fd-4538-858a-75a0f8515a8e] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
ERROR nova.compute.manager [instance: a6b37119-e3fd-4538-858a-75a0f8515a8e] self.force_reraise()
ERROR nova.compute.manager [instance: a6b37119-e3fd-4538-858a-75a0f8515a8e] File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
ERROR nova.compute.manager [instance: a6b37119-e3fd-4538-858a-75a0f8515a8e] six.reraise(self.type_, self.value, self.tb)
ERROR nova.compute.manager [instance: a6b37119-e3fd-4538-858a-75a0f8515a8e] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2022, in _build_and_run_instance
ERROR nova.compute.manager [instance: a6b37119-e3fd-4538-858a-75a0f8515a8e] instance=instance)
ERROR nova.compute.manager [instance: a6b37119-e3fd-4538-858a-75a0f8515a8e] File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__
ERROR nova.compute.manager [instance: a6b37119-e3fd-4538-858a-75a0f8515a8e] self.gen.throw(type, value, traceback)
ERROR nova.compute.manager [instance: a6b37119-e3fd-4538-858a-75a0f8515a8e] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2266, in _build_resources
ERROR nova.compute.manager [instance: a6b37119-e3fd-4538-858a-75a0f8515a8e] reason=six.text_type(exc))
ERROR nova.compute.manager [instance: a6b37119-e3fd-4538-858a-75a0f8515a8e] BuildAbortException: Build of instance a6b37119-e3fd-4538-858a-75a0f8515a8e aborted: Unexpected error while running command.
ERROR nova.compute.manager [instance: a6b37119-e3fd-4538-858a-75a0f8515a8e] Command: sudo nova-rootwrap /etc/nova/rootwrap.conf zfs destroy default/d33bca6f-lxdb-003372-ephemeral
ERROR nova.compute.manager [instance: a6b37119-e3fd-4538-858a-75a0f8515a8e] Exit code: 1
ERROR nova.compute.manager [instance: a6b37119-e3fd-4538-858a-75a0f8515a8e] Stdout: u''
ERROR nova.compute.manager [instance: a6b37119-e3fd-4538-858a-75a0f8515a8e] Stderr: u"cannot open 'default/d33bca6f-lxdb-003372-ephemeral': dataset does not exist\n"
ERROR nova.compute.manager [instance: a6b37119-e3fd-4538-858a-75a0f8515a8e]
INFO nova.compute.manager [req-5f3143bf-21e0-4d37-985f-3d81898b62bc bccab7d86ea9436a81c3345bf2f73fc5 d33bca6fdab744348a44f231da1f81ad - default default] [instance: a6b37119-e3fd-4538-858a-75a0f8515a8e] Took 1.17 seconds to deallocate network for instance.
INFO nova.scheduler.client.report [req-5f3143bf-21e0-4d37-985f-3d81898b62bc bccab7d86ea9436a81c3345bf2f73fc5 d33bca6fdab744348a44f231da1f81ad - default default] Deleted allocation for instance a6b37119-e3fd-4538-858a-75a0f8515a8e
INFO nova.compute.resource_tracker [req-a74980a4-7a0d-4532-9963-933263e061a2 - - - - -] Final resource view: name=db3 phys_ram=193409MB used_ram=1024MB phys_disk=17GB used_disk=5GB total_vcpus=48 used_vcpus=1 pci_stats=[]

Revision history for this message
Alex Kavanagh (ajkavanagh) wrote :

Hi Malte

In order to dig into this a bit further, we'll need a bit more information:

Can you add, please:

* the the configuration of your nova.conf please
* the lxd storage (lxc storage list)
* the command you issued to create the container
* the relevant bit of /var/log/nova/nova-compute.log (from the LXD host)
* The instance-xxxx/lxc.log (if you can find it -- it's usually in /var/log/lxd/instance-xxx/lxc.log)
* How the OpenStack was deployed (charms, Ansible, etc.)

Changed in nova-lxd:
status: New → Incomplete
Revision history for this message
Malte Swart (malte.swart) wrote :

* nova.conf (except generic auth specific sections like keystone, glance, neutron, messaging):

[DEFAULT]
debug = False
enabled_apis = osapi_compute,metadata
instance_name_template = %(project_id).8s-%(hostname)s-%(id)06d
lock_path = /var/lock/nova
log_dir = /var/log/nova
my_block_storage_ip = xxxxx
my_ip = xxxx
osapi_compute_unique_server_name_scope = project
osapi_compute_workers = 2
state_path = /var/lib/nova

[oslo_concurrency]
lock_path = /var/lib/nova/tmp

[lxd]
pool = default

* lxd storage is zfs:
+---------+-------------+--------+--------------------------------+---------+
| NAME | DESCRIPTION | DRIVER | SOURCE | USED BY |
 +---------+-------------+--------+--------------------------------+---------+
| default | | zfs | /var/lib/lxd/disks/default.img | 10 |
 +---------+-------------+--------+--------------------------------+---------+

* I created this VM via horizon specifying only the basic fields (flavor, image, network, ssh key and a scheduler hint to select the lxd node). The used flavor:
 VCPU: 1
 RAM: 2GB
 root disk: 5GB
 ephemeral Disk: 5GB
 space disk: 0MB

* the nova-compute log is already included (timestamps were removed for better readability).

* an instance log was not written as nova-lxd fails to create the profile
* installed with salt

For completeness as I didn't stated it explicit: booting containers works as expected for flavors without ephemeral disk.

Malte Swart (malte.swart)
Changed in nova-lxd:
status: Incomplete → New
Revision history for this message
Felipe Alfaro Solana (felipe-alfaro-gmail) wrote :

Any progress on this? I'm hitting exactly the same problem when using BTRFS:

Create LXD profile: name: instance-00000002, config: {'environment.product_name': 'OpenStack Nova', 'raw.lxc': 'lxc.console.logfile=/var/log/lxd/instance-0000000
2/console.log\n', 'limits.memory': '1024MB', 'limits.cpu': '1'}, devices: {'root': {'type': 'disk', 'path': '/', 'size': '20GB', 'pool': 'lxd'}, 'ephemeral0': {'path': '/mnt', 'source': '/var/lib/n
ova/instances/instance-00000002/storage/ephemeral0', 'type': 'disk', 'pool': 'lxd'}, 'tap72184758-a1': {'nictype': 'physical', 'hwaddr': 'fa:16:3e:96:64:d2', 'parent': 'tin72184758-a1', 'type': 'ni
c'}}

In my case, instead of /dev/sdb my NovaLXD is trying to use a source looks like a file. I think a plausible solution is to configure Nova to use ephemeral on top of Ceph, but I don't quite like it. Any ideas?

Revision history for this message
Malte Swart (malte.swart) wrote :

We are currently run a slightly patched version of nova-lxd. The attached patch is sufficient to get containers with ephemeral disk on ZFS. A similar approach might work for brtfs, too.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.