Instance creation fails with NovaException: Error mounting with libguestfs

Bug #1288352 reported by Yuriy Yekovenko
18
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
High
Unassigned
4.1.x
Fix Committed
High
Unassigned
5.0.x
Fix Committed
High
Unassigned

Bug Description

The issue was discovered with the following steps:

1. Deploy Fuel 4.0.
2. Clone tempest (Havana stable branch) to Controller node.
3. Configure tempest.
4. Run
nosetests -sv tempest/api/compute/servers/test_create_server.py

This script creates instances with code:
cls.create_server(name=cls.name,
                                     meta=cls.meta,
                                     accessIPv4=cls.accessIPv4,
                                     accessIPv6=cls.accessIPv6,
                                     personality=personality,
                                     disk_config=cls.disk_config)

This code sends request with body:
{"server": {"name": "server82800933", "imageRef": "2c97f855-a19d-4c08-8f6d-7b74df66befe", "flavorRef": 1, "OS-DCF:diskConfig": "AUTO", "accessIPv4": "1.1.1.1", "accessIPv6": "0000:0000:0000:0000:0000:babe:220.12.22.2", "metadata": {"hello": "world"}, "personality": [{"path": "/test.txt", "contents": "VGhpcyBpcyBhIHRlc3QgZmlsZS4="}]}}

Result: instance in ERROR status.
/var/log/nova-all.log contains:

<0>Mar 5 15:29:07 node-13 ¿<182>nova-nova.virt.libvirt.driver INFO: Creating image
<0>Mar 5 15:29:08 node-13 ¿<182>nova-nova.virt.libvirt.driver INFO: Injecting metadata into image 2c97f855-a19d-4c08-8f6d-7b74df66befe
<0>Mar 5 15:29:08 node-13 ¿<182>nova-nova.virt.libvirt.driver INFO: Injecting files into image 2c97f855-a19d-4c08-8f6d-7b74df66befe
<0>Mar 5 15:29:08 node-13 ¿<179>nova-nova.virt.libvirt.driver ERROR: Error injecting data into image 2c97f855-a19d-4c08-8f6d-7b74df66befe (Error mounting /var/lib/nova/instances/d5069cc1-f144-4493-8ee6-9abe32c6dd44/disk with libguestfs (command failed: LC_ALL=C '/usr/libexec/qemu-kvm' -nographic -help
errno: File exists

If qemu is located on a non-standard path, try setting the LIBGUESTFS_QEMU
environment variable. There may also be errors printed above.))
<0>Mar 5 15:29:08 node-13 ¿<179>nova-nova.compute.manager ERROR: Instance failed to spawn
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1417, in _spawn
    block_device_info)
  File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 2067, in spawn
    admin_pass=admin_password)
  File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 2500, in _create_image
    instance=instance)
  File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 2494, in _create_image
    mandatory=('files',))
  File "/usr/lib/python2.6/site-packages/nova/virt/disk/api.py", line 325, in inject_data
    fs.setup()
  File "/usr/lib/python2.6/site-packages/nova/virt/disk/vfs/guestfs.py", line 129, in setup
    {'imgfile': self.imgfile, 'e': e})
NovaException: Error mounting /var/lib/nova/instances/d5069cc1-f144-4493-8ee6-9abe32c6dd44/disk with libguestfs (command failed: LC_ALL=C '/usr/libexec/qemu-kvm' -nographic -help
errno: File exists

If qemu is located on a non-standard path, try setting the LIBGUESTFS_QEMU
environment variable. There may also be errors printed above.)

The following steps helped to fix this issue:

1. qemu-kvm is located in /usr/bin folder but libguestfs expect it in /usr/libexec/ folder.
So, on each Compute node do this :
ln -s /usr/bin/qemu-kvm /usr/libexec/qemu-kvm

2. Apply fix for bug https://bugzilla.redhat.com/show_bug.cgi?id=894241#c11 :
On each Compute node, add these to the supermin hostfiles (/usr/lib64/guestfs/supermin.d/hostfiles):
/usr/share/augeas/lenses/dist/quote.aug
/usr/share/augeas/lenses/dist/simplevars.aug

Changed in fuel:
status: New → Triaged
assignee: nobody → Fuel OSCI Team (fuel-osci)
importance: Undecided → High
milestone: none → 4.1.1
tags: added: release-notes
Revision history for this message
Vitaly Parakhin (vparakhin) wrote :

Vladimir,

Regarding the issues :

1) we're going to add creation of /usr/libexec/qemu-kvm symlink to the qemu-kvm rpm package
2) current augeas-libs version (0.9.0-4.el6) does not contain the "lenses" entries specified above, so we're going to replace it with current version (1.0.0-5.el6) from upstream
Also, in order to include these descriptions to the supermin hostfiles, we will build our own libguestfs package from CentOS 6.4 source (thus preserving the version).

Please let us know if that sounds good to you.

Roman Vyalov (r0mikiam)
Changed in fuel:
status: Triaged → In Progress
Revision history for this message
Vitaly Parakhin (vparakhin) wrote :
Roman Vyalov (r0mikiam)
Changed in fuel:
assignee: Fuel OSCI Team (fuel-osci) → Vladimir Kuklin (vkuklin)
Revision history for this message
Vladimir Kuklin (vkuklin) wrote :

please, create also backports for stable/4.1

Changed in fuel:
milestone: 4.1.1 → 5.0
tags: added: backports-4.1.1
Changed in fuel:
assignee: Vladimir Kuklin (vkuklin) → Fuel OSCI Team (fuel-osci)
Revision history for this message
Vitaly Parakhin (vparakhin) wrote :
Changed in fuel:
assignee: Fuel OSCI Team (fuel-osci) → Vladimir Kuklin (vkuklin)
Roman Vyalov (r0mikiam)
Changed in fuel:
assignee: Vladimir Kuklin (vkuklin) → Fuel QA Team (fuel-qa)
Changed in fuel:
status: In Progress → Fix Committed
Changed in fuel:
status: Fix Committed → In Progress
assignee: Fuel QA Team (fuel-qa) → Fuel OSCI Team (fuel-osci)
Roman Vyalov (r0mikiam)
Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
Meg McRoberts (dreidellhasa) wrote :

I need clarification about what to include in Release Notes.

Colin Watson (cjwatson)
Changed in fuel:
assignee: Registry Administrators (registry) → nobody
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.