Fuel for OpenStack

[library] File injection during instance launching doesn't work on Ubuntu

  1. Series 5.1.x
  2. Bug #1335697
Bug #1335697 reported by Alexander Rubtsov
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
High
Stanislav Makar
Fuel for OpenStack 6.0
5.0.x
Won't Fix
Medium
Fuel QA Team
Fuel for OpenStack 5.0.2
5.1.x
Fix Committed
High
Stanislav Makar
Fuel for OpenStack 5.1.1-updates

Bug Description

Release: Mirantis OpenStack 5.0
OS: Ubuntu
Networking: Nova-network (FlatDHCP)
Environment: KVM, 1 controller, 1 compute

Steps to reproduce:
from controller node:
echo "Hello World!" > ./test.txt
nova boot --image <image_ID> --flavor 1 --nic net-id=<network_ID> --file /root/test.txt=./test.txt instance-test

On compute node in /var/log/nova-all.log:
Jun 28 00:29:47 node-8 nova-nova.compute.manager ERROR: Instance failed to spawn
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1720, in _spawn
block_device_info)
File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2244, in spawn
admin_pass=admin_password)
File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2697, in _create_image
instance, network_info, admin_pass, files, suffix)
File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2537, in _inject_data
instance=instance)
File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 68, in __exit__
six.reraise(self.type_, self.value, self.tb)
File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 2531, in _inject_data
mandatory=('files',))
File "/usr/lib/python2.7/dist-packages/nova/virt/disk/api.py", line 353, in inject_data
fs.setup()
File "/usr/lib/python2.7/dist-packages/nova/virt/disk/vfs/guestfs.py", line 132, in setup
{'imgfile': self.imgfile, 'e': e})
NovaException: Error mounting /var/lib/nova/instances/dab50d4d-a884-4b1c-9db4-ca3c8e323f22/disk with libguestfs (external command failed, see earlier error messages)

Then we tried to create an instance without injection and it was launched successfully.
If we go on compute node and try to execute the following commands:
mkdir ./tmp
guestmount -r -a /var/lib/nova/instances/8180ff35-e371-4c69-a357-168e43e49970/disk -i /root/tmp/
We get the same error

Then we execute on the compute node:
update-guestfs-appliance
After that we can mount guest fs and we can launch an instance with injection

Nastya Urlapova (aurlapova)
Changed in fuel:
importance: Undecided → High
assignee: nobody → Fuel Library Team (fuel-library)
milestone: none → 5.0.1
Dmitry Borodaenko (angdraug)
Changed in fuel:
milestone: 5.0.1 → 5.1
Revision history for this message
Dmitry Borodaenko (angdraug) wrote :

update-guestfs-appliance was supposed to be run from postinst script of libguestfs0 deb package:

case "$1" in
    configure|reconfigure)
        . /usr/share/debconf/confmodule
        db_get libguestfs/update-appliance
        if [ "$RET" = true ]; then
            update-guestfs-appliance >&2 || true
        fi
    ;;

The libguestfs/update-appliance is set to true by puppet here:
https://github.com/stackforge/fuel-library/blob/master/deployment/puppet/nova/files/guestfs.seed
https://github.com/stackforge/fuel-library/blob/master/deployment/puppet/nova/manifests/utilities.pp

Revision history for this message
Dmitry Borodaenko (angdraug) wrote :

Please provide a diagnostic snapshot. We need to figure out why update-guestfs-appliance wasn't run during deployment, or why it ran and failed.

Changed in fuel:
status: New → Incomplete
Revision history for this message
Bogdan Dobrelya (bogdando) wrote :

could be related https://bugs.launchpad.net/fuel/+bug/1337314

Bogdan Dobrelya (bogdando)
tags: added: to-be-covered-by-tests
Dmitry Ilyin (idv1985)
summary: - File injection during instance launching doesn't work
+ [library] File injection during instance launching doesn't work
Revision history for this message
Dmitry Borodaenko (angdraug) wrote : Re: [library] File injection during instance launching doesn't work

This bug was Incomplete for over a month, lets add test coverage for this use case and close it.

Changed in fuel:
assignee: Fuel Library Team (fuel-library) → Fuel QA Team (fuel-qa)
status: Incomplete → Triaged
importance: High → Medium
Dmitry Borodaenko (angdraug)
Changed in fuel:
milestone: 5.1 → 6.0
Irina Povolotskaya (ipovolotskaya)
tags: added: release-notes
Andrey Sledzinskiy (asledzinskiy)
Changed in fuel:
assignee: Fuel QA Team (fuel-qa) → Andrey Sledzinskiy (asledzinskiy)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-main (master)

Fix proposed to branch: master
Review: https://review.openstack.org/124325

Changed in fuel:
status: Triaged → In Progress
Revision history for this message
Andrey Sledzinskiy (asledzinskiy) wrote : Re: [library] File injection during instance launching doesn't work

Seems that issue is still reproducable
It was reproduced on cluster with CentOS, Flat nova-network, Simple, 1 controller, 1 compute
Instance creation with attached file failed
Logs are attached

Revision history for this message
Andrey Sledzinskiy (asledzinskiy) wrote :
Revision history for this message
Andrey Sledzinskiy (asledzinskiy) wrote :

Also I tried to reproduce it on
{

    "build_id": "2014-09-26_00-01-46",
    "ostf_sha": "32cf2541ab8c8aa82f1e7b1d83a74e570ca2b7c9",
    "build_number": "529",
    "auth_required": true,
    "api": "1.0",
    "nailgun_sha": "9c10fc7ffd4c59d940812c580e32f4304d6ae267",
    "production": "docker",
    "fuelmain_sha": "a7f7546e157d0fc80e9a5f75aadda125648e0e64",
    "astute_sha": "a3e5da62af91b99f958ab958161d3dcec09c657b",
    "feature_groups": [
        "mirantis"
    ],
    "release": "6.0",
    "release_versions": {
        "2014.2-6.0": {
            "VERSION": {
                "build_id": "2014-09-26_00-01-46",
                "ostf_sha": "32cf2541ab8c8aa82f1e7b1d83a74e570ca2b7c9",
                "build_number": "529",
                "api": "1.0",
                "nailgun_sha": "9c10fc7ffd4c59d940812c580e32f4304d6ae267",
                "production": "docker",
                "fuelmain_sha": "a7f7546e157d0fc80e9a5f75aadda125648e0e64",
                "astute_sha": "a3e5da62af91b99f958ab958161d3dcec09c657b",
                "feature_groups": [
                    "mirantis"
                ],
                "release": "6.0",
                "fuellib_sha": "11c7260a3f91ac3c9e3327b9b21be32f6ade79e0"
            }
        }
    },
    "fuellib_sha": "11c7260a3f91ac3c9e3327b9b21be32f6ade79e0"

}

Cluster with Ubuntu, Flat nova-network, Simple, 1 controller, 1 compute
Instance was created but there was no /root/test.txt file

Logs are attached

Revision history for this message
Andrey Sledzinskiy (asledzinskiy) wrote :
Changed in fuel:
assignee: Andrey Sledzinskiy (asledzinskiy) → Fuel Library Team (fuel-library)
OpenStack Infra (hudson-openstack)
Changed in fuel:
assignee: Fuel Library Team (fuel-library) → Andrey Sledzinskiy (asledzinskiy)
Andrey Sledzinskiy (asledzinskiy)
Changed in fuel:
status: In Progress → Confirmed
assignee: Andrey Sledzinskiy (asledzinskiy) → Fuel Library Team (fuel-library)
Revision history for this message
Andrey Sledzinskiy (asledzinskiy) wrote :

Re-check it on latest master
Issue is reproduced on CentOS - instance isn't created with file injection (logs are attached)
On Ubuntu instance is created but there is no /root/test.txt file on instance
{

    "build_id": "2014-09-30_00-01-46",
    "ostf_sha": "727cd3cc93e255635fef13c80baee649ced075be",
    "build_number": "537",
    "auth_required": true,
    "api": "1.0",
    "nailgun_sha": "618d45b0cceef67b4da616167cab3161d55e9b52",
    "production": "docker",
    "fuelmain_sha": "9108f78577c7139353ace42f256e79d19564c98b",
    "astute_sha": "a3e5da62af91b99f958ab958161d3dcec09c657b",
    "feature_groups": [
        "mirantis"
    ],
    "release": "6.0",
    "release_versions": {
        "2014.2-6.0": {
            "VERSION": {
                "build_id": "2014-09-30_00-01-46",
                "ostf_sha": "727cd3cc93e255635fef13c80baee649ced075be",
                "build_number": "537",
                "api": "1.0",
                "nailgun_sha": "618d45b0cceef67b4da616167cab3161d55e9b52",
                "production": "docker",
                "fuelmain_sha": "9108f78577c7139353ace42f256e79d19564c98b",
                "astute_sha": "a3e5da62af91b99f958ab958161d3dcec09c657b",
                "feature_groups": [
                    "mirantis"
                ],
                "release": "6.0",
                "fuellib_sha": "73f2e3ca9dca9809de31471e15b58c634a81254b"
            }
        }
    },
    "fuellib_sha": "73f2e3ca9dca9809de31471e15b58c634a81254b"

}

Revision history for this message
Andrey Sledzinskiy (asledzinskiy) wrote :
Bogdan Dobrelya (bogdando)
Changed in fuel:
status: Confirmed → Triaged
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to fuel-main (master)

Reviewed: https://review.openstack.org/124325
Committed: https://git.openstack.org/cgit/stackforge/fuel-main/commit/?id=53e71f7169f8c76d18ceb42477e80a39889b29b0
Submitter: Jenkins
Branch: master

commit 53e71f7169f8c76d18ceb42477e80a39889b29b0
Author: asledzinskiy <email address hidden>
Date: Fri Sep 26 11:05:08 2014 +0300

    Add test on creating instance with
    file injection

    Change-Id: I92dfe1da854dfc662f1b8b77e85e1e295b5790e6
    Related-Bug: #1335697

Vladimir Kuklin (vkuklin)
Changed in fuel:
importance: Medium → High
Stanislav Makar (smakar)
Changed in fuel:
assignee: Fuel Library Team (fuel-library) → Stanislav Makar (smakar)
Revision history for this message
Stanislav Makar (smakar) wrote : Re: [library] File injection during instance launching doesn't work

VMs are spawning without problem, but the feature file injection doesnt wok
I got such string from logs:
<182>Nov 24 12:23:15 node-4 nova-compute Unable to import guestfsfalling back to VFSLocalFS

Stanislav Makar (smakar)
Changed in fuel:
status: Triaged → In Progress
Revision history for this message
Stanislav Makar (smakar) wrote :

Below some logs that are connected with it and are presented during vm creation in /var/log/syslog:

<3>Nov 24 11:27:08 node-4 kernel: [ 1370.718014] EXT4-fs (nbd5): VFS: Can't find ext4 filesystem
<3>Nov 24 11:27:08 node-4 kernel: [ 1370.721165] FAT-fs (nbd5): bogus number of reserved sectors
<1>Nov 24 11:27:08 node-4 kernel: [ 1370.724521] ffff88005e503000: eb 48 90 00 00 00 00 00 00 00 00 00 00 00 00 00 .H..............
<1>Nov 24 11:27:08 node-4 kernel: [ 1370.728925] ffff88005e503010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
<1>Nov 24 11:27:08 node-4 kernel: [ 1370.736931] ffff88005e503020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
<1>Nov 24 11:27:08 node-4 kernel: [ 1370.743751] ffff88005e503030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 02 ................
<1>Nov 24 11:27:08 node-4 kernel: [ 1370.749295] XFS (nbd5): Internal error xfs_sb_read_verify at line 789 of file /build/buildd/linux-lts-saucy-3.11.0/fs/xfs/xfs_mount.c. Caller 0xffffffffa000b495
<1>Nov 24 11:27:08 node-4 kernel: [ 1370.749295]
<1>Nov 24 11:27:08 node-4 kernel: [ 1370.759078] XFS (nbd5): Corruption detected. Unmount and run xfs_repair
<3>Nov 24 11:27:09 node-4 kernel: [ 1370.802595] block nbd5: Unexpected reply (ffff88005e943c08)
<27>Nov 24 11:27:09 node-4 ntpd[18331]: bind(36) AF_INET6 fe80::d41a:1fff:feea:98be%13#123 flags 0x11 failed: Cannot assign requested address
<27>Nov 24 11:27:09 node-4 ntpd[18331]: unable to create socket on br100 (19) for fe80::d41a:1fff:feea:98be#123
<27>Nov 24 11:27:11 node-4 dnsmasq[21085]: failed to load names from /var/lib/nova/networks/nova-br100.hosts: No such file or directory
<3>Nov 24 12:23:15 node-4 kernel: [ 4737.665315] EXT4-fs (nbd11): VFS: Can't find ext4 filesystem
<3>Nov 24 12:23:15 node-4 kernel: [ 4737.669327] FAT-fs (nbd11): bogus number of reserved sectors
<1>Nov 24 12:23:15 node-4 kernel: [ 4737.672965] ffff88005e840000: eb 48 90 00 00 00 00 00 00 00 00 00 00 00 00 00 .H..............
<1>Nov 24 12:23:15 node-4 kernel: [ 4737.677691] ffff88005e840010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
<1>Nov 24 12:23:15 node-4 kernel: [ 4737.684460] ffff88005e840020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
<1>Nov 24 12:23:15 node-4 kernel: [ 4737.702385] ffff88005e840030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 03 02 ................
<1>Nov 24 12:23:15 node-4 kernel: [ 4737.707152] XFS (nbd11): Internal error xfs_sb_read_verify at line 789 of file /build/buildd/linux-lts-saucy-3.11.0/fs/xfs/xfs_mount.c. Caller 0xffffffffa000b495
<1>Nov 24 12:23:15 node-4 kernel: [ 4737.707152]
<1>Nov 24 12:23:15 node-4 kernel: [ 4737.716063] XFS (nbd11): Corruption detected. Unmount and run xfs_repair
<3>Nov 24 12:23:16 node-4 kernel: [ 4737.770959] block nbd11: Unexpected reply (ffff88005e8e5c08)

and in dmesg as well.
Looks like nova uses NBD instead of guestfs.

Revision history for this message
Stanislav Makar (smakar) wrote :

- Packages are absent in ubuntu repo:
   python-libguestfs
   libguestfs-tools
- Packeges are not installed
   https://github.com/stackforge/fuel-library/blob/master/deployment/puppet/nova/manifests/utilities.pp
   https://github.com/stackforge/fuel-library/blob/master/deployment/puppet/nova/manifests/init.pp lines 284 and 382
   https://github.com/stackforge/fuel-library/blob/master/deployment/puppet/openstack/manifests/compute.pp line 241

That's why NBD is using instead of libguestfs.

Revision history for this message
Stanislav Makar (smakar) wrote :

I have installed the version 1.14 from original Ubuntu 12.04 repo and
I have caught this
WARNING nova.virt.disk.vfs.guestfs [req-7c7ca124-e736-4edf-9d6c-c1fc015f32bc None] Unable to force
 TCG mode, libguestfs too old? GuestFS instance has no attribute 'set_backend_settings'

It would be great to have 1.26 or 1.28 of below packages:
python-libguestfs
libguestfs-tools
libguestfs0

Changed in fuel:
assignee: Stanislav Makar (smakar) → Fuel OSCI Team (fuel-osci)
Revision history for this message
Stanislav Makar (smakar) wrote :

Sorry
python-guestfs

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-main (master)

Fix proposed to branch: master
Review: https://review.openstack.org/137602

Changed in fuel:
assignee: Fuel OSCI Team (fuel-osci) → Stanislav Makar (smakar)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/137606

Stanislav Makar (smakar)
Changed in fuel:
assignee: Stanislav Makar (smakar) → MOS Linux (mos-linux)
Stanislav Makar (smakar)
Changed in fuel:
assignee: MOS Linux (mos-linux) → Stanislav Makar (smakar)
Revision history for this message
Stanislav Makar (smakar) wrote : Re: [library] File injection during instance launching doesn't work

depends on https://bugs.launchpad.net/fuel/+bug/1397014

Revision history for this message
Stanislav Makar (smakar) wrote :

upgraded to 1.18 - doesn't work , the same
propose to enable this feature and use different approaches for different OS:
     CentOS - use libguestfs
     Ubuntu - use NBD due to the problems with libguestfs

Note:
On CentOS python-guestfs is installed as dependency of package openstack-nova-compute, so it isn't necessary to install, just enable by the option inject_partition=-1.
On Ubuntu - there isn't so, then we don't install it manually and set inject_partition=1. If we dont install it, nova-compute uses nbd.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/5.1)

Fix proposed to branch: stable/5.1
Review: https://review.openstack.org/138286

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/137606
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=5cc403a167ecd2d179a309d1003151e7a55567cb
Submitter: Jenkins
Branch: master

commit 5cc403a167ecd2d179a309d1003151e7a55567cb
Author: Stanislav Makar <email address hidden>
Date: Thu Nov 27 11:15:51 2014 +0000

    Enable the server personality feature

    http://docs.openstack.org/api/openstack-compute/2/content/Server_Personality-d1e2543.html
    -Use different approaches for different OS:
     CentOS - use libguestfs
     Ubuntu - use NBD due to the problems with libguestfs

    Change-Id: Id2139b415af3f8835a6d4e9ee90e81121caf7c95
    Closes-bug: #1335697

Changed in fuel:
status: In Progress → Fix Committed
Vladimir Kuklin (vkuklin)
summary: - [library] File injection during instance launching doesn't work
+ [library] File injection during instance launching doesn't work on
+ Ubuntu
Revision history for this message
Anastasia Palkina (apalkina) wrote :

Verified on ISO #49

"build_id": "2014-12-09_22-41-06", "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4", "build_number": "49", "auth_required": true, "api": "1.0", "nailgun_sha": "22bd43b89a17843f9199f92d61fc86cb0f8772f1", "production": "docker", "fuelmain_sha": "3aab16667f47dd8384904e27f70f7a87ba15f4ee", "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91", "feature_groups": ["mirantis"], "release": "6.0", "release_versions": {"2014.2-6.0": {"VERSION": {"build_id": "2014-12-09_22-41-06", "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4", "build_number": "49", "api": "1.0", "nailgun_sha": "22bd43b89a17843f9199f92d61fc86cb0f8772f1", "production": "docker", "fuelmain_sha": "3aab16667f47dd8384904e27f70f7a87ba15f4ee", "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91", "feature_groups": ["mirantis"], "release": "6.0", "fuellib_sha": "2c99931072d951301d395ebd5bf45c8d401301bb"}}}, "fuellib_sha": "2c99931072d951301d395ebd5bf45c8d401301bb"}

Revision history for this message
Anastasia Palkina (apalkina) wrote :

Verified for CentOS and Ubuntu on ISO #49

Changed in fuel:
status: Fix Committed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/5.1)

Reviewed: https://review.openstack.org/138286
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=9e50ab7a055d13362c00a10536e83d74a1565c6d
Submitter: Jenkins
Branch: stable/5.1

commit 9e50ab7a055d13362c00a10536e83d74a1565c6d
Author: Stanislav Makar <email address hidden>
Date: Thu Nov 27 11:15:51 2014 +0000

    Enable the server personality feature

    http://docs.openstack.org/api/openstack-compute/2/content/Server_Personality-d1e2543.html
    -Use different approaches for different OS:
     CentOS - use libguestfs
     Ubuntu - use NBD due to the problems with libguestfs

    Change-Id: Id2139b415af3f8835a6d4e9ee90e81121caf7c95
    Closes-bug: #1335697

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on fuel-main (master)

Change abandoned by Stanislav Makar (<email address hidden>) on branch: master
Review: https://review.openstack.org/137602
Reason: not needed

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.