OpenStack Compute (nova)

GuestFS fails to mount image for data injection

Bug #1275267 reported by Salvatore Orlando
26
This bug affects 4 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
Fix Released
Undecided
Sahid Orentino
OpenStack Compute (nova) 2014.2 "juno"

Bug Description

A GuestFS error is causing injection to fail. This result in a warning for metadata injection but results in a spawn error for key injection.

This is logged with debug level:

Exception AttributeError: "GuestFS instance has no attribute '_o'" in <bound method GuestFS.__del__ of <guestfs.GuestFS instance at 0x3ea9f38>> ignored
febootstrap-supermin-helper: ext2: parent directory not found: /lib: File not found by ext2_lookup

And causes this error: http://paste.openstack.org/show/62293/

Full logs available here: http://logs.openstack.org/58/63558/8/check/check-tempest-dsvm-neutron-pg/108e4ca/logs

Interestingly, it seems guestfs was not actually used when the relevant patches went throught the gate checks:
https://review.openstack.org/#/c/70237/
https://review.openstack.org/#/c/70354/

This was expected for patch #70354 but sounds strange for patch #70237

Finally, The traceback seeems different from that of bug 1221985

Tags: testing
Revision history for this message
Anita Kuno (anteaya) wrote :

Here is my first attempt at a logstash fingerprint for this bug: http://bit.ly/1gAUkaI

It affects the following jobs:
check-tempest-dsvm-neutron-pg 27.2%
check-tempest-dsvm-neutron-isolated 27.2%
check-tempest-dsvm-ironic-postgres-nv 14.4%
check-tempest-dsvm-ironic-nv 12.4%
check-tempest-dsvm-ironic 4.8%

Revision history for this message
Anita Kuno (anteaya) wrote :

message:"NovaException: Error mounting /opt/stack/data/nova/instances/" AND message:"/disk with libguestfs"

Revision history for this message
Anita Kuno (anteaya) wrote :

Better fingerprint: message:"NovaException: Error mounting /opt/stack/data/nova/instances/" AND message:"/disk with libguestfs" AND filename: "logs/screen-n-cpu.txt"

http://bit.ly/1eH63xA

Revision history for this message
Matt Riedemann (mriedem) wrote :

e-r query patch: https://review.openstack.org/#/c/70550/

Revision history for this message
Richard Jones (rjones-redhat) wrote :

The actual error:

    ext2: parent directory not found: /lib: File not found by ext2_lookup

In this case it means you need to run:

    sudo update-guestfs-appliance

(This is for Debian/Ubuntu only)

Revision history for this message
Salvatore Orlando (salvatore-orlando) wrote :

Thanks Richard.
I was looking for something like that yesterday, but could not find an answer by myself.

Revision history for this message
Davanum Srinivas (DIMS) (dims-v) wrote :

"This error message is usually printed when supermin runs out of space on /tmp or /var/tmp" according to http://askubuntu.com/questions/302289/error-mounting-disk-image-using-guestmount

Revision history for this message
John Garbutt (johngarbutt) wrote :

Could be a test setup issue, need more information.

tags: added: gate
tags: added: testing
removed: gate
Joe Gordon (jogo)
Changed in nova:
status: New → Confirmed
Joe Gordon (jogo)
Changed in nova:
status: Confirmed → Incomplete
Revision history for this message
Sahid Orentino (sahid-ferdjaoui) wrote :

We probably have to detect if guestfs is well configured even if we can import it

Changed in nova:
status: Incomplete → New
assignee: nobody → sahid (sahid-ferdjaoui)
OpenStack Infra (hudson-openstack)
Changed in nova:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/120467
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=6a0e6209cab2ced467b63ce9ce69a41daae669fe
Submitter: Jenkins
Branch: master

commit 6a0e6209cab2ced467b63ce9ce69a41daae669fe
Author: Sahid Orentino Ferdjaoui <email address hidden>
Date: Wed Sep 10 16:05:22 2014 +0000

    disk/vfs: ensure guestfs capabilities

    Ensures that guestfs is available and well configured. Also
    when checking for capabilities this commit introduces an in-memory
    flag to avoid repeating the process if succeeded.
    If we are able to load guestfs but something wrong happens a fatal
    error is raised.

    DocImpact: We should add a not for ubuntu like systems that use
    libguestfs to execute the command 'update-guestfs-appliance' for
    configuring it.

    Closes-Bug: #1275267
    Closes-Bug: #1157922
    Change-Id: I97b3a23829ea1f3aadfe08ca6448b35010d2f312

Changed in nova:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in nova:
milestone: none → juno-rc1
status: Fix Committed → Fix Released
Thierry Carrez (ttx)
Changed in nova:
milestone: juno-rc1 → 2014.2
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.