tripleo

Instance spawn failing with "Failed to build and run instance: libvirt.libvirtError: internal error: Process exited prior to exec: libvirt: QEMU Driver error : failed to umount devfs on /dev: Device or resource busy

Bug #2006649 reported by Sandeep Yadav
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
tripleo
Fix Released
Critical
Unassigned
tripleo antelope-1

Bug Description

Description:

Instance spawn failing with "Failed to build and run instance: libvirt.libvirtError: internal error: Process exited prior to exec: libvirt: QEMU Driver error : failed to umount devfs on /dev: Device or resource busy

Logs Snippet:

https://d91d6e9aa5357de7fdf6-27ba51e152f37c2c550101a1da9a359d.ssl.cf5.rackcdn.com/873121/1/check/tripleo-ci-centos-9-scenario001-standalone/fc75d23/logs/undercloud/var/log/extra/podman/containers/nova_compute/log/dnf.rpm.log
~~~
2023-02-08T16:37:34+0000 SUBDEBUG Installed: libvirt-libs-9.0.0-3.el9.x86_64
2023-02-08T16:37:34+0000 SUBDEBUG Installed: python3-libvirt-9.0.0-1.el9.x86_6
~~~

https://d91d6e9aa5357de7fdf6-27ba51e152f37c2c550101a1da9a359d.ssl.cf5.rackcdn.com/873121/1/check/tripleo-ci-centos-9-scenario001-standalone/fc75d23/logs/undercloud/var/log/extra/errors.txt
~~~
internal error: Process exited prior to exec: libvirt: QEMU Driver error : failed to umount devfs on /dev: Device or resource busy
~~~

This started after libvirt-libs-9.0.0-3.el9.x86_64 which have a genuine libvirt issue, Discussed more in RHBZ: https://bugzilla.redhat.com/show_bug.cgi?id=2167302

We will need to pin libvirt in Centos stream 9 job till this issue is resolved.

Tags: ci
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-quickstart (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart/+/873213

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-quickstart (master)

Reviewed: https://review.opendev.org/c/openstack/tripleo-quickstart/+/873213
Committed: https://opendev.org/openstack/tripleo-quickstart/commit/1a13b7f2c572b4b273678a37834cca9b0524582a
Submitter: "Zuul (22348)"
Branch: master

commit 1a13b7f2c572b4b273678a37834cca9b0524582a
Author: Sandeep Yadav <email address hidden>
Date: Thu Feb 9 07:16:18 2023 +0530

    Temporary Pin Libvirt to 8.10 version

    Libvirt 9 is causing errors and is acknowledged as genuine
    libvirt bug[1]

    Excluding libvirt-9* version so that it pull last known
    good version 8.10.

    Also, temporary marking undercloud upgrade as non-voting
    as its blocking merge of this patch. Undercloud upgrade job
    bug is here [2] and its fix is blocked on libvirt issue.

    [1] https://bugzilla.redhat.com/show_bug.cgi?id=2167302
    [2] https://bugs.launchpad.net/bugs/2006442

    Related-Bug: #2006649
    Change-Id: Ibdbb1dc24a3ad0eb07d802e59b9f5a0bd489ced5

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-quickstart (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart/+/873527

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-quickstart (master)

Reviewed: https://review.opendev.org/c/openstack/tripleo-quickstart/+/873527
Committed: https://opendev.org/openstack/tripleo-quickstart/commit/55c6f6e9454779b45ae8b160722a0b6949da30ac
Submitter: "Zuul (22348)"
Branch: master

commit 55c6f6e9454779b45ae8b160722a0b6949da30ac
Author: Sandeep Yadav <email address hidden>
Date: Mon Feb 13 15:29:30 2023 +0530

    Move back undercloud upgrade job to voting in tq

    Undercloud upgrade job was marked non-voting in [1] to temporary
    unblock gate in tq repo.

    Fix for undercloud upgrade job is release a new version of puppet-tripleo,
    which was done via [2].

    Moving back undercloud upgrade job to voting as fix is merged.

    [1] https://review.opendev.org/c/openstack/tripleo-quickstart/+/873213
    [2] https://review.opendev.org/c/openstack/releases/+/872858

    Related-Bug: #2006649
    Change-Id: I6e60ff9256c3c2dc8928a87f9a8d53e6f1c747b0

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to tripleo-quickstart (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/tripleo-quickstart/+/874400

Revision history for this message
Marios Andreou (marios-b) wrote :

Posted the libvirt pin revert at https://review.opendev.org/c/openstack/tripleo-quickstart/+/874400 and testing with https://review.opendev.org/c/openstack/tripleo-heat-templates/+/874416

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to tripleo-quickstart (master)

Reviewed: https://review.opendev.org/c/openstack/tripleo-quickstart/+/874400
Committed: https://opendev.org/openstack/tripleo-quickstart/commit/e1178bba3bb9dd9b3ee9dc800474dd1597396d97
Submitter: "Zuul (22348)"
Branch: master

commit e1178bba3bb9dd9b3ee9dc800474dd1597396d97
Author: Marios Andreou <email address hidden>
Date: Mon Feb 20 12:28:02 2023 +0000

    Revert "Temporary Pin Libvirt to 8.10 version"

    This reverts commit 1a13b7f2c572b4b273678a37834cca9b0524582a.

    Reason for revert: testing for https://bugs.launchpad.net/tripleo/+bug/2006649

    Change-Id: I603fac6f649c86d6a3b85a2c64a6dc8359ac1e18
    Related-Bug: 2006649

Revision history for this message
Marios Andreou (marios-b) wrote :

since we reverted the pin going to move this to fix released

Changed in tripleo:
status: Triaged → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.