[SRU] Virsh disk attach errors silently ignored

Yeah indeed it looks like the raise was used improperly. Probably a better thing to do here would be to use the excutils.save_and_reraise_exception() which provides a context manager that you can control if the exception should be re-raised or not.

Fix proposed to branch: master
Review: https://review.opendev.org/655696

Reviewed: https://review.opendev.org/655696
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=bc57ae50734fa6a70115b6369e867079fb5eb4b8
Submitter: Zuul
Branch: master

commit bc57ae50734fa6a70115b6369e867079fb5eb4b8
Author: Lee Yarwood <email address hidden>
Date: Thu Apr 25 14:42:09 2019 +0100

    libvirt: Stop ignoring unknown libvirtError exceptions during volume attach

    Id346bce6e47431988cce7001abcf29a9faf2936a attempted to introduce a
    simple breadcrumb in the logs to highlight a known Libvirt issue.
    Unfortunately this change resulted in libvirtError exceptions that
    didn't match the known issue being silently ignored.

    This change corrects this by using excutils.save_and_reraise_exception
    to ensure all libvirtError exceptions are logged and raised regardless
    of being linked to the known issue.

    Change-Id: Ib440f4f2e484312af5f393722363846f6c95b760
    Closes-Bug: #1825882

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/657049

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/657050

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/657051

Reviewed: https://review.opendev.org/657049
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=b3dedefcc58a1fc76ba37f9f8bb1ef7d238aaceb
Submitter: Zuul
Branch: stable/stein

commit b3dedefcc58a1fc76ba37f9f8bb1ef7d238aaceb
Author: Lee Yarwood <email address hidden>
Date: Thu Apr 25 14:42:09 2019 +0100

    libvirt: Stop ignoring unknown libvirtError exceptions during volume attach

    Id346bce6e47431988cce7001abcf29a9faf2936a attempted to introduce a
    simple breadcrumb in the logs to highlight a known Libvirt issue.
    Unfortunately this change resulted in libvirtError exceptions that
    didn't match the known issue being silently ignored.

    This change corrects this by using excutils.save_and_reraise_exception
    to ensure all libvirtError exceptions are logged and raised regardless
    of being linked to the known issue.

    Change-Id: Ib440f4f2e484312af5f393722363846f6c95b760
    Closes-Bug: #1825882
    (cherry picked from commit bc57ae50734fa6a70115b6369e867079fb5eb4b8)

Reviewed: https://review.opendev.org/657051
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=34b4220448395f10eb2fd39d68b3f527339ab414
Submitter: Zuul
Branch: stable/rocky

commit 34b4220448395f10eb2fd39d68b3f527339ab414
Author: Lee Yarwood <email address hidden>
Date: Thu Apr 25 14:42:09 2019 +0100

    libvirt: Stop ignoring unknown libvirtError exceptions during volume attach

    Id346bce6e47431988cce7001abcf29a9faf2936a attempted to introduce a
    simple breadcrumb in the logs to highlight a known Libvirt issue.
    Unfortunately this change resulted in libvirtError exceptions that
    didn't match the known issue being silently ignored.

    This change corrects this by using excutils.save_and_reraise_exception
    to ensure all libvirtError exceptions are logged and raised regardless
    of being linked to the known issue.

    Change-Id: Ib440f4f2e484312af5f393722363846f6c95b760
    Closes-Bug: #1825882
    (cherry picked from commit bc57ae50734fa6a70115b6369e867079fb5eb4b8)
    (cherry picked from commit b3dedefcc58a1fc76ba37f9f8bb1ef7d238aaceb)

Reviewed: https://review.opendev.org/657050
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=63b45a87b82655824e61641016430a6613cee001
Submitter: Zuul
Branch: stable/queens

commit 63b45a87b82655824e61641016430a6613cee001
Author: Lee Yarwood <email address hidden>
Date: Thu Apr 25 14:42:09 2019 +0100

    libvirt: Stop ignoring unknown libvirtError exceptions during volume attach

    Id346bce6e47431988cce7001abcf29a9faf2936a attempted to introduce a
    simple breadcrumb in the logs to highlight a known Libvirt issue.
    Unfortunately this change resulted in libvirtError exceptions that
    didn't match the known issue being silently ignored.

    This change corrects this by using excutils.save_and_reraise_exception
    to ensure all libvirtError exceptions are logged and raised regardless
    of being linked to the known issue.

    Change-Id: Ib440f4f2e484312af5f393722363846f6c95b760
    Closes-Bug: #1825882
    (cherry picked from commit bc57ae50734fa6a70115b6369e867079fb5eb4b8)
    (cherry picked from commit b3dedefcc58a1fc76ba37f9f8bb1ef7d238aaceb)
    (cherry picked from commit 34b4220448395f10eb2fd39d68b3f527339ab414)

Hello Gustavo, or anyone else affected,

Accepted nova into disco-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/nova/2:19.0.0-0ubuntu2.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-disco to verification-done-disco. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-disco. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Gustavo, or anyone else affected,

Accepted nova into cosmic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/nova/2:18.1.0-0ubuntu3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Gustavo, or anyone else affected,

Accepted nova into stein-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:stein-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-stein-needed to verification-stein-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-stein-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

This bug was fixed in the package nova - 2:19.0.0-0ubuntu5

---------------
nova (2:19.0.0-0ubuntu5) eoan; urgency=medium

  * d/p/bug_1825882.patch: Cherry-picked from upstream to ensure
    virsh disk attach does not fail silently (LP: #1825882).
  * d/p/bug_1826523.patch: Cherry-picked from upstream to ensure
    always disconnect volumes after libvirt exceptions (LP: #1826523).

 -- Sahid Orentino Ferdjaoui <email address hidden> Thu, 16 May 2019 10:41:27 +0200

Hello Gustavo, or anyone else affected,

Accepted nova into rocky-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:rocky-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-rocky-needed to verification-rocky-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-rocky-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Gustavo, or anyone else affected,

Accepted nova into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/nova/2:17.0.9-0ubuntu3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Gustavo, or anyone else affected,

Accepted nova into queens-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:queens-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-queens-needed to verification-queens-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-queens-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

I have verified the following versions:

* bionic: 17.0.9-0ubuntu3
* cosmic: 18.1.0-0ubuntu3
* disco: 19.0.0-0ubuntu2.3

The following exception is raised when the traffic to the ceph-mon is dropped by
iptables.

 libvirt.libvirtError: Timed out during operation: cannot acquire state change lock (held by remoteDispatchDomainAttachDeviceFlags)

No attachment has been found at the vms after this operation fails, no volume added to
volumes_attached property.

This bug was fixed in the package nova - 2:17.0.9-0ubuntu3

---------------
nova (2:17.0.9-0ubuntu3) bionic; urgency=medium

  * d/p/bug_1825882.patch: Cherry-picked from upstream to ensure
    virsh disk attach does not fail silently (LP: #1825882).
  * d/p/bug_1826523.patch: Cherry-picked from upstream to ensure
    always disconnect volumes after libvirt exceptions (LP: #1826523).

nova (2:17.0.9-0ubuntu2) bionic; urgency=medium

  * d/p/xenapi-agent-change-openssl-error-handling.patch: Cherry-picked from
    upstream to ensure xenapi agent only raises a RuntimeError exception
    when openssl returns a non-zero exit code (LP: #1771506).
  * d/p/skip-double-word-hacking-test.patch: Cherry-picked from upstream
    to work-around test_hacking failures with new versions of python
    (LP: #1782786).

 -- Sahid Orentino Ferdjaoui <email address hidden> Thu, 16 May 2019 11:06:11 +0200

This bug was fixed in the package nova - 2:18.1.0-0ubuntu3

---------------
nova (2:18.1.0-0ubuntu3) cosmic; urgency=medium

  * d/p/bug_1825882.patch: Cherry-picked from upstream to ensure
    virsh disk attach does not fail silently (LP: #1825882).
  * d/p/bug_1826523.patch: Cherry-picked from upstream to ensure
    always disconnect volumes after libvirt exceptions (LP: #1826523).

nova (2:18.1.0-0ubuntu2) cosmic; urgency=medium

  * d/p/xenapi-agent-change-openssl-error-handling.patch: Cherry-picked from
    upstream to ensure xenapi agent only raises a RuntimeError exception
    when openssl returns a non-zero exit code (LP: #1771506).

 -- Sahid Orentino Ferdjaoui <email address hidden> Thu, 16 May 2019 10:58:45 +0200

This issue was fixed in the openstack/nova 19.0.1 release.

OK, verified S/Q/R as commented on https://bugs.launchpad.net/bugs/1826523

The following exception is raised when the traffic to the ceph-mon is dropped by
iptables.

** libvirt.libvirtError: Timed out during operation: cannot acquire state change lock (held by remoteDispatchDomainAttachDeviceFlags)
** No attachment has been found at the vms after this operation fails, no volume added to
volumes_attached property. (https://pastebin.canonical.com/p/kkmRcfrsjT/)

This issue was fixed in the openstack/nova 18.2.1 release.

The verification of the Stable Release Update for nova has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package nova - 2:19.0.0-0ubuntu2.3~cloud0
---------------

 nova (2:19.0.0-0ubuntu2.3~cloud0) bionic-stein; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 nova (2:19.0.0-0ubuntu2.3) disco; urgency=medium
 .
   * d/p/bug_1825882.patch: Cherry-picked from upstream to ensure
     virsh disk attach does not fail silently (LP: #1825882).
   * d/p/bug_1826523.patch: Cherry-picked from upstream to ensure
     always disconnect volumes after libvirt exceptions (LP: #1826523).

The verification of the Stable Release Update for nova has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package nova - 2:17.0.9-0ubuntu3~cloud0
---------------

 nova (2:17.0.9-0ubuntu3~cloud0) xenial-queens; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 nova (2:17.0.9-0ubuntu3) bionic; urgency=medium
 .
   * d/p/bug_1825882.patch: Cherry-picked from upstream to ensure
     virsh disk attach does not fail silently (LP: #1825882).
   * d/p/bug_1826523.patch: Cherry-picked from upstream to ensure
     always disconnect volumes after libvirt exceptions (LP: #1826523).

This bug was fixed in the package nova - 2:19.0.0-0ubuntu2.3

---------------
nova (2:19.0.0-0ubuntu2.3) disco; urgency=medium

  * d/p/bug_1825882.patch: Cherry-picked from upstream to ensure
    virsh disk attach does not fail silently (LP: #1825882).
  * d/p/bug_1826523.patch: Cherry-picked from upstream to ensure
    always disconnect volumes after libvirt exceptions (LP: #1826523).

nova (2:19.0.0-0ubuntu2.2) disco; urgency=medium

  * d/p/xenapi-agent-change-openssl-error-handling.patch: Cherry-picked from
    upstream to ensure xenapi agent only raises a RuntimeError exception
    when openssl returns a non-zero exit code (LP: #1771506).

nova (2:19.0.0-0ubuntu2.1) disco; urgency=medium

  * d/gbp.conf: Create stable/stein branch.
  * d/p/eventlet-monkey-patching-should-be-as-early-as-possible.patch:
    Cherry-picked from upstream stable/stein review to fix py3+wsgi+ssl crash
    (LP: #1808951).

 -- Sahid Orentino Ferdjaoui <email address hidden> Thu, 16 May 2019 10:54:46 +0200

This issue was fixed in the openstack/nova 17.0.11 release.

This issue was fixed in the openstack/nova 20.0.0.0rc1 release candidate.