| 2019-04-22 18:44:30 |
Gustavo Randich |
bug |
|
|
added bug |
| 2019-04-23 02:01:38 |
Takashi Natsume |
tags |
|
libvirt |
|
| 2019-04-25 13:24:55 |
Matt Riedemann |
nova: status |
New |
Triaged |
|
| 2019-04-25 13:24:57 |
Matt Riedemann |
nova: importance |
Undecided |
Medium |
|
| 2019-04-25 13:27:05 |
Matt Riedemann |
nominated for series |
|
nova/stein |
|
| 2019-04-25 13:27:05 |
Matt Riedemann |
bug task added |
|
nova/stein |
|
| 2019-04-25 13:27:05 |
Matt Riedemann |
nominated for series |
|
nova/queens |
|
| 2019-04-25 13:27:05 |
Matt Riedemann |
bug task added |
|
nova/queens |
|
| 2019-04-25 13:27:05 |
Matt Riedemann |
nominated for series |
|
nova/rocky |
|
| 2019-04-25 13:27:05 |
Matt Riedemann |
bug task added |
|
nova/rocky |
|
| 2019-04-25 13:29:45 |
Matt Riedemann |
nova/queens: status |
New |
Triaged |
|
| 2019-04-25 13:29:47 |
Matt Riedemann |
nova/rocky: status |
New |
Triaged |
|
| 2019-04-25 13:29:50 |
Matt Riedemann |
nova/queens: importance |
Undecided |
Medium |
|
| 2019-04-25 13:29:53 |
Matt Riedemann |
nova/stein: status |
New |
Triaged |
|
| 2019-04-25 13:29:55 |
Matt Riedemann |
nova/stein: importance |
Undecided |
Medium |
|
| 2019-04-25 13:29:57 |
Matt Riedemann |
nova/rocky: importance |
Undecided |
Medium |
|
| 2019-04-25 13:31:00 |
Matt Riedemann |
bug |
|
|
added subscriber Lee Yarwood |
| 2019-04-25 13:51:08 |
OpenStack Infra |
nova: status |
Triaged |
In Progress |
|
| 2019-04-25 13:51:08 |
OpenStack Infra |
nova: assignee |
|
Lee Yarwood (lyarwood) |
|
| 2019-05-03 21:08:51 |
OpenStack Infra |
nova: status |
In Progress |
Fix Released |
|
| 2019-05-03 21:10:52 |
OpenStack Infra |
nova/stein: status |
Triaged |
In Progress |
|
| 2019-05-03 21:10:52 |
OpenStack Infra |
nova/stein: assignee |
|
Lee Yarwood (lyarwood) |
|
| 2019-05-03 21:11:32 |
OpenStack Infra |
nova/queens: status |
Triaged |
In Progress |
|
| 2019-05-03 21:11:32 |
OpenStack Infra |
nova/queens: assignee |
|
Lee Yarwood (lyarwood) |
|
| 2019-05-03 21:14:55 |
OpenStack Infra |
nova/rocky: status |
Triaged |
In Progress |
|
| 2019-05-03 21:14:55 |
OpenStack Infra |
nova/rocky: assignee |
|
Lee Yarwood (lyarwood) |
|
| 2019-05-04 15:21:00 |
OpenStack Infra |
nova/stein: status |
In Progress |
Fix Committed |
|
| 2019-05-05 17:41:54 |
OpenStack Infra |
nova/rocky: status |
In Progress |
Fix Committed |
|
| 2019-05-06 15:26:46 |
OpenStack Infra |
nova/queens: status |
In Progress |
Fix Committed |
|
| 2019-05-14 16:41:32 |
Jorge Niedbalski |
bug task added |
|
nova (Ubuntu) |
|
| 2019-05-14 20:37:03 |
Jorge Niedbalski |
nominated for series |
|
Ubuntu Disco |
|
| 2019-05-14 20:37:03 |
Jorge Niedbalski |
bug task added |
|
nova (Ubuntu Disco) |
|
| 2019-05-14 20:37:03 |
Jorge Niedbalski |
nominated for series |
|
Ubuntu Bionic |
|
| 2019-05-14 20:37:03 |
Jorge Niedbalski |
bug task added |
|
nova (Ubuntu Bionic) |
|
| 2019-05-14 20:37:03 |
Jorge Niedbalski |
nominated for series |
|
Ubuntu Cosmic |
|
| 2019-05-14 20:37:03 |
Jorge Niedbalski |
bug task added |
|
nova (Ubuntu Cosmic) |
|
| 2019-05-14 21:26:04 |
Jorge Niedbalski |
description |
Description
===========
The following commit (1) is causing volume attachments which fail due to libvirt device attach erros to be silently ignored and Nova report the attachment as successful.
It seems that the original intention of the commit was to log a condition and re-raise the exeption, but if the exception is of type libvirt.libvirtError and does not contain the searched pattern, the exception is ignored. If you unindent the raise statement, errors are reported again.
In our case we had ceph/apparmor configuration problems in compute nodes which prevented virsh attaching the device; volumes appeared as successfully attached but the corresponding block device missing in guests VMs. Other libvirt attach error conditions are ignored also, as when you have already occuppied device names (i.e. 'Target vdb already exists', device is busy, etc.)
(1) https://github.com/openstack/nova/commit/78891c2305bff6e16706339a9c5eca99a84e409c
Steps to reproduce
==================
This is somewhat hacky, but is a quick way to provoke a virsh attach error:
- virsh detach-disk <domain> vdb
- update nova & cinder DB as if volume is detached
- re-attach volume
- volume is marked as attached, but VM block device is missing
Expected result
===============
- Error 'libvirtError: Requested operation is not valid: target vdb already exists' should be raised, and volume not attached
Actual result
=============
- Attach successful but virsh block device not created
Environment
===========
- Openstack version Queens |
[Impact]
The following commit (1) is causing volume attachments which fail due to libvirt device attach erros to be silently ignored and Nova report the attachment as successful.
It seems that the original intention of the commit was to log a condition and re-raise the exeption, but if the exception is of type libvirt.libvirtError and does not contain the searched pattern, the exception is ignored. If you unindent the raise statement, errors are reported again.
In our case we had ceph/apparmor configuration problems in compute nodes which prevented virsh attaching the device; volumes appeared as successfully attached but the corresponding block device missing in guests VMs. Other libvirt attach error conditions are ignored also, as when you have already occuppied device names (i.e. 'Target vdb already exists', device is busy, etc.)
(1) https://github.com/openstack/nova/commit/78891c2305bff6e16706339a9c5eca99a84e409c
[Test Case]
* Deploy any OpenStack version up to Pike , which includes ceph backed cinder
* Create a guest VM (openstack server ...)
* Create a test cinder volume
$ openstack volume create test --size 10
* Force a drop on ceph traffic. Run the following command on the nova hypervisor on which the server runs.
$ iptables -A OUTPUT -d ceph-mon-addr -p tcp --dport 6800 -j DROP
* Attach the volume to a running instance.
$ openstack server add volume 7151f507-a6b7-4f6d-a4cc-fd223d9feb5d 742ff117-21ae-4d1b-a52b-5b37955716ff
* This should cause the volume attachment to fail
$ virsh domblklist instance-xxxxx
Target Source
------------------------------------------------
vda nova/7151f507-a6b7-4f6d-a4cc-fd223d9feb5d_disk
No volume should attached after this step.
* If the behavior is fixed:
* Check that openstack server show , doesn't displays the displays the volume as attached.
* Check that proper log entries states the libvirt exception and error.
* If the behavior isn't fixed:
* openstack server show <ID> , will display the volume in the volumes_attached property.
[Expected result]
* Volume attach fails and a proper exception is logged.
[Actual result]
* Volume attach fails but remains connected to the host and no further
exception gets logged.
[Regression Potential]
* We haven't identified any regression potential on this SRU.
[Other Info]
* N/A
Description |
|
| 2019-05-15 15:48:05 |
Jorge Niedbalski |
summary |
Virsh disk attach errors silently ignored |
[SRU] Virsh disk attach errors silently ignored |
|
| 2019-05-15 21:35:39 |
Jorge Niedbalski |
bug task added |
|
cloud-archive |
|
| 2019-05-16 12:33:47 |
Corey Bryant |
nova (Ubuntu Disco): status |
New |
Triaged |
|
| 2019-05-16 12:33:50 |
Corey Bryant |
nova (Ubuntu Cosmic): status |
New |
Triaged |
|
| 2019-05-16 12:33:53 |
Corey Bryant |
nova (Ubuntu Bionic): status |
New |
Triaged |
|
| 2019-05-16 12:33:57 |
Corey Bryant |
nova (Ubuntu): status |
New |
Triaged |
|
| 2019-05-16 12:33:59 |
Corey Bryant |
nova (Ubuntu): importance |
Undecided |
Medium |
|
| 2019-05-16 12:34:02 |
Corey Bryant |
nova (Ubuntu Bionic): importance |
Undecided |
Medium |
|
| 2019-05-16 12:34:06 |
Corey Bryant |
nova (Ubuntu Cosmic): importance |
Undecided |
Medium |
|
| 2019-05-16 12:34:08 |
Corey Bryant |
nova (Ubuntu Disco): importance |
Undecided |
Medium |
|
| 2019-05-16 13:08:02 |
Sahid Orentino |
cloud-archive: assignee |
|
Sahid Orentino (sahid-ferdjaoui) |
|
| 2019-05-16 13:08:05 |
Sahid Orentino |
nova (Ubuntu): assignee |
|
Sahid Orentino (sahid-ferdjaoui) |
|
| 2019-05-16 13:08:09 |
Sahid Orentino |
nova (Ubuntu Bionic): assignee |
|
Sahid Orentino (sahid-ferdjaoui) |
|
| 2019-05-16 13:08:12 |
Sahid Orentino |
nova (Ubuntu Disco): assignee |
|
Sahid Orentino (sahid-ferdjaoui) |
|
| 2019-05-16 13:08:15 |
Sahid Orentino |
nova (Ubuntu Cosmic): assignee |
|
Sahid Orentino (sahid-ferdjaoui) |
|
| 2019-05-16 13:09:49 |
Corey Bryant |
nominated for series |
|
cloud-archive/queens |
|
| 2019-05-16 13:09:49 |
Corey Bryant |
bug task added |
|
cloud-archive/queens |
|
| 2019-05-16 13:09:49 |
Corey Bryant |
nominated for series |
|
cloud-archive/rocky |
|
| 2019-05-16 13:09:49 |
Corey Bryant |
bug task added |
|
cloud-archive/rocky |
|
| 2019-05-16 13:09:49 |
Corey Bryant |
nominated for series |
|
cloud-archive/stein |
|
| 2019-05-16 13:09:49 |
Corey Bryant |
bug task added |
|
cloud-archive/stein |
|
| 2019-05-16 13:10:06 |
Corey Bryant |
cloud-archive/queens: importance |
Undecided |
Medium |
|
| 2019-05-16 13:10:06 |
Corey Bryant |
cloud-archive/queens: status |
New |
Triaged |
|
| 2019-05-16 13:10:20 |
Corey Bryant |
cloud-archive/rocky: importance |
Undecided |
Medium |
|
| 2019-05-16 13:10:20 |
Corey Bryant |
cloud-archive/rocky: status |
New |
Triaged |
|
| 2019-05-16 13:11:34 |
Corey Bryant |
cloud-archive/stein: importance |
Undecided |
Medium |
|
| 2019-05-16 13:11:34 |
Corey Bryant |
cloud-archive/stein: status |
New |
Triaged |
|
| 2019-05-16 16:22:28 |
Corey Bryant |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2019-05-16 16:24:17 |
Corey Bryant |
description |
[Impact]
The following commit (1) is causing volume attachments which fail due to libvirt device attach erros to be silently ignored and Nova report the attachment as successful.
It seems that the original intention of the commit was to log a condition and re-raise the exeption, but if the exception is of type libvirt.libvirtError and does not contain the searched pattern, the exception is ignored. If you unindent the raise statement, errors are reported again.
In our case we had ceph/apparmor configuration problems in compute nodes which prevented virsh attaching the device; volumes appeared as successfully attached but the corresponding block device missing in guests VMs. Other libvirt attach error conditions are ignored also, as when you have already occuppied device names (i.e. 'Target vdb already exists', device is busy, etc.)
(1) https://github.com/openstack/nova/commit/78891c2305bff6e16706339a9c5eca99a84e409c
[Test Case]
* Deploy any OpenStack version up to Pike , which includes ceph backed cinder
* Create a guest VM (openstack server ...)
* Create a test cinder volume
$ openstack volume create test --size 10
* Force a drop on ceph traffic. Run the following command on the nova hypervisor on which the server runs.
$ iptables -A OUTPUT -d ceph-mon-addr -p tcp --dport 6800 -j DROP
* Attach the volume to a running instance.
$ openstack server add volume 7151f507-a6b7-4f6d-a4cc-fd223d9feb5d 742ff117-21ae-4d1b-a52b-5b37955716ff
* This should cause the volume attachment to fail
$ virsh domblklist instance-xxxxx
Target Source
------------------------------------------------
vda nova/7151f507-a6b7-4f6d-a4cc-fd223d9feb5d_disk
No volume should attached after this step.
* If the behavior is fixed:
* Check that openstack server show , doesn't displays the displays the volume as attached.
* Check that proper log entries states the libvirt exception and error.
* If the behavior isn't fixed:
* openstack server show <ID> , will display the volume in the volumes_attached property.
[Expected result]
* Volume attach fails and a proper exception is logged.
[Actual result]
* Volume attach fails but remains connected to the host and no further
exception gets logged.
[Regression Potential]
* We haven't identified any regression potential on this SRU.
[Other Info]
* N/A
Description |
[Impact]
The following commit (1) is causing volume attachments which fail due to libvirt device attach erros to be silently ignored and Nova report the attachment as successful.
It seems that the original intention of the commit was to log a condition and re-raise the exeption, but if the exception is of type libvirt.libvirtError and does not contain the searched pattern, the exception is ignored. If you unindent the raise statement, errors are reported again.
In our case we had ceph/apparmor configuration problems in compute nodes which prevented virsh attaching the device; volumes appeared as successfully attached but the corresponding block device missing in guests VMs. Other libvirt attach error conditions are ignored also, as when you have already occuppied device names (i.e. 'Target vdb already exists', device is busy, etc.)
(1) https://github.com/openstack/nova/commit/78891c2305bff6e16706339a9c5eca99a84e409c
[Test Case]
* Deploy any OpenStack version up to Pike , which includes ceph backed cinder
* Create a guest VM (openstack server ...)
* Create a test cinder volume
$ openstack volume create test --size 10
* Force a drop on ceph traffic. Run the following command on the nova hypervisor on which the server runs.
$ iptables -A OUTPUT -d ceph-mon-addr -p tcp --dport 6800 -j DROP
* Attach the volume to a running instance.
$ openstack server add volume 7151f507-a6b7-4f6d-a4cc-fd223d9feb5d 742ff117-21ae-4d1b-a52b-5b37955716ff
* This should cause the volume attachment to fail
$ virsh domblklist instance-xxxxx
Target Source
------------------------------------------------
vda nova/7151f507-a6b7-4f6d-a4cc-fd223d9feb5d_disk
No volume should attached after this step.
* If the behavior is fixed:
* Check that openstack server show , doesn't displays the displays the volume as attached.
* Check that proper log entries states the libvirt exception and error.
* If the behavior isn't fixed:
* openstack server show <ID> , will display the volume in the volumes_attached property.
[Expected result]
* Volume attach fails and a proper exception is logged.
[Actual result]
* Volume attach fails but remains connected to the host and no further
exception gets logged.
[Regression Potential]
* The patches have been cherry-picked from upstream which helps to reduce the regression potential of these fixes.
[Other Info]
* N/A
Description |
|
| 2019-05-16 17:11:03 |
Łukasz Zemczak |
nova (Ubuntu Disco): status |
Triaged |
Fix Committed |
|
| 2019-05-16 17:11:08 |
Łukasz Zemczak |
bug |
|
|
added subscriber SRU Verification |
| 2019-05-16 17:11:12 |
Łukasz Zemczak |
tags |
libvirt |
libvirt verification-needed verification-needed-disco |
|
| 2019-05-16 17:17:39 |
Łukasz Zemczak |
nova (Ubuntu Cosmic): status |
Triaged |
Fix Committed |
|
| 2019-05-16 17:17:46 |
Łukasz Zemczak |
tags |
libvirt verification-needed verification-needed-disco |
libvirt verification-needed verification-needed-cosmic verification-needed-disco |
|
| 2019-05-16 17:54:45 |
Corey Bryant |
cloud-archive/stein: status |
Triaged |
Fix Committed |
|
| 2019-05-16 17:54:49 |
Corey Bryant |
tags |
libvirt verification-needed verification-needed-cosmic verification-needed-disco |
libvirt verification-needed verification-needed-cosmic verification-needed-disco verification-stein-needed |
|
| 2019-05-17 07:33:57 |
Launchpad Janitor |
nova (Ubuntu): status |
Triaged |
Fix Released |
|
| 2019-05-17 12:50:14 |
Corey Bryant |
cloud-archive/rocky: status |
Triaged |
Fix Committed |
|
| 2019-05-17 12:50:18 |
Corey Bryant |
tags |
libvirt verification-needed verification-needed-cosmic verification-needed-disco verification-stein-needed |
libvirt verification-needed verification-needed-cosmic verification-needed-disco verification-rocky-needed verification-stein-needed |
|
| 2019-05-20 15:04:55 |
Łukasz Zemczak |
nova (Ubuntu Bionic): status |
Triaged |
Fix Committed |
|
| 2019-05-20 15:05:02 |
Łukasz Zemczak |
tags |
libvirt verification-needed verification-needed-cosmic verification-needed-disco verification-rocky-needed verification-stein-needed |
libvirt verification-needed verification-needed-bionic verification-needed-cosmic verification-needed-disco verification-rocky-needed verification-stein-needed |
|
| 2019-05-20 19:04:39 |
Corey Bryant |
cloud-archive/queens: status |
Triaged |
Fix Committed |
|
| 2019-05-20 19:04:41 |
Corey Bryant |
tags |
libvirt verification-needed verification-needed-bionic verification-needed-cosmic verification-needed-disco verification-rocky-needed verification-stein-needed |
libvirt verification-needed verification-needed-bionic verification-needed-cosmic verification-needed-disco verification-queens-needed verification-rocky-needed verification-stein-needed |
|
| 2019-06-05 14:16:13 |
Edward Hope-Morley |
tags |
libvirt verification-needed verification-needed-bionic verification-needed-cosmic verification-needed-disco verification-queens-needed verification-rocky-needed verification-stein-needed |
libvirt sts-sru-needed verification-needed verification-needed-bionic verification-needed-cosmic verification-needed-disco verification-queens-needed verification-rocky-needed verification-stein-needed |
|
| 2019-06-05 14:16:21 |
Edward Hope-Morley |
tags |
libvirt sts-sru-needed verification-needed verification-needed-bionic verification-needed-cosmic verification-needed-disco verification-queens-needed verification-rocky-needed verification-stein-needed |
libvirt sts sts-sru-needed verification-needed verification-needed-bionic verification-needed-cosmic verification-needed-disco verification-queens-needed verification-rocky-needed verification-stein-needed |
|
| 2019-06-05 14:23:54 |
Jorge Niedbalski |
tags |
libvirt sts sts-sru-needed verification-needed verification-needed-bionic verification-needed-cosmic verification-needed-disco verification-queens-needed verification-rocky-needed verification-stein-needed |
libvirt verification-done-bionic verification-done-cosmic verification-done-disco verification-needed verification-queens-needed verification-rocky-needed verification-stein-needed |
|
| 2019-06-05 21:17:01 |
Launchpad Janitor |
nova (Ubuntu Bionic): status |
Fix Committed |
Fix Released |
|
| 2019-06-05 21:17:46 |
Launchpad Janitor |
nova (Ubuntu Cosmic): status |
Fix Committed |
Fix Released |
|
| 2019-06-14 22:49:17 |
Jorge Niedbalski |
tags |
libvirt verification-done-bionic verification-done-cosmic verification-done-disco verification-needed verification-queens-needed verification-rocky-needed verification-stein-needed |
libvirt verification-done verification-done-bionic verification-done-cosmic verification-done-disco verification-queens-done verification-rocky-done verification-stein-done |
|
| 2019-07-01 13:36:55 |
James Page |
cloud-archive/stein: status |
Fix Committed |
Fix Released |
|
| 2019-07-01 13:41:02 |
James Page |
cloud-archive/queens: status |
Fix Committed |
Fix Released |
|
| 2019-07-03 18:20:27 |
Launchpad Janitor |
nova (Ubuntu Disco): status |
Fix Committed |
Fix Released |
|
| 2019-07-04 04:09:36 |
James Page |
cloud-archive/rocky: status |
Fix Committed |
Fix Released |
|
