| 2017-02-17 16:21:27 |
Logan V |
bug |
|
|
added bug |
| 2017-02-17 17:36:37 |
Jamie Strandboge |
libvirt (Ubuntu): status |
New |
Incomplete |
|
| 2017-02-17 20:13:45 |
Jamie Strandboge |
libvirt (Ubuntu): status |
Incomplete |
New |
|
| 2017-02-20 08:08:22 |
Christian Ehrhardt |
bug |
|
|
added subscriber ChristianEhrhardt |
| 2017-02-20 09:41:58 |
Christian Ehrhardt |
bug task added |
|
cloud-archive |
|
| 2017-02-20 09:42:17 |
Christian Ehrhardt |
libvirt (Ubuntu): importance |
Undecided |
Critical |
|
| 2017-02-20 09:42:21 |
Christian Ehrhardt |
libvirt (Ubuntu): status |
New |
Incomplete |
|
| 2017-02-20 09:42:22 |
Christian Ehrhardt |
cloud-archive: status |
New |
Incomplete |
|
| 2017-02-20 11:39:43 |
Christian Ehrhardt |
tags |
|
regression-update |
|
| 2017-02-20 14:24:16 |
Christian Ehrhardt |
tags |
regression-update |
|
|
| 2017-02-20 14:35:20 |
Robie Basak |
bug |
|
|
added subscriber Robie Basak |
| 2017-02-20 16:16:25 |
Dr. Jens Harbott |
bug |
|
|
added subscriber Dr. Jens Rosenboom |
| 2017-02-22 14:35:45 |
Nell Jerram |
bug |
|
|
added subscriber Neil Jerram |
| 2017-03-10 07:08:52 |
Christian Ehrhardt |
bug |
|
|
added subscriber Corey Bryant |
| 2017-03-10 07:08:59 |
Christian Ehrhardt |
bug |
|
|
added subscriber James Page |
| 2017-03-17 13:35:02 |
Launchpad Janitor |
libvirt (Ubuntu): status |
Incomplete |
Fix Released |
|
| 2017-03-20 12:29:23 |
Christian Ehrhardt |
nominated for series |
|
Ubuntu Yakkety |
|
| 2017-03-20 12:29:23 |
Christian Ehrhardt |
bug task added |
|
libvirt (Ubuntu Yakkety) |
|
| 2017-03-20 12:29:29 |
Christian Ehrhardt |
libvirt (Ubuntu Yakkety): status |
New |
Triaged |
|
| 2017-03-20 12:29:31 |
Christian Ehrhardt |
libvirt (Ubuntu Yakkety): importance |
Undecided |
Medium |
|
| 2017-03-20 15:00:37 |
James Page |
cloud-archive: status |
Incomplete |
Invalid |
|
| 2017-03-20 15:01:20 |
James Page |
bug task added |
|
nova |
|
| 2017-03-21 09:43:12 |
Christian Ehrhardt |
description |
I have VMs failing to start with 2017-02-17 15:38:44.458 264015 ERROR nova.compute.manager [instance: 0c97ab16-2d30-43fa-b0e4-a064a842b5ed] libvirtError: internal error: process exited while connecting to monitor: 2017-02-17T15:38:43.907222Z qemu-system-x86_64: -netdev tap,ifname=tapf34ef99e-18,id=hostnet0,vhost=on,vhostfd=28: network script /etc/qemu-ifup failed with status 256
Log excerpt:
http://cdn.pasteraw.com/b3tw4cjefomfi3e9k09hvodrfun85z
Seems to be that /etc/qemu-ifup is being blocked by apparmor:
type=AVC msg=audit(1487347189.015:28536): apparmor="DENIED" operation="exec" profile="libvirt-4a03fea7-e966-48e4-80ac-aa138db67243" name="/etc/qemu-ifup" pid=285438 comm="qemu-system-x86" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
type=PATH msg=audit(1487347189.015:28536): item=0 name="/etc/qemu-ifup" inode=66403 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL
root@ubuntu-trusty-5773:/etc/apparmor.d/abstractions# cat /etc/apparmor.d/libvirt/libvirt-4a03fea7-e966-48e4-80ac-aa138db67243
#
# This profile is for the domain whose UUID matches this file.
#
#include <tunables/global>
profile libvirt-4a03fea7-e966-48e4-80ac-aa138db67243 {
#include <abstractions/libvirt-qemu>
#include <libvirt/libvirt-4a03fea7-e966-48e4-80ac-aa138db67243.files>
}
root@ubuntu-trusty-5773:/etc/apparmor.d/abstractions# cat /etc/apparmor.d/libvirt/libvirt-4a03fea7-e966-48e4-80ac-aa138db67243.files
# DO NOT EDIT THIS FILE DIRECTLY. IT IS MANAGED BY LIBVIRT.
"/var/log/libvirt/**/instance-00000008.log" w,
"/var/lib/libvirt/qemu/domain-instance-00000008/monitor.sock" rw,
"/var/run/libvirt/**/instance-00000008.pid" rwk,
"/run/libvirt/**/instance-00000008.pid" rwk,
"/var/run/libvirt/**/*.tunnelmigrate.dest.instance-00000008" rw,
"/run/libvirt/**/*.tunnelmigrate.dest.instance-00000008" rw,
"/var/lib/nova/instances/4a03fea7-e966-48e4-80ac-aa138db67243/console.log" rw,
"/var/lib/nova/instances/4a03fea7-e966-48e4-80ac-aa138db67243/console.log" rw,
# for qemu guest agent channel
owner "/var/lib/libvirt/qemu/channel/target/domain-instance-00000008/**" rw,
/dev/vhost-net rw,
root@ubuntu-trusty-5773:/etc/apparmor.d/abstractions# dpkg -S libvirt-qemu
libvirt-bin: /etc/apparmor.d/abstractions/libvirt-qemu
root@ubuntu-trusty-5773:/etc/apparmor.d/abstractions# dpkg -l libvirt-bin
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-=========================================-=========================-=========================-=======================================================================================
ii libvirt-bin 1.3.1-1ubuntu10.6~cloud0 amd64 programs for the libvirt library
Seeing identical behavior on Xenial
ubuntu@ubuntu-xenial-5165:~$ dpkg -l libvirt-bin
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-=========================================-=========================-=========================-=======================================================================================
ii libvirt-bin 1.3.1-1ubuntu10.8 amd64 programs for the libvirt library |
[Impact]
* Please do note that this SRU statement is about the libvirt portion
of it, this is a fix of essentially an API break from Xenial to
Yakkety. This is independent to any decision to the Openstack context
discussion about the change to drop specifying a path at all.
* Before 9c17d665fdc5f (v1.3.2 which means 1.3.1 in Xenial for us) it
was possible to have the following interface configuration:
<interface type='ethernet'/>
<script path=''/>
</interface>
This resulted in -netdev tap,script=,.. Fortunately, qemu helped
us to get away with this as it just ignored the empty script
path. However, after the commit mentioned above it's libvirtd
who is executing the script. Unfortunately without special
case-ing empty script path.
* The fix adds the special casing that qemu had into libvirts handling
of the interface definition.
[Test Case]
* That is tricky as the way openstack is using to shove that in
seems to not care on xml validation as much as e.g. virsh.
If normally adding a device like
<interface type='ethernet'/>
<script path=''/>
<model type='virtio'/>
</interface>
At least in xenial AND yakkety blocked by the XML validation.
But if trying to work around like:
<script path='""'/>
Which gave "-netdev tap,script="",id=hostnet1" on yakkety then
the fix does not apply as this is '""' and not ''.
So to add the above you have to edit it in via --skip-validate like
$ virsh edit --skip-validate zesty-on-x-test
This on onlder libvrit gave: -netdev tap,script=,id=hostnet1
Which qemu understood as nop. But newer libvirt refuses.
* Error:
error: Failed to start domain <name>
error: Cannot find '' in path: No such file or directory
* Expected:
Starting the domain as-is without calling a script,
but also without complaining about being empty.
[Regression Potential]
* Regression should be low because of:
* The fix is upstream for a while now without follow on fix
* We are essentially going back to how it was
* There is no case like "I had '' set in my setup but now it is
a no-op which makes me fail" because if one had '' it failed until
now.
* Fix is in zesty for a few days without new fallout being reported
* also it passed several levels of testing (on the case and general
regression testing)
* Due to extra xml checks a device like path='' is not even definable.
So only those who run --skip-validate or similar are affected in
the first place.
[Other Info]
* n/a
----
I have VMs failing to start with 2017-02-17 15:38:44.458 264015 ERROR nova.compute.manager [instance: 0c97ab16-2d30-43fa-b0e4-a064a842b5ed] libvirtError: internal error: process exited while connecting to monitor: 2017-02-17T15:38:43.907222Z qemu-system-x86_64: -netdev tap,ifname=tapf34ef99e-18,id=hostnet0,vhost=on,vhostfd=28: network script /etc/qemu-ifup failed with status 256
Log excerpt:
http://cdn.pasteraw.com/b3tw4cjefomfi3e9k09hvodrfun85z
Seems to be that /etc/qemu-ifup is being blocked by apparmor:
type=AVC msg=audit(1487347189.015:28536): apparmor="DENIED" operation="exec" profile="libvirt-4a03fea7-e966-48e4-80ac-aa138db67243" name="/etc/qemu-ifup" pid=285438 comm="qemu-system-x86" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
type=PATH msg=audit(1487347189.015:28536): item=0 name="/etc/qemu-ifup" inode=66403 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL
root@ubuntu-trusty-5773:/etc/apparmor.d/abstractions# cat /etc/apparmor.d/libvirt/libvirt-4a03fea7-e966-48e4-80ac-aa138db67243
#
# This profile is for the domain whose UUID matches this file.
#
#include <tunables/global>
profile libvirt-4a03fea7-e966-48e4-80ac-aa138db67243 {
#include <abstractions/libvirt-qemu>
#include <libvirt/libvirt-4a03fea7-e966-48e4-80ac-aa138db67243.files>
}
root@ubuntu-trusty-5773:/etc/apparmor.d/abstractions# cat /etc/apparmor.d/libvirt/libvirt-4a03fea7-e966-48e4-80ac-aa138db67243.files
# DO NOT EDIT THIS FILE DIRECTLY. IT IS MANAGED BY LIBVIRT.
"/var/log/libvirt/**/instance-00000008.log" w,
"/var/lib/libvirt/qemu/domain-instance-00000008/monitor.sock" rw,
"/var/run/libvirt/**/instance-00000008.pid" rwk,
"/run/libvirt/**/instance-00000008.pid" rwk,
"/var/run/libvirt/**/*.tunnelmigrate.dest.instance-00000008" rw,
"/run/libvirt/**/*.tunnelmigrate.dest.instance-00000008" rw,
"/var/lib/nova/instances/4a03fea7-e966-48e4-80ac-aa138db67243/console.log" rw,
"/var/lib/nova/instances/4a03fea7-e966-48e4-80ac-aa138db67243/console.log" rw,
# for qemu guest agent channel
owner "/var/lib/libvirt/qemu/channel/target/domain-instance-00000008/**" rw,
/dev/vhost-net rw,
root@ubuntu-trusty-5773:/etc/apparmor.d/abstractions# dpkg -S libvirt-qemu
libvirt-bin: /etc/apparmor.d/abstractions/libvirt-qemu
root@ubuntu-trusty-5773:/etc/apparmor.d/abstractions# dpkg -l libvirt-bin
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-=========================================-=========================-=========================-=======================================================================================
ii libvirt-bin 1.3.1-1ubuntu10.6~cloud0 amd64 programs for the libvirt library
Seeing identical behavior on Xenial
ubuntu@ubuntu-xenial-5165:~$ dpkg -l libvirt-bin
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-=========================================-=========================-=========================-=======================================================================================
ii libvirt-bin 1.3.1-1ubuntu10.8 amd64 programs for the libvirt library |
|
| 2017-03-21 16:20:15 |
Matt Riedemann |
nominated for series |
|
nova/newton |
|
| 2017-03-21 16:20:15 |
Matt Riedemann |
bug task added |
|
nova/newton |
|
| 2017-03-21 16:20:15 |
Matt Riedemann |
nominated for series |
|
nova/ocata |
|
| 2017-03-21 16:20:15 |
Matt Riedemann |
bug task added |
|
nova/ocata |
|
| 2017-03-21 17:27:27 |
OpenStack Infra |
nova: status |
New |
In Progress |
|
| 2017-03-21 17:27:27 |
OpenStack Infra |
nova: assignee |
|
Matt Riedemann (mriedem) |
|
| 2017-03-21 17:40:15 |
Matt Riedemann |
nova: importance |
Undecided |
High |
|
| 2017-03-21 17:40:21 |
Matt Riedemann |
nova/ocata: status |
New |
Confirmed |
|
| 2017-03-21 17:40:25 |
Matt Riedemann |
nova/newton: status |
New |
Confirmed |
|
| 2017-03-21 17:40:32 |
Matt Riedemann |
nova/newton: importance |
Undecided |
High |
|
| 2017-03-21 17:40:39 |
Matt Riedemann |
nova/ocata: importance |
Undecided |
High |
|
| 2017-03-21 19:15:45 |
OpenStack Infra |
nova/ocata: status |
Confirmed |
In Progress |
|
| 2017-03-21 19:15:45 |
OpenStack Infra |
nova/ocata: assignee |
|
Matt Riedemann (mriedem) |
|
| 2017-03-21 19:35:00 |
OpenStack Infra |
nova/newton: status |
Confirmed |
In Progress |
|
| 2017-03-21 19:35:00 |
OpenStack Infra |
nova/newton: assignee |
|
Matt Riedemann (mriedem) |
|
| 2017-03-22 01:05:47 |
OpenStack Infra |
nova: status |
In Progress |
Fix Released |
|
| 2017-03-22 09:03:26 |
James Page |
bug task added |
|
nova (Ubuntu) |
|
| 2017-03-22 09:04:52 |
James Page |
nova (Ubuntu): status |
New |
Triaged |
|
| 2017-03-22 09:04:55 |
James Page |
nova (Ubuntu Yakkety): status |
New |
Triaged |
|
| 2017-03-22 09:04:59 |
James Page |
nova (Ubuntu): importance |
Undecided |
High |
|
| 2017-03-22 09:05:05 |
James Page |
nova (Ubuntu Yakkety): importance |
Undecided |
High |
|
| 2017-03-22 09:11:08 |
James Page |
nova (Ubuntu): status |
Triaged |
Invalid |
|
| 2017-03-22 09:11:31 |
James Page |
nominated for series |
|
cloud-archive/newton |
|
| 2017-03-22 09:11:31 |
James Page |
bug task added |
|
cloud-archive/newton |
|
| 2017-03-22 09:11:42 |
James Page |
cloud-archive/newton: status |
New |
Triaged |
|
| 2017-03-22 09:11:46 |
James Page |
cloud-archive/newton: importance |
Undecided |
High |
|
| 2017-03-22 09:20:14 |
James Page |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2017-03-22 11:12:02 |
Andy Whitcroft |
nova (Ubuntu Yakkety): status |
Triaged |
Fix Committed |
|
| 2017-03-22 11:12:06 |
Andy Whitcroft |
bug |
|
|
added subscriber SRU Verification |
| 2017-03-22 11:12:13 |
Andy Whitcroft |
tags |
|
verification-needed |
|
| 2017-03-22 12:12:56 |
Chris J Arges |
libvirt (Ubuntu Yakkety): status |
Triaged |
Fix Committed |
|
| 2017-03-23 09:40:16 |
James Page |
cloud-archive/newton: status |
Triaged |
Fix Committed |
|
| 2017-03-23 09:40:19 |
James Page |
tags |
verification-needed |
verification-needed verification-newton-needed |
|
| 2017-03-24 11:46:45 |
James Page |
tags |
verification-needed verification-newton-needed |
verification-needed verification-newton-done |
|
| 2017-03-24 11:47:05 |
James Page |
tags |
verification-needed verification-newton-done |
verification-done verification-newton-done |
|
| 2017-03-24 11:47:32 |
James Page |
tags |
verification-done verification-newton-done |
verification-needed verification-newton-done |
|
| 2017-03-24 13:45:08 |
James Page |
tags |
verification-needed verification-newton-done |
verification-done verification-newton-done |
|
| 2017-03-31 05:46:13 |
Christian Ehrhardt |
description |
[Impact]
* Please do note that this SRU statement is about the libvirt portion
of it, this is a fix of essentially an API break from Xenial to
Yakkety. This is independent to any decision to the Openstack context
discussion about the change to drop specifying a path at all.
* Before 9c17d665fdc5f (v1.3.2 which means 1.3.1 in Xenial for us) it
was possible to have the following interface configuration:
<interface type='ethernet'/>
<script path=''/>
</interface>
This resulted in -netdev tap,script=,.. Fortunately, qemu helped
us to get away with this as it just ignored the empty script
path. However, after the commit mentioned above it's libvirtd
who is executing the script. Unfortunately without special
case-ing empty script path.
* The fix adds the special casing that qemu had into libvirts handling
of the interface definition.
[Test Case]
* That is tricky as the way openstack is using to shove that in
seems to not care on xml validation as much as e.g. virsh.
If normally adding a device like
<interface type='ethernet'/>
<script path=''/>
<model type='virtio'/>
</interface>
At least in xenial AND yakkety blocked by the XML validation.
But if trying to work around like:
<script path='""'/>
Which gave "-netdev tap,script="",id=hostnet1" on yakkety then
the fix does not apply as this is '""' and not ''.
So to add the above you have to edit it in via --skip-validate like
$ virsh edit --skip-validate zesty-on-x-test
This on onlder libvrit gave: -netdev tap,script=,id=hostnet1
Which qemu understood as nop. But newer libvirt refuses.
* Error:
error: Failed to start domain <name>
error: Cannot find '' in path: No such file or directory
* Expected:
Starting the domain as-is without calling a script,
but also without complaining about being empty.
[Regression Potential]
* Regression should be low because of:
* The fix is upstream for a while now without follow on fix
* We are essentially going back to how it was
* There is no case like "I had '' set in my setup but now it is
a no-op which makes me fail" because if one had '' it failed until
now.
* Fix is in zesty for a few days without new fallout being reported
* also it passed several levels of testing (on the case and general
regression testing)
* Due to extra xml checks a device like path='' is not even definable.
So only those who run --skip-validate or similar are affected in
the first place.
[Other Info]
* n/a
----
I have VMs failing to start with 2017-02-17 15:38:44.458 264015 ERROR nova.compute.manager [instance: 0c97ab16-2d30-43fa-b0e4-a064a842b5ed] libvirtError: internal error: process exited while connecting to monitor: 2017-02-17T15:38:43.907222Z qemu-system-x86_64: -netdev tap,ifname=tapf34ef99e-18,id=hostnet0,vhost=on,vhostfd=28: network script /etc/qemu-ifup failed with status 256
Log excerpt:
http://cdn.pasteraw.com/b3tw4cjefomfi3e9k09hvodrfun85z
Seems to be that /etc/qemu-ifup is being blocked by apparmor:
type=AVC msg=audit(1487347189.015:28536): apparmor="DENIED" operation="exec" profile="libvirt-4a03fea7-e966-48e4-80ac-aa138db67243" name="/etc/qemu-ifup" pid=285438 comm="qemu-system-x86" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
type=PATH msg=audit(1487347189.015:28536): item=0 name="/etc/qemu-ifup" inode=66403 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL
root@ubuntu-trusty-5773:/etc/apparmor.d/abstractions# cat /etc/apparmor.d/libvirt/libvirt-4a03fea7-e966-48e4-80ac-aa138db67243
#
# This profile is for the domain whose UUID matches this file.
#
#include <tunables/global>
profile libvirt-4a03fea7-e966-48e4-80ac-aa138db67243 {
#include <abstractions/libvirt-qemu>
#include <libvirt/libvirt-4a03fea7-e966-48e4-80ac-aa138db67243.files>
}
root@ubuntu-trusty-5773:/etc/apparmor.d/abstractions# cat /etc/apparmor.d/libvirt/libvirt-4a03fea7-e966-48e4-80ac-aa138db67243.files
# DO NOT EDIT THIS FILE DIRECTLY. IT IS MANAGED BY LIBVIRT.
"/var/log/libvirt/**/instance-00000008.log" w,
"/var/lib/libvirt/qemu/domain-instance-00000008/monitor.sock" rw,
"/var/run/libvirt/**/instance-00000008.pid" rwk,
"/run/libvirt/**/instance-00000008.pid" rwk,
"/var/run/libvirt/**/*.tunnelmigrate.dest.instance-00000008" rw,
"/run/libvirt/**/*.tunnelmigrate.dest.instance-00000008" rw,
"/var/lib/nova/instances/4a03fea7-e966-48e4-80ac-aa138db67243/console.log" rw,
"/var/lib/nova/instances/4a03fea7-e966-48e4-80ac-aa138db67243/console.log" rw,
# for qemu guest agent channel
owner "/var/lib/libvirt/qemu/channel/target/domain-instance-00000008/**" rw,
/dev/vhost-net rw,
root@ubuntu-trusty-5773:/etc/apparmor.d/abstractions# dpkg -S libvirt-qemu
libvirt-bin: /etc/apparmor.d/abstractions/libvirt-qemu
root@ubuntu-trusty-5773:/etc/apparmor.d/abstractions# dpkg -l libvirt-bin
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-=========================================-=========================-=========================-=======================================================================================
ii libvirt-bin 1.3.1-1ubuntu10.6~cloud0 amd64 programs for the libvirt library
Seeing identical behavior on Xenial
ubuntu@ubuntu-xenial-5165:~$ dpkg -l libvirt-bin
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-=========================================-=========================-=========================-=======================================================================================
ii libvirt-bin 1.3.1-1ubuntu10.8 amd64 programs for the libvirt library |
SRU - Nova
[Impact]
OpenStack deployments using vif types `tap`, `ivs`, `iovisor`, `midonet`, and `vrouter` are unable to boot instances using libvirt 1.3.1 from Ubuntu 16.04 (as used by the Newton Ubuntu Cloud Archive). Note that this impacts the nova package which is currently in yakkety-proposed/newton-proposed - the version in *-updates does not have this issue.
[Test case]
Using an OpenStack cloud deployed with one of the above SDN's boot an instance.
The instance will fail to boot with a libvirt error.
Note cloud must be deployed using the -proposed packages from the Newton UCA.
[Regression Potential]
Minimal - the patch restores the previous behaviour for older libvirt versions, ensuring compatibility with documented libvirt version baselines in OpenStack Nova.
---
SRU - libvirt
[Impact]
* Please do note that this SRU statement is about the libvirt portion
of it, this is a fix of essentially an API break from Xenial to
Yakkety. This is independent to any decision to the Openstack context
discussion about the change to drop specifying a path at all.
* Before 9c17d665fdc5f (v1.3.2 which means 1.3.1 in Xenial for us) it
was possible to have the following interface configuration:
<interface type='ethernet'/>
<script path=''/>
</interface>
This resulted in -netdev tap,script=,.. Fortunately, qemu helped
us to get away with this as it just ignored the empty script
path. However, after the commit mentioned above it's libvirtd
who is executing the script. Unfortunately without special
case-ing empty script path.
* The fix adds the special casing that qemu had into libvirts handling
of the interface definition.
[Test Case]
* That is tricky as the way openstack is using to shove that in
seems to not care on xml validation as much as e.g. virsh.
If normally adding a device like
<interface type='ethernet'/>
<script path=''/>
<model type='virtio'/>
</interface>
At least in xenial AND yakkety blocked by the XML validation.
But if trying to work around like:
<script path='""'/>
Which gave "-netdev tap,script="",id=hostnet1" on yakkety then
the fix does not apply as this is '""' and not ''.
So to add the above you have to edit it in via --skip-validate like
$ virsh edit --skip-validate zesty-on-x-test
This on onlder libvrit gave: -netdev tap,script=,id=hostnet1
Which qemu understood as nop. But newer libvirt refuses.
* Error:
error: Failed to start domain <name>
error: Cannot find '' in path: No such file or directory
* Expected:
Starting the domain as-is without calling a script,
but also without complaining about being empty.
[Regression Potential]
* Regression should be low because of:
* The fix is upstream for a while now without follow on fix
* We are essentially going back to how it was
* There is no case like "I had '' set in my setup but now it is
a no-op which makes me fail" because if one had '' it failed until
now.
* Fix is in zesty for a few days without new fallout being reported
* also it passed several levels of testing (on the case and general
regression testing)
* Due to extra xml checks a device like path='' is not even definable.
So only those who run --skip-validate or similar are affected in
the first place.
[Other Info]
* n/a
----
I have VMs failing to start with 2017-02-17 15:38:44.458 264015 ERROR nova.compute.manager [instance: 0c97ab16-2d30-43fa-b0e4-a064a842b5ed] libvirtError: internal error: process exited while connecting to monitor: 2017-02-17T15:38:43.907222Z qemu-system-x86_64: -netdev tap,ifname=tapf34ef99e-18,id=hostnet0,vhost=on,vhostfd=28: network script /etc/qemu-ifup failed with status 256
Log excerpt:
http://cdn.pasteraw.com/b3tw4cjefomfi3e9k09hvodrfun85z
Seems to be that /etc/qemu-ifup is being blocked by apparmor:
type=AVC msg=audit(1487347189.015:28536): apparmor="DENIED" operation="exec" profile="libvirt-4a03fea7-e966-48e4-80ac-aa138db67243" name="/etc/qemu-ifup" pid=285438 comm="qemu-system-x86" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
type=PATH msg=audit(1487347189.015:28536): item=0 name="/etc/qemu-ifup" inode=66403 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL
root@ubuntu-trusty-5773:/etc/apparmor.d/abstractions# cat /etc/apparmor.d/libvirt/libvirt-4a03fea7-e966-48e4-80ac-aa138db67243
#
# This profile is for the domain whose UUID matches this file.
#
#include <tunables/global>
profile libvirt-4a03fea7-e966-48e4-80ac-aa138db67243 {
#include <abstractions/libvirt-qemu>
#include <libvirt/libvirt-4a03fea7-e966-48e4-80ac-aa138db67243.files>
}
root@ubuntu-trusty-5773:/etc/apparmor.d/abstractions# cat /etc/apparmor.d/libvirt/libvirt-4a03fea7-e966-48e4-80ac-aa138db67243.files
# DO NOT EDIT THIS FILE DIRECTLY. IT IS MANAGED BY LIBVIRT.
"/var/log/libvirt/**/instance-00000008.log" w,
"/var/lib/libvirt/qemu/domain-instance-00000008/monitor.sock" rw,
"/var/run/libvirt/**/instance-00000008.pid" rwk,
"/run/libvirt/**/instance-00000008.pid" rwk,
"/var/run/libvirt/**/*.tunnelmigrate.dest.instance-00000008" rw,
"/run/libvirt/**/*.tunnelmigrate.dest.instance-00000008" rw,
"/var/lib/nova/instances/4a03fea7-e966-48e4-80ac-aa138db67243/console.log" rw,
"/var/lib/nova/instances/4a03fea7-e966-48e4-80ac-aa138db67243/console.log" rw,
# for qemu guest agent channel
owner "/var/lib/libvirt/qemu/channel/target/domain-instance-00000008/**" rw,
/dev/vhost-net rw,
root@ubuntu-trusty-5773:/etc/apparmor.d/abstractions# dpkg -S libvirt-qemu
libvirt-bin: /etc/apparmor.d/abstractions/libvirt-qemu
root@ubuntu-trusty-5773:/etc/apparmor.d/abstractions# dpkg -l libvirt-bin
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-=========================================-=========================-=========================-=======================================================================================
ii libvirt-bin 1.3.1-1ubuntu10.6~cloud0 amd64 programs for the libvirt library
Seeing identical behavior on Xenial
ubuntu@ubuntu-xenial-5165:~$ dpkg -l libvirt-bin
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-=========================================-=========================-=========================-=======================================================================================
ii libvirt-bin 1.3.1-1ubuntu10.8 amd64 programs for the libvirt library |
|
| 2017-03-31 06:29:30 |
Andy Whitcroft |
tags |
verification-done verification-newton-done |
verification-newton-done |
|
| 2017-03-31 06:29:32 |
Andy Whitcroft |
tags |
verification-newton-done |
verification-needed verification-newton-done |
|
| 2017-04-04 05:55:58 |
Christian Ehrhardt |
description |
SRU - Nova
[Impact]
OpenStack deployments using vif types `tap`, `ivs`, `iovisor`, `midonet`, and `vrouter` are unable to boot instances using libvirt 1.3.1 from Ubuntu 16.04 (as used by the Newton Ubuntu Cloud Archive). Note that this impacts the nova package which is currently in yakkety-proposed/newton-proposed - the version in *-updates does not have this issue.
[Test case]
Using an OpenStack cloud deployed with one of the above SDN's boot an instance.
The instance will fail to boot with a libvirt error.
Note cloud must be deployed using the -proposed packages from the Newton UCA.
[Regression Potential]
Minimal - the patch restores the previous behaviour for older libvirt versions, ensuring compatibility with documented libvirt version baselines in OpenStack Nova.
---
SRU - libvirt
[Impact]
* Please do note that this SRU statement is about the libvirt portion
of it, this is a fix of essentially an API break from Xenial to
Yakkety. This is independent to any decision to the Openstack context
discussion about the change to drop specifying a path at all.
* Before 9c17d665fdc5f (v1.3.2 which means 1.3.1 in Xenial for us) it
was possible to have the following interface configuration:
<interface type='ethernet'/>
<script path=''/>
</interface>
This resulted in -netdev tap,script=,.. Fortunately, qemu helped
us to get away with this as it just ignored the empty script
path. However, after the commit mentioned above it's libvirtd
who is executing the script. Unfortunately without special
case-ing empty script path.
* The fix adds the special casing that qemu had into libvirts handling
of the interface definition.
[Test Case]
* That is tricky as the way openstack is using to shove that in
seems to not care on xml validation as much as e.g. virsh.
If normally adding a device like
<interface type='ethernet'/>
<script path=''/>
<model type='virtio'/>
</interface>
At least in xenial AND yakkety blocked by the XML validation.
But if trying to work around like:
<script path='""'/>
Which gave "-netdev tap,script="",id=hostnet1" on yakkety then
the fix does not apply as this is '""' and not ''.
So to add the above you have to edit it in via --skip-validate like
$ virsh edit --skip-validate zesty-on-x-test
This on onlder libvrit gave: -netdev tap,script=,id=hostnet1
Which qemu understood as nop. But newer libvirt refuses.
* Error:
error: Failed to start domain <name>
error: Cannot find '' in path: No such file or directory
* Expected:
Starting the domain as-is without calling a script,
but also without complaining about being empty.
[Regression Potential]
* Regression should be low because of:
* The fix is upstream for a while now without follow on fix
* We are essentially going back to how it was
* There is no case like "I had '' set in my setup but now it is
a no-op which makes me fail" because if one had '' it failed until
now.
* Fix is in zesty for a few days without new fallout being reported
* also it passed several levels of testing (on the case and general
regression testing)
* Due to extra xml checks a device like path='' is not even definable.
So only those who run --skip-validate or similar are affected in
the first place.
[Other Info]
* n/a
----
I have VMs failing to start with 2017-02-17 15:38:44.458 264015 ERROR nova.compute.manager [instance: 0c97ab16-2d30-43fa-b0e4-a064a842b5ed] libvirtError: internal error: process exited while connecting to monitor: 2017-02-17T15:38:43.907222Z qemu-system-x86_64: -netdev tap,ifname=tapf34ef99e-18,id=hostnet0,vhost=on,vhostfd=28: network script /etc/qemu-ifup failed with status 256
Log excerpt:
http://cdn.pasteraw.com/b3tw4cjefomfi3e9k09hvodrfun85z
Seems to be that /etc/qemu-ifup is being blocked by apparmor:
type=AVC msg=audit(1487347189.015:28536): apparmor="DENIED" operation="exec" profile="libvirt-4a03fea7-e966-48e4-80ac-aa138db67243" name="/etc/qemu-ifup" pid=285438 comm="qemu-system-x86" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
type=PATH msg=audit(1487347189.015:28536): item=0 name="/etc/qemu-ifup" inode=66403 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL
root@ubuntu-trusty-5773:/etc/apparmor.d/abstractions# cat /etc/apparmor.d/libvirt/libvirt-4a03fea7-e966-48e4-80ac-aa138db67243
#
# This profile is for the domain whose UUID matches this file.
#
#include <tunables/global>
profile libvirt-4a03fea7-e966-48e4-80ac-aa138db67243 {
#include <abstractions/libvirt-qemu>
#include <libvirt/libvirt-4a03fea7-e966-48e4-80ac-aa138db67243.files>
}
root@ubuntu-trusty-5773:/etc/apparmor.d/abstractions# cat /etc/apparmor.d/libvirt/libvirt-4a03fea7-e966-48e4-80ac-aa138db67243.files
# DO NOT EDIT THIS FILE DIRECTLY. IT IS MANAGED BY LIBVIRT.
"/var/log/libvirt/**/instance-00000008.log" w,
"/var/lib/libvirt/qemu/domain-instance-00000008/monitor.sock" rw,
"/var/run/libvirt/**/instance-00000008.pid" rwk,
"/run/libvirt/**/instance-00000008.pid" rwk,
"/var/run/libvirt/**/*.tunnelmigrate.dest.instance-00000008" rw,
"/run/libvirt/**/*.tunnelmigrate.dest.instance-00000008" rw,
"/var/lib/nova/instances/4a03fea7-e966-48e4-80ac-aa138db67243/console.log" rw,
"/var/lib/nova/instances/4a03fea7-e966-48e4-80ac-aa138db67243/console.log" rw,
# for qemu guest agent channel
owner "/var/lib/libvirt/qemu/channel/target/domain-instance-00000008/**" rw,
/dev/vhost-net rw,
root@ubuntu-trusty-5773:/etc/apparmor.d/abstractions# dpkg -S libvirt-qemu
libvirt-bin: /etc/apparmor.d/abstractions/libvirt-qemu
root@ubuntu-trusty-5773:/etc/apparmor.d/abstractions# dpkg -l libvirt-bin
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-=========================================-=========================-=========================-=======================================================================================
ii libvirt-bin 1.3.1-1ubuntu10.6~cloud0 amd64 programs for the libvirt library
Seeing identical behavior on Xenial
ubuntu@ubuntu-xenial-5165:~$ dpkg -l libvirt-bin
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-=========================================-=========================-=========================-=======================================================================================
ii libvirt-bin 1.3.1-1ubuntu10.8 amd64 programs for the libvirt library |
SRU - Nova
[Impact]
OpenStack deployments using vif types `tap`, `ivs`, `iovisor`, `midonet`, and `vrouter` are unable to boot instances using libvirt 1.3.1 from Ubuntu 16.04 (as used by the Newton Ubuntu Cloud Archive). Note that this impacts the nova package which is currently in yakkety-proposed/newton-proposed - the version in *-updates does not have this issue.
[Test case]
Using an OpenStack cloud deployed with one of the above SDN's boot an instance.
The instance will fail to boot with a libvirt error.
Note cloud must be deployed using the -proposed packages from the Newton UCA.
[Regression Potential]
Minimal - the patch restores the previous behaviour for older libvirt versions, ensuring compatibility with documented libvirt version baselines in OpenStack Nova.
---
SRU - libvirt
[Impact]
* Please do note that this SRU statement is about the libvirt portion
of it, this is a fix of essentially an API break from Xenial to
Yakkety. This is independent to any decision to the Openstack context
discussion about the change to drop specifying a path at all.
* Before 9c17d665fdc5f (v1.3.2 which means 1.3.1 in Xenial for us) it
was possible to have the following interface configuration:
<interface type='ethernet'/>
<script path=''/>
</interface>
This resulted in -netdev tap,script=,.. Fortunately, qemu helped
us to get away with this as it just ignored the empty script
path. However, after the commit mentioned above it's libvirtd
who is executing the script. Unfortunately without special
case-ing empty script path.
* The fix adds the special casing that qemu had into libvirts handling
of the interface definition.
[Test Case]
* That is tricky as the way openstack is using to shove that in
seems to not care on xml validation as much as e.g. virsh.
If normally adding a device like
<interface type='ethernet'>
<script path=''/>
<model type='virtio'/>
</interface>
At least in xenial AND yakkety blocked by the XML validation.
But if trying to work around like with path='""'
this gives "-netdev tap,script="",id=hostnet1" on yakkety then
the fix does not apply as this is '""' and not ''.
So to add the above snippet you have to edit it in via --skip-
validate like
$ virsh edit --skip-validate zesty-on-x-test
This on older libvrit gave: -netdev tap,script=,id=hostnet1
Which qemu understood as nop. But newer libvirt refuses.
* Error:
error: Failed to start domain <name>
error: Cannot find '' in path: No such file or directory
* Expected:
Starting the domain as-is without calling a script,
but also without complaining about being empty.
[Regression Potential]
* Regression should be low because of:
* The fix is upstream for a while now without follow on fix
* We are essentially going back to how it was
* There is no case like "I had '' set in my setup but now it is
a no-op which makes me fail" because if one had '' it failed until
now.
* Fix is in zesty for a few days without new fallout being reported
* also it passed several levels of testing (on the case and general
regression testing)
* Due to extra xml checks a device like path='' is not even definable.
So only those who run --skip-validate or similar are affected in
the first place.
[Other Info]
* n/a
----
I have VMs failing to start with 2017-02-17 15:38:44.458 264015 ERROR nova.compute.manager [instance: 0c97ab16-2d30-43fa-b0e4-a064a842b5ed] libvirtError: internal error: process exited while connecting to monitor: 2017-02-17T15:38:43.907222Z qemu-system-x86_64: -netdev tap,ifname=tapf34ef99e-18,id=hostnet0,vhost=on,vhostfd=28: network script /etc/qemu-ifup failed with status 256
Log excerpt:
http://cdn.pasteraw.com/b3tw4cjefomfi3e9k09hvodrfun85z
Seems to be that /etc/qemu-ifup is being blocked by apparmor:
type=AVC msg=audit(1487347189.015:28536): apparmor="DENIED" operation="exec" profile="libvirt-4a03fea7-e966-48e4-80ac-aa138db67243" name="/etc/qemu-ifup" pid=285438 comm="qemu-system-x86" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
type=PATH msg=audit(1487347189.015:28536): item=0 name="/etc/qemu-ifup" inode=66403 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL
root@ubuntu-trusty-5773:/etc/apparmor.d/abstractions# cat /etc/apparmor.d/libvirt/libvirt-4a03fea7-e966-48e4-80ac-aa138db67243
#
# This profile is for the domain whose UUID matches this file.
#
#include <tunables/global>
profile libvirt-4a03fea7-e966-48e4-80ac-aa138db67243 {
#include <abstractions/libvirt-qemu>
#include <libvirt/libvirt-4a03fea7-e966-48e4-80ac-aa138db67243.files>
}
root@ubuntu-trusty-5773:/etc/apparmor.d/abstractions# cat /etc/apparmor.d/libvirt/libvirt-4a03fea7-e966-48e4-80ac-aa138db67243.files
# DO NOT EDIT THIS FILE DIRECTLY. IT IS MANAGED BY LIBVIRT.
"/var/log/libvirt/**/instance-00000008.log" w,
"/var/lib/libvirt/qemu/domain-instance-00000008/monitor.sock" rw,
"/var/run/libvirt/**/instance-00000008.pid" rwk,
"/run/libvirt/**/instance-00000008.pid" rwk,
"/var/run/libvirt/**/*.tunnelmigrate.dest.instance-00000008" rw,
"/run/libvirt/**/*.tunnelmigrate.dest.instance-00000008" rw,
"/var/lib/nova/instances/4a03fea7-e966-48e4-80ac-aa138db67243/console.log" rw,
"/var/lib/nova/instances/4a03fea7-e966-48e4-80ac-aa138db67243/console.log" rw,
# for qemu guest agent channel
owner "/var/lib/libvirt/qemu/channel/target/domain-instance-00000008/**" rw,
/dev/vhost-net rw,
root@ubuntu-trusty-5773:/etc/apparmor.d/abstractions# dpkg -S libvirt-qemu
libvirt-bin: /etc/apparmor.d/abstractions/libvirt-qemu
root@ubuntu-trusty-5773:/etc/apparmor.d/abstractions# dpkg -l libvirt-bin
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-=========================================-=========================-=========================-=======================================================================================
ii libvirt-bin 1.3.1-1ubuntu10.6~cloud0 amd64 programs for the libvirt library
Seeing identical behavior on Xenial
ubuntu@ubuntu-xenial-5165:~$ dpkg -l libvirt-bin
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-=========================================-=========================-=========================-=======================================================================================
ii libvirt-bin 1.3.1-1ubuntu10.8 amd64 programs for the libvirt library |
|
| 2017-04-04 05:57:02 |
Christian Ehrhardt |
tags |
verification-needed verification-newton-done |
verification-done verification-newton-done |
|
| 2017-04-04 07:37:56 |
James Page |
cloud-archive/newton: status |
Fix Committed |
Fix Released |
|
| 2017-04-05 15:01:53 |
Launchpad Janitor |
nova (Ubuntu Yakkety): status |
Fix Committed |
Fix Released |
|
| 2017-04-10 23:12:34 |
OpenStack Infra |
nova/ocata: status |
In Progress |
Fix Committed |
|
| 2017-04-21 04:34:30 |
Launchpad Janitor |
libvirt (Ubuntu Yakkety): status |
Fix Committed |
Fix Released |
|
| 2017-04-26 21:29:29 |
OpenStack Infra |
nova/newton: status |
In Progress |
Fix Committed |
|
