/etc/qemu-ifup not allowed by apparmor

/etc/qemu-ifup is a non-standard command. Can you give details of how you setup your system to use this?

WORKAROUND: add the following to /etc/apparmor.d/abstractions/libvirt-qemu:
/etc/qemu-ifup ixr,

and restart your VMs.

That has been my challenge this morning in figuring this out. As far as I know, I don't use it or need what it provides. I can't find anything in /etc/libvirt or even /etc or my Openstack venv that is calling toward qemu-ifup. I grepped thru /usr looking for 'qemu-ifup' and the only hits were from the /usr/share/doc/qemu-system-*/common/qemu-doc.html mentions.

I began to suspect a possible packaging issue when I saw that it is sourced from upstream Ubuntu packaging, so that's what prompted this bug:

root@ubuntu-trusty-5773:~# dpkg -S /etc/qemu-ifup
qemu-system-common: /etc/qemu-ifup
root@ubuntu-trusty-5773:~# dpkg -l qemu-system-common
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-=========================================-=========================-=========================-=======================================================================================
ii qemu-system-common 1:2.5+dfsg-5ubuntu10.5~cl amd64 QEMU full system emulation binaries (common files)

Also thanks for the workaround suggestion. I was looking at adding a workaround like that to the abstraction earlier but wasn't sure on the syntax :)

I wonder if https://review.openstack.org/#/c/425637/ has broken this on my version of libvirt by removing the script='' entry, which is maybe causing qemu to revert to its default of calling /etc/qemu-ifup, which as we know is not allowed by apparmor.

Thank you for responding. Marking back to New so a member of the cloud team can take a further look.

Hi Logan, thank you for your report.
And also tanks Jamie to already provide a workaround for thew apparmor case.

We just SRUed a fix to Xenial to "allow execution of those scripts".
See https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1620407

From your report I see that you likely have Clou-Archive Mitaka which gave you access to the same SRU of libvirt to 1.3.1-1ubuntu10.8.

It very much confuses me that for your case it now seems to prohibiting such an execution.
As that is just inverse to what the fix does. Never the less it is too related to consider it just an accident.

Adding Cloud-Archive task to get their expertise as well.

Marking Critical as it is a potential SRU-regression

Let me outline on the /etc/qemu-ifup a bit - or at least my understanding of it.
If you happen to run "type=ethernet" networking the script gets to be important.
See https://libvirt.org/formatdomain.html#elementsNICSEthernet
In general this is not recommended for various security reasons (you have to give libvirt/qemu more permissions/capabilities to get this working).
Now if you have type=ethernet and no script set, then the default is /etc/qemu-ifup, maybe that is how it gets this "config" in your case.

The Openstack commit you referred to could absolutely be related as well.
But it should not be in your software stack in your case right?

If applied this would make the xml instead `script=""` not set anything and yes - in that case it would then fall back to /etc/qemu-ifup which might tickle down to showing your issue.

The function the upstream fix relates to in Openstack is:
def set_vif_host_backend_ethernet_config(conf, tapname):
    """Populate a LibvirtConfigGuestInterface instance
    with host backend details for an externally configured
    host device.

Before the change it was part of "get_config(self, instance, network, mapping)" in
"class LibvirtOpenVswitchDriver(LibvirtBaseVIFDriver)"

I'm not sure but does that imply it might trigger in any openvswitch setup?
Some of our tests should have had the same then.
I'll rerun some of those later to be sure.

The libvirt change on the other hand "only" changes to NOT refuse guests right away that have a script= attribute set. See the referred bug for details.
Now it does no more refuse to start those - that should (tm) not cause the issue. Maybe it really is the openstack commit you referred to (causing it to fallback to the default).
But as I said I'd wonder if that commit is active in your Software stack at all.

I'm going off some more testing with that in mind what I documented on the bug already.

I really want to understand why it hits you now and if this is a SRU-regression, so here some questions.

Questions to Logan:
- Could you share your guest XML to help me to understand what combination drives your guest into being stopped by this apparmor DENY now?
- Did the workaround suggested by Jamie fix it for you?
- Could you also try downgrading to 1.3.1-1ubuntu10.7 if that get you working again?
- I have made assumptions, but want to confirm what are the releases you are using for Ubuntu, Cloud-Archive, Openstack?

Questions to the Cloud Archive Team:
- It didn't in mine, but did that show up in any of your usual regular tests?
- is the referred Openstack commit active in any of our cloud-archive release (if so in which)?
- Can we reproduce that somehow to iterate more quickly on it?

I was testing various type=ethernet XML configurations.

Cases:
defaultpath => <script path='/etc/qemu-ifup'/>
emptyattrib => <script path=''/>
noattrib => no script tag at all

The target statement which the error of the known bug refers to is optional, so add another set of cases with
the same three again without a <target ...> attribute called "notgt-*".

                         Pre-Fix Post-Fix
default bug 1620407 working
empty bug 1620407 working
no bug 1620407 still bug 1620407*
notgt-default working working
notgt-empty can't be defined can't be defined
notgt-no working working

*We fixed bug 1620407 with a mimimal fix intentionally, to the "no" case is "ok" to still fail.

Now the Openstack case should (IMHO) be one of the "empty" cases before the fix to openstack that was referred.
That is the path='', since notgt-empty can't be defined (xml validation) it has to be the normal "empty" case.
After the fix it should be one of the 'no' cases.

But all cases either stayed as-is or were fixed, so I don't know.
Also I had no apparmor DENIES along any of that - even when using explicitly in the *default cases.

I really need the XML that is generated to understand what might be going on.
Also please help to answer the questions I listed in commend #10

I think I found the guest definition in the attached raw log, analyzing ...

The important snippet is:
<interface type='ethernet'>
      <mac address='fa:16:3e:3b:8b:2b'/>
      <target dev='tapf34ef99e-18'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>

That is my "no" case which should still fail the same way.
Yet I think I need to run that with an externally created tap device to fully match the Openstack+Openvswitch case. Doing that now ...

Still please help to answer all the other questions in comment #10.

Even with that things are still working correctly for me:
Here the "no" case re-ran with the device being set up externally.

$ tunctl -t manualdevname
$ ip link set manualdevname up
$ ip link set manualdevname master virbr0

For the config:
    <interface type='ethernet'>
      <mac address='52:54:00:18:0d:a3'/>
      <target dev='manualdevname'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </interface>

This case worked prior and after the fix in 1.3.1-1ubuntu10.8 for me then.

FYI - while not confirmed I marked it regression-update for now until we know for sure it is not (to get the right attention).
I also pinged some Cloud Archive people on IRC to take a look.

Just wanted to make one more thing clear, the libvirt in Xenial (and here in the bug) is 1.3.1.
That means it is pre http://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=9c17d66
So the prereq to need https://review.openstack.org/#/c/425637/ is not given.
Maybe those two bite each other, but I still need some way to reproduce to fully understand.

My remaining check would be to run on Trusty+UCA?, but for that I need more confirmations what exactly is used and also another system to do so.

Summarizing open questions to stop referring back:

Questions to Logan:
- Did the workaround suggested by Jamie fix it for you?
- Could you also try downgrading to 1.3.1-1ubuntu10.7 (or before) if that get you working again?
- I have made assumptions so far, but want to confirm what are the releases you are using for Ubuntu, Cloud-Archive, Openstack?

Questions to the Cloud Archive Team:
- It did not show up in explicit or automated tests, did that show up in any of your usual tests?
- is the referred Openstack commit active in any of our cloud-archive release (if so in which)?
- Can you reproduce that somehow to iterate more quickly on it?

I have checked with our support people, but they somewhat calmed me down that this is not lighting up everywhere (?yet?). So keeping the state as is for now.

Hi Christian,

I have not tested the workaround yet. What I have done since I posted this was tried to narrow it down to the specific nova commit that I suspected triggered this. To be specific, I am deploying Newton using OpenStack-Ansible.

OSA pulls nova directly from upstream openstack git sources and does not consume Ubuntu Cloud Archive for that. However, the libvirt installed is sourced from UCA. For OSA newton on trusty, libvirt is sourced from UCA's trusty/mitaka repo. For OSA newton on xenial, libvirt comes from UCA xenial/newton.

Without changing the libvirt being installed, I have confirmed the specific nova change that triggers this is https://review.openstack.org/#/c/425637/. The guest xml that nova generates when this commit is applied is in the OP (http://cdn.pasteraw.com/b3tw4cjefomfi3e9k09hvodrfun85z). When I revert my nova SHA to the parent commit of this change, b51231c638228f67ab130a7855b9143b202733f6, without changing any libvirt bits, the VMs launch as expected since they now contain the "<script path=''/>" bits in the network interface xml.

Without digging in to libvirt too much, it seems like it should be fairly easy to reproduce by installing the latest xenial/newton or trusty/mitaka UCA sourced libvirt, and launching a VM with not "<script path=''/>" XML on the ethernet vif.

I am limited on how much time I can commit to testing other changes at this time, but I will try to test the workaround and/or the previous libvirt version you suggested asap. Hopefully this will help fill in the blanks but if you have any other questions I missed let me know!

Just found that the report was for 1.3.1-1ubuntu10.6~cloud0 and only stated to "also apply" to 1.3.1-1ubuntu10.8. By that dropping the regression-update tag.

I tried to puzzle together the timeline on this.
Thanks rbasask for discussing with me to refocus on this issue.

Timeline:
#1 libvirt passed script to qemu, qemu executed
   1.3.1 as in Xenial or UCA-Mitaka still do that
   But Openstack passed script='' and qemu silently ignored it

#2 libvirt changed, now libvirt executes
   http://libvirt.org/git/?p=libvirt.git;a=commit;h=9c17d665fdc5f
   That is in Yakkety and later.
   This had an unintentional API change, that empty scripts behave differently.

#3 Openstack adapted to that API change
   https://review.openstack.org/#/c/425637/
   Not sure - is that in Ocata only - commit in 2017?
   Now new Openstack (#3) + New Libvirt (#2) work
   But if you happen to have an old libvirt like in #1 you now have different behavior.

#4 Upstream libvirt realizes the API break and fixes it
   http://libvirt.org/git/?p=libvirt.git;a=commit;h=1d9ab0f04af310e52f80b4281751655bb3bb7601
   But backporting that would not help, this is meant for libvirt later or equal to #2

#5 IMHO openstack should either
   - detect libvirt version and do differently depending on that (keep script='' for old ones)
   - or instead of not passing script at all pass /bin/true which will work on libvirt as old as #1

I expect you have an openstack of #4 and a libvirt of #1 which due to that cause this.
I still don't see the apparmor issue on my end, but that might be an additional issue.
Even in the /bin/true case we might hit an apparmor on /bin/true.

Please everybody still try to hep sorting out questions in comment #16.

I'm having a hard time testing the workaround in #1 -- is there an additional step necessary after modifying the abstractions/libvirt-qemu file with the additional rule? Ie. some command to reload the file? I restarted apparmor, libvirt-bin, and nova-compute services and it is still failing with the same message despite the line being added to the apparmor abstraction file.

Downgrading to 1.3.1-1ubuntu10.7 on Xenial did not fix the issue for me.

I also tested hacking in conf.script = "/bin/true" to nova. You suspected it might fail with apparmor preventing /bin/true execution, I can confirm that it did indeed fail.

type=AVC msg=audit(1487951915.530:99345): apparmor="DENIED" operation="exec" profile="libvirt-5ea8f14c-73c8-4e21-9f64-28d60c1919c6" name="/bin/true" pid=802296 comm="qemu-system-x86" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
type=PATH msg=audit(1487951915.530:99345): item=0 name="/bin/true" inode=44 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL

On Fri, Feb 24, 2017 at 4:50 PM, Logan V <email address hidden> wrote:

> is there an
> additional step necessary after modifying the abstractions/libvirt-qemu
> file with the additional rule? Ie. some command to reload the file? I
> restarted apparmor, libvirt-bin, and nova-compute services and it is
> still failing with the same message despite the line being added to the
> apparmor abstraction file.
>

See https://help.ubuntu.com/lts/serverguide/apparmor.html
But you stated you restarted apparmor already.

The workflow is that on guest creation it
uses /etc/apparmor.d/libvirt/TEMPLATE.qemu to build a guest specific file.
That is then loaded for the guests qemu and at that point the (now changed)
abstraction is pulled in.

--
Christian Ehrhardt
Software Engineer, Ubuntu Server
Canonical Ltd

Yep, just to confirm with the suggested workaround applied and restarting apparmor + rebooting the system it did not work.

Thanks to you already listing an example of a full profile from your system in the description and a libvirt xml in http://cdn.pasteraw.com/b3tw4cjefomfi3e9k09hvodrfun85z both seem pretty standard and includes the abstractions/libvirt-qemu.

Maybe when you added "/etc/qemu-ifup ixr," you added it to the end which might be inside the qemu_bridge_helper child profile - it should go above that.

@Logan - In any way it might be helpful if you could share the modified abstraction file as well just to be able to have a look if there would be anything odd in your case.

@Logan - are you running upstream Openstack or a packaged one?

@Logan - you said "I also tested hacking in conf.script = "/bin/true" to nova." If you'd happen to set this to conf.script = "" I'd hope that the effective generated guest config would match what openstack did before https://review.openstack.org/#/c/425637/ and by that make it work with the libvirt 1.3.1 version again. Could you try that as well?

@Cloud-Archive Team - did you do anything special in Xenial-Ocata to avoid that issue in general?

I must've done something wrong when originally testing the abstraction fix. I re-tested based on your advice and it does indeed fix the problem. The abstraction file that I just tested, which worked, is http://cdn.pasteraw.com/9u16uk9938mp5t3x530ok0w791nkooc.

Regarding upstream vs. packaged, I went into some detail on that in #18, but basically it is upstream Openstack sourced from git, with libvirt and non-openstack bits sourced from cloud-archive.

Also yes I can confirm that reverting nova to the empty string behavior results in a working system:
    <interface type='ethernet'>
      <mac address='fa:16:3e:23:fe:4d'/>
      <script path=''/>
      <target dev='tap7f277872-98'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>

Thank you so much for confirming that Logan!
That kind of reset my sanity on understanding the issue.

But I wonder that if one uses cloud archive Ocata he would also get this combo IMO.
It would use Ocata and the libvirt 2.5 th you also have - just as you do with your combo.

I don't know why the Cloud Team is not seeing this, and I'd like to hear their opinion before releasing a fix. I'm explicitly subscribing some of them to poll their statements.

At the same time this does not stop me from preparing and testing a fix myself and with your help.
I'm preparing something and will update here once ready.

So we have:
- old openstack: sets script=''
- new openstack: sets nothing
- old libvirt: passes to qemu, which does nothing on ''
- new libvirt: executes script, but can't handle ''
- any libvirt: if nothing is set defaults to /etc/qemu-ifup

Instead of the rule to allow that to qemu, I'd prefer to backport the libvirt fix. Because essentially '' does not mean "run qemu/ifup" which would be what the apparmor change allows.
We want back to '' when set really executing nothing.
Well that will at least "allow" it to handle script='' correctly.

IMHO newer openstack is broken as setting nothing implies /etc/qemu-ifup which might not be what they wanted. Never the less from the libvirt perspective we want to allow that.

But here my brain runs into a knot while trying to prep a patch.
In your case Logan, the newer Openstack sets nothing which implies /etc/qemu-ifup.
That should be executed by libvirt which should run under the apparmor profile usr.sbin.libvirtd.
Since you can "fix" your case by adding to the libvirt-qemu abstraction shouldn't that be qemu executing it in your case.

Reading your Deny log again comm="qemu-system-x86".
Hmm, could it be that you run the "new" openstack against the "old" libvirt?
I see you said "with libvirt and non-openstack bits sourced from cloud-archive", but then I'd have expected the apparmor fail against libvirt - if any.

Don't get me wrong we might still want to backport the fix for libvirt, but I want to understand where to fix apparmor rules would be correct.

@Logan:
- Could you report the output of:
  $ dpkg -l 'libvirt-bin' 'libvirt-daemon-system' 'qemu-kvm'
- You said you run Cloud Archive - might I ask which one at the moment?

Note: a backport of the upstream libvirt fix is building here - https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/2559
Feel free to test, but I really think we need to understand the dependencies to consider the apparmor change at the right place first.

@Christian: what does the default /etc/qemu-ifup do?

Although I'm inclined to accept your argument that my recent change to Nova was a step in the wrong direction, I'm wondering why it seems to work (at least in my testing) in practice.

Also, if there was now a change in libvirt to interpret '' as "do nothing", is there something straightforward we could put in the Nova code to detect when libvirt has that fix - and so set '' - and when it doesn't - in which case best to set None.

Hi Neil,

1) on what /etc/qemu.ifup (as being the default) actually does
You can look at Debian/Ubuntus script (unchanged since 2013)
https://anonscm.debian.org/cgit/pkg-qemu/qemu.git/tree/debian/qemu-ifup.linux
AFAIK Fedora provides no default, but in most cases it is meant for custom scripts anyway, e.g. https://en.wikibooks.org/wiki/QEMU/Networking#qemu-ifup as a base to create such.

From there the TL;DR for Debian/Ubuntu is in the top quote:
 # Script to bring a network (tap) device for qemu up.
 # The idea is to add the tap device to the same bridge
 # as we have default routing to.
It has various paths with exit as a no-op e.g. if no bridge is there.

2) Why it works in tests
I outlined several combinations of libvirt and openstack above (see the timeline in comment #20) that work well together. If you are on any of those your testing will not show issues.
Also the "error" is not this script failing. It very likely would work just doing nothing or something useless in the case here (yet it is dangerous to rely on that). The issue reported here is that due to the unexpected calling of /etc/qemu-ifup (due to now path being not set) the apparmor protection kicks in and blocks it. So if you tested on a platform without apparmor or with other rules you would not have seen it.
Yet it is correct by apparmor to do so - a user is usually meant to add exceptions if needed (as the workaround for Logan does.)

3)
On detection of the fix that lets libvirt handle "" as no-op - to my knowledge - there is no externally visible thing to check.
It will be upstream with libvirt 3.1 so you can assume it is there starting from this.
But you'll never know who will have it backported or not.

4)
IMHO there is one little piece to the puzzle still missing which is want to understand why in Logans case qemu is calling it (as in newer libvirt it should do it) - see comment #28.
So I'd have expected libvirt saying "error executing '' or anything like it, not the apparmor issue he hit for qemu."
But I'll have to wait on his reply for that.

Sorry for the delay on getting this info.

root@ubuntu-xenial-6494:~# dpkg -l 'libvirt-bin' 'libvirt-daemon-system' 'qemu-kvm'
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-===================================-======================-======================-===========================================================================
ii libvirt-bin 1.3.1-1ubuntu10.8 amd64 programs for the libvirt library
un libvirt-daemon-system <none> <none> (no description available)
un qemu-kvm <none> <none> (no description available)

root@ubuntu-xenial-6494:~# cat /etc/apt/sources.list.d/mirror_lstn_net_ubuntu_cloud_archive.list
deb http://mirror.lstn.net/ubuntu-cloud-archive xenial-updates/newton main

Also in regard to: "Reading your Deny log again comm="qemu-system-x86".
Hmm, could it be that you run the "new" openstack against the "old" libvirt?
I see you said "with libvirt and non-openstack bits sourced from cloud-archive", but then I'd have expected the apparmor fail against libvirt - if any."

This is Newton UCA on Xenial running against stable/newton Nova. The openstack bits happen to be sourced from upstream (non UCA), but if UCA is pulling in stable/newton release tags at some point then I'm sure it will hit this eventually too.

Ok, that confirms my theory of new Openstack vs (too) old libvirt.
But IIRC UCA Newton should be Newton form openstack and virtualization bits from Yakkety Ubuntu release.
That would be libvirt 2.1.0-1ubuntu9.1

$ lxc launch ubuntu-daily:xenial xenial-uca-newton
$ lxc exec xenial-uca-newton -- add-apt-repository cloud-archive:newton
$ lxc exec xenial-uca-newton -- apt update
$ lxc exec xenial-uca-newton -- apt-cache policy libvirt-bin
libvirt-bin:
     1.3.1-1ubuntu10.8 500
        500 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages

Ok so Newton UCA really has not Yakketies libvirt.
But you are running Ocata level code, so their the UCA version would run into the same not having a newer libvirt.
$ lxc launch ubuntu-daily:xenial xenial-uca-ocata
$ lxc exec xenial-uca-ocata -- add-apt-repository cloud-archive:ocata
$ lxc exec xenial-uca-ocata -- apt update
$ lxc exec xenial-uca-ocata -- apt-cache policy libvirt-bin
libvirt-bin:
  Installed: (none)
  Candidate: 2.5.0-3ubuntu3~cloud0
  Version table:
     2.5.0-3ubuntu3~cloud0 500
        500 http://ubuntu-cloud.archive.canonical.com/ubuntu xenial-updates/ocata/main amd64

Here we go.
Please don't ask me why the libvirt 2.1 is not in the Newton UCA, that is up to the Cloud Archives Teams reasons and decisions. But they are subscribed and can answer that.

That explains why we are not seeing the issue when using Newton (not having the Openstack change) nor with Ocata (has Openstack but also a newer libvirt) - both cases match.
A cross version usage is not supported by UCA as far as I know.

So if you are running "ocata-level" upstream you should switch to UCA-Ocata and things hopefully should just work.

It is still right to fix the empty script execution (#4 in my timeline post) in the version that is in Zesty. Tests are mostly good, should hit zesty-proposed soon.

As it is not called on Yakkety in any supported case there is no real reason to SRU there. And it won't help you as we just clarified that the UCA-Newton still has the libvirt of Xenial.

This bug was fixed in the package libvirt - 2.5.0-3ubuntu4

---------------
libvirt (2.5.0-3ubuntu4) zesty; urgency=medium

  * d/p/ubuntu/qemu-Allow-empty-script-path-to-interface.patch: in the past
    it was possible to have <script path=''/> which now fails - fix to match
    the old behavior (LP: #1665698)

 -- Christian Ehrhardt <email address hidden> Fri, 10 Mar 2017 08:57:18 +0100

Will the libvirt fix be back-ported? We can't effectively use noscript in OpenStack with any versions missing the fix.

The change in Openstack is in Ocata only, if you run Ocata on Zesty or Xenial+UCA-Ocata you should be good already with the fix above.

I'm considering to backport it to Yakkety where libvirt is in the broken state (script is handled by libvirt, but it doesn't know about "" being meant to be a no-op). Looking into that somewhen this week.

But I won't backport to Xenial (makes no sense, the code it fixes isn't there at all).
And that means UCA-Newton on Xenial as well won't get a fix as it does not apply.
I'm open to a community driven fix for it in Xenial, but there IMO the override in the apparmor is more likely the right way to do it.

The problem that is left IMHO is that the openstack change from passing 'path=""' to "not passed at all" is suboptimal, as this is "defined" in qemu/libvirt as "fall back to default path". But that fix would have to be in Openstack.

The OpenStack change is not just in Ocata. It was also backported to Newton. Please note the merged OpenStack patch in #4 is on branch stable/newton. We need this fix in Newton UCA as well please.

Having read through the history on this bug I'm a little confused; Newton UCA does not contain any version of libvirt - Newton deployments on Xenial will use the libvirt version shipped in Xenial itself, which I think is the same version that will be used in all gates for OpenStack so I'm not clear on exactly what needs 'fixing' for the Newton UCA.

This is also reflected in the latest set of tests for Newton SRU's which are currently in-flight - I'm not seeing any unexpected test failures from tempest for the proposed packages which include the changes referenced on the stable/newton branch.

Doe the change in behaviour in OpenStack only impact on certain vif types?

Yes it only affects the "tap" vif type.

http://lists.openstack.org/pipermail/openstack-dev/2016-December/109080.html and https://bugs.launchpad.net/nova/+bug/1649527 contain additional info about the bug and nova fix.

Add what's the baseline libvirt version compatibility that nova documents?

Looks like that is defined here: https://github.com/openstack/nova/blob/stable/newton/nova/virt/libvirt/driver.py#L190-L208

newton and ocata are both:
MIN_LIBVIRT_VERSION = (1, 2, 1)
MIN_QEMU_VERSION = (1, 5, 3)

OK so reading the code and reading:

  https://wiki.openstack.org/wiki/LibvirtDistroSupportMatrix

The Newton nova libvirt driver should maintain compatibility with 1.2.1, with Pike being compatible with 1.2.9 and Queens with 1.3.1 (the current Xenial version).

Marking Cloud Archive task as Invalid (as the Newton UCA does not have any libvirt/qemu packages).

Adding bug task for Nova as its not currently compatible in stable/newton with the documented minimum libvirt version of 1.2.1.

I believe the fixes for bug 1649527 fixed things for libvirt >= 1.3.3, but broke compatibility with the documented baseline version of libvirt for Nova breaking drivers that use the 'tap' vif type on Ubuntu Xenial.

To summarize:

Before 1.3.3)
- qemu executed the script
- passing: "" got to qemu and it and knew about it being a nop
- not passing anything: means qemu will run the default path /etc/qemu-ifup

Since 1.3.3)
- libvirt executes the script
- passing "": libvirt does not know "" should be a nop and might have had issues
- not passing anything: does nothing (= nop)

Since 3.1
- passing: "": libvirt now knows "" it is meant to be a nop
- not passing anything: does nothing (still a nop)

Because openstack wants libvirt/qemu to do nothing, I think with the openstack change in place it is only compat >=1.3.3 where "not passing anything" is a nop.

(FTR we'll hold bug 1664306 - Newton stable point releases - until this is resolved one way or the other).

FYI on the delay for yakkety: To fix the issue of "" being a nop in yakkety as well requires the former SRU to complete first. But that is totally orthogonal to any of the Openstack considerations we did in the last few comments.

The former SRU was waiting in unapproved still while this SRU was already prepared here.
I checked with SRU Team and so we rejected the current upload to bundle it with the next one.

This is now as 2.1.0-1ubuntu9.3 in unapproved queue for yakkety.

To clarify my understanding of this, per comment 50 and the release note that went with the change:

https://review.openstack.org/#/c/425637/1/releasenotes/notes/libvirt-script-with-empty-path-2b49caa68b05278d.yaml

We basically want:

if libvirt < 1.3.3:
   script.path = '' (as before the change)
else:
   script.path = None (as in the change)

Correct?

@mriedem

Yes I believe that's correct.

On Tue, Mar 21, 2017 at 5:19 PM, Matt Riedemann <email address hidden>
wrote:

> if libvirt < 1.3.3:
> script.path = '' (as before the change)
> else:
> script.path = None (as in the change)
>

Sounds right to me at least.

Fix proposed to branch: master
Review: https://review.openstack.org/448203

Fix proposed to branch: stable/ocata
Review: https://review.openstack.org/448242

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/448253

Reviewed: https://review.openstack.org/448203
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=a41d265a19b7bcb1af8fc179bf864e00023c6cc6
Submitter: Jenkins
Branch: master

commit a41d265a19b7bcb1af8fc179bf864e00023c6cc6
Author: Matt Riedemann <email address hidden>
Date: Tue Mar 21 13:18:08 2017 -0400

    libvirt: conditionally set script path for ethernet vif types

    Change I4f97c05e2dec610af22a5150dd27696e1d767896 worked around
    a change introduced in libvirt 1.3.3 where the script path on
    a LibvirtConfigGuestInterface could not be the emptry string
    because libvirt would literally take that as the path and couldn't
    resolve it, when in fact it used to indicate to libvirt that the
    script path is a noop. This has been fixed in libvirt 3.1.

    On Ubuntu with libvirt<1.3.3, if the script path is None then
    it defaults to /etc/qemu-ifup which is blocked by AppArmor.

    So this change adds a conditional check when setting the script
    path value based on the libvirt version so we can straddle releases.

    Change-Id: I192c61b93bd3736fdfe16b6a6906d58997d3eef9
    Closes-Bug: #1665698
    Related-Bug: #1649527

Due to libvirt/nova combinations this bug does not impact either zesty or the ocata UCA (marking tasks Invalid).

Ubuntu SRU Information

[Impact]
OpenStack deployments using vif types `tap`, `ivs`, `iovisor`, `midonet`, and `vrouter` are unable to boot instances using libvirt 1.3.1 from Ubuntu 16.04 (as used by the Newton Ubuntu Cloud Archive). Note that this impacts the nova package which is currently in yakkety-proposed/newton-proposed - the version in *-updates does not have this issue.

[Test case]
Using an OpenStack cloud deployed with one of the above SDN's boot an instance.
The instance will fail to boot with a libvirt error.
Note cloud must be deployed using the -proposed packages from the Newton UCA.

[Regression Potential]
Minimal - the patch restores the previous behaviour for older libvirt versions, ensuring compatibility with documented libvirt version baselines in OpenStack Nova.

Hello Logan, or anyone else affected,

Accepted nova into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/nova/2:14.0.4-0ubuntu1.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Logan, or anyone else affected,

Accepted libvirt into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libvirt/2.1.0-1ubuntu9.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Logan, or anyone else affected,

Accepted nova into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/nova/2:14.0.4-0ubuntu1.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hello Logan, or anyone else affected,

Accepted nova into newton-proposed. The package will build now and be available in the Ubuntu Cloud Archive in a few hours, and then in the -proposed repository.

Please help us by testing this new package. To enable the -proposed repository:

  sudo add-apt-repository cloud-archive:newton-proposed
  sudo apt-get update

Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-newton-needed to verification-newton-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-newton-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

For the libvirt portion:
- testing of the issue of this bug itself - done
- in & cross version migration tests - done
- upgrade tests - done
- general QA regression testing - done

That said verification (for the yakkety-libvirt portion, not Nova) - done.

P.S. James will also add a statement once he could check his Tempest.
Especially to see if the rbd auth now works - that is bug 1672367 which is bundled in here.

@James - I'd consider all else done, would you set the "final" verification-done tag then?

Yakkety newton verification testing:

09:23:00 ======
09:23:00 Totals
09:23:00 ======
09:23:00 Ran: 103 tests in 1556.0000 sec.
09:23:00 - Passed: 96
09:23:00 - Skipped: 6
09:23:00 - Expected Fail: 0
09:23:00 - Unexpected Success: 0
09:23:00 - Failed: 1
09:23:00 Sum of execute time for each test: 937.5234 sec.

Single failure looks unrelated to this bug (but is consistent) so looking into that now.

Xenial newton verification testing:

10:00:49 ======
10:00:49 Totals
10:00:49 ======
10:00:49 Ran: 103 tests in 1427.0000 sec.
10:00:49 - Passed: 97
10:00:49 - Skipped: 6
10:00:49 - Expected Fail: 0
10:00:49 - Unexpected Success: 0
10:00:49 - Failed: 0
10:00:49 Sum of execute time for each test: 738.4477 sec.

So no regressions with the proposed updates for nova; however it is worth noting that just like the OpenStack nova gate, these checks do not exercise the impacted vif types.

yakkety-newton test failure was the:

 tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern.test_volume_boot_pattern

tests; I've re-run this independently and it passes OK:

======
Totals
======
Ran: 1 tests in 178.5077 sec.
 - Passed: 1
 - Skipped: 0
 - Expected Fail: 0
 - Unexpected Success: 0
 - Failed: 0
Sum of execute time for each test: 144.9156 sec.

Since we have one "issue" but two related fixes we had two SRU Templates, one in the bug description and one in comment #62 - I copied the one in the comment up and marked them as libvirt/nova so that an SRU Team member finds this information more easily.

Uploaded a no change rebuild as version 2.1.0-1ubuntu9.4 to help properly linking the other linked SRU bug as well.

Hello Logan, or anyone else affected,

Accepted libvirt into yakkety-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/libvirt/2.1.0-1ubuntu9.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

The automated proposed regression check on qemu/libvirt passed and confirmed that there should be no regression.
Also I was able to test the explicit fix of this vs proposed.
That said - verification-done (libvirt)

The verification of the Stable Release Update for nova has completed successfully and the package has now been released to -updates. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package nova - 2:14.0.4-0ubuntu1.2~cloud0
---------------

 nova (2:14.0.4-0ubuntu1.2~cloud0) xenial-newton; urgency=medium
 .
   * New update for the Ubuntu Cloud Archive.
 .
 nova (2:14.0.4-0ubuntu1.2) yakkety; urgency=medium
 .
   * d/p/libvirt-set-vlan-tag-for-macvtap.patch: Pick dependent patch
     for fix for libvirt 1.3.1 compatibility (LP: #1665698).
 .
 nova (2:14.0.4-0ubuntu1.1) yakkety; urgency=medium
 .
   * d/patches/libvirt-set-script-path.patch: Conditionally set
     the script path for ethernet vif types (LP: #1665698).

This bug was fixed in the package nova - 2:14.0.4-0ubuntu1.2

---------------
nova (2:14.0.4-0ubuntu1.2) yakkety; urgency=medium

  * d/p/libvirt-set-vlan-tag-for-macvtap.patch: Pick dependent patch
    for fix for libvirt 1.3.1 compatibility (LP: #1665698).

nova (2:14.0.4-0ubuntu1.1) yakkety; urgency=medium

  * d/patches/libvirt-set-script-path.patch: Conditionally set
    the script path for ethernet vif types (LP: #1665698).

nova (2:14.0.4-0ubuntu1) yakkety; urgency=medium

  * New upstream point release for OpenStack Newton (LP: #1664306).
  * d/control: Bump python-rfc3986 version.

 -- James Page <email address hidden> Wed, 22 Mar 2017 15:46:21 +0000

@Matt, nova: Can we get some eyes on the Newton and Ocata fix you have up? It is a pretty severe bug for me since I'm running a vif_type=ethernet plugin with regular Ubuntu distro libvirt, so I haven't been able to use any nova sha since early Feb due to this bug.

Reviewed: https://review.openstack.org/448242
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=cc495a24656893c94031f491a3fed2bc94fe1850
Submitter: Jenkins
Branch: stable/ocata

commit cc495a24656893c94031f491a3fed2bc94fe1850
Author: Matt Riedemann <email address hidden>
Date: Tue Mar 21 13:18:08 2017 -0400

    libvirt: conditionally set script path for ethernet vif types

    Change I4f97c05e2dec610af22a5150dd27696e1d767896 worked around
    a change introduced in libvirt 1.3.3 where the script path on
    a LibvirtConfigGuestInterface could not be the emptry string
    because libvirt would literally take that as the path and couldn't
    resolve it, when in fact it used to indicate to libvirt that the
    script path is a noop. This has been fixed in libvirt 3.1.

    On Ubuntu with libvirt<1.3.3, if the script path is None then
    it defaults to /etc/qemu-ifup which is blocked by AppArmor.

    So this change adds a conditional check when setting the script
    path value based on the libvirt version so we can straddle releases.

    Change-Id: I192c61b93bd3736fdfe16b6a6906d58997d3eef9
    Closes-Bug: #1665698
    Related-Bug: #1649527
    (cherry picked from commit a41d265a19b7bcb1af8fc179bf864e00023c6cc6)

This issue was fixed in the openstack/nova 15.0.3 release.

This issue was fixed in the openstack/nova 16.0.0.0b1 development milestone.

This bug was fixed in the package libvirt - 2.1.0-1ubuntu9.4

---------------
libvirt (2.1.0-1ubuntu9.4) yakkety; urgency=medium

  * no change rebuild to properly pick up all the changes since the last
    release to yakkety-updates.

libvirt (2.1.0-1ubuntu9.3) yakkety; urgency=medium

  * d/p/ubuntu/qemu-Allow-empty-script-path-to-interface.patch: in the past
    <script path=''/> meant a no-op, but libvirt now fails - fix to match
    the old behavior (LP: #1665698).

libvirt (2.1.0-1ubuntu9.2) yakkety; urgency=medium

  * d/p/ubuntu/fix-command-generation-for-rbd.patch: fix the generation
    of the command used to add disks when secrets are provided like when
    using rbd (LP: #1672367).

 -- Christian Ehrhardt <email address hidden> Fri, 31 Mar 2017 07:48:25 +0200

Reviewed: https://review.openstack.org/448253
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=8142d526dfd6f4a56dbe382d25cf4110abf57f44
Submitter: Jenkins
Branch: stable/newton

commit 8142d526dfd6f4a56dbe382d25cf4110abf57f44
Author: Matt Riedemann <email address hidden>
Date: Tue Mar 21 13:18:08 2017 -0400

    libvirt: conditionally set script path for ethernet vif types

    Change I4f97c05e2dec610af22a5150dd27696e1d767896 worked around
    a change introduced in libvirt 1.3.3 where the script path on
    a LibvirtConfigGuestInterface could not be the emptry string
    because libvirt would literally take that as the path and couldn't
    resolve it, when in fact it used to indicate to libvirt that the
    script path is a noop. This has been fixed in libvirt 3.1.

    On Ubuntu with libvirt<1.3.3, if the script path is None then
    it defaults to /etc/qemu-ifup which is blocked by AppArmor.

    So this change adds a conditional check when setting the script
    path value based on the libvirt version so we can straddle releases.

    Change-Id: I192c61b93bd3736fdfe16b6a6906d58997d3eef9
    Closes-Bug: #1665698
    Related-Bug: #1649527
    (cherry picked from commit a41d265a19b7bcb1af8fc179bf864e00023c6cc6)
    (cherry picked from commit cc495a24656893c94031f491a3fed2bc94fe1850)

This issue was fixed in the openstack/nova 14.0.6 release.