[OSSA 2016-012] qemu-img calls need to be restricted by ulimit (CVE-2015-5162)

FYI, I tested a recentish git master to see if I can trigger it and was succesful:

What I did was

 $ cd /usr/bin
 $ mv qemu-img qemu-img.real

 $ cat > qemu-img <<EOF
  #!/bin/sh

  echo "$@" >> /tmp/usage.log
  /usr/bin/time /usr/bin/qemu-img.real "$@" 2>> /tmp/usage.log
  echo >> /tmp/usage.log
  EOF

 $ chmod +x qemu-img

Then

 $ glance image-create --name afl3 --disk-format vmdk --container-format bare --file /home/berrange/afl3.img --is-public true

Glance did not appear to run qemu-img info at this stage, but when I then boot the image iwth Nova:

 $ nova boot --image afl3 --flavor m1.tiny afl3

Nova correctly refuses to boot the guest, because the disk image is larger
than the maximum it permits:

2015-04-27 14:39:00.381 TRACE nova.compute.manager [instance: aab2db9f-cbcc-4858-98e8-95e82b1d3b3b] return f(*args, **kwargs)
2015-04-27 14:39:00.381 TRACE nova.compute.manager [instance: aab2db9f-cbcc-4858-98e8-95e82b1d3b3b] File
+"/home/berrange/src/cloud/nova/nova/virt/libvirt/imagebackend.py", line 221, in fetch_func_sync
2015-04-27 14:39:00.381 TRACE nova.compute.manager [instance: aab2db9f-cbcc-4858-98e8-95e82b1d3b3b] fetch_func(target=target, *args, **kwargs)
2015-04-27 14:39:00.381 TRACE nova.compute.manager [instance: aab2db9f-cbcc-4858-98e8-95e82b1d3b3b] File
+"/home/berrange/src/cloud/nova/nova/virt/libvirt/utils.py", line 501, in fetch_image
2015-04-27 14:39:00.381 TRACE nova.compute.manager [instance: aab2db9f-cbcc-4858-98e8-95e82b1d3b3b] max_size=max_size)
2015-04-27 14:39:00.381 TRACE nova.compute.manager [instance: aab2db9f-cbcc-4858-98e8-95e82b1d3b3b] File
+"/home/berrange/src/cloud/nova/nova/virt/images.py", line 119, in fetch_to_raw
2015-04-27 14:39:00.381 TRACE nova.compute.manager [instance: aab2db9f-cbcc-4858-98e8-95e82b1d3b3b] raise exception.FlavorDiskTooSmall()
2015-04-27 14:39:00.381 TRACE nova.compute.manager [instance: aab2db9f-cbcc-4858-98e8-95e82b1d3b3b] FlavorDiskTooSmall: Flavor's disk is too small for
+requested image.
2015-04-27 14:39:00.381 TRACE nova.compute.manager [instance: aab2db9f-cbcc-4858-98e8-95e82b1d3b3b]

However in checking that maximum limit, it has indeed run qemu-img info
and suffered the memory usage DOS.

 $ cat /tmp/usage.log

 info /home/berrange/src/cloud/data/nova/instances/_base/30854c86815b92c21c8af45cf8ff5757c04046aa.part
 0.08user 0.94system 0:01.24elapsed 82%CPU (0avgtext+0avgdata 1260680maxresident)k
 0inputs+0outputs (0major+311876minor)pagefaults 0swaps

So someone needs to see just what code paths in glance might trigger the qemu-img info call. I also wonder if there is any risk to cinder, because it too has various invokations of qemu-img

Since this report concerns a possible security risk, an incomplete security advisory task has been added while the core security reviewers for the affected project or projects confirm the bug and discuss the scope of any vulnerability along with potential solutions.

Projects security liaison and original reporter have been subscribe.

While being an issue in a dependency (qemu), it's not clear whenever OpenStack is really missing restriction on the qemu-img process...

However, depending on what a malicious user can effectively do with this issue, we may want to consider this OSSA worthy. Thought ?

Excellent catch! And thanks for bringing this up, Richard. I was going to do some testing myself this week and file a corresponding bug.

This is a valid concern and something that we realized after the code was merged. One more point to consider here is that we need to run qemu-img in a sandbox vs. directly as a "glance" user on the the API node as possible in certain configurations. A approach for fixing this could be, using a chroot environment.

Here's an image which causes 4 GB to be allocated.

If you change a single byte in this image, then you can get qemu-img to crash with an allocation error:

00000000 4b 44 4d 56 01 00 00 00 03 00 00 00 01 00 ff ff |KDMV............|
00000010 ff ff 00 00 01 04 00 00 00 00 00 00 01 00 00 00 |................|
                      ^
                      +--- this byte to 0

qemu-img is run only on Glance API/worker nodes when user makes a tasks import call while the operator wants it converted into a specific format, currently.

Ref:-
http://specs.openstack.org/openstack/glance-specs/specs/kilo/conversion-of-images.html
https://review.openstack.org/#/c/159129/

> However, depending on what a malicious user can effectively do with this issue, we may want to consider this OSSA worthy. Thought ?

From Nova's POV I consider it to be OSSA worthy. Nova uses flavours to limit the amount of RAM that a user may consume. So if using the m1.tiny flavour, a guest is supposed to only have 500 MB RAM usage. The scheduler will consider this when deciding if a host has sufficient free RAM to run the guest. Now regardless of what the flavour limit is set to, the unprivileged user can consume as much as 4 GB of RAM on the host by uploading a malicious disk image. This is enough to cause OOM on the compute host and so affect running of other tenants VMs in a negative way.

Any progess on patch for this bug ?

Although this bug is marked as secret, I have published elsewhere some information about this (it was anyway an open secret that I was fuzzing qemu-img), and it is also fixed in libguestfs.

https://bugs.launchpad.net/qemu/+bug/1462944
https://bugs.launchpad.net/qemu/+bug/1462949
https://github.com/libguestfs/libguestfs/commit/9db0f807a917ec00d24297d5bf63c19065f930cf

Thanks, I will have a Glance patch this week.

This is the simplest patch I could come up with for limiting qemu-img info commands. There are various other qemu-img commands run, eg to create snapshots and the like, but I believe checking the qemu-img info command is sufficient as it is used to validate images after obtaining them from glance.

Added cinder, since the file cinder/image/image_utils.py invokes qemu-img info without protection too

Thanks Daniel, cinder-coresec is now subscribed. Can Glance and Cinder use the same prlimit fix ?

And now I've fixed the patch to make the unit tests pass again. Opps :-)

Yes, from looking at their code, the change to cinder & glance could be essentially identical.

> "util-limit package in Linux distros."

That should say "util-linux".

Here is a proposed patch for cinder

Hum, prlimit is not readily available on stable Ubuntu and Debian...

Sigh, "only" been part of util-linux since early 2012

http://karelzak.blogspot.co.uk/2012/01/prlimit1.html
https://www.kernel.org/pub/linux/utils/util-linux/v2.21/v2.21-ReleaseNotes

The alternative approach is to modify oslo.concurrency processutils.execute method to allow limits to be passed in explicitly, and then use the pre-exec function to set them between fork + exec. This is actually the more desirable architectural approach, but I didn't go that way, since while it is fine for master, it would not be something that's easily backported to stable branches due to the need to update the min required oslo.concurrency version dependancy.

I actually much prefer the ulimit argument to processutils.execute() plan. I'm going to go to the oslo PTL and ask for an opinion of backporting that.

Since this bugs requires change in 4 different projects, and the impacts is moderate, I propose we open this bug in a few days for a faster solution. Is there any objections ?

@mikal,

yes, i'd support a ulimit argument to processutils.execute and willing to help backport it to stable/kilo (and release if needed)

This is a *completely untested* patch to illustrate adding support for process resource limits to oslo.concurrency's execute() call. If people agree with this approach, I'll just submit it to gerrit against oslo, as adding this feature to olso is probably useful regardless of this OSSA

Hi, we've released a oslo.concurrency 1.8.1 (stable/kilo) with support to call a user specified preexec_fn to support this bug/fix. yes, same is in master as well.

80bfd1a Allow preexec_fn method for processutils.execute

thanks,
dims

Draft impact description:

Title: Malicious input to qemu-img may result in resource exhaustion
Reporter: Richard W.M. Jones
Product: Nova, Cinder, Glance
Affects:
    Nova: 2012.1.0 versions through 2015.1.0,
    Cinder: 2013.1.0 versions through 2015.1.0,
    Glance: 2015.1.0

Description:
Richard W.M. Jones of Red Hat reported a vulnerability that affects several
OpenStack projects including Nova, Glance, and Cinder. By providing a
maliciously crafted disk image an attacker can consume considerable amounts
of RAM and CPU time resulting in a denial of service via resource exhaustion.
Any project which makes calls to qemu-img without appropriate ulimit restrictions
in place is affected by this flaw.

I think this would more properly be (assuming the fixes won't make it in time for 2015.1.1)...

    Nova: versions through 2014.2.3, 2015.1 versions through 2015.1.1
    Cinder: versions through 2014.2.3, 2015.1 versions through 2015.1.1
    Glance: 2015.1 versions through 2015.1.1

CVE requested.

Fix proposed to branch: master
Review: https://review.openstack.org/209627

Is there a review in progress for Glance ?

Yep.

I've attached another patch which this time uses the new preexec_fn feature of oslo.concurrency 2.3.0.

This patch is *UNTESTED* beyond running the unit tests, so don't assume this actually fixes the problem without testing it in real world.

Daniel, could you please have a look at https://review.openstack.org/#/c/209627/ ?

Reviewed: https://review.openstack.org/209627
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=da217205f53f9a38a573fb151898fbbeae41021d
Submitter: Jenkins
Branch: master

commit da217205f53f9a38a573fb151898fbbeae41021d
Author: Tristan Cacqueray <email address hidden>
Date: Wed Aug 5 17:17:04 2015 +0000

    virt: Use preexec_fn to ulimit qemu-img info call

    This uses the preexec_fn oslo.concurrency execute parameter to set user limit
    before calling qemu-img info. The process is limitted to 2 seconds of cpu
    and 1GB of virtual memory

    Change-Id: Ib47f1116c94c8f76d2a4a525af24439a4aa15854
    Closes-Bug: #1449062

FYI, we are reverting the patch for this bug because it caused a spike in gate failures

https://bugs.launchpad.net/nova/+bug/1506012
https://review.openstack.org/#/c/234696/

The way https://review.openstack.org/#/c/209627/ applied the limits to qemu-img was by setting rlimits in between fork() and execve(). The problem with this is that before we reach exec() we have a full copy-on-write mapping of the nova python process. So if Nova memory usage is too high, this could tickle the limit we're setting for qemu-img.

The only way to avoid this is to *not* set limits until after we have exec'd. This would require using an external command like my original prlimit patch.

Since not all distros have prlimit, we could create a simple nova-prlimit python script todo this. We would use native prlimit if it was present in the distro (as its fast) and fallacck to nova-prlimit if missing.

It is a shame oslo.concurrency went for the preexec_fn approach, rather than my suggestion of explicitly representing resource limits in the API, as that would have allowed us to hide this compat code in oslo.concurrency, instead of dealing with it in nova, glance and cinder :-(

Can this prlimit python script could be provided by oslo or a requirement since it could be used by Cinder and Glance as well?

Note that such prlimit utility is being worked on in oslo.concurrency here: https://review.openstack.org/#/c/243829/

This issue was fixed in the openstack/nova 13.0.0.0b1 development milestone.

The proposed change did not effectively fixed that issue.

The oslo.concurrency feature we require merged in Feb 1st this year. So we can now retry fixing nova once again, and also backport it to Mitaka stable series once merged in master

Fix proposed to branch: master
Review: https://review.openstack.org/307663

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/326327

For Glance: The task API is now admin only by default since Mitaka. So, the risk of this issue to glance tasks API is significantly low. I've degraded the importance accordingly.

Reviewed: https://review.openstack.org/307663
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=068d851561addfefb2b812d91dc2011077cb6e1d
Submitter: Jenkins
Branch: master

commit 068d851561addfefb2b812d91dc2011077cb6e1d
Author: Daniel P. Berrange <email address hidden>
Date: Mon Apr 18 16:32:19 2016 +0000

    virt: set address space & CPU time limits when running qemu-img

    This uses the new 'prlimit' parameter for oslo.concurrency execute
    method, to set an address space limit of 1GB and CPU time limit
    of 2 seconds, when running qemu-img.

    This is a re-implementation of the previously reverted commit

    commit da217205f53f9a38a573fb151898fbbeae41021d
    Author: Tristan Cacqueray <email address hidden>
    Date: Wed Aug 5 17:17:04 2015 +0000

        virt: Use preexec_fn to ulimit qemu-img info call

    Closes-Bug: #1449062
    Change-Id: I135b5242af1bfdcb0ea09a6fcda21fc03a6fbe7d

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/327624

Newton is now fixed, but mitaka needs an oslo.concurrency requirements bump to 3.8.0.

For liberty, it is still using 2.6.x version and we need to first merge: https://review.openstack.org/#/c/327630/, then release a new version of olso.concurrency in order to bump requirements for stable/liberty.

Reviewed: https://review.openstack.org/326327
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=c8ec9ebf379c61d73c5671a75dd2a4e4ae1403fb
Submitter: Jenkins
Branch: stable/mitaka

commit c8ec9ebf379c61d73c5671a75dd2a4e4ae1403fb
Author: Daniel P. Berrange <email address hidden>
Date: Mon Apr 18 16:32:19 2016 +0000

    virt: set address space & CPU time limits when running qemu-img

    This uses the new 'prlimit' parameter for oslo.concurrency execute
    method, to set an address space limit of 1GB and CPU time limit
    of 2 seconds, when running qemu-img.

    This is a re-implementation of the previously reverted commit

    commit da217205f53f9a38a573fb151898fbbeae41021d
    Author: Tristan Cacqueray <email address hidden>
    Date: Wed Aug 5 17:17:04 2015 +0000

        virt: Use preexec_fn to ulimit qemu-img info call

    NOTE(mriedem): The backport depends on raising the minimum
    required oslo.concurrency>=3.7.1 which contains the prlimits
    fix for this change, the code fails without it.

    Depends-on: I157a6fc88836da73814ae46c8991ddefc5066c96
    Closes-Bug: #1449062
    Change-Id: I135b5242af1bfdcb0ea09a6fcda21fc03a6fbe7d
    (cherry picked from commit 068d851561addfefb2b812d91dc2011077cb6e1d)

This issue was fixed in the openstack/nova 13.1.0 release.

Hello Tristan, or anyone else affected,

Accepted python-oslo.concurrency into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-oslo.concurrency/3.7.1-0ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Hi Chris,

python-oslo.concurrency 3.7.1-0ubuntu1 has been tested successfully.

Thanks,
Corey

This bug was fixed in the package python-oslo.concurrency - 3.7.1-0ubuntu1

---------------
python-oslo.concurrency (3.7.1-0ubuntu1) xenial; urgency=medium

  * New upstream point release (LP: #1449062).

 -- Corey Bryant <email address hidden> Mon, 13 Jun 2016 12:34:15 -0400

The verification of the Stable Release Update for python-oslo.concurrency has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Here is a recap of all related patches,

master/newton (oslo.concurrency-3.8.0):
* https://review.openstack.org/243829 oslo.concurrency prlimit (merged in 3.7.1)
* https://review.openstack.org/307813 oslo.concurrency process limit (not merged in 3.7.1)
* https://review.openstack.org/307663 nova fix

stable/mitaka (oslo.concurrency-3.7.1):
* https://review.openstack.org/326327 nova backport
* https://review.openstack.org/327774 oslo.concurrency process limit backport

stable/liberty (oslo.concurrency-2.6.1):
* https://review.openstack.org/327624 nova backport
* https://review.openstack.org/327630 oslo.concurrency prlimit backport
* https://review.openstack.org/332222 oslo.concurrency process limit backport

It seems like we only need 327624 to move on the OSSA task. Is there something like a global requirement change we are waiting for ?

Hello Tristan, or anyone else affected,

Accepted python-oslo.concurrency into wily-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python-oslo.concurrency/2.6.1-0ubuntu1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

This issue was fixed in the openstack/nova 14.0.0.0b2 development milestone.

Just a note that the Ubuntu Cloud Archive for Liberty includes the fix with python-oslo.concurrency 2.6.1-0ubuntu1~cloud0. (Wily has gone EOL since the update from Chris Arges above.)

Based on the thread at http://lists.openstack.org/pipermail/openstack-dev/2016-September/104091.html we may need to figure out how to adjust the messaging to indicate that it was a severe enough bug to fix in stable/mitaka but that stable/liberty will be left unfixed.

Reviewed: https://review.openstack.org/327624
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=6bc37dcceca823998068167b49aec6def3112397
Submitter: Jenkins
Branch: stable/liberty

commit 6bc37dcceca823998068167b49aec6def3112397
Author: Daniel P. Berrange <email address hidden>
Date: Mon Apr 18 16:32:19 2016 +0000

    virt: set address space & CPU time limits when running qemu-img

    This uses the new 'prlimit' parameter for oslo.concurrency execute
    method, to set an address space limit of 1GB and CPU time limit
    of 2 seconds, when running qemu-img.

    This is a re-implementation of the previously reverted commit

    commit da217205f53f9a38a573fb151898fbbeae41021d
    Author: Tristan Cacqueray <email address hidden>
    Date: Wed Aug 5 17:17:04 2015 +0000

        virt: Use preexec_fn to ulimit qemu-img info call

    NOTE (kchamart) [stable/liberty]: Add a check for the presence of
    'ProcessLimits' attribute (which is only present in
    oslo.concurrency>=2.6.1; and a conditional check for 'prlimit' parameter
    in qemu_img_info() method.

    Upstream discussion[1][2] that led to merging this patch to
    stable/liberty branch.

    [1] http://lists.openstack.org/pipermail/openstack-dev/2016-September/104091.html
    [2] http://lists.openstack.org/pipermail/openstack-dev/2016-September/104303.html

    Closes-Bug: #1449062
    Change-Id: I135b5242af1bfdcb0ea09a6fcda21fc03a6fbe7d
    (cherry picked from commit 068d851561addfefb2b812d91dc2011077cb6e1d)

I'm resurrecting Grant's proposed impact description from comment #28 and updating for the year of time which has passed since. I've also edited it to remove references to Cinder and Glance... are those effectively still impacted in any supported branches? I see that the tasks API in Glance becoming admin-only in Mitaka results in this being impractical there, but what about for Liberty? And there's little input from Cinder on this bug at all but the claim is that it's exploitable there as well. Is that still the case today?

--

Title: Malicious input to qemu-img may result in resource exhaustion
Reporter: Richard W.M. Jones
Product: Nova
Affects: <=12.0.4, ==13.0.0

Description:
Richard W.M. Jones of Red Hat reported a vulnerability that affects OpenStack
Nova. By providing a maliciously crafted disk image an attacker can consume
considerable amounts of RAM and CPU time resulting in a denial of service via
resource exhaustion. Any project which makes calls to qemu-img without
appropriate ulimit restrictions in place is affected by this flaw.

Fix proposed to branch: master
Review: https://review.openstack.org/375099

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/375102

Following discussion with Sean and Hemanth, it looks like we ought to get fixes for this into supported branches of Cinder and Glance after all. Hopefully getting them merged goes quickly now that Nova has already trodden this ground and the fixes are basically identical between them.

Assuming all fixes merge before Newton releases and prior to any stable branch point releases, this is the updated proposed impact description. I'm using our YAML format here both for convenience and clarity due to the number of repos and releases involved. I've also shortened the title and referenced the affected software in it.

Should I include a note about the updated versions of oslo.concurrency required, or are the references to patches in consuming projects sufficient for this purpose? I'd like to avoid unnecessary additional complexity if possible here.

--

date: TBD

id: TBD

title: Malicious qemu-img input may exhaust resources in Cinder, Glance, Nova

description: >
  Richard W.M. Jones of Red Hat reported a vulnerability that affects OpenStack
  Cinder, Glance and Nova. By providing a maliciously crafted disk image an
  attacker can consume considerable amounts of RAM and CPU time resulting in a
  denial of service via resource exhaustion. Any project which makes calls to
  qemu-img without appropriate ulimit restrictions in place is affected by this
  flaw.

affected-products:
  - product: cinder
    version: "<=7.0.2, >=8.0.0 <=8.1.1"
  - product: glance
    version: "<=11.0.1, ==12.0.0"
  - product: nova
    version: "<=12.0.4 and ==13.0.0"

vulnerabilities:
  - cve-id: CVE-2015-5162

reporters:
  - name: Richard W.M. Jones
    affiliation: Red Hat
    reported:
      - CVE-2015-5162

issues:
  links:
    - https://launchpad.net/bugs/1449062

reviews:
  ocata:
    - https://review.openstack.org/375099 (cinder)
    - https://review.openstack.org/TBD (glance)
  newton:
    - https://review.openstack.org/375102 (cinder)
    - https://review.openstack.org/TBD (glance)
    - https://review.openstack.org/307663 (nova)
  mitaka:
    - https://review.openstack.org/TBD (cinder)
    - https://review.openstack.org/TBD (glance)
    - https://review.openstack.org/326327 (nova)
  liberty:
    - https://review.openstack.org/TBD (cinder)
    - https://review.openstack.org/TBD (glance)
    - https://review.openstack.org/327624 (nova)

notes:
  - >
    Separate Ocata patches are listed for Cinder and Glance, as they were fixed
    during the Newton release freeze after it branched from master.

Thank you for that info Jeremy, I've targetted it to the appropriate series in Glance so it's clear.

Reviewed: https://review.openstack.org/375099
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=78f17f0ad79380ee3d9c50f2670252bcc559b62b
Submitter: Jenkins
Branch: master

commit 78f17f0ad79380ee3d9c50f2670252bcc559b62b
Author: Sean McGinnis <email address hidden>
Date: Thu Sep 22 15:31:37 2016 -0500

    Limit memory & CPU when running qemu-img info

    It was found that a modified or corrupted image file can cause a DoS
    on the host when getting image info with qemu-img.

    This uses the newer 'prlimit' parameter for oslo.concurrency execute
    to set an address space limit of 1GB and CPU time limit of 2 seconds
    when running the qemu-img info command.

    Change-Id: If5b7129b266ef065642bc7898ce9dcf93722a053
    Closes-bug: #1449062

Reviewed: https://review.openstack.org/375102
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=8547444775e406a50d9d26a0003e9ba6554b0d70
Submitter: Jenkins
Branch: stable/newton

commit 8547444775e406a50d9d26a0003e9ba6554b0d70
Author: Sean McGinnis <email address hidden>
Date: Thu Sep 22 15:31:37 2016 -0500

    Limit memory & CPU when running qemu-img info

    It was found that a modified or corrupted image file can cause a DoS
    on the host when getting image info with qemu-img.

    This uses the newer 'prlimit' parameter for oslo.concurrency execute
    to set an address space limit of 1GB and CPU time limit of 2 seconds
    when running the qemu-img info command.

    Change-Id: If5b7129b266ef065642bc7898ce9dcf93722a053
    Closes-bug: #1449062
    (cherry picked from commit 78f17f0ad79380ee3d9c50f2670252bcc559b62b)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/375625

It's just a theory (credits to Brian Rosmaita) at this point, but looks like "qemu-img convert" will try to infer the format of input image if "-f" is not provided. So, "qemu-img convert" may be susceptible to the same attack. Any thoughts?

Yes, *any* qemu-img command that you run without providing '-f' will try to guess the image format. Rather than trying to figure out whether a particular invokation may or may not be susceptible to attack, the safe approach is to use '-f' every time.

Jeremy, the impact description looks good. Though note that prlimit implementation before oslo.concurrency-3.8.0 doesn't support all the required resources limit, thus I would mention that oslo.concurrency>=3.8.0 is required to fix that issue.

Hemanth, Daniel: So that means the current patches to Nova are insufficient because they missed `qemu-image convert` invocations? For example at http://git.openstack.org/cgit/openstack/nova/tree/nova/virt/xenapi/vm_utils.py#n1128

Tristan: Thanks, it looked like oslo.concurrency got backports to stable/mitaka and stable/liberty to support it too though. Is that correct? If so we may need to get much more verbose about the library versions required by the service fixes.

Jeremy: Hemanth (in comment#72) seems to have mixed up this bug (which sets limits for memory / CPU usage for `qemu-img` calls) with *another* bug[x] that is about disk image format guessing.

So, the Nova patches that fix this bug (1449062) are sufficient for the problem it is solving (setting a cap on memory / CPU usage).

[x] https://bugs.launchpad.net/nova/+bug/1415087 --
    [OSSA 2015-011] Format-guessing and file disclosure
    in image convert (CVE-2015-1850, CVE-2015-1851)

Thanks Kashyap.

That bug isn't cross listed with Glance and the current one applies to Glance in a very generic way. So, Hemanth did the right thing from our perspective. I will leave the rest of the story to the VMT team.

Reviewed: https://review.openstack.org/375526
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=69a9b659fd48aa3c1f84fc7bc9ae236b6803d31f
Submitter: Jenkins
Branch: master

commit 69a9b659fd48aa3c1f84fc7bc9ae236b6803d31f
Author: Hemanth Makkapati <email address hidden>
Date: Fri Sep 23 09:29:12 2016 -0500

    Adding constraints around qemu-img calls

    * All "qemu-img info" calls are now run under resource limitations
      that limit CPU time to 2 seconds and address space usage to 1 GB.
      This helps avoid any DoS attacks via malicious images.
    * All "qemu-img convert" calls now specify the import format so that
      it does not have to be inferred by qemu-img.

    SecurityImpact

    Change-Id: Ib900bbc05cb9ccd90c6f56ccb4bf2006e30cdc80
    Closes-Bug: #1449062

Jeremy, the missing bit for oslo-concurrency 3.7.1 is https://review.openstack.org/#/q/I164c4b35e1357a0f80ed7fe00a7ae8f49df92e31 and it was merged to stable branches. Fortunately it seems like all version >= 3.8.0 are good enough to support correct resources limit, so I guess we could just mention that.

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/377734

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/377736

Tristan: I'm still a little confused on the oslo.concurrency recommendation. Are you saying that we should suggest stable/liberty and stable/mitaka deployments to also use oslo.concurrency>=3.8.0? At the moment the tips of stable/liberty and stable/mitaka branches for oslo.concurrency are tagged 2.6.1 and 3.7.1 respectively (and that's what we have pinned in upper-constraints.txt for testing purposes as well). I don't want to imply in an advisory that all deployments should upgrade oslo.concurrency to 3.8.0 or later if there's a risk it will break liberty or mitaka deployments (which is why I was leaning toward not mentioning oslo.concurrency versions as that would just add to confusion).

I had the same question for mitaka, so had to dig a little deeper. oslo.concurrency 3.7.1 was the follow on release to 3.7.0 to include the prlimit option. So based on current requirements for stable/mitaka, it should be safe to backport this fix.

I did not investigate stable/liberty, but I'm assuming 2.6.1 is the same situation.

Reviewed: https://review.openstack.org/377734
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=6cba6b18a348e8fc2eb4ae25b636d065456a633c
Submitter: Jenkins
Branch: stable/newton

commit 6cba6b18a348e8fc2eb4ae25b636d065456a633c
Author: Hemanth Makkapati <email address hidden>
Date: Fri Sep 23 09:29:12 2016 -0500

    Adding constraints around qemu-img calls

    * All "qemu-img info" calls are now run under resource limitations
      that limit CPU time to 2 seconds and address space usage to 1 GB.
      This helps avoid any DoS attacks via malicious images.
    * All "qemu-img convert" calls now specify the import format so that
      it does not have to be inferred by qemu-img.

    SecurityImpact

    Change-Id: Ib900bbc05cb9ccd90c6f56ccb4bf2006e30cdc80
    Closes-Bug: #1449062
    (cherry picked from commit 69a9b659fd48aa3c1f84fc7bc9ae236b6803d31f)

Indeed my bad, oslo-concurrency 2.6.1, 3.7.1 and 3.8.0 are all good and referenced in the upper-constraint of supported stable branches. Thus I agree we could omit that note on the advisory.

Reviewed: https://review.openstack.org/377736
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=c90830d71969f68768d898c1c178489f602214e2
Submitter: Jenkins
Branch: stable/mitaka

commit c90830d71969f68768d898c1c178489f602214e2
Author: Hemanth Makkapati <email address hidden>
Date: Fri Sep 23 09:29:12 2016 -0500

    Adding constraints around qemu-img calls

    * All "qemu-img info" calls are now run under resource limitations
      that limit CPU time to 2 seconds and address space usage to 1 GB.
      This helps avoid any DoS attacks via malicious images.
    * All "qemu-img convert" calls now specify the import format so that
      it does not have to be inferred by qemu-img.

    SecurityImpact

    Change-Id: Ib900bbc05cb9ccd90c6f56ccb4bf2006e30cdc80
    Closes-Bug: #1449062
    (cherry picked from commit 69a9b659fd48aa3c1f84fc7bc9ae236b6803d31f)

This issue was fixed in the openstack/glance 13.0.0.0rc2 release candidate.

This issue was fixed in the openstack/cinder 9.0.0.0rc2 release candidate.

This issue was fixed in the openstack/nova 12.0.5 release.

Reviewed: https://review.openstack.org/378012
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=58311904a73f931404416649dc6ed3958adc59c8
Submitter: Jenkins
Branch: stable/liberty

commit 58311904a73f931404416649dc6ed3958adc59c8
Author: Brian Rosmaita <email address hidden>
Date: Tue Sep 27 16:11:17 2016 -0400

    Adding constraints around qemu-img calls

    * All "qemu-img info" calls are now run under resource limitations that
      limit CPU time to 2 seconds and address space usage to 1 GB. This
      helps avoid any DoS attacks via malicious images.
    * All "qemu-img convert" calls now specify the import format so that it
      does not have to be inferred by qemu-img.

    SecurityImpact

    (Hemanth did all the work on this, I'm just doing the backport.)

    Co-authored-by: Hemanth Makkapati <email address hidden>
    Closes-Bug: #1449062
    (cherry picked from commit 69a9b659fd48aa3c1f84fc7bc9ae236b6803d31f)

    Change-Id: I65f30b85439a8811545b0ca590555528631954df

Status update: it looks like all Glance and Nova fixes have merged; so too have the master and stable/newton changes for Cinder. At this point we're waiting for https://review.openstack.org/375625 (Cinder's stable/mitaka fix) to merge, and we don't seem to have a stable/liberty backport for Cinder.

Sean: were you planning to work on a liberty change for this? Or is stable/liberty of Cinder unaffected (in which case I need to adjust the impact description accordingly)?

I think the impact description is good. I will push through those backports.

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/382573

Reviewed: https://review.openstack.org/375625
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=c6adc020a67ae77e3645d4f6e80fa93b19432177
Submitter: Jenkins
Branch: stable/mitaka

commit c6adc020a67ae77e3645d4f6e80fa93b19432177
Author: Sean McGinnis <email address hidden>
Date: Thu Sep 22 15:31:37 2016 -0500

    Limit memory & CPU when running qemu-img info

    It was found that a modified or corrupted image file can cause a DoS
    on the host when getting image info with qemu-img.

    This uses the newer 'prlimit' parameter for oslo.concurrency execute
    to set an address space limit of 1GB and CPU time limit of 2 seconds
    when running the qemu-img info command.

    Change-Id: If5b7129b266ef065642bc7898ce9dcf93722a053
    Closes-bug: #1449062
    (cherry picked from commit 8547444775e406a50d9d26a0003e9ba6554b0d70)

Reviewed: https://review.openstack.org/382573
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=455b318ced717fb38dfe40014817d78fbc47dea5
Submitter: Jenkins
Branch: stable/liberty

commit 455b318ced717fb38dfe40014817d78fbc47dea5
Author: Sean McGinnis <email address hidden>
Date: Thu Sep 22 15:31:37 2016 -0500

    Limit memory & CPU when running qemu-img info

    It was found that a modified or corrupted image file can cause a DoS
    on the host when getting image info with qemu-img.

    This uses the newer 'prlimit' parameter for oslo.concurrency execute
    to set an address space limit of 1GB and CPU time limit of 2 seconds
    when running the qemu-img info command.

    Change-Id: If5b7129b266ef065642bc7898ce9dcf93722a053
    Closes-bug: #1449062
    (cherry picked from commit 8547444775e406a50d9d26a0003e9ba6554b0d70)

See comment #90 for glance fix to stable/liberty

Fix released to all stable branches

This issue was fixed in the openstack/nova 12.0.5 release.

This issue was fixed in the openstack/cinder 9.0.0 release.

This issue was fixed in the openstack/glance 13.0.0 release.

This issue was fixed in the openstack/cinder 7.0.3 release.

This issue was fixed in the openstack/cinder 10.0.0.0b1 development milestone.

This issue was fixed in the openstack/glance 14.0.0.0b1 development milestone.

This issue was fixed in the openstack/glance 13.0.0 release.

This issue was fixed in the openstack/glance 11.0.2 release.