[OSSA 2016-012] qemu-img calls need to be restricted by ulimit (CVE-2015-5162)

Bug #1449062 reported by Tristan Cacqueray
282
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
Fix Released
Medium
Sean McGinnis
Mitaka
Fix Released
Undecided
Hemanth Makkapati
Newton
Fix Released
Medium
Sean McGinnis
Glance
Fix Released
High
Hemanth Makkapati
Liberty
Fix Released
Undecided
Unassigned
Mitaka
Fix Released
High
Hemanth Makkapati
Newton
Fix Released
Critical
Hemanth Makkapati
OpenStack Compute (nova)
Fix Released
Medium
Daniel Berrange
Declined for Juno by Alan Pevec
Nominated for Kilo by Tristan Cacqueray
OpenStack Security Advisory
Fix Released
Medium
Jeremy Stanley
Ubuntu Cloud Archive
Fix Released
Medium
Unassigned
Liberty
Fix Committed
Medium
Unassigned
Mitaka
Fix Committed
Medium
Unassigned
Newton
Fix Released
Medium
Unassigned
python-oslo.concurrency (Ubuntu)
Fix Released
Medium
Unassigned
Wily
Fix Committed
Medium
Unassigned
Xenial
Fix Released
Medium
Corey Bryant
Yakkety
Fix Released
Medium
Unassigned

Bug Description

Reported via private E-mail from Richard W.M. Jones.

Turns out qemu image parser is not hardened against malicious input and can be abused to allocated an arbitrary amount of memory and/or dump a lot of information when used with "--output=json".

The solution seems to be: limit qemu-img ressource using ulimit.

Example of abuse:

-- afl1.img --

$ /usr/bin/time qemu-img info afl1.img
image: afl1.img
[...]
0.13user 0.19system 0:00.36elapsed 92%CPU (0avgtext+0avgdata 642416maxresident)k
0inputs+0outputs (0major+156927minor)pagefaults 0swaps

The original image is 516 bytes, but it causes qemu-img to allocate 640 MB.

-- afl2.img --

$ qemu-img info --output=json afl2.img | wc -l
589843

This is a 200K image which causes qemu-img info to output half a
million lines of JSON (14 MB of JSON).

Glance runs the --output=json variant of the command.

-- afl3.img --

$ /usr/bin/time qemu-img info afl3.img
image: afl3.img
[...]
0.09user 0.35system 0:00.47elapsed 94%CPU (0avgtext+0avgdata 1262388maxresident)k
0inputs+0outputs (0major+311994minor)pagefaults 0swaps

qemu-img allocates 1.3 GB (actually, a bit more if you play with
ulimit -v). It appears that you could change it to allocate
arbitrarily large amounts of RAM.

CVE References

Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote :
Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote :
Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote :
Changed in ossa:
status: New → Incomplete
Revision history for this message
Daniel Berrange (berrange) wrote :

FYI, I tested a recentish git master to see if I can trigger it and was succesful:

What I did was

 $ cd /usr/bin
 $ mv qemu-img qemu-img.real

 $ cat > qemu-img <<EOF
  #!/bin/sh

  echo "$@" >> /tmp/usage.log
  /usr/bin/time /usr/bin/qemu-img.real "$@" 2>> /tmp/usage.log
  echo >> /tmp/usage.log
  EOF

 $ chmod +x qemu-img

Then

 $ glance image-create --name afl3 --disk-format vmdk --container-format bare --file /home/berrange/afl3.img --is-public true

Glance did not appear to run qemu-img info at this stage, but when I then boot the image iwth Nova:

 $ nova boot --image afl3 --flavor m1.tiny afl3

Nova correctly refuses to boot the guest, because the disk image is larger
than the maximum it permits:

2015-04-27 14:39:00.381 TRACE nova.compute.manager [instance: aab2db9f-cbcc-4858-98e8-95e82b1d3b3b] return f(*args, **kwargs)
2015-04-27 14:39:00.381 TRACE nova.compute.manager [instance: aab2db9f-cbcc-4858-98e8-95e82b1d3b3b] File
+"/home/berrange/src/cloud/nova/nova/virt/libvirt/imagebackend.py", line 221, in fetch_func_sync
2015-04-27 14:39:00.381 TRACE nova.compute.manager [instance: aab2db9f-cbcc-4858-98e8-95e82b1d3b3b] fetch_func(target=target, *args, **kwargs)
2015-04-27 14:39:00.381 TRACE nova.compute.manager [instance: aab2db9f-cbcc-4858-98e8-95e82b1d3b3b] File
+"/home/berrange/src/cloud/nova/nova/virt/libvirt/utils.py", line 501, in fetch_image
2015-04-27 14:39:00.381 TRACE nova.compute.manager [instance: aab2db9f-cbcc-4858-98e8-95e82b1d3b3b] max_size=max_size)
2015-04-27 14:39:00.381 TRACE nova.compute.manager [instance: aab2db9f-cbcc-4858-98e8-95e82b1d3b3b] File
+"/home/berrange/src/cloud/nova/nova/virt/images.py", line 119, in fetch_to_raw
2015-04-27 14:39:00.381 TRACE nova.compute.manager [instance: aab2db9f-cbcc-4858-98e8-95e82b1d3b3b] raise exception.FlavorDiskTooSmall()
2015-04-27 14:39:00.381 TRACE nova.compute.manager [instance: aab2db9f-cbcc-4858-98e8-95e82b1d3b3b] FlavorDiskTooSmall: Flavor's disk is too small for
+requested image.
2015-04-27 14:39:00.381 TRACE nova.compute.manager [instance: aab2db9f-cbcc-4858-98e8-95e82b1d3b3b]

However in checking that maximum limit, it has indeed run qemu-img info
and suffered the memory usage DOS.

 $ cat /tmp/usage.log

 info /home/berrange/src/cloud/data/nova/instances/_base/30854c86815b92c21c8af45cf8ff5757c04046aa.part
 0.08user 0.94system 0:01.24elapsed 82%CPU (0avgtext+0avgdata 1260680maxresident)k
 0inputs+0outputs (0major+311876minor)pagefaults 0swaps

So someone needs to see just what code paths in glance might trigger the qemu-img info call. I also wonder if there is any risk to cinder, because it too has various invokations of qemu-img

Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote :

Since this report concerns a possible security risk, an incomplete security advisory task has been added while the core security reviewers for the affected project or projects confirm the bug and discuss the scope of any vulnerability along with potential solutions.

Projects security liaison and original reporter have been subscribe.

While being an issue in a dependency (qemu), it's not clear whenever OpenStack is really missing restriction on the qemu-img process...

However, depending on what a malicious user can effectively do with this issue, we may want to consider this OSSA worthy. Thought ?

Revision history for this message
Nikhil Komawar (nikhil-komawar) wrote :

Excellent catch! And thanks for bringing this up, Richard. I was going to do some testing myself this week and file a corresponding bug.

This is a valid concern and something that we realized after the code was merged. One more point to consider here is that we need to run qemu-img in a sandbox vs. directly as a "glance" user on the the API node as possible in certain configurations. A approach for fixing this could be, using a chroot environment.

Changed in glance:
status: New → Triaged
importance: Undecided → High
Revision history for this message
Richard Jones (rjones-redhat) wrote :

Here's an image which causes 4 GB to be allocated.

If you change a single byte in this image, then you can get qemu-img to crash with an allocation error:

00000000 4b 44 4d 56 01 00 00 00 03 00 00 00 01 00 ff ff |KDMV............|
00000010 ff ff 00 00 01 04 00 00 00 00 00 00 01 00 00 00 |................|
                      ^
                      +--- this byte to 0

Revision history for this message
Nikhil Komawar (nikhil-komawar) wrote :

qemu-img is run only on Glance API/worker nodes when user makes a tasks import call while the operator wants it converted into a specific format, currently.

Ref:-
http://specs.openstack.org/openstack/glance-specs/specs/kilo/conversion-of-images.html
https://review.openstack.org/#/c/159129/

Revision history for this message
Daniel Berrange (berrange) wrote :

> However, depending on what a malicious user can effectively do with this issue, we may want to consider this OSSA worthy. Thought ?

From Nova's POV I consider it to be OSSA worthy. Nova uses flavours to limit the amount of RAM that a user may consume. So if using the m1.tiny flavour, a guest is supposed to only have 500 MB RAM usage. The scheduler will consider this when deciding if a host has sufficient free RAM to run the guest. Now regardless of what the flavour limit is set to, the unprivileged user can consume as much as 4 GB of RAM on the host by uploading a malicious disk image. This is enough to cause OOM on the compute host and so affect running of other tenants VMs in a negative way.

Thierry Carrez (ttx)
Changed in ossa:
importance: Undecided → Medium
status: Incomplete → Confirmed
Changed in nova:
status: New → Confirmed
Changed in glance:
milestone: none → liberty-1
assignee: nobody → nikhil komawar (nikhil-komawar)
Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote :

Any progess on patch for this bug ?

Changed in nova:
assignee: nobody → Daniel Berrange (berrange)
Revision history for this message
Richard Jones (rjones-redhat) wrote :

Although this bug is marked as secret, I have published elsewhere some information about this (it was anyway an open secret that I was fuzzing qemu-img), and it is also fixed in libguestfs.

https://bugs.launchpad.net/qemu/+bug/1462944
https://bugs.launchpad.net/qemu/+bug/1462949
https://github.com/libguestfs/libguestfs/commit/9db0f807a917ec00d24297d5bf63c19065f930cf

Revision history for this message
Nikhil Komawar (nikhil-komawar) wrote :

Thanks, I will have a Glance patch this week.

Revision history for this message
Daniel Berrange (berrange) wrote :

This is the simplest patch I could come up with for limiting qemu-img info commands. There are various other qemu-img commands run, eg to create snapshots and the like, but I believe checking the qemu-img info command is sufficient as it is used to validate images after obtaining them from glance.

Revision history for this message
Daniel Berrange (berrange) wrote :

Added cinder, since the file cinder/image/image_utils.py invokes qemu-img info without protection too

Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote :

Thanks Daniel, cinder-coresec is now subscribed. Can Glance and Cinder use the same prlimit fix ?

Revision history for this message
Daniel Berrange (berrange) wrote :

And now I've fixed the patch to make the unit tests pass again. Opps :-)

Revision history for this message
Daniel Berrange (berrange) wrote :

Yes, from looking at their code, the change to cinder & glance could be essentially identical.

Revision history for this message
Richard Jones (rjones-redhat) wrote :

> "util-limit package in Linux distros."

That should say "util-linux".

Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote :

Here is a proposed patch for cinder

Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote :

Hum, prlimit is not readily available on stable Ubuntu and Debian...

Revision history for this message
Daniel Berrange (berrange) wrote :
Revision history for this message
Daniel Berrange (berrange) wrote :

The alternative approach is to modify oslo.concurrency processutils.execute method to allow limits to be passed in explicitly, and then use the pre-exec function to set them between fork + exec. This is actually the more desirable architectural approach, but I didn't go that way, since while it is fine for master, it would not be something that's easily backported to stable branches due to the need to update the min required oslo.concurrency version dependancy.

Changed in glance:
milestone: liberty-1 → liberty-2
Revision history for this message
Michael Still (mikal) wrote :

I actually much prefer the ulimit argument to processutils.execute() plan. I'm going to go to the oslo PTL and ask for an opinion of backporting that.

Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote :

Since this bugs requires change in 4 different projects, and the impacts is moderate, I propose we open this bug in a few days for a faster solution. Is there any objections ?

Revision history for this message
Davanum Srinivas (DIMS) (dims-v) wrote :

@mikal,

yes, i'd support a ulimit argument to processutils.execute and willing to help backport it to stable/kilo (and release if needed)

description: updated
information type: Private Security → Public Security
Revision history for this message
Daniel Berrange (berrange) wrote :

This is a *completely untested* patch to illustrate adding support for process resource limits to oslo.concurrency's execute() call. If people agree with this approach, I'll just submit it to gerrit against oslo, as adding this feature to olso is probably useful regardless of this OSSA

Revision history for this message
Davanum Srinivas (DIMS) (dims-v) wrote :

Hi, we've released a oslo.concurrency 1.8.1 (stable/kilo) with support to call a user specified preexec_fn to support this bug/fix. yes, same is in master as well.

80bfd1a Allow preexec_fn method for processutils.execute

thanks,
dims

Revision history for this message
Grant Murphy (gmurphy) wrote :

Draft impact description:

Title: Malicious input to qemu-img may result in resource exhaustion
Reporter: Richard W.M. Jones
Product: Nova, Cinder, Glance
Affects:
    Nova: 2012.1.0 versions through 2015.1.0,
    Cinder: 2013.1.0 versions through 2015.1.0,
    Glance: 2015.1.0

Description:
Richard W.M. Jones of Red Hat reported a vulnerability that affects several
OpenStack projects including Nova, Glance, and Cinder. By providing a
maliciously crafted disk image an attacker can consume considerable amounts
of RAM and CPU time resulting in a denial of service via resource exhaustion.
Any project which makes calls to qemu-img without appropriate ulimit restrictions
in place is affected by this flaw.

Revision history for this message
Jeremy Stanley (fungi) wrote :

I think this would more properly be (assuming the fixes won't make it in time for 2015.1.1)...

    Nova: versions through 2014.2.3, 2015.1 versions through 2015.1.1
    Cinder: versions through 2014.2.3, 2015.1 versions through 2015.1.1
    Glance: 2015.1 versions through 2015.1.1

Revision history for this message
Grant Murphy (gmurphy) wrote :

CVE requested.

Grant Murphy (gmurphy)
summary: - qemu-img calls need to be restricted by ulimit
+ qemu-img calls need to be restricted by ulimit (CVE-2015-5162)
Thierry Carrez (ttx)
Changed in glance:
milestone: liberty-2 → liberty-3
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.openstack.org/209627

Changed in nova:
assignee: Daniel Berrange (berrange) → Tristan Cacqueray (tristan-cacqueray)
status: Confirmed → In Progress
Thierry Carrez (ttx)
Changed in glance:
status: Triaged → In Progress
Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote : Re: qemu-img calls need to be restricted by ulimit (CVE-2015-5162)

Is there a review in progress for Glance ?

Revision history for this message
Nikhil Komawar (nikhil-komawar) wrote :

Yep.

Thierry Carrez (ttx)
Changed in glance:
milestone: liberty-3 → liberty-rc1
tags: added: liberty-rc-potential
Changed in glance:
milestone: liberty-rc1 → ongoing
Changed in nova:
assignee: Tristan Cacqueray (tristan-cacqueray) → Dan Smith (danms)
Revision history for this message
Daniel Berrange (berrange) wrote :

I've attached another patch which this time uses the new preexec_fn feature of oslo.concurrency 2.3.0.

This patch is *UNTESTED* beyond running the unit tests, so don't assume this actually fixes the problem without testing it in real world.

Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote :

Daniel, could you please have a look at https://review.openstack.org/#/c/209627/ ?

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to nova (master)

Reviewed: https://review.openstack.org/209627
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=da217205f53f9a38a573fb151898fbbeae41021d
Submitter: Jenkins
Branch: master

commit da217205f53f9a38a573fb151898fbbeae41021d
Author: Tristan Cacqueray <email address hidden>
Date: Wed Aug 5 17:17:04 2015 +0000

    virt: Use preexec_fn to ulimit qemu-img info call

    This uses the preexec_fn oslo.concurrency execute parameter to set user limit
    before calling qemu-img info. The process is limitted to 2 seconds of cpu
    and 1GB of virtual memory

    Change-Id: Ib47f1116c94c8f76d2a4a525af24439a4aa15854
    Closes-Bug: #1449062

Changed in nova:
status: In Progress → Fix Committed
Revision history for this message
Daniel Berrange (berrange) wrote : Re: qemu-img calls need to be restricted by ulimit (CVE-2015-5162)

FYI, we are reverting the patch for this bug because it caused a spike in gate failures

https://bugs.launchpad.net/nova/+bug/1506012
https://review.openstack.org/#/c/234696/

Revision history for this message
Daniel Berrange (berrange) wrote :

The way https://review.openstack.org/#/c/209627/ applied the limits to qemu-img was by setting rlimits in between fork() and execve(). The problem with this is that before we reach exec() we have a full copy-on-write mapping of the nova python process. So if Nova memory usage is too high, this could tickle the limit we're setting for qemu-img.

The only way to avoid this is to *not* set limits until after we have exec'd. This would require using an external command like my original prlimit patch.

Since not all distros have prlimit, we could create a simple nova-prlimit python script todo this. We would use native prlimit if it was present in the distro (as its fast) and fallacck to nova-prlimit if missing.

It is a shame oslo.concurrency went for the preexec_fn approach, rather than my suggestion of explicitly representing resource limits in the API, as that would have allowed us to hide this compat code in oslo.concurrency, instead of dealing with it in nova, glance and cinder :-(

Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote :

Can this prlimit python script could be provided by oslo or a requirement since it could be used by Cinder and Glance as well?

Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote :

Note that such prlimit utility is being worked on in oslo.concurrency here: https://review.openstack.org/#/c/243829/

Changed in nova:
status: Fix Committed → Fix Released
Changed in nova:
status: Fix Released → Confirmed
Changed in glance:
assignee: nikhil komawar (nikhil-komawar) → nobody
Sean Dague (sdague)
Changed in nova:
assignee: Dan Smith (danms) → nobody
importance: Undecided → Medium
Changed in nova:
assignee: nobody → Daniel Berrange (berrange)
status: Confirmed → In Progress
Changed in glance:
importance: High → Low
Changed in nova:
status: In Progress → Fix Released
tags: added: in-stable-mitaka
Changed in python-oslo.concurrency (Ubuntu Yakkety):
status: New → Fix Released
Changed in python-oslo.concurrency (Ubuntu Xenial):
status: New → Triaged
importance: Undecided → Medium
Changed in python-oslo.concurrency (Ubuntu Yakkety):
importance: Undecided → Medium
Changed in python-oslo.concurrency (Ubuntu Xenial):
assignee: nobody → Corey Bryant (corey.bryant)
Chris J Arges (arges)
Changed in python-oslo.concurrency (Ubuntu Xenial):
status: Triaged → Fix Committed
tags: added: verification-needed
tags: added: verification-done
removed: verification-needed
Changed in python-oslo.concurrency (Ubuntu Xenial):
status: Fix Committed → Fix Released
status: Fix Committed → Fix Released
Changed in python-oslo.concurrency (Ubuntu Wily):
importance: Undecided → Medium
Chris J Arges (arges)
Changed in python-oslo.concurrency (Ubuntu Wily):
status: New → Fix Committed
tags: removed: verification-done
tags: added: verification-needed
Mathew Hodson (mhodson)
affects: cinder → ubuntu-translations
no longer affects: ubuntu-translations
affects: glance → ubuntu-translations
Changed in ubuntu-translations:
milestone: ongoing → none
no longer affects: ubuntu-translations
Changed in cloud-archive:
status: New → Fix Released
importance: Undecided → Medium
Jeremy Stanley (fungi)
Changed in ossa:
status: Confirmed → Incomplete
Jeremy Stanley (fungi)
Changed in ossa:
assignee: nobody → Jeremy Stanley (fungi)
status: Incomplete → In Progress
Changed in cinder:
importance: Undecided → Medium
assignee: nobody → Sean McGinnis (sean-mcginnis)
Changed in cinder:
status: New → In Progress
25 comments hidden view all 105 comments
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/375102

Revision history for this message
Jeremy Stanley (fungi) wrote : Re: qemu-img calls need to be restricted by ulimit (CVE-2015-5162)

Following discussion with Sean and Hemanth, it looks like we ought to get fixes for this into supported branches of Cinder and Glance after all. Hopefully getting them merged goes quickly now that Nova has already trodden this ground and the fixes are basically identical between them.

Assuming all fixes merge before Newton releases and prior to any stable branch point releases, this is the updated proposed impact description. I'm using our YAML format here both for convenience and clarity due to the number of repos and releases involved. I've also shortened the title and referenced the affected software in it.

Should I include a note about the updated versions of oslo.concurrency required, or are the references to patches in consuming projects sufficient for this purpose? I'd like to avoid unnecessary additional complexity if possible here.

--

date: TBD

id: TBD

title: Malicious qemu-img input may exhaust resources in Cinder, Glance, Nova

description: >
  Richard W.M. Jones of Red Hat reported a vulnerability that affects OpenStack
  Cinder, Glance and Nova. By providing a maliciously crafted disk image an
  attacker can consume considerable amounts of RAM and CPU time resulting in a
  denial of service via resource exhaustion. Any project which makes calls to
  qemu-img without appropriate ulimit restrictions in place is affected by this
  flaw.

affected-products:
  - product: cinder
    version: "<=7.0.2, >=8.0.0 <=8.1.1"
  - product: glance
    version: "<=11.0.1, ==12.0.0"
  - product: nova
    version: "<=12.0.4 and ==13.0.0"

vulnerabilities:
  - cve-id: CVE-2015-5162

reporters:
  - name: Richard W.M. Jones
    affiliation: Red Hat
    reported:
      - CVE-2015-5162

issues:
  links:
    - https://launchpad.net/bugs/1449062

reviews:
  ocata:
    - https://review.openstack.org/375099 (cinder)
    - https://review.openstack.org/TBD (glance)
  newton:
    - https://review.openstack.org/375102 (cinder)
    - https://review.openstack.org/TBD (glance)
    - https://review.openstack.org/307663 (nova)
  mitaka:
    - https://review.openstack.org/TBD (cinder)
    - https://review.openstack.org/TBD (glance)
    - https://review.openstack.org/326327 (nova)
  liberty:
    - https://review.openstack.org/TBD (cinder)
    - https://review.openstack.org/TBD (glance)
    - https://review.openstack.org/327624 (nova)

notes:
  - >
    Separate Ocata patches are listed for Cinder and Glance, as they were fixed
    during the Newton release freeze after it branched from master.

Revision history for this message
Ian Cordasco (icordasc) wrote :

Thank you for that info Jeremy, I've targetted it to the appropriate series in Glance so it's clear.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (master)

Reviewed: https://review.openstack.org/375099
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=78f17f0ad79380ee3d9c50f2670252bcc559b62b
Submitter: Jenkins
Branch: master

commit 78f17f0ad79380ee3d9c50f2670252bcc559b62b
Author: Sean McGinnis <email address hidden>
Date: Thu Sep 22 15:31:37 2016 -0500

    Limit memory & CPU when running qemu-img info

    It was found that a modified or corrupted image file can cause a DoS
    on the host when getting image info with qemu-img.

    This uses the newer 'prlimit' parameter for oslo.concurrency execute
    to set an address space limit of 1GB and CPU time limit of 2 seconds
    when running the qemu-img info command.

    Change-Id: If5b7129b266ef065642bc7898ce9dcf93722a053
    Closes-bug: #1449062

Changed in cinder:
status: In Progress → Fix Released
Ian Cordasco (icordasc)
Changed in glance:
assignee: nobody → Hemanth Makkapati (hemanth-makkapati)
importance: Undecided → High
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (stable/newton)

Reviewed: https://review.openstack.org/375102
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=8547444775e406a50d9d26a0003e9ba6554b0d70
Submitter: Jenkins
Branch: stable/newton

commit 8547444775e406a50d9d26a0003e9ba6554b0d70
Author: Sean McGinnis <email address hidden>
Date: Thu Sep 22 15:31:37 2016 -0500

    Limit memory & CPU when running qemu-img info

    It was found that a modified or corrupted image file can cause a DoS
    on the host when getting image info with qemu-img.

    This uses the newer 'prlimit' parameter for oslo.concurrency execute
    to set an address space limit of 1GB and CPU time limit of 2 seconds
    when running the qemu-img info command.

    Change-Id: If5b7129b266ef065642bc7898ce9dcf93722a053
    Closes-bug: #1449062
    (cherry picked from commit 78f17f0ad79380ee3d9c50f2670252bcc559b62b)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/375625

Revision history for this message
Hemanth Makkapati (hemanth-makkapati) wrote : Re: qemu-img calls need to be restricted by ulimit (CVE-2015-5162)

It's just a theory (credits to Brian Rosmaita) at this point, but looks like "qemu-img convert" will try to infer the format of input image if "-f" is not provided. So, "qemu-img convert" may be susceptible to the same attack. Any thoughts?

Revision history for this message
Daniel Berrange (berrange) wrote :

Yes, *any* qemu-img command that you run without providing '-f' will try to guess the image format. Rather than trying to figure out whether a particular invokation may or may not be susceptible to attack, the safe approach is to use '-f' every time.

Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote :

Jeremy, the impact description looks good. Though note that prlimit implementation before oslo.concurrency-3.8.0 doesn't support all the required resources limit, thus I would mention that oslo.concurrency>=3.8.0 is required to fix that issue.

Revision history for this message
Jeremy Stanley (fungi) wrote :

Hemanth, Daniel: So that means the current patches to Nova are insufficient because they missed `qemu-image convert` invocations? For example at http://git.openstack.org/cgit/openstack/nova/tree/nova/virt/xenapi/vm_utils.py#n1128

Tristan: Thanks, it looked like oslo.concurrency got backports to stable/mitaka and stable/liberty to support it too though. Is that correct? If so we may need to get much more verbose about the library versions required by the service fixes.

Revision history for this message
Kashyap Chamarthy (kashyapc) wrote :

Jeremy: Hemanth (in comment#72) seems to have mixed up this bug (which sets limits for memory / CPU usage for `qemu-img` calls) with *another* bug[x] that is about disk image format guessing.

So, the Nova patches that fix this bug (1449062) are sufficient for the problem it is solving (setting a cap on memory / CPU usage).

[x] https://bugs.launchpad.net/nova/+bug/1415087 --
    [OSSA 2015-011] Format-guessing and file disclosure
    in image convert (CVE-2015-1850, CVE-2015-1851)

Revision history for this message
Nikhil Komawar (nikhil-komawar) wrote :

Thanks Kashyap.

That bug isn't cross listed with Glance and the current one applies to Glance in a very generic way. So, Hemanth did the right thing from our perspective. I will leave the rest of the story to the VMT team.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to glance (master)

Reviewed: https://review.openstack.org/375526
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=69a9b659fd48aa3c1f84fc7bc9ae236b6803d31f
Submitter: Jenkins
Branch: master

commit 69a9b659fd48aa3c1f84fc7bc9ae236b6803d31f
Author: Hemanth Makkapati <email address hidden>
Date: Fri Sep 23 09:29:12 2016 -0500

    Adding constraints around qemu-img calls

    * All "qemu-img info" calls are now run under resource limitations
      that limit CPU time to 2 seconds and address space usage to 1 GB.
      This helps avoid any DoS attacks via malicious images.
    * All "qemu-img convert" calls now specify the import format so that
      it does not have to be inferred by qemu-img.

    SecurityImpact

    Change-Id: Ib900bbc05cb9ccd90c6f56ccb4bf2006e30cdc80
    Closes-Bug: #1449062

Changed in glance:
status: In Progress → Fix Released
tags: added: newton-rc-potential
Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote : Re: qemu-img calls need to be restricted by ulimit (CVE-2015-5162)

Jeremy, the missing bit for oslo-concurrency 3.7.1 is https://review.openstack.org/#/q/I164c4b35e1357a0f80ed7fe00a7ae8f49df92e31 and it was merged to stable branches. Fortunately it seems like all version >= 3.8.0 are good enough to support correct resources limit, so I guess we could just mention that.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to glance (stable/newton)

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/377734

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to glance (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/377736

Revision history for this message
Jeremy Stanley (fungi) wrote : Re: qemu-img calls need to be restricted by ulimit (CVE-2015-5162)

Tristan: I'm still a little confused on the oslo.concurrency recommendation. Are you saying that we should suggest stable/liberty and stable/mitaka deployments to also use oslo.concurrency>=3.8.0? At the moment the tips of stable/liberty and stable/mitaka branches for oslo.concurrency are tagged 2.6.1 and 3.7.1 respectively (and that's what we have pinned in upper-constraints.txt for testing purposes as well). I don't want to imply in an advisory that all deployments should upgrade oslo.concurrency to 3.8.0 or later if there's a risk it will break liberty or mitaka deployments (which is why I was leaning toward not mentioning oslo.concurrency versions as that would just add to confusion).

Revision history for this message
Sean McGinnis (sean-mcginnis) wrote :

I had the same question for mitaka, so had to dig a little deeper. oslo.concurrency 3.7.1 was the follow on release to 3.7.0 to include the prlimit option. So based on current requirements for stable/mitaka, it should be safe to backport this fix.

I did not investigate stable/liberty, but I'm assuming 2.6.1 is the same situation.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to glance (stable/newton)

Reviewed: https://review.openstack.org/377734
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=6cba6b18a348e8fc2eb4ae25b636d065456a633c
Submitter: Jenkins
Branch: stable/newton

commit 6cba6b18a348e8fc2eb4ae25b636d065456a633c
Author: Hemanth Makkapati <email address hidden>
Date: Fri Sep 23 09:29:12 2016 -0500

    Adding constraints around qemu-img calls

    * All "qemu-img info" calls are now run under resource limitations
      that limit CPU time to 2 seconds and address space usage to 1 GB.
      This helps avoid any DoS attacks via malicious images.
    * All "qemu-img convert" calls now specify the import format so that
      it does not have to be inferred by qemu-img.

    SecurityImpact

    Change-Id: Ib900bbc05cb9ccd90c6f56ccb4bf2006e30cdc80
    Closes-Bug: #1449062
    (cherry picked from commit 69a9b659fd48aa3c1f84fc7bc9ae236b6803d31f)

Revision history for this message
Tristan Cacqueray (tristan-cacqueray) wrote : Re: qemu-img calls need to be restricted by ulimit (CVE-2015-5162)

Indeed my bad, oslo-concurrency 2.6.1, 3.7.1 and 3.8.0 are all good and referenced in the upper-constraint of supported stable branches. Thus I agree we could omit that note on the advisory.

Ian Cordasco (icordasc)
Changed in glance:
status: Fix Released → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to glance (stable/mitaka)

Reviewed: https://review.openstack.org/377736
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=c90830d71969f68768d898c1c178489f602214e2
Submitter: Jenkins
Branch: stable/mitaka

commit c90830d71969f68768d898c1c178489f602214e2
Author: Hemanth Makkapati <email address hidden>
Date: Fri Sep 23 09:29:12 2016 -0500

    Adding constraints around qemu-img calls

    * All "qemu-img info" calls are now run under resource limitations
      that limit CPU time to 2 seconds and address space usage to 1 GB.
      This helps avoid any DoS attacks via malicious images.
    * All "qemu-img convert" calls now specify the import format so that
      it does not have to be inferred by qemu-img.

    SecurityImpact

    Change-Id: Ib900bbc05cb9ccd90c6f56ccb4bf2006e30cdc80
    Closes-Bug: #1449062
    (cherry picked from commit 69a9b659fd48aa3c1f84fc7bc9ae236b6803d31f)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/glance 13.0.0.0rc2

This issue was fixed in the openstack/glance 13.0.0.0rc2 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/cinder 9.0.0.0rc2

This issue was fixed in the openstack/cinder 9.0.0.0rc2 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 12.0.5

This issue was fixed in the openstack/nova 12.0.5 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to glance (stable/liberty)

Reviewed: https://review.openstack.org/378012
Committed: https://git.openstack.org/cgit/openstack/glance/commit/?id=58311904a73f931404416649dc6ed3958adc59c8
Submitter: Jenkins
Branch: stable/liberty

commit 58311904a73f931404416649dc6ed3958adc59c8
Author: Brian Rosmaita <email address hidden>
Date: Tue Sep 27 16:11:17 2016 -0400

    Adding constraints around qemu-img calls

    * All "qemu-img info" calls are now run under resource limitations that
      limit CPU time to 2 seconds and address space usage to 1 GB. This
      helps avoid any DoS attacks via malicious images.
    * All "qemu-img convert" calls now specify the import format so that it
      does not have to be inferred by qemu-img.

    SecurityImpact

    (Hemanth did all the work on this, I'm just doing the backport.)

    Co-authored-by: Hemanth Makkapati <email address hidden>
    Closes-Bug: #1449062
    (cherry picked from commit 69a9b659fd48aa3c1f84fc7bc9ae236b6803d31f)

    Change-Id: I65f30b85439a8811545b0ca590555528631954df

Revision history for this message
Jeremy Stanley (fungi) wrote : Re: qemu-img calls need to be restricted by ulimit (CVE-2015-5162)

Status update: it looks like all Glance and Nova fixes have merged; so too have the master and stable/newton changes for Cinder. At this point we're waiting for https://review.openstack.org/375625 (Cinder's stable/mitaka fix) to merge, and we don't seem to have a stable/liberty backport for Cinder.

Sean: were you planning to work on a liberty change for this? Or is stable/liberty of Cinder unaffected (in which case I need to adjust the impact description accordingly)?

Revision history for this message
Sean McGinnis (sean-mcginnis) wrote :

I think the impact description is good. I will push through those backports.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (stable/liberty)

Fix proposed to branch: stable/liberty
Review: https://review.openstack.org/382573

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (stable/mitaka)

Reviewed: https://review.openstack.org/375625
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=c6adc020a67ae77e3645d4f6e80fa93b19432177
Submitter: Jenkins
Branch: stable/mitaka

commit c6adc020a67ae77e3645d4f6e80fa93b19432177
Author: Sean McGinnis <email address hidden>
Date: Thu Sep 22 15:31:37 2016 -0500

    Limit memory & CPU when running qemu-img info

    It was found that a modified or corrupted image file can cause a DoS
    on the host when getting image info with qemu-img.

    This uses the newer 'prlimit' parameter for oslo.concurrency execute
    to set an address space limit of 1GB and CPU time limit of 2 seconds
    when running the qemu-img info command.

    Change-Id: If5b7129b266ef065642bc7898ce9dcf93722a053
    Closes-bug: #1449062
    (cherry picked from commit 8547444775e406a50d9d26a0003e9ba6554b0d70)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to cinder (stable/liberty)

Reviewed: https://review.openstack.org/382573
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=455b318ced717fb38dfe40014817d78fbc47dea5
Submitter: Jenkins
Branch: stable/liberty

commit 455b318ced717fb38dfe40014817d78fbc47dea5
Author: Sean McGinnis <email address hidden>
Date: Thu Sep 22 15:31:37 2016 -0500

    Limit memory & CPU when running qemu-img info

    It was found that a modified or corrupted image file can cause a DoS
    on the host when getting image info with qemu-img.

    This uses the newer 'prlimit' parameter for oslo.concurrency execute
    to set an address space limit of 1GB and CPU time limit of 2 seconds
    when running the qemu-img info command.

    Change-Id: If5b7129b266ef065642bc7898ce9dcf93722a053
    Closes-bug: #1449062
    (cherry picked from commit 8547444775e406a50d9d26a0003e9ba6554b0d70)

Jeremy Stanley (fungi)
summary: - qemu-img calls need to be restricted by ulimit (CVE-2015-5162)
+ [OSSA 2016-012] qemu-img calls need to be restricted by ulimit
+ (CVE-2015-5162)
Changed in ossa:
status: In Progress → Fix Released
Revision history for this message
Brian Rosmaita (brian-rosmaita) wrote :

See comment #90 for glance fix to stable/liberty

Revision history for this message
Brian Rosmaita (brian-rosmaita) wrote :

Fix released to all stable branches

Changed in glance:
status: Fix Committed → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/nova 12.0.5

This issue was fixed in the openstack/nova 12.0.5 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/cinder 9.0.0

This issue was fixed in the openstack/cinder 9.0.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/glance 13.0.0

This issue was fixed in the openstack/glance 13.0.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/cinder 7.0.3

This issue was fixed in the openstack/cinder 7.0.3 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/cinder 10.0.0.0b1

This issue was fixed in the openstack/cinder 10.0.0.0b1 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/glance 14.0.0.0b1

This issue was fixed in the openstack/glance 14.0.0.0b1 development milestone.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/glance 13.0.0

This issue was fixed in the openstack/glance 13.0.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/glance 11.0.2

This issue was fixed in the openstack/glance 11.0.2 release.

Displaying first 40 and last 40 comments. View all 105 comments or add a comment.
This report contains Public Security information  
Everyone can see this security related information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.