[OSSA 2013-037] DoS attack via setting os_type in snapshots (CVE-2013-6437)

Sounds legit

Yep, seems legit to me, too.

I went looking through the code to figure out any implications of changing the backing file name and honestly I'm lost in image caching code, but as far as I can tell the suggested fix should work and would be fairly easy.

Phil, are you going to submit a patch for this?

Don't have anyone working on it yet, but was hoping I could get someone started this week. Open to other offers though

I'm out for the rest of the week and have a lot on my plate already. But if this sits too long I may try to tackle it.

Added Ryan Moor (HP) as he is going to start working on a patch for this

Attached is a patch that contains a solution

Never submitted tin this way before so not sure what is required

Commit message: >>
use 'os_type' in ephemeral filename only if mkfs defined

Currently for undefined os-types it will use the default mkfs command, but use the meta 'os_type' in the name of the ephemeral file (e.g. ephemeral_20_abcdef). Which can result in a lot of files (DoS?)
This change will only use 'os_type' in the ephemeral filename if there is a specific mkfs command defined, otherwise it will use 'default' (e.g. ephemeral_20_default)

Modifed the tests to test for:
  os_type=''
  os_type=None
  os_type='test' - with no mkfs command specified
  os_type='test' - with a mkfs command specified
<<

run it up in devstack and it behaves as expected

The patch from Ryan looks good to me and should address the reported issue.

Patch looks good to me too - nice work.

The patch doesn't seem to apply on stable/havana though - so I guess we need a separate patch file for that.

Any others we need ?

We'll need a stable/havana patch for this. I suspect this affects grizzly too so we'll need a stable/grizzly patch too.

master patch looks good to me, too. Thanks!

So we just need the stable backports now. moorryan, do you think you can work on some backports, too?

backported for stable/havana

backported for stable/grizzly

The stable patches look good to me - nice work Ryan.

All 3 patches look good to me. Thanks Ryan.

Proposed impact description:

Title: Nova compute DoS through ephemeral disk backing files
Reporter: Phil Day (HP)
Products: Nova
Affects: All supported versions

Description:
Phil Day from HP reported a vulnerability in the handling of ephemeral disk backing files on Nova compute nodes. By repeatedly spawning new instances of a snapshotted server with random os_type (and quickly deleting those instances), an authenticated user could generate lots of different ephemeral disk backing files and fill up compute node disks, potentially resulting in a Denial of Service against a Nova setup.

After writing the impact description it appears to me that this DoS vector could end up being costly for the attacker. I tend to consider "costly DoS" to fall in the same category as "normal usage" for a resource provider... Thoughts on that ? Should this still be considered a practical vulnerability ?

In my opinion, it comes down to whether there is typically a quota/rate-limit around these sorts of operations, and whether or not they are likely to be tracked/billed by a provider. But I don't have the answer to those (hopefully someone here with OpenStack operational background can confirm).

Hi Thierry,

A minor clarification on the impact description:

"By repeatedly spawning new instances of a snapshotted server with random os_type ..." should read " By repeatedly creating snapshots, changing the os_type to a new random value, and spawning new instances from the snapshot ..."

I don’t think its necessarily that costly - you do have to pay for each instance you create, but it can be the smallest instance needed and doesn’t have to exist any longer than the time to create it. You could probably even kill the instance before it gets to Active (It only needs to get past the creation of the ephemeral backing file) and avoid any billing at all on some systems. Likewise you don't have to keep the snapshots once the instance has been created. Because the ephemeral backing files are never cleared up (its assumed that they are small and a limited set) you don't have to do this at a fast rate to create a long term attack.

Phil

> -----Original Message-----
> From: <email address hidden> [mailto:<email address hidden>] On Behalf
> Of Thierry Carrez
> Sent: 09 December 2013 16:06
> To: Day, Phil
> Subject: [Bug 1253980] Re: DoS attack via setting os_type in snapshots.
>
> After writing the impact description it appears to me that this DoS vector
> could end up being costly for the attacker. I tend to consider "costly DoS" to
> fall in the same category as "normal usage" for a resource provider...
> Thoughts on that ? Should this still be considered a practical vulnerability ?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1253980
>
> Title:
> DoS attack via setting os_type in snapshots.
>
> Status in OpenStack Compute (Nova):
> In Progress
> Status in OpenStack Compute (nova) grizzly series:
> New
> Status in OpenStack Compute (nova) havana series:
> New
> Status in OpenStack Security Advisories:
> Triaged
>
> Bug description:
> If the os_type metadata is set of an image, the ephemeral disk backing file
> for that image will be named ephemeral_[size]_[os_type].
> Because the user can change os_type they can use this to create new
> ephemeral backing files.
> Nova image cache management does not include deleting ephemeral
> backing files (presumably because they are expected to be a small, stable
> set.
>
> Hence a user can fill the disk with ephemeral backing files via the
> following:
>
> 1) Spawn a instance
> 2) Create a snapshot from it, delete the original instance
> 3) In a loop:
> generate a random os_type
> set os_type to the snapshot
> spawn and instance from it, and then delete the instance
>
> Every iteration will generate an ephemeral backing file on a compute
> host. With a stacking scheduling policy there is a good chance of
> hitting the same host repeatedly until its disk is full.
>
> Suggested mitigation
>
> Only use “os_type” in the ephemeral file name if there is a specific
> mkfs command defined, otherwise use “default” (Currently for
> undefined os-types it will use the default mkfs command, but still
> uses os_...

New proposed impact desc:

--------------------------------------------
Title: Nova compute DoS through ephemeral disk backing files
Reporter: Phil Day (HP)
Products: Nova
Affects: All supported versions

Description:
Phil Day from HP reported a vulnerability in the handling of ephemeral disk backing files on Nova compute nodes. By repeatedly creating snapshots, changing the os_type to a new random value, and spawning new instances from the snapshot (and quickly deleting those instances), an authenticated user could generate lots of different ephemeral disk backing files and fill up compute node disks, potentially resulting in a Denial of Service against a Nova setup.
--------------------------------------------

@Phil: agreed... waiting for confirmation from other VMT members to go ahead. Would the ephemeral backing files all land on the same server (reaching a state of partial DoS quickly) or be spread across all compute nodes (reaching a state of total DoS but slowly) ?

> @Phil: agreed... waiting for confirmation from other VMT members to go
> ahead. Would the ephemeral backing files all land on the same server
> (reaching a state of partial DoS quickly) or be spread across all compute
> nodes (reaching a state of total DoS but slowly) ?

That depends entirely on the scheduling policy configured in Nova - a stacking model would hit the issue on specific nodes more quickly that a spreading model (so a quicker DoS on specific servers) - but really it’s the total DoS that is the issue which is kind of the same in either case, its just a question of if you fill from the side or from the bottom.

> -----Original Message-----
> From: <email address hidden> [mailto:<email address hidden>] On Behalf
> Of Thierry Carrez
> Sent: 10 December 2013 09:49
> To: Day, Phil
> Subject: [Bug 1253980] Re: DoS attack via setting os_type in snapshots.
>
> New proposed impact desc:
>
> --------------------------------------------
> Title: Nova compute DoS through ephemeral disk backing files
> Reporter: Phil Day (HP)
> Products: Nova
> Affects: All supported versions
>
> Description:
> Phil Day from HP reported a vulnerability in the handling of ephemeral disk
> backing files on Nova compute nodes. By repeatedly creating snapshots,
> changing the os_type to a new random value, and spawning new instances
> from the snapshot (and quickly deleting those instances), an authenticated
> user could generate lots of different ephemeral disk backing files and fill up
> compute node disks, potentially resulting in a Denial of Service against a Nova
> setup.
> --------------------------------------------
>
> @Phil: agreed... waiting for confirmation from other VMT members to go
> ahead. Would the ephemeral backing files all land on the same server
> (reaching a state of partial DoS quickly) or be spread across all compute
> nodes (reaching a state of total DoS but slowly) ?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1253980
>
> Title:
> DoS attack via setting os_type in snapshots.
>
> Status in OpenStack Compute (Nova):
> In Progress
> Status in OpenStack Compute (nova) grizzly series:
> New
> Status in OpenStack Compute (nova) havana series:
> New
> Status in OpenStack Security Advisories:
> Triaged
>
> Bug description:
> If the os_type metadata is set of an image, the ephemeral disk backing file
> for that image will be named ephemeral_[size]_[os_type].
> Because the user can change os_type they can use this to create new
> ephemeral backing files.
> Nova image cache management does not include deleting ephemeral
> backing files (presumably because they are expected to be a small, stable
> set.
>
> Hence a user can fill the disk with ephemeral backing files via the
> following:
>
> 1) Spawn a instance
> 2) Create a snapshot from it, delete the original instance
> 3) In a loop:
> generate a random os_type
> set os_type to the snapshot
> spawn and instance from it, and then delete the instance
>
> Every iteration will generate an ephemeral back...

Should the proposed impact mention that this is libvirt specific? Otherwise it lgtm.

If it is libvirt-specific, then yes it should mention it. Is this the case ?

This issue is libvirt specific. Other drivers may or may not have similar issues but the code paths are very different and can't be generalized under this bug. So adding " running the libvirt driver." or something similar to the first sentence should cover it.

OK, how about:

--------------------------------------------
Title: Nova compute DoS through ephemeral disk backing files
Reporter: Phil Day (HP)
Products: Nova
Affects: All supported versions

Description:
Phil Day from HP reported a vulnerability in the libvirt driver handling of ephemeral disk backing files on Nova compute nodes. By repeatedly creating snapshots, changing the os_type to a new random value, and spawning new instances from the snapshot (and quickly deleting those instances), an authenticated user could generate lots of different ephemeral disk backing files and fill up compute node disks, potentially resulting in a Denial of Service against a Nova setup. Only Nova setups running the libvirt driver are affected.
--------------------------------------------

LGTM - thanks Thierry

> -----Original Message-----
> From: <email address hidden> [mailto:<email address hidden>] On Behalf
> Of Thierry Carrez
> Sent: 11 December 2013 11:20
> To: Day, Phil
> Subject: [Bug 1253980] Re: DoS attack via setting os_type in snapshots.
>
> OK, how about:
>
> --------------------------------------------
> Title: Nova compute DoS through ephemeral disk backing files
> Reporter: Phil Day (HP)
> Products: Nova
> Affects: All supported versions
>
> Description:
> Phil Day from HP reported a vulnerability in the libvirt driver handling of
> ephemeral disk backing files on Nova compute nodes. By repeatedly creating
> snapshots, changing the os_type to a new random value, and spawning new
> instances from the snapshot (and quickly deleting those instances), an
> authenticated user could generate lots of different ephemeral disk backing
> files and fill up compute node disks, potentially resulting in a Denial of Service
> against a Nova setup. Only Nova setups running the libvirt driver are
> affected.
> --------------------------------------------
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1253980
>
> Title:
> DoS attack via setting os_type in snapshots.
>
> Status in OpenStack Compute (Nova):
> In Progress
> Status in OpenStack Compute (nova) grizzly series:
> New
> Status in OpenStack Compute (nova) havana series:
> New
> Status in OpenStack Security Advisories:
> Triaged
>
> Bug description:
> If the os_type metadata is set of an image, the ephemeral disk backing file
> for that image will be named ephemeral_[size]_[os_type].
> Because the user can change os_type they can use this to create new
> ephemeral backing files.
> Nova image cache management does not include deleting ephemeral
> backing files (presumably because they are expected to be a small, stable
> set.
>
> Hence a user can fill the disk with ephemeral backing files via the
> following:
>
> 1) Spawn a instance
> 2) Create a snapshot from it, delete the original instance
> 3) In a loop:
> generate a random os_type
> set os_type to the snapshot
> spawn and instance from it, and then delete the instance
>
> Every iteration will generate an ephemeral backing file on a compute
> host. With a stacking scheduling policy there is a good chance of
> hitting the same host repeatedly until its disk is full.
>
> Suggested mitigation
>
> Only use “os_type” in the ephemeral file name if there is a specific
> mkfs command defined, otherwise use “default” (Currently for
> undefined os-types it will use the default mkfs command, but still
> uses os_type in the name.
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/nova/+bug/1253980/+subscriptions

Lgtm as well.

CVE requested

CVE-2013-6437

Proposed public disclosure date/time: Wednesday, December 18, 1500UTC.

Tested:

stable/grizzly pep8: PASS
stable/grizzly py27: PASS
stable/grizzly dsvm-full: PASS
stable/grizzly dsvm-large-ops: PASS
stable/havana pep8: PASS
stable/havana py27: PASS
stable/havana dsvm-full: PASS
stable/havana dsvm-neutron: PASS
master pep8: PASS
master py27: PASS

Icehouse (development branch) fix:
https://review.openstack.org/62910

Havana fix:
https://review.openstack.org/62912

Grizzly fix:
https://review.openstack.org/62913

Reviewed: https://review.openstack.org/62910
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=6e455cd97f04bf26bbe022be17c57e089cf502f4
Submitter: Jenkins
Branch: master

commit 6e455cd97f04bf26bbe022be17c57e089cf502f4
Author: Ryan Moore <email address hidden>
Date: Fri Nov 29 14:21:19 2013 +0000

    use 'os_type' in ephemeral filename only if mkfs defined

    Currently for undefined os-types it will use the default mkfs
    command, but use the meta 'os_type' in the name of the
    ephemeral file (e.g. ephemeral_20_abcdef). Which can result
    in a lot of files (DoS?)
    This change will only use 'os_type' in the ephemeral filename
    if there is a specific mkfs command defined, otherwise it will
    use 'default' (e.g. ephemeral_20_default)

    Modifed the tests to test for:
      os_type=''
      os_type=None
      os_type='test' - with no mkfs command specified
      os_type='test' - with a mkfs command specified

    Closes-Bug: 1253980

    Change-Id: Ie4c10f99ce690c5e4ef181624bd688c38923855c

Reviewed: https://review.openstack.org/62912
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=ca38774ebcf5b67d16c202c8f218c0c433973ca9
Submitter: Jenkins
Branch: stable/havana

commit ca38774ebcf5b67d16c202c8f218c0c433973ca9
Author: Ryan Moore <email address hidden>
Date: Fri Nov 29 14:21:19 2013 +0000

    use 'os_type' in ephemeral filename only if mkfs defined

    Currently for undefined os-types it will use the default mkfs
    command, but use the meta 'os_type' in the name of the
    ephemeral file (e.g. ephemeral_20_abcdef). Which can result
    in a lot of files (DoS?)
    This change will only use 'os_type' in the ephemeral filename
    if there is a specific mkfs command defined, otherwise it will
    use 'default' (e.g. ephemeral_20_default)

    Modifed the tests to test for:
      os_type=''
      os_type=None
      os_type='test' - with no mkfs command specified
      os_type='test' - with a mkfs command specified

    Closes-Bug: 1253980

    Backport of Original Change-Id: Ie4c10f99ce690c5e4ef181624bd688c38923855c
    to stable/havana

    Change-Id: Ifa2b94e79dabd586d7e904da247d099360229313

Reviewed: https://review.openstack.org/62913
Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=3e451f1bac57d24e47171cffb3ad59bb1610d836
Submitter: Jenkins
Branch: stable/grizzly

commit 3e451f1bac57d24e47171cffb3ad59bb1610d836
Author: Ryan Moore <email address hidden>
Date: Fri Nov 29 14:21:19 2013 +0000

    use 'os_type' in ephemeral filename only if mkfs defined

    Currently for undefined os-types it will use the default mkfs
    command, but use the meta 'os_type' in the name of the
    ephemeral file (e.g. ephemeral_20_abcdef). Which can result
    in a lot of files (DoS?)
    This change will only use 'os_type' in the ephemeral filename
    if there is a specific mkfs command defined, otherwise it will
    use 'default' (e.g. ephemeral_20_default)

    Modifed the tests to test for:
      os_type=''
      os_type=None
      os_type='test' - with no mkfs command specified
      os_type='test' - with a mkfs command specified

    Closes-Bug: 1253980

    Backport of Original Change-Id: Ie4c10f99ce690c5e4ef181624bd688c38923855c
    to stable/grizzly

    Change-Id: Ia040910e90861a2987eff345ad1c01863655b124

OSSA 2013-037