[OSSA 2013-026] Some sequence of characters in console-log can DoS nova-compute (CVE-2013-4261)

mikal is working on fixing our old friend bug 832507, maybe he should take a look at this one as well.

This specific failure looks qpid specific, but I think we should probably be filtering to only printable characters for all transports. I will take a look at this ASAP.

I was debugging this issue and found out it is an instance of bug 1175808 only in grizzly. See https://bugzilla.redhat.com/show_bug.cgi?id=999164#c6

I'll propose the full backport of the patch in bug 1175808 to grizzly which fixes the issue.

Proposed fix for grizzly: https://review.openstack.org/#/c/43303/

@Xavier: so havana (master) is unaffected ? What about Folsom (not sure QPID support existed back then...) ?

@Mikal: are you taking the OSSA task too ? (i.e. draft the impact description and communicate it around once all fixes are merged ?)

Yes, the problem only appears in grizzly (and probably folsom) where the qpid implementation wasn't updated to send messages bigger than 65KB

An easy way to reproduce it is as follows:

1. Start a new instance
2. From inside the instance fill the console log (/dev/ttyS0) with as much text as possible (to make the log bigger than 65KB)
3. Ask for the full console-log (either from novaclient or horizon), which should fail with a 500 - InternalError
  - repeat #3 for at least "rpc_conn_pool_size" times until the call hangs.
4. nova-compute has no more connections in the pool and cannot be accessed

NOTE: rpc_conn_pool_size default is 30

I can confirm that I cannot duplicate this problem on trunk / rabbitmq.

Proposed impact description:

"""
Title: Potential denial of service on Nova when using Qpid
Reporter: Jaroslav Henner (Red Hat)
Products: Nova
Affects: Folsom, Grizzly

Description:
Jaroslav Henner from Red Hat reported a vulnerability in Nova when using Apache Qpid as the RPC backend. By sending malicious characters to an instance console and requesting the console log contents through the API, an authenticated user may disrupt the nova-compute node his instance is running on. This vulnerability could be leveraged in a Denial of Service attack against the cloud provider. Only Folsom and Grizzly setups using Qpid as their RPC backend are affected. Havana setups, or setups using other RPC backends (like RabbitMQ), are all unaffected.
"""

Trick question: are Cinder and Neutron (which also have the option to use Qpid as their rpc backend) vulnerable to the same DoS issue ? Or are they protected because they don't have an equivalent of nova console-log ?

Fix proposed to branch: stable/folsom
Review: https://review.openstack.org/44700

Fix proposed to branch: stable/grizzly
Review: https://review.openstack.org/44695

@Xavier: could you have a look at the impact description and check it makes sense ? Also if you have the answer to the trick question in comment 10...

@Thierry: I would only replace the part saying "malicious characters" with something like "any random text longer than 65K characters".

w.r.t. comment 10, I guess there shouldn't be a problem in other projects while we don't send user generated content like the console log but I should double check this as I am not familiar with cinder and neutron internals.

New impact description:

"""
Title: Potential denial of service on Nova when using Qpid
Reporter: Jaroslav Henner (Red Hat)
Products: Nova
Affects: Folsom, Grizzly

Description:
Jaroslav Henner from Red Hat reported a vulnerability in Nova when using Apache Qpid as the RPC backend. By sending any random text longer than 65K characters to an instance console and requesting the console log contents through the API, an authenticated user may disrupt the nova-compute node his instance is running on. This vulnerability could be leveraged in a Denial of Service attack against the cloud provider. Only Folsom and Grizzly setups using Qpid as their RPC backend are affected. Havana setups, or setups using other RPC backends (like RabbitMQ), are all unaffected.
"""

I'll try to get Cinder/neutron folks to look into the issue and confirm if they are affected.

Reviewed: https://review.openstack.org/44695
Committed: http://github.com/openstack/oslo-incubator/commit/4f97479ad078771f6f25461c95203a5d293ec08b
Submitter: Jenkins
Branch: stable/grizzly

commit 4f97479ad078771f6f25461c95203a5d293ec08b
Author: Xavier Queralt <email address hidden>
Date: Mon Sep 2 14:27:22 2013 +0200

    Fix problem when sending long messages in Qpid

    Qpid has a limitation where it cannot serialize a dict containing a
    string greater than 65535 characters. This change alters the Qpid
    implementation to JSON encode the dict before sending it, but only if
    Qpid would fail to serialize it. This maintains as much backward
    compatibility as possible, though long messages will still fail if they
    are sent to an older receiver.

    The first part of this fix was ported to Grizzly in I8b6c5734b to allow
    receiving messages from Havana using the new format. Even though this
    change will modify the message format, it will only do it when messages
    are longer than 65K which would be broken anyway and could cause serious
    bugs like the one linked below.

    Fixes bug 1215091

    Change-Id: I2f0e88435748bab631d969573d3a598d9e1f7fef

Reviewed: https://review.openstack.org/44700
Committed: http://github.com/openstack/oslo-incubator/commit/478ac3a3ec4b2dd9adb32891123b6e33c483bdf2
Submitter: Jenkins
Branch: stable/folsom

commit 478ac3a3ec4b2dd9adb32891123b6e33c483bdf2
Author: Ben Nemec <email address hidden>
Date: Thu May 9 19:06:45 2013 +0000

    Fix problem with long messages in Qpid

    Qpid has a limitation where it cannot serialize a dict containing a
    string greater than 65535 characters. This change alters the Qpid
    implementation to JSON encode the dict before sending it, but only if
    Qpid would fail to serialize it. This maintains as much backward
    compatibility as possible, though long messages will still fail if they
    are sent to an older receiver.

    Even though this change will modify the message format, it will only do
    it when messages are longer than 65K which would be broken anyway and
    could cause serious bugs like the one linked below.

    Fixes bug 1215091

    (cherry picked from commit 7ce54410485b33cffc39c7ffb96eae422b88601c)

    Conflicts:
     openstack/common/rpc/impl_qpid.py

    Change-Id: I2f0e88435748bab631d969573d3a598d9e1f7fef

Fix proposed to branch: stable/folsom
Review: https://review.openstack.org/45426

Xavier: could you clarify which one of those patches is actually the CVE fix ? I'm a bit confused (and admittedly tired too :)

Reviewed: https://review.openstack.org/45426
Committed: http://github.com/openstack/nova/commit/ef5730a4620b409a3b46e46966e3bc6f3a306464
Submitter: Jenkins
Branch: stable/folsom

commit ef5730a4620b409a3b46e46966e3bc6f3a306464
Author: Ben Nemec <email address hidden>
Date: Thu May 9 19:06:45 2013 +0000

    Fix problem with long messages in Qpid (from oslo)

    This is commit 478ac3a3e in oslo-incubator

    Qpid has a limitation where it cannot serialize a dict containing a
    string greater than 65535 characters. This change alters the Qpid
    implementation to JSON encode the dict before sending it, but only if
    Qpid would fail to serialize it. This maintains as much backward
    compatibility as possible, though long messages will still fail if they
    are sent to an older receiver.

    Even though this change will modify the message format, it will only do
    it when messages are longer than 65K which would be broken anyway and
    could cause serious bugs like the one linked below.

    Fixes bug 1215091

    Change-Id: I2f0e88435748bab631d969573d3a598d9e1f7fef

@Thierry: the patches that fix the CVE are the ones in nova:

https://review.openstack.org/#/c/45426/ (for folsom) and https://review.openstack.org/#/c/43303/ (for grizzly)

The other two are just the patches to oslo-incubator.

Reviewed: https://review.openstack.org/43303
Committed: http://github.com/openstack/nova/commit/2d949c415b97ed9649e78c880ab149d0d39c1152
Submitter: Jenkins
Branch: stable/grizzly

commit 2d949c415b97ed9649e78c880ab149d0d39c1152
Author: Xavier Queralt <email address hidden>
Date: Thu Sep 5 10:08:29 2013 +0200

    Fix Qpid when sending long messages (from oslo)

    This is commit 4f97479ad in oslo-incubator

    Qpid has a limitation where it cannot serialize a dict containing a
    string greater than 65535 characters. This change alters the Qpid
    implementation to JSON encode the dict before sending it, but only if
    Qpid would fail to serialize it. This maintains as much backward
    compatibility as possible, though long messages will still fail if they
    are sent to an older receiver.

    The first part of this fix was ported to Grizzly in Ib52e9458a to allow
    receiving messages from Havana using the new format. Even though this
    change will modify the message format, it will only do it when messages
    are longer than 65K which would be broken anyway and could cause serious
    bugs like the one linked below.

    Fixes bug 1215091

    Change-Id: I505b648c3d0e1176ec7a3fc7d1646fa5a5232261

[OSSA 2013-026]