Designate mDNS DoS through incorrect handling of large RecordSets

Bug #1471161 reported by Florian Weimer
264
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Designate
Fix Released
Critical
Kiall Mac Innes
Kilo
Fix Committed
Critical
Kiall Mac Innes
designate (Ubuntu)
Fix Released
Undecided
Unassigned

Bug Description

Designate does not enforce the DNS protocol limit concerning record set sizes when adding records for a domain name. The protocol limit is slightly less than 2**16 bytes because of some data that must accompany any record set, as part of the zone file transfer.

As a result, the rendering loop in desginate-mdns can does not make progress because the problematic record set at rrsets[i] will never fit (per designate/mdns/handler.py:RequestHandler._handle_axfr):

        # Render the results, yielding a packet after each TooBig exception.
        i, renderer = 0, None
        while i < len(rrsets):
            # No renderer? Build one
            if renderer is None:
                renderer = dns.renderer.Renderer(
                    response.id, response.flags, max_message_size)

            try:
                renderer.add_rrset(dns.renderer.ANSWER, rrsets[i])
                i += 1
            except dns.exception.TooBig:
                renderer.write_header()
                if request.had_tsig:

                yield renderer
                renderer = None

When this happens, designate-mdns will keep sending SOA records over and over again:

sendto(37, "\0\35<{\204\0\0\1\0\0\0\0\0\0\7example\3org\0\0\374\0\1", 31, 0, NULL, 0) = 31
sendto(37, "\0\35<{\204\0\0\1\0\0\0\0\0\0\7example\3org\0\0\374\0\1", 31, 0, NULL, 0) = 31
sendto(37, "\0\35<{\204\0\0\1\0\0\0\0\0\0\7example\3org\0\0\374\0\1", 31, 0, NULL, 0) = 31

Here is how I triggered this. I created a domain example.org:

+--------------------------------------+--------------+------------+
| id | name | serial |
+--------------------------------------+--------------+------------+
| ded903fc-60ef-4b5e-a37c-f3181baf9516 | example.org. | 1435315435 |
+--------------------------------------+--------------+------------+

Then I added several hundred NS records for the same subdomain, using this shell command:

for x in {1..304} ; do
  designate record-create --name 'sub.example.org.' --type NS --data \
$x'.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.'$x. \
  ded903fc-60ef-4b5e-a37c-f3181baf9516
done

In the tested configuration, PowerDNS will attempt a zone file transfer, and trigger the bug. Because it keeps receiving data, it does not seem it will ever run into a timeout (and if it does, it will try again).

Downstream bug report: https://bugzilla.redhat.com/show_bug.cgi?id=1236014

CVE References

Tim Simmons (timsim)
no longer affects: designate/kilo
Kiall Mac Innes (kiall)
Changed in designate:
status: New → Triaged
importance: Undecided → Critical
milestone: none → liberty-2
Revision history for this message
Kiall Mac Innes (kiall) wrote :

Attached a patch providing an initial candidate fix, based off master.

Tests are passing, but this has not yet been validated with BIND/PowerDNS etc

Revision history for this message
Kiall Mac Innes (kiall) wrote :

Apologies, patch is based off https://review.openstack.org/200230 rather than master

Revision history for this message
Kiall Mac Innes (kiall) wrote :

Finally, this fixes the DOS - Not the failure to apply quotas..

Revision history for this message
Kiall Mac Innes (kiall) wrote :

Affects debian: https://packages.debian.org/stretch/designate
Affects ubuntu: http://packages.ubuntu.com/vivid/designate

RedHat doesn't seem to have an equivalent package search site, but I'm assuming since they filed this issue, it affects RedHat too.

Revision history for this message
Graham Hayes (grahamhayes) wrote :

I think there is 2 parts to this bug:

1: Quotas were being bypassed as part of the v1 API.
2. If there was enough RRs in a RRSet MiniDNS went into a loop.
3. MiniDNS does not have a timeout.

1 is not really relevant - 2 should not happen in anycase.

The patch fixes 2, but not 3 - should we be include that in the fix?

Revision history for this message
Kiall Mac Innes (kiall) wrote :
Revision history for this message
Kiall Mac Innes (kiall) wrote :
Revision history for this message
Kiall Mac Innes (kiall) wrote :
Revision history for this message
Kiall Mac Innes (kiall) wrote :

I've gone and implemented the RRSet related quota enforcement for Master and Kilo, this was not previously implemented. I've included them in this bug, as publishing them now might indicate there's an issue with RRset sizes, but I don't consider them directly part of this DoS vulnerability.

i.e. An operator could choose to have infinite quotas (a valid use case), and we should support that.

Revision history for this message
Kiall Mac Innes (kiall) wrote :

Attached initial OSSA yaml based on existing documents from https://github.com/openstack/ossa/tree/master/ossa

Revision history for this message
Graham Hayes (grahamhayes) wrote :

Updated Kilo patch

Revision history for this message
Kiall Mac Innes (kiall) wrote :

All patches have now been validated as fixing the DoS.

Revision history for this message
Graham Hayes (grahamhayes) wrote :

Main Patch:

Bind9 + Master - Fix Verified
Bind9 + Kilo - Fix Verified

I have not tested the Quotas patch yet, but it should be DNS Server agnostic.

Revision history for this message
Kiall Mac Innes (kiall) wrote :

Attached reproduction script used.

Ran before applying patches, mDNS will consume 100% CPU once it's completed and zones will not propagate to the nameservers.

Ran after applying patches, mDNS will correctly propagate zones to the nameservers.

Kiall Mac Innes (kiall)
Changed in designate:
assignee: nobody → Kiall Mac Innes (kiall)
Revision history for this message
Tim Simmons (timsim) wrote :

Verified main fix with:
Bind9 + Master
Bind9 + Kilo

Verified quotas fix on:
Master
Kilo

Kiall Mac Innes (kiall)
summary: - Lack of record set size checking leads to denial of service
+ Designate mDNS DoS through incorrect handling of large RecordSets
Revision history for this message
Kiall Mac Innes (kiall) wrote :

CVE # Allocation request sent, based on VMT template. Copy of email:

A vulnerability was discovered in OpenStack Designate (see below).
In order to ensure full traceability, we need a CVE number assigned
that we can attach to private and public notifications. Please treat
the following information as confidential until further public
disclosure.

Title: Designate mDNS DoS through incorrect handling of large RecordSets
Reporter: Florian Weimer (Red Hat)
Products: OpenStack Designate
Affects: OpenStack Designate version 2015.1.0

Description:
Florian Weimer from Red Hat reported a vulnerability in Designate.
By creating a single RecordSet that exceeds the configured max
allowed DNS packet size, an authenticated user may cause the
Designate mDNS service to enter an infinite loop, triggering a DoS.

Thanks in advance,
Kiall Mac Innes

Revision history for this message
Kiall Mac Innes (kiall) wrote :

No CVE number yet, went ahead and notified downstreams. Disclosure date set for 2015-07-28, 1500UTC.

Copy of email below:

This is an advance warning of a vulnerability discovered in OpenStack
Designate, to give you, as downstream stakeholders, a chance to coordinate
the release of fixes and reduce the vulnerability window. Please treat the
following information as confidential until the proposed public
disclosure date.

Title: Designate mDNS DoS through incorrect handling of large RecordSets
Reporter: Florian Weimer (Red Hat)
Products: Designate
Affects: 2015.1.0 through 1.0.0.0b1

Description:
Florian Weimer from Red Hat reported a vulnerability in Designate.
By creating a single RecordSet that exceeds the configured max allowed DNS
packet size, an authenticated user may cause the Designate mDNS service
to enter an infinite loop, triggering a DoS.

Proposed patch:
See attached patches. Unless a flaw is discovered in them, these patches
will be merged to stable/kilo and master on the public disclosure date.

CVE: TBA
Launchpad Bug #: 1471161
RedHat Bug #: 1236014

Proposed public disclosure date/time: 2015-07-28, 1500UTC

Please do not make the issue public (or release public patches) before
this coordinated embargo date.

Regards,

--
Kiall Mac Innes
OpenStack Designate PTL

Kiall Mac Innes (kiall)
Changed in designate:
status: Triaged → In Progress
Kiall Mac Innes (kiall)
information type: Private Security → Public Security
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to designate (master)

Fix proposed to branch: master
Review: https://review.openstack.org/206578

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: master
Review: https://review.openstack.org/206579

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to designate (stable/kilo)

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/206580

Revision history for this message
OpenStack Infra (hudson-openstack) wrote :

Fix proposed to branch: stable/kilo
Review: https://review.openstack.org/206581

Revision history for this message
Kiall Mac Innes (kiall) wrote :

Public disclosure date/time (2015-07-28, 1500UTC) reached, bug has been marked public, patches uploaded + approved, and going through gate now.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to designate (stable/kilo)

Reviewed: https://review.openstack.org/206580
Committed: https://git.openstack.org/cgit/openstack/designate/commit/?id=969db41bf6d4c14d9925b801c36685e6b7d3f246
Submitter: Jenkins
Branch: stable/kilo

commit 969db41bf6d4c14d9925b801c36685e6b7d3f246
Author: Kiall Mac Innes <email address hidden>
Date: Thu Jul 9 21:39:17 2015 +0100

    Ensure a single RRSet over max_packet_size doesn't loop forever

    A single RRSet exceeding the max_packet_size configuration option would
    result in mDNS entering a infinite loop, causing a Denial of Service.

    Change-Id: Icab7cfe18a0e46cf3ec5d7087eb878e9d8653a4b
    Closes-Bug: 1471161

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to designate (master)

Reviewed: https://review.openstack.org/206578
Committed: https://git.openstack.org/cgit/openstack/designate/commit/?id=dac3fbc1f78dcd49dadebbd990db7b2d1a68e8c0
Submitter: Jenkins
Branch: master

commit dac3fbc1f78dcd49dadebbd990db7b2d1a68e8c0
Author: Kiall Mac Innes <email address hidden>
Date: Thu Jul 9 21:39:17 2015 +0100

    Ensure a single RRSet over max_packet_size doesn't loop forever

    A single RRSet exceeding the max_packet_size configuration option would
    result in mDNS entering a infinite loop, causing a Denial of Service.

    Change-Id: Icab7cfe18a0e46cf3ec5d7087eb878e9d8653a4b
    Closes-Bug: 1471161

Changed in designate:
status: In Progress → Fix Committed
tags: added: in-stable-kilo
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to designate (stable/kilo)

Reviewed: https://review.openstack.org/206581
Committed: https://git.openstack.org/cgit/openstack/designate/commit/?id=46f187f53506de1237cdd14b5d2a69e73fd8f720
Submitter: Jenkins
Branch: stable/kilo

commit 46f187f53506de1237cdd14b5d2a69e73fd8f720
Author: Kiall Mac Innes <email address hidden>
Date: Mon Jul 13 10:53:04 2015 +0100

    Ensure RecordSet quotas are enforced

    Implement RecordSet's per domain, and Records per RecordSet
    quota enforcement.

    Change-Id: If7afc70cd1ebe2e18864859de51c6ccd15c6a43c
    Partial-Bug: 1471161

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to designate (master)

Reviewed: https://review.openstack.org/206579
Committed: https://git.openstack.org/cgit/openstack/designate/commit/?id=789709f1db081222c2e4d21d287781e437b76a95
Submitter: Jenkins
Branch: master

commit 789709f1db081222c2e4d21d287781e437b76a95
Author: Kiall Mac Innes <email address hidden>
Date: Mon Jul 13 10:53:04 2015 +0100

    Ensure RecordSet quotas are enforced

    Implement RecordSet's per domain, and Records per RecordSet
    quota enforcement.

    Change-Id: If7afc70cd1ebe2e18864859de51c6ccd15c6a43c
    Partial-Bug: 1471161

Revision history for this message
Kiall Mac Innes (kiall) wrote :

Two CVE numbers have been assigned to track the two parts of this:

CVE-2015-5694 and CVE-2015-5695

https://security-tracker.debian.org/tracker/CVE-2015-5694
https://security-tracker.debian.org/tracker/CVE-2015-5695

Thierry Carrez (ttx)
Changed in designate:
status: Fix Committed → Fix Released
Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote :

The attachment "Master Branch Patch" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

tags: added: patch
Changed in designate (Ubuntu):
status: New → Confirmed
James Page (james-page)
Changed in designate (Ubuntu):
status: Confirmed → Fix Released
Thierry Carrez (ttx)
Changed in designate:
milestone: liberty-2 → 1.0.0
To post a comment you must log in.
This report contains Public Security information  
Everyone can see this security related information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.