[OSSA-2021-005] Arbitrary dnsmasq reconfiguration via extra_dhcp_opts (CVE-2021-40085)

Since this report concerns a possible security risk, an incomplete
security advisory task has been added while the core security
reviewers for the affected project or projects confirm the bug and
discuss the scope of any vulnerability along with potential
solutions.

Thx for reporting it. It seems like it should be private for now. I will work on it.

Fix like that can be backported to stable branches. In the future we may propose fix on the API side (in neutron-lib) to remove such newline chars before saving them in db.

@Slawek, I would split the opt_value string each time, instead of removing the "\n". That will take only the first part of the string, removing the tail.

>>> opt_value = "xxx\n128,aa:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb\n120,aa.cc\n128,:"
>>> opt_value.split()[0]
"xxx"

@Rodolfo, thx for review of the patch. I prepared new version of the patch. Please check it.

Naive question, but can the same bug be exploited with other characters besides linefeed? Maybe carriage returns (\r), form feeds (\f), or vertical tabs (\v) are treated similarly by dnsmasq?

dnsmasq reads the config file using `fgets` functions, so it depends on its
implementation. As far as i know, the default glibc supports only \n
character as a line delimiter

On Tue, Aug 17, 2021 at 6:41 PM Jeremy Stanley <email address hidden>
wrote:

> Naive question, but can the same bug be exploited with other characters
> besides linefeed? Maybe carriage returns (\r), form feeds (\f), or
> vertical tabs (\v) are treated similarly by dnsmasq?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1939733
>
> Title:
> Remote Code Execution via extra_dhcp_opts
>
> Status in neutron:
> New
> Status in OpenStack Security Advisory:
> Incomplete
>
> Bug description:
> This issue is being treated as a potential security risk under
> embargo. Please do not make any public mention of embargoed
> (private) security vulnerabilities before their coordinated
> publication by the OpenStack Vulnerability Management Team in the
> form of an official OpenStack Security Advisory. This includes
> discussion of the bug or associated fixes in public forums such as
> mailing lists, code review systems and bug trackers. Please also
> avoid private disclosure to other individuals not already approved
> for access to this information, and provide this same reminder to
> those who are made aware of the issue prior to publication. All
> discussion should remain confined to this private bug report, and
> any proposed fixes should be added to the bug as attachments. This
> embargo shall not extend past 2021-11-10 and will be made
> public by or on that date even if no fix is identified.
>
> Application doesnt check the input values for extra_dhcp_opts port
> parameter allowing user to use a newline character. The values from
> extra_dhcp_opts are used in rendering of opts file which is passed to
> dnsmasq as a dhcp-optsfile. Considering this, an attacker can inject
> any options to that file.
>
> The main direct impact in my opinion is that attacker can push
> arbitrary dhcp options to another instances connected to the same
> network. And due to we are able to modify our own port connected to
> external network, it is possible to push dhcp options to the instances
> of another tennants using the same external network.
>
> If we go further, there is an known buffer overflow vulnerability in
> dnsmasq
> (
> https://thekelleys.org.uk/gitweb/?p=dnsmasq.git;a=commitdiff;h=7d04e17444793a840f98a0283968b96502b112dc
> )
> which was not considered as a security issue due to attacker cannot
> control dhcp opts in most cases and therefore this vulnerability is
> still exists in most distributives (e.g Ubuntu 20.04.1). In our case
> dhcp opts is exactly what attacker can modify, so we can trigger
> buffer overflow there. I even managed to write an exploit which lead
> to a remote code execution using this buffer overflow vulnerability.
>
> Here the payload to crash dnsmasq as a proof of concept:
> ```
> PUT /v2.0/ports/9db67e0f-537c-494a-a655-c8a0c518d57e HTTP/1.1
> Host: openstack
> X-Auth-Token: TOKEN
> Content-Type: ...

Thanks. I suppose my remaining concern is more of a theoretical one in that case... dnsmasq does not expect untrusted users to supply configuration, and therefore is not overly concerned with risks related to that workflow. As a result, any bugs in dnsmasq's configuration handling become Neutron security risks and therefore OpenStack's responsibility to deal with. In the long term, this does not seem like a sustainable model under which to operate.

In the near term, I agree the proposed solution should suffice, and since it seems to be backportable to all supported stable branches I'll get started on the impact description and coordinated disclosure timeline for an embargoed security advisory.

Here's an initial draft for an impact description. Please review and comment. If this summary reasonably captures the vulnerability, I'll use it as the basis for our CVE request to MITRE, notification to downstream stakeholders, and eventual security advisory. Also, Pavel, please let me know if you have any organization you want credited with the discovery alongside your name.

Title: Exposure of dnsmasq buffer overflow via extra_dhcp_opts
Reporter: Pavel Toporkov
Products: Neutron
Affects: <16.4.1, >=17.0.0 <17.2.1, >=18.0.0 <18.1.1

Description:
Pavel Toporkov reported a vulnerability in Neutron. By supplying a specially crafted extra_dhcp_opts value, an authenticated user may trigger a configuration parsing buffer overflow in some older versions of dnsmasq, resulting in denial of service or remote code execution on the hosts where dnsmasq is running. Only deployments with dnsmasq prior to 2.81 or without commit 7d04e17 applied are affected.

On revisiting the initial bug description, I see that I've failed to capture the aspect where the user can also append lines for other configuration outside the options list itself. I'll try to come up with a better summary incorporating that.

Here's a revised impact description:

Title: Exposure of dnsmasq buffer overflow via extra_dhcp_opts
Reporter: Pavel Toporkov
Products: Neutron
Affects: <16.4.1, >=17.0.0 <17.2.1, >=18.0.0 <18.1.1

Description:
Pavel Toporkov reported a vulnerability in Neutron. By supplying a specially crafted extra_dhcp_opts value, an authenticated user may add arbitrary configuration to the dnsmasq process in order to crash the service, change parameters for other tenants sharing the same interface, or otherwise alter that daemon's behavior. This vulnerability may also be used to trigger a configuration parsing buffer overflow in versions of dnsmasq prior 2.81, which could lead to remote code execution. All Neutron deployments are affected.

I also meant to redo the title. How's this?

Title: Arbitrary dnsmasq reconfiguration via extra_dhcp_opts

I also seem to have introduced a typo in the penultimate sentence, that should say "versions of dnsmasq prior to 2.81" (I somehow lost the "to" in the most recent revision).

I reviewed the description and it seems to be OK.

Thanks Pavel, also please let me know if you have any organization you want credited with the discovery alongside your name.

Slawek: Rodolfo: If the patch in comment #5 is deemed a sufficient fix for master (Xena), then once someone attaches viable backports for stable/wallaby, stable/victoria, and stable/ussuri branches, I'll schedule a disclosure date and prepare notification under embargo for our downstream stakeholders (distribution package maintainers, public cloud security contacts, and so on).

No, please credit only my name there

Okay, this is what I'll use to request a CVE assignment from MITRE. Once viable backports for stable/wallaby, stable/victoria, and stable/ussuri branches are attached, I'll schedule a disclosure date and prepare notification under embargo for our downstream stakeholders...

Title: Arbitrary dnsmasq reconfiguration via extra_dhcp_opts
Reporter: Pavel Toporkov
Products: Neutron
Affects: <16.4.1, >=17.0.0 <17.2.1, >=18.0.0 <18.1.1

Description:
Pavel Toporkov reported a vulnerability in Neutron. By supplying a specially crafted extra_dhcp_opts value, an authenticated user may add arbitrary configuration to the dnsmasq process in order to crash the service, change parameters for other tenants sharing the same interface, or otherwise alter that daemon's behavior. This vulnerability may also be used to trigger a configuration parsing buffer overflow in versions of dnsmasq prior to 2.81, which could lead to remote code execution. All Neutron deployments are affected.

The proposed fix no longer allows the reported security issue, but can we block it at the API level too?
If the neutron API does not allow multi-line string in the API level, we can block it for further API operations.

Note that we need to consider such extra_dhcp_options is already injected, so the proposed fix is required and it looks good.

Perhaps blocking it at the API level is optional so it can be done as a follow-up after the proposed fix is applied.
What in my mind is to implement a validator like below in the API level.

 def _validate_oneline_string(data, max_len=None):
     msg = validators.validate_string(data, max_len)
     if msg:
         return msg
     lines = data.splitlines()
     if lines and lines[0] != data:
         msg = _("Multi-line string is not allowed: '%s'") % data
         LOG.debug(msg)
         return msg

Stable/wallaby backport

Stable/victoria backport

Stable/ussuri backport

@Akihiro - that was my idea also but I wanted to do it as next step later. First I wanted to propose solution which can be easily backported.
When that will be solved, I can propose API level validation, some db migration script for existing entries, etc. as a follow-up. Is that ok for You?

@Slawek, it totally works for me. Thanks.

Assuming I get a response from MITRE in the next 24 hours, I'd like to schedule the coordinated disclosure (when we'll switch this bug to public security and push the attached patches to Gerrit) for 15:00 UTC Tuesday 2021-08-31, a week from tomorrow. Is that date and time acceptable to everyone?

@Akihiro - thx.
@Jeremy - that works for me, thx a lot.

Now that MITRE has responded with a CVE assignment, the proposed patches (from comments #5, #19, #20, and #21) have been privately supplied to downstream stakeholders along with a copy of the impact description (from comment #17). We are still within our downstream notification window to be able to proceed with coordinated disclosure at 15:00 UTC on 2021-08-31.

Double-checking, is Slawek or one of the other Neutron core reviewers available to get the fix and backports from comments #5, #19, #20, and #21 pushed to Gerrit in roughly half an hour (around 14:00 UTC)? That will give me time to identify the change numbers for inclusion in the public advisory scheduled to go out at 15:00 UTC.

If there's no one available, I'll push the patches to Gerrit myself, but it would be better to have a Neutron core reviewer taking care of it and also working on expedited approvals so we can get them merged as quickly as possible (I can prioritize them in the gate pipeline once they're approved, to speed things along even more).

I heard back from Slawek in IRC and he's pushing the changes to Gerrit now.

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/806746

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/806748

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/806749

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/neutron/+/806750

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/ossa/+/806754

Reviewed: https://review.opendev.org/c/openstack/ossa/+/806754
Committed: https://opendev.org/openstack/ossa/commit/55e0ee4953446267d7900bda9cdfdc9f44a6b72d
Submitter: "Zuul (22348)"
Branch: master

commit 55e0ee4953446267d7900bda9cdfdc9f44a6b72d
Author: Jeremy Stanley <email address hidden>
Date: Tue Aug 31 13:56:02 2021 +0000

    Add OSSA-2021-005 (CVE-2021-40085)

    Change-Id: I58b8c608547e24ee144cab805d17c55045e4279a
    Closes-Bug: #1939733

https://security.openstack.org/ossa/OSSA-2021-005.html is published, announcements have been sent to the usual mailing lists, and MITRE has been notified to switch the assigned CVE public.

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/neutron/+/806707

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/c/openstack/neutron/+/806708

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/c/openstack/neutron/+/806862

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/c/openstack/neutron/+/806709

The attachment "Fix v1" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

Reviewed: https://review.opendev.org/c/openstack/neutron/+/806746
Committed: https://opendev.org/openstack/neutron/commit/df891f0593d234e01f27d7c0376d9702e178ecfb
Submitter: "Zuul (22348)"
Branch: master

commit df891f0593d234e01f27d7c0376d9702e178ecfb
Author: Slawek Kaplonski <email address hidden>
Date: Tue Aug 31 15:43:11 2021 +0200

    Remove dhcp_extra_opt value after first newline character

    Passing newline to the dnsmasq may cause security issues, especially
    that in case of Neutron that dhcp options' values are controlled by
    cloud users.
    This patch removes everything what is after first newline character
    in the dhcp_extra_opt's values before passing them to dnsmasq.

    Closes-Bug: #1939733
    Change-Id: Ifeaf258f0b5ea86f25620ac4116d618980a7272e

Reviewed: https://review.opendev.org/c/openstack/neutron/+/806748
Committed: https://opendev.org/openstack/neutron/commit/35a32a1cadf2a6bc182b3c7d7ae46e7cea73576a
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit 35a32a1cadf2a6bc182b3c7d7ae46e7cea73576a
Author: Slawek Kaplonski <email address hidden>
Date: Mon Aug 23 13:01:37 2021 +0200

    Remove dhcp_extra_opt value after first newline character

    Passing newline to the dnsmasq may cause security issues, especially
    that in case of Neutron that dhcp options' values are controlled by
    cloud users.
    This patch removes everything what is after first newline character
    in the dhcp_extra_opt's values before passing them to dnsmasq.

    Closes-Bug: #1939733
    Change-Id: Ifeaf258f0b5ea86f25620ac4116d618980a7272e
    (cherry picked from commit df891f0593d234e01f27d7c0376d9702e178ecfb)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/806749
Committed: https://opendev.org/openstack/neutron/commit/a08e21c15b766a1e3b2b64ae15f95565d9ea6688
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit a08e21c15b766a1e3b2b64ae15f95565d9ea6688
Author: Slawek Kaplonski <email address hidden>
Date: Mon Aug 23 13:01:37 2021 +0200

    Remove dhcp_extra_opt value after first newline character

    Passing newline to the dnsmasq may cause security issues, especially
    that in case of Neutron that dhcp options' values are controlled by
    cloud users.
    This patch removes everything what is after first newline character
    in the dhcp_extra_opt's values before passing them to dnsmasq.

    Closes-Bug: #1939733
    Change-Id: Ifeaf258f0b5ea86f25620ac4116d618980a7272e
    (cherry picked from commit df891f0593d234e01f27d7c0376d9702e178ecfb)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/806750
Committed: https://opendev.org/openstack/neutron/commit/6ac5bc583787b7af6273d5344fcbeef69a48e2e3
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit 6ac5bc583787b7af6273d5344fcbeef69a48e2e3
Author: Slawek Kaplonski <email address hidden>
Date: Mon Aug 23 13:01:37 2021 +0200

    Remove dhcp_extra_opt value after first newline character

    Passing newline to the dnsmasq may cause security issues, especially
    that in case of Neutron that dhcp options' values are controlled by
    cloud users.
    This patch removes everything what is after first newline character
    in the dhcp_extra_opt's values before passing them to dnsmasq.

    Closes-Bug: #1939733
    Change-Id: Ifeaf258f0b5ea86f25620ac4116d618980a7272e
    (cherry picked from commit df891f0593d234e01f27d7c0376d9702e178ecfb)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/806862
Committed: https://opendev.org/openstack/neutron/commit/e291dca5d97fc986fc7d85b8ea256a483d815277
Submitter: "Zuul (22348)"
Branch: stable/rocky

commit e291dca5d97fc986fc7d85b8ea256a483d815277
Author: Slawek Kaplonski <email address hidden>
Date: Mon Aug 23 13:01:37 2021 +0200

    Remove dhcp_extra_opt value after first newline character

    Passing newline to the dnsmasq may cause security issues, especially
    that in case of Neutron that dhcp options' values are controlled by
    cloud users.
    This patch removes everything what is after first newline character
    in the dhcp_extra_opt's values before passing them to dnsmasq.

    Conflicts:
        neutron/tests/unit/agent/linux/test_dhcp.py

    Closes-Bug: #1939733
    Change-Id: Ifeaf258f0b5ea86f25620ac4116d618980a7272e
    (cherry picked from commit df891f0593d234e01f27d7c0376d9702e178ecfb)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/806709
Committed: https://opendev.org/openstack/neutron/commit/110fed07cb83deb3abd85073cb351066713b6384
Submitter: "Zuul (22348)"
Branch: stable/queens

commit 110fed07cb83deb3abd85073cb351066713b6384
Author: Slawek Kaplonski <email address hidden>
Date: Mon Aug 23 13:01:37 2021 +0200

    Remove dhcp_extra_opt value after first newline character

    Passing newline to the dnsmasq may cause security issues, especially
    that in case of Neutron that dhcp options' values are controlled by
    cloud users.
    This patch removes everything what is after first newline character
    in the dhcp_extra_opt's values before passing them to dnsmasq.

    Conflicts:
        neutron/tests/unit/agent/linux/test_dhcp.py

    Closes-Bug: #1939733
    Change-Id: Ifeaf258f0b5ea86f25620ac4116d618980a7272e
    (cherry picked from commit df891f0593d234e01f27d7c0376d9702e178ecfb)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/806708
Committed: https://opendev.org/openstack/neutron/commit/e2d6e2d5d2388cd7c135a50129466aaa7ca85cbf
Submitter: "Zuul (22348)"
Branch: stable/stein

commit e2d6e2d5d2388cd7c135a50129466aaa7ca85cbf
Author: Slawek Kaplonski <email address hidden>
Date: Mon Aug 23 13:01:37 2021 +0200

    Remove dhcp_extra_opt value after first newline character

    Passing newline to the dnsmasq may cause security issues, especially
    that in case of Neutron that dhcp options' values are controlled by
    cloud users.
    This patch removes everything what is after first newline character
    in the dhcp_extra_opt's values before passing them to dnsmasq.

    Closes-Bug: #1939733
    Change-Id: Ifeaf258f0b5ea86f25620ac4116d618980a7272e
    (cherry picked from commit df891f0593d234e01f27d7c0376d9702e178ecfb)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/806707
Committed: https://opendev.org/openstack/neutron/commit/757d8c6e322eeda299aa5e055e38b1ed48977f2d
Submitter: "Zuul (22348)"
Branch: stable/train

commit 757d8c6e322eeda299aa5e055e38b1ed48977f2d
Author: Slawek Kaplonski <email address hidden>
Date: Mon Aug 23 13:01:37 2021 +0200

    Remove dhcp_extra_opt value after first newline character

    Passing newline to the dnsmasq may cause security issues, especially
    that in case of Neutron that dhcp options' values are controlled by
    cloud users.
    This patch removes everything what is after first newline character
    in the dhcp_extra_opt's values before passing them to dnsmasq.

    Closes-Bug: #1939733
    Change-Id: Ifeaf258f0b5ea86f25620ac4116d618980a7272e
    (cherry picked from commit df891f0593d234e01f27d7c0376d9702e178ecfb)

This issue was fixed in the openstack/neutron 16.4.1 release.

This issue was fixed in the openstack/neutron 17.2.1 release.

This issue was fixed in the openstack/neutron 18.1.1 release.

This issue was fixed in the openstack/neutron 19.0.0.0rc1 release candidate.

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron-lib/+/811708

@Slawek are you also pushing the new packages to Ubuntu Cloud Archive?

i.e. https://openstack-ci-reports.ubuntu.com/reports/cloud-archive/ussuri_versions.html does not show any 16.4.1 as of yet.

Slawek is one of the upstream Neutron developers. One of the Ubuntu package maintainers will need to take care of Ubuntu's package updates. It's probably mildly confusing that this bug report is marked as affecting the upstream project (where it's been fixed for months) but also the Ubuntu packages which still need the report triaged.

Reviewed: https://review.opendev.org/c/openstack/neutron-lib/+/811708
Committed: https://opendev.org/openstack/neutron-lib/commit/1f4c4031ee8a0dc91149dc6a83c7051db7628f23
Submitter: "Zuul (22348)"
Branch: master

commit 1f4c4031ee8a0dc91149dc6a83c7051db7628f23
Author: Slawek Kaplonski <email address hidden>
Date: Wed Sep 29 12:34:17 2021 +0200

    Add oneline_string validators

    Those new validators are used to validate extra_dhcp_opt's opt_name and
    opt_value fields to not allow multi-line strings to be set there.

    Related-Bug: #1939733
    Change-Id: I4dc3a09847205a660dc966d8eabccb4946f9bbc6

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/824641

Reviewed: https://review.opendev.org/c/openstack/neutron/+/824641
Committed: https://opendev.org/openstack/neutron/commit/ef97019c92e96d0bb17785daaf570b04c68be500
Submitter: "Zuul (22348)"
Branch: master

commit ef97019c92e96d0bb17785daaf570b04c68be500
Author: Slawek Kaplonski <email address hidden>
Date: Thu Jan 13 20:35:53 2022 +0100

    Add upgrade check for extra DHCP options

    Some time ago with patches [1] and [2] we trimmed extra_dhcp_opt name
    and value to first newline character before using them in the dnsmasq
    and later added API validator to not allow such names and/or values with
    newline character in it at all.
    This patch adds upgrade check to warn users if they have old entries
    with newline characters in the database already.

    Related-Bug: #1939733

    [1] https://review.opendev.org/c/openstack/neutron/+/806746
    [2] https://review.opendev.org/c/openstack/neutron-lib/+/811708

    Change-Id: I9a45d918b5a90f8fc50a9ec43b2a67cf582eb369

The Hirsute Hippo has reached End of Life, so this bug will not be fixed for that release.

Ubuntu CVE tracker: https://ubuntu.com/security/cve-2021-40085

This issue was fixed in the openstack/neutron queens-eol release.

This issue was fixed in the openstack/neutron rocky-eol release.

This issue was fixed in the openstack/neutron stein-eol release.

This issue was fixed in the openstack/neutron train-eol release.