[OSSA-2021-001] Anti-spoofing bypass for Open vSwitch networks (CVE-2021-20267)

Since this report concerns a possible security risk, an incomplete
security advisory task has been added while the core security
reviewers for the affected project or projects confirm the bug and
discuss the scope of any vulnerability along with potential
solutions.

I think this is an already known issue, judging from abandoned attempts to correct it years ago:

https://review.opendev.org/315828

It got a mention in bug 1502933 which you referenced, as well as other bug reports linked from the commit message there. As such, I think we should be able to safely switch to discussing this report in public. Do Neutron security reviewers agree?

@fungi: yes, I think this is similar bug to 1502933 and we can make it public IMO.
@David: is this happening for You for openvswitch fw driver or iptables_hybrid? Or both?

@slaweq: I face the issue using openvswitch fw driver but I did not test iptables_hybrid so I don't know if the issue exists there.

If you all think it should be public, I am OK with that.

This report is now being tracked in public, so it can be discussed in public meetings and on mailing lists, and changes related to it can be pushed to normal code review (ideally referencing this bug number).

Pending further confirmation, this sounds likely to be a class A report per our taxonomy (and so would be covered in a security advisory similar to OSSA-2016-009/CVE-2015-8914): https://security.openstack.org/vmt-process.html#incident-report-taxonomy

Marking this patch as "high", same as https://bugs.launchpad.net/neutron/+bug/1502933.

I was able to reproduce that bug on master. For now I did it with ovs firewall driver.
I will need to confirm if that is also the case for iptables_hybrid fw driver.

I just checked and I can't reproduce the same issue with iptables_hybrid fw driver.
So it seems that only openvswitch firewall driver is impacted by that (at least on master).

Patch proposed https://review.opendev.org/c/openstack/neutron/+/776599

I also tested OVN backend and it seems that it isn't affected. So fix is needed only for ovs firewall driver.

Patch https://review.opendev.org/c/openstack/neutron/+/776599 merged and it should solves that problem.

Fantastic, Slawek, thanks! Jeremy, have you already asked Mitre for a CVE? Otherwise, RH can issue one.

Summer: I was hoping to first see some core reviews on the proposed backports before assuming it's going to be fixed there, and as confirmation that we know for sure the extent of the problem prior to the current state of the master branch. The backport targeting stable/victoria looks like it may also fail the neutron-grenade-dvr-multinode upgrade job, it was rechecked a few minutes ago to hopefully rule that out.

Once there's at least one +2 on some of the backports I'll be more confident in drafting an accurate impact description with which to request a CVE assignment. Since this is already public, we're not under any tight embargo timeline with it, and probably not much point in having a CVE to track it until we have a fix we know we can recommend. That said, if you want to act as a CNA and issue a placeholder CVE now, just make sure to let us know the number and I'll refrain from requesting one from MITRE later.

Thanks Jeremy, CVE-2021-20267 has been assigned to this issue.

The stable/victoria backport has merged and the other backports all have positive votes from from core reviewers. Here's an initial draft for an impact description we'll be able to use in an advisory; please review and call out anything which needs correcting (and David, let me know if you have an employer or other affiliated organization you wish mentioned alongside your name)...

Title: Anti-spoofing bypass for Open vSwitch networks
Reporter: David Sinquin
Products: Neutron
Affects: <15.3.3, >=16.0.0 <16.3.1, >=17.0.0 <17.1.1

Description: David Sinquin reported a vulnerability in Neutron's default Open vSwitch firewall rules. By sending carefully crafted packets, anyone in control of a server instance connected to the virtual switch can impersonate the IPv6 addresses of other systems on the network, resulting in denial of service or in some cases possibly interception of traffic intended for other destinations. Only deployments using the Open vSwitch driver are affected.

I want to first thank everyone involved.

Regarding credits, please mention “Gandi.net” as my employee (for which I am reporting this and auditing spoofing possibilities in the first place).

Regarding Slawek Kaplonski's patch [1] that was merged in master, I am sorry I did not took time to review this further before, but I **think** there is still a potential security concern (DoS / redirection) for Neighbor Advertisements. I have not tested this using neutron so I cannot guaranty this be possible in a real deployment.

The scenario I think of would require 2 VMs : one “attacker”, one “victim” whose neighbor cache is to be poisoned (for a spoofed IP). The following is an attempt at providing a way to check for the vulnerability:

First, on attacker, add the IP to spoof. It must be unattributed to avoid races with legit owner and simplify testing.

Second, on the attacker, install scapy and start it (use `scapy` to have useful imports) and adapt then run the below lines, leaving the prompt opened for fourth step:

    spoofed_ipv6 = '' # FIXME non-ll (NOT fe80::), L2-reachable
    victim_ipv6 = '' # FIXME must be L2-reachable
    announced_mac = '' # FIXME attacker MAC

    ipv6 = IPv6(dst=victim_ipv6)
    icmpv6na = ICMPv6ND_NA(tgt=spoofed_ipv6)
    icmpv6_target_ll_addr = ICMPv6NDOptDstLLAddr(lladdr=announced_mac)
    spoofed_packet = Ether() / ip6 / icmp6na / icmpv6_target_ll_addr
    spoofed_packet.show() # not mandatory :-)

Third, on the victim, run a tcpdump (e.g. `tcpdump -nvve 'icmp6 && ip6[40] == 134'`) and while it runs, run a ping6 to the unattributed IPv6 to spoof (clear cache before if needed) and immediately continue to the next step.

Fourth, using the scapy prompt on the attacker, run (a few times by security, one should be enough but packet loss might occur):
    sendp(spoofed_packet)

Finally, check the victim tcpdump (it should be empty) and cache with e.g. `ip -6 n` (idem).

(I realize after having finished proofreading this that the same could likely be achieved through some kind of targeted IPv6 NAT for outgoing ICMPv6 packets on the attacker to avoid scapy completely.)

In the patch I had attached to this bug, I was using nd_target to rule this possible issue out but it can only be applied to Neighbor Advertisement packets (since other types do not have such value), hence the other changes I had done around NA management.

[1] https://review.opendev.org/c/openstack/neutron/+/776599

I am good with Jeremy's impact description in comment #16.

Jeremy's impact description in comment #16 also looks good to me FWIW :-)

Given David's suggestion (in comment #17) that the merged fix may be incomplete, I'd rather hold off finalizing any advisory until someone can either confirm or refute it.

Hi David,

Here is what I did exactly to try what You described in the c17: http://paste.openstack.org/show/803502/
For me it seems that I wasn't able to reproduce what You described but maybe I did something wrong there. If so, please provide me reproducer so I can check that.

Hi Slaweq,

from what I understand, one of the issue you had is the fact your VM had no route to 3000::/64 and therefore no ICMPv6 packets were sent at all.

I also realize the tcpdump command I provided was wrong, and can be replaced with e.g.:
    tcpdump -nve 'icmp6 && (ip6[40] == 135 || ip6[40] == 136)'
(135 is Neighbor Solicitation, 136 is Neighbor Advertisement, 134 was unrelated Router Advertisement…)

I have replied to your paste with what I get (with a reworked procedure I tested in a OpenStack environment), you can find it at: http://paste.openstack.org/show/803529/

The main changes I did are:
 - switching to using fe80::/64 IPs (due to my local setup, I preferred using them)
 - sending spoofed packets in a loop
 - changing NA options to match legitimate answer my VM was sending (likely useless, but it works as-is :-D)
 - some clean-up of unnecessary steps.

FTR, I did my tests using ussuri neutron version with 4b5bcff64c4725b37f094dfd2613ec58f723d304 cherry-picked on top of 4edf21a37561971991ee450b07fcfbca95922bc2 (plus internal patches that do not change openvswitch driver code).

This issue was fixed in the openstack/neutron 15.3.3 release.

This issue was fixed in the openstack/neutron 16.3.1 release.

This issue was fixed in the openstack/neutron 17.1.1 release.

I've asked in #openstack-neutron whether we can switch this back to In Progress state or similar, since it sounds like the fixes were incomplete.

Thx David. I was able to reproduce that issue with Your script.

Thx for confirming that :-)

After some more research I'm not sure if we have any way to fix that issue really as it seems for me that OVS don't allows us to match on tgt field of the ICMPv6 packets and that's where this spoofed IP address is send really in Your example.

If You have any idea on how to fix that, feel free to propose patch for it.

I think the patch I had attached to this bug report had a working fix for that using nd_target param in _add_flow method.

The code is mostly:

self._add_flow(
    table=ovs_consts.BASE_EGRESS_TABLE,
    priority=95,
    in_port=port.ofport,
    reg_port=port.ofport,
    dl_type=lib_const.ETHERTYPE_IPV6,
    nw_proto=lib_const.PROTO_NUM_IPV6_ICMP,
    icmp_type=n_const.ICMPV6_TYPE_NA,
    nd_target=allowed_ip_addr,
    actions='resubmit(,%d)' % (
        ovs_consts.ACCEPTED_EGRESS_TRAFFIC_NORMAL_TABLE)
)

and results in the following Open vSwitch rule:

table=71, hard_age=65534, priority=95,icmp6,reg5=0x4,in_port=4,icmp_type=136,nd_target=fe80::f816:3eff:fed3:1eb4,actions=resubmit(,94)

The above is as-is allowing the VM to use an IPv6 it does not own to announce its own IPs but can likely be restricted further and does not prevent it from announcing a MAC address it does not own but this requires privileges inside the VM and these privileges are anyway enough to make many other DoS for the VM itself.

This issue was fixed in the openstack/neutron 18.0.0.0rc1 release candidate.

Hi David,

I just proposed Your patch in gerrit https://review.opendev.org/c/openstack/neutron/+/783743 - please take a look and check if I didn't made any mistake there.
I also added some UT for it.

And I tested Your patch locally - it worked for me and I wasn't able to spoof address anymore with that patch using Your script described in comment #22

Thanks Slaweq, your commit looks good to me and as you tested your version (and I had tested the patch you adapted), I think we are fine.

Reviewed: https://review.opendev.org/c/openstack/neutron/+/783743
Committed: https://opendev.org/openstack/neutron/commit/ca7822e2108c151bda992ef8a6d454ec2c6d890e
Submitter: "Zuul (22348)"
Branch: master

commit ca7822e2108c151bda992ef8a6d454ec2c6d890e
Author: Slawek Kaplonski <email address hidden>
Date: Mon Mar 29 22:21:15 2021 +0200

    [ovs fw] Restrict IPv6 NA and DHCP(v6) IP and MAC source addresses

    Neighbor Advertisments are used to inform other machines of the MAC
    address to use to reach an IPv6. This commits prevents VMs from
    pretending they are assigned IPv6 they should not use.

    It also prevents sending UDP packets with spoofed IP or MAC even using
    DHCP(v6) request ports.

    Co-authored-by: David Sinquin <email address hidden>

    Closes-bug: #1902917

    Change-Id: Iffb6643359562487414460f5a7e19a7fae9f935c

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/791464

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/neutron/+/791465

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/neutron/+/791467

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/neutron/+/791468

Fix proposed to branch: stable/stein
Review: https://review.opendev.org/c/openstack/neutron/+/791500

Fix proposed to branch: stable/rocky
Review: https://review.opendev.org/c/openstack/neutron/+/791469

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/c/openstack/neutron/+/791470

Reviewed: https://review.opendev.org/c/openstack/neutron/+/791468
Committed: https://opendev.org/openstack/neutron/commit/2231a9d40f989a1e8a20f9862ff03916091b3215
Submitter: "Zuul (22348)"
Branch: stable/train

commit 2231a9d40f989a1e8a20f9862ff03916091b3215
Author: Slawek Kaplonski <email address hidden>
Date: Mon Mar 29 22:21:15 2021 +0200

    [ovs fw] Restrict IPv6 NA and DHCP(v6) IP and MAC source addresses

    Neighbor Advertisments are used to inform other machines of the MAC
    address to use to reach an IPv6. This commits prevents VMs from
    pretending they are assigned IPv6 they should not use.

    It also prevents sending UDP packets with spoofed IP or MAC even using
    DHCP(v6) request ports.

    Co-authored-by: David Sinquin <email address hidden>

    Closes-bug: #1902917

    Conflicts:
        neutron/agent/linux/openvswitch_firewall/firewall.py

    Change-Id: Iffb6643359562487414460f5a7e19a7fae9f935c
    (cherry picked from commit ca7822e2108c151bda992ef8a6d454ec2c6d890e)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/791467
Committed: https://opendev.org/openstack/neutron/commit/ac474307d3a800c30ecbabf0f47de931ad310339
Submitter: "Zuul (22348)"
Branch: stable/ussuri

commit ac474307d3a800c30ecbabf0f47de931ad310339
Author: Slawek Kaplonski <email address hidden>
Date: Mon Mar 29 22:21:15 2021 +0200

    [ovs fw] Restrict IPv6 NA and DHCP(v6) IP and MAC source addresses

    Neighbor Advertisments are used to inform other machines of the MAC
    address to use to reach an IPv6. This commits prevents VMs from
    pretending they are assigned IPv6 they should not use.

    It also prevents sending UDP packets with spoofed IP or MAC even using
    DHCP(v6) request ports.

    Co-authored-by: David Sinquin <email address hidden>

    Closes-bug: #1902917

    Change-Id: Iffb6643359562487414460f5a7e19a7fae9f935c
    (cherry picked from commit ca7822e2108c151bda992ef8a6d454ec2c6d890e)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/791469
Committed: https://opendev.org/openstack/neutron/commit/9f4a9f8f86720567a8ed5e7c5c4430c115ad3c11
Submitter: "Zuul (22348)"
Branch: stable/rocky

commit 9f4a9f8f86720567a8ed5e7c5c4430c115ad3c11
Author: Slawek Kaplonski <email address hidden>
Date: Mon Mar 29 22:21:15 2021 +0200

    [ovs fw] Restrict IPv6 NA and DHCP(v6) IP and MAC source addresses

    Neighbor Advertisments are used to inform other machines of the MAC
    address to use to reach an IPv6. This commits prevents VMs from
    pretending they are assigned IPv6 they should not use.

    It also prevents sending UDP packets with spoofed IP or MAC even using
    DHCP(v6) request ports.

    Co-authored-by: David Sinquin <email address hidden>

    Closes-bug: #1902917

    Conflicts:
        neutron/agent/linux/openvswitch_firewall/firewall.py

    Change-Id: Iffb6643359562487414460f5a7e19a7fae9f935c
    (cherry picked from commit ca7822e2108c151bda992ef8a6d454ec2c6d890e)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/791470
Committed: https://opendev.org/openstack/neutron/commit/ccea0021bc49eefde9af6568942159b52588daae
Submitter: "Zuul (22348)"
Branch: stable/queens

commit ccea0021bc49eefde9af6568942159b52588daae
Author: Slawek Kaplonski <email address hidden>
Date: Mon Mar 29 22:21:15 2021 +0200

    [ovs fw] Restrict IPv6 NA and DHCP(v6) IP and MAC source addresses

    Neighbor Advertisments are used to inform other machines of the MAC
    address to use to reach an IPv6. This commits prevents VMs from
    pretending they are assigned IPv6 they should not use.

    It also prevents sending UDP packets with spoofed IP or MAC even using
    DHCP(v6) request ports.

    Co-authored-by: David Sinquin <email address hidden>

    Closes-bug: #1902917

    Conflicts:
        neutron/agent/linux/openvswitch_firewall/firewall.py

    Change-Id: Iffb6643359562487414460f5a7e19a7fae9f935c
    (cherry picked from commit ca7822e2108c151bda992ef8a6d454ec2c6d890e)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/791500
Committed: https://opendev.org/openstack/neutron/commit/e1fe735e843b5d1bee4fa18e6011121027422203
Submitter: "Zuul (22348)"
Branch: stable/stein

commit e1fe735e843b5d1bee4fa18e6011121027422203
Author: Slawek Kaplonski <email address hidden>
Date: Mon Mar 29 22:21:15 2021 +0200

    [ovs fw] Restrict IPv6 NA and DHCP(v6) IP and MAC source addresses

    Neighbor Advertisments are used to inform other machines of the MAC
    address to use to reach an IPv6. This commits prevents VMs from
    pretending they are assigned IPv6 they should not use.

    It also prevents sending UDP packets with spoofed IP or MAC even using
    DHCP(v6) request ports.

    Co-authored-by: David Sinquin <email address hidden>

    Closes-bug: #1902917

    Conflicts:
        neutron/agent/linux/openvswitch_firewall/firewall.py

    Change-Id: Iffb6643359562487414460f5a7e19a7fae9f935c
    (cherry picked from commit ca7822e2108c151bda992ef8a6d454ec2c6d890e)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/791465
Committed: https://opendev.org/openstack/neutron/commit/eb2851561a89338c5843521128cbef3c1c7de10c
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit eb2851561a89338c5843521128cbef3c1c7de10c
Author: Slawek Kaplonski <email address hidden>
Date: Mon Mar 29 22:21:15 2021 +0200

    [ovs fw] Restrict IPv6 NA and DHCP(v6) IP and MAC source addresses

    Neighbor Advertisments are used to inform other machines of the MAC
    address to use to reach an IPv6. This commits prevents VMs from
    pretending they are assigned IPv6 they should not use.

    It also prevents sending UDP packets with spoofed IP or MAC even using
    DHCP(v6) request ports.

    Co-authored-by: David Sinquin <email address hidden>

    Closes-bug: #1902917

    Change-Id: Iffb6643359562487414460f5a7e19a7fae9f935c
    (cherry picked from commit ca7822e2108c151bda992ef8a6d454ec2c6d890e)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/791464
Committed: https://opendev.org/openstack/neutron/commit/cb746e2ca454885ba5cf653493b9bb0e91480759
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit cb746e2ca454885ba5cf653493b9bb0e91480759
Author: Slawek Kaplonski <email address hidden>
Date: Mon Mar 29 22:21:15 2021 +0200

    [ovs fw] Restrict IPv6 NA and DHCP(v6) IP and MAC source addresses

    Neighbor Advertisments are used to inform other machines of the MAC
    address to use to reach an IPv6. This commits prevents VMs from
    pretending they are assigned IPv6 they should not use.

    It also prevents sending UDP packets with spoofed IP or MAC even using
    DHCP(v6) request ports.

    Co-authored-by: David Sinquin <email address hidden>

    Closes-bug: #1902917

    Change-Id: Iffb6643359562487414460f5a7e19a7fae9f935c
    (cherry picked from commit ca7822e2108c151bda992ef8a6d454ec2c6d890e)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/ossa/+/800107

I'd like to go ahead and publish the advisory for this early next week, if anyone has time to review the proposed content here: https://review.opendev.org/800107

This issue was fixed in the openstack/neutron 16.4.0 release.

This issue was fixed in the openstack/neutron 17.2.0 release.

This issue was fixed in the openstack/neutron 18.1.0 release.

Thank you fungi, the OSSA looks correct and relevant to me.

And also thanks to everyone involved for the rest of the work :-)

Reviewed: https://review.opendev.org/c/openstack/ossa/+/800107
Committed: https://opendev.org/openstack/ossa/commit/239ec3826a74c4f3ffb1239cc574f95c6097c631
Submitter: "Zuul (22348)"
Branch: master

commit 239ec3826a74c4f3ffb1239cc574f95c6097c631
Author: Jeremy Stanley <email address hidden>
Date: Thu Jul 8 20:49:35 2021 +0000

    Add OSSA-2021-001 (CVE-2021-20267)

    Change-Id: I6bcc8392831efbdc7759b0ed5340023bb0440c85
    Closes-Bug: #1902917

The advisory is published at https://security.openstack.org/ now and copies have been sent to the usual mailing lists (oss-security, openstack-announce, openstack-discuss).

This issue was fixed in the openstack/neutron 19.0.0.0rc1 release candidate.

This issue was fixed in the openstack/neutron queens-eol release.

This issue was fixed in the openstack/neutron rocky-eol release.

This issue was fixed in the openstack/neutron stein-eol release.

This issue was fixed in the openstack/neutron train-eol release.