Compute service fails to restart if the vnic_type of a bound port changed from direct to macvtap (CVE-2022-37394)

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/849985

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/nova/+/850003

Sean: Can you elaborate on why you believe this report represents an exploitable security vulnerability? Is it that a malicious user can change the vnic_type of a port under their control and leave a time-bomb for the next time the administrator restarts the compute service, resulting in that compute host being out of service (unable to stop/start running virtual machines) until the problem can be manually rectified?

I caught up with Sean in IRC and he confirmed the situation is basically as I inferred above (exploitable by any normal authenticated user, not just limited to operator level accounts).

Since this report concerns a possible security risk, an incomplete
security advisory task has been added while the core security
reviewers for the affected project or projects confirm the bug and
discuss the scope of any vulnerability along with potential
solutions.

Title: Compute service fails to restart if the vnic_type of a bound port changed from direct to macvtap
Reporter: Balazs Gibizer (Red Hat)
Products: Nova
Affects: >=23.0.0

Description:
Balazs Gibizer with Red Hat reported a vulnerability in Nova's restart behavior when a Neutron port type is changed from "direct" to "macvtap". By creating a neutron port with vnic_type "direct", creating an instance bound to that port, and then changing the vnic_type of the bound port to "macvtap" an authenticated user may cause the compute service to fail to restart resulting in a possible denial of service.
Only Nova deployments configured with SR-IOV are affected.

@David: Your summary look good to me.
@fungi: https://review.opendev.org/c/openstack/nova/+/850003 is proposed as a mitigation of the denial of service. With that patch nova will no longer fail to restart in the reported situation.

Thanks David!

Keep in mind that the title is what will appear in the list of advisories at https://security.openstack.org/ossalist.html and will be combined with the project name, OSSA number and CVE identifier in the subject line of advisories posted to widely-read public mailing lists, so shorter is better as long as it still uniquely captures the situation, sort of like a commit message title. Maybe something along the lines of "Changing vnic_type breaks compute service restart" instead (50 characters).

For the affected versions, we assume that a fix will be backported to all stable series currently in a "managed" state (so Wallaby, Xena and Yoga in this case) and that stable point releases will be tagged to include those. Since the most recent point releases are 23.2.1, 24.1.1 and 25.0.1 we indicate that the next possible release number for each of these is not affected like so: <23.2.2, >=24.0.0 <24.1.2, >=25.0.0 <25.0.2 (if the next point release on stable/yoga ends up being 25.1.0 instead and there is never a 25.0.2 that's fine, since 25.1.0 still falls into an unaffected range strictly speaking).

Thanks for the feedback Jeremy, especially the calculation for the point releases. That was confusing me but your explanation makes perfect sense. Here’s my updated description:

Title: Changing vnic_type breaks compute service restart
Reporter: Balazs Gibizer (Red Hat)
Products: Nova
Affects: <23.2.2, >=24.0.0 <24.1.2, >=25.0.0 <25.0.2

Description:
Balazs Gibizer with Red Hat reported a vulnerability in Nova's restart behavior when a Neutron port type is changed from "direct" to "macvtap". By creating a neutron port with vnic_type "direct", creating an instance bound to that port, and then changing the vnic_type of the bound port to "macvtap" an authenticated user may cause the compute service to fail to restart resulting in a possible denial of service.
Only Nova deployments configured with SR-IOV are affected.

David's proposed impact description from comment #9 looks perfect to me. If there are no immediate objections, VMT members can proceed in requesting a CVE assignment based on that description while the master branch fix is under review, and then work on assembling an advisory once backports have been pushed.

Reviewed: https://review.opendev.org/c/openstack/nova/+/849985
Committed: https://opendev.org/openstack/nova/commit/f8c91eb75fc5504a37fc3b4be1d65d33dbc9b511
Submitter: "Zuul (22348)"
Branch: master

commit f8c91eb75fc5504a37fc3b4be1d65d33dbc9b511
Author: Balazs Gibizer <email address hidden>
Date: Fri Jul 15 12:43:58 2022 +0200

    Reproduce bug 1981813 in func env

    Related-Bug: #1981813
    Change-Id: I9367b7ed475917bdb05eb3f209ce1a4e646534e2

Reviewed: https://review.opendev.org/c/openstack/nova/+/850003
Committed: https://opendev.org/openstack/nova/commit/e43bf900dc8ca66578603bed333c56b215b1876e
Submitter: "Zuul (22348)"
Branch: master

commit e43bf900dc8ca66578603bed333c56b215b1876e
Author: Balazs Gibizer <email address hidden>
Date: Fri Jul 15 13:48:46 2022 +0200

    Gracefully ERROR in _init_instance if vnic_type changed

    If the vnic_type of a bound port changes from "direct" to "macvtap" and
    then the compute service is restarted then during _init_instance nova
    tries to plug the vif of the changed port. However as it now has macvtap
    vnic_type nova tries to look up the netdev of the parent VF. Still that
    VF is consumed by the instance so there is no such netdev on the host
    OS. This error killed the compute service at startup due to unhandled
    exception. This patch adds the exception handler, logs an ERROR and
    continue initializing other instances on the host.

    Also this patch adds a detailed ERROR log when nova detects that the
    vnic_type changed during _heal_instance_info_cache periodic.

    Closes-Bug: #1981813
    Change-Id: I1719f8eda04e8d15a3b01f0612977164c4e55e85

This issue was fixed in the openstack/nova 26.0.0.0rc1 release candidate.

Related fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/nova/+/859312

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/nova/+/859313

Related fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/nova/+/859314

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/nova/+/859315

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/nova/+/859320

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/nova/+/859321

Related fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/nova/+/869583

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/nova/+/869584

Related fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/nova/+/869585

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/nova/+/869586

Related fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/nova/+/869673

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/nova/+/869674

Reviewed: https://review.opendev.org/c/openstack/nova/+/859312
Committed: https://opendev.org/openstack/nova/commit/4954f993680c75fd9d3d507f2dcd00300c9b3d44
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 4954f993680c75fd9d3d507f2dcd00300c9b3d44
Author: Balazs Gibizer <email address hidden>
Date: Fri Jul 15 12:43:58 2022 +0200

    Reproduce bug 1981813 in func env

    There stable/yoga only change in test_pci_sriov_servers.py due to
    unittest.mock switch[1] only happened in zed.

    [1] https://review.opendev.org/q/topic:unittest.mock+status:merged+project:openstack/nova

    Related-Bug: #1981813
    Change-Id: I9367b7ed475917bdb05eb3f209ce1a4e646534e2
    (cherry picked from commit f8c91eb75fc5504a37fc3b4be1d65d33dbc9b511)

Reviewed: https://review.opendev.org/c/openstack/nova/+/859313
Committed: https://opendev.org/openstack/nova/commit/a28c82719545d5c8ee7f3ff1361b3a796e05095a
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit a28c82719545d5c8ee7f3ff1361b3a796e05095a
Author: Balazs Gibizer <email address hidden>
Date: Fri Jul 15 13:48:46 2022 +0200

    Gracefully ERROR in _init_instance if vnic_type changed

    If the vnic_type of a bound port changes from "direct" to "macvtap" and
    then the compute service is restarted then during _init_instance nova
    tries to plug the vif of the changed port. However as it now has macvtap
    vnic_type nova tries to look up the netdev of the parent VF. Still that
    VF is consumed by the instance so there is no such netdev on the host
    OS. This error killed the compute service at startup due to unhandled
    exception. This patch adds the exception handler, logs an ERROR and
    continue initializing other instances on the host.

    Also this patch adds a detailed ERROR log when nova detects that the
    vnic_type changed during _heal_instance_info_cache periodic.

    Closes-Bug: #1981813
    Change-Id: I1719f8eda04e8d15a3b01f0612977164c4e55e85
    (cherry picked from commit e43bf900dc8ca66578603bed333c56b215b1876e)

Reviewed: https://review.opendev.org/c/openstack/nova/+/859314
Committed: https://opendev.org/openstack/nova/commit/0c87681135cfb3ce61d2a0392928c1dbc1fe5fde
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 0c87681135cfb3ce61d2a0392928c1dbc1fe5fde
Author: Balazs Gibizer <email address hidden>
Date: Fri Jul 15 12:43:58 2022 +0200

    Reproduce bug 1981813 in func env

    There stable/yoga only change in test_pci_sriov_servers.py due to
    unittest.mock switch[1] only happened in zed.

    [1] https://review.opendev.org/q/topic:unittest.mock+status:merged+project:openstack/nova

    Related-Bug: #1981813
    Change-Id: I9367b7ed475917bdb05eb3f209ce1a4e646534e2
    (cherry picked from commit f8c91eb75fc5504a37fc3b4be1d65d33dbc9b511)
    (cherry picked from commit 4954f993680c75fd9d3d507f2dcd00300c9b3d44)

Reviewed: https://review.opendev.org/c/openstack/nova/+/859315
Committed: https://opendev.org/openstack/nova/commit/1a98a1a650d065a8ab3e1c474f3b9fd537dc2206
Submitter: "Zuul (22348)"
Branch: stable/xena

commit 1a98a1a650d065a8ab3e1c474f3b9fd537dc2206
Author: Balazs Gibizer <email address hidden>
Date: Fri Jul 15 13:48:46 2022 +0200

    Gracefully ERROR in _init_instance if vnic_type changed

    If the vnic_type of a bound port changes from "direct" to "macvtap" and
    then the compute service is restarted then during _init_instance nova
    tries to plug the vif of the changed port. However as it now has macvtap
    vnic_type nova tries to look up the netdev of the parent VF. Still that
    VF is consumed by the instance so there is no such netdev on the host
    OS. This error killed the compute service at startup due to unhandled
    exception. This patch adds the exception handler, logs an ERROR and
    continue initializing other instances on the host.

    Also this patch adds a detailed ERROR log when nova detects that the
    vnic_type changed during _heal_instance_info_cache periodic.

    Closes-Bug: #1981813
    Change-Id: I1719f8eda04e8d15a3b01f0612977164c4e55e85
    (cherry picked from commit e43bf900dc8ca66578603bed333c56b215b1876e)
    (cherry picked from commit a28c82719545d5c8ee7f3ff1361b3a796e05095a)

This issue was fixed in the openstack/nova 24.2.0 release.

This issue was fixed in the openstack/nova 25.1.0 release.

Even though the patch for stable/wallaby has not merged, we could go ahead and issue an advisory for this now that the branch has transitioned to extended maintenance. What does everyone think?

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/train
Review: https://review.opendev.org/c/openstack/nova/+/869673
Reason: stable/train branch of nova projects' have been tagged as End of Life. All open patches have to be abandoned in order to be able to delete the branch.

Change abandoned by "Elod Illes <email address hidden>" on branch: stable/train
Review: https://review.opendev.org/c/openstack/nova/+/869674
Reason: stable/train branch of nova projects' have been tagged as End of Life. All open patches have to be abandoned in order to be able to delete the branch.