Ironic

Cinder OSSA-2023-003 breaks Ironic Boot From Volume

Bug #2019892 reported by Julia Kreger
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Ironic
Fix Released
Critical
Julia Kreger

Bug Description

The recent security fixes to cinder blocks Ironic from detaching volumes, which is required in teardown workflows in order to ensure any attached fiber channel volumes are not inadvertently destroyed by cleaning.

Ironic independently requests volume detach/attach actions around to facilitate configuration update *as well*, since Cinder is not the only option, and depending on the infrastructure configuration, it is helpful to try and get the latest details as opposed to old configuration data.

In any event, cinder added an explicit requirement that inbound requests had to be with a service token. Initial investigation beyond the required code to choose an explicit service token, suggests we may be picking the wrong credential token in devstack runs as well.

It is worth noting, Nova had to patch this sort of behavior as well for this fix, it is just our code is different enough that it is not a direct mapping/copy-paste.

Cinder bug:
https://launchpad.net/bugs/2004555
http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-2088
Nova change: https://review.opendev.org/c/openstack/nova/+/882870 (wallaby)

Error:

May 16 17:38:00.221711 np0034044757 ironic-conductor[120302]: ERROR ironic.common.cinder [None req-28ebb999-dbc2-454a-b61c-e4cec4f7e530 None None] Failed to request detach for volume 30285a4d-9229-43e4-8b83-9525cc719bf3 from cinder for node 5b303645-5304-4aab-a5d4-cd35f52ce711: Invalid volume: Unable to detach volume. Volume status must be 'in-use' and attach_status must be 'attached' to detach. (HTTP 400) (Request-ID: req-6a58f8ba-46b0-42c8-bc44-657e8b2cb192): cinderclient.exceptions.BadRequest: Invalid volume: Unable to detach volume. Volume status must be 'in-use' and attach_status must be 'attached' to detach. (HTTP 400) (Request-ID: req-6a58f8ba-46b0-42c8-bc44-657e8b2cb192)
May 16 17:38:00.222195 np0034044757 ironic-conductor[120302]: ERROR ironic.drivers.modules.storage.cinder [None req-28ebb999-dbc2-454a-b61c-e4cec4f7e530 None None] Error detaching volume for node 5b303645-5304-4aab-a5d4-cd35f52ce711: Failed to request detach for volume 30285a4d-9229-43e4-8b83-9525cc719bf3 from cinder for node 5b303645-5304-4aab-a5d4-cd35f52ce711: Invalid volume: Unable to detach volume. Volume status must be 'in-use' and attach_status must be 'attached' to detach. (HTTP 400) (Request-ID: req-6a58f8ba-46b0-42c8-bc44-657e8b2cb192).: ironic.common.exception.StorageError: Failed to request detach for volume 30285a4d-9229-43e4-8b83-9525cc719bf3 from cinder for node 5b303645-5304-4aab-a5d4-cd35f52ce711: Invalid volume: Unable to detach volume. Volume status must be 'in-use' and attach_status must be 'attached' to detach. (HTTP 400) (Request-ID: req-6a58f8ba-46b0-42c8-bc44-657e8b2cb192)
May 16 17:38:00.227531 np0034044757 ironic-conductor[120302]: ERROR ironic.conductor.manager [None req-28ebb999-dbc2-454a-b61c-e4cec4f7e530 None None] Error in tear_down of node 5b303645-5304-4aab-a5d4-cd35f52ce711: Failed to request detach for volume 30285a4d-9229-43e4-8b83-9525cc719bf3 from cinder for node 5b303645-5304-4aab-a5d4-cd35f52ce711: Invalid volume: Unable to detach volume. Volume status must be 'in-use' and attach_status must be 'attached' to detach. (HTTP 400) (Request-ID: req-6a58f8ba-46b0-42c8-bc44-657e8b2cb192): ironic.common.exception.StorageError: Failed to request detach for volume 30285a4d-9229-43e4-8b83-9525cc719bf3 from cinder for node 5b303645-5304-4aab-a5d4-cd35f52ce711: Invalid volume: Unable to detach volume. Volume status must be 'in-use' and attach_status must be 'attached' to detach. (HTTP 400) (Request-ID: req-6a58f8ba-46b0-42c8-bc44-657e8b2cb192)
May 16 17:38:00.227531 np0034044757 ironic-conductor[120302]: ERROR ironic.conductor.manager Traceback (most recent call last):
May 16 17:38:00.227531 np0034044757 ironic-conductor[120302]: ERROR ironic.conductor.manager File "/opt/stack/ironic/ironic/common/cinder.py", line 412, in detach_volumes
May 16 17:38:00.227531 np0034044757 ironic-conductor[120302]: ERROR ironic.conductor.manager client.volumes.begin_detaching(volume_id)
May 16 17:38:00.227531 np0034044757 ironic-conductor[120302]: ERROR ironic.conductor.manager File "/usr/local/lib/python3.8/dist-packages/cinderclient/v3/volumes_base.py", line 310, in begin_detaching
May 16 17:38:00.227531 np0034044757 ironic-conductor[120302]: ERROR ironic.conductor.manager return self._action('os-begin_detaching', volume)
May 16 17:38:00.227531 np0034044757 ironic-conductor[120302]: ERROR ironic.conductor.manager File "/usr/local/lib/python3.8/dist-packages/cinderclient/v3/volumes_base.py", line 257, in _action
May 16 17:38:00.227531 np0034044757 ironic-conductor[120302]: ERROR ironic.conductor.manager resp, body = self.api.client.post(url, body=body)
May 16 17:38:00.227531 np0034044757 ironic-conductor[120302]: ERROR ironic.conductor.manager File "/usr/local/lib/python3.8/dist-packages/cinderclient/client.py", line 223, in post
May 16 17:38:00.227531 np0034044757 ironic-conductor[120302]: ERROR ironic.conductor.manager return self._cs_request(url, 'POST', **kwargs)
May 16 17:38:00.227531 np0034044757 ironic-conductor[120302]: ERROR ironic.conductor.manager File "/usr/local/lib/python3.8/dist-packages/cinderclient/client.py", line 211, in _cs_request
May 16 17:38:00.227531 np0034044757 ironic-conductor[120302]: ERROR ironic.conductor.manager return self.request(url, method, **kwargs)
May 16 17:38:00.227531 np0034044757 ironic-conductor[120302]: ERROR ironic.conductor.manager File "/usr/local/lib/python3.8/dist-packages/cinderclient/client.py", line 197, in request
May 16 17:38:00.227531 np0034044757 ironic-conductor[120302]: ERROR ironic.conductor.manager raise exceptions.from_response(resp, body)
May 16 17:38:00.227531 np0034044757 ironic-conductor[120302]: ERROR ironic.conductor.manager cinderclient.exceptions.BadRequest: Invalid volume: Unable to detach volume. Volume status must be 'in-use' and attach_status must be 'attached' to detach. (HTTP 400) (Request-ID: req-6a58f8ba-46b0-42c8-bc44-657e8b2cb192)
May 16 17:38:00.227531 np0034044757 ironic-conductor[120302]: ERROR ironic.conductor.manager
May 16 17:38:00.227531 np0034044757 ironic-conductor[120302]: ERROR ironic.conductor.manager During handling of the above exception, another exception occurred:
May 16 17:38:00.227531 np0034044757 ironic-conductor[120302]: ERROR ironic.conductor.manager
May 16 17:38:00.227531 np0034044757 ironic-conductor[120302]: ERROR ironic.conductor.manager Traceback (most recent call last):
May 16 17:38:00.227531 np0034044757 ironic-conductor[120302]: ERROR ironic.conductor.manager File "/opt/stack/ironic/ironic/conductor/manager.py", line 1083, in _do_node_tear_down
May 16 17:38:00.227531 np0034044757 ironic-conductor[120302]: ERROR ironic.conductor.manager task.driver.deploy.tear_down(task)
May 16 17:38:00.227531 np0034044757 ironic-conductor[120302]: ERROR ironic.conductor.manager File "/usr/local/lib/python3.8/dist-packages/ironic_lib/metrics.py", line 60, in wrapped
May 16 17:38:00.227531 np0034044757 ironic-conductor[120302]: ERROR ironic.conductor.manager result = f(*args, **kwargs)
May 16 17:38:00.227531 np0034044757 ironic-conductor[120302]: ERROR ironic.conductor.manager File "/opt/stack/ironic/ironic/conductor/task_manager.py", line 153, in wrapper
May 16 17:38:00.227531 np0034044757 ironic-conductor[120302]: ERROR ironic.conductor.manager return f(*args, **kwargs)
May 16 17:38:00.227531 np0034044757 ironic-conductor[120302]: ERROR ironic.conductor.manager File "/opt/stack/ironic/ironic/drivers/modules/agent_base.py", line 684, in tear_down
May 16 17:38:00.227531 np0034044757 ironic-conductor[120302]: ERROR ironic.conductor.manager task.driver.storage.detach_volumes(task)
May 16 17:38:00.227531 np0034044757 ironic-conductor[120302]: ERROR ironic.conductor.manager File "/opt/stack/ironic/ironic/drivers/modules/storage/cinder.py", line 351, in detach_volumes
May 16 17:38:00.227531 np0034044757 ironic-conductor[120302]: ERROR ironic.conductor.manager detach_volumes()
May 16 17:38:00.227531 np0034044757 ironic-conductor[120302]: ERROR ironic.conductor.manager File "/usr/local/lib/python3.8/dist-packages/tenacity/__init__.py", line 333, in wrapped_f
May 16 17:38:00.227531 np0034044757 ironic-conductor[120302]: ERROR ironic.conductor.manager return self(f, *args, **kw)
May 16 17:38:00.227531 np0034044757 ironic-conductor[120302]: ERROR ironic.conductor.manager File "/usr/local/lib/python3.8/dist-packages/tenacity/__init__.py", line 423, in __call__
May 16 17:38:00.227531 np0034044757 ironic-conductor[120302]: ERROR ironic.conductor.manager do = self.iter(retry_state=retry_state)
May 16 17:38:00.227531 np0034044757 ironic-conductor[120302]: ERROR ironic.conductor.manager File "/usr/local/lib/python3.8/dist-packages/tenacity/__init__.py", line 372, in iter
May 16 17:38:00.227531 np0034044757 ironic-conductor[120302]: ERROR ironic.conductor.manager raise retry_exc.reraise()
May 16 17:38:00.227531 np0034044757 ironic-conductor[120302]: ERROR ironic.conductor.manager File "/usr/local/lib/python3.8/dist-packages/tenacity/__init__.py", line 189, in reraise
May 16 17:38:00.227531 np0034044757 ironic-conductor[120302]: ERROR ironic.conductor.manager raise self.last_attempt.result()
May 16 17:38:00.227531 np0034044757 ironic-conductor[120302]: ERROR ironic.conductor.manager File "/usr/lib/python3.8/concurrent/futures/_base.py", line 437, in result
May 16 17:38:00.227531 np0034044757 ironic-conductor[120302]: ERROR ironic.conductor.manager return self.__get_result()
May 16 17:38:00.227531 np0034044757 ironic-conductor[120302]: ERROR ironic.conductor.manager File "/usr/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
May 16 17:38:00.227531 np0034044757 ironic-conductor[120302]: ERROR ironic.conductor.manager raise self._exception
May 16 17:38:00.227531 np0034044757 ironic-conductor[120302]: ERROR ironic.conductor.manager File "/usr/local/lib/python3.8/dist-packages/tenacity/__init__.py", line 426, in __call__
May 16 17:38:00.227531 np0034044757 ironic-conductor[120302]: ERROR ironic.conductor.manager result = fn(*args, **kwargs)
May 16 17:38:00.227531 np0034044757 ironic-conductor[120302]: ERROR ironic.conductor.manager File "/opt/stack/ironic/ironic/drivers/modules/storage/cinder.py", line 343, in detach_volumes
May 16 17:38:00.227531 np0034044757 ironic-conductor[120302]: ERROR ironic.conductor.manager LOG.error(msg)
May 16 17:38:00.227531 np0034044757 ironic-conductor[120302]: ERROR ironic.conductor.manager File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__
May 16 17:38:00.227531 np0034044757 ironic-conductor[120302]: ERROR ironic.conductor.manager self.force_reraise()
May 16 17:38:00.227531 np0034044757 ironic-conductor[120302]: ERROR ironic.conductor.manager File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise
May 16 17:38:00.227531 np0034044757 ironic-conductor[120302]: ERROR ironic.conductor.manager raise self.value
May 16 17:38:00.236239 np0034044757 ironic-conductor[120302]: ERROR ironic.conductor.manager File "/opt/stack/ironic/ironic/drivers/modules/storage/cinder.py", line 323, in detach_volumes
May 16 17:38:00.236239 np0034044757 ironic-conductor[120302]: ERROR ironic.conductor.manager cinder.detach_volumes(task, targets, connector,
May 16 17:38:00.236239 np0034044757 ironic-conductor[120302]: ERROR ironic.conductor.manager File "/opt/stack/ironic/ironic/common/cinder.py", line 414, in detach_volumes
May 16 17:38:00.236239 np0034044757 ironic-conductor[120302]: ERROR ironic.conductor.manager _handle_errors(_('Failed to request detach for volume %(vol_id)s '
May 16 17:38:00.236239 np0034044757 ironic-conductor[120302]: ERROR ironic.conductor.manager File "/opt/stack/ironic/ironic/common/cinder.py", line 388, in _handle_errors
May 16 17:38:00.236239 np0034044757 ironic-conductor[120302]: ERROR ironic.conductor.manager raise exception.StorageError(msg)
May 16 17:38:00.236239 np0034044757 ironic-conductor[120302]: ERROR ironic.conductor.manager ironic.common.exception.StorageError: Failed to request detach for volume 30285a4d-9229-43e4-8b83-9525cc719bf3 from cinder for node 5b303645-5304-4aab-a5d4-cd35f52ce711: Invalid volume: Unable to detach volume. Volume status must be 'in-use' and attach_status must be 'attached' to detach. (HTTP 400) (Request-ID: req-6a58f8ba-46b0-42c8-bc44-657e8b2cb192)
May 16 17:38:00.236239 np0034044757 ironic-conductor[120302]: ERROR ironic.conductor.manager
May 16 17:38:00.236239 np0034044757 ironic-conductor[120302]: DEBUG ironic.common.states [None req-28ebb999-dbc2-454a-b61c-e4cec4f7e530 None None] Exiting old state 'deleting' in response to event 'fail' {{(pid=120302) on_exit /opt/stack/ironic/ironic/common/states.py:328}}
May 16 17:38:00.236239 np0034044757 ironic-conductor[120302]: DEBUG ironic.common.states [None req-28ebb999-dbc2-454a-b61c-e4cec4f7e530 None None] Entering new state 'error' in response to event 'fail' {{(pid=120302) on_enter /opt/stack/ironic/ironic/common/states.py:334}}
May 16 17:38:00.253723 np0034044757 ironic-conductor[120302]: ERROR ironic.conductor.task_manager [None req-28ebb999-dbc2-454a-b61c-e4cec4f7e530 None None] Node 5b303645-5304-4aab-a5d4-cd35f52ce711 moved to provision state "error" from state "deleting"; target provision state is "None": ironic.common.exception.StorageError: Failed to request detach for volume 30285a4d-9229-43e4-8b83-9525cc719bf3 from cinder for node 5b303645-5304-4aab-a5d4-cd35f52ce711: Invalid volume: Unable to detach volume. Volume status must be 'in-use' and attach_status must be 'attached' to detach. (HTTP 400) (Request-ID: req-6a58f8ba-46b0-42c8-bc44-657e8b2cb192)

CVE References

Julia Kreger (juliaashleykreger)
Changed in ironic:
status: New → Confirmed
status: Confirmed → In Progress
assignee: nobody → Julia Kreger (juliaashleykreger)
importance: Undecided → Critical
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ironic (master)

Reviewed: https://review.opendev.org/c/openstack/ironic/+/882985
Committed: https://opendev.org/openstack/ironic/commit/9c0b4c90a19fc1db262a942a1b6a1baafc881ccc
Submitter: "Zuul (22348)"
Branch: master

commit 9c0b4c90a19fc1db262a942a1b6a1baafc881ccc
Author: Julia Kreger <email address hidden>
Date: Thu May 11 11:09:28 2023 -0700

    Fix Cinder Integration fallout from CVE-2023-2088

    In the recent change to cinder, to address CVE-2023-2088,
    cinder changed the policy rules and behavior for unbinding,
    or "detaching" a volume. This was because of a vulnerability
    in compute nodes where a volume which was in use by a VM
    could be detached outside of Nova, and nova wouldn't become
    aware the volume was detached, and the volume could be accessible
    to the next VM.

    This vulnerability doesn't apply to bare metal operations as
    volumes are attached to whole baremetal nodes with Ironic.

    We now generate and use a service token when interacting with
    Cinder which allows cinder to recognize "this request is
    coming from a fellow OpenStack service", and by-pass
    checking with Nova if the "instance" is managed by Nova,
    or Not. This allows the volumes to be attached, and detached
    as needed as part of the power operation flow and overall
    set of lifecycle operations.

    Related-Bug: 2004555
    Closes-Bug: 2019892

    Change-Id: Ib258bc9650496da989fc93b759b112d279c8b217

Changed in ironic:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ironic (stable/2023.1)

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/ironic/+/883577

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ironic (stable/zed)

Fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/ironic/+/883578

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ironic (stable/yoga)

Fix proposed to branch: stable/yoga
Review: https://review.opendev.org/c/openstack/ironic/+/883579

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ironic (stable/xena)

Fix proposed to branch: stable/xena
Review: https://review.opendev.org/c/openstack/ironic/+/883580

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ironic (stable/wallaby)

Fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/ironic/+/883581

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ironic (stable/2023.1)

Reviewed: https://review.opendev.org/c/openstack/ironic/+/883577
Committed: https://opendev.org/openstack/ironic/commit/efc1e687de91ec7393d47acaee0ba3925c87b017
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit efc1e687de91ec7393d47acaee0ba3925c87b017
Author: Julia Kreger <email address hidden>
Date: Thu May 11 11:09:28 2023 -0700

    Fix Cinder Integration fallout from CVE-2023-2088

    In the recent change to cinder, to address CVE-2023-2088,
    cinder changed the policy rules and behavior for unbinding,
    or "detaching" a volume. This was because of a vulnerability
    in compute nodes where a volume which was in use by a VM
    could be detached outside of Nova, and nova wouldn't become
    aware the volume was detached, and the volume could be accessible
    to the next VM.

    This vulnerability doesn't apply to bare metal operations as
    volumes are attached to whole baremetal nodes with Ironic.

    We now generate and use a service token when interacting with
    Cinder which allows cinder to recognize "this request is
    coming from a fellow OpenStack service", and by-pass
    checking with Nova if the "instance" is managed by Nova,
    or Not. This allows the volumes to be attached, and detached
    as needed as part of the power operation flow and overall
    set of lifecycle operations.

    Related-Bug: 2004555
    Closes-Bug: 2019892

    Change-Id: Ib258bc9650496da989fc93b759b112d279c8b217
    (cherry picked from commit 9c0b4c90a19fc1db262a942a1b6a1baafc881ccc)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ironic (stable/zed)

Reviewed: https://review.opendev.org/c/openstack/ironic/+/883578
Committed: https://opendev.org/openstack/ironic/commit/07497e1b0c1ff77fcba893897ba5b302a1fd6c6b
Submitter: "Zuul (22348)"
Branch: stable/zed

commit 07497e1b0c1ff77fcba893897ba5b302a1fd6c6b
Author: Julia Kreger <email address hidden>
Date: Thu May 11 11:09:28 2023 -0700

    Fix Cinder Integration fallout from CVE-2023-2088

    In the recent change to cinder, to address CVE-2023-2088,
    cinder changed the policy rules and behavior for unbinding,
    or "detaching" a volume. This was because of a vulnerability
    in compute nodes where a volume which was in use by a VM
    could be detached outside of Nova, and nova wouldn't become
    aware the volume was detached, and the volume could be accessible
    to the next VM.

    This vulnerability doesn't apply to bare metal operations as
    volumes are attached to whole baremetal nodes with Ironic.

    We now generate and use a service token when interacting with
    Cinder which allows cinder to recognize "this request is
    coming from a fellow OpenStack service", and by-pass
    checking with Nova if the "instance" is managed by Nova,
    or Not. This allows the volumes to be attached, and detached
    as needed as part of the power operation flow and overall
    set of lifecycle operations.

    Related-Bug: 2004555
    Closes-Bug: 2019892

    Change-Id: Ib258bc9650496da989fc93b759b112d279c8b217
    (cherry picked from commit 9c0b4c90a19fc1db262a942a1b6a1baafc881ccc)

tags: added: in-stable-zed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ironic (stable/yoga)

Reviewed: https://review.opendev.org/c/openstack/ironic/+/883579
Committed: https://opendev.org/openstack/ironic/commit/94358e471c79e4d7f169e1756b71c25943128ccb
Submitter: "Zuul (22348)"
Branch: stable/yoga

commit 94358e471c79e4d7f169e1756b71c25943128ccb
Author: Julia Kreger <email address hidden>
Date: Thu May 11 11:09:28 2023 -0700

    Fix Cinder Integration fallout from CVE-2023-2088

    In the recent change to cinder, to address CVE-2023-2088,
    cinder changed the policy rules and behavior for unbinding,
    or "detaching" a volume. This was because of a vulnerability
    in compute nodes where a volume which was in use by a VM
    could be detached outside of Nova, and nova wouldn't become
    aware the volume was detached, and the volume could be accessible
    to the next VM.

    This vulnerability doesn't apply to bare metal operations as
    volumes are attached to whole baremetal nodes with Ironic.

    We now generate and use a service token when interacting with
    Cinder which allows cinder to recognize "this request is
    coming from a fellow OpenStack service", and by-pass
    checking with Nova if the "instance" is managed by Nova,
    or Not. This allows the volumes to be attached, and detached
    as needed as part of the power operation flow and overall
    set of lifecycle operations.

    Related-Bug: 2004555
    Closes-Bug: 2019892

    Change-Id: Ib258bc9650496da989fc93b759b112d279c8b217
    (cherry picked from commit 9c0b4c90a19fc1db262a942a1b6a1baafc881ccc)

tags: added: in-stable-yoga
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ironic (stable/victoria)

Fix proposed to branch: stable/victoria
Review: https://review.opendev.org/c/openstack/ironic/+/885016

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ironic (stable/ussuri)

Fix proposed to branch: stable/ussuri
Review: https://review.opendev.org/c/openstack/ironic/+/885063

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ironic (stable/train)

Fix proposed to branch: stable/train
Review: https://review.opendev.org/c/openstack/ironic/+/885065

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ironic (stable/xena)

Reviewed: https://review.opendev.org/c/openstack/ironic/+/883580
Committed: https://opendev.org/openstack/ironic/commit/cb38746f71f5dfa346371bf06985bbbb2208af6e
Submitter: "Zuul (22348)"
Branch: stable/xena

commit cb38746f71f5dfa346371bf06985bbbb2208af6e
Author: Julia Kreger <email address hidden>
Date: Thu May 11 11:09:28 2023 -0700

    Fix Cinder Integration fallout from CVE-2023-2088

    In the recent change to cinder, to address CVE-2023-2088,
    cinder changed the policy rules and behavior for unbinding,
    or "detaching" a volume. This was because of a vulnerability
    in compute nodes where a volume which was in use by a VM
    could be detached outside of Nova, and nova wouldn't become
    aware the volume was detached, and the volume could be accessible
    to the next VM.

    This vulnerability doesn't apply to bare metal operations as
    volumes are attached to whole baremetal nodes with Ironic.

    We now generate and use a service token when interacting with
    Cinder which allows cinder to recognize "this request is
    coming from a fellow OpenStack service", and by-pass
    checking with Nova if the "instance" is managed by Nova,
    or Not. This allows the volumes to be attached, and detached
    as needed as part of the power operation flow and overall
    set of lifecycle operations.

    Note: This change is modified from the original upstream chnage
    becuse that change leverages the ability for a project_id value
    to no longer be required in the cinder URL for interactions with
    cinder, which was a requirement removed in Yoga.

    Related-Bug: 2004555
    Closes-Bug: 2019892

    Change-Id: Ib258bc9650496da989fc93b759b112d279c8b217
    (cherry picked from commit 9c0b4c90a19fc1db262a942a1b6a1baafc881ccc)

tags: added: in-stable-xena
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ironic (stable/wallaby)

Reviewed: https://review.opendev.org/c/openstack/ironic/+/883581
Committed: https://opendev.org/openstack/ironic/commit/fdaf396e88ac59a52d395ed075ddff8163a16473
Submitter: "Zuul (22348)"
Branch: stable/wallaby

commit fdaf396e88ac59a52d395ed075ddff8163a16473
Author: Julia Kreger <email address hidden>
Date: Thu May 11 11:09:28 2023 -0700

    Fix Cinder Integration fallout from CVE-2023-2088

    In the recent change to cinder, to address CVE-2023-2088,
    cinder changed the policy rules and behavior for unbinding,
    or "detaching" a volume. This was because of a vulnerability
    in compute nodes where a volume which was in use by a VM
    could be detached outside of Nova, and nova wouldn't become
    aware the volume was detached, and the volume could be accessible
    to the next VM.

    This vulnerability doesn't apply to bare metal operations as
    volumes are attached to whole baremetal nodes with Ironic.

    We now generate and use a service token when interacting with
    Cinder which allows cinder to recognize "this request is
    coming from a fellow OpenStack service", and by-pass
    checking with Nova if the "instance" is managed by Nova,
    or Not. This allows the volumes to be attached, and detached
    as needed as part of the power operation flow and overall
    set of lifecycle operations.

    Note: This change is modified from the original upstream chnage
    becuse that change leverages the ability for a project_id value
    to no longer be required in the cinder URL for interactions with
    cinder, which was a requirement removed in Yoga.

    Additional note: Disables the rescue testing on one of the wallaby
    branch jobs. Essentially is is a tempest branching, or lack their
    of issue. Master branch ironic-tempest-plugin has a fix
    which doesn't exist on tempest 29.0.0.

    Related-Bug: 2004555
    Closes-Bug: 2019892

    Change-Id: Ib258bc9650496da989fc93b759b112d279c8b217
    (cherry picked from commit 9c0b4c90a19fc1db262a942a1b6a1baafc881ccc)
    (cherry picked from commit cb38746f71f5dfa346371bf06985bbbb2208af6e)

tags: added: in-stable-wallaby
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to ironic (bugfix/21.3)

Fix proposed to branch: bugfix/21.3
Review: https://review.opendev.org/c/openstack/ironic/+/885018

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to ironic (stable/victoria)

Reviewed: https://review.opendev.org/c/openstack/ironic/+/885016
Committed: https://opendev.org/openstack/ironic/commit/9603b8612ee22dffae6e44ef380465e6d1376aed
Submitter: "Zuul (22348)"
Branch: stable/victoria

commit 9603b8612ee22dffae6e44ef380465e6d1376aed
Author: Julia Kreger <email address hidden>
Date: Thu May 11 11:09:28 2023 -0700

    Fix Cinder Integration fallout from CVE-2023-2088

    In the recent change to cinder, to address CVE-2023-2088,
    cinder changed the policy rules and behavior for unbinding,
    or "detaching" a volume. This was because of a vulnerability
    in compute nodes where a volume which was in use by a VM
    could be detached outside of Nova, and nova wouldn't become
    aware the volume was detached, and the volume could be accessible
    to the next VM.

    This vulnerability doesn't apply to bare metal operations as
    volumes are attached to whole baremetal nodes with Ironic.

    We now generate and use a service token when interacting with
    Cinder which allows cinder to recognize "this request is
    coming from a fellow OpenStack service", and by-pass
    checking with Nova if the "instance" is managed by Nova,
    or Not. This allows the volumes to be attached, and detached
    as needed as part of the power operation flow and overall
    set of lifecycle operations.

    Note: This change is modified from the original upstream chnage
    becuse that change leverages the ability for a project_id value
    to no longer be required in the cinder URL for interactions with
    cinder, which was a requirement removed in Yoga.

    Additional note: Disables the rescue testing on one of the wallaby
    branch jobs. Essentially is is a tempest branching, or lack their
    of issue. Master branch ironic-tempest-plugin has a fix
    which doesn't exist on tempest 29.0.0.

    Related-Bug: 2004555
    Closes-Bug: 2019892

    Change-Id: Ib258bc9650496da989fc93b759b112d279c8b217
    (cherry picked from commit 9c0b4c90a19fc1db262a942a1b6a1baafc881ccc)
    (cherry picked from commit cb38746f71f5dfa346371bf06985bbbb2208af6e)

tags: added: in-stable-victoria
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/ironic 22.0.0

This issue was fixed in the openstack/ironic 22.0.0 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/ironic 20.1.2

This issue was fixed in the openstack/ironic 20.1.2 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.