reset replica state not forbidden for active replica

Bug #2015328 reported by Maurice Escher
10
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Shared File Systems Service (Manila)
Fix Released
Medium
Unassigned

Bug Description

Hi,

what happened:

I had a share in a replication setup, A was my active replica (replica_state 'active'), B was my non-active replica (replica_state 'error', but the exact state doesn't matter).

I triggered 'manila share-replica-reset-state <ID_of_A>' (probably my good intention was to do something to get out of the error on B). By default this set the replica_state of A to 'out_of_sync'.

I ended up in a situation with no active replica left.
I expected some server side validation preventing me to do such a stupid thing ;)

Can a respective safeguard be implemented?

Thanks,
Maurice

P.S.:
The out_of_sync replica A went into replica_state error later during periodic replica update.
Another user deleted (with force option?) the replica A in error as part of a cleanup.
Ooops -> my share was gone, data lost.

Vida Haririan (vhariria)
Changed in manila:
importance: Undecided → Low
Vida Haririan (vhariria)
Changed in manila:
status: New → Incomplete
tags: added: low-hanging-fruit
Changed in manila:
status: Incomplete → New
importance: Low → Medium
Revision history for this message
Vida Haririan (vhariria) wrote :
Vida Haririan (vhariria)
Changed in manila:
status: New → Triaged
Revision history for this message
Goutham Pacha Ravi (gouthamr) wrote :

Hi,

Commenting back here to retain the "low-hanging-fruit" tag on this bug. In the fix, the code must check if the reset_replica_state action is being run against an "active" replica.

In https://review.opendev.org/c/openstack/manila/+/857980/3/manila/api/openstack/wsgi.py#1267 ; we're introducing a "get" operation in the mix, so we'd have the data to verify the existing replica_state; so we'd need to add the following validation:

  if (status_attr == 'replica_state'
          and resource['replica_state'] == constants.REPLICA_STATE_ACTIVE):
     msg = _("Cannot reset replica_state of active replica")
     raise webob.exc.HTTPBadRequest(explanation=msg)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to manila (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/manila/+/899320

Changed in manila:
status: Triaged → In Progress
Revision history for this message
Gray Lutalo (gray-00) wrote :
Changed in manila:
assignee: nobody → Gray Lutalo (gray-00)
assignee: Gray Lutalo (gray-00) → nobody
Changed in manila:
assignee: nobody → Gray Lutalo (gray-00)
Changed in manila:
assignee: Gray Lutalo (gray-00) → nobody
status: In Progress → New
Changed in manila:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to manila (master)

Reviewed: https://review.opendev.org/c/openstack/manila/+/899320
Committed: https://opendev.org/openstack/manila/commit/b7a1b5b2cf765b1570432b67363905251893ce42
Submitter: "Zuul (22348)"
Branch: master

commit b7a1b5b2cf765b1570432b67363905251893ce42
Author: Gray Lutalo <email address hidden>
Date: Wed Oct 25 22:33:30 2023 +0000

    Forbid resetting state of active replicas

    In a replication setup, users encountered a critical
    issue where they unintentionally reset the replica_state
    of an active replica while attempting to resolve errors
    on a non-active replica. This led to a situation with no
    active replica, causing data loss. Users expected
    server-side validation to prevent such actions.

    This commit implements the necessary validation in the
    codebase to ensure that the reset_replica_state action
    cannot be applied to active replicas, addressing the
    reported issue and improving data integrity in
    replication setups.

    Co-Authored-By: Solly <email address hidden>
    Closes-Bug: #2015328
    Change-Id: I629669476e585a834673b8c8b49ad4b0270b877f

Changed in manila:
status: In Progress → Fix Released
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to manila (stable/2023.2)

Fix proposed to branch: stable/2023.2
Review: https://review.opendev.org/c/openstack/manila/+/912743

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to manila (stable/2023.2)

Reviewed: https://review.opendev.org/c/openstack/manila/+/912743
Committed: https://opendev.org/openstack/manila/commit/d5baad9e2f90bba2831e27ad7c508313e803a84b
Submitter: "Zuul (22348)"
Branch: stable/2023.2

commit d5baad9e2f90bba2831e27ad7c508313e803a84b
Author: Gray Lutalo <email address hidden>
Date: Wed Oct 25 22:33:30 2023 +0000

    Forbid resetting state of active replicas

    In a replication setup, users encountered a critical
    issue where they unintentionally reset the replica_state
    of an active replica while attempting to resolve errors
    on a non-active replica. This led to a situation with no
    active replica, causing data loss. Users expected
    server-side validation to prevent such actions.

    This commit implements the necessary validation in the
    codebase to ensure that the reset_replica_state action
    cannot be applied to active replicas, addressing the
    reported issue and improving data integrity in
    replication setups.

    Co-Authored-By: Solly <email address hidden>
    Closes-Bug: #2015328
    Change-Id: I629669476e585a834673b8c8b49ad4b0270b877f
    (cherry picked from commit b7a1b5b2cf765b1570432b67363905251893ce42)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to manila (stable/2023.1)

Fix proposed to branch: stable/2023.1
Review: https://review.opendev.org/c/openstack/manila/+/912744

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/manila 18.0.0.0rc1

This issue was fixed in the openstack/manila 18.0.0.0rc1 release candidate.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to manila (stable/2023.1)

Reviewed: https://review.opendev.org/c/openstack/manila/+/912744
Committed: https://opendev.org/openstack/manila/commit/fc5c241a62105b83825c60cbc8c0ccdea1313bf8
Submitter: "Zuul (22348)"
Branch: stable/2023.1

commit fc5c241a62105b83825c60cbc8c0ccdea1313bf8
Author: Gray Lutalo <email address hidden>
Date: Wed Oct 25 22:33:30 2023 +0000

    Forbid resetting state of active replicas

    In a replication setup, users encountered a critical
    issue where they unintentionally reset the replica_state
    of an active replica while attempting to resolve errors
    on a non-active replica. This led to a situation with no
    active replica, causing data loss. Users expected
    server-side validation to prevent such actions.

    This commit implements the necessary validation in the
    codebase to ensure that the reset_replica_state action
    cannot be applied to active replicas, addressing the
    reported issue and improving data integrity in
    replication setups.

    Co-Authored-By: Solly <email address hidden>
    Closes-Bug: #2015328
    Change-Id: I629669476e585a834673b8c8b49ad4b0270b877f
    (cherry picked from commit b7a1b5b2cf765b1570432b67363905251893ce42)
    (cherry picked from commit d5baad9e2f90bba2831e27ad7c508313e803a84b)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/manila 16.2.1

This issue was fixed in the openstack/manila 16.2.1 release.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix included in openstack/manila 17.3.0

This issue was fixed in the openstack/manila 17.3.0 release.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.