Cinder

Failed to delete cinder backup file

Series rocky
Bug #1801316

Bug #1801316 reported by Brin Zhang on 2018-11-02

This bug affects 2 people

	Status	Importance	Assigned to
Cinder	Invalid	Undecided	Brin Zhang
Ocata	New	Undecided	Unassigned
Pike	New	Undecided	Unassigned
Queens	Fix Committed	Undecided	Sofia Enriquez
Rocky	Fix Committed	Undecided	Brin Zhang

Bug Description

The backup file failed to be deleted because the backup driver configured on the same storage backend is different.
You can reproduce the problem by following the steps below.

1.Config the "backup_driver=cinder.backup.drivers.ceph.CephBackupDriver" in cinder.conf, and then create a backup from a available/in-use volume, backup name is "vol_backup01";
2.Change the "backup_driver=cinder.backup.drivers.ceph", and then to delete the backup with "vol_backup01", there will be throw an exception (under below).

Hint: You can also change the order of setting backup_driver in steps 1 and 2 to test, and the same problem will occur.
This is the same storage backend. It is unreasonable that the delete operation cannot be performed just because the backup_driver is different.

2018-11-02 10:05:28.088 7 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2018-11-02 10:05:28.088 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
2018-11-02 10:05:28.088 7 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2018-11-02 10:05:28.088 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
2018-11-02 10:05:28.088 7 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2018-11-02 10:05:28.088 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
2018-11-02 10:05:28.088 7 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2018-11-02 10:05:28.088 7 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/backup/manager.py", line 756, in delete_backup
2018-11-02 10:05:28.088 7 ERROR oslo_messaging.rpc.server raise exception.InvalidBackup(reason=err)
2018-11-02 10:05:28.088 7 ERROR oslo_messaging.rpc.server InvalidBackup: Invalid backup: Delete backup aborted, the backup service currently configured [cinder.backup.drivers.ceph] is not the backup service that was used to create this backup [cinder.backup.drivers.ceph.CephBackupDriver].

The same problem exists with restore backup file.

See original description

Brin Zhang (zhangbailin) on 2018-11-02

Changed in cinder:
assignee:	nobody → Brin Zhang (zhangbailin)

Brin Zhang (zhangbailin) on 2018-11-02

description:

updated

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-11-02: Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.openstack.org/615117

Changed in cinder:
status:	New → In Progress

Revision history for this message

Sean McGinnis (sean-mcginnis) wrote on 2018-11-02:

This is no longer the case in master (stein) since the ability to configure the backup driver by module name was removed. Stable branches do still have this since although it was deprecated there, it was still allowed. So this would be one of the rare cases where the patch can be proposed directly against stable/rocky and cherry picked from there.

Changed in cinder:
status:	In Progress → Invalid

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-11-05: Fix proposed to cinder (stable/rocky)

Fix proposed to branch: stable/rocky
Review: https://review.openstack.org/615455

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-11-05: Change abandoned on cinder (master)

Change abandoned by Brin Zhang (<email address hidden>) on branch: master
Review: https://review.openstack.org/615117
Reason: As Sean said, this problem exists in rocky and its previous version, abandon this patch, move this to stable/rocky.
Link the rocky commit: https://review.openstack.org/#/c/615455/

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-11-26: Fix merged to cinder (stable/rocky)

Reviewed: https://review.openstack.org/615455
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=b90a13d07dcf230a509cc93a4c864a20494cf5e7
Submitter: Zuul
Branch: stable/rocky

commit b90a13d07dcf230a509cc93a4c864a20494cf5e7
Author: zhangbailin <email address hidden>
Date: Sun Nov 4 19:25:59 2018 -0500

Change the matching method of the backup driver

    The backup driver name is configured uniformly from the Stein version.
    For example, support for
    'cinder.backup.drivers.service.ServiceBackupDriver' does not support
    'cinder.backup.drivers.service'.

However, the above two backup drivers are supported in a stable version
before Rocky (including Rocky).

    For the same storage backend, if the backup driver used to create the
    backup file is 'cinder.backup.drivers.service.ServiceBackupDriver',
    the backup driver configured when restoring the backup is
    'cinder.backup.drivers.service'. The recovery of the backup file will
    fail because the drivers do not match.

Therefore, the processing of this method needs to be compatible with the
backup driver for different configurations of the same backend.

The above 'service' is a proxy, it can be one of {'swift', 'nfs',
'ceph', 'glusterfs', 'google', 'tsm', 'posix'}

Closes-Bug: #1801316
Change-Id: I1d7ee395c89beda0e50cbe26b4abb4fc7749e15e

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2018-12-03: Fix included in openstack/cinder 13.0.2

This issue was fixed in the openstack/cinder 13.0.2 release.

Revision history for this message

Alan Bishop (alan-bishop) wrote on 2019-04-23:

This issue is present in queens, and so the fix needs to be backported to
stable/queens. The problem happens when upgrading from pike to queens
*without* changing the backup_driver to the new (in queens) format that
specifies the class name instead of the module name.

In pike, create a backup with backup_driver=cinder.backup.drivers.service,
then upgrade to queens. ("service" is a placeholder for any specific backend
driver, such as swift, ceph, etc.)

In queens, backup_driver=cinder.backup.drivers.service is deprecated, but
still valid. Support for loading a driver by module name was finally
deprecated in stein (https://review.opendev.org/595372).

However, https://review.opendev.org/515577 (which fixed another bug that is
similar to this one) includes an online DB migration that updates the pike
DB entries so they presume the backup_driver includes the class name. See [1].

[1] https://review.opendev.org/#/c/515577/9/cinder/db/sqlalchemy/api.py

This creates a situation where you have queens code running with
backup_driver=cinder.backup.drivers.service, but backups created during pike
have DB entries with cinder.backup.drivers.service.ServiceBackupDriver

Revision history for this message

Alan Bishop (alan-bishop) wrote on 2019-04-23:

Edit: Support for loading a driver by module name was finally REMOVED in stein.

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-04-26: Fix proposed to cinder (stable/queens)

Fix proposed to branch: stable/queens
Review: https://review.opendev.org/656010

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-05-15: Fix merged to cinder (stable/queens)

#10

Reviewed: https://review.opendev.org/656010
Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=66133741602924c4a6509277bcb33a2f2aa042d1
Submitter: Zuul
Branch: stable/queens

commit 66133741602924c4a6509277bcb33a2f2aa042d1
Author: zhangbailin <email address hidden>
Date: Sun Nov 4 19:25:59 2018 -0500

Change the matching method of the backup driver

However, the above two backup drivers are supported in a stable version
before Rocky (including Rocky).

Therefore, the processing of this method needs to be compatible with the
backup driver for different configurations of the same backend.

The above 'service' is a proxy, it can be one of {'swift', 'nfs',
'ceph', 'glusterfs', 'google', 'tsm', 'posix'}

    Closes-Bug: #1801316
    Part of this backup comes from Change-Id:
    If1e34ea112b0cff328a58ec2fc9c3f5a5005f1c6
    Change-Id: I1d7ee395c89beda0e50cbe26b4abb4fc7749e15e
    (cherry picked from commit b90a13d07dcf230a509cc93a4c864a20494cf5e7)
    Conflicts:
        cinder/backup/manager.py

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2019-07-26: Fix included in openstack/cinder 12.0.8

#11

This issue was fixed in the openstack/cinder 12.0.8 release.

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.