Cinder

Cinder backup with ceph backend is broken in v15.0.0

Bug #1856767 reported by Michel Nicol
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
New
Undecided
Unassigned

Bug Description

When trying to create a backup, cinder backup service report en error:
2019-12-17 22:37:16.226 3592960 DEBUG oslo_messaging.rpc.server [req-210128ed-09f2-4be9-be2b-19de2249d7ff c0534e7dd84d4bc2833af9c3bc6927e3 23adb0271cff4a78aab110c2930f480e - default default] Replied failure for incoming message with id None and method: delete_backup. Time elapsed: 0.237 _process_incoming /usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py:198
2019-12-17 22:37:16.248 3592960 ERROR oslo_messaging.rpc.server [req-041bc1d2-018e-4fe7-9959-b844ffef0d61 c0534e7dd84d4bc2833af9c3bc6927e3 23adb0271cff4a78aab110c2930f480e - default default] Exception during message handling: ObjectActionError: Object action obj_load_attr failed because: attribute parent not lazy-loadable
2019-12-17 22:37:16.248 3592960 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2019-12-17 22:37:16.248 3592960 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 172, in _process_incoming
2019-12-17 22:37:16.248 3592960 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2019-12-17 22:37:16.248 3592960 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 274, in dispatch
2019-12-17 22:37:16.248 3592960 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2019-12-17 22:37:16.248 3592960 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
2019-12-17 22:37:16.248 3592960 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2019-12-17 22:37:16.248 3592960 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/cinder/backup/manager.py", line 753, in delete_backup
2019-12-17 22:37:16.248 3592960 ERROR oslo_messaging.rpc.server self._update_backup_error(backup, six.text_type(err))
2019-12-17 22:37:16.248 3592960 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-12-17 22:37:16.248 3592960 ERROR oslo_messaging.rpc.server self.force_reraise()
2019-12-17 22:37:16.248 3592960 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-12-17 22:37:16.248 3592960 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2019-12-17 22:37:16.248 3592960 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/cinder/backup/manager.py", line 750, in delete_backup
2019-12-17 22:37:16.248 3592960 ERROR oslo_messaging.rpc.server backup_service.delete_backup(backup)
2019-12-17 22:37:16.248 3592960 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/cinder/backup/drivers/ceph.py", line 1291, in delete_backup
2019-12-17 22:37:16.248 3592960 ERROR oslo_messaging.rpc.server self._try_delete_base_image(backup)
2019-12-17 22:37:16.248 3592960 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/cinder/backup/drivers/ceph.py", line 500, in _try_delete_base_image
2019-12-17 22:37:16.248 3592960 ERROR oslo_messaging.rpc.server base_name = self._get_backup_base_name(volume_id, backup=backup)
2019-12-17 22:37:16.248 3592960 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/cinder/backup/drivers/ceph.py", line 337, in _get_backup_base_name
2019-12-17 22:37:16.248 3592960 ERROR oslo_messaging.rpc.server if backup.parent:
2019-12-17 22:37:16.248 3592960 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 67, in getter
2019-12-17 22:37:16.248 3592960 ERROR oslo_messaging.rpc.server self.obj_load_attr(name)
2019-12-17 22:37:16.248 3592960 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/cinder/objects/backup.py", line 155, in obj_load_attr
2019-12-17 22:37:16.248 3592960 ERROR oslo_messaging.rpc.server reason=_('attribute %s not lazy-loadable') % attrname)
2019-12-17 22:37:16.248 3592960 ERROR oslo_messaging.rpc.server ObjectActionError: Object action obj_load_attr failed because: attribute parent not lazy-loadable
2019-12-17 22:37:16.248 3592960 ERROR oslo_messaging.rpc.server
2019-12-17 22:37:16.249 3592960 DEBUG oslo_messaging.rpc.server [req-041bc1d2-018e-4fe7-9959-b844ffef0d61 c0534e7dd84d4bc2833af9c3bc6927e3 23adb0271cff4a78aab110c2930f480e - default default] Replied failure for incoming message with id None and method: delete_backup. Time elapsed: 0.254 _process_incoming /usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py:198

the attribute parent for object backup has been added in v15.0.0 apparently. But it is not populated by any value.

If I add parent to the OPTIONAL_FIELDS, the error changes for
2019-12-17 22:29:45.388 230319 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2019-12-17 22:29:45.388 230319 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 166, in _process_incoming
2019-12-17 22:29:45.388 230319 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2019-12-17 22:29:45.388 230319 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch
2019-12-17 22:29:45.388 230319 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2019-12-17 22:29:45.388 230319 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
2019-12-17 22:29:45.388 230319 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2019-12-17 22:29:45.388 230319 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/cinder/backup/manager.py", line 753, in delete_backup
2019-12-17 22:29:45.388 230319 ERROR oslo_messaging.rpc.server self._update_backup_error(backup, six.text_type(err))
2019-12-17 22:29:45.388 230319 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2019-12-17 22:29:45.388 230319 ERROR oslo_messaging.rpc.server self.force_reraise()
2019-12-17 22:29:45.388 230319 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2019-12-17 22:29:45.388 230319 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2019-12-17 22:29:45.388 230319 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/cinder/backup/manager.py", line 750, in delete_backup
2019-12-17 22:29:45.388 230319 ERROR oslo_messaging.rpc.server backup_service.delete_backup(backup)
2019-12-17 22:29:45.388 230319 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/cinder/backup/drivers/ceph.py", line 1291, in delete_backup
2019-12-17 22:29:45.388 230319 ERROR oslo_messaging.rpc.server self._try_delete_base_image(backup)
2019-12-17 22:29:45.388 230319 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/cinder/backup/drivers/ceph.py", line 500, in _try_delete_base_image
2019-12-17 22:29:45.388 230319 ERROR oslo_messaging.rpc.server base_name = self._get_backup_base_name(volume_id, backup=backup)
2019-12-17 22:29:45.388 230319 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/cinder/backup/drivers/ceph.py", line 337, in _get_backup_base_name
2019-12-17 22:29:45.388 230319 ERROR oslo_messaging.rpc.server if backup.parent:
2019-12-17 22:29:45.388 230319 ERROR oslo_messaging.rpc.server File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 68, in getter
2019-12-17 22:29:45.388 230319 ERROR oslo_messaging.rpc.server return getattr(self, attrname)
2019-12-17 22:29:45.388 230319 ERROR oslo_messaging.rpc.server AttributeError: 'Backup' object has no attribute '_obj_parent'
2019-12-17 22:29:45.388 230319 ERROR oslo_messaging.rpc.server

To be able to perform backups (Full backups since incrementals are borken with my "fix", but at least I'm able to save volumes...) I commented out a few line in

/usr/lib/python2.7/dist-packages/cinder/backup/drivers/ceph.py:

from line 337 to 347:
        # if backup.parent:
            # Old backups don't have the base name in the service_metadata,
            # so we use the default RBD backup base
          # if backup.parent.service_metadata:
          # service_metadata = backup.parent.service_metadata
          # base_name = self._format_base_name(service_metadata)
          # else:
          # base_name = utils.convert_str("volume-%s.backup.base"
          # % volume_id)
          # return base_name

Michel Nicol (michel.nicol)
tags: added: cinder
tags: added: backup ceph
Revision history for this message
Michel Nicol (michel.nicol) wrote :

Since I had to try to get back the capability to make incremental backups, I tried a few things.
In /usr/lib/python2.7/dist-packages/cinder/backup/drivers/ceph.py, I changed the code for the function
_get_backup_base_name to:
    def _get_backup_base_name(self, volume_id, backup=None):
        # Ensure no unicode
        if backup.service_metadata:
            return self._format_base_name(backup.service_metadata)
        else:
            return utils.convert_str("volume-%s.backup.base" % volume_id)

I know this is a REALLY dirty fix, but at least I'm able to backup my volumes, both in Full backup mode and in incremental mode.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.