Ceph RBD Mirror Charm

Charm errs if any rbd pool is erasure encoded

Bug #1898504 reported by Vern Hart on 2020-10-05

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	Ceph RBD Mirror Charm	In Progress	Medium	Billy Olsen
	OpenStack Cinder-Ceph charm	In Progress	Medium	Billy Olsen

Bug Description

As near as I can tell, erasure-coded pools are not supported as the source with rbd-mirror. When the ceph-rbd-mirror charm tries to "mirror pool enable" on an ec pool we get the following error:

librbd::api::Mirror: mode_set: failed to allocate mirroring uuid: (95) Operation not supported

To work around this, I modified eligible pools in lib/charm/openstack/ceph_rbd_mirror.py to exclude pools that have an erasure_code_profile parameter:

--- lib/charm/openstack/ceph_rbd_mirror.py.orig 2020-10-05 04:05:01.126851853 +0000
+++ lib/charm/openstack/ceph_rbd_mirror.py 2020-10-05 04:05:12.739024844 +0000
@@ -71,7 +71,8 @@
         :rtype: dict
         """
         return {pool: attrs for pool, attrs in pools.items()
- if 'rbd' in attrs['applications']}
+ if ('rbd' in attrs['applications']
+ and 'erasure_code_profile' not in attrs['parameters'])}

def custom_assess_status_check(self):
"""Provide mirrored pool statistics through juju status."""

The full trace of the above error:

2020-10-05 04:15:55.440 7f14a0fce0c0 -1 librbd::api::Mirror: mode_set: failed to allocate mirroring uuid: (95) Operation not supported
Traceback (most recent call last):
  File "hooks/ceph-local-relation-changed", line 22, in <module>
    main()
  File "/var/lib/juju/agents/unit-ceph-rbd-mirror-az1-0/.venv/lib/python3.6/site-packages/charms/reactive/__init__.py", line 74, in main
    bus.dispatch(restricted=restricted_mode)
  File "/var/lib/juju/agents/unit-ceph-rbd-mirror-az1-0/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 390, in dispatch
    _invoke(other_handlers)
  File "/var/lib/juju/agents/unit-ceph-rbd-mirror-az1-0/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 359, in _invoke
    handler.invoke()
  File "/var/lib/juju/agents/unit-ceph-rbd-mirror-az1-0/.venv/lib/python3.6/site-packages/charms/reactive/bus.py", line 181, in invoke
    self._action(*args)
  File "/var/lib/juju/agents/unit-ceph-rbd-mirror-az1-0/charm/reactive/ceph_rbd_mirror_handlers.py", line 104, in configure_pools
    charm_instance.mirror_pool_enable(pool)
  File "lib/charm/openstack/ceph_rbd_mirror.py", line 155, in mirror_pool_enable
    subprocess.check_call(base_cmd + ['enable', pool, 'pool'])
  File "/usr/lib/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['rbd', '--id', 'rbd-mirror.juju-cf9afe-8-lxd-0', 'mirror', 'pool', 'enable', 'cinder-ceph-az1-arbor', 'pool']' returned non-zero exit status 95.

Tags:

Revision history for this message

Vern Hart (vern) wrote on 2021-01-13:

Subscribed field-medium to see if we can get this change (or something like it) into the charm.

Revision history for this message

Aurelien Lourot (aurelien-lourot) wrote on 2021-01-14:

Hi Vern, thanks for reporting! Could you please provide a juju deployment bundle (so we can see which config options are being used) and are there any additional steps to reproduce other than just deploying? Thanks a lot!

Revision history for this message

Vern Hart (vern) wrote on 2021-01-15:

bundle.yaml Edit (63.5 KiB, text/plain)

I have attached the sanitized bundle for this deployment. We have two ceph clusters, which is necessary for ceph-rbd-mirror, obviously, but they are in a single model. I don't believe this is germane to this bug so two models with cross model relations should also be sufficient.

We have multiple cinder-ceph charms deployed here but only one with pool-type=erasure-coded should be necessary.

Once the cinder-ceph charm creates its pool, rbd-mirror will try to mirror it and the above error will show up in the /var/log/ceph/ceph-client.rbd-mirror.*.log file.

Alternately, one could manually create a ceph pool with erasure coding and not need cinder-ceph at all.

Revision history for this message

James Page (james-page) wrote on 2021-01-15:

The lack of support for erasure coded pools is in the rbd-mirror application itself and is noted in the charm deployment guide section for use of erasure coded pools:

https://docs.openstack.org/project-deploy-guide/charm-deployment-guide/latest/app-erasure-coding.html

Revision history for this message

James Page (james-page) wrote on 2021-01-15:

(your proposed fix to exclude any pools using erasure coding makes sense)

Billy Olsen (billy-olsen) on 2021-04-03

Changed in charm-ceph-rbd-mirror:
status:	New → In Progress
assignee:	nobody → Billy Olsen (billy-olsen)
milestone:	none → 21.04
importance:	Undecided → Medium

Billy Olsen (billy-olsen) on 2021-04-09

Changed in charm-cinder-ceph:
importance:	Undecided → Medium
Changed in charm-glance:
importance:	Undecided → Medium
assignee:	nobody → Billy Olsen (billy-olsen)
Changed in charm-cinder-ceph:
assignee:	nobody → Billy Olsen (billy-olsen)
status:	New → In Progress

Alex Kavanagh (ajkavanagh) on 2021-05-03

Changed in charm-ceph-rbd-mirror:
milestone:	21.04 → none

Billy Olsen (billy-olsen) on 2022-05-13

no longer affects:

charm-glance

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2022-09-12: Change abandoned on charm-ceph-rbd-mirror (master)

Change abandoned by "James Page <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/charm-ceph-rbd-mirror/+/784630
Reason: This review is > 12 weeks without comment, and failed testing the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.