add-disk and zap-disk actions don't work for mounted filesystem paths

Bug #1838706 reported by Drew Freiberger
16
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Ceph OSD Charm
Triaged
Wishlist
Unassigned

Bug Description

When trying to replace a failed disk on an environment with 19.04 charms and using osd-devices=/path/to/mountpoint1 /path/to/mountpoint2 ..., we cannot add-disk /path/to/mountpoint2 if it had to be rebuilt due to hardware replacement.

The process would typically be:

juju run-action --wait ceph-osd/X zap-disk devices=/path/to/mountpoint2 i-really-mean-it=yes
* replace disk, reformat, and mount *
juju run-action --wait ceph-osd/X add-disk osd-devices=/path/to/mountpoint2

unfortunately, zap-disk action bombs out immediately if devices is not a block device.

zap-disk is the only way in the charm to remove that path from the kv() db of known/configured disks.

Suggest zap-disk have the ability to reduce the db disks values by the path provided at a very minimum so that add-disk doesn't skip the re-add of the disk.

Current workaround to allow for add-disk on already-configured path:

1. login to unit
2. cd /var/lib/juju/agents/unit-ceph-osd-X/charm
3. apt-get install sqlite
4. sudo sqlite3 .unit-state.db
5. enter the following in sqlite prompt:
    select data from kv where key='osd-devices';
    *capture output for update command*
    update kv set data='<list construct minus the path to zap>' where key='osd-devices';
    select data from kv where key='osd-devices';
    *validate setting is as expected*
    .quit

Revision history for this message
Drew Freiberger (afreiberger) wrote :

I've also found that add-disk does not function properly on an already mounted filesystem. It gets through the chown/chmod and ceph-osd prepare command, but fails at finding the block UUID and adding to hdparm and bombs out before updating the relation data to ceph-mon.

summary: - zap-disk action doesn't work for mounted filesystems
+ add-disk and zap-disk actions don't work for mounted filesystem paths
Changed in charm-ceph-osd:
importance: Undecided → Wishlist
status: New → Triaged
Revision history for this message
Robert Bopko (zer69) wrote :

Encountered the same problem during upgrade of old test system running xenial-queens to bionic-train.

~2 years ago I had a working charm like this (cs:ceph-osd-257):

      osd-devices: /mnt/ceph
      osd-reformat: "yes"

The reformat option doesn't work now.

Any workaround for this? I am limited to one disk where one partition is /mnt/ceph (sda1) and the second is the root fs (sda2).

In juju status I have "No block devices detected using current configuration"

When I try to set the osd-devices to /dev/sda1 I have the error:

2020-02-04 20:23:05 DEBUG config-changed Failed to find physical volume "/dev/sda1".

I tried the above workaround but:

sqlite> select data from kv where key='osd-devices';
[]
sqlite>

Many thanks.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.