Comment 6 for bug 1933914

Here's an example:

root@ceph-osd-2:/var/lib/juju/agents/unit-ceph-osd-2/charm# ls -al /dev/disk/by-dname/bca*
lrwxrwxrwx 1 root root 13 Jun 24 21:54 /dev/disk/by-dname/bcache0 -> ../../bcache6
lrwxrwxrwx 1 root root 13 Jun 24 21:54 /dev/disk/by-dname/bcache1 -> ../../bcache7
lrwxrwxrwx 1 root root 13 Jun 24 21:54 /dev/disk/by-dname/bcache2 -> ../../bcache5
lrwxrwxrwx 1 root root 13 Jun 24 21:54 /dev/disk/by-dname/bcache3 -> ../../bcache3
lrwxrwxrwx 1 root root 13 Jun 24 21:54 /dev/disk/by-dname/bcache4 -> ../../bcache4
lrwxrwxrwx 1 root root 13 Jun 24 21:54 /dev/disk/by-dname/bcache5 -> ../../bcache2
lrwxrwxrwx 1 root root 13 Jun 24 21:54 /dev/disk/by-dname/bcache6 -> ../../bcache0
lrwxrwxrwx 1 root root 13 Jun 24 21:54 /dev/disk/by-dname/bcache7 -> ../../bcache1
root@ceph-osd-2:/var/lib/juju/agents/unit-ceph-osd-2/charm# sqlite3 .unit-state.db
SQLite version 3.22.0 2018-01-22 18:45:57
Enter ".help" for usage hints.
sqlite> select data from kv where key='osd-devices';
["/dev/disk/by-dname/bcache7", "/dev/bcache0", "/dev/bcache1", "/dev/bcache5", "/dev/bcache2", "/dev/bcache4"]

This one is even tougher. I'm guessing when this was deployed, we used /dev/bcacheX, and because those got renamed upon each boot, we created the udev rules for /dev/disk/by-dname paths and then triggered this issue. As you can see ehre, some of those /dev/bcacheX's don't map to the current booted host's /dev/disk/by-dname/bcache[2-7] which is what was defined in osd-devices, so I don't even think you could use that methodology to trace this.

I think ultimately, storing /dev/bcacheX paths in osd-devices is futile, as they are renamed on each boot, and some other stragegy will be needed to ensure the charm doesn't reconfigure a disk already known, and that it knows that the disks that are configured were configured by itself. Perhaps use of the bcache UUID of the device would help.