bcache by-uuid links disappear after mounting bcache0

root@ubuntu:~# bash -x ./test-bcache-byuuid-links.sh
+ mountpoint /mnt
/mnt is a mountpoint
+ umount -f /mnt
+ '[' -e /dev/bcache0 ']'
+ wipefs -a /dev/bcache0
/dev/bcache0: 2 bytes were erased at offset 0x00000438 (ext4): 53 ef
+ true
++ ls /sys/class/block/bcache0/bcache/stop
+ bdev_stop=/sys/class/block/bcache0/bcache/stop
+ '[' -n /sys/class/block/bcache0/bcache/stop -a -e /sys/class/block/bcache0/bcache/stop ']'
+ echo 'writing 1 to /sys/class/block/bcache0/bcache/stop'
writing 1 to /sys/class/block/bcache0/bcache/stop
+ echo 1
+ udevadm settle
+ sleep 1
+ true
++ ls /sys/class/block/bcache0/bcache/stop
ls: cannot access '/sys/class/block/bcache0/bcache/stop': No such file or directory
+ bdev_stop=
+ '[' -n '' -a -e '' ']'
+ break
+ true
++ ls /sys/fs/bcache/06727454-1020-4b7b-99ef-14789f7f7f3b/stop
+ cdev_stop=/sys/fs/bcache/06727454-1020-4b7b-99ef-14789f7f7f3b/stop
+ '[' -n /sys/fs/bcache/06727454-1020-4b7b-99ef-14789f7f7f3b/stop -a -e /sys/fs/bcache/06727454-1020-4b7b-99ef-14789f7f7f3b/stop ']'
+ echo 'writing 1 to /sys/fs/bcache/06727454-1020-4b7b-99ef-14789f7f7f3b/stop'
writing 1 to /sys/fs/bcache/06727454-1020-4b7b-99ef-14789f7f7f3b/stop
+ echo 1
+ udevadm settle
+ sleep 1
+ true
++ ls '/sys/fs/bcache/*-*-*-*/stop'
ls: cannot access '/sys/fs/bcache/*-*-*-*/stop': No such file or directory
+ cdev_stop=
+ '[' -n '' -a -e '' ']'
+ break
+ (( x=0 ))
+ (( x<50 ))
+ wipefs -a /dev/sdc
/dev/sdc: 16 bytes were erased at offset 0x00001018 (bcache): c6 85 73 f6 4e 1a 45 ca 82 65 f5 7f 48 ba 6d 81
+ rc=0
+ '[' 0 = 0 ']'
+ break
+ '[' 0 '!=' 0 ']'
+ (( x=0 ))
+ (( x<50 ))
+ wipefs -a /dev/sdb
/dev/sdb: 16 bytes were erased at offset 0x00001018 (bcache): c6 85 73 f6 4e 1a 45 ca 82 65 f5 7f 48 ba 6d 81
+ rc=0
+ '[' 0 = 0 ']'
+ break
+ '[' 0 '!=' 0 ']'
+ make-bcache --wipe-bcache -C /dev/sdc -B /dev/sdb
UUID: 59306fe6-b5fd-439b-bea3-df7ee65d0b8d
Set UUID: 89103ca3-a773-4fca-a456-f10bd3ea17b2
version: 0
nbuckets: 20480
block_size: 1
bucket_size: 1024
nr_in_set: 1
nr_this_dev: 0
first_bucket: 1
UUID: 8b8fb68a-8f3c-46ba-b656-0442b511bfe7
Set UUID: 89103ca3-a773-4fca-a456-f10bd3ea17b2
version: 1
block_size: 1
data_offset: 16
+ udevadm settle --exit-if-exists /dev/bcache0
+ '[' '!' -e /dev/bcache/by-uuid ']'
+ ls -al /dev/bcache/by-uuid/
total 0
drwxr-xr-x 2 root root 60 Feb 4 23:30 .
drwxr-xr-x 3 root root 60 Feb 4 23:30 ..
lrwxrwxrwx 1 root root 13 Feb 4 23:30 8b8fb68a-8f3c-46ba-b656-0442b511bfe7 -> ../../bcache0
+ echo 'Creating filesystems on bcache0'
Creating filesystems on bcache0
+ mkfs.ext4 /dev/bcache0
mke2fs 1.45.3 (14-Jul-2019)
Discarding device blocks: done
Creating filesystem with 2621438 4k blocks and 655360 inodes
Filesystem UUID: 48988989-b276-42c1-b999-e29928353796
Superblock backups stored on blocks:
 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632

Allocating group tables: done
Writing inode tables: done
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done

+ echo 'mounting bcache0 to /mnt'
mounting bcache0 to /mnt
+ mount /dev/bcache0 /mnt
+ udevadm settle --exit-if...

I also see this on Eoan kernel, 5.3.0-29-generic, Bionic 4.15.0-76-generic does NOT show this issue.

I suppose it could be systemd-udev issue as well, Bionic has

# apt-cache policy systemd
systemd:
  Installed: 237-3ubuntu10.33
  Candidate: 237-3ubuntu10.33

vs Eoan/Focal at

# apt-cache policy systemd
systemd:
  Installed: 242-7ubuntu3.2
  Candidate: 242-7ubuntu3.2
  Version table:
 *** 242-7ubuntu3.2 500
        500 http://archive.ubuntu.com/ubuntu eoan-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     242-7ubuntu3 500
        500 http://archive.ubuntu.com/ubuntu eoan/main amd64 Packages

# apt-cache policy systemd
systemd:
  Installed: 244.1-0ubuntu2
  Candidate: 244.1-0ubuntu2
  Version table:
 *** 244.1-0ubuntu2 500
        500 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages
        100 /var/lib/dpkg/status

It's much harder to see this on 5.3.0-29, but it does happen.

I've also tried on 5.4.0-9-generic on focal and it also has this issue.

This is still occurring daily.

Adding Linux task to get kernel team's attention

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1861941

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

I've tested this with 4.15, 5.0, 5.3 and 5.4 kernels on bionic, eoan and focal and I'm getting the same results, always a failure. I've not got it to work at all on 4.15.0 so far. I'm not sure why this is.

Can you run this on a 4.15 based system and gather the output from bash -x ./test-bcache-byuuid-links.sh and provide:

1. the output of a bash -x run that works
2. the kernel version and release info

Then I'll have a go at reproducing it again.

Requested output on bionic release image (4.15-20)

Requested data from a daily cloud image with 4.15-76

I'm on Focal desktop, running kvm like so

qemu-system-x86_64 -smp 2 -m 1024 --enable-kvm \
  -drive id=disk0,if=none,format=qcow2,file=bionic-bcache-links.qcow2 \
  -device virtio-blk-pci,drive=disk0,bootindex=0 \
  -drive id=disk1,if=none,format=raw,file=bcache1.img \
  -device virtio-blk-pci,drive=disk1 \
  -drive id=disk2,if=none,format=raw,file=bcache2.img \
  -device virtio-blk-pci,drive=disk2 \
  -drive id=cdrom0,if=none,media=cdrom,file=seed.img \
  -device virtio-blk-pci,drive=cdrom0,bootindex=1 \
  -net user,hostfwd=tcp::22222-:22 -net nic,model=virtio \
  -monitor stdio -vnc 0.0.0.0:0 \
  -object rng-random,filename=/dev/urandom,id=rng0 \
  -device virtio-rng-pci,rng=rng0 \
  -serial telnet:localhost:2446,nowait,server

The images are created with:

wget https://cloud-images.ubuntu.com/daily/server/server/bionic/current/bionic-server-cloudimg-amd64.img
qemu-img create -f qcow2 -b bionic-server-cloudimg-amd64.img \
        bionic-bcache-linux.qcow2 20G
qemu-img create -f raw bcache1.img 10G
qemu-img create -f raw bcache2.img 10G

cat <<EOF >user-data
#cloud-config
password: passw0rd
chpasswd: { expire: False }
ssh_pwauth: True
ssh_authorized_keys:
 - |
   <your ssh key here>
EOF

# you'll need cloud-image-utils package installed
cat <<EOF >boot.sh
#!/bin/bash
BOOT=$1
SEED=${2-"seed.img"}
USERDATA=${3-"user-data"}

[ ! -e ${SEED} ] && {
    [ ! -e ${USERDATA} ] && {
        echo "You need a 'user-data' file"
        exit 1;
    }
    [ ! -e meta-data ] && {
         echo "instance-id: $(uuidgen || echo i-abcdefg)" > meta-data
    }
    echo "Creating seed..."
    cloud-localds ${SEED} ${USERDATA} meta-data
}

qemu-system-x86_64 -smp 2 -m 1024 --enable-kvm \
  -drive id=disk0,if=none,format=qcow2,file=${BOOT} \
  -device virtio-blk-pci,drive=disk0,bootindex=0 \
  -drive id=disk1,if=none,format=raw,file=${DISK1} \
  -device virtio-blk-pci,drive=disk1 \
  -drive id=cdrom0,if=none,media=cdrom,file=${SEED} \
  -device virtio-blk-pci,drive=cdrom0,bootindex=1 \
  -net user,hostfwd=tcp::22222-:22 \
  -net nic,model=virtio \
  -monitor stdio \
  -vnc 0.0.0.0:0 \
  -object rng-random,filename=/dev/urandom,id=rng0 \
  -device virtio-rng-pci,rng=rng0 \
  -serial telnet:localhost:2446,nowait,server \
EOF

# watch the server boot with
telnet localhost 2446

# ssh into the instance with
ssh -p 22222 ubuntu@localhost

It appears that it's always been a touch racy. Curtin does not create bcaches like the script does (make-bcache --wipe-bcache -C /dev/sdc -B /dev/sdb), rather we make the cache-dev and backing dev separately, and then attach them by echoing the cacheset uuid into the bcache device attach sysfs file. The issue is after install when we first boot the system that has one or more bcache devices, we *sometimes* don't see the /dev/bcache/by-uuid symlinks show up.

Focal _always_ fails; Eoan works sometimes, in our testing, bionic has never failed; though clearly it suffers from the same (or similar) race.

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1861941

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Thanks for the data. I suspect the racy nature is why I can't reproduce it on my H/W, perhaps the I/O and/or CPU speeds are relevant in tripping the issue.

I think we need a udev plumbing expert, I've exhausted my udev knowledge.

For older releases such as bionic we have one event:

UDEV [1304.304437] change /devices/virtual/block/bcache0 (block)
.ID_FS_TYPE_NEW=
ACTION=change
CACHED_LABEL=
CACHED_UUID=0192e0a7-9d8e-4771-9bb1-d730a2f66a9d
DEVLINKS=/dev/bcache/by-uuid/0192e0a7-9d8e-4771-9bb1-d730a2f66a9d
DEVNAME=/dev/bcache0
DEVPATH=/devices/virtual/block/bcache0
DEVTYPE=disk
DRIVER=bcache
ID_FS_TYPE=
MAJOR=251
MINOR=0
SEQNUM=4384
SUBSYSTEM=block
TAGS=:systemd:
USEC_INITIALIZED=1304237024

that creates the expected symlink.

---------

For focal, instead we get two events:

UDEV [2763.370729] change /devices/virtual/block/bcache0 (block)
ACTION=change
DEVPATH=/devices/virtual/block/bcache0
SUBSYSTEM=block
DRIVER=bcache
CACHED_UUID=1522715e-a390-40db-8d47-06fb728014e5
CACHED_LABEL=
DEVNAME=/dev/bcache0
DEVTYPE=disk
SEQNUM=5523
USEC_INITIALIZED=2763323504
.ID_FS_TYPE_NEW=
ID_FS_TYPE=
MAJOR=251
MINOR=0
DEVLINKS=/dev/bcache/by-uuid/1522715e-a390-40db-8d47-06fb728014e5
TAGS=:systemd:

KERNEL[2784.362374] change /devices/virtual/block/bcache0 (block)
ACTION=change
DEVPATH=/devices/virtual/block/bcache0
SUBSYSTEM=block
SYNTH_UUID=0
DEVNAME=/dev/bcache0
DEVTYPE=disk
SEQNUM=5524
MAJOR=251
MINOR=0

UDEV [2784.433618] change /devices/virtual/block/bcache0 (block)
ACTION=change
DEVPATH=/devices/virtual/block/bcache0
SUBSYSTEM=block
SYNTH_UUID=0
DEVNAME=/dev/bcache0
DEVTYPE=disk
SEQNUM=5524
USEC_INITIALIZED=2763323504
ID_FS_UUID=30b28bee-6a1e-423d-9d53-32c78ba5454a
ID_FS_UUID_ENC=30b28bee-6a1e-423d-9d53-32c78ba5454a
ID_FS_VERSION=1.0
ID_FS_TYPE=ext4
ID_FS_USAGE=filesystem
.ID_FS_TYPE_NEW=ext4
MAJOR=251
MINOR=0
DEVLINKS=/dev/disk/by-uuid/30b28bee-6a1e-423d-9d53-32c78ba5454a
TAGS=:systemd:

This performs:

pr 21 14:15:22 ubuntu-focal systemd-udevd[1870]: bcache0: Processing device (SEQNUM=5523, ACTION=change)
Apr 21 14:15:22 ubuntu-focal systemd-udevd[1870]: bcache0: Removing watch
Apr 21 14:15:22 ubuntu-focal systemd-udevd[1870]: value '[dmi/id]sys_vendor' is 'QEMU'
Apr 21 14:15:22 ubuntu-focal systemd-udevd[1870]: value '[dmi/id]sys_vendor' is 'QEMU'
Apr 21 14:15:22 ubuntu-focal systemd-udevd[1870]: bcache0: /usr/lib/udev/rules.d/60-persistent-storage.rules:109 Importing properties from results of builtin command 'blkid'
Apr 21 14:15:22 ubuntu-focal systemd-udevd[1870]: bcache0: Probe /dev/bcache0 with raid and offset=0
Apr 21 14:15:22 ubuntu-focal systemd-udevd[1870]: bcache0: /usr/lib/udev/rules.d/69-bcache.rules:16 Importing properties from results of 'probe-bcache -o udev /dev/bcache0'
Apr 21 14:15:22 ubuntu-focal systemd-udevd[1870]: bcache0: Starting 'probe-bcache -o udev /dev/bcache0'
Apr 21 14:15:22 ubuntu-focal systemd-udevd[1870]: Successfully forked off '(spawn)' as PID 1912.
Apr 21 14:15:22 ubuntu-focal systemd-udevd[1870]: bcache0: Process 'probe-bcache -o udev /dev/bcache0' succeeded.
Apr 21 14:15:22 ubuntu-focal systemd-udevd[1870]: bcache0: /usr/lib/udev/rules.d/69-bcache.rules:27 LINK 'bcache/by-uuid/1522715e-a390-40db-8d47-06fb728014e5'
Apr 21 14:15:22 ubuntu-focal systemd-udevd[1870]: bcache0: Handling device node '/dev/bcache0', devnum=b251:0
Apr 21 14:15:22 ubuntu-focal systemd-udevd[187...

The good news is that the bcache events are reaching udev, however the /usr/lib/udev/rules.d/60-persistent-storage.rules rules may need to be modified to ignore bcache specific events.

So, this looks like the bug to me:

Apr 21 14:15:43 ubuntu-focal systemd-udevd[1916]: bcache0: /usr/lib/udev/rules.d/60-persistent-storage.rules:112 LINK 'disk/by-uuid/30b28bee-6a1e-423d-9d53-32c78ba5454a'
Apr 21 14:15:43 ubuntu-focal systemd-udevd[1916]: bcache0: Updating old name, '/dev/bcache/by-uuid/1522715e-a390-40db-8d47-06fb728014e5' no longer belonging to '/devices/virtual/block/bcache0'
Apr 21 14:15:43 ubuntu-focal systemd-udevd[1916]: bcache0: No reference left, removing '/dev/bcache/by-uuid/1522715e-a390-40db-8d47-06fb728014e5'
Apr 21 14:15:43 ubuntu-focal systemd-udevd[1916]: bcache0: Handling device node '/dev/bcache0', devnum=b251:0

In Focal, the second event is valid, but 60-persistent-storage.rules has no rules that touch /dev/bcache/by-uuid; they don't even use the same ENV value ... so why would the by-uuid symlink rule in 60-persistent-storage.rules remove a link it doesn't have a rule for?

Do we have any more information on why we now get two events in Focal?

Following up my question; we should see both events in all kernels.

The first event is when the /dev/bcache0 is joined with a cache device, and emitts the CACHED_UUID value in the uevent; the UUID is the *backing device bcache metadata UUID* it is not related to the content contained within the backing device; this CACHED_UUID is symlinked at /dev/bcache/by-uuid

Once /dev/bcache0 is created, blkid will scan the contents of /dev/bcache0 which contains an ext4 filesystem and the FS_UUID is symlinked at /dev/disk/by-uuid.

CACHED_UUID != FS_UUID
/dev/bcache/by-uuid != /dev/disk/by-uuid

I speculate that udev has a bug where it's only looking at the basename of the symlink target and removing any other 'by-uuid' link against the device when it needs to account for the full path of the symlink since the bcache/by-uuid is different than the disk/by-uuid.

udev source package last existed in precise. systemd-udevd is now shipped in src:systemd, please file all future udev bugs against systemd as a starting point.

This appears not to be a kernel issue per-se, so any idea of an ETA for a fix with udev for this?

The reason for udev cleaning up the links is that they are missing from the change event's DEVLINKS.
Since bcache-tools manages those symlinks IMO it should keep them around.

Please take a look at the attached patch, I'll upload it if it is OK.

I guess I don't understand why we see this in focal. The two events in Colin's trace always happen on any Ubuntu kernel. We should see if we can get another udev trace on bionic that captures both CHANGE events, one will be from the bcache driver itself, and one is from the block layer. THe order and content of the change event matter.

Another thing I don't understand is why does udev drop the a symlink created by another rule? This seems like the core issue.

Looking at systemd/udev source code; udev will do a FOREACH_DEVICE_DEVLINK and check if the name is in the database file for the device. Its not clear to me yet how or when the database file get's written.

The other question I have is: if we reversed the order of the focal CHANGE events, wouldn't we see just the opposite happen (the driver=bcache change event would not have all of the devlinks from a blkid probe) and all of the /dev/disk/by-{id, uuid, ...} get removed when running

I think the patch you're proposing should work; but I don't think we've root caused why the link gets removed in the first place. Once we understand the root cause, I think we can better understand what a fix should look like.

Lastly, I think we might also test this out on mainline kernel; I wonder if our SAUCE patch for emitting the CACHED_UUID needs an update (or could be dropped).

This bug was fixed in the package bcache-tools - 1.0.8-4

---------------
bcache-tools (1.0.8-4) unstable; urgency=medium

  [ Debian Janitor ]
  * Trim trailing whitespace.
  * Bump debhelper from old 11 to 12.
  * Set debhelper-compat version in Build-Depends.

  [ Balint Reczey ]
  * Add Salsa CI configuration
  * Keep symlinks when udev changes /dev/bcacheN (LP: #1861941)
  * Drop update-initramfs trigger.
    With dh compat version 12 update-initramfs is inserted to postinst
    automatically

 -- Balint Reczey <email address hidden> Wed, 20 May 2020 14:41:11 +0200

@raharper To my understanding with the change action udev removes the DEVLINKS which are not still valid. Udev can't tell which links are still valid in cache/by-*.

The only reason udev knows about the links is because the rule added them with SYMLINK+= when the dev was created and it is a sane assumption that when a change action appears with DEVLINKS only the new DEVLINKS are valid. Hence the fix.

That doesn't explain why they show up sometimes, but not all of
the time.

There are 3 devices in play here.

* The backing device, let's say /dev/vda; this is where we want
  to store the data.
* The caching device, let's say /dev/vdb; this holds the cache.
* The bcache device; this only appears when a backing device is
  detected; via bcache-probe which reads the header on the disk
  and indicates it's a bcache device.

Now, there are a few sequences that we see:

1) make-bcache -b /dev/vda -c /deb/vdb will create a bcache0
   with the cache and backing dev at the same time and emit the
   CACHED_UUID change event which runs *after*
   60-persistent-storage, and adds the bcache/by-uuid links

2) After a reboot (or if you tear down a bcache0 by stopping them
   in sysfs) we have several scenarios where the 2 real block
   devices are found:

2a) order: backing, bcacheN, caching

    In this scenario, vda will run through
    60-persistent-storage.rules and blkid will discover
    ID_FS_TYPE=bcache, and generates only by-path, by-id links
    for the device, 69-bcache.rules will run bcache-register
    /dev/vda which informs the kernel that a backing device has
    been found.

    If the cache device has not yet been handled by udev (we
    don't know if a cache device exists, bcache can work
    without one) then the kernel will create a bcacheN
    device, but without the cache device being present, it will
    *NOT* emit a CHANGE event with CACHED_UUID value.

    Now when the cache device (vdb) is going through,
    60-persistent-storage.rules, blkid will discover the same
    settings, TYPE=bcache, generate by-path, by-id symlinks.
    Then 69-bcache.rules runs, triggers a bcache-register on vdb
    registering the cache device in the kernel and because
    the superblock on the backing device specifies the
    cache device UUID it used the kernel can bind the cache
    device into the correct bcacheN device.

    A mainline kernel (no sauce patch) does not emit a CHANGE
    event with CACHED_UUID value. Our sauce patch changes
    the kernel to emit the CHANGE event with CACHED_UUID
    whenever we bind a cache device to an existing bcacheN
    precisely so the bcache/by-* links get generated.

2b) order: cache, backing, bcacheN

   Here, when vdb is plugged in, the cache device is registered
   in the kernel, but without a backing device, no bcacheN device
   is created yet.

   Then the backing is probed, as in 2a; the bcache-register command
   will register it with the kernel, and create a bcache0, and
   since the cache device is present, mainline kernels (and ours)
   will emit the CHANGE event with CACHED_UUID

With this in-mind; it appears to me that in the focal case
we're seeing the bcache driver CHANGE event with CACHED_UUID
and then additional change events from the block layer around
the bcache0 device (but not emitted from the bcache driver code)
So I'd like to understand why ... I expect those events to occur
but not before the driver change event.

And maybe that's the race here

The happy path:

  1. udev processing CHANGE on backing device, it's bcache so we register it
  2. kernel bcache driver creates a bcache0...

Digging deeper and walking through this in a focal vm, I'm seeing some strange things.

Starting with a clean disk, and just creating the backing device like so:

make-bcache -B /dev/vdb

We see /dev/bcache0 get created with vdb as the backing device. Now, after this, I see:

/dev/bcache/by-uuid/<dev.uuid> -> ../../bcache0

This is unexpected. It should *only* appear once the bcache0 device is actively cached; that is the bcache driver should not emit CACHED_UUID, until the UUID is actually cached, which only happens once we've registered a cache device with the bcache0 device. We need to look at the kernel source to see if the SAUCE patch isn't quite right, or some other part of bcache code has changed.

Next issue, blkid now detects the bcache dev.uuid, and this shows up like so:

# udevadm test-builtin blkid /sys/class/block/vdb
...
vdb: Probe /dev/vdb with raid and offset=0
ID_FS_UUID=d7d7e025-b8d2-43cb-a5df-c240ba1418dc
ID_FS_UUID_ENC=d7d7e025-b8d2-43cb-a5df-c240ba1418dc
ID_FS_TYPE=bcache
ID_FS_USAGE=other

Note, it shows up as an FS_UUID, but it is *NOT* an fs_uuid; there is no filesystem on this device at all at this time; it does have a bcache superblock, note the FS_UUID matches the dev.uuid of the backing device.

# bcache-super-show /dev/vdb
sb.magic ok
sb.first_sector 8 [match]
sb.csum 29D6774A332A280B [match]
sb.version 1 [backing device]

dev.label (empty)
dev.uuid d7d7e025-b8d2-43cb-a5df-c240ba1418dc
dev.sectors_per_block 1
dev.sectors_per_bucket 1024
dev.data.first_sector 16
dev.data.cache_mode 0 [writethrough]
dev.data.cache_state 0 [detached]

cset.uuid 37b37bc1-e185-4825-8900-579df102b7d6

It's also curious that cset.uuid has a UUID, as there are no csets created; IMO this is also a bug and it should be 0, null, or empty.

Now, here's where the udev fights over the links. The backing device, vdb, triggers this rule in 60-persistent-storage.rules:

# by-label/by-uuid links (filesystem metadata)
ENV{ID_FS_USAGE}=="filesystem|other|crypto", ENV{ID_FS_UUID_ENC}=="?*", SYMLINK+="disk/by-uuid/$env{ID_FS_UUID_ENC}"

This creates confusion due to the CACHED_UUID also being emitted from the kernel, like so:

root@ubuntu:/run/udev/links# ls -al /dev/disk/by-uuid/d7d7e025-b8d2-43cb-a5df-c240ba1418dc
lrwxrwxrwx 1 root root 9 May 21 16:39 /dev/disk/by-uuid/d7d7e025-b8d2-43cb-a5df-c240ba1418dc -> ../../vdb
root@ubuntu:/run/udev/links# ls -al /dev/bcache/by-uuid/d7d7e025-b8d2-43cb-a5df-c240ba1418dc
lrwxrwxrwx 1 root root 13 May 21 16:39 /dev/bcache/by-uuid/d7d7e025-b8d2-43cb-a5df-c240ba1418dc -> ../../bcache0

There we have two devices both claiming the backing devices dev.uuid value, with two different block names.

So, in summary, there are two problems to fix:

1) blkid on bcache devices should not report FS_UUID values when FS_TYPE=bcache; there is no filesystem on it; this will need to be discussed upstream

2) our kernel (need to check mainline vs. ours) emits CACHED_UUID when the bcache0 is created, but no cache is attached: see this udev event:

KERNEL[2217.605118] change /devices/virtual/block/bcache0 (block)
ACTION=change
DEVPATH=/devices/virtual/block/bcache0
SUBSYSTEM=block
DRIVER=bcache
CACHED_UUID=d7d7e025-...

@Balint

I do not thing the fix you're released is correct, can you upload a new version without the scripts?

Also, we should fix make-bcache -B to ensure that cset.uuid is not initialized; that may be why the kernel thinks it should emit the CACHED_UUID if the suerpblock of the device has a cset.uuid value (versus None).

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1861941

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

OK.

I've reviewed the kernel code, and there are no unexpected changes w.r.t the CACHED_UUID change event. So I don't think we will need any kernel changes which is good.

With the small change to the 60-persistent-storage.rules to not attempt to create a /dev/disk/by-uuid symlink for the backing device; instead we want to only create a /dev/bcache/by-uuid symlink to the bcacheN device (that is associated with the backing device).

# by-label/by-uuid links (filesystem metadata), skip bcache backing,caching devices, handled in 69-bcache.rules
ENV{ID_FS_TYPE}=="bcache", GOTO="skip_fs_uuid_enc"
ENV{ID_FS_USAGE}=="filesystem|other|crypto", ENV{ID_FS_UUID_ENC}=="?*", SYMLINK+="disk/by-uuid/$env{ID_FS_UUID_ENC}"
ENV{ID_FS_USAGE}=="filesystem|other|crypto", ENV{ID_FS_LABEL_ENC}=="?*", SYMLINK+="disk/by-label/$env{ID_FS_LABEL_ENC}"
LABEL="skip_fs_uuid_enc"

And then with my previously proposed changed to 69-bcache.rules in place

https://github.com/g2p/bcache-tools/pull/29/files

The test-case works just fine.

Tarball of a source package with a fix for this issue:

bcache-tools_1.0.8.orig.tar.gz
bcache-tools_1.0.8-4ubuntu1_amd64.build
bcache-tools_1.0.8-4ubuntu1_amd64.buildinfo
bcache-tools_1.0.8-4ubuntu1_amd64.changes
bcache-tools_1.0.8-4ubuntu1_amd64.deb
bcache-tools_1.0.8-4ubuntu1.debian.tar.xz
bcache-tools_1.0.8-4ubuntu1.dsc
bcache-tools_1.0.8-4ubuntu1_source.build
bcache-tools_1.0.8-4ubuntu1_source.buildinfo
bcache-tools_1.0.8-4ubuntu1_source.changes
bcache-tools-dbgsym_1.0.8-4ubuntu1_amd64.ddeb
bcache-tools-debdiff-1.0.8-4_to_1.0.8-4ubuntu1

debdiff of the changes

Updated test to be a bit more resilient.

systemd debdiff with a fix to skip creating /dev/disk/by-uuid for bcache backing, caching devices.

@raharper Thanks, I've uploaded the systemd fix to Groovy. Could you please forward it to upstream?

Also please update bcache-tools since it is Server Team's package and I also can't upload to Debian until the next keyring refresh.

Status changed to 'Confirmed' because the bug affects multiple users.

Subscring ~field-critical as blocked by it on a customer deployment.
Seems like the duplicate of the one below
https://bugs.launchpad.net/charm-ceph-osd/+bug/1883585

The environment used is bionic/4.15.0-106-generic

$ dpkg -l | grep bcache-tools
ii bcache-tools 1.0.8-2build1 amd64 bcache userspace tools

This bug was fixed in the package systemd - 245.6-1ubuntu1

---------------
systemd (245.6-1ubuntu1) groovy; urgency=medium

  [ Balint Reczey ]
  * Merge to Ubuntu from Debian unstable
  - Dropped changes:
    * Don't try calling NULL callback in bus_wait_for_units_clear.
    * test: Skip test-boot-timestamps on permission denied.
    * From journald: stream pid change newline fix
      - debian/patches/lp1875708/journald-Increase-stdout-buffer-size-sooner-when-almost-f.patch
    * meson: initialize time-epoch to reproducible builds compatible value
  * dhclient-exit-hooks.d/timesyncd: Act only when systemd-timesyncd is enabled
    (LP: #1873031)
    File: debian/extra/dhclient-exit-hooks.d/timesyncd
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=7d5a31fb7523d1ecef0d2932df5efa3b99322919
  * debian/udev.postinst: Allow kvm to be an already present non-system group
    (LP: #1880541)
    File: debian/udev.postinst
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=8b5c31828d4323ddb719326b1316c179b7cdbdef
  * Cherry-pick reversion of detecting WSL by the presence of "microsoft" in osrelease
    File: debian/patches/Revert-detect-virt-also-detect-microsoft-as-WSL.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=e13f4452698b84ecb0ca6b7950b5f0c42c02c2cf

  [ Ryan Harper ]
  * Skip over by-uuid/by-label symlinks for bcache backing and cache devices
    (LP: #1861941)
    File: debian/patches/lp1861941-dont-generate-disk-byuuid-for-bcache-uuid.patch
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=4b9a0e888ca40eaed561ec48aece23218fab18ff

systemd (245.6-1) unstable; urgency=medium

  [ Michael Biebl ]
  * New upstream version 245.6
  * Rebase patches

  [ Balint Reczey ]
  * debian/tests/boot-and-services: Handle missing fstab (LP: #1877078)

 -- Balint Reczey <email address hidden> Thu, 11 Jun 2020 12:54:02 +0200

Is it possible to backport this ^ to Bionic?

Reopening against bcache-tools according to @raharper's comment about preferring a different fix.
https://bugs.launchpad.net/ubuntu/groovy/+source/bcache-tools/+bug/1861941/comments/28

@raharper I've forwarded the systemd fix for you with minimal tidying of the commit message https://github.com/systemd/systemd/pull/16317

On Tue, Jun 30, 2020 at 6:35 AM Balint Reczey <email address hidden>
wrote:

> @raharper I've forwarded the systemd fix for you with minimal tidying of
> the commit message https://github.com/systemd/systemd/pull/16317

Thanks!

>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1861941
>
> Title:
> bcache by-uuid links disappear after mounting bcache0
>
> To manage notifications about this bug go to:
>
> https://bugs.launchpad.net/ubuntu/+source/bcache-tools/+bug/1861941/+subscriptions
>

@rbalint @raharper a commit is in the ubuntu-focal git repo for this:
https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=3c228b1d9c5d83d7f0f7b7d8f2053a81de57f33c

I was prepping focal for upload, but on reading the last few comments, should I revert that commit from the focal branch and upload the next systemd focal version without it? Or should it be included in the focal upload? Do we need to hold off on updating focal until it's resolved upstream?

@ddstreet

I'm not sure where upstream is going just yet. For Ubuntu; I think

1) Adjusting the bcache-tools patch to use the full path to bcache-super-show should change;

2) If we fix (1) then I think we can drop the systemd patch from a bug fixing perspective; on the openSUSE image I did testing on; the skip of the dev/disk/by-uuid symlink for backing devices does not *break* the /dev/bcache/by-uuid ; it was missing due to the updated 69-bcache.rules not working.

3) IMO, blkid should not report FS_UUID for bcache backing/cache devices since it's not a Filesystem; we could patch that in blkid directly instead; alternatively since older blkid's may or maynot emit FS_UUID for bcache backing/caching devices the patch to systemd we're caring will avoid creating a useless dev/disk/by-uuid symlink to those devices.

Ok, I'm going to revert this from the queued-up focal commits before uploading, as IIUC the /dev/disk/by-uuid/ symlink shouldn't actually matter, since the symlink that should be used is /dev/bcache/by-uuid/. And based on comment 47 it sounds like that's accurate, and the real fix needs to go into bcache-utils to properly create the dev/bcache/by-uuid/ symlink.

This can be added back to the focal queue if I've misread the patch.

I've removed the linux projects from this bug at it's been determined to not be a kernel issue (see comment 47 for a good summary).

I'm having similar issues to this bug and those described by Dmitrii in https://bugs.launchpad.net/charm-ceph-osd/+bug/1883585 specifically comment #2 and the last comment.

It appears that if I run 'udevadm trigger --subsystem-match=block', I get my by-dname devices for bcaches, but if I run udevadm trigger w/out a subsystem match, something is triggering later than the block subsystem that is removing the links.

Here are the runs with --verbose to show what appears to be getting probed on each run:

https://pastebin.ubuntu.com/p/VPvSKRfGt4/

This is with 5.3.0-62 kernel on Bionic.

I also have core and canonical-livepatch snaps installed as did the environment where Dmitrii has run into this.

Relevant package revisions for comment #50:
bcache-tools 1.0.8-2build1
snapd 2.45.1+18.04.2
systemd 237-3ubuntu10.41
udev 237-3ubuntu10.41

and snaps:
Name Version Rev Tracking Publisher Notes
canonical-livepatch 9.5.5 95 latest/stable canonical✓ -
core 16-2.45.2 9665 latest/stable canonical✓ core

Just for the record, last week I tried mitigation/workaround as described in https://paste.ubuntu.com/p/JTB6vYbbkV/ on focal. It failed exactly the same as without those changes. I reported back to Rafael.

Thanks Marian... I'm now currently working on this issue and will provide feedback soon here...

Alright, summarizing this so far, we have the following fixes landed in Groovy:

udev (systemd pkg):

- 60-persistent-storage.rules: skip_bcache_fs_type:
  - /dev/disk/by-uuid/_ and /dev/disk/by-label/_ not created for bcache devs

Since it does not make sense to have the caching device symlink like if it was a regular filesystem disk to be used:

(k)rafaeldtinoco@bcache:~$ ls -lah /dev/disk/by-uuid
total 0
drwxr-xr-x 2 root root 100 Jul 21 18:49 .
drwxr-xr-x 7 root root 140 Jul 21 17:39 ..
lrwxrwxrwx 1 root root 10 Jul 21 17:39 022E-07E1 -> ../../vda1
lrwxrwxrwx 1 root root 13 Jul 21 18:49 50b98f0d-c0e0-49d2-9856-8ac6bc91e232 -> ../../bcache0
lrwxrwxrwx 1 root root 10 Jul 21 17:39 67993d4f-fa55-49b1-886e-c692f88393ba -> ../../vda2

- 69-bcache.rules:
  - if any other driver - than bcache - touches /dev/bcache[0-9]*
    bcache/by-uuid/_ (BCACHE_CACHED_UUID and LABEL) symlinks are
    recreated.

To address the udev event responsible for wiping /dev/bcache/by-uuid, like:

KERNEL[4175.725930] change /devices/virtual/block/bcache0 (block)
ACTION=change
DEVPATH=/devices/virtual/block/bcache0
SUBSYSTEM=block
SYNTH_UUID=0
DEVNAME=/dev/bcache0
DEVTYPE=disk
SEQNUM=6746
MAJOR=251
MINOR=0

UDEV [4175.758078] change /devices/virtual/block/bcache0 (block)
ACTION=change
DEVPATH=/devices/virtual/block/bcache0
SUBSYSTEM=block
SYNTH_UUID=0
DEVNAME=/dev/bcache0
DEVTYPE=disk
SEQNUM=6746
USEC_INITIALIZED=4151438558
ID_FS_UUID=50b98f0d-c0e0-49d2-9856-8ac6bc91e232
ID_FS_UUID_ENC=50b98f0d-c0e0-49d2-9856-8ac6bc91e232
ID_FS_VERSION=1.0
ID_FS_TYPE=ext4
ID_FS_USAGE=filesystem
MAJOR=251
MINOR=0
DEVLINKS=/dev/disk/by-uuid/50b98f0d-c0e0-49d2-9856-8ac6bc91e232
TAGS=:systemd:

@Rafael

For bcache-tools changes, bcache-export-cached-uuid needs the full path to bcache-super-show, as PATH is not exported when running from udev.

Status from old attempts to solve same nature issues:

----

Older (2018) merge request from @raharper:

https://github.com/koverstreet/bcache-tools/pull/1

addressing the fact that kernel uevents would not always emit
CACHED_UUID parameters, making udev to delete (whenever that happens)
/dev/bcache/{by-uuid,by-label} symlinks.

This last MR pointed to previous related bugs:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=890446
https://bugs.launchpad.net/curtin/+bug/1728742

And to an upstream kernel patch:

https://lore.kernel.org/patchwork/patch/921298/

to

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1729145

that wasn't accepted upstream.

Even not being accepted upstream, the SRU was attempted:

LP: #1729145

https://lists.ubuntu.com/archives/kernel-team/2017-December/088680.html
https://lists.ubuntu.com/archives/kernel-team/2017-December/088679.html

Both were NACKED.

Attempted again:

https://lists.ubuntu.com/archives/kernel-team/2017-December/088682.html
https://lists.ubuntu.com/archives/kernel-team/2017-December/088683.html

NACKED again.

And a v2 was sent:

https://lists.ubuntu.com/archives/kernel-team/2017-December/088751.html
https://lists.ubuntu.com/archives/kernel-team/2017-December/088750.html
https://lists.ubuntu.com/archives/kernel-team/2017-December/088749.html

and acked in January 2018 by Coling:

https://lists.ubuntu.com/archives/kernel-team/2018-January/089492.html

but not upstreamed.

BIONIC contains the fix:

commit ed9333e1b583
Author: Ryan Harper <email address hidden>
Date: Mon Dec 11 12:12:01 2017

    UBUNTU: SAUCE: (no-up) bcache: decouple emitting a cached_dev CHANGE uevent

    BugLink: http://bugs.launchpad.net/bugs/1729145

    - decouple emitting a cached_dev CHANGE uevent which includes dev.uuid
      and dev.label from bch_cached_dev_run() which only happens when a
      bcacheX device is bound to the actual backing block device (bcache0 -> vdb)

    - update bch_cached_dev_run() to invoke bch_cached_dev_emit_change() as
      needed; no functional code path changes here

    - Modify register_bcache to detect a re-registering of a bcache
      cached_dev, and in that case call bcache_cached_dev_emit_change() to

    Signed-off-by: Ryan Harper <email address hidden>
    Signed-off-by: Joseph Salisbury <email address hidden>
    Acked-by: Colin Ian King <email address hidden>
    Acked-by: Stefan Bader <email address hidden>
    Signed-off-by: Khalid Elmously <email address hidden>
    [ saf: fix incorrect indentation ]
    Signed-off-by: Seth Forshee <email address hidden>

FOCAL contains the fix:

commit 67553dcd7905
Author: Ryan Harper <email address hidden>
Date: Mon Dec 11 12:12:01 2017

    UBUNTU: SAUCE: (no-up) bcache: decouple emitting a cached_dev CHANGE uevent

GROOVY contains the fix:

commit 67553dcd7905
Author: Ryan Harper <email address hidden>
Date: Mon Dec 11 12:12:01 2017

    UBUNTU: SAUCE: (no-up) bcache: decouple emitting a cached_dev CHANGE uevent

----

So, the kernel patch wasn't accepted, nor bcache-tools patch by
@raharper, the bcache-export-cached.

----

New Upstream summary from ...

@raharper,

Great to know.. thanks! I'm in the "reading phase".. I think I managed to read all the history until now.. now I'm reading your analysis for this bug more carefully, knowing the history...

@raharper,

quick question... I see that you are trying to come up with something nor relying in the kernel fix (to throw the variables again after pivot root, etc..)

https://github.com/g2p/bcache-tools/pull/29/#issue-169277283

why are you merging in

"github.com/g2p" ? and not in "github.com/koverstreet" ?

which one is the "current official ?"

And now, after reading it all, I do follow your comment:

https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1861941/comments/47

with the proposed changed. As you're probably not fully dedicated to this, and I can be, I'll give your suggestions a try, possibly among with other things, and post findings here with how I think we should move (considering upstream hasn't been very eager to fix this situation).

@Rafael

It's in both places:

https://github.com/koverstreet/bcache-tools/pull/1

I can update the PR there as well; though I don't think upstream cares as Kent's working on bcachefs instead of bcache AFAICT.

> I see that you are trying to come up with something nor relying in the kernel fix (to throw the variables again after pivot root, etc..)

I think after the systemd discussion upstream:

https://github.com/systemd/systemd/pull/16317

The kernel uevent for CACHED_UUID isn't the right way to go long term; all of the information we want is present in the superblock of the devices as they are probed; If we have a 69-bcache.rules with a helper that uses bcache-super-show; we could drop the Ubuntu sauce patch we have; it's just noise (and now it's somewhat unreliable, this bug shows that).

Alright. +1 on you last comment. Thanks for the feedback! I'll work on that!

TODOs (currently being started):

1 - adjust/add bcache-tools patch to use bcache-super-show (full path)
    (bcache-tools - bionic | focal | groovy)

1.1 - Ubuntu kernel patches for CACHED UUID resubmission could be removed
      (linux - bionic | focal | groovy) IF (1) is done and SRU'ed

1.2 - Work with upstream so they merge @raharper's patch

2 - drop systemd patch for /dev/disk/by-uuid | /dev/bcache/by-uuid ??
    (systemd - bionic | focal | groovy)
    - I prefer to keep this patch and open upstream discussion for (3)

3 - blkid should not report FS_UUID for bcache backing devices (not a filesytem)
   (blkid - bionic | focal | groovy)
   - would avoid useless dev/disk/by-uuid, just like the systemd patch
   - blkid would have to be SRUed to all or systemd patch kept
   - will drive upstream discussion

4 - provide a temporary PPA with all fixes before SRU happens (MAAS tests)

Upstream patch submitted (amended from Ryan's previous patch) with a minor PATH fix:

- https://github.com/koverstreet/bcache-tools/pull/6/

Debian patch submitted (removed previous patch and proposed new one based on what was proposed to upstream):

https://salsa.debian.org/debian/bcache-tools/-/merge_requests/2

Tests are good, I could run testcase multiple times and wasn't able to reproduce the original issue.

I'm setting this bug to In Progress so I can better track reverted fix and the new one in both packages for all Ubuntu versions.

Groovy Merge Request:

BUG: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1861941
PPA: https://launchpad.net/~rafaeldtinoco/+archive/ubuntu/lp1861941
GROOVY: https://tinyurl.com/yxonp5hz

Focal Merge Request:

FOCAL: https://tinyurl.com/y3tbd3un

PPA containing (for now) fixes for Groovy and Focal:

https://launchpad.net/~rafaeldtinoco/+archive/ubuntu/lp1861941

will contain fix for Bionic as well.

Testcase: https://paste.ubuntu.com/p/37KGy2Smnp/

Testcase passes for both versions when having PPA pkgs installed.

About blkid fix: Ryan has explained carefully how to obtain FS UUID from the backing device at:

https://github.com/systemd/systemd/pull/16317#issuecomment-655574875

and about bcache-export-cached wrapper he created for simulating CACHED_UUID and CACHED_LABEL (at 69-bcache.rules) and to create /dev/bcache/{by-uuid,by-label}/{CACHED_UUID,CACHED_LABEL} symlinks.

He explained that having /dev/bcache/{by-uuid,by-label} is mandatory to maintain same names for multiple bcache'd environments over reboots.

Currently, with the bcache-export-cached fix applied, it is *not* mandatory to have systemd-udev fix to correctly create bcache symlinks (or not) at /dev/disk/{by-uuid,by-label}: libblkid could have more information regarding bcache backing devices, but judging by upstream developers disposition they don't seem too worried about having that feature since ID_FS_TYPE=bcache is already provided. (So, having /dev/disk/by-uuid/XXX pointing to bcache backing devices would continue happening...)

Summary:

Having the bcache-tools fix is enough for this bug (LP: #1861941) and there could be changes to systemd-udev (I doubt judging by discussions at https://github.com/systemd/systemd/pull/16317), or libblkid, that could provide more information regarding bcache device types. If those changes happen, we would need to revisit this bug.

IMHO,

I like the "skip_bcache_fs_type" approach done by Ryan to systemd-udev. I'll maintain that as a delta to udev until https://github.com/systemd/systemd/pull/16317 is fixed (if ever).

So, for now, PPA contains fix for bcache-tools for Groovy and Focal (and will for Bionic) and that is enough to make /dev/bcache/XXX to work and be stable after reboots. I'll keep the udev fix for Groovy and SRU that fix for Focal and Bionic and upload to the PPA as well.

After all that is done, I'll tell kernel team to revert the bcache patch on Bionic, Focal and Groovy kernels since it is not needed and will reduce their delta and reduce chances of regressions over time.

systemd (udev) is fixed for Groovy:

systemd (245.6-1ubuntu1) groovy; urgency=medium

[ Ryan Harper ]
* Skip over by-uuid/by-label symlinks for bcache backing and cache devices (LP: #1861941)
  File: debian/patches/lp1861941-dont-generate-disk-byuuid-for-bcache-uuid.patch
  https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=4b9a0e888ca40eae

BUG: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1861941
TESTCASE: https://paste.ubuntu.com/p/37KGy2Smnp/
PPA: https://launchpad.net/~rafaeldtinoco/+archive/ubuntu/lp1861941

BCACHE-TOOLS:

GROOVY: https://tinyurl.com/yxonp5hz
FOCAL: https://tinyurl.com/y3tbd3un

SYSTEMD-UDEV:

GROOVY: patch rewording only (https://tinyurl.com/y3ln9ttu)
FOCAL: https://tinyurl.com/yxkz6tzl

Just spoke to @ddstreet about the SYSTEMD-UDEV merges. We are discussing if we should keep the udev delta (from upstream) or not, considering that the issue will be already fixed by bcache-tools and udev change would only serve not to have backing devices at /dev/disk/by-uuid and /dev/disk/by-label (good for end user, but to add burden to ubuntu-devel).

I tested ppa:rafaeldtinoco/lp1861941 several times today on a focal setup where it failed 100%.
Glad to report it passes without issue every single time now.
Thank you

SUMMARY:

BUG: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1861941
TESTCASE: https://paste.ubuntu.com/p/37KGy2Smnp/
PPA: https://launchpad.net/~rafaeldtinoco/+archive/ubuntu/lp1861941

BCACHE-TOOLS:

GROOVY: https://tinyurl.com/yxonp5hz (needs-review for an upload)
FOCAL: https://tinyurl.com/y3tbd3un (needs-review for SRU)
BIONIC: Will be done after Groovy and Focal are reviewed.

SYSTEMD-UDEV:

GROOVY: https://tinyurl.com/y4w22s57 (merged patch rewording, will wait next upload)
FOCAL: https://tinyurl.com/y2uqbyll (merged patch, will ask for an upload - SRU)
BIONIC: Will be done after Groovy and Focal are reviewed.

KERNEL:

After Groovy, Focal and Bionic are fixed (bcache-tools and systemd-udev), I'll send a SRU request for 3 kernels to remove bcache patches created for old bcache/udev rules bug (not needed after this bcache-tools fix).

This bug was fixed in the package bcache-tools - 1.0.8-4ubuntu1

---------------
bcache-tools (1.0.8-4ubuntu1) groovy; urgency=medium

  [ Ryan Harper ]
  * Revert "Keep symlinks when udev changes /dev/bcacheN"
  * Add helper script to read bcache devs superblock (LP: #1861941)

 -- Rafael David Tinoco <email address hidden> Thu, 23 Jul 2020 00:14:31 +0000

Hello Ryan, or anyone else affected,

Accepted bcache-tools into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/bcache-tools/1.0.8-3ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

VERIFICATION FOR FOCAL:

(k)rafaeldtinoco@bcachefocal:~$ apt-cache policy bcache-tools
bcache-tools:
  Installed: (none)
  Candidate: 1.0.8-3ubuntu0.1
  Version table:
     1.0.8-3ubuntu0.1 500
        500 http://br.archive.ubuntu.com/ubuntu focal-proposed/main amd64 Packages
     1.0.8-3 500
        500 http://br.archive.ubuntu.com/ubuntu focal/main amd64 Packages

(k)rafaeldtinoco@bcachefocal:~$ apt-get install bcache-tools

(k)rafaeldtinoco@bcachefocal:~/scripts$ sudo ./test-bcache.sh
/mnt is not a mountpoint
/dev/bcache0: 2 bytes were erased at offset 0x00000438 (ext4): 53 ef
writing 1 to /sys/class/block/bcache0/bcache/stop
writing 1 to /sys/fs/bcache/6fcbc055-be04-4527-9c73-788c2419b292/stop
/dev/vdc: 16 bytes were erased at offset 0x00001018 (bcache): c6 85 73 f6 4e 1a 45 ca 82 65 f5 7f 48 ba 6d 81
10+0 records in
10+0 records out
10485760 bytes (10 MB, 10 MiB) copied, 0.0204717 s, 512 MB/s
/dev/vdb: 16 bytes were erased at offset 0x00001018 (bcache): c6 85 73 f6 4e 1a 45 ca 82 65 f5 7f 48 ba 6d 81
10+0 records in
10+0 records out
10485760 bytes (10 MB, 10 MiB) copied, 0.0092515 s, 1.1 GB/s
UUID: 73cc1c6f-ded6-4874-b76a-465408cfbd19
Set UUID: a2400c1f-e83b-4086-a58d-0b2ebba97a9c
version: 0
nbuckets: 2048
block_size: 1
bucket_size: 1024
nr_in_set: 1
nr_this_dev: 0
first_bucket: 1
UUID: 5f51a399-d65d-49ec-af3b-6b2b28c39605
Set UUID: a2400c1f-e83b-4086-a58d-0b2ebba97a9c
version: 1
block_size: 1
data_offset: 16
/dev/vdc
tee: /sys/fs/bcache/register: Invalid argument
/dev/vdb
tee: /sys/fs/bcache/register: Invalid argument
total 0
drwxr-xr-x 2 root root 60 Aug 5 21:12 .
drwxr-xr-x 3 root root 60 Aug 5 21:12 ..
lrwxrwxrwx 1 root root 13 Aug 5 21:12 5f51a399-d65d-49ec-af3b-6b2b28c39605 -> ../../bcache0
Creating filesystems on bcache0
mke2fs 1.45.5 (07-Jan-2020)
Discarding device blocks: done
Creating filesystem with 2621438 4k blocks and 655360 inodes
Filesystem UUID: 1b609131-52ca-43f7-96b2-4dc931f2553f
Superblock backups stored on blocks:
 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632

Allocating group tables: done
Writing inode tables: done
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done

mounting bcache0 to /mnt
total 0
drwxr-xr-x 2 root root 60 Aug 5 21:12 .
drwxr-xr-x 3 root root 60 Aug 5 21:12 ..
lrwxrwxrwx 1 root root 13 Aug 5 21:12 5f51a399-d65d-49ec-af3b-6b2b28c39605 -> ../../bcache0
Everything OK

All good!

CURRENT STATUS:

SUMMARY:

BUG: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1861941
TESTCASE: https://paste.ubuntu.com/p/37KGy2Smnp/
PPA: https://launchpad.net/~rafaeldtinoco/+archive/ubuntu/lp1861941

BCACHE-TOOLS:

GROOVY: https://tinyurl.com/yxonp5hz (merged)
FOCAL: https://tinyurl.com/y3tbd3un (committed + verified)
BIONIC: https://tinyurl.com/y58fm4l5 (being reviewed)

SYSTEMD-UDEV:

GROOVY: https://tinyurl.com/y4w22s57 (merged patch rewording, will wait next upload)
FOCAL: https://tinyurl.com/y2uqbyll (merged patch, will ask for an upload - SRU)
BIONIC: BEING WORKED

KERNEL:

Decided that won't be changed as the udev bcache changes won't cause any harm. Will ask for the patch to be dropped from groovy's kernel since there isn't any need to keep it.

SRU TEAM:

Summary is the one above. The reason why Bionic SRU makes sense was discussed with @paelzer during merge review:

https://code.launchpad.net/~rafaeldtinoco/ubuntu/+source/bcache-tools/+git/bcache-tools/+merge/388774/comments/1022492

Thanks for reviewing the SRU.

@rafaeldtinoco

Deescalated to high.

This bug was fixed in the package bcache-tools - 1.0.8-3ubuntu0.1

---------------
bcache-tools (1.0.8-3ubuntu0.1) focal; urgency=medium

  [ Ryan Harper ]
  * Add helper script to read bcache devs superblock (LP: #1861941)

 -- Rafael David Tinoco <email address hidden> Thu, 23 Jul 2020 00:33:50 +0000

The verification of the Stable Release Update for bcache-tools has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Hello Ryan, or anyone else affected,

Accepted bcache-tools into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/bcache-tools/1.0.8-2ubuntu0.18.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Bionic verification:

$ apt changelog bcache-tools | head -10

bcache-tools (1.0.8-2ubuntu0.18.04.1) bionic; urgency=medium

  [ Ryan Harper ]
  * Add helper script to read bcache devs superblock (LP: #1861941)

 -- Rafael David Tinoco <email address hidden> Wed, 05 Aug 2020 17:44:05 -0300

bcache-tools (1.0.8-2build1) artful; urgency=medium

----

$ dpkg-query -f '${Package}###${Version}\n' -W 'bcache-tools'
bcache-tools###1.0.8-2ubuntu0.18.04.1

----
$ sudo ./test-bcache.sh
/mnt is a mountpoint
/dev/bcache0: 2 bytes were erased at offset 0x00000438 (ext4): 53 ef
writing 1 to /sys/class/block/bcache0/bcache/stop
writing 1 to /sys/fs/bcache/60412675-9103-47b0-9794-91fcfb12d50d/stop
/dev/vdc: 16 bytes were erased at offset 0x00001018 (bcache): c6 85 73 f6 4e 1a 45 ca 82 65 f5 7f 48 ba 6d 81
10+0 records in
10+0 records out
10485760 bytes (10 MB, 10 MiB) copied, 0.0173157 s, 606 MB/s
/dev/vdb: 16 bytes were erased at offset 0x00001018 (bcache): c6 85 73 f6 4e 1a 45 ca 82 65 f5 7f 48 ba 6d 81
10+0 records in
10+0 records out
10485760 bytes (10 MB, 10 MiB) copied, 0.00634546 s, 1.7 GB/s
UUID: c3758ea3-0e2c-42e3-bb77-da27292d2476
Set UUID: b2b21f55-3599-461a-b162-8377d85fbc34
version: 0
nbuckets: 2048
block_size: 1
bucket_size: 1024
nr_in_set: 1
nr_this_dev: 0
first_bucket: 1
UUID: f4001b7e-386e-4ca3-a5e0-d53f100d156f
Set UUID: b2b21f55-3599-461a-b162-8377d85fbc34
version: 1
block_size: 1
data_offset: 16
/dev/vdc
tee: /sys/fs/bcache/register: Invalid argument
/dev/vdb
tee: /sys/fs/bcache/register: Invalid argument
total 0
drwxr-xr-x 2 root root 60 Aug 21 14:03 .
drwxr-xr-x 3 root root 60 Aug 21 14:03 ..
lrwxrwxrwx 1 root root 13 Aug 21 14:03 f4001b7e-386e-4ca3-a5e0-d53f100d156f -> ../../bcache0
Creating filesystems on bcache0
mke2fs 1.44.1 (24-Mar-2018)
Discarding device blocks: done
Creating filesystem with 262142 4k blocks and 65536 inodes
Filesystem UUID: 4ade91a9-6176-4a93-a34f-158fa08d0091
Superblock backups stored on blocks:
 32768, 98304, 163840, 229376

Allocating group tables: done
Writing inode tables: done
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: done

mounting bcache0 to /mnt
total 0
drwxr-xr-x 2 root root 60 Aug 21 14:03 .
drwxr-xr-x 3 root root 60 Aug 21 14:03 ..
lrwxrwxrwx 1 root root 13 Aug 21 14:03 f4001b7e-386e-4ca3-a5e0-d53f100d156f -> ../../bcache0
Everything OK
----

This bug was fixed in the package bcache-tools - 1.0.8-2ubuntu0.18.04.1

---------------
bcache-tools (1.0.8-2ubuntu0.18.04.1) bionic; urgency=medium

  [ Ryan Harper ]
  * Add helper script to read bcache devs superblock (LP: #1861941)

 -- Rafael David Tinoco <email address hidden> Wed, 05 Aug 2020 17:44:05 -0300

This bug was fixed in the package linux - 5.8.0-18.19

---------------
linux (5.8.0-18.19) groovy; urgency=medium

  * groovy/linux: 5.8.0-18.19 -proposed tracker (LP: #1893047)

  * Packaging resync (LP: #1786013)
    - update dkms package versions

  * Groovy update: v5.8.4 upstream stable release (LP: #1893048)
    - drm/vgem: Replace opencoded version of drm_gem_dumb_map_offset()
    - drm/panel-simple: Fix inverted V/H SYNC for Frida FRD350H54004 panel
    - drm/ast: Remove unused code paths for AST 1180
    - drm/ast: Initialize DRAM type before posting GPU
    - khugepaged: adjust VM_BUG_ON_MM() in __khugepaged_enter()
    - ALSA: hda: avoid reset of sdo_limit
    - ALSA: hda/realtek: Add quirk for Samsung Galaxy Flex Book
    - ALSA: hda/realtek: Add quirk for Samsung Galaxy Book Ion
    - can: j1939: transport: j1939_session_tx_dat(): fix use-after-free read in
      j1939_tp_txtimer()
    - can: j1939: socket: j1939_sk_bind(): make sure ml_priv is allocated
    - spi: Prevent adding devices below an unregistering controller
    - io_uring: find and cancel head link async work on files exit
    - mm/vunmap: add cond_resched() in vunmap_pmd_range
    - romfs: fix uninitialized memory leak in romfs_dev_read()
    - kernel/relay.c: fix memleak on destroy relay channel
    - uprobes: __replace_page() avoid BUG in munlock_vma_page()
    - squashfs: avoid bio_alloc() failure with 1Mbyte blocks
    - mm: include CMA pages in lowmem_reserve at boot
    - mm, page_alloc: fix core hung in free_pcppages_bulk()
    - ASoC: amd: renoir: restore two more registers during resume
    - RDMA/hfi1: Correct an interlock issue for TID RDMA WRITE request
    - opp: Enable resources again if they were disabled earlier
    - opp: Put opp table in dev_pm_opp_set_rate() for empty tables
    - opp: Put opp table in dev_pm_opp_set_rate() if _set_opp_bw() fails
    - ext4: do not block RWF_NOWAIT dio write on unallocated space
    - ext4: fix checking of directory entry validity for inline directories
    - jbd2: add the missing unlock_buffer() in the error path of
      jbd2_write_superblock()
    - scsi: zfcp: Fix use-after-free in request timeout handlers
    - selftests: kvm: Use a shorter encoding to clear RAX
    - s390/pci: fix zpci_bus_link_virtfn()
    - s390/pci: re-introduce zpci_remove_device()
    - s390/pci: fix PF/VF linking on hot plug
    - s390/pci: ignore stale configuration request event
    - mm/memory.c: skip spurious TLB flush for retried page fault
    - drm: amdgpu: Use the correct size when allocating memory
    - drm/amdgpu/display: use GFP_ATOMIC in dcn20_validate_bandwidth_internal
    - drm/amd/display: Fix incorrect backlight register offset for DCN
    - drm/amd/display: Fix EDID parsing after resume from suspend
    - drm/amd/display: Blank stream before destroying HDCP session
    - drm/amd/display: Fix DFPstate hang due to view port changed
    - drm/amd/display: fix pow() crashing when given base 0
    - drm/i915/pmu: Prefer drm_WARN_ON over WARN_ON
    - drm/i915: Provide the perf pmu.module
    - scsi: ufs: Add DELAY_BEFORE_LPM quirk for Micron devices
    - scsi: target: tcmu: Fix crash in tcmu_flush_dcache_range on ARM
  ...

Hello Ryan, or anyone else affected,

Accepted systemd into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/systemd/245.4-4ubuntu3.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

All autopkgtests for the newly accepted systemd (245.4-4ubuntu3.3) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

linux-hwe-5.8/5.8.0-25.26~20.04.1 (armhf)
python-dbusmock/0.19-1 (armhf)
lxc/1:4.0.2-0ubuntu1 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#systemd

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

root@lp1861941-f:~# dpkg -l systemd|grep systemd
ii systemd 245.4-4ubuntu3.2 amd64 system and service manager
root@lp1861941-f:~# dpkg -l udev|grep udev
ii udev 245.4-4ubuntu3.2 amd64 /dev/ and hotplug management daemon
root@lp1861941-f:~# ls -l /dev/disk/by-uuid/$(ls /dev/bcache/by-uuid/)
lrwxrwxrwx 1 root root 9 Nov 2 17:16 /dev/disk/by-uuid/651c5532-0a15-4d56-acdf-9c8b2bb559f1 -> ../../vdc

root@lp1861941-f:~# dpkg -l systemd|grep systemd
ii systemd 245.4-4ubuntu3.3 amd64 system and service manager
root@lp1861941-f:~# dpkg -l udev|grep udev
ii udev 245.4-4ubuntu3.3 amd64 /dev/ and hotplug management daemon
root@lp1861941-f:~# ls -l /dev/disk/by-uuid/$(ls /dev/bcache/by-uuid/)
ls: cannot access '/dev/disk/by-uuid/651c5532-0a15-4d56-acdf-9c8b2bb559f1': No such file or directory

This bug was fixed in the package systemd - 245.4-4ubuntu3.3

---------------
systemd (245.4-4ubuntu3.3) focal; urgency=medium

  [ Rafael David Tinoco ]
  * d/p/lp1861941-dont-generate-disk-byuuid-for-bcache-uuid.patch:
    Reworded and reintroduced patch to fully explain delta is NOT a fix to
    LP: #1861941 if the bcache-tools patch exists, but should be kept anyway
    as the change makes sense for a better experience to end user.
    (LP: #1861941)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=f8f64b3b58a04a83b1c426818b9affc41e0bff6c

  [ Dan Streetman ]
  * d/p/lp1882596-man-fix-some-manvolnum.patch:
    - fix some man section references (LP: #1882596)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=3959ec95eff78d38ec4409807f151572afe83fe9
  * d/p/lp1895418-correct-resolved-conf-cache-default.patch:
    - fix resolved.conf default Cache= value (LP: #1895418)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=ebe274a2b01658ee39b372d7033c35209510b028
  * d/p/lp1897744-resolve-enable-RES_TRUSTAD-towards-the-127.0.0.53-st.patch:
    - add resolv.conf 'trust-ad' option (LP: #1897744)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=f6acc8c620b80adab7b048352d85e722b5ba8214
  * d/t/*:
    - Update tests to fix false negatives (LP: #1892358)
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=cee6c31a6caec7888270c9fa8757105ab950ed0c
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=a1c1a2bb0ff27faf84fe94583631dfd0f1f4ed8f
    https://git.launchpad.net/~ubuntu-core-dev/ubuntu/+source/systemd/commit/?id=9417ce996766c133c2a33d4102ce1494f3166774

 -- Dan Streetman <email address hidden> Thu, 08 Oct 2020 16:14:56 -0400