/dev/bcache/by-uuid links not created after reboot

This change was made by a bot.

To recreate:

Add a second disk (any size will do) and split the disk into 4 equal partitions. Then

make-bcache -C /dev/vdb1
make-bcache -B /dev/vdb2
make-bcache -B /dev/vdb3
make-bcache -B /dev/vdb4

Note, you can see the proper symlinks created at this time:

ls -al /dev/bcache/by-uuid/

Reboot

ls -al /dev/bcache/by-uuid does not exist only /dev/bcache{0,1,2}

Did this issue start happening after an update/upgrade? Was there a prior kernel version where you were not having this particular problem?

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v4.14 kernel[0].

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.14-rc7

This fails on Xenial -> Artful. I've not yet tested Trusty, but I suspect it's just a latent bug in the bcache driver.

I'll grab the upstream kernel and provide results.

Mainline kernel does not help; it fails the same way and also has a bug that's fixed in the Ubuntu kernel:

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1667078

Here's the details from the mainline test:

ubuntu@a-unsolar-melonie:~$ dpkg --list | grep linux-image
ii linux-image-4.13.0-16-generic 4.13.0-16.19 amd64 Linux kernel image for version 4.13.0 on 64 bit x86 SMP
ii linux-image-4.14.0-041400rc7-generic 4.14.0-041400rc7.201710292231 amd64 Linux kernel image for version 4.14.0 on 64 bit x86 SMP
ii linux-image-virtual 4.13.0.16.17 amd64 Virtual Linux kernel image
ubuntu@a-unsolar-melonie:~$ uname -r
4.14.0-041400rc7-generic
ubuntu@a-unsolar-melonie:~$ uname -a
Linux a-unsolar-melonie 4.14.0-041400rc7-generic #201710292231 SMP Sun Oct 29 22:32:07 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
ubuntu@a-unsolar-melonie:~$ ls -al /dev/bcache*
brw-rw---- 1 root disk 251, 0 Nov 1 14:45 /dev/bcache0
brw-rw---- 1 root disk 251, 16 Nov 1 14:45 /dev/bcache16
brw-rw---- 1 root disk 251, 32 Nov 1 14:45 /dev/bcache32
ubuntu@a-unsolar-melonie:~$ dmesg | grep bcache
[ 3.099823] bcache: register_bdev() registered backing device vdb4
[ 3.159769] bcache: bch_journal_replay() journal replay done, 0 keys in 1 entries, seq 2
[ 3.170818] bcache: register_cache() registered cache device vdb1
[ 3.173682] bcache: register_bdev() registered backing device vdb3
[ 3.183034] bcache: register_bdev() registered backing device vdb2
[ 8.271351] bcache: register_bcache() error opening /dev/vdb1: device already registered
[ 8.296933] bcache: register_bcache() error opening /dev/vdb2: device already registered
[ 8.345949] bcache: register_bcache() error opening /dev/vdb3: device already registered
[ 8.353927] bcache: register_bcache() error opening /dev/vdb4: device already registered

Just to clarify on UUIDs - there are 3 types of UUIDs in question:

1. cache set UUIDs (upstream docs mention that multiple cache devices per cache set are not supported but a configuration with M:N as in <cache_devices>:<backing_devices> has been taken into account)
2. cache device superblock UUIDs;
3. backing device superblock UUIDs.

We are interested in uevents containing backing device superblock UUIDs to reliably create by-uuid symlinks and, ideally, have persistent /dev/bcache<n> names based on those UUIDs (although the latter is out of scope for this particular bug).

The following code in bcache-tools provides more clarity on how suberblock UUIDs are generated in userspace:

https://github.com/g2p/bcache-tools/blob/v1.0.8/make-bcache.c#L448-L458 loops over cache devices and backing devices calling write_sb for each device
https://github.com/g2p/bcache-tools/blob/v1.0.8/make-bcache.c#L170-L174 (write_sb which takes device type into account)
https://github.com/g2p/bcache-tools/blob/v1.0.8/make-bcache.c#L211-L221 - selects a type of suberblock which is either cache or backing device (cdev or bdev), writes bcache magic, generates a ***suberblock UUID***, writes a passed cache set UUID into in-memory suberblock representation along with block size and bucket size
https://github.com/g2p/bcache-tools/blob/v1.0.8/make-bcache.c#L284-L288 (pwrites the in-memory representation to a given block device)

There is a scenario where a real rootfs is located on a bcache device, however, for that we need to register a bcache device at the initrd stage which already happens now. Then we'd locate a file system on it and do pivot_root and so on.

The bcache<i> naming, I believe, is not guaranteed at this point unless we have a rule that says so.

Side-tracking to our field use-cases, we need persistence in /dev/bcache<i> names based on superblock UUIDs. So, I expect /dev/bcache/<i> names to be persisted by UUID on first discovery (which corresponds to MAAS deploy stage, not commissioning as in case of disk serial numbers).

However, we also expect bcache<i> names to match names in MAAS which may not happen in this scenario because <backing-dev-name> : bcache<i> mapping is not enforced.

Going back to https://bugs.launchpad.net/curtin/+bug/1728742, I think we can break it down into two problems:

1. bcache device numbers are not static across reboots and we need a static mapping of superblock UUID to bcache<i> for a given device. This requires CACHED_UUID to be present in uevent environment which is only possible during a successful registration where this code path is triggered. As a result of rootfs on bcache requirement, this makes sense to do at the initrd stage before we have to do pivot_root to the real rootfs.

Doing something like that when systemd is running post pivot_root and /dev devtmpfs transfer to the real rootfs doesn't sound right to me as we have this problem with double registration. In summary, I think /dev/bcache/by-uuid/ symlinks for bcache devices that exist on initial boot should be created via udev rules in initrd.

This is what this bug is about.

2. bcache device names may not match the ones in MAAS. This has implications for our use of Juju Storage functionality when we need device special files with static names without file systems or partition tables present. After commissioning in MAAS there's already metadata present about a given machine - disk serial numbers are gathered (if present, this is not guaranteed and block driver-specific AFAIK but a sane assumption to make) and device names that were assigned during ephemeral image boot are presented and stored in a database with associated serial numbers available for querying to set up dname symlinks on deployment.

In order to make <backing-dev-name> : bcache<i> mapping static we need to essentially have a mapping of disk serial numbers to bcache superblock UUIDs which are in turn mapped to bcache<i> names.

I would say that https://bugs.launchpad.net/curtin/+bug/1728742 is about p.2.

====

The rationale for p. 1 is that the init script sets up devtmpfs initially which then gets moved over to the real rootfs (init-bottom script) before pivot_root is performed. systemd then runs its mount point set up code which checks if a given entry in its hard-coded table of mount points is already a mount point and skips its setup if this is the case. So anything set up during initrd stage will stay there after systemd runs as devtmpfs is moved and reused.

https://git.launchpad.net/~usd-import-team/ubuntu/+source/systemd/tree/src/core/mount-setup.c?h=applied/ubuntu/xenial-updates#...

I reviewed this once again:

1. persistent /dev/bcache<i> - not possible or needed due to how kernel enumeration works
2. our userspace handles /dev/by-dname/<symlink> properly so we just need CACHED_UUID given to us during coldplug in some way to properly create /dev/bcache/by-uuid/<link> symlinks and consequently by-dname symlinks.

3. bcache, like almost all block device drivers, gets "struct device_type disk_type" by using alloc_disk from genhd.c which gives us standard contents for the uevent file. Partitions are a notable exception and use "struct device_type part_type" which has a .uevent field (http://elixir.free-electrons.com/linux/v4.14.2/source/block/partition-generic.c#L223) that allows customization of uevent file contents in a bus-, class- and type-specific manner (http://elixir.free-electrons.com/linux/v4.14.2/source/drivers/base/core.c#L903).

cat /sys/class/block/nvme0n1/uevent
MAJOR=259
MINOR=0
DEVNAME=nvme0n1
DEVTYPE=disk

as opposed to the following with partitions

cat /sys/class/block/nvme0n1p1/uevent
MAJOR=259
MINOR=1
DEVNAME=nvme0n1p1
DEVTYPE=partition
PARTN=1
PARTNAME=Basic data partition

While we could have a new block device type of "bcache" with a custom uevent content that had CACHED_UUID of a superblock this would be a userspace breakage which is hardly upstreamable.

So, the kernel fix could look like this:

* instead of just failing with "device already registered"
* the kernel would emit another uevent with CACHED_UUID which would be processed another time to set up a symlink

It seems like this approach with "synthetic uevents" may be used for other purposes as well with changes that landed in 4.13

http://elixir.free-electrons.com/linux/v4.14.2/source/Documentation/ABI/testing/sysfs-uevent
"Enable passing additional variables for synthetic uevents that are generated by writing /sys/.../uevent file

Recognized extended format is ACTION [UUID [KEY=VALUE ...]

Users: udev, userspace tools generating synthetic uevents"

We do run udevd at the initramfs stage in init-top and stop it in init-bottom:

➜ scripts git:(ubuntu/xenial-updates) ✗ grep -RiP udev
init-bottom/udev:# Stop udevd, we'll miss a few events while we run init, but we catch up
init-bottom/udev:udevadm control --exit

init-top/udev:SYSTEMD_LOG_LEVEL=$log_level /lib/systemd/systemd-udevd --daemon --resolve-names=never
init-top/udev:udevadm trigger --type=subsystems --action=add
init-top/udev:udevadm trigger --type=devices --action=add
init-top/udev:udevadm settle || true

And, from what I can see after adding some logging to the udev rule and updating initramfs, KOBJ_CHANGE events generated by the bcache driver are processed by 69-bcache.rules:

http://paste.ubuntu.com/26080228/ (rule)

https://paste.ubuntu.com/26080236/ (dmesg)
[ 1.789254] AFTER DRIVER BCACHE: devpath: /devices/virtual/block/bcache0 UUID: dd563676-f731-420c-a65d-93992321eb14, LABEL:
[ 1.790563] AFTER DRIVER BCACHE: devpath: /devices/virtual/block/bcache1 UUID: 3aec199f-d1d7-452f-a56d-9c946c72cae1, LABEL:
[ 1.794265] PROBING BCACHE /devices/virtual/block/bcache3
[ 1.795957] BEFORE DRIVER BCACHE /devices/virtual/block/bcache3
[ 1.798215] AFTER DRIVER BCACHE: devpath: /devices/virtual/block/bcache3 UUID: 2c8f2de3-a1a5-4f53-ad00-09294eb19512, LABEL:
[ 1.801311] PROBING BCACHE /devices/virtual/block/bcache2
[ 1.803180] BEFORE DRIVER BCACHE /devices/virtual/block/bcache2
[ 1.804731] AFTER DRIVER BCACHE: devpath: /devices/virtual/block/bcache2 UUID: dcdb4830-78f2-494a-905b-efa549b484fc, LABEL:

So the UUID is present in the environment of the CHANGE uevent but the symlink is not present when I ssh into the system after the boot process.

I wonder why that happens as devtmpfs mounted at /dev is moved to the new rootfs before init script exits and systemd is started - so if it's created on that devtmpfs, it should be there at the end:

https://git.launchpad.net/~usd-import-team/ubuntu/+source/systemd/tree/debian/extra/initramfs-tools/scripts/init-bottom/udev?h=applied/ubuntu/xenial-updates

Let's assume there is a systemd fault here too, thus marking systemd as affected. I will try to play around with this, but may get to this by December 14th the earliest.

After booting with break=bottom kernel parameter I can see that symlinks are in fact present at the initramfs stage.

They are gone afterwards post pivot_root & systemd start (the bcache dir isn't even present while it was there)

ubuntu@maas-xenial4:~$ ls /dev/disk/
by-dname by-id by-label by-path by-uuid

Just in case also checked with this rule that creates links at /dev /dev/disk/by-uuid /dev/bcache/by-uuid https://paste.ubuntu.com/26081756/

They are all there originally (see the screenshot).

But not there afterwards. So it's not just a problem with a different path.

https://paste.ubuntu.com/26081775/

ubuntu@maas-xenial4:~$ tree /dev/ | grep bcache
├── bcache0
├── bcache1
├── bcache2
├── bcache3
│   ├── 251:0 -> ../bcache0
│   ├── 251:1 -> ../bcache1
│   ├── 251:2 -> ../bcache2
│   ├── 251:3 -> ../bcache3
│   │   ├── bcache0 -> ../../bcache0
│   │   ├── bcache1 -> ../../bcache1
│   │   ├── bcache2 -> ../../bcache2
│   │   ├── bcache3 -> ../../bcache3

So, /dev/bcache/by-uuid is not getting created.

That's the same kernel bug I filed.

And, if they were, I think they'd get moved properly.

init-bottom/udev script does the following:

# Stop udevd, we'll miss a few events while we run init, but we catch up
udevadm control --exit

# move the /dev tmpfs to the rootfs
mount -n -o move /dev ${rootmnt}/dev

That "move" operation migrates the entire mounted directory to a new
location
so, if /dev/bcache/by-uuid existed, it would be present when systemd init
runs.

AFAICT, there's not fix here without the kernel side.

On Thu, Nov 30, 2017 at 10:09 AM, Dmitrii Shcherbakov <
<email address hidden>> wrote:

> Just in case also checked with this rule that creates links at /dev
> /dev/disk/by-uuid /dev/bcache/by-uuid https://paste.ubuntu.com/26081756/
>
> They are all there originally (see the screenshot).
>
> But not there afterwards. So it's not just a problem with a different
> path.
>
> https://paste.ubuntu.com/26081775/
>
> ubuntu@maas-xenial4:~$ tree /dev/ | grep bcache
> ├── bcache0
> ├── bcache1
> ├── bcache2
> ├── bcache3
> │ ├── 251:0 -> ../bcache0
> │ ├── 251:1 -> ../bcache1
> │ ├── 251:2 -> ../bcache2
> │ ├── 251:3 -> ../bcache3
> │ │ ├── bcache0 -> ../../bcache0
> │ │ ├── bcache1 -> ../../bcache1
> │ │ ├── bcache2 -> ../../bcache2
> │ │ ├── bcache3 -> ../../bcache3
>
>
> ** Attachment added: "bcache-multiple-links.png"
> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/
> 1729145/+attachment/5016592/+files/bcache-multiple-links.png
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1729145
>
> Title:
> /dev/bcache/by-uuid links not created after reboot
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/
> 1729145/+subscriptions
>

Investigating why /dev doesn't have the correct links even if they're
present led me to look at
/usr/share/initramfs-tools/scripts/init-bottom/udev (provided via the udev
dpkg from systemd)

In there, the mount -n -o move is used to migrate the /dev from initramfs
to the rootfs/dev

This bug:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=844775

Indicated that the initramfs may have util-linux version of mount which
needs to use
mount --move instead. This is addressed in newer systemd packages (in
bionic for sure)

And the mount line now looks like this:

# move the /dev tmpfs to the rootfs; fall back to util-linux mount that does
# not understand -o move
mount -n -o move /dev ${rootmnt}/dev || mount -n --move /dev ${rootmnt}/dev

Let's see if that helps resolve the "missing" symlinks.

On Thu, Nov 30, 2017 at 10:27 AM, Ryan Harper <email address hidden>
wrote:

> So, /dev/bcache/by-uuid is not getting created.
>
> That's the same kernel bug I filed.
>
> And, if they were, I think they'd get moved properly.
>
> init-bottom/udev script does the following:
>
> # Stop udevd, we'll miss a few events while we run init, but we catch up
> udevadm control --exit
>
> # move the /dev tmpfs to the rootfs
> mount -n -o move /dev ${rootmnt}/dev
>
> That "move" operation migrates the entire mounted directory to a new
> location
> so, if /dev/bcache/by-uuid existed, it would be present when systemd init
> runs.
>
> AFAICT, there's not fix here without the kernel side.
>
> On Thu, Nov 30, 2017 at 10:09 AM, Dmitrii Shcherbakov <
> <email address hidden>> wrote:
>
>> Just in case also checked with this rule that creates links at /dev
>> /dev/disk/by-uuid /dev/bcache/by-uuid https://paste.ubuntu.com/26081756/
>>
>> They are all there originally (see the screenshot).
>>
>> But not there afterwards. So it's not just a problem with a different
>> path.
>>
>> https://paste.ubuntu.com/26081775/
>>
>> ubuntu@maas-xenial4:~$ tree /dev/ | grep bcache
>> ├── bcache0
>> ├── bcache1
>> ├── bcache2
>> ├── bcache3
>> │ ├── 251:0 -> ../bcache0
>> │ ├── 251:1 -> ../bcache1
>> │ ├── 251:2 -> ../bcache2
>> │ ├── 251:3 -> ../bcache3
>> │ │ ├── bcache0 -> ../../bcache0
>> │ │ ├── bcache1 -> ../../bcache1
>> │ │ ├── bcache2 -> ../../bcache2
>> │ │ ├── bcache3 -> ../../bcache3
>>
>>
>> ** Attachment added: "bcache-multiple-links.png"
>> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/172914
>> 5/+attachment/5016592/+files/bcache-multiple-links.png
>>
>> --
>> You received this bug notification because you are subscribed to the bug
>> report.
>> https://bugs.launchpad.net/bugs/1729145
>>
>> Title:
>> /dev/bcache/by-uuid links not created after reboot
>>
>> To manage notifications about this bug go to:
>> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1729145
>> /+subscriptions
>>
>
>

Ryan,

As discussed in IRC, I tweaked locations for testing in #11 and added all paths in #14.

Looks like after `mount -n -o move /dev ${rootmnt}/dev` ${rootmnt}/dev has the proper content (see the bcache rule as well)

https://paste.ubuntu.com/26082611/
== before ==
lrwxrwxrwx 1 13 /dev/bcache/by-uuid/aa04ab39-a42c-446f-aff5-addbd9c80e06 -> ../../bcache0
lrwxrwxrwx 1 13 /dev/bcache/by-uuid/835afacf-6213-48a9-864c-43733822668b -> ../../bcache3
lrwxrwxrwx 1 13 /dev/bcache/by-uuid/727212eb-0f0b-4d20-9191-f0577a38454e -> ../../bcache1
lrwxrwxrwx 1 13 /dev/bcache/by-uuid/2963855b-3d2b-4387-abd1-3ae788919de4 -> ../../bcache2

== after ==
lrwxrwxrwx 1 13 /root/dev/bcache/by-uuid/aa04ab39-a42c-446f-aff5-addbd9c80e06 -> ../../bcache0
lrwxrwxrwx 1 13 /root/dev/bcache/by-uuid/835afacf-6213-48a9-864c-43733822668b -> ../../bcache3
lrwxrwxrwx 1 13 /root/dev/bcache/by-uuid/727212eb-0f0b-4d20-9191-f0577a38454e -> ../../bcache1
lrwxrwxrwx 1 13 /root/dev/bcache/by-uuid/2963855b-3d2b-4387-abd1-3ae788919de4 -> ../../bcache2

So the problem must be further down.

And about the only thing I see in the init script is run-init from klibc:

# Chain to real filesystem
exec run-init ${drop_caps} ${rootmnt} ${init} "$@" ${recovery:+--startup-event=recovery} <${rootmnt}/dev/console >${rootmnt}/dev/console 2>&1

So it's either somewhere in run-init code

https://git.launchpad.net/~usd-import-team/ubuntu/+source/klibc/tree/usr/kinit/run-init/runinitlib.c?h=applied/ubuntu/xenial-updates#n158

or in systemd.

Looks like masking systemd-udevd removes this behavior. Need to figure out where it actually clears everything out.

ubuntu@maas-xenial4:~$ sudo systemctl mask systemd-udevd
11:32 PM Created symlink from /etc/systemd/system/systemd-udevd.service to /dev/null.

# reboot

ubuntu@maas-xenial4:~$ ls /dev/bcache/by-uuid/
2963855b-3d2b-4387-abd1-3ae788919de4 727212eb-0f0b-4d20-9191-f0577a38454e 835afacf-6213-48a9-864c-43733822668b aa04ab39-a42c-446f-aff5-addbd9c80e06

It looks like there is some ordering issues:

This is a grep through /run/udev/links ; these are checked by udev-dev

# find . -name 'b250*'
./\x2fdisk\x2fby-uuid\x2f0a270acb-56b8-4498-8bad-b3bb149fe869/b250:1
./\x2fdisk\x2fby-uuid\x2f92b0868d-7e56-4956-8e55-2c90ebee4a72/b250:0
./\x2fbcache\x2fby-uuid\x2f92d882d8-38cd-4537-847b-6f9c40ba67b4/b250:1
./\x2fbcache\x2fby-uuid\x2f57e009b1-6bf4-42ea-abe0-334b10941a0b/b250:0

So both /dev/bcache/by-uuid and /dev/disk/by-uuid both point to the bcache
device (b250:0)

udevd shows this:

Nov 30 23:39:10.738290 ubuntu systemd-udevd[465]: LINK
'disk/by-uuid/0a270acb-56b8-4498-8bad-b3bb149fe869'
/lib/udev/rules.d/60-persistent-storage.rules:79
Nov 30 23:39:10.738304 ubuntu systemd-udevd[465]: update old name,
'/dev/bcache/by-uuid/92d882d8-38cd-4537-847b-6f9c40ba67b4' no longer
belonging to '/devices/virtual/block/bcache1'
Nov 30 23:39:10.738321 ubuntu systemd-udevd[465]: no reference left, remove
'/dev/bcache/by-uuid/92d882d8-38cd-4537-847b-6f9c40ba67b4'
Nov 30 23:39:10.738361 ubuntu systemd-udevd[465]: found 'b250:1' claiming
'/run/udev/links/\x2fdisk\x2fby-uuid\x2f0a270acb-56b8-4498-8bad-b3bb149fe869'
Nov 30 23:39:10.738370 ubuntu systemd-udevd[465]: creating link
'/dev/disk/by-uuid/0a270acb-56b8-4498-8bad-b3bb149fe869' to '/dev/bcache1'
Nov 30 23:39:10.738380 ubuntu systemd-udevd[465]: preserve already existing
symlink '/dev/disk/by-uuid/0a270acb-56b8-4498-8bad-b3bb149fe869' to
'../../bcache1'
Nov 30 23:39:10.743215 ubuntu systemd-udevd[487]: LINK
'disk/by-uuid/92b0868d-7e56-4956-8e55-2c90ebee4a72'
/lib/udev/rules.d/60-persistent-storage.rules:79
Nov 30 23:39:10.743228 ubuntu systemd-udevd[487]: update old name,
'/dev/bcache/by-uuid/57e009b1-6bf4-42ea-abe0-334b10941a0b' no longer
belonging to '/devices/virtual/block/bcache0'
Nov 30 23:39:10.743247 ubuntu systemd-udevd[487]: no reference left, remove
'/dev/bcache/by-uuid/57e009b1-6bf4-42ea-abe0-334b10941a0b'
Nov 30 23:39:10.743291 ubuntu systemd-udevd[487]: found 'b250:0' claiming
'/run/udev/links/\x2fdisk\x2fby-uuid\x2f92b0868d-7e56-4956-8e55-2c90ebee4a72'
Nov 30 23:39:10.743302 ubuntu systemd-udevd[487]: creating link
'/dev/disk/by-uuid/92b0868d-7e56-4956-8e55-2c90ebee4a72' to '/dev/bcache0'
Nov 30 23:39:10.743312 ubuntu systemd-udevd[487]: preserve already existing
symlink '/dev/disk/by-uuid/92b0868d-7e56-4956-8e55-2c90ebee4a72' to
'../../bcache0'

It would seem that the 'no longer belonging to ...bcache1' is the trigger
for the removal
however, 250:1 is indeed bcache1 device.

# ls -al /dev/bcache1
brw-rw---- 1 root disk 250, 1 Nov 30 23:39 /dev/bcache1

Looking at the systemd/udev/udev-event.c it appears that for entries in
the current udev database that don't have a new event associated with them
get the call to remove the old link

void udev_event_execute_rules()
  ...
  if (major(udev_device_get_devnum(dev)) > 0) {
                        bool apply;

                        /* remove/update possible left-over symlinks from
old database entry */
                        if (event->dev_db != NULL)
                                udev_node_update_old_links(dev,
event->dev_db);

Will need to crawl through the db to see what the the event->dev_db en...

Dec 01 05:07:25.679368 ubuntu systemd-udevd[474]: LINK
'disk/by-uuid/0a270acb-56b8-4498-8bad-b3bb149fe869'
/lib/udev/rules.d/60-persistent-storage.rules:79
Dec 01 05:07:25.683008 ubuntu systemd-udevd[474]: WARK:
dev_old_name=/dev/bcache/by-uuid/92d882d8-38cd-4537-847b-6f9c40ba67b4
dev_old_path=/devices/virtual/block/bcache0
Dec 01 05:07:25.683042 ubuntu systemd-udevd[474]: WARK:
dev_name=/dev/disk/by-uuid/0a270acb-56b8-4498-8bad-b3bb149fe869
dev_path=/devices/virtual/block/bcache0
Dec 01 05:07:25.683066 ubuntu systemd-udevd[474]: WARK:
streq(name=/dev/bcache/by-uuid/92d882d8-38cd-4537-847b-6f9c40ba67b4,
name_current=/dev/disk/by-uuid/0a270acb-56b8-4498-8bad-b3bb149fe869)
Dec 01 05:07:25.683076 ubuntu systemd-udevd[474]: update old name,
'/dev/bcache/by-uuid/92d882d8-38cd-4537-847b-6f9c40ba67b4' no longer
belonging to '/devices/virtual/block/bcache0'
Dec 01 05:07:25.683085 ubuntu systemd-udevd[474]: WARK:
link_update(name=/dev/bcache/by-uuid/92d882d8-38cd-4537-847b-6f9c40ba67b4,
false
Dec 01 05:07:25.683094 ubuntu systemd-udevd[474]: no reference left, remove
'/dev/bcache/by-uuid/92d882d8-38cd-4537-847b-6f9c40ba67b4'
Dec 01 05:07:25.683104 ubuntu systemd-udevd[474]: WARK:
dev_old_name=/dev/disk/by-uuid/0a270acb-56b8-4498-8bad-b3bb149fe869
dev_old_path=/devices/virtual/block/bcache0
Dec 01 05:07:25.683113 ubuntu systemd-udevd[474]: WARK:
dev_name=/dev/disk/by-uuid/0a270acb-56b8-4498-8bad-b3bb149fe869
dev_path=/devices/virtual/block/bcache0
Dec 01 05:07:25.683122 ubuntu systemd-udevd[474]: WARK:
streq(name=/dev/disk/by-uuid/0a270acb-56b8-4498-8bad-b3bb149fe869,
name_current=/dev/disk/by-uuid/0a270acb-56b8-4498-8bad-b3bb149fe869)
Dec 01 05:07:25.683131 ubuntu systemd-udevd[474]: WARK: found match!
continue
Dec 01 05:07:25.683146 ubuntu systemd-udevd[474]: handling device node
'/dev/bcache0', devnum=b250:0, mode=0660, uid=0, gid=6
Dec 01 05:07:25.683155 ubuntu systemd-udevd[474]: set permissions
/dev/bcache0, 060660, uid=0, gid=6
Dec 01 05:07:25.683165 ubuntu systemd-udevd[474]: preserve already existing
symlink '/dev/block/250:0' to '../bcache0'
Dec 01 05:07:25.683174 ubuntu systemd-udevd[474]: found 'b250:0' claiming
'/run/udev/links/\x2fdisk\x2fby-uuid\x2f0a270acb-56b8-4498-8bad-b3bb149fe869'
Dec 01 05:07:25.683183 ubuntu systemd-udevd[474]: creating link
'/dev/disk/by-uuid/0a270acb-56b8-4498-8bad-b3bb149fe869' to '/dev/bcache0'
Dec 01 05:07:25.683193 ubuntu systemd-udevd[474]: preserve already existing
symlink '/dev/disk/by-uuid/0a270acb-56b8-4498-8bad-b3bb149fe869' to
'../../bcache0'
Dec 01 05:07:25.683201 ubuntu systemd-udevd[474]: created db file
'/run/udev/data/b250:0' for '/devices/virtual/block/bcache0'
Dec 01 05:07:25.683211 ubuntu systemd[1]:
dev-disk-by\x2duuid-0a270acb\x2d56b8\x2d4498\x2d8bad\x2db3bb149fe869.device:
Changed dead -> plugged
Dec 01 05:07:25.683226 ubuntu systemd[1]: dev-bcache0.device: Changed dead
-> plugged
Dec 01 05:07:25.683234 ubuntu systemd[1]:
sys-devices-virtual-block-bcache0.device: Changed dead -> plugged

The new dev (from /deb/bcache0) only has a by-uuid devlink as the UEVENT
that generates the CACHED_UUID devlink is not recorded/discovered
after the original binding of the cache device and backing store. ...

Untested patch. Hoping to convey the change that's needed.

- decouple emitting a cached_dev CHANGE uevent which includes dev.uuid and dev.label
   from bch_cached_dev_run() which only happens when a bcacheX device is bound to the
   actual backing block device (bcache0 -> vdb)

- update bch_cached_dev_run() to invoke bch_cached_dev_emit_change() as needed; no functional
  code path changes here

- Modify register_bcache to detect a re-registering of a bcache cached_dev, and in that case
  call bcache_cached_dev_emit_change() to emit the uevent needed to trigger SYMLINK generation

The attachment "bcache_always_emit_backing_dev_change_uevent.diff" seems to be a patch. If it isn't, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are a member of the ~ubuntu-reviewers, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issues please contact him.]

@Ryan

I tried to build an Artful test kernel with your patch. However, there were compilation errors:

ubuntu-artful/drivers/md/bcache/super.c:868:9: error: 'env' undeclared (first use in this function); did you mean '_end'?
   kfree(env[1]);
         ^~~
         _end

Did you have any issues compiling?

It looks like env[] was declared in bch_cached_dev_emit_change(), but not in bch_cached_dev_run().

Hi Joseph,

Sorry, I didn't give that a compile either; I just wanted to show what the
change could look like;
Let me see if I can get that to at least compile.

On Mon, Dec 4, 2017 at 11:18 AM, Joseph Salisbury <
<email address hidden>> wrote:

> It looks like env[] was declared in bch_cached_dev_emit_change(), but
> not in bch_cached_dev_run().
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1729145
>
> Title:
> /dev/bcache/by-uuid links not created after reboot
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/
> 1729145/+subscriptions
>

Looks like those two kfree's in dev_run can be dropped since that was an
exit after kmalloc'ing env entries which are now only done in

bch_cached_dev_emit_change()

which is only called by dev_run after it knows that the device is not
yet running.

On Mon, Dec 4, 2017 at 11:35 AM, Ryan Harper <email address hidden>
wrote:

> Hi Joseph,
>
> Sorry, I didn't give that a compile either; I just wanted to show what the
> change could look like;
> Let me see if I can get that to at least compile.
>
>
> On Mon, Dec 4, 2017 at 11:18 AM, Joseph Salisbury <
> <email address hidden>> wrote:
>
>> It looks like env[] was declared in bch_cached_dev_emit_change(), but
>> not in bch_cached_dev_run().
>>
>> --
>> You received this bug notification because you are subscribed to the bug
>> report.
>> https://bugs.launchpad.net/bugs/1729145
>>
>> Title:
>> /dev/bcache/by-uuid links not created after reboot
>>
>> To manage notifications about this bug go to:
>> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1729145
>> /+subscriptions
>>
>
>

Revised patch, should fix error with kfree on env

I built an Artful test kernel with the path provided by Ryan. The test kernel can be downloaded from:

http://kernel.ubuntu.com/~jsalisbury/lp1729145/

Can those that can reproduce this bug give this test kernel a try?

Thanks! I'll give it a try today.

On Tue, Dec 5, 2017 at 3:43 PM, Joseph Salisbury <
<email address hidden>> wrote:

> I built an Artful test kernel with the path provided by Ryan. The test
> kernel can be downloaded from:
>
> http://kernel.ubuntu.com/~jsalisbury/lp1729145/
>
> Can those that can reproduce this bug give this test kernel a try?
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1729145
>
> Title:
> /dev/bcache/by-uuid links not created after reboot
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/
> 1729145/+subscriptions
>

Woot!

I think that did the trick.

ubuntu@ubuntu:~$ cat /etc/cloud/build.info
build_name: server
serial: 20171122
ubuntu@ubuntu:~$ lsb_release -rd
Description: Ubuntu 17.10
Release: 17.10
ubuntu@ubuntu:~$ set -o vi
ubuntu@ubuntu:~$ dpkg --list | grep linux-image
ii linux-image-4.13.0-17-generic 4.13.0-17.20~lp1739145
                 amd64 Linux kernel image for version 4.13.0 on 64
bit x86 SMP
ii linux-image-virtual 4.13.0.17.18
                 amd64 Virtual Linux kernel image
ubuntu@ubuntu:~$ dpkg --list | grep bcache-tools
ii bcache-tools 1.0.8-2build1
                amd64 bcache userspace tools

ubuntu@ubuntu:~$ sudo tree /dev/bcache
/dev/bcache
├── by-label
│ ├── backing0 -> ../../bcache1
│ └── backing1 -> ../../bcache0
└── by-uuid
    ├── 1375d23f-4780-46d7-9fca-c397f8333685 -> ../../bcache2
    ├── 57e009b1-6bf4-42ea-abe0-334b10941a0b -> ../../bcache0
    ├── 7ce7dc32-7da9-42a8-899a-5d21ed7ea714 -> ../../bcache3
    └── 92d882d8-38cd-4537-847b-6f9c40ba67b4 -> ../../bcache1

2 directories, 6 files

On Tue, Dec 5, 2017 at 4:00 PM, Ryan Harper <email address hidden>
wrote:

> Thanks! I'll give it a try today.
>
> On Tue, Dec 5, 2017 at 3:43 PM, Joseph Salisbury <
> <email address hidden>> wrote:
>
>> I built an Artful test kernel with the path provided by Ryan. The test
>> kernel can be downloaded from:
>>
>> http://kernel.ubuntu.com/~jsalisbury/lp1729145/
>>
>> Can those that can reproduce this bug give this test kernel a try?
>>
>> --
>> You received this bug notification because you are subscribed to the bug
>> report.
>> https://bugs.launchpad.net/bugs/1729145
>>
>> Title:
>> /dev/bcache/by-uuid links not created after reboot
>>
>> To manage notifications about this bug go to:
>> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1729145
>> /+subscriptions
>>
>
>

I attached an update to your patch, Ryan.

I added a subject, commit log and couple of missing semi colons / parenthesis.

Can you look it over and see if you approve of the log and subject? I'll see if it applies to the other Ubuntu releases and builds on them as well.

In the final patch we submit for SRU, it will also include your Signed-off-by, I just forgot to add it in the attached one.

Thanks for doing the cleanup; Patch looks good and I approve.

Signed-off-by: Ryan Harper <email address hidden>

On Tue, Dec 5, 2017 at 4:59 PM, Joseph Salisbury <
<email address hidden>> wrote:

> In the final patch we submit for SRU, it will also include your Signed-
> off-by, I just forgot to add it in the attached one.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1729145
>
> Title:
> /dev/bcache/by-uuid links not created after reboot
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/
> 1729145/+subscriptions
>

I built Xenial, Zesty and Bionic test kernels with your patch. They can also be downloaded from:

http://kernel.ubuntu.com/~jsalisbury/lp1729145/

Can those that can reproduce this bug give this test kernel a try?

Zesty required upstream commit 4b758df21 as a prereq. Zesty never got it, since upstream 4.10 is EOL. Zesty would be the more important test kernel to try due to this difference.

Thanks again.

Here's the Zesty test; all looks good.

ubuntu@ubuntu:~$ cat /etc/cloud/build.info
build_name: server
serial: 20171207
ubuntu@ubuntu:~$ uname -a
Linux ubuntu 4.10.0-40-generic #44~lp1729145 SMP Wed Dec 6 16:21:45 UTC
2017 x86_64 x86_64 x86_64 GNU/Linux
ubuntu@ubuntu:~$ tree /dev/bcache
/dev/bcache [error opening dir]

0 directories, 0 files

$ sudo dpkg --install
linux-image-4.10.0-40-generic_4.10.0-40.44~lp1729145_amd64.deb
dpkg: warning: downgrading linux-image-4.10.0-40-generic from 4.10.0-40.44
to 4.10.0-40.44~lp1729145
(Reading database ... 56846 files and directories currently installed.)
Preparing to unpack
linux-image-4.10.0-40-generic_4.10.0-40.44~lp1729145_amd64.deb ...
Done.
Unpacking linux-image-4.10.0-40-generic (4.10.0-40.44~lp1729145) over
(4.10.0-40.44) ...
Examining /etc/kernel/postrm.d .
run-parts: executing /etc/kernel/postrm.d/initramfs-tools 4.10.0-40-generic
/boot/vmlinuz-4.10.0-40-generic
run-parts: executing /etc/kernel/postrm.d/x-grub-legacy-ec2
4.10.0-40-generic /boot/vmlinuz-4.10.0-40-generic
run-parts: executing /etc/kernel/postrm.d/zz-update-grub 4.10.0-40-generic
/boot/vmlinuz-4.10.0-40-generic
Setting up linux-image-4.10.0-40-generic (4.10.0-40.44~lp1729145) ...
Running depmod.
update-initramfs: deferring update (hook will be called later)
Not updating initrd symbolic links since we are being updated/reinstalled
(4.10.0-40.44 was configured last, according to dpkg)
Not updating image symbolic links since we are being updated/reinstalled
(4.10.0-40.44 was configured last, according to dpkg)
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/apt-auto-removal
4.10.0-40-generic /boot/vmlinuz-4.10.0-40-generic
run-parts: executing /etc/kernel/postinst.d/initramfs-tools
4.10.0-40-generic /boot/vmlinuz-4.10.0-40-generic
update-initramfs: Generating /boot/initrd.img-4.10.0-40-generic
W: mdadm: /etc/mdadm/mdadm.conf defines no arrays.
run-parts: executing /etc/kernel/postinst.d/unattended-upgrades
4.10.0-40-generic /boot/vmlinuz-4.10.0-40-generic
run-parts: executing /etc/kernel/postinst.d/update-notifier
4.10.0-40-generic /boot/vmlinuz-4.10.0-40-generic
run-parts: executing /etc/kernel/postinst.d/x-grub-legacy-ec2
4.10.0-40-generic /boot/vmlinuz-4.10.0-40-generic
Searching for GRUB installation directory ... found: /boot/grub
Searching for default file ... found: /boot/grub/default
Testing for an existing GRUB menu.lst file ... found: /boot/grub/menu.lst
Searching for splash image ... none found, skipping ...
Found kernel: /boot/vmlinuz-4.10.0-40-generic
Found kernel: /boot/vmlinuz-4.10.0-40-generic
Updating /boot/grub/menu.lst ... done

run-parts: executing /etc/kernel/postinst.d/zz-update-grub
4.10.0-40-generic /boot/vmlinuz-4.10.0-40-generic
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.10.0-40-generic
Found initrd image: /boot/initrd.img-4.10.0-40-generic
done
ubuntu@ubuntu:~$ sudo reboot
...

ubuntu@ubuntu:~$ dpkg --list | grep linux-image
ii linux-image-4.10.0-40-generic 4.10.0-40.44~lp1729145
                 amd64 Linux kernel image for version 4.10.0 on 64
bit x86 SMP
ii linux-image-virtual 4.10.0.40.40
         ...

Tested the xenial update. I had one boot where the links didn't get
created, but I cannot recreate that issue now.

On Thu, Dec 7, 2017 at 9:56 AM, Ryan Harper <email address hidden>
wrote:

> Here's the Zesty test; all looks good.
>
> ubuntu@ubuntu:~$ cat /etc/cloud/build.info
> build_name: server
> serial: 20171207
> ubuntu@ubuntu:~$ uname -a
> Linux ubuntu 4.10.0-40-generic #44~lp1729145 SMP Wed Dec 6 16:21:45 UTC
> 2017 x86_64 x86_64 x86_64 GNU/Linux
> ubuntu@ubuntu:~$ tree /dev/bcache
> /dev/bcache [error opening dir]
>
> 0 directories, 0 files
>
> $ sudo dpkg --install linux-image-4.10.0-40-generic_
> 4.10.0-40.44~lp1729145_amd64.deb
> dpkg: warning: downgrading linux-image-4.10.0-40-generic from 4.10.0-40.44
> to 4.10.0-40.44~lp1729145
> (Reading database ... 56846 files and directories currently installed.)
> Preparing to unpack linux-image-4.10.0-40-generic_
> 4.10.0-40.44~lp1729145_amd64.deb ...
> Done.
> Unpacking linux-image-4.10.0-40-generic (4.10.0-40.44~lp1729145) over
> (4.10.0-40.44) ...
> Examining /etc/kernel/postrm.d .
> run-parts: executing /etc/kernel/postrm.d/initramfs-tools
> 4.10.0-40-generic /boot/vmlinuz-4.10.0-40-generic
> run-parts: executing /etc/kernel/postrm.d/x-grub-legacy-ec2
> 4.10.0-40-generic /boot/vmlinuz-4.10.0-40-generic
> run-parts: executing /etc/kernel/postrm.d/zz-update-grub
> 4.10.0-40-generic /boot/vmlinuz-4.10.0-40-generic
> Setting up linux-image-4.10.0-40-generic (4.10.0-40.44~lp1729145) ...
> Running depmod.
> update-initramfs: deferring update (hook will be called later)
> Not updating initrd symbolic links since we are being updated/reinstalled
> (4.10.0-40.44 was configured last, according to dpkg)
> Not updating image symbolic links since we are being updated/reinstalled
> (4.10.0-40.44 was configured last, according to dpkg)
> Examining /etc/kernel/postinst.d.
> run-parts: executing /etc/kernel/postinst.d/apt-auto-removal
> 4.10.0-40-generic /boot/vmlinuz-4.10.0-40-generic
> run-parts: executing /etc/kernel/postinst.d/initramfs-tools
> 4.10.0-40-generic /boot/vmlinuz-4.10.0-40-generic
> update-initramfs: Generating /boot/initrd.img-4.10.0-40-generic
> W: mdadm: /etc/mdadm/mdadm.conf defines no arrays.
> run-parts: executing /etc/kernel/postinst.d/unattended-upgrades
> 4.10.0-40-generic /boot/vmlinuz-4.10.0-40-generic
> run-parts: executing /etc/kernel/postinst.d/update-notifier
> 4.10.0-40-generic /boot/vmlinuz-4.10.0-40-generic
> run-parts: executing /etc/kernel/postinst.d/x-grub-legacy-ec2
> 4.10.0-40-generic /boot/vmlinuz-4.10.0-40-generic
> Searching for GRUB installation directory ... found: /boot/grub
> Searching for default file ... found: /boot/grub/default
> Testing for an existing GRUB menu.lst file ... found: /boot/grub/menu.lst
> Searching for splash image ... none found, skipping ...
> Found kernel: /boot/vmlinuz-4.10.0-40-generic
> Found kernel: /boot/vmlinuz-4.10.0-40-generic
> Updating /boot/grub/menu.lst ... done
>
> run-parts: executing /etc/kernel/postinst.d/zz-update-grub
> 4.10.0-40-generic /boot/vmlinuz-4.10.0-40-generic
> Generating grub configuration file ...
> Found linux image: /boot/vmlinuz-4.10.0-40-generic
> Found initrd image: /boot/initrd.img-4.10.0-40...

Thanks for testing and the patch, Ryan. I can submit and SRU request.

We should also send this upstream for inclusion in the mainline kernel. Is that something you think you can do? If not, I can do it for you.

Submitted SRU requests:
https://lists.ubuntu.com/archives/kernel-team/2017-December/088680.html
https://lists.ubuntu.com/archives/kernel-team/2017-December/088682.html

Confirmed bionic works as expected.

I suspect you can send that upstream with my SoB faster than I can.
Definitely interested in seeing if they'll take something like that.

On Thu, Dec 7, 2017 at 1:21 PM, Joseph Salisbury <
<email address hidden>> wrote:

> Thanks for testing and the patch, Ryan. I can submit and SRU request.
>
> We should also send this upstream for inclusion in the mainline kernel.
> Is that something you think you can do? If not, I can do it for you.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1729145
>
> Title:
> /dev/bcache/by-uuid links not created after reboot
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/
> 1729145/+subscriptions
>

Once the kernel is fixed, are there any changes that are required to systemd/udev?

On Tue, Dec 12, 2017 at 5:52 AM, Dimitri John Ledkov <<email address hidden>
> wrote:

> Once the kernel is fixed, are there any changes that are required to
> systemd/udev?
>

No changes needed.

>
> ** Changed in: systemd (Ubuntu Bionic)
> Status: New => Incomplete
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1729145
>
> Title:
> /dev/bcache/by-uuid links not created after reboot
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/
> 1729145/+subscriptions
>

@jsalisbury Did you get a chance to forward the patch upstream? We've a userspace fix in the bcache-tools package for now; I've sent that upstream here:

https://github.com/koverstreet/bcache-tools/pull/1

Kent was asking about the kernel fix, to which I pointed to the patch here.

@raharper, I got side tracked and it got pushed down on my list. I can still send it upstream if you want. It's currently in Ubuntu as a SAUCE patch.

On Fri, Feb 16, 2018 at 12:14 PM, Joseph Salisbury <
<email address hidden>> wrote:

> @raharper, I got side tracked and it got pushed down on my list. I can
> still send it upstream if you want. It's currently in Ubuntu as a SAUCE
> patch.
>

That would be great!
Thanks,

>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1729145
>
> Title:
> /dev/bcache/by-uuid links not created after reboot
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/
> 1729145/+subscriptions
>

Joseph forwarded this upstream. LKML mailing list post https://lkml.org/lkml/2018/3/1/812 .

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-xenial' to 'verification-done-xenial'. If the problem still exists, change the tag 'verification-needed-xenial' to 'verification-failed-xenial'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-artful' to 'verification-done-artful'. If the problem still exists, change the tag 'verification-needed-artful' to 'verification-failed-artful'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

Not sure if this was supposed to also be fixed in bionic-proposed yet,
but I assumed so and
it's not working there; I'll test -proposed in artful and xenial next.

ubuntu@ubuntu:~$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu Bionic Beaver (development branch)"
ubuntu@ubuntu:~$ uname -a
Linux ubuntu 4.15.0-13-generic #14-Ubuntu SMP Sat Mar 17 13:44:27 UTC
2018 x86_64 x86_64 x86_64 GNU/Linux

ubuntu@ubuntu:~$ apt-cache policy linux-image-virtual
linux-image-virtual:
  Installed: 4.15.0.13.14
  Candidate: 4.15.0.13.14
  Version table:
 *** 4.15.0.13.14 500
        500 http://archive.ubuntu.com/ubuntu bionic-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     4.15.0.12.13 500
        500 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages
ubuntu@ubuntu:~$ uname -a
Linux ubuntu 4.15.0-13-generic #14-Ubuntu SMP Sat Mar 17 13:44:27 UTC
2018 x86_64 x86_64 x86_64 GNU/Linux

ubuntu@ubuntu:~$ ls -al /dev/bcache*
brw-rw---- 1 root disk 251, 0 Mar 19 18:09 /dev/bcache0
brw-rw---- 1 root disk 251, 128 Mar 19 18:09 /dev/bcache1
brw-rw---- 1 root disk 251, 256 Mar 19 18:09 /dev/bcache2
brw-rw---- 1 root disk 251, 384 Mar 19 18:09 /dev/bcache3
ubuntu@ubuntu:~$ dmesg | grep bcache
[ 1.005037] bcache: register_bdev() registered backing device sdb
[ 1.011078] bcache: bch_journal_replay() journal replay done, 0
keys in 1 entries, seq 5
[ 1.023834] bcache: bch_cached_dev_attach() Caching sdb as bcache0
on set 0dbdcc17-7af0-4318-bd57-bbdf5c7f1bfa
[ 1.029237] bcache: register_cache() registered cache device sda
[ 1.029555] bcache: register_bdev() registered backing device sdd
[ 1.050543] bcache: bch_cached_dev_attach() Caching sdd as bcache1
on set 0dbdcc17-7af0-4318-bd57-bbdf5c7f1bfa
[ 1.052124] bcache: register_bdev() registered backing device sde
[ 1.054044] bcache: register_bdev() registered backing device sdf
[ 1.058398] bcache: bch_journal_replay() journal replay done, 0
keys in 1 entries, seq 5
[ 1.068618] bcache: bch_cached_dev_attach() Caching sdf as bcache3
on set d36b419d-a49d-42e2-95af-710138a20700
[ 1.070652] bcache: bch_cached_dev_attach() Caching sde as bcache2
on set d36b419d-a49d-42e2-95af-710138a20700
[ 1.076326] bcache: register_cache() registered cache device sdc
[ 4.090280] bcache: register_bcache() error /dev/sdc: device
already registered
[ 4.091070] bcache: register_bcache() error /dev/sdd: device
already registered
[ 4.094397] bcache: register_bcache() error /dev/sde: device
already registered
[ 4.095253] bcache: register_bcache() error /dev/sda: device
already registered
[ 4.103178] bcache: register_bcache() error /dev/sdb: device
already registered
[ 4.115378] bcache: register_bcache() error /dev/sdf: device
already registered

On Mon, Mar 19, 2018 at 5:56 AM, Stefan Bader
<email address hidden> wrote:
> This bug is awaiting verification that the kernel in -proposed solves
> the problem. Please test the kernel and update this bug with the
> results. If the problem is solved, change the tag 'verification-needed-
> artful' to 'verification-done-artful'. If the problem still exists,
> chang...

Artful verified:

ubuntu@ubuntu:~$ lsb_release -rd
Description: Ubuntu 17.10
Release: 17.10
ubuntu@ubuntu:~$ uname -a
Linux ubuntu 4.13.0-38-generic #43-Ubuntu SMP Wed Mar 14 15:20:44 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
ubuntu@ubuntu:~$ apt-cache policy linux-image-virtual
linux-image-virtual:
  Installed: 4.13.0.38.41
  Candidate: 4.13.0.38.41
  Version table:
 *** 4.13.0.38.41 500
        500 http://archive.ubuntu.com/ubuntu artful-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     4.13.0.37.40 500
        500 http://archive.ubuntu.com/ubuntu artful-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu artful-security/main amd64 Packages
     4.13.0.16.17 500
        500 http://archive.ubuntu.com/ubuntu artful/main amd64 Packages

ubuntu@ubuntu:~$ ls -al /dev/bcache/by-*
/dev/bcache/by-label:
total 0
drwxr-xr-x 2 root root 120 Mar 19 18:32 .
drwxr-xr-x 4 root root 80 Mar 19 18:32 ..
lrwxrwxrwx 1 root root 13 Mar 19 18:32 backing-sdb -> ../../bcache1
lrwxrwxrwx 1 root root 13 Mar 19 18:32 backing-sdd -> ../../bcache0
lrwxrwxrwx 1 root root 13 Mar 19 18:32 backing-sde -> ../../bcache3
lrwxrwxrwx 1 root root 13 Mar 19 18:32 backing-sdf -> ../../bcache2

/dev/bcache/by-uuid:
total 0
drwxr-xr-x 2 root root 120 Mar 19 18:32 .
drwxr-xr-x 4 root root 80 Mar 19 18:32 ..
lrwxrwxrwx 1 root root 13 Mar 19 18:32 1f531b44-437f-49cc-b7db-16db6d699c41 -> ../../bcache2
lrwxrwxrwx 1 root root 13 Mar 19 18:32 31474eb7-687d-4bb7-8c6b-b4efd472e860 -> ../../bcache1
lrwxrwxrwx 1 root root 13 Mar 19 18:32 3653ce1f-b6aa-43ad-9a91-506530db2d50 -> ../../bcache0
lrwxrwxrwx 1 root root 13 Mar 19 18:32 94041d44-4bb2-4e4d-8641-21d4ae410595 -> ../../bcache3

ubuntu@ubuntu:~$ ls -al /dev/bcache/by-label/
total 0
drwxr-xr-x 2 root root 120 Mar 19 18:32 .
drwxr-xr-x 4 root root 80 Mar 19 18:32 ..
lrwxrwxrwx 1 root root 13 Mar 19 18:32 backing-sdb -> ../../bcache1
lrwxrwxrwx 1 root root 13 Mar 19 18:32 backing-sdd -> ../../bcache0
lrwxrwxrwx 1 root root 13 Mar 19 18:32 backing-sde -> ../../bcache3
lrwxrwxrwx 1 root root 13 Mar 19 18:32 backing-sdf -> ../../bcache2

ubuntu@ubuntu:~$ ls -al /dev/bcache/by-label/backing-sdb
lrwxrwxrwx 1 root root 13 Mar 19 18:32 /dev/bcache/by-label/backing-sdb -> ../../bcache1
ubuntu@ubuntu:~$ ls -al /sys/class/block/bcache1/slaves/
total 0
drwxr-xr-x 2 root root 0 Mar 19 18:36 .
drwxr-xr-x 8 root root 0 Mar 19 18:36 ..
lrwxrwxrwx 1 root root 0 Mar 19 18:36 sda -> ../../../../pci0000:00/0000:00:04.0/virtio1/host2/target2:0:0/2:0:0:0/block/sda
lrwxrwxrwx 1 root root 0 Mar 19 18:36 sdb -> ../../../../pci0000:00/0000:00:04.0/virtio1/host2/target2:0:1/2:0:1:0/block/sdb
ubuntu@ubuntu:~$ ls -al /dev/bcache/by-label/backing-sdd
lrwxrwxrwx 1 root root 13 Mar 19 18:32 /dev/bcache/by-label/backing-sdd -> ../../bcache0
ubuntu@ubuntu:~$ ls -al /sys/class/block/bcache0/slaves/
total 0
drwxr-xr-x 2 root root 0 Mar 19 18:36 .
drwxr-xr-x 8 root root 0 Mar 19 18:36 ..
lrwxrwxrwx 1 root root 0 Mar 19 18:36 sda -> ../../../../pci0000:00/0000:00:04.0/virtio1/host2/target2:0:0/2:0:0:0/block/sda
lrwxrwxrwx 1 root root 0 Mar 19 18:36 sdd -> ../../../../pci0000:00/0000:00:04.0/virtio1/host2/target2:0:3/2:0:3:0/b...

Xenial verified.

root@ubuntu:~# lsb_release -rd
Description: Ubuntu 16.04.4 LTS
Release: 16.04
root@ubuntu:~# uname -a
Linux ubuntu 4.4.0-117-generic #141-Ubuntu SMP Tue Mar 13 11:58:07 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
root@ubuntu:~# apt-cache policy linux-image-virtual
linux-image-virtual:
  Installed: 4.4.0.117.123
  Candidate: 4.4.0.117.123
  Version table:
 *** 4.4.0.117.123 500
        500 http://archive.ubuntu.com/ubuntu xenial-proposed/main amd64 Packages
        100 /var/lib/dpkg/status
     4.4.0.116.122 500
        500 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
        500 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages
     4.4.0.21.22 500
        500 http://archive.ubuntu.com/ubuntu xenial/main amd64 Packages
root@ubuntu:~# ls -al /dev/bcache/*
/dev/bcache/by-label:
total 0
drwxr-xr-x 2 root root 120 Mar 19 19:07 .
drwxr-xr-x 4 root root 80 Mar 19 19:07 ..
lrwxrwxrwx 1 root root 13 Mar 19 19:07 backing-sdb -> ../../bcache0
lrwxrwxrwx 1 root root 13 Mar 19 19:07 backing-sdd -> ../../bcache3
lrwxrwxrwx 1 root root 13 Mar 19 19:07 backing-sde -> ../../bcache2
lrwxrwxrwx 1 root root 13 Mar 19 19:07 backing-sdf -> ../../bcache1

/dev/bcache/by-uuid:
total 0
drwxr-xr-x 2 root root 120 Mar 19 19:07 .
drwxr-xr-x 4 root root 80 Mar 19 19:07 ..
lrwxrwxrwx 1 root root 13 Mar 19 19:07 1f531b44-437f-49cc-b7db-16db6d699c41 -> ../../bcache1
lrwxrwxrwx 1 root root 13 Mar 19 19:07 31474eb7-687d-4bb7-8c6b-b4efd472e860 -> ../../bcache0
lrwxrwxrwx 1 root root 13 Mar 19 19:07 3653ce1f-b6aa-43ad-9a91-506530db2d50 -> ../../bcache3
lrwxrwxrwx 1 root root 13 Mar 19 19:07 94041d44-4bb2-4e4d-8641-21d4ae410595 -> ../../bcache2
root@ubuntu:~# ls -al /dev/bcache/by-label/backing-sdb
lrwxrwxrwx 1 root root 13 Mar 19 19:07 /dev/bcache/by-label/backing-sdb -> ../../bcache0
root@ubuntu:~# ls -al /sys/class/block/bcache0/slaves
total 0
drwxr-xr-x 2 root root 0 Mar 19 19:08 .
drwxr-xr-x 8 root root 0 Mar 19 19:07 ..
lrwxrwxrwx 1 root root 0 Mar 19 19:08 sda -> ../../../../pci0000:00/0000:00:04.0/virtio1/host2/target2:0:0/2:0:0:0/block/sda
lrwxrwxrwx 1 root root 0 Mar 19 19:08 sdb -> ../../../../pci0000:00/0000:00:04.0/virtio1/host2/target2:0:1/2:0:1:0/block/sdb
root@ubuntu:~# ls -al /dev/bcache/by-label/backing-sdd
lrwxrwxrwx 1 root root 13 Mar 19 19:07 /dev/bcache/by-label/backing-sdd -> ../../bcache3
root@ubuntu:~# ls -al /sys/class/block/bcache3/slaves
total 0
drwxr-xr-x 2 root root 0 Mar 19 19:09 .
drwxr-xr-x 8 root root 0 Mar 19 19:07 ..
lrwxrwxrwx 1 root root 0 Mar 19 19:09 sda -> ../../../../pci0000:00/0000:00:04.0/virtio1/host2/target2:0:0/2:0:0:0/block/sda
lrwxrwxrwx 1 root root 0 Mar 19 19:09 sdd -> ../../../../pci0000:00/0000:00:04.0/virtio1/host2/target2:0:3/2:0:3:0/block/sdd
root@ubuntu:~# ls -al /dev/bcache/by-label/backing-sde
lrwxrwxrwx 1 root root 13 Mar 19 19:07 /dev/bcache/by-label/backing-sde -> ../../bcache2
root@ubuntu:~# ls -al /sys/class/block/bcache2/slaves
total 0
drwxr-xr-x 2 root root 0 Mar 19 19:09 .
drwxr-xr-x 8 root root 0 Mar 19 19:07 ..
lrwxrwxrwx 1 root root 0 Mar 19 19:09 sdc -> ../../../../pci0000:00/0000:00:04.0/virtio1/host2/target2:0:2/2:0:2:0/block/sdc
lrwxrwxrw...

This bug was fixed in the package linux - 4.13.0-38.43

---------------
linux (4.13.0-38.43) artful; urgency=medium

  * linux: 4.13.0-38.43 -proposed tracker (LP: #1755762)

  * Servers going OOM after updating kernel from 4.10 to 4.13 (LP: #1748408)
    - i40e: Fix memory leak related filter programming status
    - i40e: Add programming descriptors to cleaned_count

  * [SRU] Lenovo E41 Mic mute hotkey is not responding (LP: #1753347)
    - platform/x86: ideapad-laptop: Increase timeout to wait for EC answer

  * fails to dump with latest kpti fixes (LP: #1750021)
    - kdump: write correct address of mem_section into vmcoreinfo

  * headset mic can't be detected on two Dell machines (LP: #1748807)
    - ALSA: hda/realtek - Support headset mode for ALC215/ALC285/ALC289
    - ALSA: hda - Fix headset mic detection problem for two Dell machines
    - ALSA: hda - Fix a wrong FIXUP for alc289 on Dell machines

  * CIFS SMB2/SMB3 does not work for domain based DFS (LP: #1747572)
    - CIFS: make IPC a regular tcon
    - CIFS: use tcon_ipc instead of use_ipc parameter of SMB2_ioctl
    - CIFS: dump IPC tcon in debug proc file

  * i2c-thunderx: erroneous error message "unhandled state: 0" (LP: #1754076)
    - i2c: octeon: Prevent error message on bus error

  * hisi_sas: Add disk LED support (LP: #1752695)
    - scsi: hisi_sas: directly attached disk LED feature for v2 hw

  * EDAC, sb_edac: Backport 1 patch to Ubuntu 17.10 (Fix missing DIMM sysfs
    entries with KNL SNC2/SNC4 mode) (LP: #1743856)
    - EDAC, sb_edac: Fix missing DIMM sysfs entries with KNL SNC2/SNC4 mode

  * [regression] Colour banding and artefacts appear system-wide on an Asus
    Zenbook UX303LA with Intel HD 4400 graphics (LP: #1749420)
    - drm/edid: Add 6 bpc quirk for CPT panel in Asus UX303LA

  * DVB Card with SAA7146 chipset not working (LP: #1742316)
    - vmalloc: fix __GFP_HIGHMEM usage for vmalloc_32 on 32b systems

  * [Asus UX360UA] battery status in unity-panel is not changing when battery is
    being charged (LP: #1661876) // AC adapter status not detected on Asus
    ZenBook UX410UAK (LP: #1745032)
    - ACPI / battery: Add quirk for Asus UX360UA and UX410UAK

  * ASUS UX305LA - Battery state not detected correctly (LP: #1482390)
    - ACPI / battery: Add quirk for Asus GL502VSK and UX305LA

  * support thunderx2 vendor pmu events (LP: #1747523)
    - perf pmu: Extract function to get JSON alias map
    - perf pmu: Pass pmu as a parameter to get_cpuid_str()
    - perf tools arm64: Add support for get_cpuid_str function.
    - perf pmu: Add helper function is_pmu_core to detect PMU CORE devices
    - perf vendor events arm64: Add ThunderX2 implementation defined pmu core
      events
    - perf pmu: Add check for valid cpuid in perf_pmu__find_map()

  * lpfc.ko module doesn't work (LP: #1746970)
    - scsi: lpfc: Fix loop mode target discovery

  * Ubuntu 17.10 crashes on vmalloc.c (LP: #1739498)
    - powerpc/mm/book3s64: Make KERN_IO_START a variable
    - powerpc/mm/slb: Move comment next to the code it's referring to
    - powerpc/mm/hash64: Make vmalloc 56T on hash

  * ethtool -p fails to light NIC LED on HiSilicon D05 systems (LP: #1748567)
    - net...

Is this also fixed in bionic yet?

On Tue, Apr 3, 2018 at 9:10 AM, Launchpad Bug Tracker
<email address hidden> wrote:
> This bug was fixed in the package linux - 4.13.0-38.43
>
> ---------------
> linux (4.13.0-38.43) artful; urgency=medium
>
> * linux: 4.13.0-38.43 -proposed tracker (LP: #1755762)
>
> * Servers going OOM after updating kernel from 4.10 to 4.13 (LP: #1748408)
> - i40e: Fix memory leak related filter programming status
> - i40e: Add programming descriptors to cleaned_count
>
> * [SRU] Lenovo E41 Mic mute hotkey is not responding (LP: #1753347)
> - platform/x86: ideapad-laptop: Increase timeout to wait for EC answer
>
> * fails to dump with latest kpti fixes (LP: #1750021)
> - kdump: write correct address of mem_section into vmcoreinfo
>
> * headset mic can't be detected on two Dell machines (LP: #1748807)
> - ALSA: hda/realtek - Support headset mode for ALC215/ALC285/ALC289
> - ALSA: hda - Fix headset mic detection problem for two Dell machines
> - ALSA: hda - Fix a wrong FIXUP for alc289 on Dell machines
>
> * CIFS SMB2/SMB3 does not work for domain based DFS (LP: #1747572)
> - CIFS: make IPC a regular tcon
> - CIFS: use tcon_ipc instead of use_ipc parameter of SMB2_ioctl
> - CIFS: dump IPC tcon in debug proc file
>
> * i2c-thunderx: erroneous error message "unhandled state: 0" (LP: #1754076)
> - i2c: octeon: Prevent error message on bus error
>
> * hisi_sas: Add disk LED support (LP: #1752695)
> - scsi: hisi_sas: directly attached disk LED feature for v2 hw
>
> * EDAC, sb_edac: Backport 1 patch to Ubuntu 17.10 (Fix missing DIMM sysfs
> entries with KNL SNC2/SNC4 mode) (LP: #1743856)
> - EDAC, sb_edac: Fix missing DIMM sysfs entries with KNL SNC2/SNC4 mode
>
> * [regression] Colour banding and artefacts appear system-wide on an Asus
> Zenbook UX303LA with Intel HD 4400 graphics (LP: #1749420)
> - drm/edid: Add 6 bpc quirk for CPT panel in Asus UX303LA
>
> * DVB Card with SAA7146 chipset not working (LP: #1742316)
> - vmalloc: fix __GFP_HIGHMEM usage for vmalloc_32 on 32b systems
>
> * [Asus UX360UA] battery status in unity-panel is not changing when battery is
> being charged (LP: #1661876) // AC adapter status not detected on Asus
> ZenBook UX410UAK (LP: #1745032)
> - ACPI / battery: Add quirk for Asus UX360UA and UX410UAK
>
> * ASUS UX305LA - Battery state not detected correctly (LP: #1482390)
> - ACPI / battery: Add quirk for Asus GL502VSK and UX305LA
>
> * support thunderx2 vendor pmu events (LP: #1747523)
> - perf pmu: Extract function to get JSON alias map
> - perf pmu: Pass pmu as a parameter to get_cpuid_str()
> - perf tools arm64: Add support for get_cpuid_str function.
> - perf pmu: Add helper function is_pmu_core to detect PMU CORE devices
> - perf vendor events arm64: Add ThunderX2 implementation defined pmu core
> events
> - perf pmu: Add check for valid cpuid in perf_pmu__find_map()
>
> * lpfc.ko module doesn't work (LP: #1746970)
> - scsi: lpfc: Fix loop mode target discovery
>
> * Ubuntu 17.10 crashes on vmalloc.c (LP: #1739498)
> - powerpc/mm/book3s64...

This bug was fixed in the package linux - 4.4.0-119.143

---------------
linux (4.4.0-119.143) xenial; urgency=medium

  * linux: 4.4.0-119.143 -proposed tracker (LP: #1760327)

  * Dell XPS 13 9360 bluetooth scan can not detect any device (LP: #1759821)
    - Revert "Bluetooth: btusb: fix QCA Rome suspend/resume"

linux (4.4.0-118.142) xenial; urgency=medium

  * linux: 4.4.0-118.142 -proposed tracker (LP: #1759607)

  * Kernel panic with AWS 4.4.0-1053 / 4.4.0-1015 (Trusty) (LP: #1758869)
    - x86/microcode/AMD: Do not load when running on a hypervisor

  * CVE-2018-8043
    - net: phy: mdio-bcm-unimac: fix potential NULL dereference in
      unimac_mdio_probe()

linux (4.4.0-117.141) xenial; urgency=medium

  * linux: 4.4.0-117.141 -proposed tracker (LP: #1755208)

  * Xenial update to 4.4.114 stable release (LP: #1754592)
    - x86/asm/32: Make sync_core() handle missing CPUID on all 32-bit kernels
    - usbip: prevent vhci_hcd driver from leaking a socket pointer address
    - usbip: Fix implicit fallthrough warning
    - usbip: Fix potential format overflow in userspace tools
    - x86/microcode/intel: Fix BDW late-loading revision check
    - x86/retpoline: Fill RSB on context switch for affected CPUs
    - sched/deadline: Use the revised wakeup rule for suspending constrained dl
      tasks
    - can: af_can: can_rcv(): replace WARN_ONCE by pr_warn_once
    - can: af_can: canfd_rcv(): replace WARN_ONCE by pr_warn_once
    - PM / sleep: declare __tracedata symbols as char[] rather than char
    - time: Avoid undefined behaviour in ktime_add_safe()
    - timers: Plug locking race vs. timer migration
    - Prevent timer value 0 for MWAITX
    - drivers: base: cacheinfo: fix x86 with CONFIG_OF enabled
    - drivers: base: cacheinfo: fix boot error message when acpi is enabled
    - PCI: layerscape: Add "fsl,ls2085a-pcie" compatible ID
    - PCI: layerscape: Fix MSG TLP drop setting
    - mmc: sdhci-of-esdhc: add/remove some quirks according to vendor version
    - fs/select: add vmalloc fallback for select(2)
    - hwpoison, memcg: forcibly uncharge LRU pages
    - cma: fix calculation of aligned offset
    - mm, page_alloc: fix potential false positive in __zone_watermark_ok
    - ipc: msg, make msgrcv work with LONG_MIN
    - x86/ioapic: Fix incorrect pointers in ioapic_setup_resources()
    - ACPI / processor: Avoid reserving IO regions too early
    - ACPI / scan: Prefer devices without _HID/_CID for _ADR matching
    - ACPICA: Namespace: fix operand cache leak
    - netfilter: x_tables: speed up jump target validation
    - netfilter: arp_tables: fix invoking 32bit "iptable -P INPUT ACCEPT" failed
      in 64bit kernel
    - netfilter: nf_dup_ipv6: set again FLOWI_FLAG_KNOWN_NH at flowi6_flags
    - netfilter: nf_ct_expect: remove the redundant slash when policy name is
      empty
    - netfilter: nfnetlink_queue: reject verdict request from different portid
    - netfilter: restart search if moved to other chain
    - netfilter: nf_conntrack_sip: extend request line validation
    - netfilter: use fwmark_reflect in nf_send_reset
    - ext2: Don't clear SGID when inheriting ACLs
    - reiserfs: fix race in prealloc discard
    - re...

This bug was fixed in the package linux - 4.15.0-15.16

---------------
linux (4.15.0-15.16) bionic; urgency=medium

  * linux: 4.15.0-15.16 -proposed tracker (LP: #1761177)

  * FFe: Enable configuring resume offset via sysfs (LP: #1760106)
    - PM / hibernate: Make passing hibernate offsets more friendly

  * /dev/bcache/by-uuid links not created after reboot (LP: #1729145)
    - SAUCE: (no-up) bcache: decouple emitting a cached_dev CHANGE uevent

  * Ubuntu18.04:POWER9:DD2.2 - Unable to start a KVM guest with default machine
    type(pseries-bionic) complaining "KVM implementation does not support
    Transactional Memory, try cap-htm=off" (kvm) (LP: #1752026)
    - powerpc: Use feature bit for RTC presence rather than timebase presence
    - powerpc: Book E: Remove unused CPU_FTR_L2CSR bit
    - powerpc: Free up CPU feature bits on 64-bit machines
    - powerpc: Add CPU feature bits for TM bug workarounds on POWER9 v2.2
    - powerpc/powernv: Provide a way to force a core into SMT4 mode
    - KVM: PPC: Book3S HV: Work around transactional memory bugs in POWER9
    - KVM: PPC: Book3S HV: Work around XER[SO] bug in fake suspend mode
    - KVM: PPC: Book3S HV: Work around TEXASR bug in fake suspend state

  * Important Kernel fixes to be backported for Power9 (kvm) (LP: #1758910)
    - powerpc/mm: Fixup tlbie vs store ordering issue on POWER9

  * Ubuntu 18.04 - IO Hang on some namespaces when running HTX with 16
    namespaces (Bolt / NVMe) (LP: #1757497)
    - powerpc/64s: Fix lost pending interrupt due to race causing lost update to
      irq_happened

  * fwts-efi-runtime-dkms 18.03.00-0ubuntu1: fwts-efi-runtime-dkms kernel module
    failed to build (LP: #1760876)
    - [Packaging] include the retpoline extractor in the headers

linux (4.15.0-14.15) bionic; urgency=medium

  * linux: 4.15.0-14.15 -proposed tracker (LP: #1760678)

  * [Bionic] mlx4 ETH - mlnx_qos failed when set some TC to vendor
    (LP: #1758662)
    - net/mlx4_en: Change default QoS settings

  * AT_BASE_PLATFORM in AUXV is absent on kernels available on Ubuntu 17.10
    (LP: #1759312)
    - powerpc/64s: Fix NULL AT_BASE_PLATFORM when using DT CPU features

  * Bionic update to 4.15.15 stable release (LP: #1760585)
    - net: dsa: Fix dsa_is_user_port() test inversion
    - openvswitch: meter: fix the incorrect calculation of max delta_t
    - qed: Fix MPA unalign flow in case header is split across two packets.
    - tcp: purge write queue upon aborting the connection
    - qed: Fix non TCP packets should be dropped on iWARP ll2 connection
    - sysfs: symlink: export sysfs_create_link_nowarn()
    - net: phy: relax error checking when creating sysfs link netdev->phydev
    - devlink: Remove redundant free on error path
    - macvlan: filter out unsupported feature flags
    - net: ipv6: keep sk status consistent after datagram connect failure
    - ipv6: old_dport should be a __be16 in __ip6_datagram_connect()
    - ipv6: sr: fix NULL pointer dereference when setting encap source address
    - ipv6: sr: fix scheduling in RCU when creating seg6 lwtunnel state
    - mlxsw: spectrum_buffers: Set a minimum quota for CPU port traffic
    - net: phy: Tell caller result ...

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-bionic' to 'verification-done-bionic'. If the problem still exists, change the tag 'verification-needed-bionic' to 'verification-failed-bionic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug was erroneously marked for verification in bionic; verification is not required and verification-needed-bionic is being removed.

For older releases such as bionic we have one event:

UDEV [1304.304437] change /devices/virtual/block/bcache0 (block)
.ID_FS_TYPE_NEW=
ACTION=change
CACHED_LABEL=
CACHED_UUID=0192e0a7-9d8e-4771-9bb1-d730a2f66a9d
DEVLINKS=/dev/bcache/by-uuid/0192e0a7-9d8e-4771-9bb1-d730a2f66a9d
DEVNAME=/dev/bcache0
DEVPATH=/devices/virtual/block/bcache0
DEVTYPE=disk
DRIVER=bcache
ID_FS_TYPE=
MAJOR=251
MINOR=0
SEQNUM=4384
SUBSYSTEM=block
TAGS=:systemd:
USEC_INITIALIZED=1304237024

that creates the expected symlink.

---------

For focal, instead we get two events:

UDEV [2763.370729] change /devices/virtual/block/bcache0 (block)
ACTION=change
DEVPATH=/devices/virtual/block/bcache0
SUBSYSTEM=block
DRIVER=bcache
CACHED_UUID=1522715e-a390-40db-8d47-06fb728014e5
CACHED_LABEL=
DEVNAME=/dev/bcache0
DEVTYPE=disk
SEQNUM=5523
USEC_INITIALIZED=2763323504
.ID_FS_TYPE_NEW=
ID_FS_TYPE=
MAJOR=251
MINOR=0
DEVLINKS=/dev/bcache/by-uuid/1522715e-a390-40db-8d47-06fb728014e5
TAGS=:systemd:

KERNEL[2784.362374] change /devices/virtual/block/bcache0 (block)
ACTION=change
DEVPATH=/devices/virtual/block/bcache0
SUBSYSTEM=block
SYNTH_UUID=0
DEVNAME=/dev/bcache0
DEVTYPE=disk
SEQNUM=5524
MAJOR=251
MINOR=0

UDEV [2784.433618] change /devices/virtual/block/bcache0 (block)
ACTION=change
DEVPATH=/devices/virtual/block/bcache0
SUBSYSTEM=block
SYNTH_UUID=0
DEVNAME=/dev/bcache0
DEVTYPE=disk
SEQNUM=5524
USEC_INITIALIZED=2763323504
ID_FS_UUID=30b28bee-6a1e-423d-9d53-32c78ba5454a
ID_FS_UUID_ENC=30b28bee-6a1e-423d-9d53-32c78ba5454a
ID_FS_VERSION=1.0
ID_FS_TYPE=ext4
ID_FS_USAGE=filesystem
.ID_FS_TYPE_NEW=ext4
MAJOR=251
MINOR=0
DEVLINKS=/dev/disk/by-uuid/30b28bee-6a1e-423d-9d53-32c78ba5454a
TAGS=:systemd:

This performs:

pr 21 14:15:22 ubuntu-focal systemd-udevd[1870]: bcache0: Processing device (SEQNUM=5523, ACTION=change)
Apr 21 14:15:22 ubuntu-focal systemd-udevd[1870]: bcache0: Removing watch
Apr 21 14:15:22 ubuntu-focal systemd-udevd[1870]: value '[dmi/id]sys_vendor' is 'QEMU'
Apr 21 14:15:22 ubuntu-focal systemd-udevd[1870]: value '[dmi/id]sys_vendor' is 'QEMU'
Apr 21 14:15:22 ubuntu-focal systemd-udevd[1870]: bcache0: /usr/lib/udev/rules.d/60-persistent-storage.rules:109 Importing properties from results of builtin command 'blkid'
Apr 21 14:15:22 ubuntu-focal systemd-udevd[1870]: bcache0: Probe /dev/bcache0 with raid and offset=0
Apr 21 14:15:22 ubuntu-focal systemd-udevd[1870]: bcache0: /usr/lib/udev/rules.d/69-bcache.rules:16 Importing properties from results of 'probe-bcache -o udev /dev/bcache0'
Apr 21 14:15:22 ubuntu-focal systemd-udevd[1870]: bcache0: Starting 'probe-bcache -o udev /dev/bcache0'
Apr 21 14:15:22 ubuntu-focal systemd-udevd[1870]: Successfully forked off '(spawn)' as PID 1912.
Apr 21 14:15:22 ubuntu-focal systemd-udevd[1870]: bcache0: Process 'probe-bcache -o udev /dev/bcache0' succeeded.
Apr 21 14:15:22 ubuntu-focal systemd-udevd[1870]: bcache0: /usr/lib/udev/rules.d/69-bcache.rules:27 LINK 'bcache/by-uuid/1522715e-a390-40db-8d47-06fb728014e5'
Apr 21 14:15:22 ubuntu-focal systemd-udevd[1870]: bcache0: Handling device node '/dev/bcache0', devnum=b251:0
Apr 21 14:15:22 ubuntu-focal systemd-udevd[1870]: bcache0: Preserve already existing symlink '/dev/block/251:0...

Perhaps we need some udev plumbing expertise to resolve this.

Ignore my above 2 comments, posted these to the wrong bug.

Anyone interested in this bug, please take a look at:

https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1861941

Good summary of the issue:

https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1861941/comments/27
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1861941/comments/47
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1861941/comments/56
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1861941/comments/62

And the fix made at bcache-tools for that bug (kernel patch made to fix this bug will asked to be removed soon, as that fix solves this issue).