bcache makes the whole io system hang after long run time

Problem still exists on 4.4.0-97-generic. And in my attempt to reproduce it on a VM, I got a corrupt kernel instead, which sounds worse (could create some persistent damage, not just a hang).

Attached are the 3 scripts to reproduce the corrupt kernel, and an extra one to attach if the first failed.

- Make a VM with at least 2GB RAM.
- Run crash-setup first (it uses a loop device backed by something in /dev/shm so it doesn't have to wear out an SSD).
- Then verify the cache is attached, or attach it if not.
    Output of these 2 should match:
        bcache-super-show /dev/loop0 | awk '$1=="cset.uuid"{print $2}'
        basename $(readlink /sys/block/bcache0/bcache/cache)
    This should say [writeback]:
        cat /sys/block/bcache0/bcache/cache_mode
    eg. writethrough [writeback] writearound none
- Then run the setup-grep and the setup-fio at the same time, like in screen or 2 terminals. The grep is not intended to be only information output... I strongly believe that reading the files in sysfs causes crashing, as I have caused hangs and other problems many times this way in the
past.
- wait a while... could be as soon as 20 minutes, or some hours.

So far I made this VM die 3 times... it wasn't possible ot get the console output or dmesg as I wanted and I didn't save a screenshot either the first 2 times... and the 3rd time also failed, even with a serial console set up, so attached is a screenshot.

tested again with a better serial console setup, and here's the text version like that screenshot:

[ 744.779536] Kernel panic - not syncing: stack-protector: Kernel stack is corrupted in: ffffffffc00d3ccc
[ 744.779536]
[ 744.780013] CPU: 0 PID: 5087 Comm: grep Not tainted 4.4.0-97-generic #120~14.04.1-Ubuntu
[ 744.780013] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.10.3-20171021_125229-anatol 04/01/2014
[ 744.780013] 0000000000000000 ffff8800791afc48 ffffffff813df51c ffffffff81cbae98
[ 744.780013] 000000ad5f71c8e7 ffff8800791afcc0 ffffffff8118356c 000000ad00000010
[ 744.780013] ffff8800791afcd0 ffff8800791afc70 ffff8800367a6b40 ffffffffc00d3ccc
[ 744.780013] Call Trace:
[ 744.780013] [<ffffffff813df51c>] dump_stack+0x63/0x87
[ 744.780013] [<ffffffff8118356c>] panic+0xc8/0x20f
[ 744.786378] [<ffffffffc00d3ccc>] ? __bch_cached_dev_show+0x4ac/0x4b0 [bcache]
[ 744.786877] [<ffffffff8107e82b>] __stack_chk_fail+0x1b/0x20
[ 744.786877] [<ffffffffc00d3ccc>] __bch_cached_dev_show+0x4ac/0x4b0 [bcache]
[ 744.786877] [<ffffffff8120fe69>] ? path_openat+0x2e9/0x12d0
[ 744.786877] [<ffffffffc00d3d01>] bch_cached_dev_show+0x31/0x50 [bcache]
[ 744.786877] [<ffffffff8127f822>] sysfs_kf_seq_show+0xc2/0x1a0
[ 744.786877] [<ffffffff8127e043>] kernfs_seq_show+0x23/0x30
[ 744.786877] [<ffffffff812255bb>] seq_read+0xeb/0x360
[ 744.786877] [<ffffffff8127e7dd>] kernfs_fop_read+0x10d/0x170
[ 744.786877] [<ffffffff81201b18>] __vfs_read+0x18/0x40
[ 744.786877] [<ffffffff812020cf>] vfs_read+0x7f/0x130
[ 744.793893] [<ffffffff81202ea6>] SyS_read+0x46/0xa0
[ 744.794753] [<ffffffff8180f4f6>] entry_SYSCALL_64_fastpath+0x16/0x75
[ 744.795617] Kernel Offset: disabled
[ 744.795617] ---[ end Kernel panic - not syncing: stack-protector: Kernel stack is corrupted in: ffffffffc00d3ccc
[ 744.795617]

Hi there was a slight issue in the repro relying on sda5, I changed that to be on shm as well.
2x800M are easy to spare. Hope it triggers with that as well.

# Get a 3G guest with full console logging
$ uvt-simplestreams-libvirt --verbose sync --source http://cloud-images.ubuntu.com/daily arch=amd64 label=daily release=xenial
$ uvt-kvm create --password=ubuntu --log-console-output --memory 3096 xenial-bcache-test release=xenial arch=amd64 label=daily

# setup guest into the crash in the guest
$ sudo apt update
$ sudo apt install fio
$ wget https://bugs.launchpad.net/ubuntu/+source/linux-lts-xenial/+bug/1724173/+attachment/5008126/+files/bcachecrash.tgz
$ tar xvf bcachecrash.tgz
# Add in the fixed version of crash-setup from http://paste.ubuntu.com/26063669/
$ sudo ./bcachecrash/crash-setup
$ sudo ./bcachecrash/crash-attach

# now on two consoles
$ sudo ./bcachecrash/crash-grep
$ sudo ./bcachecrash/crash-fio

Not sure if it triggers the bug, but with those fixes it runs ...
Will let you know if it crashed the kernel before I needed to recycle the machine.

This one sets up a slower device which might be important according to IRC discussions
http://paste.ubuntu.com/26063765/

Status changed to 'Confirmed' because the bug affects multiple users.

We're also seeing this with 4.4.0-111-generic (on Trusty), and a very similar hardware profile. The boxes in question are running Swift with a large (millions) number of objects all approx 32k in size.

I'm currently fio'ing in a test environment to try to reproduce this away from production.

@xavpaice Is anyone else with the problem also using isdct (Intel® SSD Data Center Tool for NVMe)? I have not had the problem since disabling it from running regularly. I was using it to check the wear level on regular intervals before.

And the reason I thought of that was that finally one time I was able to run a few shell commands shortly after a hang, and the only things that would hang were the ones directly related to the NVMe (surely tested isdct, probably tested parted, dd, etc.). (and then eventually everything was hung as normal)