curtin fails to setup bcache when unclean bcache from previous install exists

I was in hurry to get rolling, so didn't have enough time to collect more info. More robust reproducer could be created when I have some time.

> 4. quick erase all disks (erasing a few megabytes at the beginning and
the end of the drives with MAAS in my case)

Can your provide the curtin configuration of the first deployment and the
second?

In general, if maas is sending wipe: superblock

Curtin handles recursively wiping and clearing holders at each step of the
way.

The full curtin log would be helpful to see what the discovered block tree
looks like
curtin as curtin starts wiping devices.

On Thu, Sep 21, 2017 at 9:52 AM, Nobuto Murata <email address hidden>
wrote:

> I was in hurry to get rolling, so didn't have enough time to collect
> more info. More robust reproducer could be created when I have some
> time.
>
> --
> You received this bug notification because you are subscribed to curtin.
> Matching subscriptions: curtin-bugs-all
> https://bugs.launchpad.net/bugs/1718699
>
> Title:
> curtin fails to setup bcache when unclean bcache from previous install
> exists
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/curtin/+bug/1718699/+subscriptions
>

I can confirm this issue. Encountered it myself on one node. After booting to the rescue mode I found out that a bcache signature was still present.

ubuntu@jkt01z03nova003:~$ sudo blkid
/dev/sdd: UUID="bf3a9242-cb68-40b9-b2a3-d4c72894fb79" TYPE="bcache"

Interestingly, the commissioning process always succeeds while the deployment process consistently fails. During deployment there are multiple devices: /dev/sdd and /dev/sdf present.

Error messsage (verbose curtin log):
http://paste.ubuntu.com/25730909/

      finish: cmd-install/stage-partitioning/builtin/cmd-block-meta: FAIL: removing previous storage devices
        finish: cmd-install/stage-partitioning/builtin/cmd-block-meta: FAIL: curtin command block-meta
        Traceback (most recent call last):
          File "/curtin/curtin/commands/main.py", line 215, in main
            ret = args.func(args)
          File "/curtin/curtin/commands/block_meta.py", line 67, in block_meta
            meta_custom(args)
          File "/curtin/curtin/commands/block_meta.py", line 1186, in meta_custom
            clear_holders.clear_holders(disk_paths)
...
            return next(self.gen)
          File "/curtin/curtin/block/__init__.py", line 807, in exclusive_open
            fd = os.open(path, os.O_RDWR | os.O_EXCL)
        OSError: [Errno 16] Device or resource busy: '/dev/sdd'
        [Errno 16] Device or resource busy: '/dev/sdd'

curtin config + lsscsi in the rescue mode:
http://paste.ubuntu.com/25730923/

ubuntu@jkt01z03nova003:~$ sudo blkid
/dev/sdd: UUID="bf3a9242-cb68-40b9-b2a3-d4c72894fb79" TYPE="bcache"

After a certain point you may start to blame hardware which isn't correct.

`wipefs -a /dev/sdd` successfully removes the bcache signature and solves the issue.

Can you provide the curtin configuration sent to the end point?

For maas 2.0+

maas <session> machine get-curtin-config <system-id>

I have reproduced the issue on my test bed. Will attach some info here.

Disk layout settings in MAAS.

How to reproduce:

$ maas admin machine deploy retc7f
$ maas admin machine get-curtin-config retc7f > curtin-config.yaml

-> first deployment succeeds

$ maas admin machine release retc7f erase=true quick_erase=true
-> quick erase succeeds

$ maas admin machine deploy retc7f
$ maas admin machine get-curtin-config retc7f > curtin-config_after_erase.yaml

-> second deployment with the same configuration fails with:
An error occured handling 'vda-part2': OSError - [Errno 16] Device or resource busy: '/dev/vda2'

On Fri, Oct 13, 2017 at 1:34 PM, Nobuto Murata <email address hidden>
wrote:

> How to reproduce:
>
> $ maas admin machine deploy retc7f
> $ maas admin machine get-curtin-config retc7f > curtin-config.yaml
>
> -> first deployment succeeds
>
> $ maas admin machine release retc7f erase=true quick_erase=true
> -> quick erase succeeds
>

Do you have the log for this operation?

When I attempt to perform the quick-wipe on disks that have been configured
in bcache mode
that operation fails. I'm interested in if you're also seeing this.

>
> $ maas admin machine deploy retc7f
> $ maas admin machine get-curtin-config retc7f >
> curtin-config_after_erase.yaml
>

There is no change in config (which is good).

>
> -> second deployment with the same configuration fails with:
> An error occured handling 'vda-part2': OSError - [Errno 16] Device or
> resource busy: '/dev/vda2'
>

Do you have the entire log here? In the early part of the curtin install
process, curtin dumps
the block device tree layout w.r.t which devices has a holder (like
bcache). I'd like to see what curtin
saw on this second run.

> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1718699
>
> Title:
> curtin fails to setup bcache when unclean bcache from previous install
> exists
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/curtin/+bug/1718699/+subscriptions
>

OK. I've recreated this. Certainly a subtle bug w.r.t how bcache retains its claim to a device.

First, for a workaround, I believe if you *skip* the quick erase, maas/curtin will already issue wipe: superblock which will detect the previous bcache devices (they've not been wiped yet) and shut them down properly. Please see if that helps. Alternatively, modifying the size of the partitions between runs will prevent the bcache metadata from being found.

In terms of what's happening, here goes.

After the quick-erase, we've "hidden" two bcache devices, first a backing device on /dev/vda2, and second a cache set device on /dev/vdb.

The cache set device does not become active automatically has it's not paired with a backing device which would normally result in a bcacheN device being found on the system and the underlying devices (vda, vdb) would have block devices in /sys/class/block/<kname>/holders

However since they're not active, no holder is found and curtin then attempts to open the device exclusively. This fails as the device has a bcache cache set on the device and /sys/class/block/<kname>/bcache exists and /sys/class/block/<kname>/bcache/set points to the cache which needs to be stopped to release the underlying device.

The same is true for the backing device. These buried bcache devices are not found due to the partitioning data being wiped by the quick-erase, thus when curtin starts partitioning the device at the exact same offsets, the kernel then reads the superblock of the newly created partition and the kernel bcache drive detects the metadata and then "opens" the device and updates sysfs.

Currently working on some changes to curtin to handle this particular case.

This bug is believed to be fixed in curtin in 17.1. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

A fix for this bug is being SRU to Ubuntu 16.04 and 17.10 under bug 1743618.

I'm facing this bug on a 33-server openstack deployment using xenial-queens.
Curtin version: 18.1-17-gae48e86f-0ubuntu1~16.04.1

As per comment #14, the fix seems to be applied to version 17.1. Therefore, I can point that the problem still persists.

The failure is intermittent, meaning generally MAAS seems to correctly release servers.
But, sometimes, servers get stuck on pending during deployment and fail all subsequent deployments with errors similar to: https://pastebin.canonical.com/p/w6CJXBtHTf/

Applying the fix: maas <session> machine release <system_id> erase=true
(without quick_erase) solves the issue but is very slow.

Please attach curtin verbose logs and your curtin config.

https://discourse.maas.io/t/getting-curtin-debug-logs/169

@raharper
Please, reference to: https://bugs.launchpad.net/curtin/+bug/1815018
I am adding all the info I've collected there on that bug report instead