Comment 0 for bug 1926591

Brief Description
-----------------

After restore playbook was executed, the unlock failed because docker-lv resize was not successful.

Severity
--------

Critical

Steps to Reproduce
------------------

Install a system

Backup it

Run restore playbook

Unlock controller

Expected Behavior
------------------

Unlock succeeds

Actual Behavior
----------------

Unlock fails

Reproducibility
---------------

100%

System Configuration
--------------------

AIO-SX, AIO-DX

Branch/Pull Time/Commit
-----------------------

stx master build on "2021-03-21"

Last Pass
---------

2021-04-15 worked, did not test between 15-20

Timestamp/Logs
--------------

Puppet wanted to resize docker-lv from 20G to 30G and failed at unlock:

2021-04-22T04:59:09.337 Info: 2021-04-22 04:59:09 +0000 Logical_volume[docker-lv](provider=lvm): Current: value=20.0, unit=G, kibi=20971520
2021-04-22T04:59:09.342 Info: 2021-04-22 04:59:09 +0000 Logical_volume[docker-lv](provider=lvm): New: value=30.0, unit=G, kibi=31457280
2021-04-22T04:59:09.345 Debug: 2021-04-22 04:59:09 +0000 Executing: '/usr/sbin/lvs --noheading -o vg_extent_size --units k /dev/cgts-vg/docker-lv'
2021-04-22T04:59:09.363 Debug: 2021-04-22 04:59:09 +0000 Executing: '/usr/sbin/lvextend -L 31457280k /dev/cgts-vg/docker-lv'
2021-04-22T04:59:09.417 Debug: 2021-04-22 04:59:09 +0000 Executing: 'umount /dev/cgts-vg/docker-lv'
2021-04-22T04:59:09.425 Debug: 2021-04-22 04:59:09 +0000 Executing: 'fsadm -y check /dev/cgts-vg/docker-lv'
2021-04-22T04:59:09.471 Debug: 2021-04-22 04:59:09 +0000 Executing: 'fsadm -y resize /dev/cgts-vg/docker-lv 31457280k'
2021-04-22T04:59:09.507 Notice: 2021-04-22 04:59:09 +0000 /Stage[main]/Platform::Filesystem::Docker/Platform::Filesystem[docker-lv]/Logical_volume[docker-lv]/size: size changed '20G' to '30G'
2021-04-22T04:59:09.512 Debug: 2021-04-22 04:59:09 +0000 /Stage[main]/Platform::Filesystem::Docker/Platform::Filesystem[docker-lv]/Logical_volume[docker-lv]: The container Platform::Filesystem[docker-lv] will propagate my refresh event
2021-04-22T04:59:09.516 Debug: 2021-04-22 04:59:09 +0000 Class[Platform::Lvm::Vg::Cgts_vg]: The container Stage[main] will propagate my refresh event
2021-04-22T04:59:09.520 Debug: 2021-04-22 04:59:09 +0000 Executing: '/usr/sbin/lvs cgts-vg'
2021-04-22T04:59:09.536 Debug: 2021-04-22 04:59:09 +0000 Executing: '/usr/sbin/lvs --noheading --unit g /dev/cgts-vg/etcd-lv'
2021-04-22T04:59:09.560 Debug: 2021-04-22 04:59:09 +0000 Executing: '/usr/sbin/lvs cgts-vg'
2021-04-22T04:59:09.584 Debug: 2021-04-22 04:59:09 +0000 Executing: '/usr/sbin/lvs --noheading --unit g /dev/cgts-vg/kubelet-lv'
2021-04-22T04:59:09.611 Debug: 2021-04-22 04:59:09 +0000 Executing: '/usr/sbin/lvs --noheading --unit g /dev/cgts-vg/kubelet-lv'
2021-04-22T04:59:09.632 Info: 2021-04-22 04:59:09 +0000 Logical_volume[kubelet-lv](provider=lvm): Current: value=2.0, unit=G, kibi=2097152
2021-04-22T04:59:09.638 Info: 2021-04-22 04:59:09 +0000 Logical_volume[kubelet-lv](provider=lvm): New: value=10.0, unit=G, kibi=10485760
2021-04-22T04:59:09.642 Debug: 2021-04-22 04:59:09 +0000 Executing: '/usr/sbin/lvs --noheading -o vg_extent_size --units k /dev/cgts-vg/kubelet-lv'
2021-04-22T04:59:09.653 Debug: 2021-04-22 04:59:09 +0000 Executing: '/usr/sbin/lvextend -L 10485760k /dev/cgts-vg/kubelet-lv'
2021-04-22T04:59:09.710 Debug: 2021-04-22 04:59:09 +0000 Executing: 'umount /dev/cgts-vg/kubelet-lv'
2021-04-22T04:59:10.060 Debug: 2021-04-22 04:59:10 +0000 Executing: 'fsadm -y check /dev/cgts-vg/kubelet-lv'
2021-04-22T04:59:10.166 Debug: 2021-04-22 04:59:10 +0000 Executing: 'fsadm -y resize /dev/cgts-vg/kubelet-lv 10485760k'
2021-04-22T04:59:10.355 Notice: 2021-04-22 04:59:10 +0000 /Stage[main]/Platform::Filesystem::Kubelet/Platform::Filesystem[kubelet-lv]/Logical_volume[kubelet-lv]/size: size changed '2G' to '10G'
2021-04-22T04:59:10.359 Debug: 2021-04-22 04:59:10 +0000 /Stage[main]/Platform::Filesystem::Kubelet/Platform::Filesystem[kubelet-lv]/Logical_volume[kubelet-lv]: The container Platform::Filesystem[kubelet-lv] will propagate my refresh event
2021-04-22T04:59:10.364 Debug: 2021-04-22 04:59:10 +0000 Exec[wipe start of device kubelet-lv](provider=posix): Executing check 'test ! -e /etc/platform/.kubelet-lv'
2021-04-22T04:59:10.367 Debug: 2021-04-22 04:59:10 +0000 Executing: 'test ! -e /etc/platform/.kubelet-lv'
2021-04-22T04:59:10.371 Debug: 2021-04-22 04:59:10 +0000 Exec[wipe end of device kubelet-lv](provider=posix): Executing check 'test ! -e /etc/platform/.kubelet-lv'
2021-04-22T04:59:10.376 Debug: 2021-04-22 04:59:10 +0000 Executing: 'test ! -e /etc/platform/.kubelet-lv'
2021-04-22T04:59:10.379 Debug: 2021-04-22 04:59:10 +0000 Exec[mark lv as wiped kubelet-lv:](provider=posix): Executing check 'test ! -e /etc/platform/.kubelet-lv'
2021-04-22T04:59:10.382 Debug: 2021-04-22 04:59:10 +0000 Executing: 'test ! -e /etc/platform/.kubelet-lv'
2021-04-22T04:59:10.385 Debug: 2021-04-22 04:59:10 +0000 Exec[wipe start of device docker-lv](provider=posix): Executing check 'test ! -e /etc/platform/.docker-lv'
2021-04-22T04:59:10.389 Debug: 2021-04-22 04:59:10 +0000 Executing: 'test ! -e /etc/platform/.docker-lv'
2021-04-22T04:59:10.394 Debug: 2021-04-22 04:59:10 +0000 Executing: '/usr/sbin/lvs cgts-vg'
2021-04-22T04:59:10.415 Debug: 2021-04-22 04:59:10 +0000 Executing: '/usr/sbin/lvs --noheading --unit g /dev/cgts-vg/extension-lv'
2021-04-22T04:59:10.441 Debug: 2021-04-22 04:59:10 +0000 Exec[wipe end of device docker-lv](provider=posix): Executing check 'test ! -e /etc/platform/.docker-lv'
2021-04-22T04:59:10.445 Debug: 2021-04-22 04:59:10 +0000 Executing: 'test ! -e /etc/platform/.docker-lv'
2021-04-22T04:59:10.457 Debug: 2021-04-22 04:59:10 +0000 Exec[mark lv as wiped docker-lv:](provider=posix): Executing check 'test ! -e /etc/platform/.docker-lv'
2021-04-22T04:59:10.466 Debug: 2021-04-22 04:59:10 +0000 Executing: 'test ! -e /etc/platform/.docker-lv'
2021-04-22T04:59:10.470 Debug: 2021-04-22 04:59:10 +0000 Executing: '/usr/sbin/blkid /dev/cgts-vg/docker-lv'
2021-04-22T04:59:10.476 Debug: 2021-04-22 04:59:10 +0000 Executing: 'mkfs.xfs /dev/cgts-vg/docker-lv -n ftype=1'
2021-04-22T04:59:10.485 Error: 2021-04-22 04:59:10 +0000 Execution of 'mkfs.xfs /dev/cgts-vg/docker-lv -n ftype=1' returned 1: mkfs.xfs: /dev/cgts-vg/docker-lv contains a mounted filesystem
2021-04-22T04:59:10.489 Usage: mkfs.xfs
2021-04-22T04:59:10.492 /* blocksize */ [-b log=n|size=num]

Firstly, the resize should not be triggered here, in an older load (2021-04-15 )the docker-lv was 30G before unlock.

For some reason docker-lv fs type is drbd and I think it's wrong, it should be xfs:

controller-0:/var/log# blkid /dev/cgts-vg/docker-lv
/dev/cgts-vg/docker-lv: UUID="c5c72dc8a5af335b" TYPE="drbd"

Possible similar bug: https://bugs.launchpad.net/starlingx/+bug/1883825

Test Activity
-------------

Developer Testing