LVM in 16.04 cannot mount LVM Cached Volumes created in 15.10

Here is the package info for LVM2 on LDME Betsy:

dpkg-query -l lvm2
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-============-============-=================================
ii lvm2 2.02.111-2.2 amd64 Linux Logical Volume Manager

aptitude show lvm2
Package: lvm2
State: installed
Automatically installed: no
Multi-Arch: foreign
Version: 2.02.111-2.2
Priority: optional
Section: admin
Maintainer: Debian LVM Team <email address hidden>
Architecture: amd64
Uncompressed Size: 1,663 k
Depends: libc6 (>= 2.15), libdevmapper-event1.02.1 (>= 2:1.02.74),
         libdevmapper1.02.1 (>= 2:1.02.90), libreadline5 (>= 5.2), libudev1 (>=
         183), init-system-helpers (>= 1.18~), lsb-base, dmsetup (> 2:1.02.47),
         dmeventd (> 2:1.02.90), initscripts (>= 2.88dsf-13.3)
Suggests: thin-provisioning-tools
Conflicts: lvm2
Breaks: initramfs-tools (< 0.99), initramfs-tools (< 0.99),
        thin-provisioning-tools (< 0.3.2), thin-provisioning-tools (< 0.3.2)
Provides: lvm2
Provided by: lvm2
Description: Linux Logical Volume Manager
 This is LVM2, the rewrite of The Linux Logical Volume Manager. LVM supports
 enterprise level volume management of disk and disk subsystems by grouping
 arbitrary disks into volume groups. The total capacity of volume groups can be
 allocated to logical volumes, which are accessed as regular block devices.
Homepage: http://sources.redhat.com/lvm2/

Tags: admin::filesystem, interface::commandline, role::program, scope::utility,
      uitoolkit::ncurses

Hmmm... I wonder whether thin-provisioning-tools was missing on the Ubuntu Xenial install ISO. Here's that package:

aptitude show thin-provisioning-tools
Package: thin-provisioning-tools
State: installed
Automatically installed: no
Version: 0.3.2-1
Priority: optional
Section: admin
Maintainer: Debian LVM Team <email address hidden>
Architecture: amd64
Uncompressed Size: 17.1 M
Depends: libc6 (>= 2.14), libexpat1 (>= 2.0.1), libgcc1 (>= 1:4.1.1), libstdc++6
         (>= 4.6)
Conflicts: thin-provisioning-tools
Description: Tools for handling thinly provisioned device-mapper meta-data
 This package contains tools to handle meta-data from the device-mapper thin
 target. This target allows the use of a single backing store for multple
 thinly provisioned volumes. Numerous snapshots can be taken from such volumes.
 The tools can check the meta-data for consistency, repair damaged information
 and dump or restore the meta-data in textual form.

I vaguely recall having to manually install that when I set up these LVM cache's.

Installing the thin-provisioning-tools after booting the Live USB did not help. I still get the same error when checking the LVM partition status with 'pvs'.

Status changed to 'Confirmed' because the bug affects multiple users.

As I understand it, this bug affects any cached volumes created using the lvm2 package from 15.10 and then attempting to mount that volume under 16.04.

I have this issue on two machines. As a workaroundon one of the machines I was able to mount the volume after booting into a 15.10 live environment, converting the lvs back to uncached volumes, booting back to 16.04 and converting to cached volumes again.

Looking at the Fedora bug report this is caused by new cache metadata attributes missing in the metadata of volumes created using previous lvm2 version. A patch exists that adds default attributes to the cache metadata if they don't already exists and this will resolve the issue.

Link to patch: https://www.redhat.com/archives/lvm-devel/2015-November/msg00098.html

I'll check upstream to see if a bugreport exists for this lvm2 package.

At the very least, there should be a caveat in the 16.04 upgrade notes to state that any cached lvm volumes should have the caches removed before the upgrade.

The patch referenced above does not have a related bug report in the upstream project.

The earliest release version the patch is included in is lvm2 2.02.135

I've restored my system to a working state by building lvm2 from upstream source that includes the referenced patch and using the compiled lvm to do a config backup which added the missing metadata attributes then using native lvm (lvm2 2.02.133-1ubuntu7) to restore the config backup.

All is working now using lvm2 2.02.133-1ubuntu7 package.

guy@mediaserver:~/tools$ sudo lvs
  LV lv_cache has invalid cache's feature flag.
  LV lv_cache is missing cache policy name.
  Internal error: LV segments corrupted in lv_cache.

guy@mediaserver:~/tools$ sudo ./lvm vgcfgbackup -v
    Using volume group(s) on command line.
    Cache is missing cache policy, using mq.
    Cache is missing cache mode, using writethrough.
    Creating volume group backup "/etc/lvm/backup/vg1" (seqno 13).
  Volume group "vg1" successfully backed up.

guy@mediaserver:~/tools$ sudo vgcfgrestore vg1
  Restored volume group vg1

guy@mediaserver:~/tools$ sudo lvs
  LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert
  media vg1 Cwi---C--- 500.00g [lv_cache] [media_corig]

The suggested workaround in https://bugs.launchpad.net/ubuntu/+source/lvm2/+bug/1556602/comments/8 did not work for me. What did work however was to build a modified lvm2 package using the aforementioned patch, then boot from external media, install the modified lvm2 on the externally booted ubuntu, then mount the system's main fs using the modified lvm, then install the modified lvm2 again this time on the system's fs using chroot.

This process is cumbersome and I wish ubuntu would include this patch by default.

LVM caching enabled in Ubuntu 16.04 (or, to be completely honest, Linux Mint 18) is also incompatible with the older caching implementation in Linux Mint Debian Edition 2 (i.e. Debian Jessie). I hoped the following procedure would create an LVM cache that works for both Jessie and Xenial:

1) Under Jessie, remove the cache from the LVM volume.
2) Reboot into Xenial and add an LVM cache to the volume. (This works fine).
3) Reboot back to Jessie to use the LVM volume with caching created under Xenial.

My goal is to support dual boot between jessie and xenial with LVM caching enabled on my data volumes.

This doesn't work. The LVM volume comes up a disabled or offline under Jessie and there's no apparent way to bring it up without removing the cache. (I can easily recreate this on my scratch/test machine in case there's useful data to be gathered).

So, LVM caching between debian jessie and ubuntu xenial is neither forward or backwards compatible.