Cryptoswap not working in Bionic

The code that I used to force failure, memtest.c, is attached to the bug report.

This change was made by a bot.

It seems that cryptswap is working correctly on 17.10 with the kernel 4.13.0-36-generic on a different machine using the same test as described above. I tried the same kernel (4.13.0-36) on 18.04 (machine used in original post) and it still fails. Of course, I checked that the swap appears in /proc/meminfo. Hope this provides a clue.

[FYI, I'm using 4.13.0-36 rather than 4.13.0-38 because of a bad firmware interaction with 4.13.0-37 and 4.13.0-38 that results in CPU lockups]

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v4.16 kernel[0].

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.16

Yes, the problem does occur in mainline 4.16 and I have added the requested tag.

If anything, 4.16.1 seems slower than 4.15 release kernel even for no cryptswap. It could be that my disk is too slow for cryptswap, but it did work in 17.10.

The obvious work around is not to use cryptswap, of course.

Looks like the same issue as reported on lp:#1736072

I'd like to perform a bisect to figure out what commit caused this regression. We need to identify the earliest kernel where the issue started happening as well as the latest kernel that did not have this issue.

Can you test the following kernels and report back? We are looking for the first kernel version that exhibits this bug:

v4.14 Final: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.14/
v4.15-rc1: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.15-rc1/
v4.15-rc4: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.15-rc4/
v4.15 Final: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.15/

You don't have to test every kernel, just up until the kernel that first has this bug.

Thanks in advance!

The first kernel on that list that exhibits the bug is the first one:
4.14 Final.  Recall: the bug results in a full up hang.  The disk stops
swapping (it's audible).   I can reboot with sysctl SUB however.

I then tested 4.13.0-36 from the Ubuntu 17.10 respoitory, followed by
4.11.12-041112-generic from mainline.  Both had the bug.

I also tested each of these without cryptswap (i.e. normal swap) and
they did not hang.  The system ground to a halt, but I could hear the
disk swapping and the process eventually completed.

I then booted an 16.04 live usb which has kernel 4.10.  I had to make a
cryptswap using cryptsetup on the swap partition of the hard disk, to
test it.   But this one worked! The behavior was a bit different that
more recent kernels: after allocating 6 GB of the 10 GB that I
requested, the system invoked the oom-killer on the process.  However,
it did not hang.

I then grabbed 4.10.1-041001-generic from mainline and tested it.  This
worked the same as the live usb.  So 4.10 does not have the bug,
although the oom-killer is invoked to kill the process grabbing the memory.

I have an older machine with an SSD.  This does not exhibit the bug on
Ubuntu 17.10 running 4.13.0-36.  However, this disk is _fast_ (Samsung
EVO), so I'm guessing that there is some tuning problem, maybe, in how
the kernel handles swap speed vs memory allocation requests???  I really
don't know.

--M

On 04/11/2018 11:18 AM, Joseph Salisbury wrote:
> I'd like to perform a bisect to figure out what commit caused this
> regression. We need to identify the earliest kernel where the issue
> started happening as well as the latest kernel that did not have this
> issue.
>
> Can you test the following kernels and report back? We are looking for
> the first kernel version that exhibits this bug:
>
> v4.14 Final: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.14/
> v4.15-rc1: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.15-rc1/
> v4.15-rc4: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.15-rc4/
> v4.15 Final: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.15/
>
> You don't have to test every kernel, just up until the kernel that first
> has this bug.
>
> Thanks in advance!
>
> ** Tags added: performing-bisect
>

--
Martin Weinberg
6 Grass Hill Rd
West Whately, MA
010039

BTW, I _do_ realize that you want a starting point for bisection; I'm
sorry this is turning out so messy.

On 04/11/2018 05:51 PM, Martin Weinberg wrote:
> The first kernel on that list that exhibits the bug is the first one:
> 4.14 Final.  Recall: the bug results in a full up hang.  The disk stops
> swapping (it's audible).   I can reboot with sysctl SUB however.
>
> I then tested 4.13.0-36 from the Ubuntu 17.10 respoitory, followed by
> 4.11.12-041112-generic from mainline.  Both had the bug.
>
> I also tested each of these without cryptswap (i.e. normal swap) and
> they did not hang.  The system ground to a halt, but I could hear the
> disk swapping and the process eventually completed.
>
> I then booted an 16.04 live usb which has kernel 4.10.  I had to make a
> cryptswap using cryptsetup on the swap partition of the hard disk, to
> test it.   But this one worked! The behavior was a bit different that
> more recent kernels: after allocating 6 GB of the 10 GB that I
> requested, the system invoked the oom-killer on the process.  However,
> it did not hang.
>
> I then grabbed 4.10.1-041001-generic from mainline and tested it.  This
> worked the same as the live usb.  So 4.10 does not have the bug,
> although the oom-killer is invoked to kill the process grabbing the memory.
>
> I have an older machine with an SSD.  This does not exhibit the bug on
> Ubuntu 17.10 running 4.13.0-36.  However, this disk is _fast_ (Samsung
> EVO), so I'm guessing that there is some tuning problem, maybe, in how
> the kernel handles swap speed vs memory allocation requests???  I really
> don't know.
>
> --M
>
>
> On 04/11/2018 11:18 AM, Joseph Salisbury wrote:
>> I'd like to perform a bisect to figure out what commit caused this
>> regression. We need to identify the earliest kernel where the issue
>> started happening as well as the latest kernel that did not have this
>> issue.
>>
>> Can you test the following kernels and report back? We are looking for
>> the first kernel version that exhibits this bug:
>>
>> v4.14 Final: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.14/
>> v4.15-rc1: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.15-rc1/
>> v4.15-rc4: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.15-rc4/
>> v4.15 Final: http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.15/
>>
>> You don't have to test every kernel, just up until the kernel that first
>> has this bug.
>>
>> Thanks in advance!
>>
>> ** Tags added: performing-bisect
>>

--
Martin Weinberg
6 Grass Hill Rd
West Whately, MA
010039

In response to #7, I don't think this is the same issue. The cryptswap initiates with no difficulties, timeouts, etc. on boot. When I first installed Bionic, ubiquity did not correctly install the swap, and I filed #1759253 on that issue.

Thanks for finding the mainline 4.10 does not have the bug and 4.11 final does. Can you also test v4.11-rc1, so we can find which release candidate introduced the bug?

http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.11-rc1

Some good and bad news: I have been testing this in a running Gnome 3 environment. I decided to start testing on console. On console, all kernels seem to fail using crypt swap. This suggests to me that failure must depend on current memory usage, dirty ratio, or some such runtime condition.

However, the good news, I have kernel messages on console. I'm attaching a screen shot, since none of them get written to syslog. A typical message is: task dm_write:xxx blocked for more than 120 seconds

There are no problems when not using dm_crypt for swap, i.e. 'normal' swap.

At the risk of saying something completely unrelated; any change this is a regression of #1310058 where the UUID got overwritten? It came back a few times around 2015, leading me to believe that the devs don't actually use this method, and might not test for this scenario, making a regression possible.

-update-

https://bugs.launchpad.net/ubuntu/+source/ecryptfs-utils/+bug/1310058 that is.