Kernel Panic with linux-image-4.15.0-60-generic when specifying nameserver in docker-compose

Status changed to 'Confirmed' because the bug affects multiple users.

Tested using the docker-compose.yml with original docker-compose.yml with every container but unbound using -dns.

Test1: Crashed (took less than 1 minutes to crash)
Fully updated ubuntu, using Scaleway's ubuntu image. (4.15.0-60-generic)
Crash: https://gist.github.com/FingerlessGlov3s/f829642cfc6390c975ca89be3b0d685a

Test2: Crashed (took less than 5 seconds to crash)
Fully updated ubuntu, using Scaleway's ubuntu image. (4.15.0-60-generic)
Also turned off appamour, SELinux not installed (systemctl disable apparmor.service and rebooted)
Crash: https://gist.github.com/FingerlessGlov3s/fe3b5e6b89986b397e20bc5474049584

Test3: Working
Fully updated ubuntu, using Scaleway's ubuntu image. using old kernel (4.15.0-58-generic)

Just ask if you want any more information or testing.

We were just affected by this bug on one of our server. I was able to get some information from the kernel log (see attachment).

It appears to crash at net/ipv4/ip_output.c:636:

Sep 4 07:54:04 gonzo kernel: [ 145.912955] kernel BUG at /build/linux-5mCauq/linux-4.15.0/net/ipv4/ip_output.c:636!
Sep 4 07:54:04 gonzo kernel: [ 146.005576] invalid opcode: 0000 [#1] SMP PTI

I was affected by this (or a very similar bug), but it only happened on the first HTTP request to a docker-compose managed docker container.

Thanks!!!

Thank you for fixing this, most appreciated.

When will the fix be available on the repos? On the next kernel release?

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-bionic' to 'verification-done-bionic'. If the problem still exists, change the tag 'verification-needed-bionic' to 'verification-failed-bionic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

Fixed it for me (via 4.15.0-62-generic).

I can also confirm that it's now working. Thanks a lot for the fast fix, @cascardo!

Oh. I was able to set the status to released? :-D Sorry. Not released yet.

We updated last night (4.15.0-60) and started having issues this morning... issue has been fixed with update to 5.0.0.27 HWE. We’re up and running again, but trying to figure out where the issue stems from. Is the issue caused from a bug or an attack?

It's possible this same issue is responsible for this crash in WireGuard: https://lists.zx2c4.com/pipermail/wireguard/2019-September/004495.html

The last status change happened on accident. The fix has not been released, yet. I cannot change the status back to "Fix Committed" though. Can anyone help with this?

This bug has to be dealt with a higher degree of urgency. It impacts any machine that performs any type of NAT. Is not application specific though.

I agree with Taher (in https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1842447/comments/15), this bug seems to impact a lot of systems (my colo host was kernel panic restarting about every 75-90 minutes, all weekend). It has a NAT firewall on it (for the hosted VMs), but no Docker/Wireguard, etc. My guess for the 75-90 minutes is that is how long it took to dirty enough memory that the relevant value just happened not to already be 0 (NULL). (Curiously I installed -60 on Thursday last week, and the first issue didn't happen until Friday, so it's been worse over the weekend than the first 24 hours. I enabled kernel.panic=15 after the first issue, to automate recovery, and was *extremely* glad I did so.)

Honestly I'd suggest withdrawing -60 as it's very unstable in a lot of common configurations. And also suggest expediting the release of -62, which AFAICT just contains the one line fix for the bug in -60.

Now it's Monday morning (and thus I can get into the colo if needed), I've upgraded the colo system to the proposed -62 version, and crossing my fingers the system is more stable as a result.

In case it helps others, I found I needed to:

(a) https://wiki.ubuntu.com/Testing/EnableProposed (changing "xenial" to "bionic", for 18.04 LTS, including enabling the low priority pin of bionic-proposed); and

(b) sudo apt-get install linux-generic/bionic-proposed linux-signed-generic/bionic-proposed linux-headers-generic/bionic-proposed

(without at least two of those three, the proposed update metapackages wouldn't install due to conflicts; I'm not sure if linux-signed-generic is needed, but it's still installed, so I chose to keep it in sync.)

That list of packages found by looking for 4.15.0-60 versioned packages that didn't have that version in their package name (ie, to find the generic metapackages).

Ewen

PS: Reboots (due to kernel panic, and kernel.panic=15 sysctl) over the weekend:

-=- cut here -=-
ewen@naosr620:~$ last | grep reboot
reboot system boot 4.15.0-62-generi Mon Sep 9 10:43 still running
reboot system boot 4.15.0-60-generi Mon Sep 9 10:14 - 10:39 (00:25)
reboot system boot 4.15.0-60-generi Mon Sep 9 08:48 - 10:09 (01:21)
reboot system boot 4.15.0-60-generi Mon Sep 9 07:33 - 10:09 (02:36)
reboot system boot 4.15.0-60-generi Mon Sep 9 06:18 - 10:09 (03:51)
reboot system boot 4.15.0-60-generi Mon Sep 9 05:03 - 10:09 (05:06)
reboot system boot 4.15.0-60-generi Mon Sep 9 03:48 - 10:09 (06:21)
reboot system boot 4.15.0-60-generi Mon Sep 9 02:33 - 10:09 (07:36)
reboot system boot 4.15.0-60-generi Mon Sep 9 01:13 - 10:09 (08:56)
reboot system boot 4.15.0-60-generi Sun Sep 8 23:58 - 10:09 (10:11)
reboot system boot 4.15.0-60-generi Sun Sep 8 22:43 - 10:09 (11:26)
reboot system boot 4.15.0-60-generi Sun Sep 8 21:28 - 10:09 (12:41)
reboot system boot 4.15.0-60-generi Sun Sep 8 20:08 - 10:09 (14:01)
reboot system boot 4.15.0-60-generi Sun Sep 8 18:53 - 10:09 (15:16)
reboot system boot 4.15.0-60-generi Sun Sep 8 17:38 - 10:09 (16:31)
reboot system boot 4.15.0-60-generi Sun Sep 8 16:23 - 10:09 (17:46)
reboot system boot 4.15.0-60-generi Sun Sep 8...

FTR, I think this is the fix in -62:

https://kernel.ubuntu.com/git/ubuntu/ubuntu-bionic.git/commit/?h=master-next&id=b502cfeffec81be8564189e5498fd3f252b27900

and it appears to be the only change from -60 to -62:

-=- cut here -=-
ewen@naosr620:~$ zcat /usr/share/doc/linux-headers-4.15.0-62-generic/changelog.Debian.gz | head -9
linux (4.15.0-62.69) bionic; urgency=medium

  * bionic/linux: 4.15.0-62.69 -proposed tracker (LP: #1842746)

  * Kernel Panic with linux-image-4.15.0-60-generic when specifying nameserver
    in docker-compose (LP: #1842447)
    - ip: frags: fix crash in ip_do_fragment()

 -- Khalid Elmously <email address hidden> Wed, 04 Sep 2019 16:11:43 -0400
ewen@naosr620:~$
-=- cut here -=-

It's a one line fix.

Ewen

FTR, 4.15.0-62 seems *much* better than 4.15.0-60. With 4.15.0-60 this system was kernel panic restarting every 75-90 minutes; now it's been up since I installed 4.15.0-62, over 5 hours ago:

-=- cut here -=-
ewen@naosr620:~$ uname -r
4.15.0-62-generic
ewen@naosr620:~$ uptime
 16:09:54 up 5:26, 1 user, load average: 0.24, 0.25, 0.24
ewen@naosr620:~$
-=- cut here -=-

That one line fix seems important :-)

Ewen

Could you please release 4.15.0-62 into the main repo? It's still in the proposed. Thank you.

This bug was fixed in the package linux - 4.15.0-62.69

---------------
linux (4.15.0-62.69) bionic; urgency=medium

  * bionic/linux: 4.15.0-62.69 -proposed tracker (LP: #1842746)

  * Kernel Panic with linux-image-4.15.0-60-generic when specifying nameserver
    in docker-compose (LP: #1842447)
    - ip: frags: fix crash in ip_do_fragment()

 -- Khalid Elmously <email address hidden> Wed, 04 Sep 2019 16:11:43 -0400

Will this be available for xenial soon?

I'm seeing the same kernel panic on linux-aws 4.15.0.1048.47 as well

Hi, @uxian.

linux-aws 4.15.0-1047 has the bug, and I can reproduce it. linux-aws 4.15.0-1048, however, has the fix and should not present the bug. I can confirm I wasn't able to reproduce the issue. Can you send any logs?

Thanks.
Cascardo.

Disregard. Looks like 4.15.0.1048 is installed but running uname -a shows 4.15.0.1047 so the box just hasn't been rebooted since the update and since it's an AWS ASG it's just reverting back to the 1047 version from the AMI every time it panics and AWS re-creates the instance. Time to rebuild the AMI with the new kernel.

4.15.0-62.69~16.04.1 is available on -updates https://kernel.ubuntu.com/sru/sru-report.html

For the xenial/linux-hwe kernel, 4.15.0-62.69~16.04.1 with this fix has been published. Are you seeing this issue in xenial's 4.4 kernel.