[Hyper-V] srcu: Lock srcu_data structure in srcu_gp_start()

Hi, Josh. Do you have an specific workload that triggers that issue?

It's a heavy database workload from a online site, so it is difficult to make a simple repro for.

I built a test kernel with commit eb4c2382272ae7 from linux-next. This commit relies on commit d633198088bd9 for the definition of spin_lock_rcu_node and it's corresponding unlock. That commit was added to mainline in v4.16-rc1

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1802021

It sounds like this bug is difficult to reproduce, but it would be great if the affected customer was willing to test this kernel.

Note about installing test kernels:
• If the test kernel is prior to 4.15(Bionic) you need to install the linux-image and linux-image-extra .deb packages.
• If the test kernel is 4.15(Bionic) or newer, you need to install the linux-modules, linux-modules-extra and linux-image-unsigned .deb packages.

Thanks in advance!

Joe, is your kernel different than the one Gavin Guo built for us here? https://launchpad.net/~mimi0213kimo/+archive/ubuntu/sf00204509-rcu-backport

We have enabled extra debugging and given that kernel to our internal customer who attempting to repro. Since the repro takes a very long time it is difficult to decide when the fix is working or not.

Paul McKenney upstream has submitted a pull request for this patch (and others) to go into 4.21. Getting some "burn in" time upstream hasn't really started in earnest yet, but there is no negative discussion about the PR, and I am tempted to get this into the regular based on Paul's comment: "Lock srcu_data structure in srcu_gp_start(), fixing a an extremely rare but also extremely embarrassing concurrency bug, courtesy of Dennis Krein."

Hi Joshua,

I just reviewed the commit Joseph provided in the launchpad. It's the
same as the two patches I backported.

eb4c2382272a srcu: Lock srcu_data structure in srcu_gp_start()
d633198088bd srcu: Prohibit call_srcu() use under raw spinlocks

The commit id eb4c2382272a is the latest linux-next commit id. When I
backported the above two patches, the eb4c2382272a was in Paul's
rcu-next tree. So, for the SRU process, Joseph's backport are the formal one.

Hi Joshua,

I will touched base with Gavin to compare our trees. My test kernel is the current Azure kernel with two commits applied: d633198088bd9 and eb4c2382272ae7. Commit eb4c2382272ae7 being the patch from
Dennis Krein in linux-next:

eb4c2382272a ("srcu: Lock srcu_data structure in srcu_gp_start()")

Gavin and I both had the same set of commits. I can submit and SRU request for this if you don't want to wait for the testing, since it could take a long time. If I submit it this week, it won't land in the Azure kernel until the next SRU cycle in the new year. Just let us know what you think.

Here are the dates for the next cycle:

cycle: 14-Jan through 03-Feb
====================================================================
         11-Jan Last day for kernel commits for this cycle.
14-Jan - 18-Jan Kernel prep week.
21-Jan - 01-Feb Bug verification & Regression testing.
         31-Jan Release 18.04.2 kernels to -updates
         04-Feb Release remaining kernels to -updates.

Looking at these dates, we may want to SRU it this week, due to a Company shutdown between 24-Dec and 06-Jan.

Hi, Josh.

Did you have any feedback from the customer regarding the test kernel?

How do you want to proceed with that?

No new instances of the problem on the test cluster for many weeks. Let's move forward with this change.

Will this fix be available for Linux 4.15.0-generic x86_64, or is it available already?

I am currently on Linux 4.15.0-43-generic x86_64 and on some servers I have this issue, others are fine, I am not sure what triggers the problem but when it triggers kworker, dockerd, systemd, go in uninterruptible sleep and I need to reboot the servers to recover from the issue.

After a while the issue reappears, so I would like to patch the servers as fast as possible.
Thanks!

Using Ubuntu 16.04.5 LTS btw.

The fix was picked up for upstream stable 4.19.15 and 4.20.2. I would expect the generic kernels to eventually pick up this fix.

Hi, Josh. Should we apply the fixes for the 4.15 and 4.18 linux-azure kernel then?

Yes, please, per comment #8

Status changed to 'Confirmed' because the bug affects multiple users.

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1802021

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

https://lists.ubuntu.com/archives/kernel-team/2019-February/098264.html

Hey guys, I am not sure how to do this but can you also make a patch for Xenial, 4.15.0-generic, for linux-hwe package? We have the same problem and we can't upgrade the boxes to a newer release so this would really help us.

Can't run the logs collection tool since the system is stuck in D state and times out. (Also the tool is not installed on the systems). We just need this fix back-ported on 4.15.0-generic for Xenial on linux-hwe package (we don't use linux-azure)

Hi, @overlord.

I changed it to "linux" instead because xenial/linux-hwe is simply a backport of bionic/linux, so we need to apply the fix to bionic/linux first and that will be include to xenial/linux-hwe automatically.

Marcelo @mhcerri,

Would you be able to provide a test kernel for bionic/linux-hwe so that @lazamarius1 can provide test results for -generic?

I'll be happy to do that as well if you're short on time right now.
(I guess the patchset is the same you posted for linux-azure.)

Thanks,
Mauricio

Thanks Marcelo! However I still see the unassigned package is linux-hwe, and I can't add Xenial tag.
I also notices I received an update today from 4.15.0-43-generic to 4.15.0-45-generic but I cannot tell if this update has the fix from this bug.
Could you help me with some directions on this? Where could I check ?
Thanks!

Mauricio @mfo, is there a way to reproduce the issue easily in order to test it? I was not able to find it.
The only way I can tell the issue is there or not is to apply the patch and wait for the servers to "hit" the problem (could take days or weeks...), and when that happens, in my case, docker tasks will end up in D state and load average will go to 100 very fast, then also a certain kworker will hit the D state and possibly in the end the init/systemd will go also in D state, and the only recovery action is to restart the box.

Hi overlord. An easy way to check the updates included in released kernels is to look at the "-changes" mailing list for your Ubuntu release.

In this situation it would be https://lists.ubuntu.com/archives/xenial-changes/

And you can find that new kernel here:

https://lists.ubuntu.com/archives/xenial-changes/2019-February/023480.html

You can see this kernel 4.15.0-45 does not include the fix for this LP bug #1802021.

@overlord: AFAIK, there is no simple reproducer test case for this issue. The ideal testing scenario for a bug and fix like this one would be for each user who reported this issue to use a test kernel with the fix in their environment and report back if the issue still manifests or not after some time has passed with that test kernel in your affected environment.

Thanks David, that was my intention to patch the systems and wait for it to reproduce (usually we get the issue back in 3-8 days or so...)
Also thanks for the pointers for checking the release notes for certain releases.
Is there a connection between the changes listed in the second link and the code base for those changes? I mean to say maybe the LP bug does not appear in the changes because it was not tagged with a correct tag for it to show up but maybe the code fix is already there ? Does this make sense? I know the code fix it's already available starting with 4.19.16 but I don't know how the back-porting is handled.

You can clone the ubuntu-xenial kernel:

git clone git://kernel.ubuntu.com/ubuntu/ubuntu-xenial.git

And then grep for the commit you're looking for. There's a few different ways to do it, I do:

git log --oneline | grep "Expose SMT control init function"

This bug is missing log files that will aid in diagnosing the problem. While running an Ubuntu kernel (not a mainline or third-party kernel) please enter the following command in a terminal window:

apport-collect 1802021

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Hi Marcelo (@mhcerri),

We have another user who confirmed the 2 patches submitted for linux-azure also fix the problem on linux(-generic).

  srcu: Prohibit call_srcu() use under raw spinlocks
  srcu: Lock srcu_data structure in srcu_gp_start()

Could they be submitted for linux as well?

Thank you very much,
Mauricio

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-bionic' to 'verification-done-bionic'. If the problem still exists, change the tag 'verification-needed-bionic' to 'verification-failed-bionic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

Hi @brad-figg,

Can we get a proposed fix for Xenial linux(-generic) package?
Or that is planned after the bionic one?

Hi @lazamarius1,

The fix for linux generic should be applied in the next kernel SRU cycle.
The current cycle ends on late February [1].

[1] https://kernel.ubuntu.com/

@lazamarius1: Just to clarify, the fix is scheduled to go in the 4.15 kernel in Bionic which is the same kernel as the Xenial HWE kernel. So there's no need to add anything to the Affects section. You will see a new linux-hwe 4.15 kernel in xenial-proposed once this is ready to test. Thanks!

This bug was fixed in the package linux-azure - 4.18.0-1011.11

---------------
linux-azure (4.18.0-1011.11) cosmic; urgency=medium

  * linux-azure: 4.18.0-1011.11 -proposed tracker (LP: #1816081)

  * 4.15.0-1037 does not see all PCI devices on GPU VMs (LP: #1816106)
    - Revert "PCI: hv: Make sure the bus domain is really unique"

linux-azure (4.18.0-1009.9) cosmic; urgency=medium

  * Allow I/O schedulers to be loaded with modprobe in linux-azure
    (LP: #1813211)
    - [Config] linux-azure: Enable all IO schedulers as modules

  * [Hyper-V] srcu: Lock srcu_data structure in srcu_gp_start() (LP: #1802021)
    - srcu: Lock srcu_data structure in srcu_gp_start()

  * CONFIG_SECURITY_SELINUX_DISABLE should be disabled on 4.15/4.18 Azure
    (LP: #1813866)
    - [Config]: disable CONFIG_SECURITY_SELINUX_DISABLE

  [ Ubuntu: 4.18.0-15.16 ]

  * Ubuntu boot failure. 4.18.0-14 boot stalls. (does not boot) (LP: #1814555)
    - Revert "drm/i915/ringbuffer: Delay after EMIT_INVALIDATE for gen4/gen5"
  * Userspace break as a result of missing patch backport (LP: #1813873)
    - tty: Don't hold ldisc lock in tty_reopen() if ldisc present

 -- Stefan Bader <email address hidden> Fri, 15 Feb 2019 17:16:24 +0100

This bug was fixed in the package linux-azure - 4.18.0-1011.11~18.04.1

---------------
linux-azure (4.18.0-1011.11~18.04.1) bionic; urgency=medium

  * linux-azure: 4.18.0-1011.11~18.04.1 -proposed tracker (LP: #1816080)

  * Miscellaneous Ubuntu changes
    - Start new release

  [ Ubuntu: 4.18.0-1011.11 ]

  * linux-azure: 4.18.0-1011.11 -proposed tracker (LP: #1816081)
  * 4.15.0-1037 does not see all PCI devices on GPU VMs (LP: #1816106)
    - Revert "PCI: hv: Make sure the bus domain is really unique"

linux-azure (4.18.0-1009.9~18.04.1) bionic; urgency=medium

  * linux-azure: 4.18.0-1009.9~18.04.1 -proposed tracker (LP: #1814733)

  * Packaging resync (LP: #1786013)
    - [Packaging] update helper scripts

  * Move bionic/linux-azure to 4.18 (LP: #1815451)
    - [Packaging] bionic/linux-azure is now a backport of cosmic/linux-azure

  * Miscellaneous Ubuntu changes
    - Start new release

  [ Ubuntu: 4.18.0-1009.9 ]

  * Allow I/O schedulers to be loaded with modprobe in linux-azure
    (LP: #1813211)
    - [Config] linux-azure: Enable all IO schedulers as modules
  * [Hyper-V] srcu: Lock srcu_data structure in srcu_gp_start() (LP: #1802021)
    - srcu: Lock srcu_data structure in srcu_gp_start()
  * CONFIG_SECURITY_SELINUX_DISABLE should be disabled on 4.15/4.18 Azure
    (LP: #1813866)
    - [Config]: disable CONFIG_SECURITY_SELINUX_DISABLE
  * Ubuntu boot failure. 4.18.0-14 boot stalls. (does not boot) (LP: #1814555)
    - Revert "drm/i915/ringbuffer: Delay after EMIT_INVALIDATE for gen4/gen5"
  * Userspace break as a result of missing patch backport (LP: #1813873)
    - tty: Don't hold ldisc lock in tty_reopen() if ldisc present

linux-azure (4.18.0-1008.8~18.04.1) bionic; urgency=medium

  * linux-azure-edge: 4.18.0-1008.8~18.04.1 -proposed tracker (LP: #1811410)

  [ Ubuntu: 4.18.0-1008.8 ]

  * linux-azure: 4.18.0-1008.8 -proposed tracker (LP: #1811415)
  * Cosmic update: 4.18.19 upstream stable release (LP: #1810820)
    - [Config] Update config after 4.18.0-14.15 rebase
  * Packaging resync (LP: #1786013)
    - [Packaging] update helper scripts
  * linux: 4.18.0-14.15 -proposed tracker (LP: #1811406)
  * CPU hard lockup with rigorous writes to NVMe drive (LP: #1810998)
    - blk-wbt: Avoid lock contention and thundering herd issue in wbt_wait
    - blk-wbt: move disable check into get_limit()
    - blk-wbt: use wq_has_sleeper() for wq active check
    - blk-wbt: fix has-sleeper queueing check
    - blk-wbt: abstract out end IO completion handler
    - blk-wbt: improve waking of tasks
  * To reduce the Realtek USB cardreader power consumption (LP: #1811337)
    - mmc: core: Introduce MMC_CAP_SYNC_RUNTIME_PM
    - mmc: rtsx_usb_sdmmc: Don't runtime resume the device while changing led
    - mmc: rtsx_usb_sdmmc: Re-work runtime PM support
    - mmc: rtsx_usb_sdmmc: Re-work card detection/removal support
    - memstick: rtsx_usb_ms: Add missing pm_runtime_disable() in probe function
    - misc: rtsx_usb: Use USB remote wakeup signaling for card insertion detection
    - memstick: Prevent memstick host from getting runtime suspended during card
      detection
    - memstick: rtsx_usb_ms: Use ms_dev() helper
    - memstick: rtsx_...

This bug was fixed in the package linux-azure - 4.18.0-1011.11

---------------
linux-azure (4.18.0-1011.11) cosmic; urgency=medium

  * linux-azure: 4.18.0-1011.11 -proposed tracker (LP: #1816081)

  * 4.15.0-1037 does not see all PCI devices on GPU VMs (LP: #1816106)
    - Revert "PCI: hv: Make sure the bus domain is really unique"

linux-azure (4.18.0-1009.9) cosmic; urgency=medium

  * Allow I/O schedulers to be loaded with modprobe in linux-azure
    (LP: #1813211)
    - [Config] linux-azure: Enable all IO schedulers as modules

  * [Hyper-V] srcu: Lock srcu_data structure in srcu_gp_start() (LP: #1802021)
    - srcu: Lock srcu_data structure in srcu_gp_start()

  * CONFIG_SECURITY_SELINUX_DISABLE should be disabled on 4.15/4.18 Azure
    (LP: #1813866)
    - [Config]: disable CONFIG_SECURITY_SELINUX_DISABLE

  [ Ubuntu: 4.18.0-15.16 ]

  * Ubuntu boot failure. 4.18.0-14 boot stalls. (does not boot) (LP: #1814555)
    - Revert "drm/i915/ringbuffer: Delay after EMIT_INVALIDATE for gen4/gen5"
  * Userspace break as a result of missing patch backport (LP: #1813873)
    - tty: Don't hold ldisc lock in tty_reopen() if ldisc present

 -- Stefan Bader <email address hidden> Fri, 15 Feb 2019 17:16:24 +0100

This is the final reminder to please verify that the kernel in -proposed resolves the issue for which you've filed this bug report. Canonical is planning to release these kernels early next week. Thank you in advance!

Do you know when a fix for linux-xenial will be available? Looks like that's the only one remaining.

Andrei, I will discuss with engineering to confirm availability and get back to you

This bug was fixed in the package linux-azure - 4.15.0-1040.44

---------------
linux-azure (4.15.0-1040.44) xenial; urgency=medium

  * linux-azure: 4.15.0-1040.44 -proposed tracker (LP: #1817038)

  * Packaging resync (LP: #1786013)
    - [Packaging] resync retpoline extraction

  * CONFIG_SECURITY_SELINUX_DISABLE should be disabled on 4.15/4.18 Azure
    (LP: #1813866)
    - [Config]: disable CONFIG_SECURITY_SELINUX_DISABLE
    - [Config] Update configs

  * Allow I/O schedulers to be loaded with modprobe in linux-azure
    (LP: #1813211)
    - [Config] linux-azure: Enable all IO schedulers as modules

  * [Hyper-V] srcu: Lock srcu_data structure in srcu_gp_start() (LP: #1802021)
    - srcu: Prohibit call_srcu() use under raw spinlocks
    - srcu: Lock srcu_data structure in srcu_gp_start()

  [ Ubuntu: 4.15.0-46.49 ]

  * linux: 4.15.0-46.49 -proposed tracker (LP: #1814726)
  * mprotect fails on ext4 with dax (LP: #1799237)
    - x86/speculation/l1tf: Exempt zeroed PTEs from inversion
  * kernel BUG at /build/linux-vxxS7y/linux-4.15.0/mm/slub.c:296! (LP: #1812086)
    - iscsi target: fix session creation failure handling
    - scsi: iscsi: target: Set conn->sess to NULL when iscsi_login_set_conn_values
      fails
    - scsi: iscsi: target: Fix conn_ops double free
  * user_copy in user from ubuntu_kernel_selftests failed on KVM kernel
    (LP: #1812198)
    - selftests: user: return Kselftest Skip code for skipped tests
    - selftests: kselftest: change KSFT_SKIP=4 instead of KSFT_PASS
    - selftests: kselftest: Remove outdated comment
  * RTL8822BE WiFi Disabled in Kernel 4.18.0-12 (LP: #1806472)
    - SAUCE: staging: rtlwifi: allow RTLWIFI_DEBUG_ST to be disabled
    - [Config] CONFIG_RTLWIFI_DEBUG_ST=n
    - SAUCE: Add r8822be to signature inclusion list
  * kernel oops in bcache module (LP: #1793901)
    - SAUCE: bcache: never writeback a discard operation
  * CVE-2018-18397
    - userfaultfd: use ENOENT instead of EFAULT if the atomic copy user fails
    - userfaultfd: shmem: allocate anonymous memory for MAP_PRIVATE shmem
    - userfaultfd: shmem/hugetlbfs: only allow to register VM_MAYWRITE vmas
    - userfaultfd: shmem: add i_size checks
    - userfaultfd: shmem: UFFDIO_COPY: set the page dirty if VM_WRITE is not set
  * Ignore "incomplete report" from Elan touchpanels (LP: #1813733)
    - HID: i2c-hid: Ignore input report if there's no data present on Elan
      touchpanels
  * Vsock connect fails with ENODEV for large CID (LP: #1813934)
    - vhost/vsock: fix vhost vsock cid hashing inconsistent
  * SRU: Fix thinkpad 11e 3rd boot hang (LP: #1804604)
    - ACPI / LPSS: Force LPSS quirks on boot
  * Bionic update: upstream stable patchset 2019-01-17 (LP: #1812229)
    - scsi: sd_zbc: Fix variable type and bogus comment
    - KVM/Eventfd: Avoid crash when assign and deassign specific eventfd in
      parallel.
    - x86/apm: Don't access __preempt_count with zeroed fs
    - x86/events/intel/ds: Fix bts_interrupt_threshold alignment
    - x86/MCE: Remove min interval polling limitation
    - fat: fix memory allocation failure handling of match_strdup()
    - ALSA: hda/realtek - Add Panasonic CF-SZ6 headset jack quirk
    - ARCv2:...

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-cosmic' to 'verification-done-cosmic'. If the problem still exists, change the tag 'verification-needed-cosmic' to 'verification-failed-cosmic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

Hi Brad @brad-figg,

5 days are not enough to test this bug.
Citing David:
"@overlord: AFAIK, there is no simple reproducer test case for this issue. The ideal testing scenario for a bug and fix like this one would be for each user who reported this issue to use a test kernel with the fix in their environment and report back if the issue still manifests or not after some time has passed with that test kernel in your affected environment."

In most of the cases systems showed the bug symptoms after 40+ days uptime.
Plus, is there a xenial fix for linux-hwe yet? We really need this patch in 4.15 kernel.

Thanks,
Marius

Hi Marius @lazamarius1,

Per the kernel.ubuntu.com schedule, the version for Bionic/linux -> Xenial/linux-hwe should land soon.

You can verify the version/timestamps for each package/release at the bottom of these pages
(the linux-hwe version comes a bit after the corresponding linux version)

https://launchpad.net/ubuntu/+source/linux
https://launchpad.net/ubuntu/+source/linux-hwe

As far as testing, yes, this issue might take longer to reproduce, but initial testing from another user that happened in order to first submit the fix to Ubuntu showed good results, so it's previously good sign of it invidivudally.
The integration of it with other fixes, i.e., testing with it in -proposed, will be done by that other user as well, so collectively w/ your testing that might increase chances of the issue still happening or not.
There's also regression testing of the kernel builds, which can spot failures, so that collaborates too.

Hope this helps,
Mauricio

@lazamarius1,

Actually linux-hwe for Bionic with this fix has just been uploaded.
See in https://launchpad.net/ubuntu/+source/linux-hwe

Changelog

linux-hwe (4.15.0-47.50~16.04.1) xenial; urgency=medium
...
  * [Hyper-V] srcu: Lock srcu_data structure in srcu_gp_start() (LP: #1802021)
    - srcu: Prohibit call_srcu() use under raw spinlocks
    - srcu: Lock srcu_data structure in srcu_gp_start()
...

Thanks Mauricio! @mfo

Will begin deploying this and let you guys know as soon as possible.

@mfo

Just to confirm that we installed the right proposed version of the kernel as this doesn't have any steps to reproduce. It only reproduces randomly after a certain amount of time.
These are the steps we followed to install

echo 'deb http://archive.ubuntu.com/ubuntu/ xenial-proposed restricted main multiverse universe' > /etc/apt/sources.list.d/kernel.list
echo -e "Package: *\nPin: release a=xenial-proposed\nPin-Priority: 400" > /etc/apt/preferences.d/proposed-updates
apt-get -t xenial-proposed install -y linux-image-unsigned-4.15.0-47-generic
reboot

Installed packages
dpkg -l | grep -i 4.15.0-47
ii linux-image-unsigned-4.15.0-47-generic 4.15.0-47.50~16.04.1 amd64 Linux kernel image for version 4.15.0 on 64 bit x86 SMP
ii linux-modules-4.15.0-47-generic 4.15.0-47.50~16.04.1 amd64 Linux kernel extra modules for version 4.15.0 on 64 bit x86 SMP

dpkg -l | grep -i hwe
ii linux-generic-hwe-16.04 4.15.0.45.66 amd64 Complete Generic Linux kernel and headers
ii linux-headers-generic-hwe-16.04 4.15.0.45.66 amd64 Generic Linux kernel headers
ii linux-image-generic-hwe-16.04 4.15.0.45.66 amd64 Generic Linux kernel image

uname -a
Linux myhostname 4.15.0-47-generic #50~16.04.1-Ubuntu SMP Fri Mar 15 16:06:21 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

As Marius mentioned we already deployed this patch on several instances and now we are monitoring them to see if this still happens. At least 2 of them were affected by the bug before patching so we had to reboot them.
If the above information (previous post) is correct and considering the time until this might reproduce or not, I think you might want to include this fix into existing release cycle.

I was experiencing a situation with moby dockerd entering a state similar to comment 0 here, running kubernetes and linux kernel 4.15.0-1037-azure. This was an 8 node cluster. Observed with combinations:

kubernetes 1.11.5 + moby runtime 3.0.1 + Ubuntu 16.04.5
kubernetes 1.11.7 + moby runtime 3.0.4 + Ubuntu 16.04.10

The longest window between outages prior was 4 days, with the shortest being less than a day.

I have observed 2 weeks of uptime on 8 nodes without observation of the original symptoms since upgrading the kubernetes node kernel to 4.15.0-1040-azure. I am confident the kernel patch has resolved our problem.

Ref https://github.com/moby/moby/issues/38750 and https://github.com/Azure/AKS/issues/838, both closed.

Updating bug tags to verification done.

As mentioned by users in this LP bug, the verification period of 5 days is _usually_ not enough to reproduce this problem, however, we have some datapoints that support the fix is good.

1) The fix has been first delivery in linux-azure, 3 weeks ago, and has reportedly resolved the issue for @alanjcastonguay: the issue was experienced within 4 days at the most, and hasn't happened for 2 weeks in 8 nodes (which is statistically very positive; and it helps that the fix is not specific to -azure).

2) One of the users who reported this in linux (-generic), has verified a test kernel with this fix for weeks, based upon which the fix has been submitted after linux-azure had it. The same user has verified -proposed for about a week now, and it's looking good.

3) Users in this LP bug have been running the -proposed kernel in multiple nodes for about a week now too, and haven't hit the issue yet.

On top of 1), with 2) and 3) combined, and the schedule for -proposed verification, this seems to be a reasonable compromise between results and test time.

cheers,
Mauricio

This bug was fixed in the package linux - 4.15.0-47.50

---------------
linux (4.15.0-47.50) bionic; urgency=medium

  * linux: 4.15.0-47.50 -proposed tracker (LP: #1819716)

  * Packaging resync (LP: #1786013)
    - [Packaging] resync getabis
    - [Packaging] update helper scripts
    - [Packaging] resync retpoline extraction

  * C++ demangling support missing from perf (LP: #1396654)
    - [Packaging] fix a mistype

  * arm-smmu-v3 arm-smmu-v3.3.auto: CMD_SYNC timeout (LP: #1818162)
    - iommu/arm-smmu-v3: Fix unexpected CMD_SYNC timeout

  * Crash in nvme_irq_check() when using threaded interrupts (LP: #1818747)
    - nvme-pci: fix out of bounds access in nvme_cqe_pending

  * CVE-2019-9213
    - mm: enforce min addr even if capable() in expand_downwards()

  * CVE-2019-3460
    - Bluetooth: Check L2CAP option sizes returned from l2cap_get_conf_opt

  * amdgpu with mst WARNING on blanking (LP: #1814308)
    - drm/amd/display: Don't use dc_link in link_encoder
    - drm/amd/display: Move wait for hpd ready out from edp power control.
    - drm/amd/display: eDP sequence BL off first then DP blank.
    - drm/amd/display: Fix unused variable compilation error
    - drm/amd/display: Fix warning about misaligned code
    - drm/amd/display: Fix MST dp_blank REG_WAIT timeout

  * tun/tap: unable to manage carrier state from userland (LP: #1806392)
    - tun: implement carrier change

  * CVE-2019-8980
    - exec: Fix mem leak in kernel_read_file

  * raw_skew in timer from the ubuntu_kernel_selftests failed on Bionic
    (LP: #1811194)
    - selftest: timers: Tweak raw_skew to SKIP when ADJ_OFFSET/other clock
      adjustments are in progress

  * [Packaging] Allow overlay of config annotations (LP: #1752072)
    - [Packaging] config-check: Add an include directive

  * CVE-2019-7308
    - bpf: move {prev_,}insn_idx into verifier env
    - bpf: move tmp variable into ax register in interpreter
    - bpf: enable access to ax register also from verifier rewrite
    - bpf: restrict map value pointer arithmetic for unprivileged
    - bpf: restrict stack pointer arithmetic for unprivileged
    - bpf: restrict unknown scalars of mixed signed bounds for unprivileged
    - bpf: fix check_map_access smin_value test when pointer contains offset
    - bpf: prevent out of bounds speculation on pointer arithmetic
    - bpf: fix sanitation of alu op with pointer / scalar type from different
      paths
    - bpf: add various test cases to selftests

  * CVE-2017-5753
    - bpf: properly enforce index mask to prevent out-of-bounds speculation
    - bpf: fix inner map masking to prevent oob under speculation

  * BPF: kernel pointer leak to unprivileged userspace (LP: #1815259)
    - bpf/verifier: disallow pointer subtraction

  * squashfs hardening (LP: #1816756)
    - squashfs: more metadata hardening
    - squashfs metadata 2: electric boogaloo
    - squashfs: more metadata hardening
    - Squashfs: Compute expected length from inode size rather than block length

  * efi/arm/arm64: Allow SetVirtualAddressMap() to be omitted (LP: #1814982)
    - efi/arm/arm64: Allow SetVirtualAddressMap() to be omitted

  * Update ENA driver to version 2.0.3K (LP: #1816806)...

This bug was fixed in the package linux - 4.18.0-17.18

---------------
linux (4.18.0-17.18) cosmic; urgency=medium

  * linux: 4.18.0-17.18 -proposed tracker (LP: #1819624)

  * Packaging resync (LP: #1786013)
    - [Packaging] resync getabis
    - [Packaging] update helper scripts

  * C++ demangling support missing from perf (LP: #1396654)
    - [Packaging] fix a mistype

  * arm-smmu-v3 arm-smmu-v3.3.auto: CMD_SYNC timeout (LP: #1818162)
    - iommu/arm-smmu-v3: Fix unexpected CMD_SYNC timeout

  * Crash in nvme_irq_check() when using threaded interrupts (LP: #1818747)
    - nvme-pci: fix out of bounds access in nvme_cqe_pending

  * CVE-2019-9003
    - ipmi: fix use-after-free of user->release_barrier.rda

  * CVE-2019-9162
    - netfilter: nf_nat_snmp_basic: add missing length checks in ASN.1 cbs

  * CVE-2019-9213
    - mm: enforce min addr even if capable() in expand_downwards()

  * CVE-2019-3460
    - Bluetooth: Check L2CAP option sizes returned from l2cap_get_conf_opt

  * tun/tap: unable to manage carrier state from userland (LP: #1806392)
    - tun: implement carrier change

  * CVE-2019-8980
    - exec: Fix mem leak in kernel_read_file

  * [Packaging] Allow overlay of config annotations (LP: #1752072)
    - [Packaging] config-check: Add an include directive

  * amdgpu with mst WARNING on blanking (LP: #1814308)
    - drm/amd/display: Fix MST dp_blank REG_WAIT timeout

  * CVE-2019-7308
    - bpf: move {prev_,}insn_idx into verifier env
    - bpf: move tmp variable into ax register in interpreter
    - bpf: enable access to ax register also from verifier rewrite
    - bpf: restrict map value pointer arithmetic for unprivileged
    - bpf: restrict stack pointer arithmetic for unprivileged
    - bpf: restrict unknown scalars of mixed signed bounds for unprivileged
    - bpf: fix check_map_access smin_value test when pointer contains offset
    - bpf: prevent out of bounds speculation on pointer arithmetic
    - bpf: fix sanitation of alu op with pointer / scalar type from different
      paths
    - bpf: add various test cases to test_verifier
    - bpf: add various test cases to selftests

  * CVE-2017-5753
    - bpf: fix inner map masking to prevent oob under speculation

  * Use memblock quirk instead of delayed allocation for GICv3 LPI tables
    (LP: #1816425)
    - efi/arm: Revert "Defer persistent reservations until after paging_init()"
    - arm64, mm, efi: Account for GICv3 LPI tables in static memblock reserve
      table

  * efi/arm/arm64: Allow SetVirtualAddressMap() to be omitted (LP: #1814982)
    - efi/arm/arm64: Allow SetVirtualAddressMap() to be omitted

  * Update ENA driver to version 2.0.3K (LP: #1816806)
    - net: ena: update driver version from 2.0.2 to 2.0.3
    - net: ena: fix race between link up and device initalization
    - net: ena: fix crash during failed resume from hibernation

  * Silent "Unknown key" message when pressing keyboard backlight hotkey
    (LP: #1817063)
    - platform/x86: dell-wmi: Ignore new keyboard backlight change event

  * CVE-2018-19824
    - ALSA: usb-audio: Fix UAF decrement if card has no live interfaces in card.c

  * CVE-2019-3459
    - Bluetooth: Verify that l2cap_get...

Status changed to 'Confirmed' because the bug affects multiple users.