Kernel panic

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1275879

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

Do you happen to know if this issue also happens in the latest Trusty kernel?

I am unable to successfully run the 'apport-collect 1275879' command as it its only through the terminal and tries to open up a browser.

As for the issue showing up on the Trusty kernel, I've just launched a server with it to test it out. It usually takes around a day or two for the kernel panic to show up. I'll post the kernel panic log if it shows up in Trusy.

Looks like the kernel panic also shows up in Trusty Tahr; the AWS ubuntu image I used for trusty was ami-c8d8ea8d. I have attached the system log which contains the kernel panic towards the end

Chili_Man: I've been looking at the xen-netfront driver which produces the panic, and as far as I can tell this is a bug elsewhere in the networking stack and not in xen-netfront. Exactly where the problem comes from is still unknown, as the panic is disconnected from the source of the bad data.

However if true I should be able to produce similar data without using Xen. I've built a modified kernel which will hopefully detect the bad data when it is created and point to the code which created it. I'm running this right now in a kvm virtual machine with a minimal haproxy/nginx setup and downloading a large file from the VM in a tight loop. If you can provide more details about your setup I may be able to tweak my test to more closely match your use case.

It would also be helpful if you could reproduce the issue while running with my modified kernel and then give me the kernel log. I've uploaded the package to the location below if you're able to try it out. You probably need only the linux-image package and not linux-image-extra, though the safest bet is to see what you already have installed and install the same thing.

http://people.canonical.com/~sforshee/lp1275879/linux-3.11.0-17.31+lp1275879v201402101139/

Thanks!

Hey Seth,

I've launched a server with only the linux-image package you provided. I'll attach the kernel logs once the kernel panic shows up.

As for the the setup with the server, it is basically the load balancer in our system. Nginx's sole purpose is to provide SSL termination, to which it then forwards all requests to Haproxy. Hapryxy then routes requests to rails, nodejs, and python flask backends. Haproxy's configuration is dynamically generated every 5 minutes depending on whether new backends were added to it's cluster , which are discovered through a chef-client run.

If you need any more details about the server please let me know. Your help is very much appreciated!.

The kernel panic has occurred again, so I've attached the kern.log and the amazon AWS system log showing the panic.

The AWS system log.

Thanks for getting those logs. Unfortunately I'm still not able to reproduce the problem.

The logs show that the data which triggers the panic is read from a pipe, but the problem happens on the other side, before or when the data enters the pipe. I've been looking at code and haven't identified the cause so far, though there's still more code to investigate.

While I'm doing that it would be helpful if you could reproduce the problem again with the test kernel below. This should tell us where the problematic data comes from. I also added some printouts to provide additional information at the point where the panic occurs. Hopefully this gets us to the root cause and not to another layer of indirection.

http://people.canonical.com/~sforshee/lp1275879/linux-3.11.0-17.31+lp1275879v201402141025/

I found some code that looks suspicious. The kernel below has a fix, so please try it out and let me know if it resolves the issue. The kernel also contains the extra debugging from the previous build (with a fix for an off-by-one error), so if the problem does happen again please give me the logs. Thanks!

http://people.canonical.com/~sforshee/lp1275879/linux-3.11.0-18.32+lp1275879v201402201000/

Chili_Man: Did you ever get a chance to test the kernel in comment #11?

Seth: Sorry for the long delay, I'll get on it by next week.

I have applied the kernel fix that you provided to the server and its been running for about a week now without any issues. I'll let it run for a week more to see if the issue pops up again.

Great, thanks the update. I'll wait to see your final results.

No kernel panics at all from the nginx haproxy server this week. The kernel patch you provided has made the issue go away. Do need any logs or something from the server? Thanks for the kernel fix, I really appreciate it !

Nope, no logs needed. Thanks for testing.

Diego: I've been reviewing this patch with newer kernel versions, and I don't believe it should actually make any difference there (and would probably add a bug as well). Have you tried 14.04 to see if the problem still exists there?

Not yet, we will be evaluating 14.04 for the load balancer in the upcoming weeks. I'll post here if we find any issues with it.

We just tried using ubuntu 14.04 as the load balancer and unfortunately the kernel panic issue was happening there. I've attached a log of the kernel panic.

I am also running into this kernel panic. We are trying to update our infrastructure from 12.04 to 14.04, and our load balancer boxes, which are HAProxy only (I've tried 1.4.x and 1.5.x of HAProxy). The servers work perfectly well, but as soon as I expose the server to the Internet via ELB, the server is guaranteed to kernel panic within 5 minutes. I have been unable to trigger the panic outside of ELB traffic.

$ uname -a
Linux ip-10-154-139-57 3.13.0-37-generic #64-Ubuntu SMP Mon Sep 22 21:28:38 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

I'd like to add to this that the upcoming kernel update, 3.13.0-37 from the proposed package set, also has this same problem. This is on an Amazon AWS m1.small box.

I am also able to trigger the kernel panic through a simple `ab` invocation. With the params of -c 10 -n 1000 I have yet to go through two full runs without a panic, so it's not related to ELB at all, just an eventual failure of the networking stack.

Attached is the kernel panic log for this newest kernel.

Jason: Since it sounds like you can reliably reproduce the problem maybe you can help me do some diagnosis. I think I got in the general neighborhood of the problem before but didn't quite find the cause. If I give you a kernel with some extra debug logging added, can you reproduce the issue and send me the logs?

I went ahead an posted a build at http://people.canonical.com/~sforshee/lp1275879/linux-3.13.0-37.64+lp1275879v201410021107/. It may be rather verbose, but hopefully it will show us where the bad data originates. Please reproduce the issue with this kernel and send me the log. Thanks!

Python Software Foundation Infrastructure is experiencing the same kernel panic on Ubuntu 14.04 with haproxy. Opening a bug to mark as duplicate.

Ernest: If you can give me logs when using the test kernel from comment #24 it should help with tracking down where this problem originates. Thanks!

Seth: I went a head and applied the patch from comment #24 on one of our load balancers. I've been running it for a few days, so I've attached the logs for it here.

Diego: Thanks for testing. I had a look at the logs, but all I was able to find out was that I had an off-by-one error in some of the checks I added that causes them to trigger on full pages in addition to cases which actually cause the problem. As a result I think most if not all of the warnings in your logs are false positives.

Sorry for the mistake. I've fixed that and uploaded a new build, so anyone who has been testing the last one should switch to this one.

http://people.canonical.com/~sforshee/lp1275879/linux-3.13.0-38.65+lp1278579v201410160939/

Seth: I'm having issues installing the linux headers from this package

 http://people.canonical.com/~sforshee/lp1275879/linux-3.13.0-38.65+lp1278579v201410160939/linux-headers-3.13.0-38-generic_3.13.0-38.65+lp1278579v201410160939_amd64.deb

I keep getting this error when I try to install it:

----------------------------------------------------------------------------------------
root@aws-load-balancer:~ 2014-10-20 18:01:08
# dpkg -i install linux-headers-3.13.0-38-generic_3.13.0-38.65+lp1278579v201410160939_amd64.deb
dpkg: error processing archive install (--install):
 cannot access archive: No such file or directory
Selecting previously unselected package linux-headers-3.13.0-38-generic.
(Reading database ... 109756 files and directories currently installed.)
Preparing to unpack linux-headers-3.13.0-38-generic_3.13.0-38.65+lp1278579v201410160939_amd64.deb ...
Unpacking linux-headers-3.13.0-38-generic (3.13.0-38.65+lp1278579v201410160939) ...
dpkg: dependency problems prevent configuration of linux-headers-3.13.0-38-generic:
 linux-headers-3.13.0-38-generic depends on linux-headers-3.13.0-38; however:
  Package linux-headers-3.13.0-38 is not installed.

dpkg: error processing package linux-headers-3.13.0-38-generic (--install):
 dependency problems - leaving unconfigured
Errors were encountered while processing:
 install
 linux-headers-3.13.0-38-generic
----------------------------------------------------------------------------------------------------

After that I try to do `apt-get install -f` , but it just ends up removing the package

--------------------------------------------------------------------------------------------
root@staging-aws-load-balance:~ 2014-10-20 18:01:20
# apt-get install -f
Reading package lists... Done
Building dependency tree
Reading state information... Done
Correcting dependencies... Done
The following packages were automatically installed and are no longer required:
  libnet-cups-perl python-bson python-bson-ext python-gridfs python-pymongo
  python-pymongo-ext
Use 'apt-get autoremove' to remove them.
The following packages will be REMOVED:
  linux-headers-3.13.0-38-generic
0 upgraded, 0 newly installed, 1 to remove and 6 not upgraded.
1 not fully installed or removed.
After this operation, 3,752 kB disk space will be freed.

You probably don't need the headers package, unless you're using dkms packages or something like that, so you can probably just omit that package when you install. If you really do need it then I'll need to build the other headers package which it depends on.

Seth: Here's the new kernel log after applying your patch to the server.

We've been running this as well, however the server hasn't shut down since we've been running it. Is the log still useful or should we wait until it shuts down?

Donald: The patches catch the kernel exception, so you shouldn't see it shut down. But you should be able to see the caught kernel panics in the kern.log

It turns out that the check which generates the panic here is likely invalid, and the code will handle the situation just fine, so the check can be removed. I've prepared a kernel with removes that check (and a related one), which I've temporarily replaced with a warning for testing so we can detect that the condition has been encountered. Please test this kernel and wait for a similar message and stack trace to appear in dmesg, and confirm that other than the appearance of the warning everything continues working fine. Thanks!

http://people.canonical.com/~sforshee/lp1278531/linux-3.13.0-14.34+lp1278531v201402280817/

It seems I pasted in the wrong link in comment #34. Here's the correct link:

http://people.canonical.com/~sforshee/lp1275879/linux-3.13.0-40.68+lp1275879v201411101105/

Apologies for the mistake.

Seth: I've been running the kernel patch you provided in comment #35 for the last 2 weeks and I have not been able to see any kernel panics or debug outputs in dmesg, syslog, or the kernel log .

On Tue, Nov 25, 2014 at 06:40:36PM -0000, Diego Rodriguez wrote:
> Seth: I've been running the kernel patch you provided in comment #35
> for the last 2 weeks and I have not been able to see any kernel panics
> or debug outputs in dmesg, syslog, or the kernel log .

Thanks for testing. It sounds like you haven't hit the condition yet, as
it should result in a similar splat into dmesg but then continue on and
work as expected. At least that's the expectation, and I'm hoping we can
verify it before applying the changes.

I've tested the fix that Seth has and it resolved our issue completely. The first time that 14.04 had been stable on our HAProxy load balancer boxes.

Err, I accidently set this to Fix Released trying to click to see if there was any information about if that meant "Fix Released in the archives" or "Fix Released in a 14.04.N patch" and now it won't let me change it back.

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-trusty' to 'verification-done-trusty'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed-utopic' to 'verification-done-utopic'.

If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed.

See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you!

So, I don't have a problem testing this on trusty-proposed but the 5 day thing might be an issue. The kernel panic happened randomly after some period of time, it wasn't something I could trigger on demand. All I can do is run a server and wait some period of time and see if it kernel panics or not. Is that good enough?

Brad: I'm testing the proposed kernel fix for trusty right now, but as Donald mentioned, 5 days might not be enough to test it since the kernel panic occurs randomly after a period of time.

Seth: I've been testing the proposed kernel fix for the past few days, I'm not getting any kernel panics but I am seeing many of the following log lines in the kernel.log:

Dec 23 01:36:12 lb kernel: [280745.518414] xen_netfront: xennet: skb rides the rocket: 19 slots
Dec 23 01:36:17 lb kernel: [280750.146841] xen_netfront: xennet: skb rides the rocket: 19 slots
Dec 23 01:36:17 lb kernel: [280750.148361] xen_netfront: xennet: skb rides the rocket: 19 slots
Dec 23 01:36:17 lb kernel: [280750.487324] xen_netfront: xennet: skb rides the rocket: 19 slots
Dec 23 01:36:17 lb kernel: [280750.557000] xen_netfront: xennet: skb rides the rocket: 19 slots
Dec 23 01:36:17 lb kernel: [280750.557319] xen_netfront: xennet: skb rides the rocket: 19 slots
Dec 23 01:36:20 lb kernel: [280752.961168] xen_netfront: xennet: skb rides the rocket: 19 slots
Dec 23 01:36:24 lb kernel: [280757.305653] xen_netfront: xennet: skb rides the rocket: 19 slots
Dec 23 01:36:24 lb kernel: [280757.306978] xen_netfront: xennet: skb rides the rocket: 19 slots
Dec 23 01:36:24 lb kernel: [280757.308023] xen_netfront: xennet: skb rides the rocket: 19 slots
Dec 23 01:36:24 lb kernel: [280757.310167] xen_netfront: xennet: skb rides the rocket: 19 slots
Dec 23 01:36:24 lb kernel: [280757.517154] xen_netfront: xennet: skb rides the rocket: 19 slots
Dec 23 01:36:24 lb kernel: [280757.519087] xen_netfront: xennet: skb rides the rocket: 19 slots
Dec 23 01:36:33 lb kernel: [280766.589360] xen_netfront: xennet: skb rides the rocket: 19 slots
Dec 23 01:36:33 lb kernel: [280766.591679] xen_netfront: xennet: skb rides the rocket: 19 slots
Dec 23 01:36:35 lb kernel: [280767.943873] xen_netfront: xennet: skb rides the rocket: 19 slots
Dec 23 01:36:35 lb kernel: [280767.949281] xen_netfront: xennet: skb rides the rocket: 19 slots
Dec 23 01:36:35 lb kernel: [280767.999322] xen_netfront: xennet: skb rides the rocket: 19 slots
Dec 23 01:36:35 lb kernel: [280768.019297] xen_netfront: xennet: skb rides the rocket: 19 slots
Dec 23 01:36:47 lb kernel: [280780.506279] xen_netfront: xennet: skb rides the rocket: 19 slots
Dec 23 01:36:47 lb kernel: [280780.507321] xen_netfront: xennet: skb rides the rocket: 19 slots
Dec 23 01:36:47 lb kernel: [280780.510664] xen_netfront: xennet: skb rides the rocket: 19 slots
Dec 23 01:36:51 lb kernel: [280783.849895] xen_netfront: xennet: skb rides the rocket: 19 slots
Dec 23 01:37:09 lb kernel: [280802.234283] xen_netfront: xennet: skb rides the rocket: 20 slots
Dec 23 01:38:35 lb kernel: [280888.357129] xen_netfront: xennet: skb rides the rocket: 19 slots
Dec 23 01:38:38 lb kernel: [280891.041640] xen_netfront: xennet: skb rides the rocket: 19 slots
Dec 23 01:38:41 lb kernel: [280894.528434] xen_netfront: xennet: skb rides the rocket: 19 slots
Dec 23 01:38:49 lb kernel: [280902.511241] xen_netfront: xennet: skb rides the rocket: 19 slots
Dec 23 01:39:27 lb kernel: [280940.435533] xen_netfront: xennet: skb rides the rocket: 19 slots
Dec 23 01:39:40 lb kernel: [280953.579076] xen_netfront: xennet: skb rides the rocket: 19 slots

Seems that the `xen_netfront: xennet: skb rides the rocket: 19 slots` is a different issue thats already been reported:

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1317811

This bug was fixed in the package linux - 3.16.0-29.39

---------------
linux (3.16.0-29.39) utopic; urgency=low

  [ Kamal Mostafa ]

  * Release Tracking Bug
    - LP: #1402822

  [ AceLan Kao ]

  * SAUCE: Add use_native_backlight quirk for HP ProBook 6570b
    - LP: #1359010

  [ Andy Whitcroft ]

  * Revert "SAUCE: (no-up) arm64: optimized copy_to_user and copy_from_user
    assembly code"
    - LP: #1398596
  * [Config] updateconfigs to balance CONFIG_SCOM_DEBUGFS

  [ Paolo Pisati ]

  * [Config] armhf: VIRTIO_[BALLOON|MMIO]=y

  [ Upstream Kernel Changes ]

  * Revert "arm64: Make default dma_ops to be noncoherent"
    - LP: #1386490
  * Revert "percpu: free percpu allocation info for uniprocessor system"
    - LP: #1401079
  * ath3k: Add support of MCI 13d3:3408 bt device
    - LP: #1395465
  * x86: kvm: use alternatives for VMCALL vs. VMMCALL if kernel text is
    read-only
    - LP: #1379340
  * cpufreq: Allow stop CPU callback to be used by all cpufreq drivers
    - LP: #1397928
  * cpufreq: powernv: Set the pstate of the last hotplugged out cpu in
    policy->cpus to minimum
    - LP: #1397928
  * cpufreq: powernv: Set the cpus to nominal frequency during reboot/kexec
    - LP: #1397928
  * xen-netfront: Remove BUGs on paged skb data which crosses a page
    boundary
    - LP: #1275879
  * ACPI / blacklist: blacklist Win8 OSI for Dell Vostro 3546
    - LP: #1383589
  * iwlwifi: add device / firmware to fw-error-dump file
    - LP: #1399440
  * iwlwifi: rename iwl_mvm_fw_error_next_data
    - LP: #1399440
  * iwlwifi: pcie: add firmware monitor capabilities
    - LP: #1399440
  * iwlwifi: remove wrong comment about alignment in iwl-fw-error-dump.h
    - LP: #1399440
  * iwlwifi: mvm: don't collect logs in the interrupt thread
    - LP: #1399440
  * iwlwifi: mvm: kill iwl_mvm_fw_error_rxf_dump
    - LP: #1399440
  * iwlwifi: mvm: update layout of firmware error dump
    - LP: #1399440
  * powerpc/pseries: Fix endiannes issue in RTAS call from xmon
    - LP: #1396235
  * mmc: sdhci-pci-o2micro: Fix Dell E5440 issue
    - LP: #1346067
  * mfd: rtsx: Fix PM suspend for 5227 & 5249
    - LP: #1359052
  * samsung-laptop: Add broken-acpi-video quirk for NC210/NC110
    - LP: #1401079
  * acer-wmi: Add acpi_backlight=video quirk for the Acer KAV80
    - LP: #1401079
  * pinctrl: baytrail: show output gpio state correctly on Intel Baytrail
    - LP: #1401079
  * ALSA: hda - Add dock support for Thinkpad T440 (17aa:2212)
    - LP: #1401079
  * ALSA: hda - Add ultra dock support for Thinkpad X240.
    - LP: #1401079
  * rbd: Fix error recovery in rbd_obj_read_sync()
    - LP: #1401079
  * ds3000: fix LNB supply voltage on Tevii S480 on initialization
    - LP: #1401079
  * powerpc: do_notify_resume can be called with bad thread_info flags
    argument
    - LP: #1401079
  * powerpc/powernv: Properly fix LPC debugfs endianness
    - LP: #1401079
  * irqchip: armada-370-xp: Fix MSI interrupt handling
    - LP: #1401079
  * irqchip: armada-370-xp: Fix MPIC interrupt handling
    - LP: #1401079
  * USB: kobil_sct: fix non-atomic allocation in write path
    - LP: #1401079
  * USB: opticon: fix non-atomic allocation in write path
    - LP: #14010...

This bug was fixed in the package linux - 3.13.0-44.73

---------------
linux (3.13.0-44.73) trusty; urgency=low

  [ Kamal Mostafa ]

  * Release Tracking Bug
    - LP: #1402872

  [ AceLan Kao ]

  * SAUCE: Add use_native_backlight quirk for HP ProBook 6570b
    - LP: #1359010

  [ Andy Whitcroft ]

  * Revert "SAUCE: (no-up) arm64: optimized copy_to_user and copy_from_user
    assembly code"
    - LP: #1398596
  * [Config] updateconfigs to balance CONFIG_SCOM_DEBUGFS

  [ Upstream Kernel Changes ]

  * iwlwifi: mvm: fix merge damage
    - LP: #1393317
  * iwlwifi: remove IWL_UCODE_TLV_FLAGS_SCHED_SCAN flag
    - LP: #1393317
  * iwlwifi: mvm: disable scheduled scan to prevent firmware crash
    - LP: #1393317
  * iwlwifi: mvm: enable scheduled scan on newest firmware
    - LP: #1393317
  * x86: kvm: use alternatives for VMCALL vs. VMMCALL if kernel text is
    read-only
    - LP: #1379340
  * phylib: introduce PHY_INTERFACE_MODE_XGMII for 10G PHY
    - LP: #1381084
  * of: make of_get_phy_mode parse 'phy-connection-type'
    - LP: #1381084
  * xen-netfront: Remove BUGs on paged skb data which crosses a page
    boundary
    - LP: #1275879
  * ACPI / blacklist: blacklist Win8 OSI for Dell Vostro 3546
    - LP: #1383589
  * powerpc/pseries: Fix endiannes issue in RTAS call from xmon
    - LP: #1396235
  * mmc: sdhci-pci-o2micro: Fix Dell E5440 issue
    - LP: #1346067
  * mfd: rtsx: Fix PM suspend for 5227 & 5249
    - LP: #1359052
  * drivers:scsi:storvsc: Fix a bug in handling ring buffer failures that
    may result in I/O freeze
    - LP: #1400289
  * arm64: optimized copy_to_user and copy_from_user assembly code
    - LP: #1400349
  * net:socket: set msg_namelen to 0 if msg_name is passed as NULL in
    msghdr struct from userland.
    - LP: #1335478
  * drm/radeon: initialize sadb to NULL in the audio code
    - LP: #1402714
  * powerpc/vphn: NUMA node code expects big-endian
    - LP: #1401150
  * ALSA: usb-audio: Fix device_del() sysfs warnings at disconnect
    - LP: #1402853
  * ALSA: hda - Add mute LED pin quirk for HP 15 touchsmart
    - LP: #1334950, #1402853
  * rcu: Make callers awaken grace-period kthread
    - LP: #1402853
  * rcu: Use rcu_gp_kthread_wake() to wake up grace period kthreads
    - LP: #1402853
  * net: sctp: fix NULL pointer dereference in af->from_addr_param on
    malformed packet
    - LP: #1402853
  * KVM: x86: Don't report guest userspace emulation error to userspace
    - LP: #1402853
  * [media] ttusb-dec: buffer overflow in ioctl
    - LP: #1402853
  * arm64: __clear_user: handle exceptions on strb
    - LP: #1402853
  * ARM: pxa: fix hang on startup with DEBUG_LL
    - LP: #1402853
  * samsung-laptop: Add broken-acpi-video quirk for NC210/NC110
    - LP: #1402853
  * acer-wmi: Add Aspire 5741 to video_vendor_dmi_table
    - LP: #1402853
  * acer-wmi: Add acpi_backlight=video quirk for the Acer KAV80
    - LP: #1402853
  * rbd: Fix error recovery in rbd_obj_read_sync()
    - LP: #1402853
  * [media] ds3000: fix LNB supply voltage on Tevii S480 on initialization
    - LP: #1402853
  * powerpc: do_notify_resume can be called with bad thread_info flags
    argument
    - LP: #1402853
  * USB: kobil_sct: f...

This bug was nominated against a series that is no longer supported, ie vivid. The bug task representing the vivid nomination is being closed as Won't Fix.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.