xenial kernel crash on HP BL460c G7 (qla24xx problem?)

This bug is missing log files that will aid in diagnosing the problem. From a terminal window please run:

apport-collect 1554003

and then change the status of the bug to 'Confirmed'.

If, due to the nature of the issue you have encountered, you are unable to run this command, please add a comment stating that fact and change the bug status to 'Confirmed'.

This change has been made by an automated script, maintained by the Ubuntu Kernel Team.

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

apport information

I boot newer (4.4.0-11 with bootparam: modprobe.blacklist=qla2xxx )

manual modporbe:

root@kvm2:~# modprobe qla2xxx
[ 247.431349] qla2xxx [0000:06:00.0]-0034:3: MSI-X: Unsupported ISP 2432 SSVID/SSDID (0x103C,0x1705).
[ 247.807564] qla2xxx [0000:06:00.1]-0034:4: MSI-X: Unsupported ISP 2432 SSVID/SSDID (0x103C,0x1705).
[ 248.200048] [<ffffffff8101bd5a>] oops_end+0xca/0xd0
[ 248.200115] [<ffffffff8106a485>] no_context+0x135/0x380
[ 248.200181] [<ffffffff8106a750>] __bad_area_nosemaphore+0x80/0x1f0
[ 248.200
[ 248.200249] [<ffffffff8106a8d3>] bad_area_nosemaphore+0x13/0x20
[ 248.200316] [<ffffffff8106abab>] __do_page_fault+0xcb/0x440
[ 248.200382] [<ffffffff8106af42>] do_page_fault+0x22/0x30
[ 248.200449] [<ff
[ 248.200449] [<ffffffff8182baf8>] page_fault+0x28/0x30
[ 248.200530] [<ffffffffc072de29>] ? qla24xx_process_response_queue+0x49/0x4a0 [qla2xxx]
[ 248.200615] [<ffffffff810c
[ 248.200615] [<ffffffff810ca0fe>] ? cpuacct_charge+0x4e/0x60
[ 248.200681] [<ffffffff810c0a77>] ? dequeue_rt_stack+0xc7/0x230
[ 248.200749] [<ffffffff810c008f>] ? update_curr_rt+0x13f/0x1d0
[ 248.200830] [<ffffffffc0730581>] qla
[ 248.200830] [<ffffffffc0730581>] qla24xx_intr_handler+0x101/0x300 [qla2xxx]
[ 248.200899] [<ffffffff81824db9>] ? __schedule+0x389/0xae0
[ 248.200965] [<ffffffff810dbfa0>] ? irq_finalize_oneshot.part.35+0xd0/0xd0
[ 248.201034] [<ffffffff810dbfc9>] irq_forced_th
[ 248.201034] [<ffffffff810dbfc9>] irq_forced_thread_fn+0x29/0x70
[ 248.201101] [<ffffffff810dc328>] irq_thread+0x138/0x1b0
[ 248.201167] [<ffffffff810dc140>] ? wake_threads_waitq+0x30/0x30
[ 248.201234] [<ffffffff810dc1f0>] ? irq_thread_dtor+0xb0/
[ 248.201234] [<ffffffff810dc1f0>] ? irq_thread_dtor+0xb0/0xb0
[ 248.201301] [<ffffffff810a0448>] kthread+0xd8/0xf0
[ 248.201367] [<ffffffff810a0370>] ? kthread_worker_fn+0x170/0x170
[ 248.201435] [<ffffffff81829d0f>] ret_from_fork+0x3f/0x70
[ 248.201502] [<ffffffff810a0370>] ? kthread_worker_fn+0x170/0x170
[ 248.201569] ---
[ end trace ba42d5cb2a09b7a9 ]---
</>hpiLO-> exit

I try build original kernel from kernel.org(4.4.4). Configure with make-kpkg.
This kernel working perfectly.

Attached dmesg and config.

Attila - Please try the test kernel at http://people.canonical.com/~rtg/lp1554003/1/. Be sure to install both linux-image and linux-image-extra. This kernel has the following patches applied:

Revert "qla2xxx: Fix warning reported by static checker"
Revert "qla2xxx: Fix TMR ABORT interaction issue between qla2xxx and TCM"
Revert "qla2xxx: Use ATIO type to send correct tmr response"
Revert "qla2xxx: use TARGET_SCF_USE_CPUID flag to indiate CPU Affinity"

Reverting all qla2xxx patches since v4.4 isn't really an option since it would impact bug #1541456. If this test kernel doesn't work for you, then it would be helpful to know what was the last version that did work. The post v4.4 qla2xxx updates were first released in linux 4.4.0-4.19, so I'd start with a version prior to that.

Tim

I try kernel 4.4.0-12 (ubuntu branch). Same problem occured as what I experienced with the previous versions.

My tests:
Ubuntu branch: 4.4.0-8 - 4.4.0.12 kernel crashed
Kernel.org 4.4.4 kernel booting without problem
Kernel.org 4.5rc7 kernel crashed, same with ubuntu branch!

Does 4.4.0-7 work ?

4.4.0-7 crash

I checked older kernels:
4.4.0-3 good!
4.4.0-4 crash

I started a kernel bisect between 4.4.0-3 and 4.4.0-4. The kernel bisect will require testing of about 6 test kernels.

I built the first test kernel, up to the following commit:
b1aa887dffc9cb81d82b72bdf2af6c9e4b5a0294

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1554003

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

Just confirming this on an IBM HS21 with the nightly from 14-Mar-2016 07:01 @ http://cdimage.ubuntu.com/ubuntu-server/daily/current/xenial-server-amd64.iso
IBM Part Number 26R0892

I try 4.4.0-4.18~lp1554003Commitb1aa887df kernel, is good!

Attached dmesg.

I built the next test kernel, up to the following commit:
17c4da325caf67876458a05baa833a35931f3133

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1554003

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

4.4.1-040401-generic #201603161907 is crashed
attached screenshot

more screenshot

I built the next test kernel, up to the following commit:
fa87c4229357b70340ca35411a1c9013be535e11

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1554003

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

lp1554003Commitfa87c422 is good, dmesg attached.

I built the next test kernel, up to the following commit:
bdddded9ad074a8be28e0a3097598238a02c47c6

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1554003

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

I am also having this issue.

Ubuntu 4.4.0-4.18~lp1554003Commitbdddded9-generic 4.4.1

is good for me.

same, good!

I built the next test kernel, up to the following commit:
d3b9bcd9c7d8497eda30f5172503c871501e77b4

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1554003

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

lp1554003Commitd3b9bcd9c works fine, no panics.

same, good!

I built the next test kernel, up to the following commit:
b11f66aebbf8369707ddde3a9a7bb01b6cb41d58

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1554003

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

 b11f66aebbf8369707ddde3a9a7bb01b6cb41d58 is good!

The bisect reported the following commit as the first bad commit:

commit cdb898c52d1dfad4b4800b83a58b3fe5d352edde
Author: Quinn Tran <email address hidden>
Date: Thu Dec 17 14:57:05 2015 -0500

    qla2xxx: Add irq affinity notification

I'll build a test kernel with a revert of this commit for testing.

I built a Xenial test kernel with a revert of commit cdb898c5. However, it also required three other commits:

        5327c7d qla2xxx: use TARGET_SCF_USE_CPUID flag to indiate CPU Affinity
 9095ada target/transport: add flag to indicate CPU Affinity is observed
 fb3269b qla2xxx: Add selective command queuing

The test kernel can be downloaded from:

http://kernel.ubuntu.com/~jsalisbury/lp1554003

Can you test that kernel and report back if it has the bug or not?

Revert build is good! Dmesg attached.

Works fine here also.

Upstream sent a patch that may fix this bug without needed to revert any commits. I built a Xenial test kernel with this patch, which can be downloaded from:

http://kernel.ubuntu.com/~jsalisbury/lp1554003/

Can folks affected by this bug test this kernel and see if it fixes the bug?

Works for me.

[ 98.715393] qla2xxx [0000:00:00.0]-0005: : QLogic Fibre Channel HBA Driver: 8.07.00.26-k.
[ 98.715608] qla2xxx [0000:10:00.0]-001d: : Found an ISP2432 irq 18 iobase 0xffffc90003612000.
[ 98.716407] qla2xxx [0000:10:00.0]-0034:1: MSI-X: Unsupported ISP 2432 SSVID/SSDID (0x103C,0x1705).
[ 99.112157] scsi host1: qla2xxx
[ 99.112872] qla2xxx [0000:10:00.0]-00fb:1: QLogic QMH2462 - PCI-Express Dual Channel 4Gb Fibre Channel Mezzanine HBA.
[ 99.112883] qla2xxx [0000:10:00.0]-00fc:1: ISP2432: PCIe (2.5GT/s x4) @ 0000:10:00.0 hdma+ host#=1 fw=8.03.00 (9496).
[ 99.113074] qla2xxx [0000:10:00.1]-001d: : Found an ISP2432 irq 19 iobase 0xffffc9000361e000.
[ 99.113264] qla2xxx [0000:10:00.1]-0034:2: MSI-X: Unsupported ISP 2432 SSVID/SSDID (0x103C,0x1705).
[ 99.500141] scsi host2: qla2xxx
[ 99.500825] qla2xxx [0000:10:00.1]-00fb:2: QLogic QMH2462 - PCI-Express Dual Channel 4Gb Fibre Channel Mezzanine HBA.
[ 99.500836] qla2xxx [0000:10:00.1]-00fc:2: ISP2432: PCIe (2.5GT/s x4) @ 0000:10:00.1 hdma+ host#=2 fw=8.03.00 (9496).
[ 100.717278] qla2xxx [0000:10:00.0]-500a:1: LOOP UP detected (4 Gbps).
[ 101.105997] qla2xxx [0000:10:00.1]-500a:2: LOOP UP detected (4 Gbps).

lp1554003V2UpstreamPatch is good, dmesg attached.

This bug was fixed in the package linux - 4.4.0-21.37

---------------
linux (4.4.0-21.37) xenial; urgency=low

  [ Tim Gardner ]

  * Release Tracking Bug
    - LP: #1571791

  * linux: MokSBState is ignored (LP: #1571691)
    - SAUCE: (noup) MODSIGN: Import certificates from UEFI Secure Boot
    - SAUCE: (noup) efi: Disable secure boot if shim is in insecure mode
    - SAUCE: (noup) Display MOKSBState when disabled

linux (4.4.0-20.36) xenial; urgency=low

  [ Tim Gardner ]

  * Release Tracking Bug
    - LP: #1571069

  * sysfs mount failure during stateful lxd snapshots (LP: #1570906)
    - SAUCE: kernfs: Do not match superblock in another user namespace when
      mounting

  * Kernel Panic in Ubuntu 16.04 netboot installer (LP: #1570441)
    - x86/topology: Fix logical package mapping
    - x86/topology: Fix Intel HT disable
    - x86/topology: Use total_cpus not nr_cpu_ids for logical packages
    - xen/apic: Provide Xen-specific version of cpu_present_to_apicid APIC op
    - x86/topology: Fix AMD core count

  * [regression]: Failed to call clock_adjtime(): Invalid argument
    (LP: #1566465)
    - ntp: Fix ADJ_SETOFFSET being used w/ ADJ_NANO

linux (4.4.0-19.35) xenial; urgency=low

  [ Tim Gardner ]

  * Release Tracking Bug
    - LP: #1570348

  * CVE-2016-2847 (LP: #1554260)
    - pipe: limit the per-user amount of pages allocated in pipes

  * xenial kernel crash on HP BL460c G7 (qla24xx problem?) (LP: #1554003)
    - SAUCE: (noup) qla2xxx: Add irq affinity notification V2

  * arm64: guest hangs when ntpd is running (LP: #1549494)
    - SAUCE: (noup) KVM: arm/arm64: Handle forward time correction gracefully

  * linux: Enforce signed module loading when UEFI secure boot (LP: #1566221)
    - [Config] CONFIG_EFI_SECURE_BOOT_SIG_ENFORCE=y

  * s390/cpumf: Fix lpp detection (LP: #1555344)
    - s390/facilities: use stfl mnemonic instead of insn magic
    - s390/facilities: always use lowcore's stfle field for storing facility bits
    - s390/cpumf: Fix lpp detection

  * s390x kernel image needs weightwatchers (LP: #1536245)
    - [Config] s390x: Use compressed kernel bzImage

  * Surelock GA2 SP1: surelock02p05: Not seeing sgX devices for LUNs after
    upgrading to Ubuntu 16.04 (LP: #1567581)
    - Revert "UBUNTU: SAUCE: (noup) powerpc/pci: Assign fixed PHB number based on
      device-tree properties"

  * Backport upstream bugfixes to ubuntu-16.04 (LP: #1555765)
    - cpufreq: powernv: Define per_cpu chip pointer to optimize hot-path
    - Revert "cpufreq: postfix policy directory with the first CPU in related_cpus"
    - cpufreq: powernv: Add sysfs attributes to show throttle stats

  * systemd-modules-load.service: Failing due to missing module 'ib_iser' (LP: #1566468)
    - [Config] Add ib_iser to generic inclusion list

  * thunderx nic performance improvements (LP: #1567093)
    - net: thunderx: Set recevie buffer page usage count in bulk
    - net: thunderx: Adjust nicvf structure to reduce cache misses

  * fixes for thunderx nic in multiqueue mode (LP: #1567091)
    - net: thunderx: Fix for multiqset not configured upon interface toggle
    - net: thunderx: Fix for HW TSO not enabled for secondary qsets
    - net: thund...