That one completed its first run, but then crashed when bringing CPU 14 back online, with the following dmesg output:
[ 163.176945] ------------[ cut here ]------------ [ 163.176949] kernel BUG at /home/jsalisbury/bugs/lp1733662/ubuntu-artful/mm/slub.c:3878! [ 163.178043] invalid opcode: 0000 [#1] SMP [ 163.178995] Modules linked in: nls_iso8859_1 intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass intel_cstate joydev input_leds shpchp ipmi_ssif intel_rapl_perf acpi_power_meter lpc_ich ipmi_si ipmi_devintf ipmi_msghandler acpi_pad mac_hid mei_me mei ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear ses enclosure scsi_transport_sas mgag200 ttm drm_kms_helper crct10dif_pclmul crc32_pclmul ghash_clmulni_intel syscopyarea pcbc sysfillrect fnic aesni_intel hid_generic sysimgblt igb fb_sys_fops aes_x86_64 dca usbhid crypto_simd i2c_algo_bit glue_helper libfcoe hid ahci ptp libfc mxm_wmi cryptd libahci [ 163.186785] drm pps_core enic scsi_transport_fc megaraid_sas wmi [ 163.188025] CPU: 14 PID: 93 Comm: cpuhp/14 Not tainted 4.13.0-13-generic #14~lp1733662Commite6108d5475696 [ 163.189294] Hardware name: Cisco Systems Inc UCSC-C240-M4L/UCSC-C240-M4L, BIOS C240M4.2.0.10c.0.032320160820 03/23/2016 [ 163.190606] task: ffff8dbaf809c5c0 task.stack: ffffae2acc8a8000 [ 163.191926] RIP: 0010:kfree+0x11c/0x160 [ 163.193255] RSP: 0000:ffffae2acc8abb80 EFLAGS: 00010246 [ 163.194600] RAX: fffff9cb3bff0020 RBX: ffff8dba00000000 RCX: ffffae2acc8abb60 [ 163.195954] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000728480000000 [ 163.197311] RBP: ffffae2acc8abb98 R08: ffffae2acc8abaec R09: 0000000000000002 [ 163.198703] R10: fffff9cb3c000000 R11: 0000000000000000 R12: ffff8d9aff94beb0 [ 163.200096] R13: ffffffffa6f2034b R14: ffff8dbaf27e4318 R15: ffff8dbaf27e4200 [ 163.201497] FS: 0000000000000000(0000) GS:ffff8dbaff380000(0000) knlGS:0000000000000000 [ 163.202919] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 163.204351] CR2: 0000000000000000 CR3: 000000101aa09000 CR4: 00000000001406e0 [ 163.205802] Call Trace: [ 163.207253] acpi_ns_get_node_unlocked+0xac/0xd8 [ 163.208704] ? kernfs_add_one+0xe4/0x130 [ 163.210183] ? down_timeout+0x37/0x60 [ 163.211644] ? acpi_os_wait_semaphore+0x4c/0x70 [ 163.213098] acpi_ns_get_node+0x41/0x58 [ 163.214550] ? acpi_ns_get_node+0x41/0x58 [ 163.216016] acpi_get_handle+0x95/0xbe [ 163.217486] acpi_has_method+0x25/0x40 [ 163.218932] acpi_processor_get_performance_info+0x57/0x580 [ 163.220391] ? wrmsrl_on_cpu+0x57/0x70 [ 163.221870] acpi_processor_register_performance+0x5e/0xd0 [ 163.223354] __intel_pstate_cpu_init.part.16+0xed/0x2e0 [ 163.224835] ? intel_pstate_init_cpu+0xc9/0x2d0 [ 163.226323] intel_pstate_cpu_init+0x24/0x40 [ 163.227819] cpufreq_online+0xd8/0x750 [ 163.229301] ? cpufreq_online+0x750/0x750 [ 163.230781] cpuhp_cpufreq_online+0xe/0x20 [ 163.232262] cpuhp_invoke_callback+0x84/0x3b0 [ 163.233758] cpuhp_up_callbacks+0x36/0xc0 [ 163.235254] cpuhp_thread_fun+0xd4/0xe0 [ 163.236731] smpboot_thread_fn+0xec/0x160 [ 163.238210] kthread+0x125/0x140 [ 163.239693] ? sort_range+0x30/0x30 [ 163.241165] ? kthread_create_on_node+0x70/0x70 [ 163.242629] ret_from_fork+0x25/0x30 [ 163.244061] Code: 08 49 83 c4 18 48 89 da 4c 89 ee ff d0 49 8b 04 24 48 85 c0 75 e6 e9 0e ff ff ff 49 8b 02 f6 c4 80 75 0a 49 8b 42 20 a8 01 75 02 <0f> 0b 49 8b 02 31 f6 f6 c4 80 74 04 41 8b 72 6c 4c 89 d7 e8 2c [ 163.247030] RIP: kfree+0x11c/0x160 RSP: ffffae2acc8abb80 [ 163.248463] ---[ end trace e22fa4721cb983b5 ]--- [ 168.454846] ------------[ cut here ]------------ [ 168.456219] kernel BUG at /home/jsalisbury/bugs/lp1733662/ubuntu-artful/mm/slub.c:3878! [ 168.457561] invalid opcode: 0000 [#2] SMP [ 168.458849] Modules linked in: nls_iso8859_1 intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass intel_cstate joydev input_leds shpchp ipmi_ssif intel_rapl_perf acpi_power_meter lpc_ich ipmi_si ipmi_devintf ipmi_msghandler acpi_pad mac_hid mei_me mei ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear ses enclosure scsi_transport_sas mgag200 ttm drm_kms_helper crct10dif_pclmul crc32_pclmul ghash_clmulni_intel syscopyarea pcbc sysfillrect fnic aesni_intel hid_generic sysimgblt igb fb_sys_fops aes_x86_64 dca usbhid crypto_simd i2c_algo_bit glue_helper libfcoe hid ahci ptp libfc mxm_wmi cryptd libahci [ 168.468659] drm pps_core enic scsi_transport_fc megaraid_sas wmi [ 168.470126] CPU: 0 PID: 2683 Comm: irqbalance Tainted: G D 4.13.0-13-generic #14~lp1733662Commite6108d5475696 [ 168.471648] Hardware name: Cisco Systems Inc UCSC-C240-M4L/UCSC-C240-M4L, BIOS C240M4.2.0.10c.0.032320160820 03/23/2016 [ 168.473183] task: ffff8dbae2bf9740 task.stack: ffffae2acf51c000 [ 168.474734] RIP: 0010:kfree+0x11c/0x160 [ 168.476246] RSP: 0018:ffffae2acf51fa08 EFLAGS: 00010246 [ 168.477765] RAX: fffff9cb3bff0020 RBX: ffff8dba00000000 RCX: 0000000000000000 [ 168.479292] RDX: 0000000000000000 RSI: ffff8dbae313ed10 RDI: 0000728480000000 [ 168.480797] RBP: ffffae2acf51fa20 R08: ffff8dbae2a5bac8 R09: 0000000180220021 [ 168.482306] R10: fffff9cb3c000000 R11: 0000000000000001 R12: ffff8dbaf2f60960 [ 168.483831] R13: ffffffffa6bdd4e0 R14: ffff8dbae33fbcd8 R15: ffff8dbae33fae00 [ 168.485365] FS: 00007f342d25a740(0000) GS:ffff8d9affc00000(0000) knlGS:0000000000000000 [ 168.486926] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 168.488478] CR2: 0000560651c9f3a8 CR3: 0000003ff4879000 CR4: 00000000001406f0 [ 168.490066] Call Trace: [ 168.491641] kfree_const+0x20/0x30 [ 168.493227] kernfs_put+0x71/0x180 [ 168.494793] kernfs_dop_release+0x12/0x20 [ 168.496367] __dentry_kill+0xe5/0x150 [ 168.497925] shrink_dentry_list+0x11f/0x2e0 [ 168.499478] d_invalidate+0x67/0x110 [ 168.501018] lookup_fast+0x2b9/0x310 [ 168.502552] ? dput.part.23+0x2d/0x1e0 [ 168.504096] walk_component+0x49/0x340 [ 168.505624] ? kernfs_iop_permission+0x4f/0x60 [ 168.507170] link_path_walk+0x1bc/0x590 [ 168.508703] ? path_init+0x177/0x2f0 [ 168.510248] path_lookupat+0x56/0x1f0 [ 168.511794] filename_lookup+0xb6/0x190 [ 168.513341] ? sprintf+0x51/0x70 [ 168.514885] ? __check_object_size+0xaf/0x1b0 [ 168.516429] ? strncpy_from_user+0x4d/0x170 [ 168.517968] user_path_at_empty+0x36/0x40 [ 168.519514] ? user_path_at_empty+0x36/0x40 [ 168.521020] vfs_statx+0x76/0xe0 [ 168.522481] SYSC_newstat+0x3d/0x70 [ 168.523922] ? ____fput+0xe/0x10 [ 168.525346] ? task_work_run+0x7b/0x90 [ 168.526777] ? exit_to_usermode_loop+0x9b/0xd0 [ 168.528186] SyS_newstat+0xe/0x10 [ 168.529565] entry_SYSCALL_64_fastpath+0x1e/0xa9 [ 168.530924] RIP: 0033:0x7f342c34abb5 [ 168.532229] RSP: 002b:00007ffcd3f64668 EFLAGS: 00000246 ORIG_RAX: 0000000000000004 [ 168.533535] RAX: ffffffffffffffda RBX: 0000000000b95fa0 RCX: 00007f342c34abb5 [ 168.534805] RDX: 00007ffcd3f646c0 RSI: 00007ffcd3f646c0 RDI: 00007ffcd3f65f50 [ 168.536043] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000038 [ 168.537240] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 [ 168.538390] R13: 00007ffcd3f64f6b R14: 0000000000b95fa0 R15: 0000000000b96250 [ 168.539524] Code: 08 49 83 c4 18 48 89 da 4c 89 ee ff d0 49 8b 04 24 48 85 c0 75 e6 e9 0e ff ff ff 49 8b 02 f6 c4 80 75 0a 49 8b 42 20 a8 01 75 02 <0f> 0b 49 8b 02 31 f6 f6 c4 80 74 04 41 8b 72 6c 4c 89 d7 e8 2c [ 168.541855] RIP: kfree+0x11c/0x160 RSP: ffffae2acf51fa08 [ 168.543000] ---[ end trace e22fa4721cb983b6 ]---
The system is semi-responsive; bash continues to run, but most external commands seem to hang. Thus, I've rebooted via the BMC.
That one completed its first run, but then crashed when bringing CPU 14 back online, with the following dmesg output:
[ 163.176945] ------------[ cut here ]------------ y/bugs/ lp1733662/ ubuntu- artful/ mm/slub. c:3878! temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass intel_cstate joydev input_leds shpchp ipmi_ssif intel_rapl_perf acpi_power_meter lpc_ich ipmi_si ipmi_devintf ipmi_msghandler acpi_pad mac_hid mei_me mei ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_ iscsi autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear ses enclosure scsi_transport_sas mgag200 ttm drm_kms_helper crct10dif_pclmul crc32_pclmul ghash_clmulni_intel syscopyarea pcbc sysfillrect fnic aesni_intel hid_generic sysimgblt igb fb_sys_fops aes_x86_64 dca usbhid crypto_simd i2c_algo_bit glue_helper libfcoe hid ahci ptp libfc mxm_wmi cryptd libahci mmite6108d54756 96 M4L/UCSC- C240-M4L, BIOS C240M4. 2.0.10c. 0.032320160820 03/23/2016 0x11c/0x160 8abb80 EFLAGS: 00010246 0(0000) GS:ffff8dbaff38 0000(0000) knlGS:000000000 0000000 get_node_ unlocked+ 0xac/0xd8 add_one+ 0xe4/0x130 0x37/0x60 wait_semaphore+ 0x4c/0x70 get_node+ 0x41/0x58 get_node+ 0x41/0x58 handle+ 0x95/0xbe method+ 0x25/0x40 get_performance _info+0x57/ 0x580 on_cpu+ 0x57/0x70 register_ performance+ 0x5e/0xd0 pstate_ cpu_init. part.16+ 0xed/0x2e0 init_cpu+ 0xc9/0x2d0 cpu_init+ 0x24/0x40 online+ 0xd8/0x750 online+ 0x750/0x750 online+ 0xe/0x20 callback+ 0x84/0x3b0 callbacks+ 0x36/0xc0 fun+0xd4/ 0xe0 thread_ fn+0xec/ 0x160 0x30/0x30 create_ on_node+ 0x70/0x70 fork+0x25/ 0x30 y/bugs/ lp1733662/ ubuntu- artful/ mm/slub. c:3878! temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass intel_cstate joydev input_leds shpchp ipmi_ssif intel_rapl_perf acpi_power_meter lpc_ich ipmi_si ipmi_devintf ipmi_msghandler acpi_pad mac_hid mei_me mei ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_ iscsi autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear ses enclosure scsi_transport_sas mgag200 ttm drm_kms_helper crct10dif_pclmul crc32_pclmul ghash_clmulni_intel syscopyarea pcbc sysfillrect fnic aesni_intel hid_generic sysimgblt igb fb_sys_fops aes_x86_64 dca usbhid crypto_simd i2c_algo_bit glue_helper libfcoe hid ahci ptp libfc mxm_wmi cryptd libahci mmite6108d54756 96 M4L/UCSC- C240-M4L, BIOS C240M4. 2.0.10c. 0.032320160820 03/23/2016 0x11c/0x160 51fa08 EFLAGS: 00010246 0(0000) GS:ffff8d9affc0 0000(0000) knlGS:000000000 0000000 0x20/0x30 put+0x71/ 0x180 dop_release+ 0x12/0x20 kill+0xe5/ 0x150 dentry_ list+0x11f/ 0x2e0 0x67/0x110 fast+0x2b9/ 0x310 23+0x2d/ 0x1e0 0x49/0x340 iop_permission+ 0x4f/0x60 walk+0x1bc/ 0x590 0x177/0x2f0 0x56/0x1f0 lookup+ 0xb6/0x190 object_ size+0xaf/ 0x1b0 from_user+ 0x4d/0x170 at_empty+ 0x36/0x40 at_empty+ 0x36/0x40 0x3d/0x70 run+0x7b/ 0x90 usermode_ loop+0x9b/ 0xd0 0xe/0x10 64_fastpath+ 0x1e/0xa9 f64668 EFLAGS: 00000246 ORIG_RAX: 0000000000000004
[ 163.176949] kernel BUG at /home/jsalisbur
[ 163.178043] invalid opcode: 0000 [#1] SMP
[ 163.178995] Modules linked in: nls_iso8859_1 intel_rapl x86_pkg_
[ 163.186785] drm pps_core enic scsi_transport_fc megaraid_sas wmi
[ 163.188025] CPU: 14 PID: 93 Comm: cpuhp/14 Not tainted 4.13.0-13-generic #14~lp1733662Co
[ 163.189294] Hardware name: Cisco Systems Inc UCSC-C240-
[ 163.190606] task: ffff8dbaf809c5c0 task.stack: ffffae2acc8a8000
[ 163.191926] RIP: 0010:kfree+
[ 163.193255] RSP: 0000:ffffae2acc
[ 163.194600] RAX: fffff9cb3bff0020 RBX: ffff8dba00000000 RCX: ffffae2acc8abb60
[ 163.195954] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000728480000000
[ 163.197311] RBP: ffffae2acc8abb98 R08: ffffae2acc8abaec R09: 0000000000000002
[ 163.198703] R10: fffff9cb3c000000 R11: 0000000000000000 R12: ffff8d9aff94beb0
[ 163.200096] R13: ffffffffa6f2034b R14: ffff8dbaf27e4318 R15: ffff8dbaf27e4200
[ 163.201497] FS: 000000000000000
[ 163.202919] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 163.204351] CR2: 0000000000000000 CR3: 000000101aa09000 CR4: 00000000001406e0
[ 163.205802] Call Trace:
[ 163.207253] acpi_ns_
[ 163.208704] ? kernfs_
[ 163.210183] ? down_timeout+
[ 163.211644] ? acpi_os_
[ 163.213098] acpi_ns_
[ 163.214550] ? acpi_ns_
[ 163.216016] acpi_get_
[ 163.217486] acpi_has_
[ 163.218932] acpi_processor_
[ 163.220391] ? wrmsrl_
[ 163.221870] acpi_processor_
[ 163.223354] __intel_
[ 163.224835] ? intel_pstate_
[ 163.226323] intel_pstate_
[ 163.227819] cpufreq_
[ 163.229301] ? cpufreq_
[ 163.230781] cpuhp_cpufreq_
[ 163.232262] cpuhp_invoke_
[ 163.233758] cpuhp_up_
[ 163.235254] cpuhp_thread_
[ 163.236731] smpboot_
[ 163.238210] kthread+0x125/0x140
[ 163.239693] ? sort_range+
[ 163.241165] ? kthread_
[ 163.242629] ret_from_
[ 163.244061] Code: 08 49 83 c4 18 48 89 da 4c 89 ee ff d0 49 8b 04 24 48 85 c0 75 e6 e9 0e ff ff ff 49 8b 02 f6 c4 80 75 0a 49 8b 42 20 a8 01 75 02 <0f> 0b 49 8b 02 31 f6 f6 c4 80 74 04 41 8b 72 6c 4c 89 d7 e8 2c
[ 163.247030] RIP: kfree+0x11c/0x160 RSP: ffffae2acc8abb80
[ 163.248463] ---[ end trace e22fa4721cb983b5 ]---
[ 168.454846] ------------[ cut here ]------------
[ 168.456219] kernel BUG at /home/jsalisbur
[ 168.457561] invalid opcode: 0000 [#2] SMP
[ 168.458849] Modules linked in: nls_iso8859_1 intel_rapl x86_pkg_
[ 168.468659] drm pps_core enic scsi_transport_fc megaraid_sas wmi
[ 168.470126] CPU: 0 PID: 2683 Comm: irqbalance Tainted: G D 4.13.0-13-generic #14~lp1733662Co
[ 168.471648] Hardware name: Cisco Systems Inc UCSC-C240-
[ 168.473183] task: ffff8dbae2bf9740 task.stack: ffffae2acf51c000
[ 168.474734] RIP: 0010:kfree+
[ 168.476246] RSP: 0018:ffffae2acf
[ 168.477765] RAX: fffff9cb3bff0020 RBX: ffff8dba00000000 RCX: 0000000000000000
[ 168.479292] RDX: 0000000000000000 RSI: ffff8dbae313ed10 RDI: 0000728480000000
[ 168.480797] RBP: ffffae2acf51fa20 R08: ffff8dbae2a5bac8 R09: 0000000180220021
[ 168.482306] R10: fffff9cb3c000000 R11: 0000000000000001 R12: ffff8dbaf2f60960
[ 168.483831] R13: ffffffffa6bdd4e0 R14: ffff8dbae33fbcd8 R15: ffff8dbae33fae00
[ 168.485365] FS: 00007f342d25a74
[ 168.486926] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 168.488478] CR2: 0000560651c9f3a8 CR3: 0000003ff4879000 CR4: 00000000001406f0
[ 168.490066] Call Trace:
[ 168.491641] kfree_const+
[ 168.493227] kernfs_
[ 168.494793] kernfs_
[ 168.496367] __dentry_
[ 168.497925] shrink_
[ 168.499478] d_invalidate+
[ 168.501018] lookup_
[ 168.502552] ? dput.part.
[ 168.504096] walk_component+
[ 168.505624] ? kernfs_
[ 168.507170] link_path_
[ 168.508703] ? path_init+
[ 168.510248] path_lookupat+
[ 168.511794] filename_
[ 168.513341] ? sprintf+0x51/0x70
[ 168.514885] ? __check_
[ 168.516429] ? strncpy_
[ 168.517968] user_path_
[ 168.519514] ? user_path_
[ 168.521020] vfs_statx+0x76/0xe0
[ 168.522481] SYSC_newstat+
[ 168.523922] ? ____fput+0xe/0x10
[ 168.525346] ? task_work_
[ 168.526777] ? exit_to_
[ 168.528186] SyS_newstat+
[ 168.529565] entry_SYSCALL_
[ 168.530924] RIP: 0033:0x7f342c34abb5
[ 168.532229] RSP: 002b:00007ffcd3
[ 168.533535] RAX: ffffffffffffffda RBX: 0000000000b95fa0 RCX: 00007f342c34abb5
[ 168.534805] RDX: 00007ffcd3f646c0 RSI: 00007ffcd3f646c0 RDI: 00007ffcd3f65f50
[ 168.536043] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000038
[ 168.537240] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
[ 168.538390] R13: 00007ffcd3f64f6b R14: 0000000000b95fa0 R15: 0000000000b96250
[ 168.539524] Code: 08 49 83 c4 18 48 89 da 4c 89 ee ff d0 49 8b 04 24 48 85 c0 75 e6 e9 0e ff ff ff 49 8b 02 f6 c4 80 75 0a 49 8b 42 20 a8 01 75 02 <0f> 0b 49 8b 02 31 f6 f6 c4 80 74 04 41 8b 72 6c 4c 89 d7 e8 2c
[ 168.541855] RIP: kfree+0x11c/0x160 RSP: ffffae2acf51fa08
[ 168.543000] ---[ end trace e22fa4721cb983b6 ]---
The system is semi-responsive; bash continues to run, but most external commands seem to hang. Thus, I've rebooted via the BMC.