kernel crash in ipv4_dst_destroy

Bug #1514363 reported by Tom Eichhorn
28
This bug affects 6 people
Affects Status Importance Assigned to Milestone
linux-lts-vivid (Ubuntu)
Confirmed
Undecided
Unassigned

Bug Description

Hi,

we have seen a GPF resulting in ipv4_dst_destroy:

messages-2015-11-08:Nov 8 20:23:05 bbk130129 kernel: [205505.064821] device tap406dbc6f-a8 left promiscuous mode
messages-2015-11-08:Nov 8 20:23:08 bbk130129 kernel: [205505.534610] general protection fault: 0000 [#1] SMP
messages-2015-11-08:Nov 8 20:23:08 bbk130129 kernel: [205505.578139] Modules linked in: kvm_intel kvm vport_vxlan vport_gre gre openvswitch dm_multipath scsi_dh nbd sch_fq_codel xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat bridge ebtable_filter ebtables ipmi_devintf btrfs xor raid6_pq ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs libcrc32c vhost_net vhost macvtap macvlan ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables ipt_REJECT nf_reject_ipv4 xt_limit xt_multiport xt_CT iptable_raw xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack iptable_filter ip_tables x_tables mlx4_en(OE) mlx4_core(OE) mlx_compat(OE) isofs ipmi_ssif intel_rapl iosf_mbi x86_pkg_temp_thermal intel_powerclamp coretemp crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd sb_edac edac_core lpc_ich hpilo ioatdma ipmi_si 8250_fintek ipmi_msghandler wmi acpi_power_meter mac_hid shpchp bonding 8021q garp mrp stp llc lp parport nls_iso8859_1 ixgbe dca vxlan ip6_udp_tunnel tg3 udp_tunnel mpt3sas(OE) raid_class ptp configfs pps_core mdio scsi_transport_sas
messages-2015-11-08:Nov 8 20:23:08 bbk130129 kernel: [205506.275481] CPU: 42 PID: 258 Comm: rcuos/35 Tainted: G OE 3.19.0-32-generic #37~14.04.1-Ubuntu
messages-2015-11-08:Nov 8 20:23:08 bbk130129 kernel: [205506.349044] Hardware name: HP ProLiant DL380 Gen9, BIOS P89 07/20/2015
messages-2015-11-08:Nov 8 20:23:08 bbk130129 kernel: [205506.404252] task: ffff882fa5823110 ti: ffff882fa5848000 task.ti: ffff882fa5848000
messages-2015-11-08:Nov 8 20:23:08 bbk130129 kernel: [205506.463760] RIP: 0010:[<ffffffff816de782>] [<ffffffff816de782>] ipv4_dst_destroy+0x42/0x80
messages-2015-11-08:Nov 8 20:23:08 bbk130129 kernel: [205506.527658] RSP: 0018:ffff882fa584bde8 EFLAGS: 00010246
messages-2015-11-08:Nov 8 20:23:08 bbk130129 kernel: [205506.578843] RAX: dead000000200200 RBX: ffff882f1e961200 RCX: 0000000000000060
messages-2015-11-08:Nov 8 20:23:08 bbk130129 kernel: [205506.636886] RDX: dead000000100100 RSI: 00000000fffffe01 RDI: ffffffff81fc5d84
messages-2015-11-08:Nov 8 20:23:08 bbk130129 kernel: [205506.696883] RBP: ffff882fa584bdf8 R08: ffff882f1e961200 R09: 00000001802a000b
messages-2015-11-08:Nov 8 20:23:08 bbk130129 kernel: [205506.756193] R10: ffffffff816b30f5 R11: ffffea00bc7a5800 R12: 0000000000000000
messages-2015-11-08:Nov 8 20:23:08 bbk130129 kernel: [205506.814938] R13: 0000000000000001 R14: ffff882f1e961080 R15: ffff882fa5823110
messages-2015-11-08:Nov 8 20:23:08 bbk130129 kernel: [205506.878741] FS: 0000000000000000(0000) GS:ffff885fbf440000(0000) knlGS:0000000000000000
messages-2015-11-08:Nov 8 20:23:08 bbk130129 kernel: [205506.943046] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
messages-2015-11-08:Nov 8 20:23:08 bbk130129 kernel: [205506.997859] CR2: 00007ffd99ac3640 CR3: 0000000002c16000 CR4: 00000000001427e0
messages-2015-11-08:Nov 8 20:23:08 bbk130129 kernel: [205507.057616] Stack:
messages-2015-11-08:Nov 8 20:23:08 bbk130129 kernel: [205507.093614] 0000000000000000 ffff882f1e961200 ffff882fa584be18 ffffffff816b30d2
messages-2015-11-08:Nov 8 20:23:08 bbk130129 kernel: [205507.155762] ffff882f1e961200 ffff882fbfcf4bc0 ffff882fa584be28 ffffffff816b33ae
messages-2015-11-08:Nov 8 20:23:08 bbk130129 kernel: [205507.217537] ffff882fa584beb8 ffffffff810d542f ffff882fa5823110 ffff882fa584be78
messages-2015-11-08:Nov 8 20:23:08 bbk130129 kernel: [205507.281530] Call Trace:
messages-2015-11-08:Nov 8 20:23:08 bbk130129 kernel: [205507.321761] [<ffffffff816b30d2>] dst_destroy+0x32/0xe0
messages-2015-11-08:Nov 8 20:23:08 bbk130129 kernel: [205507.373239] [<ffffffff816b33ae>] dst_destroy_rcu+0xe/0x20
messages-2015-11-08:Nov 8 20:23:08 bbk130129 kernel: [205507.424174] [<ffffffff810d542f>] rcu_nocb_kthread+0x19f/0x590
messages-2015-11-08:Nov 8 20:23:08 bbk130129 kernel: [205507.474698] [<ffffffff810b4e30>] ? prepare_to_wait_event+0x110/0x110
messages-2015-11-08:Nov 8 20:23:08 bbk130129 kernel: [205507.530284] [<ffffffff810d5290>] ? rcu_spawn_all_nocb_kthreads.part.54+0x160/0x160
messages-2015-11-08:Nov 8 20:23:08 bbk130129 kernel: [205507.589559] [<ffffffff81093822>] kthread+0xd2/0xf0
messages-2015-11-08:Nov 8 20:23:08 bbk130129 kernel: [205507.635397] [<ffffffff81093750>] ? kthread_create_on_node+0x1c0/0x1c0
messages-2015-11-08:Nov 8 20:23:08 bbk130129 kernel: [205507.691201] [<ffffffff817b6d18>] ret_from_fork+0x58/0x90
messages-2015-11-08:Nov 8 20:23:08 bbk130129 kernel: [205507.741015] [<ffffffff81093750>] ? kthread_create_on_node+0x1c0/0x1c0
messages-2015-11-08:Nov 8 20:23:08 bbk130129 kernel: [205507.798002] Code: 08 48 39 87 b0 00 00 00 74 4f 48 c7 c7 84 5d fc 81 e8 a3 82 0d 00 48 8b 83 b8 00 00 00 48 8b 93 b0 00 00 00 48 c7 c7 84 5d fc 81 <48> 89 42 08 48 89 10 48 b8 00 01 10 00 00 00 ad de 48 89 83 b0
messages-2015-11-08:Nov 8 20:23:08 bbk130129 kernel: [205508.001730] RSP <ffff882fa584bde8>
messages-2015-11-08:Nov 8 20:23:08 bbk130129 kernel: [205508.087352] ---[ end trace 645de9b56a152109 ]---

What has happened is that openstack has destroyed a VM and a network interface has been removed from the kernel:

auth-2015-11-08.log:Nov 8 20:23:01 bbk130129 sudo: nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/nova-rootwrap /etc/nova/rootwrap.conf mm-ctl --unbind-port 406dbc6f-a8fb-42f1-8f8d-a668698c7676
auth-2015-11-08.log:Nov 8 20:23:01 bbk130129 sudo: pam_unix(sudo:session): session opened for user root by (uid=0)
auth-2015-11-08.log:Nov 8 20:23:05 bbk130129 sudo: pam_unix(sudo:session): session closed for user root
auth-2015-11-08.log:Nov 8 20:23:05 bbk130129 sudo: nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/nova-rootwrap /etc/nova/rootwrap.conf ip link delete tap406dbc6f-a8

Some googling had brought up http://lists.openwall.net/netdev/2014/01/22/29 and http://lists.openwall.net/netdev/2014/01/18/5,
which might here also be the problem.

I have not yet been able to reproduce this bug, but will update this bug if it happens again.

ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: linux-image-3.19.0-32-generic 3.19.0-32.37~14.04.1
ProcVersionSignature: Ubuntu 3.19.0-32.37~14.04.1-generic 3.19.8-ckt7
Uname: Linux 3.19.0-32-generic x86_64
ApportVersion: 2.14.1-0ubuntu3.18
Architecture: amd64
Date: Mon Nov 9 09:09:04 2015
SourcePackage: linux-lts-vivid
UpgradeStatus: No upgrade log present (probably fresh install)

Revision history for this message
Tom Eichhorn (tomeichhorn) wrote :
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in linux-lts-vivid (Ubuntu):
status: New → Confirmed
Revision history for this message
csmcd5 (geek-n) wrote :

On our product we have recently seen kernel crashes showing the same call chain as above. It happened on two different hosts only 26 minutes apart.

We have a Python program using scapy/sniff. Its effect is to repeatedly put all interfaces into promiscuous mode and then take them out. There are about 10 interfaces, mostly vlan, and it does not create/delete interfaces. This has been running on 7 hosts every 1-3 minutes for days.

Linux ir6 3.16.0-44-iridium #59~14.04.1 SMP PREEMPT Tue May 23 01:10:24 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

I've seen other Google results for crashes in ipv4_dst_destroy that are preceded by "promiscuous mode" notices in the kernel log.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.