Juniper Openstack

[14.04 3058] kernel crash in agent while running port mirroring testcase

Bug #1685181 reported by Suresh
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juniper Openstack
Status tracked in Trunk
R2.21.x
Fix Committed
Critical
Divakar Dharanalakota
Juniper Openstack r2.21.3
R2.22.x
Fix Committed
Critical
Divakar Dharanalakota
Juniper Openstack r2.22.3
R3.0
Fix Committed
Critical
Divakar Dharanalakota
Juniper Openstack r3.0.4.0
R3.1
Fix Committed
Critical
Divakar Dharanalakota
Juniper Openstack r3.1.3.0
R3.2
Fix Committed
Critical
Divakar Dharanalakota
Juniper Openstack r3.2.4.0
R3.2.3.x
Fix Committed
Critical
Divakar Dharanalakota
Juniper Openstack r3.2.3.1
Trunk
Fix Committed
Critical
Divakar Dharanalakota
Juniper Openstack r4.0.0.0-fcs "r4.0.0.0"

Bug Description

Kernel crash seen in compute node while running test_intf_mirror_src_cn1vn1_dst_cn2vn2_analyzer_cn3vn3 testcase in test_intf_mirror script.

  Created 2 VM in nodea35 and mirroring the packets in nodea34. kernel crash is seen in nodea35 when pinging between VM's.

  Copied the dump and dmesg logs of the crash to stack@10.204.216.49 (stack/stack@123) at /home/stack/suresh/kernel_crash/201704211406

Logs:
====
> /contrail-test/common/servicechain/mirror/verify.py(307)verify_port_mirroring()
-> session = ssh(host['host_ip'], host['username'], host['password'])
(Pdb) l
302 svm = mirr_vm.vm_obj
303 if svm.status == 'ACTIVE':
304 svm_name = svm.name
305 host = self.get_svm_compute(svm_name)
306 tapintf = self.get_svm_tapintf(svm_name)
307 -> session = ssh(host['host_ip'], host['username'], host['password'])
308 pcap = self.start_tcpdump(session, tapintf)
309 assert src_vm.ping_with_certainty(dst_vm.vm_ip, count=5, size='1400')
310 self.logger.info('Ping from %s to %s executed with c=5, expected mirrored packets 5 Ingress,5 Egress count = 10'
311 % (src_vm.vm_ip, dst_vm.vm_ip))
312 exp_count = 10
(Pdb) n
> /contrail-test/common/servicechain/mirror/verify.py(308)verify_port_mirroring()
-> pcap = self.start_tcpdump(session, tapintf)
(Pdb) n
2017-04-21 13:55:24,959 - INFO - Staring tcpdump to capture the mirrored packets.
> /contrail-test/common/servicechain/mirror/verify.py(309)verify_port_mirroring()
-> assert src_vm.ping_with_certainty(dst_vm.vm_ip, count=5, size='1400')
(Pdb) n
2017-04-21 13:57:33,050 - WARNING - Unable to run command ping -s 1400 -c 5 -W 1 204.200.183.3: Command failed to finish in 120 seconds
2017-04-21 13:57:38,053 - WARNING - Got exception in ping_to_ip:argument of type 'NoneType' is not iterable
2017-04-21 13:57:39,060 - WARNING - Unable to run command ping -s 1400 -c 5 -W 1 204.200.183.3: Low level socket error connecting to host 10.204.216.31 on port 22: Unable to connect to port 22 on 10.204.216.31 (tried 1 time)
2017-04-21 13:57:44,065 - WARNING - Got exception in ping_to_ip:argument of type 'NoneType' is not iterable
2017-04-21 13:57:45,071 - WARNING - Unable to run command ping -s 1400 -c 5 -W 1 204.200.183.3: Low level socket error connecting to host 10.204.216.31 on port 22: Unable to connect to port 22 on 10.204.216.31 (tried 1 time)

Sudheendra Rao (sudheendra-k)
tags: added: blocker
tags: added: vrouter
Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] master

Review in progress for https://review.opencontrail.org/30790
Submitter: Anand H. Krishnan (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/30790
Committed: http://github.com/Juniper/contrail-vrouter/commit/e18e6557a2e1053eabab946c292d452cdb8df325
Submitter: Zuul (<email address hidden>)
Branch: master

commit e18e6557a2e1053eabab946c292d452cdb8df325
Author: Anand H. Krishnan <email address hidden>
Date: Thu Apr 27 12:07:55 2017 +0530

Memset the control block to zeroes before calling skb_segment

skb_segment casts part of skb->cb to some structure to determine
tunnel header length. Having remnants of packet structure in cb
results in wrong tunnel header length calculation and thus wrong
memory copies to random memory locations. This problem is with
newer kernels (4.x).

Change-Id: Ifc2c7a4c5ea448cd57df88e51bb82a7f62c97cc6
Closes-Bug: #1685181

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.2

Review in progress for https://review.opencontrail.org/31041
Submitter: Anand H. Krishnan (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/31041
Committed: http://github.com/Juniper/contrail-vrouter/commit/3fbe0e1ff87a2c4751cec45a442e03ee16f2eb55
Submitter: Zuul (<email address hidden>)
Branch: R3.2

commit 3fbe0e1ff87a2c4751cec45a442e03ee16f2eb55
Author: Anand H. Krishnan <email address hidden>
Date: Thu Apr 27 12:07:55 2017 +0530

Memset the control block to zeroes before calling skb_segment

skb_segment casts part of skb->cb to some structure to determine
tunnel header length. Having remnants of packet structure in cb
results in wrong tunnel header length calculation and thus wrong
memory copies to random memory locations. This problem is with
newer kernels (4.x).

Change-Id: Ifc2c7a4c5ea448cd57df88e51bb82a7f62c97cc6
Closes-Bug: #1685181

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R2.21.x

Review in progress for https://review.opencontrail.org/31152
Submitter: Anand H. Krishnan (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R2.22.x

Review in progress for https://review.opencontrail.org/31153
Submitter: Anand H. Krishnan (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.0

Review in progress for https://review.opencontrail.org/31154
Submitter: Anand H. Krishnan (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.1

Review in progress for https://review.opencontrail.org/31155
Submitter: Anand H. Krishnan (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/31154
Committed: http://github.com/Juniper/contrail-vrouter/commit/d7ea2cce0b162e064f5b0201cc692fb260939ae0
Submitter: Zuul (<email address hidden>)
Branch: R3.0

commit d7ea2cce0b162e064f5b0201cc692fb260939ae0
Author: Anand H. Krishnan <email address hidden>
Date: Thu Apr 27 12:07:55 2017 +0530

Memset the control block to zeroes before calling skb_segment

skb_segment casts part of skb->cb to some structure to determine
tunnel header length. Having remnants of packet structure in cb
results in wrong tunnel header length calculation and thus wrong
memory copies to random memory locations. This problem is with
newer kernels (4.x).

Change-Id: Ifc2c7a4c5ea448cd57df88e51bb82a7f62c97cc6
Closes-Bug: #1685181

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/31152
Committed: http://github.com/Juniper/contrail-vrouter/commit/808c4646efc7dcae45630542c0e123ec02139600
Submitter: Zuul (<email address hidden>)
Branch: R2.21.x

commit 808c4646efc7dcae45630542c0e123ec02139600
Author: Anand H. Krishnan <email address hidden>
Date: Thu Apr 27 12:07:55 2017 +0530

Memset the control block to zeroes before calling skb_segment

skb_segment casts part of skb->cb to some structure to determine
tunnel header length. Having remnants of packet structure in cb
results in wrong tunnel header length calculation and thus wrong
memory copies to random memory locations. This problem is with
newer kernels (4.x).

Change-Id: Ifc2c7a4c5ea448cd57df88e51bb82a7f62c97cc6
Closes-Bug: #1685181

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/31155
Committed: http://github.com/Juniper/contrail-vrouter/commit/a6eb88fcf43a827da615768e34eef1697192183b
Submitter: Zuul (<email address hidden>)
Branch: R3.1

commit a6eb88fcf43a827da615768e34eef1697192183b
Author: Anand H. Krishnan <email address hidden>
Date: Thu Apr 27 12:07:55 2017 +0530

Memset the control block to zeroes before calling skb_segment

skb_segment casts part of skb->cb to some structure to determine
tunnel header length. Having remnants of packet structure in cb
results in wrong tunnel header length calculation and thus wrong
memory copies to random memory locations. This problem is with
newer kernels (4.x).

Change-Id: Ifc2c7a4c5ea448cd57df88e51bb82a7f62c97cc6
Closes-Bug: #1685181

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote :

Reviewed: https://review.opencontrail.org/31153
Committed: http://github.com/Juniper/contrail-vrouter/commit/e431f08a615b3956c29f962efe820c18a42aac60
Submitter: Zuul (<email address hidden>)
Branch: R2.22.x

commit e431f08a615b3956c29f962efe820c18a42aac60
Author: Anand H. Krishnan <email address hidden>
Date: Thu Apr 27 12:07:55 2017 +0530

Memset the control block to zeroes before calling skb_segment

skb_segment casts part of skb->cb to some structure to determine
tunnel header length. Having remnants of packet structure in cb
results in wrong tunnel header length calculation and thus wrong
memory copies to random memory locations. This problem is with
newer kernels (4.x).

Change-Id: Ifc2c7a4c5ea448cd57df88e51bb82a7f62c97cc6
Closes-Bug: #1685181

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : [Review update] R3.2.3.x

Review in progress for https://review.opencontrail.org/32593
Submitter: Vinay Vithal Mahuli (<email address hidden>)

Revision history for this message
OpenContrail Admin (ci-admin-f) wrote : A change has been merged

Reviewed: https://review.opencontrail.org/32593
Committed: http://github.com/Juniper/contrail-vrouter/commit/4fb89e127de1a70aba07b0835ce86bf876bbf66b
Submitter: Zuul (<email address hidden>)
Branch: R3.2.3.x

commit 4fb89e127de1a70aba07b0835ce86bf876bbf66b
Author: Anand H. Krishnan <email address hidden>
Date: Thu Apr 27 12:07:55 2017 +0530

Memset the control block to zeroes before calling skb_segment

skb_segment casts part of skb->cb to some structure to determine
tunnel header length. Having remnants of packet structure in cb
results in wrong tunnel header length calculation and thus wrong
memory copies to random memory locations. This problem is with
newer kernels (4.x).

Change-Id: Ifc2c7a4c5ea448cd57df88e51bb82a7f62c97cc6
Closes-Bug: #1685181

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.