R1.10-build-22-ubuntu-havana-ssh to vrouter stuck

Hi Team,

We are seeing the issue with R1.10 build-22 ubuntu Havana multimode sanity. One of the vrouter a6s19 not able to connect through ssh. Can someone take a look at the setup.

Console error shows libvirtd blocked morethan 120 sec. screenshot attached

A6s32, a6s33, a6s38, a6s19, a6s22

Vrouter node: a6s19 and a6s22
Console of a6s19: 10.84.6.69

Note: Aborted all other sanity runs on this setup to debug this issue. Will run the necessary sanity locally.

Thanks,
Shaju
From: Shaju V.K
Sent: Thursday, August 21, 2014 4:08 PM
To: Anand H Krishnan; Ashish Ranjan; Hari Prasad Killi; Rajagopalan Sivaramakrishnan
Cc: Divakar Dharanalakota; Praveen K V
Subject: RE: Ubuntu havana-2304 build- lost ssh connectivity from cfgm to vrouter

Hi Anand,

I couldn’t hit the ssh issue with latest ubuntu build 2305. But in console of vrouter nodes (a5s9 and a5s10) some errors related to ext4 file system. Could you please check the vrouter console- http://10.84.6.159/ and http://10.84.6.160/

Thanks,
Shaju

From: Anand H Krishnan
Sent: Thursday, August 21, 2014 10:27 AM
To: Ashish Ranjan; Hari Prasad Killi; Rajagopalan Sivaramakrishnan
Cc: Shaju V.K; Divakar Dharanalakota; Praveen K V
Subject: RE: Ubuntu havana-2304 build- lost ssh connectivity from cfgm to vrouter

The system has recovered now. The traces aren't really helpful. Hence, if
we are able to reproduce, it will be helpful. Shaju?

Thanks,
Anand
________________________________________
From: Ashish Ranjan
Sent: Thursday, August 21, 2014 10:27 PM
To: Hari Prasad Killi; Anand H Krishnan; Rajagopalan Sivaramakrishnan
Cc: Shaju V.K; Divakar Dharanalakota; Praveen K V
Subject: Re: Ubuntu havana-2304 build- lost ssh connectivity from cfgm to vrouter

Anand, Raja we need to look at this at highest priority.
This will gate R1.10 release.

Shaju will help repro if needed.

Ashish

On Aug 21, 2014, at 12:49 AM, Ashish Ranjan <email address hidden> wrote:

BTW this is new kernel.. We have not done sanity on this
3.11.0.22 kernel.. So we need to look into this.

Ashish

On Aug 21, 2014, at 12:39 AM, Hari Prasad Killi <email address hidden> wrote:

Ssh asks for passwd and gets struck. Dmesg shows attached trace – quite a few entries like this are present. Kernel thread locked ?

Regards,
Hari

From: Ashish Ranjan <email address hidden>
Date: Thursday, August 21, 2014 12:36 PM
To: Hari Prasad Killi <email address hidden>
Cc: "Shaju V.K" <email address hidden>
Subject: Re: Ubuntu havana-2304 build- lost ssh connectivity from cfgm to vrouter

See if we can try to debug this.
Its reproducible.

On Aug 20, 2014, at 6:54 PM, Shaju V.K <email address hidden> wrote:

Hi Team,

I am seeing ssh connectivity issue in multinode setup with Ubuntu havan build 2304. Sanity ran 47 test cases and 43 test cases are passed 1 failed and 3 skipped. While executing the 48th test case it lost the connectivity.

Not able to ssh a5s9 compute node from build machine or cfgm. Ping is working. Can someone take a look at below nodes. Tested file and logs are in node a5s6.