Platform system tests failed with timeout

Bug #1442206 reported by Timur Nurlygayanov
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Mirantis OpenStack
Fix Released
Critical
Timur Nurlygayanov
6.0.x
Fix Released
Critical
Timur Nurlygayanov

Bug Description

We can see that system tests for platform components failed with the timeout error, example:
http://jenkins-product.srt.mirantis.net:8080/view/6.1_swarm/job/6.1.system_test.ubuntu.services_ha/87/console

we can see that the timeout is 600 minutes for all system tests on Ubuntu HA configurations (for different components) and looks like we need to increase this timeout to 1000 minutes.

Changed in mos:
assignee: nobody → Timur Nurlygayanov (tnurlygayanov)
importance: Undecided → Critical
milestone: none → 6.1
status: New → Confirmed
status: Confirmed → Fix Committed
status: Fix Committed → Fix Released
status: Fix Released → Confirmed
Revision history for this message
Timur Nurlygayanov (tnurlygayanov) wrote :

I have checked how we use te time during the tests and have found that most of the time were lost on:

1 hour 20 minutes - deployment of OpenStack cluster with Sahara
1 hour - execution of OSTF tests for Sahara
1 hour 9 minutes - deployment of OpenStack cluster with Murano
1 hour 12 minutes - deployment of OpenStack cluster with Ceilometer #1
1 hour 10 minutes - deployment of OpenStack cluster with Ceilometer #2

So, as we can see we have added new system tests and these tests requires more time for execution, it means that we need just increase the timeout for the jenkins job with the system tests for this type of tests.

We also need to improve the configuration of compute node on the test environment because execution of Sahara OSTF tests takes 11 hour and it is too long.

Changed in mos:
assignee: Timur Nurlygayanov (tnurlygayanov) → Fuel DevOps (fuel-devops)
Revision history for this message
Timur Nurlygayanov (tnurlygayanov) wrote :

And we also can see that we have VMs with 1 vCPU for OpenStack nodes:

tnurlygayanov@srv36-bud:~$ virsh dumpxml 6.1.system_test.ubuntu.services_ha.88.2015-04-10_02-00-45_slave-01 | grep cpu
  <vcpu placement='static'>1</vcpu>
  <cpu mode='host-model'>
  </cpu>

We need to increase these parameters to at least 2 vCPU for each node.

Revision history for this message
Timur Nurlygayanov (tnurlygayanov) wrote :

We need to set SLAVE_NODE_CPU=2 for all services jobs and set timeout to 1200 minutes.

Revision history for this message
Timur Nurlygayanov (tnurlygayanov) wrote :
Changed in mos:
status: Confirmed → Fix Committed
status: Fix Committed → In Progress
Changed in mos:
assignee: Fuel DevOps (fuel-devops) → Aleksandra Fedorova (afedorova)
Changed in mos:
assignee: Aleksandra Fedorova (afedorova) → Timur Nurlygayanov (tnurlygayanov)
Revision history for this message
Timur Nurlygayanov (tnurlygayanov) wrote :

Fixed, verified and looks like it works :)

Changed in mos:
status: In Progress → Fix Released
Revision history for this message
Dennis Dmitriev (ddmitriev) wrote :
Changed in mos:
status: Fix Released → Confirmed
Revision history for this message
Dennis Dmitriev (ddmitriev) wrote :
Revision history for this message
Timur Nurlygayanov (tnurlygayanov) wrote :

Hi Denis,

the issue was successfully solved and the issues which aoy are mentioned - it is another known issue: https://bugs.launchpad.net/mos/+bug/1443360 - it is in progress now.

Changed in mos:
status: Confirmed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.