[system tests] Need to increase wait for galera is UP in ceph destructive tests

Bug #1463054 reported by Andrey Sledzinskiy
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Released
Medium
Tatyanka
6.1.x
Won't Fix
Medium
MOS QA Team
7.0.x
Fix Released
Medium
Tatyanka

Bug Description

In ha_sequential_rabbit_master_failover test in line https://github.com/stackforge/fuel-qa/blob/master/fuelweb_test/tests/tests_strength/test_failover_base.py#L723, 748, 765 we are waiting 300 seconds that is not enough, we should increase it to 600 seconds

Error Message

Failed 5 OSTF tests; should fail 2 tests. Names of failed tests: [{u'Check data replication over mysql (failure)': u'Can not connect to mysql. Please check that mysql is running and there is connectivity by management network Please refer to OpenStack logs for more details.'}, {u'Check amount of tables in databases is the same on each node (failure)': u'Can list tables Please refer to OpenStack logs for more details.'}, {u'Check galera environment state (failure)': u'Verification of galera cluster node status failed Please refer to OpenStack logs for more details.'}, {u'RabbitMQ availability (failure)': u'Number of controllers is not equal to number of cluster nodes.'}, {u'RabbitMQ replication (failure)': u'Failed to connect to 5673 port on host 10.109.0.5 Please refer to OpenStack logs for more details.'}]

Stacktrace

Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/case.py", line 331, in run
    testMethod()
  File "/usr/lib/python2.7/unittest/case.py", line 1043, in runTest
    self._testFunc()
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/home/jenkins/workspace/6.1.system_test.ubuntu.ha_destructive_ceph_neutron/fuelweb_test/helpers/decorators.py", line 70, in wrapper
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/6.1.system_test.ubuntu.ha_destructive_ceph_neutron/fuelweb_test/tests/tests_strength/test_failover_with_ceph.py", line 71, in ha_ceph_neutron_rabbit_master_destroy
    super(self.__class__, self).ha_sequential_rabbit_master_failover()
  File "/home/jenkins/workspace/6.1.system_test.ubuntu.ha_destructive_ceph_neutron/fuelweb_test/tests/tests_strength/test_failover_base.py", line 726, in ha_sequential_rabbit_master_failover
    test_sets=['ha'], should_fail=2)
  File "/home/jenkins/workspace/6.1.system_test.ubuntu.ha_destructive_ceph_neutron/fuelweb_test/__init__.py", line 48, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/6.1.system_test.ubuntu.ha_destructive_ceph_neutron/fuelweb_test/models/fuel_web_client.py", line 981, in run_ostf
    failed_test_name=failed_test_name)
  File "/home/jenkins/workspace/6.1.system_test.ubuntu.ha_destructive_ceph_neutron/fuelweb_test/__init__.py", line 48, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/6.1.system_test.ubuntu.ha_destructive_ceph_neutron/fuelweb_test/models/fuel_web_client.py", line 232, in assert_ostf_run
    failed_tests_res))
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/asserts.py", line 163, in assert_true
    raise ASSERTION_ERROR(message)
AssertionError: Failed 5 OSTF tests; should fail 2 tests. Names of failed tests: [{u'Check data replication over mysql (failure)': u'Can not connect to mysql. Please check that mysql is running and there is connectivity by management network Please refer to OpenStack logs for more details.'}, {u'Check amount of tables in databases is the same on each node (failure)': u'Can list tables Please refer to OpenStack logs for more details.'}, {u'Check galera environment state (failure)': u'Verification of galera cluster node status failed Please refer to OpenStack logs for more details.'}, {u'RabbitMQ availability (failure)': u'Number of controllers is not equal to number of cluster nodes.'}, {u'RabbitMQ replication (failure)': u'Failed to connect to 5673 port on host 10.109.0.5 Please refer to OpenStack logs for more details.'}]

Changed in fuel:
status: New → Confirmed
tags: added: non-release
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-qa (master)

Fix proposed to branch: master
Review: https://review.openstack.org/203724

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-qa (master)

Reviewed: https://review.openstack.org/203724
Committed: https://git.openstack.org/cgit/stackforge/fuel-qa/commit/?id=345a5013cfb5df33e860921cbbf8d377588d5017
Submitter: Jenkins
Branch: master

commit 345a5013cfb5df33e860921cbbf8d377588d5017
Author: Tatyana Leontovich <email address hidden>
Date: Mon Jul 20 19:48:52 2015 +0300

    Increase timeout in ceph destructive tests

    In ha_sequential_rabbit_master_failover test in line
    test_failover_base.py lines: L723, L748, L765 we are
    waiting 300 seconds that is not enough, we should increase it to 600 seconds.

    Change-Id: I4714f58f242fb6fdf51c3c6fe39f53ed43473251
    Closes-Bug: #1463054

Changed in fuel:
status: In Progress → Fix Committed
tags: added: area-qa
removed: non-release
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.