If test 'ha_corosync_stability_check' fails on step '3. Start corosync on the first controller' it raises the following exception:
Traceback (most recent call last):
File "/usr/lib/python2.7/unittest/case.py", line 331, in run
testMethod()
File "/usr/lib/python2.7/unittest/case.py", line 1043, in runTest
self._testFunc()
File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
compatability.capture_type_error(s_func)
File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
func()
File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
func(test_case.state.get_state())
File "/home/jenkins/workspace/8.0.system_test.ubuntu.ha_neutron_destructive/fuelweb_test/helpers/decorators.py", line 81, in wrapper
result = func(*args, **kwargs)
File "/home/jenkins/workspace/8.0.system_test.ubuntu.ha_neutron_destructive/fuelweb_test/tests/tests_strength/test_failover.py", line 330, in ha_corosync_stability_check
super(self.__class__, self).ha_corosync_stability_check()
File "/home/jenkins/workspace/8.0.system_test.ubuntu.ha_neutron_destructive/fuelweb_test/tests/tests_strength/test_failover_base.py", line 1245, in ha_corosync_stability_check
'log, count: {0}'.format(count)), timeout=20)
File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/devops/helpers/helpers.py", line 108, in _wait
return raising_predicate()
File "/home/jenkins/workspace/8.0.system_test.ubuntu.ha_neutron_destructive/fuelweb_test/tests/tests_strength/test_failover_base.py", line 1245, in <lambda>
'log, count: {0}'.format(count)), timeout=20)
File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/asserts.py", line 163, in assert_true
raise ASSERTION_ERROR(message)
AssertionError: Corosync was not started on controller, see debug log, count: 3
The message says there is something helpful for investigation in debug logs, but actually nothing is logged about corosync start:
2015-12-08 02:18:32,289 - DEBUG test_failover_base.py:1189 -- Status of pacemaker nodes on node slave-02_controller: {'Standby': [], 'Offline': ['node-1.test.domain.local'], 'Online': ['node-2.test.domain.local', 'node-3.test.domain.local']}
2015-12-08 02:18:32,289 - INFO sftp.py:129 -- [chan 0] sftp session closed.
2015-12-08 02:18:32,292 - DEBUG __init__.py:54 -- Calling: generate_logs with args: ([<class 'fuelweb_test.models.nailgun_client.NailgunClient'>(0x7fc8646d4750), url:None],) {}
We need to improve tests and add logging of remote command stdour/stderr here:
https://github.com/openstack/fuel-qa/blob/81f1c0d739678e70c479b222601b27c2a40a62e0/fuelweb_test/tests/tests_strength/test_failover_base.py#L1235-L1239
Fix proposed to branch: master /review. openstack. org/260991
Review: https:/