Test http://jenkins-product.srt.mirantis.net:8080/view/5.0_swarm/job/5.0_fuelmain.system_test.centos.thread_3/29/testReport/junit/%28root%29/ceph_ha/ceph_ha/ failed with unclear waiting timeout error
....
....
....
File "/home/jenkins/venv-nailgun-tests/local/lib/python2.7/site-packages/devops/helpers/helpers.py", line 95, in wait
raise TimeoutError("Waiting timed out")
TimeoutError: Waiting timed out
Several things should be done:
1. Add error message to the ceph_health check in case of failure(we can use sdterr to know what happens)
2. In this case ceph health was warn according to mon-0 has feature time, after sync time - on all nodes ceph health is fine -so we need to add dsome workaround here like:
try:
wait(....)
except TimeoutError:
if 'clocks not synchronized ' in output from ceph -w command:
run sync time command(method) and try again to verify health
Below we can see why ceph health is warn
[root@node-4 ~]# ceph -w
cluster 7cec7fda-d7ca-4836-b954-55373a0e95f3
health HEALTH_WARN clock skew detected on mon.node-2, mon.node-4
monmap e3: 3 mons at {node-1=10.108.17.3:6789/0,node-2=10.108.17.4:6789/0,node-4=10.108.17.6:6789/0}, election epoch 16, quorum 0,1,2 node-1,node-2,node-4
osdmap e65: 12 osds: 12 up, 12 in
pgmap v104: 3264 pgs: 3264 active+clean; 14464 KB data, 25085 MB used, 568 GB / 592 GB avail
mdsmap e1: 0/0/1 up
2014-07-01 03:15:57.938597 mon.1 [WRN] message from mon.0 was stamped 0.231240s in the future, clocks not synchronized
^C[root@node-4 ~]# exit
logout
Connection to node-4 closed.
[root@nailgun ~]# ssh node-1
Warning: Permanently added 'node-1' (RSA) to the list of known hosts.
Last login: Tue Jul 1 03:18:29 2014 from 10.108.15.2
[root@node-1 ~]# sudo hwclock --show
Tue 01 Jul 2014 11:19:28 AM UTC -0.624457 seconds
[root@node-1 ~]# date
Tue Jul 1 03:23:01 UTC 2014
Fix proposed to branch: master
Review: https:/