CephRadosGW.ceph_rados_gw() does not wait for Ceph to become available for OS services

http://jenkins-product.srt.mirantis.net:8080/job/6.1.staging.ubuntu.bvt_2/108/consoleText

6.1 staging job failed with:

======================================================================
FAIL: Deploy ceph HA with RadosGW for objects
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 296, in testng_method_mistake_capture_func
    compatability.capture_type_error(s_func)
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/compatability/exceptions_2_6.py", line 27, in capture_type_error
    func()
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/proboscis/case.py", line 350, in func
    func(test_case.state.get_state())
  File "/home/jenkins/workspace/6.1.staging.ubuntu.bvt_2/fuelweb_test/helpers/decorators.py", line 65, in wrapper
    return func(*args, **kwargs)
  File "/home/jenkins/workspace/6.1.staging.ubuntu.bvt_2/fuelweb_test/tests/test_ceph.py", line 291, in ceph_rados_gw
    self.fuel_web.run_ostf(cluster_id=cluster_id)
  File "/home/jenkins/workspace/6.1.staging.ubuntu.bvt_2/fuelweb_test/__init__.py", line 48, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/6.1.staging.ubuntu.bvt_2/fuelweb_test/models/fuel_web_client.py", line 601, in run_ostf
    failed_test_name=failed_test_name)
  File "/home/jenkins/workspace/6.1.staging.ubuntu.bvt_2/fuelweb_test/__init__.py", line 48, in wrapped
    result = func(*args, **kwargs)
  File "/home/jenkins/workspace/6.1.staging.ubuntu.bvt_2/fuelweb_test/models/fuel_web_client.py", line 197, in assert_ostf_run
    failed_tests_res))
AssertionError: Failed tests, fails: 1 should fail: 0 failed tests name: [{u'Create volume and boot instance from it (failure)': u'Failed to get to expected status. In error state. Please refer to OpenStack logs for more details.'}]

cinder-scheduler contains the following error:

2015-02-27 09:50:44.101 9193 ERROR cinder.scheduler.flows.create_volume [req-e0d56160-3dc9-4149-946c-825079893a32 7784bc5b6a284bdbb9f79c912c0b0f83 75e3b6fdc7504d75aa1eabe3abc5656e - - -] Failed to run task cinder.scheduler.flows.create_volume.ScheduleCreateVolumeTask;volume:create: No valid host was found. No weighed hosts available

The problem is that last report from cinder-volume was:

2015-02-27 09:50:17.526 9193 DEBUG cinder.scheduler.host_manager [req-0817c820-8669-4834-8060-336ba8d5f0ec - - - - -] Received volume service update from rbd:volumes: {u'volume_backend_name': u'DEFAULT', u'free_
capacity_gb': 0, u'driver_version': u'1.1.0', u'total_capacity_gb': 0, u'reserved_percentage': 0, u'vendor_name': u'Open Source', u'storage_protocol': u'ceph'} update_service_capabilities /usr/lib/python2.7/dist
-packages/cinder/scheduler/host_manager.py:434

which means last time cinder-volume checked Ceph status (by default, every 60 seconds), there was no space available.

Looks like we need to add another wait to ceph_rados_gw(), which would wait before 'ceph df' reports available space for our pools (images, volumes) and give OS service another 60 seconds to update the information in the DB.