Zaza tests need to implement retries when external problems occur (test env flakiness)

Have you looked through the logs about why the newly created volume went into an Error state?

@chris yes, although I did not find it relevant because it worked on previous gate tests and the same one when run manually.

cinder-volume.log shows:
"""
2020-07-13 11:39:17.610 18925 ERROR cinder.cmd.volume [req-ecfbb173-3674-46e2-ba8d-b3f89555c23e - - - - -] Volume service juju-3c71ff-zaza-8c0ecc927ef0-0@lvm failed to start.: DBNonExistentTable: (sqlite3.OperationalError) no such table: services [SQL: u'SELECT services.created_at AS services_created_at, services.de
leted_at AS services_deleted_at, services.deleted AS services_deleted, services.id AS services_id, services.cluster_name AS services_cluster_name, services.host AS services_host, services.binary AS services_binary, services.updated_at AS services_updated_at, services.topic AS services_topic, services.report_count AS
 services_report_count, services.disabled AS services_disabled, services.availability_zone AS services_availability_zone, services.disabled_reason AS services_disabled_reason, services.modified_at AS services_modified_at, services.rpc_current_version AS services_rpc_current_version, services.object_current_version A
S services_object_current_version, services.replication_status AS services_replication_status, services.active_backend_id AS services_active_backend_id, services.frozen AS services_frozen \nFROM services \nWHERE services.deleted = 0 AND services.binary = ?'] [parameters: ('cinder-scheduler',)]
"""

And the same log also shows:
"""
2020-07-13 11:42:40.214 24040 ERROR oslo.messaging._drivers.impl_rabbit [req-f26ade91-573f-4847-b041-6e81199e6d95 - - - - -] [1abe28df-dca5-406e-b261-16179df0d178] AMQP server on 127.0.0.1:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 1 seconds. Client port: None: error: [Errno 111] ECONNREFUSED
"""

This happened after a successful test to pause and resume cinder services.

Given that it sounds like the service isn't actually available, I'd suggest that the pause/resume test is incorrectly passing

I tried to run charm-recheck-full and it now failed on xenial-queens with error:
"""
OSError: [Errno 113] Connect call failed ('252.10.0.1', 17070)
"""

That error happened when async_block_until_wl_status_info_starts_with tried to connect to the model. Everytime, it's a different error on a different point.

WRT to pause/resume not working fine, I manually run xenial-pike (commented all gate tests except the xenial-pike one) and ran "tox -e func", which passed.