There are mysql errors in the c-bak logs when the volume goes into 'error_restoring' status: http://logs.openstack.org/82/202782/15/gate/gate-tempest-dsvm-neutron-full/372a95f/logs/screen-c-bak.txt.gz?level=TRACE 2015-08-10 21:36:37.420 ERROR oslo_messaging.rpc.dispatcher [req-1abbfe4f-97f2-4184-b598-522a1e0e2c19 None] Exception during message handling: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last): 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher executor_callback)) 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher executor_callback) 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args) 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher return f(*args, **kwargs) 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/cinder/cinder/backup/manager.py", line 437, in restore_backup 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher backup.save() 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 119, in __exit__ 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb) 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/cinder/cinder/backup/manager.py", line 431, in restore_backup 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher backup_service) 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher return f(*args, **kwargs) 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/cinder/cinder/volume/drivers/lvm.py", line 500, in restore_backup 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher backup_service.restore(backup, volume['id'], volume_file) 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/cinder/cinder/backup/chunkeddriver.py", line 684, in restore 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher self.put_metadata(volume_id, volume_meta) 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/cinder/cinder/backup/driver.py", line 336, in put_metadata 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher self.backup_meta_api.put(volume_id, json_metadata) 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/cinder/cinder/backup/driver.py", line 315, in put 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher func(meta_container[type], volume_id, fields) 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/cinder/cinder/backup/driver.py", line 177, in _restore_vol_base_meta 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher self.db.volume_update(self.context, volume_id, metadata) 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/cinder/cinder/db/api.py", line 245, in volume_update 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher return IMPL.volume_update(context, volume_id, values) 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/cinder/cinder/db/sqlalchemy/api.py", line 182, in wrapper 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher return f(*args, **kwargs) 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/cinder/cinder/db/sqlalchemy/api.py", line 1696, in volume_update 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher volume_ref = _volume_get(context, volume_id, session=session) 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/cinder/cinder/db/sqlalchemy/api.py", line 182, in wrapper 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher return f(*args, **kwargs) 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher File "/opt/stack/new/cinder/cinder/db/sqlalchemy/api.py", line 1278, in _volume_get 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher filter_by(id=volume_id).\ 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2445, in first 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher ret = list(self[0:1]) 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2281, in __getitem__ 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher return list(res) 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2516, in __iter__ 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher return self._execute_and_instances(context) 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2529, in _execute_and_instances 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher close_with_result=True) 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2520, in _connection_from_session 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher **kw) 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 882, in connection 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher execution_options=execution_options) 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 887, in _connection_for_bind 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher engine, execution_options) 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 334, in _connection_for_bind 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher conn = bind.contextual_connect() 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2039, in contextual_connect 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher self._wrap_pool_connect(self.pool.connect, None), 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2078, in _wrap_pool_connect 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher e, dialect, self) 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1401, in _handle_dbapi_exception_noconnection 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher util.raise_from_cause(newraise, exc_info) 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher reraise(type(exception), exception, tb=exc_tb) 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 2074, in _wrap_pool_connect 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher return fn() 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 376, in connect 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher return _ConnectionFairy._checkout(self) 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 713, in _checkout 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher fairy = _ConnectionRecord.checkout(pool) 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 480, in checkout 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher rec = pool._do_get() 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 1060, in _do_get 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher self._dec_overflow() 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__ 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher compat.reraise(exc_type, exc_value, exc_tb) 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 1057, in _do_get 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher return self._create_connection() 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 323, in _create_connection 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher return _ConnectionRecord(self) 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 449, in __init__ 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher self.connection = self.__connect() 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/pool.py", line 607, in __connect 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher connection = self.__pool._invoke_creator(self) 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/strategies.py", line 97, in connect 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher return dialect.connect(*cargs, **cparams) 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 385, in connect 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher return self.dbapi.connect(*cargs, **cparams) 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/pymysql/__init__.py", line 88, in Connect 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher return Connection(*args, **kwargs) 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 644, in __init__ 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher self._connect() 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 838, in _connect 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher self._request_authentication() 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 1021, in _request_authentication 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher auth_packet = self._read_packet() 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 882, in _read_packet 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher packet_header = self._read_bytes(4) 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 909, in _read_bytes 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher 2013, "Lost connection to MySQL server during query") 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') 2015-08-10 21:36:37.420 9136 ERROR oslo_messaging.rpc.dispatcher So that's probably the root issue here.