Cinder

HPE Lefthand driver fails to create volume backup concurrently

Bug #1662148 reported by Vijay Ladani
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
New
Undecided
Unassigned

Bug Description

While running rally test "create_and_restore_volume_backup" concurrently, test failed with below error

2017-01-11 00:36:26.176 _[00;32mDEBUG cinder.volume.drive...2017-01-11 00:36:26.176 DEBUG cinder.volume.drivers.hpe.hpe_lefthand_iscsi [req-7392f27b-ad00-4272-9940-f2b693e039d6 3c6f12c2c200420ab203999a7cf4b9a5 c606c14351a04fccb8210f2a9b8743ba] <== initialize_connection: exception (209ms) VolumeBackendAPIException(u'Error (HTTP 500) SERVER_ALREADY_EXISTS - The server with the name "csimbe13-b14" already exists. Use a unique name and try again.',) from (pid=5717) trace_logging_wrapper /opt/stack/cinder/cinder/utils.py:901
2017-01-11 00:36:26.176 ERROR cinder.volume.manager [req-7392f27b-ad00-4272-9940-f2b693e039d6 3c6f12c2c200420ab203999a7cf4b9a5 c606c14351a04fccb8210f2a9b8743ba] Driver initialize connection failed (error: Error (HTTP 500) SERVER_ALREADY_EXISTS - The server with the name "csimbe13-b14" already exists. Use a unique name and try again.).
2017-01-11 00:36:26.176 TRACE cinder.volume.manager Traceback (most recent call last):
2017-01-11 00:36:26.176 TRACE cinder.volume.manager  File "/opt/stack/cinder/cinder/volume/manager.py", line 1429, in initialize_connection
2017-01-11 00:36:26.176 TRACE cinder.volume.manager  conn_info = self.driver.initialize_connection(volume, connector)
2017-01-11 00:36:26.176 TRACE cinder.volume.manager  File "/opt/stack/cinder/cinder/utils.py", line 895, in trace_logging_wrapper
2017-01-11 00:36:26.176 TRACE cinder.volume.manager  result = f(*args, **kwargs)
2017-01-11 00:36:26.176 TRACE cinder.volume.manager  File "/opt/stack/cinder/cinder/volume/drivers/hpe/hpe_lefthand_iscsi.py", line 746, in initialize_connection
2017-01-11 00:36:26.176 TRACE cinder.volume.manager  raise exception.VolumeBackendAPIException(ex)
2017-01-11 00:36:26.176 TRACE cinder.volume.manager VolumeBackendAPIException: Error (HTTP 500) SERVER_ALREADY_EXISTS - The server with the name "csimbe13-b14" already exists. Use a unique name and try again.
2017-01-11 00:36:26.176 TRACE cinder.volume.manager 
2017-01-11 00:36:26.178 ERROR oslo_messaging.rpc.server [req-7392f27b-ad00-4272-9940-f2b693e039d6 3c6f12c2c200420ab203999a7cf4b9a5 c606c14351a04fccb8210f2a9b8743ba] Exception during message handling
2017-01-11 00:36:26.178 TRACE oslo_messaging.rpc.server Traceback (most recent call last):
2017-01-11 00:36:26.178 TRACE oslo_messaging.rpc.server  File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 133, in _process_incoming
2017-01-11 00:36:26.178 TRACE oslo_messaging.rpc.server  res = self.dispatcher.dispatch(message)
2017-01-11 00:36:26.178 TRACE oslo_messaging.rpc.server  File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 150, in dispatch
2017-01-11 00:36:26.178 TRACE oslo_messaging.rpc.server  return self._do_dispatch(endpoint, method, ctxt, args)
2017-01-11 00:36:26.178 TRACE oslo_messaging.rpc.server  File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 121, in _do_dispatch
2017-01-11 00:36:26.178 TRACE oslo_messaging.rpc.server  result = func(ctxt, **new_args)
2017-01-11 00:36:26.178 TRACE oslo_messaging.rpc.server  File "/opt/stack/cinder/cinder/volume/manager.py", line 4400, in initialize_connection
2017-01-11 00:36:26.178 TRACE oslo_messaging.rpc.server  connector, volume=volume)
2017-01-11 00:36:26.178 TRACE oslo_messaging.rpc.server  File "/opt/stack/cinder/cinder/volume/manager.py", line 1437, in initialize_connection
2017-01-11 00:36:26.178 TRACE oslo_messaging.rpc.server  raise exception.VolumeBackendAPIException(data=err_msg)
2017-01-11 00:36:26.178 TRACE oslo_messaging.rpc.server VolumeBackendAPIException: Bad or unexpected response from the storage volume backend API: Driver initialize connection failed (error: Error (HTTP 500) SERVER_ALREADY_EXISTS - The server with the name "csimbe13-b14" already exists. Use a unique name and try again.).
2017-01-11 00:36:26.178 TRACE oslo_messaging.rpc.server 

Revision history for this message
Vijay Ladani (vijay-ladani) wrote :
Revision history for this message
Vijay Ladani (vijay-ladani) wrote :

As per the previous bug https://bugs.launchpad.net/cinder/+bug/1658888 filed against HPE 3PAR driver. Cause of failure is, multiple threads are trying to enter in initialize_connection OR terminate_connection method at the same time which leads to an above error.

Fix is to apply lock on those methods.

Vijay Ladani (vijay-ladani)
Changed in cinder:
assignee: nobody → Vijay Ladani (vijay-ladani)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to cinder (master)

Fix proposed to branch: master
Review: https://review.openstack.org/430060

Changed in cinder:
status: New → In Progress
Revision history for this message
Sean McGinnis (sean-mcginnis) wrote : Bug Assignee Expired

Unassigning due to no activity for > 6 months.

Changed in cinder:
assignee: Vijay Ladani (vijay-ladani) → nobody
Sean McGinnis (sean-mcginnis)
Changed in cinder:
status: In Progress → New
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on cinder (master)

Change abandoned by Sean McGinnis (<email address hidden>) on branch: master
Review: https://review.openstack.org/430060
Reason: No response for several months. Feel free to restore and address comments if you wish to pursue.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.