Create a volume in openstack and its state is 'available'.
=============================
cinder list
+--------------------------------------+-----------+--------+------+-------------+----------+-------------+
| ID | Status | Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+-----------+--------+------+-------------+----------+-------------+
| 201f32cd-7268-46ea-b240-d7415afb5cf4 | available | wsvol2 | 1 | vnxfc1 | false | |
| 8d7259e7-9b6b-4caf-b8c9-e06d5499d349 | available | wsvol1 | 1 | vnxfc1 | false | |
+--------------------------------------+-----------+--------+------+-------------+----------+-------------
=============================
However the volume is not created on the array.
And below errors logged in cinder.
==============================
[10:00 AM] Wan, Sam
Feb 02 20:58:03 vnx-fc-074 cinder-volume[1486261]: ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) [943/2215]
Feb 02 20:58:03 vnx-fc-074 cinder-volume[1486261]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
Feb 02 20:58:03 vnx-fc-074 cinder-volume[1486261]: ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
Feb 02 20:58:03 vnx-fc-074 cinder-volume[1486261]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch Feb 02 20:58:03 vnx-fc-074 cinder-volume[1486261]: ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
Feb 02 20:58:03 vnx-fc-074 cinder-volume[1486261]: ERROR oslo_messaging.rpc.server File "<decorator-gen-751>", line 2, in create_volume Feb 02 20:58:03 vnx-fc-074 cinder-volume[1486261]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/cinder/cinder/objects/cleanable.py", line 208, in wrapper
Feb 02 20:58:03 vnx-fc-074 cinder-volume[1486261]: ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) Feb 02 20:58:03 vnx-fc-074 cinder-volume[1486261]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/cinder/cinder/volume/manager.py", line 747, in create_volume
Feb 02 20:58:03 vnx-fc-074 cinder-volume[1486261]: ERROR oslo_messaging.rpc.server _run_flow()
Feb 02 20:58:03 vnx-fc-074 cinder-volume[1486261]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/cinder/cinder/volume/manager.py", line 739, in _run_flow
Feb 02 20:58:03 vnx-fc-074 cinder-volume[1486261]: ERROR oslo_messaging.rpc.server flow_engine.run()
Feb 02 20:58:03 vnx-fc-074 cinder-volume[1486261]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run
Feb 02 20:58:03 vnx-fc-074 cinder-volume[1486261]: ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout):
Feb 02 20:58:03 vnx-fc-074 cinder-volume[1486261]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter
Feb 02 20:58:03 vnx-fc-074 cinder-volume[1486261]: ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures)
Feb 02 20:58:03 vnx-fc-074 cinder-volume[1486261]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/taskflow/types/failure.py", line 339, in reraise_if_any
Feb 02 20:58:03 vnx-fc-074 cinder-volume[1486261]: ERROR oslo_messaging.rpc.server failures[0].reraise()
Feb 02 20:58:03 vnx-fc-074 cinder-volume[1486261]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/taskflow/types/failure.py", line 346, in reraise
Feb 02 20:58:03 vnx-fc-074 cinder-volume[1486261]: ERROR oslo_messaging.rpc.server six.reraise(*self._exc_info)
Feb 02 20:58:03 vnx-fc-074 cinder-volume[1486261]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/six.py", line 703, in reraise
Feb 02 20:58:03 vnx-fc-074 cinder-volume[1486261]: ERROR oslo_messaging.rpc.server raise value
Feb 02 20:58:03 vnx-fc-074 cinder-volume[1486261]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task
Feb 02 20:58:03 vnx-fc-074 cinder-volume[1486261]: ERROR oslo_messaging.rpc.server result = task.execute(**arguments)
Feb 02 20:58:03 vnx-fc-074 cinder-volume[1486261]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/cinder/cinder/volume/flows/manager/create_volume.py", line 1122, in execute
Feb 02 20:58:03 vnx-fc-074 cinder-volume[1486261]: ERROR oslo_messaging.rpc.server model_update = self._create_raw_volume(
Feb 02 20:58:03 vnx-fc-074 cinder-volume[1486261]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/cinder/cinder/volume/flows/manager/create_volume.py", line 1084, in _create_raw_volume
Feb 02 20:58:03 vnx-fc-074 cinder-volume[1486261]: ERROR oslo_messaging.rpc.server self.message.create(
Feb 02 20:58:03 vnx-fc-074 cinder-volume[1486261]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
Feb 02 20:58:03 vnx-fc-074 cinder-volume[1486261]: ERROR oslo_messaging.rpc.server self.force_reraise()
Feb 02 20:58:03 vnx-fc-074 cinder-volume[1486261]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
Feb 02 20:58:03 vnx-fc-074 cinder-volume[1486261]: ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
Feb 02 20:58:03 vnx-fc-074 cinder-volume[1486261]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/six.py", line 703, in reraise
Feb 02 20:58:03 vnx-fc-074 cinder-volume[1486261]: ERROR oslo_messaging.rpc.server raise value
Feb 02 20:58:03 vnx-fc-074 cinder-volume[1486261]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/cinder/cinder/volume/flows/manager/create_volume.py", line 1081, in _create_raw_volume
Feb 02 20:58:03 vnx-fc-074 cinder-volume[1486261]: ERROR oslo_messaging.rpc.server ret = self.driver.create_volume(volume)
Feb 02 20:58:03 vnx-fc-074 cinder-volume[1486261]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/cinder/cinder/volume/drivers/dell_emc/vnx/driver.py", line 125, in create_volume
Feb 02 20:58:03 vnx-fc-074 cinder-volume[1486261]: ERROR oslo_messaging.rpc.server return self.adapter.create_volume(volume)
Feb 02 20:58:03 vnx-fc-074 cinder-volume[1486261]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/cinder/cinder/volume/drivers/dell_emc/vnx/adapter.py", line 256, in create_volume
Feb 02 20:58:03 vnx-fc-074 cinder-volume[1486261]: ERROR oslo_messaging.rpc.server lun = self.client.create_lun(
Feb 02 20:58:03 vnx-fc-074 cinder-volume[1486261]: ERROR oslo_messaging.rpc.server File "/opt/stack/new/cinder/cinder/volume/drivers/dell_emc/vnx/client.py", line 106, in create_lun
Feb 02 20:58:03 vnx-fc-074 cinder-volume[1486261]: ERROR oslo_messaging.rpc.server lun = pool.create_lun(lun_name=name,
Feb 02 20:58:03 vnx-fc-074 cinder-volume[1486261]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/storops/vnx/resource/block_pool.py", line 156, in create_lun
Feb 02 20:58:03 vnx-fc-074 cinder-volume[1486261]: ERROR oslo_messaging.rpc.server ex.raise_if_err(ret, 'error creating lun.',
Feb 02 20:58:03 vnx-fc-074 cinder-volume[1486261]: ERROR oslo_messaging.rpc.server File "/usr/local/lib/python3.8/dist-packages/storops/exception.py", line 227, in raise_if_err
Feb 02 20:58:03 vnx-fc-074 cinder-volume[1486261]: ERROR oslo_messaging.rpc.server raise ex_clz(msg)
Feb 02 20:58:03 vnx-fc-074 cinder-volume[1486261]: ERROR oslo_messaging.rpc.server storops.exception.VNXCreateLunError: error creating lun. detail:
Feb 02 20:58:03 vnx-fc-074 cinder-volume[1486261]: ERROR oslo_messaging.rpc.server Cannot use the security file. Check the -secfilepath option.
Feb 02 20:58:03 vnx-fc-074 cinder-volume[1486261]: ERROR oslo_messaging.rpc.server Cannot use the security file. Check the -secfilepath option.
Feb 02 20:58:03 vnx-fc-074 cinder-volume[1486261]: ERROR oslo_messaging.rpc.server
================================================
Two issues here.
Issue#1:
Security file path issue, every time VNX driver executes naviseccli command, "Cannot use the security file. Check the -secfilepath option." error will occur.
Re-configure security file path could fix it.
Issue#2:
Issue#1 will cause the original issue you reported: volume status is available, but LUN doesn't exist in VNX.
This is the root cause:
1) When cinder created volume, VNX driver executed "lun -create" command to create LUN, LUN created successfully but security file error was returned (Issue#1).
2) Cinder reverted the volume creation task, VNX driver executed "lun -destroy" command to delete the LUN, but the destroy command took about 4 seconds.
3) Meanwhile, cinder retried to create the volume, a new volume creation task was triggered, but the LUN was being destroyed, VNX returned LUN name in use error, VNX driver used the existed LUN as the LUN of the new volume, so volume created successfully this time. The volume status became available.
4) But the LUN finally got destroyed, now as you reported, the volume status is available, but the LUN didn't exist on VNX.
For issue#1, re-configure the security file path could fix it.
For issue#2, VNX driver doesn't handle the scenario of creating the existed but being destroyed LUN. We could make a patch to enhance it, but it is risky, it may introduce regression issue, so I think if the CI environment is recovered, we will not make this change.