[Dell PowerFlex] [scaleio]: password appears in plain text when creating a volume from an image

Hi Jean,
Is this happening also on generic-nfs or this is only happening with Dell PowerFlex?

Rajat suggested that this could also be a Nova bug, because also does some formatting while attaching the volume,  so I'm adding the project to the bug.

Cheers,

This is powerflex issue as this the password used by our client driver to access the volume from the powerflex backend.

As this is PowerFlex, this is block-only.

It seems that the password is displayed every time the driver tries to attach or detach a volume.

The following logs are display when executing an image to volume operation:
openstack volume create --size 32 --type powerflex --image cirros-0.5.2-x86_64-disk volpflex

When attaching!
Jan 25 10:32:26 zed-devstask cinder-volume[45668]: INFO cinder.volume.drivers.dell_emc.powerflex.driver [None req-bc7b1480-5cae-4fa3-b451-c2a36a79e285 admin None] Call os-brick to attach PowerFlex volume.
Jan 25 10:32:26 zed-devstask cinder-volume[45668]: DEBUG cinder.volume.drivers.dell_emc.powerflex.utils [None req-bc7b1480-5cae-4fa3-b451-c2a36a79e285 admin None] Converted id 2d59df0e-2ebc-4868-ab74-c9aa91a66fdf to PowerFlex OS name LVnfDi68SGirdMmqkaZv3w==. {{(pid=45668) id_to_base64 /opt/stack/cinder/cinder/volume/drivers/dell_emc/powerflex/utils.py:45}}
Jan 25 10:32:26 zed-devstask cinder-volume[45668]: DEBUG os_brick.initiator.connectors.scaleio [None req-bc7b1480-5cae-4fa3-b451-c2a36a79e285 admin None] ==> connect_volume: call "{'self': <os_brick.initiator.connectors.scaleio.ScaleIOConnector object at 0x7fc4e784b160>, 'connection_properties': {'scaleIO_volname': 'LVnfDi68SGirdMmqkaZv3w==', 'hostIP': None, 'serverIP': '10.225.108.171', 'serverPort': 443, 'serverUsername': 'admin', 'iopsLimit': None, 'bandwidthLimit': None, 'scaleIO_volume_id': '5aafd80d00000004', 'config_group': 'powerflex', 'failed_over': False, 'verify_certificate': False, 'certificate_path': None}}" {{(pid=45668) trace_logging_wrapper /usr/local/lib/python3.8/dist-packages/os_brick/utils.py:174}}
Jan 25 10:32:26 zed-devstask cinder-volume[45668]: DEBUG os_brick.initiator.connectors.base [None req-bc7b1480-5cae-4fa3-b451-c2a36a79e285 admin None] Acquiring lock "scaleio" by "os_brick.initiator.connectors.scaleio.ScaleIOConnector.connect_volume" {{(pid=45668) inner /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/base.py:68}}
Jan 25 10:32:26 zed-devstask cinder-volume[45668]: DEBUG os_brick.initiator.connectors.base [None req-bc7b1480-5cae-4fa3-b451-c2a36a79e285 admin None] Lock "scaleio" acquired by "os_brick.initiator.connectors.scaleio.ScaleIOConnector.connect_volume" :: waited 0.001s {{(pid=45668) inner /usr/local/lib/python3.8/dist-packages/os_brick/initiator/connectors/base.py:73}}
Jan 25 10:32:26 zed-devstask cinder-volume[45668]: INFO os_brick.initiator.connectors.scaleio [None req-bc7b1480-5cae-4fa3-b451-c2a36a79e285 admin None] Get ScaleIO connector password from configuration file
Jan 25 10:32:26 zed-devstask cinder-volume[45668]: DEBUG oslo.privsep.daemon [-] privsep: reply[7f6c8418-69aa-46d4-8595-4f9e02109658]: (4, 'Password123!') {{(pid=46568) _call_back /usr/local/lib/python3.8/dist-packages/oslo_privsep/daemon.py:501}}

When detaching:

Jan 25 10:32:39 zed-devstask cinder-volume[45668]: INFO cinder.volume.drivers.dell_emc.powerflex.driver [None req-bc7b1480-5cae-4fa3-b451-c2a36a79e285 admin None] Call os-brick to detach PowerFlex volume.
Jan 25 10:32:39 zed-devstask cinder-volume[45668]: DEBUG cinder.volume.drivers.dell_emc.powerflex.utils [None req-bc7b1480-5cae-4fa3-b451-c2a36a79e285 admin None] Converted id 2d59df0e-2ebc-4868-ab74-c9aa91a66fdf to Power...

This doesn't happen when attaching or detaching a volume with nova.

This has something to do between os_brick, privsep and cinder. I need to investigate on why this password appears in clear.

The issue happens only when creating a volume from an image at the time it attaches or detaches the volume.

Seems there is something wrong between cinder and privsep called by os_brick.

Cinder calls os-brick:

https://github.com/openstack/cinder/blob/master/cinder/volume/drivers/dell_emc/powerflex/driver.py#L1199

Then os-brick initiator makes a privilege call since it needs to access a restricted file

https://github.com/openstack/os-brick/blob/master/os_brick/initiator/connectors/scaleio.py#L90

It communicates with oslo_privsep to fetch the required information

https://github.com/openstack/os-brick/blob/master/os_brick/privileged/scaleio.py#L78

Then this is where the password is displayed in clear in the logs

https://github.com/openstack/oslo.privsep/blob/master/oslo_privsep/daemon.py#L501

The issue happens only when creating a volume from an image at the time it attaches or detaches the volume.
Seems there is something wrong between cinder and privsep called by os_brick.

Cinder calls os-brick:
https://github.com/openstack/cinder/blob/master/cinder/volume/drivers/dell_emc/powerflex/driver.py#L1199

Then os-brick initiator makes a privilege call since it needs to access a restricted file

https://github.com/openstack/os-brick/blob/master/os_brick/initiator/connectors/scaleio.py#L90

It communicates with oslo_privsep to fetch the required information

https://github.com/openstack/os-brick/blob/master/os_brick/privileged/scaleio.py#L78

Then this is where the password is displayed in clear in the logs

https://github.com/openstack/oslo.privsep/blob/master/oslo_privsep/daemon.py#L501

I think the fix for this is to introduce a new privsep entrypoint into os_brick/privileged/__init__.py and pass in a different logger_name which is configured to only log at ERROR level, which should avoid logging the output.

Do you have any link to an example of such an implementation?

Flagged this as a security hardening opportunity, class B3 in our taxonomy since it seems to only affect debug-level logging. https://security.openstack.org/vmt-process.html#report-taxonomy