Unable to delete encrypted volume (barbican)

Latest devstack queens.

Since the volume created as encrypted , we expect that admin tenant will be able to delete the volume.

Steps to reproduce:
# create a volume type (admin tenant)
# create encryption type - > (admin tenant)
encryption_prov = nova.volume.encryptors.luks.LuksEncryptor
key_size = 256
cipher = aes-xts-plain64
control_location = front-end

# add non-admin tenant to the role creator

# create a volume (non-admin) tenant.

Here are some shows:
Tenants:
89517260b9504830a0d3154cf061f36d | tempest-TestEncryptedCinderVolumes-57557031

cinder backend:
[root@localhost devstack]# cinder get-pools --detail
+-----------------------------+------------------------------------------------------------------------------+
| Property | Value |
+-----------------------------+------------------------------------------------------------------------------+
| QoS_support | False |
| allocated_capacity_gb | 1 |
| backend_state | up |
| driver_version | 3.0.0 |
| filter_function | None |
| free_capacity_gb | 22.8 |
| goodness_function | None |
| location_info | LVMVolumeDriver:localhost.localdomain.localdomain:stack-volumes-lvm_2:thin:0 |
| max_over_subscription_ratio | 20.0 |
| multiattach | True |
| name | localhost.localdomain.localdomain@lvm_2#lvm_2 |
| pool_name | lvm_2 |
| provisioned_capacity_gb | 1.0 |
| reserved_percentage | 0 |
| storage_protocol | iSCSI |
| thick_provisioning_support | False |
| thin_provisioning_support | True |
| timestamp | 2018-06-07T05:12:27.189877 |
| total_capacity_gb | 22.8 |
| total_volumes | 2 |
| vendor_name | Open Source |
| volume_backend_name | lvm_2 ...

This is probably not a cinder bug, but is a side effect of using barbican. Cinder has its own policies for who can delete a resource (such as a volume), and barbican has its own policies that can be more restrictive. Barbican has stricter rules about deleting another user's secrets than cinder has about deleting another user's volumes. Here, cinder is refusing to delete the volume because the user is not authorized to delete the secret associated with the volume. This is done in order to avoid "leaking" an encryption key.

Would it be really an issue if cinder would delete the volume anyways? The encryption key is generated on volume create, so the key should be worthless after the volume is gone?
Maybe a temporary solution until the admin user is able to delete user secrets.
As far as I know, it is not possible to delete barbican user secrets with an admin user, making the management of encrypted volumes really annoying.

Agreed this is quite annoying behaviour. Would it be acceptable to return a 403 to the cinder API caller if we get a 403 from barbican?

@awalende, the concern is that "abandoned" encryption keys are still consuming a resource, notably in the HSM where they might be stored. Cinder's behavior is intended to highlight the fact that there's an issue deleting the key in order to avoid it becoming abandoned. I don't know about the specifics of barbican's policies regarding whether an admin can delete a user secret.

@mgoddard, I agree the behavior does not make for a good user experience. The problem is volume deletion is inherently an asynchronous operation, and at the cinder-api layer it can only do so much validation before it returns 202 when it passes the deletion request on to the cinder-volume service. And in this situation, cinder-volume deletes the key.

Alan, as far as I can tell, the key is deleted in the API service synchronously, so it would be possible to alter the return code.

https://opendev.org/openstack/cinder/src/commit/9abb00521849616b4b8662d65480e67c5848a530/cinder/volume/api.py#L522

we are hitting that as well

IMO it all boils down to default API access policies in Barbican are generally more strict than policies in other services using Barbican in respect for (global) admin actions.

I see at least 3 possible solutions, but we need to discuss consequences of each of them for the security model and use cases this functionality is intended to solve:

1) (as already mentioned) just ignore key deletion errors and orphan them with appropriate WARNING in the logs
2) change (default) api policies in Barbican to match its consumers and allow the same global admin to delete whatever secret it has to delete
3) store these auto-generated secrets under service user, at least make this choice available to operators via config option for new deployments. These secrets are basically transparent to the user, who may be oblivious to them being created and used in the first place, and user has no option to pass her own secret to the volume create / attach call, so why store them in the user's project at all? This of course highly depends on what use cases this encryption is actually trying to solve, and while it seems to me the best option, I may be missing something here...