Barbican with Live Migration issue

Bug #1784873 reported by Helen Walsh
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Cinder
New
Undecided
Unassigned

Bug Description

I am having some issues with key management in a multinode devstack (from master branch 27th July ’18) environment where Barbican is the configured key_manager. I have followed setup instructions from the following pages:
https://docs.openstack.org/barbican/latest/contributor/devstack.html (manual configuration)
https://docs.openstack.org/cinder/latest/configuration/block-storage/volume-encryption.html

So far:
• Unencrypted block volumes can be attached to instances on any compute node
• Instances with unencrypted volumes can also be live migrated to other compute node
• Encrypted bootable volumes created successfully
• Instances can be launched using these encrypted volumes when the instance is spawned on demo_machine1 (controller & compute node)
• Instances cannot be launched using encrypted volumes when the instance is spawned on demo_machine2 or demo_machine3 (compute only), the same failure can be seen in nova logs from both compute nodes:

Jul 30 14:35:18 demo_machine2 nova-compute[25686]: DEBUG cinderclient.v3.client [None req-3c977faa-a64c-4536-82c8-d1dbaf856b99 admin admin] GET call to cinderv3 for http://10.0.0.63/volume/v3/3f22a0262a7b4832a08c24ac0295cbd9/volumes/296148bf-edb8-4c9f-88c2-44464907f7e7/encryption used request id req-71fa7f20-c0bc-46c3-9f07-5866344d31a1 {{(pid=25686) request /usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py:844}}

Jul 30 14:35:18 demo_machine2 nova-compute[25686]: DEBUG os_brick.encryptors [None req-3c977faa-a64c-4536-82c8-d1dbaf856b99 admin admin] Using volume encryption metadata '{u'cipher': u'aes-xts-plain64', u'encryption_key_id': u'da7ee21c-67ff-4d74-95a0-18ee6c25d85a', u'provider': u'luks', u'key_size': 256, u'control_location': u'front-end'}' for connection: {'status': u'attaching', 'detached_at': u'', u'volume_id': u'296148bf-edb8-4c9f-88c2-44464907f7e7', 'attach_mode': u'null', 'driver_volume_type': u'iscsi', 'instance': u'e0dc6eac-09bb-4232-bea7-7b8b161cfa31', 'attached_at': u'2018-07-30T13:35:17.000000', 'serial': u'296148bf-edb8-4c9f-88c2-44464907f7e7', 'data': {'device_path': '/dev/disk/by-id/scsi-SEMC_SYMMETRIX_900049_wy000', u'target_discovered': True, u'encrypted': True, u'qos_specs': None, u'target_iqn': u'iqn.1992-04.com.emc:600009700bcbb7112504018f00000000', u'target_portal': u'192.168.0.60:3260', u'volume_id': u'296148bf-edb8-4c9f-88c2-44464907f7e7', u'target_lun': 1, u'access_mode': u'rw'}} {{(pid=25686) get_encryption_metadata /usr/local/lib/python2.7/dist-packages/os_brick/encryptors/__init__.py:125}}

Jul 30 14:35:18 demo_machine2 nova-compute[25686]: WARNING keystoneauth.identity.generic.base [None req-3c977faa-a64c-4536-82c8-d1dbaf856b99 admin admin] Failed to discover available identity versions when contacting http://localhost/identity/v3. Attempting to parse version from URL.: NotFound: Not Found (HTTP 404)

Jul 30 14:35:18 demo_machine2 nova-compute[25686]: ERROR castellan.key_manager.barbican_key_manager [None req-3c977faa-a64c-4536-82c8-d1dbaf856b99 admin admin] Error creating Barbican client: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Not Found (HTTP 404): DiscoveryFailure: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Not Found (HTTP 404)

All instance of Nova have [key_manager] configured as follows:
[key_manager]
backend = barbican
auth_url = http://10.0.0.63/identity/
### Tried with and without the below config options, same result
# auth_type = password
# password = devstack
# username = barbican

Any assistance here would be greatly appreciated, I have spent a lot of time looking for some additional information for the use of Barbican in multinode devstack environments or with live migration but there is nothing out there, everything is for all-in-one environments and I’m not having any issues when everything is on one node. I am wondering if at this point there is something I am missing in terms of services in a multinode devstack environment, qualification of barbican in a multinode environment is outside of the recommended test config but following the docs it looks very straight forward.

Some information on the three nodes in my environment are below, if there is any other information I can provide let me know, thanks for the help!

Node & Service Breakdown
Node 1 (Controller & Compute)
stack@demo_machine1:~$ openstack service list
+----------------------------------+-------------+----------------+
| ID | Name | Type |
+----------------------------------+-------------+----------------+
| 43a1334c755c4c81969565097cc9c30c | cinder | volume |
| 52a8927c09154e33900f24c7c95a9f8b | cinderv2 | volumev2 |
| 5427a9dff3b6477197062e1747843c4d | nova_legacy | compute_legacy |
| 5b319b6d50634661998fdd8dc70a85e3 | nova | compute |
| 5ffbb2e9f7c84c9e9601ab7aba0cf5e1 | placement | placement |
| 787fd29afe2f41b0bb44f9c301fd22c5 | cinderv3 | volumev3 |
| 96813e167b8842aba9d8b94fad67904f | neutron | network |
| 993e615a03cc49e3be94840c0b82636b | swift | object-store |
| b3834468ffc44f30b792459611f5f4e9 | cinder | block-storage |
| cab9ff9e175f4566a1865ea35a377d0d | barbican | key-manager |
| d12f710b815442fb970c22087b6e8f4f | glance | image |
| eb80de21e42b4e978985db979b175f79 | keystone | identity |
+----------------------------------+-------------+----------------+

stack@demo_machine1:~$ openstack endpoint list
+----------------------------------+-----------+--------------+----------------+---------+-----------+-------------------------------------------------+
| ID | Region | Service Name | Service Type | Enabled | Interface | URL |
+----------------------------------+-----------+--------------+----------------+---------+-----------+-------------------------------------------------+
| 00b276609956454d8d80dd0dde0df231 | RegionOne | cinder | volume | True | public | http://10.0.0.63/volume/v1/$(project_id)s |
| 18e5d431143d47ed980ee0ffbf0d03d7 | RegionOne | barbican | key-manager | True | public | http://10.0.0.63/key-manager |
| 20cfe0a80cc94b6eb8ea8e6784839198 | RegionOne | barbican | key-manager | True | internal | http://10.0.0.63/key-manager |
| 3a740b472e7349f19d0cf110c1792122 | RegionOne | cinderv3 | volumev3 | True | public | http://10.0.0.63/volume/v3/$(project_id)s |
| 4d957921fe894abba296331869f82f7f | RegionOne | cinderv2 | volumev2 | True | public | http://10.0.0.63/volume/v2/$(project_id)s |
| 4df258794fde476ab82502c682848e58 | RegionOne | swift | object-store | True | admin | http://10.0.0.63:8080 |
| 719eabec7cb94580af9f928278589878 | RegionOne | keystone | identity | True | public | http://10.0.0.63/identity |
| 792f4c99085f4b008643b08aff463759 | RegionOne | keystone | identity | True | admin | http://10.0.0.63/identity |
| 9e8c27c6e22f4a70865bfcdd815ed3c0 | RegionOne | cinder | block-storage | True | public | http://10.0.0.63/volume/v3/$(project_id)s |
| a271f19f29d443a0b5545626584389d7 | RegionOne | glance | image | True | public | http://10.0.0.63/image |
| a975403a2ff149bb88ce2d2227d17a80 | RegionOne | nova | compute | True | public | http://10.0.0.63/compute/v2.1 |
| b65b46e83b4547588eb694d63cb5cdd5 | RegionOne | swift | object-store | True | public | http://10.0.0.63:8080/v1/AUTH_$(project_id)s |
| bfd1f91ba18b4bc0bc83586ee358a73c | RegionOne | placement | placement | True | public | http://10.0.0.63/placement |
| d38a11dcfe824fe28f70b45422277d26 | RegionOne | nova_legacy | compute_legacy | True | public | http://10.0.0.63/compute/v2/$(project_id)s |
| ea9139e670e84ff39d1c052347a04695 | RegionOne | neutron | network | True | public | http://10.0.0.63:9696/ |
+----------------------------------+-----------+--------------+----------------+---------+-----------+-------------------------------------------------+

      stack@demo_machine1:~$ openstack secret store
+---------------+---------------------------------------------------------------------------------+
| Field | Value |
+---------------+---------------------------------------------------------------------------------+
| Secret href | http://10.0.0.63/key-manager/v1/secrets/72a3955b-a494-4352-b1f6-ae3f322e5656 |
| Name | None |
| Created | 2018-07-30T12:58:33+00:00 |
| Status | ACTIVE |
| Content types | None |
| Algorithm | aes |
| Bit length | 256 |
| Secret type | opaque |
| Mode | cbc |
| Expiration | None |
+---------------+---------------------------------------------------------------------------------+

Node 2 & 3 (Compute Only)
Services:
      stack@demo_machine2:~$ sudo systemctl list-unit-files | grep devstack@*
      <email address hidden> enabled
      <email address hidden> enabled
      <email address hidden> enabled

      stack@demo_machine3:~$ sudo systemctl list-unit-files | grep devstack@*
      <email address hidden> enabled
      <email address hidden> enabled
      <email address hidden> enabled

********************************************************************

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.