| 2023-06-22 15:26:54 |
Jorge Merlino |
bug |
|
|
added bug |
| 2023-06-22 15:27:22 |
Jorge Merlino |
description |
There is a bug with the service token configuration when cinder is separated between cinder and cinder-volume.
Plain cider uses the identity-service relation with keystone while cinder-volume uses identity-credentials.
The token created with the credentials obtained from identity-credentials fails to authenticate on keystone.
I can reproduce this on a focal/yoga openstack installation using cinder-volume and cinder-ceph but it also happens with pure-storage. Using the default service token configuration cinder-volume can not create a volume based on an image because glance rejects the service token. This can be fixed by removing the [service-user] section on cinder-volume or by using the same credentials as those obtained by the identity-service relation.
This is the exception shown on the cinder-volume.log:
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server [req-a49bac91-585d-47bd-9ed2-5a858f4ca2de 1d0e71f8854a4cddab77ac23b88dd841 38560b4f0be54e17af9a90ca2e90144f - - -] Exception during message handling: keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-f0458e09-ee73-4506-b053-f7b6553f6317)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "<decorator-gen-719>", line 2, in create_volume
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/objects/cleanable.py", line 208, in wrapper
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server result = f(*args, **kwargs)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/volume/manager.py", line 815, in create_volume
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server _run_flow()
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/volume/manager.py", line 807, in _run_flow
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server flow_engine.run()
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout):
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/taskflow/types/failure.py", line 339, in reraise_if_any
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server failures[0].reraise()
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/taskflow/types/failure.py", line 346, in reraise
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server six.reraise(*self._exc_info)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/six.py", line 719, in reraise
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server raise value
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server result = task.execute(**arguments)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/volume/flows/manager/create_volume.py", line 322, in execute
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server 'image_location': image_service.get_location(context,
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/image/glance.py", line 336, in get_location
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server _reraise_translated_image_exception(image_id)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/image/glance.py", line 635, in _reraise_translated_image_exception
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server raise new_exc.with_traceback(exc_trace)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/image/glance.py", line 334, in get_location
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server image_meta = client.call(context, 'get', image_id)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/image/glance.py", line 236, in call
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server return getattr(controller, method)(*args, **kwargs)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/glanceclient/v2/images.py", line 197, in get
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server return self._get(image_id)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/glanceclient/common/utils.py", line 670, in inner
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server return RequestIdProxy(wrapped(*args, **kwargs))
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/glanceclient/v2/images.py", line 190, in _get
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server resp, body = self.http_client.get(url, headers=header)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 395, in get
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server return self.request(url, 'GET', **kwargs)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/glanceclient/common/http.py", line 361, in request
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server resp = super(SessionClient,
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 257, in request
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server return self.session.request(url, method, **kwargs)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 790, in request
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server auth_headers = self.get_auth_headers(auth)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1201, in get_auth_headers
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server return auth.get_headers(self, **kwargs)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/service_token.py", line 29, in get_headers
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server token = self.service_auth.get_token(session, **kwargs)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 88, in get_token
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server return self.get_access(session).auth_token
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 134, in get_access
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server self.auth_ref = self.get_auth_ref(session)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/identity/generic/base.py", line 208, in get_auth_ref
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server return self._plugin.get_auth_ref(session, **kwargs)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/identity/v3/base.py", line 187, in get_auth_ref
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server resp = session.post(token_url, json=body, headers=headers,
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1149, in post
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server return self.request(url, 'POST', **kwargs)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 986, in request
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server raise exceptions.from_response(resp, method, url)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-f0458e09-ee73-4506-b053-f7b6553f6317) |
There is a bug with the service token configuration when cinder is separated between cinder and cinder-volume.
Plain cinder uses the identity-service relation with keystone while cinder-volume uses identity-credentials.
The token created with the credentials obtained from identity-credentials fails to authenticate on keystone.
I can reproduce this on a focal/yoga openstack installation using cinder-volume and cinder-ceph but it also happens with pure-storage. Using the default service token configuration cinder-volume can not create a volume based on an image because glance rejects the service token. This can be fixed by removing the [service-user] section on cinder-volume or by using the same credentials as those obtained by the identity-service relation.
This is the exception shown on the cinder-volume.log:
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server [req-a49bac91-585d-47bd-9ed2-5a858f4ca2de 1d0e71f8854a4cddab77ac23b88dd841 38560b4f0be54e17af9a90ca2e90144f - - -] Exception during message handling: keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-f0458e09-ee73-4506-b053-f7b6553f6317)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "<decorator-gen-719>", line 2, in create_volume
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/objects/cleanable.py", line 208, in wrapper
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server result = f(*args, **kwargs)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/volume/manager.py", line 815, in create_volume
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server _run_flow()
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/volume/manager.py", line 807, in _run_flow
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server flow_engine.run()
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout):
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/taskflow/types/failure.py", line 339, in reraise_if_any
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server failures[0].reraise()
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/taskflow/types/failure.py", line 346, in reraise
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server six.reraise(*self._exc_info)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/six.py", line 719, in reraise
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server raise value
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server result = task.execute(**arguments)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/volume/flows/manager/create_volume.py", line 322, in execute
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server 'image_location': image_service.get_location(context,
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/image/glance.py", line 336, in get_location
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server _reraise_translated_image_exception(image_id)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/image/glance.py", line 635, in _reraise_translated_image_exception
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server raise new_exc.with_traceback(exc_trace)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/image/glance.py", line 334, in get_location
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server image_meta = client.call(context, 'get', image_id)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/image/glance.py", line 236, in call
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server return getattr(controller, method)(*args, **kwargs)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/glanceclient/v2/images.py", line 197, in get
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server return self._get(image_id)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/glanceclient/common/utils.py", line 670, in inner
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server return RequestIdProxy(wrapped(*args, **kwargs))
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/glanceclient/v2/images.py", line 190, in _get
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server resp, body = self.http_client.get(url, headers=header)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 395, in get
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server return self.request(url, 'GET', **kwargs)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/glanceclient/common/http.py", line 361, in request
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server resp = super(SessionClient,
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 257, in request
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server return self.session.request(url, method, **kwargs)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 790, in request
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server auth_headers = self.get_auth_headers(auth)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1201, in get_auth_headers
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server return auth.get_headers(self, **kwargs)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/service_token.py", line 29, in get_headers
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server token = self.service_auth.get_token(session, **kwargs)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 88, in get_token
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server return self.get_access(session).auth_token
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 134, in get_access
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server self.auth_ref = self.get_auth_ref(session)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/identity/generic/base.py", line 208, in get_auth_ref
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server return self._plugin.get_auth_ref(session, **kwargs)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/identity/v3/base.py", line 187, in get_auth_ref
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server resp = session.post(token_url, json=body, headers=headers,
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1149, in post
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server return self.request(url, 'POST', **kwargs)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 986, in request
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server raise exceptions.from_response(resp, method, url)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-f0458e09-ee73-4506-b053-f7b6553f6317) |
|
| 2023-06-22 15:28:35 |
Felipe Reyes |
nominated for series |
|
charm-cinder/yoga |
|
| 2023-06-22 15:28:35 |
Felipe Reyes |
bug task added |
|
charm-cinder/yoga |
|
| 2023-06-22 15:28:35 |
Felipe Reyes |
nominated for series |
|
charm-cinder/zed |
|
| 2023-06-22 15:28:35 |
Felipe Reyes |
bug task added |
|
charm-cinder/zed |
|
| 2023-06-22 15:28:35 |
Felipe Reyes |
nominated for series |
|
charm-cinder/2023.1 |
|
| 2023-06-22 15:28:35 |
Felipe Reyes |
bug task added |
|
charm-cinder/2023.1 |
|
| 2023-06-22 15:28:42 |
Felipe Reyes |
nominated for series |
|
charm-cinder/trunk |
|
| 2023-06-22 15:28:42 |
Felipe Reyes |
bug task added |
|
charm-cinder/trunk |
|
| 2023-06-22 19:43:11 |
Jorge Merlino |
description |
There is a bug with the service token configuration when cinder is separated between cinder and cinder-volume.
Plain cinder uses the identity-service relation with keystone while cinder-volume uses identity-credentials.
The token created with the credentials obtained from identity-credentials fails to authenticate on keystone.
I can reproduce this on a focal/yoga openstack installation using cinder-volume and cinder-ceph but it also happens with pure-storage. Using the default service token configuration cinder-volume can not create a volume based on an image because glance rejects the service token. This can be fixed by removing the [service-user] section on cinder-volume or by using the same credentials as those obtained by the identity-service relation.
This is the exception shown on the cinder-volume.log:
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server [req-a49bac91-585d-47bd-9ed2-5a858f4ca2de 1d0e71f8854a4cddab77ac23b88dd841 38560b4f0be54e17af9a90ca2e90144f - - -] Exception during message handling: keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-f0458e09-ee73-4506-b053-f7b6553f6317)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "<decorator-gen-719>", line 2, in create_volume
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/objects/cleanable.py", line 208, in wrapper
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server result = f(*args, **kwargs)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/volume/manager.py", line 815, in create_volume
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server _run_flow()
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/volume/manager.py", line 807, in _run_flow
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server flow_engine.run()
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout):
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/taskflow/types/failure.py", line 339, in reraise_if_any
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server failures[0].reraise()
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/taskflow/types/failure.py", line 346, in reraise
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server six.reraise(*self._exc_info)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/six.py", line 719, in reraise
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server raise value
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server result = task.execute(**arguments)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/volume/flows/manager/create_volume.py", line 322, in execute
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server 'image_location': image_service.get_location(context,
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/image/glance.py", line 336, in get_location
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server _reraise_translated_image_exception(image_id)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/image/glance.py", line 635, in _reraise_translated_image_exception
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server raise new_exc.with_traceback(exc_trace)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/image/glance.py", line 334, in get_location
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server image_meta = client.call(context, 'get', image_id)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/image/glance.py", line 236, in call
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server return getattr(controller, method)(*args, **kwargs)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/glanceclient/v2/images.py", line 197, in get
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server return self._get(image_id)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/glanceclient/common/utils.py", line 670, in inner
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server return RequestIdProxy(wrapped(*args, **kwargs))
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/glanceclient/v2/images.py", line 190, in _get
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server resp, body = self.http_client.get(url, headers=header)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 395, in get
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server return self.request(url, 'GET', **kwargs)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/glanceclient/common/http.py", line 361, in request
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server resp = super(SessionClient,
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 257, in request
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server return self.session.request(url, method, **kwargs)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 790, in request
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server auth_headers = self.get_auth_headers(auth)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1201, in get_auth_headers
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server return auth.get_headers(self, **kwargs)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/service_token.py", line 29, in get_headers
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server token = self.service_auth.get_token(session, **kwargs)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 88, in get_token
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server return self.get_access(session).auth_token
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 134, in get_access
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server self.auth_ref = self.get_auth_ref(session)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/identity/generic/base.py", line 208, in get_auth_ref
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server return self._plugin.get_auth_ref(session, **kwargs)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/identity/v3/base.py", line 187, in get_auth_ref
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server resp = session.post(token_url, json=body, headers=headers,
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1149, in post
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server return self.request(url, 'POST', **kwargs)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 986, in request
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server raise exceptions.from_response(resp, method, url)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-f0458e09-ee73-4506-b053-f7b6553f6317) |
There is a bug with the service token configuration when cinder is separated between cinder and cinder-volume.
Plain cinder uses the identity-service relation with keystone while cinder-volume uses identity-credentials.
Cinder-volume can't create a token with the credentials obtained from identity-credentials. They fail to authenticate on keystone.
I can reproduce this on a focal/yoga openstack installation using cinder-volume and cinder-ceph but it also happens with pure-storage. Using the default service token configuration cinder-volume can not create a volume based on an image because glance rejects the service token. This can be fixed by removing the [service-user] section on cinder-volume or by using the same credentials as those obtained by the identity-service relation.
This is the exception shown on the cinder-volume.log:
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server [req-a49bac91-585d-47bd-9ed2-5a858f4ca2de 1d0e71f8854a4cddab77ac23b88dd841 38560b4f0be54e17af9a90ca2e90144f - - -] Exception during message handling: keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-f0458e09-ee73-4506-b053-f7b6553f6317)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "<decorator-gen-719>", line 2, in create_volume
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/objects/cleanable.py", line 208, in wrapper
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server result = f(*args, **kwargs)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/volume/manager.py", line 815, in create_volume
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server _run_flow()
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/volume/manager.py", line 807, in _run_flow
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server flow_engine.run()
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout):
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/taskflow/types/failure.py", line 339, in reraise_if_any
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server failures[0].reraise()
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/taskflow/types/failure.py", line 346, in reraise
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server six.reraise(*self._exc_info)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/six.py", line 719, in reraise
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server raise value
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server result = task.execute(**arguments)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/volume/flows/manager/create_volume.py", line 322, in execute
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server 'image_location': image_service.get_location(context,
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/image/glance.py", line 336, in get_location
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server _reraise_translated_image_exception(image_id)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/image/glance.py", line 635, in _reraise_translated_image_exception
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server raise new_exc.with_traceback(exc_trace)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/image/glance.py", line 334, in get_location
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server image_meta = client.call(context, 'get', image_id)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/image/glance.py", line 236, in call
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server return getattr(controller, method)(*args, **kwargs)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/glanceclient/v2/images.py", line 197, in get
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server return self._get(image_id)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/glanceclient/common/utils.py", line 670, in inner
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server return RequestIdProxy(wrapped(*args, **kwargs))
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/glanceclient/v2/images.py", line 190, in _get
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server resp, body = self.http_client.get(url, headers=header)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 395, in get
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server return self.request(url, 'GET', **kwargs)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/glanceclient/common/http.py", line 361, in request
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server resp = super(SessionClient,
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 257, in request
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server return self.session.request(url, method, **kwargs)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 790, in request
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server auth_headers = self.get_auth_headers(auth)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1201, in get_auth_headers
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server return auth.get_headers(self, **kwargs)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/service_token.py", line 29, in get_headers
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server token = self.service_auth.get_token(session, **kwargs)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 88, in get_token
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server return self.get_access(session).auth_token
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 134, in get_access
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server self.auth_ref = self.get_auth_ref(session)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/identity/generic/base.py", line 208, in get_auth_ref
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server return self._plugin.get_auth_ref(session, **kwargs)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/identity/v3/base.py", line 187, in get_auth_ref
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server resp = session.post(token_url, json=body, headers=headers,
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1149, in post
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server return self.request(url, 'POST', **kwargs)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 986, in request
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server raise exceptions.from_response(resp, method, url)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-f0458e09-ee73-4506-b053-f7b6553f6317) |
|
| 2023-06-22 19:58:09 |
Pedro Victor Lourenço Fragola |
tags |
|
sts |
|
| 2023-06-22 20:12:02 |
Jorge Merlino |
description |
There is a bug with the service token configuration when cinder is separated between cinder and cinder-volume.
Plain cinder uses the identity-service relation with keystone while cinder-volume uses identity-credentials.
Cinder-volume can't create a token with the credentials obtained from identity-credentials. They fail to authenticate on keystone.
I can reproduce this on a focal/yoga openstack installation using cinder-volume and cinder-ceph but it also happens with pure-storage. Using the default service token configuration cinder-volume can not create a volume based on an image because glance rejects the service token. This can be fixed by removing the [service-user] section on cinder-volume or by using the same credentials as those obtained by the identity-service relation.
This is the exception shown on the cinder-volume.log:
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server [req-a49bac91-585d-47bd-9ed2-5a858f4ca2de 1d0e71f8854a4cddab77ac23b88dd841 38560b4f0be54e17af9a90ca2e90144f - - -] Exception during message handling: keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-f0458e09-ee73-4506-b053-f7b6553f6317)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "<decorator-gen-719>", line 2, in create_volume
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/objects/cleanable.py", line 208, in wrapper
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server result = f(*args, **kwargs)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/volume/manager.py", line 815, in create_volume
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server _run_flow()
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/volume/manager.py", line 807, in _run_flow
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server flow_engine.run()
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout):
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/taskflow/types/failure.py", line 339, in reraise_if_any
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server failures[0].reraise()
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/taskflow/types/failure.py", line 346, in reraise
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server six.reraise(*self._exc_info)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/six.py", line 719, in reraise
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server raise value
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server result = task.execute(**arguments)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/volume/flows/manager/create_volume.py", line 322, in execute
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server 'image_location': image_service.get_location(context,
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/image/glance.py", line 336, in get_location
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server _reraise_translated_image_exception(image_id)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/image/glance.py", line 635, in _reraise_translated_image_exception
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server raise new_exc.with_traceback(exc_trace)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/image/glance.py", line 334, in get_location
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server image_meta = client.call(context, 'get', image_id)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/image/glance.py", line 236, in call
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server return getattr(controller, method)(*args, **kwargs)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/glanceclient/v2/images.py", line 197, in get
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server return self._get(image_id)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/glanceclient/common/utils.py", line 670, in inner
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server return RequestIdProxy(wrapped(*args, **kwargs))
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/glanceclient/v2/images.py", line 190, in _get
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server resp, body = self.http_client.get(url, headers=header)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 395, in get
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server return self.request(url, 'GET', **kwargs)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/glanceclient/common/http.py", line 361, in request
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server resp = super(SessionClient,
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 257, in request
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server return self.session.request(url, method, **kwargs)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 790, in request
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server auth_headers = self.get_auth_headers(auth)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1201, in get_auth_headers
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server return auth.get_headers(self, **kwargs)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/service_token.py", line 29, in get_headers
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server token = self.service_auth.get_token(session, **kwargs)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 88, in get_token
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server return self.get_access(session).auth_token
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 134, in get_access
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server self.auth_ref = self.get_auth_ref(session)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/identity/generic/base.py", line 208, in get_auth_ref
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server return self._plugin.get_auth_ref(session, **kwargs)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/identity/v3/base.py", line 187, in get_auth_ref
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server resp = session.post(token_url, json=body, headers=headers,
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1149, in post
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server return self.request(url, 'POST', **kwargs)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 986, in request
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server raise exceptions.from_response(resp, method, url)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-f0458e09-ee73-4506-b053-f7b6553f6317) |
There is a bug with the service token configuration when cinder is separated between cinder and cinder-volume.
Plain cinder uses the identity-service relation with keystone while cinder-volume uses identity-credentials.
Cinder-volume can't create a token with the credentials obtained from identity-credentials. They fail to authenticate on keystone.
I can reproduce this on a focal/yoga openstack installation using cinder-volume and cinder-ceph but it also happens with pure-storage. Using the default service token configuration cinder-volume can not create a volume based on an image because it can't create the service token. This can be fixed by removing the [service-user] section on cinder-volume or by using the same credentials as those obtained by the identity-service relation.
This is the exception shown on the cinder-volume.log:
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server [req-a49bac91-585d-47bd-9ed2-5a858f4ca2de 1d0e71f8854a4cddab77ac23b88dd841 38560b4f0be54e17af9a90ca2e90144f - - -] Exception during message handling: keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-f0458e09-ee73-4506-b053-f7b6553f6317)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "<decorator-gen-719>", line 2, in create_volume
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/objects/cleanable.py", line 208, in wrapper
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server result = f(*args, **kwargs)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/volume/manager.py", line 815, in create_volume
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server _run_flow()
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/volume/manager.py", line 807, in _run_flow
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server flow_engine.run()
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/taskflow/engines/action_engine/engine.py", line 247, in run
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout):
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/taskflow/types/failure.py", line 339, in reraise_if_any
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server failures[0].reraise()
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/taskflow/types/failure.py", line 346, in reraise
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server six.reraise(*self._exc_info)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/six.py", line 719, in reraise
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server raise value
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server result = task.execute(**arguments)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/volume/flows/manager/create_volume.py", line 322, in execute
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server 'image_location': image_service.get_location(context,
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/image/glance.py", line 336, in get_location
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server _reraise_translated_image_exception(image_id)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/image/glance.py", line 635, in _reraise_translated_image_exception
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server raise new_exc.with_traceback(exc_trace)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/image/glance.py", line 334, in get_location
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server image_meta = client.call(context, 'get', image_id)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/cinder/image/glance.py", line 236, in call
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server return getattr(controller, method)(*args, **kwargs)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/glanceclient/v2/images.py", line 197, in get
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server return self._get(image_id)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/glanceclient/common/utils.py", line 670, in inner
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server return RequestIdProxy(wrapped(*args, **kwargs))
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/glanceclient/v2/images.py", line 190, in _get
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server resp, body = self.http_client.get(url, headers=header)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 395, in get
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server return self.request(url, 'GET', **kwargs)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/glanceclient/common/http.py", line 361, in request
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server resp = super(SessionClient,
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 257, in request
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server return self.session.request(url, method, **kwargs)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 790, in request
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server auth_headers = self.get_auth_headers(auth)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1201, in get_auth_headers
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server return auth.get_headers(self, **kwargs)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/service_token.py", line 29, in get_headers
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server token = self.service_auth.get_token(session, **kwargs)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 88, in get_token
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server return self.get_access(session).auth_token
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 134, in get_access
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server self.auth_ref = self.get_auth_ref(session)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/identity/generic/base.py", line 208, in get_auth_ref
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server return self._plugin.get_auth_ref(session, **kwargs)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/identity/v3/base.py", line 187, in get_auth_ref
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server resp = session.post(token_url, json=body, headers=headers,
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1149, in post
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server return self.request(url, 'POST', **kwargs)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 986, in request
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server raise exceptions.from_response(resp, method, url)
2023-06-22 14:24:28.231 544373 ERROR oslo_messaging.rpc.server keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-f0458e09-ee73-4506-b053-f7b6553f6317) |
|
| 2023-06-27 21:53:58 |
Jorge Merlino |
charm-cinder/trunk: status |
New |
Fix Committed |
|
| 2023-07-06 17:33:30 |
Corey Bryant |
charm-cinder/2023.1: status |
New |
Fix Released |
|
| 2023-07-06 17:33:37 |
Corey Bryant |
charm-cinder/zed: status |
New |
Fix Released |
|
| 2023-07-06 17:33:39 |
Corey Bryant |
charm-cinder/yoga: status |
New |
Fix Released |
|
| 2023-08-18 01:25:42 |
Felipe Reyes |
summary |
Service tokens do not work on cider-volume charm |
Service tokens do not work on cinder-volume charm |
|
