Ubuntu
magnum package

Fixed interval looping call 'magnum.service.periodic.ClusterUpdateJob.update_status' failed: magnum.common.exception.AuthorizationFailure

  1. Focal (20.04)
  2. Bug #2039599
Bug #2039599 reported by Felipe Reyes
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
magnum (Ubuntu)
Invalid
Undecided
Unassigned
Focal
In Progress
High
Felipe Reyes

Bug Description

[Impact]

When deleting a cluster (from the web dashboard or CLI), the cluster stays in DELETE_IN_PROGRESS , internally magnum-conductor issues calls to heat to delete the stacks, although the periodic job that tracks the progress fails when the trust has been deleted, this prevents magnum-conductor to query Heat.

Stacktrace:

2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall [req-5d099ed5-c878-454f-91b7-6979bd674f62 - - - - -] Fixed interval looping call 'magnum.service.periodic.ClusterUpdateJob.update_status' failed: magnum.common.exception.AuthorizationFailure: unexpected keystone client error occurred: You are not authorized to perform the requested action. (HTTP 403) (Request-ID: req-6bfd2d88-87fa-4d21-959e-e8e994077823)
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall Traceback (most recent call last):
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/magnum/common/exception.py", line 57, in wrapped
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall return func(*args, **kw)
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/magnum/common/clients.py", line 110, in heat
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall endpoint = self.url_for(service_type='orchestration',
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/magnum/common/clients.py", line 48, in url_for
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall return self.keystone().session.get_endpoint(**kwargs)
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1225, in get_endpoint
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall return auth.get_endpoint(self, **kwargs)
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 375, in get_endpoint
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall endpoint_data = self.get_endpoint_data(
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 271, in get_endpoint_data
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall service_catalog = self.get_access(session).service_catalog
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/keystoneauth1/identity/base.py", line 134, in get_access
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall self.auth_ref = self.get_auth_ref(session)
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/keystoneauth1/identity/v3/base.py", line 183, in get_auth_ref
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall resp = session.post(token_url, json=body, headers=headers,
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 1131, in post
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall return self.request(url, 'POST', **kwargs)
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 968, in request
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall raise exceptions.from_response(resp, method, url)
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall keystoneauth1.exceptions.http.Forbidden: You are not authorized to perform the requested action. (HTTP 403) (Request-ID: req-6bfd2d88-87fa-4d21-959e-e8e994077823)
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall During handling of the above exception, another exception occurred:
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall Traceback (most recent call last):
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/oslo_service/loopingcall.py", line 150, in _run_loop
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall result = func(*self.args, **self.kw)
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/magnum/service/periodic.py", line 71, in update_status
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall cdriver.update_cluster_status(self.ctx, self.cluster)
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/magnum/drivers/heat/driver.py", line 131, in update_cluster_status
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall poller.poll_and_check()
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/magnum/drivers/heat/driver.py", line 470, in poll_and_check
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall status = self.extract_nodegroup_status()
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/magnum/drivers/heat/driver.py", line 495, in extract_nodegroup_status
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall stack = self.openstack_client.heat().stacks.get(
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall File "/usr/lib/python3/dist-packages/magnum/common/exception.py", line 62, in wrapped
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall raise AuthorizationFailure(
2023-09-08 12:09:34.989 3605161 ERROR oslo.service.loopingcall magnum.common.exception.AuthorizationFailure: unexpected keystone client error occurred: You are not authorized to perform the requested action. (HTTP 403) (Request-ID: req-6bfd2d88-87fa-4d21-959e-e8e994077823)

[Test Plan]

TBD

[Where problems could occur]

TBD

[Other Info]

Upstream bug: https://storyboard.openstack.org/#!/story/2010232
Fixed by: https://opendev.org/openstack/magnum/commit/1ed78a44384ae2617e49acb9031fa3f0964f12f7

This bug was fixed in upstream during the OpenStack Zed (Ubuntu 22.10) development cycle and backported to OpenStack Yoga, Xena and Wallaby

Ubuntu 22.04 has the magnum-14.1.0 which corresponds to OpenStack Yoga, the fix backported ( https://opendev.org/openstack/magnum/commit/4d30bef60227cd8246f4caa00857761fa56b7d51 ) is available

$ git tag --contains 4d30bef60227cd8246f4caa00857761fa56b7d51
14.1.0
14.1.1
$ rmadison magnum | grep jammy
 magnum | 14.0.0-0ubuntu1 | jammy/universe | source
 magnum | 14.1.0-0ubuntu1 | jammy-updates/universe | source
 magnum | 14.1.1-0ubuntu1 | jammy-proposed/universe | source

See original description
Felipe Reyes (freyes)
Changed in magnum (Ubuntu):
status: New → Invalid
Felipe Reyes (freyes)
description: updated
Changed in magnum (Ubuntu Focal):
assignee: nobody → Felipe Reyes (freyes)
summary: Fixed interval looping call
'magnum.service.periodic.ClusterUpdateJob.update_status' failed:
- magnum.common.exception.AuthorizationFailure: unexpected keystone client
- error occurred: You are not authorized to perform the requested action
+ magnum.common.exception.AuthorizationFailure
Felipe Reyes (freyes)
Changed in magnum (Ubuntu Focal):
importance: Undecided → High
status: New → In Progress
summary: - Fixed interval looping call
+ [SRU] Fixed interval looping call
'magnum.service.periodic.ClusterUpdateJob.update_status' failed:
magnum.common.exception.AuthorizationFailure
Revision history for this message
Felipe Reyes (freyes) wrote : Re: [SRU] Fixed interval looping call 'magnum.service.periodic.ClusterUpdateJob.update_status' failed: magnum.common.exception.AuthorizationFailure

PPA for testing purposes available at https://launchpad.net/~freyes/+archive/ubuntu/lp2039599

Revision history for this message
Felipe Reyes (freyes) wrote : Re: [Bug 2039599] Re: [SRU] Fixed interval looping call 'magnum.service.periodic.ClusterUpdateJob.update_status' failed: magnum.common.exception.AuthorizationFailure

I've been reading the patch and I find something different between the expectations of failure in
the patch and the stacktrace posted in the description.

Specifically the patch contains this line:

+ except exception.AuthorizationFailure as e:
+ trust_ex = ("Could not find trust: %s" % self.cluster.trust_id)
[...]
+ if trust_ex in str(e):
[...]

While the error in the stacktrace is not about "could not find trust", it's simply a "you are not
authorized to perform the requested action".

magnum.common.exception.AuthorizationFailure: unexpected keystone client error occurred: You are not
authorized to perform the requested action. (HTTP 403)

I will try to get copies of the keystone logs for 2023-09-08 to know why keystone didn't allow the
access.

summary: - [SRU] Fixed interval looping call
+ Fixed interval looping call
'magnum.service.periodic.ClusterUpdateJob.update_status' failed:
magnum.common.exception.AuthorizationFailure
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.