"AttributeError: 'Enforcer' object has no attribute '_warning_emitted'" when keystone API run with > 1 wsgi threads
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
OpenStack Identity (keystone) |
New
|
Undecided
|
Unassigned |
Bug Description
Seeing this in the gate while testing a change that sets wsgi threads to 16 in the wsgi config [1]:
[uwsgi]
threads = 16
In keystone, we get "ERROR keystone AttributeError: 'Enforcer' object has no attribute '_warning_emitted'" and a 500 error returned from the API. The problem seems to be around the global policy enforcer object code [2]:
@property
def _enforcer(self):
# The raw oslo-policy enforcer object
if self.__ENFORCER is None:
# NOTE(cmurphy) when running in the keystone server, suppress
# deprecation warnings for individual policy rules. Instead, we log
# a single notification at enforcement time indicating the
# oslo.policy tools the operator can use to detect and resolve
# deprecated policies. If there is no request context here, that
# means external tooling such as the oslo.policy tools are running
# this code, in which case we do want the full deprecation warnings
# emitted for individual polcy rules.
if flask.has_
# NOTE(cmurphy) Tests may explicitly disable these warnings to
# prevent an explosion of test logs
if self.suppress_
return self.__ENFORCER
With multiple wsgi threads, it can get into a state where self.__
def _emit_warning():
if not self._enforcer.
Full traceback from an example job run [4]:
Aug 22 01:06:34.637238 np0035008942 <email address hidden>[47694]: DEBUG keystone.
Aug 22 01:06:34.671985 np0035008942 <email address hidden>[47694]: CRITICAL keystone [None req-e7086405-
Aug 22 01:06:34.671985 np0035008942 <email address hidden>[47694]: ERROR keystone Traceback (most recent call last):
Aug 22 01:06:34.671985 np0035008942 <email address hidden>[47694]: ERROR keystone File "/opt/stack/
Aug 22 01:06:34.671985 np0035008942 <email address hidden>[47694]: ERROR keystone return self.wsgi_
Aug 22 01:06:34.671985 np0035008942 <email address hidden>[47694]: ERROR keystone File "/opt/stack/
Aug 22 01:06:34.671985 np0035008942 <email address hidden>[47694]: ERROR keystone return self.app(environ, start_response)
Aug 22 01:06:34.671985 np0035008942 <email address hidden>[47694]: ERROR keystone File "/opt/stack/
Aug 22 01:06:34.671985 np0035008942 <email address hidden>[47694]: ERROR keystone resp = self.call_func(req, *args, **kw)
Aug 22 01:06:34.671985 np0035008942 <email address hidden>[47694]: ERROR keystone File "/opt/stack/
Aug 22 01:06:34.671985 np0035008942 <email address hidden>[47694]: ERROR keystone return self.func(req, *args, **kwargs)
Aug 22 01:06:34.671985 np0035008942 <email address hidden>[47694]: ERROR keystone File "/opt/stack/
Aug 22 01:06:34.671985 np0035008942 <email address hidden>[47694]: ERROR keystone response = req.get_
Aug 22 01:06:34.671985 np0035008942 <email address hidden>[47694]: ERROR keystone File "/opt/stack/
Aug 22 01:06:34.671985 np0035008942 <email address hidden>[47694]: ERROR keystone status, headers, app_iter = self.call_
Aug 22 01:06:34.671985 np0035008942 <email address hidden>[47694]: ERROR keystone File "/opt/stack/
Aug 22 01:06:34.671985 np0035008942 <email address hidden>[47694]: ERROR keystone app_iter = application(
Aug 22 01:06:34.671985 np0035008942 <email address hidden>[47694]: ERROR keystone File "/opt/stack/
Aug 22 01:06:34.671985 np0035008942 <email address hidden>[47694]: ERROR keystone return resp(environ, start_response)
Aug 22 01:06:34.671985 np0035008942 <email address hidden>[47694]: ERROR keystone File "/opt/stack/
Aug 22 01:06:34.675440 np0035008942 <email address hidden>[47694]: ERROR keystone resp = self.call_func(req, *args, **kw)
Aug 22 01:06:34.675440 np0035008942 <email address hidden>[47694]: ERROR keystone File "/opt/stack/
Aug 22 01:06:34.675440 np0035008942 <email address hidden>[47694]: ERROR keystone return self.func(req, *args, **kwargs)
Aug 22 01:06:34.675440 np0035008942 <email address hidden>[47694]: ERROR keystone File "/opt/stack/
Aug 22 01:06:34.675440 np0035008942 <email address hidden>[47694]: ERROR keystone response = req.get_
Aug 22 01:06:34.675440 np0035008942 <email address hidden>[47694]: ERROR keystone File "/opt/stack/
Aug 22 01:06:34.675440 np0035008942 <email address hidden>[47694]: ERROR keystone status, headers, app_iter = self.call_
Aug 22 01:06:34.675440 np0035008942 <email address hidden>[47694]: ERROR keystone File "/opt/stack/
Aug 22 01:06:34.675440 np0035008942 <email address hidden>[47694]: ERROR keystone app_iter = application(
Aug 22 01:06:34.675440 np0035008942 <email address hidden>[47694]: ERROR keystone File "/opt/stack/
Aug 22 01:06:34.675440 np0035008942 <email address hidden>[47694]: ERROR keystone resp = self.call_func(req, *args, **kw)
Aug 22 01:06:34.675440 np0035008942 <email address hidden>[47694]: ERROR keystone File "/opt/stack/
Aug 22 01:06:34.675440 np0035008942 <email address hidden>[47694]: ERROR keystone return self.func(req, *args, **kwargs)
Aug 22 01:06:34.675440 np0035008942 <email address hidden>[47694]: ERROR keystone File "/opt/stack/
Aug 22 01:06:34.675440 np0035008942 <email address hidden>[47694]: ERROR keystone return request.
Aug 22 01:06:34.675440 np0035008942 <email address hidden>[47694]: ERROR keystone File "/opt/stack/
Aug 22 01:06:34.675440 np0035008942 <email address hidden>[47694]: ERROR keystone status, headers, app_iter = self.call_
Aug 22 01:06:34.675440 np0035008942 <email address hidden>[47694]: ERROR keystone File "/opt/stack/
Aug 22 01:06:34.675440 np0035008942 <email address hidden>[47694]: ERROR keystone app_iter = application(
Aug 22 01:06:34.675440 np0035008942 <email address hidden>[47694]: ERROR keystone File "/opt/stack/
Aug 22 01:06:34.678696 np0035008942 <email address hidden>[47694]: ERROR keystone resp = self.call_func(req, *args, **kw)
Aug 22 01:06:34.678696 np0035008942 <email address hidden>[47694]: ERROR keystone File "/opt/stack/
Aug 22 01:06:34.678696 np0035008942 <email address hidden>[47694]: ERROR keystone return self.func(req, *args, **kwargs)
Aug 22 01:06:34.678696 np0035008942 <email address hidden>[47694]: ERROR keystone File "/opt/stack/
Aug 22 01:06:34.678696 np0035008942 <email address hidden>[47694]: ERROR keystone response = req.get_
Aug 22 01:06:34.678696 np0035008942 <email address hidden>[47694]: ERROR keystone File "/opt/stack/
Aug 22 01:06:34.678696 np0035008942 <email address hidden>[47694]: ERROR keystone status, headers, app_iter = self.call_
Aug 22 01:06:34.678696 np0035008942 <email address hidden>[47694]: ERROR keystone File "/opt/stack/
Aug 22 01:06:34.678696 np0035008942 <email address hidden>[47694]: ERROR keystone app_iter = application(
Aug 22 01:06:34.678696 np0035008942 <email address hidden>[47694]: ERROR keystone File "/opt/stack/
Aug 22 01:06:34.678696 np0035008942 <email address hidden>[47694]: ERROR keystone return self.app(environ, start_response)
Aug 22 01:06:34.678696 np0035008942 <email address hidden>[47694]: ERROR keystone File "/opt/stack/
Aug 22 01:06:34.678696 np0035008942 <email address hidden>[47694]: ERROR keystone resp = self.call_func(req, *args, **kw)
Aug 22 01:06:34.678696 np0035008942 <email address hidden>[47694]: ERROR keystone File "/opt/stack/
Aug 22 01:06:34.678696 np0035008942 <email address hidden>[47694]: ERROR keystone return self.func(req, *args, **kwargs)
Aug 22 01:06:34.678696 np0035008942 <email address hidden>[47694]: ERROR keystone File "/opt/stack/
Aug 22 01:06:34.678696 np0035008942 <email address hidden>[47694]: ERROR keystone response = req.get_
Aug 22 01:06:34.678696 np0035008942 <email address hidden>[47694]: ERROR keystone File "/opt/stack/
Aug 22 01:06:34.678696 np0035008942 <email address hidden>[47694]: ERROR keystone status, headers, app_iter = self.call_
Aug 22 01:06:34.681823 np0035008942 <email address hidden>[47694]: ERROR keystone File "/opt/stack/
Aug 22 01:06:34.681823 np0035008942 <email address hidden>[47694]: ERROR keystone app_iter = application(
Aug 22 01:06:34.681823 np0035008942 <email address hidden>[47694]: ERROR keystone File "/opt/stack/
Aug 22 01:06:34.681823 np0035008942 <email address hidden>[47694]: ERROR keystone return app(environ, start_response)
Aug 22 01:06:34.681823 np0035008942 <email address hidden>[47694]: ERROR keystone File "/opt/stack/
Aug 22 01:06:34.681823 np0035008942 <email address hidden>[47694]: ERROR keystone response = self.handle_
Aug 22 01:06:34.681823 np0035008942 <email address hidden>[47694]: ERROR keystone File "/opt/stack/
Aug 22 01:06:34.681823 np0035008942 <email address hidden>[47694]: ERROR keystone return original_handler(e)
Aug 22 01:06:34.681823 np0035008942 <email address hidden>[47694]: ERROR keystone File "/opt/stack/
Aug 22 01:06:34.681823 np0035008942 <email address hidden>[47694]: ERROR keystone return original_handler(e)
Aug 22 01:06:34.681823 np0035008942 <email address hidden>[47694]: ERROR keystone File "/opt/stack/
Aug 22 01:06:34.681823 np0035008942 <email address hidden>[47694]: ERROR keystone return original_handler(e)
Aug 22 01:06:34.681823 np0035008942 <email address hidden>[47694]: ERROR keystone [Previous line repeated 28 more times]
Aug 22 01:06:34.681823 np0035008942 <email address hidden>[47694]: ERROR keystone File "/opt/stack/
Aug 22 01:06:34.681823 np0035008942 <email address hidden>[47694]: ERROR keystone response = self.full_
Aug 22 01:06:34.681823 np0035008942 <email address hidden>[47694]: ERROR keystone File "/opt/stack/
Aug 22 01:06:34.681823 np0035008942 <email address hidden>[47694]: ERROR keystone rv = self.handle_
Aug 22 01:06:34.681823 np0035008942 <email address hidden>[47694]: ERROR keystone File "/opt/stack/
Aug 22 01:06:34.681823 np0035008942 <email address hidden>[47694]: ERROR keystone return original_handler(e)
Aug 22 01:06:34.681823 np0035008942 <email address hidden>[47694]: ERROR keystone File "/opt/stack/
Aug 22 01:06:34.685577 np0035008942 <email address hidden>[47694]: ERROR keystone return original_handler(e)
Aug 22 01:06:34.685577 np0035008942 <email address hidden>[47694]: ERROR keystone File "/opt/stack/
Aug 22 01:06:34.685577 np0035008942 <email address hidden>[47694]: ERROR keystone return original_handler(e)
Aug 22 01:06:34.685577 np0035008942 <email address hidden>[47694]: ERROR keystone [Previous line repeated 28 more times]
Aug 22 01:06:34.685577 np0035008942 <email address hidden>[47694]: ERROR keystone File "/opt/stack/
Aug 22 01:06:34.685577 np0035008942 <email address hidden>[47694]: ERROR keystone rv = self.dispatch_
Aug 22 01:06:34.685577 np0035008942 <email address hidden>[47694]: ERROR keystone File "/opt/stack/
Aug 22 01:06:34.685577 np0035008942 <email address hidden>[47694]: ERROR keystone return self.ensure_
Aug 22 01:06:34.685577 np0035008942 <email address hidden>[47694]: ERROR keystone File "/opt/stack/
Aug 22 01:06:34.685577 np0035008942 <email address hidden>[47694]: ERROR keystone resp = resource(*args, **kwargs)
Aug 22 01:06:34.685577 np0035008942 <email address hidden>[47694]: ERROR keystone File "/opt/stack/
Aug 22 01:06:34.685577 np0035008942 <email address hidden>[47694]: ERROR keystone return current_
Aug 22 01:06:34.685577 np0035008942 <email address hidden>[47694]: ERROR keystone File "/opt/stack/
Aug 22 01:06:34.685577 np0035008942 <email address hidden>[47694]: ERROR keystone resp = meth(*args, **kwargs)
Aug 22 01:06:34.685577 np0035008942 <email address hidden>[47694]: ERROR keystone File "/opt/stack/
Aug 22 01:06:34.685577 np0035008942 <email address hidden>[47694]: ERROR keystone ENFORCER.
Aug 22 01:06:34.685577 np0035008942 <email address hidden>[47694]: ERROR keystone File "/opt/stack/
Aug 22 01:06:34.685577 np0035008942 <email address hidden>[47694]: ERROR keystone enforcer_
Aug 22 01:06:34.685577 np0035008942 <email address hidden>[47694]: ERROR keystone File "/opt/stack/
Aug 22 01:06:34.685577 np0035008942 <email address hidden>[47694]: ERROR keystone self._check_
Aug 22 01:06:34.689279 np0035008942 <email address hidden>[47694]: ERROR keystone File "/opt/stack/
Aug 22 01:06:34.689279 np0035008942 <email address hidden>[47694]: ERROR keystone _emit_warning()
Aug 22 01:06:34.689279 np0035008942 <email address hidden>[47694]: ERROR keystone File "/opt/stack/
Aug 22 01:06:34.689279 np0035008942 <email address hidden>[47694]: ERROR keystone if not self._enforcer.
Aug 22 01:06:34.689279 np0035008942 <email address hidden>[47694]: ERROR keystone AttributeError: 'Enforcer' object has no attribute '_warning_emitted'
Aug 22 01:06:34.689279 np0035008942 <email address hidden>[47694]: ERROR keystone
Aug 22 01:06:34.689279 np0035008942 <email address hidden>[47694]: [pid: 47694|app: 0|req: 2/15] 10.209.128.207 () {68 vars in 1290 bytes} [Tue Aug 22 01:06:34 2023] POST /identity/
[1] https:/
[2] https:/
[3] https:/
[4] https:/