OpenStack Compute (nova)

FileNotFoundError: [Errno 2] No such file or directory: '/etc/nova/policy.json' when policy file removed

Bug #1856119 reported by melanie witt
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
OpenStack Compute (nova)
In Progress
Low
Akhil Gudise

Bug Description

Reported in the #openstack-nova channel today, that if a policy.json file is added without restarting the nova-api service, things work fine, but if the file is removed without restarting the service, the following error is raised:

2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack [req-46b41bde-4689-438f-805b-23b0a500066e 1ea06b07c73149ca9c6753e07c30383a 2962d44b73db4e1d884498b8ce000a69 - 5080f063d9f84290a8233e16a0ff39a2 5080f063d9f84290a8233e16a0ff39a2] Caught error: [Errno 2] No such file or directory: '/etc/nova/policy.json': FileNotFoundError: [Errno 2] No such file or directory: '/etc/nova/policy.json'
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack Traceback (most recent call last):
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack File "/usr/lib/python3/dist-packages/nova/api/openstack/__init__.py", line 95, in __call__
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack return req.get_response(self.application)
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack File "/usr/lib/python3/dist-packages/webob/request.py", line 1314, in send
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack application, catch_exc_info=False)
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack File "/usr/lib/python3/dist-packages/webob/request.py", line 1278, in call_application
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack app_iter = application(self.environ, start_response)
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack File "/usr/lib/python3/dist-packages/webob/dec.py", line 143, in __call__
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack return resp(environ, start_response)
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack File "/usr/lib/python3/dist-packages/webob/dec.py", line 129, in __call__
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack resp = self.call_func(req, *args, **kw)
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack File "/usr/lib/python3/dist-packages/webob/dec.py", line 193, in call_func
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack return self.func(req, *args, **kwargs)
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack File "/usr/lib/python3/dist-packages/keystonemiddleware/auth_token/__init__.py", line 341, in __call__
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack response = req.get_response(self._app)
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack File "/usr/lib/python3/dist-packages/webob/request.py", line 1314, in send
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack application, catch_exc_info=False)
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack File "/usr/lib/python3/dist-packages/webob/request.py", line 1278, in call_application
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack app_iter = application(self.environ, start_response)
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack File "/usr/lib/python3/dist-packages/webob/dec.py", line 129, in __call__
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack resp = self.call_func(req, *args, **kw)
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack File "/usr/lib/python3/dist-packages/webob/dec.py", line 193, in call_func
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack return self.func(req, *args, **kwargs)
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack File "/usr/lib/python3/dist-packages/nova/api/auth.py", line 99, in __call__
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack service_catalog=service_catalog)
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack File "/usr/lib/python3/dist-packages/debtcollector/renames.py", line 43, in decorator
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack return wrapped(*args, **kwargs)
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack File "/usr/lib/python3/dist-packages/debtcollector/renames.py", line 43, in decorator
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack return wrapped(*args, **kwargs)
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack File "/usr/lib/python3/dist-packages/debtcollector/renames.py", line 43, in decorator
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack return wrapped(*args, **kwargs)
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack [Previous line repeated 2 more times]
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack File "/usr/lib/python3/dist-packages/oslo_context/context.py", line 466, in from_environ
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack return cls(**kwargs)
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack File "/usr/lib/python3/dist-packages/nova/context.py", line 141, in __init__
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack self.is_admin = policy.check_is_admin(self)
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack File "/usr/lib/python3/dist-packages/nova/policy.py", line 192, in check_is_admin
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack return _ENFORCER.authorize('context_is_admin', target, credentials)
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack File "/usr/lib/python3/dist-packages/oslo_policy/policy.py", line 1043, in authorize
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack *args, **kwargs)
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack File "/usr/lib/python3/dist-packages/oslo_policy/policy.py", line 855, in enforce
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack self.load_rules()
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack File "/usr/lib/python3/dist-packages/oslo_policy/policy.py", line 568, in load_rules
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack overwrite=self.overwrite)
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack File "/usr/lib/python3/dist-packages/oslo_policy/policy.py", line 797, in _load_policy_file
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack self._file_cache, path, force_reload=force_reload)
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack File "/usr/lib/python3/dist-packages/oslo_policy/_cache_handler.py", line 36, in read_cached_file
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack mtime = os.path.getmtime(filename)
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack File "/usr/lib/python3.6/genericpath.py", line 55, in getmtime
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack return os.stat(filename).st_mtime
2019-12-12 03:08:52.634 72789 ERROR nova.api.openstack FileNotFoundError: [Errno 2] No such file or directory: '/etc/nova/policy.json'

Happening due to some file caching done by oslo.policy. Looks like we need to handle that somehow in nova.

Revision history for this message
melanie witt (melwitt) wrote :

Reproduce steps:

1. Add a policy.json override while nova-api is running and verify the override works by calling the relevant REST API
2. Remove the policy.json file and try the same REST API request again and get the "[Errno 2] No such file or directory: '/etc/nova/policy.json'" error

Expected Result:

No error when calling the REST API

Actual Result:

[Errno 2] No such file or directory: '/etc/nova/policy.json'

Revision history for this message
Matt Riedemann (mriedem) wrote :

Should probably move all calls to _ENFORCER.authorize to a separate _authorize method or something and handle FileNotFoundError and if we hit that, null out _ENFORCER, run through init() and then retry the auth check.

tags: added: policy
Matt Riedemann (mriedem)
Changed in nova:
status: New → Confirmed
Akhil Gudise (akhil-g)
Changed in nova:
assignee: nobody → Akhil Gudise (akhil-g)
Revision history for this message
melanie witt (melwitt) wrote :

> Should probably move all calls to _ENFORCER.authorize to a separate _authorize method or something and handle FileNotFoundError and if we hit that, null out _ENFORCER, run through init() and then retry the auth check.

Here's some more detail about what's involved in this suggested approach:

1. Create a new helper method (for example, name it _authorize) and use it to call _ENFORCER.authorize.
2. Inside the new _authorize method, try-except around _ENFORCER.authorize and catch FileNotFoundError. If caught, call reset() and then call init(), then call _ENFORCER.authorize again and return the result.
3. Find and replace all _ENFORCER.authorize calls with calls of the new _authorize method.
4. Add unit tests to validate the new code. The test of the new code should fail without the bug fix and pass with the bug fix.

Hope this helps.

https://github.com/openstack/nova/blob/db916926e541fe5d2ddf5d04654dd7a7caabba42/nova/policy.py#L178

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.opendev.org/739460

Changed in nova:
status: Confirmed → In Progress
melanie witt (melwitt)
Changed in nova:
importance: Undecided → Low
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on nova (master)

Change abandoned by Akhil Gudise (<email address hidden>) on branch: master
Review: https://review.opendev.org/739460

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to nova (master)

Fix proposed to branch: master
Review: https://review.opendev.org/749080

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.