While collecting Guru Meditation report (GMR) using "kill -USR2 <pid>" command on a system configured with SELinux in 'enforcing' mode we get the below error stack trace
nova-api: Traceback (most recent call last):
nova-api: File "/usr/lib/python2.7/site-packages/oslo_reports/guru_meditation_report.py", line 211, in handle_signa l
nova-api: res = cls(version, frame).run()
nova-api: File "/usr/lib/python2.7/site-packages/oslo_reports/guru_meditation_report.py", line 259, in run
nova-api: return super(GuruMeditation, self).run()
nova-api: File "/usr/lib/python2.7/site-packages/oslo_reports/report.py", line 77, in run
nova-api: return "\n".join(six.text_type(sect) for sect in self.sections)
nova-api: File "/usr/lib/python2.7/site-packages/oslo_reports/report.py", line 77, in <genexpr>
nova-api: return "\n".join(six.text_type(sect) for sect in self.sections)
nova-api: File "/usr/lib/python2.7/site-packages/oslo_reports/report.py", line 102, in __str__
nova-api: return self.view(self.generator())
nova-api: File "/usr/lib/python2.7/site-packages/oslo_reports/report.py", line 131, in newgen
nova-api: res = gen()
nova-api: File "/usr/lib/python2.7/site-packages/oslo_reports/generators/process.py", line 38, in __call__
nova-api: return pm.ProcessModel(psutil.Process(os.getpid()))
nova-api: File "/usr/lib/python2.7/site-packages/oslo_reports/models/process.py", line 66, in __init__
nova-api: children = process.children()
nova-api: File "/usr/lib64/python2.7/site-packages/psutil/__init__.py", line 336, in wrapper
nova-api: return fun(self, *args, **kwargs)
nova-api: File "/usr/lib64/python2.7/site-packages/psutil/__init__.py", line 913, in children
nova-api: **if p.ppid() == self.pid:**
nova-api: File "/usr/lib64/python2.7/site-packages/psutil/_common.py", line 293, in wrapper
nova-api: return fun(self)
nova-api: File "/usr/lib64/python2.7/site-packages/psutil/__init__.py", line 622, in ppid
nova-api: return self._proc.ppid()
nova-api: File "/usr/lib64/python2.7/site-packages/psutil/_pslinux.py", line 1092, in wrapper
nova-api: **raise AccessDenied(self.pid, self._name)**
nova-api: **AccessDenied: psutil.AccessDenied (pid=1)**
nova-api: **Unable to run Guru Meditation Report!**
This happens because the nova-api process is trying to locate a specific dir (pid number) in /proc & it compares every dir with the dir it wants to find. In this process, it encounters one directory for which it is not having access/search privilege according to SELinux rules and as psutils didnot catch this "AccessDenied" exception, the whole thing fails. Ideally, we would want to catch this exception and proceed with the search. For this I had raised a defect on psutils https://github.com/giampaolo/psutil/issues/1246. Giampolo has a valid point in the discussion we had there. I think this exception must be handled in oslo reports when its using psutils to collect GMR.
There have been other defects raised around this issue as well like https://bugzilla.redhat.com/show_bug.cgi?id=1292787, but they haven't come to conclusion.
Adding a fix in psutil might not be the right fix for this as psutil library should throw an error whenever it encounters one rather than curb it. The error could be because of different reasons. Fixing this at nova service level also might not be the right place as there are other Openstack services that uses oslo-reports and would run into the same problem. Considering this oslo-reports might be a suitable layer to provide the fix.