httpd leaks open files

We have also noticed this problem.
We did an strace on horizon and found that the connections made by horizon to keystone were not getting closed.

Digging deeper, we found 2 very similar commits that fix connection closure issues related to keystone.

https://github.com/openstack/python-keystoneclient/commit/8fcacdc7c74f5ac68e8e55ea8c15918c452411fe

and

https://github.com/openstack/keystoneauth/commit/dbcbf414ac8423e97d77d0bda8157be5350530f0

I think that the commit in keystoneauth is incomplete, it's missing the destination of the move of the _FakeRequestSession class and so it's just a removal.

In Newton, horizon is using keystoneclient to get keystone sessions and we don't see the connection leaks.
Somewhere between Newton and Queens, horizon switched to keystoneauth and lost the _FakeRequestSession hack and so we are now seeing a regression where horizon is leaking connections again.

I am attaching a patch that we are currently testing, it re-introduces the _FakeRequestSession class into keystoneauth.
The patch is a little naive, I don't know what to do with the instance attribute "adapters" of _FakeRequestSession, I just set it to an empty list because it is being referenced by code elsewhere. It seems to be working for now.

We should fix keystoneclient. KeystoneAuth is not doing anything wrong here. I am against a "temp hack" like this. Secondarily, please submit this patch to gerrit at review.openstack.org so that it can be considered. Posting patches here is unlikely to be seen as easily/readily.

Thanks for looking at this.

For the record, I also don't like this hack. However, horizon is using keystoneauth and not keystoneclient so I don't see how fixing keystoneclient will help here. I still think it's keystoneauth that needs to be fixed.

Also, I hesitate to submit the patch to gerrit as it makes the tests fail when I run tox.

@mchlumsky thanks for including the patch for session.py. We are able to use it in a packaged Rocky deploy and successfully overcome the file descriptor leak associated with the keyauth connections. This successfully resolved our too many open files error being caused by unbounded growth of open files due to lack of close() after connect()'s to keystone.

I am also seeing the same issue, that the /proc/<memcached pid>/fd/ count increasing with each Horizon login. I agree we need to fix keystoneauth.

I've seen this issue in multiple production Rocky environments.

I found a commit in keystoneauth that properly closes requests sessions (which closes the socket/file handle):

https://opendev.org/openstack/keystoneauth/commit/b2b5ad3cb1ff05e08c22973bb079125214ba7bcf

This commit is included in keystoneauth1==3.18.0. I have tested upgrading to 3.18.0 and initial outlook is good--I'm seeing apache workers open new sockets/file handles and then close them as expected.

Will report back once I have deployed 3.18.0 in more environments and it has been running for a while.