OpenStack Identity (keystone)

[ldappool] Downing one of the configured LDAP servers causes a persistent failure

Bug #1953627 reported by Grzegorz Grasza
16
This bug affects 3 people
Affects Status Importance Assigned to Milestone
OpenStack Identity (keystone)
In Progress
Undecided
Unassigned

Bug Description

If a server disconnects after a pooled connection is created, it fails withe an ldap.TIMEOUT which is Unhandled.

The ReconnectLDAPObject used by ldappool only catches the ldap.SERVER_DOWN exception in _apply_method_s which is applied to synchronous methods, whereas ldap.TIMEOUT is properly caught only during the initial connection.

To test this I did the following:

1. Create a new interface with a new IP address for the ldap server

2. set this as the first server in the url list and set pool_connection_timeout in the domain [ldap] configuration

3. Do a couple of:

  time openstack user list --domain=Users

to fill the connections in the pools in all WSGI processes

4. remove the created IP address from the interface

5. try an openstack user list again

Result:

the pooled connection consistently fails with:

gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: CRITICAL keystone [None req-7ad1997a-e23e-436b-a9ad-72f68a52abf8 demo admin] Unhandled error: ldap.TIMEOUT
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone Traceback (most recent call last):
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 2091, in __call__
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone return self.wsgi_app(environ, start_response)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/werkzeug/middleware/proxy_fix.py", line 187, in __call__
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone return self.app(environ, start_response)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/webob/dec.py", line 129, in __call__
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone resp = self.call_func(req, *args, **kw)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/webob/dec.py", line 193, in call_func
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone return self.func(req, *args, **kwargs)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/oslo_middleware/base.py", line 124, in __call__
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone response = req.get_response(self.application)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/webob/request.py", line 1313, in send
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone status, headers, app_iter = self.call_application(
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/webob/request.py", line 1278, in call_application
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone app_iter = application(self.environ, start_response)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/webob/dec.py", line 143, in __call__
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone return resp(environ, start_response)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/webob/dec.py", line 129, in __call__
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone resp = self.call_func(req, *args, **kw)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/webob/dec.py", line 193, in call_func
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone return self.func(req, *args, **kwargs)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/oslo_middleware/base.py", line 124, in __call__
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone response = req.get_response(self.application)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/webob/request.py", line 1313, in send
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone status, headers, app_iter = self.call_application(
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/webob/request.py", line 1278, in call_application
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone app_iter = application(self.environ, start_response)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/webob/dec.py", line 129, in __call__
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone resp = self.call_func(req, *args, **kw)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/webob/dec.py", line 193, in call_func
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone return self.func(req, *args, **kwargs)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/osprofiler/web.py", line 112, in __call__
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone return request.get_response(self.application)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/webob/request.py", line 1313, in send
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone status, headers, app_iter = self.call_application(
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/webob/request.py", line 1278, in call_application
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone app_iter = application(self.environ, start_response)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/webob/dec.py", line 129, in __call__
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone resp = self.call_func(req, *args, **kw)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/webob/dec.py", line 193, in call_func
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone return self.func(req, *args, **kwargs)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/oslo_middleware/request_id.py", line 58, in __call__
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone response = req.get_response(self.application)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/webob/request.py", line 1313, in send
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone status, headers, app_iter = self.call_application(
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/webob/request.py", line 1278, in call_application gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone app_iter = application(self.environ, start_response)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/opt/stack/keystone/keystone/server/flask/request_processing/middleware/url_normalize.py",[50/1978]
 in __call__
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone return self.app(environ, start_response)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/webob/dec.py", line 129, in __call__
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone resp = self.call_func(req, *args, **kw)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/webob/dec.py", line 193, in call_func
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone return self.func(req, *args, **kwargs)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/keystonemiddleware/auth_token/__init__.py", line 341, in __c
all__
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone response = req.get_response(self._app)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/webob/request.py", line 1313, in send
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone status, headers, app_iter = self.call_application(
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/webob/request.py", line 1278, in call_application
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone app_iter = application(self.environ, start_response)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/werkzeug/middleware/dispatcher.py", line 78, in __call__
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone return app(environ, start_response)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 2076, in wsgi_app
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone response = self.handle_exception(e)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/flask_restful/__init__.py", line 271, in error_router
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone return original_handler(e)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/flask_restful/__init__.py", line 271, in error_router
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone return original_handler(e)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/flask_restful/__init__.py", line 271, in error_router
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone return original_handler(e)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone [Previous line repeated 27 more times]
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 2073, in wsgi_app
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone response = self.full_dispatch_request()
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1518, in full_dispatch_request
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone rv = self.handle_user_exception(e)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/flask_restful/__init__.py", line 271, in error_router
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone return original_handler(e)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/flask_restful/__init__.py", line 271, in error_router
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone return original_handler(e)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/flask_restful/__init__.py", line 271, in error_router
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone return original_handler(e)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone [Previous line repeated 27 more times]
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1516, in full_dispatch_request
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone rv = self.dispatch_request()
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1502, in dispatch_request
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/flask_restful/__init__.py", line 467, in wrapper
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone resp = resource(*args, **kwargs)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/flask/views.py", line 84, in view
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone resp = meth(*args, **kwargs)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/opt/stack/keystone/keystone/api/users.py", line 183, in get
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone return self._list_users()
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/opt/stack/keystone/keystone/api/users.py", line 214, in _list_users
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone refs = PROVIDERS.identity_api.list_users(
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone __ret_val = __f(*args, **kwargs)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone return f(self, *args, **kwargs)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone return f(self, *args, **kwargs)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/opt/stack/keystone/keystone/identity/core.py", line 1108, in list_users
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone ref_list = self._handle_shadow_and_local_users(driver, hints)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/opt/stack/keystone/keystone/identity/core.py", line 1091, in _handle_shadow_and_local_users
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone return driver.list_users(hints) + fed_res
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/opt/stack/keystone/keystone/identity/backends/ldap/core.py", line 85, in list_users
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone return self.user.get_all_filtered(hints)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/opt/stack/keystone/keystone/identity/backends/ldap/core.py", line 328, in get_all_filtered
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone for user in self.get_all(query, hints)]
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/opt/stack/keystone/keystone/identity/backends/ldap/core.py", line 319, in get_all
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone objs = super(UserApi, self).get_all(ldap_filter=ldap_filter,
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/opt/stack/keystone/keystone/identity/backends/ldap/common.py", line 1934, in get_all
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone return super(EnabledEmuMixIn, self).get_all(ldap_filter, hints)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/opt/stack/keystone/keystone/identity/backends/ldap/common.py", line 1622, in get_all
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone for x in self._ldap_get_all(hints, ldap_filter)]
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/opt/stack/keystone/keystone/common/driver_hints.py", line 42, in wrapper
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone return f(self, hints, *args, **kwargs)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/opt/stack/keystone/keystone/identity/backends/ldap/common.py", line 1572, in _ldap_get_all
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone res = conn.search_s(self.tree_dn,
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/opt/stack/keystone/keystone/identity/backends/ldap/common.py", line 985, in search_s
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone ldap_result = self.conn.search_s(base, scope, filterstr,
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/opt/stack/keystone/keystone/identity/backends/ldap/common.py", line 679, in wrapper
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone return func(self, conn, *args, **kwargs)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/opt/stack/keystone/keystone/identity/backends/ldap/common.py", line 813, in search_s
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone return conn.search_s(base, scope, filterstr, attrlist,
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/ldap/ldapobject.py", line 631, in search_s
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone return self.search_ext_s(base,scope,filterstr,attrlist,attrsonly,None,None,timeout=self.timeout)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/ldap/ldapobject.py", line 1030, in search_ext_s
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone return self._apply_method_s(SimpleLDAPObject.search_ext_s,*args,**kwargs)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/ldap/ldapobject.py", line 968, in _apply_method_s
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone return func(self,*args,**kwargs)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/ldap/ldapobject.py", line 625, in search_ext_s
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone return self.result(msgid,all=1,timeout=timeout)[1]
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/ldap/ldapobject.py", line 535, in result
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone resp_type, resp_data, resp_msgid = self.result2(msgid,all,timeout)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/ldap/ldapobject.py", line 539, in result2
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone resp_type, resp_data, resp_msgid, resp_ctrls = self.result3(msgid,all,timeout)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/ldap/ldapobject.py", line 543, in result3
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone resp_type, resp_data, resp_msgid, decoded_resp_ctrls, retoid, retval = self.result4(
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/ldap/ldapobject.py", line 553, in result4
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone ldap_result = self._ldap_call(self._l.result4,msgid,all,timeout,add_ctrls,add_intermediates,add_extop)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone File "/usr/local/lib/python3.9/dist-packages/ldap/ldapobject.py", line 128, in _ldap_call
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone result = func(*args,**kwargs)
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone ldap.TIMEOUT
gru 08 14:21:01 ggrasza-ubuntu2 <email address hidden>[1270797]: ERROR keystone

Tags: ldap
Revision history for this message
Grzegorz Grasza (xek) wrote :

Also added an issue to python-ldap https://github.com/python-ldap/python-ldap/issues/446

Revision history for this message
Sven Kieske (s-kieske) wrote :

Keystone should imho fail gracefully and retry indefinitely if one backend goes down for whatever reason.

Most backend failures are temporary, e.g. during maintenance windows.

Revision history for this message
Sven Kieske (s-kieske) wrote :

if I read this correctly:

> There is another issue, which is triggered, when re-using a connection after a ldap.SERVER_DOWN connection. Then the reconnection does not apply as well[..]

https://github.com/python-ldap/python-ldap/pull/267#issuecomment-520436367

a workaround for keystone might be, to use a new connection and not reuse an existing connection, when encountering ldap.SERVER_DOWN.

I don't know if it is feasible though, as I'm not familiar with the keystone codebase.

Revision history for this message
Sven Kieske (s-kieske) wrote :
Download full text (10.5 KiB)

Could at least the logic get changed how keystone uses python-ldap?

Usecase:

I have 4 LDAP Servers configured

3 are down for maintenance.

I get alerts with stacktraces like:

2022-08-24 04:06:44.860 7 ERROR keystonemiddleware.auth_token [req-b4703136-ac14-44d5-bc85-665d852cb41e - c512d8855ab44fd7af3d2ad10583d6e4 - 1831760463a0498e8b0b006eff434cf8 1831760463a0498e8b0b006eff434cf8] Bad response code while vali
dating token: 504 Gateway Timeout (HTTP 504): keystoneauth1.exceptions.http.GatewayTimeout: Gateway Timeout (HTTP 504)
2022-08-24 04:06:44.860 7 WARNING keystonemiddleware.auth_token [req-b4703136-ac14-44d5-bc85-665d852cb41e - c512d8855ab44fd7af3d2ad10583d6e4 - 1831760463a0498e8b0b006eff434cf8 1831760463a0498e8b0b006eff434cf8] Identity response: <html><
body><h1>504 Gateway Time-out</h1>
The server didn't respond in time.
</body></html>
: keystoneauth1.exceptions.http.GatewayTimeout: Gateway Timeout (HTTP 504)
2022-08-24 04:06:44.860 7 CRITICAL keystonemiddleware.auth_token [req-b4703136-ac14-44d5-bc85-665d852cb41e - c512d8855ab44fd7af3d2ad10583d6e4 - 1831760463a0498e8b0b006eff434cf8 1831760463a0498e8b0b006eff434cf8] Unable to validate token:
 Failed to fetch token data from identity server: keystonemiddleware.auth_token._exceptions.ServiceError: Failed to fetch token data from identity server

in octavia.

I get this in keystone.log:

2022-08-24 04:07:02.711 31 ERROR ldappool Traceback (most recent call last):
2022-08-24 04:07:02.711 31 ERROR ldappool File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 1196, in _apply_method_s
2022-08-24 04:07:02.711 31 ERROR ldappool return func(self,*args,**kwargs)
2022-08-24 04:07:02.711 31 ERROR ldappool File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 443, in simple_bind_s
2022-08-24 04:07:02.711 31 ERROR ldappool msgid = self.simple_bind(who,cred,serverctrls,clientctrls)
2022-08-24 04:07:02.711 31 ERROR ldappool File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 437, in simple_bind
2022-08-24 04:07:02.711 31 ERROR ldappool return self._ldap_call(self._l.simple_bind,who,cred,RequestControlTuples(serverctrls),RequestControlTuples(clientctrls))
2022-08-24 04:07:02.711 31 ERROR ldappool File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 329, in _ldap_call
2022-08-24 04:07:02.711 31 ERROR ldappool reraise(exc_type, exc_value, exc_traceback)
2022-08-24 04:07:02.711 31 ERROR ldappool File "/usr/lib/python3/dist-packages/ldap/compat.py", line 44, in reraise
2022-08-24 04:07:02.711 31 ERROR ldappool raise exc_value
2022-08-24 04:07:02.711 31 ERROR ldappool File "/usr/lib/python3/dist-packages/ldap/ldapobject.py", line 313, in _ldap_call
2022-08-24 04:07:02.711 31 ERROR ldappool result = func(*args,**kwargs)
2022-08-24 04:07:02.711 31 ERROR ldappool ldap.SERVER_DOWN: {'desc': "Can't contact LDAP server", 'errno': 107, 'info': 'Transport endpoint is not connected'}
2022-08-24 04:07:02.711 31 ERROR ldappool
2022-08-24 04:07:02.711 31 ERROR ldappool During handling of the above exception, another exception occurred:
2022-08-24 04:07:02.711 31 ERROR ldappool
2022-08-24 04:07:02.711 31 ERROR ldappool Traceback (most recent...

Revision history for this message
Grzegorz Grasza (xek) wrote :

Looks like this change is related:
https://review.opendev.org/c/openstack/keystone/+/860118

OpenStack Infra (hudson-openstack)
Changed in keystone:
status: New → In Progress
Revision history for this message
Sven Kieske (s-kieske) wrote :

so I think I fixed the related change, hopefully someone can look at it soon. I can't remove the "WIP" label though, as I'm not the change owner.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.