| 2022-12-05 14:16:32 |
Mustafa Kemal Gilor |
bug |
|
|
added bug |
| 2022-12-05 14:16:49 |
Mustafa Kemal Gilor |
summary |
PooledLDAPHandler.result3 does not release pool connection back when an exception raises |
PooledLDAPHandler.result3 does not release pool connection back when an exception is raised |
|
| 2022-12-05 14:16:57 |
Mustafa Kemal Gilor |
keystone: assignee |
|
Mustafa Kemal Gilor (mustafakemalgilor) |
|
| 2022-12-06 14:57:46 |
OpenStack Infra |
keystone: status |
New |
In Progress |
|
| 2023-02-23 21:39:46 |
OpenStack Infra |
keystone: status |
In Progress |
Fix Released |
|
| 2023-03-20 11:19:40 |
OpenStack Infra |
tags |
sts |
in-stable-zed sts |
|
| 2023-03-24 18:14:44 |
OpenStack Infra |
tags |
in-stable-zed sts |
in-stable-yoga in-stable-zed sts |
|
| 2023-05-19 16:13:34 |
OpenStack Infra |
tags |
in-stable-yoga in-stable-zed sts |
in-stable-xena in-stable-yoga in-stable-zed sts |
|
| 2023-09-12 08:59:43 |
Mustafa Kemal Gilor |
bug task added |
|
keystone (Ubuntu) |
|
| 2023-09-12 09:04:28 |
Mustafa Kemal Gilor |
nominated for series |
|
Ubuntu Jammy |
|
| 2023-09-12 09:04:28 |
Mustafa Kemal Gilor |
bug task added |
|
keystone (Ubuntu Jammy) |
|
| 2023-09-12 09:04:28 |
Mustafa Kemal Gilor |
nominated for series |
|
Ubuntu Focal |
|
| 2023-09-12 09:04:28 |
Mustafa Kemal Gilor |
bug task added |
|
keystone (Ubuntu Focal) |
|
| 2023-09-12 09:05:04 |
Mustafa Kemal Gilor |
bug task added |
|
cloud-archive |
|
| 2023-09-12 09:06:04 |
Mustafa Kemal Gilor |
nominated for series |
|
cloud-archive/victoria |
|
| 2023-09-12 09:06:04 |
Mustafa Kemal Gilor |
bug task added |
|
cloud-archive/victoria |
|
| 2023-09-12 09:06:04 |
Mustafa Kemal Gilor |
nominated for series |
|
cloud-archive/zed |
|
| 2023-09-12 09:06:04 |
Mustafa Kemal Gilor |
bug task added |
|
cloud-archive/zed |
|
| 2023-09-12 09:06:04 |
Mustafa Kemal Gilor |
nominated for series |
|
cloud-archive/ussuri |
|
| 2023-09-12 09:06:04 |
Mustafa Kemal Gilor |
bug task added |
|
cloud-archive/ussuri |
|
| 2023-09-12 09:06:04 |
Mustafa Kemal Gilor |
nominated for series |
|
cloud-archive/xena |
|
| 2023-09-12 09:06:04 |
Mustafa Kemal Gilor |
bug task added |
|
cloud-archive/xena |
|
| 2023-09-12 09:06:04 |
Mustafa Kemal Gilor |
nominated for series |
|
cloud-archive/wallaby |
|
| 2023-09-12 09:06:04 |
Mustafa Kemal Gilor |
bug task added |
|
cloud-archive/wallaby |
|
| 2023-09-12 09:06:04 |
Mustafa Kemal Gilor |
nominated for series |
|
cloud-archive/yoga |
|
| 2023-09-12 09:06:04 |
Mustafa Kemal Gilor |
bug task added |
|
cloud-archive/yoga |
|
| 2023-10-05 07:46:33 |
Mustafa Kemal Gilor |
summary |
PooledLDAPHandler.result3 does not release pool connection back when an exception is raised |
[SRU] PooledLDAPHandler.result3 does not release pool connection back when an exception is raised |
|
| 2023-10-05 07:46:50 |
Mustafa Kemal Gilor |
description |
This is a follow-up issue for LP#1896125.
This problem has happened when LDAP connection pooling is on (use_pool=True), page_size > 0 and pool_connection_timeout is < 'ldap server response time'. The scenario is as follows:
- An user tries to log in to a domain that is attached to LDAP backend.
- LDAP server does not respond in `pool_connection_timeout` seconds, causing LDAP connection to raise a ldap.TIMEOUT() exception
- From now on, all subsequent LDAP requests will fail with ldappool.MaxConnectionReachedError
An in-depth analysis explains why it happens:
- LDAP query initiated for user login request with BaseLdap._ldap_get() function call, which grabs a connection with self.get_connection() and invokes conn.search_s()
- conn.search_s() invokes conn._paged_search_s() since page_size is > 0
- conn._paged_search_s() calls conn.search_ext() (PooledLDAPHandler.search_ext) method
- conn.search_ext() initiates an asynchronous LDAP request and returns an AsynchronousMessage object to the _paged_search_s(), representing the request.
- conn._paged_search_s() tries to obtain asynchronous LDAP request results via calling conn.result3() (PooledLDAPHandler.result3)
- conn.result3() calls message.connection.result3()
- the server cannot respond in pool_connection_timeout seconds,
- message.connection.result3() raises a ldap.TIMEOUT(), causes subsequent connection release function, message.clean() to be not called
- the connection is kept active forever, subsequent requests cannot use it anymore
Reproducer:
- Deploy an LDAP server of your choice
- Fill it with many data so the search takes more than `pool_connection_timeout` seconds
- Define a keystone domain with the LDAP driver with following options:
[ldap]
use_pool = True
page_size = 100
pool_connection_timeout = 3
pool_retry_max = 3
pool_size = 10
- Point the domain to the LDAP server
- Try to login to the OpenStack dashboard, or try to do anything that uses the LDAP user
- Observe the /var/log/apache2/keystone_error.log, it should contain ldap.TIMEOUT() stack traces followed by `ldappool.MaxConnectionReachedError` stack traces
Known workarounds:
- Disable LDAP pooling by setting use_pool=Flase
- Set page_size to 0 |
[Impact]
This SRU is a backport of https://review.opendev.org/c/openstack/keystone/+/866723 to the respective Ubuntu and UCA releases. The patch is merged to the all respective upstream branches (master & stable/[u,v,w,x,y,z]).
This SRU intends to fix a denial-of-service bug that happens when keystone uses pooled ldap connections. In pooled ldap connection mode, keystone borrows a connection from the pool, do the LDAP operation and release it back to the pool. But, if an exception or error happens while the LDAP connection is still borrowed, Keystone fails to release the connection back to the pool, hogging it forever. If this happens for all the pooled connections, the connection pool will be exhausted and Keystone will no longer be able to perform LDAP operations.
The fix corrects this behavior by allowing the connection to release back to the pool even if an exception/error happens during the LDAP operation.
[Test Case]
- Deploy an LDAP server of your choice
- Fill it with many data so the search takes more than `pool_connection_timeout` seconds
- Define a keystone domain with the LDAP driver with following options:
[ldap]
use_pool = True
page_size = 100
pool_connection_timeout = 3
pool_retry_max = 3
pool_size = 10
- Point the domain to the LDAP server
- Try to login to the OpenStack dashboard, or try to do anything that uses the LDAP user
- Observe the /var/log/apache2/keystone_error.log, it should contain ldap.TIMEOUT() stack traces followed by `ldappool.MaxConnectionReachedError` stack traces
To confirm the fix, repeat the scenario and observe that the "/var/log/apache2/keystone_error.log" does not contain `ldappool.MaxConnectionReachedError` stack traces and LDAP operation in motion is successful (e.g. OpenStack Dashboard login)
[Regression Potential]
The patch is quite trivial and should not affect any deployment in a negative way. The LDAP pool functionality can be disabled by setting "use_pool=False" in case of any regression. |
|
| 2023-10-05 07:49:00 |
Mustafa Kemal Gilor |
attachment added |
|
focal.debdiff https://bugs.launchpad.net/keystone/+bug/1998789/+attachment/5706973/+files/focal.debdiff |
|
| 2023-10-05 07:49:17 |
Mustafa Kemal Gilor |
attachment added |
|
jammy.debdiff https://bugs.launchpad.net/keystone/+bug/1998789/+attachment/5706974/+files/jammy.debdiff |
|
| 2023-10-05 07:51:00 |
Mustafa Kemal Gilor |
attachment added |
|
uca_bionic_ussuri.debdiff https://bugs.launchpad.net/keystone/+bug/1998789/+attachment/5706975/+files/uca_bionic_ussuri.debdiff |
|
| 2023-10-05 07:52:05 |
Mustafa Kemal Gilor |
attachment added |
|
uca_focal_victoria.debdiff https://bugs.launchpad.net/keystone/+bug/1998789/+attachment/5706976/+files/uca_focal_victoria.debdiff |
|
| 2023-10-05 07:52:26 |
Mustafa Kemal Gilor |
attachment added |
|
uca_focal_wallaby.debdiff https://bugs.launchpad.net/keystone/+bug/1998789/+attachment/5706977/+files/uca_focal_wallaby.debdiff |
|
| 2023-10-05 07:52:45 |
Mustafa Kemal Gilor |
attachment added |
|
uca_focal_xena.debdiff https://bugs.launchpad.net/keystone/+bug/1998789/+attachment/5706978/+files/uca_focal_xena.debdiff |
|
| 2023-10-05 07:53:04 |
Mustafa Kemal Gilor |
attachment added |
|
uca_jammy_yoga.debdiff https://bugs.launchpad.net/keystone/+bug/1998789/+attachment/5706979/+files/uca_jammy_yoga.debdiff |
|
| 2023-10-05 07:53:19 |
Mustafa Kemal Gilor |
attachment added |
|
uca_jammy_zed.debdiff https://bugs.launchpad.net/keystone/+bug/1998789/+attachment/5706980/+files/uca_jammy_zed.debdiff |
|
| 2023-10-05 08:19:52 |
Ubuntu Foundations Team Bug Bot |
tags |
in-stable-xena in-stable-yoga in-stable-zed sts |
in-stable-xena in-stable-yoga in-stable-zed patch sts |
|
| 2023-10-05 08:19:57 |
Ubuntu Foundations Team Bug Bot |
bug |
|
|
added subscriber Ubuntu Sponsors |
| 2023-10-09 06:24:17 |
OpenStack Infra |
cloud-archive/yoga: status |
New |
Fix Released |
|
| 2023-10-09 06:24:52 |
OpenStack Infra |
cloud-archive/zed: status |
New |
Fix Released |
|
| 2023-10-16 13:36:51 |
Edward Hope-Morley |
cloud-archive/yoga: status |
Fix Released |
New |
|
| 2023-10-16 13:38:21 |
Edward Hope-Morley |
cloud-archive/zed: status |
Fix Released |
New |
|
| 2023-10-16 13:39:27 |
Edward Hope-Morley |
nominated for series |
|
cloud-archive/antelope |
|
| 2023-10-16 13:39:27 |
Edward Hope-Morley |
bug task added |
|
cloud-archive/antelope |
|
| 2023-10-16 13:39:35 |
Edward Hope-Morley |
cloud-archive/antelope: status |
New |
Fix Released |
|
| 2023-10-16 13:40:23 |
Edward Hope-Morley |
nominated for series |
|
Ubuntu Lunar |
|
| 2023-10-16 13:40:23 |
Edward Hope-Morley |
bug task added |
|
keystone (Ubuntu Lunar) |
|
| 2023-10-16 13:40:34 |
Edward Hope-Morley |
keystone (Ubuntu Lunar): status |
New |
Fix Released |
|
| 2023-10-16 14:02:45 |
Corey Bryant |
keystone (Ubuntu Lunar): status |
Fix Released |
Fix Committed |
|
| 2023-10-16 14:03:01 |
Corey Bryant |
cloud-archive/antelope: status |
Fix Released |
Fix Committed |
|
| 2023-10-16 14:03:19 |
Corey Bryant |
keystone (Ubuntu Jammy): status |
New |
Fix Committed |
|
| 2023-10-16 14:03:26 |
Corey Bryant |
cloud-archive/victoria: status |
New |
Fix Committed |
|
| 2023-10-16 14:03:35 |
Corey Bryant |
cloud-archive/wallaby: status |
New |
Fix Committed |
|
| 2023-10-16 14:03:45 |
Corey Bryant |
cloud-archive/xena: status |
New |
Fix Committed |
|
| 2023-10-16 14:03:53 |
Corey Bryant |
cloud-archive/yoga: status |
New |
Fix Committed |
|
| 2023-10-16 14:04:01 |
Corey Bryant |
cloud-archive/zed: status |
New |
Fix Committed |
|
| 2023-10-16 14:04:16 |
Corey Bryant |
cloud-archive/ussuri: status |
New |
Triaged |
|
| 2023-10-16 14:04:53 |
Corey Bryant |
keystone (Ubuntu): status |
New |
Fix Released |
|
| 2023-10-16 14:05:03 |
Corey Bryant |
cloud-archive: status |
New |
Fix Released |
|
| 2023-10-16 14:05:20 |
Corey Bryant |
keystone (Ubuntu Focal): status |
New |
Triaged |
|
| 2023-10-16 14:23:18 |
Corey Bryant |
bug |
|
|
added subscriber Ubuntu Stable Release Updates Team |
| 2023-10-16 15:46:41 |
Ubuntu Archive Robot |
bug |
|
|
added subscriber Corey Bryant |
| 2023-10-17 13:56:16 |
Lukas Märdian |
removed subscriber Ubuntu Sponsors |
|
|
|
