ldap_install_tls occasionally fails due to watchdog timeout when using ad_use_ldaps with tls

Apport file attached

Thank you for taking the time to report this bug and helping to make Ubuntu better. It seems that your bug report is not filed about a specific source package though, rather it is just filed against Ubuntu in general. It is important that bug reports be filed about source packages so that people interested in the package can find the bugs about it. You can find some hints about determining what package your bug might be about at https://wiki.ubuntu.com/Bugs/FindRightPackage. You might also ask for help in the #ubuntu-bugs irc channel on Freenode.

To change the source package that this bug is filed about visit https://bugs.launchpad.net/ubuntu/+bug/1921494/+editstatus and add the package name in the text box next to the word Package.

[This is an automated message. I apologize if it reached you inappropriately; please just reply to this message indicating so.]

Status changed to 'Confirmed' because the bug affects multiple users.

Hi Rex,

Thank you for taking the time to file a bug report.

Would you mind also providing the access logs for the server when the connection is attempted?

Moreover, could you confirm that you can also use ldapsearch with TLS; e.g., `ldapsearch -x -Z`?

Since there is not enough information in your report to begin triage or to
differentiate between a local configuration problem and a bug in Ubuntu, I
am marking this bug as "Incomplete". We would be grateful if you would:
provide a more complete description of the problem, explain why you
believe this is a bug in Ubuntu rather than a problem specific to your
system, and then change the bug status back to "New".

For local configuration issues, you can find assistance here:
http://www.ubuntu.com/support/community

Hi Rex,

Looking closer at the logging which you provided when debug_level = 4, the important part is:

[sss_ldap_init_sys_connect_done] (0x0020): ldap_install_tls failed: [Connect error] [(unknown error code)]

This looks very similar to this upstream bug report:
https://github.com/SSSD/sssd/issues/5531

In which case, I believe the below commit should fix the issue:

commit da55e3e69707de416b7949d08c165c950090bbb6
From: Iker Pedrosa <email address hidden>
Date: Wed, 3 Mar 2021 15:34:49 +0100
Subject: ldap: retry ldap_install_tls() when watchdog interruption
Link: https://github.com/SSSD/sssd/commit/da55e3e69707de416b7949d08c165c950090bbb6

I will make some test packages for Bionic, Focal, Hirsute and Impish. Will you be able to try some test packages? I will also try and reproduce myself, but this looks like a unreliable race condition between the watchdog and ldap_install_tls().

Thanks,
Matthew

Hi Athos and Matthew.
The original Ubuntu server I used for the testing back in February/March has been deleted, so I will need to create a new server and set it up to (hopefully) replicate the issue seen and reported in the bug report I filed.
Due to ongoing work commitments I will need to do this over the next few days.
Once the server is available, tested and the problem I see is reproduced, then it can certainly be used for any testing you desire.
I will also respond in full with regards to more info and the ldapsearch query result.

Regards,
Rex

Hello,
i am also affected from this bug and like to test the testbuilds
ldapsearch can conntect to the ldap server.

With my sssd config i can success connect from RHEL8 and 7 but have problems with different ubuntu and debian versions
if you need more information i can try to provide this to

my sssd ldap log

Hi Rex and Snakekick,

I have built test packages for Bionic, Focal, Hirsute and Impish, with the below commit included:

commit da55e3e69707de416b7949d08c165c950090bbb6
From: Iker Pedrosa <email address hidden>
Date: Wed, 3 Mar 2021 15:34:49 +0100
Subject: ldap: retry ldap_install_tls() when watchdog interruption
Link: https://github.com/SSSD/sssd/commit/da55e3e69707de416b7949d08c165c950090bbb6

Test packages can be found in the below ppa:

https://launchpad.net/~mruffell/+archive/ubuntu/lp1921494-test

Can you please test the new packages and report back and let me know if they solve your problem?

Please note, these test packages are NOT SUPPORTED by Canonical and are for TEST PURPOSES ONLY. ONLY install in a dedicated test environment.

Instructions to Install:

1) sudo add-apt-repository ppa:mruffell/lp1921494-test
2) sudo apt update
3) sudo apt upgrade # or just install the sssd packages, up to you, there may be many of them.
4) sudo apt-cache policy sssd | grep Installed
For Impish:
2.4.1-2ubuntu4+lp1921494v20211011b1
For Hirsute:
2.4.0-1ubuntu6.1+lp1921494v20211011b1
For Focal:
2.2.3-3ubuntu0.7+lp1921494v20211011b2
For Bionic:
1.16.1-1ubuntu1.8+lp1921494v20211011b1

If the test packages work, I will submit the patches for SRU and we can get this fixed in official packages.

Thanks,
Matthew

Hello Matthew Ruffell,
thank you for creating this patch, but for me this solve not the problem ;(
i try it with Ubuntu 20.04.3 LTS
sssd Version Installed: 2.2.3-3ubuntu0.7+lp1921494v20211011b2
the log look similary for me, ldap_install_tls failed: [Connect error] [unknown error]

i attached a log file with debug lvl 7, i hope you can see the problem

thank you

Hi Athos,
I have created two Ubuntu test servers (Ubu 18.04.6 and 20.04.3) and managed to reproduce the original issue I reported for this bug.
Am not sure access logs will provide anything due sssd fails to start due to the aforementioned startup errors "Could not start TLS encryption. (unknown error code)".
When trying an ldapsearch with TLS I get:
# ldapsearch -x -Z -v -H ldaps://ad-server-01:636 -b "dc=company,dc=com" "(sAMAccountName=myUserName)" cn
ldap_initialize( ldaps://ad-server-01:636/??base )
ldap_start_tls: Operations error (1)
        additional info: 00000000: LdapErr: DSID-0C09127F, comment: TLS or SSL already in effect, data 0, v3839
filter: (sAMAccountName=myUserName)
requesting: cn
# extended LDIF
#
# LDAPv3
# base <dc=helsemn,dc=no> with scope subtree
# filter: (sAMAccountName=myUserName)
# requesting: cn
#

# search result
search: 3
result: 1 Operations error
text: 000004DC: LdapErr: DSID-0C090A71, comment: In order to perform this opera
 tion a successful bind must be completed on the connection., data 0, v3839

# numResponses: 1

The status of this bug has been changed from Incomplete and is now "in progress", where there is another user (Snakekick) that is also experiencing this issue.

Hi Matthew,
Unfortunately I get the same error as originally reported when using the lp1921494-test patches for SSSD.
# apt-cache policy sssd | grep Installed
  Installed: 2.2.3-3ubuntu0.7+lp1921494v20211011b2

systemctl stop sssd
rm -rf /var/lib/sss/db/*
systemctl start sssd
systemctl status sssd - Could not start TLS encryption. (unknown error code)
Attached is also a washed debug level 4 sssd.log from the startup of sssd with the patchset applied.
Something I see is that my logs complains about connecting to port 389, whereas Snakekicks recent log tries connecting to port 636 "marking port 636 to server x as not working".

Hello there,

Matthew and I spent a non-trivial amount of time trying to reproduce this bug. Let me tell you what we did, what we found, and then maybe you can help us progress.

First of all, it is really important that we find a reproducer so that we can: (1) figure out what exactly is going on, (2) possibly find a patch or write a fix for the problem, and (3) drive the SRU process to its completion and get the fix release for all affected Ubuntu versions.

Having said that, here is what I did (and, to the best of my knowledge, what Matthew also tried):

1) I installed Windows Server 2019 in a VM. I configured Active Directory, DHCP and DNS. I also configured the Certificate Authority in it.

Bear in mind that this is a test environment, so the configuration I did was basic. For AD, I chose to "Add a new forest" using "Windows Server 2016" as the functional level, and with DNS and Global Catalog capabilities actived. For the Certificate Authority, I chose to generate a root certificate (I don't remember exactly the other options, but the root certificate is important).

2) I created an LXD container running Ubuntu Focal that shared the same network bridge as the Windows Server 2019 VM. Promptly, the container acquired an IP from the DHCP I had configured in the Windows VM.

3) I joined the AD realm using "realm join win-ad-example.adtest.local". sssd and other dependencies were automatically installed, and the process finished successfully.

4) I then went to the Windows VM, opened certsrv, found the certificate for the machine and exported it. I copied the certificate into the LXD container, put it into the right place (/usr/local/share/ca-certificates/) and ran update-ca-certificates. I noticed that the command added 1 certificate to the chain.

5) I edited /etc/sssd/sssd.conf and added the following options to the domain section:

ldap_tls_cacert = /etc/ssl/certs/ca-certificates.crt
ad_use_ldaps = True
debug_level = 4

6) I then restarted sssd. And I noticed the error manifesting! At that point, I thought I had reproduced the bug (keep reading, though). I started to investigate what could be happening.

7) After several minutes thinking I was on the right track, I decided to try and run the "ldapsearch" command provided above:

$ ldapsearch -x -Z -v -H ldaps://win-ad-example.adtest.local:636

And to my surprise, I noticed that the command could not connect to the Windows server. I then started debugging things, and quickly found that the problem was that the TLS certificate from the server could not be validated. Something was wrong...

8) I went back to the Windows VM and poked around certsrv. I noticed that I had exported the certificate for the machine, but not the root certificate. I decided to give it a try.

9) After having imported the root certificate into the LXD container, I restarted sssd and, much to my surprise, everything worked out of the box (using the sssd package from the archive). Back to square one...

As you can see, I have a working AD DC and I can successfully connect to it using a regular Focal container (I also tried with a Focal VM, with the same results). I spent a few more hours trying to tw...

Hi Rex, Snakekick,

Thanks for testing that previous test package, it was actually very interesting to hear that it didn't work, and it came as a real surprise.

I have been working with Sergio on reproducing this locally, and as you can see, we haven't been able to reproduce just yet.

Along with answering Sergio's questions, could you also please try the below test package? It is a build of the current upstream sssd master branch, at version 2.6.0 which was released a few days ago.

As I am writing this, the package is pending to be built, so please check that the package has built before you start installing anything.

https://launchpad.net/~mruffell/+archive/ubuntu/lp1921494-test-upstream

Please note, these test packages are NOT SUPPORTED by Canonical and are for TEST PURPOSES ONLY. ONLY install in a dedicated test environment.

Now, as this is actually 2.6.0 and not 2.2.3-ubuntu0.7, you may find some major differences, and your configuration files might need changing, please keep it in mind.

Instructions to Install (on Focal):
1) sudo add-apt-repository ppa:mruffell/lp1921494-test-upstream
2) sudo apt update
3) sudo apt upgrade # or just install the sssd packages, up to you, there may be many of them.
4) sudo apt-cache policy sssd | grep Installed
2.2.3-3ubuntu0.7+lp1921494v20211020b1.really.2.6.0

I just built it for Focal. Please don't install it on any production system, just a VM that you can snapshot and restore ideally.

Thanks,
Matthew

Hello Sergio, Hello Matthew

thanks for your help and the time you invest.
But my configuration is a bit different from the creator of this ticket.
We have only the same error message ;(
Sorry if that didn't come across clearly in the past.
I can't and didn't want to connect to this domain, we only use sssd with ldap as provider
I sent you my sssd.conf last week, but here is the relevant part

[sssd]
config_file_version = 2
domains = xxx
services = nss,pam,ssh
reconnection_retries = 3
#debug_level = 5

[pam]

[nss]
filter_users = bin,daemon,ftp,games,haldaemon,lp,mail,messagebus,nobody,ntp,polkituser,postfix,root,sshd,wwwrun,at,dergraf,abix,amboscl,sysnrpe,dnsmasq,hpsmh,ambosrtu,vmon,man,news,uucp
filter_groups = root,bin,daemon,sys,tty,disk,lp,www,kmem,wheel,mail,news,uuscp,shadow,dialout,audio,floppy,cdrom,console,utmp,public,video,games,xok,trusted,modem,ftp,man,users,nobody,nogroup,messagebus,haldaemon,sshd,tape,postfix,maildrop,polkituser,ntp,at,dergraf,abix,amboscl,sysnrpe,hpsmh,ambosrtu,vmon,winbind,ntadmin

[domain/xxx]
#debug_level = 7
id_provider = ldap
auth_provider = ldap
chpass_provider = ldap
access_provider = simple
ldap_uri = ldaps://xx.xx.de:636,
ldap_search_base = dc=xx,dc=xx,dc=xx
ldap_schema = ad
ldap_id_mapping = True
fallback_homedir = /home/%u@%d
default_shell = /bin/bash
ldap_idmap_range_min = 100000000
ldap_idmap_range_max = 2100000000
ldap_idmap_range_size = 2000000000
ldap_idmap_default_domain_sid = S-1-5-21-32142354-212345234-839522115
ldap_idmap_default_domain = xx.xx.de
enumerate = False
ignore_group_members = True
ldap_idmap_autorid_compat = True
ldap_default_bind_dn = xxx
ldap_default_authtok_type = obfuscated_password
ldap_default_authtok = xxx
use_fully_qualified_names = False
case_sensitive = false
ldap_tls_cacertdir = /etc/ssl/certs
#ldap_tls_cacert = /etc/ssl/certs/Domain-Root.crt
ldap_id_use_start_tls = True
ldap_tls_reqcert = demand
ldap_tls_cipher_suite = ECDHE-RSA-AES256-GCM-SHA384
simple_allow_groups = xxx
sudo_provider = ldap
autofs_provider = ldap
resolver_provider = ldap

I try differtent settings with ldap_tls_cacertdir or ldap_tls_cacer (only the domain root crt or the ca-certificates.crt but it ends with the same error the same with different ldap_tls_reqcert settings

I have attached a screenshot of the working ldapsearch
( ldapsearch -x -b "dc=xx,dc=xx,dc=xx" -H ldaps://xx.xx.xx:636 -D user@domain -W "objectclass=*" -d1

Hi Snakekick,

Sergio and I have continued to attempt to reproduce, and we have modeled our config based on yours, and we think we see the issue. ldapsearch works, but sssd does not.

If we comment out the line below in the sssd config, then sssd starts working.

ldap_tls_cipher_suite = ECDHE-RSA-AES256-GCM-SHA384

Can you comment this line out with a # and restart sssd.service?

We took a look at this list of ciphersuites that w2k19 is sending, and ECDHE-RSA-AES256-GCM-SHA384 is included in that list, but if we change ldap_tls_cipher_suite to something like HIGH:-SSLv2, tls handshake still fails. It only works when we comment out ldap_tls_cipher_suite entirely.

We have also seen the same happen with Fedora 34 and Fedora 35 running in LXC containers, and the same thing happens with the sssd 2.6.0 upstream package in my previous comment.

So please, comment out ldap_tls_cipher_suite and see if ti makes a difference.

Could you also see if you see the same under Fedora, and not just RHEL or CentOS?

Also, how are you installing your CA certificates on the client? Are you copying them to /usr/local/share/ca-certificates/ and running update-ca-certificates?

Thanks,
Matthew

Hi Matthew.
i try to disable ldap_tls_cipher_suite ...and it works ! fantastic.
I start now a new clean Focal installation and upload it to lab and test again it´s take a litte time.
But thank you, we are on the right way.

"Also, how are you installing your CA certificates on the client? Are you copying them to /usr/local/share/ca-certificates/ and running update-ca-certificates?"

Yes that is the way. sometimes i run dpkg-reconfigure ca-certificates because its feel thats is need to but i am unsure

after focal test i will test fedora 34 to

Hello Matthew Hi Sergio,
i have now tested Fedora 34 and it is working without delete ldap_tls_cipher_suite setting
the only thing i change in the sssd.conf is

ldap_tls_cacert = /etc/pki/tls/certs/ca-bundle.crt

i have tested a fresh installed focal and yes the cipher suite parameter is the problem (when disabled it´s work)

i attach the fedora log, and here i see the first time this: "is a secure channel. No need to run START_TLS" maybe this helps

sssd.x86_64 2.5.2-2.fc34 @updates
sssd-ad.x86_64 2.5.2-2.fc34 @updates
sssd-client.x86_64 2.5.2-2.fc34 @updates
sssd-common.x86_64 2.5.2-2.fc34 @updates
sssd-common-pac.x86_64 2.5.2-2.fc34 @updates
sssd-ipa.x86_64 2.5.2-2.fc34 @updates
sssd-kcm.x86_64 2.5.2-2.fc34 @updates
sssd-krb5.x86_64 2.5.2-2.fc34 @updates
sssd-krb5-common.x86_64 2.5.2-2.fc34 @updates
sssd-ldap.x86_64 2.5.2-2.fc34 @updates
sssd-nfs-idmap.x86_64 2.5.2-2.fc34 @updates
sssd-proxy.x86_64 2.5.2-2.fc34 @updates

Hi all. Apologies for delay in response, but workload has been getting in the way.

I appreciate the time used to reproduce this - I have also used a non-trivial amount of time on this issue also. Especially annoying and frustrating is that ad_use_ldaps just worked 'out of the box' for all of my RH7, RH8 and CentOS7 servers.
The SSL cert used on my Ubuntu 18.04 and 20.04 servers is the same as used by my RH/CentOS srvrs for ad_use_ldaps with port 636. The only difference is on my RH/CentOS servers the SSL cert is used 'standalone' and not merged, as I have done for Ubuntu, within the ca-certificates bundle. Although I have also tested with the SSL cert outside of the ca-certificate bundle without any joy on my Ubuntu 18.04 and 20.04 servers.

An initial install of Matthews2.2.3-3ubuntu0.7+lp1921494v20211020b1.really.2.6.0 packages threw up multiple issues immediately - so I am currently trawling through all of the previous release notes to figure out what has changed, so that sssd can start without complaining. But my time today on this issue was limited and it has now run out. So I will need to come back to testing with 2.2.3-3ubuntu0.7+lp1921494v20211020b1.really.2.6.0 and let you know the results.

Hello Rex,
for me the Solution was to change the ldap tls cipher parameter.
You can easaly check it when you disable ldap_tls_cipher_suite. After this the connection work for me.
After a while of search i found that Ubuntu or sssd changed the ssl tool or his paramerts

Not Openssl is used anymore (like redhat) , GNUtls is used (or his parameters)

When I use gnutls parameter i can connect to our domain
ldap_tls_cipher_suite = NORMAL
if you want it more specific you can use like this ( if its pass to your domain settings)
ldap_tls_cipher_suite = NONE:+VERS-TLS-ALL:+AES-256-GCM:+SIGN-ALL:+COMP-NUL

Hi all,
I've done a fresh test, using a new Ubuntu 20.04 server updated today.
Results were the same as I first reported back in March 2021. Although this time I am using the latest available SSSD (2.2.3-3ubuntu0.8) and adcli (0.9.0-1ubuntu0.20.04.1) packages.
When setting ad_use_ldaps I still got the unhelpful "Could not start TLS encryption. (unknown error code)" returned. I also tried Snakekicks suggestion for ldap_tls_cipher_suite, but unfortunately that did not resolve my issue.

I took this Ubu 20.04 server and did an in-place upgrade to 21.04. Surprisingly I could log onto the upgraded server with no changes to my Ubuntu 20.04 SSSD config files (and repeated with emptying the SSSD cache, restarting SSSD).
The latest available SSSD (2.4.0-1ubuntu6.1) and adcli (0.9.0-1ubuntu2) packages for 21.04 were used.
However when adding ad_use_ldaps to sssd.conf and restarting sssd.service I again got "Could not start TLS encryption. (unknown error code)" returned along with identical errors in the logs (debug_level = 4).

I think the problem is somewhere within my root cert-to-AD-server and/or the way Ubuntu is interpreting the response back from the AD server.
But as mentioned earlier, my RH servers using the same root cert with ad_use_ldaps do not have this issue.
An annoying issue...

The Hirsute Hippo has reached End of Life, so this bug will not be fixed for that release.

This was resolved for me and my particular situation. AD guys activated LDAPS without warning which left me scrambling.

For my situation, the problem was that my RootCA cert was SHA1. Testing via openssl doesn't fail due SHA1, but SSSD + Ubuntu stops when a SHA1 cert is used.

You can check your certificate with # openssl x509 -text -in /path/to/cert/RootCA.pem | grep Signature

The solution was to use a SHA2 cert. With a SHA256 root CA cert in place, ad_use_ldaps worked without error - or with those terribly uninformative error messages initially seen. This is confirmed as working on Ubu 18.04, 20.04 and 22.04 for me.
As an aside, LDAP-over-TLS also worked (without ad_use_ldaps set) once a SHA2 root cert was used.

Ubuntu 21.10 (Impish Indri) has reached end of life, so this bug will not be fixed for that specific release.