Authentication fails after upgrading sssd to 2.2.3-3ubuntu0.6

Thank you for your report and sorry for the issue caused by this SRU.

In focal GPO are enforced by default. But the original version of SSSD was not fully implementing the MS spec and GPO where actually not enforced. This was the case for the default domain policy which defines in particular the security and password policy which where not followed.

This is what this SRU addressed by backporting most the GPO fixes from recent version of SSSD and created this unfortunate side effect.

This problem will occur when users will upgrade from 20.04 to 22.04 anyway.

Adding 'ad_gpo_access_control = permissive' to sssd.conf is one way to fix it. Another way is to make sure the default domain policy is properly defined and that the client can access it.

However we will have a look at a Samba4 setup to try to understand if there is a general problem with this setup and how it can be addressed before 22.04 is released.

We found issues too and work around it by using 'access_provider = permit' instead of 'access_provider = ad'.

Hello,

I have the same issues on a fleet on Ubuntu 20.04 machines joined to a Univention Corporate Server domain. UCS uses Samba4 as a background.

Is there any security concern about the 'ad_gpo_access_control = permissive' or 'access_provider = permit' in sssd.conf?

Thank you for looking into this issue

I just got hit by this. While 'ad_gpo_access_control = permissive' lets users log in again, it is not ideal.

It appears that Samba AD does not create GptTmpl.inf except for policies which actually set something in "Security Settings". Hence a policy with nothing no security settings will not have the file, which will lead SSSD to get "file not found" when trying to download it and fail PAM auth.

I don't know if Samba is in the wrong here, but anyway treating GptTmpl.inf not found as "no security settings" seems like a reasonable approach.

Anyway, a deb-changelog warning for this kind of change in a minor-minor release would have been most welcome.

OK, after spending a few hours investigating this bug I think I can write a summary of my findings now.

First, thanks everybody who have commented on the bug.

It took me a while to set up a local Samba AD DC with GPO enabled in a VM, and then a container which joins the AD using SSSD (realm + adcli), but I finally have something working here.

I can easily reproduce the problem when I enable GPO on the Samba AD DC and then try to do a regular user login on the client. I can also confirm that Valtteri's analysis seems good indeed: the Samba AD DC doesn't create the GptTmpl.inf file automatically, and SSSD doesn't handle this very well (since https://github.com/SSSD/sssd/issues/4356, apparently). I haven't tested it myself, but I have read reports from users saying that Windows machines don't have problems with this non-existing file.

After investigating a bit more, I found https://bugzilla.redhat.com/show_bug.cgi?id=1839805. This is basically the same bug report as this one, but for Fedora. In there, you can see that one of the developers mentions that SSSD should not be affected by the non-existence of this file, just like Windows isn't. I agree. The only thing I could find in the upstream git repo that addresses this problem is https://github.com/SSSD/sssd/pull/844, which has apparently been abandoned.

Now, I understand that this is a frustrating situation for some users and I agree that we could have mentioned this change in a NEWS file or something more visible. Right now, there are two possible workarounds that I know of here:

1) The easiest one, as said above, is to use the following setting in your /etc/sssd/sssd.conf:

ad_gpo_access_control = permissive

This will make SSSD try to perform GPO operations, but not fail if it can't.

2) The other possible solution, as explained by Sumit Bose on that Fedora bug, is to create the GptTpml.inf file by hand with the following contents:

-------- snip --------
[Unicode]
Unicode=yes
[Version]
signature="$CHICAGO$"
Revision=1
-------- snip --------

which, according to Sumit, represents just an empty policy file with a valid header. I tested this workaround here and it seems to work:

# cd /var/lib/samba/sysvol/test.lan/Policies/\{31B2F340-016D-11D2-945F-00C04FB984F9\}/MACHINE/
# mkdir -p Microsoft/Windows\ NT/SecEdit/
# cd Microsoft/Windows\ NT/SecEdit/
# cat > GptTmpl.inf << EOF
[Unicode]
Unicode=yes
[Version]
signature="$CHICAGO$"
Revision=1
EOF

Then, in the client machine:

# login
sambaclientdc1 login: <email address hidden>
Password:
...
<email address hidden>@sambaclientdc1:~$

I think it is also important to mention that this problem would be introduced anyway if/when you updated to a newer version of Ubuntu, because the recent SSSD versions all have this GPO code in place. As far as I have investigated, it doesn't seem to me that Samba is at fault here.

I will try to talk to the SSSD developers and see if they have any plans to tackle this problem.

Still waiting on upstream.

https://github.com/SSSD/sssd/pull/6051/files seems to be the new PR to address this, and it's closed. Have to check if it was merged into one of the stable release branches, or even better, if it's part of a release.

These releases have the fix:

$ git tag --contains 81d67a592315740d13e48b9a29e29baa8cc35638
2.7.0
2.7.1

I added a task to, in the meantime, update the server guide with the available workarounds for now.

I also just added an entry to the 22.04 release notes at https://discourse.ubuntu.com/t/jammy-jellyfish-release-notes/24668

I added a note to the SSSD docs at https://discourse.ubuntu.com/t/sssd-and-active-directory/27888 which will be live in ubuntu.com/server/docs shortly. I almost changed my mind, since the bug will eventually be fixed and then it won't be needed in the docs anymore. I'll remove the note again once this bug is fixed in 22.04.

Another user reported being affected by this bug on #ubuntu-server today. I plan to start working on SRUing this tomorrow.

Status changed to 'Confirmed' because the bug affects multiple users.

Impish will go EOL in less than a month, so I'm marking its task as Won't Fix.

This bug was fixed in the package sssd - 2.7.1-2ubuntu1

---------------
sssd (2.7.1-2ubuntu1) kinetic; urgency=medium

  * Merge with Debian unstable (LP: #1971327, #1934997). Remaining changes:
    - d/control: Drop libgdm-dev Build-Depend on i386.
    - d/control: Don't build sssd-tools on i386, now uninstallable due
      to added python3-{click,systemd} dependencies.
  * Dropped changes:
    - d/rules: Disable lto, not ready upstream.
    [ Incorporated by Debian ]

 -- Sergio Durigan Junior <email address hidden> Tue, 14 Jun 2022 16:59:20 -0400

Hello r0mulux, or anyone else affected,

Accepted sssd into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/sssd/2.6.3-1ubuntu3.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Jammy verification

root@j-client-1934997:~# apt-cache policy sssd
sssd:
  Installed: 2.6.3-1ubuntu3
  Candidate: 2.6.3-1ubuntu3
  Version table:
 *** 2.6.3-1ubuntu3 500
        500 http://archive.ubuntu.com/ubuntu jammy/main amd64 Packages

With the packages from jammy, the login fails with a system error:

root@j-client-1934997:~# login <email address hidden>
Password:

System error

And I get the error in the logs about the missing gpo file:
  * (2022-07-02 0:21:26): [gpo_child[4606]] [copy_smb_file_to_gpo_cache] (0x0400): [RID#7] smb_uri: smb://sambadc.test.lan/sysvol/test.lan/Policies/{31B2F340-016D-11D2-945F-00C04FB984F9}/MACHINE/Microsoft/Windows NT/SecEdit/GptTmpl.inf
   * (2022-07-02 0:21:26): [gpo_child[4606]] [copy_smb_file_to_gpo_cache] (0x0020): [RID#7] smbc_getFunctionOpen failed [2][No such file or directory]

root@j-client-1934997:~# login <email address hidden>
Password:
Welcome to Ubuntu 22.04 LTS (GNU/Linux 5.15.0-39-generic x86_64)
(...)
<email address hidden>@j-client-1934997:~$

I then update sssd to the version in proposed:
root@j-client-1934997:~# apt-cache policy sssd
sssd:
  Installed: 2.6.3-1ubuntu3.1
  Candidate: 2.6.3-1ubuntu3.1
  Version table:
 *** 2.6.3-1ubuntu3.1 500
        500 http://archive.ubuntu.com/ubuntu jammy-proposed/main amd64 Packages

Remove the "ad_gpo_access_control = permissive" config which allowed it to work, restart sssd.

Attempt the login again. This time the login works, and the logs stay silent:
root@j-client-1934997:~# login <email address hidden>
Password:
Welcome to Ubuntu 22.04 LTS (GNU/Linux 5.15.0-39-generic x86_64)
(...)
Last login: Sat Jul 2 00:23:05 UTC 2022 on pts/3
<email address hidden>@j-client-1934997:~$

jammy verification succeeded.

Changed tag from verification-jammy-jammy (notice the two jammy) per verification-done-jammy (to conclude the verification). I hope it helps.

Ops, thanks :)

An upload of sssd to focal-proposed has been rejected from the upload queue for the following reason: "Patch lp1934997-authentication-fails-gpo-non-existent.patch appears to have unrelated change to Makefile.am; I'm unsure if this makes any functional change to the build, but would appear to be a bit of a landmine for potential future uuploads".

Hi Chris,
I do not think this change is unrelated.

The pre-fix code used "strerror" the new one might handle sssd specific error codes and therefore used "sss_strerror" which is provided by util/util_errors[.h/.c] and therefore has to be linked into it.
So adding that is required and IMHO fine to be done in that patch.

util/signal.c on the other hand only provides BlockSignals & CatchSignal which isn't used in any of the gpo_child_* code so it was removed while touching the list of files to link.

Is that sufficient to let is pass on a second try, or did I miss some underlying problem?

Hello r0mulux, or anyone else affected,

Accepted sssd into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/sssd/2.2.3-3ubuntu0.9 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Focal verification

Reproducing the bug:
root@sssdclient:~# apt-cache policy sssd
sssd:
  Installed: 2.2.3-3ubuntu0.8
  Candidate: 2.2.3-3ubuntu0.8
  Version table:
 *** 2.2.3-3ubuntu0.8 500
        500 http://br.archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
        500 http://br.archive.ubuntu.com/ubuntu focal-security/main amd64 Packages
        100 /var/lib/dpkg/status

Login fails:
root@sssdclient:~# getent passwd <email address hidden>
<email address hidden>:*:906401103:906400513:testuser:/<email address hidden>:/bin/bash

root@sssdclient:~# login <email address hidden>
Password:

System error

Logs show it tried to fetch a gpo file and failed:
root@sssdclient:~# cat /var/log/sssd/gpo_child.log
(Wed Jul 6 14:07:59 2022) [gpo_child[2738]] [copy_smb_file_to_gpo_cache] (0x0020): smbc_getFunctionOpen failed [2][No such file or directory]
(Wed Jul 6 14:07:59 2022) [gpo_child[2738]] [perform_smb_operations] (0x0020): copy_smb_file_to_gpo_cache failed [2][No such file or directory]
(Wed Jul 6 14:07:59 2022) [gpo_child[2738]] [main] (0x0020): perform_smb_operations failed.[2][No such file or directory].
(Wed Jul 6 14:07:59 2022) [gpo_child[2738]] [main] (0x0020): gpo_child failed!

Adding the "ad_gpo_access_control = permissive" workaround to /etc/sssd/sssd.conf and restarting makes it work:

root@sssdclient:~# login <email address hidden>
Password:
Welcome to Ubuntu 20.04.4 LTS (GNU/Linux 5.4.0-1070-kvm x86_64)
(...)
Creating directory '/<email address hidden>'.
<email address hidden>@sssdclient:~$ id
uid=906401103(<email address hidden>) gid=906400513(domain <email address hidden>) groups=906400513(domain <email address hidden>)

Removing the workaround from /etc/sssd/sssd.conf, restarting sssd, purging the cache, confirming the failure one more time:
root@sssdclient:~# vi /etc/sssd/sssd.conf
root@sssdclient:~# systemctl restart sssd
root@sssdclient:~# sssctl cache-remove -o -p -s
Creating backup of local data...
Removing cache files...
root@sssdclient:~# login <email address hidden>
Password:

System error

And now updating sssd to the version in proposed:
root@sssdclient:~# apt-cache policy sssd
sssd:
  Installed: 2.2.3-3ubuntu0.9
  Candidate: 2.2.3-3ubuntu0.9
  Version table:
 *** 2.2.3-3ubuntu0.9 500
        500 http://br.archive.ubuntu.com/ubuntu focal-proposed/main amd64 Packages
        100 /var/lib/dpkg/status

Login now works without the workaround:

root@sssdclient:~# grep permissive /etc/sssd/sssd.conf

root@sssdclient:~# login <email address hidden>
Password:
Welcome to Ubuntu 20.04.4 LTS (GNU/Linux 5.4.0-1070-kvm x86_64)
(...)
Last login: Wed Jul 6 14:10:29 UTC 2022 on pts/0
<email address hidden>@sssdclient:~$ id
uid=906401103(<email address hidden>) gid=906400513(domain <email address hidden>) groups=906400513(domain <email address hidden>)

And the gpo_child.log (after bumping debug to 2 again) are silent this time.

Focal verification succeeded.

This bug was fixed in the package sssd - 2.6.3-1ubuntu3.1

---------------
sssd (2.6.3-1ubuntu3.1) jammy; urgency=medium

  * d/p/lp1934997-authentication-fails-gpo-non-existent.patch:
    Fix authentication failure when GPO is enabled and
    SecEdit/GptTmpl.inf is missing (LP: #1934997).
  * d/p/lp1979350-GPO-ignore-non-ascii-symbols-in-GPT.INI.patch:
    Ignore non-ASCII characters in GPT.INI. (LP: #1979350)
  * Fix "sssctl analyze" (LP: #1979453)
    [ Timo Aaltonen ]
    - d/{python3-libipa-hbac.install,python3-sss.install,
      sssd-tools.install,rules}: Fix Python installation directory
      path.
    [ Sergio Durigan Junior ]
    - d/p/lp1979453-fix-shebang-on-sss_analyze.patch: Fix shebang on
      sss_analyze.

 -- Sergio Durigan Junior <email address hidden> Wed, 22 Jun 2022 12:31:22 -0400

The verification of the Stable Release Update for sssd has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package sssd - 2.2.3-3ubuntu0.9

---------------
sssd (2.2.3-3ubuntu0.9) focal; urgency=medium

  * d/p/lp1934997-authentication-fails-gpo-non-existent.patch:
    Fix authentication failure when GPO is enabled and
    SecEdit/GptTmpl.inf is missing (LP: #1934997).
  * d/p/lp1979350-GPO-ignore-non-ascii-symbols-in-GPT.INI.patch:
    Ignore non-ASCII characters in GPT.INI. (LP: #1979350)

 -- Sergio Durigan Junior <email address hidden> Tue, 21 Jun 2022 14:29:52 -0400