ntlm_auth --helper-protocol=squid-2.5-ntlmssp report segfault

Status changed to 'Confirmed' because the bug affects multiple users.

This broke web browsing for my 325 employees this morning. Trying to research how to roll back to 2:4.1.6+dfsg-1ubuntu2. Please provide instructions if you can to help others affected. Thanks for reporting this and your help!

Yep, we had to turn off single signon

Just so I can try and reproduce this in a test environment, are you authenticating to a Windows AD, or to a Samba AD?

Windows AD

i have rolled back and all good now.

I can't seem to reproduce this issue. Can someone get a backtrace?

Are you able to reproduce it by using ntlm_auth directly on the command line, like so?:

$ sudo ntlm_auth --helper-protocol=squid-2.5-basic
MYDOMAIN\testuser mypassword

$ sudo ntlm_auth --helper-protocol=squid-2.5-ntlmssp
MYDOMAIN\testuser mypassword

When I run the commands on the command line they don't exit and pressing return gives:

ERR

Pressing "?" gives (for ntlmssp, not basic):

BH Query invalid

Typing about anything else gives me (again, only for ntlmssp, not basic):

BH SPNEGO request invalid prefix

I tried installing gdb and creating a backtrace, but I'm not sure what I'm doing - can't see to get anything useful. I'm reading this may be because apport is enabled?

It looks like apport creates files under /var/crash/_usr_bin_ntlm_auth.13.crash automatically - I've attached this hoping it has what you need.

I am seeing the same errors. I am using ntlm_auth for Moodle access.

When attempting to use NTLM, I show this in the processes:

www-data 9866 9856 0 08:52 ? 00:00:00 [ntlm_auth] <defunct>

Same problem. Am running:

Ubuntu Server 14.04.1 LTS
Windbind: 2:4.3.9+dfsg-0ubuntu0.14.04.1
Samba: 2:4.3.9+dfsg-0ubuntu0.14.04.1
Squid3: 3.3.8-1ubuntu6.6

Authenticating against Active Directory - has been working really well for the last 18 months, then stopped working about a week ago.

Errors in cache.log:
2016/05/09 06:20:07| Too few ntlmauthenticator processes are running (need 1/10)
2016/05/09 06:20:07| Starting new helpers
2016/05/09 06:20:07| helperOpenServers: Starting 1/10 'ntlm_auth' processes
2016/05/09 06:20:07| ERROR: NTLM Authentication Helper '0x7f313ea68318' crashed!.
2016/05/09 06:20:07| ERROR: NTLM Authentication validating user. Error returned 'BH Internal error'
2016/05/09 06:20:08| WARNING: ntlmauthenticator #1 exited

Errors in syslog:
May 9 06:20:09 optsquidproxy kernel: [228590.127125] ntlm_auth[8850]: segfault at 8 ip 00007f201ec729b0 sp 00007ffda249aae8 error 4 in libsamba-security.so.0[7f201ec67000+1b000]

Squid is using pure NTLM authentication (taken from squid.conf):
### pure ntlm authentication
auth_param ntlm program /usr/bin/ntlm_auth --diagnostics --helper-protocol=squid-2.5-ntlmssp --domain=MYDOMAIN
auth_param ntlm children 10
auth_param ntlm keep_alive off

running an Apache2 server with NTLM authentication against an AD, stopped working with 500 Internal Server error since the Samba upgrade.

Apache config:
AuthType NTLM
AuthName "..."
NTLMAuth on
NTLMAuthHelper "/usr/bin/ntlm_auth --helper-protocol=squid-.5-ntlmssp"
NTLMBasicAuthoritative on
require valid-user

Syslog:
May 9 08:41:17 intranet-gvp kernel: [438844.377831] ntlm_auth[23011]: segfault at 8 ip 00007f512efcf9b0 sp 00007ffc8017cc78 error 4 in libsamba-security.so.0[7f512efc4000+1b000]
May 9 08:41:23 intranet-gvp kernel: [438849.956323] ntlm_auth[23148]: segfault at 8 ip 00007f39b08e99b0 sp 00007fff4ba86bb8 error 4 in libsamba-security.so.0[7f39b08de000+1b000]
May 9 08:41:30 intranet-gvp kernel: [438856.430960] ntlm_auth[23240]: segfault at 8 ip 00007f96a55309b0 sp 00007ffe0a7eaa98 error 4 in libsamba-security.so.0[7f96a5525000+1b000]
May 9 08:43:30 intranet-gvp kernel: [438977.462065] ntlm_auth[25264]: segfault at 8 ip 00007f874faf29b0 sp 00007ffd417d0478 error 4 in libsamba-security.so.0[7f874fae7000+1b000]
May 9 08:45:03 intranet-gvp kernel: [439070.043363] ntlm_auth[28559]: segfault at 8 ip 00007fb5af4769b0 sp 00007ffcc2b84918 error 4 in libsamba-security.so.0[7fb5af46b000+1b000]
May 9 08:47:12 intranet-gvp kernel: [439199.384723] ntlm_auth[30675]: segfault at 8 ip 00007f357d1439b0 sp 00007ffef1ea7c98 error 4 in libsamba-security.so.0[7f357d138000+1b000]
May 9 08:47:25 intranet-gvp kernel: [439211.944010] ntlm_auth[30822]: segfault at 8 ip 00007f89a24e49b0 sp 00007ffffb32b3c8 error 4 in libsamba-security.so.0[7f89a24d9000+1b000]
May 9 08:50:12 intranet-gvp kernel: [439379.146404] ntlm_auth[1121]: segfault at 8 ip 00007fdb6f7d19b0 sp 00007ffcf44e5728 error 4 in libsamba-security.so.0[7fdb6f7c6000+1b000]

After rollback to Samba packages 4.3.8, the system is running again.

$ pgrep ntlm -a
2192 ntlm_auth
4951 /usr/bin/ntlm_auth --helper-protocol=squid-2.5-ntlmssp
4952 /usr/bin/ntlm_auth --helper-protocol=squid-2.5-ntlmssp
4953 /usr/bin/ntlm_auth --helper-protocol=squid-2.5-ntlmssp
6074 /usr/bin/ntlm_auth --helper-protocol=squid-2.5-ntlmssp
6503 /usr/bin/ntlm_auth --helper-protocol=squid-2.5-ntlmssp
6596 /usr/bin/ntlm_auth --helper-protocol=squid-2.5-ntlmssp
15376 /usr/bin/ntlm_auth --helper-protocol=squid-2.5-ntlmssp
17037 /usr/bin/ntlm_auth --helper-protocol=squid-2.5-ntlmssp
17149 /usr/bin/ntlm_auth --helper-protocol=squid-2.5-ntlmssp

We're getting the same thing on 16.04 - narrowed it down to the same segfault. Here's a strace of the apache process which should be useful, it seems to work fine on the first challenge but then fail on the second

strace: Process 6911 attached
accept4(4, {sa_family=AF_INET6, sin6_port=htons(42500), inet_pton(AF_INET6, "::ffff:10.10.0.210", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [28], SOCK_CLOEXEC) = 26
gettimeofday({1462796465, 907646}, NULL) = 0
getsockname(26, {sa_family=AF_INET6, sin6_port=htons(80), inet_pton(AF_INET6, "::ffff:10.10.0.24", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [28]) = 0
fcntl(26, F_GETFL) = 0x2 (flags O_RDWR)
fcntl(26, F_SETFL, O_RDWR|O_NONBLOCK) = 0
gettimeofday({1462796465, 908578}, NULL) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1810dac000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1810daa000
read(26, "GET /ntlm-auth HTTP/1.1\r\nHost: jmk-dev-web\r\nUser-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:46.0) Gecko/20100101 Firefox/46.0\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nAccept-Language: en-US,en;q=0.5\r\nAccept-Encoding: gzip, deflate\r\nReferer: http://jmk-dev-web/\r\nConnection: keep-alive\r\nCache-Control: max-age=0\r\n\r\n", 8000) = 354
gettimeofday({1462796465, 909153}, NULL) = 0
gettimeofday({1462796465, 909345}, NULL) = 0
gettimeofday({1462796465, 909434}, NULL) = 0
stat("/var/www/jmk/ntlm-auth", {st_dev=makedev(202, 1), st_ino=263914, st_mode=S_IFDIR|0770, st_nlink=2, st_uid=101143, st_gid=33, st_blksize=4096, st_blocks=8, st_size=4096, st_atime=2016/05/09-09:16:32, st_mtime=2016/04/22-12:27:32.672865000, st_ctime=2016/04/22-12:27:32.672865000}) = 0
open("/var/www/.htaccess", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
open("/var/www/jmk/.htaccess", O_RDONLY|O_CLOEXEC) = 27
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1810da8000
fstat(27, {st_dev=makedev(202, 1), st_ino=912136, st_mode=S_IFREG|0770, st_nlink=1, st_uid=101143, st_gid=33, st_blksize=4096, st_blocks=8, st_size=28, st_atime=2016/05/09-10:29:18.820558786, st_mtime=2016/05/09-10:29:15.964543846, st_ctime=2016/05/09-10:29:15.964543846}) = 0
read(27, "AddHandler php7.0-fcgi .php\n", 4096) = 28
read(27, "", 4096) = 0
gettimeofday({1462796465, 910687}, NULL) = 0
close(27) = 0
open("/var/www/jmk/ntlm-auth/.htaccess", O_RDONLY|O_CLOEXEC) = 27
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1810da6000
fstat(27, {st_dev=makedev(202, 1), st_ino=269927, st_mode=S_IFREG|0770, st_nlink=1, st_uid=101143, st_gid=33, st_blksize=4096, st_blocks=8, st_size=170, st_atime=2016/05/09-09:16:32.004000000, st_mtime=2016/04/22-13:54:12.424865000, st_ctime=2016/04/22-13:54:12.424865000}) = 0
read(27, "AuthName \"NTLM Test\"\r\nNTLMAuth on\r\nNTLMAuthHelper \"/usr/bin/ntlm_auth --helper-protocol=squid-2.5-ntlmssp\"\r\nNTLMBasicAuthoritative on\r\nAuthType NTLM\r\nRequire valid-user\r\n", 4096) = 170
read(27, "", 4096) = 0
close(27) = ...

Could someone who is experiencing this regression with Samba 4.3.9 please report it to the Samba developers here:

https://bugzilla.samba.org/

Please attach the bug report with this one once it's been filed.

I'd file the bug myself, but they may require more details on the problematic environment.

Thanks!

Have submitted to the Samba developers: https://bugzilla.samba.org/show_bug.cgi?id=11912

Could anyone give a quick guide on how to downgrade samba/winbind to workaround this issue while it is fixed?

My downgrade command:

aptitude -V install winbind=2:4.1.6+dfsg-1ubuntu2 samba=2:4.1.6+dfsg-1ubuntu2 libwbclient0=2:4.1.6+dfsg-1ubuntu2 samba-libs=2:4.1.6+dfsg-1ubuntu2 python-samba=2:4.1.6+dfsg-1ubuntu2 samba-vfs-modules=2:4.1.6+dfsg-1ubuntu2 libldb1=1:1.1.16-1 samba-common=2:4.1.6+dfsg-1ubuntu2 samba-common-bin=2:4.1.6+dfsg-1ubuntu2 samba-dsdb-modules=2:4.1.6+dfsg-1ubuntu2 python-ldb=1:1.1.16-1 libnss-winbind=2:4.1.6+dfsg-1ubuntu2 libpam-winbind=2:4.1.6+dfsg-1ubuntu2

Upstream has a patch in their bug now

I have uploaded test packages that include the upstream fix to the following PPA:

https://launchpad.net/~mdeslaur/+archive/ubuntu/testing

Once they finish building, could someone please test them and confirm it fixes the issue?

Once someone has tested them successfully, I will publish them as security updates.

Thanks!

Just tried and still getting the same problem:
samba:
  Installed: 2:4.3.9+dfsg-0ubuntu0.14.04.2~ppa1
  Candidate: 2:4.3.9+dfsg-0ubuntu0.14.04.2~ppa1
  Version table:
 *** 2:4.3.9+dfsg-0ubuntu0.14.04.2~ppa1 0
        500 http://ppa.launchpad.net/mdeslaur/testing/ubuntu/ trusty/main amd64 Packages
        100 /var/lib/dpkg/status
     2:4.3.9+dfsg-0ubuntu0.14.04.1 0
        500 http://optubunturepository.MYDOMAIN.net/ubuntu/ trusty-updates/main amd64 Packages
        500 http://optubunturepository.MYDOMAIN.net/ubuntu/ trusty-security/main amd64 Packages
     2:4.1.6+dfsg-1ubuntu2 0
        500 http://optubunturepository.MYDOMAIN.net/ubuntu/ trusty/main amd64 Packages

syslog:
May 10 14:31:11 optsquidproxy kernel: [ 206.928248] ntlm_auth[2264]: segfault at 8 ip 00007f68e2aba9b0 sp 00007fff384ec2c8 error 4 in libsamba-security.so.0[7f68e2aaf000+1b000]

cache.log:
2016/05/10 14:32:42| WARNING: ntlmauthenticator #1 exited
2016/05/10 14:32:42| Too few ntlmauthenticator processes are running (need 1/10)
2016/05/10 14:32:42| Starting new helpers
2016/05/10 14:32:42| helperOpenServers: Starting 1/10 'ntlm_auth' processes
2016/05/10 14:32:42| ERROR: NTLM Authentication Helper '0x7f8368efb268' crashed!.
2016/05/10 14:32:42| ERROR: NTLM Authentication validating user. Error returned 'BH Internal error'

Thanks for testing Paul. Did you update all the binary packages in the PPA, including winbind and samba-libs, and did you reboot or restart all processes including squid?

Yep - added your testing PPA, then:

sudo apt-get update
sudo apt-get upgrade samba (which brought in all dependencies)

Then rebooted the server for good measure :)

The above error messages were from after the reboot, and I'm still getting issues when trying to authenticate via the squid proxy.

Thanks Paul. Could you please add a comment to the upstream bug?

Done :)

It seems there're two similar bugs, I've created
https://bugzilla.samba.org/show_bug.cgi?id=11914 to track the 2nd problem

I have uploaded a test package for trusty that includes the proposed fix for Samba bug #11914 to the following PPA:

https://launchpad.net/~mdeslaur/+archive/ubuntu/testing

Thanks Marc - appears to work for me!

This bug was fixed in the package samba - 2:4.3.9+dfsg-0ubuntu0.14.04.3

---------------
samba (2:4.3.9+dfsg-0ubuntu0.14.04.3) trusty-security; urgency=medium

  * SECURITY REGRESSION: NTLM authentication issues (LP: #1578576)
    - debian/patches/samba-bug11912.patch: let msrpc_parse() return
      talloc'ed empty strings in libcli/auth/msrpc_parse.c.
    - debian/patches/samba-bug11914.patch: make
      ntlm_auth_generate_session_info() more complete in
      source3/utils/ntlm_auth.c.
  * debian/rules: work around amd64 build failure (LP: #1585174)

 -- Marc Deslauriers <email address hidden> Tue, 24 May 2016 07:47:59 -0400

This bug was fixed in the package samba - 2:4.3.9+dfsg-0ubuntu0.16.04.2

---------------
samba (2:4.3.9+dfsg-0ubuntu0.16.04.2) xenial-security; urgency=medium

  * SECURITY REGRESSION: NTLM authentication issues (LP: #1578576)
    - debian/patches/samba-bug11912.patch: let msrpc_parse() return
      talloc'ed empty strings in libcli/auth/msrpc_parse.c.
    - debian/patches/samba-bug11914.patch: make
      ntlm_auth_generate_session_info() more complete in
      source3/utils/ntlm_auth.c.

 -- Marc Deslauriers <email address hidden> Fri, 20 May 2016 07:31:37 -0400

This bug was fixed in the package samba - 2:4.3.9+dfsg-0ubuntu0.15.10.2

---------------
samba (2:4.3.9+dfsg-0ubuntu0.15.10.2) wily-security; urgency=medium

  * SECURITY REGRESSION: NTLM authentication issues (LP: #1578576)
    - debian/patches/samba-bug11912.patch: let msrpc_parse() return
      talloc'ed empty strings in libcli/auth/msrpc_parse.c.
    - debian/patches/samba-bug11914.patch: make
      ntlm_auth_generate_session_info() more complete in
      source3/utils/ntlm_auth.c.

 -- Marc Deslauriers <email address hidden> Fri, 20 May 2016 08:09:44 -0400

This bug was fixed in the package samba - 2:4.3.9+dfsg-0ubuntu1

---------------
samba (2:4.3.9+dfsg-0ubuntu1) yakkety; urgency=medium

  * SECURITY REGRESSION: Updated to 4.3.9 to fix multiple regressions in
    the previous security updates. (LP: #1577739)
    - debian/control: bump tevent Build-Depends to 0.9.28.
  * SECURITY REGRESSION: NTLM authentication issues (LP: #1578576)
    - debian/patches/samba-bug11912.patch: let msrpc_parse() return
      talloc'ed empty strings in libcli/auth/msrpc_parse.c.
    - debian/patches/samba-bug11914.patch: make
      ntlm_auth_generate_session_info() more complete in
      source3/utils/ntlm_auth.c.

 -- Marc Deslauriers <email address hidden> Wed, 25 May 2016 09:29:15 -0400