monitor_read: unpermitted request 48 on server while attempting GSSAPI key exchange

Hello Niklas,

Thank you for taking the time to file a bug report.

While the symptoms experienced here seem similar to the ones reported in https://bugzilla.redhat.com/show_bug.cgi?id=1162620, the patch that fixed the latter is present in the version of the package for which you reported the issue.

Therefore, would you mind providing additional information, such as configuration files? More importantly, we would be interested in a reproducer for the issue.
Can you reproduce it without using ansible?

Since there is not enough information in your report to begin triage or to
differentiate between a local configuration problem and a bug in Ubuntu, I
am marking this bug as "Incomplete". We would be grateful if you would:
provide a more complete description of the problem, explain why you
believe this is a bug in Ubuntu rather than a problem specific to your
system, and then change the bug status back to "New".

For local configuration issues, you can find assistance here:
http://www.ubuntu.com/support/communit

Hello Athos,

thanks for looking into this!

This is reproducible without Ansible, that was just use-case that brought up the issue. I've further narrowed it down to the following setup:

Server:
/usr/sbin/sshd -d -p 2222 -f /dev/null -o GSSAPIKeyExchange=yes -o GSSAPIAuthentication=yes

Client:
ssh -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex root@compute-test -v -p 2222 -o GSSAPIKeyExchange=yes -F /dev/null

I think this should make it independent from my local config, right? Obviously there is also Kerberos involved, which I would call configured pretty standard in our environment, but I can have a look at that config as well, if this is desired.

The problem will not arise when:
- The client has no valid Kerberos-Key (unset KRB5CCNAME)
- If any of the the GSSAPI* options is missing on client or server
- If the order of "gssapi-with-mic,gssapi-keyex" is switched (!)

Hello Niklas!

You're welcome again. Thanks for adding more information... Good to know we can put Ansible apart from this.

Anyway, we would need more information about the Kerberos configuration you mentioned before as you noticed it is involved because we don't have the complete picture to reproduce the issue. Also sshd configuration is needed for both client and server.

In the time we receive that, I will mark the report as "Incomplete", but thanks a lot for progressing on it.

When you submit new information, please mark the bug as "New" so we can continue with it.

So, I give this a try and attempted to reproduce the issue.

I set up a VM acting as the KDC, and configured sshd in it with the following options:

GSSAPIAuthentication yes
GSSAPICleanupCredentials yes
GSSAPIKeyExchange yes

I then configured an LXD container to act as the krb5 client. I created a user "john" both in the KDC and in the client, then was able to verify that kinit was working fine. With that out of the way, I tried to connect via ssh to the KDC:

$ ssh -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex -o GSSAPIKeyExchange=yes krb5.test.lan

The connection worked. I did the RH bug and tried to check if there was anything else I could do, but apparently the bug should have manifested with what I did. I also tried to start sshd by hand using the options you mentioned (plus "-o UsePam=yes"), to no avail. So I'm a bit lost here, and would also appreciate more info.

Thanks.

Thanks Sergio for trying to reproduce this! I'm a bit puzzled, why the bug did not trigger in your case. I'll try to reproduce this in a clean VM as well now. One important thing might be that I tried to login as "root", for which I did not have a Kerberos-Ticket, so a "permission denied" would be expected, unless something like "publickey" is included in PreferredAuthentications.

@Miriam: The SSH-configuration should be irrelevant, because of the options "-f /dev/null" and "-F /dev/null". The Kerberos-config could be part of this. I'll try to reproduce this on a clean machine and post better steps for reproducing afterwards.

Thank you all for helping with this!

Ok, I managed to reproduces this in a clean "ubuntu:latest" docker container. Steps to reproduce are below. During testing, I noticed that I aliased "ssh" to "ssh -K -X", and that "-K" (or equivalently "-o GSSAPIAuthentication=yes") is crucial. This changes the problematic SSH client command to

ssh -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex root@ac3f9944f201 -v -p 2222 -o GSSAPIKeyExchange=yes -o GSSAPIAuthentication=yes -F /dev/null

Complete steps to reproduce (container ac3f9944f201 is the server, IP 1.2.3.4 is the IP of the container host; this needs to be adapted):

Server:

podman run -it -p 2222:2222,8888:88 ubuntu

apt update
apt install openssh-server krb5-kdc krb5-admin-server
touch /etc/krb5kdc/kadm5.acl
touch /etc/krb5kdc/kadm5.dict
krb5_newrealm
kadmin.local

addprinc user
addprinc -randkey host/ac3f9944f201
ktadd -k /etc/krb5.keytab host/ac3f9944f201
exit

mkdir /run/sshd
/usr/sbin/sshd -d -p 2222 -f /dev/null -o GSSAPIKeyExchange=yes -o GSSAPIAuthentication=yes

Client:

podman run -it ubuntu

apt update
apt install openssh-client krb5-user
kinit user
echo "1.2.3.4 ac3f9944f201" >> /etc/hosts

ssh -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex root@ac3f9944f201 -v -p 2222 -o GSSAPIKeyExchange=yes -o GSSAPIAuthentication=yes -F /dev/null

Notice "monitor_read: unpermitted request 48" on the server, and "Connection closed by 1.2.3.4 port 2222" on the client (instead of the expected "permission denied).

Hi Niklas,

Thanks for putting in the effort into finding a reproducer for the reported issue.

I could indeed reproduce the issue you have been experiencing. I am attaching a couple scripts to aid others to reproduce the bug (this includes a README file with further instructions).

Interestingly, if you swap the preferred authentications order to read

PreferredAuthentications=gssapi-keyex,gssapi-with-mic

The bug will not manifest itself.

Next, I will verify if other branches at https://github.com/openssh-gsskex/openssh-gsskex are also affected. If this is the case, we should report the issue there.

The issue is reproducible in the latest published versions of openssh carrying the patches in https://github.com/openssh-gsskex/openssh-gsskex for Ubuntu (impish), Debian (unstable), and Fedora (rawhide).

I filed a bug report in https://github.com/openssh-gsskex/openssh-gsskex/issues/20 to make sure the gsskex patch upstream is aware of this issue.

Dmitry Belyavskiy proposed a patch for this issue at https://github.com/openssh-gsskex/openssh-gsskex/pull/21.

I created a PPA with the proposed fix at https://launchpad.net/~athos-ribeiro/+archive/ubuntu/openssh-gssapi-fix/+packages and I can confirm it does fix the reproducer proposed in this bug.

Moreover, running the server with

/usr/sbin/sshd -d -p 2222 -f /dev/null -o GSSAPIKeyExchange=yes -o GSSAPIAuthentication=yes -o PasswordAuthentication=yes -o PermitRootLogin=yes

And logging in as root, will prompt for the root password and get you a proper ssh connection.

Finally, I also ran the available openssh dep8 test suite to ensure the patch would not introduce covered regrerssions.

autopkgtest [17:57:18]: @@@@@@@@@@@@@@@@@@@@ summary
regress PASS

Niklas, it would be really nice if you could also test the proposed patch to confirm it does fix the reported issue.

Thanks for the PPA! I currently don't have an impish system at hand, would it be possible to build it for focal as well?

Hi Niklas,

I just pushed the focal patched package to that same PPA. Note that they are only available for x86_64 and i386. Let me know if you need it for any other platforms.

Thanks for building the packages for focal.

I can confirm that this fixes the problem for me!

Woooooooot, thanks for confirming Niklas. Athos, would you mind prepping an MP for this, et al? TIA, good sir! \o/

Thanks, Niklas!

Utkarsh, Paride: Since this seems to be a low priority issue, I am waiting to see if we get a couple more eyes into https://github.com/openssh-gsskex/openssh-gsskex/pull/21 before adding this one in our delta (this could even go into Debian first and then we can start preparing SRUs). Therefore, I am also removing the server-next tag from this one.

The Hirsute Hippo has reached End of Life, so this bug will not be fixed for that release.

That PR is still pending review, no movements there unfortunately.