Backport: SSL: use of the SSL_OP_IGNORE_UNEXPECTED_EOF option.

Status changed to 'Confirmed' because the bug affects multiple users.

Thanks for reporting this bug, Daniel.

We will also need to address this one for kinetic.

A similar issue was reported for php, which we are addressing in LP: #1975626

I added this to the server team backlog so someone can start working on it soon. In the meantime, would you be able to provide a simple reproducer for the bug?

I think this can be reproduced with the following command (replace $URL with an HTTPS protected URL to the nginx in question):

```
timeout -s KILL 0.2s curl -v -K <(echo verbose;for i in {1..20}; do echo url = "$URL"; echo -o /dev/null; done)
```

At least I didn't get any "unexpected eof while reading" errors reported in the error log of nginx for an older version (1.14.0-0ubuntu1.9) on Ubuntu Bionic.

Hi Daniel,

Thanks for the reproducer.

I've packaged the patch and posted it to this PPA for testing:

    https://launchpad.net/~bryce/+archive/ubuntu/nginx-fix-lp1981457

I'd appreciate it if you could verify it does indeed fix the issue.

Also, if you're doing anything unusual with your https setup, it would also be super helpful if you could share your config snippets or outline the process you're using to set things up.

The patch is available in upstream's release-1.22.0, which is now uploaded to kinetic-proposed:

 nginx | 1.18.0-6ubuntu14 | jammy
 nginx | 1.18.0-6ubuntu14.1 | jammy-security
 nginx | 1.18.0-6ubuntu14.1 | jammy-updates
 nginx | 1.20.2-2ubuntu2 | kinetic
 nginx | 1.22.0-1ubuntu1 | kinetic-proposed

I'm keeping an eye on the transition for this from kinetic-proposed to kinetic in case there are any issues.

Hi Bryce,

Thank you for providing the fixed version in your PPA. I've installed this version and so far, the error messages have not been recorded again yet. Neither for the reproducer nor for the real production scenario.

However, since in the real scenario it occurred together with a remote third party intermediate box, it is rather difficult to test.

Attached, you may find some config snippets from the production scenario, although I don't consider this configuration unusual.

 nginx | 1.22.0-1ubuntu1 | kinetic

The new nginx with this patch has fully landed to kinetic.

Hi Daniel,

Thanks for verifying the fix against your reproducer, and understood about difficulty of verifying the fix in production. In the week since your post have you seen any recurrence of the original issue?

Do you feel the PPA is an adequate enough fix for your purposes, or should we proceed with attempting to get an SRU accepted? I notice while no one else has commented on this bug, there's 4 marked as affected, so I imagine others will find benefit to having this fix.

Hi Bryce,

On our testing environment, I haven't seen the issue arise again, which leads me to believe, this patch resolves it entirely. Additionally, I've again pushed for feedback from the remote side.

Is this PPA kept in sync with the patch/security releases of Ubuntu 22.04 LTS as well? If it is, this might be an option. Otherwise, I would favour having it released in the official repository.

I'll get back once I have confirmation from the remote side as well.

The PPA is just for this specific bug, and is not going to be kept in sync with security updates. Glad to hear it seems to have resolved the problem you've been seeing though.

The main issue for getting this into the official release is that SRU policy tends to be strict about having defined methods to reproduce the bug and verify the fix. If there is not a well-defined reproducer for this bug, then there's a strong chance the SRU team will decline it.

I'm going to leave the server team subscribed to this bug for now, in hopes possibly myself or one of my team mates may find some time some day to work on finding how to reproduce it in our environment. If you find a way to reproduce it synthetically, let us know; that could help a lot.

If you're interested, or just for reference, the SRU process is documented at https://wiki.ubuntu.com/StableReleaseUpdates .

Hello,

I would like to voice my concern about this bug, I would very much like it to be included in Ubuntu 22.04 LTS.
If you google the relevant terms you'll see that many users have this problem, often related to either nginx or php.
I've used the PPA to resolve the issue on our production servers, but this isn't a long term solution.
Otherwise everyone using nginx on Ubuntu 22.04 LTS will be forced to use the upstream nginx repositories instead, which isn't ideal either.

BTW As soon as I upgraded nginx and openssl, my server logs were spammed full of these errors, from all over the world, so I suspect this would show up in anyone's logs.

Best regards,

jgb1984

@bryce, I took the suggested reproducer in comment 3 and pushed it to the extreme:

while :; do timeout -s KILL 0.2s curl -v -K <(echo verbose;for i in {1..2000}; do echo url = "$URL"; echo -o /dev/null; done); done

With nginx 1.18.0-6ubuntu14.1, this immediately leads to hundreds of 'unexpected eof while reading' in the logs.

With nginx 1.18.0-6ubuntu14.2~jammy1 (from your PPA), I've been hammering it for several minutes and the error is not coming back.

Please note that before using the reproducer, the error happened "naturally" as it was triggered by regular clients, multiple times a day.

My biggest concern was that the value of SSL_OP_IGNORE_UNEXPECTED_EOF might change from OpenSSL 1.1.x to 3.0.x but it looks like it has the same value in all our packages (at least according to whatever I've got checked out at the moment):

$ rg 'define.*SSL_OP_IGNORE_UNEXPECTED_EOF'
openssl_3.0.3-0ubuntu1/include/openssl/ssl.h.in
335:# define SSL_OP_IGNORE_UNEXPECTED_EOF SSL_OP_BIT(7)

openssl_3.0.0-1ubuntu1/include/openssl/ssl.h.in
335:# define SSL_OP_IGNORE_UNEXPECTED_EOF SSL_OP_BIT(7)

openssl_3.0.2-0ubuntu1/include/openssl/ssl.h.in
335:# define SSL_OP_IGNORE_UNEXPECTED_EOF SSL_OP_BIT(7)

openssl_3.0.0-1ubuntu2/include/openssl/ssl.h.in
335:# define SSL_OP_IGNORE_UNEXPECTED_EOF SSL_OP_BIT(7)

openssl_3.0.1-0ubuntu1/include/openssl/ssl.h.in
335:# define SSL_OP_IGNORE_UNEXPECTED_EOF SSL_OP_BIT(7)

openssl_3.0.2-0ubuntu1.2/include/openssl/ssl.h.in
335:# define SSL_OP_IGNORE_UNEXPECTED_EOF SSL_OP_BIT(7)

openssl_3.0.3-5ubuntu2/include/openssl/ssl.h.in
335:# define SSL_OP_IGNORE_UNEXPECTED_EOF SSL_OP_BIT(7)

openssl_3.0.2-0ubuntu1.1/include/openssl/ssl.h.in
335:# define SSL_OP_IGNORE_UNEXPECTED_EOF SSL_OP_BIT(7)

It makes sense to me. A general-purpose webserver can't realistically know whatever every hosted application it runs will do with an improperly-terminated tls session: some might care a *lot*, some might not care at all, and most people just serve files with it and really don't care.

If upstream doesn't care to pass this along to the hosted applications, that's a strong vote for us doing the same.

Thanks

Package with fix has been uploaded to Jammy, and SRU text has been prepared.

Hello Daniel, or anyone else affected,

Accepted nginx into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/nginx/1.18.0-6ubuntu14.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-jammy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

I enabled jammy-proposed:

...
Get:1 http://us.archive.ubuntu.com/ubuntu jammy-proposed/universe amd64 nginx-light amd64 1.18.0-6ubuntu14.2 [457 kB]
Get:2 http://us.archive.ubuntu.com/ubuntu jammy-proposed/main amd64 libnginx-mod-http-geoip2 amd64 1.18.0-6ubuntu14.2 [11.8 kB]
Get:3 http://us.archive.ubuntu.com/ubuntu jammy-proposed/universe amd64 libnginx-mod-http-echo amd64 1.18.0-6ubuntu14.2 [22.7 kB]
Get:4 http://us.archive.ubuntu.com/ubuntu jammy-proposed/main amd64 nginx-common all 1.18.0-6ubuntu14.2 [40.0 kB]
...

Then followed the steps to reproduce:

 while :; do timeout -s KILL 0.2s curl -v -K <(echo verbose;for i in {1..2000}; do echo url = "$URL"; echo -o /dev/null; done); done

And no 'unexpected eof while reading' error showed in the logs.

So I'm marking this as verified, thanks Bryce!

I originally experienced this issue on version 1.18.0-6ubuntu14.1.

After enabling jammy-proposed and upgrading to 1.18.0-6ubuntu14.2 I no longer see the "unexpected eof while reading" errors. I also don't see any other new issues.

Having updated nginx to 1.18.0-6ubuntu14.2 from jammy-proposed as well, I can confirm that neither do I observe these error messages anymore:

```
apt install --only-upgrade libnginx-*=1.18.0-6ubuntu14.2 nginx*=1.18.0-6ubuntu14.2
```

Using the curl command to reproduce the issue, the error log now stays clean.

This bug was fixed in the package nginx - 1.18.0-6ubuntu14.2

---------------
nginx (1.18.0-6ubuntu14.2) jammy; urgency=medium

  * d/p/ssl-op-ignore-unexpected-eof-option.patch: Add compatibility
    flag for certain clients that don't close their connection properly.
    (LP: #1981457)

 -- Bryce Harrington <email address hidden> Tue, 02 Aug 2022 15:24:53 -0700

The verification of the Stable Release Update for nginx has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.