GnuTLS recv error (-9): A TLS packet with unexpected length was received

Bug #1111882 reported by TJ
172
This bug affects 36 people
Affects Status Importance Assigned to Milestone
apt (Ubuntu)
Invalid
Undecided
Unassigned
curl (Ubuntu)
Invalid
Undecided
Unassigned
git (Ubuntu)
Confirmed
Undecided
Unassigned
gnutls26 (Ubuntu)
Confirmed
Undecided
Unassigned

Bug Description

On Precise 12.04 whilst attempting:

GIT_CURL_VERBOSE=1 git clone -v https://git01.codeplex.com/typescript

the operation fails after the final git pack-file has been received and the already-created repository is deleted from the file system.

...
> POST /typescript/git-upload-pack HTTP/1.1
User-Agent: git/1.8.1.2.433.g9808ce0.dirty
Host: git01.codeplex.com
Accept-Encoding: gzip
Content-Type: application/x-git-upload-pack-request
Accept: application/x-git-upload-pack-result
Content-Length: 611

* upload completely sent off: 611out of 611 bytes
< HTTP/1.1 200 OK
< Cache-Control: no-cache, max-age=0, must-revalidate
< Pragma: no-cache
< Content-Type: application/x-git-upload-pack-result
< Expires: Fri, 01 Jan 1980 00:00:00 GMT
< Server: Microsoft-IIS/7.5
< X-Powered-By: ASP.NET
< Date: Thu, 31 Jan 2013 21:43:55 GMT
< Connection: close
<
remote: Counting objects: 149766, done.
remote: Compressing objects: 100% (10580/10580), done.
* GnuTLS recv error (-9): A TLS packet with unexpected length was received.
* Closing connection #0
remote: Total 149766 (delta 138201), reused 149559 (delta 138077)
Receiving objects: 100% (149766/149766), 198.98 MiB | 361 KiB/s, done.
error: RPC failed; result=56, HTTP code = 200
Resolving deltas: 100% (138201/138201), done.

git exits at this point but it deletes the entire cloned ./typescript directory.

I tried building the latest git binary and included an additional debug option in "http.c" that allowed me to set the protocol version using an environment option:

CURLOPT_SSLVERSION=1 git clone ...

where 1 = TLSv1, 2 = SSLv2, 3 = SSLv3.

I tried each protocol but the result was the same.

The knock-on bug here is that git ought not to delete what it has fetched - in this case more than 250MB of data.

I did try to build the latest gnutls but it needs a very recent version of libnettle which has the "rsa_decrypt_tr" function. I stopped at that point since I don't want to get into dependency and library version issues.

Revision history for this message
TJ (tj) wrote :

I've added the other packages that git depends upon since this might need to be dealt with higher up than libgnutls.

git < libcurl3-gnutls < libgnutls26

Revision history for this message
Anders Kaseorg (andersk) wrote :

Fascinating. I can reproduce, but I recommend you take this to the upstream Git mailing list, since they will more likely be able to help you figure out what’s going wrong.

http://git-scm.com/community

Changed in git (Ubuntu):
status: New → Confirmed
Revision history for this message
TJ (tj) wrote :

This is a gnutls issue; it could affect any application that makes use of it.

I've already mentioned it on the git developers mailing list and it has been seen once or twice before affecting git.

Additional research seems to indicate this is a known intentional gnutls behaviour (that has been modified in very recent gnutls that makes use of a recent libnettle - as mentioned above). The issue is, apparently, the random size padding of packets to prevent communications compromise for stream ciphers.

Unfortunately the changes required are far too invasive for an SRU so we'll have to make do with a work-around.

I installed stunnel4 (which depends on openssl rather than gnutls) and created a reverse-proxy (client in stunnel terminology):

$ cat /etc/stunnel/rp-codeplex.com.conf
client = yes

[http]
accept = 8888
connect = git01.codeplex.com:443
TIMEOUTclose = 0

$ sudo sed -i 's/\(ENABLED\).*/\1=1/' /etc/default/stunnel4
$ sudo service stunnel4 restart

$ GIT_CURL_VERBOSE=1 git clone -v http://localhost:8888/typescript

...
> POST http://localhost:8888/typescript/git-upload-pack HTTP/1.1
User-Agent: git/1.8.1.2.433.g9808ce0.dirty
Host: localhost:8888
Accept-Encoding: gzip
Proxy-Connection: Keep-Alive
Content-Type: application/x-git-upload-pack-request
Accept: application/x-git-upload-pack-result
Content-Length: 611

* upload completely sent off: 611out of 611 bytes
< HTTP/1.1 200 OK
< Cache-Control: no-cache, max-age=0, must-revalidate
< Pragma: no-cache
< Content-Type: application/x-git-upload-pack-result
< Expires: Fri, 01 Jan 1980 00:00:00 GMT
< Server: Microsoft-IIS/7.5
< X-Powered-By: ASP.NET
< Date: Thu, 31 Jan 2013 23:38:19 GMT
< Connection: close
<
remote: Counting objects: 149798, done.
remote: Compressing objects: 100% (10612/10612), done.
remote: Total 149798 (delta 138221), reused 149558 (delta 138077)
* Closing connection #0
Receiving objects: 100% (149798/149798), 198.99 MiB | 640 KiB/s, done.
Resolving deltas: 100% (138221/138221), done.
Checking out files: 100% (2851/2851), done.

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in curl (Ubuntu):
status: New → Confirmed
Changed in gnutls26 (Ubuntu):
status: New → Confirmed
Revision history for this message
Teräslilja (m-matti-a-lehtonen) wrote :

With "GIT push", I got
> * Done waiting for 100-continue
> < HTTP/1.1 100 Continue
> * GnuTLS recv error (-9): A TLS packet with unexpected length was received.
> * Closing connection 6
> PUT XXX failed, aborting (56/100)

Didn't fix after upgrading (from 13.04) to and recompiling following libraries
nettle-2.7.1
gnutls-3.2.3
curl-7.32.0
git version 1.8.3.4

Revision history for this message
Mirco Bauer (meebey) wrote :

This also failed for me like this:

git clone https://git01.codeplex.com/katanaproject
Cloning into 'katanaproject'...
remote: Counting objects: 21605, done.
remote: Compressing objects: 100% (3386/3386), done.
remote: Total 21605 (delta 17488), reused 21605 (delta 17488)
Receiving objects: 100% (21605/21605), 6.47 MiB | 913.00 KiB/s, done.
error: RPC failed; result=56, HTTP code = 200
Resolving deltas: 100% (17488/17488), done.

I could workaround this by doing:
mkdir katanaproject
cd katanaproject
git init
git remote add -f origin https://git01.codeplex.com/katanaproject
git fetch
git checkout -B master origin/master

Revision history for this message
Giuseppe Pasquino (giuseppe-pasquino) wrote :

This issue is still present in Ubuntu 14.04 LTS and git version 1.9.1

Revision history for this message
Wolverine (nareshtechs) wrote :

To clarify things, no the issue is not within GIT itself. The issue comes from libcurl3-openssl. Most of the GIT binaries that are distributed (either from GIT PPA or Ubuntu) is compiled to work with libcurl3-openssl.

Hence, it's necessary to recompile the GIT from source and link it to libcurl4-openssl. The issue can only be solved after Ubuntu or Git PPA is compiled against libcurl4-openssl instead of libcurl3-openssl.

Revision history for this message
Jonathan Elchison (jelchison) wrote :

Thanks, Wolverine.

I've confirmed that the workaround posted at http://askubuntu.com/a/187199 succeeds in the following environment:
* git 1.9.1-1
* Ubuntu 14.04 LTS on i686

For the record, the workaround I'm posting recompiles git with libcurl4-openssl-dev (instead of libcurl4-gnutls-dev).

Revision history for this message
thulle (thulle) wrote :

I think this bug also affects ngIRCd.
Connecting to a SSL-enabled ngIRCd using gnutls-cli fails on;
*** Fatal error: A TLS packet with unexpected length was received.
Connecting with openssl s_client fails with;
140615615354528:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:177:

Recompiling ngIRCd against libgnutls28-dev fixes the problem.

Revision history for this message
Anders Kaseorg (andersk) wrote :

Unfortunately, Git compiled with OpenSSL cannot be distributed for licensing reasons.

http://lintian.debian.org/tags/possible-gpl-code-linked-with-openssl.html
https://people.gnome.org/~markmc/openssl-and-the-gpl.html

So we will need to find some other solution.

Revision history for this message
Vindicator (vindicator) wrote :

I just encountered this issue with Ubuntu 15.04 64-bit while trying to clone nuget: https://nuget.codeplex.com/workitem/8339

Jeff (liujinfengjeff)
Changed in curl (Ubuntu):
status: Confirmed → In Progress
status: In Progress → Confirmed
Revision history for this message
Jeff (liujinfengjeff) wrote :

This also failed for me like this:
cd /home/jeff/jef
git clone https://github.com/torch/distro.git ~/torch --recursive

error: RPC failed; curl 56 GnuTLS recv error (-9): A TLS packet with unexpected length was received.
fatal: The remote end hung up unexpectedly
fatal: early EOF
fatal: index-pack failed
fatal: clone of 'https://github.com/torch/cutorch.git' into submodule path 'extra/cutorch' failed

last,I could workaround this by doing:
cd /home/jeff
git clone https://github.com/torch/distro.git ~/torch --recursive

and fix it. Sometimes because of network failure.I hope my answer can help you.

Revision history for this message
Jeff (liujinfengjeff) wrote :

Sometimes because of network failure,try more than once maybe you can fix it.

Revision history for this message
Joe Damato (q-joe) wrote :

This bug in gnutls also affects users of apt-transport-https which is used by APT for accessing repositories via TLS. Fixing this bug would be a great help for SSL-enabled APT repositories.

Revision history for this message
Julian Andres Klode (juliank) wrote :

Nothing to be done for apt here due to legal reasons. I'm marking this as invalid, but might just remove the task completely later on

Changed in apt (Ubuntu):
status: New → Invalid
Revision history for this message
David Ayers (ayers) wrote :

TJ stated in early 2013:

"Additional research seems to indicate this is a known intentional gnutls behaviour (that has been modified in very recent gnutls that makes use of a recent libnettle - as mentioned above). The issue is, apparently, the random size padding of packets to prevent communications compromise for stream ciphers.

Unfortunately the changes required are far too invasive for an SRU so we'll have to make do with a work-around."

Where do these changes need be implemented? In GNUTLS?
Has anyone with sufficient understanding opened an informed report upstream at https://gitlab.com/gnutls/gnutls/issues that could be referenced here?
If not, would someone be willing / capable to do that?

Thanks!

Revision history for this message
Lakshmi Kannan (lakshmi-9) wrote :

We are hitting this error roughly 1% of the times when running our CI.

Err:1 https://packagecloud.io/StackStorm/staging-enterprise-unstable/ubuntu xenial/main amd64 st2flow amd64 2.2dev-1
  GnuTLS recv error (-54): Error in the pull function.

Revision history for this message
Adam Voss (vossad01) wrote :

This issue has shown up on StackOverflow. The advice there is to rebuild git :(

http://stackoverflow.com/q/38378914/1072626

I doubt it will be useful but here is the verbose output from my failure: https://gist.github.com/vossad01/cb65180966f63eeb10c24b8ebf8957b8

Revision history for this message
Adam Voss (vossad01) wrote :

My case ended up not being this bug. I recompiled with OpenSSL (devopscube.com/gnutls-handshake-failed-aws-codecommit/) and then got the following error instead:

Writing objects: 100% (1103/1103), 1.09 MiB | 0 bytes/s, done.
Total 1103 (delta 653), reused 1001 (delta 606)
error: RPC failed; curl 56 SSL read: error:00000000:lib(0):func(0):reason(0), errno 104
fatal: The remote end hung up unexpectedly
fatal: The remote end hung up unexpectedly
Everything up-to-date

I am guessing it was because I was pushing so many commits (upstream changes). I deleted my fork on GitHub and recreated it so that it would be up-to-date, then had no problem pushing my changes.

Revision history for this message
Joe Damato (q-joe) wrote :

I've changed the status for APT back to "Confirmed" - I've seen many cases of this bug reported. I don't follow the comment about legal reasons (#17). Ubuntu already ships apt-transport-https, which links against gnutls and suffers from this bug; no new code causing any new legal issues would be pulled in. Fixing this bug in gnutls would also fix it in apt-transport-https, so IMO, APT is affected by this bug just the same as git and curl.

Changed in apt (Ubuntu):
status: Invalid → Confirmed
Revision history for this message
Julian Andres Klode (juliank) wrote :

While this does affect apt, there is no actionable item here, as we can't switch to OpenSSL for legal reasons, and won't switch to another more obscure TLS library. Fixing the GnuTLS bug is dealt with by the gnutls task.

Curl and Git are not my topic.

Changed in apt (Ubuntu):
status: Confirmed → Invalid
Revision history for this message
Julian Andres Klode (juliank) wrote :

That said, curl already has a built against openssl, so the problem is "solved" there. The gnutls variant is of course affected, but that's for GnuTLS to fix.

Changed in curl (Ubuntu):
status: Confirmed → Invalid
Revision history for this message
Joe Damato (q-joe) wrote :

OpenSSL was recently relicensed: https://www.openssl.org/blog/blog/2017/03/22/license/. I'm not a lawyer, but it would be nice to get one to weigh in on how this affects the legal issues mentioned.

Revision history for this message
Julian Andres Klode (juliank) wrote :

They are in the process of relicensing - asking for permissions. This might go on for some years or might never conclude.

Revision history for this message
Adrian Wilkins (adrian-wilkins) wrote :

Also seeing this with libgnutls30 and git 2.17.1-ubuntu0 on 18.04

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.