HAProxy 2.0.13 does not close connection even though "connection: close" is sent, leaves many connections in CLOSE-WAIT state for HEAD-method requests

Bug #1919468 reported by Malte Schmidt
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
haproxy (Ubuntu)
Fix Released
Undecided
Unassigned
Focal
Confirmed
Medium
Unassigned
Groovy
Fix Released
Undecided
Unassigned
Hirsute
Fix Released
Undecided
Unassigned
Impish
Fix Released
Undecided
Unassigned

Bug Description

I figured that the current latest HAProxy (2.0.13-2ubuntu0.1) for focal has a problem closing connections, even if "connection: close" is being sent, the connection is not closed afterwards. The problem does not exist if "no option http-use-htx" is set (disabling HTX). The problem only affects the HEAD-method.

Using 2.0.20 (2.0.20-1ppa1~focal, https://haproxy.debian.net/) of vbernat's Debian HAProxy repo, this works flawless with and without HTX.

I marked this as security relevant because I believe that this bug can be used for DoS, just like Slowloris, but much easier, because no effort is required to keep the connection open.

Here is a minimal (working with 2.0.20-1ppa1~focal, not working with 2.0.13-2ubuntu0.1) example cfg:

====
global
        user haproxy
        group haproxy
        daemon

frontend test
        bind *:80
        mode http
====

Here is a minimal working example cfg:

====
global
        user haproxy
        group haproxy
        daemon

frontend test
        bind *:80
        mode http
        no option http-use-htx
====

Testing can be done multiple ways:

curl will not close the connection:

====
curl -v -XHEAD http://10.0.0.1/
====

curl will close the connection:

====
curl -v -I http://10.0.0.1/
====

To exclude any curl related problems:

====
telnet 10.0.0.1 80
HEAD / HTTP/1.1
====

To summarize:

I expect to happen: Connections get closed proactively by server if "connection: close" header is sent.
This happens: Connections stay open and in CLOSE-WAIT on server.

Tags: patch haproxy

Related branches

Malte Schmidt (maltris)
description: updated
description: updated
Revision history for this message
Malte Schmidt (maltris) wrote :

This was created 20 days ago. When can I expect somebody to look into it?

information type: Private Security → Public Security
Revision history for this message
Seth Arnold (seth-arnold) wrote :

Hello Malte, on a first read I don't think this is a security issue: if a client is responsible for adding a "connection: close" header to the messages, a malicious client could just as easily issue requests without this header, no?

Thanks

Revision history for this message
Malte Schmidt (maltris) wrote :

Hello Seth,

unfortunately, this is incorrect.

   HTTP/1.1 defines the "close" connection option for the sender to
   signal that the connection will be closed after completion of the
   response.

https://tools.ietf.org/html/rfc2616#section-14.10

But the server with Ubuntus HAProxy 2.0.13 will not proactively do it. Therefore, as described above, it is very easy for a malicious application to fill up the servers memory and connection pool.

With my minimal working example and a HAProxy docker container, you may easily and rather quickly be able to reproduce that. If this is expected behaviour, then I'm sorry for this ticket, which can be closed. (Migrated all my HAProxy based endpoints to a version not having this problem ;) )

Revision history for this message
Seth Arnold (seth-arnold) wrote :

Hello Malte, no, it sounds like you found a real bug that probably deserves to be fixed; I'm just not familiar enough with haproxy to know if this is crossing security boundaries.

Server team, any thoughts?

Thanks

Revision history for this message
Paride Legovini (paride) wrote :

Hello Malte, thanks for this bug report and for providing some minimal steps to reproduce it. I could reproduce the issue you described and verified that it doesn't happen when using vbernat's PPA, as you stated. I checked the haproxy changelog for the 2.0 branch [1] and its full git history [2] hoping to find the fixing change, but I had no luck, hwever I did spot a number of commits that could possibly be relevant. I also checked the GitHub issue tracker [3] and the past Debian bugs for the haproxy package [4] but again with no luck.

If we want to to find a minimal fix to this bug to SRU I think we'll have to:

 - bisect across releases [3] by manually compiling each;
 - bisect across commits of the first fixed release
 - hope that the commit(s) fixing this have a clear and
   limited (= SRUable) scope.

However I'm not sure this is the best way forward. Looking again at the changelog [1] basically all the changes would be nice to ship. I'll check with the team to decide how to proceed, for the moment I'm setting the status of this bug to Confirmed.

[1] http://www.haproxy.org/download/2.0/src/CHANGELOG
[2] http://git.haproxy.org/?p=haproxy-2.0.git
[3] https://github.com/haproxy/haproxy/issues
[4] https://bugs.debian.org/cgi-bin/pkgreport.cgi?archive=both;dist=unstable;package=haproxy
[5] https://www.haproxy.org/download/2.0/src/

Changed in haproxy (Ubuntu):
status: New → Confirmed
tags: added: server-triage-discuss
Revision history for this message
Simon Déziel (sdeziel) wrote :

@paride, I know of at least another problem affecting a Ubuntu user who reported to upstream [1]. Upstream have included the fix which should make it in 2.0.22. Once it is released, I'd be happy to work with you on having this new version shipped in Focal.

[1] https://github.com/haproxy/haproxy/issues/1197

Revision history for this message
Simon Déziel (sdeziel) wrote :

I'm attaching a debdiff to have Focal upgraded from 2.0.13 to 2.0.22 (latest upstream 2.0 stable).

Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote :

The attachment "lp1919468.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

tags: added: patch
Revision history for this message
Utkarsh Gupta (utkarsh) wrote :

Hi Simon,

Thanks for working on this! And wow, the debdiff looks huge but well, I don't expect it to be any smaller either, considering that it's a 2.0.13 -> 2.0.22. But well, I'd still copy the SRU team to take their opinion on this and to actually confirm whether they'd be OK with this being SRU'd to Focal or not.

Anyway, the upstream changelog looks good and honestly, I like the fact how they neatly put each commit categorized into subsets, making it very clear what that commit is about.

I could swing this by one of the members of the SRU team tonight and maybe once they are onboard with this, we could then workout the other things (backporting, testing, checking, uploading, et al.), hope that sounds OK to you?

Revision history for this message
Simon Déziel (sdeziel) wrote :

Hello Utkarsh,

Yes, that sounds like a good plan, thanks. I forgot to mention, but aside from the changelog, they also maintain this wonderful list of fixed bugs: https://www.haproxy.org/bugs/bugs-2.0.13.html
Let me know if I can help with this SRU ;)

Thank you!

Revision history for this message
Bryce Harrington (bryce) wrote :

Hi Simon,

The server team discussed this bug and we do agree the debdiff in this case looks good, and having a point release of haproxy for the stable release likely would bring many good fixes to users in addition to this particular one, even though it doesn't appear that those issues have been reported against ubuntu in launchpad, yet.

That said, process-wise for getting a SRU accepted and released, it will assuredly be faster to do it with the specific fix you identified in comment #6 (i.e. https://github.com/FireBurn/haproxy/commit/c6eedcceef97f6a5a03e2b8b35e32ec31470483c.patch; attached), at least as a first step. Utkarsh is interested in taking a shot at solving this bug in this fashion.

If you're interested, the specific process we will need to follow for getting the full point release in, is documented at https://wiki.ubuntu.com/StableReleaseUpdates#New_upstream_microreleases. There are four key points listed there that we will need to demonstrate proof for. Along with that, knowing how to reliably reproduce some of the other bugs mentioned in the v2.0.13 release notes on Ubuntu 20.04 would help in making the justification for its acceptance.

Changed in haproxy (Ubuntu):
assignee: nobody → Utkarsh Gupta (utkarsh)
Revision history for this message
Bryce Harrington (bryce) wrote :

Utkarsh, a few next steps:

a. Hone the test case in the bug description to a good 'paint-by-numbers' level of detail

b. Identify if other Ubuntu releases in addition to focal will need this fix. We have 2.2.x in groovy and hirsute, so presumably it already has the fix, but doublecheck in code. Bionic is back on 1.8.8, but worth checking if it is similarly affected (maybe via the test case from (a)). Update the bug report adding the affected series.

c. Using the test case, verify the issue exists without the patch and is resolved with the patch, for the affected series, and carry on the usual SRU process from there.

Revision history for this message
Utkarsh Gupta (utkarsh) wrote :

Right, thanks, Bryce! \o/

Simon, hey, I mean to ask, do you intend to work on the SRU (just this fix for now) as well? I could help you land that patch if you're interested and if not, I could just go ahead and do it. And then later, we discuss the microrelease route in a bit detail and use your debdiff to then proceed from there. Let me know what you think or ask away if you have any questions! :D

Revision history for this message
Simon Déziel (sdeziel) wrote : Re: [Bug 1919468] Re: HAProxy 2.0.13 does not close connection even though "connection: close" is sent, leaves many connections in CLOSE-WAIT state for HEAD-method requests

@Utkarsh, I'm not affected by this bug. I am interested in the MRE to
fix the other 305 bugs waiting to bite me or others ;)

Mathew Hodson (mhodson)
information type: Public Security → Public
Changed in haproxy (Ubuntu):
importance: Undecided → Medium
Utkarsh Gupta (utkarsh)
Changed in haproxy (Ubuntu Focal):
assignee: nobody → Utkarsh Gupta (utkarsh)
Changed in haproxy (Ubuntu Groovy):
status: New → Fix Released
Changed in haproxy (Ubuntu Hirsute):
status: New → Fix Released
Changed in haproxy (Ubuntu Focal):
status: New → Confirmed
Changed in haproxy (Ubuntu Impish):
status: Confirmed → Fix Released
Changed in haproxy (Ubuntu Focal):
importance: Undecided → Medium
Changed in haproxy (Ubuntu Impish):
importance: Medium → Undecided
assignee: Utkarsh Gupta (utkarsh) → nobody
Revision history for this message
Paride Legovini (paride) wrote :

Per server-triage-discussion: this *may( be a case where a "new upstream microrelease" is acceptable, but the requirements are to be verified:

https://wiki.ubuntu.com/StableReleaseUpdates#New_upstream_microreleases

tags: removed: server-triage-discuss
Utkarsh Gupta (utkarsh)
Changed in haproxy (Ubuntu Focal):
assignee: Utkarsh Gupta (utkarsh) → nobody
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.