HAProxy 2.0.13 does not close connection even though "connection: close" is sent, leaves many connections in CLOSE-WAIT state for HEAD-method requests

This was created 20 days ago. When can I expect somebody to look into it?

Hello Malte, on a first read I don't think this is a security issue: if a client is responsible for adding a "connection: close" header to the messages, a malicious client could just as easily issue requests without this header, no?

Thanks

Hello Seth,

unfortunately, this is incorrect.

   HTTP/1.1 defines the "close" connection option for the sender to
   signal that the connection will be closed after completion of the
   response.

https://tools.ietf.org/html/rfc2616#section-14.10

But the server with Ubuntus HAProxy 2.0.13 will not proactively do it. Therefore, as described above, it is very easy for a malicious application to fill up the servers memory and connection pool.

With my minimal working example and a HAProxy docker container, you may easily and rather quickly be able to reproduce that. If this is expected behaviour, then I'm sorry for this ticket, which can be closed. (Migrated all my HAProxy based endpoints to a version not having this problem ;) )

Hello Malte, no, it sounds like you found a real bug that probably deserves to be fixed; I'm just not familiar enough with haproxy to know if this is crossing security boundaries.

Server team, any thoughts?

Thanks

Hello Malte, thanks for this bug report and for providing some minimal steps to reproduce it. I could reproduce the issue you described and verified that it doesn't happen when using vbernat's PPA, as you stated. I checked the haproxy changelog for the 2.0 branch [1] and its full git history [2] hoping to find the fixing change, but I had no luck, hwever I did spot a number of commits that could possibly be relevant. I also checked the GitHub issue tracker [3] and the past Debian bugs for the haproxy package [4] but again with no luck.

If we want to to find a minimal fix to this bug to SRU I think we'll have to:

 - bisect across releases [3] by manually compiling each;
 - bisect across commits of the first fixed release
 - hope that the commit(s) fixing this have a clear and
   limited (= SRUable) scope.

However I'm not sure this is the best way forward. Looking again at the changelog [1] basically all the changes would be nice to ship. I'll check with the team to decide how to proceed, for the moment I'm setting the status of this bug to Confirmed.

[1] http://www.haproxy.org/download/2.0/src/CHANGELOG
[2] http://git.haproxy.org/?p=haproxy-2.0.git
[3] https://github.com/haproxy/haproxy/issues
[4] https://bugs.debian.org/cgi-bin/pkgreport.cgi?archive=both;dist=unstable;package=haproxy
[5] https://www.haproxy.org/download/2.0/src/

@paride, I know of at least another problem affecting a Ubuntu user who reported to upstream [1]. Upstream have included the fix which should make it in 2.0.22. Once it is released, I'd be happy to work with you on having this new version shipped in Focal.

[1] https://github.com/haproxy/haproxy/issues/1197

I'm attaching a debdiff to have Focal upgraded from 2.0.13 to 2.0.22 (latest upstream 2.0 stable).

The attachment "lp1919468.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team.

[This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.]

Hi Simon,

Thanks for working on this! And wow, the debdiff looks huge but well, I don't expect it to be any smaller either, considering that it's a 2.0.13 -> 2.0.22. But well, I'd still copy the SRU team to take their opinion on this and to actually confirm whether they'd be OK with this being SRU'd to Focal or not.

Anyway, the upstream changelog looks good and honestly, I like the fact how they neatly put each commit categorized into subsets, making it very clear what that commit is about.

I could swing this by one of the members of the SRU team tonight and maybe once they are onboard with this, we could then workout the other things (backporting, testing, checking, uploading, et al.), hope that sounds OK to you?

Hello Utkarsh,

Yes, that sounds like a good plan, thanks. I forgot to mention, but aside from the changelog, they also maintain this wonderful list of fixed bugs: https://www.haproxy.org/bugs/bugs-2.0.13.html
Let me know if I can help with this SRU ;)

Thank you!

Hi Simon,

The server team discussed this bug and we do agree the debdiff in this case looks good, and having a point release of haproxy for the stable release likely would bring many good fixes to users in addition to this particular one, even though it doesn't appear that those issues have been reported against ubuntu in launchpad, yet.

That said, process-wise for getting a SRU accepted and released, it will assuredly be faster to do it with the specific fix you identified in comment #6 (i.e. https://github.com/FireBurn/haproxy/commit/c6eedcceef97f6a5a03e2b8b35e32ec31470483c.patch; attached), at least as a first step. Utkarsh is interested in taking a shot at solving this bug in this fashion.

If you're interested, the specific process we will need to follow for getting the full point release in, is documented at https://wiki.ubuntu.com/StableReleaseUpdates#New_upstream_microreleases. There are four key points listed there that we will need to demonstrate proof for. Along with that, knowing how to reliably reproduce some of the other bugs mentioned in the v2.0.13 release notes on Ubuntu 20.04 would help in making the justification for its acceptance.

Utkarsh, a few next steps:

a. Hone the test case in the bug description to a good 'paint-by-numbers' level of detail

b. Identify if other Ubuntu releases in addition to focal will need this fix. We have 2.2.x in groovy and hirsute, so presumably it already has the fix, but doublecheck in code. Bionic is back on 1.8.8, but worth checking if it is similarly affected (maybe via the test case from (a)). Update the bug report adding the affected series.

c. Using the test case, verify the issue exists without the patch and is resolved with the patch, for the affected series, and carry on the usual SRU process from there.

Right, thanks, Bryce! \o/

Simon, hey, I mean to ask, do you intend to work on the SRU (just this fix for now) as well? I could help you land that patch if you're interested and if not, I could just go ahead and do it. And then later, we discuss the microrelease route in a bit detail and use your debdiff to then proceed from there. Let me know what you think or ask away if you have any questions! :D

@Utkarsh, I'm not affected by this bug. I am interested in the MRE to
fix the other 305 bugs waiting to bite me or others ;)

Per server-triage-discussion: this *may( be a case where a "new upstream microrelease" is acceptable, but the requirements are to be verified:

https://wiki.ubuntu.com/StableReleaseUpdates#New_upstream_microreleases