HAProxy init script does not work correctly with nbproc configuration option

Bug #1481737 reported by Peter G. Marczis
36
This bug affects 7 people
Affects Status Importance Assigned to Milestone
haproxy (Ubuntu)
Fix Released
Medium
James Page
Trusty
Fix Released
Medium
Unassigned
Vivid
Won't Fix
Medium
Louis Bouchard
Wily
Fix Released
Medium
James Page
Xenial
Fix Released
Medium
James Page

Bug Description

[SRU justification]
The sysVinit script does not completely stop processes when nbproc > 1

[Impact]
Stopping haproxy without this fix leaves all but one process running

[Fix]
Use temporary pidfile in a loop containing one single PID as the --pid option is not available for start-stop-daemon.

[Test Case]
Set nbproc > 4 in /etc/haproxy/haproxy.cfg
resart haproxy & verify that there is more than one haproxy process
Run :
$ ps aux | grep haproxy | grep -v grep | wc -l
4
$ service haproxy stop
$ ps aux | grep haproxy | grep -v grep | wc -l

Without the patch, the result of the last command will be 3. With the fix, it will be 0

[Regression]
The regression risk is in the use of a temporary file instead of the start-stop-daemon option.

It has been tested with TMPDIR="/tmp/My Dir" to check for spaces in directory names.

[Original description of the problem]
In case you have more than one process enabled in the haproxy config, the init script fails to stop all of those processes.
So to clarify you need to have this in your haproxy.cfg:

"""
...
global
        maxconn 32000
        ulimit-n 65536
        user haproxy
        group haproxy
        nbproc 2

...
"""

the problem is more visible if you set the nbproc to higher number.

service haproxy stop --> will only stop on of the haproxy processes.

The problem is so that start-stop-daemon can't handle pid files with multiple lines. Only stopping the first one. HAProxy does write all the pids started by itself into the pid file, so the problem is not in HAProxy, but in the start-stop-daemon or more likely in the init scrip of haproxy.
One solution or workaround is to remove pidfile option of start-stop-daemon in the init script, than it wil work as killall and will stop the haproxy processes properly.

To proof you can try this sequence:

service haproxy start #if its not running
ps ax | grep haproxy | grep -v grep | wc -l #this should report 2
service haproxy restart
ps ax | grep haproxy | grep -v grep | wc -l #this will report 3

The workaround as a diff:
root@ubi1:/opt# diff /etc/init.d/haproxy /etc/init.d/haproxy.orig
62c62
< --retry 5 --exec $HAPROXY || ret=$?
---
> --retry 5 --pidfile $PIDFILE --exec $HAPROXY || ret=$?

extra infos:
root@ubi1:/opt# lsb_release -rd
Description: Ubuntu 14.04.3 LTS
Release: 14.04

root@ubi1:/opt# apt-cache policy haproxy
haproxy:
  Installed: 1.4.24-2ubuntu0.2
  Candidate: 1.4.24-2ubuntu0.2
  Version table:
     1.5.3-1~ubuntu14.04.1 0
        100 http://us.archive.ubuntu.com/ubuntu/ trusty-backports/main amd64 Packages
 *** 1.4.24-2ubuntu0.2 0
        500 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     1.4.24-2 0
        500 http://us.archive.ubuntu.com/ubuntu/ trusty/main amd64 Packages

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in haproxy (Ubuntu):
status: New → Confirmed
Revision history for this message
Peter G. Marczis (peter-marczis) wrote :

Any additional info needed ?
I'm happy to help with anything !
Should be an upstart config for haproxy ? Or you plan to shift away from upstart ? (I remember to read that somewhere...)
Br,
 Peter.

Revision history for this message
Robie Basak (racb) wrote :

Thank you for taking the time to report this bug and helping to make Ubuntu better.

We're in sync with Debian on the haproxy package so this bug should be reported there. For Ubuntu, managing haproxy with systemd units would presumably be easiest; this could also be contributed to Debian.

Workaround: change /etc/init.d/haproxy as you described. Importance -> Medium since a workaround is available.

tags: added: needs-upstream-report
summary: - HAProxy init script is not working properly
+ HAProxy init script does not work correctly with nbproc configuration
+ option
Revision history for this message
Peter G. Marczis (peter-marczis) wrote :

Is it really so that I should report the bug to debian via email ?
https://www.debian.org/Bugs/Reporting

Strange... please confirm.
Br,
 Peter.

Revision history for this message
Robie Basak (racb) wrote :

> Is it really so that I should report the bug to debian via email ?

Assuming the bug does reproduce on Debian and there isn't a bug filed there already, then yes - someone needs to in order to drive this bug towards a resolution. If you don't have time for this then that's fine - this bug can remain open until someone else gets to it.

Revision history for this message
Peter G. Marczis (peter-marczis) wrote :

No, no, I really want to get my hands dirty. Than first of all I will try to reproduce on Debian, and then check with them how it goes... In fact, should I use the newest mainstream version of Debian ? Or do you have any kind of mappings between Debian and Ubuntu releases ?

Revision history for this message
Louis Bouchard (louis) wrote :

Peter, you will not be able to reproduce on Debian unless you test versions prior to Stable.

The reason why the stop script does not stop at all processes is because it relied on the --pidfile option and this one only kills the first PID. When nproc > 1, the PIDFILE contains all PIDs. This is why debian introduced the use of --pid in a loop for all haproxy PIDs.

Now, the --pid option that is introduced in dpkg version 1.17.6 as outlined in the man page of start-stop-daemon :

       [--pid] pid
              Check for a process with the specified pid (since version 1.17.6). The pid must be a number greater than 0.

This version is available in all Debian releases since Stable but only available in Ubuntu Wily. A fix for this situation is available for haproxy 1.5 which is in trusty-backport (see LP: #1494141). I will backport that fix for the stable release and ask for an SRU on this one.

Revision history for this message
Peter G. Marczis (peter-marczis) wrote :

Hi Louis !
Yes, I have the same understanding of the problem ! So if the fix is available, and you do port it, I guess you can close the ticket :)
Thanks a lot for you work and effort !
Br,
 Peter.

Louis Bouchard (louis)
Changed in haproxy (Ubuntu Trusty):
importance: Undecided → Medium
status: New → In Progress
assignee: nobody → Louis Bouchard (louis-bouchard)
Changed in haproxy (Ubuntu):
status: Confirmed → Invalid
Louis Bouchard (louis)
description: updated
Revision history for this message
Louis Bouchard (louis) wrote :
Revision history for this message
Mathieu Trudel-Lapierre (cyphermox) wrote :

sponsored.

Mathew Hodson (mhodson)
tags: added: trusty
removed: haproxy
Changed in haproxy (Ubuntu):
status: Invalid → Fix Released
importance: Undecided → Medium
Mathew Hodson (mhodson)
tags: removed: needs-upstream-report
Louis Bouchard (louis)
Changed in haproxy (Ubuntu Vivid):
assignee: nobody → Louis Bouchard (louis-bouchard)
importance: Undecided → Medium
status: New → In Progress
Revision history for this message
James Page (james-page) wrote :

Please can we ensure that this is actually fixed in the current development release as well; we're backporting haproxy to 14.04 as part of the Ubuntu Cloud Archive, so having backwards compat in the latest packages with 14.04 is important.

Revision history for this message
James Page (james-page) wrote :

Related bug 1477198

Revision history for this message
James Page (james-page) wrote :

There's a small niggle in the proposed change for trusty; the 'clean' exit handling will override the exit code from the init script, resulting in the return code being 0 all of the time; dropping the exit from within 'clean' will resolve this.

I've uploaded Louis' fix with this small improvement to xenial

Changed in haproxy (Ubuntu Xenial):
status: Fix Released → In Progress
Changed in haproxy (Ubuntu Wily):
importance: Undecided → Medium
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package haproxy - 1.6.2-2ubuntu3

---------------
haproxy (1.6.2-2ubuntu3) xenial; urgency=medium

  * d/haproxy.init: Ensure that cleanup does not override exit status
    of init script.

 -- James Page <email address hidden> Mon, 09 Nov 2015 14:35:15 +0000

Changed in haproxy (Ubuntu Xenial):
status: In Progress → Fix Released
James Page (james-page)
Changed in haproxy (Ubuntu Wily):
status: New → In Progress
Changed in haproxy (Ubuntu Xenial):
assignee: nobody → James Page (james-page)
Changed in haproxy (Ubuntu Wily):
assignee: nobody → James Page (james-page)
Revision history for this message
Chris J Arges (arges) wrote : Please test proposed package

Hello Peter, or anyone else affected,

Accepted haproxy into wily-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/haproxy/1.5.14-1ubuntu0.15.10.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in haproxy (Ubuntu Wily):
status: In Progress → Fix Committed
tags: added: verification-needed
James Page (james-page)
tags: added: verification-done
removed: verification-needed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package haproxy - 1.5.14-1ubuntu0.15.10.1

---------------
haproxy (1.5.14-1ubuntu0.15.10.1) wily; urgency=medium

  * Ensure that haproxy processes are terminated correctly when executing
    stop/restart operations, easing backports to pre-systemd versions of
    Ubuntu (LP: #1477198, #1481737).

 -- James Page <email address hidden> Mon, 09 Nov 2015 16:51:46 +0000

Changed in haproxy (Ubuntu Wily):
status: Fix Committed → Fix Released
Revision history for this message
Chris J Arges (arges) wrote : Update Released

The verification of the Stable Release Update for haproxy has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Chris J Arges (arges) wrote : Please test proposed package

Hello Peter, or anyone else affected,

Accepted haproxy into trusty-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/haproxy/1.4.24-2ubuntu0.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in haproxy (Ubuntu Trusty):
status: In Progress → Fix Committed
tags: removed: verification-done
tags: added: verification-needed
Revision history for this message
Chris J Arges (arges) wrote :

Can someone post an SRU for vivid? Thanks

Louis Bouchard (louis)
tags: added: verification-done
removed: verification-needed
Changed in haproxy (Ubuntu Trusty):
assignee: Louis Bouchard (louis-bouchard) → nobody
Revision history for this message
Louis Bouchard (louis) wrote :

ok, I'm on the vivid SRU

Revision history for this message
Louis Bouchard (louis) wrote :

debdiff for vivid SRU

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package haproxy - 1.4.24-2ubuntu0.3

---------------
haproxy (1.4.24-2ubuntu0.3) trusty; urgency=medium

  * debian/haproxy.init:
    - Loops through all PIDs present in the PIDFILE when
      nproc > 1 (LP: #1481737)

 -- Louis Bouchard <email address hidden> Wed, 23 Sep 2015 17:45:41 +0200

Changed in haproxy (Ubuntu Trusty):
status: Fix Committed → Fix Released
Revision history for this message
Chris J Arges (arges) wrote :

Sponsored for vivid

Revision history for this message
Brian Murray (brian-murray) wrote :

Hello Peter, or anyone else affected,

Accepted haproxy into vivid-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/haproxy/1.5.10-1ubuntu0.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in haproxy (Ubuntu Vivid):
status: In Progress → Fix Committed
tags: removed: verification-done
tags: added: verification-needed
Revision history for this message
Ubuntu Foundations Team Bug Bot (crichton) wrote : [haproxy/vivid] verification still needed

The fix for this bug has been awaiting testing feedback in the -proposed repository for vivid for more than 90 days. Please test this fix and update the bug appropriately with the results. In the event that the fix for this bug is still not verified 15 days from now, the package will be removed from the -proposed repository.

tags: added: removal-candidate
Revision history for this message
Martin Pitt (pitti) wrote : Proposed package removed from archive

The version of haproxy in the proposed pocket of Vivid that was purported to fix this bug report has been removed because the bugs that were to be fixed by the upload were not verified in a timely (105 days) fashion.

Changed in haproxy (Ubuntu Vivid):
status: Fix Committed → Won't Fix
tags: removed: verification-needed
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.