Change haproxy health check from TCP to HTTP

It hits code freeze in 6.0. Moving to 6.1

The issue is pretty critical and better to be kept in 6.0. Having one of 3 servers in hanging state affects service availability and this needs to be improved.

Here's how it was done for radosgw:
https://review.openstack.org/#/c/130885/2/deployment/puppet/openstack/manifests/ha/radosgw.pp

Currently only 'horizon' and 'mysql' backends use L7 checks.

For the following backends:

cinder-api
glance-api
heat-api
heat-api-cfn
heat-api-cloudwatch
keystone-1
keystone-2
neutron
nova-api-2
nova-metadata-api

it could be enabled with the following options right away (tested on live Juno env):

option => ['httpchk', 'httplog','httpclose'],

In addition I suggest to use these options to lower HTTP check rate (default interval is 2s):

balancermember_options => inter 10s fastinter 2s downinter 3s rise 3 fall 3

Fix proposed to branch: master
Review: https://review.openstack.org/136298

Reviewed: https://review.openstack.org/136298
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=539460f99cba534363650d6fa322cb0bb252d39e
Submitter: Jenkins
Branch: master

commit 539460f99cba534363650d6fa322cb0bb252d39e
Author: Aleksandr Didenko <email address hidden>
Date: Fri Nov 21 12:45:22 2014 +0200

    Change haproxy health check from TCP to HTTP

    Switch from L4 healtch checks to L7 where possible.

    Change-Id: Ie2523d05a6a52cdac08b0d8c70c68642f64bcb8f
    Closes-bug: #1394195

This issue also affects 5.1.x, I reproduced it during performance testing: under high load keystone on one controller crashed and begun to return 504 error. But HAProxy still continued to send requests to it:

root@node-2:~# echo "show stat" | nc -U /var/lib/haproxy/stats | awk -F',' '{if (NR == 1 || $1 ~/keystone/){printf "%14s %14s %10s %10s %10s %10s %10s %10s %10s %10s %10s\n", $1,$2,$8,$13,$14,$15,$18,$22,$23,$25,$28}}'
      # pxname svname stot ereq econ eresp status chkfail chkdown downtime iid
    keystone-1 FRONTEND 12898 0 OPEN 3
    keystone-1 node-2 4308 0 769 UP 0 0 0 3
    keystone-1 node-3 4295 0 0 UP 0 1 809 3
    keystone-1 node-6 4295 0 0 UP 0 1 817 3
    keystone-1 BACKEND 12898 0 769 UP 0 0 3
    keystone-2 FRONTEND 135668 0 OPEN 4
    keystone-2 node-2 44786 0 8305 UP 147 16 191 4
    keystone-2 node-3 46007 0 6 UP 0 1 807 4
    keystone-2 node-6 46011 0 8 UP 0 1 817 4
    keystone-2 BACKEND 135668 0 8319 UP 0 0 4

So lots of attempts to create new instances failed:

2014-12-02 17:52:14.069 20474 INFO rally.benchmark.runners.base [-] Task b40d77e3-e267-4fa2-9bdd-9f10ccb9c68e | ITER: 846 END: Error GetResourceFailure: Failed to get the resource <Server: rally_novaserver_jpxlzicqprsefvsy>: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-6829db25-b851-44db-aa6b-2f37dbb890c3)
2014-12-02 17:52:14.069 20474 INFO rally.benchmark.runners.base [-] Task b40d77e3-e267-4fa2-9bdd-9f10ccb9c68e | ITER: 923 START
2014-12-02 17:52:14.247 20476 INFO rally.benchmark.runners.base [-] Task b40d77e3-e267-4fa2-9bdd-9f10ccb9c68e | ITER: 818 END: Error GatewayTimeout: Gateway Timeout (HTTP 504)

Fix proposed to branch: stable/5.1
Review: https://review.openstack.org/138713

Reviewed: https://review.openstack.org/138713
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=bc49271672643d79bb61929897d936c8e98615dd
Submitter: Jenkins
Branch: stable/5.1

commit bc49271672643d79bb61929897d936c8e98615dd
Author: Aleksandr Didenko <email address hidden>
Date: Fri Nov 21 12:45:22 2014 +0200

    Change haproxy health check from TCP to HTTP

    Switch from L4 healtch checks to L7 where possible.

    Change-Id: Ie2523d05a6a52cdac08b0d8c70c68642f64bcb8f
    Closes-bug: #1394195