Gnocchi Charm

layer 7 haproxy backend checks

Bug #1880610 reported by Andrea Ieri
16
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Gnocchi Charm
Fix Committed
Undecided
Gabriel Cocenza
OpenStack Charm Guide
Triaged
High
Unassigned
OpenStack Cinder Charm
Fix Committed
Undecided
Gabriel Cocenza
OpenStack Designate Charm
Fix Committed
Undecided
Gabriel Cocenza
OpenStack Glance Charm
Fix Committed
Undecided
Gabriel Cocenza
OpenStack HA Cluster Charm
Invalid
Undecided
Unassigned
OpenStack Heat Charm
Fix Committed
Undecided
Gabriel Cocenza
OpenStack Ironic API Charm
Fix Committed
Undecided
Gabriel Cocenza
OpenStack Keystone Charm
Fix Committed
Undecided
Gabriel Cocenza
OpenStack Neutron API Charm
Fix Committed
Undecided
Gabriel Cocenza
OpenStack Nova Cloud Controller Charm
Fix Committed
Undecided
Gabriel Cocenza
OpenStack Swift Proxy Charm
Fix Committed
Undecided
Gabriel Cocenza

Bug Description

The current haproxy config adds health checks for all backends. These are however the default TCP checks[0], which succeed simply if a connection to the specified port is possible, and don't go any further in trying to assess if the backend service is actually healthy.

In some partial failure scenarios - e.g. extreme load or memory pressure - TCP checks may succeed where more thorough ones would not, causing API traffic to be routed to backends that aren't actually able to process it.

HAproxy offers the possibility of reconfiguring the layer 4 checks into layer 7 ones by using the httpchk option[1]. This is still fairly basic, but configuring the http-check option[4] makes the check a bit more accurate, while service-specific checks are available for some backends such as mysql[2] and postgres[3]. Finally, external scripts[5] can be invoked for a fully customized check routine.

In order to improve the control plane resilience we should gradually switch to more advanced backend checks. Plain TCP checks should remain the default, but principal charms should provide more accurate check specifications via their relation to hacluster. Depending on the service, a properly configured httpchk may suffice, while in some cases a fully custom script may be more appropriate.

This is not in contrast with giving more control to pacemaker via openstack resource agents, as described in bug 1880611.

[0] https://www.haproxy.com/blog/the-four-essential-sections-of-an-haproxy-configuration/
[1] https://www.haproxy.com/documentation/hapee/latest/onepage/#4-option%20httpchk
[2] https://www.haproxy.com/documentation/hapee/latest/onepage/#4.2-option%20mysql-check
[3] https://www.haproxy.com/documentation/hapee/latest/onepage/#4.2-option%20pgsql-check
[4] https://www.haproxy.com/documentation/hapee/latest/onepage/#4.2-http-check%20expect
[5] https://www.haproxy.com/documentation/hapee/latest/onepage/#external-check%20command

See original description
Tags: bseng-436
Andrea Ieri (aieri)
affects: charm-designate → charm-hacluster
description: updated
Jose Guedez (jfguedez)
Changed in charm-hacluster:
assignee: nobody → Jose Guedez (jfguedez)
Jose Guedez (jfguedez)
Changed in charm-glance:
assignee: nobody → Jose Guedez (jfguedez)
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-glance (master)

Fix proposed to branch: master
Review: https://review.opendev.org/750224

Revision history for this message
Jose Guedez (jfguedez) wrote :

Marking invalid for 'hacluster' as it doesn't handle haproxy configurations (this is handled by the principal charms)

Changed in charm-hacluster:
status: New → Invalid
Revision history for this message
Jose Guedez (jfguedez) wrote :

commenting to send notification to field:medium

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on charm-glance (master)

Change abandoned by "James Page <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/charm-glance/+/750224
Reason: This review is > 12 weeks without comment, and failed testing the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

Andrea Ieri (aieri)
Changed in charm-hacluster:
assignee: Jose Guedez (jfguedez) → nobody
Changed in charm-glance:
assignee: Jose Guedez (jfguedez) → nobody
Revision history for this message
Andrea Ieri (aieri) wrote :

for the record this bug affects all principals using hacluster, so the bug may be in a shared library

tags: added: bseng-436
Revision history for this message
Jose Guedez (jfguedez) wrote :

We had another event when having the L7 checks would have been helpful. Since the balancing scheme is "leastconn" [0], having a bad backend that is not completely dead (i.e. accepts network connections, but just never replies until a timeout is reached), will lead to the situation of new connections to the loadbalancer being pointed to a "broken" backend.

We have recently seen this in production clouds using the gnocchi charm, but should be similar for other services (principal charms). In this particular case the requests were being continuously pointed to the unresponsive backend, even though the others were healthy (just 1 out of 3 backends was unhealthy). Since the unhealthy gnocchi backend service was still accepting network connections, haproxy continued sending it requests. Pausing the bad backend restored service to clients while the problem with the unhealthy backend was resolved.

Having a L7 check would make haproxy realise that a backend is unhealthy quickly and requests would be then be routed to the other 2 healthy backends. This would be similar to how readiness checks in Kubernetes (typically L7), would prevent this situation from happening.

[0] https://github.com/juju/charm-helpers/blob/master/charmhelpers/contrib/openstack/templates/haproxy.cfg#L79

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-glance (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-glance/+/862521

Gabriel Cocenza (gabrielcocenza)
Changed in charm-glance:
assignee: nobody → Gabriel Angelo Sgarbi Cocenza (gabrielcocenza)
Revision history for this message
Gabriel Cocenza (gabrielcocenza) wrote :

I've opened two PR against charm-layer-openstack-api[0] and charm-layer-openstack[1] in order to give layer 7 haproxy backend checks on reactive openstack-api charms

[0] https://review.opendev.org/c/openstack/charm-layer-openstack-api/+/862750
[1] https://review.opendev.org/c/openstack/charm-layer-openstack/+/862752

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-glance (master)

Reviewed: https://review.opendev.org/c/openstack/charm-glance/+/862521
Committed: https://opendev.org/openstack/charm-glance/commit/96a2bb3af9a9b3b5756e5d618371187fa6e09b8f
Submitter: "Zuul (22348)"
Branch: master

commit 96a2bb3af9a9b3b5756e5d618371187fa6e09b8f
Author: Gabriel Cocenza <email address hidden>
Date: Mon Oct 24 12:04:05 2022 -0300

    Add support for HAProxy L7 checks

    This change add several configuration options to enable HTTP checks
    to the HAProxy configuration, instead of the default TCP connection
    checks (which continue to be the default)

    Closes-Bug: #1880610
    Change-Id: I81d7fd67029dd5025f95b41d788b03ce4b6038bb

Changed in charm-glance:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-gnocchi (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-gnocchi/+/874030

Changed in charm-gnocchi:
status: New → In Progress
Gabriel Cocenza (gabrielcocenza)
Changed in charm-gnocchi:
assignee: nobody → Gabriel Angelo Sgarbi Cocenza (gabrielcocenza)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-keystone (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-keystone/+/874035

Changed in charm-keystone:
status: New → In Progress
Gabriel Cocenza (gabrielcocenza)
Changed in charm-keystone:
assignee: nobody → Gabriel Angelo Sgarbi Cocenza (gabrielcocenza)
Revision history for this message
Gabriel Cocenza (gabrielcocenza) wrote :

charm-ceilometer removed haproxy since ocata, so it doesn't make sense to add layer 7 check to this charm.

Changed in charm-ceilometer:
status: New → Invalid
Revision history for this message
Gabriel Cocenza (gabrielcocenza) wrote :

charm-openstack-dashboard has its own haproxy.cfg template that it's not from charmhelpers.
The template uses `listen` for dash_insecure and dash_secure and as the documentation says[0]:

A "listen" section defines a complete proxy with its frontend and backend
parts combined in one section. It is generally useful for TCP-only traffic.

[0] http://cbonte.github.io/haproxy-dconv/1.6/configuration.html#4

Changed in charm-openstack-dashboard:
status: New → Invalid
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-cinder (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-cinder/+/874177

Changed in charm-cinder:
status: New → In Progress
Gabriel Cocenza (gabrielcocenza)
Changed in charm-cinder:
assignee: nobody → Gabriel Angelo Sgarbi Cocenza (gabrielcocenza)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-designate (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-designate/+/874247

Changed in charm-designate:
status: New → In Progress
Gabriel Cocenza (gabrielcocenza)
Changed in charm-designate:
assignee: nobody → Gabriel Angelo Sgarbi Cocenza (gabrielcocenza)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-heat (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-heat/+/874262

Changed in charm-heat:
status: New → In Progress
Gabriel Cocenza (gabrielcocenza)
Changed in charm-neutron-api:
assignee: nobody → Gabriel Angelo Sgarbi Cocenza (gabrielcocenza)
OpenStack Infra (hudson-openstack)
Changed in charm-neutron-api:
status: New → In Progress
Gabriel Cocenza (gabrielcocenza)
no longer affects: charm-neutron-gateway
Gabriel Cocenza (gabrielcocenza)
no longer affects: charm-percona-cluster
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-ironic-api (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-ironic-api/+/874967

Changed in charm-ironic-api:
status: New → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-swift-proxy (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-swift-proxy/+/874973

Changed in charm-swift-proxy:
status: New → In Progress
Gabriel Cocenza (gabrielcocenza)
no longer affects: charm-ceilometer
no longer affects: charm-openstack-dashboard
Changed in charm-heat:
assignee: nobody → Gabriel Angelo Sgarbi Cocenza (gabrielcocenza)
Changed in charm-ironic-api:
assignee: nobody → Gabriel Angelo Sgarbi Cocenza (gabrielcocenza)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to charm-nova-cloud-controller (master)

Fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/charm-nova-cloud-controller/+/875299

Changed in charm-nova-cloud-controller:
status: New → In Progress
Gabriel Cocenza (gabrielcocenza)
Changed in charm-nova-cloud-controller:
assignee: nobody → Gabriel Angelo Sgarbi Cocenza (gabrielcocenza)
Changed in charm-swift-proxy:
assignee: nobody → Gabriel Angelo Sgarbi Cocenza (gabrielcocenza)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-ironic-api (master)

Reviewed: https://review.opendev.org/c/openstack/charm-ironic-api/+/874967
Committed: https://opendev.org/openstack/charm-ironic-api/commit/65379959f2bcb9d317f1ece0b07ab887bf8b47c9
Submitter: "Zuul (22348)"
Branch: master

commit 65379959f2bcb9d317f1ece0b07ab887bf8b47c9
Author: Gabriel Cocenza <email address hidden>
Date: Fri Feb 17 15:02:32 2023 -0300

    Add support for HAProxy L7 checks

    This change add several configuration options to enable HTTP checks
    to the HAProxy configuration, instead of the default TCP connection
    checks.

    Closes-Bug: #1880610
    Change-Id: I5544d9b4c5b9030c0e5b619829152edd01fa0889

Changed in charm-ironic-api:
status: In Progress → Fix Committed
Changed in charm-keystone:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-keystone (master)

Reviewed: https://review.opendev.org/c/openstack/charm-keystone/+/874035
Committed: https://opendev.org/openstack/charm-keystone/commit/04480c4ff4b37ad6aef619d452522721a728b587
Submitter: "Zuul (22348)"
Branch: master

commit 04480c4ff4b37ad6aef619d452522721a728b587
Author: Gabriel Cocenza <email address hidden>
Date: Wed Feb 15 19:09:25 2023 -0300

    Add support for HAProxy L7 checks

    This change add several configuration options to enable HTTP checks
    to the HAProxy configuration, instead of the default TCP connection
    checks (which continue to be the default)

    Closes-Bug: #1880610
    Change-Id: I50a9442ae66da71793a5e9904d23c26d1fbbdf42

Changed in charm-designate:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-designate (master)

Reviewed: https://review.opendev.org/c/openstack/charm-designate/+/874247
Committed: https://opendev.org/openstack/charm-designate/commit/6711b12054e2da999bf7f606344f6bb9af2768b7
Submitter: "Zuul (22348)"
Branch: master

commit 6711b12054e2da999bf7f606344f6bb9af2768b7
Author: Gabriel Cocenza <email address hidden>
Date: Fri Feb 17 10:21:23 2023 -0300

    Add support for HAProxy L7 checks

    This change add several configuration options to enable HTTP checks
    to the HAProxy configuration, instead of the default TCP connection
    checks (which continue to be the default).

    Closes-Bug: #1880610
    Change-Id: Ibadd5648cdf69eef42b05e29772dcae9897d668b

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-swift-proxy (master)

Reviewed: https://review.opendev.org/c/openstack/charm-swift-proxy/+/874973
Committed: https://opendev.org/openstack/charm-swift-proxy/commit/c29a1c7b8e953b2055763c16d04786b800300f9d
Submitter: "Zuul (22348)"
Branch: master

commit c29a1c7b8e953b2055763c16d04786b800300f9d
Author: Gabriel Cocenza <email address hidden>
Date: Thu Feb 23 21:06:33 2023 -0300

    Add support for HAProxy L7 checks

    This change add several configuration options to enable HTTP checks
    to the HAProxy configuration, instead of the default TCP connection
    checks.

    Closes-Bug: #1880610
    Change-Id: Ie1f53d38c1c45ce3b2ddf5041eaa04a7f3139066

Changed in charm-swift-proxy:
status: In Progress → Fix Committed
Changed in charm-heat:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-heat (master)

Reviewed: https://review.opendev.org/c/openstack/charm-heat/+/874262
Committed: https://opendev.org/openstack/charm-heat/commit/19cf71dc79bcba91132dcd769eeb99e18b09b754
Submitter: "Zuul (22348)"
Branch: master

commit 19cf71dc79bcba91132dcd769eeb99e18b09b754
Author: Gabriel Cocenza <email address hidden>
Date: Fri Feb 17 11:38:13 2023 -0300

    Add support for HAProxy L7 checks

    This change add several configuration options to enable HTTP checks
    to the HAProxy configuration, instead of the default TCP connection
    checks (which continue to be the default). It also enables /healthcheck
    endpoint for heat-api and heat-cfn-api on openstack releases >= queens.

    Closes-Bug: #1880610
    Change-Id: I94c9418c82cdddd5a5d9ed400ab47889bfb225b1

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-gnocchi (master)

Reviewed: https://review.opendev.org/c/openstack/charm-gnocchi/+/874030
Committed: https://opendev.org/openstack/charm-gnocchi/commit/b33b6a213f6768072d21827ea38b6d28f35f9d0e
Submitter: "Zuul (22348)"
Branch: master

commit b33b6a213f6768072d21827ea38b6d28f35f9d0e
Author: Gabriel Cocenza <email address hidden>
Date: Wed Feb 15 17:54:19 2023 -0300

    This change add several configuration options to enable HTTP checks
    to the HAProxy configuration, instead of the default TCP connection
    checks (which continue to be the default)

    Closes-Bug: #1880610
    Change-Id: If7841f710d0dfffdc7ee6b424cb77fc34aeae823

Changed in charm-gnocchi:
status: In Progress → Fix Committed
Revision history for this message
Felipe Reyes (freyes) wrote :

Hi Gabriel, thanks for the large piece of work you've done to support healthchecks, since we wouldn't want this new functionality goes inadvertently released, I added a task for the charm-guide, so this get documented in the release-notes ;-)

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-neutron-api (master)

Reviewed: https://review.opendev.org/c/openstack/charm-neutron-api/+/874309
Committed: https://opendev.org/openstack/charm-neutron-api/commit/7330c5badfa72ac2112389838f6e4b64d3f45e3c
Submitter: "Zuul (22348)"
Branch: master

commit 7330c5badfa72ac2112389838f6e4b64d3f45e3c
Author: Gabriel Cocenza <email address hidden>
Date: Fri Feb 17 15:27:55 2023 -0300

    Add support for HAProxy L7 checks

    This change add several configuration options to enable HTTP checks
    to the HAProxy configuration, instead of the default TCP connection
    checks (which continue to be the default). It also enables /healthcheck
    endpoint for neutron-api.

    Closes-Bug: #1880610

    Change-Id: Ia820d8c2ca709d6b358b1c80d770624568d9a85b

Changed in charm-neutron-api:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-nova-cloud-controller (master)

Reviewed: https://review.opendev.org/c/openstack/charm-nova-cloud-controller/+/875299
Committed: https://opendev.org/openstack/charm-nova-cloud-controller/commit/468d648655f420230f3018123ac0ec29bdb0ca87
Submitter: "Zuul (22348)"
Branch: master

commit 468d648655f420230f3018123ac0ec29bdb0ca87
Author: Gabriel Cocenza <email address hidden>
Date: Thu Feb 23 21:17:56 2023 -0300

    Add support for HAProxy L7 checks

    This change add several configuration options to enable HTTP checks
    to the HAProxy configuration, instead of the default TCP connection
    checks.

    Closes-Bug: #1880610
    Change-Id: I4a947c5b52eb3283c08a0d39cc9bf14695a63eab

Changed in charm-nova-cloud-controller:
status: In Progress → Fix Committed
Changed in charm-cinder:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to charm-cinder (master)

Reviewed: https://review.opendev.org/c/openstack/charm-cinder/+/874177
Committed: https://opendev.org/openstack/charm-cinder/commit/51e32a4f7e8d7256fd6189197b49f1965f63199b
Submitter: "Zuul (22348)"
Branch: master

commit 51e32a4f7e8d7256fd6189197b49f1965f63199b
Author: Gabriel Cocenza <email address hidden>
Date: Thu Feb 16 18:18:29 2023 -0300

    Add support for HAProxy L7 checks

    This change add several configuration options to enable HTTP checks
    to the HAProxy configuration, instead of the default TCP connection
    checks (which continue to be the default). It also enables /healthcheck
    endpoint for cinder-api on openstack releases >= ocata.

    Closes-Bug: #1880610
    Change-Id: I9d118f70fc1390be7b800ad20ae20e77818adac7

Alex Kavanagh (ajkavanagh)
Changed in charm-guide:
importance: Undecided → High
status: New → Triaged
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.