Comment 0 for bug 1880610

Revision history for this message
Andrea Ieri (aieri) wrote :

The current haproxy config adds health checks for all backends. These are however the default TCP checks[0], which succeed simply if a connection to the specified port is possible, and don't go any further in trying to assess if the backend service is actually healthy.

In some partial failure scenarios - e.g. extreme load or memory pressure - TCP checks may succeed where more thorough ones would not, causing API traffic to be routed to backends that aren't actually able to process it.

HAproxy offers the possibility of reconfiguring the layer 4 checks into layer 7 ones by using the httpchk option[1]. This is still fairly basic, but configuring the http-check option[4] makes the check a bit more accurate, while service-specific checks are available for some backends such as mysql[2] and postgres[3]. Finally, external scripts[5] can be invoked for a fully customized check routine.

In order to improve the control plane resilience we should gradually switch to more advanced backend checks. Plain TCP checks should remain the default, but principal charms should provide more accurate check specifications via their relation to hacluster. Depending on the service, a properly configured httpchk may suffice, while in some cases a fully custom script may be more appropriate.

[0] https://www.haproxy.com/blog/the-four-essential-sections-of-an-haproxy-configuration/
[1] https://www.haproxy.com/documentation/hapee/latest/onepage/#4-option%20httpchk
[2] https://www.haproxy.com/documentation/hapee/latest/onepage/#4.2-option%20mysql-check
[3] https://www.haproxy.com/documentation/hapee/latest/onepage/#4.2-option%20pgsql-check
[4] https://www.haproxy.com/documentation/hapee/latest/onepage/#4.2-http-check%20expect
[5] https://www.haproxy.com/documentation/hapee/latest/onepage/#external-check%20command