| 2020-05-25 22:23:16 |
Andrea Ieri |
bug |
|
|
added bug |
| 2020-05-25 22:23:38 |
Andrea Ieri |
bug |
|
|
added subscriber Canonical IS BootStack |
| 2020-05-25 22:23:45 |
Andrea Ieri |
bug |
|
|
added subscriber The Canonical Sysadmins |
| 2020-05-25 22:39:40 |
Andrea Ieri |
affects |
charm-designate |
charm-hacluster |
|
| 2020-05-25 22:40:09 |
Andrea Ieri |
description |
The current haproxy config adds health checks for all backends. These are however the default TCP checks[0], which succeed simply if a connection to the specified port is possible, and don't go any further in trying to assess if the backend service is actually healthy.
In some partial failure scenarios - e.g. extreme load or memory pressure - TCP checks may succeed where more thorough ones would not, causing API traffic to be routed to backends that aren't actually able to process it.
HAproxy offers the possibility of reconfiguring the layer 4 checks into layer 7 ones by using the httpchk option[1]. This is still fairly basic, but configuring the http-check option[4] makes the check a bit more accurate, while service-specific checks are available for some backends such as mysql[2] and postgres[3]. Finally, external scripts[5] can be invoked for a fully customized check routine.
In order to improve the control plane resilience we should gradually switch to more advanced backend checks. Plain TCP checks should remain the default, but principal charms should provide more accurate check specifications via their relation to hacluster. Depending on the service, a properly configured httpchk may suffice, while in some cases a fully custom script may be more appropriate.
[0] https://www.haproxy.com/blog/the-four-essential-sections-of-an-haproxy-configuration/
[1] https://www.haproxy.com/documentation/hapee/latest/onepage/#4-option%20httpchk
[2] https://www.haproxy.com/documentation/hapee/latest/onepage/#4.2-option%20mysql-check
[3] https://www.haproxy.com/documentation/hapee/latest/onepage/#4.2-option%20pgsql-check
[4] https://www.haproxy.com/documentation/hapee/latest/onepage/#4.2-http-check%20expect
[5] https://www.haproxy.com/documentation/hapee/latest/onepage/#external-check%20command |
The current haproxy config adds health checks for all backends. These are however the default TCP checks[0], which succeed simply if a connection to the specified port is possible, and don't go any further in trying to assess if the backend service is actually healthy.
In some partial failure scenarios - e.g. extreme load or memory pressure - TCP checks may succeed where more thorough ones would not, causing API traffic to be routed to backends that aren't actually able to process it.
HAproxy offers the possibility of reconfiguring the layer 4 checks into layer 7 ones by using the httpchk option[1]. This is still fairly basic, but configuring the http-check option[4] makes the check a bit more accurate, while service-specific checks are available for some backends such as mysql[2] and postgres[3]. Finally, external scripts[5] can be invoked for a fully customized check routine.
In order to improve the control plane resilience we should gradually switch to more advanced backend checks. Plain TCP checks should remain the default, but principal charms should provide more accurate check specifications via their relation to hacluster. Depending on the service, a properly configured httpchk may suffice, while in some cases a fully custom script may be more appropriate.
This is not in contrast with giving more control to pacemaker via openstack resource agents, as described in bug 1880611.
[0] https://www.haproxy.com/blog/the-four-essential-sections-of-an-haproxy-configuration/
[1] https://www.haproxy.com/documentation/hapee/latest/onepage/#4-option%20httpchk
[2] https://www.haproxy.com/documentation/hapee/latest/onepage/#4.2-option%20mysql-check
[3] https://www.haproxy.com/documentation/hapee/latest/onepage/#4.2-option%20pgsql-check
[4] https://www.haproxy.com/documentation/hapee/latest/onepage/#4.2-http-check%20expect
[5] https://www.haproxy.com/documentation/hapee/latest/onepage/#external-check%20command |
|
| 2020-09-02 00:05:45 |
Jose Guedez |
charm-hacluster: assignee |
|
Jose Guedez (jfguedez) |
|
| 2020-09-08 00:32:51 |
Jose Guedez |
bug task added |
|
charm-glance |
|
| 2020-09-08 00:41:08 |
Jose Guedez |
charm-glance: assignee |
|
Jose Guedez (jfguedez) |
|
| 2020-09-08 00:41:36 |
Jose Guedez |
charm-glance: status |
New |
In Progress |
|
| 2020-09-08 04:47:44 |
Jose Guedez |
charm-hacluster: status |
New |
Invalid |
|
| 2020-09-09 02:03:53 |
Jose Guedez |
bug |
|
|
added subscriber Canonical Field Medium |
| 2022-09-27 01:28:50 |
Andrea Ieri |
charm-hacluster: assignee |
Jose Guedez (jfguedez) |
|
|
| 2022-09-27 01:28:52 |
Andrea Ieri |
charm-glance: assignee |
Jose Guedez (jfguedez) |
|
|
| 2022-09-27 01:30:05 |
Andrea Ieri |
bug task added |
|
charm-gnocchi |
|
| 2022-09-27 21:10:55 |
Andrea Ieri |
tags |
|
bseng-436 |
|
| 2022-10-03 14:49:49 |
Junien Fridrick |
bug |
|
|
added subscriber Junien Fridrick |
| 2022-10-24 20:52:10 |
Gabriel Cocenza |
charm-glance: assignee |
|
Gabriel Angelo Sgarbi Cocenza (gabrielcocenza) |
|
| 2023-01-18 19:04:12 |
OpenStack Infra |
charm-glance: status |
In Progress |
Fix Committed |
|
| 2023-02-15 18:13:31 |
Gabriel Cocenza |
bug task added |
|
charm-ceilometer |
|
| 2023-02-15 18:14:53 |
Gabriel Cocenza |
bug task added |
|
charm-nova-cloud-controller |
|
| 2023-02-15 18:15:55 |
Gabriel Cocenza |
bug task added |
|
charm-keystone |
|
| 2023-02-15 18:16:29 |
Gabriel Cocenza |
bug task added |
|
charm-neutron-api |
|
| 2023-02-15 18:17:14 |
Gabriel Cocenza |
bug task added |
|
charm-openstack-dashboard |
|
| 2023-02-15 18:17:45 |
Gabriel Cocenza |
bug task added |
|
charm-cinder |
|
| 2023-02-15 18:18:19 |
Gabriel Cocenza |
bug task added |
|
charm-heat |
|
| 2023-02-15 18:18:47 |
Gabriel Cocenza |
bug task added |
|
charm-swift-proxy |
|
| 2023-02-15 18:19:22 |
Gabriel Cocenza |
bug task added |
|
charm-designate |
|
| 2023-02-15 18:19:44 |
Gabriel Cocenza |
bug task added |
|
charm-neutron-gateway |
|
| 2023-02-15 18:20:03 |
Gabriel Cocenza |
bug task added |
|
charm-percona-cluster |
|
| 2023-02-15 18:20:53 |
Gabriel Cocenza |
bug task added |
|
charm-ironic-api |
|
| 2023-02-15 20:56:10 |
OpenStack Infra |
charm-gnocchi: status |
New |
In Progress |
|
| 2023-02-15 21:00:36 |
Gabriel Cocenza |
charm-gnocchi: assignee |
|
Gabriel Angelo Sgarbi Cocenza (gabrielcocenza) |
|
| 2023-02-15 22:10:48 |
OpenStack Infra |
charm-keystone: status |
New |
In Progress |
|
| 2023-02-15 22:13:51 |
Gabriel Cocenza |
charm-keystone: assignee |
|
Gabriel Angelo Sgarbi Cocenza (gabrielcocenza) |
|
| 2023-02-16 04:13:01 |
Nobuto Murata |
bug |
|
|
added subscriber Nobuto Murata |
| 2023-02-16 15:21:38 |
Gabriel Cocenza |
charm-ceilometer: status |
New |
Invalid |
|
| 2023-02-16 21:51:28 |
Gabriel Cocenza |
charm-openstack-dashboard: status |
New |
Invalid |
|
| 2023-02-17 00:17:46 |
OpenStack Infra |
charm-cinder: status |
New |
In Progress |
|
| 2023-02-17 00:37:20 |
Gabriel Cocenza |
charm-cinder: assignee |
|
Gabriel Angelo Sgarbi Cocenza (gabrielcocenza) |
|
| 2023-02-17 13:24:03 |
OpenStack Infra |
charm-designate: status |
New |
In Progress |
|
| 2023-02-17 13:24:31 |
Gabriel Cocenza |
charm-designate: assignee |
|
Gabriel Angelo Sgarbi Cocenza (gabrielcocenza) |
|
| 2023-02-17 15:05:33 |
OpenStack Infra |
charm-heat: status |
New |
In Progress |
|
| 2023-02-17 20:46:35 |
Gabriel Cocenza |
charm-neutron-api: assignee |
|
Gabriel Angelo Sgarbi Cocenza (gabrielcocenza) |
|
| 2023-02-17 20:53:52 |
OpenStack Infra |
charm-neutron-api: status |
New |
In Progress |
|
| 2023-02-23 19:23:10 |
Gabriel Cocenza |
bug task deleted |
charm-neutron-gateway |
|
|
| 2023-02-23 20:04:27 |
Gabriel Cocenza |
bug task deleted |
charm-percona-cluster |
|
|
| 2023-02-23 22:40:35 |
OpenStack Infra |
charm-ironic-api: status |
New |
In Progress |
|
| 2023-02-24 00:07:53 |
OpenStack Infra |
charm-swift-proxy: status |
New |
In Progress |
|
| 2023-02-24 02:35:56 |
Gabriel Cocenza |
bug task deleted |
charm-ceilometer |
|
|
| 2023-02-24 02:36:03 |
Gabriel Cocenza |
bug task deleted |
charm-openstack-dashboard |
|
|
| 2023-02-24 02:36:17 |
Gabriel Cocenza |
charm-heat: assignee |
|
Gabriel Angelo Sgarbi Cocenza (gabrielcocenza) |
|
| 2023-02-24 02:36:27 |
Gabriel Cocenza |
charm-ironic-api: assignee |
|
Gabriel Angelo Sgarbi Cocenza (gabrielcocenza) |
|
| 2023-02-25 15:24:14 |
OpenStack Infra |
charm-nova-cloud-controller: status |
New |
In Progress |
|
| 2023-02-27 02:06:31 |
Gabriel Cocenza |
charm-nova-cloud-controller: assignee |
|
Gabriel Angelo Sgarbi Cocenza (gabrielcocenza) |
|
| 2023-02-27 02:06:41 |
Gabriel Cocenza |
charm-swift-proxy: assignee |
|
Gabriel Angelo Sgarbi Cocenza (gabrielcocenza) |
|
| 2023-02-28 09:26:07 |
OpenStack Infra |
charm-ironic-api: status |
In Progress |
Fix Committed |
|
| 2023-02-28 09:26:37 |
OpenStack Infra |
charm-keystone: status |
In Progress |
Fix Committed |
|
| 2023-02-28 09:27:31 |
OpenStack Infra |
charm-designate: status |
In Progress |
Fix Committed |
|
| 2023-02-28 09:37:07 |
OpenStack Infra |
charm-swift-proxy: status |
In Progress |
Fix Committed |
|
| 2023-02-28 09:40:04 |
OpenStack Infra |
charm-heat: status |
In Progress |
Fix Committed |
|
| 2023-03-13 07:30:51 |
OpenStack Infra |
charm-gnocchi: status |
In Progress |
Fix Committed |
|
| 2023-03-13 15:45:05 |
Felipe Reyes |
bug task added |
|
charm-guide |
|
| 2023-03-29 08:03:09 |
OpenStack Infra |
charm-neutron-api: status |
In Progress |
Fix Committed |
|
| 2023-03-29 15:11:27 |
OpenStack Infra |
charm-nova-cloud-controller: status |
In Progress |
Fix Committed |
|
| 2023-03-29 15:14:58 |
OpenStack Infra |
charm-cinder: status |
In Progress |
Fix Committed |
|
| 2023-06-16 10:30:18 |
Alex Kavanagh |
charm-guide: importance |
Undecided |
High |
|
| 2023-06-16 10:30:18 |
Alex Kavanagh |
charm-guide: status |
New |
Triaged |
|
