Openstack API Charms have a naive haproxy connection timeout
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
ceilometer (Juju Charms Collection) |
Fix Released
|
High
|
Unassigned | ||
ceph-radosgw (Juju Charms Collection) |
Fix Released
|
High
|
Unassigned | ||
cinder (Juju Charms Collection) |
Fix Released
|
High
|
Unassigned | ||
glance (Juju Charms Collection) |
Fix Released
|
High
|
Unassigned | ||
keystone (Juju Charms Collection) |
Fix Released
|
High
|
Unassigned | ||
neutron-api (Juju Charms Collection) |
Fix Released
|
High
|
Unassigned | ||
nova-cloud-controller (Juju Charms Collection) |
Fix Released
|
High
|
Unassigned | ||
openstack-dashboard (Juju Charms Collection) |
Fix Released
|
High
|
Unassigned | ||
swift-proxy (Juju Charms Collection) |
Fix Released
|
High
|
Unassigned |
Bug Description
On OpenStack API charms haproxy sets timeout connection to 1 second. This is reasonable only in an ideal scenario. Connections to backends on a busy cloud may take much longer. Even haproxy docs suggest 4 to 5 seconds [1].
defaults
log global
mode tcp
option tcplog
option dontlognull
retries 3
timeout queue 1000
timeout connect 1000
This causes mysterious errors like the following:
On neutron-api:
ConnectionRefused: Unable to establish connection to http://
On keystone:
ERROR: HTTPConnectionP
On nova-cloud-
requests.
When max_retries is forced > 1 it becomes only slightly less opaque:
urllib3.
Set timeout connect (and timeout queue while we are at it) to sane defaults and optionally make this configurable.
[1] https:/
Related branches
- James Page: Approve
- David Ames (community): Needs Resubmitting
-
Diff: 56 lines (+22/-9)2 files modifiedcharmhelpers/contrib/openstack/context.py (+6/-0)
charmhelpers/contrib/openstack/templates/haproxy.cfg (+16/-9)
- OpenStack Charmers: Pending requested
-
Diff: 79 lines (+34/-9)3 files modifiedconfig.yaml (+12/-0)
hooks/charmhelpers/contrib/openstack/context.py (+6/-0)
hooks/charmhelpers/contrib/openstack/templates/haproxy.cfg (+16/-9)
- OpenStack Charmers: Pending requested
-
Diff: 103 lines (+48/-11)4 files modifiedconfig.yaml (+24/-0)
hooks/charmhelpers/contrib/openstack/context.py (+6/-0)
hooks/charmhelpers/contrib/openstack/templates/haproxy.cfg (+16/-9)
hooks/neutron_api_utils.py (+2/-2)
- OpenStack Charmers: Pending requested
-
Diff: 88 lines (+46/-9)3 files modifiedcharmhelpers/contrib/openstack/context.py (+6/-0)
charmhelpers/contrib/openstack/templates/haproxy.cfg (+16/-9)
config.yaml (+24/-0)
- OpenStack Charmers: Pending requested
-
Diff: 79 lines (+34/-9)3 files modifiedcharmhelpers/contrib/openstack/context.py (+6/-0)
charmhelpers/contrib/openstack/templates/haproxy.cfg (+16/-9)
config.yaml (+12/-0)
- OpenStack Charmers: Pending requested
-
Diff: 88 lines (+46/-9)3 files modifiedconfig.yaml (+24/-0)
hooks/charmhelpers/contrib/openstack/context.py (+6/-0)
hooks/charmhelpers/contrib/openstack/templates/haproxy.cfg (+16/-9)
- OpenStack Charmers: Pending requested
-
Diff: 89 lines (+47/-9)3 files modifiedconfig.yaml (+25/-0)
hooks/charmhelpers/contrib/openstack/context.py (+6/-0)
hooks/charmhelpers/contrib/openstack/templates/haproxy.cfg (+16/-9)
- OpenStack Charmers: Pending requested
-
Diff: 88 lines (+46/-9)3 files modifiedconfig.yaml (+24/-0)
hooks/charmhelpers/contrib/openstack/context.py (+6/-0)
hooks/charmhelpers/contrib/openstack/templates/haproxy.cfg (+16/-9)
- OpenStack Charmers: Pending requested
-
Diff: 98 lines (+49/-9) (has conflicts)3 files modifiedcharmhelpers/contrib/openstack/context.py (+6/-0)
charmhelpers/contrib/openstack/templates/haproxy.cfg (+16/-9)
config.yaml (+27/-0)
- OpenStack Charmers: Pending requested
-
Diff: 88 lines (+46/-9)3 files modifiedcharmhelpers/contrib/openstack/context.py (+6/-0)
charmhelpers/contrib/openstack/templates/haproxy.cfg (+16/-9)
config.yaml (+24/-0)
description: | updated |
affects: | charms → nova-cloud-controller (Juju Charms Collection) |
description: | updated |
Changed in glance (Juju Charms Collection): | |
status: | New → Invalid |
Changed in nova-cloud-controller (Juju Charms Collection): | |
status: | New → Invalid |
Changed in cinder (Juju Charms Collection): | |
status: | New → In Progress |
Changed in keystone (Juju Charms Collection): | |
status: | New → In Progress |
Changed in neutron-api (Juju Charms Collection): | |
status: | New → In Progress |
Changed in cinder (Juju Charms Collection): | |
importance: | Undecided → High |
Changed in keystone (Juju Charms Collection): | |
importance: | Undecided → High |
Changed in neutron-api (Juju Charms Collection): | |
importance: | Undecided → High |
Changed in cinder (Juju Charms Collection): | |
assignee: | nobody → Liam Young (gnuoy) |
Changed in keystone (Juju Charms Collection): | |
assignee: | nobody → Liam Young (gnuoy) |
Changed in neutron-api (Juju Charms Collection): | |
assignee: | nobody → Liam Young (gnuoy) |
Changed in cinder (Juju Charms Collection): | |
milestone: | none → 16.01 |
Changed in keystone (Juju Charms Collection): | |
milestone: | none → 16.01 |
Changed in neutron-api (Juju Charms Collection): | |
milestone: | none → 16.01 |
Changed in ceph-radosgw (Juju Charms Collection): | |
status: | New → Confirmed |
importance: | Undecided → High |
Changed in openstack-dashboard (Juju Charms Collection): | |
status: | New → Confirmed |
importance: | Undecided → High |
Changed in swift-proxy (Juju Charms Collection): | |
status: | New → Confirmed |
importance: | Undecided → High |
Changed in openstack-dashboard (Juju Charms Collection): | |
milestone: | none → 16.01 |
Changed in swift-proxy (Juju Charms Collection): | |
milestone: | none → 16.01 |
Changed in ceph-radosgw (Juju Charms Collection): | |
milestone: | none → 16.01 |
Changed in ceilometer (Juju Charms Collection): | |
importance: | Undecided → High |
milestone: | none → 16.01 |
status: | New → Confirmed |
Changed in ceilometer (Juju Charms Collection): | |
status: | Confirmed → Fix Committed |
Changed in ceph-radosgw (Juju Charms Collection): | |
status: | Confirmed → Fix Committed |
Changed in cinder (Juju Charms Collection): | |
status: | Confirmed → Fix Committed |
Changed in glance (Juju Charms Collection): | |
status: | Confirmed → Fix Committed |
Changed in keystone (Juju Charms Collection): | |
status: | Confirmed → Fix Committed |
Changed in neutron-api (Juju Charms Collection): | |
status: | Confirmed → Fix Committed |
Changed in nova-cloud-controller (Juju Charms Collection): | |
status: | Confirmed → Fix Committed |
Changed in openstack-dashboard (Juju Charms Collection): | |
status: | Confirmed → Fix Committed |
Changed in swift-proxy (Juju Charms Collection): | |
status: | Confirmed → Fix Committed |
Changed in nova-cloud-controller (Juju Charms Collection): | |
status: | Fix Committed → Fix Released |
Changed in neutron-api (Juju Charms Collection): | |
status: | Fix Committed → Fix Released |
Changed in keystone (Juju Charms Collection): | |
status: | Fix Committed → Fix Released |
Changed in glance (Juju Charms Collection): | |
status: | Fix Committed → Fix Released |
Changed in cinder (Juju Charms Collection): | |
status: | Fix Committed → Fix Released |
Changed in openstack-dashboard (Juju Charms Collection): | |
status: | Fix Committed → Fix Released |
Changed in ceph-radosgw (Juju Charms Collection): | |
status: | Fix Committed → Fix Released |
Changed in swift-proxy (Juju Charms Collection): | |
status: | Fix Committed → Fix Released |
Changed in ceilometer (Juju Charms Collection): | |
status: | Fix Committed → Fix Released |
Yes, I think a charm config option (and/or a more sane timeout default value) would be great, for all os-charms which employ haproxy.
I've observed the "''" return during times of elevated load, in neutron-api, nova and dashboard.