"stick on src" haproxy option is a cause of slow keystone

Bug #1582202 reported by Leontiy Istomin
18
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Fuel for OpenStack
Fix Committed
Critical
Alex Schultz
Mitaka
Fix Released
Critical
Alex Schultz

Bug Description

Detailed bug description:
 stick on src is a cause of slow performance of kystone. We performed rally scenario which boots an instance with secgroup with 10 rules and a cinder volume, then migrate the instance. (http://paste.openstack.org/show/497202/)
When the option persists in /etc/haproxy/conf.d/020-keystone-1.cfg and /etc/haproxy/conf.d/030-keystone-2.cfg files (during rally test):
openstack user list - real 0m27.563s
nova list --all-tenant - real 0m47.577s
root@node-179:~# for i in 77 96 179; do grep keystone-2 /var/log/haproxy.log | grep -c node-$i; done
1124
313
2283
When I commented out two lines in the files (during rally test):
  #stick on src
  #stick-table type ip size 200k expire 2m
openstack user list - real 0m1.704s
nova list --all-tenant - real 0m6.011s
Round Robin works well

Steps to reproduce:
1. Fuel 9.0-308 has been deployed
2. curl -s 'https://review.openstack.org/gitweb?p=openstack/fuel-web.git;a=patch;h=6106dfa026b042dac26ed77354321115b78aae5b' | patch -b -d /usr/lib/python2.7/site-packages -p2
rpm -Uvh epel-release-latest-7.noarch.rpm
sed -i s/^enabled=1/enabled=0/g /etc/yum.repos.d/epel.repo
yum --enablerepo=epel install uwsgi uwsgi-plugin-python python-uwsgidecorator
service receiverd restart && service nailgun restart
due https://bugs.launchpad.net/fuel/+bug/1569859 and https://bugs.launchpad.net/fuel/+bug/1570509
fuel-agent and fuel-library have been updated: http://paste.openstack.org/show/496825/ due https://bugs.launchpad.net/fuel/+bug/1543233 and https://bugs.launchpad.net/fuel/+bug/1574999
3. patch has been applied to keep rotated logs http://paste.openstack.org/show/495857/
4. patch has been applied to increase rsyslog chunks to 4к http://paste.openstack.org/show/496901/
5. LMA, ElasticSearch, Grafana plugins have been installed:
yum -y install createrepo rpm rpm-build dpkg-devel git
easy_install pip
pip install fuel-plugin-builder
git clone https://github.com/openstack/fuel-plugin-lma-collector.git
fpb --check ./fuel-plugin-lma-collector
fpb --build ./fuel-plugin-lma-collector
fuel plugins --install ./fuel-plugin-lma-collector/*.noarch.rpm
git clone https://github.com/openstack/fuel-plugin-elasticsearch-kibana.git
fpb --check ./fuel-plugin-elasticsearch-kibana
fpb --build ./fuel-plugin-elasticsearch-kibana
fuel plugins --install ./fuel-plugin-elasticsearch-kibana/*.noarch.rpm
git clone https://github.com/openstack/fuel-plugin-influxdb-grafana.git
fpb --check ./fuel-plugin-influxdb-grafana
fpb --build ./fuel-plugin-influxdb-grafana
fuel plugins --install ./fuel-plugin-influxdb-grafana/*.noarch.rpm
patch -b -d /var/www/nailgun/plugins/lma_collector-0.10/ -p1 < lma.patch (http://paste.openstack.org/show/495328/)
fuel plugins --sync
6. deploy cluster 3 controllers, 20 computes+Ceph, 172 computes, vxlan+DVR, Ceph for all
7. Perform rally tests and execute openstack user list command during the testing
Expected results:
 The command shows results in 5 seconds
Actual result:
 The command shows results in ~50 seconds (timeout 60)
Reproducibility:
 Each time
Workaround:
 Comment out lines if /etc/haproxy/conf.d/020-keystone-1.cfg and /etc/haproxy/conf.d/030-keystone-2.cfg files:
  #stick on src
  #stick-table type ip size 200k expire 2m
Impact:
 OpenStack cluster performance
Description of the environment:
 Operation system: ubuntu
 Versions of components: MOS 9.0
 Reference architecture: 3 controllers, 20 computes+Ceph, 172 computes, vxlan+DVR, Ceph for all
 Network model: vxlan+DVR
 Related projects installed: LMA
Additional information:
 -

description: updated
Dmitry Klenov (dklenov)
tags: added: area-library
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (master)

Fix proposed to branch: master
Review: https://review.openstack.org/316947

Changed in fuel:
assignee: Fuel Sustaining (fuel-sustaining-team) → Alex Schultz (alex-schultz)
status: Confirmed → In Progress
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (master)

Reviewed: https://review.openstack.org/316947
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=8ebe5f15474319fafe2cdae2fc74ee98d63dcb80
Submitter: Jenkins
Branch: master

commit 8ebe5f15474319fafe2cdae2fc74ee98d63dcb80
Author: Alex Schultz <email address hidden>
Date: Mon May 16 09:54:12 2016 -0600

    Make keystone session stickiness optional

    As part of Bug 1527717, we added in support of session stickiness for
    keystone to allow for federation support in keystone. This introduced
    performance issues with keystone and should not be configured by
    default. This change disables the stickiness by default but allows for
    it to be enabled by adding a 'federation' key to the keystone hiera
    hash.

    DocImpact: Keystone HAProxy session stickiness can be enabled by adding
    a 'federation' key to the keystone hiera configuration. By default
    session stickiness is disabled.

    Change-Id: Id43446a9f923417b8fa140ffcfddb6797f1b4e48
    Related-Bug: #1527717
    Closes-Bug: #1582202

Changed in fuel:
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix proposed to fuel-library (stable/mitaka)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/318716

Dmitry Pyzhov (dpyzhov)
no longer affects: fuel/newton
Changed in fuel:
importance: High → Critical
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to fuel-library (stable/mitaka)

Reviewed: https://review.openstack.org/318716
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=264a310432e3ab1bfbdae85d2af4faa4d2852521
Submitter: Jenkins
Branch: stable/mitaka

commit 264a310432e3ab1bfbdae85d2af4faa4d2852521
Author: Alex Schultz <email address hidden>
Date: Mon May 16 09:54:12 2016 -0600

    Make keystone session stickiness optional

    As part of Bug 1527717, we added in support of session stickiness for
    keystone to allow for federation support in keystone. This introduced
    performance issues with keystone and should not be configured by
    default. This change disables the stickiness by default but allows for
    it to be enabled by adding a 'federation' key to the keystone hiera
    hash.

    DocImpact: Keystone HAProxy session stickiness can be enabled by adding
    a 'federation' key to the keystone hiera configuration. By default
    session stickiness is disabled.

    Change-Id: Id43446a9f923417b8fa140ffcfddb6797f1b4e48
    Related-Bug: #1527717
    Closes-Bug: #1582202
    (cherry picked from commit 8ebe5f15474319fafe2cdae2fc74ee98d63dcb80)

Revision history for this message
Michael Semenov (msemenov) wrote :

Verified on sclae env-10 (200 nodes)
Average keystone RPS is ~280, however it is ​*sum for get token + validate token*​, so rate for one iteration of obtaining and validation token is ​* ~ 140TPS*​

Since RPS is >100, considered as verified.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.