[library] Performance issue in Keystone: let's use httpd / apache

In keystone logs multiply occurrences of eventlet tracebacks like this one http://paste.openstack.org/show/77411/

I run the same rally/keystone task on the fresh deployed HA-env with one controller.
Deployment by FUEL gives better results, than devstack (see attach).

+1, need to use httpd/apache for Keystone to have an ability to scale Keystone API service.

-1, we already use Ha-proxy as reverse proxy for keystone and another Openstack services. Any httpd server (Apache/Nginx/etc) do the same -- unlock working thread (by quick fetching response from keystone) and send it to the client as possible.

Sergey, yes, but how yu can increase the number of workers of Keystone API service if you want to support large workload?

In Nova, for exanple, we can set the number of API workers in the config.
For Keystone we have only one proccess, and this proccess allows only ~150 requests per seccond.
It is really bad 'peroformance' for production environments with 10 000 + users.

In HA we have at least 3 controller nodes.

but I really missed that keystone hasn't "workers" option in his config.

Sergey, yes. And Apache/httpd will allow to configure number of workers for Keystone API on HA/nonHA environments.

Looks like this bug is duplicate for https://bugs.launchpad.net/fuel/+bug/1316857

This is too major to change after Feature Freeze for 6.0. Moving to 'next' release.

It seems that Juno keystone already has multiple workers, so it's severity is lower for 6.0 release, but we might want to introduce frontend proxy for 6.1 release to support bigger environments

Ryan,
you've lowered priority of this issue. Do you have any info that it's not High for us in 6.1? Especially I'd love to know info from scale lab, so we decide based on data we have (not just feelings). Please don't lower priorities unless you provide reasoning (and please push others do the same). Thanks.

Mike,
It seems that as we can now have multiple workers (as mentioned by Vladimir) this bug isn't as severe. Also the related bug mentioned by Oleksii that prevents us from using Apache with Keystone at all only has a medium priority which is why I changed this one to match.

I've heard from services team, and from Keystone Core guy at the summit that it is still a problem, even if you have multiple workers, and the right solution is to put keystone behind Apache.
That's why I insist on real measurements, as I am still getting contradicting information.

That all makes sense. Moving back to high, sorry for the confusion.

Folks, might this be related to the https://bugs.launchpad.net/mos/+bug/1413341 ?
We had issues on scale lab (Keystone stopped working after continuous high load on it) and it turned out, that the issue was in the incorrect order of eventlet monkey-patching of python logging.

Our issue seems to be gone after this, may you guys refresh the bug or have the additional investigation on it due to the things Keystone team has found?

Thank you for the feedback, @Dina. I set this bug to incomplete as it requires additional investigating

Confirmed status fits better, as we don't want an occasional expiration of this one.

Backports must be provided as well

Nope, backports will not be there as it requires modification of reference architecture. We will not configure apache for keystone in 6.0.x release.

Sorry for confusion, my point was that the *fix* should be backported in case of this issue is related with https://bugs.launchpad.net/mos/+bug/1413341

here's the note from keystone ptl to the operators list with lots of good reasons on why moving to Apache is a good idea and the deprecation path forward.

http://markmail.org/message/pur56ut2ikzpaglu

Fix proposed to branch: master
Review: https://review.openstack.org/173314

Fix proposed to branch: master
Change author: Alexander Didenko <email address hidden>
Review: https://review.fuel-infra.org/6251

Fix proposed to branch: master
Review: https://review.fuel-infra.org/6251

Fix proposed to branch: openstack-ci/fuel-6.1/2014.2
Change author: Alexander Didenko <email address hidden>
Review: https://review.fuel-infra.org/6326

Reviewed: https://review.openstack.org/178554
Committed: https://git.openstack.org/cgit/stackforge/fuel-ostf/commit/?id=ccecd5d22bcd44a46dfdfad31b2f0f259bd66cf9
Submitter: Jenkins
Branch: master

commit ccecd5d22bcd44a46dfdfad31b2f0f259bd66cf9
Author: Tatyana Leontovich <email address hidden>
Date: Wed Apr 29 11:34:39 2015 +0300

    Check if cookies has cfrtoken

    Try to get cfrtoken atribute from cookies,
    if we do not have such just try to login without it

    Related-Bug: 1313662
    Change-Id: I978a0cf2e4f96561113f0f05ad0d0947b45548e4

Folks, that is the status of this bug? The related commit looks stuck with -2 "Do not patch binaries inside puppet", any progress on this?

We're blocked by this patch not being reviewed/merged:
https://review.fuel-infra.org/#/c/6326/

Reviewed: https://review.fuel-infra.org/6326
Submitter: Dmitry Burmistrov <email address hidden>
Branch: openstack-ci/fuel-6.1/2014.2

Commit: f8b2c05606e6476e58fb536dafcf3c32b3e27a43
Author: Alexander Didenko <email address hidden>
Date: Tue May 12 12:37:45 2015

Add keystone/wsgi.py to deb package

We need to ship wsgi keystone script and config example with the
package. We have them in RPM already, so need to change deb only.

Partial-bug: #1313662

Change-Id: If08509abd042461a521a1cda3298dcd71a9d28c5

Change abandoned by Alexander Didenko <email address hidden> on branch: master
Review: https://review.fuel-infra.org/6251
Reason: Invalid

Reviewed: https://review.openstack.org/173314
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=4b1a79095994169189d18e7f5849c2c089263fd0
Submitter: Jenkins
Branch: master

commit 4b1a79095994169189d18e7f5849c2c089263fd0
Author: Dmitry Ilyin <email address hidden>
Date: Fri Apr 3 20:39:08 2015 +0300

    Enable keystone wsgi support

    * New task to deploy Apache and configure MPM
    * Run keystone as Apache vhost
    * Pull wsgi threads and priority configuration from upstream
      change-id: Ib05ac81381e169845b44b2ef7cb810a4d5db17de
    * Pull custom file source for wsgi scripts from upstream
      change-id: I941bf8804982e9081812e076f7a736f413220047
    * Fix for keystone spec tests (use concat 1.2.1 in fixtures)
    * Use keystone.py wsgi script from packages instead of downloading
      it from puppet module. Requires deb package with this patch:
      https://review.fuel-infra.org/6251
      Until it's megred, we'll use upstream wsgi script for Debian

    Change-Id: I85008079b0e922a4518c696a097238500132fa04
    Closes-Bug: 1313662

Verified on MOS 6.1 ISO #429
Steps to Verify:
root@node-1:~# ps -ef | grep keystone
keystone 17914 17911 0 14:47 ? 00:00:01 /usr/sbin/apache2 -k start
keystone 17915 17911 0 14:47 ? 00:00:01 /usr/sbin/apache2 -k start
root 30659 19123 0 15:06 pts/20 00:00:00 grep --color=auto keystone

Note, that this feature was reverted from the 6.1 as it doesn't fit into the HCF plans and its HA impact on memcache_pool backend was not tested as appropriate. The related bug reverted this is https://bugs.launchpad.net/fuel/+bug/1459977

Folks, any progress on this? I suggest to switch to apache as early as possible.

Fix proposed to branch: master
Review: https://review.openstack.org/204111

Reviewed: https://review.openstack.org/204111
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=67f1c37c54a2ba65829379eae505831e57c5776d
Submitter: Jenkins
Branch: master

commit 67f1c37c54a2ba65829379eae505831e57c5776d
Author: Aleksandr Didenko <email address hidden>
Date: Tue Jul 21 17:24:49 2015 +0300

    Run keystone under Apache wsgi

    Configure keystone under Apache wsgi.

    Change-Id: Iad9324cf1646965d67b1c55c73c4650bdf42a5b5
    Closes-Bug: 1313662

Verified on MOS 7.0 ISO #294