Horizon is slow at small scale

Thank you very much for a very detailed bug report with videos!

The painful slowness of Overview panel is a well-know issue, as well horrible Quotas/Defaults implementation. As for the Instances panel, how many instances were there on a single table page? Currently that heavy influences the total performance of page rendering.

Another interesting thing I've seen was that Projects table took 6+ seconds to render, that's particularly interesting because that page requires almost no other sources of data (contrary to Instances, for example). Could you fetch the list of projects using CLI on the same env and report back the total time?

Unfortunately, Horizon slowness is not a thing that could be fixed easily within the scope of just one bug. It's going to be a slow and tedious process, involving many changes, bugfixes and blueprints. Still I'd prefer to have this bug to remind us the actual numbers.

I've fetched project list with `keystone tenant-list` and using Horizon several times: it seems that this time is pretty close, ~4 seconds.

> As for the Instances panel, how many instances were there on a single table page?

Overall, I had ~20 instances at that time. But you may check latest video: I removed all instances from cluster but delay was too big as well.

Timur, please take a closer look. 20 instances is way too few...

This seems like a legitimate assumption:
https://ask.openstack.org/en/question/91577/horizon-slow-causes/

the link above is related to a MOS deployment and concerns the 'workers' configuration of openstack services running on the controllers - it seems each service will run one worker for each cpu core and this leads to swapping, in MOS 9.0 my 3 controllers have 24GB RAM and 24 cores, disk swapping happens at rest immediately after deployment. this should be a critical fix as it could break disks.

I'm seeing similar issues in another 9.0 installation. Are there any updates for this issue? What information is needed to continue troubleshooting?

Michael, the approach we (me and Andrey Grebennikov) took to see what things cause the most lags was the following one:

* in case of Horizon HA ensure that there is only one node working (to simplify further debugging)
* ensure that 'DEBUG' level of logging is set for all LOGGING['loggers'] related to various python-clients in /etc/openstack-dashboard/local_settings.py
* ensure also that whichever handler that is actually used for writing horizon logs (the ones defined within LOGGING['handlers']) has 'DEBUG' level enabled as well.
* preferrably truncate horizon logs to start fresh
* load the page you're experiencing difficulties with
* among all the logged calls that have been made while rendering the page, find the ones that took most time.

Parsing Horizon logs to see which calls take which time is actually the most tedious part of the process. In late N/early O (actually it was early O, but can be easily backported to the tagged N) release Horizon received a new 'OpenStack Profiler' panel which allows to do the same task from Horizon itself - reload the page with profiler switched on and then see the which calls were responsible for the rendering delays through a nice graphs.

I can confirm this issue even in MOS 9.2 on clean cloud with no resources - horizon tabs take several seconds to load. Since I have already done extensive analysis of Horizon slowness in past, I can try to do the same with 9.2 and provide some insights. I will contact Nikita Konovalov regarding this.

UPD: after several refreshes all pages start to respond very quickly in MOS 9.2, main request usually takes < 1s to complete, the rest is css/js (which should be loaded from cache, I have different results in Chrome vs FF - FF shows as cached while Chrome keeps reloading js and css, chaching only .jpg. Or at least reports it this way, while I did not disable cache). Therefore I agree with the opinion that this behavior is due to swapping. However in each environment it can be different. In each particular case slowness has to be analyzed individually.

Dear Dmitry, kindly tell how did you analyze slowness of horizon.
On my installation I see that 'instances' page with 6 instances taking 5-6 secs to render. Is it slow or fast? Other pages taking 2-3 seconds.