Horizon makes duplicated and extra requests

Fix proposed to branch: openstack-ci/fuel-8.0/liberty
Change author: Timur Sufiev <email address hidden>
Review: https://review.fuel-infra.org/22191

Fix proposed to branch: openstack-ci/fuel-8.0/liberty
Change author: Timur Sufiev <email address hidden>
Review: https://review.fuel-infra.org/22192

Two above patches should solve the issue of duplicated requests in network-related views. Andrey, please confirm if they really do.

Change abandoned by Timur Sufiev <email address hidden> on branch: openstack-ci/fuel-8.0/liberty
Review: https://review.fuel-infra.org/22192
Reason: The patch seems to cause more problems than it solves.

Here is the list of all CRs which are fixing the issues related to this bug:

https://review.fuel-infra.org/#/c/22191
https://review.fuel-infra.org/#/c/22495
https://review.fuel-infra.org/#/c/22496
https://review.fuel-infra.org/#/c/22520
https://review.fuel-infra.org/#/c/22573
https://review.fuel-infra.org/#/c/22582
https://review.fuel-infra.org/#/c/22619

Reviewed: https://review.fuel-infra.org/22191
Submitter: Pkgs Jenkins <email address hidden>
Branch: openstack-ci/fuel-8.0/liberty

Commit: 17dd25621ea70afbfcf71d6bf6aa04d8f31ac764
Author: Timur Sufiev <email address hidden>
Date: Thu Jun 30 15:09:33 2016

Fix duplicating <network_entity>_list() calls in api.network

The solution is trivial - to add @memoize decorator to the following
calls:
* api.neutron.subnet_list
* api.neutron.port_list
* api.network.tenant_floating_ip_list
* api.network.security_group_list

Partial-Bug: #1593456
Change-Id: I73498ec07166e8910a700b577b8b32f46d032d86

Reviewed: https://review.fuel-infra.org/22520
Submitter: Pkgs Jenkins <email address hidden>
Branch: openstack-ci/fuel-8.0/liberty

Commit: 9101c861262771aae0638e4a4ab2b52c72a4b2b0
Author: Timur Sufiev <email address hidden>
Date: Thu Jun 30 16:38:00 2016

Optimize Instances panel render by means of threads parallelizing

Partial-Bug: #1593456
Change-Id: I525a594052f3fc9770e59b41ca5e5478a2184f19

Reviewed: https://review.fuel-infra.org/22573
Submitter: Pkgs Jenkins <email address hidden>
Branch: openstack-ci/fuel-8.0/liberty

Commit: bbce16424241d4aade4ee4f72c90b0ac21395149
Author: Timur Sufiev <email address hidden>
Date: Tue Jul 5 15:59:20 2016

Parallelize with threads fetching networks and routers data

Also parallelize subnets / ports / agents data methods which were
previously run for Network Details MultiTable page sequentially. In
base tables views check if every data been gathered is a
ThreadedPromise, if it is, wait until it's calculated (once all
promises are started).

Partial-Bug: #1593456
Change-Id: I7004d8983d13ff10718ab6dbe4af1384c2c03a72

Verified on MOS 8.0 + MU2 updates using steps from bug description.

Configuration:
MOS 8.0 + Contrail plugin
1 controller node, 1 compute + cinder node, 1 contrail-control + contrail-config + contrail-db node.

Actual results:
Login takes 2-3 sec, each table takes 1+ secs.

Release Note

Several performance tweaks were made so that Horizon would render certain pages faster. First, more network-related API calls are memoized within one request, so that they are no longer duplicated in case both Quotas Usages and Networks panel both request them in the same request. Second, Quota Usages was extensively rewritten to make API calls to Nova, Cinder and Neutron in parallel, so that Quota Usage calls complete several times faster. The same parallelization technique was employed for Instances and Networks pages, where requests to Neutron were made, since they were the main contributor to Horizon's performance degradation. Third, memoize decorator was fixed to work with parallelized calls and do not start a duplicate call in case there is one already in progress (by means of threading locks).

Fix proposed to branch: 9.0/mitaka
Change author: Timur Sufiev <email address hidden>
Review: https://review.fuel-infra.org/23521

Fix proposed to branch: 9.0/mitaka
Change author: Timur Sufiev <email address hidden>
Review: https://review.fuel-infra.org/23522

Fix proposed to branch: 9.0/mitaka
Change author: Timur Sufiev <email address hidden>
Review: https://review.fuel-infra.org/23523

Fix proposed to branch: 9.0/mitaka
Change author: Timur Sufiev <email address hidden>
Review: https://review.fuel-infra.org/23524

Fix proposed to branch: 9.0/mitaka
Change author: Timur Sufiev <email address hidden>
Review: https://review.fuel-infra.org/23525

Fix proposed to branch: 9.0/mitaka
Change author: Timur Sufiev <email address hidden>
Review: https://review.fuel-infra.org/23526

Fix proposed to branch: 9.0/mitaka
Change author: Timur Sufiev <email address hidden>
Review: https://review.fuel-infra.org/23527

Fix proposed to branch: openstack-ci/fuel-7.0/2015.1.0
Change author: Alex Ermolov <email address hidden>
Review: https://review.fuel-infra.org/23580

Fix proposed to branch: openstack-ci/fuel-7.0/2015.1.0
Change author: Alex Ermolov <email address hidden>
Review: https://review.fuel-infra.org/23606

Reviewed: https://review.fuel-infra.org/23521
Submitter: Pkgs Jenkins <email address hidden>
Branch: 9.0/mitaka

Commit: 9c26934dc131c9a2062be117a07fdb576acf0719
Author: Timur Sufiev <email address hidden>
Date: Fri Jul 22 18:59:47 2016

Fix duplicating <network_entity>_list() calls in api.network

The solution is trivial - to add @memoize decorator to the following
calls:
* api.neutron.subnet_list
* api.neutron.port_list
* api.network.tenant_floating_ip_list
* api.network.security_group_list

Partial-Bug: #1593456
Change-Id: I73498ec07166e8910a700b577b8b32f46d032d86

Reviewed: https://review.fuel-infra.org/23522
Submitter: Pkgs Jenkins <email address hidden>
Branch: 9.0/mitaka

Commit: 4ade0e1ace5fb2e34c87e2fb7f1105f439d1cb01
Author: Timur Sufiev <email address hidden>
Date: Fri Jul 22 19:19:17 2016

Prepare for parallelization of fetching quotas / usages data

Partial-Bug: #1593456
Change-Id: I0c120f2a1db87f8c660b4ca99ebd27f03ede8398

Reviewed: https://review.fuel-infra.org/23523
Submitter: Pkgs Jenkins <email address hidden>
Branch: 9.0/mitaka

Commit: 386bf222530201cd90201dc1289b8f3ac1c82ffd
Author: Timur Sufiev <email address hidden>
Date: Fri Jul 22 19:52:34 2016

Parallelize quotas fetching using threads

Partial-Bug: #1593456
Change-Id: I140a192a3afaf379c2dc2664abfc4515abe9320b

Reviewed: https://review.fuel-infra.org/23524
Submitter: Pkgs Jenkins <email address hidden>
Branch: 9.0/mitaka

Commit: 53e05f34e0b613582c144f42354280069bd29a20
Author: Timur Sufiev <email address hidden>
Date: Fri Jul 22 20:04:09 2016

Optimize Instances panel render by means of threads parallelizing

Partial-Bug: #1593456
Change-Id: I525a594052f3fc9770e59b41ca5e5478a2184f19

Reviewed: https://review.fuel-infra.org/23525
Submitter: Pkgs Jenkins <email address hidden>
Branch: 9.0/mitaka

Commit: 0a6ee7fd284ec154deeddc1d7615ed12bf5bd70d
Author: Timur Sufiev <email address hidden>
Date: Mon Jul 25 11:37:37 2016

Parallelize with threads fetching networks and routers data

Also parallelize subnets / ports / agents data methods which were
previously run for Network Details MultiTable page sequentially. In
base tables views check if every data been gathered is a
ThreadedPromise, if it is, wait until it's calculated (once all
promises are started).

Partial-Bug: #1593456
Change-Id: I7004d8983d13ff10718ab6dbe4af1384c2c03a72

Reviewed: https://review.fuel-infra.org/23526
Submitter: Pkgs Jenkins <email address hidden>
Branch: 9.0/mitaka

Commit: 76c5e16a044033cc85d4e0e0ad89cb76558edc67
Author: Timur Sufiev <email address hidden>
Date: Mon Jul 25 12:09:13 2016

Prefetch quota and limits calls in parallel with fetching table data

Rendering table actions often involves expensive calls to quotas, so
getting them in advance and at the same time while getting table data
can provide a huge boost to the performance of the table rendering.

Partial-Bug: #1593456
Change-Id: Iaf09668bd856f2781dd7b5d15035bf137486c7cf

Reviewed: https://review.fuel-infra.org/23527
Submitter: Pkgs Jenkins <email address hidden>
Branch: 9.0/mitaka

Commit: 448c11af1a8cd5e48a8bff69788d72389080561e
Author: Timur Sufiev <email address hidden>
Date: Mon Jul 25 12:14:21 2016

Make @memoize thread-aware

If 2 parallel threads simultaneously make an expensive call to a
memoized function with arguments for which the returned value wasn's
cached before, 2 calls are effectively made in parallel.

It's better to avoid such a situation and if one thread already
initiated an expensive call, block on it in another thread and once
the call in the first thread is completed, unblock and reuse the value
which is cached now.

Partial-Bug: #1593456
Change-Id: I99c6d4eac5cce7bdc9ad059d1d08d2f457e7f492

Fix proposed to branch: openstack-ci/fuel-7.0/2015.1.0
Change author: Alex Ermolov <email address hidden>
Review: https://review.fuel-infra.org/23720

Fix proposed to branch: openstack-ci/fuel-7.0/2015.1.0
Change author: Alex Ermolov <email address hidden>
Review: https://review.fuel-infra.org/23721

Fix proposed to branch: openstack-ci/fuel-7.0/2015.1.0
Change author: Alex Ermolov <email address hidden>
Review: https://review.fuel-infra.org/23722

Fix proposed to branch: openstack-ci/fuel-7.0/2015.1.0
Change author: Alex Ermolov <email address hidden>
Review: https://review.fuel-infra.org/23723

Fix proposed to branch: openstack-ci/fuel-7.0/2015.1.0
Change author: Alex Ermolov <email address hidden>
Review: https://review.fuel-infra.org/23724

Won't fix for 6.x series as they are unsupported and the issue has medium importance.

Verified on MOS 7.0 + mu6 updates.

Configuration:
MOS 7.0 + Contrail plugin
1 controller node, 1 compute + cinder node, 1 contrail-control + contrail-config + contrail-db node(reconfigured disk space).

Actual results:
Switching between pages and login take 1,5% less time than before, each table takes ~3 secs.

Ekaterina, that doesn't look like an improvement that we've been looking for (1.5% decrease in loading time). Is there any MOS 7.0 setup with Contrail that took a considerable time (10+ secs) to render Network/* pages before these patches were applied? I mean, if we start with setup which initially taken around 3 seconds to render pages, we would hardly notice any difference after the patches have been applied.

Fixed in 7.0 by: https://review.fuel-infra.org/#/c/26115/

Fix proposed to branch: mcp/newton
Change author: Timur Sufiev <email address hidden>
Review: https://review.fuel-infra.org/33897

Fix proposed to branch: mcp/newton
Change author: Timur Sufiev <email address hidden>
Review: https://review.fuel-infra.org/33911

Fix proposed to branch: 11.0/ocata
Change author: Timur Sufiev <email address hidden>
Review: https://review.fuel-infra.org/33934

Fix proposed to branch: 11.0/ocata
Change author: Timur Sufiev <email address hidden>
Review: https://review.fuel-infra.org/33950

Fix proposed to branch: mcp/newton
Change author: Timur Sufiev <email address hidden>
Review: https://review.fuel-infra.org/33969

Fix proposed to branch: 11.0/ocata
Change author: Timur Sufiev <email address hidden>
Review: https://review.fuel-infra.org/33990

Fix proposed to branch: 11.0/ocata
Change author: Timur Sufiev <email address hidden>
Review: https://review.fuel-infra.org/34064

Fix proposed to branch: 11.0/ocata
Change author: Timur Sufiev <email address hidden>
Review: https://review.fuel-infra.org/34087

Fix proposed to branch: mcp/newton
Change author: Timur Sufiev <email address hidden>
Review: https://review.fuel-infra.org/34092

Fix proposed to branch: mcp/newton
Change author: Timur Sufiev <email address hidden>
Review: https://review.fuel-infra.org/34128

Fix proposed to branch: 11.0/ocata
Change author: Timur Sufiev <email address hidden>
Review: https://review.fuel-infra.org/34136

Fix proposed to branch: mcp/newton
Change author: Timur Sufiev <email address hidden>
Review: https://review.fuel-infra.org/34190

Fix proposed to branch: 11.0/ocata
Change author: Timur Sufiev <email address hidden>
Review: https://review.fuel-infra.org/34196

Fix proposed to branch: mcp/newton
Change author: Timur Sufiev <email address hidden>
Review: https://review.fuel-infra.org/34230

Fix proposed to branch: mcp/ocata
Change author: Timur Sufiev <email address hidden>
Review: https://review.fuel-infra.org/34728

Fix proposed to branch: mcp/ocata
Change author: Timur Sufiev <email address hidden>
Review: https://review.fuel-infra.org/34745

Fix proposed to branch: mcp/ocata
Change author: Timur Sufiev <email address hidden>
Review: https://review.fuel-infra.org/34767

Fix proposed to branch: mcp/ocata
Change author: Timur Sufiev <email address hidden>
Review: https://review.fuel-infra.org/34808

Fix proposed to branch: mcp/ocata
Change author: Timur Sufiev <email address hidden>
Review: https://review.fuel-infra.org/34824

Fix proposed to branch: mcp/ocata
Change author: Timur Sufiev <email address hidden>
Review: https://review.fuel-infra.org/34853

Fix proposed to branch: mcp/ocata
Change author: Timur Sufiev <email address hidden>
Review: https://review.fuel-infra.org/34875

Change abandoned by Ihor Kalnytskyi <email address hidden> on branch: 11.0/ocata
Review: https://review.fuel-infra.org/34196
Reason: 11.0/ocata is obsolete. We use mcp/ocata instead.

Change abandoned by Ihor Kalnytskyi <email address hidden> on branch: 11.0/ocata
Review: https://review.fuel-infra.org/33934
Reason: 11.0/ocata is obsolete. We use mcp/ocata instead.

Change abandoned by Ihor Kalnytskyi <email address hidden> on branch: 11.0/ocata
Review: https://review.fuel-infra.org/33950
Reason: 11.0/ocata is obsolete. We use mcp/ocata instead.

Change abandoned by Ihor Kalnytskyi <email address hidden> on branch: 11.0/ocata
Review: https://review.fuel-infra.org/33990
Reason: 11.0/ocata is obsolete. We use mcp/ocata instead.

Change abandoned by Ihor Kalnytskyi <email address hidden> on branch: 11.0/ocata
Review: https://review.fuel-infra.org/34064
Reason: 11.0/ocata is obsolete. We use mcp/ocata instead.

Change abandoned by Ihor Kalnytskyi <email address hidden> on branch: 11.0/ocata
Review: https://review.fuel-infra.org/34087
Reason: 11.0/ocata is obsolete. We use mcp/ocata instead.

Change abandoned by Ihor Kalnytskyi <email address hidden> on branch: 11.0/ocata
Review: https://review.fuel-infra.org/34136
Reason: 11.0/ocata is obsolete. We use mcp/ocata instead.

Reviewed: https://review.fuel-infra.org/34875
Submitter: Pkgs Jenkins <email address hidden>
Branch: mcp/ocata

Commit: 47115a271bfa77875f48493c902aeb21ad9e6356
Author: Ivan Kolodyazhny <email address hidden>
Date: Thu Apr 27 10:23:53 2017

Fix duplicating <network_entity>_list() calls in api.network

The solution is trivial - to add @memoize decorator to the following
calls:
* api.neutron.subnet_list
* api.neutron.port_list
* api.network.tenant_floating_ip_list
* api.network.security_group_list

Partial-Bug: #1593456
Change-Id: I73498ec07166e8910a700b577b8b32f46d032d86

Reviewed: https://review.fuel-infra.org/34808
Submitter: Pkgs Jenkins <email address hidden>
Branch: mcp/ocata

Commit: 7d03f243c68c9c1052bcb8f7dba17f240e930b4d
Author: Timur Sufiev <email address hidden>
Date: Wed Apr 26 16:22:51 2017

Make @memoize thread-aware

If 2 parallel threads simultaneously make an expensive call to a
memoized function with arguments for which the returned value wasn's
cached before, 2 calls are effectively made in parallel.

It's better to avoid such a situation and if one thread already
initiated an expensive call, block on it in another thread and once
the call in the first thread is completed, unblock and reuse the value
which is cached now.

Partial-Bug: #1593456
Change-Id: I99c6d4eac5cce7bdc9ad059d1d08d2f457e7f492

Reviewed: https://review.fuel-infra.org/34824
Submitter: Pkgs Jenkins <email address hidden>
Branch: mcp/ocata

Commit: fc39869f9baef022c5de5e74b9852afcba84abe6
Author: Timur Sufiev <email address hidden>
Date: Fri Apr 28 11:13:07 2017

Prepare for parallelization of fetching quotas / usages data

Partial-Bug: #1593456
Change-Id: I0c120f2a1db87f8c660b4ca99ebd27f03ede8398

Change abandoned by Ivan Kolodyazhny <email address hidden> on branch: mcp/ocata
Review: https://review.fuel-infra.org/34745
Reason: Will be implemented in scope of https://mirantis.jira.com/browse/PROD-10715

Change abandoned by Ivan Kolodyazhny <email address hidden> on branch: mcp/ocata
Review: https://review.fuel-infra.org/34728
Reason: Will be implemented in scope of https://mirantis.jira.com/browse/PROD-10715

Change abandoned by Ivan Kolodyazhny <email address hidden> on branch: mcp/ocata
Review: https://review.fuel-infra.org/34853
Reason: Will be implemented in scope of https://mirantis.jira.com/browse/PROD-10715

Change abandoned by Ivan Kolodyazhny <email address hidden> on branch: mcp/ocata
Review: https://review.fuel-infra.org/34767
Reason: Will be implemented in scope of https://mirantis.jira.com/browse/PROD-10715

Change abandoned by Michael Dovgal <email address hidden> on branch: mcp/newton
Review: https://review.fuel-infra.org/33969

Change abandoned by Michael Dovgal <email address hidden> on branch: mcp/newton
Review: https://review.fuel-infra.org/34190

Change abandoned by Michael Dovgal <email address hidden> on branch: mcp/newton
Review: https://review.fuel-infra.org/33911

Change abandoned by Michael Dovgal <email address hidden> on branch: mcp/newton
Review: https://review.fuel-infra.org/33897

Change abandoned by Michael Dovgal <email address hidden> on branch: mcp/newton
Review: https://review.fuel-infra.org/34128

Reviewed: https://review.fuel-infra.org/34230
Submitter: Pkgs Jenkins <email address hidden>
Branch: mcp/newton

Commit: a03565216edade22ac40c11fcdade3a48600b155
Author: Timur Sufiev <email address hidden>
Date: Mon May 8 13:19:07 2017

Fix duplicating <network_entity>_list() calls in api.network

The solution is trivial - to add @memoize decorator to the following
calls:
* api.neutron.subnet_list
* api.neutron.port_list
* api.network.tenant_floating_ip_list
* api.network.security_group_list

Partial-Bug: #1593456
Change-Id: I73498ec07166e8910a700b577b8b32f46d032d86

Reviewed: https://review.fuel-infra.org/34092
Submitter: Pkgs Jenkins <email address hidden>
Branch: mcp/newton

Commit: 17cc1106d1aa1c3975cce466d70abc20a5a5d778
Author: Timur Sufiev <email address hidden>
Date: Mon May 8 12:25:46 2017

Make @memoize thread-aware

If 2 parallel threads simultaneously make an expensive call to a
memoized function with arguments for which the returned value wasn's
cached before, 2 calls are effectively made in parallel.

It's better to avoid such a situation and if one thread already
initiated an expensive call, block on it in another thread and once
the call in the first thread is completed, unblock and reuse the value
which is cached now.

Partial-Bug: #1593456
Change-Id: I99c6d4eac5cce7bdc9ad059d1d08d2f457e7f492