AZ list performance optimization: avoid double service list DB fetch
Assume number of services can be large (10000 as in the bug description),
this patch removes second service_get_all call.
zone_hosts changed from dict of lists to dict of sets.
The HostAPI instance from the API controller is also passed to the
get_availability_zones method so it does not have to recreate it
per call (this is both for a slight performance gain but mostly also
for test sanity).
On devstack with 10000 services patch decreased response time twice.
The test_availability_zone_detail unit test is updated to assert that
services are only retrieved twice (once for enabled, once for disabled).
While in there, the expected response dict is formatted for readability
and a duplicate zone/host is added to make sure duplicates are handled
for available services. To ensure service_get_all is called only twice,
the low-level DB API service_get_all stub is replaced with a mock and
the mock is changed to be on the HostAPI.service_get_all method which
is (1) what the API controller code is actually using and (2) allows the
test to only mock the instance of the HostAPI being tested - trying to
mock the DB API service_get_all method causes intermittent failures
in unrelated tests because of the global nature of that mock.
There is another opportunity for optimizing get_availability_zones which
is marked with a TODO but left for a separate patch.
Co-Authored-By: Matt Riedemann <email address hidden>
Partial-Bug: #1801897
Change-Id: Ib9a9a9a79499272d740a64cc0b909f0299a237d1
(cherry picked from commit 74cefe4266a613d4c2afbb0c791e16eb7789aef4)
Reviewed: https:/ /review. opendev. org/656382 /git.openstack. org/cgit/ openstack/ nova/commit/ ?id=c280d747fb2 3f1abaaf91eea7f 6d11e716c6db42
Committed: https:/
Submitter: Zuul
Branch: stable/stein
commit c280d747fb23f1a baaf91eea7f6d11 e716c6db42
Author: Andrey Volkov <email address hidden>
Date: Thu Feb 14 15:39:45 2019 +0300
AZ list performance optimization: avoid double service list DB fetch
Assume number of services can be large (10000 as in the bug description),
this patch removes second service_get_all call.
zone_hosts changed from dict of lists to dict of sets.
The HostAPI instance from the API controller is also passed to the availability_ zones method so it does not have to recreate it
get_
per call (this is both for a slight performance gain but mostly also
for test sanity).
On devstack with 10000 services patch decreased response time twice.
openstack availability zone list --long --timing
...
Before:
+-- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- -----+- ------- ---+ ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- -----+- ------- ---+ 192.168. 0.45/identity | 0.006816 | 192.168. 0.45/identity/ v3/auth/ tokens | 0.456708 | 192.168. 0.45/identity/ v3/auth/ tokens | 0.087485 | 172.18. 237.203/ compute/ v2.1/os- availability- zone/detail | 95.667192 | 172.18. 237.203/ volume/ v2/e2671d37ee2c 4374bd153364526 1f1d4/os- availability- zone | 0.036528 | ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- -----+- ------- ---+
| URL | Seconds |
+--
| GET http://
| POST http://
| POST http://
| GET http://
| GET http://
| Total | 96.254729 |
+--
After:
+-- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- -----+- ------- ---+ ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- -----+- ------- ---+ 192.168. 0.45/identity | 0.020215 | 192.168. 0.45/identity/ v3/auth/ tokens | 0.102987 | 192.168. 0.45/identity/ v3/auth/ tokens | 0.111899 | 172.18. 237.203/ compute/ v2.1/os- availability- zone/detail | 39.346657 | 172.18. 237.203/ volume/ v2/e2671d37ee2c 4374bd153364526 1f1d4/os- availability- zone | 0.026403 | ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- ------- -----+- ------- ---+
| URL | Seconds |
+--
| GET http://
| POST http://
| POST http://
| GET http://
| GET http://
| Total | 39.608161 |
+--
The test_availabili ty_zone_ detail unit test is updated to assert that service_ get_all method which
services are only retrieved twice (once for enabled, once for disabled).
While in there, the expected response dict is formatted for readability
and a duplicate zone/host is added to make sure duplicates are handled
for available services. To ensure service_get_all is called only twice,
the low-level DB API service_get_all stub is replaced with a mock and
the mock is changed to be on the HostAPI.
is (1) what the API controller code is actually using and (2) allows the
test to only mock the instance of the HostAPI being tested - trying to
mock the DB API service_get_all method causes intermittent failures
in unrelated tests because of the global nature of that mock.
There is another opportunity for optimizing get_availabilit y_zones which
is marked with a TODO but left for a separate patch.
Co-Authored-By: Matt Riedemann <email address hidden>
Partial-Bug: #1801897 2d740a64cc0b909 f0299a237d1 4c2afbb0c791e16 eb7789aef4)
Change-Id: Ib9a9a9a7949927
(cherry picked from commit 74cefe4266a613d