API Server Performance Optimization Placeholder bug

Review in progress for https://review.opencontrail.org/20974
Submitter: Ranjeet R (<email address hidden>)

Review in progress for https://review.opencontrail.org/21028
Submitter: Ranjeet R (<email address hidden>)

Reviewed: https://review.opencontrail.org/21028
Committed: http://github.org/Juniper/contrail-controller/commit/7d6cd7b34e579bbc2c4019e2c4d76679eee0102f
Submitter: Zuul
Branch: master

commit 7d6cd7b34e579bbc2c4019e2c4d76679eee0102f
Author: Ranjeet R <email address hidden>
Date: Wed Jun 8 16:29:49 2016 -0700

Fixes: API Server Performance Optimization Placeholder bug

Test setup
20000 VMs 20000 VMIs 20000 Instance IPs
API server startup takes 6 minutes 7 seconds to start

This commit has two improvements -
1. In Dbe_resync, we yield the greenlet to make sure that
Zookeeper heartbeats are working. But during startup, ZK
heartbeats are not relevant. Improves perf from 6:07 to 4:35.

2. In IFMAP publish call, the cache organization was not
efficient. We use the following schema now.

id_to_metas = { 'id': //id1
{'meta_name':
{'': prop_value} or // If the meta is a prop
{id2:ref_value} //If the meta is a ref
changes in object_delete/update to take care of this organization.
Improves perf from 4:35 to 3:35
Updating UTs to reflect this/

3. Eduoard's change of using ujson instead of simplejson in
https://review.opencontrail.org/21049 brings down the time to
another 20 seconds, 3:35 to 315

4. Not sending the VM's UVE trace during startup
is saving another 25 seconds
3:15 to 2:50

Change-Id: I473ed25b09f7b59e82e2bc77e2ce1f89d9be0661
Partial-Bug: 1589014

Review in progress for https://review.opencontrail.org/21274
Submitter: Ranjeet R (<email address hidden>)

Review in progress for https://review.opencontrail.org/21279
Submitter: Ranjeet R (<email address hidden>)

Reviewed: https://review.opencontrail.org/21274
Committed: http://github.org/Juniper/contrail-controller/commit/825f26b3f6ab67683030dfcbfb7107185da6d1e9
Submitter: Zuul
Branch: R3.0

commit 825f26b3f6ab67683030dfcbfb7107185da6d1e9
Author: Ranjeet R <email address hidden>
Date: Fri Jun 17 10:58:02 2016 -0700

Fixes: API Server Performance Optimization Placeholder bug

Test setup
20000 VMs 20000 VMIs 20000 Instance IPs
API server startup takes 6 minutes 7 seconds to start

This commit has two improvements -
1. In Dbe_resync, we yield the greenlet to make sure that
Zookeeper heartbeats are working. But during startup, ZK
heartbeats are not relevant. Improves perf from 6:07 to 4:35.

2. In IFMAP publish call, the cache organization was not
efficient. We use the following schema now.

id_to_metas = { 'id': //id1
{'meta_name':
{'': prop_value} or // If the meta is a prop
{id2:ref_value} //If the meta is a ref
changes in object_delete/update to take care of this organization.
Improves perf from 4:35 to 3:35
Updating UTs to reflect this/

3. Eduoard's change of using ujson instead of simplejson in
https://review.opencontrail.org/21049 brings down the time to
another 20 seconds, 3:35 to 315

4. Not sending the VM's UVE trace during startup
is saving another 25 seconds
3:15 to 2:50

Change-Id: I23a6f1c039683624aee23b7fe16539f0aa1f7537
Partial-Bug: 1589014

Review in progress for https://review.opencontrail.org/21303
Submitter: Ranjeet R (<email address hidden>)

Review in progress for https://review.opencontrail.org/21279
Submitter: Ranjeet R (<email address hidden>)

Reviewed: https://review.opencontrail.org/21279
Committed: http://github.org/Juniper/contrail-provisioning/commit/b07adcb09ec263581421c543169fde008c4d0641
Submitter: Zuul
Branch: master

commit b07adcb09ec263581421c543169fde008c4d0641
Author: Ranjeet R <email address hidden>
Date: Fri Jun 17 19:01:27 2016 -0700

Fixes: API Server Performance Optimization Placeholder bug

Only producer for IfMap server is the API server. IfMap
server need not validate the XML sent by API server.

Change-Id: I1543b09478863a51942a56ba143cc50962762c11
Partial-Bug: 1589014

Review in progress for https://review.opencontrail.org/21349
Submitter: ?douard Thuleau (<email address hidden>)

Review in progress for https://review.opencontrail.org/21349
Submitter: Édouard Thuleau (<email address hidden>)

Review in progress for https://review.opencontrail.org/21490
Submitter: Édouard Thuleau (<email address hidden>)

Reviewed: https://review.opencontrail.org/21349
Committed: http://github.org/Juniper/contrail-controller/commit/7c2b1e37c9e81341974ff7afcf2cbff1671ce587
Submitter: Zuul
Branch: master

commit 7c2b1e37c9e81341974ff7afcf2cbff1671ce587
Author: Édouard Thuleau <email address hidden>
Date: Wed Jun 22 18:45:21 2016 +0200

[VNC API server] Optimize sending UVEs during initializing

Update the method 'vnc_cfg_ifmap.VncDbClient.dbe_uve_trace' that's
sending UVEs to be faster and use a greenlet group to parallelized it

Change-Id: I548711065a9d46b4119b8636fc9e23f67d9d2835
Partial-Bug: #1589014

Review in progress for https://review.opencontrail.org/21548
Submitter: Édouard Thuleau (<email address hidden>)

Review in progress for https://review.opencontrail.org/21550
Submitter: Hampapur Ajay (<email address hidden>)

Reviewed: https://review.opencontrail.org/21550
Committed: http://github.org/Juniper/contrail-generateDS/commit/7db3d53ef47158fd203ba15d53b2d5c5651bd7fc
Submitter: Zuul
Branch: master

commit 7db3d53ef47158fd203ba15d53b2d5c5651bd7fc
Author: Hampapur Ajay <email address hidden>
Date: Wed Jun 29 11:36:30 2016 -0700

Use unicode for key access in kwargs for from_dict to reduce time
spent in utf_8:decode(). Also and don't expand to params for same
reason. Prefer values in params_dict to arg values so from_dict is
more efficient.

Change-Id: I00ed5d7825aab4919f84fd9416ebd6c5a017cb00
Partial-Bug: 1589014

Review in progress for https://review.opencontrail.org/21548
Submitter: Édouard Thuleau (<email address hidden>)

Reviewed: https://review.opencontrail.org/21490
Committed: http://github.org/Juniper/contrail-controller/commit/10c96256a83c61ff15790afb6cbda8dc326a9a56
Submitter: Zuul
Branch: master

commit 10c96256a83c61ff15790afb6cbda8dc326a9a56
Author: Édouard Thuleau <email address hidden>
Date: Tue Jun 28 16:21:47 2016 +0200

[VNC API server] Improve cassandra object_read method

Use pycassa 'multiget' method when it's possible.
Two cases can be distinguished:
- cannot determine return size when fetching multi keys with start and
finish column conditions. In that case try to fetch data with
multiget and if it raises 'OverflowError' try to fetch data with
'xget' method.
- when the column length is defined, we calculate if we can fetch all
data in one 'multiget' call, if not use all necessary 'get' calls

Change-Id: I2497ea365e25e7cfa6abd8c35b36929f7b5e7806
Partial-Bug: #1589014

Reviewed: https://review.opencontrail.org/21548
Committed: http://github.org/Juniper/contrail-controller/commit/cad59fa27ea7847010b95ef41b216355bc0120e9
Submitter: Zuul
Branch: master

commit cad59fa27ea7847010b95ef41b216355bc0120e9
Author: Édouard Thuleau <email address hidden>
Date: Wed Jun 29 18:37:30 2016 +0200

[VNC API server] Implement xml [un]escape special characters

Replace XML sax [un]escape methods by a method that check if string need
to be [un]escaped before calling string replace method.

Change-Id: Ie67c632b02308877e9b1998dc70450dc90fb86d6
Partial-bug: #1589014

Review in progress for https://review.opencontrail.org/21720
Submitter: Hampapur Ajay (<email address hidden>)

Review in progress for https://review.opencontrail.org/21722
Submitter: Hampapur Ajay (<email address hidden>)

Reviewed: https://review.opencontrail.org/21722
Committed: http://github.org/Juniper/contrail-controller/commit/c61d6902ddd74d3d08baea92d36bf80641e164dd
Submitter: Zuul
Branch: master

commit c61d6902ddd74d3d08baea92d36bf80641e164dd
Author: Hampapur Ajay <email address hidden>
Date: Thu Jul 7 07:02:49 2016 -0700

Check for ip-addr membership in subnet using IPNetwork instead of
IPSet as it is way more efficient (improving port-list operations).

Change-Id: Ic73a32e0c371ddae3a00f4eff56f7e774f36b9ba
Partial-Bug: 1589014

Review in progress for https://review.opencontrail.org/21767
Submitter: Ranjeet R (<email address hidden>)

Reviewed: https://review.opencontrail.org/21767
Committed: http://github.org/Juniper/contrail-controller/commit/a1ac0e02647317d0fe9f8ea15e285862c97021c6
Submitter: Zuul
Branch: master

commit a1ac0e02647317d0fe9f8ea15e285862c97021c6
Author: Ranjeet R <email address hidden>
Date: Fri Jul 8 13:19:57 2016 -0700

Fixes: Not sending IdPerms and Perms2 to IFMAP improves perf. API Server Performance Optimization Placeholder bug

Certain common fields like IdPerms and perms2 are not relevant
to be published or published completely to ifmap.

We are still publishing it to IfMAP since they are part of the
schema. Addiing infra to skip publishing fields not relevant
to IfMap completely or partially publish it to IfMap.

For 20k VMs, 20k VMIs, and 60k objects in total, the Init time
stands at 1:28

Change-Id: Ib36aafd4c612472f9d7fc53b74274dbc62a67f7c
Partial-Bug: 1589014

Review in progress for https://review.opencontrail.org/21923
Submitter: Sachin Bansal (<email address hidden>)

Reviewed: https://review.opencontrail.org/21923
Committed: http://github.org/Juniper/contrail-controller/commit/8b90cf3fbbcf2d3bdff360744558e06a7fe90de7
Submitter: Zuul
Branch: master

commit 8b90cf3fbbcf2d3bdff360744558e06a7fe90de7
Author: Sachin Bansal <email address hidden>
Date: Tue Jul 12 17:40:51 2016 -0700

Fix broken vnc_openstack unit tests

1. Do not create default sg. This is created by api server
2. If subnet_uuid is specified as None, generate one

Change-Id: Ib190829beeacb0a28bf74b2da52ad792b049fcca
Related-Bug: 1589014

Review in progress for https://review.opencontrail.org/22048
Submitter: Sachin Bansal (<email address hidden>)

Reviewed: https://review.opencontrail.org/22048
Committed: http://github.org/Juniper/contrail-controller/commit/0eae591ef56420ff4a9674b63a029db0f9edb7e9
Submitter: Zuul
Branch: R3.1

commit 0eae591ef56420ff4a9674b63a029db0f9edb7e9
Author: Sachin Bansal <email address hidden>
Date: Tue Jul 12 17:40:51 2016 -0700

Fix broken vnc_openstack unit tests

1. Ignore RefsExistError while creating default SG
2. If subnet_uuid is specified as None, generate one
3. Create new tenants in keystone mock so that they are not deleted by resync thread

Change-Id: Ib190829beeacb0a28bf74b2da52ad792b049fcca
Related-Bug: 1589014
Closes-Bug: 1603380
(cherry picked from commit 8b90cf3fbbcf2d3bdff360744558e06a7fe90de7)
(cherry picked from commit 16aaa54dc373b02a4e89d9e81e3b1018ab208e74)

Review in progress for https://review.opencontrail.org/21720
Submitter: Hampapur Ajay (<email address hidden>)

Reviewed: https://review.opencontrail.org/21720
Committed: http://github.org/Juniper/contrail-controller/commit/61c9332d345f0062f529b346ebc37934e2b228ed
Submitter: Zuul
Branch: master

commit 61c9332d345f0062f529b346ebc37934e2b228ed
Author: Hampapur Ajay <email address hidden>
Date: Thu Jul 7 06:11:19 2016 -0700

Set autopack_names and _values to False when using obj_uuid_cf to
avoid pycassa having to decode from stored values as it is always
of string/unicode type. This speeds up list and startup operations.
Pass additional cf kwargs to pycassa as is.

Change-Id: I5b35f49d5ad5e8b772aa7e881124f4910f96192a
Partial-Bug: 1589014

Review in progress for https://review.opencontrail.org/22834
Submitter: Hampapur Ajay (<email address hidden>)

Review in progress for https://review.opencontrail.org/22835
Submitter: Hampapur Ajay (<email address hidden>)

Review in progress for https://review.opencontrail.org/22836
Submitter: Hampapur Ajay (<email address hidden>)

Reviewed: https://review.opencontrail.org/22836
Committed: http://github.org/Juniper/contrail-controller/commit/667a3cf249ad689f9c6a7c90268ac633226a7a48
Submitter: Zuul
Branch: master

commit 667a3cf249ad689f9c6a7c90268ac633226a7a48
Author: Hampapur Ajay <email address hidden>
Date: Wed Aug 3 18:33:59 2016 -0700

Check membership in set instead of list, in scale *way* more performant.

Change-Id: I23dc40fa954e8d82be9b460affb57ff44673d6ee
Partial-Bug: 1589014

Review in progress for https://review.opencontrail.org/22904
Submitter: Sachin Bansal (<email address hidden>)

Review in progress for https://review.opencontrail.org/22905
Submitter: Hampapur Ajay (<email address hidden>)

Reviewed: https://review.opencontrail.org/22835
Committed: http://github.org/Juniper/contrail-controller/commit/f547ee90978b6f710fa2a8c2d115b7a044f62e9c
Submitter: Zuul
Branch: R3.1

commit f547ee90978b6f710fa2a8c2d115b7a044f62e9c
Author: Hampapur Ajay <email address hidden>
Date: Thu Jul 7 06:11:19 2016 -0700

Set autopack_names and _values to False when using obj_uuid_cf to
avoid pycassa having to decode from stored values as it is always
of string/unicode type. This speeds up list and startup operations.
Pass additional cf kwargs to pycassa as is.

Change-Id: I5b35f49d5ad5e8b772aa7e881124f4910f96192a
Partial-Bug: 1589014
(cherry picked from commit 61c9332d345f0062f529b346ebc37934e2b228ed)

Reviewed: https://review.opencontrail.org/22905
Committed: http://github.org/Juniper/contrail-generateDS/commit/b47a1ef8312d9266382c5043d1952246b6c1419a
Submitter: Zuul
Branch: R3.1

commit b47a1ef8312d9266382c5043d1952246b6c1419a
Author: Hampapur Ajay <email address hidden>
Date: Tue Aug 2 12:13:07 2016 -0700

Use params_dict for nested type member.
For e.g. in class IdPermsType:__init__
instead of
if isinstance(_permissions, dict):
obj = PermType(**_permissions)
self.permissions = obj
use
if isinstance(_permissions, dict):
obj = PermType(params_dict=_permissions)
self.permissions = obj
this saves on unicode to string conversion.

Change-Id: I52d644115d757fd9b59a472e2967baf959976d8f
Partial-Bug: 1589014
(cherry picked from commit 4a09a88c162467ee4f53c88ee01b9e5cf0f41b8b)

Reviewed: https://review.opencontrail.org/22904
Committed: http://github.org/Juniper/contrail-controller/commit/28ee8bbec1ac1ee91c08cf54f288680956f4cf98
Submitter: Zuul
Branch: R3.1

commit 28ee8bbec1ac1ee91c08cf54f288680956f4cf98
Author: Hampapur Ajay <email address hidden>
Date: Wed Aug 3 18:33:59 2016 -0700

Check membership in set instead of list, in scale *way* more performant.

Change-Id: I23dc40fa954e8d82be9b460affb57ff44673d6ee
Partial-Bug: 1589014
(cherry picked from commit 667a3cf249ad689f9c6a7c90268ac633226a7a48)

Review in progress for https://review.opencontrail.org/23120
Submitter: Sachin Bansal (<email address hidden>)

Reviewed: https://review.opencontrail.org/23120
Committed: http://github.org/Juniper/contrail-controller/commit/7dc702bc1bce98a0955401fe355c5da7a8a947f8
Submitter: Zuul
Branch: master

commit 7dc702bc1bce98a0955401fe355c5da7a8a947f8
Author: Sachin Bansal <email address hidden>
Date: Tue Aug 9 15:40:49 2016 -0700

Build a uuid-fqname cache in all config daemons on start up

This can help in converting from uuid to type or fq name later
Partial-Bug: 1589014

Change-Id: Ib984d16a75e61dbfe732c199770ff71967e3b2f8

Reviewed: https://review.opencontrail.org/22834
Committed: http://github.org/Juniper/contrail-generateDS/commit/4a09a88c162467ee4f53c88ee01b9e5cf0f41b8b
Submitter: Zuul
Branch: master

commit 4a09a88c162467ee4f53c88ee01b9e5cf0f41b8b
Author: Hampapur Ajay <email address hidden>
Date: Tue Aug 2 12:13:07 2016 -0700

Use params_dict for nested type member.
For e.g. in class IdPermsType:__init__
instead of
if isinstance(_permissions, dict):
obj = PermType(**_permissions)
self.permissions = obj
use
if isinstance(_permissions, dict):
obj = PermType(params_dict=_permissions)
self.permissions = obj
this saves on unicode to string conversion.

Change-Id: I52d644115d757fd9b59a472e2967baf959976d8f
Partial-Bug: 1589014

Review in progress for https://review.opencontrail.org/23701
Submitter: Hampapur Ajay (<email address hidden>)

Related Bugs:
https://bugs.launchpad.net/juniperopenstack/+bug/1605723 https://bugs.launchpad.net/juniperopenstack/+bug/1605725

Review in progress for https://review.opencontrail.org/23701
Submitter: Hampapur Ajay (<email address hidden>)

Reviewed: https://review.opencontrail.org/23701
Committed: http://github.org/Juniper/contrail-controller/commit/32ca40a8060db41f9e0548bed8abf1fe98c1960c
Submitter: Zuul
Branch: master

commit 32ca40a8060db41f9e0548bed8abf1fe98c1960c
Author: Hampapur Ajay <email address hidden>
Date: Wed Aug 24 17:05:07 2016 -0700

Add support for workers for contrail-api.

With this by changing numprocs in [program:contrail-api]
section in /etc/contrail/supervisord_config_files/contrail-api.ini
more processes can be used to serve northbound API.

Only worker 0 is consumer to rabbitmq and publishes to ifmap.
Other workers will serve REST contrail API and publish to
rabbitmq (for create/update/delete).

Partial-Bug: 1589014

Change-Id: I2a8c060ca04e13bc7b7700fc6eeaa673aa18ed32

Review in progress for https://review.opencontrail.org/24090
Submitter: Hampapur Ajay (<email address hidden>)

Reviewed: https://review.opencontrail.org/24090
Committed: http://github.org/Juniper/contrail-controller/commit/ee0227c5bc3a04084821982c0f5f9df46841e8b1
Submitter: Zuul
Branch: master

commit ee0227c5bc3a04084821982c0f5f9df46841e8b1
Author: Hampapur Ajay <email address hidden>
Date: Mon Sep 12 17:00:10 2016 -0700

Only contrail-api worker 0 should sync domains/projects from keystone.

Change-Id: Idae4fde682dd5b35a75e5cacda57fc640682b715
Partial-Bug: 1589014

Review in progress for https://review.opencontrail.org/24162
Submitter: Hampapur Ajay (<email address hidden>)

Review in progress for https://review.opencontrail.org/24163
Submitter: Hampapur Ajay (<email address hidden>)

Review in progress for https://review.opencontrail.org/24164
Submitter: Hampapur Ajay (<email address hidden>)

Reviewed: https://review.opencontrail.org/24164
Committed: http://github.org/Juniper/contrail-controller/commit/c6579f3a3934f583fed61e9f0c8a0dbb75b0cfd5
Submitter: Zuul
Branch: R3.1

commit c6579f3a3934f583fed61e9f0c8a0dbb75b0cfd5
Author: Sachin Bansal <email address hidden>
Date: Tue Aug 9 15:40:49 2016 -0700

Build a uuid-fqname cache in all config daemons on start up

This can help in converting from uuid to type or fq name later
Partial-Bug: 1589014

Change-Id: Ib984d16a75e61dbfe732c199770ff71967e3b2f8
(cherry picked from commit 7dc702bc1bce98a0955401fe355c5da7a8a947f8)

Review in progress for https://review.opencontrail.org/24214
Submitter: Hampapur Ajay (<email address hidden>)

Review in progress for https://review.opencontrail.org/24215
Submitter: Hampapur Ajay (<email address hidden>)

Review in progress for https://review.opencontrail.org/24229
Submitter: Hampapur Ajay (<email address hidden>)

Reviewed: https://review.opencontrail.org/24214
Committed: http://github.org/Juniper/contrail-controller/commit/3a65b5af0dbe992f82782637eb521104e8a178f0
Submitter: Zuul
Branch: R3.1

commit 3a65b5af0dbe992f82782637eb521104e8a178f0
Author: Hampapur Ajay <email address hidden>
Date: Wed Aug 24 17:05:07 2016 -0700

Add support for workers for contrail-api.

With this by changing numprocs in [program:contrail-api]
section in /etc/contrail/supervisord_config_files/contrail-api.ini
more processes can be used to serve northbound API.

Only worker 0 is consumer to rabbitmq and publishes to ifmap.
Other workers will serve REST contrail API and publish to
rabbitmq (for create/update/delete).

Partial-Bug: 1589014

(cherry picked from commit 32ca40a8060db41f9e0548bed8abf1fe98c1960c)

Change-Id: I94141b95b06cceacfc2fcdf19a0fa3f09c66241e

Reviewed: https://review.opencontrail.org/24215
Committed: http://github.org/Juniper/contrail-controller/commit/c12eb58c26d54c8610c2110e3d997b15c90371a6
Submitter: Zuul
Branch: R3.1

commit c12eb58c26d54c8610c2110e3d997b15c90371a6
Author: Hampapur Ajay <email address hidden>
Date: Mon Sep 12 17:00:10 2016 -0700

Only contrail-api worker 0 should sync domains/projects from keystone.

Partial-Bug: 1589014
(cherry picked from commit ee0227c5bc3a04084821982c0f5f9df46841e8b1)

Change-Id: Id5d15bde8558aaed9d93ee3c4bbc682189d8f1eb

Review in progress for https://review.opencontrail.org/24316
Submitter: Sachin Bansal (<email address hidden>)

Reviewed: https://review.opencontrail.org/24316
Committed: http://github.org/Juniper/contrail-controller/commit/a3e5ccb680317813ebfadd7425b73941da053963
Submitter: Zuul
Branch: master

commit a3e5ccb680317813ebfadd7425b73941da053963
Author: Sachin Bansal <email address hidden>
Date: Tue Sep 20 14:01:57 2016 -0700

Remove uuid-fqname cache in schema transformer

Remove uuid to fq name cache in schema transformer since now we
have a cache in vnc_cassandra. This cache is already maintained
in the rabbitmq subscribe functions, so we also don't need to
overload locate and delete methods in DBBaseST class.

Partial-Bug: 1589014

Change-Id: I9f753eedefde11696452f799076099aaff593597

Reviewed: https://review.opencontrail.org/24229
Committed: http://github.org/Juniper/contrail-controller/commit/fd681cd4bc4ca922246aa9d23a436fa12729f6df
Submitter: Zuul
Branch: R3.0

commit fd681cd4bc4ca922246aa9d23a436fa12729f6df
Author: Sachin Bansal <email address hidden>
Date: Tue Aug 9 15:40:49 2016 -0700

Build a uuid-fqname cache in all config daemons on start up

This can help in converting from uuid to type or fq name later
Partial-Bug: 1589014

Change-Id: Ib984d16a75e61dbfe732c199770ff71967e3b2f8
(cherry picked from commit 7dc702bc1bce98a0955401fe355c5da7a8a947f8)