analytics config from cassandra

Review in progress for https://review.opencontrail.org/33549
Submitter: Sundaresan Rajangam (<email address hidden>)

Reviewed: https://review.opencontrail.org/33549
Committed: http://github.com/Juniper/contrail-controller/commit/09c7291e2321db7d890b2d4130c0df933a2ec980
Submitter: Zuul (<email address hidden>)
Branch: master

commit 09c7291e2321db7d890b2d4130c0df933a2ec980
Author: Sundaresan Rajangam <email address hidden>
Date: Tue Jul 11 17:35:20 2017 -0700

Blueprint for analytics config from Cassandra

This spec provides details about the new scheme, where analytics
services would read the user configuration from the config database.

Change-Id: Id061895cad7f0ef735ca5e51bcbbde7f528e9ab3
Partial-Bug: #1703709

Review in progress for https://review.opencontrail.org/34033
Submitter: Zhiqiang Cui (<email address hidden>)

Review in progress for https://review.opencontrail.org/34324
Submitter: Sundaresan Rajangam (<email address hidden>)

Review in progress for https://review.opencontrail.org/34328
Submitter: Sundaresan Rajangam (<email address hidden>)

Reviewed: https://review.opencontrail.org/34328
Committed: http://github.com/Juniper/contrail-controller/commit/67a2e47b13ff5fc24a5bc5152dda4781eb431a01
Submitter: Zuul (<email address hidden>)
Branch: master

commit 67a2e47b13ff5fc24a5bc5152dda4781eb431a01
Author: Sundaresan Rajangam <email address hidden>
Date: Fri Aug 4 16:55:53 2017 -0700

Modify class VncAmqpHandle constructor

VncAmqpHandle takes service's config as an argument (args) in the
constructor and extracts rabbitmq parameters and other config parameters
like trace_file from the args. We plan to use VncAmqpHandle in analytics
services and the rabbitmq parameter names are not the same as the config
service. So, instead of passing args to VncAmqpHandle, pass the rabbitmq
config as dict.

Change-Id: Ie24f08c7d027925a17b204cbe994c0cb72e7a53c
Related-Bug: #1703709

Review in progress for https://review.opencontrail.org/34369
Submitter: Sundaresan Rajangam (<email address hidden>)

Review in progress for https://review.opencontrail.org/34033
Submitter: Zhiqiang Cui (<email address hidden>)

Reviewed: https://review.opencontrail.org/34369
Committed: http://github.com/Juniper/contrail-ansible-internal/commit/7517668643b85bcc823e41d35a1ec5d6b12d8859
Submitter: Zuul (<email address hidden>)
Branch: master

commit 7517668643b85bcc823e41d35a1ec5d6b12d8859
Author: Sundaresan Rajangam <email address hidden>
Date: Mon Aug 7 12:10:56 2017 -0700

Add rabbitmq and config cassandra in analytics

At present, analytics services read user configuration directly from
contrail-api through REST API. We are changing the scheme to read the
user configuration from config cassandra and receive config change
notifications from rabbitmq.

Change-Id: I3096ca2f7fe982b6c1e3f39930e223cdf83356b2
Partial-Bug: #1703709

Review in progress for https://review.opencontrail.org/34450
Submitter: Zhiqiang Cui (<email address hidden>)

Review in progress for https://review.opencontrail.org/34033
Submitter: Zhiqiang Cui (<email address hidden>)

Review in progress for https://review.opencontrail.org/34608
Submitter: Sundaresan Rajangam (<email address hidden>)

Reviewed: https://review.opencontrail.org/34608
Committed: http://github.com/Juniper/contrail-controller/commit/ac5023f90a43f19af197eca3c0e4e284dc03b8c7
Submitter: Zuul (<email address hidden>)
Branch: master

commit ac5023f90a43f19af197eca3c0e4e284dc03b8c7
Author: Sundaresan Rajangam <email address hidden>
Date: Tue Aug 15 14:13:18 2017 -0700

alarm-gen changes to read user config from cassandra

At present, contrail-alarm-gen reads the user configuration from
contrail-api and listens to config notifications from rabbitmq.
The following changes are made in contrail-alarm-gen:
- No longer connects to contrail-api.
- On startup, reads the user configuration directly from config
cassandra database.
- Connects to rabbitmq to listen for user config changes for alarm
objects.

Change-Id: I31d5ff39410c5e937c15eea6aa5ccf74f0e8e7e9
Partial-Bug: #1703709

Review in progress for https://review.opencontrail.org/34033
Submitter: Zhiqiang Cui (<email address hidden>)

Review in progress for https://review.opencontrail.org/34896
Submitter: Zhiqiang Cui (<email address hidden>)

Review in progress for https://review.opencontrail.org/34033
Submitter: Zhiqiang Cui (<email address hidden>)

Review in progress for https://review.opencontrail.org/34896
Submitter: Zhiqiang Cui (<email address hidden>)

Review in progress for https://review.opencontrail.org/34033
Submitter: Zhiqiang Cui (<email address hidden>)

Reviewed: https://review.opencontrail.org/34033
Committed: http://github.com/Juniper/contrail-controller/commit/77b4298630b556345b1940cbdbf6f1a641e623af
Submitter: Zuul (<email address hidden>)
Branch: master

commit 77b4298630b556345b1940cbdbf6f1a641e623af
Author: zcui <email address hidden>
Date: Wed Jul 26 10:28:45 2017 -0700

Add rabbitmq and cassandra config in contrail-collector

At present, contrail-collector service reads the configuration from the
contrail-api periodically.We are changing the scheme to get the config
notification from rabbitmq and read the configurationfrom config
db (cassandra)

This is the first commit to decouple config client management from
ifmap. The implement move common code of config client management to
src/config/, and at same time move json parser related code to json
parser base class. Class json parser base involve json parse's common
input and output. User need implement derived class and rewrite
virtual function receive to deal with json document output, and
wite virtual fuction:
setup_schema_graph_filter
setup_schema_wrapper_property_info
setup_objector_filter
to filter object/class needed by user.

Change detail:
1. Except config_json_parser*, move other files of ifmap/client to
config/config-client, involve sandesh file.
2. decouple some attribute of ifmap .sandesh to config-client-mgr/
.sandes
3. generate config_json_parse_base*, move class ConfigClientManager's

link_name_map_
parent_name_map_
wrapper_field_map_
obj_type_to_read_
and data-related function

to class config_json_parse_base since they are the input data of json
decode and encode, lib user can implement derived class and rewrite

setup_schema_graph_filter
setup_schema_wrapper_property_info
setup_objector_filter

to setup minimal input data for json decode and encode.
4. rewrite delete colume code but keep cache.
Old code according to cache to generate ifmap DB request list
directly, which couple common fucntion to ifmap. This implement
to define output of delete as ConfigCass2JsonAdapter, which is
same to add/update. To implement this, we change FieldDetailMap
key from string to JsonAdapterDataType, this can keep all json
generate infos with minor change. And when need delete one colomn,
we generate ConfigCass2JsonAdapter but add_update is set to false.
Here, we have a special senario: propertymap/property list. If
list/map is not empty, ifmap DB can delete old priorities and
add/update new priorities internal, only with one update message,
only when list/map is empty, a deleted message is needed.
For example, to one property map with property 1,2,3, and after
that, update with property 4,5, only one message with update 4,5
is enough to ifmap DB. Ifmap DB will delete 1,2,3 and create 4,5.
But from lib common view, since cache has been implemented
internal in cassadra DB client, we should notify update and delete
by property. To this case, we need notify update 4,5, and notify
1,2,3 delete at same time for other lib user. To ifmap DB, we give
a new API: IsListOrMapPropertyEmpty with parameter uuid and
metaname to check if property map/list is empty.

Change-Id: Ib07801587a8dcb6d0121d8608a283e03841ce8d8
Partial-Bug: 1703709

Review in progress for https://review.opencontrail.org/35205
Submitter: Zhiqiang Cui (<email address hidden>)

Review in progress for https://review.opencontrail.org/35591
Submitter: Sundaresan Rajangam (<email address hidden>)

Review in progress for https://review.opencontrail.org/35205
Submitter: Zhiqiang Cui (<email address hidden>)

Reviewed: https://review.opencontrail.org/35591
Committed: http://github.com/Juniper/contrail-controller/commit/eb5cf95db5b1dd8f45544aeaa98cadfda304a3a7
Submitter: Zuul (<email address hidden>)
Branch: master

commit eb5cf95db5b1dd8f45544aeaa98cadfda304a3a7
Author: Sundaresan Rajangam <email address hidden>
Date: Wed Sep 13 14:30:32 2017 -0700

analytics changes to read config from cassandra

At present, contrail-snmp-collector and contrail-topology reads the user
configuration from contrail-api periodically. This patch changes
the scheme to read the user configuration from config database
(cassandra).
- contrail-snmp-collector and contrail-topology no longer connect to
contrail-api.
- On startup, reads the user configuration directly from config
cassandra database.
- Connects to rabbitmq to listen for the user configuration changes for
the desired user configuration objects.

Partial-Bug: #1703709

This patch contains the changes to read user configuration from

Change-Id: I6294dce8c5759bc40538ca93ce7057640bd88d42

Review in progress for https://review2.opencontrail.org/34033
Submitter: Krzysztof Klimonda (<email address hidden>)

Reviewed: https://review2.opencontrail.org/34033
Committed: http://github.com/Juniper/contrail-controller/commit/77b4298630b556345b1940cbdbf6f1a641e623af
Submitter: Krzysztof Klimonda (<email address hidden>)
Branch: master

commit 77b4298630b556345b1940cbdbf6f1a641e623af
Author: zcui <email address hidden>
Date: Wed Jul 26 10:28:45 2017 -0700

Add rabbitmq and cassandra config in contrail-collector

At present, contrail-collector service reads the configuration from the
contrail-api periodically.We are changing the scheme to get the config
notification from rabbitmq and read the configurationfrom config
db (cassandra)

This is the first commit to decouple config client management from
ifmap. The implement move common code of config client management to
src/config/, and at same time move json parser related code to json
parser base class. Class json parser base involve json parse's common
input and output. User need implement derived class and rewrite
virtual function receive to deal with json document output, and
wite virtual fuction:
setup_schema_graph_filter
setup_schema_wrapper_property_info
setup_objector_filter
to filter object/class needed by user.

Change detail:
1. Except config_json_parser*, move other files of ifmap/client to
config/config-client, involve sandesh file.
2. decouple some attribute of ifmap .sandesh to config-client-mgr/
.sandes
3. generate config_json_parse_base*, move class ConfigClientManager's

link_name_map_
parent_name_map_
wrapper_field_map_
obj_type_to_read_
and data-related function

to class config_json_parse_base since they are the input data of json
decode and encode, lib user can implement derived class and rewrite

setup_schema_graph_filter
setup_schema_wrapper_property_info
setup_objector_filter

to setup minimal input data for json decode and encode.
4. rewrite delete colume code but keep cache.
Old code according to cache to generate ifmap DB request list
directly, which couple common fucntion to ifmap. This implement
to define output of delete as ConfigCass2JsonAdapter, which is
same to add/update. To implement this, we change FieldDetailMap
key from string to JsonAdapterDataType, this can keep all json
generate infos with minor change. And when need delete one colomn,
we generate ConfigCass2JsonAdapter but add_update is set to false.
Here, we have a special senario: propertymap/property list. If
list/map is not empty, ifmap DB can delete old priorities and
add/update new priorities internal, only with one update message,
only when list/map is empty, a deleted message is needed.
For example, to one property map with property 1,2,3, and after
that, update with property 4,5, only one message with update 4,5
is enough to ifmap DB. Ifmap DB will delete 1,2,3 and create 4,5.
But from lib common view, since cache has been implemented
internal in cassadra DB client, we should notify update and delete
by property. To this case, we need notify update 4,5, and notify
1,2,3 delete at same time for other lib user. To ifmap DB, we give
a new API: IsListOrMapPropertyEmpty with parameter uuid and
metaname to check if property map/list is empty.

Change-Id: Ib07801587a8dcb6d0121d8608a283e03841ce8d8
Parti...