control-node functionality to retrieve config via cassandra

Review in progress for https://review.opencontrail.org/24799
Submitter: Tapan Karwa (<email address hidden>)

Reviewed: https://review.opencontrail.org/24799
Committed: http://github.org/Juniper/contrail-controller/commit/a559aba3a53b841c6edd151235f5a220dfba66ea
Submitter: Zuul
Branch: master

commit a559aba3a53b841c6edd151235f5a220dfba66ea
Author: Tapan Karwa <email address hidden>
Date: Mon Oct 10 16:30:52 2016 -0700

Adding a few classes for control-node retrieving config via cassandra.

Partial-Bug: #1632470

Change-Id: I42c50e29885897685a869a7ac656c5afdf6f2d74

Review in progress for https://review.opencontrail.org/25317
Submitter: Tapan Karwa (<email address hidden>)

Reviewed: https://review.opencontrail.org/25317
Committed: http://github.org/Juniper/contrail-controller/commit/2a4eeab6eb44073076db19545c8b8616c6145a2b
Submitter: Zuul
Branch: master

commit 2a4eeab6eb44073076db19545c8b8616c6145a2b
Author: Tapan Karwa <email address hidden>
Date: Tue Oct 25 13:54:53 2016 -0700

Code for control-node retrieving config via cassandra.

Add code to process config options for cassandra and start using
Db_UseColumnfamily with string.

Change-Id: Ibdfbac4860955301bab5d396b2ad988a30da8b6a
Partial-Bug: #1632470

Review in progress for https://review.opencontrail.org/26061
Submitter: Hampapur Ajay (<email address hidden>)

Reviewed: https://review.opencontrail.org/26061
Committed: http://github.org/Juniper/contrail-controller/commit/db8c947ea94365dd246195225348988697ceb991
Submitter: Zuul
Branch: master

commit db8c947ea94365dd246195225348988697ceb991
Author: Hampapur Ajay <email address hidden>
Date: Mon Nov 14 16:23:13 2016 -0800

Add specification for receiving configuration data from cassandra
to control-node. Submitting on behalf of tkarwa@

Change-Id: I2148f61007e1852a8c2595e50d82964fb50f5c3d
Partial-Bug: 1632470

Review in progress for https://review.opencontrail.org/26179
Submitter: Tapan Karwa (<email address hidden>)

Reviewed: https://review.opencontrail.org/26179
Committed: http://github.org/Juniper/contrail-controller/commit/63d6a8dca6a6fafad5d2ba39536ef8141961bf16
Submitter: Zuul (<email address hidden>)
Branch: master

commit 63d6a8dca6a6fafad5d2ba39536ef8141961bf16
Author: Tapan Karwa <email address hidden>
Date: Wed Nov 16 15:20:56 2016 -0800

Code for control-node retrieving config via cassandra.

Add code to parse json messages. Add tests and json test data.

Change-Id: I1cb667a37dd4021e34bc9e303d25987394b8b0e9
Partial-Bug: #1632470

Review in progress for https://review.opencontrail.org/26670
Submitter: Tapan Karwa (<email address hidden>)

Review in progress for https://review.opencontrail.org/26846
Submitter: Tapan Karwa (<email address hidden>)

Review in progress for https://review.opencontrail.org/26670
Submitter: Tapan Karwa (<email address hidden>)

Review in progress for https://review.opencontrail.org/27106
Submitter: Tapan Karwa (<email address hidden>)

Reviewed: https://review.opencontrail.org/27106
Committed: http://github.org/Juniper/contrail-controller/commit/90867da664858e624c5a228bae3b430fc71bd474
Submitter: Zuul (<email address hidden>)
Branch: master

commit 90867da664858e624c5a228bae3b430fc71bd474
Author: Tapan Karwa <email address hidden>
Date: Fri Dec 9 14:31:13 2016 -0800

Code for control-node retrieving config via cassandra.

Add code to read one row from cassandra and process the read columns. Also, add
code to convert the incoming column data to a json document that will finally
be fed to the json parser. The conversion is done by the ConfigCass2JsonAdapter
class and handles the quirks in the data stored in cassandra.
Also, add code to read all-rows on startup (bulk sync).

Change-Id: I0ea706554024e314772147fb35c722d008c05fd8
Partial-Bug: #1632470

Review in progress for https://review.opencontrail.org/26846
Submitter: Tapan Karwa (<email address hidden>)

Review in progress for https://review.opencontrail.org/26670
Submitter: Tapan Karwa (<email address hidden>)

Reviewed: https://review.opencontrail.org/26846
Committed: http://github.org/Juniper/contrail-controller/commit/daa49914ad6ba15d67e23125a861c73bbbae09ba
Submitter: Zuul (<email address hidden>)
Branch: master

commit daa49914ad6ba15d67e23125a861c73bbbae09ba
Author: Tapan Karwa <email address hidden>
Date: Mon Dec 5 14:29:28 2016 -0800

Code for control-node retrieving config via cassandra.

Add code to add links to parent.
Also, add code to perform a diff between previous and current json messages to
decide if properties are getting deleted. Add test for this.

Change-Id: I516a170aa97efe89564e10aa6dbf3e2a51569b12
Partial-Bug: #1632470

Reviewed: https://review.opencontrail.org/26670
Committed: http://github.org/Juniper/contrail-generateDS/commit/95a47374423462d878f0d778767470163e5efedb
Submitter: Zuul (<email address hidden>)
Branch: master

commit 95a47374423462d878f0d778767470163e5efedb
Author: Tapan Karwa <email address hidden>
Date: Thu Dec 1 11:30:39 2016 -0800

Code to generate C++ code to handle input config data in json format.

Change-Id: I9787f1c318e964463b2499557094f0ce34778d6a
Partial-Bug: #1632470

Review in progress for https://review.opencontrail.org/27577
Submitter: Tapan Karwa (<email address hidden>)

Review in progress for https://review.opencontrail.org/27579
Submitter: Prakash Bailkeri (<email address hidden>)

Review in progress for https://review.opencontrail.org/27581
Submitter: Prakash Bailkeri (<email address hidden>)

Review in progress for https://review.opencontrail.org/27577
Submitter: Tapan Karwa (<email address hidden>)

Review in progress for https://review.opencontrail.org/27649
Submitter: Prakash Bailkeri (<email address hidden>)

Reviewed: https://review.opencontrail.org/27577
Committed: http://github.org/Juniper/contrail-controller/commit/3a80c41f9d6230c74ce2c1ba4ee82ffb491a2ff1
Submitter: Zuul (<email address hidden>)
Branch: master

commit 3a80c41f9d6230c74ce2c1ba4ee82ffb491a2ff1
Author: Tapan Karwa <email address hidden>
Date: Fri Dec 30 09:30:11 2016 -0800

Code for control-node retrieving config via cassandra.

Change-Id: Iff4aa698ee794bfe3c97beccaec416703357b45d
Partial-Bug: #1632470

Reviewed: https://review.opencontrail.org/27579
Committed: http://github.org/Juniper/contrail-third-party/commit/2f9757f1888e6769f8cd7d5eb9df620ff2966723
Submitter: Zuul (<email address hidden>)
Branch: master

commit 2f9757f1888e6769f8cd7d5eb9df620ff2966723
Author: Prakash M Bailkeri <email address hidden>
Date: Mon Jan 2 12:22:07 2017 +0000

Add rabbitmq-c and SimpleAmqpClient for c++

Add third party package detail for reading rabbitmq messages

Change-Id: I2162d82b4650a9d32870751c30fb6c0950a2e540
Partial-Bug: #1632470

Review in progress for https://review.opencontrail.org/27649
Submitter: Prakash Bailkeri (<email address hidden>)

Reviewed: https://review.opencontrail.org/27581
Committed: http://github.org/Juniper/contrail-generateDS/commit/6fc0546163555845062c08ef6264c9285a295944
Submitter: Zuul (<email address hidden>)
Branch: master

commit 6fc0546163555845062c08ef6264c9285a295944
Author: Prakash Bailkeri <email address hidden>
Date: Mon Jan 2 23:02:59 2017 +0530

generateDS changes for getting wrapped fields

Auto generate a map of field to wrapped field for map and list property

Change-Id: I85f67652a1335aa74d6554e344a54f2e742cea2f
Partial-Bug: #1632470

Review in progress for https://review.opencontrail.org/27649
Submitter: Prakash Bailkeri (<email address hidden>)

Reviewed: https://review.opencontrail.org/27649
Committed: http://github.org/Juniper/contrail-controller/commit/e3bd9a324db8ceea8fffbdcd86f1972efd9d832e
Submitter: Zuul (<email address hidden>)
Branch: master

commit e3bd9a324db8ceea8fffbdcd86f1972efd9d832e
Author: Prakash M Bailkeri <email address hidden>
Date: Wed Jan 4 18:19:38 2017 +0000

Config read from cassandra

Changes in this commit:
1. Json encoding for refs, list property, map property and parent
2. Bulk sync code. [Reading from fqname table, Starting multiple task to read in parallel etc]
3. Thirdparty packages for AMQP client and SConscript for same
4. Class for reading Rabbit messages

Other dependent review in progress
https://review.opencontrail.org/#/c/27581/
https://review.opencontrail.org/#/c/27579/

Change-Id: Idce47fd59fcdfd7d778451c6e1ff220d2b1897a1
Partial-Bug: #1632470

Review in progress for https://review.opencontrail.org/27770
Submitter: Ananth Suryanarayana (<email address hidden>)

Review in progress for https://review.opencontrail.org/27776
Submitter: Ananth Suryanarayana (<email address hidden>)

Reviewed: https://review.opencontrail.org/29712
Committed: http://github.org/Juniper/contrail-controller/commit/af9d2265afa87ec39c234b76fb20782e0a2b26ae
Submitter: Zuul (<email address hidden>)
Branch: master

commit af9d2265afa87ec39c234b76fb20782e0a2b26ae
Author: Ananth Suryanarayana <email address hidden>
Date: Sun Mar 19 20:19:42 2017 -0700

Remove an unncessary call to dirty cache

Also clear input request_list uuid list in
ConfigCassandraClient::ReadUuidTableRows (code review comments for another
review)

Change-Id: I12e0764712e78f10eeb93e427c2c4079b7f986f2
Partial-Bug: #1632470

Review in progress for https://review.opencontrail.org/29741
Submitter: Ananth Suryanarayana (<email address hidden>)

Review in progress for https://review.opencontrail.org/29617
Submitter: Ananth Suryanarayana (<email address hidden>)

Review in progress for https://review.opencontrail.org/29697
Submitter: Ananth Suryanarayana (<email address hidden>)

Reviewed: https://review.opencontrail.org/29617
Committed: http://github.org/Juniper/contrail-controller/commit/aa800ba6d7e50045ac3777962dee3c0c8fc40308
Submitter: Zuul (<email address hidden>)
Branch: master

commit aa800ba6d7e50045ac3777962dee3c0c8fc40308
Author: Ananth Suryanarayana <email address hidden>
Date: Wed Mar 15 12:47:09 2017 -0700

Add config_cassandra_client_test unit test to cover config_cassandra_client.cc

Add specific tests to cover fq-name and uuid-table from cassandra db.
Simulate many possible scenarios using parametertized gtests. code coverage
reports 94% for this file when all ifmap UTs are run including this new test

Fix request list corruption issue there by found while running this test.

In the current code, assumption is that the entry being deleted in
RemoveObjReqEntry() is always at the front of the uuid_read_list_. This is not
the case when (after bunching a few entries), we encounter delete/eor. In that
case, delete/eor could be any where in the middle/end of the list (as its
previous entries are still not processed). But by removing wrong head of the
list, we not only take wrong entry from the list we also end up deleted entry
to remain in the list. Code can crash later on, when this freed entry is
accessed.

This is fixed by storing list of iterators to the entries in the list, instead
of entries themselves. By this, we can still delete the entry in constant time
irrespective of where it is present in the list. (IOW, we still get pop()
equivalent semantics like how it is the current code)

Change-Id: I66f6adb07eaf185e7fdbf9209b54b382747e1621
Partial-Bug: #1632470

Review in progress for https://review.opencontrail.org/29697
Submitter: Ananth Suryanarayana (<email address hidden>)

Review in progress for https://review.opencontrail.org/29741
Submitter: Ananth Suryanarayana (<email address hidden>)

Review in progress for https://review.opencontrail.org/29697
Submitter: Ananth Suryanarayana (<email address hidden>)

Reviewed: https://review.opencontrail.org/29697
Committed: http://github.org/Juniper/contrail-controller/commit/8cd721b7df8db04c00ae57924de39226e421b043
Submitter: Zuul (<email address hidden>)
Branch: master

commit 8cd721b7df8db04c00ae57924de39226e421b043
Author: Ananth Suryanarayana <email address hidden>
Date: Fri Mar 17 11:50:32 2017 -0700

Add config_amqp_client_test unit test to test ConfigAmqClient class

Create ConfigAmqpChannel class to invoke AmqpClient::.*() and virtualize them
to override them in config_amqp_client_test. Cover coveraged improved to 87%
for config_amqp_client.cc with this test

Change-Id: Ie9003714d84cfb57d9fb2ad013c62f69fb7ea15c
Partial-Bug: #1632470

Reviewed: https://review.opencontrail.org/29741
Committed: http://github.org/Juniper/contrail-controller/commit/300d42efc3b10377800379b96aecb39ff073724e
Submitter: Zuul (<email address hidden>)
Branch: master

commit 300d42efc3b10377800379b96aecb39ff073724e
Author: Ananth Suryanarayana <email address hidden>
Date: Mon Mar 20 15:00:27 2017 -0700

Remove some of spurious includes

Partial-Bug: #1632470
Change-Id: I47fd227c37f0ef7ad70a0bdbe81b2ed392684193

Review in progress for https://review.opencontrail.org/29903
Submitter: Ananth Suryanarayana (<email address hidden>)

Review in progress for https://review.opencontrail.org/29929
Submitter: Ananth Suryanarayana (<email address hidden>)

Review in progress for https://review.opencontrail.org/29947
Submitter: Ananth Suryanarayana (<email address hidden>)

Review in progress for https://review.opencontrail.org/29929
Submitter: Ananth Suryanarayana (<email address hidden>)

Reviewed: https://review.opencontrail.org/29947
Committed: http://github.org/Juniper/contrail-controller/commit/172d76517805fbde0b1efaf40af90b49276b9f58
Submitter: Zuul (<email address hidden>)
Branch: master

commit 172d76517805fbde0b1efaf40af90b49276b9f58
Author: Ananth Suryanarayana <email address hidden>
Date: Wed Mar 29 12:17:11 2017 -0700

Initialize bool terminate_ in ConfigAmqpClient::ConfigAmqpClient()

This is used only in tests though, to terminate rabbit-mq msg receive loop

Change-Id: Ia7738b23afaf320f96b3d3f7f3035075ec0146a6
Partial-Bug: #1632470

Review in progress for https://review.opencontrail.org/29929
Submitter: Ananth Suryanarayana (<email address hidden>)

Reviewed: https://review.opencontrail.org/29929
Committed: http://github.org/Juniper/contrail-controller/commit/256d0a19a0947c47b78c78814baede8327b3c76e
Submitter: Zuul (<email address hidden>)
Branch: master

commit 256d0a19a0947c47b78c78814baede8327b3c76e
Author: Ananth Suryanarayana <email address hidden>
Date: Mon Mar 27 15:34:17 2017 -0700

Add config_cassandra_client_reader_test unit test

Enqueue various combinations of updates and deletes for a set of uuids.
Pause work-queues and then resume to make sure that uuids work items get
queued up before they get processed. Add duplicate uuid operations and ensure
that they get processed correctly

Change-Id: I2fd2c36012fd4b4cf97ea5c0f668612bafa0b04f
Partial-Bug: #1632470

Review in progress for https://review.opencontrail.org/30057
Submitter: Prakash Bailkeri (<email address hidden>)

Reviewed: https://review.opencontrail.org/30057
Committed: http://github.org/Juniper/contrail-controller/commit/0cd7be4765fa170265889b95886a33e6a15a16d1
Submitter: Zuul (<email address hidden>)
Branch: master

commit 0cd7be4765fa170265889b95886a33e6a15a16d1
Author: Prakash Bailkeri <email address hidden>
Date: Mon Apr 3 15:01:47 2017 +0530

Remove the UUID proces list

In the current code, UUID request-list or process-list is stored in two container.
1. List - Ensure the order of read/update
2. SET - To supress duplicate insertion of UUID to UUID process list.

Currently, since the UUID reading is done in multiple partition, the ordering of
read is not ensured. Code does handle the out of order reading of UUID in the
config reader task.

Remove the need for list(time ordered list).

Change-Id: Ia4734b3da90d58e7ab3f4c1bbe870f6f86b297da
Related-bug: #1632470

Review in progress for https://review.opencontrail.org/30206
Submitter: Prakash Bailkeri (<email address hidden>)

Reviewed: https://review.opencontrail.org/30206
Committed: http://github.org/Juniper/contrail-controller/commit/09e794df47482c0c3cdee13bf09dfde940d5c140
Submitter: Zuul (<email address hidden>)
Branch: master

commit 09e794df47482c0c3cdee13bf09dfde940d5c140
Author: Prakash Bailkeri <email address hidden>
Date: Thu Apr 6 15:55:38 2017 +0530

Introduce partition concept for config workers

1. Code re-org in ConfigCassandraClient class
2. Introduce Partition for ConfigCassandraClient worker. Move all datastructure
related to workers to Partition. e.g. obj cache, obj_req_set, task trigger and
work queue.
3. Remove iterator list RemoveObjReqEntries. Instead, lookup and erase from set.
4. Fix memory leak in config_cassandra_client_reader_test

Change-Id: Id4eb741956e53e88158043b61635fc9ca4ee7d7e
Related-bug: #1632470

Review in progress for https://review.opencontrail.org/30266
Submitter: Nischal Sheth (<email address hidden>)

Review in progress for https://review.opencontrail.org/30268
Submitter: Prakash Bailkeri (<email address hidden>)

Reviewed: https://review.opencontrail.org/30268
Committed: http://github.org/Juniper/contrail-controller/commit/a634ce74595c83fa07701de9a206767fe06cccf9
Submitter: Zuul (<email address hidden>)
Branch: master

commit a634ce74595c83fa07701de9a206767fe06cccf9
Author: Prakash Bailkeri <email address hidden>
Date: Sat Apr 8 10:32:46 2017 +0530

Fix compilation error in bgp_stress_test

1. Error got introduced due to code re-org of ConfigCassandraClient
2. Incorporate review comments

Change-Id: I254e834309d8e0c5f65b6f8e066c2b402be5cc19
Related-bug: #1632470

Review in progress for https://review.opencontrail.org/30319
Submitter: Prakash Bailkeri (<email address hidden>)

Reviewed: https://review.opencontrail.org/30319
Committed: http://github.org/Juniper/contrail-controller/commit/129ba5c178f7969c02a845de604b44ac4608563b
Submitter: Zuul (<email address hidden>)
Branch: master

commit 129ba5c178f7969c02a845de604b44ac4608563b
Author: Prakash Bailkeri <email address hidden>
Date: Wed Apr 5 10:29:12 2017 +0530

Introspect for config client

Introduce introspect pages for config cassandra client for following data
1. FQName Cache
2. Object Cache

Added test cases for introspect validation

TODO: Basic test for ReqIterate is added for both the request. Need to add test
for bulk data to validate batching.

Related-bug: #1632470

Change-Id: I4992047fc34af2dab4b825c9186d7969c65e933e

Review in progress for https://review.opencontrail.org/30618
Submitter: Prakash Bailkeri (<email address hidden>)

Reviewed: https://review.opencontrail.org/30618
Committed: http://github.com/Juniper/contrail-controller/commit/15abf924b5a54373721639673682543837b355cd
Submitter: Zuul (<email address hidden>)
Branch: master

commit 15abf924b5a54373721639673682543837b355cd
Author: Prakash Bailkeri <email address hidden>
Date: Thu Apr 20 11:48:45 2017 +0530

Config cassandra client: Fix compile error in optimization build

Compile Error while making production build

In file included from /usr/include/c++/4.8/algorithm:62:0,
from /usr/include/boost/smart_ptr/shared_ptr.hpp:42,
from /usr/include/boost/shared_ptr.hpp:17,
from build/include/sandesh/sandesh_types.h:13,
from controller/src/ifmap/ifmap_server_show.cc:5:
/usr/include/c++/4.8/bits/stl_algo.h: In function ‘void std::inplace_merge(_BIter, _BIter, _BIter, _Compare) [with _BIter = __gnu_cxx::__normal_iterator<ConfigDBUUIDCacheEntry*, std::vector<ConfigDBUUIDCacheEntry> >; _Compare = boost::_bi::bind_t<bool, bool (*)(const ConfigDBUUIDCacheEntry&, const ConfigDBUUIDCacheEntry&), boost::_bi::list2<boost::arg<1>, boost::arg<2> > >]’:
/usr/include/c++/4.8/bits/stl_algo.h:3242:5: internal compiler error: Max. number of generated reload insns per insn is achieved (90)

}
^
Please submit a full bug report,
with preprocessed source if appropriate.
See <file:///usr/share/doc/gcc-4.8/README.Bugs> for instructions.

Fix:
Remove the boost bind and pass function pointer to Sort method.

Change-Id: Ia513b3fc85ccea189880b82d78b5a155a29416ca
Related-bug: #1632470

Review in progress for https://review.opencontrail.org/31625
Submitter: Vinay Vithal Mahuli (<email address hidden>)

Reviewed: https://review.opencontrail.org/31625
Committed: http://github.com/Juniper/contrail-third-party-cache/commit/c1cedeaf68bdfd40394d88660bd1a60fec1c93b0
Submitter: Zuul (<email address hidden>)
Branch: master

commit c1cedeaf68bdfd40394d88660bd1a60fec1c93b0
Author: Ananth Suryanarayana <email address hidden>
Date: Mon Jan 23 16:22:18 2017 -0800

Add rapidjson v1.1.0 code

https://github.com/miloyip/rapidjson
commit f54b0e47a08782a6131cc3d60f94d038fa6e0a51
Merge: ab791ae 17254e0
Author: Milo Yip <email address hidden>
Date: Thu Aug 25 15:59:05 2016 +0800

Merge pull request #714 from miloyip/version1.1.0

Version 1.1.0

Change-Id: Icdffc6381bce207dea72aaa6bc2f4291c1b02849
Partial-Bug: #1632470

Review in progress for https://review.opencontrail.org/36605
Submitter: Ananth Suryanarayana (<email address hidden>)

Review in progress for https://review.opencontrail.org/36606
Submitter: Ananth Suryanarayana (<email address hidden>)

Review in progress for https://review.opencontrail.org/36607
Submitter: Ananth Suryanarayana (<email address hidden>)

Reviewed: https://review.opencontrail.org/36607
Committed: http://github.com/Juniper/contrail-controller/commit/7d92395488f804f8768c614ff0868185ed600dc4
Submitter: Zuul (<email address hidden>)
Branch: R4.1

commit 7d92395488f804f8768c614ff0868185ed600dc4
Author: Ananth Suryanarayana <email address hidden>
Date: Tue Oct 17 16:54:18 2017 -0700

Change default project to default-domain:admin

Change-Id: I70ffdf6e132bce883496e7fa1de854eec0158738
Partial-Bug: #1632470

Reviewed: https://review.opencontrail.org/36605
Committed: http://github.com/Juniper/contrail-controller/commit/0c2b6fd7589e200b5db8685d7524faa798be21af
Submitter: Zuul (<email address hidden>)
Branch: R4.0

commit 0c2b6fd7589e200b5db8685d7524faa798be21af
Author: Ananth Suryanarayana <email address hidden>
Date: Tue Oct 17 16:54:18 2017 -0700

Change default project to default-domain:admin

Change-Id: I70ffdf6e132bce883496e7fa1de854eec0158738
Partial-Bug: #1632470

Reviewed: https://review.opencontrail.org/36606
Committed: http://github.com/Juniper/contrail-controller/commit/d7264b5827b9a62ba52b1757cd653814b617a27e
Submitter: Zuul (<email address hidden>)
Branch: master

commit d7264b5827b9a62ba52b1757cd653814b617a27e
Author: Ananth Suryanarayana <email address hidden>
Date: Tue Oct 17 16:54:18 2017 -0700

Change default project to default-domain:admin

Change-Id: I70ffdf6e132bce883496e7fa1de854eec0158738
Partial-Bug: #1632470