Comment 39 for bug 1755649

Reviewed: https://review.opencontrail.org/43761
Committed: http://github.com/Juniper/contrail-sandesh/commit/3be6ccb47163040e34fce049dd6d7e21e4f9dea9
Submitter: Zuul (<email address hidden>)
Branch: R4.1

commit 3be6ccb47163040e34fce049dd6d7e21e4f9dea9
Author: zcui <email address hidden>
Date: Mon Jun 4 14:04:43 2018 -0700

contrail-collector crash immediately after provisioning

root cause:
Race condition problem:
To state_machine_,
(1) alloced by sandesh_connection.
(2) used by generator
When problem happen, generator receive Resource update message,
and enqueue resouece update to state_machine_, at same time,
update stats immedietly. This action will try to get mutex
sometime, it will lead CPU yield. We call this as thread 1.
At same time, connection close is triggered, and destructor
function will be triggered. And destructure will call termial
and all memory will be released related to this connection.
We call this as thread 2.
When thread 2 finished and thread 1 go ahead, crash will happen.

Solution:
Designer of state_machine should consider this problem. So state
Machine destructure is separated two steps:
(1) call terminal to free memory alloced by its substruct.
(2) start a timer to free state machine self.
Between step1 and step2, deleted_ is used to check state machine
can be used or not.
We add a shutdown fucntion for stats structure to pass this state.

Closes-Bug: 1755649

Change-Id: I599461f0a37adc21d2b68a5ca20d66ccaf4f6e51