SYMC: Multiple HAProxy processes getting spawned for single LBaaS

Also, no errors whatsoever in the vrouter-agent log files

Assigning to Vivek Garg to re-create internally.

Observed the haproxy config file for the lbaas with many instances didn't seem correct. Attaching the config observed:

Review in progress for https://review.opencontrail.org/14410
Submitter: Divakar Dharanalakota (<email address hidden>)

Review in progress for https://review.opencontrail.org/14440
Submitter: Divakar Dharanalakota (<email address hidden>)

Review in progress for https://review.opencontrail.org/14442
Submitter: Divakar Dharanalakota (<email address hidden>)

Review in progress for https://review.opencontrail.org/14444
Submitter: Divakar Dharanalakota (<email address hidden>)

Review in progress for https://review.opencontrail.org/14442
Submitter: Divakar Dharanalakota (<email address hidden>)

Reviewed: https://review.opencontrail.org/14442
Committed: http://github.org/Juniper/contrail-controller/commit/9bbe8d2ec3add95af5602fc8a8ed77c2dcd68be6
Submitter: Zuul
Branch: R2.20

commit 9bbe8d2ec3add95af5602fc8a8ed77c2dcd68be6
Author: Divakar <email address hidden>
Date: Thu Oct 15 16:11:50 2015 +0530

Taking the haproxy process id from config.pid file

The haproxy config is updated by providing the new configuration file
and old haproxy process id. This starts a new haproxy process letting
the old haproxy process to handle the old sessions. New haproxy handles
the new sessions. If a new configuration update comes while both old and
new haproxy processes are runnning, another haproxy process needs to be started
by providing the latest running haproxy process id.

In the existing update code, the haproxy process id is found by "ps"
output. This is resulating in not getting the latest haproxy process id,
but an older process id. This is leading to second haproxy process never
getting killed leaving two haproxies running for ever.

The latest haproxy process that is handling the new session is always
updated in a file by haproxy itself. As a fix, this process id used to
start a new haproxy rather taking the pid from "ps". When the
loadbalancer VIP is deleted, Agent deletes the directory where haproxy
config file, stats file and pid file are stored. This can result in
vrouter_netns.py script failing as pid is not found to stop the haproxy.
To overcome this, the config file, pid file and sock file are stored not
in an explicit directory corresponding to the pool. Only config fike and
sock file are deleted by Agent and pid file is deleted by vrouter_netns
script.

Change-Id: Id7e1a74a4f076f02c052f860ab5b183adf950640
closes-bug: #1495702

Reviewed: https://review.opencontrail.org/14444
Committed: http://github.org/Juniper/contrail-controller/commit/6083a6b34aba1eecd9bea3bc4712553da3c9dc4d
Submitter: Zuul
Branch: R2.0

commit 6083a6b34aba1eecd9bea3bc4712553da3c9dc4d
Author: Divakar <email address hidden>
Date: Thu Oct 15 16:38:59 2015 +0530

Taking the haproxy process id from config.pid file

The haproxy config is updated by providing the new configuration file
and old haproxy process id. This starts a new haproxy process letting
the old haproxy process to handle the old sessions. New haproxy handles
the new sessions. If a new configuration update comes while both old and
new haproxy processes are runnning, another haproxy process needs to be started
by providing the latest running haproxy process id.

In the existing update code, the haproxy process id is found by "ps"
output. This is resulating in not getting the latest haproxy process id,
but an older process id. This is leading to second haproxy process never
getting killed leaving two haproxies running for ever.

The latest haproxy process that is handling the new session is always
updated in a file by haproxy itself. As a fix, this process id used to
start a new haproxy rather taking the pid from "ps". When the
loadbalancer VIP is deleted, Agent deletes the directory where haproxy
config file, stats file and pid file are stored. This can result in
vrouter_netns.py script failing as pid is not found to stop the haproxy.
To overcome this, the config file, pid file and sock file are stored not
in an explicit directory corresponding to the pool. Only config fike and
sock file are deleted by Agent and pid file is deleted by vrouter_netns
script.

Change-Id: I2f9bda829f90859ae55991e7e4313bc2119c44a1
closes-bug: #1495702

Review in progress for https://review.opencontrail.org/14877
Submitter: Rudra Rugge (<email address hidden>)

Reviewed: https://review.opencontrail.org/14877
Committed: http://github.org/Juniper/contrail-controller/commit/2bfbb468a25a5779e2eec15d935ace0990e00d87
Submitter: Zuul
Branch: R2.21.x

commit 2bfbb468a25a5779e2eec15d935ace0990e00d87
Author: Divakar <email address hidden>
Date: Thu Oct 15 16:11:50 2015 +0530

Taking the haproxy process id from config.pid file

The haproxy config is updated by providing the new configuration file
and old haproxy process id. This starts a new haproxy process letting
the old haproxy process to handle the old sessions. New haproxy handles
the new sessions. If a new configuration update comes while both old and
new haproxy processes are runnning, another haproxy process needs to be started
by providing the latest running haproxy process id.

In the existing update code, the haproxy process id is found by "ps"
output. This is resulating in not getting the latest haproxy process id,
but an older process id. This is leading to second haproxy process never
getting killed leaving two haproxies running for ever.

The latest haproxy process that is handling the new session is always
updated in a file by haproxy itself. As a fix, this process id used to
start a new haproxy rather taking the pid from "ps". When the
loadbalancer VIP is deleted, Agent deletes the directory where haproxy
config file, stats file and pid file are stored. This can result in
vrouter_netns.py script failing as pid is not found to stop the haproxy.
To overcome this, the config file, pid file and sock file are stored not
in an explicit directory corresponding to the pool. Only config fike and
sock file are deleted by Agent and pid file is deleted by vrouter_netns
script.

Change-Id: Id7e1a74a4f076f02c052f860ab5b183adf950640
closes-bug: #1495702

Review in progress for https://review.opencontrail.org/17834
Submitter: Divakar Dharanalakota (<email address hidden>)

Reviewed: https://review.opencontrail.org/17834
Committed: http://github.org/Juniper/contrail-controller/commit/8a30644055aeec528a6223223adc5a1d9f76b925
Submitter: Zuul
Branch: R3.0

commit 8a30644055aeec528a6223223adc5a1d9f76b925
Author: Divakar <email address hidden>
Date: Thu Oct 15 16:11:50 2015 +0530

Taking the haproxy process id from config.pid file

The haproxy config is updated by providing the new configuration file
and old haproxy process id. This starts a new haproxy process letting
the old haproxy process to handle the old sessions. New haproxy handles
the new sessions. If a new configuration update comes while both old and
new haproxy processes are runnning, another haproxy process needs to be started
by providing the latest running haproxy process id.

In the existing update code, the haproxy process id is found by "ps"
output. This is resulating in not getting the latest haproxy process id,
but an older process id. This is leading to second haproxy process never
getting killed leaving two haproxies running for ever.

The latest haproxy process that is handling the new session is always
updated in a file by haproxy itself. As a fix, this process id used to
start a new haproxy rather taking the pid from "ps". When the
loadbalancer VIP is deleted, Agent deletes the directory where haproxy
config file, stats file and pid file are stored. This can result in
vrouter_netns.py script failing as pid is not found to stop the haproxy.
To overcome this, the config file, pid file and sock file are stored not
in an explicit directory corresponding to the pool. Only config fike and
sock file are deleted by Agent and pid file is deleted by vrouter_netns
script.

closes-bug: #1495702

Conflicts:
 src/nodemgr/vrouter_nodemgr/haproxy_stats.py
 src/vnsw/agent/oper/instance_manager.cc
 src/vnsw/agent/oper/netns_instance_adapter.cc
 src/vnsw/agent/oper/test/instance_manager_test.cc
 src/vnsw/opencontrail-vrouter-netns/opencontrail_vrouter_netns/vrouter_netns.py

Change-Id: If5fd9be0c5229aa68f4b104d3a566f6ac612814e

Review in progress for https://review.opencontrail.org/18000
Submitter: Divakar Dharanalakota (<email address hidden>)

Reviewed: https://review.opencontrail.org/18000
Committed: http://github.org/Juniper/contrail-controller/commit/04f36c5282f0b4efc9a798c922020151dda26a7a
Submitter: Zuul
Branch: master

commit 04f36c5282f0b4efc9a798c922020151dda26a7a
Author: Divakar <email address hidden>
Date: Thu Oct 15 16:11:50 2015 +0530

Taking the haproxy process id from config.pid file

The haproxy config is updated by providing the new configuration file
and old haproxy process id. This starts a new haproxy process letting
the old haproxy process to handle the old sessions. New haproxy handles
the new sessions. If a new configuration update comes while both old and
new haproxy processes are runnning, another haproxy process needs to be started
by providing the latest running haproxy process id.

In the existing update code, the haproxy process id is found by "ps"
output. This is resulating in not getting the latest haproxy process id,
but an older process id. This is leading to second haproxy process never
getting killed leaving two haproxies running for ever.

The latest haproxy process that is handling the new session is always
updated in a file by haproxy itself. As a fix, this process id used to
start a new haproxy rather taking the pid from "ps". When the
loadbalancer VIP is deleted, Agent deletes the directory where haproxy
config file, stats file and pid file are stored. This can result in
vrouter_netns.py script failing as pid is not found to stop the haproxy.
To overcome this, the config file, pid file and sock file are stored not
in an explicit directory corresponding to the pool. Only config fike and
sock file are deleted by Agent and pid file is deleted by vrouter_netns
script.

Change-Id: Iebc6230e0eef73c4e18b18e4fc0ca65b8af6b4e4
closes-bug: #1495702