Custom nodegroup tests failing sporadically

Starting from 3:40 in logs on node 1 it's been constantly seeing logs where agent gets terminated

2016-12-27T03:40:19.605173+00:00 err: 2016-12-27 03:40:19.584 3701 ERROR neutron.agent.linux.utils [-] Exit code: 1; Stdin: hard_timeout=0,idle_timeout=0,priority=0,table=24,cookie=13505880170845670788,actions=drop; Stdout: ; Stderr: ovs-ofctl: br-int is not a bridge or
a socket
2016-12-27T03:40:19.605173+00:00 err: 2016-12-27 03:40:19.584 3701 ERROR neutron.agent.common.ovs_lib [-] Unable to execute ['ovs-ofctl', 'add-flows', 'br-int', '-']. Exception: Exit code: 1; Stdin: hard_timeout=0,idle_timeout=0,priority=0,table=24,cookie=135058801708456
70788,actions=drop; Stdout: ; Stderr: ovs-ofctl: br-int is not a bridge or a socket
2016-12-27T03:40:19.605173+00:00 debug: 2016-12-27 03:40:19.590 3701 DEBUG neutron.agent.linux.utils [req-3da6051a-4253-4cae-a0bf-39fe8db2c969 - - - - -] Running command (rootwrap daemon): ['ovs-vsctl', '--timeout=10', '--oneline', '--format=json', '--', 'list-br'] execu
te_rootwrap_daemon /usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:100
2016-12-27T03:40:19.605173+00:00 debug: 2016-12-27 03:40:19.603 3701 DEBUG neutron.agent.linux.utils [req-3da6051a-4253-4cae-a0bf-39fe8db2c969 - - - - -] Exit code: 0 execute /usr/lib/python2.7/dist-packages/neutron/agent/linux/utils.py:142
2016-12-27T03:40:19.605722+00:00 info: 2016-12-27 03:40:19.604 3701 INFO neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-3da6051a-4253-4cae-a0bf-39fe8db2c969 - - - - -] Mapping physical network physnet2 to bridge br-prv
2016-12-27T03:40:19.606037+00:00 err: 2016-12-27 03:40:19.604 3701 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent [req-3da6051a-4253-4cae-a0bf-39fe8db2c969 - - - - -] Bridge br-prv for physical network physnet2 does not exist. Agent terminated!
2016-12-27T03:40:19.607427+00:00 info: 2016-12-27 03:40:19.606 3701 INFO oslo_rootwrap.client [req-3da6051a-4253-4cae-a0bf-39fe8db2c969 - - - - -] Stopping rootwrap daemon process with pid=3709

Very similar to https://bugs.launchpad.net/fuel/+bug/1555162

On reverted snapshot on rebooted compute node I found that br-prv definition has beeing missed in
root@node-2:~# ovs-vsctl show
e1d97732-f42f-46c5-9604-a5004e144cfb
    Bridge br-int
        fail_mode: secure
        Port br-int
            Interface br-int
                type: internal
        Port int-br-prv
            Interface int-br-prv
                type: patch
                options: {peer=phy-br-prv}
    ovs_version: "2.6.1"

Suggest to reopen bug https://bugs.launchpad.net/fuel/+bug/1555162 for 9.2

I think that this bug needs to be splitted into 2 different ones. The first one (related to dpdk) might require investigation whether it is a duplicate of https://bugs.launchpad.net/mos/+bug/1652937. The latter should be investigated with regards to node group addition

So the root cause of the latter issue with the node groups is that some of the node groups do not have connectivity to the management VIP, while computes from the other node group have.

So, the root cause of the issue is that, for some reason, haproxy is not restarted with new set of 'other_networks' parameter which controls the networks that should be routed through the host out of haproxy namespace and thus the packets do not reach newly added compute nodes from the new node group.

At the same time, puppet reconfigures haproxy resource, but it is not restarted for some reason.

So the issue is that haproxy does not set ip routes on reload which is triggered by pacemaker

Fix proposed to branch: master
Review: https://review.openstack.org/416687

Reviewed: https://review.openstack.org/416687
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=a4ad4193684a133beb80aec977e0f910ce2fcab6
Submitter: Jenkins
Branch: master

commit a4ad4193684a133beb80aec977e0f910ce2fcab6
Author: Vladimir Kuklin <email address hidden>
Date: Wed Jan 4 20:47:26 2017 +0300

    Fix multirack routes installation for vrouter and haproxy

    1. Both scripts do not flush ip route table for non-local
       routes, which makes them non-idempotent
    2. Haproxy did not add routes on reload

    Change-Id: I498870b45ac47e6d6d8808d18964f3c2777c930c
    Closes-bug: #1652765

Fix proposed to branch: stable/newton
Review: https://review.openstack.org/417028

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/417029

Reviewed: https://review.openstack.org/417029
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=864c8e8696c20e2906803b707e5f6c5c67d7cf87
Submitter: Jenkins
Branch: stable/mitaka

commit 864c8e8696c20e2906803b707e5f6c5c67d7cf87
Author: Vladimir Kuklin <email address hidden>
Date: Wed Jan 4 20:47:26 2017 +0300

    Fix multirack routes installation for vrouter and haproxy

    1. Both scripts do not flush ip route table for non-local
       routes, which makes them non-idempotent
    2. Haproxy did not add routes on reload

    Change-Id: I498870b45ac47e6d6d8808d18964f3c2777c930c
    Closes-bug: #1652765
    (cherry picked from commit a4ad4193684a133beb80aec977e0f910ce2fcab6)

Reviewed: https://review.openstack.org/417028
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=73e7860944c48ce3eecbbfd8b1ebb4aae2c76ce6
Submitter: Jenkins
Branch: stable/newton

commit 73e7860944c48ce3eecbbfd8b1ebb4aae2c76ce6
Author: Vladimir Kuklin <email address hidden>
Date: Wed Jan 4 20:47:26 2017 +0300

    Fix multirack routes installation for vrouter and haproxy

    1. Both scripts do not flush ip route table for non-local
       routes, which makes them non-idempotent
    2. Haproxy did not add routes on reload

    Change-Id: I498870b45ac47e6d6d8808d18964f3c2777c930c
    Closes-bug: #1652765
    (cherry picked from commit a4ad4193684a133beb80aec977e0f910ce2fcab6)

The last 5 runs of [1] are green. Moved to Fix Released for 9.2

[1] https://product-ci.infra.mirantis.net/job/9.x.system_test.ubuntu.support_dpdk/

This issue was fixed in the openstack/fuel-library 11.0.0.0rc1 release candidate.