VMs are losing connectivity and their IPs after HA failover

Should be related https://bugs.launchpad.net/fuel/+bug/1370510

Should this be included into Release notes?
It there any workaround?

If the behavior is confirmed, then this bug is Critical.

Do we really have this behaviour confirmed?

Any update on confirming if this is still present after fix for bug #1370510 was merged (9/22 for 6.0, 10/28 for 5.1.1)?

Our QA guys in Mellanox tested Fuel 5.1 GA with a few failovers and it passed ok.
We still haven't tested this on v6.0 / v5.1.1, we'll update as soon as we get to that.

Thanks Gil! Do you have any ETA when you could test at least 5.1.1 ?

First as I said before it worked on Fuel 5.1 GA, I verified again with QA this is correct.
I can't really commit on an ETA, I'll try pushing QA to prioritize this to be done in the next week..

I can't reproduce this behavior on 6.0.

Update: re v5.1.1 we couldn't reproduce it, re v6.0 we haven't got to it yet.

Thanks Gil, marking as Incomplete in 5.1.x.

Can't reproduce on 5.1.1 and 6.

This was reproduced again here with 6.0 RC3 (build 56), build info is below.
* I can't attach snapshot do to problems in launchpad site, I'll try again later *

The bug isn't always reproduces, but happened here more than once with this build.
I noticed this error on pacemaker log:
<29>Dec 24 15:32:29 node-4 lrmd[26113]: notice: operation_finished: p_neutron-dhcp-agent_start_0:38178:stderr [ cat: /var/run/resource-agents/neutron-agent-dhcp/neutron-agent-dhcp.pid: No such file or directory ]

I saw once there was no dnsmasq on any of the controllers (not even the master), the second time it happened (this snapshot) the IP addresses on instances got back (and the dnsmasq service on the master controller too) after more than 1 hour.

{"build_id": "2014-12-18_01-32-01", "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4", "build_number": "56", "auth_required": true, "api": "1.0", "nailgun_sha": "5f91157daa6798ff522ca9f6d34e7e135f150a90", "production": "docker", "fuelmain_sha": "45caacadb878abfbd9d60e134d72229698b469c9", "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91", "feature_groups": ["mirantis"], "release": "6.0", "release_versions": {"2014.2-6.0": {"VERSION": {"build_id": "2014-12-18_01-32-01", "ostf_sha": "a9afb68710d809570460c29d6c3293219d3624d4", "build_number": "56", "api": "1.0", "nailgun_sha": "5f91157daa6798ff522ca9f6d34e7e135f150a90", "production": "docker", "fuelmain_sha": "45caacadb878abfbd9d60e134d72229698b469c9", "astute_sha": "16b252d93be6aaa73030b8100cf8c5ca6a970a91", "feature_groups": ["mirantis"], "release": "6.0", "fuellib_sha": "73332192a257ea02c40a39885c502ad1ebdf3eda"}}}, "fuellib_sha": "73332192a257ea02c40a39885c502ad1ebdf3eda"}

Fix proposed to branch: master
Review: https://review.openstack.org/143996

Fix proposed to branch: stable/6.0
Review: https://review.openstack.org/143998

The bug is reproduced on Mellanox enabled environments with some VM images (it seems CirrOS is not affected but Ubuntu is).
DHCP lease time in Fuel 6.0 is 60 seconds.
During a Controller failover Dnsmasq can start up to 10 minutes. That's why VMs loose IP addresses and connectivity. As a workaround, this patch can be applied https://review.openstack.org/#/c/143996/

Taking into account all above I decrease the Bug importance to High.

https://review.openstack.org/#/c/144094/1 - Known Issue is described here.
Gil and Andrey, please review it and put +1 once everything looks good.

Reviewed: https://review.openstack.org/143996
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=238a26c07394bf141acdd1db8de28cb6651c60d5
Submitter: Jenkins
Branch: master

commit 238a26c07394bf141acdd1db8de28cb6651c60d5
Author: Sergii Golovatiuk <email address hidden>
Date: Thu Dec 25 16:41:59 2014 +0100

    Increase lease time for dhcp agent

    Usually failover takes a couple of minutes. 120 seconds is not enough so
    during failover VMs loose IP address, breaking the connectivity.
    Increasing the value to 600 allows

    * Minimaze the number of DHCPDISCOVER requests during failover
    * Minimaze the load on dnsmasq as it will be less frequently renewing IP
      addresses for VMs

    Change-Id: I7dc57e3f79ed8ee5e83c26e0c80d6a44e0840b4a
    Closes-Bug: 1371104

Reviewed: https://review.openstack.org/144094
Committed: https://git.openstack.org/cgit/stackforge/fuel-docs/commit/?id=e4516d379983af2a933a6ca1030e8138c498a416
Submitter: Jenkins
Branch: master

commit e4516d379983af2a933a6ca1030e8138c498a416
Author: Irina Povolotskaya <email address hidden>
Date: Fri Dec 26 10:56:13 2014 +0300

    Release Notes 6.0 -- a short DHCP timeout issue is discovered

    Related bug: #1371104
    Change-Id: I353c40230e6364b952f89464264655a3d1a97efe

Reviewed: https://review.openstack.org/143998
Committed: https://git.openstack.org/cgit/stackforge/fuel-library/commit/?id=20f0b5a5201f65ee7fc7a1e5f081c46ca59b4726
Submitter: Jenkins
Branch: stable/6.0

commit 20f0b5a5201f65ee7fc7a1e5f081c46ca59b4726
Author: Sergii Golovatiuk <email address hidden>
Date: Thu Dec 25 16:41:59 2014 +0100

    Increase lease time for dhcp agent

    Usually failover takes a couple of minutes. 120 seconds is not enough so
    during failover VMs loose IP address, breaking the connectivity.
    Increasing the value to 600 allows

    * Minimaze the number of DHCPDISCOVER requests during failover
    * Minimaze the load on dnsmasq as it will be less frequently renewing IP
      addresses for VMs

    Change-Id: I7dc57e3f79ed8ee5e83c26e0c80d6a44e0840b4a
    Closes-Bug: 1371104
    Signed-off-by: Sergii Golovatiuk <email address hidden>