FuelMasterMigrate failed Network verification not avaliable because nodes ["1", "2"] not avaliable via mcollective

(This check performed automatically)
Please, make sure that bug description contains the following sections filled in with the appropriate data related to the bug you are describing:

actual result

expected result

For more detailed information on the contents of each of the listed sections see https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Here_is_how_you_file_a_bug

According provided snapshot there is no issues
According tail of nailgun.test.domain.local/var/log/fuel-migrate.log everything finished OK, cloned fuel could see both nodes.
Also according node-1/var/log/atop/atop_current there was near 1 minute between finishing data-sync phase and snapshot creating. It lets us suspect that test was performed too early

Yo may use after first reboot following check:

until ssh -tt ${admin_net_ip} fuel node; do ;sleep 20 do

Fix proposed to branch: master
Review: https://review.openstack.org/313593

I've look at all "red" jobs logs.

There are 3 main causes of failure:

1 Qemu related problems (for example hung during coping) .
2 After migration nailgun begins answering (i. e. to work). But not all nodes have time to update their status before test starts
3 Busy indicator /tmp/notready is present only last couple minutes of whole migration time.

So it was added new flag /tmp/migration-done which appears exactly when migration process is finished and nailgun on cone starts working.

But after that, additionally in tests code is necessary to add checking of node status. Inside script itself it is impossible because cluster configuration and current state may be different. During testing configuration of cluster is known

It allows avoid 2 and 3 cases.

According 1 there is only common recommendation - update software on Jenkins slave host, increase memory and cpus on compute node

Fix proposed to branch: master
Review: https://review.openstack.org/317340

Reviewed: https://review.openstack.org/313593
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=ecf9c0d908ece88f21202e59c17109ec098321a7
Submitter: Jenkins
Branch: master

commit ecf9c0d908ece88f21202e59c17109ec098321a7
Author: Peter Zhurba <email address hidden>
Date: Fri May 6 17:40:17 2016 +0300

    Add new flag migrate process finishing.

    For qa test it is necessary to determine when migration process finished.
    For this purposes /not_ready was used. But it has some disadvantages, it
    covers not whole migration process. And there was possibility to mistake.
    So was added new flag which appears after finishing of migration. Old
    was left because of old tests. Both of them are used only for testing
    propouses and don't affect migration process.

    Partial-Bug: #1576276
    Change-Id: I49b62dc743111cb4c41489c528e48f8d56c2f9df

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/341244

Reviewed: https://review.openstack.org/341244
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=23766969e4df642dea41d371598d442f49efaf8c
Submitter: Jenkins
Branch: stable/mitaka

commit 23766969e4df642dea41d371598d442f49efaf8c
Author: Peter Zhurba <email address hidden>
Date: Fri May 6 17:40:17 2016 +0300

    Add new flag migrate process finishing.

    For qa test it is necessary to determine when migration process finished.
    For this purposes /not_ready was used. But it has some disadvantages, it
    covers not whole migration process. And there was possibility to mistake.
    So was added new flag which appears after finishing of migration. Old
    was left because of old tests. Both of them are used only for testing
    propouses and don't affect migration process.

    Partial-Bug: #1576276
    Change-Id: I49b62dc743111cb4c41489c528e48f8d56c2f9df
    (cherry picked from commit ecf9c0d908ece88f21202e59c17109ec098321a7)

Fix proposed to branch: stable/mitaka
Review: https://review.openstack.org/344647

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/344649

Reviewed: https://review.openstack.org/317340
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=573ded4401abd7e0f1090e31333d65ab160d4bc7
Submitter: Jenkins
Branch: master

commit 573ded4401abd7e0f1090e31333d65ab160d4bc7
Author: zatserklyany <email address hidden>
Date: Tue May 17 12:19:50 2016 +0300

    Refactor finding flag of finishing migrate process

    Change-Id: I17d37d7604aee4d5efae7fa4123621f0c2c630d4
    Partial-Bug: #1576276
    Partial-Bug: #1603488

Verified: https://custom-ci.infra.mirantis.net/job/10.0.custom.system_test/686

Reviewed: https://review.openstack.org/344649
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=2ef2befa5acbfa282a5d2523f6dd335034cde2a8
Submitter: Jenkins
Branch: stable/8.0

commit 2ef2befa5acbfa282a5d2523f6dd335034cde2a8
Author: zatserklyany <email address hidden>
Date: Tue May 17 12:19:50 2016 +0300

    Refactor finding flag of finishing migrate process

    Change-Id: I17d37d7604aee4d5efae7fa4123621f0c2c630d4
    Partial-Bug: #1576276
    Partial-Bug: #1603488

Reviewed: https://review.openstack.org/344647
Committed: https://git.openstack.org/cgit/openstack/fuel-qa/commit/?id=7a88c7476dced6292dd9013ef5e7cd031f19c446
Submitter: Jenkins
Branch: stable/mitaka

commit 7a88c7476dced6292dd9013ef5e7cd031f19c446
Author: zatserklyany <email address hidden>
Date: Tue May 17 12:19:50 2016 +0300

    Refactor finding flag of finishing migrate process

    Change-Id: I17d37d7604aee4d5efae7fa4123621f0c2c630d4
    Partial-Bug: #1576276
    Partial-Bug: #1603488