Verify network fails after backup-reinstall-restore Fuel

According the gate state, this error blocks all patches.
So change priority to Critical

Also there is a such traceback displayed after traceback mentioned in bug description for several jobs:

Exception in thread Thread-13 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_innerException in thread Thread-37 (most likely raised during interpreter shutdown):Exception in thread Thread-45 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/paramiko/transport.py", line 1777, in run
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'error'

 Exception in thread Thread-21 (most likely raised during interpreter shutdown):
  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/paramiko/transport.py", line 1777, in runException in thread Thread-29 (most likely raised during interpreter shutdown):

Traceback (most recent call last):Traceback (most recent call last):
Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner

  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner

  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner

  File "/home/jenkins/venv-nailgun-tests-2.9/local/lib/python2.7/site-packages/paramiko/transport.py", line 1777, in run+ ec=1

I don't think, that it's different bug, so I did not create one more bug for this issue.

Where is the fix? Why it is "in progress"? Could you please share the link to code review?

Timur, Looks like it was my mistake. I have compared description this bug with original traceback and wrongly decided, that it's the same error. I reanalyzed it and founded another bug https://bugs.launchpad.net/fuel/+bug/1537386.
I will recheck it again and probably will update doc with fail results.

Hm. We have another traceback for this bug https://bugs.launchpad.net/fuel/+bug/1537386.

For now i can't reproduce bug because i can't restore backup

https://bugs.launchpad.net/fuel/+bug/1538052

And second and dockrerctl restarts container instead stops.

Fix proposed to branch: master
Review: https://review.openstack.org/273037

Fix proposed to branch: stable/8.0
Review: https://review.openstack.org/274010

Reviewed: https://review.openstack.org/273037
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=4207659be2a20dbd0c3bd0f20dc50751694b0061
Submitter: Jenkins
Branch: master

commit 4207659be2a20dbd0c3bd0f20dc50751694b0061
Author: Peter Zhurba <email address hidden>
Date: Wed Jan 27 15:19:52 2016 +0200

    Changes in dockerct script:

      * Add posibility use systemctl for container start , stop instead using docker because systemd restarts container if it is stopped not by systemctl.
      * Fix free space checking

    Change-Id: I17fcb58ccd2384caebb277e904b3877e5ec461f1
    Closes-Bug: #1536314
    Closes-Bug: #1538052

Reviewed: https://review.openstack.org/274010
Committed: https://git.openstack.org/cgit/openstack/fuel-library/commit/?id=fd86b36c07f3088be57a89179c9792be4b525f81
Submitter: Jenkins
Branch: stable/8.0

commit fd86b36c07f3088be57a89179c9792be4b525f81
Author: Peter Zhurba <email address hidden>
Date: Wed Jan 27 15:19:52 2016 +0200

    Changes in dockerctl script:

      * Add possibility use systemctl for container start , stop instead using docker because systemd restarts container if it is stopped not by systemctl.
      * Fix free space checking

    Change-Id: I17fcb58ccd2384caebb277e904b3877e5ec461f1
    Closes-Bug: #1536314
    Closes-Bug: #1538052
    (cherry picked from commit 4207659be2a20dbd0c3bd0f20dc50751694b0061)

Verification is blocked by https://bugs.launchpad.net/fuel/+bug/1541539.

Verified on ISO #550
Network verification is successful after backup restore.

VERSION:
  feature_groups:
    - mirantis
  production: "docker"
  release: "8.0"
  api: "1.0"
  build_number: "550"
  build_id: "550"
  fuel-nailgun_sha: "ed2e0cde96ae7bc064e689f7409470e69c57772e"
  python-fuelclient_sha: "4f234669cfe88a9406f4e438b1e1f74f1ef484a5"
  fuel-agent_sha: "658be72c4b42d3e1436b86ac4567ab914bfb451b"
  fuel-nailgun-agent_sha: "b2bb466fd5bd92da614cdbd819d6999c510ebfb1"
  astute_sha: "b81577a5b7857c4be8748492bae1dec2fa89b446"
  fuel-library_sha: "33634ec27be77ecfb0b56b7e07497ad86d1fdcd3"
  fuel-ostf_sha: "3bc76a63a9e7d195ff34eadc29552f4235fa6c52"
  fuel-mirror_sha: "fb45b80d7bee5899d931f926e5c9512e2b442749"
  fuelmenu_sha: "78ffc73065a9674b707c081d128cb7eea611474f"
  shotgun_sha: "63645dea384a37dde5c01d4f8905566978e5d906"
  network-checker_sha: "a43cf96cd9532f10794dce736350bf5bed350e9d"
  fuel-upgrade_sha: "616a7490ec7199f69759e97e42f9b97dfc87e85b"
  fuelmain_sha: "87dfb6bc25d4650264f09c338ed77c21a3d6fe87"

Re-produced on 8.0 RC2/RTM (build #570).

Already deployed (ready) nodes don't answer via mcollective after master node restore, until they (agents) are restarted. Mcollective on slaves "hangs", here is the latest message in logs before restart:

D, [2016-02-24T16:46:04.975556 #7574] DEBUG -- : rabbitmq.rb:225:in `receive' Waiting for a message from RabbitMQ

Full log is attached.

Moved to MOS 8.0 updates, because MOS 8.0 release is comming.

Won't be verified on 9.0

fuel-9.0-mos-429-2016-06-02_00-19-00.iso

[root@nailgun ~]# dockerctl backup
Deprecation Warning:
*dockerctl* command is deprecated and will be removed in next major release.
We are not using docker containers any more.

[root@nailgun ~]# octane fuel-backup --to /tmp/fuel-backup.9.0.tar.gz

[root@fuel ~]# octane fuel-restore --from /tmp/fuel-backup.9.0.tar.gz --admin-password admin
2016-06-02 08:57:18 INFO octane.util.subprocess Started process docker[13032]: docker ps --all --filter status=running '--format="{{.Names}}"'
2016-06-02 08:57:19 INFO octane.util.subprocess Process docker finished with return value 0
2016-06-02 08:57:19 ERROR octane Required running containers: rsync, postgres, ostf, rabbitmq, nginx, keystone, mcollective, cobbler, rsyslog, astute, nailgun

[root@fuel ~]# docker ps --all
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

Actually _this_ issue has no connections with 9.0 backup-restore procedure - so for 9.0 we can "Invalid" this.