[4.0.0.0-3038] Provisioning fails as 'fab setup_all' fails during prov_config()

Is this reproducible? Could you please confirm it wasn't a network issue? I see this in the logs:

ssh: connect to host 10.204.216.232 port 22: Connection timed out

Yes this is reproducible (I am seeing it each time while provisioning) and there wasn't any network issue. Seen on 3039 too -

2017-02-14 02:00:58:661673: [root@10.204.216.232] out: [localhost] local: chkconfig nova-compute on
2017-02-14 02:00:58:661934: [root@10.204.216.232] out: [localhost] local: service nova-compute restart
2017-02-14 02:00:58:662075: [root@10.204.216.232] out: nova-compute stop/waiting
2017-02-14 02:00:58:694211: [root@10.204.216.232] out: nova-compute start/running, process 27934
2017-02-14 02:00:58:726012: [root@10.204.216.232] out: [localhost] local: chkconfig supervisor-vrouter on
2017-02-14 02:00:58:726164: [root@10.204.216.232] out:
2017-02-14 02:00:58:757877:
2017-02-14 02:00:58:758489: No env.dpdk section in testbed file, skipping the configuration...
2017-02-14 02:00:58:758699: [root@10.204.216.232] Executing task 'prov_config'
2017-02-14 02:00:58:759079: [root@10.204.216.232] sudo: hostname
2017-02-14 02:00:58:759447: [root@10.204.216.232] out: nodek12
2017-02-14 02:00:58:809587: [root@10.204.216.232] out:
2017-02-14 02:00:58:809964: Write failed: Broken pipe
+ rv=255
+ return 255
+ debug_and_die 'fab setup_interface or setup_all task failed'
+ local 'message=fab setup_interface or setup_all task failed'

Hi Nipa,

As I said earlier in the afternoon, vrouter kernel module is getting loaded in the kernel even before the reboot.
Which causes the vhost0 to be created with the same ip as the original physical interface.(vhost0 and <physicalInt> is holding the same ip).
Which results in two routes to same network one via vhost0 and other via <physicalInt>. So the compute nodes become unreachable.

We will need agent/vrouter teams help in debugging this and identify what causes the kernel module to be loaded.

Pavana/Nipa,

If you have the setup in failed state please share it with agent/vrouter team.

Thanks,
Ignatious

Review in progress for https://review.opencontrail.org/29046
Submitter: Atul Moghe (<email address hidden>)

Reviewed: https://review.opencontrail.org/29046
Committed: http://github.org/Juniper/contrail-provisioning/commit/1a1deea8aa2a60dd5f0cce204bc894fbb16d1457
Submitter: Zuul (<email address hidden>)
Branch: master

commit 1a1deea8aa2a60dd5f0cce204bc894fbb16d1457
Author: Atul Moghe <email address hidden>
Date: Wed Feb 22 23:04:21 2017 +0000

removed vrouter restart from compute-server-setup,

this was causing to loose connectivity even before setup is completed
Closes-Bug: #1663464

Change-Id: Id2bc15205e5b37b7f609a13a1b1964680ab80777