[library] HA deployment has failed on 3-rd controller with error: Timeout of deployment is exceeded (GRE Latency)

analysis of the puppet apply logs shows that a lot of time is spent on the tasks that are reading disks are extremely slow.
need logs from another env to confirm this bug.

Top lines:
01 - 94.18 sec (1.57 min) 4001 (/Stage[main]/Heat::Engine/Service[heat-engine]) Triggered 'refresh' from 36 events
02 - 81.11 sec (1.35 min) 120 (/Stage[netconfig]/L23network::L2/Package[kmod-openvswitch]/ensure) created
03 - 59.20 sec (0.99 min) 3744 (/Stage[main]/Galera/Service[mysql]/enable) enable changed 'true' to 'true'
04 - 56.11 sec (0.94 min) 4000 (/Stage[main]/Heat::Engine/Service[heat-engine]/enable) enable changed 'true' to 'true'
05 - 46.71 sec (0.78 min) 209 (/Firewall[105 nova ]/ensure) created
06 - 38.82 sec (0.65 min) 3591 (/Stage[main]/Rabbitmq::Server/Package[rabbitmq-server]/ensure) created
07 - 31.81 sec (0.53 min) 150 (L3_if_downup[eth2](provider=ruby)) Interface 'eth2' up.
08 - 30.03 sec (0.50 min) 189 (/Stage[corosync_setup]/Osnailyfacter::Cluster_ha::Virtual_ips/Cluster::Virtual_ips[management_old]/Cluster::Virtual_ip[management_old]/Cs_shadow[vip__management_old]/cib) defined 'cib' as 'vip__management_old'
09 - 28.58 sec (0.48 min) 3916 (/Stage[main]/Glance::Api/Glance_cache_config[DEFAULT/auth_url]/ensure) created
10 - 27.07 sec (0.45 min) 3738 (/Stage[main]/Galera/Package[MySQL-server]/ensure) created
Sum Total: 1161.86 sec (19.36 min)

I have one more environment from ISO 72.
Deployment has failed on 3-rd controller again.

Let's analyse where the issue is.

Again, according to the logs, the issue is with reporter's environment performance as even simpliest tasks involving, e.g. running 'depmod' command which simply reads a bunch of kernel modules and generates a file, wait for about several minutes, that clearly indicates performance problems. Thus we need logs from another env, not reporter's one. Thus I am marking this bug as incomplete.

Even we are facing the similar issues though we are using GRE implementation.

I found that communication between the controller-1 & 2 are very fast, whereas between 1 & 3 and 2 & 3 are very slow because of which ceph-deploy times-out and then issues with missing keystone tables. Communications from puppet master is instant for all the controllers. There isn't any slowness as such with the controller-3. The ping between the nodes too work fine, no packet loss.

See a timeout after 18 minutes:

2014-03-18 23:50:19,123 [ceph_deploy.cli][INFO ] Invoked (1.2.7): /usr/bin/ceph-deploy --overwrite-conf config pull node-1
2014-03-18 23:50:19,123 [ceph_deploy.config][DEBUG ] Checking node-1 for /etc/ceph/ceph.conf
2014-03-18 23:50:19,124 [ceph_deploy.sudo_pushy][DEBUG ] will use a remote connection without sudo
2014-03-19 00:08:09,975 [ceph_deploy.config][ERROR ] Unable to pull /etc/ceph/ceph.conf from node-1
2014-03-19 00:08:09,976 [ceph_deploy][ERROR ] GenericError: Failed to fetch config from 1 hosts

SSH from 1 <-> 3 & 2 <->3 takes >2 minutes whereas 1<->2 & fuel <-> controllers is instant.

See samples below, where node-1, node-2 & node-4 are controller-1, 2 & 3 respectively.

=====Fuel======
[root@fuel ~]# time ssh node-1 exit
real 0m0.103s
user 0m0.032s
sys 0m0.003s
[root@fuel ~]# time ssh node-2 exit
real 0m0.102s
user 0m0.030s
sys 0m0.004s
[root@fuel ~]# time ssh node-4 exit
real 0m0.103s
user 0m0.029s
sys 0m0.005s

=====Controller-1======
[root@node-1 ~]# time ssh fuel exit
root@fuel's password:
real 0m2.938s
user 0m0.004s
sys 0m0.007s
[root@node-1 ~]# time ssh node-2 exit
real 0m0.083s
user 0m0.013s
sys 0m0.002s
[root@node-1 ~]# time ssh node-4 exit
real 2m7.389s
user 0m0.010s
sys 0m0.005s

=====Controller-2======
[root@node-2 ~]# time ssh fuel exit
root@fuel's password:
real 0m2.997s
user 0m0.006s
sys 0m0.004s
[root@node-2 ~]# time ssh node-1 exit
real 0m0.084s
user 0m0.009s
sys 0m0.006s
[root@node-2 ~]# time ssh node-4 exit
real 2m7.060s
user 0m0.012s
sys 0m0.004s

=====Controller-3======
[root@node-4 ~]# time ssh fuel exit
root@fuel's password:
real 0m2.652s
user 0m0.007s
sys 0m0.003s
[root@node-4 ~]# time ssh node-1 exit
real 2m9.584s
user 0m0.012s
sys 0m0.003s
[root@node-4 ~]# time ssh node-2 exit
real 2m9.586s
user 0m0.013s
sys 0m0.002s

Arminder, could you please collect diagnostic snapshot, FUEL version and post it in the another bug as your case is not related to this bug.

It sounds like Neutron VLAN, if you have VLAN tagged networks, specifically management network, you need to test with one of the VLAN splinters mode when using CentOS.

Please reopen this if you see this again, and ideally get someone in #fuel-dev to debug your env. I'm closing this as Invalid for now.

I have got exactly the same thing in mu setup with fuel-5.0

Is there any fix / patches available for this - that I can apply on my system ?

Amrita, Please describe your env and if possible attach a support bundle

As far as this issue affects 4.1 initially and 5.0 later, Perhaps it should be nominated for all related series as well

Cannot reproduce that on Fuel 5.1. Deployment was OK. OSTF sanity tests also was OK.
Fuel version:

{"build_id": "2014-08-11_13-56-51",
"ostf_sha": "acf52a59e04fa74d2ed2b68ea225f4d24403b264",
"build_number": "423",
"auth_required": true,
"api": "1.0",
"nailgun_sha": "2741cdc0f0615263db2f176899d406207ec4ac04",
"production": "docker",
"fuelmain_sha": "9d4463400b4924159c978af43855e48bcf2a84b2",
"astute_sha": "b52910642d6de941444901b0f20e95ebbcb2b2e9",
"feature_groups": ["mirantis"],
"release": "5.1",
"fuellib_sha": "d9b93edb53c44900bd5bc2c25e7c8af0a1310645"}

No reproducer for 2 months (since June 6), back to Invalid.