[ 4.0.3.0-119] Provisioning fails when openstack_manage_amqp is set to False in cluster.json

/opt/contrail/contrail_server_manager/provision_status.sh

+-------------+------------------+---------------+-------------------+
| id | status | ip_address | mac_address |
+-------------+------------------+---------------+-------------------+
| nodel11-vm1 | provision_failed | 10.204.217.31 | 52:53:58:01:00:01 |
| nodel11-vm2 | provision_failed | 10.204.217.32 | 52:53:58:01:00:02 |
| nodel11-vm3 | provision_failed | 10.204.217.33 | 52:53:58:01:00:03 |
| nodel11-vm4 | provision_failed | 10.204.217.34 | 52:53:58:01:00:04 |
| nodel11-vm5 | provision_failed | 10.204.217.36 | 52:53:58:01:00:05 |
| nodel11-vm6 | provision_failed | 10.204.217.37 | 52:53:58:01:00:06 |
+-------------+------------------+---------------+-------------------+

+-------------+-----------------+------------------------+-------------------------+---------------------------------------------------------------------------------------------------------------+
| id | cluster_id | role_being_provisioned | roles_pending_provision | provisioned_roles |
+-------------+-----------------+------------------------+-------------------------+---------------------------------------------------------------------------------------------------------------+
| nodel11-vm1 | nodel11_cluster | None | [] | [u'haproxy', u'pre_exec_vnc_galera', u'post_provision', u'post_exec_vnc_galera', u'keepalived', u'openstack'] |
| nodel11-vm2 | nodel11_cluster | None | [] | [u'haproxy', u'pre_exec_vnc_galera', u'post_provision', u'post_exec_vnc_galera', u'keepalived', u'openstack'] |
| nodel11-vm3 | nodel11_cluster | None | [] | [u'haproxy', u'pre_exec_vnc_galera', u'post_provision', u'post_exec_vnc_galera', u'keepalived', u'openstack'] |
+-------------+-----------------+------------------------+-------------------------+---------------------------------------------------------------------------------------------------------------+
root@nodei5:/var/log/contrail/sm_provisioning#

root@nodel11-vm2:~# docker ps -a

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c5e8066f5dc4 10.204.217.117:5100/regress_image-contrail-analytics:R4.1 "/bin/sh -c /entry..." 2 days ago Up 2 days analytics
4796829dc25a 10.204.217.117:5100/regress_image-contrail-analyticsdb:R4.1 "/bin/sh -c /entry..." 2 days ago Up 2 days analyticsdb
1f5102118a2b 10.204.217.117:5100/regress_image-contrail-controller:R4.1 "/bin/sh -c /entry..." 2 days ago Up About a minute controller
'.

<---Controller keeps restarting as rabbitmq is unable to come up in the controller..Rabbitmq was already started/up on host(openstack) so the port was not available to be used in the controller.

When the following steps were executed, the system recovered:

1) root@nodel11-vm2:~# kill -9 11903 beam <--stop beam and epmd on all the openstack nodes
root@nodel11-vm2:~# kill -9 11932 epmd
2) restart all the controllers

3) Start provisioning

root@nodel11-vm1:~# ps -aux | grep rabbitmq
rabbitmq 6303 0.0 0.0 7508 320 ? S 06:45 0:00 /usr/lib/erlang/erts-5.10.4/bin/epmd -daemon
rabbitmq 6443 4.0 0.1 2173812 45024 ? Sl 06:45 0:02 /usr/lib/erlang/erts-5.10.4/bin/beam.smp -W w -K true -A30 -P 1048576 -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.5.0/sbin/../ebin -noshell -noinput -s rabbit boot -sname rabbit@nodel11-vm1 -boot start_sasl -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/<email address hidden>"} -rabbit sasl_error_logger {file,"/<email address hidden>"} -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/lib/rabbitmq_server-3.5.0/sbin/../plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rabbit@nodel11-vm1-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@nodel11-vm1" -kernel inet_dist_listen_min 25672 -kernel inet_dist_listen_max 25672
rabbitmq 7265 0.0 0.0 7472 440 ? Ss 06:45 0:00 inet_gethost 4
rabbitmq 7267 0.0 0.0 9564 644 ? S 06:45 0:00 inet_gethost 4

root@nodel11-vm1:~# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ff1af24013de 10.204.217.117:5100/regress_image-contrail-analytics:R4.1 "/bin/sh -c /entry..." 2 days ago Up About a minute analytics
2175f7e6531f 10.204.217.117:5100/regress_image-contrail-analyticsdb:R4.1 "/bin/sh -c /entry..." 2 days ago Up About a minute analyticsdb
b045a4716a03 10.204.217.117:5100/regress_image-contrail-controller:R4.1 "/bin/sh -c /entry..." 2 days ago Restarting (1) 22 seconds ago controller
root@nodel11-vm1:~#

root@nodel11-vm1:~# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ff1af24013de 10.204.217.117:5100/regress_image-contrail-analytics:R4.1 "/bin/sh -c /entry..." 2 days ago Up About a minute analytics
2175f7e6531f 10.204.217.117:5100/regress_image-contrail-analyticsdb:R4.1 "/bin/sh -c /entry..." 2 days ago Up About a minute analyticsdb
b045a4716a03 10.204.217.117:5100/regress_image-contrail-controller:R4.1 "/bin/sh -c /entry..." 2 days ago Up 4 seconds controller

root@nodel11-vm1:~# ps -aux | grep beam
rabbitmq 6443 3.0 0.1 2174836 44960 ? Sl 06:45 0:02 /usr/lib/erlang/erts-5.10.4/bin/beam.smp -W w -K true -A30 -P 1048576 -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.5.0/sbin/../ebin -noshell -noinput -s rabbit boot -sname rabbit@nodel11-vm1 -boot start_sasl -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/<email address hidden>"} -rabbit sasl_error_logger {file,"/<email address hidden>"} -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/lib/rabbitmq_server-3.5.0/sbin/../plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rabbit@nodel11-vm1-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@nodel11-vm1" -kernel inet_dist_listen_min 25672 -kernel inet_dist_listen_max 25672
root 21542 0.0 0.0 10480 908 pts/7 S+ 06:47 0:00 grep --color=auto beam
root@nodel11-vm1:~# ps -aux | grep epmd
rabbitmq 6303 0.0 0.0 7508 320 ? S 06:45 0:00 /usr/lib/erlang/erts-5.10.4/bin/epmd -daemon
root 23143 0.0 0.0 10480 912 pts/7 S+ 06:47 0:00 grep --color=auto epmd

root@nodel11-vm1:~# kill -9 6443
root@nodel11-vm1:~# kill -9 6303
root@nodel11-vm1:~# docker restart controller
controller

root@nodel11-vm1:~# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ff1af24013de 10.204.217.117:5100/regress_image-contrail-analytics:R4.1 "/bin/sh -c /entry..." 2 days ago Up 3 minutes analytics
2175f7e6531f 10.204.217.117:5100/regress_image-contrail-analyticsdb:R4.1 "/bin/sh -c /entry..." 2 days ago Up 3 minutes analyticsdb
b045a4716a03 10.204.217.117:5100/regress_image-contrail-controller:R4.1 "/bin/sh -c /entry..." 2 days ago Up 41 seconds controller

root@nodel11-vm1:~# docker top b045a4716a03 | grep rabbit <--rabbitmq came up in the controller as expected

root 30394 30393 0 06:48 ? 00:00:00 /bin/sh /etc/init.d/rabbitmq-server start
root 30512 30394 0 06:48 ? 00:00:00 su rabbitmq -s /bin/sh -c /usr/lib/rabbitmq/bin/rabbitmqctl "wait" "/var/run/rabbitmq/pid"
message+ 30513 27741 4 06:48 ? 00:00:00 /usr/lib/erlang/erts-5.10.4/bin/beam.smp -W w -K true -A30 -P 1048576 -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.5.0/sbin/../ebin -noshell -noinput -s rabbit boot -sname rabbit@nodel11-vm1 -boot start_sasl -config /etc/rabbitmq/rabbitmq -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/<email address hidden>"} -rabbit sasl_error_logger {file,"/<email address hidden>"} -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/lib/rabbitmq_server-3.5.0/sbin/../plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rabbit@nodel11-vm1-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@nodel11-vm1" -kernel inet_dist_listen_min 25672 -kernel inet_dist_listen_max 25672
message+ 30523 30512 0 06:48 ? 00:00:00 sh -c /usr/lib/rabbitmq/bin/rabbitmqctl "wait" "/var/run/rabbitmq/pid"
message+ 30524 30523 1 06:48 ? 00:00:00 /usr/lib/erlang/erts-5.10.4/bin/beam.smp -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.5.0/sbin/../ebin -noshell -noinput -hidden -boot start_clean -sasl errlog_type error -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@nodel11-vm1" -s rabbit_control_main -nodename rabbit@nodel11-vm1 -extra wait /var/run/rabbitmq/pid
root@nodel11-vm1:~#

root@nodel11-vm2:~# docker top 1f5102118a2b | grep rabbitmq
message+ 26683 9571 2 06:47 ? 00:00:06 /usr/lib/erlang/erts-5.10.4/bin/beam.smp -W w -K true -A30 -P 1048576 -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.5.0/sbin/../ebin -noshell -noinput -s rabbit boot -sname rabbit@nodel11-vm2 -boot start_sasl -config /etc/rabbitmq/rabbitmq -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/<email address hidden>"} -rabbit sasl_error_logger {file,"/<email address hidden>"} -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/lib/rabbitmq_server-3.5.0/sbin/../plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rabbit@nodel11-vm2-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@nodel11-vm2" -kernel inet_dist_listen_min 25672 -kernel inet_dist_listen_max 25672

root@nodel11-vm3:~# docker top a826d0add47c | grep rabbit
message+ 19775 9374 2 06:55 ? 00:00:02 /usr/lib/erlang/erts-5.10.4/bin/beam.smp -W w -K true -A30 -P 1048576 -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.5.0/sbin/../ebin -noshell -noinput -s rabbit boot -sname rabbit@nodel11-vm3 -boot start_sasl -config /etc/rabbitmq/rabbitmq -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/<email address hidden>"} -rabbit sasl_error_logger {file,"/<email address hidden>"} -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/lib/rabbitmq_server-3.5.0/sbin/../plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rabbit@nodel11-vm3-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@nodel11-vm3" -kernel inet_dist_listen_min 25672 -kernel inet_dist_listen_max 25672
root@nodel11-vm3:~#

root@nodel11-vm2:~# docker top 1f5102118a2b | grep rabbitmq <--no rabbitmq in the controller

<-- As openstack_manage_amqp is set to False, rabbitmq should have been running in the controllers not on the openstack nodes..

root@nodel11-vm2:~# ps -aux | grep rabbitmq <---rabbitmq running on the host

rabbitmq 11903 0.0 0.0 9176 2248 ? S Nov24 0:03 /usr/lib/erlang/erts-5.10.4/bin/epmd -daemon
rabbitmq 11932 1.0 0.1 2189172 65528 ? Sl Nov24 42:02 /usr/lib/erlang/erts-5.10.4/bin/beam.smp -W w -K true -A30 -P 1048576 -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.5.0/sbin/../ebin -noshell -noinput -s rabbit boot -sname rabbit@nodel11-vm2 -boot start_sasl -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/<email address hidden>"} -rabbit sasl_error_logger {file,"/<email address hidden>"} -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/lib/rabbitmq_server-3.5.0/sbin/../plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rabbit@nodel11-vm2-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@nodel11-vm2" -kernel inet_dist_listen_min 25672 -kernel inet_dist_listen_max 25672
rabbitmq 12143 0.0 0.0 7472 436 ? Ss Nov24 0:11 inet_gethost 4
rabbitmq 12144 0.0 0.0 9564 640 ? S Nov24 0:20 inet_gethost 4
root 14032 0.0 0.0 10480 916 pts/9 S+ 05:47 0:00 grep --color=auto rabbitmq

root@nodel11-vm2:~# netstat -anp | grep 11932
tcp 0 0 0.0.0.0:25672 0.0.0.0:* LISTEN 11932/beam.smp
tcp 0 0 127.0.0.1:39364 127.0.0.1:4369 ESTABLISHED 11932/beam.smp
tcp6 0 0 :::5672 :::* LISTEN 11932/beam.smp
tcp6 13 0 10.204.217.32:5672 10.204.217.32:51382 CLOSE_WAIT 11932/beam.smp
tcp6 13 0 10.204.217.32:5672 10.204.217.33:59963 CLOSE_WAIT 11932/beam.smp
tcp6 13 0 10.204.217.32:5672 10.204.217.33:59953 CLOSE_WAIT 11932/beam.smp
tcp6 13 0 10.204.217.32:5672 10.204.217.32:51385 CLOSE_WAIT 11932/beam.smp
root@nodel11-vm2:~# docker exec -it controller bash

root@nodel11-vm2:~# docker top 1f5102118a2b | grep rabbitmq

root@nodel11-vm2:~# netstat -anp | grep 5672
tcp 0 0 0.0.0.0:25672 0.0.0.0:* LISTEN 11932/beam.smp
tcp 0 0 10.204.217.32:39966 10.204.217.33:5672 FIN_WAIT2 -
tcp 0 1 10.204.217.32:33269 10.204.217.31:5672 SYN_SENT 28023/python
tcp 0 1 10.204.217.32:33207 10.204.217.31:5672 SYN_SENT 27643/python
tcp 0 1 10.204.217.32:33299 10.204.217.31:5672 SYN_SENT 22881/python
tcp 0 1 10.204.217.32:33401 10.204.217.31:5672 SYN_SENT 27855/python
tcp 0 0 10.204.217.32:34955 10.204.217.32:5672 FIN_WAIT2 -
tcp6 0 0 :::5672 :::* LISTEN 11932/beam.smp
tcp6 13 0 10.204.217.32:5672 10.204.217.33:42699 CLOSE_WAIT 11932/beam.smp
tcp6 13 0 10.204.217.32:5672 10.204.217.32:34955 CLOSE_WAIT 11932/beam.smp
root@nodel11-vm2:~# ps -aux | grep epmd
rabbitmq 11903 0.0 0.0 9176 2248 ? S Nov24 0:03 /usr/lib/erlang/erts-5.10.4/bin/epmd -daemon
root 28954 0.0 0.0 10480 912 pts/9 S+ 05:51 0:00 grep --color=auto epmd
root@nodel11-vm2:~# netstat -anp | grep 11903
tcp6 0 0 :::4369 :::* LISTEN 11903/epmd
tcp6 0 0 127.0.0.1:4369 127.0.0.1:39364 ESTABLISHED 11903/epmd
unix 2 [ ] DGRAM 1874550 11903/epmd

root@nodel11-vm2:~# docker top 1f5102118a2b | grep epmd
root@nodel11-vm2:~# docker top 1f5102118a2b | grep beam

root@nodel11-vm2:~# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c5e8066f5dc4 10.204.217.117:5100/regress_image-contrail-analytics:R4.1 "/bin/sh -c /entry..." 2 days ago Up 2 days analytics
4796829dc25a 10.204.217.117:5100/regress_image-contrail-analyticsdb:R4.1 "/bin/sh -c /entry..." 2 days ago Up 2 days analyticsdb
1f5102118a2b 10.204.217.117:5100/regress_image-contrail-controller:R4.1 "/bin/sh -c /entry..." 2 days ago Up 2 minutes controller

root@nodel11-vm2:~# ps -aux | grep beam
rabbitmq 11932 1.0 0.1 2189172 65528 ? Sl Nov24 42:05 /usr/lib/erlang/erts-5.10.4/bin/beam.smp -W w -K true -A30 -P 1048576 -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.5.0/sbin/../ebin -noshell -noinput -s rabbit boot -sname rabbit@nodel11-vm2 -boot start_sasl -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/<email address hidden>"} -rabbit sasl_error_logger {file,"/<email address hidden>"} -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/lib/rabbitmq_server-3.5.0/sbin/../plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rabbit@nodel11-vm2-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@nodel11-vm2" -kernel inet_dist_listen_min 25672 -kernel inet_dist_listen_max 25672
root 32564 0.0 0.0 10480 908 pts/9 S+ 05:53 0:00 grep --color=auto beam
root@nodel11-vm2:~# ps -aux | grep epmd
rabbitmq 11903 0.0 0.0 9176 2248 ? S Nov24 0:03 /usr/lib/erlang/erts-5.10.4/bin/epmd -daemon
root 32697 0.0 0.0 10480 908 pts/9 S+ 05:53 0:00 grep --color=auto epmd
root@nodel11-vm2:~# kill -9 11903
root@nodel11-vm2:~# kill -9 11932

root@nodel11-vm2:~# docker top 1f5102118a2b | grep beam

After re provisioning using the cmd server-manager provision -F --cluster_id nodel11_cluster regress_image

Contrail-api on one of the controllers could not come up, but after restart the same, the process recovered.

root@nodel11-vm2(controller):/# contrail-status
== Contrail Control ==
contrail-control active
contrail-control-nodemgr active
contrail-dns active
contrail-named active

== Contrail Config ==
contrail-api:0 initializing (Generic Connection:Keystone[] connection down)
contrail-config-nodemgr active
contrail-device-manager backup
contrail-schema active
contrail-svc-monitor backup

== Contrail Config Database==
contrail-database: active

== Contrail Web UI ==
contrail-webui active
contrail-webui-middleware active

root@nodel11-vm2(controller):/# service contrail-api restart
contrail-api:0: stopped
contrail-api:0: started

root@nodel11-vm2(controller):/# contrail-status
== Contrail Control ==
contrail-control active
contrail-control-nodemgr active
contrail-dns active
contrail-named active

== Contrail Config ==
contrail-api:0 active
contrail-config-nodemgr active
contrail-device-manager backup
contrail-schema active
contrail-svc-monitor backup

== Contrail Config Database==
contrail-database: active

== Contrail Web UI ==
contrail-webui active
contrail-webui-middleware active

root@nodel11-vm2(controller):/#

root@nodel11-vm2(controller):/# contrail-status
== Contrail Control ==
contrail-control active
contrail-control-nodemgr active
contrail-dns active
contrail-named active

== Contrail Config ==
contrail-api:0 initializing (Generic Connection:Keystone[] connection down)
contrail-config-nodemgr active
contrail-device-manager backup
contrail-schema active
contrail-svc-monitor

Logs and json copied :

/home/bhushana/Documents/technical/bugs/bug-id