Juniper Openstack

  1. Bug #1734608
  2. Comment #0

Comment 0 for bug 1734608

Revision history for this message
Ankit Jain (ankitja) wrote :

/opt/contrail/contrail_server_manager/provision_status.sh

+-------------+------------------+---------------+-------------------+
| id | status | ip_address | mac_address |
 +-------------+------------------+---------------+-------------------+
| nodel11-vm1 | provision_failed | 10.204.217.31 | 52:53:58:01:00:01 |
| nodel11-vm2 | provision_failed | 10.204.217.32 | 52:53:58:01:00:02 |
| nodel11-vm3 | provision_failed | 10.204.217.33 | 52:53:58:01:00:03 |
| nodel11-vm4 | provision_failed | 10.204.217.34 | 52:53:58:01:00:04 |
| nodel11-vm5 | provision_failed | 10.204.217.36 | 52:53:58:01:00:05 |
| nodel11-vm6 | provision_failed | 10.204.217.37 | 52:53:58:01:00:06 |
 +-------------+------------------+---------------+-------------------+

+-------------+-----------------+------------------------+-------------------------+---------------------------------------------------------------------------------------------------------------+
| id | cluster_id | role_being_provisioned | roles_pending_provision | provisioned_roles |
 +-------------+-----------------+------------------------+-------------------------+---------------------------------------------------------------------------------------------------------------+
| nodel11-vm1 | nodel11_cluster | None | [] | [u'haproxy', u'pre_exec_vnc_galera', u'post_provision', u'post_exec_vnc_galera', u'keepalived', u'openstack'] |
| nodel11-vm2 | nodel11_cluster | None | [] | [u'haproxy', u'pre_exec_vnc_galera', u'post_provision', u'post_exec_vnc_galera', u'keepalived', u'openstack'] |
| nodel11-vm3 | nodel11_cluster | None | [] | [u'haproxy', u'pre_exec_vnc_galera', u'post_provision', u'post_exec_vnc_galera', u'keepalived', u'openstack'] |
 +-------------+-----------------+------------------------+-------------------------+---------------------------------------------------------------------------------------------------------------+
root@nodei5:/var/log/contrail/sm_provisioning#

root@nodel11-vm2:~# docker ps -a

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c5e8066f5dc4 10.204.217.117:5100/regress_image-contrail-analytics:R4.1 "/bin/sh -c /entry..." 2 days ago Up 2 days analytics
4796829dc25a 10.204.217.117:5100/regress_image-contrail-analyticsdb:R4.1 "/bin/sh -c /entry..." 2 days ago Up 2 days analyticsdb
1f5102118a2b 10.204.217.117:5100/regress_image-contrail-controller:R4.1 "/bin/sh -c /entry..." 2 days ago Up About a minute controller
'.

<---Controller keeps restarting as rabbitmq is unable to come up in the controller..Rabbitmq was already started/up on host(openstack) so the port was not available to be used in the controller.

When the following steps were executed, the system recovered:

1) root@nodel11-vm2:~# kill -9 11903 beam <--stop beam and epmd on all the openstack nodes
root@nodel11-vm2:~# kill -9 11932 epmd
2) restart all the controllers

3) Start provisioning

root@nodel11-vm2:~# docker top 1f5102118a2b | grep rabbitmq <--no rabbitmq in the controller

<-- As openstack_manage_amqp is set to False, rabbitmq should have been running in the controllers not on the openstack nodes..

root@nodel11-vm2:~# ps -aux | grep rabbitmq <---rabbitmq running on the host

rabbitmq 11903 0.0 0.0 9176 2248 ? S Nov24 0:03 /usr/lib/erlang/erts-5.10.4/bin/epmd -daemon
rabbitmq 11932 1.0 0.1 2189172 65528 ? Sl Nov24 42:02 /usr/lib/erlang/erts-5.10.4/bin/beam.smp -W w -K true -A30 -P 1048576 -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.5.0/sbin/../ebin -noshell -noinput -s rabbit boot -sname rabbit@nodel11-vm2 -boot start_sasl -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/<email address hidden>"} -rabbit sasl_error_logger {file,"/<email address hidden>"} -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/lib/rabbitmq_server-3.5.0/sbin/../plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rabbit@nodel11-vm2-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@nodel11-vm2" -kernel inet_dist_listen_min 25672 -kernel inet_dist_listen_max 25672
rabbitmq 12143 0.0 0.0 7472 436 ? Ss Nov24 0:11 inet_gethost 4
rabbitmq 12144 0.0 0.0 9564 640 ? S Nov24 0:20 inet_gethost 4
root 14032 0.0 0.0 10480 916 pts/9 S+ 05:47 0:00 grep --color=auto rabbitmq

root@nodel11-vm2:~# netstat -anp | grep 11932
tcp 0 0 0.0.0.0:25672 0.0.0.0:* LISTEN 11932/beam.smp
tcp 0 0 127.0.0.1:39364 127.0.0.1:4369 ESTABLISHED 11932/beam.smp
tcp6 0 0 :::5672 :::* LISTEN 11932/beam.smp
tcp6 13 0 10.204.217.32:5672 10.204.217.32:51382 CLOSE_WAIT 11932/beam.smp
tcp6 13 0 10.204.217.32:5672 10.204.217.33:59963 CLOSE_WAIT 11932/beam.smp
tcp6 13 0 10.204.217.32:5672 10.204.217.33:59953 CLOSE_WAIT 11932/beam.smp
tcp6 13 0 10.204.217.32:5672 10.204.217.32:51385 CLOSE_WAIT 11932/beam.smp
root@nodel11-vm2:~# docker exec -it controller bash

root@nodel11-vm2:~# docker top 1f5102118a2b | grep rabbitmq

root@nodel11-vm2:~# netstat -anp | grep 5672
tcp 0 0 0.0.0.0:25672 0.0.0.0:* LISTEN 11932/beam.smp
tcp 0 0 10.204.217.32:39966 10.204.217.33:5672 FIN_WAIT2 -
tcp 0 1 10.204.217.32:33269 10.204.217.31:5672 SYN_SENT 28023/python
tcp 0 1 10.204.217.32:33207 10.204.217.31:5672 SYN_SENT 27643/python
tcp 0 1 10.204.217.32:33299 10.204.217.31:5672 SYN_SENT 22881/python
tcp 0 1 10.204.217.32:33401 10.204.217.31:5672 SYN_SENT 27855/python
tcp 0 0 10.204.217.32:34955 10.204.217.32:5672 FIN_WAIT2 -
tcp6 0 0 :::5672 :::* LISTEN 11932/beam.smp
tcp6 13 0 10.204.217.32:5672 10.204.217.33:42699 CLOSE_WAIT 11932/beam.smp
tcp6 13 0 10.204.217.32:5672 10.204.217.32:34955 CLOSE_WAIT 11932/beam.smp
root@nodel11-vm2:~# ps -aux | grep epmd
rabbitmq 11903 0.0 0.0 9176 2248 ? S Nov24 0:03 /usr/lib/erlang/erts-5.10.4/bin/epmd -daemon
root 28954 0.0 0.0 10480 912 pts/9 S+ 05:51 0:00 grep --color=auto epmd
root@nodel11-vm2:~# netstat -anp | grep 11903
tcp6 0 0 :::4369 :::* LISTEN 11903/epmd
tcp6 0 0 127.0.0.1:4369 127.0.0.1:39364 ESTABLISHED 11903/epmd
unix 2 [ ] DGRAM 1874550 11903/epmd

root@nodel11-vm2:~# docker top 1f5102118a2b | grep epmd
root@nodel11-vm2:~# docker top 1f5102118a2b | grep beam

root@nodel11-vm2:~# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c5e8066f5dc4 10.204.217.117:5100/regress_image-contrail-analytics:R4.1 "/bin/sh -c /entry..." 2 days ago Up 2 days analytics
4796829dc25a 10.204.217.117:5100/regress_image-contrail-analyticsdb:R4.1 "/bin/sh -c /entry..." 2 days ago Up 2 days analyticsdb
1f5102118a2b 10.204.217.117:5100/regress_image-contrail-controller:R4.1 "/bin/sh -c /entry..." 2 days ago Up 2 minutes controller

root@nodel11-vm2:~# ps -aux | grep beam
rabbitmq 11932 1.0 0.1 2189172 65528 ? Sl Nov24 42:05 /usr/lib/erlang/erts-5.10.4/bin/beam.smp -W w -K true -A30 -P 1048576 -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.5.0/sbin/../ebin -noshell -noinput -s rabbit boot -sname rabbit@nodel11-vm2 -boot start_sasl -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/<email address hidden>"} -rabbit sasl_error_logger {file,"/<email address hidden>"} -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/lib/rabbitmq_server-3.5.0/sbin/../plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rabbit@nodel11-vm2-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@nodel11-vm2" -kernel inet_dist_listen_min 25672 -kernel inet_dist_listen_max 25672
root 32564 0.0 0.0 10480 908 pts/9 S+ 05:53 0:00 grep --color=auto beam
root@nodel11-vm2:~# ps -aux | grep epmd
rabbitmq 11903 0.0 0.0 9176 2248 ? S Nov24 0:03 /usr/lib/erlang/erts-5.10.4/bin/epmd -daemon
root 32697 0.0 0.0 10480 908 pts/9 S+ 05:53 0:00 grep --color=auto epmd
root@nodel11-vm2:~# kill -9 11903
root@nodel11-vm2:~# kill -9 11932

root@nodel11-vm2:~# docker top 1f5102118a2b | grep beam

Logs and json copied :

/home/bhushana/Documents/technical/bugs/bug-id