[ 4.0.3.0-119] Provisioning fails when openstack_manage_amqp is set to False in cluster.json
Affects | Status | Importance | Assigned to | Milestone | ||
---|---|---|---|---|---|---|
Juniper Openstack | Status tracked in Trunk | |||||
R4.0 |
New
|
Undecided
|
Abhay Joshi | |||
Trunk |
New
|
Undecided
|
Abhay Joshi |
Bug Description
/opt/contrail/
+------
| id | status | ip_address | mac_address |
+------
| nodel11-vm1 | provision_failed | 10.204.217.31 | 52:53:58:01:00:01 |
| nodel11-vm2 | provision_failed | 10.204.217.32 | 52:53:58:01:00:02 |
| nodel11-vm3 | provision_failed | 10.204.217.33 | 52:53:58:01:00:03 |
| nodel11-vm4 | provision_failed | 10.204.217.34 | 52:53:58:01:00:04 |
| nodel11-vm5 | provision_failed | 10.204.217.36 | 52:53:58:01:00:05 |
| nodel11-vm6 | provision_failed | 10.204.217.37 | 52:53:58:01:00:06 |
+------
+------
| id | cluster_id | role_being_
+------
| nodel11-vm1 | nodel11_cluster | None | [] | [u'haproxy', u'pre_exec_
| nodel11-vm2 | nodel11_cluster | None | [] | [u'haproxy', u'pre_exec_
| nodel11-vm3 | nodel11_cluster | None | [] | [u'haproxy', u'pre_exec_
+------
root@nodei5:
root@nodel11-vm2:~# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c5e8066f5dc4 10.204.
4796829dc25a 10.204.
1f5102118a2b 10.204.
'.
<---Controller keeps restarting as rabbitmq is unable to come up in the controller.
When the following steps were executed, the system recovered:
1) root@nodel11-vm2:~# kill -9 11903 beam <--stop beam and epmd on all the openstack nodes
root@nodel11-vm2:~# kill -9 11932 epmd
2) restart all the controllers
3) Start provisioning
root@nodel11-vm1:~# ps -aux | grep rabbitmq
rabbitmq 6303 0.0 0.0 7508 320 ? S 06:45 0:00 /usr/lib/
rabbitmq 6443 4.0 0.1 2173812 45024 ? Sl 06:45 0:02 /usr/lib/
rabbitmq 7265 0.0 0.0 7472 440 ? Ss 06:45 0:00 inet_gethost 4
rabbitmq 7267 0.0 0.0 9564 644 ? S 06:45 0:00 inet_gethost 4
root@nodel11-vm1:~# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ff1af24013de 10.204.
2175f7e6531f 10.204.
b045a4716a03 10.204.
root@nodel11-vm1:~#
root@nodel11-vm1:~# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ff1af24013de 10.204.
2175f7e6531f 10.204.
b045a4716a03 10.204.
root@nodel11-vm1:~# ps -aux | grep beam
rabbitmq 6443 3.0 0.1 2174836 44960 ? Sl 06:45 0:02 /usr/lib/
root 21542 0.0 0.0 10480 908 pts/7 S+ 06:47 0:00 grep --color=auto beam
root@nodel11-vm1:~# ps -aux | grep epmd
rabbitmq 6303 0.0 0.0 7508 320 ? S 06:45 0:00 /usr/lib/
root 23143 0.0 0.0 10480 912 pts/7 S+ 06:47 0:00 grep --color=auto epmd
root@nodel11-vm1:~# kill -9 6443
root@nodel11-vm1:~# kill -9 6303
root@nodel11-vm1:~# docker restart controller
controller
root@nodel11-vm1:~# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ff1af24013de 10.204.
2175f7e6531f 10.204.
b045a4716a03 10.204.
root@nodel11-vm1:~# docker top b045a4716a03 | grep rabbit <--rabbitmq came up in the controller as expected
root 30394 30393 0 06:48 ? 00:00:00 /bin/sh /etc/init.
root 30512 30394 0 06:48 ? 00:00:00 su rabbitmq -s /bin/sh -c /usr/lib/
message+ 30513 27741 4 06:48 ? 00:00:00 /usr/lib/
message+ 30523 30512 0 06:48 ? 00:00:00 sh -c /usr/lib/
message+ 30524 30523 1 06:48 ? 00:00:00 /usr/lib/
root@nodel11-vm1:~#
root@nodel11-vm2:~# docker top 1f5102118a2b | grep rabbitmq
message+ 26683 9571 2 06:47 ? 00:00:06 /usr/lib/
root@nodel11-vm3:~# docker top a826d0add47c | grep rabbit
message+ 19775 9374 2 06:55 ? 00:00:02 /usr/lib/
root@nodel11-vm3:~#
root@nodel11-vm2:~# docker top 1f5102118a2b | grep rabbitmq <--no rabbitmq in the controller
<-- As openstack_
root@nodel11-vm2:~# ps -aux | grep rabbitmq <---rabbitmq running on the host
rabbitmq 11903 0.0 0.0 9176 2248 ? S Nov24 0:03 /usr/lib/
rabbitmq 11932 1.0 0.1 2189172 65528 ? Sl Nov24 42:02 /usr/lib/
rabbitmq 12143 0.0 0.0 7472 436 ? Ss Nov24 0:11 inet_gethost 4
rabbitmq 12144 0.0 0.0 9564 640 ? S Nov24 0:20 inet_gethost 4
root 14032 0.0 0.0 10480 916 pts/9 S+ 05:47 0:00 grep --color=auto rabbitmq
root@nodel11-vm2:~# netstat -anp | grep 11932
tcp 0 0 0.0.0.0:25672 0.0.0.0:* LISTEN 11932/beam.smp
tcp 0 0 127.0.0.1:39364 127.0.0.1:4369 ESTABLISHED 11932/beam.smp
tcp6 0 0 :::5672 :::* LISTEN 11932/beam.smp
tcp6 13 0 10.204.217.32:5672 10.204.217.32:51382 CLOSE_WAIT 11932/beam.smp
tcp6 13 0 10.204.217.32:5672 10.204.217.33:59963 CLOSE_WAIT 11932/beam.smp
tcp6 13 0 10.204.217.32:5672 10.204.217.33:59953 CLOSE_WAIT 11932/beam.smp
tcp6 13 0 10.204.217.32:5672 10.204.217.32:51385 CLOSE_WAIT 11932/beam.smp
root@nodel11-vm2:~# docker exec -it controller bash
root@nodel11-vm2:~# docker top 1f5102118a2b | grep rabbitmq
root@nodel11-vm2:~# netstat -anp | grep 5672
tcp 0 0 0.0.0.0:25672 0.0.0.0:* LISTEN 11932/beam.smp
tcp 0 0 10.204.217.32:39966 10.204.217.33:5672 FIN_WAIT2 -
tcp 0 1 10.204.217.32:33269 10.204.217.31:5672 SYN_SENT 28023/python
tcp 0 1 10.204.217.32:33207 10.204.217.31:5672 SYN_SENT 27643/python
tcp 0 1 10.204.217.32:33299 10.204.217.31:5672 SYN_SENT 22881/python
tcp 0 1 10.204.217.32:33401 10.204.217.31:5672 SYN_SENT 27855/python
tcp 0 0 10.204.217.32:34955 10.204.217.32:5672 FIN_WAIT2 -
tcp6 0 0 :::5672 :::* LISTEN 11932/beam.smp
tcp6 13 0 10.204.217.32:5672 10.204.217.33:42699 CLOSE_WAIT 11932/beam.smp
tcp6 13 0 10.204.217.32:5672 10.204.217.32:34955 CLOSE_WAIT 11932/beam.smp
root@nodel11-vm2:~# ps -aux | grep epmd
rabbitmq 11903 0.0 0.0 9176 2248 ? S Nov24 0:03 /usr/lib/
root 28954 0.0 0.0 10480 912 pts/9 S+ 05:51 0:00 grep --color=auto epmd
root@nodel11-vm2:~# netstat -anp | grep 11903
tcp6 0 0 :::4369 :::* LISTEN 11903/epmd
tcp6 0 0 127.0.0.1:4369 127.0.0.1:39364 ESTABLISHED 11903/epmd
unix 2 [ ] DGRAM 1874550 11903/epmd
root@nodel11-vm2:~# docker top 1f5102118a2b | grep epmd
root@nodel11-vm2:~# docker top 1f5102118a2b | grep beam
root@nodel11-vm2:~# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c5e8066f5dc4 10.204.
4796829dc25a 10.204.
1f5102118a2b 10.204.
root@nodel11-vm2:~# ps -aux | grep beam
rabbitmq 11932 1.0 0.1 2189172 65528 ? Sl Nov24 42:05 /usr/lib/
root 32564 0.0 0.0 10480 908 pts/9 S+ 05:53 0:00 grep --color=auto beam
root@nodel11-vm2:~# ps -aux | grep epmd
rabbitmq 11903 0.0 0.0 9176 2248 ? S Nov24 0:03 /usr/lib/
root 32697 0.0 0.0 10480 908 pts/9 S+ 05:53 0:00 grep --color=auto epmd
root@nodel11-vm2:~# kill -9 11903
root@nodel11-vm2:~# kill -9 11932
root@nodel11-vm2:~# docker top 1f5102118a2b | grep beam
After re provisioning using the cmd server-manager provision -F --cluster_id nodel11_cluster regress_image
Contrail-api on one of the controllers could not come up, but after restart the same, the process recovered.
root@nodel11-
== Contrail Control ==
contrail-control active
contrail-
contrail-dns active
contrail-named active
== Contrail Config ==
contrail-api:0 initializing (Generic Connection:
contrail-
contrail-
contrail-schema active
contrail-
== Contrail Config Database==
contrail-database: active
== Contrail Web UI ==
contrail-webui active
contrail-
root@nodel11-
contrail-api:0: stopped
contrail-api:0: started
root@nodel11-
== Contrail Control ==
contrail-control active
contrail-
contrail-dns active
contrail-named active
== Contrail Config ==
contrail-api:0 active
contrail-
contrail-
contrail-schema active
contrail-
== Contrail Config Database==
contrail-database: active
== Contrail Web UI ==
contrail-webui active
contrail-
root@nodel11-
root@nodel11-
== Contrail Control ==
contrail-control active
contrail-
contrail-dns active
contrail-named active
== Contrail Config ==
contrail-api:0 initializing (Generic Connection:
contrail-
contrail-
contrail-schema active
contrail-
Logs and json copied :
/home/bhushana/