I still saw this with cs:trusty/rabbitmq-server-48
juju status was showing this about rabbit:
rabbitmq-server/0 active idle 1.25.5 2/lxc/0 5672/tcp 10.96.6.63 Unit is ready and clustered
rabbitmq-server/1 maintenance executing 1.25.5 1/lxc/2 5672/tcp 10.96.8.95 Starting RabbitMQ
rabbitmq-server/2 active idle 1.25.5 5/lxc/0 5672/tcp 10.96.9.186 Unit is ready and clustered
and the logs on rabbitmq-server/1 were full of:
Mnesia('rabbit@juju-machine-1-lxc-2'): ** ERROR ** Mnesia on 'rabbit@juju-machine-1-lxc-2' could not connect to node(s) ['rabbit@juju-machine-5-lxc-0']
/etc/hosts on rabbitmq-server/1 had:
# cat /etc/hosts
127.0.0.1 localhost
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
10.96.8.95 juju-machine-1-lxc-2
10.96.6.63 juju-machine-2-lxc-0
I still saw this with cs:trusty/ rabbitmq- server- 48
juju status was showing this about rabbit:
rabbitmq-server/0 active idle 1.25.5 2/lxc/0 5672/tcp 10.96.6.63 Unit is ready and clustered
rabbitmq-server/1 maintenance executing 1.25.5 1/lxc/2 5672/tcp 10.96.8.95 Starting RabbitMQ
rabbitmq-server/2 active idle 1.25.5 5/lxc/0 5672/tcp 10.96.9.186 Unit is ready and clustered
and the logs on rabbitmq-server/1 were full of: 'rabbit@ juju-machine- 1-lxc-2' ): ** ERROR ** Mnesia on 'rabbit@ juju-machine- 1-lxc-2' could not connect to node(s) ['rabbit@ juju-machine- 5-lxc-0' ]
Mnesia(
/etc/hosts on rabbitmq-server/1 had:
# cat /etc/hosts
127.0.0.1 localhost
# The following lines are desirable for IPv6 capable hosts 1-lxc-2 2-lxc-0
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
10.96.8.95 juju-machine-
10.96.6.63 juju-machine-
Process listing was: juju/tools/ machine- 1-lxc-2/ jujud machine --data-dir /var/lib/juju --machine-id 1/lxc/2 --debug juju/tools/ unit-rabbitmq- server- 1/jujud unit --data-dir /var/lib/juju --unit-name rabbitmq-server/1 --debug juju/agents/ unit-rabbitmq- server- 1/charm/ hooks/cluster- relation- changed rabbitmqctl start_app rabbitmq/ bin/rabbitmqctl "start_app" rabbitmq/ bin/rabbitmqctl "start_app" erlang/ erts-5. 10.4/bin/ beam.smp -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitm erlang/ erts-5. 10.4/bin/ epmd -daemon rabbitmq- server erlang/ erts-5. 10.4/bin/ beam.smp -W w -K true -A30 -P 1048576 -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /us
root 922 0.0 0.1 511780 22576 ? Ssl 16:07 0:00 /var/lib/
root 1025 0.0 0.1 487008 22728 ? Ssl 16:20 0:00 /var/lib/
root 10406 0.0 0.1 92464 20136 ? S 16:26 0:02 \_ /usr/bin/python /var/lib/
root 11222 0.0 0.0 4440 668 ? S 16:27 0:00 \_ /bin/sh /usr/sbin/
root 11230 0.0 0.0 61672 1676 ? S 16:27 0:00 \_ su rabbitmq -s /bin/sh -c /usr/lib/
rabbitmq 11231 0.0 0.0 4440 644 ? Ss 16:27 0:00 \_ sh -c /usr/lib/
rabbitmq 11232 0.0 0.1 370876 24084 ? Sl 16:27 0:00 \_ /usr/lib/
rabbitmq 11264 0.0 0.0 7456 428 ? Ss 16:27 0:00 \_ inet_gethost 4
rabbitmq 11265 0.0 0.0 9548 632 ? S 16:27 0:00 \_ inet_gethost 4
rabbitmq 8002 0.0 0.0 7492 320 ? S 16:24 0:00 /usr/lib/
rabbitmq 10803 0.0 0.0 4440 648 ? S 16:26 0:00 /bin/sh /usr/sbin/
rabbitmq 10834 19.2 0.3 2174496 55720 ? Sl 16:26 9:16 \_ /usr/lib/
rabbitmq 10943 1.6 0.0 7456 428 ? Ss 16:26 0:46 \_ inet_gethost 4
rabbitmq 10944 5.2 0.0 13776 784 ? S 16:26 2:32 \_ inet_gethost 4
I added this to /etc/hosts: 5-lxc-0
# added manually by andreas
10.96.9.186 juju-machine-
And just by saving that file, rabbitmq-server/1 unwedged and the deployment proceeded.