Comment 10 for bug 1592200

Revision history for this message
Andreas Hasenack (ahasenack) wrote : Re: rabbitmq_ctl stop didn't work and subsequent start gets stuck

I still saw this with cs:trusty/rabbitmq-server-48

juju status was showing this about rabbit:
rabbitmq-server/0 active idle 1.25.5 2/lxc/0 5672/tcp 10.96.6.63 Unit is ready and clustered
rabbitmq-server/1 maintenance executing 1.25.5 1/lxc/2 5672/tcp 10.96.8.95 Starting RabbitMQ
rabbitmq-server/2 active idle 1.25.5 5/lxc/0 5672/tcp 10.96.9.186 Unit is ready and clustered

and the logs on rabbitmq-server/1 were full of:
Mnesia('rabbit@juju-machine-1-lxc-2'): ** ERROR ** Mnesia on 'rabbit@juju-machine-1-lxc-2' could not connect to node(s) ['rabbit@juju-machine-5-lxc-0']

/etc/hosts on rabbitmq-server/1 had:
# cat /etc/hosts
127.0.0.1 localhost

# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts
10.96.8.95 juju-machine-1-lxc-2
10.96.6.63 juju-machine-2-lxc-0

Process listing was:
root 922 0.0 0.1 511780 22576 ? Ssl 16:07 0:00 /var/lib/juju/tools/machine-1-lxc-2/jujud machine --data-dir /var/lib/juju --machine-id 1/lxc/2 --debug
root 1025 0.0 0.1 487008 22728 ? Ssl 16:20 0:00 /var/lib/juju/tools/unit-rabbitmq-server-1/jujud unit --data-dir /var/lib/juju --unit-name rabbitmq-server/1 --debug
root 10406 0.0 0.1 92464 20136 ? S 16:26 0:02 \_ /usr/bin/python /var/lib/juju/agents/unit-rabbitmq-server-1/charm/hooks/cluster-relation-changed
root 11222 0.0 0.0 4440 668 ? S 16:27 0:00 \_ /bin/sh /usr/sbin/rabbitmqctl start_app
root 11230 0.0 0.0 61672 1676 ? S 16:27 0:00 \_ su rabbitmq -s /bin/sh -c /usr/lib/rabbitmq/bin/rabbitmqctl "start_app"
rabbitmq 11231 0.0 0.0 4440 644 ? Ss 16:27 0:00 \_ sh -c /usr/lib/rabbitmq/bin/rabbitmqctl "start_app"
rabbitmq 11232 0.0 0.1 370876 24084 ? Sl 16:27 0:00 \_ /usr/lib/erlang/erts-5.10.4/bin/beam.smp -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitm
rabbitmq 11264 0.0 0.0 7456 428 ? Ss 16:27 0:00 \_ inet_gethost 4
rabbitmq 11265 0.0 0.0 9548 632 ? S 16:27 0:00 \_ inet_gethost 4
rabbitmq 8002 0.0 0.0 7492 320 ? S 16:24 0:00 /usr/lib/erlang/erts-5.10.4/bin/epmd -daemon
rabbitmq 10803 0.0 0.0 4440 648 ? S 16:26 0:00 /bin/sh /usr/sbin/rabbitmq-server
rabbitmq 10834 19.2 0.3 2174496 55720 ? Sl 16:26 9:16 \_ /usr/lib/erlang/erts-5.10.4/bin/beam.smp -W w -K true -A30 -P 1048576 -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /us
rabbitmq 10943 1.6 0.0 7456 428 ? Ss 16:26 0:46 \_ inet_gethost 4
rabbitmq 10944 5.2 0.0 13776 784 ? S 16:26 2:32 \_ inet_gethost 4

I added this to /etc/hosts:
# added manually by andreas
10.96.9.186 juju-machine-5-lxc-0

And just by saving that file, rabbitmq-server/1 unwedged and the deployment proceeded.