service restart error not caught, waits forever

I induced a failure to get the charm moving on, then tried a restart via the shell:

root@juju-machine-0-lxc-3:/var/log/rabbitmq# service rabbitmq-server restart
 * Restarting message broker rabbitmq-server [fail]
root@juju-machine-0-lxc-3:/var/log/rabbitmq# echo $?
1

It failed as it should, and for the same reason:
# cat /var/log/rabbitmq/startup_log
ERROR: epmd error for host "euphoric-hook": nxdomain (non-existing domain)

So why didn't the charm catch it? I don't know, the code looks fine at first glance.

def service(action, service_name):
    """Control a system service"""
    if init_is_systemd():
        cmd = ['systemctl', action, service_name]
    else:
        cmd = ['service', service_name, action]
    return subprocess.call(cmd) == 0

I'm attaching the rabbit unit logs.

It really didn't notice the failure:
2016-04-26 19:44:02 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["Starting rabbitmq-server."]
2016-04-26 19:44:02 INFO juju-log Starting rabbitmq-server.
2016-04-26 19:44:02 INFO config-changed * Restarting message broker rabbitmq-server
2016-04-26 19:44:04 INFO config-changed ...fail!
2016-04-26 19:44:04 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "INFO" "getting local nodename for ip address: 10.96.12.161"]
2016-04-26 19:44:04 INFO juju-log getting local nodename for ip address: 10.96.12.161
2016-04-26 19:44:04 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "INFO" "local nodename: euphoric-hook"]
2016-04-26 19:44:04 INFO juju-log local nodename: euphoric-hook
2016-04-26 19:44:04 INFO worker.uniter.jujuc server.go:173 running hook tool "status-set" ["maintenance" "Waiting for rabbitmq app to start: /<email address hidden>"]
2016-04-26 19:44:04 INFO worker.uniter.jujuc server.go:173 running hook tool "juju-log" ["-l" "DEBUG" "Running ['/usr/sbin/rabbitmqctl', 'wait', '/<email address hidden>']"]
2016-04-26 19:44:04 DEBUG juju-log Running ['/usr/sbin/rabbitmqctl', 'wait', '/<email address hidden>']
2016-04-26 19:44:04 INFO config-changed Waiting for 'rabbit@euphoric-hook' ...

When rabbit is deployed with juju1, the following happens differently:
a) /etc/resolv.conf gets a "search <domain>" entry, that an lxc under MAAS deployed with juju2 does not

b) the lxc IP is just a normal IP obtained via DHCP, instead of a device in MAAS:
Apr 27 20:00:31 atlas dhcpd: DHCPREQUEST for 10.96.9.136 (10.96.0.10) from 00:16:3e:55:8e:a4 (juju-machine-0-lxc-7) via br0

When deployed with juju2, the container gets an IP from a "maas device":
Apr 26 19:39:28 atlas maas.node_query: [INFO] shawmut: Power is on.
Apr 26 19:41:08 atlas maas.api: [INFO] euphoric-hook: Added new device (parent: shawmut)
Apr 26 19:41:09 atlas maas.dns: [INFO] Generating new DNS zone file for scapestack

This is what triggered this rabbit bug: no "search <domain>" entry in /etc/resolv.conf. One can argue that it's not a bug in this charm then, but the charm should have failed right away when the restart action failed, instead of getting stuck forever. So there is still a charm bug.

rabbitmq/erlang is very sensitive to broken/poorly configured forward/reverse DNS.

We can't/should not try to fix that specific in the charm (that's more of a general charm environment problem for Juju IMHO), but the change made in commit 38912a540c8d259cf25ef29adc55c3c5ff55157b does introduce a 180 second timeout on all wait operations, which should mean that the charm enters an error start if RMQ fails to startup due to this type of issue.

Failure in service_restart is async to the subprocess call that makes it (so can't fix that one).

Timeout of 180 on wait should fix bug 2.

Marking Fix Released (this was included in the 16.10 charm release).