MOS 9.0 installation - RabbitMQ hostname issue

While testing the latest Mirantis release (9.0, Mitaka) I ran into some issues while doing a test environment deployment. After some investigation, I noticed something sucpisious about RabbitMQ behaviour.

After a couple of unsuccessfull deployments I noticed that rabbitmq-server keeps failing after the initial setup. I believe the reason is the misconfigured NODENAME parameter which doesn't use the system hostname correctly. Following are the steps that should show what's happening:

# Boot system from MOS9 ISO. At the Fuel menu set the following options:
DNS & Hostname >> Hostname >> "fuelsrv"
DNS & Hostname >> External DNS >> leave blank
DNS & Hostname >> Hostname to test DNS >> leave blank
Bootstrap Image >> Select "Skip buildinig bootstrap image"
Set Root Password >> Quit Setup >> Save and Continue + press "F8" afterwards

# The Fuel should install everything correctly. When the installation finishes, log into the system and verify hostname and RabbitMQ configuration:
[root@fuel ~]# hostname
fuel.domain.tld

[root@fuel ~]# hostnamectl
   Static hostname: fuelsrv
Transient hostname: fuel.domain.tld
         Icon name: computer-vm
           Chassis: vm
        Machine ID: 5e522aabe4bf4367a8ed779dcd1c21b6
           Boot ID: 187546a671094d7ea6433a608f0da8f5
    Virtualization: kvm
  Operating System: CentOS Linux 7 (Core)
       CPE OS Name: cpe:/o:centos:centos:7
            Kernel: Linux 3.10.0-327.18.2.el7.x86_64
      Architecture: x86-64

[root@fuel ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
10.20.0.2 fuelsrv.domain.tld fuelsrv

[root@fuel ~]# cat /etc/rabbitmq/rabbitmq-env.conf
ERL_EPMD_ADDRESS=10.20.0.2
NODENAME=rabbit@fuel
NODE_IP_ADDRESS=10.20.0.2
NODE_PORT=5672
RABBITMQ_SERVER_ERL_ARGS=+K true +A30 +P 1048576

[root@fuel ~]# systemctl status rabbitmq-server
● rabbitmq-server.service - RabbitMQ broker
   Loaded: loaded (/usr/lib/systemd/system/rabbitmq-server.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/rabbitmq-server.service.d
           └─limits.conf
   Active: active (running) since Sun 2016-07-17 11:30:34 UTC; 9min ago
 Main PID: 16598 (beam)
   Status: "Initialized"
   CGroup: /system.slice/rabbitmq-server.service
           ├─16598 /usr/lib64/erlang/erts-7.1/bin/beam -W w -A 64 -P 1048576 -K true -- -r...
           ├─16778 inet_gethost 4
           └─16779 inet_gethost 4

Jul 17 11:30:31 fuel.domain.tld rabbitmq-server[16598]: RabbitMQ 3.6.1. Copyright (C) 200....
Jul 17 11:30:31 fuel.domain.tld rabbitmq-server[16598]: ## ## Licensed under the MP.../
Jul 17 11:30:31 fuel.domain.tld rabbitmq-server[16598]: ## ##
Jul 17 11:30:31 fuel.domain.tld rabbitmq-server[16598]: ########## Logs: /var/log/rabbit...g
Jul 17 11:30:31 fuel.domain.tld rabbitmq-server[16598]: ###### ## /var/log/rabbit...g
Jul 17 11:30:31 fuel.domain.tld rabbitmq-server[16598]: ##########
Jul 17 11:30:31 fuel.domain.tld rabbitmq-server[16598]: Starting broker...
Jul 17 11:30:34 fuel.domain.tld rabbitmq-server[16598]: systemd unit for activation check..."
Jul 17 11:30:34 fuel.domain.tld systemd[1]: Started RabbitMQ broker.
Jul 17 11:30:34 fuel.domain.tld rabbitmq-server[16598]: completed with 7 plugins.
Hint: Some lines were ellipsized, use -l to show in full.

# So far everything looks ok. Notice that the system hostname is currently at the default value ("fuel"). Now, reboot the system, log into back into it and check the forementioned configuration again:
[root@fuelsrv ~]# hostname
fuelsrv

[root@fuelsrv ~]# hostnamectl
   Static hostname: fuelsrv
         Icon name: computer-vm
           Chassis: vm
        Machine ID: 5e522aabe4bf4367a8ed779dcd1c21b6
           Boot ID: f32fc83db9b24d05bd4070da8f8cc423
    Virtualization: kvm
  Operating System: CentOS Linux 7 (Core)
       CPE OS Name: cpe:/o:centos:centos:7
            Kernel: Linux 3.10.0-327.18.2.el7.x86_64
      Architecture: x86-64

[root@fuelsrv ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
10.20.0.2 fuelsrv.domain.tld fuelsrv

[root@fuelsrv ~]# cat /etc/rabbitmq/rabbitmq-env.conf
ERL_EPMD_ADDRESS=10.20.0.2
NODENAME=rabbit@fuel
NODE_IP_ADDRESS=10.20.0.2
NODE_PORT=5672
RABBITMQ_SERVER_ERL_ARGS=+K true +A30 +P 1048576

[root@fuelsrv ~]# systemctl status rabbitmq-server
● rabbitmq-server.service - RabbitMQ broker
   Loaded: loaded (/usr/lib/systemd/system/rabbitmq-server.service; enabled; vendor preset: d isabled)
  Drop-In: /etc/systemd/system/rabbitmq-server.service.d
           └─limits.conf
   Active: failed (Result: start-limit) since Sun 2016-07-17 11:42:12 UTC; 31s ago
  Process: 2933 ExecStop=/usr/lib/rabbitmq/bin/rabbitmqctl stop (code=exited, status=69)
  Process: 2825 ExecStart=/usr/lib/rabbitmq/bin/rabbitmq-server (code=exited, status=1/FAILUR E)
 Main PID: 2825 (code=exited, status=1/FAILURE)
   Status: "Exited."

Jul 17 11:42:12 fuelsrv systemd[1]: rabbitmq-server.service scheduled restart job.
Jul 17 11:42:12 fuelsrv systemd[1]: rabbitmq-server.service changed auto-restart -> dead
Jul 17 11:42:12 fuelsrv systemd[1]: Job rabbitmq-server.service/restart finished, result=done
Jul 17 11:42:12 fuelsrv systemd[1]: Converting job rabbitmq-server.service/restart -> r...art
Jul 17 11:42:12 fuelsrv systemd[1]: start request repeated too quickly for rabbitmq-ser...ice
Jul 17 11:42:12 fuelsrv systemd[1]: rabbitmq-server.service changed dead -> failed
Jul 17 11:42:12 fuelsrv systemd[1]: Job rabbitmq-server.service/start finished, result=failed
Jul 17 11:42:12 fuelsrv systemd[1]: Failed to start RabbitMQ broker.
Jul 17 11:42:12 fuelsrv systemd[1]: Unit rabbitmq-server.service entered failed state.
Jul 17 11:42:12 fuelsrv systemd[1]: rabbitmq-server.service failed.
Hint: Some lines were ellipsized, use -l to show in full.

# Notice that rabbitmq-server is now not working currently. Doing an service restart via systemctl doesn't help, but starting the service directly indicates what could be the problem:
[root@fuelsrv ~]# systemctl restart rabbitmq-server
Job for rabbitmq-server.service failed because start of the service was attempted too often. See "systemctl status rabbitmq-server.service" and "journalctl -xe" for details.
To force a start use "systemctl reset-failed rabbitmq-server.service" followed by "systemctl start rabbitmq-server.service" again.

[root@fuelsrv ~]# runuser -u rabbitmq /usr/lib/rabbitmq/bin/rabbitmq-server
ERROR: epmd error for host fuel: nxdomain (non-existing domain)

# If we take a look at the RabbitMQ configuration we can see that it's actually using the wrong hostname for it's NODENAME parameter (notice that the system hostname is now set to "fuelsrv", as set during the Fuel installation):
[root@fuelsrv ~]# cat /etc/rabbitmq/rabbitmq-env.conf
ERL_EPMD_ADDRESS=10.20.0.2
NODENAME=rabbit@fuel
NODE_IP_ADDRESS=10.20.0.2
NODE_PORT=5672
RABBITMQ_SERVER_ERL_ARGS=+K true +A30 +P 1048576

# After a simple change to the rabbitmq-env.conf the service should start sucessfully and work fine afterwards (and "survive" future reboots).
[root@fuelsrv ~]# cat /etc/rabbitmq/rabbitmq-env.conf
ERL_EPMD_ADDRESS=10.20.0.2
-NODENAME=rabbit@fuel
+NODENAME=rabbit@fuelsrv
NODE_IP_ADDRESS=10.20.0.2
NODE_PORT=5672
RABBITMQ_SERVER_ERL_ARGS=+K true +A30 +P 1048576

[root@fuelsrv ~]# runuser -u rabbitmq /usr/lib/rabbitmq/bin/rabbitmq-server

              RabbitMQ 3.6.1. Copyright (C) 2007-2016 Pivotal Software, Inc.
  ## ## Licensed under the MPL. See http://www.rabbitmq.com/
  ## ##
  ########## Logs: /<email address hidden>
  ###### ## /<email address hidden>
  ##########
              Starting broker...
 completed with 7 plugins.
^CStopping and halting node rabbit@fuelsrv ...
Gracefully halting Erlang VM

[root@fuelsrv ~]# grep -i fuelsrv /var/log/rabbitmq/rabbit@fuel*
/<email address hidden>:node : rabbit@fuelsrv
/<email address hidden>:log : /<email address hidden>
/<email address hidden>:sasl log : /<email address hidden>
/<email address hidden>:database dir : /var/lib/rabbitmq/mnesia/rabbit@fuelsrv
/<email address hidden>:Database directory at /var/lib/rabbitmq/mnesia/rabbit@fuelsrv is empty. Initialising from scratch...

[root@fuelsrv ~]# systemctl start rabbitmq-server
[root@fuelsrv ~]# systemctl status rabbitmq-server
● rabbitmq-server.service - RabbitMQ broker
   Loaded: loaded (/usr/lib/systemd/system/rabbitmq-server.service; enabled; vendor preset: d isabled)
  Drop-In: /etc/systemd/system/rabbitmq-server.service.d
           └─limits.conf
   Active: active (running) since Sun 2016-07-17 11:43:52 UTC; 5s ago
  Process: 2933 ExecStop=/usr/lib/rabbitmq/bin/rabbitmqctl stop (code=exited, status=69)
 Main PID: 3260 (beam)
   Status: "Initialized"
   CGroup: /system.slice/rabbitmq-server.service
           ├─3260 /usr/lib64/erlang/erts-7.1/bin/beam -W w -A 64 -P 1048576 -K true -- -ro...
           ├─3446 inet_gethost 4
           └─3447 inet_gethost 4

Jul 17 11:43:52 fuelsrv systemd[1]: rabbitmq-server.service: Got notification message f...60)
Jul 17 11:43:52 fuelsrv systemd[1]: rabbitmq-server.service: got MAINPID=3260
Jul 17 11:43:52 fuelsrv systemd[1]: rabbitmq-server.service: got READY=1
Jul 17 11:43:52 fuelsrv systemd[1]: rabbitmq-server.service changed start -> running
Jul 17 11:43:52 fuelsrv systemd[1]: Job rabbitmq-server.service/start finished, result=done
Jul 17 11:43:52 fuelsrv systemd[1]: Started RabbitMQ broker.
Jul 17 11:43:52 fuelsrv systemd[1]: rabbitmq-server.service: got STATUS=Initialized
Jul 17 11:43:52 fuelsrv rabbitmq-server[3260]: Unexpected status from systemd "Showing o...et
Jul 17 11:43:52 fuelsrv rabbitmq-server[3260]: systemd READY notification failed, beware...ts
Jul 17 11:43:52 fuelsrv rabbitmq-server[3260]: completed with 7 plugins.
Hint: Some lines were ellipsized, use -l to show in full.

Of course, this doesn't happen if the default hostname is used for the installation. Could somebody please try to reproduce and (possibly) confirm the described behaviour?