Relies on DNS to resolve own hostname

So, this works just fine:

@cached
def get_host_ip(hostname=None):
    if config('prefer-ipv6'):
        return get_ipv6_addr()[0]

    hostname = hostname or unit_get('private-address')
    try:
        # Test to see if already an IPv4 address
        socket.inet_aton(hostname)
        return hostname
    except socket.error:
        return socket.gethostbyname(hostname)

If my analysis here is correct, then it doesn't just break Ceph; instead it would break all of the affected charms when hosts are not DNS-resolvable. Which never happens when using MAAS and the MAAS host acts as one's DNS server, but would probably be the case with most (all?) non-MAAS Juju providers.

This function is used to resolve IP addresses from hostnames; I'll take a look and see what's changed recently, but this approach was put together over a few cycles of testing against MAAS and OpenStack providers and provided the most generally applicable approach to resolving an IP address.

AFAICT this code has not changed in quite some time; so I'm not sure what's causing the change in behaviour that you are seeing.

Which provider are you using? It would be helpful to understand so we can see the full context.

Having a final attempt using gethostbyname might be a sensible improvement - from memory this might return 127.0.1.1 on quite a few of the Juju providers, but that's worth testing with to see.

"What if the other node *is* in DNS, but is a CNAME?"

The way that query is done means that a CNAME record gets resolved down into an underlying A record automatically.

MAAS used to structure dns in this way, with the actual hostname being a CNAME that pointed to an auto-generated A record.

The problem pops up when using the "manual" provider. I've just spun up a simplified test environment to confirm the problem; the exact Juju config is here if it helps: https://www.hastexo.com/resources/hints-and-kinks/ubuntu-openstack-juju-4-nodes/

After running that setup, cinder-volume and neutron-gateway are broken:

$ juju stat neutron-gateway --format=oneline

- neutron-gateway/0: charlie.example.com (error)

$ juju stat cinder-volume --format=oneline

- cinder-volume/0: daisy.example.com (error)

$ juju debug-log --replay | grep NXDOMAIN
unit-neutron-gateway-0[22632]: 2016-01-28 18:50:56 INFO unit.neutron-gateway/0.install logger.go:40 raise NXDOMAIN
unit-neutron-gateway-0[22632]: 2016-01-28 18:50:56 INFO unit.neutron-gateway/0.install logger.go:40 dns.resolver.NXDOMAIN
unit-cinder-volume-0[22570]: 2016-01-28 18:52:39 INFO unit.cinder-volume/0.install logger.go:40 raise NXDOMAIN
unit-cinder-volume-0[22570]: 2016-01-28 18:52:39 INFO unit.cinder-volume/0.install logger.go:40 dns.resolver.NXDOMAIN

By the way: why exactly is this just an INFO message when it's an unhandled exception that, according to the comment, "kills the hook"?

All nodes have an /etc/hosts file that makes them mutually resolvable by gethostbyname(), but they use 8.8.8.8 and 8.8.4.4 for their DNS servers, which means they obviously can't resolve each other's host names via DNS.

Forgot to add evidence that in the test environment nova-volume is broken too.

training@deploy:~$ juju stat nova-compute --format=oneline

- nova-compute/0: bob.example.com (error)

unit-nova-compute-0[1355]: 2016-01-28 19:15:01 INFO unit.nova-compute/0.config-changed logger.go:40 raise NXDOMAIN
unit-nova-compute-0[1355]: 2016-01-28 19:15:01 INFO unit.nova-compute/0.config-changed logger.go:40 raise NXDOMAIN
unit-nova-compute-0[1355]: 2016-01-28 19:15:01 INFO unit.nova-compute/0.config-changed logger.go:40 dns.resolver.NXDOMAIN
unit-nova-compute-0[1355]: 2016-01-28 19:15:01 INFO unit.nova-compute/0.config-changed logger.go:40 dns.resolver.NXDOMAIN

Just as a reference point. It appears that what this charm helper *really* wants to do is get the host's "default" IP address. Recognizing that a reverse name lookup is a rather poor way of finding that out, Ansible does an ip route get to two well-known IP addresses (v4 and v6) to determine the outgoing interface for the default route, and then returns that interface's address as ansible_default_ipv4.address, and ansible_default_ipv6.address.

See https://github.com/ansible/ansible/blob/devel/lib/ansible/module_utils/facts.py#L1918 for details.

I just ran into this bug as well. The function is ipv4 only which is unfortunate. If DNS fails to resolve it falls back on this: https://docs.python.org/3/library/socket.html#socket.gethostbyname which is v4 only.

Marking the charm-ceph task wontfix as the ceph charm has been removed from support for a while now