All three ceph-mon units not clustered

Adding autopilot-log-analyser so that we remember to write an analyser issue for this.

@fginther - can you please provide a full juju status, including tabular? Trying to understand the placement, metal vs container, etc.

Also, a pointer to the specific code which adds the charm config to ceph-* in your model will be helpful in reproducing. That, or a bundle if this was deployed with a bundle.yaml. Thank you.

Fwiw, I get 404 trying: https://ci.lscape.net/job/landscape-system-tests/4981

This one in json, so you can parse it with python and print or search whatever you want

The ceph-* charm config is trickier, as it's indeed code that does that.

The code is in https://bazaar.launchpad.net/~landscape/landscape/trunk/files/head:/canonical/landscape/model/openstack/services.py

@ahasenack - in that case can you paste the resultant charm config for ceph-*? I'm not able to derive exactly what those values are from the code.

Please also attach the corresponding ceph-osd.tar.gz log archive. Thanks again.

Sorry, can't show you charm config as the deployment isn't live anymore. We want to include that data in our log collection script eventually.

ceph-{mon,osd,radosgw} logs

Thanks for the added info, I believe that will be enough for the storage charm folks to triage.

## To summarize log spelunking:
3 ceph-mons in containers
6 ceph-osds
xenial-mitaka
Juju 2.1-beta4

## ceph-mon syslog says:
Jan 11 20:28:33 juju-bd739b-5-lxd-5 systemd[1]: Started Ceph cluster monitor daemon.
Jan 11 20:28:34 juju-bd739b-5-lxd-5 ceph-mon[61101]: 2017-01-11 20:28:34.021764 7f3dc8945580 -1 did not load config file, using default settings.
Jan 11 20:28:34 juju-bd739b-5-lxd-5 ceph-mon[61101]: monitor data directory at '/var/lib/ceph/mon/ceph-juju-bd739b-5-lxd-5' does not exist: have you run 'mkfs'?
Jan 11 20:28:34 juju-bd739b-5-lxd-5 systemd[1]: ceph-mon.service: Main process exited, code=exited, status=1/FAILURE
Jan 11 20:28:34 juju-bd739b-5-lxd-5 systemd[1]: ceph-mon.service: Unit entered failed state.
Jan 11 20:28:34 juju-bd739b-5-lxd-5 systemd[1]: ceph-mon.service: Failed with result 'exit-code'.

And with that, I'll turn it over to a ceph charm SME. :-)

Was the `monitor-count` setting in ceph-mon's config.yaml set to something other than the default of 3? I don't see any log messages saying that the monitors were bootstrapping. The only thing the prevents that is setting the monitor-count.

monitor-count in ceph-mon is set to 3:

NUMBER_OF_STORAGE_REPLICAS = 3
...
    "ceph-mon": ServiceInfo(
        u"ceph-mon",
        config={"monitor-count": str(NUMBER_OF_STORAGE_REPLICAS),
                "source": OPENSTACK_SOURCE},
        count=NUMBER_OF_STORAGE_REPLICAS),

Beisner helped me find that the mon count is indeed 3. The other thing that would prevent bootstrapping is if the mon relation failed to query the other related mons. Do you have the relation data showing from each mon's perspective how many other related units it sees? That would be really helpful in narrowing this down.

No, this run isn't live anymore. All we have is essentially juju status and /var/log/** and some of /etc/** from all units.

To be honest the mon's won't even try to bootstrap until they can see each other over the relation data, so I suspect this is something network-ish with regards to LXD containers in Juju 2.1.x

Might be pertinent:

mon:14: still waiting for leader to setup keys

Which is odd:

      ceph-mon/0:
        juju-status: {current: idle, since: '11 Jan 2017 20:18:05Z', version: 2.1-beta4}
        leader: true

as that message is from the unit log on mon0.

So it looks like when the config-changed hook ran on each unit, none of the units where actually the leader - I've never see this behaviour from either the charm or juju.

Was this a one-off error or has this problem been seen multiple times?

Definitely something todo with wonky leadership election:

ceph-mon-0/var/log/juju/unit-ceph-mon-0.log:2017-01-11 17:15:44 INFO juju-log still waiting for leader to setup keys
ceph-mon-0/var/log/juju/unit-ceph-mon-0.log:2017-01-11 17:20:02 INFO juju-log mon:14: still waiting for leader to setup keys
ceph-mon-0/var/log/juju/unit-ceph-mon-0.log:2017-01-11 17:20:19 INFO juju-log mon:14: still waiting for leader to setup keys

ceph-mon-1/var/log/juju/unit-ceph-mon-1.log:2017-01-11 17:19:30 INFO juju-log still waiting for leader to setup keys
ceph-mon-1/var/log/juju/unit-ceph-mon-1.log:2017-01-11 17:19:47 INFO juju-log mon:14: still waiting for leader to setup keys
ceph-mon-1/var/log/juju/unit-ceph-mon-1.log:2017-01-11 17:20:01 INFO juju-log mon:14: still waiting for leader to setup keys
ceph-mon-1/var/log/juju/unit-ceph-mon-1.log:2017-01-11 17:20:10 INFO juju-log mon:14: still waiting for leader to setup keys

ceph-mon-2/var/log/juju/unit-ceph-mon-2.log:2017-01-11 17:19:28 INFO juju-log still waiting for leader to setup keys
ceph-mon-2/var/log/juju/unit-ceph-mon-2.log:2017-01-11 17:19:42 INFO juju-log mon:14: still waiting for leader to setup keys
ceph-mon-2/var/log/juju/unit-ceph-mon-2.log:2017-01-11 17:19:54 INFO juju-log mon:14: still waiting for leader to setup keys
ceph-mon-2/var/log/juju/unit-ceph-mon-2.log:2017-01-11 17:20:07 INFO juju-log mon:14: still waiting for leader to setup keys

all three units where waiting for a leader to appear and bootstrap the data for the cluster.

I believe the call at 2017-01-11 17:15:44 by ceph-mon/0 should have resulted in it being declared the lead unit.