quickstart deployment fails to add relations when bootstrap goes "down"

This issue may relate to bug 1455260.

Thanks for the triage Curtis! Just another data point here.. I fired up a Trusty vagrant image with ppa:juju/stable and see the same problem -- the bootstrap/juju-gui unit (0) state goes "down", but all units continue deploying and their states eventually become "started". However, no relations are created. Same deployment as before:

$ juju quickstart u/bigdata-dev/apache-core-batch-processing --debug

I see similar output with 1.23.3 (though less informative than 1.24):

machines:
  "0":
    agent-state: down
    agent-state-info: (started)
    agent-version: 1.23.3
    dns-name: 10.55.60.13
    instance-id: c3608547-1d59-4c6e-92db-03fb01f4c06c
    instance-state: ACTIVE
    series: trusty
    hardware: arch=amd64 cpu-cores=1 mem=2048M root-disk=10240M availability-zone=nova
    state-server-member-status: has-vote
services:
  juju-gui:
    charm: cs:trusty/juju-gui-27
    exposed: true
    units:
      juju-gui/0:
        agent-state: down
        agent-state-info: (started)
        agent-version: 1.23.3
        machine: "0"
        open-ports:
        - 80/tcp
        - 443/tcp
        public-address: 10.55.60.13

Versions, etc:

vagrant@vagrant-ubuntu-trusty-64:~$ cat /etc/issue
Ubuntu 14.04.2 LTS \n \l

vagrant@vagrant-ubuntu-trusty-64:~$ dpkg -l | grep juju
ii juju 1.23.3-0ubuntu1~14.04.1~juju1 all next generation service orchestration system
ii juju-core 1.23.3-0ubuntu1~14.04.1~juju1 amd64 Juju is devops distilled - client
ii juju-local 1.23.3-0ubuntu1~14.04.1~juju1 all dependency package for the Juju local provider
ii juju-mongodb 2.4.9-0ubuntu3 amd64 MongoDB object/document-oriented database for Juju
ii juju-quickstart 2.1.1+bzr133+ppa36~ubuntu14.04.1 all Easy configuration of Juju environments
ii jujubundlelib 0.1.8-1 all A Python library for working with Juju bundles.
ii python-jujuclient 0.50.1-2 amd64 Python API client for juju-core

vagrant@vagrant-ubuntu-trusty-64:~$ pip list | grep juju
juju-quickstart (2.1.1)
jujubundlelib (0.1.8)
jujuclient (0.50.1)

Fwiw, I have *only* seen this with canonistack, and it takes 3+ hours before the units in my deployment go "started". I have no problem deploying the same bundle to Azure or locally, though deployments to those substrates complete in < 1 hour. I'm not sure if this means my problem is related to the openstack provider or if it's due to the length of time for the deployment.

tl;dr; canonistack sucks for real work

IIRC canonistack is *way* overcommitted and the fact that the deployment takes 3-4 times longer on canonistack compared with other clouds just reinforces that. It may be that due to severe I/O or CPU contention the agent pingers simply cannot get their message through to the state server before timing out, hence the "agent is lost" message. Also, it may be that the relations are still being started but the deployment is taking a long time to do so. The agent state going to started doesn't necessarily mean that everything is finished being wired up yet.

I am not convinced this is a canonistack issue. I have encountered similar problems, missing relations, on multiple occasions on local .It was also reported by a customer two weeks ago.

@Ian,

Thanks for the comments.
Agreed Canonistack is over-committed, but given Amir's comment 5 that we are seeing this outside of Canonistack how can we make the state server more reliable in handling this situation, and/or make it more visible to the Juju Admin, and/or possible action(s) to try and resolve?

-thanks,
Antonio

@Antonio

Agree 100% with need to handle this situation better. My motivation in the comment was to provide justification for moving this bug off 1.24. Solving the problem is likely to involve non-trivial engineering effort so would implementation during a development iteration, rather than during pre release stabilisation.

If we can reproduce this issue, I'd also like to gather information of the various deployment scenarios and also log files so we can diagnose the root cause etc.

@ ian,
I just had another customer reporting a similar issue. Sadly this time the issue was reported by a customer interested in signing CPP. Forwarding the email to you.

I was able to reproduce an issue similar to what has been reported. I hope attached log files help

yet again we had to manually add relations to avoid a potentially embarrassing moment during a big data demo by Sam.