juju-deployer

deployer doesn't always honor global deployment timeout

Bug #1241721 reported by Paul Czarkowski on 2013-10-18

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
	juju-deployer	Triaged	Low	Unassigned

Bug Description

deploying the following never finishes.

https://github.com/paulczar/charm-championship/blob/master/monitoringstack.yaml

```
juju-deployer -v -W -c monitoringstack.yaml monitoringstack
2013-10-18 10:59:03 Using runtime GoEnvironment
2013-10-18 10:59:03 Starting deployment of monitoringstack
2013-10-18 10:59:07 Connected to environment
2013-10-18 10:59:07 Getting charms...
2013-10-18 10:59:07 Cache dir /home/pczarkowski/.juju/.deployer-store-cache/cs_~paulcz_precise_sensu-agent-1
2013-10-18 10:59:07 Cache dir /home/pczarkowski/.juju/.deployer-store-cache/cs_~paulcz_precise_kibana-7
2013-10-18 10:59:07 Cache dir /home/pczarkowski/.juju/.deployer-store-cache/cs_~paulcz_precise_logstash-indexer-8
2013-10-18 10:59:07 Cache dir /home/pczarkowski/.juju/.deployer-store-cache/cs_~paulcz_precise_graphite-1
2013-10-18 10:59:07 Cache dir /home/pczarkowski/.juju/.deployer-store-cache/cs_precise_rabbitmq-server-14
2013-10-18 10:59:07 Cache dir /home/pczarkowski/.juju/.deployer-store-cache/cs_~paulcz_precise_elasticsearch-4
2013-10-18 10:59:07 Cache dir /home/pczarkowski/.juju/.deployer-store-cache/cs_precise_juju-gui-78
2013-10-18 10:59:07 Retrieving store charm cs:precise/juju-gui-78
2013-10-18 10:59:13 Cache dir /home/pczarkowski/.juju/.deployer-store-cache/cs_~paulcz_precise_logstash-agent-5
2013-10-18 10:59:13 Retrieving store charm cs:~paulcz/precise/logstash-agent-5
2013-10-18 10:59:13 Cache dir /home/pczarkowski/.juju/.deployer-store-cache/cs_~paulcz_precise_sensu-server-1
2013-10-18 10:59:13 Retrieving store charm cs:~paulcz/precise/sensu-server-1
2013-10-18 10:59:14 Resolving configuration
2013-10-18 10:59:14 Deploying services...
2013-10-18 10:59:15 Deploying service sensu-agent using cs:~paulcz/precise/sensu-agent-1
2013-10-18 10:59:21 Deploying service kibana using cs:~paulcz/precise/kibana-7
2013-10-18 10:59:32 Deploying service logstash-indexer using cs:~paulcz/precise/logstash-indexer-8
2013-10-18 10:59:43 Deploying service graphite using cs:~paulcz/precise/graphite-1
2013-10-18 10:59:55 Deploying service rabbit using cs:precise/rabbitmq-server-14
2013-10-18 11:00:07 Deploying service elasticsearch using cs:~paulcz/precise/elasticsearch-4
2013-10-18 11:00:18 Deploying service juju-gui using cs:precise/juju-gui-78
2013-10-18 11:00:53 Deploying service logstash-agent using cs:~paulcz/precise/logstash-agent-5
2013-10-18 11:00:59 Deploying service sensu-server using cs:~paulcz/precise/sensu-server-1
2013-10-18 11:03:30 Delta machine: 2 change:started
2013-10-18 11:03:31 Delta unit: logstash-indexer/0 change:pending
2013-10-18 11:04:15 Delta machine: 1 change:started
2013-10-18 11:04:20 Delta unit: kibana/0 change:pending
2013-10-18 11:04:21 Delta unit: logstash-indexer/0 change:started
2013-10-18 11:04:31 Delta unit: kibana/0 change:pending
2013-10-18 11:04:31 Delta unit: kibana/0 change:started
2013-10-18 11:04:51 Delta machine: 3 change:started
2013-10-18 11:04:51 Delta unit: graphite/0 change:pending
2013-10-18 11:04:56 Delta unit: graphite/0 change:pending
2013-10-18 11:05:51 Delta machine: 5 change:started
2013-10-18 11:05:51 Delta unit: elasticsearch/0 change:pending
2013-10-18 11:06:31 Delta machine: 6 change:started
2013-10-18 11:06:31 Delta unit: juju-gui/0 change:pending
2013-10-18 11:06:36 Delta unit: juju-gui/0 change:pending
2013-10-18 11:07:06 Delta unit: elasticsearch/0 change:installed
2013-10-18 11:07:11 Delta unit: elasticsearch/0 change:installed
2013-10-18 11:07:16 Delta unit: elasticsearch/0 change:started
2013-10-18 11:07:31 Delta unit: juju-gui/0 change:installed
2013-10-18 11:07:36 Delta unit: juju-gui/0 change:installed
2013-10-18 11:07:36 Delta unit: juju-gui/0 change:started
2013-10-18 11:08:11 Delta unit: graphite/0 change:installed
2013-10-18 11:08:11 Delta unit: graphite/0 change:installed
2013-10-18 11:08:16 Delta unit: graphite/0 change:started
2013-10-18 11:09:16 Delta machine: 7 change:started
2013-10-18 11:09:21 Delta unit: sensu-server/0 change:pending
2013-10-18 11:13:36 Delta unit: sensu-server/0 change:pending
2013-10-18 11:13:36 Delta unit: sensu-server/0 change:started
```

and then it just sits there.... current time writing this is ` date
Fri Oct 18 11:40:34 CDT 2013` and the deploy hasn't progressed since 11:13.

Revision history for this message

Paul Czarkowski (paulcz) wrote on 2013-10-18:

juju debug-log output Edit (1.1 MiB, text/plain)

Revision history for this message

Paul Czarkowski (paulcz) wrote on 2013-10-18:

Download full text (4.2 KiB)

juju status
environment: amazon
machines:
  "0":
    agent-state: started
    agent-version: 1.16.0
    dns-name: ec2-54-224-88-249.compute-1.amazonaws.com
    instance-id: i-67196f1f
    instance-state: running
    series: precise
    hardware: arch=amd64 cpu-cores=1 cpu-power=100 mem=1740M root-disk=8192M
  "1":
    agent-state: started
    agent-version: 1.16.0
    dns-name: ec2-54-205-165-46.compute-1.amazonaws.com
    instance-id: i-d2c521aa
    instance-state: running
    series: precise
    hardware: arch=amd64 cpu-cores=1 cpu-power=100 mem=1740M root-disk=8192M
  "2":
    agent-state: started
    agent-version: 1.16.0
    dns-name: ec2-54-224-84-112.compute-1.amazonaws.com
    instance-id: i-2d8d4657
    instance-state: running
    series: precise
    hardware: arch=amd64 cpu-cores=1 cpu-power=100 mem=1740M root-disk=8192M
  "3":
    agent-state: started
    agent-version: 1.16.0
    dns-name: ec2-50-17-68-0.compute-1.amazonaws.com
    instance-id: i-723d9a15
    instance-state: running
    series: precise
    hardware: arch=amd64 cpu-cores=1 cpu-power=100 mem=1740M root-disk=8192M
  "4":
    agent-state-info: '(error: invalid URL "http://cloud-images.ubuntu.com/releases/streams/v1/index.sjson"
      not found)'
    instance-id: pending
    series: precise
  "5":
    agent-state: started
    agent-version: 1.16.0
    dns-name: ec2-54-205-194-197.compute-1.amazonaws.com
    instance-id: i-4ec52136
    instance-state: running
    series: precise
    hardware: arch=amd64 cpu-cores=1 cpu-power=100 mem=1740M root-disk=8192M
  "6":
    agent-state: started
    agent-version: 1.16.0
    dns-name: ec2-54-234-215-130.compute-1.amazonaws.com
    instance-id: i-ca2591ac
    instance-state: running
    series: precise
    hardware: arch=amd64 cpu-cores=1 cpu-power=100 mem=1740M root-disk=8192M
  "7":
    agent-state: started
    agent-version: 1.16.0
    dns-name: ec2-54-205-113-224.compute-1.amazonaws.com
    instance-id: i-0e3e9969
    instance-state: running
    series: precise
    hardware: arch=amd64 cpu-cores=1 cpu-power=100 mem=1740M root-disk=8192M
services:
  elasticsearch:
    charm: cs:~paulcz/precise/elasticsearch-4
    exposed: false
    units:
      elasticsearch/0:
        agent-state: started
        agent-version: 1.16.0
        machine: "5"
        open-ports:
        - 9200/tcp
        - 9300/tcp
        public-address: ec2-54-205-194-197.compute-1.amazonaws.com
  graphite:
    charm: cs:~paulcz/precise/graphite-1
    exposed: false
    units:
      graphite/0:
        agent-state: started
        agent-version: 1.16.0
        machine: "3"
        open-ports:
        - 80/tcp
        public-address: ec2-50-17-68-0.compute-1.amazonaws.com
  juju-gui:
    charm: cs:precise/juju-gui-78
    exposed: false
    units:
      juju-gui/0:
        agent-state: started
        agent-version: 1.16.0
        machine: "6"
        open-ports:
        - 80/tcp
        - 443/tcp
        public-address: ec2-54-234-215-130.compute-1.amazonaws.com
  kibana:
    charm: cs:~paulcz/precise/kibana-7
    exposed: false
    units:
      kibana/0:
        agent-state: started
        agent-version: 1.16.0
...

juju status                                                
environment: amazon
machines:
  "0":
    agent-state: started
    agent-version: 1.16.0
    dns-name: ec2-54-224-88-249.compute-1.amazonaws.com
    instance-id: i-67196f1f
    instance-state: running
    series: precise
    hardware: arch=amd64 cpu-cores=1 cpu-power=100 mem=1740M root-disk=8192M
  "1":
    agent-state: started
    agent-version: 1.16.0
    dns-name: ec2-54-205-165-46.compute-1.amazonaws.com
    instance-id: i-d2c521aa
    instance-state: running
    series: precise
    hardware: arch=amd64 cpu-cores=1 cpu-power=100 mem=1740M root-disk=8192M
  "2":
    agent-state: started
    agent-version: 1.16.0
    dns-name: ec2-54-224-84-112.compute-1.amazonaws.com
    instance-id: i-2d8d4657
    instance-state: running
    series: precise
    hardware: arch=amd64 cpu-cores=1 cpu-power=100 mem=1740M root-disk=8192M
  "3":
    agent-state: started
    agent-version: 1.16.0
    dns-name: ec2-50-17-68-0.compute-1.amazonaws.com
    instance-id: i-723d9a15
    instance-state: running
    series: precise
    hardware: arch=amd64 cpu-cores=1 cpu-power=100 mem=1740M root-disk=8192M
  "4":
    agent-state-info: '(error: invalid URL "http://cloud-images.ubuntu.com/releases/streams/v1/index.sjson"
      not found)'
    instance-id: pending
    series: precise
  "5":
    agent-state: started
    agent-version: 1.16.0
    dns-name: ec2-54-205-194-197.compute-1.amazonaws.com
    instance-id: i-4ec52136
    instance-state: running
    series: precise
    hardware: arch=amd64 cpu-cores=1 cpu-power=100 mem=1740M root-disk=8192M
  "6":
    agent-state: started
    agent-version: 1.16.0
    dns-name: ec2-54-234-215-130.compute-1.amazonaws.com
    instance-id: i-ca2591ac
    instance-state: running
    series: precise
    hardware: arch=amd64 cpu-cores=1 cpu-power=100 mem=1740M root-disk=8192M
  "7":
    agent-state: started
    agent-version: 1.16.0
    dns-name: ec2-54-205-113-224.compute-1.amazonaws.com
    instance-id: i-0e3e9969
    instance-state: running
    series: precise
    hardware: arch=amd64 cpu-cores=1 cpu-power=100 mem=1740M root-disk=8192M
services:
  elasticsearch:
    charm: cs:~paulcz/precise/elasticsearch-4
    exposed: false
    units:
      elasticsearch/0:
        agent-state: started
        agent-version: 1.16.0
        machine: "5"
        open-ports:
        - 9200/tcp
        - 9300/tcp
        public-address: ec2-54-205-194-197.compute-1.amazonaws.com
  graphite:
    charm: cs:~paulcz/precise/graphite-1
    exposed: false
    units:
      graphite/0:
        agent-state: started
        agent-version: 1.16.0
        machine: "3"
        open-ports:
        - 80/tcp
        public-address: ec2-50-17-68-0.compute-1.amazonaws.com
  juju-gui:
    charm: cs:precise/juju-gui-78
    exposed: false
    units:
      juju-gui/0:
        agent-state: started
        agent-version: 1.16.0
        machine: "6"
        open-ports:
        - 80/tcp
        - 443/tcp
        public-address: ec2-54-234-215-130.compute-1.amazonaws.com
  kibana:
    charm: cs:~paulcz/precise/kibana-7
    exposed: false
    units:
      kibana/0:
        agent-state: started
        agent-version: 1.16.0
        machine: "1"
        open-ports:
        - 80/tcp
        public-address: ec2-54-205-165-46.compute-1.amazonaws.com
  logstash-agent:
    charm: cs:~paulcz/precise/logstash-agent-5
    exposed: false
  logstash-indexer:
    charm: cs:~paulcz/precise/logstash-indexer-8
    exposed: false
    units:
      logstash-indexer/0:
        agent-state: started
        agent-version: 1.16.0
        machine: "2"
        public-address: ec2-54-224-84-112.compute-1.amazonaws.com
  rabbit:
    charm: cs:precise/rabbitmq-server-14
    exposed: false
    relations:
      cluster:
      - rabbit
    units:
      rabbit/0:
        agent-state: pending
        machine: "4"
  sensu-agent:
    charm: cs:~paulcz/precise/sensu-agent-1
    exposed: false
  sensu-server:
    charm: cs:~paulcz/precise/sensu-server-1
    exposed: false
    units:
      sensu-server/0:
        agent-state: started
        agent-version: 1.16.0
        machine: "7"
        open-ports:
        - 4567/tcp
        - 8080/tcp
        public-address: ec2-54-205-113-224.compute-1.amazonaws.com

Revision history for this message

Paul Czarkowski (paulcz) wrote on 2013-10-18:

Download full text (3.7 KiB)

juju-deployer finally finishes ... but all services still not brought up. From where it left off above ....

2013-10-18 11:09:21 Delta unit: sensu-server/0 change:pending
2013-10-18 11:13:36 Delta unit: sensu-server/0 change:pending
2013-10-18 11:13:36 Delta unit: sensu-server/0 change:started

2013-10-18 11:56:24 Adding units...
2013-10-18 11:56:25 Config specifies num units for subordinate: sensu-agent
2013-10-18 11:56:25 Service 'kibana' does not need any more units added.
2013-10-18 11:56:25 Service 'logstash-indexer' does not need any more units added.
2013-10-18 11:56:25 Service 'graphite' does not need any more units added.
2013-10-18 11:56:25 Service 'rabbit' does not need any more units added.
2013-10-18 11:56:25 Service 'elasticsearch' does not need any more units added.
2013-10-18 11:56:25 Service 'juju-gui' does not need any more units added.
2013-10-18 11:56:25 Config specifies num units for subordinate: logstash-agent
2013-10-18 11:56:25 Service 'sensu-server' does not need any more units added.
2013-10-18 11:56:25 Adding relations...
2013-10-18 11:56:25 Adding relation elasticsearch:cluster <-> logstash-indexer:cluster
2013-10-18 11:56:25 Waiting 5s before next relation
2013-10-18 11:56:30 Adding relation logstash-agent:juju-info <-> rabbit:juju-info
2013-10-18 11:56:30 Waiting 5s before next relation
2013-10-18 11:56:35 Adding relation logstash-agent:juju-info <-> juju-gui:juju-info
2013-10-18 11:56:36 Waiting 5s before next relation
2013-10-18 11:56:41 Adding relation logstash-agent:juju-info <-> sensu-server:juju-info
2013-10-18 11:56:41 Waiting 5s before next relation
2013-10-18 11:56:46 Adding relation logstash-agent:juju-info <-> graphite:juju-info
2013-10-18 11:56:46 Waiting 5s before next relation
2013-10-18 11:56:51 Adding relation logstash-agent:juju-info <-> kibana:juju-info
2013-10-18 11:56:51 Waiting 5s before next relation
2013-10-18 11:56:56 Adding relation logstash-agent:juju-info <-> elasticsearch:juju-info
2013-10-18 11:56:56 Waiting 5s before next relation
2013-10-18 11:57:01 Adding relation sensu-agent:juju-info <-> logstash-indexer:juju-info
2013-10-18 11:57:01 Waiting 5s before next relation
2013-10-18 11:57:06 Adding relation sensu-agent:juju-info <-> elasticsearch:juju-info
2013-10-18 11:57:06 Waiting 5s before next relation
2013-10-18 11:57:11 Adding relation sensu-agent:juju-info <-> juju-gui:juju-info
2013-10-18 11:57:11 Waiting 5s before next relation
2013-10-18 11:57:16 Adding relation sensu-server:amqp <-> rabbit:amqp
2013-10-18 11:57:16 Waiting 5s before next relation
2013-10-18 11:57:21 Adding relation graphite:amqp <-> rabbit:amqp
2013-10-18 11:57:21 Waiting 5s before next relation
2013-10-18 11:57:26 Adding relation logstash-indexer:amqp <-> rabbit:amqp
2013-10-18 11:57:26 Waiting 5s before next relation
2013-10-18 11:57:31 Adding relation logstash-agent:amqp <-> rabbit:amqp
2013-10-18 11:57:31 Waiting 5s before next relation
2013-10-18 11:57:36 Adding relation sensu-agent:juju-info <-> rabbit:juju-info
2013-10-18 11:57:36 Waiting 5s before next relation
2013-10-18 11:57:41 Adding relation sensu-agent:amqp <-> rabbit:amqp
2013-10-18 11:57:41 Waitin...

juju-deployer finally finishes ...  but all services still not brought up.   From where it left off above ....

2013-10-18 11:09:21  Delta unit: sensu-server/0 change:pending
2013-10-18 11:13:36  Delta unit: sensu-server/0 change:pending
2013-10-18 11:13:36  Delta unit: sensu-server/0 change:started

2013-10-18 11:56:24 Adding units...
2013-10-18 11:56:25 Config specifies num units for subordinate: sensu-agent
2013-10-18 11:56:25  Service 'kibana' does not need any more units added.
2013-10-18 11:56:25  Service 'logstash-indexer' does not need any more units added.
2013-10-18 11:56:25  Service 'graphite' does not need any more units added.
2013-10-18 11:56:25  Service 'rabbit' does not need any more units added.
2013-10-18 11:56:25  Service 'elasticsearch' does not need any more units added.
2013-10-18 11:56:25  Service 'juju-gui' does not need any more units added.
2013-10-18 11:56:25 Config specifies num units for subordinate: logstash-agent
2013-10-18 11:56:25  Service 'sensu-server' does not need any more units added.
2013-10-18 11:56:25 Adding relations...
2013-10-18 11:56:25  Adding relation elasticsearch:cluster <-> logstash-indexer:cluster
2013-10-18 11:56:25  Waiting 5s before next relation
2013-10-18 11:56:30  Adding relation logstash-agent:juju-info <-> rabbit:juju-info
2013-10-18 11:56:30  Waiting 5s before next relation
2013-10-18 11:56:35  Adding relation logstash-agent:juju-info <-> juju-gui:juju-info
2013-10-18 11:56:36  Waiting 5s before next relation
2013-10-18 11:56:41  Adding relation logstash-agent:juju-info <-> sensu-server:juju-info
2013-10-18 11:56:41  Waiting 5s before next relation
2013-10-18 11:56:46  Adding relation logstash-agent:juju-info <-> graphite:juju-info
2013-10-18 11:56:46  Waiting 5s before next relation
2013-10-18 11:56:51  Adding relation logstash-agent:juju-info <-> kibana:juju-info
2013-10-18 11:56:51  Waiting 5s before next relation
2013-10-18 11:56:56  Adding relation logstash-agent:juju-info <-> elasticsearch:juju-info
2013-10-18 11:56:56  Waiting 5s before next relation
2013-10-18 11:57:01  Adding relation sensu-agent:juju-info <-> logstash-indexer:juju-info
2013-10-18 11:57:01  Waiting 5s before next relation
2013-10-18 11:57:06  Adding relation sensu-agent:juju-info <-> elasticsearch:juju-info
2013-10-18 11:57:06  Waiting 5s before next relation
2013-10-18 11:57:11  Adding relation sensu-agent:juju-info <-> juju-gui:juju-info
2013-10-18 11:57:11  Waiting 5s before next relation
2013-10-18 11:57:16  Adding relation sensu-server:amqp <-> rabbit:amqp
2013-10-18 11:57:16  Waiting 5s before next relation
2013-10-18 11:57:21  Adding relation graphite:amqp <-> rabbit:amqp
2013-10-18 11:57:21  Waiting 5s before next relation
2013-10-18 11:57:26  Adding relation logstash-indexer:amqp <-> rabbit:amqp
2013-10-18 11:57:26  Waiting 5s before next relation
2013-10-18 11:57:31  Adding relation logstash-agent:amqp <-> rabbit:amqp
2013-10-18 11:57:31  Waiting 5s before next relation
2013-10-18 11:57:36  Adding relation sensu-agent:juju-info <-> rabbit:juju-info
2013-10-18 11:57:36  Waiting 5s before next relation
2013-10-18 11:57:41  Adding relation sensu-agent:amqp <-> rabbit:amqp
2013-10-18 11:57:41  Waiting 5s before next relation
2013-10-18 11:57:46  Adding relation elasticsearch:rest <-> kibana:rest
2013-10-18 11:57:46  Waiting 5s before next relation
2013-10-18 11:57:51  Adding relation sensu-agent:juju-info <-> graphite:juju-info
2013-10-18 11:57:52  Waiting 5s before next relation
2013-10-18 11:57:57  Adding relation sensu-agent:juju-info <-> kibana:juju-info
2013-10-18 11:57:57  Waiting 5s before next relation
2013-10-18 11:58:02 Waiting for units to be started
2013-10-18 11:58:02 Reached deployment timeout.. exiting
2013-10-18 11:58:02 Deployment stopped. run time: 3538.40

Revision history for this message

Martin Packman (gz) wrote on 2013-10-18:

"4":
     agent-state-info: '(error: invalid URL "http://cloud-images.ubuntu.com/releases/streams/v1/index.sjson"
       not found)'
     instance-id: pending
     series: precise

That url resolves for me, but it's probably spurious. I suspect machine-4 just failed to provision due to EC2 instability, deleting it and retrying the rabbit deploy may be enough. Unfortunately the debug-log doesn't contain anything incriminating about the actual start instance failure.

Revision history for this message

Kapil Thangavelu (hazmat) wrote on 2014-02-22:

afaics this isn't a deployer issue but a juju-core one. closing.

Revision history for this message

Kapil Thangavelu (hazmat) wrote on 2014-02-22:

well technically it could be a deployment timeout issue.

summary:	- juju-deployer never finishes + deployer doesn't always honor global deployment timeout
Changed in juju-deployer:
status:	New → Triaged
importance:	Undecided → Low

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Bug attachments

juju debug-log output Edit

Add attachment

Remote bug watches

Bug watches keep track of this bug in other bug trackers.