juju-core

juju deployment of systems that does not meet maas base constraints

Bug #1411042 reported by Larry Michel
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
juju-core
Triaged
Critical
Ian Booth
juju-core 1.21-beta5

Bug Description

When doing an OpenStack deployment, our maas base constraints of hw-staging was totally ignored. All the servers selected for the bootstrap node and the OpenStack deployment did not meet the constraints.

Some console output showing maas base tag:
============================================================================
2015-01-14 22:55:14,199651: Base MAAS tag: hw-staging
2015-01-14 22:55:14,200009: converted slave label ci-oil-slave1 to maas user oil-slave-1
2015-01-14 22:55:15,111365: found 34 nodes with tag=hw-staging
2015-01-14 22:55:15,111848: restricting to 0 nodes owned by oil-slave-1
2015-01-14 22:55:15,112040: no nodes owned by oil-slave-1 in tag hw-staging
Exiting with: 0
+ rm -rf /var/lib/jenkins/.juju/environments
+ mv run_config artifacts/run_config.parameter
+ oil-ci generate run_config --out run_config --environment ci-oil-slave1
2015-01-14 22:55:15,548 [INFO] oil_ci.cli: Constructing parameters from jenkins job environment
2015-01-14 22:55:15,549 [INFO] oil_ci.cli: Generating run config with deploy parameters: {'block-storage': 'cinder-iscsi', 'networking': 'neutron-openvswitch', 'openstack-release': 'juno', 'image-storage': 'glance-ceph', 'ubuntu-release': 'trusty', 'compute': 'nova-kvm'}
2015-01-14 22:55:15,549 [INFO] oil_ci.cli: Generating run_config networking for env: ci-oil-slave1
2015-01-14 22:55:15,549 [INFO] oil_ci.cli: Allocating networks for env: ci-oil-slave1
2015-01-14 22:55:15,603 [INFO] oil_ci.cli: OIL Network Allocating 2 ranges of 32 IPs per environment
2015-01-14 22:55:15,618 [INFO] oil_ci.cli: Calculating deployment cost
2015-01-14 22:55:15,914 [INFO] oil_ci.placement.density: ceph units of 3 to machines: [1, 2, 3]
2015-01-14 22:55:15,915 [INFO] oil_ci.placement.density: Base services:
2015-01-14 22:55:15,915 [INFO] oil_ci.placement.density: 3 units of nova-compute to: ['ceph=0', 'ceph=1', 'ceph=2']
2015-01-14 22:55:15,915 [INFO] oil_ci.placement.density: LXCed services:
2015-01-14 22:55:15,915 [INFO] oil_ci.placement.density: 1 units of ceilometer to: ['lxc:ceph=0']
2015-01-14 22:55:15,915 [INFO] oil_ci.placement.density: 1 units of glance to: ['lxc:ceph=1']
2015-01-14 22:55:15,916 [INFO] oil_ci.placement.density: 1 units of heat to: ['lxc:ceph=2']
2015-01-14 22:55:15,916 [INFO] oil_ci.placement.density: 1 units of keystone to: ['lxc:cinder=0']
2015-01-14 22:55:15,916 [INFO] oil_ci.placement.density: 1 units of mongodb to: ['lxc:neutron-gateway=0']
2015-01-14 22:55:15,916 [INFO] oil_ci.placement.density: 1 units of mysql to: ['lxc:nova-cloud-controller=0']
2015-01-14 22:55:15,916 [INFO] oil_ci.placement.density: 1 units of neutron-api to: ['lxc:ceph=0']
2015-01-14 22:55:15,916 [INFO] oil_ci.placement.density: 1 units of openstack-dashboard to: ['lxc:ceph=1']
2015-01-14 22:55:15,917 [INFO] oil_ci.placement.density: 1 units of rabbitmq-server to: ['lxc:ceph=2']
2015-01-14 22:55:15,922 [INFO] oil_ci.placement.density: Deployment will require 7 machines (6 + bootstrap)
+ cp -av run_config artifacts/run_config
`run_config' -> `artifacts/run_config'
+ cp -av oil_nodes artifacts/oil_nodes
`oil_nodes' -> `artifacts/oil_nodes'
+ oil-ci do deploy --environment ci-oil-slave1 --run_config artifacts/run_config --artifacts_dir ./artifacts
2015-01-14 22:55:16,753 [INFO] oil_ci.deploy.oil_deployer: Bootstrapping new juju environment
2015-01-14 22:55:17,011 [INFO] oil_ci.juju.client: Boostrapping new environment
Bootstrapping environment "ci-oil-slave1"
Starting new instance for initial state server
Launching instance
WARNING no architecture was specified, acquiring an arbitrary node
 - /MAAS/api/1.0/nodes/node-8bdda5f8-6bb0-11e4-83f3-00163eca07b6/
Installing Juju agent on bootstrap instance
Waiting for address
Attempting to connect to mokoi-vm-02.oil:22
Attempting to connect to mokoi-vm-02.oil:22
============================================================================

For example (I'll attach entire yaml file as well):
============================================================================
environment: ci-oil-slave1
machines:
  '0':
    agent-state: started
    agent-version: 1.21-beta4
    dns-name: mokoi-vm-02.oil
    hardware: arch=amd64 cpu-cores=2 mem=12288M tags=virtual
    instance-id: /MAAS/api/1.0/nodes/node-8bdda5f8-6bb0-11e4-83f3-00163eca07b6/
    series: precise
    state-server-member-status: has-vote
  '1':
    agent-state: started
    agent-version: 1.21-beta4
    containers:
      1/lxc/0:
        agent-state: started
        agent-version: 1.21-beta4
        dns-name: 10.245.44.246
        hardware: arch=amd64
        instance-id: juju-machine-1-lxc-0
        series: trusty
      1/lxc/1:
        agent-state: started
        agent-version: 1.21-beta4
        dns-name: 10.245.44.248
        hardware: arch=amd64
        instance-id: juju-machine-1-lxc-1
        series: trusty
    dns-name: zahhak.oil
    hardware: arch=amd64 cpu-cores=4 mem=32768M tags=hardware-dell-poweredge-R210,disable_biosdevname,hw-standby,bug1398879
    instance-id: /MAAS/api/1.0/nodes/node-6491e170-ae10-11e3-9074-00163efc5068/
    series: trusty
==========================================================================

We are using juju 1.21-beta4 and maas version 1.7.1~rc1+bzr3313-0ubuntu1~trusty1

Note that this issue has not been observed with Juju 1.20.x and maas version 1.7.1~rc1+bzr3313-0ubuntu1~trusty1

Revision history for this message
Larry Michel (lmic) wrote :

juju-status.yaml file

tags: added: oil
Revision history for this message
Larry Michel (lmic) wrote :
Revision history for this message
Larry Michel (lmic) wrote :

output from 'maas root tag nodes -k hw-staging'

Revision history for this message
John George (jog) wrote :
Download full text (3.1 KiB)

I was able to confirm the reported behaviour.

1.20.14 honours '--contraints tag=...' and errors when the request can not be fulfilled.

$ juju --show-log bootstrap -e maas-env2-weekly --constraints tags=medium
2015-01-15 00:31:38 INFO juju.cmd supercommand.go:37 running juju [1.20.14-trusty-amd64 gc]
2015-01-15 00:31:43 INFO juju.environs.bootstrap bootstrap.go:48 bootstrapping environment "maas-env2-weekly"
2015-01-15 00:31:43 INFO juju.environs.tools tools.go:187 filtering tools by released version
2015-01-15 00:31:43 INFO juju.environs.tools tools.go:87 reading tools with major.minor version 1.20
2015-01-15 00:31:43 INFO juju.environs.tools tools.go:98 filtering tools by series: trusty
2015-01-15 00:31:43 INFO juju.utils http.go:59 hostname SSL verification enabled
2015-01-15 00:31:43 INFO juju.utils http.go:59 hostname SSL verification enabled
2015-01-15 00:31:43 INFO juju.utils http.go:59 hostname SSL verification enabled
2015-01-15 00:31:43 INFO juju.utils http.go:59 hostname SSL verification enabled
2015-01-15 00:31:43 INFO juju.environs.bootstrap bootstrap.go:60 newest version: 1.20.14
2015-01-15 00:31:44 INFO juju.environs.bootstrap bootstrap.go:88 picked bootstrap tools version: 1.20.14
Launching instance
2015-01-15 00:31:50 ERROR juju.provider.common bootstrap.go:136 bootstrap failed: cannot start bootstrap instance: cannot run instances: gomaasapi: got error back from server: 409 CONFLICT (No available node matches constraints: tags=medium)
Bootstrap failed, destroying environment
2015-01-15 00:31:50 INFO juju.provider.common destroy.go:15 destroying environment "maas-env2-weekly"
2015-01-15 00:31:50 ERROR juju.cmd supercommand.go:323 cannot start bootstrap instance: cannot run instances: gomaasapi: got error back from server: 409 CONFLICT (No available node matches constraints: tags=medium)

1.21-beta4 does not:
juju --show-log bootstrap -e maas-env2-weekly --constraints tags=medium
2015-01-15 00:34:35 INFO juju.cmd supercommand.go:37 running juju [1.21-beta4-trusty-amd64 gc]
2015-01-15 00:34:35 WARNING juju.environs config.go:127 Config attribute "tools-metadata-url" (https://swift.canonistack.canonical.com/v1/AUTH_526ad877f3e3464589dc1145dfeaac60/juju-dist/weekly/tools) is deprecated.
It is replaced by "agent-metadata-url" attribute.
Your configuration should be updated to set "agent-metadata-url" as follows
agent-metadata-url: https://swift.canonistack.canonical.com/v1/AUTH_526ad877f3e3464589dc1145dfeaac60/juju-dist/weekly/tools.
2015-01-15 00:34:35 INFO juju.network network.go:106 setting prefer-ipv6 to false
Bootstrapping environment "maas-env2-weekly"
2015-01-15 00:34:40 INFO juju.environs.tools tools.go:86 reading tools with major.minor version 1.21
2015-01-15 00:34:40 INFO juju.utils http.go:59 hostname SSL verification enabled
2015-01-15 00:34:40 INFO juju.utils http.go:59 hostname SSL verification enabled
2015-01-15 00:34:40 INFO juju.utils http.go:59 hostname SSL verification enabled
2015-01-15 00:34:40 INFO juju.utils http.go:59 hostname SSL verification enabled
2015-01-15 00:34:40 INFO juju.utils http.go:59 hostname SSL verification enabled
Starting new instance for initial state server
Launching instance
.
.
...

Read more...

Revision history for this message
John George (jog) wrote :

oops here is the full output from 1.21-beta4:

juju --show-log bootstrap -e maas-env2-weekly --constraints tags=medium
2015-01-15 00:34:35 INFO juju.cmd supercommand.go:37 running juju [1.21-beta4-trusty-amd64 gc]
2015-01-15 00:34:35 WARNING juju.environs config.go:127 Config attribute "tools-metadata-url" (https://swift.canonistack.canonical.com/v1/AUTH_526ad877f3e3464589dc1145dfeaac60/juju-dist/weekly/tools) is deprecated.
It is replaced by "agent-metadata-url" attribute.
Your configuration should be updated to set "agent-metadata-url" as follows
agent-metadata-url: https://swift.canonistack.canonical.com/v1/AUTH_526ad877f3e3464589dc1145dfeaac60/juju-dist/weekly/tools.
2015-01-15 00:34:35 INFO juju.network network.go:106 setting prefer-ipv6 to false
Bootstrapping environment "maas-env2-weekly"
2015-01-15 00:34:40 INFO juju.environs.tools tools.go:86 reading tools with major.minor version 1.21
2015-01-15 00:34:40 INFO juju.utils http.go:59 hostname SSL verification enabled
2015-01-15 00:34:40 INFO juju.utils http.go:59 hostname SSL verification enabled
2015-01-15 00:34:40 INFO juju.utils http.go:59 hostname SSL verification enabled
2015-01-15 00:34:40 INFO juju.utils http.go:59 hostname SSL verification enabled
2015-01-15 00:34:40 INFO juju.utils http.go:59 hostname SSL verification enabled
Starting new instance for initial state server
Launching instance
2015-01-15 00:34:41 WARNING juju.provider.maas environ.go:576 no architecture was specified, acquiring an arbitrary node
2015-01-15 00:34:46 INFO juju.provider.maas environ.go:884 could not acquire a node in zone "default", trying another zone
2015-01-15 00:34:46 WARNING juju.provider.maas environ.go:576 no architecture was specified, acquiring an arbitrary node
 - /MAAS/api/1.0/nodes/node-4ddd31ea-9790-11e4-8d20-525400985998/
2015-01-15 00:34:47 INFO juju.environs.bootstrap bootstrap.go:178 newest version: 1.21-beta5
2015-01-15 00:34:47 WARNING juju.environs.bootstrap bootstrap.go:201 failed to find 1.21-beta4 tools, will attempt to use 1.21-beta5
2015-01-15 00:34:47 INFO juju.environs.bootstrap bootstrap.go:206 picked bootstrap tools version: 1.21-beta5
Installing Juju agent on bootstrap instance
Waiting for address
Attempting to connect to maas-node-107.maas:22
Attempting to connect to maas-node-107.maas:22
Attempting to connect to 10.0.60.120:22
Warning: Permanently added '10.0.60.120' (ECDSA) to the list of known hosts.
Logging to /var/log/cloud-init-output.log on remote host
Running apt-get update
Running apt-get upgrade
Installing package: curl
Installing package: cpu-checker
Installing package: bridge-utils
Installing package: rsyslog-gnutls
Fetching tools: curl -sSfw 'tools from %{url_effective} downloaded: HTTP %{http_code}; time %{time_total}s; size %{size_download} bytes; speed %{speed_download} bytes/s ' --retry 10 -o $bin/tools.tar.gz <[https://swift.canonistack.canonical.com/v1/AUTH_526ad877f3e3464589dc1145dfeaac60/juju-dist/weekly/tools/releases/juju-1.21-beta5-trusty-amd64.tgz]>
Bootstrapping Juju machine agent
Starting Juju machine agent (jujud-machine-0)
Bootstrap complete

Changed in juju-core:
status: New → Triaged
importance: Undecided → Critical
milestone: none → 1.21-beta5
assignee: nobody → Ian Booth (wallyworld)
tags: added: bootstrap constraints maas-provider
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.