destroying Azure environment took down other environment

Bug #1398820 reported by Brad Crittenden
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
juju-core
Fix Released
High
Ian Booth
1.21
Fix Released
High
Ian Booth

Bug Description

Two environments were active in Azure using the same credentials but different storage-account-name settings. (A sanitized version of environments.yaml is attached.) The two environments were named azure-ci and azure-ci-1. When destroying the former both environments were taken down.

Similar to the report in bug 1335885 I got a warning that the security group could not be deleted as it referenced the storage account associated with the azure-ci-1 environment.

The two environments had six and five machines respectively. After the destroy-environment was run all 11 cloud services and virtual machines were gone. The environments/azure-ci.jenv file was deleted but azure-ci-1.jenv remained. 'juju status -e azure-ci-1' hung and did not report any results as its state server was gone.

The full JUJU_HOME environment as it existed *before* the destroy-environment was executed can be found at chinstrap:~bac/juju-gui-ci.

One thing to note is the first environment used the default True value for availability-sets-enabled while the second one used availability-sets-enabled: false. That was done so we could co-locate Apache on one of the Jenkins servers.

I have tried unsuccessfully to reproduce the issue using minimal environments. One attempt involved bootstrapping two Azure environments with no other machines and tearing down the first. Another involved using juju-quickstart to bring up two environments, each with a second machine deploying the juju-gui. A final attempt repeated the last but had availability-sets-enabled setup as in the original failure. All three experiments failed to reproduce the issue.

% juju --version
1.20.12-utopic-amd64

WORK AROUND:

Avoid dash-numbers in envs names because they can appear to be machines and groups in another env. azure-ci-a or azure-ci-01 are safe.

Revision history for this message
Brad Crittenden (bac) wrote :
Revision history for this message
Curtis Hovey (sinzui) wrote :

I suspect the envs names are the issue here. There were problems in openstack in the past were machines from another env were taken down because one name is a subset of the other. Juju appends "-1", "-2" to the machines and groups is creates, and the other env has "-1"; it looks like a subset.

In my own experience, I avoid -number. I use letters or some other characters to separate like a 0, -01.

tags: added: azure-provider
tags: added: destroy-environment
Changed in juju-core:
status: New → Triaged
milestone: none → 1.22
importance: Undecided → High
description: updated
Brad Crittenden (bac)
description: updated
description: updated
Revision history for this message
Brad Crittenden (bac) wrote :

I have reproduced the problem by creating environments named azure and azure-1. Destroying azure deletes azure-1 also. Note if the experiment uses environments named azure-1 and azure-2 then the problem is not reproduced.

After the destroy-environment, 'juju status -e azure-1' does not terminate and all cloud services and virtual machines are no longer shown in the Azure dashboard.

Here is what I saw:

bac@chuleta:~/bzr/azure-test$ juju switch
azure-1
bac@chuleta:~/bzr/azure-test$ juju status
environment: azure-1
machines:
  "0":
    agent-state: started
    agent-version: 1.20.13
    dns-name: juju-azure-1-vahumlx6ii.cloudapp.net
    instance-id: juju-azure-1-vahumlx6ii-jujuxu2b0ao0ahdzctlpv2f48t94z8pf6shb18vbf85s79cp33
    instance-state: ReadyRole
    series: trusty
    hardware: arch=amd64 cpu-cores=1 mem=1792M root-disk=130048M
    state-server-member-status: has-vote
services: {}
bac@chuleta:~/bzr/azure-test$ juju status -e azure
environment: azure
machines:
  "0":
    agent-state: started
    agent-version: 1.20.13
    dns-name: juju-azure-t8bv2yyz60.cloudapp.net
    instance-id: juju-azure-t8bv2yyz60-jujuk7433f0hnbqzzmavzogs1tbwaedtjik3ldgbcoon42mcjp
    instance-state: ReadyRole
    series: trusty
    hardware: arch=amd64 cpu-cores=1 mem=1792M root-disk=130048M
    state-server-member-status: has-vote
services: {}
bac@chuleta:~/bzr/azure-test$ ls environments
azure-1.jenv azure-2.jenv azure.jenv
bac@chuleta:~/bzr/azure-test$ juju destroy-environment azure
WARNING! this command will destroy the "azure" environment (type: azure)
This includes all machines, services, data and other resources.

Continue [y/N]? y

Ian Booth (wallyworld)
Changed in juju-core:
assignee: nobody → Ian Booth (wallyworld)
Ian Booth (wallyworld)
Changed in juju-core:
status: Triaged → In Progress
Ian Booth (wallyworld)
Changed in juju-core:
status: In Progress → Fix Committed
Curtis Hovey (sinzui)
Changed in juju-core:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.