Cannot terminate/remove broken state server after ensure-availability

Bug #1449633 reported by Jorge Niedbalski
16
This bug affects 3 people
Affects Status Importance Assigned to Milestone
Canonical Juju
Incomplete
Undecided
Unassigned
juju-core
Won't Fix
Medium
Unassigned

Bug Description

[Environment]

Trusty 14.04
juju-core 1.20.11-0ubuntu1~14.04.1~juju1 amd64 Juju is devops distilled - client
ii maas-cli 1.5.4+bzr2294-0ubuntu1.1 all MAAS command line API tool
ii maas-cluster-controller 1.5.4+bzr2294-0ubuntu1.1 all MAAS server cluster controller
ii maas-common 1.5.4+bzr2294-0ubuntu1.1 all MAAS server common files
ii maas-dhcp 1.5.4+bzr2294-0ubuntu1.1 all MAAS DHCP server
ii maas-dns 1.5.4+bzr2294-0ubuntu1.1 all MAAS DNS server
ii maas-region-controller 1.5.4+bzr2294-0ubuntu1.1 all MAAS server complete region controller
ii maas-region-controller-min 1.5.4+bzr2294-0ubuntu1.1 all MAAS Server minimum region controller

[Description]

$ juju ensure-availability --constraints 'tags=bootstrap" -n 5

One of the node fails to bootstrap because there are no matching tags are availables:

"85":
agent-state-info: 'cannot run instances: gomaasapi: got error back from server:
409 CONFLICT (No matching node is available.)'
instance-id: pending
series: trusty
state-server-member-status: no-vote

This is a totally valid and possible scenario on which one of the requested state servers fails.

At this point 3 of 5 nodes are already available as state servers, but if you try to remove/terminate the machine 85, even by forcing is not possible to remove this machine from the

$ juju terminate-machine 85 --force
ERROR no machines were destroyed: machine 85 is required by the environment

Curtis Hovey (sinzui)
tags: added: destroy-machine ensure-availability
Changed in juju-core:
status: New → Triaged
importance: Undecided → Medium
tags: added: sts
removed: cts
tags: added: sts-needs-review
tags: removed: sts-needs-review
Curtis Hovey (sinzui)
tags: added: jujuqa
Changed in juju-core:
status: Triaged → Won't Fix
Revision history for this message
Anastasia (anastasia-macmood) wrote :

@Jorge,
You have updated the bug :D
Does it mean that you have a reproducible scenario for Juju 2.x? Could you please share it?
It might be worthwhile re-wording the bug with Juju 2.x terminology for clarity \o/

Changed in juju:
status: New → Incomplete
tags: added: eda
Revision history for this message
David (dberardozzi) wrote :

Same problem here with Juju 2.0.2-yakkety-amd64

First controller created using:
juju bootstrap ovh juju-controller --metadata-source ~/simplestreams --bootstrap-series yakkety

On an Openstack cloud provider (OVH)

juju status
Machine State DNS Inst id Series AZ
0 started xx.xx.xx.xxx e3ce647b-0458-436f-a0cc-0df6b037972c yakkety nova
1 down pending yakkety
2 down pending yakkety
3 down pending yakkety
4 down pending yakkety

The problem raised after using
juju enable-ha
maintaining machines: 0
adding machines: 1, 2

No error shown (made 2 times without option, that is why I have machines 3 and 4).

I am completely stuck, no way to remove old down machines:
juju remove-machine 4 --force
ERROR no machines were destroyed: machine is required by the model

Is anything more needed for a reproductible scenario?

Revision history for this message
Anastasia (anastasia-macmood) wrote :

We track Juju 2.x issues in "juju" project and what you describe looks like a duplicate of bug # 1658033. I'll mark it as such.

Thank you for update, David!

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Related blueprints

Remote bug watches

Bug watches keep track of this bug in other bug trackers.