Juju HA - Unable to remove controller machines in 'down' state

I was facing same issue on Openstack cloud (OVH public cloud), and it seems that the cause was that I added a new Neutron network after initial bootstrap. I assumes this causes the controller to be confused with the network it should use for further instance provisioning.

After Bootstraping with --config network=<networkID> no more issue enabling HA.

@David,

Thank you for the update. I'll mark this bug as Invalid since Juju works as expected with --config network=...

So while the original submitter was able to work around this, it is still true that if machines are added as controllers and then fail to provision, they get stuck because we refuse to let you remove controller machines, but they aren't up for you to take them out of the HA group.
Probably peer-grouper needs a way to flag these as no-longer-important so that we can remove them. (see bug #1648799 or bug #1449633)

Is reproduction still needed? I can get this in an scenario where I
3 out of 3 machines allocated in MAAS.
Then enable-ha. It then asks maas for two extra nodes, which it will never get. I then `juju enable-ha --to 1,2`. I get the mention that node 3 and 4 are demoted, but they never get removed.

A patch along the way.

https://github.com/juju/juju/pull/8625

earlier patch that implemented initial support for "juju remove-machine"
https://github.com/juju/juju/pull/8557

A related patch
https://github.com/juju/juju/pull/8627

In 2.4 beta 2 (and probably beta1), I just tested:
 juju bootstrap lxd
 juju enable-ha
 lxc stop juju-XXXXX-1
 juju status
 juju enable-ha # does nothing
 juju remove-machine 1 # properly removes machine 1 from being listed as a controller
 juju enable-ha # creates a 3rd machine

Just like all machines, if the machine agent isn't running, then the machine will be 'stuck' in a 'down' state:

$ juju status
Model Controller Cloud/Region Version SLA
controller lxd lxd 2.4-beta2.1 unsupported

App Version Status Scale Charm Store Rev OS Notes

Unit Workload Agent Machine Public address Ports Message

Machine State DNS Inst id Series AZ Message
0 started 10.16.17.214 juju-10a670-0 xenial Running
1 down 10.16.17.89 juju-10a670-1 xenial Stopped
2 started 10.16.17.189 juju-10a670-2 xenial Running
3 started 10.16.17.42 juju-10a670-3 xenial Running

But it is removed as a controller:
$ juju show-controller
lxd:
...
  controller-machines:
    "0":
      instance-id: juju-10a670-0
      ha-status: ha-enabled
    "2":
      instance-id: juju-10a670-2
      ha-status: ha-enabled
    "3":
      instance-id: juju-10a670-3
      ha-status: ha-enabled

To actually purge the machine (like you need to do for normal application machines), you can use "juju remove-machine --force":
$ juju status
Model Controller Cloud/Region Version SLA
controller lxd lxd 2.4-beta2.1 unsupported

App Version Status Scale Charm Store Rev OS Notes

Unit Workload Agent Machine Public address Ports Message

Machine State DNS Inst id Series AZ Message
0 started 10.16.17.214 juju-10a670-0 xenial Running
2 started 10.16.17.189 juju-10a670-2 xenial Running
3 started 10.16.17.42 juju-10a670-3 xenial Running

(note that remove-machine --force triggers a 'cleanup' action, so it does not immediately remove the machine. So an immediate 'status' will not show it as gone, but after a few seconds it does go away.)