load balancers created for Kube API via the loadbalancer endpoint are not deleted

There is explicit cleanup logic in the charm ([1] and [2]) but it seems like the stop hook may not be having a chance to complete that. Options there are to move the cleanup earlier, to the relation hooks, and to provide an action to do an explicit manual cleanup, like the AWS integrator charm has. If it's a hook race condition during model destroy, then it may still require a manual intervention in the teardown to ensure that the cleanup is run before the destroy-model command is run, but it should be escalated up to the Juju team if that's the case.

Alternatively, if the cleanup is failing, we need to figure out why. As discussed on IRC, you're going to try running the specific cleanup command that the charm uses (openstack loadbalancer delete --cascade $lb_name) to see if that same "immutable" error occurs.

[1]: https://github.com/juju-solutions/charm-openstack-integrator/blob/master/reactive/openstack.py#L125-L128

[2]: https://github.com/juju-solutions/charm-openstack-integrator/blob/master/lib/charms/layer/openstack.py#L140-L154

Running 'openstack loadbalancer delete --cascade <$lb-id>' worked.

It removed the old loadbalancer.

It should be noted, that after the loadbalancer is deleted with --cascade, the security group for this loadbalancer still exists.

---

$ openstack security group list
...
| 3c677ea8-cad7-4c53-aa0c-91ae07f5e8db | openstack-integrator-e2bfb0599b95-kubernetes-master | openstack-integrator-e2bfb0599b95-kubernetes-master | 4c204b4cf3e141c68fbb60aaadb2264b | [] |
...

---

This is still requiring a manual clean up.

Hrm. It seems that the cleanup code originally didn't cleanup the SG because it was global rather than per-LB [1]. However, it seems that is no longer the case and a new SG is used for each LB [2]. That does need to be fixed.

[1]: https://github.com/juju-solutions/charm-openstack-integrator/blob/master/lib/charms/layer/openstack.py#L141-L142

[2]: https://github.com/juju-solutions/charm-openstack-integrator/blob/master/lib/charms/layer/openstack.py#L419

The fact that the LB itself is not being cleaned up indicates that the stop hook [1] is either not running or being interrupted, during model destruction. I'm attaching Juju to this bug because my expectation is that the destruction would wait until the stop hook has run.

Just to confirm, Jeff: I trust that you are using neither --force nor --no-wait when destroying the model?

[1]: https://github.com/juju-solutions/charm-openstack-integrator/blob/master/reactive/openstack.py#L125-L128

No switches are used. Simply 'juju destroy-model kubernetes'

Adding the AWS integrator as well because it has a similar cleanup stop hook [1] which is also known to not get run during model destruction (which is why we added the purge-iam-entities to that charm).

[1]: https://github.com/juju-solutions/charm-aws-integrator/blob/master/reactive/aws.py#L154-L161

I should note that when trying to debug this in the past, I had issues with `juju debug-logs` terminating output before the model was completely destroyed. At the time, it was not entirely clear if that was just a communication issue or not, but I think the implementation of debug-logs has improved since then so we might be able to get some better info.

openstack-integrator log grabbed by 'juju debug-log --include openstack-integrator'

It starts right before the destroy-model command was issues (with no switches).

We'll investigate any Juju issue post 2.8. Adding to the 2.8.1 milestone initially so the bug doesn't get lost.

With the loadbalancer not being deleted, this ended up triggering a bug in Octavia which I've filed upstream, hoping for an Ussuri backport if there's a fix in master.

Tagging charm-octavia to monitor https://storyboard.openstack.org/#!/story/2009128 which provides details of what I've found which relates to this bug.

This is not an Octavia charm issue, unless it is believed that the charm is configuring the octavia application incorrectly. If this *is* the case, then please re-open the bug on the octavia charm.