Canonical Juju

destroy-controller fails on microk8s

Bug #1865146 reported by Tom Haddon
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Canonical Juju
Triaged
Low
Unassigned

Bug Description

I'm running the following versions:

$ snap list microk8s
Name Version Rev Tracking Publisher Notes
microk8s v1.17.3 1214 stable canonical✓ classic
$ snap list juju
Name Version Rev Tracking Publisher Notes
juju 2.8-beta1+develop-4cd46ad 10782 edge canonical✓ classic

I've created a k8s controller, and deployed a charm into the model. I then saw a config-changed hook error, so decided to destroy the controller (I'm iterating on developing a charm locally). However, it's still running approximately 15 minutes later:

$ juju status -m is-pd-bot
Model Controller Cloud/Region Version SLA Timestamp
is-pd-bot micro microk8s/localhost 2.8-beta1 unsupported 13:33:45+01:00

App Version Status Scale Charm Store Rev OS Address Notes
pd-bot error 1 pd-bot local 0 kubernetes hook failed: "config-changed"

Unit Workload Agent Address Ports Message
pd-bot/0* error idle hook failed: "config-changed"

$ juju destroy-model is-pd-bot
WARNING! This command will destroy the "is-pd-bot" model.
This includes all containers, applications, data and other resources.

Continue [y/N]? y
Destroying model
Waiting for model to be removed, 1 application(s)...............................
................................................................................
................................................................................
................................................................................
................................................................................
...........................................

Checking microk8s the operator pod is still running:

$ microk8s.kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
container-registry registry-d7d7c8bc9-twt24 1/1 Running 4 2d
controller-micro controller-0 2/2 Running 1 25m
is-pd-bot pd-bot-operator-0 1/1 Running 0 15m
kube-system coredns-7b67f9f8c-85qg9 1/1 Running 5 2d2h
kube-system hostpath-provisioner-7b9cb5cdb4-spxrg 1/1 Running 4 2d

Revision history for this message
Tom Haddon (mthaddon) wrote :

Running juju status in a separate window I see "attempt 1 to destroy model failed (will retry): model not empty, found 1 application (model not empty)", fwiw.

Revision history for this message
Tom Haddon (mthaddon) wrote :

And now the command completed "ERROR timeout after 30m0s timeout" and juju status' notes still reports "attempt 1 to destroy model failed (will retry): model not empty, found 1 application (model not empty)"

Revision history for this message
Tom Haddon (mthaddon) wrote :

Eventually this model was deleted. I was making live changes to the pod to debug my config-changed errors, and I *think* when I actually fixed those to the point where the pod wasn't erroring any more, it was immediately terminated.

I span up another model got that to the point were the charm was running without any errors, and I was able to destroy the model without problems this time.

So perhaps this blocks on a failed hook?

Revision history for this message
Ian Booth (wallyworld) wrote :

juju will block if there's any hook errors etc as it tries to allow units to terminate gracefully, running the cleanup steps such as terminating relations etc.

If there's an error, you need to use --force to tell juju to try cleanly, but if that fails, continue anyway.

I'll mark this as Invalid on that basis - please feel free to re-open if necessary.

Changed in juju:
status: New → Invalid
Revision history for this message
Tom Haddon (mthaddon) wrote :

Could the "note" field say something more useful? Currently it says "attempt 1 to destroy model failed (will retry): model not empty, found 1 application (model not empty)" but this wasn't obvious to me that it was blocking on a failed hook.

Changed in juju:
status: Invalid → New
Revision history for this message
Ian Booth (wallyworld) wrote :

We can look to add a bit of extra info to the model destroy progress messages. It's not totally trivial since the api used by the destroy command only feeds back back counts of remaining apps, machines etc. So we'll need to rev the api version and change data formats. juju status will of course surface any model related errors so that's one place to look in the meantime.

Changed in juju:
milestone: none → 2.8-beta1
status: New → Triaged
importance: Undecided → Medium
tags: added: destroy-model
tags: added: ui
Ian Booth (wallyworld)
Changed in juju:
milestone: 2.8-beta1 → 2.8.1
Tim Penhey (thumper)
Changed in juju:
milestone: 2.8.1 → 2.8-next
Revision history for this message
Canonical Juju QA Bot (juju-qa-bot) wrote :

This bug has not been updated in 2 years, so we're marking it Low importance. If you believe this is incorrect, please update the importance.

Changed in juju:
importance: Medium → Low
tags: added: expirebugs-bot
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.