Machine agent state changes not included in the mega-watcher

We'll investigate if we can get this information over the watcher. If it's not exposed maybe we can request core support it.

I confirm this information is not sent as part of the mega-watcher data, so there is currently no way for the GUI to detect the machine was stopped/destroyed from outside of Juju.

See https://github.com/juju/juju/blob/master/apiserver/client/status.go#L746

Assigning this bug to juju-core.

Looked at this a bit with Menno, and we had expected that this was just not implemented. However, it looks like the machine agent status should be included, and there might just be a bug preventing it from being reported properly. Will look more in the morning.

From Menno:

I remember seeing this when I was working on status just after I started with Canonical. The "down" status isn't set or decided within state and is never reflected in the database. Instead the API server switches out the status when generating the result of the FullStatus API call. See the large comment I wrote to explain this towards the bottom of processAgent in apiserver/client/status.go. It even mentioned that the down status won't be seen by clients using a watcher.

So the "down" status is kinda of synthetic when it really shouldn't be. To fix this ticket, "down" needs to be somehow reflected in the database. If that's done then the AllWatcher API will start reporting machine or unit change events when the agent goes down (I don't think anything needs to change with the watcher code at all).

This might not be easy, especially to do efficiently (this might require some changes in state/presence). Something running in the state server needs to notice when any agent presence has changed and then update the agent's status in the database. It'll need to remember the previous status so that when the agent comes back the old status can be restored (the agent might not set the status back to "started" again itself).

This is the kind of thing that is worth talking to Will or John about as they will probably have some thoughts on how it should be done.

Sent an email to John and Will today to start the conversation around how to address this issue.

The conversation around this continues, as it is turning out to be quite a challenge to handle this appropriately.

There is still a lot of discussion around this, and we are aiming to have it complete as part of larger observability work in 1.26

Fixing this bug requires a substantial change to how we manage / track agent presence, and will likely need to be addressed in a feature in a future release.

Although code referred to here has been re-factored, the issue was not addressed in Juju 2.1.
I reproduced on AWS following original steps.

Removing 2.1 milestone as we will not be addressing this issue in 2.1.

An update to comment # 2, on develop (2.3-alpha1), the code can now be found at https://github.com/juju/juju/blob/develop/apiserver/facades/client/client/status.go#L722

This has indeed not been fixed yet. The analysis in comment #4 is still accurate.

this is affecting jaas metrics
Any news on that bug ?

@fabricematrat

Just that this is moved to the beta2 milestone and still something we're working to fit into the polish of 2.5 final. We're looking to cut beta1 today and so this didn't make the deadline this week.

This bug has not been updated in 2 years, so we're marking it Low importance. If you believe this is incorrect, please update the importance.