juju status <service_name> broken

Can you attach the following?
1 - Full machine-0.log
2 - Full output of juju status
3 - Output of juju status wordpress --debug

Setting to incomplete, pending the additional info requested in comment #1

Some info in those logs is sensitive so I prefer not to publish them here. I archived them and let developers know in private where to find them.

Reviewed the logs, and wasn't able to find any obvious answer. Will need to look into later. Any devs interested in assisting can contact me for the logs.

I see what the problem is now. This was injected when the unit status was split into unit / unit agent status. Starting in 1.24, we key the unit status with an id of "u#<unit name>#charm", whereas older versions just used "u#<unit name>". The key "u#<unit name>" is now used for the unit agent status.

There does not seem to be an upgrade step to create the unit / workload statuses, so when we try to query the unit status, we get an error back. In running the full status, this error is ignored, and a nil status is displayed for the workload. You can see this when we just run juju status: http://paste.ubuntu.com/13538895/
In the above paste, the ubuntu and ubuntu2 workload-status is nil. But, ubuntu3, which was deployed post-upgrade to 1.25, has a valid status.

When running juju status to query a specific service, it will error because the workload status does not exist.

An upgrade step is needed to correctly create the unit / workload status. If needed, it may be possible to create a script to insert the values into an existing environment. But, I need to defer that to someone who has more knowledge of mongodb internals. Going to ask Menno for help on this part.

This is also a test escape that should have a testcase added to the juju test suite.

fwereade is working on a script to repair an environment after this is hit in an upgrade.

When targeting the upgrade step in 1.25 I cannot see a way to distinguish between 1. a correct agent statusDoc and 2. a unit statusDoc with a agent style id - as the id is the only distinguishing feature between the two.

I imagine fwereade will hit this when writing the fix-it script.

unitAgentGlobalKey was introduced in 1.22-alpha1, before then we just had unitGlobalKey. That is, Prior to 1.22-alpha1 we didn't have an agent status. If we targeted 1.22 then the upgrade step could grab all docs from statusC with format u#<name> and update them to u#<name>#charm

Cheryl just pointed out that each unit needs to have a unit agent status doc and a workload (unit status) doc. So we can:

1 - generate a list of all the units
2 - query the status for each unit (not agent status)
3 - if it returns not found, then create the status

The fix for this is underway, and should be included in 1.25.2.

William kindly put together a script to directly modify mongo to add in the missing statuses to work around this problem in the meantime. You can contact me directly to get the script until I figure out where to put it.

I have verified that running the script after an upgrade resolves the unit not found error.

pr: http://reviews.vapour.ws/r/3279

I confirm the fix is effective. For large environments where hooks have failed, here's a one-liner after running the fix to re-run all the failed hooks, which should return the agents to normal (idle) state:
for i in `juju status --format tabular|grep failed|awk '{print $1;}'`;do echo Fixing $i;juju resolved -r $i;done

Just a note to those using the script to insert the workload status - you'll see a bogus time in the "since" field: 31 Dec 1969 18:00:00-06:00

Status changed to 'Confirmed' because the bug affects multiple users.

Not fixing in 2.0, as users will be required to upgrade to 1.25.2 before upgrading to 2.0

WARNING: don't use the script in comment #16, but instead use this one:

http://pastebin.ubuntu.com/24186473/

This is a copy of the earlier script but adds in the txn-revno and txn-queue fields that are required for when juju wants to update the status to something new.

Should I mark comment #16 as "hidden" so it doesn't give people the wrong idea?

Great idea, John \o/ Marked as hidden now!