Deployer fails because juju thinks it is upgrading

The term "upgrade" may mean one of two things:

1. Juju running upgrade steps to upgrade an older environment
2. Juju upgrading agent tools

The message in this bug refers to item 1, but with the change in behaviour of Juju bootstrap, is now poorly worded.
What happens now is:

1. Juju bootstraps and starts the machine agent on the bootstrap node
2. the machine agent delays activating the until:
  a. any upgrade steps are run
  b. it has determined that no agent upgrades are needed <-- this is new
3. once all upgrade (agent or steps) related tasks are finished, the full api is enabled

So if a deploy is attempted before the full api is enabled, the "upgrade in progress" error is returned.

Before the above change, the deployer would connect immediately after bootstrap and if an implicit upgrade were done, the deployer would be disconnected part way through it's deployment process.

Now what happens is more correct - any attempt to do work with Juju while the state server is not ready is rejected up front, rather than accepting a connection and then disconnecting.

The same response in this bug would happen if the user typed fast and did a deploy immediately after bootstrap - they would be told to try again in a sort time.

Ideally here the deployer would "do the right thing" and retry.

I question whether this is a regression - the same "error" or behaviour would always been possible/likely if the user did:

juju upgrade-juju && juju-deploy

The deployer would receive the message about an upgrade being in progress.

Without the change in behaviour

juju bootstrap && juju-deploy

also failed, but in a way that is less obvious and useful - a unexpected disconnect.

Now the state server doesn't even accept the deploy request in the first place until it is ready to act on it, and instead returns an error which the deployer can retry on.

A solution would be to delay the API, limited or otherwise, availability until after the agent upgrade check. But this would have 2 issues:

1. juju status would be slightly delayed until it started working
2. would still not solve the agent upgrade error that the deployer would get when an agent upgrade is running

The limited API would still be available while the upgrade steps are running, but not until after the agent upgrade check completes.

Interestingly, it doesn't always fail. On AWS:

juju bootstrap --upload-tools && juju --debug deployer --deploy-delay 10 --config ~/landscape-scalable.yaml

works fine.

Another option is to teach the juju-deployer how to retry on upgrade errors. This would be much easier to implement.

The python-jujuclient code has been modified so that if any RPC call results in an "upgrade in progress" error, then the call will be retried.

This change improves the robustness of the deployer overall.

https://code.launchpad.net/~wallyworld/python-jujuclient/retry-on-upgrade/+merge/260658

I changed the deployer test to call client.wait_for_started() after bootstrap. We can see that 17 seconds can pass before the the bootstrapped state-server is ready.

I added code to juju bootstrap to delay the exit of the bootstrap command until the API is fully available. This will also alleviate the problem, without the need for a delay in deployer.