Bootstrap on Joyent often fails with "model is not bootstrapped"

This is NOT a bug, it's only a warning and the logs show that:

2016-03-21 05:51:42 WARNING juju.api api.go:127 discarding API open error: space discovery still in progress

The CI job however treats this as an error and immediately destroys the newly successfully bootstrapped model.

If you look at the rest of the command output, you'll see that the bootstrap fails with "model is not bootstrapped" after encountering this error.

2016-03-21 11:56:11 WARNING juju.api api.go:127 discarding API open error: space discovery still in progress
2016-03-21 11:56:11 INFO juju.provider.common destroy.go:22 destroying model "joyent-deploy-precise-amd64"
2016-03-21 11:56:11 INFO juju.provider.common destroy.go:33 destroying instances
2016-03-21 11:56:23 INFO juju.provider.common destroy.go:53 destroying storage
2016-03-21 11:56:23 ERROR cmd supercommand.go:448 model is not bootstrapped

This is our top cause of failure for the week, at 88 matches currently:

<http://reports.vapour.ws/releases/top-issues>

A couple are spurious, but most are hard bootstrap failures on joyent with no obvious cause, and only the spaces warning standing out in the log. That may be a red herring, but we do have a intermittent but persistent issue getting a successful bootstrap on Joyent.

I'd be happy to add more debugging if possible, but nothing CI does aborts bootstrap in response to log output from juju.

I think this bug is now for a slightly different problem than when originally reported. It seems that bootstrapping is failing with "model is not bootstrapped", starting after the switch over to use tags to track controller instances, rather than storage.

I suspect that there's some timing window where while the instance is coming up, the filtering on tags.JujuController == true won't return the machine that's just been provisioned.

If this is a true limitation in Joyent (i.e. we can't guarantee filtering on tags that were just set), we could add the "model is not bootstrapped" error to the list of errors we retry on when waiting for the API to become available in the bootstrap command.

Okay, this is actually a bug in the joyent provider. We are ignoring the tags passed into StartInstance (which contain the is-controller tag).

We need to append the joyent specific tags to this list, not ignore them.

Verified that properly setting (and filtering) on the tags fixed the bootstrap issue for me locally. Anastasia is working on making the changes.

PR: https://github.com/juju/juju/pull/4842