i/o timeout errors can cause non-atomic service deploys

It is not atomic indeed - it can fail mid-way for various reasons, like "charm not found" (which leaves the service "empty" but existing).

1.22.8 and 1.22.6 was tested and failed

Dropped the milestore as it was not set correctly: there are no plans (that I know of) to fix it by the 1.26.0 release.
Consider that to fix the issue properly we need to have a (cooperating) persistency layer with a reliable support for transactions across multiple collections, including the ability to rollback such transactions.

@Dimiter: out of curiosity, I guess the "cooperating persistency layer" you are alluding to is the juju/state package?

Because the underlying technologies (mongodb + mgo/txn) do seem to have reliable support for transactions across multiple collections, afaiu. The problem seems a "factoring" one in juju/state which afaics basically ends up doing:

RunTransaction(<ADD SERVICE OPS>)
RunTransaction(<ATTACH FETCHED CHARM TO SERVICE OPS>
RunTransaction(<SET SERVICE CONFIG OPS>)

as opposed to:

RunTransaction(
    <ADD SERVICE OPS> + <ATTACH FETCHED CHARM TO SERVICE OPS> + <SET SERVICE CONFIG OPS>)

Is this the case?

I'm looking into this one now.

Note that users have sometimes experienced the timeout causing the service's settings set at deploy time to not actually get set.

It's worth noting that MongoDB i/o timeout errors always happen after the ensure-availability has been issued. As each new member of the replicaset comes up MongoDB breaks all connections to it, causing these errors. Juju should handle this better.

Could that be what's happening here? Was the replicaset coming up when the deploy was requested?

On Sunday, September 6, 2015, Menno Smits <email address hidden> wrote:
>
> Could that be what's happening here? Was the replicaset coming up when
> the deploy was requested?
>
>
In this case, no. Non HA juju.

--
David Britton <email address hidden>

FYI, I have a change that adds service settings to the same transaction as the service creation, so there's no possible failure in between the two. Now working on getting unit creation into the same transaction. Assigning units to machines is going to be done asynchronously by a worker to ensure retries in the case of timeouts etc.

Great to see service creation and config atomic. FWIW Landscape sets NumUnits to 0 when deploying a service (and the add units later with a separate API call), so atomicity of the unit creation part is not critical for our use case.

The first half of this bug is landing in 1.24 today (the part about making sure the settings get applied to the service atomically with service deployment).

And this has now landed in 1.24.

The second half of this bug - making unit assignment atomic(ish) is being tracked here: https://bugs.launchpad.net/juju-core/+bug/1497312

Nate - has this landed in master?