Error causes deployments to fail

I'm seeing instances of this error in our automated testing of Landscape autopilot openstack deployments. The most recent example was with juju 1:2.1~rc1, but I also have runs with 1:2.1~beta4. Also, I first found lp:1604915, which is duped to this bug. I've seen this error four times between beta4 and rc1.

The error message seen in one of the juju unit logs is:
[from build 5173 landscape-0-inner-logs/ceilometer-1/var/log/juju/unit-ceilometer-1.log]
2017-02-02 05:12:48 ERROR juju.worker.dependency engine.go:547 "leadership-tracker" manifold worker returned unexpected error: leadership failure: lease manager stopped
2017-02-02 05:12:48 ERROR juju.worker.uniter agent.go:28 resolver loop error: could not acquire lock: cancelled acquiring mutex

The juju-status message for that unit is also set to "resolver loop error". In each instance of hitting this, it appears to be a different application. The error causes the entire deployment to fail.

Builds associated with this failure:
 - https://ci.lscape.net/job/landscape-system-tests/5263
 - https://ci.lscape.net/job/landscape-system-tests/5173
 - https://ci.lscape.net/job/landscape-system-tests/5139
 - https://ci.lscape.net/job/landscape-system-tests/5132

Trimmed logs from https://ci.lscape.net/job/landscape-system-tests/5263.

Speaking of leadership issues, it could be related to https://bugs.launchpad.net/juju/+bug/1654116 (just a remark)

We'll address this in 2.1.1 due early March. Putting Importance higher based on the feedback from Interested Parties.

Affects landscape `15-20% of the time, TestUniterSteadyStateUpgradeRelations: could not acquire lock

This is happening because a mutex acquire is being cancelled while delaying. The uniter tries to grab the hook lock, but is blocked by the test (in waitHooks). It delays 250ms and while it's delayed the test kills the uniter, which makes the delayed acquire call return ErrCancelled.

That is returned by the worker.Stop call in util_test.go, and so the test fails.

I'm changing the test code so that it won't fail if it gets ErrCancelled from the stop call.

PR: https://github.com/juju/juju/pull/7037