KvStore() is not mocked in unit_tests leading to random test failures

Example of solutions in other charms:
https://review.opendev.org/c/openstack/charm-hacluster/+/769967
https://review.opendev.org/c/openstack/charm-percona-cluster/+/764384

From vault charm unit test. KvStore needs mocking on it too:

{4} unit_tests.test_reactive_vault_handlers.TestHandlers.test_send_vault_url_and_ca_ext [5.153165s] ... FAILED

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):

      File "/home/zuul/src/opendev.org/openstack/charm-vault/.tox/py37/lib/python3.7/site-packages/mock/mock.py", line 1346, in patched
    return func(*newargs, **newkeywargs)

      File "/home/zuul/src/opendev.org/openstack/charm-vault/unit_tests/test_reactive_vault_handlers.py", line 739, in test_send_vault_url_and_ca_ext
    handlers.send_vault_url_and_ca()

      File "src/reactive/vault_handlers.py", line 615, in send_vault_url_and_ca
    vault_url = vault.get_api_url()

      File "src/lib/charm/vault.py", line 130, in get_vault_url
    if charms.reactive.is_state('vault.ssl.available'):

      File "/home/zuul/src/opendev.org/openstack/charm-vault/.tox/py37/lib/python3.7/site-packages/charms/reactive/flags.py", line 287, in is_state
    return is_flag_set(state)

      File "/home/zuul/src/opendev.org/openstack/charm-vault/.tox/py37/lib/python3.7/site-packages/charms/reactive/flags.py", line 208, in is_flag_set
    return any_flags_set(flag)

      File "/home/zuul/src/opendev.org/openstack/charm-vault/.tox/py37/lib/python3.7/site-packages/charms/reactive/flags.py", line 223, in any_flags_set
    active_flags = get_flags()

      File "/home/zuul/src/opendev.org/openstack/charm-vault/.tox/py37/lib/python3.7/site-packages/charms/reactive/flags.py", line 232, in get_flags
    flags = unitdata.kv().getrange('reactive.states.', strip=True) or {}

      File "/home/zuul/src/opendev.org/openstack/charm-vault/.tox/py37/lib/python3.7/site-packages/charmhelpers/core/unitdata.py", line 524, in kv
    _KV = Storage()

      File "/home/zuul/src/opendev.org/openstack/charm-vault/.tox/py37/lib/python3.7/site-packages/charmhelpers/core/unitdata.py", line 189, in __init__
    self._init()

      File "/home/zuul/src/opendev.org/openstack/charm-vault/.tox/py37/lib/python3.7/site-packages/charmhelpers/core/unitdata.py", line 389, in _init
    )''')

    sqlite3.OperationalError: database is locked

Failing tests are:

{1} unit_tests.test_reactive_vault_handlers.TestHandlers.test_configure_vault [5.085344s] ... FAILED
{4} unit_tests.test_reactive_vault_handlers.TestHandlers.test_send_vault_url_and_ca_ext [5.153165s] ... FAILED
6} unit_tests.test_reactive_vault_handlers.TestHandlers.test_send_vault_url_and_ca_hostname [5.055302s] ... FAILED
{7} unit_tests.test_reactive_vault_handlers.TestHandlers.test_loadbalancer [5.200378s] ... FAILED

You can make it use an in-memory database by setting this environment var:

UNIT_STATE_DB=:memory:

Or by doing the following before kv() is called:

charmhelpers.core.unitdata._KV = charmhelpers.core.unitdata.Storage(':memory:')

Perhaps this could be done by default in Zaza?

Reference: https://charm-helpers.readthedocs.io/en/latest/api/charmhelpers.core.unitdata.html#charmhelpers.core.unitdata.Storage

I'm trying to reproduce the error for charm-vault locally, but cannot. Alex, do you have any more information about where you see this sqlite error, or if you can reliably reproduce it?

Samuel, see the trace above in comment #2; it shows exactly where the lock occurred. However, it being a race-hazard bug means that it'll be hard to re-produce especially if you have a fast desktop/laptop as that'll make the locks smaller and thus less likely to occur at the same time.

One way to force a lock is just to do write a tight loop for say, 10 seconds, that constantly writes to the sql db (using the kv store) in at least two tests, limit the concurrency to 2 threads and that'll almost certainly result in a failure. However, it's probably not necessary to do that.

Either just use Cory's solution (#3) or better, make sure it is mocked out, as we are not actually testing the SQL database, and it's locking behaviour may change over versions. Thanks!

After merging in https://github.com/juju/charm-helpers/pull/823 to charmhelpers, this shouldn't be a problem anymore.

> After merging in https://github.com/juju/charm-helpers/pull/823 to charmhelpers, this shouldn't be a problem anymore.

This only affects the main development branch. As we use stable branches for stable versions, we may need to consider backporting the above change to stable branches.