Comment 1 for bug 1949807

Looking at layer-vault-kv [1] , it seems that when `update-status` hooks run, every unit will contact vault and collect some app layer key-value data. Also, each unit collects its 'hash' of the data to determine if the value has been locally consumed by the charm. Those app layer kv hashes are only updated by the leader [2], and not every unit.

All units should read the same k-v store[3] on every hook, as well as their own unit's hash on every hook.

I believe its possible that the non-leaders are reading a different hash value than the hash generated from the value in the the app storage, which causes it to believe there is a change.

this layer tries to keep up with a shared k-v store and hashes the values to see if they have "changed" or not. When the non-leader wakes, it reads the KV store, sees that the `encryption_key` hash doesn't match the previous hash and declares it "changed".

can the kv-store in vault be interrogated?
Yes

```
vault kv get charm-kubernetes-master/kv/app
vault kv get charm-kubernetes-master/kv/app-hashes/7
vault kv get charm-kubernetes-master/kv/app-hashes/8
vault kv get charm-kubernetes-master/kv/app-hashes/9
```

Result:
```
ubuntu@juju-f7d8a7-lma-4:~$ for i in app app-hashes/{7,8,9}; do vault kv get -address http://172.16.100.4:8200 charm-kubernetes-master/kv/${i}; done
========= Data =========
Key Value
--- -----
encryption_key s[...]j
========= Data =========
Key Value
--- -----
encryption_key c[...]1
No value found at charm-kubernetes-master/kv/app-hashes/8
No value found at charm-kubernetes-master/kv/app-hashes/9
```

as somewhat expected, the non-leaders don't write their hash so they always appear "changed".

Work-around
```
HASH=$(vault kv get -field=encryption_key charm-kubernetes-master/kv/app-hashes/7)
vault kv put charm-kubernetes-master/kv/app-hashes/8 encryption_key=$HASH
vault kv put charm-kubernetes-master/kv/app-hashes/9 encryption_key=$HASH
```

[1] https://github.com/juju-solutions/layer-vault-kv/blob/e22c18b133070ce354cebbda864a5aa8a4b60398/lib/charms/layer/vault_kv.py#L101
[2] https://github.com/juju-solutions/layer-vault-kv/blob/e22c18b133070ce354cebbda864a5aa8a4b60398/reactive/vault_kv.py#L55
[3] https://github.com/juju-solutions/layer-vault-kv/blob/e22c18b133070ce354cebbda864a5aa8a4b60398/reactive/vault_kv.py#L49