clock updated concurrently on a non-ha controller

Triaging and adding to milestone for further investigation.

It's possible that we are firing off a worker twice, which might be responsible for other raft oddities. This is definitely worth digging into!

Got a reproducer upgrading from 2.8 HEAD to develop HEAD.

This should be a transient error.

When the global clock updater advances time via the Raft store, it sends what it thinks is the old global time in the command.

If this time does not match the Raft FSM last known time, the concurrent update error is returned. The mechanism protects against concurrent updates, which is reflected in the error message, but that doesn't *quite* match what is happening here.

It actually results in the updater syncing its last known time with the FSM and retrying with a back-off.

From what I can tell, this is possible after restarting controller machine agents and restoring the Raft FSM from a snapshot, in which case the updater has a different value from the FSM.

This is pertinent log capture following a restart.
https://pastebin.canonical.com/p/DjnWdt5rsW/

I can see some issues here.

1. The lease manager makes claims against what appears to be an empty FSM *before* it has completed restoring from its snapshot - you can see the "no leasing found, claiming.." entries.

2. The initial clock advancement, the worker having been instantiated with an empty FSM, is made with an old time of 0001-01-01 00:00:00 +0000 UTC, which is incorrect following the FSM restoration (mentioned above).

3. Raft leader election occurs, relevant workers bounce, there are lease claim time-outs following.

The logs show convergence and success following this jitter, but we don't really want large models spammed with time-out errors every time there is a Raft leader election.

I've proposed https://github.com/juju/juju/pull/12248 in light of this ticket.

It doesn't change the fundamentals, but I think uses better nomenclature.