When the global clock updater advances time via the Raft store, it sends what it thinks is the old global time in the command.
If this time does not match the Raft FSM last known time, the concurrent update error is returned. The mechanism protects against concurrent updates, which is reflected in the error message, but that doesn't *quite* match what is happening here.
It actually results in the updater syncing its last known time with the FSM and retrying with a back-off.
From what I can tell, this is possible after restarting controller machine agents and restoring the Raft FSM from a snapshot, in which case the updater has a different value from the FSM.
This should be a transient error.
When the global clock updater advances time via the Raft store, it sends what it thinks is the old global time in the command.
If this time does not match the Raft FSM last known time, the concurrent update error is returned. The mechanism protects against concurrent updates, which is reflected in the error message, but that doesn't *quite* match what is happening here.
It actually results in the updater syncing its last known time with the FSM and retrying with a back-off.
From what I can tell, this is possible after restarting controller machine agents and restoring the Raft FSM from a snapshot, in which case the updater has a different value from the FSM.