Heartbeats stop working when you mess with the system time. If a monotonic clock were used, they would continue to work when the system time was changed.
Steps to reproduce:
* List the nova services ('nova-manage service list'). Note that the 'State' for each services is a happy face ':-)'.
* Move the time ahead (for example 2 hours in the future), and then list the nova services again. Note that heartbeats continue to work and use the future time (see 'Updated_At').
* Revert back to the actual time, and list the nova services again. Note that all heartbeats stop, and have a 'State' of 'XXX'.
* The heartbeats will start again in 2 hours when the actual time catches up to the future time, or if you restart the services.
Heartbeats stop working when you mess with the system time. If a monotonic clock were used, they would continue to work when the system time was changed.
Steps to reproduce:
* List the nova services ('nova-manage service list'). Note that the 'State' for each services is a happy face ':-)'.
* Move the time ahead (for example 2 hours in the future), and then list the nova services again. Note that heartbeats continue to work and use the future time (see 'Updated_At').
* Revert back to the actual time, and list the nova services again. Note that all heartbeats stop, and have a 'State' of 'XXX'.
* The heartbeats will start again in 2 hours when the actual time catches up to the future time, or if you restart the services.
Here's example output demonstrating the issue:
http:// paste.openstack .org/show/ 477404/
See bug #1450438 for more context:
https:/ /bugs.launchpad .net/oslo. service/ +bug/1450438
Long story short: looping call is using the built-in time rather than a monotonic clock for sleeps.
https:/ /github. com/openstack/ oslo.service/ blob/3d79348dae 4d36bcaf4e52515 3abf74ad4bd182a /oslo_service/ loopingcall. py#L122
Oslo Service: version 0.11 76fefb66a91995d 6612bb4ab2)
Nova: Master (commit 2c3f9c339cae245