Comment 0 for bug 1510234

Revision history for this message
Diana Clarke (diana-clarke) wrote :

Heartbeats stop working when you mess with the system time. If a monotonic clock were used, they would continue to work when the system time was changed.

Steps to reproduce:

* List the nova services ('nova-manage service list'). Note that the 'State' for each services is a happy face ':-)'.
* Move the time ahead (for example 2 hours in the future), and then list the nova services again. Note that heartbeats continue to work and use the future time (see 'Updated_At').
* Revert back to the actual time, and list the nova services again. Note that all heartbeats stop, and have a 'State' of 'XXX'.
* The heartbeats will start again in 2 hours when the actual time catches up to the future time, or if you restart the services.

Here's example output demonstrating the issue:

    http://paste.openstack.org/show/477404/

See bug #1450438 for more context:

    https://bugs.launchpad.net/oslo.service/+bug/1450438

Long story short: looping call is using the built-in time rather than a monotonic clock for sleeps.

    https://github.com/openstack/oslo.service/blob/3d79348dae4d36bcaf4e525153abf74ad4bd182a/oslo_service/loopingcall.py#L122

Oslo Service: version 0.11
Nova: Master (commit 2c3f9c339cae24576fefb66a91995d6612bb4ab2)