Release: Mirantis OpenStack 4.0
Steps to reproduce:
Launch instances (as many as you can). Assign floating IPs to these instances.
Check how much time is needed for displaying all of them in Horizon
Expected result:
Information about floating IPs appears immediately after assignment to an instance
Actual result:
There is a delay between floating IPs assignment and displaying they both in Horizon and "nova list"
There is the more detailed description:
Network information is placed in "InstanceInfoCache" entry in the database
Nova compute runs periodic task for updating network information every 60 seconds (by default)
But due to implementation of task, only one instance can be processed in each run.
So every new instance on node will increase update period for specific instance by 1 minute
There is a snippet from nova-compute.log
2014-05-23 02:04:47 DEBUG nova.compute.manager [req-fb3efb94-0ebf-4633-b10a-160ed4914847 None None] [instance: 3de0cbfd-b92b-412c-bd4c-d8d59d617055] Updated the info_cache for instance _heal_instance_info_cache /usr/lib/python2.7/dist-packages/nova/compute/manager.py:4364
2014-05-23 02:05:50 DEBUG nova.compute.manager [-] [instance: 0d6fd7ae-a84a-4ed2-9c14-0342531f10f2] Updated the info_cache for instance _heal_instance_info_cache /usr/lib/python2.7/dist-packages/nova/compute/manager.py:4364
2014-05-23 02:06:53 DEBUG nova.compute.manager [-] [instance: 4b63b037-c617-436c-9522-a812e80b8026] Updated the info_cache for instance _heal_instance_info_cache /usr/lib/python2.7/dist-packages/nova/compute/manager.py:4364
The configuration option "heal_instance_info_cache_interval" in the nova.conf is intended to changing frequency of this periodic task. But this option doesn't work due to this bug: https://bugs.launchpad.net/nova/+bug/1274317
Anyway, applying of patch to fix this bug is just workaround that is able to reduce delay.
Also, if the interval is too small, it can lead to some nova-compute service are sporadically going down (It's not confirmed that there is the root cause, but issue appeared after changing the value of "heal_instance_info_cache_interval" to 20 seconds)
I verified how it works in Mirantis OpenStack 5.0
There are the following messages are writed in the nova-compute.log every minute:
<183>May 30 17:40:28 node-1 nova-nova.openstack.common.periodic_task DEBUG: Running periodic task ComputeManager._heal_instance_info_cache
<183>May 30 17:40:28 node-1 nova-nova.compute.manager DEBUG: Starting heal instance info cache
<183>May 30 17:40:28 node-1 nova-nova.compute.manager DEBUG: Rebuilding the list of instances to heal
It seems this logic eliminates subsequent info cache update for each instance
Information about Floating IPs appears in Horizon IMMEDIATELY (before next info cache update happens)
Also I've checked that the value of "heal_instance_info_cache_interval" can be configured out-of-box (without any patches)
Would you please backport code from 5.0 to the previous versions (4.x)
Also, backport to 4.0 is required