Various things come up in IRC every once in a while about configuration options that need to be tweaked at large scale (blizzard, cern, etc) which once you hit hundreds or thousands of compute nodes need to be changed to avoid killing the control plane.
One such option is this:
https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.heal_instance_info_cache_interval
From a blizzard operator:
(3:04:18 PM) eandersson: mriedem, we had to set heal_instance_info_cache high because it was killing our control plane
(3:05:41 PM) eandersson: It was getting real heavy on large sites with 1k nodes
(3:06:26 PM) eandersson: We also ended up adding a variance
Similarly, CERN had to totally disable this one:
https://docs.openstack.org/nova/latest/configuration/config.html#compute.resource_provider_association_refresh
And rely on SIGHUP / restart of the service if they needed to refresh that cache.
We should put these things in the admin docs as we come across them so we don't forget about this stuff when new operators/users come along and hit scaling issues.
https:/