SecurityGroup deletion causes bulk_pull of SG rules by all the agents

Hi Guillaume,

Thank you for submitting the bug. For completness
- what OSP version are you using? Probably does not matter that much since the bulk_pull has not changed much
- How big is your enviroment? How many rpc workers do you have on each node? How many security rules per group? Basically more details to reproduce the performance issue.
- Agents logs could be useful as well if handy.

Thanks!

Here's a proposed patch (I don't know why it doesn't appear in the bug's comments): https://review.opendev.org/c/openstack/neutron/+/883235

For the sake of completeness:
we observed the problem in Stein and Xena, but the code indicates this issue exists in the subsequent versions as well.

Without being too specific, our environment spans up to several thousands agents. I don't have clear numbers about how many rules per group we have, but I would hazard a guess at at least 2.

With regard to the code-path triggering the issue:

When a security group is deleted, a notification is sent to all the agents triggering the callbacks of SecurityGroupServerAPIShim [1]. All the agents receiving the notification proceed with the SecurityGroupServerAPIShim._clear_child_sg_rules callback which itself is using
self.rcache.get_resources to look for SecurityGroupRules belonging to the deleted SecurityGroup [2].

Now, the first thing RemoteResourceCache.get_resources does is call _flood_cache_for_query [3] which looks in its own cache if it already queried neutron-rpc for these resources and if not, finally, performs the unwelcome bulk_pull.

[1] https://github.com/openstack/neutron/blob/fd21c905ca9016092d48d3f4442bae6d4abb42e3/neutron/api/rpc/handlers/securitygroups_rpc.py#L247
[2] https://github.com/openstack/neutron/blob/fd21c905ca9016092d48d3f4442bae6d4abb42e3/neutron/api/rpc/handlers/securitygroups_rpc.py#L306
[3] https://github.com/openstack/neutron/blob/fd21c905ca9016092d48d3f4442bae6d4abb42e3/neutron/agent/resource_cache.py#L127

Thanks for taking care for this issue

Just to make it more visible, related patch:
https://review.opendev.org/c/openstack/neutron/+/883235

This issue was fixed in the openstack/neutron 23.0.0.0b3 development milestone.