neutron

[rfe] rate-limit metadata API

Bug #1989199 reported by Guillaume Espanel
12
This bug affects 2 people
Affects Status Importance Assigned to Milestone
neutron
Fix Committed
Wishlist
Miguel Lavalle

Bug Description

At the moment, there is no limit on how many requests the metadata-agent
will handle. Some users may sometimes run scripts in their instances that
try to query the metadata endpoint at high rate (for example a bugged k8s
cloud controller manager), causing an increased load on some or all the
component above the metadata-agent.

We'd like to add some rate-limiting around the metadata-agent level to give
some protection to the other components. We were thinking to implement
that through a change to the metadata's haproxy's configuration using
stick-tables. In that case, the rate-limited queries would get a 429
without even being passed to the metadata-agent itself.

Revision history for this message
Lajos Katona (lajos-katona) wrote :

We discussed this RFE during our drivers meeting, see the logs:
https://meetings.opendev.org/meetings/neutron_drivers/2022/neutron_drivers.2022-09-09-14.00.log.html#l-51

The agreement was to approve this RFE, and ask for a spec where the details can be discussed. thanks for proposing.

tags: added: rfe-approved
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron-specs (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron-specs/+/856831

Miguel Lavalle (minsel)
Changed in neutron:
status: New → In Progress
importance: Undecided → Wishlist
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron-specs (master)

Reviewed: https://review.opendev.org/c/openstack/neutron-specs/+/856831
Committed: https://opendev.org/openstack/neutron-specs/commit/9462d10a7b1c65c877bec5002315fafcc49ac28e
Submitter: "Zuul (22348)"
Branch: master

commit 9462d10a7b1c65c877bec5002315fafcc49ac28e
Author: Guillaume Espanel <email address hidden>
Date: Fri Sep 9 17:53:23 2022 +0200

    Add spec for metadata-rate-limit

    Related-Bug: #1989199
    Change-Id: I31fe80a2fa4c495acfab5886a304866c7da0d04b

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (master)

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/858879

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (master)

Change abandoned by "Slawek Kaplonski <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/neutron/+/858879
Reason: This review is > 4 weeks without comment, and failed Zuul jobs the last time it was checked. We are abandoning this for now. Feel free to reactivate the review by pressing the restore button and leaving a 'recheck' comment to get fresh test results.

Revision history for this message
Miguel Lavalle (minsel) wrote :

During the Neutron Drivers meeting on April 21st, 2023 (https://meetings.opendev.org/meetings/neutron_drivers/2023/neutron_drivers.2023-04-21-14.00.log.html#l-176), it was agreed that the implementation of this RFE will support rate limiting for IPv4 and IPv6, but not at the same time. The user will have to choose whether rate limiting will be carried out in the deployment on IPv4 or IPv6. This is due to a limitation in the open source version of HAProxy, that only provides a maximum of 3 sticky counters. To support rate limiting for IPV4 and IPv6 simultaneously, 4 sticky counters would be needed.

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix merged to neutron (master)

Reviewed: https://review.opendev.org/c/openstack/neutron/+/858879
Committed: https://opendev.org/openstack/neutron/commit/5f4a41326d7b1da03e1929b4fd8bcdaf69da19ab
Submitter: "Zuul (22348)"
Branch: master

commit 5f4a41326d7b1da03e1929b4fd8bcdaf69da19ab
Author: Guillaume Espanel <email address hidden>
Date: Thu Sep 22 11:27:04 2022 +0200

    Add rate-limiting to metadata agents

    Requests handled by the metadata-agents can now be rate-limited by
    source-ip. This is done to protect the OpenStack control plane against
    VMs querying the metadata endpoint in an overly enthusiastic way.

    Co-authored-by: Miguel Lavalle <email address hidden>

    Related-Bug: #1989199
    Change-Id: I748ccfa8b50496dcbcbe41fd22f84249a4d46b11

Miguel Lavalle (minsel)
Changed in neutron:
assignee: nobody → Miguel Lavalle (minsel)
status: In Progress → Fix Committed
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/wallaby)

Related fix proposed to branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/892754

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (stable/wallaby)

Change abandoned by "Miguel Lavalle <email address hidden>" on branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/892754

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Related fix proposed to neutron (stable/zed)

Related fix proposed to branch: stable/zed
Review: https://review.opendev.org/c/openstack/neutron/+/892868

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (stable/zed)

Change abandoned by "Miguel Lavalle <email address hidden>" on branch: stable/zed
Review: https://review.opendev.org/c/openstack/neutron/+/892868

Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Change abandoned on neutron (stable/wallaby)

Change abandoned by "Miguel Lavalle <email address hidden>" on branch: stable/wallaby
Review: https://review.opendev.org/c/openstack/neutron/+/892754

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.