Landscape Client

have client message broker back-off when receiving 5xx responses

Bug #1947399 reported by Simon Poirier
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Landscape Client
Fix Released
High
Kevin Nasto
Landscape Server
Fix Released
Undecided
Kevin Nasto
Landscape Server 19.11
landscape-client (Ubuntu)
Fix Released
Undecided
Unassigned

Bug Description

Current client tries sending messages on schedule. When we get a massive backlog of messages (due to outage or bugs) we are basically getting DDOS-ed, getting 503 messages.

As an operator of landscape SaaS, I would like to have at least those 2 throttling mechanisms added to clients when 5xx status are received

- an exponential back-off timer of the broker (urgent or standard exchanges) are queued to avoid hammering the message-system with too many simultaneous clients.
- a reverse-exponential reduction of the broker max_messages to reduce server load and keep the broker "nice"

This might lengthen recovery from downtime, but should make the recovery smoother.

Revision history for this message
Kevin Nasto (silverdrake11) wrote :

The pull request is here https://github.com/CanonicalLtd/landscape-client/pull/107

Changed in landscape-client:
assignee: nobody → Kevin Nasto (silverdrake11)
Kevin Nasto (silverdrake11)
Changed in landscape:
assignee: nobody → Kevin Nasto (silverdrake11)
Simon Poirier (simpoir)
Changed in landscape:
status: New → In Progress
Changed in landscape-client:
status: Confirmed → In Progress
🤖 Landscape Builder (landscape-builder)
Changed in landscape:
milestone: none → 19.11
status: In Progress → Fix Committed
Kevin Nasto (silverdrake11)
Changed in landscape-client:
status: In Progress → Fix Committed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package landscape-client - 23.02-0ubuntu1

---------------
landscape-client (23.02-0ubuntu1) lunar; urgency=medium

  * New upstream release 23.02:
    - Preventing the generation of large messages and logs that can overwhelm
      Landscape Server (LP: #1995775)
    - Improved MOTD slowdown on machines with many tap network interfaces
      (LP: #2006396)
    - No longer using deprecated apt-key when storing trusted GPG keys
      (LP: #1973202)
    - Fixed issue recognising Parallels VMs as Virtual Machine clients
      (LP: #1827909)
    - Fixes for incorrect logfile rotation config (LP: #1968189)
    - Client-side backoff handling to moderate traffic to Landscape Server
      during high load (LP: #1947399)
    - Avoid sending empty messages when catching up to expected next message
      (LP: #1917540)
    - --is-registered CLI option to quickly check if client is registered
      (LP: #1912516)
    - Can now report Ubuntu Pro attachment information if the version of
      Landscape Server it is registered to supports this (LP: #2006401)
    - Packages installed as dependencies as part of package profiles are now
      appropriately autoremovable (LP: #1878957)
    - Registration timeouts give an error instead of timing out (LP: #1889464)
    - RHEV hypervisor VMs are now recognized as virtual machines (LP: #1884116)
    - Doing a Landscape-driven release upgrade from a release running python 2
      to one running python 3 no longer hangs forever (LP: #1943291)

 -- Mitch Burton <email address hidden> Wed, 08 Feb 2023 10:23:31 -0800

Changed in landscape-client (Ubuntu):
status: New → Fix Released
Mitch Burton (mitchburton)
Changed in landscape-client:
status: Fix Committed → Fix Released
Changed in landscape:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.