misbehaving backend services are able to DOS external services
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| Bazaar Subversion Plugin |
Triaged
|
High
|
Unassigned | ||
| Launchpad CSCVS |
Triaged
|
Low
|
Unassigned | ||
| Launchpad code imports |
Triaged
|
Low
|
Unassigned | ||
Bug Description
Symptoms
========
Launchpad backend services like the code importer, bug watchers, tarball finders are all expected to communicate with external sites at fairly high frequencies.
We will from time to time have bugs in those services that result in excessively high attempted requests being made to those sites. When this happens we generally get firewalled which is unpleasant for everyone.
Solution
========
As part of being good net citizens we need to ensure that when that happens our services are throttled in some fashion. Simple request count throttling won't cover all requests (e.g. asking svn to perform an expensive server side operation may still be an issue) but its an improvement over our current catchall (revisions imported in one session are throttled).
Implementation
==============
We probably want to use an external catchall - e.g. squid or something - so that many different services can all be throttled in some sensible fashion.
bzr-svn and cscvs also speak svn: and pserver: which means that we either need a proxy protocol implementation for them or some in-process policy layer (that doesn't lose state when a specific import worker fails).
| Changed in launchpad: | |
| importance: | Undecided → Critical |
| tags: | added: canonical-losa-lp |
| description: | updated |
| Robert Collins (lifeless) wrote Re: no safety net for misbehaved backend services | #1 |
| Changed in launchpad: | |
| importance: | Critical → High |
| status: | New → Triaged |
| summary: |
- code import servers DoS-ing third party services + no safety net for misbehaved backend services |
| description: | updated |
| Changed in launchpad-cscvs: | |
| status: | New → Triaged |
| importance: | Undecided → High |
| Changed in bzr-svn: | |
| importance: | Undecided → High |
| summary: |
- no safety net for misbehaved backend services + misbehaving backend services are able to DOS external services |
| Robert Collins (lifeless) wrote | #2 |
| Martin Pool (mbp) wrote | #3 |
| Changed in bzr-svn: | |
| status: | New → Triaged |
| assignee: | nobody → Jelmer Vernooij (jelmer) |
| Jelmer Vernooij (jelmer) wrote | #4 |
| Changed in bzr-svn: | |
| assignee: | Jelmer Vernooij (jelmer) → nobody |
| Robert Collins (lifeless) wrote | #5 |
| Changed in launchpad-cscvs: | |
| importance: | High → Low |
| Changed in launchpad: | |
| importance: | High → Low |
| affects: | launchpad → lp-codeimport |
There is a separate bug (I don't have the number handy sorry) about the specific behaviour in bzr-svn. The way that dos occured was a bzr-import-export triggering a retry of a known-failing import to see if it worked with the updates made to bzr-svn over the last while. (It didn't).
Our normal protections *did* operate : we didn't run the import too often, and we didn't import too many revisions, but we encountered a failure mode on a different dimension : importing a single revision went beserk. I'm repurposing this bug to cover a more general catchall to protect against unknown bugs like this (was) and other such things.