The basic problem is that we have quick a few memory hungry scripts that have to be run at the same time (this, language pack exporter, poimport, etc.) and that means we often get into the situation where we have to kill some jobs to avoid OOM killer kicking in. Assigning priority to reducing memory usage on any of these single scripts is difficult because on their own they're not causing critical problems, but together they are. And it's a problem that only gets worse as Launchpad usage increases - we had hoped to solve this by moving the scripts to a newer and faster server (forster -> loganberry), but we're already finding this isn't enough to fix the issue.
The basic problem is that we have quick a few memory hungry scripts that have to be run at the same time (this, language pack exporter, poimport, etc.) and that means we often get into the situation where we have to kill some jobs to avoid OOM killer kicking in. Assigning priority to reducing memory usage on any of these single scripts is difficult because on their own they're not causing critical problems, but together they are. And it's a problem that only gets worse as Launchpad usage increases - we had hoped to solve this by moving the scripts to a newer and faster server (forster -> loganberry), but we're already finding this isn't enough to fix the issue.