Memory usage of the karma update script foaf-update-karma-cache.py seems excessive

Link to the graph for the click happy - https://lpstats.canonical.com/graphs/KarmaUpdateRSS/

That's kind of expected as the script sucks all rows from the Karma table into memory to later iterate on them.

Are you saying that there isn't a more efficient way of doing this? I don't think this kind of memory usage should be considered "okay" as we don't have unlimited hardware resources.

I'm just stating why the script uses so much memory (not surprisingly)
so that whoever is assigned to fix it knows where to start.

As Salgado stated, this is expected. I believe the scaling issue is growing into a bug. We do not intend to address this at this time, but we need to consider new ways to accomplishing this task. At the January 2009 teamleads sprint, we decided to review how karma work after Launchpad 3.0 is released. Is there are clear performance point to indicate that this bug is high? I would prefer to address this before this becomes critical.

The basic problem is that we have quick a few memory hungry scripts that have to be run at the same time (this, language pack exporter, poimport, etc.) and that means we often get into the situation where we have to kill some jobs to avoid OOM killer kicking in. Assigning priority to reducing memory usage on any of these single scripts is difficult because on their own they're not causing critical problems, but together they are. And it's a problem that only gets worse as Launchpad usage increases - we had hoped to solve this by moving the scripts to a newer and faster server (forster -> loganberry), but we're already finding this isn't enough to fix the issue.

This issue now appear to be high enough to postpone new feature work.

I think we will investigate batching in this script.

2.5 goals will block work on this. It may be addresses in late 2.5, but I cannot commit to it. 2.6 is a realistic goal.