Record memory usage information when branch puller exceeds memory limits

Bug #390443 reported by Jonathan Lange
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Launchpad itself
Triaged
Low
Unassigned

Bug Description

We've recently had problems with the branch puller using way too much memory. Although we've worked around these problems by providing rlimits, it would be nice to dig at the underlying causes. Here's one way of doing that:

  - Define a "soft" memory limit that is below the rlimit but still high.
  - On each activity event in the puller worker, check memory usage to see that it's below that soft limit
  - If it exceeds that soft limit, use py_memory_dump to get all the information we can about the current memory usage dumped to a file.
  - Trigger an OOPS, terminating the worker process.
  - Make sure the OOPS report includes a pointer to this file, and that the file is synced to devpad.

It's possible that some of the code written for this could be generically useful as a tool to debug extreme memory usage. If so, it should perhaps be incorporated into a hypothetical lazr.oops. However, in the meantime, let's just do this for the puller.

Revision history for this message
Robert Collins (lifeless) wrote :

A simpler way would be to configure core files and examine the core file as a post mortem.

Changed in launchpad:
importance: Medium → Low
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.