OOPS in MailingListAPIView

Bug #259440 reported by Barry Warsaw
8
Affects Status Importance Assigned to Milestone
Launchpad itself
Fix Released
Critical
Barry Warsaw

Bug Description

Soft and hard timeouts in getMembershipInformation. See OOPS-961S1637 OOPS-961S1641 OOPS-961S1641 OOPS-661S1642 and so on.

Barry Warsaw (barry)
Changed in launchpad:
assignee: nobody → barry
importance: Undecided → Critical
milestone: none → 2.1.8
status: New → Confirmed
Revision history for this message
Barry Warsaw (barry) wrote :

With confirmation from Gustavo, it seems like the .union() call in getSenderAddresses() is ignoring the prejoins. Recommendation is to rewrite the method to use native Storm queries.

Revision history for this message
Barry Warsaw (barry) wrote :

Er, the .union() call in the SQLObject compatibility layer.

Changed in launchpad:
status: Confirmed → Triaged
Revision history for this message
Barry Warsaw (barry) wrote :

I tried and failed to rewrite the queries using the storm native api. The .union() call defeated me. On Francis's suggestion, I used the old SQLObject API and changed the union call to:

team_members.union(approved_posters).prejoin(['person'])

while this doesn't fail, it also doesn't help reduce the query count, as verified by custom tracers that counted the total number of queries.

Given that I've spent a day on this and failed, Francis suggests we leave it until we get OOPSes on production, so I'm bumping down the importance and removing the milestone.

Changed in launchpad:
importance: Critical → Medium
milestone: 2.1.8 → none
Revision history for this message
Diogo Matsubara (matsubara) wrote :

Tom Haddon brought to my attention today that we're generating 6gb/day OOPS reports in the internal xmlrpc server (arsenic). I've updated oops-tools to include the internal xmlrpc OOPSes. We can see by OOPS-969XMLP1805 that this bug occurs on production

Changed in launchpad:
importance: Medium → Critical
Barry Warsaw (barry)
Changed in launchpad:
milestone: none → 2.1.9
status: Triaged → In Progress
Revision history for this message
Christian Reis (kiko) wrote :

Moral of the story: when you see something OOPS on staging, it means it's serious, even if you don't know what exactly is going on!

Barry Warsaw (barry)
Changed in launchpad-foundations:
status: In Progress → Fix Committed
Barry Warsaw (barry)
Changed in launchpad-foundations:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.