Payment notification processor should handle DB connection resets
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Software Center Agent |
Fix Released
|
High
|
Anthony Lenton |
Bug Description
On 7/03 at 9:09AM the prod DB was reset. At that point the payment notification processor service failed with the following error:
2012-07-03 09:09:09,690 - root - ERROR - Process payment notifications failed:
2012-07-03 09:09:09,850 - root - ERROR - Traceback (most recent call last):
2012-07-03 09:09:09,850 - root - ERROR - File "/srv/sc.
yment_notificat
2012-07-03 09:09:09,850 - root - ERROR - call_command(
2012-07-03 09:09:09,850 - root - ERROR - File "/srv/sc.
_init__.py", line 166, in call_command
2012-07-03 09:09:09,850 - root - ERROR - return klass.execute(
2012-07-03 09:09:09,850 - root - ERROR - File "/srv/sc.
ase.py", line 220, in execute
2012-07-03 09:09:09,850 - root - ERROR - output = self.handle(*args, **options)
2012-07-03 09:09:09,851 - root - ERROR - File "/srv/sc.
s/process-
2012-07-03 09:09:09,851 - root - ERROR - include_
2012-07-03 09:09:09,851 - root - ERROR - File "/srv/sc.
2012-07-03 09:09:09,851 - root - ERROR - % subscriptions.
2012-07-03 09:09:09,851 - root - ERROR - File "/srv/sc.
2012-07-03 09:09:09,851 - root - ERROR - return self.query.
2012-07-03 09:09:09,851 - root - ERROR - File "/srv/sc.
2012-07-03 09:09:09,851 - root - ERROR - number = obj.get_
2012-07-03 09:09:09,851 - root - ERROR - File "/srv/sc.
2012-07-03 09:09:09,851 - root - ERROR - result = query.get_
2012-07-03 09:09:09,851 - root - ERROR - File "/srv/sc.
2012-07-03 09:09:09,851 - root - ERROR - cursor.execute(sql, params)
2012-07-03 09:09:09,851 - root - ERROR - File "/srv/sc.
2012-07-03 09:09:09,852 - root - ERROR - return self.cursor.
2012-07-03 09:09:09,852 - root - ERROR - DatabaseError: terminating connection due to administrator command
2012-07-03 09:09:09,852 - root - ERROR - SSL connection has been closed unexpectedly
2012-07-03 09:09:09,852 - root - ERROR -
2012-07-03 09:09:09,852 - root - ERROR -
And continue to fail like that for the rest of the day, until the process was reset due to log rotation.
The payment notification processor should handle DB connection resets. I'll file a separate bug to improve monitoring so that we find out about this kind of issues faster.
Changed in software-center-agent: | |
status: | New → Confirmed |
importance: | Undecided → High |
Changed in software-center-agent: | |
status: | Confirmed → Fix Released |
Changed in software-center-agent: | |
milestone: | none → 12.08 |
Changed in software-center-agent: | |
assignee: | nobody → Anthony Lenton (elachuni) |