Payment notification processor should handle DB connection resets

Bug #1030019 reported by Anthony Lenton
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Software Center Agent
Fix Released
High
Anthony Lenton

Bug Description

On 7/03 at 9:09AM the prod DB was reset. At that point the payment notification processor service failed with the following error:

2012-07-03 09:09:09,690 - root - ERROR - Process payment notifications failed:
2012-07-03 09:09:09,850 - root - ERROR - Traceback (most recent call last):
2012-07-03 09:09:09,850 - root - ERROR - File "/srv/sc.ubuntu.com/production/software-center-agent/src/softwarecenteragent/scripts/process_pa
yment_notifications", line 37, in <module>
2012-07-03 09:09:09,850 - root - ERROR - call_command('process-payment-notifications')
2012-07-03 09:09:09,850 - root - ERROR - File "/srv/sc.ubuntu.com/production/software-center-agent/sourcecode/django/django/core/management/_
_init__.py", line 166, in call_command
2012-07-03 09:09:09,850 - root - ERROR - return klass.execute(*args, **defaults)
2012-07-03 09:09:09,850 - root - ERROR - File "/srv/sc.ubuntu.com/production/software-center-agent/sourcecode/django/django/core/management/b
ase.py", line 220, in execute
2012-07-03 09:09:09,850 - root - ERROR - output = self.handle(*args, **options)
2012-07-03 09:09:09,851 - root - ERROR - File "/srv/sc.ubuntu.com/production/software-center-agent/src/softwarecenteragent/management/command
s/process-payment-notifications.py", line 35, in handle
2012-07-03 09:09:09,851 - root - ERROR - include_initiated=include_initiated)
2012-07-03 09:09:09,851 - root - ERROR - File "/srv/sc.ubuntu.com/production/software-center-agent/src/softwarecenteragent/managers.py", line 176, in process_payment_notifications
2012-07-03 09:09:09,851 - root - ERROR - % subscriptions.count())
2012-07-03 09:09:09,851 - root - ERROR - File "/srv/sc.ubuntu.com/production/software-center-agent/sourcecode/django/django/db/models/query.py", line 334, in count
2012-07-03 09:09:09,851 - root - ERROR - return self.query.get_count(using=self.db)
2012-07-03 09:09:09,851 - root - ERROR - File "/srv/sc.ubuntu.com/production/software-center-agent/sourcecode/django/django/db/models/sql/query.py", line 401, in get_count
2012-07-03 09:09:09,851 - root - ERROR - number = obj.get_aggregation(using=using)[None]
2012-07-03 09:09:09,851 - root - ERROR - File "/srv/sc.ubuntu.com/production/software-center-agent/sourcecode/django/django/db/models/sql/query.py", line 367, in get_aggregation
2012-07-03 09:09:09,851 - root - ERROR - result = query.get_compiler(using).execute_sql(SINGLE)
2012-07-03 09:09:09,851 - root - ERROR - File "/srv/sc.ubuntu.com/production/software-center-agent/sourcecode/django/django/db/models/sql/compiler.py", line 735, in execute_sql
2012-07-03 09:09:09,851 - root - ERROR - cursor.execute(sql, params)
2012-07-03 09:09:09,851 - root - ERROR - File "/srv/sc.ubuntu.com/production/software-center-agent/sourcecode/django/django/db/backends/postgresql_psycopg2/base.py", line 44, in execute
2012-07-03 09:09:09,852 - root - ERROR - return self.cursor.execute(query, args)
2012-07-03 09:09:09,852 - root - ERROR - DatabaseError: terminating connection due to administrator command
2012-07-03 09:09:09,852 - root - ERROR - SSL connection has been closed unexpectedly
2012-07-03 09:09:09,852 - root - ERROR -
2012-07-03 09:09:09,852 - root - ERROR -

And continue to fail like that for the rest of the day, until the process was reset due to log rotation.

The payment notification processor should handle DB connection resets. I'll file a separate bug to improve monitoring so that we find out about this kind of issues faster.

Changed in software-center-agent:
status: New → Confirmed
importance: Undecided → High
Dave Morley (davmor2)
Changed in software-center-agent:
status: Confirmed → Fix Released
Dave Morley (davmor2)
Changed in software-center-agent:
milestone: none → 12.08
Changed in software-center-agent:
assignee: nobody → Anthony Lenton (elachuni)
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.