crash using hashed oops id as appinstance

Bug #889982 reported by Robert Collins
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
python-oops-tools
Fix Released
Critical
Robert Collins

Bug Description

Traceback (most recent call last):
  File "bin/amqp2disk", line 49, in <module>
    oopstools.scripts.amqp2disk.main()
  File "/srv/lp-oops.canonical.com/cgi-bin/lpoops/src/oopstools/scripts/amqp2disk.py", line 113, in main
    receiver.run_forever()
  File "/srv/lp-oops.canonical.com/cgi-bin/lpoops/eggs/oops_amqp-0.0.4-py2.6.egg/oops_amqp/receiver.py", line 86, in run_forever
    self._run_forever()
  File "/srv/lp-oops.canonical.com/cgi-bin/lpoops/eggs/oops_amqp-0.0.4-py2.6.egg/oops_amqp/receiver.py", line 108, in _run_forever
    self.channel.wait()
  File "/srv/lp-oops.canonical.com/cgi-bin/lpoops/eggs/amqplib-0.6.1-py2.6.egg/amqplib/client_0_8/abstract_channel.py", line 107, in wait
    return amqp_method(self, args, content)
  File "/srv/lp-oops.canonical.com/cgi-bin/lpoops/eggs/amqplib-0.6.1-py2.6.egg/amqplib/client_0_8/channel.py", line 1973, in _basic_deliver
    func(msg)
  File "/srv/lp-oops.canonical.com/cgi-bin/lpoops/eggs/oops_amqp-0.0.4-py2.6.egg/oops_amqp/receiver.py", line 71, in handle_report
    self.config.publish(report)
  File "/srv/lp-oops.canonical.com/cgi-bin/lpoops/eggs/oops-0.0.9-py2.6.egg/oops/config.py", line 184, in publish
    id = publisher(report)
  File "/srv/lp-oops.canonical.com/cgi-bin/lpoops/src/oopstools/scripts/amqp2disk.py", line 129, in db_publisher
    report, report['datedir_repo_filepath'])
  File "/srv/lp-oops.canonical.com/cgi-bin/lpoops/src/oopstools/oops/models.py", line 441, in parsed_oops_to_model_oops
    data, req_vars, statements, traceback = _get_oops_tuple(parsed_oops)
  File "/srv/lp-oops.canonical.com/cgi-bin/lpoops/src/oopstools/oops/models.py", line 300, in _get_oops_tuple
    prefix.save()
  File "/srv/lp-oops.canonical.com/cgi-bin/lpoops/parts/django/django/db/models/base.py", line 460, in save
    self.save_base(using=using, force_insert=force_insert, force_update=force_update)
  File "/srv/lp-oops.canonical.com/cgi-bin/lpoops/parts/django/django/db/models/base.py", line 553, in save_base
    result = manager._insert(values, return_id=update_pk, using=using)
  File "/srv/lp-oops.canonical.com/cgi-bin/lpoops/parts/django/django/db/models/manager.py", line 195, in _insert
    return insert_query(self.model, values, **kwargs)
  File "/srv/lp-oops.canonical.com/cgi-bin/lpoops/parts/django/django/db/models/query.py", line 1436, in insert_query
    return query.get_compiler(using=using).execute_sql(return_id)
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/srv/lp-oops.canonical.com/cgi-bin/lpoops/parts/django/django/db/models/sql/compiler.py", line 791, in execute_sql
    cursor = super(SQLInsertCompiler, self).execute_sql(None)
  File "/srv/lp-oops.canonical.com/cgi-bin/lpoops/parts/django/django/db/models/sql/compiler.py", line 735, in execute_sql
    cursor.execute(sql, params)
  File "/srv/lp-oops.canonical.com/cgi-bin/lpoops/parts/django/django/db/backends/postgresql_psycopg2/base.py", line 44, in execute
    return self.cursor.execute(query, args)
django.db.utils.DatabaseError: value too long for type character varying(20)

Related branches

Revision history for this message
Robert Collins (lifeless) wrote :

I patched the orm to get this out:
SELECT "oops_prefix"."id", "oops_prefix"."value", "oops_prefix"."appinstance_id" FROM "oops_prefix" WHERE "oops_prefix"."value" = %s ('F5054932567230492013CE',)
SELECT "oops_appinstance"."id", "oops_appinstance"."title" FROM "oops_appinstance" WHERE "oops_appinstance"."title" = %s ('F5054932567230492013CE',)
INSERT INTO "oops_prefix" ("value", "appinstance_id") VALUES (%s, %s) ('F5054932567230492013CE', 44)

So it looks like the schema needs a tweak to support appinstances that are not noddily short.

This is the file on disk (different because of different python dictionary order; may want to tweak that)
grep -i F5054932567230492013CE -r /srv/oops-amqp/staging/
/srv/oops-amqp/staging/2011-11-09/OOPS-914c6f3d5bba6a35f70d10c60f21d0d7

>>> import bson
>>> content = file('/srv/oops-amqp/staging/2011-11-09/OOPS-914c6f3d5bba6a35f70d10c60f21d0d7', 'rb').read()
>>> d = bson.loads(content)
>>> d.keys()
[u'revno', u'req_vars', u'reporter', u'url', u'branch_nick', u'timeline', u'hostname', u'value', u'ignore', u'time', u'duration', u'type', u'id', u'tb_text']
>>> d['reporter']
>>> d['hostname']
'tellurium'
u'req_vars': [['<oops-message-268>', "Failure: error(111, 'Connection refused')"], ['path', '/srv/bazaar.staging.launchpad.net/staging/launchpad/cronscripts/translations-export-to-branch.py'], ['script_name', 'translations-export-to-branch']],

So this is a misconfigured/oddly configured translations script, trying to dynamically allocate an appserver instance when no sensible reporter exists.

I think we should change the code to not use the oops id, but instead use the hostname or something - or just 'unknown' - its less likely to proliferate.

summary: - crash inserting report into DB
+ crash using hashed oops id as appinstance
Revision history for this message
Robert Collins (lifeless) wrote :

F5054932567230492013CE happens to match the oops tools oops re:
[a-z]+\d+[a-z]+

this was intended to match e.g. LPNET123465SUFFIXA

:(.

We've deployed the new code base everywhere, so everything should be using reporter in LP, but U1 haven't yet.

As a workaround, I'm going to hardcode in a 'looks like a hash' check (on length) and coerce that to UNKNOWN.

Revision history for this message
Robert Collins (lifeless) wrote :

beware assumptions -
>>> d['id']
'OOPS-68383510f5054932567230492013ce90'

Revision history for this message
Robert Collins (lifeless) wrote :

(which is the full id - the cutoff bit that matches the regex is fromthe 10th char on).

Changed in python-oops-tools:
status: Triaged → Fix Committed
assignee: nobody → Robert Collins (lifeless)
Revision history for this message
Robert Collins (lifeless) wrote :

This is deployed on the LP oops-tools instance.

Changed in python-oops-tools:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.