get-s3-ci-results inefficiently determines what needs to be scanned

Bug #1604127 reported by Aaron Bentley
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Juju Reports
Triaged
Low
Unassigned

Bug Description

Recently, get-s3-ci-results began failing with:
2016-07-17 23:54:15,659 ERROR [jujureports.jobs] BSON document too large (16782472 bytes) - the connected server supports BSON document sizes up to 16777216 bytes.

I have worked around this by breaking up a single request into multiple requests.
https://code.launchpad.net/~abentley/juju-reports/fix-list-no-match/+merge/300362

However, my workaround is just avoiding the size limitation, not fixing the inefficiency.

We have considered at least two alternatives that would be more efficient:
1. Only consider recent logs, e.g. those in the past 30 days
2. Only consider logs that sync_file considers new. This would be the most efficient, but is not an idempotent operation; sync_file will only ever return a given key once, so robustness can be an issue.

Full traceback:
2016-07-17 23:54:15,659 ERROR [jujureports.jobs] BSON document too large (16782472 bytes) - the connected server supports BSON document sizes up to 16777216 bytes.
Traceback (most recent call last):
  File "/home/ubuntu/juju-reports-repo/juju-reports-332-VxIJlv/local/lib/python2.7/site-packages/jujureports-0.0-py2.7.egg/jujureports/jobs/utils.py", line 208, in log_exceptions
    yield
  File "/home/ubuntu/juju-reports-repo/juju-reports-332-VxIJlv/local/lib/python2.7/site-packages/jujureports-0.0-py2.7.egg/jujureports/jobs/results.py", line 181, in main
    analyse_outcomes(ini, db, artifacts)
  File "/home/ubuntu/juju-reports-repo/juju-reports-332-VxIJlv/local/lib/python2.7/site-packages/jujureports-0.0-py2.7.egg/jujureports/jobs/results.py", line 144, in analyse_outcomes
    for attempt_key in list_no_match(db, attempts_for_logs(artifacts, 2592)):
  File "/home/ubuntu/juju-reports-repo/juju-reports-332-VxIJlv/local/lib/python2.7/site-packages/jujureports-0.0-py2.7.egg/jujureports/models.py", line 300, in list_no_match
    db.outcome_matches.find({'_id': {'$in': id_list}}, fields=[])
  File "/home/ubuntu/juju-reports-repo/juju-reports-332-VxIJlv/local/lib/python2.7/site-packages/jujureports-0.0-py2.7.egg/jujureports/models.py", line 299, in <setcomp>
    AttemptKey(**a['_id']) for a in
  File "/home/ubuntu/juju-reports-repo/juju-reports-332-VxIJlv/local/lib/python2.7/site-packages/pymongo/cursor.py", line 904, in next
    if len(self.__data) or self._refresh():
  File "/home/ubuntu/juju-reports-repo/juju-reports-332-VxIJlv/local/lib/python2.7/site-packages/pymongo/cursor.py", line 848, in _refresh
    self.__uuid_subtype))
  File "/home/ubuntu/juju-reports-repo/juju-reports-332-VxIJlv/local/lib/python2.7/site-packages/pymongo/cursor.py", line 782, in __send_message
    res = client._send_message_with_response(message, **kwargs)
  File "/home/ubuntu/juju-reports-repo/juju-reports-332-VxIJlv/local/lib/python2.7/site-packages/pymongo/mongo_client.py", line 1042, in _send_message_with_response
    response = self.__send_and_receive(message, sock_info)
  File "/home/ubuntu/juju-reports-repo/juju-reports-332-VxIJlv/local/lib/python2.7/site-packages/pymongo/mongo_client.py", line 1017, in __send_and_receive
    (request_id, data) = self.__check_bson_size(message)
  File "/home/ubuntu/juju-reports-repo/juju-reports-332-VxIJlv/local/lib/python2.7/site-packages/pymongo/mongo_client.py", line 929, in __check_bson_size
    (max_doc_size, self.__max_bson_size))
InvalidDocument: BSON document too large (16782472 bytes) - the connected server supports BSON document sizes up to 16777216 bytes.

Changed in juju-reports:
importance: High → Medium
importance: Medium → Low
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.