get-s3-ci-results inefficiently determines what needs to be scanned
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Juju Reports |
Triaged
|
Low
|
Unassigned |
Bug Description
Recently, get-s3-ci-results began failing with:
2016-07-17 23:54:15,659 ERROR [jujureports.jobs] BSON document too large (16782472 bytes) - the connected server supports BSON document sizes up to 16777216 bytes.
I have worked around this by breaking up a single request into multiple requests.
https:/
However, my workaround is just avoiding the size limitation, not fixing the inefficiency.
We have considered at least two alternatives that would be more efficient:
1. Only consider recent logs, e.g. those in the past 30 days
2. Only consider logs that sync_file considers new. This would be the most efficient, but is not an idempotent operation; sync_file will only ever return a given key once, so robustness can be an issue.
Full traceback:
2016-07-17 23:54:15,659 ERROR [jujureports.jobs] BSON document too large (16782472 bytes) - the connected server supports BSON document sizes up to 16777216 bytes.
Traceback (most recent call last):
File "/home/
yield
File "/home/
analyse_
File "/home/
for attempt_key in list_no_match(db, attempts_
File "/home/
db.
File "/home/
AttemptKey(
File "/home/
if len(self.__data) or self._refresh():
File "/home/
self.
File "/home/
res = client.
File "/home/
response = self.__
File "/home/
(request_id, data) = self.__
File "/home/
(max_doc_size, self.__
InvalidDocument: BSON document too large (16782472 bytes) - the connected server supports BSON document sizes up to 16777216 bytes.
Changed in juju-reports: | |
importance: | High → Medium |
importance: | Medium → Low |