Vandelay: generating queue summary can bloat cstore memory and CPU usage

Bug #1581552 reported by Galen Charlton
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
Evergreen
New
Undecided
Unassigned

Bug Description

When inspecting a MARC import queue, if the queue has a large number of records, the process of calculating the summary counts (particularly the total number of records and the number of imported records) can cause the cstore backend to bloat.

This is because open-ils.vandelay.bib_queue.summary.retrieve uses makes a couple calls (e.g., open-ils.cstore.direct.vandelay.queued_bib_record.id_list.atomic) to retrieve the list of IDs of records in the queue, then counts then, then tosses the idlist away. When doing this on a queue containing over 100,000 records, this is pretty wasteful:

- it's slow
- it sends a lot of data through XMPP that mostly gets discarded
- it bloats the cstore backends that process the idlist requests

A simple count via json_query would be better.

Evergreen 2.9.1 (and later)

Galen Charlton (gmc)
tags: added: cataloging performance vandelay
Elaine Hardy (ehardy)
tags: added: cat-importexport
removed: cataloging vandelay
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.