Finish jsonindex Phase One

Bug #1650315 reported by Paul Everitt
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
KARL4
Fix Released
Medium
Jim Fulton

Bug Description

1) Finish the pickle->jsonb conversion

(WAG 8 hours)

There are some outstanding changes that need to be integrated with master, with test changes and additions including:

- Bug fixes

- Using integers in persistent references for better interoperability with RelStorage

- Avoiding but allowing for (in case of actual cycles) pickle-internal references.

- Investigate unicode issue

2) Asynchronous update of JSON data

- Avoids need to update RelStorage now

- Dedicated table with columns:
  * zoid: bigint
  * class_name: text
  * ghost_pickle: bytea
  * state: jsonb

- Process that polls notifications (with timeouts) to keep json up to date

3) Search API

Use dedicated PG connection for now, but would be interesting to use RelStorage read connection
API returns iterator
Buffered to allow prefetch in advance of iterator
Creates ghosts in object cache and prefetches them as iteration proceeds. If you only iterate over the first N of a many objects, you’ll only create N+a few objects.
The column ghost_pickle above allows search results to contain enough data to create ghosts.
If there’s time, implement prefetch for RelStorage
Requires a very small change to ZODB. Connection.get grows an optional argument supplying a ghost pickle (class name and __new__ args, if any) so that get can create a ghost without a database load.

Tags: jsonindex
Changed in karl4:
milestone: 027 → 026
Revision history for this message
Paul Everitt (paul-agendaless) wrote :

Jim, do we need to keep this ticket open?

Changed in karl4:
milestone: 026 → 027
Revision history for this message
Jim Fulton (jim-zope) wrote :

Nope.

BTW, I did end up using the RelStorage's load connection. (I got the relevant monkey patch reviewed by Jason.)

Changed in karl4:
status: New → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.