Finish jsonindex Phase One
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
KARL4 |
Fix Released
|
Medium
|
Jim Fulton |
Bug Description
1) Finish the pickle->jsonb conversion
(WAG 8 hours)
There are some outstanding changes that need to be integrated with master, with test changes and additions including:
- Bug fixes
- Using integers in persistent references for better interoperability with RelStorage
- Avoiding but allowing for (in case of actual cycles) pickle-internal references.
- Investigate unicode issue
2) Asynchronous update of JSON data
- Avoids need to update RelStorage now
- Dedicated table with columns:
* zoid: bigint
* class_name: text
* ghost_pickle: bytea
* state: jsonb
- Process that polls notifications (with timeouts) to keep json up to date
3) Search API
Use dedicated PG connection for now, but would be interesting to use RelStorage read connection
API returns iterator
Buffered to allow prefetch in advance of iterator
Creates ghosts in object cache and prefetches them as iteration proceeds. If you only iterate over the first N of a many objects, you’ll only create N+a few objects.
The column ghost_pickle above allows search results to contain enough data to create ghosts.
If there’s time, implement prefetch for RelStorage
Requires a very small change to ZODB. Connection.get grows an optional argument supplying a ghost pickle (class name and __new__ args, if any) so that get can create a ghost without a database load.
Changed in karl4: | |
milestone: | 027 → 026 |
Changed in karl4: | |
status: | New → Fix Released |
Jim, do we need to keep this ticket open?