ZODB

Clock skew and packing

Bug #143481 reported by Lalo Martins
0
Affects Status Importance Assigned to Milestone
ZODB
New
Low
Unassigned

Bug Description

If you have transactions with a timestamp in the future (probably due to clock skew), they will never be packed away, at least not until you get to that date again.

If you compound that with an application-level bug that for some reason causes the database file to grow too fast, then you'll have a really huge and slow server and no easy way to fix it.

Proposed solution: on ZODB.fspack.GC.buildPackIndex, consider transactions that are in the future as infinitely old. Something like this:

See original description
Tags: bug database
Revision history for this message
Lalo Martins (lalo.martins) wrote :
Revision history for this message
Tim Peters (tim-one) wrote :

There's more about this patch on the zodb-dev list:

http://mail.zope.org/pipermail/zodb-dev/2005-September/009248.html

If the patch works, it might be nice to have an option to explicitly ask for it, but I wouldn't want to assume that "time in the future means infinitely old" by default.

In all, if I had time to look at this, I'd rather spend it on automating a way to force timestamps in badly skewed databases back to sane times.

Revision history for this message
Thomas Güttler (hv-tbz-pariv) wrote :

Uploaded: BaseStorage_reset_tids.patch

I had the problem, too. The clock was set to the year 2009 for very short, but now all new tids are from 2009.

Tim Peter suggested, that you can solve the problem by doing
a storage.copyTransactionsFrom by setting the tid to None.
See bottom of:
http://mail.zope.org/pipermail/zodb-dev/2005-September/009225.html

This is a small path to BaseStorage with gives the method copyTransactionsFrom a new keyword argument "reset_tids".

If you use the method once, all tids are reset to the current time.
This means, you don't need the patch zodb-pack-future.patch anymore. I prefere this, because you only need to repair the database once, while other patch need to installed for ever.

Tres Seaver (tseaver)
affects: zope2 → zodb
Revision history for this message
Jim Fulton (jim-zope) wrote :

A better patch would set tids > now to about now, making sure their order is preserved.

A better patch would also have a test. :)

Doing this as part of pack has the operational advantage that it doesn't require down time.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.