Testrepository

times.dbm incompatible based on Python version it was created with

Bug #1212909 reported by clayg
52
This bug affects 12 people
Affects Status Importance Assigned to Milestone
Testrepository
Triaged
Critical
Unassigned

Bug Description

On my deadsnakes [1] python2.6I get can't get my py26 environment to run.

    py26 runtests: commands[0] | python setup.py testr --testr-args=
    running testr
    running=${PYTHON:-python} -m subunit.run discover -t ./ ./tests
    No module named _bsddb
    error: testr failed (3)
    ERROR: InvocationError: '/home/clayg/.python-swiftclient/.tox/py26/bin/python setup.py testr --testr-args='
    ERROR: py26: commands failed

What seems to be going on is just that using anydbm with the stock system python (py27) is getting dbhash and it works just fine, then when the py26 env loops around and uses anydbm it inspects the db and tries to open it with bsddb even though it's anydbm._defaultmod is dmb (or gdbm w/e depending on the ImportError). I mean it's easy to say deadsnakes is just broken, but it's sort of a great way to get you some python2.6 on an ubuntu system I wonder if you might just work around it by avoiding bsddb explicitly?

I mean, it's easy to get wrong, even macosx has a broken bsddb [2]:

    clayg:~/Workspace/swift$ python -c 'import sys; print sys.version; import bsddb'
    2.7.2 (default, Oct 11 2012, 20:14:37)
    [GCC 4.2.1 Compatible Apple Clang 4.0 (tags/Apple/clang-418.0.60)]
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/bsddb/__init__.py", line 67, in <module>
        import _bsddb
    ImportError: No module named _bsddb

I tried to compile python2.6 with myself with "bsddb185 bsddbmodule.c", but it's kind of a chore to get the bsddb-dev 4.0 includes these days cause everything is deprecated :\

Don't tell anyone, but what I ended up doing was just hacking my local system anydb._names to just not look at hashdb, I think other folks just hack hashdb to use bsddb3 [3]

1. https://launchpad.net/~fkrull/+archive/deadsnakes
2. http://stackoverflow.com/questions/814041/how-to-fix-the-broken-bsddb-install-in-the-default-python-package-on-mac-os-x-10
3. http://marc-abramowitz.com/archives/2007/11/28/hacking-os-xs-python-dbhash-and-bsddb-modules-to-work/

Revision history for this message
clayg (clay-gerrard) wrote :
Revision history for this message
Robert Collins (lifeless) wrote :

You can get a backtrace with
TESTR_PDB=1 testr run

Revision history for this message
Robert Collins (lifeless) wrote :

That, said, there is also bug 1229445 and I think this is arguably a dupe.

Could you check on your system, does
dbm (in python 2.x)
and dbm.ndbm (in python3.x)
work?

Thanks!

Changed in testrepository:
status: New → Triaged
importance: Undecided → Critical
Revision history for this message
John Garbutt (johngarbutt) wrote :

I was seeing this same error, but in a different way, and managed to fix it up. Basically default python27 in the mac seems broken.

On my mac, I saw this because I used port to install python27 but not virtualenv, installing the port version of virtualenv brought the fix into my virtualenv when running ./run-tests.sh

Interestingly tox correctly picked up the port install of python27, so never showed this issue.

Revision history for this message
Robert Collins (lifeless) wrote :

So, I don't have a mac - I really need guidance on where to take this :(

Revision history for this message
clayg (clay-gerrard) wrote :

well, i ran into this again in a different context coming at it from https://bugs.launchpad.net/testrepository/+bug/1229445 this time... In all fairness I wonder if some of this really should just be filed against openstack - since we're the ones making all the noise.

I think maybe instead of anydbm if testrepository/repository/file.py explicitly chose a specific format it thinks is going to be well supported across various builds of 2.6 - 3.4 when creating a db it might have a better time of it.

For existing db's I guess you just open it with anydb and get whatever is in there. But when making the db the first time (or after someone deleted it trying to dig themselves out of this mess) it seems like just picking whatever python's happen to be running first's default isn't always the most widely supported choice.

OTOH, maybe tox should be isolating this better? Is there a good reason to support sharing this db across runs of different virtualenv's with different python interpreters?

my launchpad says i'm subscribed to this bug - so you if you need any more information or debugging from my end - maybe I'll see the next update.

Revision history for this message
Robert Collins (lifeless) wrote :

So the problem is that I tried that. I couldn't find a db format supported across Windows MacOSX Linux 2.x and 3.x when I last looked into this.

I would *totally* accept a patch to default to such a format if there is one.

Some prior context, but sadly I can't find the prior bug about this specific incompat issue.

See https://bugs.launchpad.net/testrepository/+bug/1091500
https://bugs.launchpad.net/testrepository/+bug/1094330

Robert Collins (lifeless)
summary: - ImportError _bsddb with deadsnakes python2.6 anydbm
+ times.dbm incompatible based on Python version it was created with
Revision history for this message
Morgan Fainberg (mdrnstm) wrote :

Can we simply make the timing db (and the output) python-version specific? e.g. py2.7 gets a py27 dir, a py3.4 gets a py34 dir. It doesn't have to work across platforms just isolate per python-version.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.