duplicity 0.6.16 lists all backup sets as incomplete

Added the output of duplicity collection-status after downgrading to duplicity 0.6.15. Now there are no incomplete backup sets and all backup sets contain volumes.

It seems I omitted some basic information in the original post.

Distribution: Ubuntu 10.04.3 LTS
Platform: x86-64
Package version: 0.6.16-0ubuntu0ppa6~lucid1

Options used when invoking duplicity: --full-if-older-than 6M --volsize 1024 --asynchronous-upload --exclude-globbing-filelist [filelist name] --archive-dir [archive directory] --exclude [archive directory] --name [name] --no-encryption --exclude-other-filesystems

The problem seems to be reproducible easily:

megari@miyoko:/tmp$ mkdir duplicity-test
megari@miyoko:/tmp$ cd duplicity-test/
megari@miyoko:/tmp/duplicity-test$ ls
megari@miyoko:/tmp/duplicity-test$ mkdir cache
megari@miyoko:/tmp/duplicity-test$ mkdir a
megari@miyoko:/tmp/duplicity-test$ mkdir b
megari@miyoko:/tmp/duplicity-test$ cat > a/foo.txt
Hello!
megari@miyoko:/tmp/duplicity-test$ duplicity --archive-dir=/tmp/duplicity-test/cache --name test --no-encryption a file:///tmp/duplicity-test/b
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: none
No signatures found, switching to full backup.
Warning, found incomplete backup sets, probably left from aborted session
--------------[ Backup Statistics ]--------------
StartTime 1319589284.00 (Wed Oct 26 03:34:43 2011)
EndTime 1319589284.02 (Wed Oct 26 03:34:44 2011)
ElapsedTime 0.03 (0.03 seconds)
SourceFiles 2
SourceFileSize 4103 (4.01 KB)
NewFiles 2
NewFileSize 4103 (4.01 KB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 2
RawDeltaSize 7 (7 bytes)
TotalDestinationSizeChange 150 (150 bytes)
Errors 0
-------------------------------------------------

megari@miyoko:/tmp/duplicity-test$ duplicity collection-status --archive-dir=/tmp/duplicity-test/cache --name test --no-encryption file:///tmp/duplicity-test/bLocal and Remote metadata are synchronized, no sync needed.
Warning, found incomplete backup sets, probably left from aborted session
Last full backup date: Wed Oct 26 03:34:43 2011
Collection Status
-----------------
Connecting with backend: LocalBackend
Archive dir: /tmp/duplicity-test/cache/test

Found 0 secondary backup chains.

Found primary backup chain with matching signature chain:
-------------------------
Chain start time: Wed Oct 26 03:34:43 2011
Chain end time: Wed Oct 26 03:34:43 2011
Number of contained backup sets: 1
Total number of contained volumes: 0
 Type of backup set: Time: Num volumes:
-------------------------
Also found 0 backup sets not part of any chain,
and 1 incomplete backup set.
These may be deleted by running duplicity with the "cleanup" command.
megari@miyoko:/tmp/duplicity-test$ nano a/foo.txt
megari@miyoko:/tmp/duplicity-test$ duplicity --archive-dir=/tmp/duplicity-test/cache --name test --no-encryption a file:///tmp/duplicity-test/b
Local and Remote metadata are synchronized, no sync needed.
Warning, found incomplete backup sets, probably left from aborted session
Last full backup date: Wed Oct 26 03:34:43 2011
--------------[ Backup Statistics ]--------------
StartTime 1319589353.91 (Wed Oct 26 03:35:53 2011)
EndTime 1319589353.92 (Wed Oct 26 03:35:53 2011)
ElapsedTime 0.00 (0.00 seconds)
SourceFiles 2
SourceFileSize 4116 (4.02 KB)
NewFiles 0
NewFileSize 0 (0 bytes)
DeletedFiles 0
ChangedFiles 1
ChangedFileSize 20 (20 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 1
RawDeltaSize 27 (27 bytes)
TotalDestinationSizeChange 128 (128 bytes)
Errors 0
-------------------------------------------------

megari@miyoko:/tmp/duplicity-test$ duplicity collection-status --archive-dir=/tmp/duplicity-test/cache --name test --no-encryption file:///tmp/dupl...

Furthermore, it looks like duplicity 0.6.16 fails to restore from backup even in the simple scenario above.

I reported this behavior under bug 498933, but it's exactly what is presented here and megari has presented much fuller information. I'll post something similar there - not sure if it's actually related or not.

Reproduced the failure(s) in the simple testcase using the source tarball (duplicity-0.6.16.tar.gz) on Fedora 13 i686, so the problem doesn't seem to be specific to the PPA version (0.6.16-0ubuntu0ppa6~lucid1).

Also, as 0.6.16 fails even with backups created by itself, it doesn't seem like this is caused by a buggy old version producing invalid backup chains.

Thank you for the bug report. I'm looking into the issue now.

Any update regarding this? The nature of the bug is such that I can't be sure if I can safely run my regular backups. Is there a chance of corruption if I make backups with 0.6.16?

Status changed to 'Confirmed' because the bug affects multiple users.

Big "me too" here. Blew several hours trying to debug this, thinking it was a cygwin largefiles issue.

I get similar behavior with Ubuntu 10.04; with duplicity PPA; which would have brought in the newer duplicity code.

Oh, I use Deja-dup and get the recurring full-backups and cannot do Restores with message "Restore Failed"

Similar behavior on Ubuntu 10.04 and Deja-dup with Duplicity PPA enabled. Also, if I do succeed in getting a message of a complete backup; I am not able to do any restores; message is always "Restore Failed".

I am seeing messages like this in duplicity's -v9 output:

Ignoring incremental Backupset (start_time: Mon Aug 8 17:17:14 2011; needed: Mon Aug 8 02:51:33 2011)
Ignoring incremental Backupset (start_time: Mon Aug 15 01:03:29 2011; needed: Mon Aug 8 02:51:33 2011)
Ignoring incremental Backupset (start_time: Mon Aug 15 03:20:23 2011; needed: Mon Aug 8 02:51:33 2011)
Ignoring incremental Backupset (start_time: Mon Aug 22 03:10:32 2011; needed: Mon Aug 8 02:51:33 2011)
Ignoring incremental Backupset (start_time: Mon Aug 29 02:42:40 2011; needed: Mon Aug 8 02:51:33 2011)
Ignoring incremental Backupset (start_time: Tue Sep 13 04:00:14 2011; needed: Mon Aug 8 02:51:33 2011)
Ignoring incremental Backupset (start_time: Mon Sep 19 03:11:19 2011; needed: Mon Aug 8 02:51:33 2011)
Ignoring incremental Backupset (start_time: Sun Sep 25 23:39:14 2011; needed: Mon Aug 8 02:51:33 2011)
Ignoring incremental Backupset (start_time: Mon Oct 3 01:51:53 2011; needed: Mon Aug 8 02:51:33 2011)
Ignoring incremental Backupset (start_time: Mon Oct 10 00:50:22 2011; needed: Mon Aug 8 02:51:33 2011)
Ignoring incremental Backupset (start_time: Mon Oct 17 03:48:49 2011; needed: Mon Aug 8 02:51:33 2011)

Bug #885670 might be the same as this one.

I also confirm what Mike O'Donnell said: restoring always fails (with a KeyError).