Comment 21 for bug 896728

Hi,

I have just started to get this issue as well with the latest version (0.7.10-0ubuntu0ppa1240~ubun) of duplicity from the Ubuntu PPA. At first it happened on a full backup but now it's also happened on an incremental backup.

It's a fairly modest size of files to backup (SourceFiles 80196 / SourceFileSize 2583652257 (2.41 GB)) and in the past has only taken just over 1 minute to complete an incremental backup but seems to require a lot of memory.

Is there a way configure duplicity to use less memory but take longer to run the backup?

My server is:

Ubuntu 14.04.5 LTS
Python 2.7.6 (default, Oct 26 2016, 20:30:19)
duplicity 0.7.10

Here is typical memory usage when duplicity is not running:

free -mh
             total used free shared buffers cached
Mem: 615M 555M 60M 102M 23M 287M
-/+ buffers/cache: 244M 371M
Swap: 0B 0B 0B

Here is incremental backup output for the most recent failed backup:

Local and Remote metadata are synchronised, no sync needed.
Last full backup date: Sun Dec 4 01:30:47 2016
No extraneous files found, nothing deleted in cleanup.
Local and Remote metadata are synchronised, no sync needed.
Last full backup date: Sun Dec 4 01:30:47 2016
There are backup set(s) at time(s):
Sun Sep 11 01:30:46 2016
Mon Sep 12 01:30:52 2016
Tue Sep 13 01:30:52 2016
Wed Sep 14 01:30:52 2016
Thu Sep 15 01:30:47 2016
Fri Sep 16 01:30:49 2016
Which can't be deleted because newer sets depend on them.
No old backup sets found, nothing deleted.
Exception in thread Thread-1 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
  File "/usr/lib/python2.7/dist-packages/paramiko/transport.py", line 1613, in run
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'error'
Local and Remote metadata are synchronised, no sync needed.
Last full backup date: Sun Dec 4 01:30:47 2016
Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1546, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1540, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1391, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1521, in do_backup
    check_last_manifest(col_stats) # not needed for full backup
  File "/usr/bin/duplicity", line 1222, in check_last_manifest
    last_backup_set.check_manifests()
  File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 199, in check_manifests
    remote_manifest = self.get_remote_manifest()
  File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 234, in get_remote_manifest
    manifest_buffer = self.backend.get_data(self.remote_manifest_name)
  File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line 677, in get_data
    fin = self.get_fileobj_read(filename, parseresults)
  File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line 671, in get_fileobj_read
    return tdp.filtered_open_with_delete("rb")
  File "/usr/lib/python2.7/dist-packages/duplicity/dup_temp.py", line 119, in filtered_open_with_delete
    fh = FileobjHooked(path.DupPath.filtered_open(self, mode))
  File "/usr/lib/python2.7/dist-packages/duplicity/path.py", line 772, in filtered_open
    return gpg.GPGFile(False, self, gpg_profile)
  File "/usr/lib/python2.7/dist-packages/duplicity/gpg.py", line 182, in __init__
    'logger': self.logger_fp})
  File "/usr/lib/python2.7/dist-packages/duplicity/gpginterface.py", line 374, in run
    create_fhs, attach_fhs)
  File "/usr/lib/python2.7/dist-packages/duplicity/gpginterface.py", line 414, in _attach_fork_exec
    process.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory

Here is typical incremental backup output for a successful backup:

Local and Remote metadata are synchronised, no sync needed.
Last full backup date: Sun Dec 4 01:30:47 2016
No extraneous files found, nothing deleted in cleanup.
Local and Remote metadata are synchronised, no sync needed.
Last full backup date: Sun Dec 4 01:30:47 2016
There are backup set(s) at time(s):
Sun Sep 11 01:30:46 2016
Mon Sep 12 01:30:52 2016
Tue Sep 13 01:30:52 2016
Wed Sep 14 01:30:52 2016
Thu Sep 15 01:30:47 2016
Which can't be deleted because newer sets depend on them.
No old backup sets found, nothing deleted.
Local and Remote metadata are synchronised, no sync needed.
Last full backup date: Sun Dec 4 01:30:47 2016
--------------[ Backup Statistics ]--------------
StartTime 1481725863.33 (Thu Dec 15 01:31:03 2016)
EndTime 1481725934.69 (Thu Dec 15 01:32:14 2016)
ElapsedTime 71.36 (1 minute 11.36 seconds)
SourceFiles 80196
SourceFileSize 2583652257 (2.41 GB)
NewFiles 286
NewFileSize 15810582 (15.1 MB)
DeletedFiles 3
ChangedFiles 35
ChangedFileSize 914609 (893 KB)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 324
RawDeltaSize 15275397 (14.6 MB)
TotalDestinationSizeChange 9606518 (9.16 MB)
Errors 0
-------------------------------------------------