OSError Cannot Allocate Memory GPG

Bug #896728 reported by bongo
74
This bug affects 14 people
Affects Status Importance Assigned to Milestone
Duplicity
Fix Released
Medium
Unassigned
Debian
Fix Released
Unknown

Bug Description

I use the deja-dup frontend for backing up my homedir. When starting the first backup over ssh, the following error occurs while the system is in no way out of ram/swap (>2G free).
(This is _NOT_ a duplicate of bug 780270! The person who filed the latter one was able to do backups, I am not even able to finish a backup)

Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1347, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1330, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1300, in main
    full_backup(col_stats)
  File "/usr/bin/duplicity", line 469, in full_backup
    globals.backend)
  File "/usr/bin/duplicity", line 347, in write_multivol
    globals.gpg_profile, globals.volsize)
  File "/usr/lib/python2.7/site-packages/duplicity/gpg.py", line 313, in GPGWriteFile
    file = GPGFile(True, path.Path(filename), profile)
  File "/usr/lib/python2.7/site-packages/duplicity/gpg.py", line 143, in __init__
    'logger': self.logger_fp})
  File "/usr/lib/python2.7/site-packages/duplicity/GnuPGInterface.py", line 365, in run
    create_fhs, attach_fhs)
  File "/usr/lib/python2.7/site-packages/duplicity/GnuPGInterface.py", line 407, in _attach_fork_exec
    process.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory

The used version of duplicity is 0.6.17 on ArchLinux.´
Python 3.2.2; 2.7.2 (installed as python and python2)

Revision history for this message
Chris McGinley (cemcginley) wrote :

I am also seeing this error and it is in fact consuming all available physical memory + my 2GB pagefile before it fails. Typically this happens if I have a virtual machine running as well as other apps when the backup is kicked off (i.e., a good amount of memory already in use).

Here are details about my install:
  Gentoo Linux - kernel 3.1.2
  Python 2.7.2
  GnuPG 2.0.18
  duplicity 0.6.16

I really like duplicity and all that it does, but this issue has prevented me from completely automating my backups and could mean an alternative solution is necessary.

Revision history for this message
bongo (bong0) wrote :

I could sort of backtrace the error down to a group of rather big files (8Gib-10Gib VMs) which I excluded for testing. With this exclusion, backup works, but I am still wondering why It seems that either the python gpg bindings (implementation?) or duplicity cache the file in memory (for no reason?). Using the gpg commandline tool this has never happened to me, so IMHO we can say It isn't essential.

Revision history for this message
Chris McGinley (cemcginley) wrote :

Contrary to your findings, I only use the duplicity command line and it happens to me. I haven't put enough research in to say if I could tie it to big files or anything...but, I know that I definitely have some big files (VM snapshots) that are included during my backup process...though, according to verbose logs, duplicity already stored those off and is on to other smaller files when the memory is consumed.

Revision history for this message
bongo (bong0) wrote :

The problem persists despite my last impressions: Here's a longer backlog with verbosity=5:

Error accessing possibly locked file /lost+found
Error accessing possibly locked file /root
Cleanup of temporary file /home/$USER/.cache/deja-dup/b46a916b73394f83c5a77cd450ead9b9/duplicity-u2A9yE-tempdir/mktemp-tzl_EL-1 failed
Cleanup of temporary file /home/$USER/.cache/deja-dup/b46a916b73394f83c5a77cd450ead9b9/duplicity-8Ihkbv-tempdir/mktemp-P253Cq-1 failed
AsyncScheduler: running task synchronously (asynchronicity disabled)
Writing sftp://backup@HOST/mnt/backup/duplicity-inc.20111212T004634Z.to.20111212T182905Z.vol1.difftar.gpg
Deleting /tmp/duplicity-TlYUdU-tempdir/mktemp-3qPCXA-3
AsyncScheduler: task completed successfully
Processed volume 1
Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1377, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1370, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1352, in main
    incremental_backup(sig_chain)
  File "/usr/bin/duplicity", line 575, in incremental_backup
    new_sig_outfp.to_remote()
  File "/usr/lib/python2.7/site-packages/duplicity/dup_temp.py", line 181, in to_remote
    gpg.GPGWriteFile(src_iter, tgt.name, globals.gpg_profile, size = sys.maxint)
  File "/usr/lib/python2.7/site-packages/duplicity/gpg.py", line 313, in GPGWriteFile
    file = GPGFile(True, path.Path(filename), profile)
  File "/usr/lib/python2.7/site-packages/duplicity/gpg.py", line 143, in __init__
    'logger': self.logger_fp})
  File "/usr/lib/python2.7/site-packages/duplicity/GnuPGInterface.py", line 365, in run
    create_fhs, attach_fhs)
  File "/usr/lib/python2.7/site-packages/duplicity/GnuPGInterface.py", line 407, in _attach_fork_exec
    process.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory

Revision history for this message
Peter Cech (cechptr) wrote :

I upgraded from 0.6.14 to 0.6.16 and then 0.6.17 and noticed that the new version requires much more RAM than the old one. CPU load increased as well. The backup slowed down due to constant swapping, with full backup taking almost whole day instead of few hours. I tried to replace

/usr/share/pyshared/duplicity/tarfile.py

with the tarfile.py from version 0.6.14 and it returned the RAM consumption to previous levels. Could you try the same and see if it helps?

Revision history for this message
Kenneth Loafman (kenneth-loafman) wrote :

When this happens, what does the process list look like just before? Are there a bunch of extra GPG processes present? These should max out at 3 or 4 at the most.

Revision history for this message
bongo (bong0) wrote :

@Kenneth At least for now, I am unable to monitor my process list changes during the time of about 3h - 4h. I am in need of a tool which would help me doing that. If you know a solution for that problem, please post it.

Changed in debian:
status: Unknown → Confirmed
Revision history for this message
Olivier Berger (olivierberger) wrote :

To me this looks very much similar to #908228 also

Changed in debian:
status: Confirmed → Fix Released
Revision history for this message
Bradley Paul (bradhuh91) wrote :

Still getting the OSError. My logs are much the same and I got this error with duplicity v**18 since I'm running that today and I haven't seen it pop up on an update since at least 2 days ago when I started noticing this message.

(log output):
{
Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1403, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1396, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1366, in main
    full_backup(col_stats)
  File "/usr/bin/duplicity", line 500, in full_backup
    globals.backend)
  File "/usr/bin/duplicity", line 378, in write_multivol
    globals.gpg_profile, globals.volsize)
  File "/usr/lib/python2.7/dist-packages/duplicity/gpg.py", line 313, in GPGWriteFile
    file = GPGFile(True, path.Path(filename), profile)
  File "/usr/lib/python2.7/dist-packages/duplicity/gpg.py", line 143, in __init__
    'logger': self.logger_fp})
  File "/usr/lib/python2.7/dist-packages/GnuPGInterface.py", line 360, in run
    create_fhs, attach_fhs)
  File "/usr/lib/python2.7/dist-packages/GnuPGInterface.py", line 402, in _attach_fork_exec
    process.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory
}

Seems the only difference is in the lines affected.

Anyway, thanks

Revision history for this message
Aaron Swartz (aaronsw) wrote :

I'm getting this with 0.6.19.

Revision history for this message
Kenneth Loafman (kenneth-loafman) wrote : Re: [Bug 896728] Re: OSError Cannot Allocate Memory GPG

Is the version you got from the duplicity repository or from an upstream
repository?

On Wed, Jun 6, 2012 at 6:04 PM, Aaron Swartz <email address hidden> wrote:

> I'm getting this with 0.6.19.
>
> --
> You received this bug notification because you are subscribed to
> Duplicity.
> https://bugs.launchpad.net/bugs/896728
>
> Title:
> OSError Cannot Allocate Memory GPG
>
> Status in Déjà Dup Backup Tool:
> New
> Status in Duplicity - Bandwidth Efficient Encrypted Backup:
> New
> Status in Debian GNU/Linux:
> Fix Released
>
> Bug description:
> I use the deja-dup frontend for backing up my homedir. When starting the
> first backup over ssh, the following error occurs while the system is in no
> way out of ram/swap (>2G free).
> (This is _NOT_ a duplicate of bug 780270! The person who filed the latter
> one was able to do backups, I am not even able to finish a backup)
>
> Traceback (most recent call last):
> File "/usr/bin/duplicity", line 1347, in <module>
> with_tempdir(main)
> File "/usr/bin/duplicity", line 1330, in with_tempdir
> fn()
> File "/usr/bin/duplicity", line 1300, in main
> full_backup(col_stats)
> File "/usr/bin/duplicity", line 469, in full_backup
> globals.backend)
> File "/usr/bin/duplicity", line 347, in write_multivol
> globals.gpg_profile, globals.volsize)
> File "/usr/lib/python2.7/site-packages/duplicity/gpg.py", line 313, in
> GPGWriteFile
> file = GPGFile(True, path.Path(filename), profile)
> File "/usr/lib/python2.7/site-packages/duplicity/gpg.py", line 143, in
> __init__
> 'logger': self.logger_fp})
> File "/usr/lib/python2.7/site-packages/duplicity/GnuPGInterface.py",
> line 365, in run
> create_fhs, attach_fhs)
> File "/usr/lib/python2.7/site-packages/duplicity/GnuPGInterface.py",
> line 407, in _attach_fork_exec
> process.pid = os.fork()
> OSError: [Errno 12] Cannot allocate memory
>
> The used version of duplicity is 0.6.17 on ArchLinux.´
> Python 3.2.2; 2.7.2 (installed as python and python2)
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/deja-dup/+bug/896728/+subscriptions
>

Revision history for this message
Aaron Swartz (aaronsw) wrote :
Revision history for this message
orkica (orkica) wrote :

Hey,

I'm getting this with the newest duplicity installed from source.

# duplicity --version
duplicity 0.6.24

Traceback (most recent call last):
  File "/usr/local/bin/duplicity", line 1502, in <module>
    with_tempdir(main)
  File "/usr/local/bin/duplicity", line 1496, in with_tempdir
    fn()
  File "/usr/local/bin/duplicity", line 1345, in main
    do_backup(action)
  File "/usr/local/bin/duplicity", line 1466, in do_backup
    full_backup(col_stats)
  File "/usr/local/bin/duplicity", line 538, in full_backup
    globals.backend)
  File "/usr/local/bin/duplicity", line 399, in write_multivol
    globals.gpg_profile, globals.volsize)
  File "/usr/local/lib/python2.7/dist-packages/duplicity/gpg.py", line 324, in GPGWriteFile
    file = GPGFile(True, path.Path(filename), profile)
  File "/usr/local/lib/python2.7/dist-packages/duplicity/gpg.py", line 155, in __init__
    'logger': self.logger_fp})
  File "/usr/local/lib/python2.7/dist-packages/duplicity/gpginterface.py", line 369, in run
    create_fhs, attach_fhs)
  File "/usr/local/lib/python2.7/dist-packages/duplicity/gpginterface.py", line 411, in _attach_fork_exec
    process.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory

Revision history for this message
Andrew Rankin (5-andrew-9) wrote :

So I hit this as well. As it turned out my backup command was attempting to backup /proc, /sys, and /dev which was causing the process grow forever. Removing them fixed the issue for me.

Andrew

Revision history for this message
Jonas H (jonash) wrote :

Just hit this with 0.7.05. Still trying to figure out what was the reason.

Revision history for this message
Kenneth Loafman (kenneth-loafman) wrote :

Can you verify that 0.7.05 came from the duplicity PPA? Somewhere else?

What command line? What OS? What Python?

Revision history for this message
Jonas H (jonash) wrote :

The issue was that duplicity was trying to back up /var/lib/docker/devicemapper/data, which is only a couple of hundred MiB large, but maye it's a special device? I excluded it from the backup and now it went through successfully.

It's duplicity from the Arch Linux repositories. Command line, from duply: /usr/bin/duplicity --name XXX --verbosity 4 -v9 --exclude-filelist /root/.duply/XXX/exclude / s3://s3.amazonaws.com/YYY

Python 2.7.11

Revision history for this message
Stefano Stalio (stefano-stalio) wrote :

Same happens to me with duplicity 0.7.06 downloaded from duplicity.org, python 2.7.3 and a swift backend.
I managed to perform a full backup of an 800GB storage area (but needed to exclude a 127GB VM image).
After that incremental backups fail. There are a few big files (>10GB) in the tree and the host, a Debian 7 VM, has 2GB RAM and no swap.

Traceback (most recent call last):
  File "/usr/local/bin/duplicity", line 1532, in <module>
    with_tempdir(main)
  File "/usr/local/bin/duplicity", line 1526, in with_tempdir
    fn()
  File "/usr/local/bin/duplicity", line 1380, in main
    do_backup(action)
  File "/usr/local/bin/duplicity", line 1508, in do_backup
    incremental_backup(sig_chain)
  File "/usr/local/bin/duplicity", line 662, in incremental_backup
    globals.backend)
  File "/usr/local/bin/duplicity", line 425, in write_multivol
    at_end = gpg.GPGWriteFile(tarblock_iter, tdp.name, globals.gpg_profile, globals.volsize)
  File "/usr/local/lib/python2.7/dist-packages/duplicity/gpg.py", line 332, in GPGWriteFile
    file = GPGFile(True, path.Path(filename), profile)
  File "/usr/local/lib/python2.7/dist-packages/duplicity/gpg.py", line 163, in __init__
    'logger': self.logger_fp})
  File "/usr/local/lib/python2.7/dist-packages/duplicity/gpginterface.py", line 374, in run
    create_fhs, attach_fhs)
  File "/usr/local/lib/python2.7/dist-packages/duplicity/gpginterface.py", line 414, in _attach_fork_exec
    process.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory

Revision history for this message
Kenneth Loafman (kenneth-loafman) wrote :
Download full text (3.6 KiB)

What is duplicity.org? Try download from Launchpad instead,
https://launchpad.net/duplicity/+download

On Thu, Mar 10, 2016 at 1:59 AM, Stefano Stalio <<email address hidden>
> wrote:

> Same happens to me with duplicity 0.7.06 downloaded from duplicity.org,
> python 2.7.3 and a swift backend.
> I managed to perform a full backup of an 800GB storage area (but needed to
> exclude a 127GB VM image).
> After that incremental backups fail. There are a few big files (>10GB) in
> the tree and the host, a Debian 7 VM, has 2GB RAM and no swap.
>
> Traceback (most recent call last):
> File "/usr/local/bin/duplicity", line 1532, in <module>
> with_tempdir(main)
> File "/usr/local/bin/duplicity", line 1526, in with_tempdir
> fn()
> File "/usr/local/bin/duplicity", line 1380, in main
> do_backup(action)
> File "/usr/local/bin/duplicity", line 1508, in do_backup
> incremental_backup(sig_chain)
> File "/usr/local/bin/duplicity", line 662, in incremental_backup
> globals.backend)
> File "/usr/local/bin/duplicity", line 425, in write_multivol
> at_end = gpg.GPGWriteFile(tarblock_iter, tdp.name,
> globals.gpg_profile, globals.volsize)
> File "/usr/local/lib/python2.7/dist-packages/duplicity/gpg.py", line
> 332, in GPGWriteFile
> file = GPGFile(True, path.Path(filename), profile)
> File "/usr/local/lib/python2.7/dist-packages/duplicity/gpg.py", line
> 163, in __init__
> 'logger': self.logger_fp})
> File "/usr/local/lib/python2.7/dist-packages/duplicity/gpginterface.py",
> line 374, in run
> create_fhs, attach_fhs)
> File "/usr/local/lib/python2.7/dist-packages/duplicity/gpginterface.py",
> line 414, in _attach_fork_exec
> process.pid = os.fork()
> OSError: [Errno 12] Cannot allocate memory
>
> --
> You received this bug notification because you are subscribed to
> Duplicity.
> https://bugs.launchpad.net/bugs/896728
>
> Title:
> OSError Cannot Allocate Memory GPG
>
> Status in Déjà Dup:
> New
> Status in Duplicity:
> New
> Status in Debian:
> Fix Released
>
> Bug description:
> I use the deja-dup frontend for backing up my homedir. When starting the
> first backup over ssh, the following error occurs while the system is in no
> way out of ram/swap (>2G free).
> (This is _NOT_ a duplicate of bug 780270! The person who filed the
> latter one was able to do backups, I am not even able to finish a backup)
>
> Traceback (most recent call last):
> File "/usr/bin/duplicity", line 1347, in <module>
> with_tempdir(main)
> File "/usr/bin/duplicity", line 1330, in with_tempdir
> fn()
> File "/usr/bin/duplicity", line 1300, in main
> full_backup(col_stats)
> File "/usr/bin/duplicity", line 469, in full_backup
> globals.backend)
> File "/usr/bin/duplicity", line 347, in write_multivol
> globals.gpg_profile, globals.volsize)
> File "/usr/lib/python2.7/site-packages/duplicity/gpg.py", line 313, in
> GPGWriteFile
> file = GPGFile(True, path.Path(filename), profile)
> File "/usr/lib/python2.7/site-packages/duplicity/gpg.py", line 143, in
> __init__
> 'logger': self.logger_fp})
> File "/usr/lib/python2.7/site-packages/duplicity...

Read more...

Revision history for this message
dev169 (6465766e756c6c) wrote :

I am also hitting this.

### Backing up /srv/www/
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Fri Jul 22 01:00:34 2016
Reuse configured PASSPHRASE as SIGN_PASSPHRASE
Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1546, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1540, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1391, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1522, in do_backup
    incremental_backup(sig_chain)
  File "/usr/bin/duplicity", line 671, in incremental_backup
    globals.backend)
  File "/usr/bin/duplicity", line 432, in write_multivol
    globals.volsize)
  File "/usr/lib/python2.7/dist-packages/duplicity/gpg.py", line 332, in GPGWriteFile
    file = GPGFile(True, path.Path(filename), profile)
  File "/usr/lib/python2.7/dist-packages/duplicity/gpg.py", line 163, in __init__
    'logger': self.logger_fp})
  File "/usr/lib/python2.7/dist-packages/duplicity/gpginterface.py", line 374, in run
    create_fhs, attach_fhs)
  File "/usr/lib/python2.7/dist-packages/duplicity/gpginterface.py", line 414, in _attach_fork_exec
    process.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory

My system:

Ubuntu 12.04
Python 2.7.3
python-boto 2.35.2

I also updated to duplicity 0.7.09 (via apt) but no luck.

The issue started after almost a year or running flawlessly every day via cron. Note that duplicity chokes on a specific directory, that contains 124327 files in multiple subfolders with a total size of 6 GB.

Maybe the increasing file count has something to do with it? Please advise.

Revision history for this message
Simon Watson (simononline) wrote :
Download full text (4.9 KiB)

Hi,

I have just started to get this issue as well with the latest version (0.7.10-0ubuntu0ppa1240~ubun) of duplicity from the Ubuntu PPA. At first it happened on a full backup but now it's also happened on an incremental backup.

It's a fairly modest size of files to backup (SourceFiles 80196 / SourceFileSize 2583652257 (2.41 GB)) and in the past has only taken just over 1 minute to complete an incremental backup but seems to require a lot of memory.

Is there a way configure duplicity to use less memory but take longer to run the backup?

My server is:

Ubuntu 14.04.5 LTS
Python 2.7.6 (default, Oct 26 2016, 20:30:19)
duplicity 0.7.10

Here is typical memory usage when duplicity is not running:

free -mh
             total used free shared buffers cached
Mem: 615M 555M 60M 102M 23M 287M
-/+ buffers/cache: 244M 371M
Swap: 0B 0B 0B

Here is incremental backup output for the most recent failed backup:

Local and Remote metadata are synchronised, no sync needed.
Last full backup date: Sun Dec 4 01:30:47 2016
No extraneous files found, nothing deleted in cleanup.
Local and Remote metadata are synchronised, no sync needed.
Last full backup date: Sun Dec 4 01:30:47 2016
There are backup set(s) at time(s):
Sun Sep 11 01:30:46 2016
Mon Sep 12 01:30:52 2016
Tue Sep 13 01:30:52 2016
Wed Sep 14 01:30:52 2016
Thu Sep 15 01:30:47 2016
Fri Sep 16 01:30:49 2016
Which can't be deleted because newer sets depend on them.
No old backup sets found, nothing deleted.
Exception in thread Thread-1 (most likely raised during interpreter shutdown):
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
  File "/usr/lib/python2.7/dist-packages/paramiko/transport.py", line 1613, in run
<type 'exceptions.AttributeError'>: 'NoneType' object has no attribute 'error'
Local and Remote metadata are synchronised, no sync needed.
Last full backup date: Sun Dec 4 01:30:47 2016
Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1546, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1540, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1391, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1521, in do_backup
    check_last_manifest(col_stats) # not needed for full backup
  File "/usr/bin/duplicity", line 1222, in check_last_manifest
    last_backup_set.check_manifests()
  File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 199, in check_manifests
    remote_manifest = self.get_remote_manifest()
  File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 234, in get_remote_manifest
    manifest_buffer = self.backend.get_data(self.remote_manifest_name)
  File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line 677, in get_data
    fin = self.get_fileobj_read(filename, parseresults)
  File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line 671, in get_fileobj_read
    return tdp.filtered_open_with_delete("rb")
  File "/usr/lib/python2.7/dist-packages/duplicity/dup_temp.py", line 119, in filtered_open_with_delete
...

Read more...

Revision history for this message
Kenneth Loafman (kenneth-loafman) wrote :

I'm assuming you mean https://code.launchpad.net/~duplicity-team/+recipe/duplicity-releases

Did you "$ sudo apt-get purge duplicity" before installing from the PPA? Old versions might be hanging around and causing problems.

What is your command line? Are you running in a VM?

You really only have 615M of memory and 60M free? I'm fairly certain not much will run in that. I've personally never tried on less than 2GB memory, and even that is tight at times.

Revision history for this message
Simon Watson (simononline) wrote :

Yes, that is the PPA that I am installing from.

It's a small Xen VM instance used as a web server that has been running duplicity for over a year but as the SourceFileSize has grown over time it has required more memory. It seems that when duplicity runs it causes a memory usage spike that is driving the memory requirements for the server. So using duplicity is great however it's having a direct affect on the hosting costs as memory capacity needs to be increased as the duplicity SourceFileSize grows over time. Which means a reasonable amount of free memory must always be available for the nightly backup job.

It's 60M memory free including buffers/cache. Interestingly now that I have upgraded the VM instance memory to 1024M duplicity is running fine again even though it now has around 66M memory free. The buffers/cache always seem to soak up free memory so for some time I have reloaded the uwsgi app service and mysql service just before running duplicity in order to free up more memory in an attempt to reduce the growing memory requirements
that duplicity places on the server.

I've only ever installed duplicity from the PPA so I don't think there would be any old versions still installed.

The command line that I'm running is:
duplicity full --exclude /root/.cache/duplicity --include /root --include /home --include /etc --include /other/dirs --exclude '**' ${ENCRYPT_KEY_PARAM} / ${REMOTE_TARGET}

It would be interesting to find out what the memory requirements of duplicity are based on
factors such as SourceFileSize. Do the memory requirements of duplicity always increase as SourceFileSize increases? Or is there a certain point where duplicity memory usage stops increasing with SourceFileSize?

Revision history for this message
Kenneth Loafman (kenneth-loafman) wrote :

Yes, the current duplicity memory use increases with backup size, roughly 8 bytes per 64k plus structure overhead. This will be fixed in 0.8.0.

Changed in duplicity:
assignee: nobody → Kenneth Loafman (kenneth-loafman)
importance: Undecided → Medium
milestone: none → 0.8.00
status: New → In Progress
Vej (vej)
Changed in deja-dup:
status: New → Confirmed
importance: Undecided → High
Revision history for this message
Simon Watson (simononline) wrote :

@kenneth-loafman Ok great. Thanks for confirming.

Revision history for this message
pensielolaterale (fabeo) wrote :

Hi,
I'm incurring in the same memory error, while waiting for the new release, is there any workaround?

I'm running:
Ubuntu 16.04.2 LTS
duplicity 0.7.06

RAM 16GB, and at least 10GB available for duplicity

These are my last success statistics
--------------[ Backup Statistics ]--------------
StartTime 1492461636.94 (Mon Apr 17 22:40:36 2017)
EndTime 1492468936.02 (Tue Apr 18 00:42:16 2017)
ElapsedTime 7299.09 (2 hours 1 minute 39.09 seconds)
SourceFiles 479868
SourceFileSize 378726403704 (353 GB)
NewFiles 20721
NewFileSize 44027221516 (41.0 GB)
DeletedFiles 9667
ChangedFiles 11995
ChangedFileSize 12150909767 (11.3 GB)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 42383
RawDeltaSize 46988789815 (43.8 GB)
TotalDestinationSizeChange 45052504938 (42.0 GB)
Errors 0
-------------------------------------------------

Michael Terry (mterry)
no longer affects: deja-dup
Revision history for this message
Frank Bicknell (fbicknel) wrote :

Sorry for the "me, too", but I thought maybe confirmation might be of some value. While the file it horked on (../Win7-03.vdi) is a particularly gnarly backup for duplicity during incrementals due to its size and complexity, the vm was not running at the time.

duplicity 0.7.06 (December 07, 2015)
Linux mumble 4.4.0-96-generic #119-Ubuntu SMP Tue Sep 12 14:59:54 UTC 2017 x86_64 x86_64
/usr/bin/python2 2.7.12 (default, Nov 19 2016, 06:48:10)
...
AsyncScheduler: instantiating at concurrency 1
A .
A .VirtualBox
M .VirtualBox/HardDisks/Win7-03.vdi
AsyncScheduler: scheduling task for asynchronous execution
Processed volume 1
Writing duplicity-inc.20171006T052438Z.to.20171009T051723Z.vol1.difftar.gpg
Deleting /tmp/duplicity-nfDI19-tempdir/mktemp-a9VBx7-3
AsyncScheduler: task execution done (success: True)
AsyncScheduler: scheduling task for asynchronous execution
Processed volume 2
Writing duplicity-inc.20171006T052438Z.to.20171009T051723Z.vol2.difftar.gpg
Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1532, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1526, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1380, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1508, in do_backup
    incremental_backup(sig_chain)
  File "/usr/bin/duplicity", line 662, in incremental_backup
    globals.backend)
  File "/usr/bin/duplicity", line 425, in write_multivol
    at_end = gpg.GPGWriteFile(tarblock_iter, tdp.name, globals.gpg_profile, globals.volsize)
  File "/usr/lib/python2.7/dist-packages/duplicity/gpg.py", line 332, in GPGWriteFile
    file = GPGFile(True, path.Path(filename), profile)
  File "/usr/lib/python2.7/dist-packages/duplicity/gpg.py", line 163, in __init__
    'logger': self.logger_fp})
  File "/usr/lib/python2.7/dist-packages/duplicity/gpginterface.py", line 374, in run
    create_fhs, attach_fhs)
  File "/usr/lib/python2.7/dist-packages/duplicity/gpginterface.py", line 414, in _attach_fork_exec
    process.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory

Changed in duplicity:
assignee: Kenneth Loafman (kenneth-loafman) → nobody
milestone: 0.8.00 → 0.7.17
status: In Progress → Fix Committed
Changed in duplicity:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.