Deja Dup restore fails with "too many files open" error

Thanks for the report, Pete! This is likely a bug that was half-fixed in duplicity 0.5.18 (which you should have in Alpha 4). The rest of the fix was in 0.6.00 (which will be in Karmic+1). A possible workaround is to run:
ulimit -n 4096
or some such number to make it work.

However... I can't totally clear Deja Dup. Deja Dup exacerbates the problem by rarely (i.e. never) making a fresh full backup (and causing lots of open files as a result when restoring -- though I'd argue that's a duplicity issue as well). That's fixed in trunk and will be released with Deja Dup 11.0.

Ken, please note the linked Debian report from Gonzalo, a Debian user who is using duplicity 0.6.04 and deja-dup 10.1. Gonzalo says that even with those versions and doing just a single full backup and then restore (i.e. not a long chain of incrementals), he is getting this error.

Are there other sources of problems that would cause this error? And isn't it sort of a design flaw of duplicity that it theoretically is open to this problem? Couldn't it close the file handles on files to which it isn't actively writing? That seems like it would solve the issue.

Reopened this bug since it appears not to be fixed. Will look into it ASAP.

Just as a workaround, please try 'sudo ulimit -n 4096' before running the restore. Let me know if that works.

I don't think Gonzalo is subscribed to this bug, but he knows about it and may read it. I already told him to try ulimit -n 4096 in the Debian bug, but he didn't mention whether it worked for him. Since he later ran into this problem again, I'm betting he didn't try it (since that has seemed to work for everybody else with this problem).

Michael Terry wrote:
> I don't think Gonzalo is subscribed to this bug, but he knows about it
> and may read it. I already told him to try ulimit -n 4096 in the Debian
> bug, but he didn't mention whether it worked for him. Since he later
> ran into this problem again, I'm betting he didn't try it (since that
> has seemed to work for everybody else with this problem).

I replied to the Debian bug just now. Perhaps he'll see it. This
appears to be a totally different problem from the long incremental
backup chains.

I have tried to reproduce this and am unable to do so. I'm going to need more information as to what system he's on, Python versions, etc. I'm just not seeing any more than about 70 open file descriptors at any one time, and most of those are Python libraries and objects.

Folks, this might be a more fundamental issue, since the maximum number of open files is limited by default to 1024 on the system.

We from the Amarok team had this bug report:

https://bugs.kde.org/show_bug.cgi?id=211026

and this can be solved by adding the following lines to /etc/security/limits.conf

* soft nofile 16384
* hard nofile 16384

BTW, the same will happen on a Fedora system, which has the same limitation. Seems we inherit a limitation from the nineties here...

Please try this out.

Deja Dup's side of things has been fixed with 11.0 -- we now make full backups occasionally.