Restore fails with “too many open files”
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Duplicity |
Invalid
|
Undecided
|
Unassigned | ||
duplicity (Ubuntu) |
New
|
Low
|
Unassigned |
Bug Description
I use duplicity with Backblaze B2, have an incremental backup every hour and a full backup every 90 days. I recently tried to restore a file to the state that it had 14 days ago. Eventually it failed saying “too many open files”.
Apparently the last full backup was 2020-03-08 15:00:05Z. So that means that _a lot_ of incremental backups needed to be resolved. These are the last lines of output:
Get: duplicity-
Verarbeitung des lokalen Manifest b'/home/
1 Volumen in Manifest gefunden
Get: duplicity-
Verarbeitung des lokalen Manifest b'/home/
1 Volumen in Manifest gefunden
Get: duplicity-
Traceback (innermost last):
File "/usr/bin/
with_
File "/usr/bin/
fn()
File "/usr/lib64/
do_
File "/usr/lib64/
restore(
File "/usr/lib64/
if not patchdir.
File "/usr/lib64/
for ropath in rop_iter:
File "/usr/lib64/
for patch_seq in collated:
File "/usr/lib64/
setrorps(
File "/usr/lib64/
elems[i] = next(iter_list[i])
File "/usr/lib64/
tarinfo_list = [next(tar_iter)]
File "/usr/lib64/
self.
File "/usr/lib64/
self.current_fp = next(self.
File "/usr/lib64/
yield restore_
File "/usr/lib64/
fileobj = tdp.filtered_
File "/usr/lib64/
fh = FileobjHooked(
File "/usr/lib64/
return gpg.GPGFile(False, self, gpg_profile)
File "/usr/lib64/
p1 = gnupg.run(
File "/usr/lib64/
process = self._attach_
File "/usr/lib64/
pipe = os.pipe()
OSError: [Errno 24] Too many open files
I would assume that it stumbles over the 1024 file descriptor limit and it just cannot resolve a backup with that many incremental backups. Basically with the current state of the software my incremental backups are worthless because I cannot restore. Is there something that I can do except doing less frequent incremental updates and more full backups?
Changed in duplicity (Ubuntu): | |
importance: | Undecided → Low |
Changed in duplicity: | |
status: | New → Opinion |
Changed in duplicity: | |
status: | Opinion → Invalid |
This is a known issue with duplicity. You will need to increase your systems number of open files since it's set so low. I'd suggest 16384 or higher if you have a lot of incrementals. I'd also suggest limiting the number of incrementals and doing more full backups.