Duplicity fails with UnicodeDecodeError in uexc function

Status changed to 'Confirmed' because the bug affects multiple users.

Failed with an unknown error.

Traceback (innermost last):
  File "/usr/bin/duplicity", line 1555, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1541, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1393, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1523, in do_backup
    incremental_backup(sig_chain)
  File "/usr/bin/duplicity", line 668, in incremental_backup
    globals.backend)
  File "/usr/bin/duplicity", line 454, in write_multivol
    (tdp, dest_filename, vol_num)))
  File "/usr/lib/python2.7/dist-packages/duplicity/asyncscheduler.py", line 146, in schedule_task
    return self.__run_synchronously(fn, params)
  File "/usr/lib/python2.7/dist-packages/duplicity/asyncscheduler.py", line 172, in __run_synchronously
    ret = fn(*params)
  File "/usr/bin/duplicity", line 453, in <lambda>
    vol_num: put(tdp, dest_filename, vol_num),
  File "/usr/bin/duplicity", line 342, in put
    backend.put(tdp, dest_filename)
  File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line 395, in inner_retry
    % (n, e.__class__.__name__, util.uexc(e)))
  File "/usr/lib/python2.7/dist-packages/duplicity/util.py", line 79, in uexc
    return ufn(unicode(e).encode('utf-8'))
 UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 37: ordinal not in range(128)

I had an error with the same look in the last couple of days with the daily backup:

"Failed with an unknown error.
Traceback (innermost last):
  File "/usr/bin/duplicity", line 1555, in <module>
    with_tempdir(main)"
etc.

I found my external drives became read-only. I have a dual-boot Win10/Ubuntu system. Reboots/clean exit of Windows did not help, drives were still read only, and the backup process remained broken, but I realized that the Fast Boot option is reactivated. It should have been the April 2018 Windows update I just installed some days before... After I turned Fast Boot back off, backup has completed, seems to be perfect again.

Affected by ascii error too.

I am experiencing this problem too. I have an English language Ubuntu 18.04 install (fresh) with Japanese language added.

Traceback (innermost last):
  File "/usr/bin/duplicity", line 1555, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1541, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1393, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1511, in do_backup
    full_backup(col_stats)
  File "/usr/bin/duplicity", line 572, in full_backup
    globals.backend)
  File "/usr/bin/duplicity", line 454, in write_multivol
    (tdp, dest_filename, vol_num)))
  File "/usr/lib/python2.7/dist-packages/duplicity/asyncscheduler.py", line 146, in schedule_task
    return self.__run_synchronously(fn, params)
  File "/usr/lib/python2.7/dist-packages/duplicity/asyncscheduler.py", line 172, in __run_synchronously
    ret = fn(*params)
  File "/usr/bin/duplicity", line 453, in <lambda>
    vol_num: put(tdp, dest_filename, vol_num),
  File "/usr/bin/duplicity", line 342, in put
    backend.put(tdp, dest_filename)
  File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line 395, in inner_retry
    % (n, e.__class__.__name__, util.uexc(e)))
  File "/usr/lib/python2.7/dist-packages/duplicity/util.py", line 79, in uexc
    return ufn(unicode(e).encode('utf-8'))
 UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 37: ordinal not in range(128)

Hello all,

same issue here with Slovak environment and user with sudo rights.
I found workaround by changing storage location to folder, which is located in home directory and thus user has direct access to it.

It looks like access rights issues.

Andrej

Hello All - me too - identical error output to Bud Roth above. Not enough space available to try Andrej's workaround, unfortunately.
Nick

May be connected to #1440372 – please port deja-dup to python3

Similar to what is described also on https://bugzilla.redhat.com/show_bug.cgi?id=1470873

This is the error message I receive when trying to use backup:

Traceback (innermost last):
  File "/usr/bin/duplicity", line 1555, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1541, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1393, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1511, in do_backup
    full_backup(col_stats)
  File "/usr/bin/duplicity", line 563, in full_backup
    bytes_written = dummy_backup(tarblock_iter)
  File "/usr/bin/duplicity", line 236, in dummy_backup
    while tarblock_iter.next():
  File "/usr/lib/python2.7/dist-packages/duplicity/diffdir.py", line 523, in next
    result = self.process(self.input_iter.next())
  File "/usr/lib/python2.7/dist-packages/duplicity/diffdir.py", line 195, in get_delta_iter
    for new_path, sig_path in collated:
  File "/usr/lib/python2.7/dist-packages/duplicity/diffdir.py", line 286, in collate2iters
    relem2 = riter2.next()
  File "/usr/lib/python2.7/dist-packages/duplicity/diffdir.py", line 237, in sigtar2path_iter
    tf = util.make_tarfile("r", sigtarobj)
  File "/usr/lib/python2.7/dist-packages/duplicity/util.py", line 123, in make_tarfile
    tf = tarfile.TarFile("arbitrary", mode, fp)
  File "/usr/lib/python2.7/tarfile.py", line 1543, in __init__
    self.name = os.path.abspath(name) if name else None
  File "/usr/lib/python2.7/posixpath.py", line 364, in abspath
    cwd = os.getcwd()
 OSError: [Errno 2] No such file or directory

Same here unbuntu 18.04 lts french

Traceback (innermost last):
  File "/usr/bin/duplicity", line 1555, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1541, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1393, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1511, in do_backup
    full_backup(col_stats)
  File "/usr/bin/duplicity", line 572, in full_backup
    globals.backend)
  File "/usr/bin/duplicity", line 454, in write_multivol
    (tdp, dest_filename, vol_num)))
  File "/usr/lib/python2.7/dist-packages/duplicity/asyncscheduler.py", line 146, in schedule_task
    return self.__run_synchronously(fn, params)
  File "/usr/lib/python2.7/dist-packages/duplicity/asyncscheduler.py", line 172, in __run_synchronously
    ret = fn(*params)
  File "/usr/bin/duplicity", line 453, in <lambda>
    vol_num: put(tdp, dest_filename, vol_num),
  File "/usr/bin/duplicity", line 342, in put
    backend.put(tdp, dest_filename)
  File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line 392, in inner_retry
    util.uexc(e)), code=code, extra=extra)
  File "/usr/lib/python2.7/dist-packages/duplicity/util.py", line 79, in uexc
    return ufn(unicode(e).encode('utf-8'))
 UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 36: ordinal not in range(128)

Thank you everybody for your bug reports and comments.

I believe that this bug is already fixed in the 0.8-series, but we have not yet released a package for this.

The code triggering this is used when there is an exception/error message containing a non-ASCII character (for example a filename), so this is sometimes 'fixed' by fixing the underlying problem. This is why some commenters say the problem went away when they resolved permissions issues. I do not know what has changed that is causing this to break for people it did not break for previously.

The reality is that version 0.7 and below were never actually written to accommodate non-ASCII characters and the fact it worked at all was largely luck. The 0.8 series does add proper support for non-ASCII, with the internals converted to Unicode and Python 3 support.

Many thanks to Pete Zaitcev (zaitcev) in Bug #1797928 for proposing a patch for this, which I had overlooked.

I have applied this in the branch here: lp:~aaron-whitehouse/duplicity/07-uexc-fix

It would be helpful if you can please test to see if this fixes your problem as follows:
1. Create a new destination directory to test with.
2. Test that duplicity fails as you have reported when using this destination.
3. Pull down a copy of the branch with the fix:
bzr branch lp:~aaron-whitehouse/duplicity/07-uexc-fix dup_patched
4. cd dup_patched/duplicity
5. run ./compilec.py
6. change directory back to wherever you were running duplicity from before
5. Test with the patched duplicity:
PYTHONPATH=[full path/]dup_patched [full path/]dup_patched/bin/duplicity [all of your options/arguments]

Please let me know how you get on.

== Simple illustration of failure ==
A simple example triggering this bug is to create a couple of empty directories, say "source1" and "dest".

Then try:
duplicity --exclude=bänana source1/ file://dest/

This should give a FilePrefixError (hideous in version 0.7 and much more informative in 0.8), but instead says:
Traceback (most recent call last):
  File "/usr/bin/duplicity", line 1611, in <module>
    if "Forced assertion for testing" in util.uexc(e):
  File "/usr/lib/python2.7/dist-packages/duplicity/util.py", line 79, in uexc
    return ufn(unicode(e).encode('utf-8'))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 1: ordinal not in range(128)

The 0.8-series was affected by this (as I mentioned above, fixing the underlying problem avoids this error and a number of these have been fixed, so it is harder to trigger).

I have just proposed a fix for it be merged into the 0.8-series:
https://code.launchpad.net/~aaron-whitehouse/duplicity/08-uexc-fix

I would still appreciate any feedback on the 0.7-series branch above. I am more nervous about making changes in our stable branch without feedback.

Hello,
what is needed for the fix to land in ubuntu's duplicity package ? This bug has bee fixed mainstream and the fix is available in other distro, so if any help needed, please tell me where to start.

Let's try to land the change in Ubuntu disco and see how it's going there first before condidering SRUing

This bug was fixed in the package duplicity - 0.7.18.2-1ubuntu3.1

---------------
duplicity (0.7.18.2-1ubuntu3.1) disco; urgency=medium

  * debian/patches/bzr_unicode_error.patch:
    - backport an upstream change to fix UnicodeDecodeError
      (lp: #1770929)

 -- Sebastien Bacher <email address hidden> Fri, 12 Apr 2019 16:37:20 +0200

Still, or again, appearing in 0.7.19

duplicity --version
duplicity 0.7.19

deja-dup --version
deja-dup 37.0

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.2 LTS
Release: 18.04
Codename: bionic

Traceback (innermost last):
  File "/usr/bin/duplicity", line 1581, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1567, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1419, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1537, in do_backup
    full_backup(col_stats)
  File "/usr/bin/duplicity", line 577, in full_backup
    globals.backend)
  File "/usr/bin/duplicity", line 459, in write_multivol
    (tdp, dest_filename, vol_num)))
  File "/usr/lib/python2.7/dist-packages/duplicity/asyncscheduler.py", line 146, in schedule_task
    return self.__run_synchronously(fn, params)
  File "/usr/lib/python2.7/dist-packages/duplicity/asyncscheduler.py", line 172, in __run_synchronously
    ret = fn(*params)
  File "/usr/bin/duplicity", line 458, in <lambda>
    vol_num: put(tdp, dest_filename, vol_num),
  File "/usr/bin/duplicity", line 347, in put
    backend.put(tdp, dest_filename)
  File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line 395, in inner_retry
    % (n, e.__class__.__name__, util.uexc(e)))
  File "/usr/lib/python2.7/dist-packages/duplicity/util.py", line 79, in uexc
    return ufn(unicode(e).encode('utf-8'))
 UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 37: ordinal not in range(128)

I can also confirm that this is still happening with Ubuntu 16.04 LTS

duplicity-backup.sh v1.6.0
duplicity 0.7.19

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.6 LTS
Release: 16.04
Codename: xenial

-------[ Program versions ]-------
duplicity-backup.sh v1.6.0
duplicity 0.7.19
----------------------------------

LFTP version is 4.6.3a
Traceback (innermost last):
  File "/usr/bin/duplicity", line 1581, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1567, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1419, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1441, in do_backup
    action).set_values()
  File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 684, in set_values
    backend_filename_list = self.backend.list()
  File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line 395, in inner_retry
    % (n, e.__class__.__name__, util.uexc(e)))
  File "/usr/lib/python2.7/dist-packages/duplicity/util.py", line 79, in uexc
    return ufn(unicode(e).encode('utf-8'))
 UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 301: ordinal not in range(128)

----------------[ Duplicity Cleanup ]----------------
LFTP version is 4.6.3a
Traceback (innermost last):
  File "/usr/bin/duplicity", line 1581, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1567, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1419, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1441, in do_backup
    action).set_values()
  File "/usr/lib/python2.7/dist-packages/duplicity/collections.py", line 684, in set_values
    backend_filename_list = self.backend.list()
  File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line 395, in inner_retry
    % (n, e.__class__.__name__, util.uexc(e)))
  File "/usr/lib/python2.7/dist-packages/duplicity/util.py", line 79, in uexc
    return ufn(unicode(e).encode('utf-8'))
 UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 301: ordinal not in range(128)

All, to clarify the current position, this is fixed in the 0.8 series, but I set out the steps to test the fix in comment #15 and was waiting on testing feedback before merging the patch back into the 0.7 series. Nobody gave me any testing feedback. It is therefore not yet fixed in the 0.7 series.

It looks as though Ubuntu has separately patched this in Disco only (comment #19).

So it will not work in Bionic or Xenial.

We are now so close to a 0.8 release that I am inclined to wait for that and encourage affected users to move straight to that, rather than risk breaking people who are sticking to the 0.7 series.

All,

This should be fixed in 0.8.00, which has now been released. This is the first 0.8-series release, so expect issues and test it out on data you can afford to lose (if it comes to that).

I have now packaged up this version as a snap. You can install it by typing:
snap install duplicity --classic --beta
Please uninstall your current version of duplicity to ensure you are using the snap instead of one from apt etc.
(This will work out of the box on nearly all versions of Ubuntu. If you use a different distro and have never installed snaps before, visit this page:
https://docs.snapcraft.io/installing-snapd )

This includes the latest duplicity and all dependencies wrapped up in a self-contained bundle. You should not need to install anything else for this to work (including for all backends).

If you do test this, it would be very helpful if you could please comment here, even if the only feedback is that all worked as expected.

If you are giving any feedback, please also give the output of:
which duplicity
and
snap list | grep duplicity
and let me know which backends you used.

To add on to Aaron's comment, if you are coming here because you hit this bug while using deja-dup, you can install a version of deja-dup that bundles in duplicity 0.8.0, which has the fix:

snap install deja-dup --classic --candidate

Hello,

I receive the same error on Kubuntu 18.04, so I purge both duplicity 0.7 and deja-dup and install the snap version of them, as instructed by Michael and Aaron. Nevertheless, I still get the same error when I run "snap run duplicity". Here's the output:

$ which duplicity
/snap/bin/duplicity

$ snap list | grep duplicity
duplicity 0.8.02 11 beta aaron-whitehouse classic

$ snap run duplicity
Traceback (innermost last):
  File "/snap/duplicity/11/bin/duplicity", line 1711, in <module>
    with_tempdir(main)
  File "/snap/duplicity/11/bin/duplicity", line 1699, in with_tempdir
    tempdir.default().cleanup()
  File "/snap/duplicity/11/lib/python2.7/site-packages/duplicity/tempdir.py", line 63, in default
    _defaultInstance = TemporaryDirectory(temproot=globals.temproot)
  File "/snap/duplicity/11/lib/python2.7/site-packages/duplicity/tempdir.py", line 135, in __init__
    log.Info(_(u"Using temporary directory %s") % self.__dir)
 UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 16: ordinal not in range(128)

Hello,
in view of the above-mentioned post, it seems to me that the problem is not fully solved and therefore I will describe my findings. Today I tried to set up my small server backup to a second using duplicity package. I used some example for writing tiny script.

#!/bin/bash
export PASSPHRASE=some_pwd
export FTP_PASSWORD=some_ftp_pwd
duplicity ~/SQLBACKUP ftp://some_user@10.0.0.250/SQLBACKUP
unset PASSPHRASE
unset FTP_PASSWORD

When I ran the script, I got the errors listed above, but a directory SQLBACKUP was created on the target server. The permissions of this directory were okay (some similar bug), so I tried to run the script a second time and the backup went without errors. Other backups were already running without errors too. In my opinion, the problem may be that it fails to create a backup directory on the target server fast enough. As a target server I used ftp server on Windows once and second ftp server on Ubuntu 18.04 and the result was always the same (physically three machines on network: one with duplicity and two ftp servers). When I prepared SQLBACKUP directory on the target server manually, the backup went without errors too.

machine with duplicity:
-----------------------
Ubuntu 18.04.4 LTS
Python 2.7.17
duplicity 0.7.17 (from Ubuntu repository)
lftp version 4.8.1

Perhaps a small loop in the depths of duplicity package, that tests whether a directory has already been created, would help :-).

I'm still experiencing this bug.

Backup failed. I can't find any processing logs, so I'm not able to know which file caused the problem...

The dialog box shows:

Traceback (innermost last):
  File "/usr/bin/duplicity", line 1555, in <module>
    with_tempdir(main)
  File "/usr/bin/duplicity", line 1541, in with_tempdir
    fn()
  File "/usr/bin/duplicity", line 1393, in main
    do_backup(action)
  File "/usr/bin/duplicity", line 1511, in do_backup
    full_backup(col_stats)
  File "/usr/bin/duplicity", line 572, in full_backup
    globals.backend)
  File "/usr/bin/duplicity", line 454, in write_multivol
    (tdp, dest_filename, vol_num)))
  File "/usr/lib/python2.7/dist-packages/duplicity/asyncscheduler.py", line 146, in schedule_task
    return self.__run_synchronously(fn, params)
  File "/usr/lib/python2.7/dist-packages/duplicity/asyncscheduler.py", line 172, in __run_synchronously
    ret = fn(*params)
  File "/usr/bin/duplicity", line 453, in <lambda>
    vol_num: put(tdp, dest_filename, vol_num),
  File "/usr/bin/duplicity", line 342, in put
    backend.put(tdp, dest_filename)
  File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line 395, in inner_retry
    % (n, e.__class__.__name__, util.uexc(e)))
  File "/usr/lib/python2.7/dist-packages/duplicity/util.py", line 79, in uexc
    return ufn(unicode(e).encode('utf-8'))
 UnicodeDecodeError: 'ascii' codec can't decode byte 0xe8 in position 27: ordinal not in range(128)

Environment:

bluet@ocisly:~/.cache/deja-dup$ dpkg-query -W deja-dup duplicity
deja-dup 37.1-2fakesync1ubuntu0.1
duplicity 0.7.17-0ubuntu1.1

bluet@ocisly:~/.cache/deja-dup$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.4 LTS
Release: 18.04
Codename: bionic
bluet@ocisly:~/.cache/deja-dup$ uname -a
Linux ocisly 5.3.0-61-generic #55~18.04.1-Ubuntu SMP Mon Jun 22 16:40:20 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

bluet@ocisly:~/.cache/deja-dup$ locale
LANG=zh_TW.UTF-8
LANGUAGE=zh_TW:en
LC_CTYPE="zh_TW.UTF-8"
LC_NUMERIC=zh_TW.UTF-8
LC_TIME=zh_TW.UTF-8
LC_COLLATE="zh_TW.UTF-8"
LC_MONETARY=zh_TW.UTF-8
LC_MESSAGES="zh_TW.UTF-8"
LC_PAPER=zh_TW.UTF-8
LC_NAME=zh_TW.UTF-8
LC_ADDRESS=zh_TW.UTF-8
LC_TELEPHONE=zh_TW.UTF-8
LC_MEASUREMENT=zh_TW.UTF-8
LC_IDENTIFICATION=zh_TW.UTF-8
LC_ALL=

https://answers.launchpad.net/deja-dup/+question/691652

Hello all,

Apologies, I missed these recent messages.

For background, the error UnicodeDecodeError just means that a non-ASCII character is hitting a piece of code that was not designed to handle it. That is not surprising -- prior to the 0.8 series, very little of the code was designed for non-ASCII characters. What that means is that you may be seeing a UnicodeDecodeError even though this bug is fixed, because somewhere else in the code needs fixing.

Victor Douglas (jhngalt): This is an interesting error and thank you very much for following our instructions on testing the snap and giving the relevant details. Can you please test with the latest stable duplicity snap and, if you still have a problem, report the bug in our new tracker (https://gitlab.com/duplicity/duplicity/-/issues )? Note that "snap run duplicity" does not actually do anything without arguments (even working correctly it gives:
Command line error: Expected 2 args, got 0
Enter 'duplicity --help' for help screen.), but it still should not be giving an error like that. It looks as though the error was related to non-ASCII in your temporary directory name not being handled correctly.

Martin Theiner (mathesubu1) and BlueT - Matthew Lien - 練喆明 (bluet) -- you are both using an old (pre 0.8 series) version of duplicity/deja dup. Please uninstall the system version (e.g. "sudo apt-get purge duplicity") and install the stable snap (e.g. "sudo snap install duplicity —classic") and test again.

Here's hoping a non-snap version gets fixed!

I have ubuntu 18.04 with ru locale and same issue
How I fixed it:
install snap versions
snap install duplicity --classic
snap install deja-dup --classic

remove apt version of deja-dup
sudo apt purge deja-dup

backup (option)
sudo mv duplicity duplicity_apt

link duplicity bin from snap pkg
sudo ln /snap/bin/duplicity /usr/bin/duplicity

For me this bug was actually triggered by another new bug in lftp caused by glibc upgate, see #1902832 which was obfuscated by this bug.

I found the patch from 19.04 disco which was mentioned in #19 which fixed the unicode error for be and revealed the other bug.

https://launchpad.net/ubuntu/+source/duplicity/0.7.18.2-1ubuntu3.1
The change related to unicode is quite small, patch file "02-unicode":
##############################################################################
Author: Alexander Zangerl <email address hidden>
Subject: fix unicode decode-does-encode-and-fail problems with iso8859 locales

--- a/bin/duplicity
+++ b/bin/duplicity
@@ -29,6 +29,11 @@

 import gzip
 import os
+
+# override locale to avoid bug #682837, until
+# the logger finally deals with locales cleanly
+os.environ['LC_ALL']="POSIX"
+
 import sys
 import time
 import types
##############################################################################