Unicode decode error during upgrade to 14.04 if sources.list contains non-ascii characters and locale is non-US

Thanks for your report.

Could you open a terminal and run the command:
find /etc/apt

and paste the output in this bug report? Thanks.

I found the reason: I had polish character in comment in /etc/apt/sources.list. Just sth like:

# Wyłączone w trakcie upgrade do saucy
deb ....

(after I removed this comment, the error went away, now I get „can't calculate upgrade” but that's another story)

I cannot reproduce by just adding this comment, could you restore this file as it was when this crash occurred and attach it to this bug report?

I can reproduce if I set a non-US locale like fr_FR.utf8. Setting to triage.

(the latter caused by mate and cinnamon installed from PPA)

OK, this bug is about unicodedecodeerror when non-ascii comment is present in /etc/apt/sources.list

The crash is actually in apt-clone.

Here's a patch I don't have 100% confidence in although it does work. I'm cautious because of the work done to fix bug https://bugs.launchpad.net/ubuntu/+source/ubuntu-release-upgrader/+bug/1212025.

From the comment which I see in this patch ("open in unicode mode..."), it looks like somebody intended to go towards handling file content as unicode. Your patch reverts it towards binary. Maybe it would be better to attempt input decoding (guessing the encoding somehow, at least considering utf-8 and locale-preferred encoding)? I don't remember what/how this exactly works in python3, so feel free to ignore this remark.

(sidenote: I am fairly curious what would happen if I saved some config files as utf-16 ;-))

Thanks Jibel and Brian - I can only reproduce this if I do not have "language-pack-fr" installed on precise.
If that is the case, then the command output is:
$ python -c 'import sys;print(sys.getfilesystemencoding())'
ANSI_X3.4-1968

and that leads python to fail with the above stacktrace.

I pushed a branch lp:~mvo/apt-clone/lp1309447 that contains a testcase, however, I can only make it work if:

http://bazaar.launchpad.net/~mvo/apt-clone/lp1309447/view/head:/tests/test_lp1309447.py#L11

the locale is set to something unsupported or non-utf8, i.e. if the sys.getfilesystemencoding() is not utf8.

So its worthwhile to try to find out if something is unsetting LANG or LC_ALL when update-manager is run.
If its a common problem we should probably also add a workaround like:
http://bazaar.launchpad.net/~mvo/software-properties/lp1069019/revision/901

I pushed a fix for the issues I was able to reproduce here:
  lp:~mvo/ubuntu-release-upgrader/lp1309447
it will use C.UTF-8 when the LANG is unset or set to C to ensure that the release upgrader does not fall over on a utf8 sources.list file.

Bug #1243795 - the same exactly ?

Hello Mekk, or anyone else affected,

Accepted ubuntu-release-upgrader into trusty-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/ubuntu-release-upgrader/1:0.220.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

This bug was fixed in the package ubuntu-release-upgrader - 1:14.10.2

---------------
ubuntu-release-upgrader (1:14.10.2) utopic; urgency=low

  * debian/test/control:
    - add missing pep8 test dependency
  * tests/test_xorg_fix_intrepid.py:
    -
 -- Michael Vogt <email address hidden> Sat, 03 May 2014 08:37:51 +0200

Still crashes in Trusty with http://launchpad.net/ubuntu/+source/ubuntu-release-upgrader/1:0.220.3

I used the test case in the bug description and ran the command
LANG=fr_FR.utf8 do-release-upgrade -d

To test the version of the release upgrader from the -proposed repository you need to use 'do-release-upgrade -p'.

I've test this myself using the test case from the bug description and 'do-release-upgrade -p' and it did in fact fail.

I noticed the following traceback in my log file:

2014-05-05 10:59:56,198 INFO Using config files '['./DistUpgrade.cfg']'
2014-05-05 10:59:56,198 INFO uname information: 'Linux clean-saucy-amd64 3.11.0-19-generic #33-Ubuntu SMP Tue Mar 11 18:48:34 UTC 2014 x86_64'
2014-05-05 10:59:56,198 INFO apt version: '0.9.9.1~ubuntu1'
2014-05-05 10:59:56,198 INFO python version: '2.7.5+ (default, Feb 27 2014, 19:37:08)
[GCC 4.8.1]'
2014-05-05 10:59:56,199 INFO release-upgrader version '0.220.3' started
2014-05-05 10:59:56,199 ERROR getting the encoding failed
Traceback (most recent call last):
  File "/tmp/ubuntu-release-upgrader-cyewkp/DistUpgrade/DistUpgradeView.py", line 40, in <module>
    locale.setlocale(locale.LC_ALL, "")
  File "/usr/lib/python2.7/locale.py", line 547, in setlocale
    return _setlocale(category, locale)
Error: unsupported locale setting
2014-05-05 10:59:56,200 INFO locale: 'None' 'None'

How do you test in Trusty? When I run

do-release-upgrade -p

I get "No new release found"

@Matthew - sorry if you are running trusty then you just use 'do-release-upgrade -d' as the fix is in the main pocket for utopic. If you are using saucy then you need to use 'do-release-upgrade -p' to download the release upgrader from the trusty -proposed pocket.

I confirm my own comment #18 that this really did fail.

Thanks Brian and Mathew and sorry for the trouble. The issue is not actually the release upgrader but the apt-clone file its including. The push a branch to lp:~mvo/apt-clone/lp1324833 that fixes the bug. The release-upgrader needs to be rebuild/uploaded with this fix (and a updated build-dependency to ensure that really the latest version with the fix of apt-clone is made part of the release upgrader tarball).

I am still getting the same error even when I start the upgrader with `do-release-upgrade -p` from saucy:

File
"/tmp/ubuntu-release-upgrader-qwht26/DistUpgrade/sourceslist.py",
line 254, in str
line += " #" + self.comment

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position
27: ordinal not in range(128)
=== Command detached from window (Wed Jul 2 08:56:01 2014) ===
=== Command terminated with exit status 1 (Wed Jul 2 08:56:01 2014) ===

I should also add that I have no-non ascii characters in my sources.list. I get this error even when I delete all the sources.

This bug was fixed in the package apt-clone - 0.3.1~ubuntu12

---------------
apt-clone (0.3.1~ubuntu12) utopic; urgency=medium

  [ Brian Murray ]
  * Add an option to replace the mirror used in /etc/apt/sources.list with one
    of your choosing. (LP: #1290584)
  * debian/tests/control: add in missing make dependency

  [ Michael Vogt ]
  * Resolve unicode decode error for sources.list files if locale is
    unsupported. (LP: #1309447)
 -- Brian Murray <email address hidden> Wed, 20 Aug 2014 13:54:48 -0700

@Michael - Do you have any ideas on how we can test the apt-clone part of this fix? We need to get the fixed apt-clone into trusty for the release-upgrader to use it, if I understand things correctly.

Hello Mekk, or anyone else affected,

Accepted apt-clone into trusty-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/apt-clone/0.3.1~ubuntu11.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

I've tested the apt-clone part of this fix by running "LANG='' test_lp1309447.py" on a 14.04 system with the version of apt-clone from the release pocket and observed a traceback. I then installed the version of apt-clone from -proposed and ran the same test and did not receive a traceback.

This bug was fixed in the package apt-clone - 0.3.1~ubuntu11.1

---------------
apt-clone (0.3.1~ubuntu11.1) trusty-proposed; urgency=medium

  * Resolve unicode decode error for sources.list files if locale is
    unsupported. (LP: #1309447)
 -- Brian Murray <email address hidden> Wed, 20 Aug 2014 14:21:21 -0700

The verification of the Stable Release Update for apt-clone has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Is there anything left to do for the ubuntu-release-upgrader task of this bug?

Michael said in comment #22, "The release-upgrader needs to be rebuild/uploaded with this fix (and a updated build-dependency to ensure that really the latest version with the fix of apt-clone is made part of the release upgrader tarball)."

Yes, we would need to update the Trusty version of ubuntu-release-upgrader for people to get the newest version of apt-clone. However, it is only necessary for people upgrading from Saucy to Trusty. I went looking for failures like this for people upgrading from Saucy to Trusty in the Ubuntu Error Tracker and was unable to find any because we stopped accepting crash reports from Saucy. Subsequently, we do not really have an idea of how many people are still affected by this specific bug report and therefore whether or not it is worth updating ubuntu-release-upgrader.

I added bug #1278280 as a duplicate, because I think it is the same issue it has the same symptoms and test case.

Looking closer, the traceback on those bugs is different. It's a similar Unicode issue, but maybe not the exact same one. I made them duplicates of bug #1243795 instead.