be smart at guessing encoding

As discussed with LartiQ in #bzr, I think bzr-gtk should be smarter at guessing how to decode arbitrary file contents. Generally, the logic should look like:

1. Look for a BOM. If we find one, we can be confident that the encoding is utf-something. BOM are normally found in utf-16 and utf-32 files, but LartiQ reports that it's sometimes used in utf-8 documents as well (although it makes no sense, since utf-8 fixes the bit ordering).

2. Try decoding with utf-8. I do not know of any encoding/language that normally (in non-pathological documents) produce data that is valid utf-8.

3. Optionally, more heuristics. Some text editors looks for patterns in the document to guess the encoding. I believe emacs has some magic of that sort.

4. Try the locale encoding, as provided by sys.getpreferredencoding (per j-a-meinel)

5. If that still does not work, decode('ascii', 'replace').

FINALLY: always display a control that shows the encoding, and provides direct user control to override the automatic detection. The choices should at least include utf-8, the locale encoding, and explicit input of any arbitrary encoding supported by Python. Optionally, the choices could include a list of user-configurable favourite encodings.

I think behaviour should be predictable by the user and trying to be smarter than the user is hard. If it is user-specifiable, with the initial encoding guessed by bzr-gtk, I'm all for it.

Is this possible to bzr-gtk to read encoding from a sort of config file or from environment variable (like $ENCODING)?

Yes, though the problem is that the encoding might be different for different branches or even for different files in a branch.

Adding to Jelmer's comment in reply to bialix:
And that the encoding of files in a bzr branch may not bear any relation to the user's locale setting. User locale is good for interepreting user input, and for encoding unicode text to the user's preferred encoding.

But decoding of source files is a different problem than the one addressed by the user's locale. Source file data is not user input (it's rather like untrusted data off the internet) and it's definitely not unicode (otherwise we would not be worrying about decoding it).

Well, I understand your pain in wide sense of guessing encoding. But it's too big problem. Make problem smaller.

If I work on branch and I precisely know what encoding files are -- how I can tell to bzr-gtk this info? I think support for giving this info could be first small but important step towards encodings support.

Right now bzr-gtk simply unusable with my russian-text files.

On Thu, 2006-10-05 at 15:05 +0000, bialix wrote:
> Well, I understand your pain in wide sense of guessing encoding. But
> it's too big problem. Make problem smaller.
>
> If I work on branch and I precisely know what encoding files are -- how
> I can tell to bzr-gtk this info? I think support for giving this info
> could be first small but important step towards encodings support.
>
> Right now bzr-gtk simply unusable with my russian-text files.
That's why I think the best way to solve this is as ddaa proposed:

- try to guess an encoding (maybe always UTF-8?)
- allow the user (in the diff window) to change the encoding from the
default

Cheers,

Jelmer
--
Jelmer Vernooij <email address hidden> - http://samba.org/~jelmer/

That is steps 2, 5 and FINALLY in the bug description.

Jelmer Vernooij пишет:
> On Thu, 2006-10-05 at 15:05 +0000, bialix wrote:
>> Well, I understand your pain in wide sense of guessing encoding. But
>> it's too big problem. Make problem smaller.
>>
>> If I work on branch and I precisely know what encoding files are -- how
>> I can tell to bzr-gtk this info? I think support for giving this info
>> could be first small but important step towards encodings support.
>>
>> Right now bzr-gtk simply unusable with my russian-text files.
> That's why I think the best way to solve this is as ddaa proposed:
>
> - try to guess an encoding (maybe always UTF-8?)
> - allow the user (in the diff window) to change the encoding from the
> default

It's not only in diff window.
gannotate also works bad with non-utf-8 text files.

--
Alexander