better support for UTF-16 in qdiff, qannotate, qcat

Philippe Lhoste wrote in bzr ML:
-------------------------------------------------------
I was puzzled because I had a simple .reg file (exported by regedit) which I hacked to add support for a new source code extension (icon, editor/compiler, etc.), and Bazaar was seeing it as binary although my editor shown only CR and LF control chars...

The Bazaar User Reference mentions (casually) that binary status is guessed by content (I suppose looking for some control chars at first bytes, as usual).

When I opened the file with a hex editor, I saw the reason: it is an UTF-16 file with Bom (0xFF 0xFE).

It is annoying because I cannot do diffs (it says just "Binary files ... differ and qdiff shows nothing -- at least I can do an external diff), cats are strange (letters are double spaced -- qcat shows a hex view), etc.
-----------------------------------------------------

So I think we could handle UTF-16 better in QBzr and provide autodetection of such files by BOM markers.