Hyphenation of Russian text doesn't work in Firefox due to unrecognized encoding

If it works after extracting the Firefox patterns to /usr/share/hyphen, then it's a bug in the system patterns

lohmatii netizen, thank you for taking the time to report this bug and helping to make Ubuntu better. Please execute the following command, as it will automatically gather debugging information, in a terminal:
apport-collect 1107859
When reporting bugs in the future please use apport by using 'ubuntu-bug' and the name of the package affected. You can learn more about this functionality at https://wiki.ubuntu.com/ReportingBugs.

apport information

apport information

Status changed to 'Confirmed' because the bug affects multiple users.

A bit more information on this bug.

1. A workaround is to force the execution of the following JS code on every web page:

var defaultLang = 'ru';

// set LANG attribute if missing
var head=document.getElementsByTagName('head').item(0);
var html=document.getElementsByTagName('html').item(0);
var lang = html.getAttribute('xml:lang');
if (lang != '' && lang != null) {
html.lang = lang;
} else if (html.lang == '') {
html.lang = defaultLang;
}
html.lang = html.lang.substr(0, 2);

(borrowed from http://txt.arboreus.com/2012/03/20/force-css-hyphenation-in-firefox-greasemonkey-script.html)

To do so, you can use Firefox extension 'dotjs': https://addons.mozilla.org/en-US/firefox/addon/dotjs/

2. Even then Russian hyphenation works with errors. On many occasions, hyphen is misplaced one or even two letters from its correct location: проб-ежал instead of про-бе-жал; трен-ировок instead of тре-ни-ро-вок; лаге-рь instead of ла-герь, обслуж-ивающего instead of об-слу-жи-ва-ю-ще-го; палатк-ами instead of па-лат-ка-ми; ск-оростью instead of ско-ростью, etc.

I have an impression that in Ubuntu 12.04 errors took place almost always. In Ubuntu 14.10, they constitute about 20% of all occasions.

However, these errors are only found in Ubuntu Firefox. If you download the vanilla Firefox from mozilla.org and run it using the same profile, hyphenation is correct.

Hyphenation for Russian has for a while been provided by the hunspell-ru source package. Does this issue appear in Ubuntu 14.04 or later?

Sorry, I meant hyphen-ru.

@gunnarhj, I only have 12.04 upgraded to 15.04 and the problem persists there.

Thanks, Dmitri.

I attached an alternative dictionary (from the libreoffice-dictionaries source package). The currently installed file is:

/usr/share/hyphen/hyph_ru_RU.dic

As a test, can you please replace it with the attached file (after having unzipped it with gunzip), and let us know if it makes a difference.

No, Gunnar, your file produces the same erroneous hyphenation as the original one.

But I think I know now, what is going wrong. The dictionary provided in hyphen-ru package (and yours, too) is in the old single-byte encoding KOI8-R. Firefox does check the encoding (given in the first line of the dictionary file), but calculates the offset of the hyphen position in bytes instead of Unicode characters (or the other way round, I'm not yet sure).

So, if we take the stock hyph_ru_RU.dic from the Ubuntu package hyphen-ru, convert it to UTF-8 and edit the first line from 'KOI8-R' to 'UTF-8', Firefox hyphenates the text correctly.

That's very interesting, Dmitri. Can you then install hyphen-ru from this PPA:

https://launchpad.net/~gunnarhj/+archive/ubuntu/hyphen-ru

and let us know if it works. If it works on Firefox, can you also test with e.g. LibreOffice.

ping

Anybody who can test if the PPA package fixes this issue?

Hello. Sorry for the later reply. Your first package, hyphen-ru_20030310-1ubuntu2~ppa_all.deb, contains the dictionary not in UTF-8 encoding. The second one, hyphen-ru_20030310-1ubuntu2~ppa2_all.deb, works as supposed.

As for LibreOffice, it hyphenates Russian text correctly with both your dictionary and the one installed from Ubuntu repository (KOI8-R-encoded). So, the bug is probably in Firefox, not in the dictionary.

Thanks for letting us know!

Yeah, after my "ping" I realized that I had made a mistake with the first PPA package. I had let gedit guess the encoding, and gedit guessed wrong (thought it was ISO-8859-15). So maybe there is some ambiguity which cheats both Firefox and gedit.

Anyway, since the second PPA package works, let's ask for help to get it into the archive, may it be a workaround or not.

@Mattia: Do you think this should be forwarded to hyphen-ru in Debian, or should we take this opportunity to fix it in lo-dicts instead and start building hyphen-ru in lo-dicts both in Debian and Ubuntu?

I agree it's more of a firefox issue than a hyphen-ru thing, whatever we don to hyphen-ru would be a workaround. Still, I have a feeling firefox will take forever to be fixed…

I wouldn't like adding such a patch to lo-dicts, if such a thing would be needed there I'd try to do it a build time instead, and I'd do so very sadly.
Then, taking over a package in debian a kind of social problem :(, it needs to be done slowly and with great care, otherwise bad things may happen. If I were you in the meantime I'd
1) upload that hyphen-ru of yours to close this bug
2) open a bug against upstream lo-dicts to ask to convert the file to UTF-8

On 2016-03-14 12:23, Mattia Rizzolo wrote:
> I wouldn't like adding such a patch to lo-dicts, if such a thing
> would be needed there I'd try to do it a build time instead, and I'd
> do so very sadly.

Ack. With "fix it in lo-dicts" I actually meant upstream.

> Then, taking over a package in debian a kind of social problem :(, it
> needs to be done slowly and with great care, otherwise bad things may
> happen.

I happily hand over to you to handle that. ;)

> If I were you in the meantime I'd
> 1) upload that hyphen-ru of yours to close this bug

It's already in the sponsor queue.
http://reqorts.qa.ubuntu.com/reports/sponsoring

> 2) open a bug against upstream lo-dicts to ask to convert the file to
> UTF-8

If I understand it correctly, they don't want bug reports against extensions, but suggest that you contact the author instead... Possibly I'll do it, but it will have to wait.

Thanks Gunnar, it seems like firefox should deal with other encoding but if the encoding change is not confusing other users or the dict (unsure which other programs use it) then I guess it's fine to do as a workaround, uploading

This bug was fixed in the package hyphen-ru - 20030310-1ubuntu2

---------------
hyphen-ru (20030310-1ubuntu2) xenial; urgency=low

  * debian/patches/convert-to-utf8.diff:
    Convert pattern to UTF-8 (LP: #1107859).

 -- Gunnar Hjalmarsson <email address hidden> Sun, 13 Mar 2016 04:34:00 +0100

I don't agree this is a "wishlist" thing. Firefox should be able to use KOI8-R encoded hyphenation dicts as other applications are, so this is indeed a bug.

Further to comment #20, is this now resolved in Firefox?

This issue has sat incomplete for nearly three years without any response to comment #21 so it is being closed as "Invalid".

Please feel free to re-open this bug report if this is still an issue when using a currently supported release of Firefox and Ubuntu.