solr treats diacriticals as word breaks

at least two other tokenizing problems:

subject headings with ampersands, e.g. "sports & recreation", retrieve zero when clicked on. Note that there are headings with ampersands but no surrounding spaces ("Sports&Recreation"), and these can be retrieved by putting together the two words without the ampersand ("sportsrecreation"). This latter does not retrieve the ones with spaces around the ampersand.

some subject headings with slashes have this same problem, e.g. "Children's Books/Ages 4-8 Fiction". However, others, e.g. "Health/Fitness" work fine. The search "healthfitness" retrieves books with "health/fitness".

totally unclear to me how solr tokenizes.

The issue with ampersands in those subject links is unrelated, it's caused by thingrepr making unicode with escaped entities, which hash into different facet tokens than the unescaped versions, so the links don't find anything. That is discussed in #378841.

How solr tokenizes: http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters -- the part about whitespace is pretty clear, but I'm still not sure what solr does with slashes or ampersands.

One of the problems could be if we are using the de-composed Unicode forms, and solr expects pre-composed. So we would have a letter followed by an accent character, rather than have the two combined in a single unicode character. We may want to switch to pre-composed if that is the case.

Pre-composed could present some problems for transliterations (which is what we see with the nasa+jainism case) -- sometimes there isn't a pre-composed equivalent because the transliterations are artificial. But we'd probably still get better search results for most cases.

Note: Edward already has on his list to re-normalize the data to switch to pre-composed unicode, so that will solve the nara + jainism problem.

Karen, the issue with the ampersands in the subject links is discussed in bug #378841. It has absolutely nothing to do with solr tokenization.

I agree that switching normalization will help with these diacriticals and probably with some other issues.

Edward - thoughts?

If needed, keep open.