Grapheme Usage

Draft

The goal is to allow the use of the appropriate grapheme clusters for given tasks, for a given language. See http://unicode.org/cldr/trac/ticket/2142. Please leave any feedback as comments on that ticket.

The idea is that we have explicit boundaries that represent certain common behaviors (codepoint breaks, or legacy grapheme cluster breaks), and we also have associations for a given language between a particular function and the explicit boundaries that should be used in that language for that function.

Here is a proposal for the structure in LDML:

<characters>

...

<grapheme-usage type="count">extended</grapheme-usage> <!-- when counting 'user characters' -->

<grapheme-usage type="drop-cap">legacy</grapheme-usage> <!-- paragraph drop-caps -->

<grapheme-usage type="selection">aksara</grapheme-usage> <!-- selection boundaries: highlighting, keyboard arrows, cut&paste -->

<grapheme-usage type="backspace">codepoint</grapheme-usage> <!-- delete previous character -->

<grapheme-usage type="delete">extended</grapheme-usage> <!-- delete next character -->

...

</characters>

The above would be tailorable per locale.

In segments/root.xml we have GraphemeClusterBreak. We interpret that as extended grapheme clusters for compatibility. We then add rules for:

• • LegacyGraphemeClusterBreak // as per UAX#29

  • AksaraGraphemeClusterBreak // the virama character connects extended clusters

  • CodepointGraphemeClusterBreak // constant, trivial, probably usually implemented in code

  • ExemplarGraphemeClusterBreak // uses the CLDR exemplar set in addition to extended clusters.

These would also be tailorable per locale (except CodePoint), but should be more rarely done.

Clients like ICU would add new constants for getting BreakIterators (or equivalents). These would be both corresponding to the new explicit rules:

• • legacy

  • extended = 'user-character'

  • aksara

  • codepoint

  • exemplar

And to the new 'function-based' breaks:

• • character_count

  • character_drop_cap

  • character_selection

  • character_backspace

  • character_delete

Related bugs

• • #2142, Alternate Grapheme Clusters (pedberg, 2.0)

  • #2975, Support legacy grapheme break (pedberg, 2.0)

  • #2825, Add aksha grapheme break (pedberg, 2.0)

  • #2992, Grapheme Clusters or a new break type - TR29 vs TR18? [about language-specific treatment of digraphs as clusters - ]

  • #2406, Add locale keywords to specify the type (or variant) of word & grapheme break (pedberg, 2.0)

  • There is also the suggestion to add another type which is beyond the scope of CLDR - a cluster type that treats ligatures as single clusters. This depends on font behavior.