“Ignore Punctuation” Options

This page has moved to https://unicode-org.github.io/icu/userguide/collation/customization/ignorepunct.html

The contents below is out of date.

By default, spaces and punctuation characters add primary (base character) differences. Such characters sort less-than digits and letters. For example, the default collation yields “De Anza” < “de-luge” < “deanza”.

UCA/CLDR/ICU provide several options for “ignore punctuation” collation settings, also known as Variable Weighting or Alternate Handling. These options change the sorting behavior of “variable” characters algorithmically. “Variable” characters are those with low (but non-zero) primary weights up to a threshold, the “variable top”. By default, CLDR and ICU treat spaces and punctuation as variable. (This can be changed via API.) The DUCET also includes most symbols.

Non-Ignorable

The default behavior in CLDR & ICU, shown above, is to not ignore punctuation (alternate=non-ignorable) but to map variable characters to their normal primary collation elements.

All of the following options cause variable characters to be ignored on levels 1..3. Only when strings compare equal up to the tertiary level may variable characters make a difference, depending on the options.

Blanked

The simplest option is to “ignore punctuation” completely, as if all variable characters (and following combining marks) had been removed from the input strings before comparing them.

For example: “De Anza” = “De-Anza” = “DeAnza”.

In ICU, this option is selected with alternate=shifted and strength=primary|secondary|tertiary. (ICU does not support Blanked combined with strength=identical.)

The implementation “blanks” out all weights of the variable characters’ collation elements.

With all of the following options, variable characters are ignored on levels 1..3 but add distinctions on level 4 (quaternary level).

Shifted

Among strings that compare tertiary-equal, that is, they contain the same letters, accents and casing:

Sorts all variable characters less-than (before) regular characters.
Appending a variable character makes a string sort greater-than the string without it.
Inserting a variable character makes a string sort less-than the string without it.
Inserting a variable character earlier in a string makes it sort less-than inserting the variable character later in the string.

The result is similar to Merging Sort Keys (with shorter prefixes sorting less-than longer ones), like in last-name+first-name sorting, except only among tertiary-equal strings.

For example: “de-luge” < “delu-ge” < “deluge” < “deluge-”.

In ICU, this option is selected with alternate=shifted and strength=quaternary|identical.

The implementation “shifts” the primary weight p of the collation element [p, s, t, q] of each variable characters down three levels: [0, 0, 0, p]. Regular characters with primary collation elements get a high quaternary weight, higher than that of any variable character.

Note that this behavior is different from collation on secondary and tertiary level, because normal collation elements get low secondary & tertiary weights but high quaternary weights. Adding an accent difference anywhere makes a string sort greater-than the string without it, and adding an accent difference earlier makes it sort greater-than adding it later. For example, “deanza” < “deanzä” < “deänza” < “dëanza”. (Compare the ‘ä’/‘ë’ positions here with the ‘-’ positions above.)

Shift-Trimmed

Note: This method is not currently implemented in ICU.

Among strings that compare tertiary-equal:

Sorts variable characters sometimes less-than, sometimes greater-than regular characters.
Inserting a variable character anywhere makes a string sort greater-than the string without it. (The string without variable characters gets an empty quaternary level.)
Inserting a variable character earlier in a string makes it sort less-than inserting the variable character later in the string.

For example: “deluge” < “de-luge” < “delu-ge” < “deluge-”.

The Shift-Trimmed method works like Shifted, except that trailing high-quaternary weights (from regular characters) are removed (trimmed). Compared with Shifted, the Shift-Trimmed method sorts strings without variable characters before ones with variable characters added, rather than producing the equivalent of Merging Sort Keys.

Shift-Trimmed is more complicated to implement than all of the other options: When comparing strings, a lookahead (or equivalent) is needed to determine whether a non-variable character gets a zero quaternary weight (if no variables follow) or a high quaternary weight (if at least one variable follows). When building sort keys, trailing high/common quaternary weights are trimmed (backed out) at the end of the quaternary level.

Variable-After

Note: This method is not currently implemented in ICU.

Among strings that compare tertiary-equal:

Sorts all variable characters greater-than (after) regular characters.
Inserting a variable character anywhere makes a string sort greater-than the string without it. (Like Shift-Trimmed.)
Inserting a variable character earlier in a string makes it sort greater-than inserting the variable character later in the string. (Like accent differences.)

For example: “deluge” < “deluge-” < “delu-ge” < “de-luge”.

The implementation “shifts” the primary weight p of the collation element [p, s, t, q] of each variable characters down three levels: [0, 0, 0, p]. Regular characters with primary collation elements get a low quaternary weight, lower than that of any variable character. This is consistent with collation on secondary and tertiary levels but unlike Merging Sort Keys.

This method extends the UCA well-formedness condition 2 to apply to quaternary weights. (UCA versions before UCA 6.2 did not limit WF2 to secondary & tertiary weights, which meant that several of the Variable Weighting options technically created ill-formed quaternary weights.)