Alex Rivera | Logout

Is comparing two byte[] of utf-8 encoded strings the same as comparing two unicode strings?

Asked 2010-08-13T16:44:06.973
10

I found this in the wikipedia article on utf-8:

Sorting of UTF-8 strings as arrays of unsigned bytes will produce the same results as sorting them based on Unicode code points.

That would lead me to believe that for comparison purposes (sorting, binary search, etc) that comparing two byte arrays (i.e. byte-by-byte like memcmp) of utf-8 encoded strings would give the same results as comparing the actual unicode strings.

Is this true?

Edit
Report

1 Answer

8

It is the same as a code-point for code-point comparison, that is to say one that pays no attention to case-folding, cultural orderings, composition, or anything other than the Unicode value.

This is pretty useless when considering strings as a piece of human-readable text, but sometimes you just want to be able to put the strings into an ordering, as some algorithms (binary search as you say) need a consistent ordering, but the details of that consistent ordering is not significant.

It is important to note though, that the ordinal comparison on strings offered by .NET works on the UTF-16 used internally which does not maintain code-point ordering. If we compare a string with just the character U+FF61 and a string with just the character U+10002, then .NET will store the latter as surrogate pairs, of 0xD800 and 0XDC02.

Hence:

string.CompareOrdinal("\U0000ff61", "\U00010002");

and

string.Compare("\U0000ff61", "\U00010002", StringComparison.Ordinal);

both return values great than zero, even though the former is lower in code-point value than the latter (I used the \U form rather than the \u form to make that clearer).

If by "the actual unicode strings" you mean the .NET UTF-16 strings, then the answer to your question is no, for the opposite reason to that which led to your thinking it might work.

answered 2010-08-13T17:03:28.193

Your Answer