Alex Rivera | Logout

Why is python decode replacing more than the invalid bytes from an encoded string?

Asked 2010-03-30T17:33:46.723
27

Trying to decode an invalid encoded utf-8 html page gives different results in python, firefox and chrome.

The invalid encoded fragment from test page looks like 'PREFIX\xe3\xabSUFFIX'

>>> fragment = 'PREFIX\xe3\xabSUFFIX'
>>> fragment.decode('utf-8', 'strict')
...
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 6-8: invalid data

UPDATE: This question concluded in a bug report to Python unicode component. The Issue is reported to be fixed in Python 2.7.11 and 3.5.2.

What follows is the replacement policies used to handle decoding errors in Python, Firefox and Chrome. Note how they differs, and specially how python builtin removes the valid S (plus the invalid sequence of bytes).

Python

The builtin replace error handler replaces the invalid \xe3\xab plus the S from SUFFIX by U+FFFD

>>> fragment.decode('utf-8', 'replace')
u'PREFIX\ufffdUFFIX'
>>> print _
PREFIX�UFFIX

Browsers

To tests how browsers decode the invalid sequence of bytes will use a cgi script:

#!/usr/bin/env python
print """\
Content-Type: text/plain; charset=utf-8

PREFIX\xe3\xabSUFFIX"""

Firefox and Chrome browsers rendered:

PREFIX�SUFFIX

Why builtin replace error handler for str.decode is removing the S from SUFFIX

(Was UPDATE 1)

According to wikipedia UTF-8 (thanks mjv), the following ranges of bytes are used to indicate the start of a sequence of bytes

  • 0xC2-0x
python security unicode screen-scraping
Edit
Report

1 Answer

9

the 0xE3 byte is one (of the possible) first bytes indicative of a 3-bytes character.

Apparently Python's decode logic takes these three bytes and tries to decode them. They turn out to not match an actual code point ("character") and that is why Python produces a UnicodeDecodeError and emits a substitution character
It appears, however that in doing so, Python's decode logic doesn't adhere to the recommendation of the Unicode Consortium with regards to substitution characters for "ill-formed" UTF-8 sequences.

See UTF-8 article on Wikipedia for background info about UTF-8 encoding.

New (final?) Edit: re the UniCode Consortium's recommended practice for replacement characters (PR121)
(BTW, congrats to dangra to keep digging and digging and hence making the question better)
Both dangra and I were partially incorrect, in our own way, regarding the interpretation of this recommendation; my latest insight is that indeed the recommendation also speaks to trying and "re-synchronize".
The key concept is that of the maximal subpart [of an ill-formed sequence].
In view of the (lone) example supplied in the PR121 document, the "maximal subpart" implies not reading-in the bytes which could not possibly be part of a sequence. For example, the 5th byte in the sequence, 0xE1 could NOT possibly be a "second, third or fourth byte of a sequence" since it isn't in the x80-xBF range, and hence this terminates the ill-formed sequence which started with xF1. Then one must try and start a new sequence with the xE1 etc. Similarly, upon hitting the x62 which too cannot be

answered 2010-03-30T17:50:46.250

Your Answer