Edit Question - KnowledgeHub

Title Body

Trying to decode an invalid encoded utf-8 html page gives different results in python, firefox and chrome. The invalid encoded fragment from test page looks like 'PREFIX\xe3\xabSUFFIX' >>> fragment = 'PREFIX\xe3\xabSUFFIX' >>> fragment.decode('utf-8', 'strict') ... UnicodeDecodeError: 'utf8' codec can't decode bytes in position 6-8: invalid data UPDATE : This question concluded in a bug report to Python unicode component. The Issue is reported to be fixed in Python 2.7.11 and 3.5.2. What follows is the replacement policies used to handle decoding errors in Python, Firefox and Chrome. Note how they differs, and specially how python builtin removes the valid S (plus the invalid sequence of bytes). Python The builtin replace error handler replaces the invalid \xe3\xab plus the S from SUFFIX by U+FFFD >>> fragment.decode('utf-8', 'replace') u'PREFIX\ufffdUFFIX' >>> print _ PREFIX�UFFIX Browsers To tests how browsers decode the invalid sequence of bytes will use a cgi script: #!/usr/bin/env python print """\ Content-Type: text/plain; charset=utf-8 PREFIX\xe3\xabSUFFIX""" Firefox and Chrome browsers rendered: PREFIX�SUFFIX Why builtin replace error handler for str.decode is removing the S from SUFFIX (Was UPDATE 1) According to wikipedia UTF-8 (thanks mjv), the following ranges of bytes are used to indicate the start of a sequence of bytes 0xC2-0x

Tags (comma-separated) Cancel