Alex Rivera | Logout

Why doesn't this conversion to utf8 work?

Asked 2011-08-23T15:24:24.237
16

I have a subprocess command that outputs some characters such as '\xf1'. I'm trying to decode it as utf8 but I get an error.

s = '\xf1'
s.decode('utf-8')

The above throws:

UnicodeDecodeError: 'utf8' codec can't decode byte 0xf1 in position 0: unexpected end of data

It works when I use 'latin-1' but shouldn't utf8 work as well? My understanding is that latin1 is a subset of utf8.

Am I missing something here?

EDIT:

print s # ñ
repr(s) # returns "'\\xa9'"
python unicode encoding utf-8
Edit
Report

1 Answer

4

UTF-8 is not a subset of Latin-1. UTF-8 encodes ASCII with the same single bytes. For all other code points, it's all multiple bytes.

Put simply, \xf1 is not valid UTF-8, as Python tells you. "Unexpected end of input" indicates that this byte marks the beginning of a multi-byte sequence which is not provided.

I recommend you read up on UTF-8.

answered 2011-08-23T15:50:28.143

Your Answer