How to remove non UTF-8 characters from text file

Asked 2012-10-21T16:16:55.300

102

I have a bunch of Arabic, English, Russian files which are encoded in utf-8. Trying to process these files using a Perl script, I get this error:

Malformed UTF-8 character (fatal)

Manually checking the content of these files, I found some strange characters in them. Now I'm looking for a way to automatically remove these characters from the files.

Is there anyway to do it?

Edit Report

1 Answer

0

Your method must read byte by byte and fully understand and appreciate the byte wise construction of characters. The simplest method is to use an editor which will read anything but only output UTF-8 characters. Textpad is one choice.

answered 2013-02-04T12:15:20.150

How to remove non UTF-8 characters from text file

1 Answer

Your Answer