Edit Question - KnowledgeHub

Title Body

I know there are multiple similar questions about this error, and I've tried many of them without luck. The problem I'm having involves the byte \xA1 and is throwing ArgumentError: invalid byte sequence in UTF-8 I've tried the following with no success: "\xA1".encode('UTF-8', :undef => :replace, :invalid => :replace, :replace => "").sub('', '') "\xA1".encode('UTF-8', :undef => :replace, :invalid => :replace, :replace => "").force_encoding('UTF-8').sub('', '') "\xA1".encode('UTF-8', :undef => :replace, :invalid => :replace, :replace => "").encode('UTF-8').sub('', '') Each line throws the error for me. What am I doing wrong? UPDATE: The above lines fail only in IRB. However, I modified my application to encode lines of a CVS file using the same String#encode method and arguments, and I get the same error when reading the line from a file (note: it works if you perform the operations on the same string w/o using IO). bad_line = "col1\tcol2\tbad\xa1" bad_line.sub('', '') # does NOT fail puts bad_line # => col1 col2 bad? tmp = Tempfile.new 'foo' # write the line to a file to emulate real problem tmp.puts bad_line tmp.close tmp2 = Tempfile.new 'bar' begin IO.foreach tmp.path do |line| line.encode!('UTF-8', :undef => :replace, :invalid => :replace, :replace => "") line.sub('', '') # fail: invalid byte sequence in UTF-8 tmp2.puts line end tmp2.close # this would fail if the above error didn't halt execution CSV.foreach(tmp2.path) do |row| puts row.inspect # fail: invalid byte sequence in UTF-8 end ensure tmp.unlink tmp2.close tmp2.unlink end

Tags (comma-separated) Cancel