I need to get all the text from a page using BeautifulSoup. At BeautifulSoup's documentation, it showed that you could do soup.get_text() to do this. When I tried doing this on reddit.com, I got this error:


UnicodeEncodeError in soup.py:16
  'cp932' codec can't encode character u'\xa0' in position 2262: illegal multibyte sequence

I get errors like that on most of the sites I checked.
I got similar errors when I did soup.prettify() too, but I fixed it by changing it to soup.prettify('UTF-8'). Is there any way to fix this? Thanks in advance!

Update June 24
I've found a bit of code that seems to work for other people, but I still need to use UTF-8 instead of the default. Code:


texts = soup.findAll(text=True)

   def visible(element):
      if element.parent.name in ['style', 'script', '[document]', 'head', 'title']:
         return False
      elif re.match('', str(element)): return False
      elif re.match('\n', str(element)): return False
      return True

   visible_texts = filter(visible, texts)

   print visible_texts

Error is different, though. Progress?


UnicodeEncodeError in soup.py:29
'ascii' codec can't encode character u'\xbb' in position 1: ordinal not in range
(128)
python python-2.7 beautifulsoup
Edit
Report