Edit Question - KnowledgeHub

Title Body

I need to get all the text from a page using BeautifulSoup. At BeautifulSoup's documentation, it showed that you could do soup.get_text() to do this. When I tried doing this on reddit.com, I got this error: UnicodeEncodeError in soup.py:16 'cp932' codec can't encode character u'\xa0' in position 2262: illegal multibyte sequence I get errors like that on most of the sites I checked. I got similar errors when I did soup.prettify() too, but I fixed it by changing it to soup.prettify('UTF-8') . Is there any way to fix this? Thanks in advance! Update June 24 I've found a bit of code that seems to work for other people, but I still need to use UTF-8 instead of the default. Code: texts = soup.findAll(text=True) def visible(element): if element.parent.name in ['style', 'script', '[document]', 'head', 'title']: return False elif re.match('', str(element)): return False elif re.match('\n', str(element)): return False return True visible_texts = filter(visible, texts) print visible_texts Error is different, though. Progress? UnicodeEncodeError in soup.py:29 'ascii' codec can't encode character u'\xbb' in position 1: ordinal not in range (128)

Tags (comma-separated) Cancel