Edit Question - KnowledgeHub

Title Body

I am attempting to use BeautifulSoup to parse through a DOM tree and extract the names of authors. Below is a snippet of HTML to show the structure of the code I'm going to scrape. <html> <body> <div class="list-authors"> <span class="descriptor">Authors:</span> <a href="/find/astro-ph/1/au:+Lin_D/0/1/0/all/0/1">Dacheng Lin</a>, <a href="/find/astro-ph/1/au:+Remillard_R/0/1/0/all/0/1">Ronald A. Remillard</a>, <a href="/find/astro-ph/1/au:+Homan_J/0/1/0/all/0/1">Jeroen Homan</a> </div> <div class="list-authors"> <span class="descriptor">Authors:</span> <a href="/find/astro-ph/1/au:+Kosovichev_A/0/1/0/all/0/1">A.G. Kosovichev</a> </div>  </body> </html> My point of confusion is that when I do soup.find, it finds the first occurrence of the div tag that I'm searching for. After that, I search for all 'a' link tags. At this stage, how do I extract the authors names from each of the link tags and print them out? Is there a way to do it using BeautifulSoup or do I need to use Regex? How do I continue iterating over every other other div tag and extract the authors names? import re import urllib2,sys from BeautifulSoup import BeautifulSoup, NavigableString html = urllib2.urlopen(address).read() soup = BeautifulSoup(html) try: authordiv = soup.find('div', attrs={'class': 'list-authors'}) links=tds.findAll('a') for link in links: print ''.join(link[0].contents) #Iterate through entire page and print authors except IOError: print 'IO error'

Tags (comma-separated) Cancel