Finding Information
10 AI Speak: Search Engine Indexing
A search engine takes keywords (the search query) entered into the search box, and tries to find the web documents that answer the information. It then displays the information in an easily accessible form, with the most relevant page at the top. In order to do this, the search engine has to start by finding documents on the web and tagging them so that they are easy to retrieve. Let us see in broad strokes what is involved in this process:
Step 1: Web crawlers find and download documents
After a user enters a search query, it is too late to look at all the content available on the internet1. The web documents are looked at beforehand, and their content is broken down and stored in different slots. Once the query is available, all that needs to be done is to match what is in the query with what is in the slots.
Web crawlers are pieces of code that find and download documents from the web. They start with a set of website addresses (URLs) and look inside them for links to new web pages. Then, they download and look inside the new pages for more links. Provided the starting list was diverse enough, crawlers end up visiting every site that allows access to them, often multiple times, looking for updates.
Step 2: The document gets transformed into multiple pieces
The document downloaded by the crawler might be a clearly structured web page with its own description of content, author, date, etc. It can also be a badly scanned image of an old library book. Search engines can usually read a hundred different document types1. They convert these into html or xml and store them in tables (called BigTable in the case of Google).
A table is made up of smaller sections called tablets in which each row of the tablet is dedicated to one web page. These rows are arranged in some order which is recorded along with a log for updates. Each column has specific information related to the webpage which can help with matching the document content to the contents of a future query. The columns contain:
-
- The website address which may, by itself, give a good description of the contents in the page, if it is a home page with representative content or a side page with associated content;
- Titles, headings and words in bold face that describe important content;
- Metadata of the page. This is information about the page that is not part of the main content, such as the document type (eg, email or web page), document structure and features such as document length, keywords, author names and publishing date;
- Description of links from other pages to this page which provide succinct text regarding different aspects of the page content. More links, more descriptions and more columns used. The presence of links is also used for ranking, to determine how popular a web page is. (Take a look at Google’s Pagerank, a ranking system that uses links to and from a page to gauge quality and popularity).
- People’s names, company