How Search Engines Work

Summary of this Page

Before we go any further, it would be a good idea for all of our readers to know exactly how search engines work. It is good to understand what is happening behind the scenes when you type a query into Google. This page will help you understand this proccess.

Two Kinds of Search Engines

Crawler Based Search Engine

A crawler based search engine is what you could call Google, Yahoo, MSN, and others. A crawler based search engine is composed of a crawler, which crawls the web, jumping from page to page, from link to link. Then the search engine indexes the page, or updates the page if anything on the page has changed if it was previously indexed. This is the most common type of search engine. The search engines that use this technique usually store a copy of your pages on their servers for further reference of keywords, etc.

Human-Powered Search Engines
Human-Powered Search Engines depend on humans to power them. They are much different than crawler based search engines, because it is usually much, much harder to get into then a normal search engine that indexes pages. An example of a human-powered search engine is DMOZ. DMOZ is a directory which is human edited, but is very hard to get into. But their website has a feature to search through all the sites, based upon what the site describtion is. These type of search engines don't always index, or keep a copy of a whole page. They often just keep your link, page title, and have a human or someone who is submitting the site pick keywords, which are reviewed. Then the results are based upon those peices of info.

Searching the Web

Searching the Web Isn't Actually Searching the Web

For anyone who is just begining SEO, I thought I would mention this clearly, in case you havn't already caught on. Search engines don't actually search the web. They search through an index the search engine keeps on its servers, and then find matches to the keywords you typed in, depending on how their algorithm is. So just remember, when searching the web, the search engine isn't actually going out into the internet and finding websites that match your query, it searches through its own index on its own servers.

Searching Enigine's Searching Software/Algorithm

Search engines have programs and algorithms to sift through millions upon millions of pages, deciding what pages best fit the query you entered. The software takes all the relevent pages, then ranks them from the most relevent and helpful (at least in the eyes of the engine), to the least relevent, and the least useful.

"Sponsored Links"

This topic will be discussed at length further down the road in this guide. But for right now, just remember that "Sponsored Links" are the links you see on search engines that arn't actually pages that were indexed and ranked through the search engine. Sponsored Links are paid for by the owner of the site. The owner picks the most relevent terms and keywords for his site, and then agrees to pay a certain amount of money for every person that clicks his link. So the sites come up only when people use the keywords or queries he picked. But don't worry. Search engines label these links as "Sponsored Links" or "Advertisments" instead of mixing them in with the non-pad results.

Different Search Engines and Different Results?

You may be asking now "Why do search engines bring me different results for the same query?" Well the answer is easy. Each search engine has its own unique algorithm, and its own unique way to rank websites and index them. But they all use basiclly the same principles of ranking.