How Do Web Search Sites Work?

How do search engines work?

The world wide web is a big place. If you know the web address, or URL, of a site you can find it by typing it into the address bar along the top of your browser. But what if you don’t know the URL?

You can find pages by following links from other pages but usually it is easier to search for things using a search engine.

These are programs that search an index of the world wide web for keywords and display the results in order.

Finding things online

It is useful to think about the words that you use in your search. Too few and you will get too many results, too many and you may get no results at all.

When you want to find the answer to a question, such as, ‘What countries did the Vikings raid?’ You need to think about the important keywords.

You could just type the whole sentence into the search engine but it will probably ignore some words, like joining words and punctuation. Instead it might save time to search for keywords like 'countries', 'Vikings' and 'raid'.

Top tips for evaluating online content

Find out how search engines work in this short video.

How search engines make an index

To find what you’re after, a search engine will scan its index of webpages for content related to your search.

A search engine makes this index using a program called a ‘web crawler’. This automatically browses the web and stores information about the pages it visits.

Every time a web crawler visits a webpage, it makes a copy of it and adds its URL to an index. Once this is done, the web crawler follows all the links on the page, repeating the process of copying, indexing and then following the links. It keeps doing this, building up a huge index of many webpages as it goes.

Some websites stop web crawlers from visiting them. These pages will be left out of the index, along with pages that no-one links to.

The information that the web crawler puts together is then used by search engines. It becomes the search engine’s index. Every webpage recommended by a search engine has been visited by a web crawler.

Web crawlers automatically browse the web and store information about the pages they visit.

How do search engines order results?

Search engines sort results to show you the ones they think are the most useful.

PageRank is the best known algorithm which is used to improve web search results. In simple terms, PageRank is a popularity contest. The more links that point to a webpage, the more useful it will seem. This means it will appear higher up in the results.

The webpages on the first page of results are those that PageRank thinks are the best.

Search engines also pay attention to lots of other ‘signals’ when working out the order to show you results. For example how often the page is updated and if it is from a trustworthy domain.

There are many search engines to choose from. Different search engines use different algorithms. This means that some sites will give their results in a different order, or they may even show completely different results altogether.

Search engines use algorithms to sort the results and try to place the links which are most useful to you at the top of the page.