Indexing Learn how Search Engines Work, What They Like & What They Don't

Learn how search engines work, what they like and importantly, what they don't. Browse our helpful resources for learning about search engine indexing.

Posted On: 2nd July 2020
Updated On: 7th July 2020

What is a Website Crawl?

Unfortunately, Search Engines like Google don’t magically know what websites exist across the internet, what information they provide and what keywords they’re targeting. This is where a web crawl comes in.

You may have heard of spiders, robots and bots, but these are web crawlers (a web crawl is far less exciting than imagining thousands of spiders collecting information across the web!). Whatever you call it, the function of them is the same: to crawl across the world wide web and deliver appropriate pages to the search engine results page.

Web crawlers gather information about the page such as the metadata, content and internal links and store them in their index so that search engine algorithms can decide when to show them for a particular search. Factors such as internal and external links help crawlers identify which pages are important so can have a big effect of keyword rankings.

Think of it like a library. Crawlers are looking at each section of the internet and reading all the pages available so they know what information is where. This way, when someone searches for something like ‘SEO Peterborough‘, they’ll know which pages to show to the user.

As you can imagine, there’s a lot of information web crawlers need to gather, especially when websites are constantly updating their information, services and resources. This why each website has a limited timeframe and ‘crawl budget’ available. How can we get the best from this? We make it as easy as possible for crawlers to understand what our pages are about so they can rank them higher in the search results – Search Engine Optimisation.

Tools like Google Search Console and Bing Webmaster Tools help website owners to communicate with the search engines by enabling them to list all the pages on their website they want (and don’t want) crawlers to look at – otherwise known as a sitemap. They also let you submit new changes to page as a way of saying “I’ve updated my page, please take a look!”.

A web crawl is an essential part of getting your website listed in Search Engines – just make sure they know what to list you for!