What is Google crawling and indexing (crawling vs Indexing)

A search engine is a system run by programs which crawl the web for information including web pages, images, files, videos and other detectable documents. So crawling and indexing are very important, a crawler detects a new document or an update on a document, its data is stored on the search engine server in a process referred to as caching. The documents in the cache are then ranked in order of their importance on Search Engine Result Page (SERP).

What is Google Crawling and Indexing (crawling vs Indexing)

google-crawling-and-indexing what is Crawling and Indexing

What is Google Crawling?

Crawling basically means following a path. Crawling is the process by which search engines discover  content on the web, such as new sites or pages, changes to existing sites, and dead links.

In SEO, crawling means following your links and crawling around your website. When bots come to any website (any page), they follow other linked pages also on that website.

This is one of the reason why we create site maps, as they contain all of the links in the blog or website and Google’s bots can use them to look deeply into a website.

We can stop crawling and indexing certain parts of our site is by using the robots.txt file. For this, search engine uses a program which can be referred as  ‘crawler’, ‘bot’ or ‘spider’ (each search engine has its own type) which follows an algorithmic process to determine which sites to crawl and how often to that.

As search engine’s crawler moves through our website it will detect and record any links it finds on these pages and add them to the list that will be crawled later. This is how every new content is discovered.

Search bots wait upon signals from previously indexed pages, such as links, to be notified about the new content. So if you have created a new page on your website and linked to it from an existing page or the main menu, this would be a signal for the search bots that they should come visit and index it.

New pages can also be introduced to bots through Sitemaps and robots.txt files. Platforms such as WordPress will automatically alert search engines that you have created a new page.

Detection can be accelerated by verifying your website with search engines using Google Webmaster Tools or Bing Webmaster Tools.


What is Google Indexing?


Indexing is the process of adding webpages into Google search. Depending upon which meta tag you have used (index or NO-index), google will crawl and index your pages accordingly. A no-index tag means that the webpage will not be added into the web search’s index.

By default, every WordPress post and page is indexed.

A trick for ranking higher in search engines is to let only the important parts of your website be indexed. Do not index unnecessary archives like tags, categories, and all other useless pages.

Once a search engine processes each of the pages it crawls, it compiles a massive index of all the words it sees and their location on each page. It is essentially a database of billions of web pages.

This extracted content is then stored with the information which then organised and interpreted by the search engine’s algorithm to measure its importance compared to other similar pages.

Servers which are based all around the world allow users to access these pages almost instantaneously. Storing and sorting this information requires significant space and both Microsoft and Google have over a million of servers.


Crawling and Indexing serves two purposes –

  • to return results related to a search engine user’s query
  • to rank those results in order of importance and relevancy

The order of ranking is dependent with each search engine’s ranking algorithm. These algorithms are highly complex formulas, made even more advanced by the relationship your website has with external sites and its on-page SEO factors.

To sum up, indexing exists to ensure that users questions are promptly answered as quickly as possible.


Why Google can’t crawl and index pages


  • Badly written title, meta tags or author tags
  • Connectivity or DNS issue
  • Low pagerank
  • No or incorrect robots.txt file
  • Incorrectly configured URL parameters
  • Duplicate content


Have any query or suggestion.. Just comment below.

Like this post? Don’t forget to share it!

Leave a Reply