Slow Down Your Crawl during a Site Audit: WHYs and HOWs

June 8, 2020
Site audits take longer if you slow down your crawl but it helps your real-time users face less frequent speed issues. Know WHYs and HOWs of crawl-delay.

A site audit is a great way to ensure whether your website is fully optimized for your users. However, while conducting an audit, potential speed issues may arise for your real-time site visitors. Read more to find about crawl rate, crawl delay, and why and how you should slow down your crawl during a site audit.

slow down your crawl man on wall

What is Crawl Rate

Crawl-rate is the time elapsed between two separate bots sending requests to your website. Simply put, how many requests/second bot sends to your site while crawling it. For example, 10 requests per second. Hence, the higher crawl rate, the faster a bot crawls your website.

What is a Crawl Delay

Crawl Delay, on the other hand, means to signal the bot to wait for some time before sending another request. Such directives are sent to the crawlers so that they don’t overload the server. They are basically signals to slow down your crawl.

The interpretation of the crawl delay directive varies across different search engines. However, the basic idea remains the same.

Crawl Delay using BacklinkSEO

BacklinkSEO has 3 options available for slowing down your crawl during a site audit i.e. Default delay, follow robots.txt, and 1 URL/2 seconds.

The default delay option is what the name suggests: the bots will crawl your site at a default rate. By default rate, we mean that the bots will wait 1 second before crawling another page.

Now you might have a robots.txt file in your site. The file is also known as the Robots Exclusion Protocol. In theory, this protocol defines a roadmap for the bots to follow.
a road map of sorts, which each robot must follow. You can also specify the crawl delay in your protocol.

If you want our the BacklinkSEO bots to follow the protocol, just select the “respect robots.txt crawl-delay” option. The bots will do the rest.

Now if you don’t have such a directive in your robots.txt file, you can inform BacklinkSEO to crawl 1 page per 2 seconds. Doing so will significantly increase the length of the SEO Audit. However, here is why you should slow down your crawl during an audit.

Why Slow Down Your Crawl during a Site Audit

Suppose you have a lot of pages on your site. You have linked many of the pages from the index. Therefore, a bot while crawling your site may generate an excessive number of requests in a short period of time.

Usually hosting resources are monitored on an hourly basis. Such high website traffic can deplete your hosting resources. Therefore, it is a good idea to slow down your crawl. To do so, you can set the crawl delay to 1-2 seconds. Thus, the bots can crawl your website at a moderate pace without causing peaks frequently.

Crawl-delay directive causes a site audit to take longer than usual. However, doing so would help the real-time visitors on your website facing less frequent speed issues.

Crawl Delay Across Different Search Engines

Google

Google does not support the crawl-delay directive. Thus, Google bots will ignore the directive. However, if you want to stop other malicious bots to crawl your website, you can lower the crawl rate from the Google Search Console.

Here’s how you can slow down your crawl rate in Google Search Console.

  1. Log into the console.
  2. Select the website you want to optimize.
  3. You will find a slider for the crawl rate that ranges from Low to High
  4. The slider will also show the numeric values i.e. request/second
  5. You can also choose the default crawl rate

Baidu

Similarly, Baidu also doesn’t support the crawl delay directive. As with Google, You have to log into Baidu Webmaster Tools to reset your crawl rate. The process is almost similar to that of Google.

Yandex

Yandex supports crawl delay. For example, if you set a crawl delay to x seconds, the bots will wait x seconds before requesting another URL. You can also perform the same function in Yandex.Webmaster. The tool allows you to reset the crawl rate for any website.

Yahoo & Bing

The interpretation of crawl delay in Bing and Yahoo is a bit different from that of Yandex. If you set the crawl delay to x second, the search engines will divide 24 hours into x-second windows. Then, they will allow bots to crawl a maximum of one page within that window.

You might also like

A Comprehensive Guide to Search Engine Visibility Strategy

Public
A Comprehensive Guide to Search Engine Optimization Strategy Search Engine Optimization is key to visibility, but it has so much more value than that alone. When you attract more traffic to your website, that, right there, is your chance to convert that traffic into leads and customers. Deeply connected to SEO is Public Relations, another […]
December 13, 2021

SERP: A Definitive Guide To Search Engine Results Page (Updated 2021)

Public
To have your place on SERP requires an overview of how these work and why search engines use them. Find more about some of the SERP components and the requirements that your content needs to meet in this definitive guide.
July 30, 2021

Affiliate Marketing In 2021: What Is It And How To Get Started?

Public
Photographer: Campaign Creators | Source: Unsplash Waking up at an ungodly hour and driving through total gridlock to reach your office on time? Or you might have transitioned to staring at your screen for almost two-third of the day and slogging through mind-numbing emails after emails until the sweet release at five o’clock. Sounds terrible? […]
July 23, 2021
1 2 3 21
Backlink SEO analyzes the links pointing to a website, and provides insights into their quality, quantity, and potential impact on SEO performance.
Have a question?
Contact us at
© 2014-2023, All Rights Reserved