Scheduled Recrawls of your Website: WHYs, HOWs and WHENs

June 17, 2020
In scheduled crawl mode, the bots crawl your site at a scheduled time. Know why you need scheduled recrawls of your website on a regular basis and how.

Google crawlers are diligent workers. Any new article on an authoritative domain is crawled and indexed pretty fast. However, sometimes it’s better if you have scheduled recrawls of your website. Why? The basic answer is to remain updated on website health status.

men women vector scheduled recrawl text

Crawl Modes

The Google Search Console endorses two types of crawling:

  • Continuous crawl
  • Scheduled crawl

Continuous Crawl Vs. Scheduled Crawl

In this mode, the bots are crawling your website 24/7. Thereby, they ensure that new content is indexed as fast as possible. Contrarily, the bots crawl your site at a scheduled time in scheduled crawl mode.

After all the URLs are indexed, the search appliance does not recrawl any URLs. Moreover, crawling stops when the crawlers reach the license limit. However, it is necessary to perform scheduled recrawls from time to time.

Things to Bear in Mind about Scheduled Recrawls

In this post, you will learn why, how, and when to schedule recrawls of your website. But before we start, keep in mind two things.

  1. You cannot force Google to recrawl or index your site immediately. You must remain patient.
  2. Make sure your website is fresh and easily navigable. Moreover, you must link the new content to anything on the web.

Prompting Google to recrawl is not a regular task. However, if you have a site that is continually changing, it might be a good idea. In some cases, scheduled recrawls will speed up the indexing process.

When to Schedule Recrawls for your Site

Major Restructuring of your Site

It is a good idea to perform scheduled recrawls after massive changes in the site architecture. For example, a page migrating to https from http. Because the change is significant enough to affect all pages.

Merger or Acquisition

Sometimes you might need to change domain. Often changing domain occurs due to a merger/acquisition by a third-party. Therefore, you can ask Google for scheduled recrawls of both the website. Doing so makes things easier for Google.

Re-Branding

Rebranding of your site affects all of your pages. Because it requires a head-to-toe transformation and renovation. Often rebranding even affects the site architecture and URLs. Despite the domain remains the same, a scheduled recrawl should be done.

Website Redesign

As with rebranding, a website redesign demands substantial changes in HTML code or JavaScript. Therefore, you should opt for re-indexing the site.

Content Audit

During a content audit, many pages are removed or redirected. Hence, scheduling recrawls after a thorough content audit is a good decision. You should let the bots know about the changes, after all.

When to Schedule Recrawls for Individual Pages

Often you need to recrawl only specific individual page(s). There are 4 cases when you can request scheduled recrawls for individual pages.

Firstly, trendy content needs to be indexed faster than usual. Therefore, you can request an index for articles in the following categories i.e. breaking news, happening now, etc.

Secondly, if you add pre-defined tags i.e. schema markup in your existing post, a re-indexing might be a good option. Therefore you can have scheduled recrawls of that individual page.

Thirdly, because of significant external changes, you might update a post more drastically than usual. Hence, you can ask Google to re-index the specific post.

Finally, to comply with new standards, a webmaster may change the page title or meta description. Changing the page title demands a re-indexing. Similarly, changing meta descriptions demand scheduled recrawls too.

How to Perform Scheduled Recrawls on your Site

Scheduled Crawl

For scheduling a crawl, log into your Google Search Console. In the Admin Console page,

  • Go to Content Sources.
  • Click Web Crawl.
  • Then go to the Crawl Schedule tab.

Now you can schedule the crawl. For example, you can select the time and day of the crawling. You can also specify the maximum duration of the crawling.

Scheduled Recrawls

URL Inspection Tool

Firstly, open the URL Inspection Tool. The tool is handy when you have a few URLs to re-index. To ask Google to recrawl your individual page-

  1. Inspect the URL using the tool.
  2. Now, click Request indexing.
  3. The URL Inspection Tool will run a preliminary test. The test will determine if the URL has any obvious indexing issues.
  4. If there are no such issues, the page will be submitted for re-indexing.
  5. However, the tool will ask you to fix the issue if any.

As we said, requesting a recrawl does not translate to fast indexing. Rather, the crawlers inherently prioritized useful and relevant content.

Admin Console

There are few other methods up our sleeves for submitting your URLs for recrawling.

  • In the Admin Console, select Content Sources > Web Crawl > Start and Block URLs > Start URL > Recrawl
  • Content Sources > Web Crawl > Freshness Tuning > Recrawl these URL Patterns
  • To monitor the crawl, select Recrawl this URL > Index > Diagnostics > Index Diagnostics.

Adding a Sitemap

For submitting many URLs at a time, you can add a sitemap. However, scheduled recrawls don’t have any effect if your sitemap is the same as before. Therefore, mark your updated pages with <lastmod> in the site map.

Here are 3 ways that you can use the sitemap for recrawling purposes.

1. Using the Sitemap report.
2. The next method involves the ping tool. Copy-paste the following URL in a new browser window.


http://www.google.com/ping?sitemap=<full_URL_of_sitemap>

For example,

http://www.google.com/ping?sitemap=https://backlinkseo.com/sitemap.xml

Make sure that your sitemap is fully accessible. After submission, you will receive a notification.

3. The last method is using the robot.txt file. For this, you have to insert the following line in your robot.txt file. sitemap:


http://domain.com/my_sitemap.xml


Google will automatically find it next time it crawls your website.

A word of advice: Resubmitting URLs more than once does not make any difference. Moreover, do not use any of the abovementioned methods if you haven’t posted any significant updates. Finally, the answer to how much time before Google responds to your scheduled crawls depends.

As long as you make useful and high-quality content, your updates may be re-indexed in less than 30 minutes even.

You might also like

A Comprehensive Guide to Search Engine Visibility Strategy

Public
A Comprehensive Guide to Search Engine Optimization Strategy Search Engine Optimization is key to visibility, but it has so much more value than that alone. When you attract more traffic to your website, that, right there, is your chance to convert that traffic into leads and customers. Deeply connected to SEO is Public Relations, another […]
December 13, 2021

SERP: A Definitive Guide To Search Engine Results Page (Updated 2021)

Public
To have your place on SERP requires an overview of how these work and why search engines use them. Find more about some of the SERP components and the requirements that your content needs to meet in this definitive guide.
July 30, 2021

Affiliate Marketing In 2021: What Is It And How To Get Started?

Public
Photographer: Campaign Creators | Source: Unsplash Waking up at an ungodly hour and driving through total gridlock to reach your office on time? Or you might have transitioned to staring at your screen for almost two-third of the day and slogging through mind-numbing emails after emails until the sweet release at five o’clock. Sounds terrible? […]
July 23, 2021
1 2 3 21
Backlink SEO analyzes the links pointing to a website, and provides insights into their quality, quantity, and potential impact on SEO performance.
Have a question?
Contact us at
© 2014-2023, All Rights Reserved