Google crawlers are diligent workers. Any new article on an authoritative domain is crawled and indexed pretty fast. However, sometimes it’s better if you have scheduled recrawls of your website. Why? The basic answer is to remain updated on website health status.
The Google Search Console endorses two types of crawling:
In this mode, the bots are crawling your website 24/7. Thereby, they ensure that new content is indexed as fast as possible. Contrarily, the bots crawl your site at a scheduled time in scheduled crawl mode.
After all the URLs are indexed, the search appliance does not recrawl any URLs. Moreover, crawling stops when the crawlers reach the license limit. However, it is necessary to perform scheduled recrawls from time to time.
In this post, you will learn why, how, and when to schedule recrawls of your website. But before we start, keep in mind two things.
Prompting Google to recrawl is not a regular task. However, if you have a site that is continually changing, it might be a good idea. In some cases, scheduled recrawls will speed up the indexing process.
It is a good idea to perform scheduled recrawls after massive changes in the site architecture. For example, a page migrating to https from http. Because the change is significant enough to affect all pages.
Sometimes you might need to change domain. Often changing domain occurs due to a merger/acquisition by a third-party. Therefore, you can ask Google for scheduled recrawls of both the website. Doing so makes things easier for Google.
Rebranding of your site affects all of your pages. Because it requires a head-to-toe transformation and renovation. Often rebranding even affects the site architecture and URLs. Despite the domain remains the same, a scheduled recrawl should be done.
As with rebranding, a website redesign demands substantial changes in HTML code or JavaScript. Therefore, you should opt for re-indexing the site.
During a content audit, many pages are removed or redirected. Hence, scheduling recrawls after a thorough content audit is a good decision. You should let the bots know about the changes, after all.
Often you need to recrawl only specific individual page(s). There are 4 cases when you can request scheduled recrawls for individual pages.
Firstly, trendy content needs to be indexed faster than usual. Therefore, you can request an index for articles in the following categories i.e. breaking news, happening now, etc.
Secondly, if you add pre-defined tags i.e. schema markup in your existing post, a re-indexing might be a good option. Therefore you can have scheduled recrawls of that individual page.
Thirdly, because of significant external changes, you might update a post more drastically than usual. Hence, you can ask Google to re-index the specific post.
Finally, to comply with new standards, a webmaster may change the page title or meta description. Changing the page title demands a re-indexing. Similarly, changing meta descriptions demand scheduled recrawls too.
For scheduling a crawl, log into your Google Search Console. In the Admin Console page,
Now you can schedule the crawl. For example, you can select the time and day of the crawling. You can also specify the maximum duration of the crawling.
Firstly, open the URL Inspection Tool. The tool is handy when you have a few URLs to re-index. To ask Google to recrawl your individual page-
As we said, requesting a recrawl does not translate to fast indexing. Rather, the crawlers inherently prioritized useful and relevant content.
There are few other methods up our sleeves for submitting your URLs for recrawling.
For submitting many URLs at a time, you can add a sitemap. However, scheduled recrawls don’t have any effect if your sitemap is the same as before. Therefore, mark your updated pages with <lastmod> in the site map.
Here are 3 ways that you can use the sitemap for recrawling purposes.
1. Using the Sitemap report.
2. The next method involves the ping tool. Copy-paste the following URL in a new browser window.
http://www.google.com/ping?sitemap=<full_URL_of_sitemap>
For example,
http://www.google.com/ping?sitemap=https://backlinkseo.com/sitemap.xml
Make sure that your sitemap is fully accessible. After submission, you will receive a notification.
3. The last method is using the robot.txt file. For this, you have to insert the following line in your robot.txt file. sitemap:
http://domain.com/my_sitemap.xml
Google will automatically find it next time it crawls your website.
A word of advice: Resubmitting URLs more than once does not make any difference. Moreover, do not use any of the abovementioned methods if you haven’t posted any significant updates. Finally, the answer to how much time before Google responds to your scheduled crawls depends.
As long as you make useful and high-quality content, your updates may be re-indexed in less than 30 minutes even.
