Google crawlers are diligent workers. Any new article on an authoritative domain is crawled and indexed pretty fast. However, sometimes it’s better if you have scheduled recrawls of your website. Why? The basic answer is to remain updated on website health status.
The Google Search Console endorses two types of crawling:
- Continuous crawl
- Scheduled crawl
Continuous Crawl Vs. Scheduled Crawl
In this mode, the bots are crawling your website 24/7. Thereby, they ensure that new content is indexed as fast as possible. Contrarily, the bots crawl your site at a scheduled time in scheduled crawl mode.
After all the URLs are indexed, the search appliance does not recrawl any URLs. Moreover, crawling stops when the crawlers reach the license limit. However, it is necessary to perform scheduled recrawls from time to time.
Things to Bear in Mind about Scheduled Recrawls
In this post, you will learn why, how, and when to schedule recrawls of your website. But before we start, keep in mind two things.
- You cannot force Google to recrawl or index your site immediately. You must remain patient.
- Make sure your website is fresh and easily navigable. Moreover, you must link the new content to anything on the web.
Prompting Google to recrawl is not a regular task. However, if you have a site that is continually changing, it might be a good idea. In some cases, scheduled recrawls will speed up the indexing process.
When to Schedule Recrawls for your Site
Major Restructuring of your Site
It is a good idea to perform scheduled recrawls after massive changes in the site architecture. For example, a page migrating to https from http. Because the change is significant enough to affect all pages.
Merger or Acquisition
Sometimes you might need to change domain. Often changing domain occurs due to a merger/acquisition by a third-party. Therefore, you can ask Google for scheduled recrawls of both the website. Doing so makes things easier for Google.
Rebranding of your site affects all of your pages. Because it requires a head-to-toe transformation and renovation. Often rebranding even affects the site architecture and URLs. Despite the domain remains the same, a scheduled recrawl should be done.
During a content audit, many pages are removed or redirected. Hence, scheduling recrawls after a thorough content audit is a good decision. You should let the bots know about the changes, after all.
When to Schedule Recrawls for Individual Pages
Often you need to recrawl only specific individual page(s). There are 4 cases when you can request scheduled recrawls for individual pages.
Firstly, trendy content needs to be indexed faster than usual. Therefore, you can request an index for articles in the following categories i.e. breaking news, happening now, etc.
Secondly, if you add pre-defined tags i.e. schema markup in your existing post, a re-indexing might be a good option. Therefore you can have scheduled recrawls of that individual page.
Thirdly, because of significant external changes, you might update a post more drastically than usual. Hence, you can ask Google to re-index the specific post.
Finally, to comply with new standards, a webmaster may change the page title or meta description. Changing the page title demands a re-indexing. Similarly, changing meta descriptions demand scheduled recrawls too.
How to Perform Scheduled Recrawls on your Site
For scheduling a crawl, log into your Google Search Console. In the Admin Console page,
- Go to Content Sources.
- Click Web Crawl.
- Then go to the Crawl Schedule tab.
Now you can schedule the crawl. For example, you can select the time and day of the crawling. You can also specify the maximum duration of the crawling.
URL Inspection Tool
Firstly, open the URL Inspection Tool. The tool is handy when you have a few URLs to re-index. To ask Google to recrawl your individual page-
- Inspect the URL using the tool.
- Now, click Request indexing.
- The URL Inspection Tool will run a preliminary test. The test will determine if the URL has any obvious indexing issues.
- If there are no such issues, the page will be submitted for re-indexing.
- However, the tool will ask you to fix the issue if any.
As we said, requesting a recrawl does not translate to fast indexing. Rather, the crawlers inherently prioritized useful and relevant content.
There are few other methods up our sleeves for submitting your URLs for recrawling.
- In the Admin Console, select Content Sources > Web Crawl > Start and Block URLs > Start URL > Recrawl
- Content Sources > Web Crawl > Freshness Tuning > Recrawl these URL Patterns
- To monitor the crawl, select Recrawl this URL > Index > Diagnostics > Index Diagnostics.
Adding a Sitemap
For submitting many URLs at a time, you can add a sitemap. However, scheduled recrawls don’t have any effect if your sitemap is the same as before. Therefore, mark your updated pages with <lastmod> in the site map.
Here are 3 ways that you can use the sitemap for recrawling purposes.
1. Using the Sitemap report.
2. The next method involves the ping tool. Copy-paste the following URL in a new browser window.
Make sure that your sitemap is fully accessible. After submission, you will receive a notification.
3. The last method is using the robot.txt file. For this, you have to insert the following line in your robot.txt file. sitemap:
Google will automatically find it next time it crawls your website.
A word of advice: Resubmitting URLs more than once does not make any difference. Moreover, do not use any of the abovementioned methods if you haven’t posted any significant updates. Finally, the answer to how much time before Google responds to your scheduled crawls depends.
As long as you make useful and high-quality content, your updates may be re-indexed in less than 30 minutes even.