As a pioneer in search innovation, Google announced the decision to deprecate the crawl rate limiter tool in Search Console on January 8th, 2024. The tool has been available for over a decade, but with the vast improvements Google have made to their crawling logic and other tools now available, the need for the crawl rate limiter tool has diminished.
Making room for Googlebot
Googlebot is a generic name used to cover Google’s two types of web crawlers:
- Googlebot Desktop
- Googlebot Smartphone
Googlebot was designed to run simultaneously by thousands of machines to improve performance and scale as the web continues to grow. The purpose of this tool, much like the crawl rate limiter, is to collect documents from the web to build a searchable index for Google’s search engine.
What was the crawl rate limiter used for?
Introduced over 15 years ago, the crawl rate limiter tool’s purpose was to provide publishers with a way to control Googlebot crawling so that it didn’t overwhelm the server.
Google’s announcement stated that crawling algorithms have reached a point where Googlebot can automatically detect when a server is reaching capacity and take action to slow down the crawl rate.
Optimise your site for Googlebot
Here are some of our most important SEO strategies to make Googlebot’s job easier:
- Improve website visibility with the “Allow: /” command in the robots.txt file.
- Avoid using the ‘nofollow’ tag on internal links on your site. This specifically tells crawlers not to follow them back to their source.
- Create a sitemap for your website. This enables Googlebot to learn about your site and find all available content.
- Utilise Google Search Console.
In summary
Improving your site with Googlebot – and other crawlers – in mind will improve website traffic, conversions and sales. The decision to deprecate the crawl rate limiter tool was made to improve the usability of Search Console as it is less cluttered with tools that are rarely used. Publishers who continue to have problems with Googlebot’s crawl rate can still use the Googlebot report form to send feedback to Google. Publishers who continue to have problems with Googlebot’s crawl rate can still use the Googlebot report form to send feedback to Google.
If you are interested in improving your site visibility, contact us today to learn about our services.
