What Is Crawl Budget?

Date First Published: 19th November 2022

Topic: Web Design & Development

Subtopic: SEO

Article Type: Computer Terms & Definitions

Difficulty Level: 6/10

Learn more about what crawl budget is in this article.

Crawl budget is how fast a search engine bot can crawl a website and the maximum number of pages it can crawl in a given period of time. For example, Google might crawl up to 10 pages a day on a small website and up to 5000 pages a day on a large website. Once the budget has been reached, the search engine bot will stop accessing the pages of the website for a certain period of time and move on to another website. The crawl budget of a website is automatically established by Google based on the size of the website, the performance of the website (e.g. the page speed, how often pages are updated, and how many errors Google comes across), and the number of sites that link to a website.

The two main aspects of crawl budget are:

• Crawl rate - The maximum number of requests per second a search engine bot can make to a website (e.g. 5 requests per second).
• Crawl demand - How important a website is to a search engine bot. This depends on the factors listed above.

Can I Adjust Google Crawl Rate?

Not anymore. Google has removed the old search console where it was possible to adjust the crawl rate by opening the Crawl Rate Settings page for the property. You may find some online guides that show a screenshot of the crawl rate settings page that can be accessed here, but this is outdated and it will now only display a page asking you to select a verified property and will then route you to the proper report, as shown below. Google will automatically optimise the crawl rate for websites and when Google crawls pages, it rarely slows websites down since it runs in the background.

Crawl budget is not something that most website owners have to worry about as Google is already good at finding and crawling pages. But, there are some reasons why website owners would need to monitor crawl budget. These include:

• They run a large website - A large website with over 10,000 pages makes it harder for Google to find them all. If the crawl budget is low, it will take a long time for Google to crawl them all and notice changes on certain pages.
• They have added a lot of new pages - If someone has recently added hundreds or thousands of new pages, they need to make sure that they have the right crawl budget to crawl and index all of them.
• They have lots of redirects, which can use up their crawl budget.