Google recently released a podcast on crawl budgets and the factors that affect how Google indexes content.
Martin Splitt and Gary Illyes of Google provided us with their insights on indexing the web, as understood from Google’s point of view.
In this post, we’ll look at what a crawl budget is and what you can do to optimize it. We’ll also look at the indexing process and how Google stores and finds content.
So what is a crawl budget?
The number of pages Google will crawl on your website daily is known as a crawl budget. This figure fluctuates a little bit from day to day, but overall, it is pretty consistent. Google could visit five pages on your website per day, 10,000 pages, or even 5,000,000 pages in a single day.
The size of your site, the health of your site (determined by the number of errors Google finds), and the number of links pointing to your site all play a role in deciding how many pages Google crawls – your ‘budget.’
The concept of a crawl budget was something created outside of Google by the search community.
On the podcast, Gary Illyes explained that there wasn’t any one thing internally within Google that corresponded with the idea of a crawl budget. Instead, what was happening inside Google involved a multitude of metrics.
However, he went on to explain that after much deliberation and collaboration within teams, Google settled on the definition of crawl budget as being: “the number of URLs Googlebot can and is willing or is instructed to crawl, for a given site.”
Should I be worried about it?
Gary and Martin talked about how the vast majority of sites don’t need to worry about the crawl budget.
They blamed blogs within the search industry for creating a false sense of fear regarding the crawl budget when, in reality, “over 90% of sites on the internet don’t have to worry about it”. Google is VERY good at finding and indexing pages.
So relax!
Nevertheless, there are a few circumstances in which you may need to pay attention to the crawl budget:
> You run a big site: If you have a website, such as an e-commerce site, with over 10,000 pages, Google can struggle to find all of them.
> You have a lot of redirects: A high number of redirects and redirect chains can eat up your crawl budget.
> You’ve added a lot of new pages: If you recently launched a new addition to your website that has hundreds of pages, you want to make sure that you have the crawl budget to swiftly index them all.
How can I maximize my crawl budget?
As previously mentioned, most sites do not need to worry about their crawl budget. However, if you believe you’re an exception, here are a few ways in which you can maximize your crawl budget:
> Improving site speed: Slow-loading pages use up valuable Googlebot time. Increasing your site’s speed not only improves the users’ experience but also increases the crawl rate.
> Using a lot of internal links: Pages with numerous internal and external links directed to them are prioritized by Googlebot. Your site’s internal links direct Googlebot to all of the pages that you want to be indexed.
> You’ve added a lot of new pages: If you recently launched a new addition to your website that has hundreds of pages, you want to make sure that you have the crawl budget to swiftly index them all.
What is indexing?
Search engines organize the data and websites they are aware of through indexing. It involves gathering information with the purpose of building an index of pages or other data. Indexing is a big part of a typical search engine process, arguably the most crucial part, as content that is not indexed cannot rank in search results.
The indexing process involves crawlers from Google going from link to link, looking for new web pages. They use a sitemap or former tracking data to find content and information.
After processing the data, they analyze it based on a variety of criteria, including the quality of the content, keywords, meta tags, and word count. This data is then saved and used to populate SERPs in the future.
How can I optimize my website for indexing?
Making a crawler’s job as simple as possible is only going to benefit your site. It’s important to avoid placing any barriers that hinder your site from getting indexed. Ways that you can optimize your website for indexing include:> Blocking pages that you don’t want to be indexed: Low-quality pages can be harmful to your website’s SEO. A no-index tag or a 301 redirect can block pages from being indexed.
> Creating and submitting a sitemap: This will ensure that web crawlers can find the appropriate pages on your website and steer clear of the ones you don’t want people to see. Here, the use of canonical tags and robot meta tags is crucial. This will also assist you in prioritizing your pages in order of importance.
> Using Google Search Console: You can use Google Search Console (GSC) to check if your pages are being crawled effectively. If they aren’t, you can make adjustments to ensure the right pages on your website are being indexed quickly. In addition, by providing GSC with the URL you want Googlebot to visit and pressing fetch, you can affect indexing directly yourself. Crawlers will visit your site and swiftly index it as a result.
Gary reiterated on the podcast that Google doesn’t have infinite space and cannot index everything, and only wants to index content that its algorithm determines may be searched for at some point. Therefore, it’s important to be selective by indexing only the content that matters.
