What the 'Crawl Stats' section in Search Console tells you about whether Google actually likes your website

Here's something most website owners never think about: Google doesn't actually have to visit your site. It chooses to. And how often it bothers turning up, how many pages it looks at, and how patient it is whilst waiting for your server to respond – all of this tells you quite a lot about where you stand in Google's estimation.

The Crawl Stats section in Search Console is where Google gives you a fairly transparent window into this relationship. It's tucked away in the Settings menu, which is probably why hardly anyone looks at it. But if you want to know whether Google considers your site worth the bother, this is where you'll find out.

What crawl budget actually means

Google has a limited amount of time and server resources to spend crawling the entire internet. Even for a company with Google's infrastructure, there are limits. So it has to make decisions about which sites get visited, how often, and how thoroughly.

This is your crawl budget – essentially, how many pages Google is willing to fetch from your site within a given timeframe. If Google thinks your site is important, frequently updated, and generally worth paying attention to, you get a larger budget. If it thinks your site is a bit rubbish, slow, or just not very interesting, you get less.

The slightly uncomfortable truth is that you can publish new content all day long, but if Google isn't actually coming round to look at it, you're essentially shouting into the void.

The three numbers that matter

The Crawl Stats report gives you three main metrics, and whilst they look rather dry and technical, they're actually telling you a story about your relationship with Google.

Total crawl requests

This is simply how many times Googlebot fetched something from your site each day. The graph shows you the last 90 days, so you can spot trends rather than getting worked up about daily fluctuations.

If this number is going up over time, it generally means Google is finding more of your content interesting enough to check regularly. If it's dropping, that's worth investigating. Maybe you've stopped publishing as frequently. Maybe your content quality has dropped. Or maybe there's a technical issue making Google think twice about visiting.

For a small site – say, a few hundred pages – you might see a few hundred crawl requests per day. For larger sites, it could be thousands or tens of thousands. There's no universal "good" number, but the trend matters more than the absolute value.

Total download size

This tells you how much data Google downloaded from your site. If you've got lots of images, scripts, and other resources, this number will be higher. If you're running a fairly lean text-based site, it'll be lower.

What you're looking for here is whether the ratio makes sense. If Google is making thousands of requests but downloading very little data, it might be hitting a lot of error pages or redirects. If it's downloading enormous amounts of data relative to the number of requests, your pages might be bloated with unnecessary resources.

Average response time

This is the big one that people overlook. It measures how long your server takes to start sending data back after Google requests a page. Note that this isn't how long it takes to download the entire page – it's just the initial response.

If this number is creeping upwards, Google is experiencing your site as slow. And here's the thing: Google isn't patient. If your server is sluggish, Google will simply reduce how often it bothers visiting. Why would it waste time waiting around for your server to wake up when it could be crawling faster sites instead?

You want this number to be under 200 milliseconds if possible. Anything consistently over 500 milliseconds is a bit concerning. If you're regularly seeing numbers in the seconds rather than milliseconds, you've got a serious problem that's almost certainly affecting your crawl budget.

What the trends tell you

Looking at a single day's worth of data doesn't tell you much. Google's crawling behaviour fluctuates for all sorts of reasons. What you want to do is look at the trends over weeks and months.

If your crawl requests are steadily declining, Google is losing interest. This might be because you're not publishing fresh content, or because the content you are publishing isn't getting engagement, or because technical issues are making your site less crawlable.

If your response time is gradually increasing, your server is struggling. This might be because you're getting more traffic (which is good), but your hosting can't keep up (which is bad). Or it might be because your site has become bloated with plugins, scripts, and database queries that are slowing everything down.

The ideal pattern is stable or slightly increasing crawl requests, combined with consistently fast response times. This tells you that Google considers your site reliable and worth visiting regularly.

Why response time matters more than people think

Of the three metrics, response time is probably the most actionable and the most frequently ignored. People obsess over content and links, but if your server is slow, you're essentially telling Google not to bother visiting very often.

Think about it from Google's perspective. It's trying to keep its index fresh across billions of pages. If your site takes 800 milliseconds to respond whilst your competitor's takes 100 milliseconds, Google can crawl eight of their pages in the time it takes to fetch one of yours. Why would it allocate the same crawl budget to both sites?

This is why response time is effectively a multiplier on everything else you do. You can create brilliant content, build excellent links, and have perfect technical SEO, but if your server is sluggish, Google will still deprioritise you.

What to do if the numbers are disappointing

If you're seeing declining crawl requests or increasing response times, you need to work out why. Start with the technical basics: check your server response times using tools like WebPageTest or GTmetrix. If your hosting is slow, upgrade it or move to a better host. This is not the place to economise.

Look at your robots.txt file and make sure you're not accidentally blocking important content. Check Search Console for crawl errors – if Google is hitting lots of 404s or 500 errors, it'll reduce how often it visits.

Review your site structure. If you've got thousands of low-quality pages that nobody cares about, Google will waste your crawl budget fetching rubbish instead of your good content. Sometimes the solution is to delete or noindex the pages that don't deserve to rank.

Make sure your XML sitemap is up to date and submitted to Search Console. Whilst Google doesn't need your sitemap to find pages, it does use it as a signal about what you consider important and when it was last updated.

The bottom line

The Crawl Stats section won't tell you everything about your site's SEO health, but it will tell you something that's arguably more fundamental: whether Google thinks you're worth the effort. If those numbers are healthy and stable, you're in good shape. If they're declining, you've got a problem that needs addressing before you worry about anything else.

Most people spend their time obsessing over rankings and traffic whilst ignoring whether Google is actually bothering to crawl their new content. That's a bit like worrying about what you'll say at the party whilst ignoring the fact that you haven't been invited. Check your Crawl Stats occasionally. If the trend is downward, fix your server and your technical SEO before you do anything else.

Let us do all the hard work for you

Of course, we understand that not everyone will be interested or even want to learn about how console works. And that is why we offer an ongoing management service so that we take care of all of this for you. After we've built your website we then look after all of the content (including social media) and keep tracks on how console is indexing your site. To find out more about both our websites and our ongoing management. See below!

How We Build Your New Website

Ongoing Website Management Prices

Where do I find Crawl Stats in Google Search Console?

It's in the Settings menu. Most people miss it because it's tucked away there rather than being prominently displayed on the main dashboard.

Why is Google crawling my site less than it used to?

It depends on a few things. You might have stopped publishing fresh content, your server response time could be getting slower, or technical issues like crawl errors might be putting Google off. Check your Crawl Stats trends and server performance first.

Does crawl budget matter for small websites?

Yes, it does. Even small sites need Google to actually visit and fetch their new content, otherwise you're publishing into the void. If Google isn't bothering to crawl your pages regularly, they won't get indexed or ranked no matter how good they are.