Crawl Budget: What Google's Rewritten Documentation Actually Changes

In short — On July 22, 2026, Google rewrote its crawl budget guide and added three statements it had never published before: every site starts with the same default crawl capacity limit, deliberately conservative; that capacity is shared across all of Google's crawlers; and returning 304 (Not Modified) is now an explicit recommendation for saving crawl resources. The documentation also moved: it no longer sits under Search Central but on a dedicated site, developers.google.com/crawling, formalizing that crawling is shared infrastructure serving Search, Shopping, News, AdSense and Gemini. As of July 26, 2026, the localized versions of the page are still the March 5, 2026 revision and contain none of these additions. None of these mechanisms are directly measurable in Search Console: only server logs show which Google crawler is actually consuming your capacity, and at what cost.

A documentation update is almost never news. This one is, for a simple reason: Google didn't just reword its crawl budget page, it put in writing three mechanisms the SEO community had been inferring from server logs for years without ever being able to cite a source. The change was spotted on July 22, 2026 by Barry Schwartz and Glenn Gabe. It belongs to a series of edits that began in November 2025, all pointing the same way: Google wants to crawl the web more cheaply, and it is starting to explain how it rations.

This article doesn't paraphrase the documentation. It compares the previous version to the new one, isolates what is genuinely new, and turns each addition into a concrete check to run against your log files.

The documentation left Search for a crawling infrastructure section

The crawl budget guide no longer lives under developers.google.com/search/. Since December 2025 it has been hosted on a separate site, developers.google.com/crawling/, alongside the robots.txt specification, the user-agent reference, HTTP status code behavior and DNS error troubleshooting. Google justifies the move by noting that its crawling infrastructure serves many products beyond Search: Shopping, News, Gemini, AdSense.

This isn't cosmetic reorganization, it's a change of scope. Crawl budget stops being an SEO concept and becomes an infrastructure concept. The practical consequence shows up in the page's closing section, which now talks about optimizing content quality "for the Google product you're targeting": an ecommerce catalog and a news site no longer pull the same lever, because they aren't crawled by the same robots or for the same reasons.

Every site starts with the same deliberately conservative limit

This is the single newest sentence in the update: every site begins with the same default crawl capacity limit, described as conservative, and Google's systems adjust it automatically over time if demand grows and the site stays healthy. This had never been documented.

The implication is direct in three situations. A new site isn't throttled because it's new — everyone is throttled at the start, and the ceiling rises based on server behavior. A migration to a new hostname restarts from that same floor, since budget is calculated per hostname. And capacity lost after a server incident isn't restored instantly: the adjustment is described as gradual, with no timeframe given.

Crawl capacity is shared across all of Google's crawlers

The second addition is probably the most consequential: each Google crawler has its own crawl demand, but the capacity limit is common to all of them. High demand from one crawler therefore reduces what's left for the others.

In practice, AdsBot crawling your dynamic ad targets, Storebot-Google working through your merchant feeds and Googlebot-Image refetching your visuals all draw from the same pool of server time as Googlebot. An ecommerce site launching a large Shopping campaign can see crawling of its editorial pages slow down without a single SEO change having been made. Until now this was an empirical observation from log files; it is now documented behavior.

Crawl budget is measured in connection time, not URL count

The previous version defined the capacity limit as a maximum number of parallel connections plus a delay between fetches. The new version defines it as the total amount of time your server spends holding connections open for Google, factoring in both the number of simultaneous connections and how long each one lasts. Google also surfaces the internal name of this limit: hostload, the same term behind the "Hostload exceeded" message in the URL Inspection tool.

The difference isn't semantic. Under the old model you reasoned in requests; under the new one, the unit is the second of connection. A page that is slow to generate structurally costs more than a fast one, at identical URL volume. This formalizes a trade-off log analysis has shown for years: lowering the average response time served to crawlers widens coverage mechanically, whereas pruning a handful of URLs barely moves it.

429 responses and latency lower your capacity

The old wording simply said the limit went up if a site responded quickly for a while, and down if it slowed or returned server errors. The new wording is considerably more precise: it names latency and Time-to-First-Byte as health indicators, and adds 429 (Too Many Requests) to the list of signals that lower the limit, alongside 5xx errors.

That detail deserves an audit of its own. Plenty of sites serve 429 responses to Googlebot without knowing it, because a web application firewall, a CDN rate-limiting rule or an anti-scraping module reads a crawl spike as an attack. The site then throttles itself, and the effect lingers, since the limit recovers gradually. These responses appear nowhere except in server logs, filtered on verified Google IP ranges.

Google now explicitly recommends HTTP caching and 304 responses

The section on load efficiency gained two recommendations that weren't there before. The first is obvious enough: optimize server response times. The second is more interesting: use HTTP caching and support the 304 (Not Modified) status code. If a page hasn't changed since Google last crawled it, a 304 tells Google to reuse its cached copy, saving your server's bandwidth and resources.

Google's crawler reference spells out the expected implementation: its infrastructure supports the ETag / If-None-Match pair and the Last-Modified / If-Modified-Since pair, and recommends ETag first because it avoids date formatting problems. In other words, Google is no longer only asking you to reduce the number of URLs it has to crawl — it's asking you to make each recrawl cheaper. That is consistent with a budget now denominated in connection time.

The myths now live on their own page, along with an apparent contradiction

The "myths and facts" section that used to close the old guide was split off in December 2025 into a standalone page. It keeps its reference answers: the non-standard crawl-delay rule isn't processed by Google's crawlers, compressing sitemaps doesn't increase crawl budget, crawling isn't a ranking signal, and alternate URLs such as AMP or hreflang as well as embedded resources (CSS, JavaScript, XHR fetches) do consume budget.

One item on that page deserves careful reading, because it is frequently miscited: Google states that pages serving 4xx status codes, except 429, don't waste crawl budget, on the grounds that the request returned a status code and no content. That appears to contradict the crawl budget guide, which asks you to eliminate soft 404 errors. There is no contradiction: a true 404 costs one bare request and takes the URL out of the queue, whereas a soft 404 returns a 200 with a full page — real connection time — and keeps being crawled indefinitely. The lesson is that the cost of an error depends on the status code served, not on what the page looks like.

What the documentation still doesn't say

Three blind spots remain, and naming them is more useful than filling them with guesses. Google gives no numbers: not the value of the default limit, not the response time threshold considered problematic, not the recovery period after an incident, which is described only as gradual. Google also doesn't specify exactly which crawlers fall inside the shared capacity: the wording covers all crawlers, but the status of an agent like Google-Extended, which governs whether your content trains Gemini models, isn't spelled out. Finally, the documentation says nothing about non-Google crawlers, even though they consume the same server resources.

That last point is the most structurally important in 2026. GPTBot, ClaudeBot, PerplexityBot or Bytespider don't consume Google's crawl budget in the strict sense, since that allowance is calculated by Google for Google. But they stretch your response times and can trigger the errors and 429 responses that do lower the limit Google grants you. AI crawler load therefore affects your Googlebot crawling indirectly, through the server health channel.

Localized versions are one revision behind

As of July 26, 2026, the French page carries a March 5, 2026 date, against July 22 for the English version. It still describes the capacity limit as a number of parallel connections plus a waiting delay, and mentions neither the shared default floor, nor capacity sharing between crawlers, nor 429 responses, nor the HTTP caching and 304 recommendation.

This is a habit worth building into any documentation watch: translations ship with a variable lag, and an article written in good faith from a localized page can be obsolete on the day it publishes. The last-updated date sits at the bottom of every page in every language, and the comparison takes ten seconds. The official crawling documentation changelog also publishes an RSS feed, which remains the most reliable way not to miss a silent rewrite like this one.

What this changes in your log analysis

None of the July 2026 additions can be verified in Search Console. The Crawl Stats report does break down by Googlebot type, but it aggregates, samples, caps at ninety days, and obviously shows no crawler outside Google. The new documentation only becomes actionable when checked against your own logs. Four verifications follow directly from the text.

Break crawl down by Google user agent, not just "Googlebot." Since capacity is shared, you need each crawler's share: Googlebot, Googlebot-Image, Googlebot-News, AdsBot-Google, Storebot-Google, Google-Extended. If one of them captures an unexpected fraction of requests, you have your explanation for a drop in crawling elsewhere.

Look for 429 responses served to verified Google IPs. Their mere presence means your infrastructure is throttling Google, and the documentation now states that this lowers your limit. It's the best effort-to-impact fix on this list.

Measure response time as served to crawlers, not as measured by your performance tools. Crawlers hit poorly cached URLs at scale: deep pagination, facets, archives. Their real TTFB there is often far worse than on the pages your visitors see. The $request_time field on Nginx or %D on Apache has to be present in your log format for this measurement to exist at all; without it, Google's unit of account stays invisible to you.

Track the share of 304 responses served to crawlers. A share of zero means your server exposes no usable ETag or Last-Modified header, and that every recrawl is billed at the price of a full response. This is the lever Google explicitly added in July, and it takes a single status-code aggregation per crawler to check.

→ The Crawl Lab log analyzer automatically segments crawling by bot and by status code: see each Google agent's share of your capacity, misfired 429 responses and your 304 rate in seconds. Everything runs in your browser, with no upload.

Read next

• Server Log Analysis for SEO: The Complete Guide (2026)
• 404 and 5xx Errors Seen by Crawlers: Spotting Them in SEO Logs
• SEO Log Analysis: Which KPIs to Track and How to Read Them