Crawl budget on a brand-new site: 1,722 crawler requests, analysed

In short — Between 20 and 27 July 2026 we logged every crawler request hitting crawl-lab.com, a site that had gone live thirteen days earlier: 1,916 raw hits, 1,722 after removing a vulnerability scanner and our own tooling traffic. Three results stand out. Google accounts for 63.4% of all crawling, but Googlebot is only 23.1% of it: three quarters comes from GoogleOther, and 86.2% of GoogleOther's requests are not pages at all but framework payloads. The site served zero 304 and zero 429 responses across 1,722 replies, which makes Google's July 2026 HTTP caching recommendation inapplicable as-is on a Next.js stack hosted on Cloudflare. And during the three days we served 500 errors, GoogleOther's crawl volume dropped 81% while Googlebot's fell only 16%: shared capacity is not shared equally.

On 22 July 2026, Google rewrote its crawl budget documentation and put three previously unpublished mechanisms in writing. We covered them in what Google's rewritten documentation actually changes. That left the question a practitioner actually cares about: do those statements hold up against real logs?

This article answers with our own data. We captured every crawler request our site received during its first two weeks, then replayed those logs through our own analyzer. No figure quoted here comes from a third-party tool, a panel or an estimate: these are 1,722 log lines we own, on a site whose every deployment we control — including the two outages that turned into a natural experiment.

The setup: every crawler hit captured at the source, with no sampling

The site runs on a Cloudflare Worker. The free plan gives no access to raw access logs, so we produce them ourselves: a custom entry point wraps the request handler, detects crawler user agents and writes one Combined Log line per hit into object storage. Human traffic is never logged, which settles the personal-data question at the source.

The observation window runs from 20 July 2026 at 17:23 UTC — when the bucket was created — to 27 July 2026, eight calendar days. The domain went live on 14 July 2026, so on the last day of measurement the site is thirteen days old. It has roughly sixty indexable URLs, two language versions, five blog posts and twenty-four crawler reference pages. This is a new site in the strict sense: no acquired backlinks, no crawl history, no brand awareness.

This method has a decisive advantage over Search Console: no sampling, no aggregation, no ninety-day window, and it sees non-Google crawlers. It also has a limitation we own up to below: it records the response once produced, so it cannot record its own failures.

What we removed from the dataset, and why

Of 1,916 raw hits, 194 do not belong to the object of study and were excluded before any calculation. Publishing a case study without that step would mean publishing numbers that are roughly 10% wrong.

132 hits came from a vulnerability scanner, sent from a single IP address in twenty-three seconds under thirteen different crawler identities — Googlebot, Bingbot, GPTBot, ChatGPT-User, OAI-SearchBot, PerplexityBot, ClaudeBot, CCBot, YandexBot, Baiduspider, DeepSeekBot, xAI-SearchBot and Google-CloudVertexBot. Its targets leave no doubt: /.env, /.git/config, /terraform.tfstate, /firebase-adminsdk.json, a POST to /graphql. On its own it accounts for 6.9% of the raw log and 89% of every spoofed hit in the period.

38 hits came from our own workstation: a Screaming Frog verification crawl, a few fetches from a development agent, and social sharing previews. A further 24 hits are Claude-User fetches from Anthropic's servers: as our Claude-User reference page explains, those are not crawler visits but retrievals triggered by a human asking an assistant a question — and it is reasonable to assume that human was us, working on the site. We removed all of them.

That cleanup already yields a lesson found in no documentation: on a low-traffic site, your own tooling plus a single scanner can account for a tenth of your apparent crawl volume. Any conclusion drawn before cleaning is a conclusion about your own noise.

Google's claims, tested against our logs

The table below summarises our verdicts. Each row is detailed in the matching section.

Confirmed: capacity is shared, and Googlebot consumes only a quarter of it

Across the 1,722 hits we kept, Google accounts for 1,091, or 63.4% of all crawling. But the internal split is the opposite of what habit suggests: GoogleOther accounts for 835 requests, or 76.5% of Google's crawl, against 252 for Googlebot, or 23.1%, and 4 for Googlebot-Image. Three out of four Google requests are not Googlebot requests.

The clearest evidence of shared capacity is not in the volumes but in the addresses. Those 1,091 Google hits come from just twelve IP addresses, all inside the operator's published ranges, and the same addresses serve Googlebot and GoogleOther interchangeably. A single one of them issued 284 GoogleOther requests, 77 Googlebot requests and one Googlebot-Image request. Two addresses concentrate 66% of Google's entire crawl. The hostload described in the documentation is therefore not an accounting abstraction: it is a shared queue, materialised by a shared pool of machines.

The operational consequence is immediate. A crawl report that only counts lines containing "Googlebot" understates the real load Google puts on this site by a factor of four. Any crawl budget measurement must segment by full user agent, not by brand.

New finding: 86% of GoogleOther's requests are framework payloads, not pages

This is the most surprising result of the study, and it appears in no documentation. Of GoogleOther's 835 requests, 720 carry a `?_rsc=` parameter, or 86.2%. That parameter is not ours: Next.js generates it to fetch the React Server Components payload for a route. A crawler that executes the page's JavaScript triggers prefetching of the visible links, and each prefetch becomes a separate HTTP request in our logs.

Across the whole site, 42.9% of all crawler traffic received in eight days consists of these framework-internal requests. They point to no new URL: they are the same pages, requested a second time in a different format. Google's documentation does state that embedded resources — CSS, JavaScript, XHR calls — consume crawl budget; our logs show that in a modern server-rendered application this line item can outweigh the pages themselves.

This is not waste in the sense of a misconfiguration: these requests are the framework working as designed and must not be blocked. But in a unit of account expressed as server connection time, it is budget spent, and it is invisible to anyone who only looks at canonical URLs. It is also a trap for log analysis tools: we had to fix our own, which initially classified those requests as "parameterised URLs to fix" and made a healthy site look like a massive budget waster.

Confirmed, with a priority order the documentation does not describe

From 23 July at 19:57 UTC to 26 July at 20:43 UTC, our site served 69 500 errors to crawlers, caused by two distinct infrastructure defects unrelated to content. We did not cause them on purpose, but they are the natural experiment Google's documentation does not let you run deliberately: what happens when a young site serves server errors to Google for three days?

Reading that table honestly requires one caveat: the 23 July peak is explained by publishing twenty-four pages that day, so by a rise in crawl demand, not capacity alone. The meaningful comparison is 24 to 25 July, at constant content. GoogleOther then falls from 206 to 39 requests, a drop of 81%, while Googlebot goes from 51 to 43, or 16%. The day after the fix, GoogleOther climbs back to 86 while Googlebot keeps sliding to 29.

What we take from it, presented for what it is — an observation on a single site, not proof of causality: Googlebot maintains a remarkably stable floor, between 29 and 51 requests a day across the whole period regardless of the error rate served, while GoogleOther varies by a factor of twenty-one. If capacity really is shared, it is not shared symmetrically: the variable share is absorbed by the secondary crawler, and search indexing is protected last. The documentation says crawlers share one envelope; our logs suggest there is an order of service inside that envelope.

One detail makes the diagnosis harder and is worth knowing: our 500 errors happened at cache expiry, not continuously. Twelve URLs therefore served both 200 and 500 responses within the same window, which is the hardest case to spot — the site looks perfectly up to any manual check. Only crossing URL against response code in the logs reveals it, as we detail in our guide to 404 and 5xx errors seen by crawlers.

Refuted in practice: zero 304 responses out of 1,722, and it is not negligence

The most concrete novelty in the July 2026 documentation is the recommendation to support HTTP caching and the 304 (Not Modified) status, so that a re-crawl does not cost a full response. Across our 1,722 responses, the number of `304`s served is exactly zero.

We checked why by querying the live site rather than assuming. No HTML page on the site emits an ETag or Last-Modified header: not the on-demand rendered pages, not the fully static ones, which carry only a Cache-Control with s-maxage. A request carrying If-Modified-Since gets a full 200. The only file on the site that exposes an ETag is robots.txt, and that ETag is returned unquoted — a form that does not conform to RFC 9110 — so a client echoing it back in If-None-Match still receives a 200.

The conclusion is not that we misconfigured the site, but that Google's recommendation is currently inapplicable by default to an entire class of modern stacks. A Next.js site deployed at the edge provides no cache validator for its HTML, and nothing in the standard configuration lets you add one without writing code. For the millions of sites built on that kind of foundation, the July 2026 recommendation will remain a dead letter until frameworks implement it upstream. An Apache or Nginx server serving static files, meanwhile, has honoured it forever without knowing.

The audit corollary is simple: a zero share of 304 in your logs is not necessarily a defect to fix, it is first of all information about your hosting. Check whether your server can even emit a validator before concluding negligence.

Qualified: a deliberately low default limit does not show up at discovery

The documentation states that every site starts at the same default capacity limit, described as conservative, then adjusted over time. Our site is the textbook case: thirteen days old, no backlinks, no history. We expected frugal crawling.

The sharpest observation says otherwise. We published an article on 26 July 2026 at 21:49:36 UTC — the creation timestamp in our CMS. Googlebot's first hit on that URL is stamped the same day at 21:55:38, from a verified Google IP address: six minutes and two seconds after publication. The English version was reached two minutes later. No manual Search Console submission was made in that interval.

There is no formal contradiction with the documentation: capacity caps throughput, it does not govern discovery latency, and Google was crawling the site heavily at that precise moment. But for a practitioner the nuance matters: on a technically healthy new site properly wired through its sitemaps, discovery speed is not the limiting factor. The capacity ceiling, if it exists, never showed up here as slowness to discover new content. It showed up as total volume, and above all as responsiveness to errors.

Another observation fits that reading: on the day we published twenty-four pages at once, Google went from 118 to 363 daily requests, a threefold increase in twenty-four hours. Crawl demand follows publication very closely, with no apparent observation period.

Confirmed: 4xx costs almost nothing, but it wrecks the diagnosis

Google's myths page states that 4xx responses, other than 429, do not waste crawl budget. Our figures agree: after cleaning, the site served only 19 `404` responses, or 1.1% of the crawl, across 13 distinct URLs. Eight of them are stale JavaScript files requested by crawlers still holding HTML from an earlier deployment. Seven correspond to two old blog URLs left without a redirect. The rest is noise.

The real lesson is elsewhere. Before cleaning, the same site showed 101 `404` responses across 80 URLs, the vast majority pointing at `/.env`, `/.git/config` or `/firebase-adminsdk.json` — and those requests carried the user agents of Googlebot, Bingbot, GPTBot and PerplexityBot. A hurried analyst would have concluded that AI crawlers waste crawl budget on non-existent technical paths. That conclusion would have been entirely wrong: those 132 requests came from a single IP address in twenty-three seconds.

That is why IP-range authenticity verification is not a refinement but a prerequisite. In our data, 1,296 hits were verifiable; only 7 were identified as spoofed after cleaning, against 64 before — meaning 89% of all spoofing came from that one scanner. An audit that skips this step is not measuring Google's behaviour, it is measuring its impersonators.

Nearly a quarter of the budget goes to robots.txt and sitemaps

On a seventy-page site, the surprise is in the distribution. 417 requests out of 1,722, or 24.2% of all crawler traffic, target `robots.txt` or the sitemaps — files that contain no editorial content at all. It is the site's second largest line item after the homepage.

The per-crawler detail is even more telling. Googlebot spent 122 of its 252 requests on `robots.txt`, or 48.4% of its activity on our site; GoogleOther never requested that file once — further confirmation that the two crawlers share infrastructure, one fetching the directives the other benefits from. On the AI side, ClaudeBot spent 36% of its requests on robots.txt and 38% on sitemaps, nearly three quarters of its activity on crawl infrastructure rather than pages. Bingbot spent 84% of its time in the sitemaps.

This is a small-site characteristic: the fixed cost of checking directives does not shrink with page count, so it becomes proportionally enormous. It cannot be fixed — a crawler must re-read robots.txt — but it must be excluded from any waste calculation, or a perfectly healthy site will appear to squander three quarters of its budget. We work through the reasoning in our guide to SEO log analysis KPIs.

What our logs cannot measure

The unit of account Google described in July 2026 is connection time: the total time your server holds connections open for its crawlers. Our logs do not contain that information. The format we generate carries neither processing duration nor response size — the byte field is - on all 1,722 lines, because responses are streamed and have no known length when the line is written. We can therefore neither measure our consumption in Google's unit nor compare it page by page.

There is worse, and it is intrinsic to any logging done by the application itself: our logs cannot record their own outages. Before 23 July, our Worker regularly exceeded its CPU quota when a crawler chained pages, producing infrastructure errors served before our code ever ran. Those errors appear nowhere in the 1,722 lines analysed: the site was falling over under crawl load without leaving a trace in its own journal.

We flag it because it is exactly the scenario Google's documentation describes — latency and errors lower your capacity — and we are unable to quantify it with our own data. For anyone running Apache or Nginx, the answer is one configuration field: %D or $request_time in the log format, and Google's unit of account becomes measurable. It is the first setting we recommend before any serious analysis, and it is covered in our guide to getting your server logs.

The two-week verdict, in ten numbers

If we had to reduce these two weeks to three lessons for a site that is just starting, here they are. First, your crawl budget is not spent where you think it is: on our site, 24.2% went to directive files and 42.9% to framework-generated requests, before a single article page was served. Second, server health produces a measurable effect within forty-eight hours: three days of 500 errors cost 81% of GoogleOther's crawl volume, with recovery beginning the day after the fix. Third, none of these numbers were visible in Search Console: not GoogleOther's share, not the framework requests, not the spoofed identities, not the complete absence of 304.

We will republish this study at three months, with the same protocol on the same property, to see whether the capacity granted evolves once a crawl history exists. That is the only honest way to test the most intriguing claim in the July 2026 documentation: a progressive adjustment over time, for which Google gives neither a delay nor a magnitude.

→ Replay your own logs in the Crawl Lab analyzer: segmentation by crawler and response code, IP-range authenticity checks, framework request detection. Everything runs in your browser — no file is ever uploaded to a server.

Read next

• Crawl Budget: What Google's Rewritten Documentation Actually Changes
• Server log analysis for SEO: the complete guide (2026)
• 404 and 5xx errors seen by crawlers: spotting them in your SEO logs
• SEO log analysis: which KPIs to track and how to interpret them