Crawl Lab
FR
Server logs

Jul 22, 2026

404 and 5xx errors seen by crawlers: spotting them in your SEO logs

By Louis Gerecht

Contents
In brief — 404 and 5xx errors served to robots are often only visible in the logs: a page can return 200 to a visitor and an error to Googlebot (cache/CDN, user-agent blocking, geolocation, JavaScript rendering). To spot them, isolate IP-verified robots, filter codes other than 200/304, then group by URL to tell a one-off incident from a chronic problem. What matters is not the raw error count but their nature and location: a single 5xx on a strategic page outweighs ten 404s on old URLs.

Some errors are visible nowhere. You open the page in your browser: it displays perfectly, code 200. But when Googlebot requests it, the server returns a 404 or a 500. This kind of silent gap escapes most tools, because they test the site like a visitor, not like a robot. Only the logs record what robots actually receive. This article explains how to flush out these errors and why they weigh heavily on SEO.

Understanding HTTP error codes

HTTP status codes are the language your server uses to answer robots.

  • 404 (Not Found): the resource doesn't exist. Occasional, it's normal; recurring on important URLs, it's an internal linking or sitemap problem.
  • 410 (Gone): the resource was deliberately removed. A stronger, cleaner signal than a 404 to tell Google to forget a URL.
  • 500 (Internal Server Error): the server crashed on this request. In the engine's eyes, a sign of fragility.
  • 503 (Service Unavailable): temporary unavailability. Acceptable once in a while (maintenance), worrying if it persists.

The SEO principle is simple: Googlebot reads repeated errors as a signal of degraded quality. A site that regularly serves 5xx sees its crawl frequency shrink, because the engine slows down to avoid worsening a server load it assumes is struggling.

The silent case: 200 for humans, an error for robots

The gap between a page that returns 200 to a human and an error to a robot is the hardest error to detect — and often the most costly. Several causes can produce it:

  • Cache and CDN: a cached version serves a 200 to humans while the origin returns an error to robots, or the other way around.
  • User-agent blocking: a misconfigured security rule refuses Googlebot thinking it's blocking a scraper, and answers 403 or 404.
  • Geolocation: the server routes differently by IP; Google's IPs, often American, land on a different version.
  • JavaScript rendering: the page builds client-side; without executing the script, the robot receives an empty shell or an error.

None of these silent error cases shows up when you test the site yourself. Only the log line, which records the code actually served to Googlebot's IP, reveals them. It's the clearest illustration of why logs matter: they see the site through the robot's eyes, not yours.

The method to spot them

1. Isolate verified robots

Start by keeping only genuine robots, authenticated by their IP — otherwise you'd be chasing errors served to fake Googlebots, which is pointless. This preliminary step is described in the article on identifying Googlebot and unmasking fake ones.

2. Filter on non-200 codes

Then extract every robot request whose code isn't a 200 (or a 304, the normal not-modified response). You get the raw list of friction points engines are hitting.

3. Group, date, prioritize

Aggregate by URL to surface recurring errors, date them to tell a one-off incident from a chronic problem, and prioritize by the importance of the affected pages. A 5xx on a pillar page crawled daily comes before a 404 on an old forgotten URL.

Fix according to the cause

Treatment depends on the diagnosis. A 404 on a URL that should exist calls for a 301 redirect to the relevant equivalent, or fixing the faulty internal link. A permanently removed URL is better served by a 410 than a 404, to speed up its exit from the index. A recurring 5xx belongs to infrastructure: application logs, server capacity, slow queries. As for 200/error gaps, they're solved by aligning configuration (cache, security, geolocation) on identical behavior for humans and robots.

Careful with redirects: fixing a 404 with a 301 is good, but chaining redirects creates a chain that consumes crawl budget. Every hop is one more request. This link between errors, redirects and budget is explored further in the crawl budget cluster.

Error tracking to build into the long run

Errors seen by robots are not a one-off audit but an indicator to monitor. A redesign, a migration, a CMS update or a new security rule can introduce them overnight. Comparing the response code distribution from one period to the next — as we recommend in the KPI article — catches these regressions before they eat into your rankings.

Do I have too many HTTP errors in my SEO logs?

No site shows zero errors, and that's not the goal. 404s always exist: stale external links, removed old URLs, robot probes. So the question isn't “are there errors?” but “what share, and on which pages?”. A simple benchmark: the vast majority of responses served to robots should be 200s (or 304s). As soon as error codes and redirects capture a noticeable fraction of the crawl, budget dissipates and a quality signal degrades.

More than raw volume, what matters is the nature and location of the errors. Ten 404s on unimportant old URLs are anecdotal; a single recurring 5xx on a pillar page crawled daily is a priority. The rule: weigh each error by the importance of the affected page and its crawl frequency. That cross-weighting turns a long error list into a fix queue sorted by impact.

A step-by-step diagnosis

Take a typical case. After a migration, organic traffic drops with no obvious explanation in Search Console. Log analysis then follows a clear thread. First step: the code distribution reveals a sharp rise in 301s. Second step: grouping by URL uncovers redirect chains — the old URL redirects to a second one, which redirects to the final page. Each page now costs two crawl requests instead of one. Third step: a few old URLs, forgotten in the redirect table, flat-out answer 404, severing the link between the migrated content and the authority it had built.

The remedy for these redirect chains follows from the diagnosis: flatten the chains so each old URL points to its final destination in a single redirect, and turn the faulty 404s back into proper 301s. Invisible in a classic audit, this problem jumps out in the logs, and fixing it often restores traffic within weeks.

410 or 404: choosing the right signal

When a page is gone for good, two answers are possible, and the choice has consequences. A 404 tells the engine “I can't find this page”; it will come back to test it several times before giving up, out of caution. A 410 says “this page was deliberately removed”; the signal is stronger and more final, and Google generally drops it from the index faster. For an assumed removal — a product pulled from the catalog, an obsolete article unpublished — the 410 is the clean choice; keep the 404 for involuntary or uncertain absences. This level of finesse, detailed in the article on HTTP status codes, avoids leaving dead URLs lingering in the index and diluting the site's perceived quality.

Prevention over cure: the risky moments

Errors seen by robots don't happen at random. They surge around specific events: site migration, CMS change, security or firewall rule update, information architecture overhaul, CDN deployment. At each of these moments, it's wise to schedule a log analysis in the following days rather than wait for a traffic drop to sound the alarm. Catching a response code regression early means fixing it before it costs rankings — the opposite of the usual reflex of diagnosing once the damage is done.

→ The Crawl Lab log analyzer automatically sorts response codes by robot and by date: spot at a glance the 404s and 5xx Googlebot encounters, including the ones invisible from your browser.

Also read