Server log analysis, working for your SEO and AI search.
Crawl Lab is a free SEO toolbox that analyzes server logs directly in the browser, with no upload and no sign-up. Its log analyzer reads Apache and Nginx access.log files and shows which pages Googlebot and AI crawlers — GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot — actually crawl: crawl frequency, errors served to bots, and wasted crawl budget. Because processing happens locally, the IP addresses contained in the logs never leave the user's machine.
Nothing is uploaded
Parsing runs in your browser. Your logs — and the IP addresses they contain — never leave your machine. Privacy-safe by construction, with no account and no third-party server.
Built for AI crawlers
GPTBot, ClaudeBot, PerplexityBot, Google-Extended… A maintained signature base separates AI crawl from classic search engine traffic like Googlebot.
Free, even at 1 GB
No server-side size limit, no compute bill. A full month of logs streams through without slowing your browser down.
Log analysis, concretely
Your server logs are the only source of truth about crawling
Every time a search engine robot visits a page, your server writes it to a log file (access.log). It is the most reliable record there is: no estimation, no sampling, just what actually happened. A server log analyzer turns those millions of raw lines into a readable SEO diagnosis.
With AI search engines, the stakes double. Beyond Googlebot and Bingbot, a new generation of crawlers (GPTBot for ChatGPT, ClaudeBot for Anthropic, PerplexityBot, Google-Extended) roams the web to train models or cite sources. Knowing which ones visit your site, how often and on which pages has become a real SEO topic — and a GEO one (Generative Engine Optimization).
What Google actually crawls
Classic reports estimate crawl activity. Your server logs prove it, line by line: which URLs Googlebot visits, how often, and which ones it ignores entirely.
A better-spent crawl budget
If bots spend their time on URL parameters, useless files or error pages, they crawl your strategic pages less. Log analysis exposes that waste.
Errors only robots see
A page that returns 404 or 500 to Googlebot but 200 to your visitors goes unnoticed everywhere — except in the logs. We surface them, dated and grouped by bot.
The tools
A growing SEO toolbox
Free tools — no account, respectful of your data — to steer your SEO in the AI search era. We start with logs; more are coming.
llms.txt generator
Generate an optimized llms.txt file to guide AI crawlers to your most important content.
AI citation audit
Check how your brand is cited in ChatGPT, Perplexity and Google AI Overviews answers.
Ready to see your crawl in plain sight?
Drop a log file and get a diagnosis in seconds. Nothing is sent, nothing is stored.