The blog
SEO in the AI search era
GEO, AI crawlers and visibility in generative answers.
How to Detect AI Crawlers in Your Server Logs: The Complete Method
GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot: invisible in Google Analytics, visible in your logs. How to identify them and verify authenticity by IP range.
Cloudflare Blocks AI Crawlers by Default on 15 September 2026: What Changes
On 15 September 2026, Cloudflare blocks Training and Agent by default on ad-monetised pages. Googlebot, Applebot and Bingbot can be caught by the same setting. How to check in your server logs.
Crawl budget on a brand-new site: 1,722 crawler requests, analysed
We logged every crawler hit on our site during its first two weeks. 1,722 requests later: what our logs confirm, qualify and refute in Google's July 2026 crawl budget documentation.
Crawl Budget: What Google's Rewritten Documentation Actually Changes
Google rewrote its crawl budget documentation on July 22, 2026. Shared capacity across all crawlers, an identical starting limit for every site, HTTP caching: what changed and what to check in your logs.
404 and 5xx errors seen by crawlers: spotting them in your SEO logs
A page that returns 200 to humans but 404 or 500 to Googlebot goes unnoticed everywhere — except in the logs. Here's how to spot and fix these errors.
How to get your server logs for SEO (Apache, Nginx, cPanel, CDN)
Shared hosting, VPS, cloud or CDN: every method to retrieve your raw server logs and start your SEO analysis without uploading anything.
SEO log analysis: which KPIs to track and how to interpret them
Crawl volume, ignored pages, wasted budget, response codes, frequency by page type: the essential KPIs of log analysis and how to interpret them.
Server log analysis for SEO: the complete guide (2026)
Your server logs are the only proof of how Googlebot and AI crawlers actually crawl your site. Here's how to analyze them, step by step, to steer your SEO.