The blog

Robots.txt & crawler control

Crawler control: robots.txt, network-level blocking, and the trade-offs between model training and citation in AI answers.

Cloudflare Blocks AI Crawlers by Default on 15 September 2026: What Changes

On 15 September 2026, Cloudflare blocks Training and Agent by default on ad-monetised pages. Googlebot, Applebot and Bingbot can be caught by the same setting. How to check in your server logs.

Jul 29, 2026