Crawl Lab
FR
Server logs

Jul 22, 2026

How to get your server logs for SEO (Apache, Nginx, cPanel, CDN)

By Louis Gerecht

Contents
In brief — how you retrieve your server logs depends on your hosting: Raw Access Logs on shared hosting (cPanel/Plesk), direct SSH/FTP access on a VPS or dedicated server, an export to enable in the cloud (AWS S3, Google Cloud), and caution with CDNs like Cloudflare, where part of the robot traffic never reaches the origin server. Aim for at least 30 days of logs and check the format (Combined Log Format) before analyzing. Since logs contain IP addresses — personal data under the GDPR and Quebec's Law 25 — favor local processing over sending them to a third party.

Before you can analyze your server logs for SEO insights, you first have to get your hands on them. It's a step many SEO guides gloss over, as if logs fell from the sky in a tidy CSV file. In reality, how you retrieve your logs changes completely depending on your hosting type — and that's often where log analysis projects stall before they even start. This guide covers all the common situations: shared hosting, VPS, cloud, CDN, and the traps to avoid along the way.

Why getting your server logs matters for SEO

A server log file is the raw trace of every HTTP request your server receives: humans, robots, APIs — everything goes through it. That exhaustiveness is what makes it such a precious source for SEO, far superior to Search Console's estimates and sampling. But a badly retrieved log file — truncated, incomplete over the period, or stripped of the user-agent field — distorts the entire analysis that follows. Better to start on solid ground.

Three questions to ask before diving in:

  • Where are my logs physically stored? On the web server, on a CDN, or scattered across several load balancers?
  • What period should I retrieve? A minimum of 30 days is recommended to capture normal crawl cycles.
  • Am I allowed to access them as-is? Logs contain IP addresses, hence personal data under the GDPR / Law 25 — a point we come back to below.

Shared hosting: cPanel, Plesk and others

It's the most common case for small sites, and paradoxically often the worst-equipped for logs. On shared hosting (OVH, o2switch, Hostinger, PlanetHoster...), raw logs are generally not enabled by default, for disk-space reasons on machines shared by hundreds of customers.

In cPanel, the usual path is Metrics > Raw Access Logs. Two options: download the current day's archive, or enable “Archive logs” to keep a rolling history rather than a single file overwritten every night. Without that option, you often only get the last 24 hours — far too little for a reliable analysis.

In Plesk, the equivalent lives under Websites & Domains > Logs, with a rotation system configurable right in the interface (number of files kept, rotation frequency).

With some low-cost shared hosts, raw logs simply aren't available to the user — only aggregated statistics like AWStats or Webalizer, unusable for real SEO analysis because they don't go down to the individual log line. In that case, the only recourse is contacting support, or considering a move to a VPS.

VPS and dedicated servers: direct SSH/FTP access

On a VPS or dedicated server, you have full access to the file system, hence to raw logs. It's the most comfortable setup for SEO analysis.

The standard locations:

  • Apache: /var/log/apache2/access.log (Debian/Ubuntu) or /var/log/httpd/access_log (CentOS/RHEL)
  • Nginx: /var/log/nginx/access.log

To retrieve them, two main methods:

  1. SFTP/FTP with a client like FileZilla or Cyberduck, connecting straight to the log directory.
  2. SSH with scp or rsync, better suited if you want to automate regular retrieval:

rsync -avz user@myserver.com:/var/log/nginx/access.log* ./local-logs/

Mind the rotation system: most servers use logrotate, which archives and compresses older logs as access.log.1, access.log.2.gz, and so on. If you only grab the current access.log, you lose all the history that has already rotated. Include the .gz archives in your retrieval — they decompress easily with gunzip.

Cloud environments and managed hosting

On AWS, Google Cloud, Azure or managed platforms like WP Engine, Kinsta or Vercel, the logic changes again: logs are often not simple files on disk but streams to enable and export through dedicated services.

  • AWS: ELB/ALB logs are enabled via the Access Logs option in the console, written straight to an S3 bucket. For CloudFront (often used up front), same idea: standard or real-time logs to S3 or Kinesis.
  • Google Cloud: Cloud Logging centralizes logs, exportable to BigQuery or Cloud Storage for bulk processing.
  • Managed WordPress hosting (WP Engine, Kinsta): raw logs are rarely accessible directly; you generally go through their log dashboard, or ask support for an extraction over a given period.

These cloud and managed environments therefore require an activation step upstream — logs aren't always retained by default if the export was never configured.

The CDN case: your logs may not be where you think

It's the most common trap in 2026: if your site sits behind a CDN like Cloudflare, Fastly or Akamai, a good share of the traffic (notably robots) first hits the CDN's servers, not your origin. Analyzing only your origin server's logs then means ignoring part of the crawl.

On Cloudflare, you need to enable Logpush (paid plans) to continuously export Edge logs to external storage (S3, GCS...), or use Logs Explorer for one-off consultation. On the free plan, raw log access is very limited — something to check before launching an ambitious analysis.

Ideally, you combine both sources: the CDN's logs for the full picture of edge crawl, and the origin server's logs for whatever wasn't cached. We dig deeper into this in the article on identifying Googlebot in your logs, where the CDN-vs-origin distinction becomes crucial to avoid false positives.

Format, volume and retention: what to check before analyzing

Once the files are in hand, a few checks are in order before the analysis itself:

  • The log format: Apache and Nginx default to the Combined Log Format, but it may have been customized. Check that the essential fields are present: IP, timestamp, method, URL, status code, user-agent, referer.
  • The period covered: ideally 30 full days to smooth out crawl variations. A one-week export can paint a misleading picture, especially for sites with irregular crawl.
  • The volume: a site with millions of pages can generate files of several gigabytes. That's exactly why processing locally in the browser becomes relevant — no need to upload that kind of volume to a third-party service, a point detailed in the article on log analysis without upload.

Once your logs are retrieved, there's no need to upload them to a third-party server to use them. The Crawl Lab log analyzer processes your .log or .gz files directly in the browser, even at several gigabytes: your IPs and crawl data never leave your machine. It's the fastest entry point to turn those raw files into actionable KPIs, with no setup and no command line.

GDPR: logs contain personal data

A raw log file contains IP addresses, which are legally personal data in Europe. Two good practices to know:

  • Anonymize IPs at extraction time if you need to share the files or store them long-term (e.g. by truncating the last octet in IPv4).
  • Limit the retention period of raw, non-anonymized logs, consistent with your data retention policy.

That's also why tools that process logs directly in the browser, with no upload to a third-party server, offer a concrete advantage: IPs never leave your machine.

In short

Getting your server logs for SEO is anything but universal: shared, VPS, cloud or CDN each impose their own method, with their own traps — rotation not included, forgotten CDN logs, retention too short. Once this step is mastered, head to the guide on reading an access.log file to understand what each line says, then to the KPIs to track to turn that raw data into an SEO action plan.

Also read