Analyze server logs for crawl budget waste

Identify wasted Googlebot crawls on 404s, internal search pages, and redirect loops. Use it to protect crawl budget on large websites.

You getCrawl Distribution, wasted Hits and technical Fixes

Prompt

Prompt
Act as a technical SEO log analyst. Analyze the pasted web server log sample to identify Googlebot crawl budget waste.

- Calculate total requests per HTTP status code and summarize crawler focus.
- Identify repeated crawls on non-canonical URLs, internal search parameters, faceted navigation, or static assets.
- Flag high-latency URLs that slow down crawl rate.
- Provide concrete robots.txt or server-level fixes to block or redirect bad crawler traffic.

Output format:
1. Crawl Distribution: table with Status Code, Request Count, Percentage, Assessment.
2. Wasted Hits: bullet list of specific URL patterns draining bot activity.
3. Technical Fixes: code block with recommended robots.txt disallow rules or Nginx blocks.
4. Verification Steps: 3 steps to monitor improvements in Google Search Console.

Here is the server log excerpt:

Example output

Example for: 50 lines of e-commerce Nginx access logs containing Googlebot entries

  1. Crawl Distribution:
Status CodeRequest CountPercentageAssessment
2002856%Normal content crawl
3011122%Chain waste on legacy product IDs
404918%Crawling deleted catalog pages
50024%High priority server timeout
  1. Wasted Hits:
  • Faceted search loops: URLs containing ?filter_size= and &sort= crawled repeatedly.
  • Expired promotion pages: Googlebot requesting 2021 seasonal discount URLs returning 404.
  • Uncached asset crawls: high frequency of requests directly hitting /wp-json/ endpoints.
  1. Technical Fixes:
# Add to robots.txt
User-agent: Googlebot
Disallow: /*?*filter_
Disallow: /*?*sort=
Disallow: /wp-json/
  1. Verification Steps:
  2. Check Crawl Stats report in Google Search Console for drops in average response time.
  3. Monitor 'By response' chart to ensure 404 and 500 crawl percentages fall below 5%.
  4. Compare total daily requests in server logs one week after robots.txt update.
Written by AI from this prompt. Your results will vary.

Paste 20 to 100 rows of Apache, Nginx, or Cloudflare access logs right after the prompt in ChatGPT, Claude, or Gemini. In a follow-up, ask for tailored log-parsing bash commands for your specific log format.

Make it repeatable

Keep using this prompt without pasting it in every time.

Keep exploring

More in SEO
SEO

Build a site migration SEO QA checklist

Create a comprehensive quality assurance checklist to protect organic rankings during site migrations. Use this before staging sites push to production.