Find accidental noindex tags and crawl blocks

Step-by-step audit plan to catch staging noindex tags, disallow directives, and canonical leaks before production launch.

You getPre-Deploy Checklist, terminal Commands and CI/CD Gate Rule

Prompt

Prompt
Act as a technical webmaster. Provide a practical checklist to prevent staging search-engine blocks and noindex tags from leaking into production.

- Cover robots.txt directives, HTTP response headers (X-Robots-Tag), HTML meta tags, and canonical tags.
- Include shell curl commands to verify live headers and status codes quickly.
- Detail CI/CD deployment checks to automate detection before code merges.
- State how to verify the site immediately after DNS switch without waiting for search console alerts.

Output format:
1. Pre-Deploy Checklist: numbered verification steps.
2. Terminal Commands: code block with curl and grep snippets to test headers.
3. CI/CD Gate Rule: short config script or pseudo-rule for deployment pipelines.
4. Post-Launch Verification: 4 fast checks to run within 10 minutes of release.

Example output

  1. Pre-Deploy Checklist:
  2. Remove 'Disallow: /' from production-targeted robots.txt files.
  3. Scan compiled templates for <meta name="robots" content="noindex, nofollow">.
  4. Check server configs (Nginx/Apache/Cloudflare) for hardcoded X-Robots-Tag headers.
  5. Ensure canonical tags resolve to the production domain rather than staging subdomains.
  6. Terminal Commands:
# Check response headers for noindex or staging headers
curl -s -I https://example.com | grep -iE 'x-robots-tag|noindex'

# Check robots.txt status
curl -s https://example.com/robots.txt | grep -i 'disallow'

# Check HTML source for meta robots
curl -s https://example.com | grep -i '<meta.*robots'
  1. CI/CD Gate Rule:
# Fails pipeline if noindex is found in built static assets
if grep -rqi 'content="noindex"' ./dist/; then
  echo "Error: Staging noindex found in production build."; exit 1;
fi
  1. Post-Launch Verification:
  2. Inspect live home page source for correct canonical URL.
  3. Fetch robots.txt directly in an incognito window.
  4. Request indexation for one page in Search Console to verify fetch availability.
  5. Check server access logs for Googlebot 200 responses.
Written by AI from this prompt. Your results will vary.

Run this prompt in ChatGPT, Claude, or Gemini before any major code push or domain switch. Follow up by sharing your deployment platform (e.g. GitHub Actions, Vercel) to get exact automated test scripts.

Make it repeatable

Keep using this prompt without pasting it in every time.

Keep exploring

More in SEO