Check If You Can Scrape a Website and Plan a Polite Scraper First

You get a Go or Don't verdict based on robots.txt, terms and privacy risks, a list of API or export alternatives, and a step-by-step polite scraping plan.

Prompt

Prompt
Act as a careful data engineer who helps non-developers collect web data responsibly.

Before writing any code, help me decide whether scraping a website is allowed and sensible, then plan a polite scraper. First ask me these questions in one message and wait for my answers:
1. Which website and which pages do you want data from?
2. What exact fields do you need, and how often?
3. What will you use the data for: personal, research or commercial?
4. Can you paste the site's robots.txt and any terms of service text about bots, scraping or automated access?

Rules:
- Check robots.txt against the exact paths I need and quote the relevant lines. If I did not paste it, tell me where to find it.
- Look for an official API, data export, RSS feed or download first, and recommend it over scraping when it covers my fields.
- Flag any personal data (names, emails, photos, profiles) and explain the privacy risk in plain words.
- Set a specific delay between requests, a user agent with contact details, and a stop condition.
- Give one verdict: Go, Go with limits, or Don't scrape.

Output format:
1. Verdict (one line with the main reason)
2. Permissions table: Source | What it says | Impact on my plan
3. Alternatives to scraping
4. Personal-data risks
5. Polite scraper plan as numbered steps, no code yet
6. Open questions

This is not legal advice; for commercial use, personal data or pages behind a login, check with a lawyer. End by listing your assumptions and anything you could not verify.

Example output

Example for: a bakery owner who wants weekly flour prices from the fictional supplier site millstonesupply.example, for internal cost tracking.

1. Verdict

Go with limits. Product pages are allowed in robots.txt and contain no personal data, but the terms ask for written permission for automated access, so email the supplier first.

2. Permissions table

SourceWhat it saysImpact on my plan
robots.txtUser-agent: * / Disallow: /account/ / Disallow: /cart/ / Crawl-delay: 10/products/flour/ is allowed; wait at least 10 seconds between requests
robots.txtSitemap: /sitemap-products.xmlUse the sitemap to list pages instead of crawling links
Terms, section 7.2"Automated access requires prior written consent."Ask permission before running anything
Terms, section 9Content may not be republishedFine for internal use only; do not publish the prices

3. Alternatives to scraping

  1. Wholesale price list PDF: the account area offers a monthly download. Covers price and pack size, so it may replace scraping entirely.
  2. Ask your sales rep for a weekly CSV by email.
  3. No public API was mentioned on the site; ask whether a trade API exists.

4. Personal-data risks

  • Product pages show only product names, prices and pack sizes: low risk.
  • Customer reviews include first names and towns. Do not collect the review section.

5. Polite scraper plan

  1. Email the supplier describing the pages, frequency (once a week) and purpose. Keep their reply.
  2. Read the 14 flour URLs from the product sitemap.
  3. Set the user agent to BakeryPriceCheck/1.0 (owner@example-bakery.test).
  4. Request one page every 15 seconds, Mondays at 6:00 a.m. (about 4 minutes total).
  5. Save only name, pack size, price and date to a spreadsheet.
  6. Stop on any 403 or 429 response, or after 3 errors in a row, and do not retry until next week.
  7. Recheck robots.txt and the terms every 3 months.

6. Open questions

  • Do trade prices only appear when logged in? If so, do not scrape; use the PDF.
  • Will the supplier allow weekly access in writing?

Assumptions: the pasted robots.txt and terms are current; prices on public pages match what you pay. I could not verify whether the site blocks bots technically.

Written by AI from this prompt. Your results will vary.

Tips and common edits

Why this works

  • Asking for robots.txt and terms text up front means the verdict quotes real rules instead of guessing about the site.
  • Ranking APIs, exports and feeds above scraping often removes the need for a scraper at all.
  • Fixed delay, user agent and stop conditions turn "be polite" into checkable settings you can copy into code.

Use before building any scraper with ChatGPT, Claude or Gemini. Have the site URL, the fields you need, and copies of its robots.txt and terms of service section on automated access ready to paste.

Common edits

  • Add "then write the scraper in Python with requests and BeautifulSoup" once the verdict is Go.
  • Add your country or region so the privacy section mentions rules like GDPR or CCPA.

Prompt details

By
Prompts-GPT editors
Published

Did this prompt work?

Keep exploring

More in Coding