Check If You Can Scrape a Website and Plan a Polite Scraper First
You get a Go or Don't verdict based on robots.txt, terms and privacy risks, a list of API or export alternatives, and a step-by-step polite scraping plan.
Prompt
Act as a careful data engineer who helps non-developers collect web data responsibly.
Before writing any code, help me decide whether scraping a website is allowed and sensible, then plan a polite scraper. First ask me these questions in one message and wait for my answers:
1. Which website and which pages do you want data from?
2. What exact fields do you need, and how often?
3. What will you use the data for: personal, research or commercial?
4. Can you paste the site's robots.txt and any terms of service text about bots, scraping or automated access?
Rules:
- Check robots.txt against the exact paths I need and quote the relevant lines. If I did not paste it, tell me where to find it.
- Look for an official API, data export, RSS feed or download first, and recommend it over scraping when it covers my fields.
- Flag any personal data (names, emails, photos, profiles) and explain the privacy risk in plain words.
- Set a specific delay between requests, a user agent with contact details, and a stop condition.
- Give one verdict: Go, Go with limits, or Don't scrape.
Output format:
1. Verdict (one line with the main reason)
2. Permissions table: Source | What it says | Impact on my plan
3. Alternatives to scraping
4. Personal-data risks
5. Polite scraper plan as numbered steps, no code yet
6. Open questions
This is not legal advice; for commercial use, personal data or pages behind a login, check with a lawyer. End by listing your assumptions and anything you could not verify.Example output
Tips and common edits
Why this works
- Asking for robots.txt and terms text up front means the verdict quotes real rules instead of guessing about the site.
- Ranking APIs, exports and feeds above scraping often removes the need for a scraper at all.
- Fixed delay, user agent and stop conditions turn "be polite" into checkable settings you can copy into code.
Use before building any scraper with ChatGPT, Claude or Gemini. Have the site URL, the fields you need, and copies of its robots.txt and terms of service section on automated access ready to paste.
Common edits
- Add "then write the scraper in Python with requests and BeautifulSoup" once the verdict is Go.
- Add your country or region so the privacy section mentions rules like GDPR or CCPA.
Prompt details
- By
- Prompts-GPT editors
- Published
Did this prompt work?
Keep exploring
More like this
Build an Excel Formula from a Plain-English Description of What You Need
You get a working Excel formula, versions for older Excel and Google Sheets, a one-row worked example, and a table of the blank and error cases it handles.
Write a Google Apps Script to Automate a Repetitive Google Sheets Task
You get a commented Google Apps Script built around your sheet names, columns and timing, plus install steps, a test plan on a copy, and undo steps.
Batch Rename Files in a Folder with a Script You Can Preview First
You get a PowerShell or Bash rename script that shows every old-to-new name before changing anything, writes an undo log, and includes plain run steps.
Turn Your Inbox Problems into Gmail or Outlook Rules You Can Set Up Today
Get a numbered set of Gmail filters or Outlook rules with exact conditions, actions, run order, conflict warnings, setup clicks and a test for each rule.
Write a Safe Automation Script with Dry Run and Error Handling
Get a commented shell or Python script for your repetitive task, with a dry-run flag, clear error handling and a test command to try on sample files first.
Combine and Clean Multiple CSV or Excel Files into One Table with Power Query
You get a header mapping table, numbered Power Query clicks, commented M code that fixes mismatched columns and duplicates, and refresh steps for new files.