BestPromptFinder

Polite Web Scraper in Python

Quality, usefulness and confidence are AI evaluations out of 100 — not user ratings. Community results come from visitors' "worked / didn't work" votes below. How scoring works →

The prompt

Write a Python script to collect data from a website, the polite and legal way. Website and pages: [URL pattern, e.g. a public product listing or public notices page] Data I need: [fields, e.g. title, price, date] How many pages: [ ] Output: [CSV / JSON / Google Sheet] Requirements: 1. First check whether the site offers an API or a data download; if so, use that instead and tell me. 2. Check robots.txt and skip paths it disallows. Remind me to read the site's terms of use. 3. Use requests + BeautifulSoup (or Playwright only if the page needs JavaScript), with a clear User-Agent naming my project. 4. Wait 2-5 seconds between requests, retry gently on errors, and stop after [N] pages. 5. Don't log in, bypass paywalls or CAPTCHAs, or collect personal data. 6. Save results progressively so a crash doesn't lose everything. 7. Clear comments and a README section: install, run, change the target. Add a small test using a saved sample HTML file.

Why this prompt

• Hand-picked, editorially reviewed prompt
• Written for ChatGPT, Claude, Gemini

Source

Created by BestPromptFinder — editorial prompt, not user-generated. See our content & source policy.

Related Coding prompts