Scrapy
Scrapy is a web scraping framework used to extract structured data from websites. It runs independently of the platform and requires version Python 3.11 or above. The project is maintained by Zyte (formerly Scrapinghub) and numerous contributors.
Scrapy
100scrapy · Web Scraping & Automation
Common commercial products in this category: Bright Data, Apify, ScrapingBee
- You can automatically pull structured data from websites.
- You can manage data debugging processes in Python-based projects.
- With cross-platform support, you can do scrapings on different operating systems.
- ↓ NegativeHacker News
Scrapy Tips from the Pros
“Much like apples to oranges. Nokogiri is a tag soup parser. Scrapy is a web scraping framework. In addition to tag soup parsing, Scrapy handles a slew of things such as text encoding problems, retrying urls that fail owing to network problems (if you wish), dispatching requests…”
- • NeutralHacker News
Scrapy Tips from the Pros
“> would you be better off using a search engine crawler like Nutch for this? Fwiw we recently met a client who tried Scrapy + Frontera vs Nutch, and their assessment was that Scrapy + Frontera is twice as fast. Here's a deck on Frontera FYI: Aside: don't hesitate to get in touch…”
Verified against README · Oct 26, 2026 · source