Crawlee
Crawlee is a library that allows you to manage web scraping and browser automation processes from end to end. It helps create scanners that exhibit human-like behavior against modern bot protections and stores data permanently in machine-readable formats. Applications are available for TypeScript and Python languages.
Crawlee
95apify · Web Scraping & Automation
Common commercial products in this category: Bright Data, Apify, ScrapingBee
- You can collect data by scanning links on websites.
- You can shoot content by interacting with JavaScript-density sites.
- With configurable settings, you can create scalable browsers that fit your project.
- • NeutralHacker News
Show HN: Crawlee – Web scraping and browser automation library for Node.js
“In that they're trying to be crawling frameworks, and for sure Scrapy allows headed crawling via Splash, it's just not something I've needed or advocate for Scrapy also has a long lineage of extensions, which maybe Crawlee will gain as it increases in popularity but I didn't see…”
- • NeutralHacker News
Show HN: Crawlee – Web scraping and browser automation library for Node.js
“Hi! Very cool project. Just out of curiosity, what trips up Crawlee on CreepJS? I haven't heard of anyone actually using it in production (actually don't think it's meant for production use). It's certainly overzealous in its aggregate "trust score", but (a) it seems like a good…”