Spider
Spider is a simultaneous priority web browsing engine developed with Rust. It processes incoming pages by flow method, JavaScript only runs when necessary and can scale from a single script to a distributed structure. It can be run locally or used with cloud infrastructure which includes features such as proxy and unblocking.
Spider
65spider-rs · Web Scraping & Automation
Common commercial products in this category: Bright Data, Apify, ScrapingBee
- You can simultaneously crawl web pages through a single script or distributed worker fleet.
- You can collect pages in flow by running JavaScript on pages that are needed.
- You can prototype in the local environment and switch to cloud infrastructure with a single configuration change.
- • No published release
- ↓ NegativeHacker News
Spider venom kills varroa mites without harming honeybees
“Spider venom saving bees was not on my bingo card, but this is genuinely cool. Obviously "works on mites in the lab" is a long way from "works safely and cheaply in a real hive", but varroa is such a nasty problem that even a new direction is worth paying attention to”
- • NeutralGitHub Issue
Also extract urls that are pointing to other domains? [CLI]
“I only get 'internal links'. Is there a way to get external links too?”