Scrapyd
Scrapyd is a service daemon used to disperse and run Scrapy spiders. It allows to load Scrapy projects and control spiders via a JSON API.
Scrapyd
100scrapy · Web Scraping & Automation
Common commercial products in this category: Bright Data, Apify, ScrapingBee
- You run spiders by uploading Scrapy projects to the server
- You control Scrapy spiders using JSON API
- You manage Scrapy projects and spiders through a remote service
- • NeutralGitHub Issue
Deploy failed (400)
“i have old service , use python 3.7.1, and use scrapyd deploy is good recently i have new service , and user python 3.7.3, scrapyd can run in new service . but when deploy have failed who's can tell what's mean about Deploy failed (400), i don't find any information about this…”
- • NeutralGitHub Issue
Wrong type adaption for BLOB in scrapyd sqlite3 utils
“When I execute `curl -d project=default -d spider=spider1` I go to scrapyd console and I see the Exception: ``` [-] Unhandled Error Traceback (most recent call last): File "/home/scrapyV1/venv/lib/python2.7/site-packages/twisted/internet/defer.py", line 150, in maybeDeferred…”