Crawl4AI
Crawl4AI is an open-source web browser and scraper that extracts data from websites, producing outputs in a clean Markdown format for RAG, AI agents, and data pipelines. It works fast with asynchronous browser pool and caching mechanisms; it offers crash recovery and fast URL reconnaissance modes for deep scanning operations. LLM compliant smart outputs and optimizes developers' data collection processes by providing full control.
Crawl4AI
100unclecode · Web Scraping & Automation
Common commercial products in this category: Bright Data, Apify, ScrapingBee
- You convert content from websites into a clean Markdown format for LLM and RAG systems.
- In deep scanning processes, you collect large-scale data with crash recovery and cache features.
- Thanks to the asynchronous browser pool and fast URL discovery mode, you can quickly scan websites.
- ↓ NegativeGitHub Issue
[Bug]: Error: Page.content: Target page, context or browser has been closed
“### crawl4ai version 0.5.0.post4 ### Expected Behavior Crawler should crawl ### Current Behavior I get the following error [ERROR]... × | Error: ┌───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ ×…”
- • NeutralGitHub Issue
Error using crawler: maximum recursion depth exceeded
“I'm getting the `maximum recursion depth exceeded` error in the crawling application I have. See _portion_ of the trace below (I'm only including a portion because the whole thing is insanely long, and the lines repeat over and over again, so not too useful). Some important…”
Verified against README · Oct 26, 2026 · source