Open source Bright Data alternatives

Puppeteer Support for Firefox

“Well the truth is it's both. We had to change Firefox so it could be automated with WebDriver BiDi. The Puppeteer team had to change Puppeteer in order to implement a WebDriver BiDi backend, and to enable specific support for downloading and launching Firefox. As the article…”

Puppeteer Support for Firefox

“I'm not convinced. It looks like v0.10.0 contains ~half of Google's Puppeteer code and even in the latest release[0]the core package references Google's copyright several hundred times. Conceptually, the core, the bridge between a node server and the injected Chrome DevTools…”

Scrapling: Fast, Adaptive Web Scraping for Python

“Scrapling is a high-performance, intelligent web scraping library for Python that automatically adapts to website changes while significantly outperforming popular alternatives. Whether you're a beginner or an expert, Scrapling provides powerful features while maintaining…”

"'TextHandlers' object has no attribute 'partition'" error when trying to fetch page

“### Have you searched if there an existing issue for this? - [x] I have searched the existing issues ### Python version (python --version) Python 3.13.1 ### Scrapling version (scrapling.__version__) 0.2.95 ### Dependencies version (pip3 freeze) aiohappyeyeballs==2.4.4…”

[Bug]: Error: Page.content: Target page, context or browser has been closed

“### crawl4ai version 0.5.0.post4 ### Expected Behavior Crawler should crawl ### Current Behavior I get the following error [ERROR]... × | Error: ┌───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ ×…”

Error using crawler: maximum recursion depth exceeded

“I'm getting the `maximum recursion depth exceeded` error in the crawling application I have. See _portion_ of the trace below (I'm only including a portion because the whole thing is insanely long, and the lines repeat over and over again, so not too useful). Some important…”

Scrapy Tips from the Pros

“Much like apples to oranges. Nokogiri is a tag soup parser. Scrapy is a web scraping framework. In addition to tag soup parsing, Scrapy handles a slew of things such as text encoding problems, retrying urls that fail owing to network problems (if you wish), dispatching requests…”

Scrapy Tips from the Pros

“> would you be better off using a search engine crawler like Nutch for this? Fwiw we recently met a client who tried Scrapy + Frontera vs Nutch, and their assessment was that Scrapy + Frontera is twice as fast. Here's a deck on Frontera FYI: Aside: don't hesitate to get in touch…”

Consider using system trust stores by default in 3.0.0.

“It's been raised repeatedly, mostly by people using Linux systems, that it's annoying that requests doesn't use the system trust store and instead uses the one that certifi ships. This is an understandable position. I have some personal attachment to the certifi approach, but…”

Specify password for SSL client side certificate

“As far as I know currently it's not possible to specify the password for the client side certificate you're using for authentication. This is a bit of a problem because you typically always want to password protect your .pem file which contains the private key. `openssl` won't…”

mitmproxy adds --emudhra option

“For the public WiFi threat vector (e.g. a rogue WiFi AP running MITMproxy to sniff your traffic) you would be protected by vpn. But the broader threat, e.g. older & unpatched devices with these flimsy CAs installed & not upgradable or revocable – tough cookies.”

mitmproxy adds --emudhra option

“> I prefer haproxy to mitmproxy. It's faster If you’re proxying all the traffic in your network it makes sense, naturally. mitmproxy is usually used for debugging and/or reverse engineering (e.g. with a specific app).”

Browser DevTools MCP: 78% less tokens than Playwright MCP, faster and consistent

“Yeah, you’re right. The most of the gain is from the execute tool (batching many steps in one call), not from CDP vs Playwright. If Playwright MCP added a similar sandboxed “run script that calls other tools” and the model used it the same way, token use and turn count would…”

Browser DevTools MCP: 78% less tokens than Playwright MCP, faster and consistent

“We ran the same verification task with Claude Code, same prompt, 3 runs per MCP. Browser DevTools MCP: ~330K tokens, 12 turns, ~2m45s. Playwright MCP: ~1.5M tokens, 48–52 turns, ~6m. So 78% fewer tokens, 76% fewer turns, ~57% faster. We compare task-level totals rather than…”

Show HN: Vibium – Browser automation for AI and humans, by Selenium's creator

“Also, as someone on the periphery of Selenium (mostly via WebDriver), some of the challenge is that Selenium has a huge amount of test code already written for it — and making radical API changes would break every test already written for it, and at that point you’re effectively…”

Show HN: Vibium – Browser automation for AI and humans, by Selenium's creator

“The good old days of browser automation! Thanks a lot for Selenium and all the best for Vibium :)”

[1.14.0+] pydantic ValidationError with SmartScraperGraph

“**Describe the bug** Since version 1.14.0 (tested here on 1.15.0), SmartScraperGraph on OpenAI stopped working with a pydantic-related error: ``` /venv/lib/python3.12/site-packages/google_crc32c/__init__.py:29: RuntimeWarning: As the c extension couldn't be imported,…”

v1.15.0: OmniScraperGraph not working: Error parsing input keys for ImageToText

“**Describe the bug** OmniScraperGraph throws error. Tested on minimal example on GitHub. [omni_scraper_openai.py]( **To Reproduce** ```bash mkdir test cd test python3 -m venv venv source venv/bin/activate ``` ```bash pip install scrapegraphai \ "scrapegraphai[burr]" \…”

f.lux for iOS is no longer available

“I like having an intuitive, easy-to-use WIMP system for my common tasks. I don't want to read my e-mail in a terminal window, or use some X11 chat app with a programmer-designed UI. I also like being able to get a terminal window to fiddle with the guts when I want it. Some…”

f.lux for iOS is no longer available

“Well, Redshift is a lightweight program that also works, and has a simple-to-use shell command. That's attractive to some Linux-users :) And it goes all the way down to 1000K, leaving only the red channel :) Which is not particularly useful, but it does make Redshift literally…”

ArchiveBox is evolving: the future of self-hosted internet archives

“>ArchiveBox is also gearing up to support real cryptographic signing of archives That's a really interesting point. The gut reaction is "why are we wasting time on adding a nice-to-have such as a very fancy cousin of the MD5 checksum when the real meat of the time & effort is…”

ArchiveBox is evolving: the future of self-hosted internet archives

“I've tried to get started with ArchiveBox many times, but it was always quite buggy (Not working in Safari, a bit clunky to run,...) but I've noticed a lot of updates in the past months so I'm excited about it moving forward and giving it another shot.”

Show HN: Stagehand – an open source browser automation framework powered by AI

“Yes and no. Getting a VLM to work on the web would definitely be great, but it comes with its own problems, mainly around developing and acting on bounding boxes. We have vision as a default fallback for Stagehand, but we've found that the screenshot sent to the VLM often has to…”

Show HN: Stagehand – an open source browser automation framework powered by AI

“Yes! These are both phenomenal projects, and kudos to their authors as well. Stagehand is different in that it makes fine-grained control a first-class citizen. Often times, you want to control the exact steps a web agent takes. Our experience using other tools was that the only…”

Colly – Scraping Framework for Golang

“Please breakup your main `colly.go` file into separate parts. If possible you shouldn't have a 30 line imports definition covering everything from cookies and regex to html and sync access. Make sure to use DNS caching on the box else add it in Go. Colly only supports a single…”

Colly – Scraping Framework for Golang

“Good performance matters if you have decent networking infrastructure or your server has limited resources. Bandwidth and IP limits are the most common bottle necks, but these can be solved using multiple proxies and ssh tunnels. Colly has built in support for switching proxies…”

The tragic death of an actor: Molière (2013)

“>"A year later, Molière’s self-mockery has grown more explicit. The new play is The School for Wives Criticised, a short, informal sketch, ridiculing Molière’s critics in an argument about The School for Wives. Significantly, Molière didn’t defend his own play onstage. Instead,…”

SurfSense a Knowledge Graph Brain for World Wide Web Surfers

“When I’m browsing the internet, I tend to save a ton of content—but remembering when and what I saved? Is hard to remember! That’s where SurfSense comes in. SurfSense is like a Knowledge Graph Brain for anything you see on the World Wide Web. Now, you’ll never forget any…”