Apache Spark
Apache Spark is a combined analysis engine for large-scale data processing. Scala offers high-level API for Java and Python (R API marked as deprecated). Spark for SQL and DataFrame includes high-end tools such as SQL, pandas for pandas workloads API on Spark, MLlib for machine learning, GraphX for graph processing, and Structured Streaming for flow processing.
Apache Spark
100apache · Data Pipelines & ETL
Common commercial products in this category: Fivetran, Informatica, Stitch
- You manage data analysis processes using Spark SQL and DataFrame structures.
- You develop machine learning models with the MLlib library.
- You process data streams through Structured Streaming.
- • High number of open issues (742)
- • NeutralHacker News
Amazon's exabyte-scale migration from Apache Spark to Ray on EC2
“I remember when everyone shifted from Apache Hadoop to Apache Spark. This seems like a possibly similar sea change. I am not sure if many other users will embrace Ray over Spark, but this is a sign that people are looking to either improve Spark on some fundamental levels, or…”
- • NeutralHacker News
BlazingSQL – GPU SQL Engine Now Runs Over 20X Faster Than Apache Spark
“Further down in the article: In our original demo, we ran a V100 GPU against 8 CPU nodes. The new T4 GPUs cut our cost in half, which meant we reduced the Apache Spark cluster to 4 CPU nodes in order to maintain price parity. Even with the reduced GPU memory, the whole workload…”
Verified against README · Oct 26, 2026 · source
My note
Alternative pages
More in: Data Pipelines & ETL
- Apache Flink★ 26,441 · today
- rocketride-server★ 19,018 · today
- Luigi★ 18,780 · 16 days ago
- OpenMetadata★ 15,637 · today
- cocoindex★ 11,788 · today
- ElectricSQL★ 10,420 · 24 days ago