docetl
DocETL is a declarative and agent-based map-download system developed to process large amounts of structured and unstructured data with the help of language models. It coordinates operations written in natural language, processing data in parallel and automatically optimizing the pipeline. It implements optimizations such as model change, prompt rewrite, and task parsing in order to increase accuracy and reduce cost.
ucbepic · Data Pipelines & ETL
Common commercial products in this category: Fivetran, Informatica, Stitch
- With natural language expressions, you process large data sets by defining operations such as map, reduce, and filter.
- You create data processing pipelines using Python API or YAML configuration files.
- Visually edit requests through the DocWrangler interface and observe the results in real time.
- ↑ PositiveGitHub Issue
Add Support for PDF Input with OCR Capabilities
“## Objective Extend DocETL to support PDF files as input, including the ability to extract text using OCR for scanned documents or images within PDFs. ## Background Currently, DocETL supports structured data formats like JSON. Adding PDF support with OCR will greatly expand our…”
- ↑ PositiveGitHub Issue
Add flexible support for other LLM providers
“First of all, many thanks for sharing this project! It's a much better execution of an idea we (University of Groningen, Center for Information Technology) had floating internally. We are currently serving models through vLLM for our research staff and this would be a great tool…”
My note
Alternative pages
More in: Data Pipelines & ETL
- Apache Spark★ 44,376 · today
- Apache Flink★ 26,441 · today
- rocketride-server★ 19,018 · today
- Luigi★ 18,780 · 16 days ago
- OpenMetadata★ 15,637 · today
- cocoindex★ 11,788 · today