data-juicer

Data-Juicer is a modular data operating system that processes text, images, audio and video data for basic models. It supports processes of clearing pre-training compilations on the web scale, compiling agent interaction tracks, and preparing RAG indexes specific to the domain. The structure, which can be scaled from a single laptop to a thousand-node clusters, offers YAML-based pipelines and over 200 operators.