Back to Devoured - June 22, 2026
DEVOURED
Data-Juicer: The Data Operating System for the Foundation Model Era (Tool)

Data-Juicer: The Data Operating System for the Foundation Model Era (Tool)

Data ai GitHub
Data-Juicer provides a Ray-native framework for curating multimodal training data through composable YAML recipes and distributed processing.
What: The tool includes over 200 operators for tasks like deduplication, cleaning, and synthesizing text, image, and video, scaling from local machines to large Ray clusters.
Why it matters: The transition from monolithic data scripts to modular, version-controlled 'data infrastructure' is becoming standard as model training shifts from massive scale to high-quality curation.
Takeaway: Install the framework with `uv pip install py-data-juicer` to define your own cleaning pipelines via YAML.
Decoder
  • Ray: An open-source unified framework for scaling AI and Python applications.
  • Multimodal: Systems capable of processing and synthesizing multiple types of data, such as text, images, and audio.
Original article

Data-Juicer: The Data Operating System for the Foundation Model Era

Multimodal | Cloud-Native | AI-Ready | Large-Scale

Data-Juicer (DJ) transforms raw data chaos into AI-ready intelligence. It treats data processing as composable infrastructure—providing modular building blocks to clean, synthesize, and analyze data across the entire AI lifecycle, unlocking latent value in every byte.

Whether you’re deduplicating web-scale pre-training corpora, curating agent interaction traces, or preparing domain-specific RAG indices, DJ scales seamlessly from your laptop to thousand-node clusters—no glue code required.

Alibaba Cloud PAI has deeply integrated Data-Juicer into its data processing products.

🚀 Quick Start

Zero-install exploration:

  • JupyterLab Playground with Tutorials
  • Ask DJ Copilot

Install & run:

uv pip install py-data-juicer
dj-process --config demos/process_simple/process.yaml

Or compose in Python:

from data_juicer.core.data import NestedDataset
from data_juicer.ops.filter import TextLengthFilter
from data_juicer.ops.mapper import WhitespaceNormalizationMapper

ds = NestedDataset.from_dict({
    "text": ["Short", "This passes the filter.", "Text   with   spaces"]
})
res_ds = ds.process([
    TextLengthFilter(min_len=10),
    WhitespaceNormalizationMapper()
])

for s in res_ds:
    print(s)

✨ Why Data-Juicer?

1. Modular & Extensible Architecture

  • 200+ operators spanning text, image, audio, video, and multimodal data
  • Recipe-first: Reproducible YAML pipelines you can version, share, and fork like code
  • Composable: Drop in a single operator, chain complex workflows, or orchestrate full pipelines
  • Hot-reload: Iterate on operators without pipeline restarts

2. Full-Spectrum Data Intelligence

  • Foundation Models: Pre-training, fine-tuning, RL, and evaluation-grade curation
  • Agent Systems: Clean tool traces, structure context, de-identification, and quality gating
  • RAG & Analytics: Extraction, normalization, semantic chunking, deduplication, and data profiling

3. Production-Ready Performance

  • Scale: Process 70B samples in 2h on 50 Ray nodes (6400 cores)
  • Efficiency: Deduplicate 5TB in 2.8h using 1280 cores
  • Optimization: Automatic OP fusion (2-10x speedup), adaptive parallelism, CUDA acceleration, robustness
  • Observability: Built-in tracing for debugging, auditing, and iterative improvement

📰 News

[2026-05-29] Release v1.5.2: Semantic LLM OPs, Cross-doc Line Dedup & Leaner Dependencies

  • 🧹 New Deduplicator — Added DocumentLineDeduplicator for cross-document line-level dedup, removing boilerplate lines (templates, copyright notices, navigation bars) by global document frequency.
  • 🤖 Agent Data Quality Toolkit — Shipped interaction-quality OPs & recipe, a bad-case HTML report, and more robust JSONL / HuggingFace meta loading.
  • 📦 Leaner & Faster Install — Slimmed the default dependency set (Ray, audio, spaCy, av, etc. moved to on-demand extras) to speed up installation.
  • 🐳 Stability & Robustness Fixes — Library-safe error handling (raise over exit(1)), Ray init/temp-dir fixes, valid API params (drop invalid max_new_tokens), PyArrow 20+ batch JSON reading, local-path aesthetics model support, and more performance/bug fixes.
  • 🧠 Semantic LLM Operators — Introduced llm_extract_mapper, llm_condition_filter, and llm_structured_ops with unified llm_* naming and configurable inference strategies (join/agg/top-k planned).

🔌 Users & Ecosystems

Data-Juicer plugs into your existing stack and evolves with community contributions:

Extensions

  • data-juicer-agents — DJ Copilot and agentic workflows
  • data-juicer-hub — Community recipes and best practices
  • data-juicer-sandbox — Data-model co-development with feedback loops

Frameworks & Platforms

AgentScope · Apache Arrow · Apache HDFS · Apache Hudi · Apache Iceberg · Apache Paimon · Alibaba PAI · Delta Lake · DiffSynth-Studio · EasyAnimate · Eval-Scope · Huawei Ascend · Hugging Face · LanceDB · LLaMA-Factory · ModelScope · ModelScope Swift · NVIDIA NeMo · Ray · RM-Gallery · Trinity-RFT · Volcano Engine

📖 Citation

@inproceedings{djv1,
  title={Data-Juicer: A One-Stop Data Processing System for Large Language Models},
  author={Chen, Daoyuan and Huang, Yilun and Ma, Zhijian and Chen, Hesen and Pan, Xuchen and Ge, Ce and Gao, Dawei and Xie, Yuexiang and Liu, Zhaoyang and Gao, Jinyang and Li, Yaliang and Ding, Bolin and Zhou, Jingren},
  booktitle={SIGMOD},
  year={2024}
}

@article{djv2,
  title={Data-Juicer 2.0: Cloud-Scale Adaptive Data Processing for and with Foundation Models},
  author={Chen, Daoyuan and Huang, Yilun and Pan, Xuchen and Jiang, Nana and Wang, Haibin and Zhang, Yilei and Ge, Ce and Chen, Yushuo and Zhang, Wenhao and Ma, Zhijian and Huang, Jun and Lin, Wei and Li, Yaliang and Ding, Bolin and Zhou, Jingren},
  journal={NeurIPS},
  year={2025}
}
Explore all 69 articles in today's digest

Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi