Back to Devoured - May 21, 2026
DEVOURED
A Bitter Lesson for Data Filtering

A Bitter Lesson for Data Filtering

AI machine-learningresearchdata arXiv
New scaling studies suggest data filtering might be unnecessary for large model pretraining, as ample compute allows models to benefit even from low-quality and distractor data.
What: Researchers Christopher Mohri, John Duchi, and Tatsunori Hashimoto found that in high compute, data-scarce scenarios, large parameter models benefit from including low-quality and distractor data. This challenges the common belief that strict data filtering is essential for pretraining.
Why it matters: This "bitter lesson" shifts the paradigm for large model training, suggesting that raw computational power can compensate for, and even benefit from, less curated datasets, potentially simplifying data pipeline efforts in resource-rich environments.
Takeaway: If you have access to significant compute resources for large model pretraining, reconsider aggressive data filtering strategies, as including lower-quality data might be more beneficial than previously thought.
Deep dive
  • A new research paper titled "A Bitter Lesson for Data Filtering" investigates the role of data filtering in large model pretraining.
  • The study focuses on scaling in a high-compute, data-scarce environment.
  • Contrary to prevailing wisdom, the authors found that the optimal data filter in such conditions is "no data filter."
  • Large parameter models, when sufficiently trained with ample compute, appear to not only tolerate low-quality and distractor data but actually benefit from it.
  • This suggests that computational resources can effectively mitigate the perceived negative impact of "poor" or unfiltered data.
  • The findings could simplify data preparation pipelines for organizations with substantial compute infrastructure.
Decoder
  • Pretraining: The initial phase of training a machine learning model on a very large and diverse dataset, allowing it to learn general features and representations before being fine-tuned for specific tasks.
  • Distractor data: Data that is not directly relevant to the primary learning objective or contains noise, but which the model might still derive useful information from when given sufficient capacity and training.
Original article

A Bitter Lesson for Data Filtering

Abstract:We investigate data filtering for large model pretraining via new scaling studies that target the high compute, data-scarce regime. In spite of an apparently common belief that filtering data to include only high-quality information is essential, our experiments suggest that with enough compute, the best data filter is no data filter. We find that sufficiently trained large parameter models not only tolerate low-quality and distractor data, but in fact benefit from nominally ``poor'' data.

Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI)

Cite as: arXiv:2605.19407 [cs.LG] (or arXiv:2605.19407v1 [cs.LG] for this version)

https://doi.org/10.48550/arXiv.2605.19407 Focus to learn more arXiv-issued DOI via DataCite

Submission history

From: Christopher Mohri

Tue, 19 May 2026 06:02:36 UTC (458 KB)

Access Paper:

  • View PDF
  • HTML (experimental)
  • TeX Source

Which authors of this paper are endorsers?

Explore all 58 articles in today's digest

Fresh dev news, AI-digested daily. Never miss a trend at devoured.fyi