Data Filtering

Data filtering is the process of systematically removing low-quality, irrelevant, harmful, or redundant examples from a training dataset to improve model performance. In the era of large-scale web-scraped data, filtering has become one of the most impactful steps in the ML pipeline — the quality of training data often matters more than its quantity.

Common Filtering Criteria

• Language Detection: Remove text in unintended languages using tools like fastText language identification.
• Quality Scoring: Use heuristics or classifiers to score text quality — remove content that is too short, too repetitive, mostly URLs/boilerplate, or poorly formatted.
• Toxicity Filtering: Remove text containing hate speech, explicit content, or violence using classifiers like Perspective API.
• Deduplication: Remove exact and near-duplicate content (see data deduplication).
• Perplexity Filtering: Remove text with very high or very low perplexity as measured by a reference language model — extreme perplexity often indicates garbage or trivial content.
• Domain Filtering: Select or exclude specific domains (e.g., keep educational content, remove social media spam).

Impact on Model Quality

• The Llama training pipeline applies extensive filtering to Common Crawl data, keeping only ~5% of raw web text.
• Phi models from Microsoft demonstrated that a small, highly filtered dataset can train models competitive with those trained on much larger, less filtered data.
• DCLM (DataComp for Language Models) showed that better data filtering algorithms consistently lead to better model performance.

Best Practices

• Multiple Passes: Apply filtering in stages — cheap heuristic filters first, expensive classifier-based filters later.
• Sample Inspection: Manually inspect random samples of filtered-in and filtered-out data to verify filter quality.
• Filter Logging: Track why each example was removed to enable analysis and adjustment.

Data filtering is increasingly recognized as one of the highest-ROI activities in ML development — clean data reduces training time, improves performance, and reduces harmful outputs.