NeurIPS 2022 | Meta AI, Stanford & Tübingen U Beat Neural Scaling Laws via Data Pruning

Studies on various machine learning models have shown that applying neural scaling laws — increasing compute, model size, and pretraining data — can reduce errors and significantly improve model performance. But this approach is far from ideal, as the benefits of scaling are relatively weak and unsustainable considering the huge additional…