opinion | May 27, 2026 Why Learning from Data Gets Harder in the Tail Lessons for Autonomous Driving Prof. Shai Shalev-Shwartz and Prof. Amnon Shashua Knowing the obstacle exists means we can navigate around it. The scaling paradox There's a puzzling gap in how machine learning systems improve. Classical learning theory tells us that as we collect more data, our models should get better at a steady, predictable rate—roughly proportional to how much new data we gather. But empirical evidence paints a different picture. Language models and end-to-end learning systems in the real world often improve much more slowly than theory predicts, requiring orders of magnitude more data for only marginal improvements. This isn't a failure of implementation—it's a fundamental consequence of how Stochastic Gradient Descent (SGD) interacts with certain problem structures. Understanding why matters deeply for autonomous driving, especially if one relies on end-to-end imitation learning to handle the full complexity of real-world scenarios. Grounding in empirical reality: the Chinchilla Law The severity of this problem becomes concrete when we look at empirical scaling laws. Recent research on large language models (the Chinchilla law) has shown how error decreases with the number of training examples \(T\), finding that error scales as \(T^{-\alpha}\) where \(\alpha < 0.1\). An exponent of \(0.1\) means convergence is orders of magnitude slower than classical learning theory predicts. To appreciate the implications: suppose we want to reduce error from \(10^{-2}\) to \(10^{-4}\)—a hundred-fold improvement. With \(\alpha = 0.1\), this requires: Twenty orders of magnitude more data. For systems requiring very high accuracy—like autonomous driving, where safety demands error rates far below \(10^{-4}\)—this scaling becomes prohibitively expensive. The theoretical construction below explains why this slow convergence arises from the interaction of weak supervision and heavy-tailed data when trained with SGD. The two properties that slow everything