Deep Learning on a Data Diet

On the other hand, unsupervised learning is a paradigm that aims at learning to generate meaningful and comprehensible representations solely from inputs. Unsupervised learning remains one of the most challenging tasks in modern machine learning and deep learning despite the recent success, in particular, of self-supervised learning, which is currently widely used in many applications, including image and speech recognition, natural language processing, and recommendation systems. Due to multiple moving pieces, unsupervised learning is complicated and lacks reproducibility, scalability, and explainability. Yet, self-supervised learning is limited by unintelligibility, numerous hyperparameters inconsistent among architectures and datasets, and a lack of theoretical guarantees. More precisely, DIET is highly sensitive to the strength of data augmentation, similar to self-supervised learning, and the convergence is slower than self-supervised learning, but label smoothing helps.