New insights into training dynamics of deep classifiers

In the study, the authors focused on two types of deep classifiers: fully connected deep networks and convolutional neural networks (CNNs). A previous study examined the structural properties that develop in large neural networks at the final stages of training. Co-author and MIT McGovern Institute postdoc Akshay Rangamani states, “Our analysis shows that neural collapse emerges from the minimization of the square loss with highly expressive deep neural networks. Thus far, the fact that CNNs and not dense networks represent the success story of deep networks has been almost completely ignored by machine learning theory. Instead, the theory presented here suggests that this is an important insight in why deep networks work as well as they do.