A closer look at memorization in deep networks

We examine the role of memorization in deep learning, drawing connections to capacity, generalization, and adversarial robustness. While deep networks are capable of memorizing noise data, our results suggest that they tend to prioritize learning simple patterns first. In our experiments, we expose qualitative differences in gradient-based optimization of deep neural networks (DNNs) on noise vs. real data. We also demonstrate that for appropriately tuned explicit regularization (e.g., dropout) we can degrade DNN training performance on noise datasets without compromising generalization on real data. Our analysis suggests that the notions of effective capacity which are dataset independent are unlikely to…

• CI
  Canadian Institute for Advanced Research
• SA
  Samsung Advanced Institute of Technology
• ID
  Institut de Valorisation des Données
• S
  Samsung
• NS
  Natural Sciences and Engineering Research Council of Canada