reviewIEEE Transactions on Pattern Analysis and Machine IntelligenceFeb 28, 2023Closed access

Normalization Techniques in Training DNNs: Methodology, Analysis and Application

LHLei HuangJQJie QinYZYi ZhouFZFan ZhuLLLi Liu

Beihang University · Nanjing University of Aeronautics and Astronautics · +3 more institutions

PubMed
Indexed incrossrefpubmed

Abstract

Normalization techniques are essential for accelerating the training and improving the generalization of deep neural networks (DNNs), and have successfully been used in various applications. This paper reviews and comments on the past, present and future of normalization methods in the context of DNN training. We provide a unified picture of the main motivation behind different approaches from the perspective of optimization, and present a taxonomy for understanding the similarities and differences between them. Specifically, we decompose the pipeline of the most representative normalizing activation methods into three components: the normalization area partitioning, normalization operation and normalization…

Citation impact

456
total citations
FWCI
50.88
Percentile
100%
References
484
Citations per year

Authors

6

Topics & keywords

Topics
Keywords
  • Normalization (sociology)
  • Computer science
  • Artificial intelligence
  • Machine learning
  • Deep neural networks
  • Artificial neural network
No related works found for this paper.

Funding