Deep Learning with Limited Numerical Precision

Training of large-scale deep neural networks is often constrained by the available computational resources. We study the effect of limited preci-sion data representation and computation on neu-ral network training. Within the context of low-precision fixed-point computations, we observe the rounding scheme to play a crucial role in de-termining the network’s behavior during train-ing. Our results show that deep networks can be trained using only 16-bit wide fixed-point num-ber representation when using stochastic round-ing, and incur little to no degradation in the classification accuracy. We also demonstrate an energy-efficient hardware accelerator that imple-ments low-precision fixed-point arithmetic with…