We present a novel methodology based on a Taylor expansion of the network
output for obtaining analytical expressions for the expected value of the
network weights and output under stochastic training. Using these analytical
expressions the effects of the hyperparameters and the noise variance of the
optimization algorithm on the performance of the deep neural network are
studied. In the early phases of training with a small noise coefficient, the
output is equivalent to a linear model. In this case the network can generalize
better due to the noise preventing the output from fully converging on the
train data, however the noise does not result in any explicit regularization.
In the later training stages, when higher order approximations are required,
the impact of the noise becomes more significant, i.e. in a model which is
non-linear in the weights noise can regularize the output function resulting in
better generalization as witnessed by its influence on the weight Hessian, a
commonly used metric for generalization capabilities.
Akionux
All senooken JP Social content and data are available under the