To understand the statistical behavior of the neural network training process, I adopt back-propagation as a basic method of the neural network training process, and analyze the evolution of the probability density in a Fokker-Planck equation. And I choose a simple feed-forward neural network which consists of two input neurons, one output neuron, and two weights. The evolution of the probability density in a Fokker-Planck equation can be understood as a distribution, which represents the various paths that weights change per one sample data in a back-propagation method. The Probability density spreads out to the direction of error minimum weight point in the beginning. After some numerical iterations the peak located at error minimum point increases while the other peaks decrease very rapidly. Finally only one sharp peak located at error minimum point remains. And at that point, all jump moments in a Fokker-Planck equation have zero value.
