Variance normalising pre-training of neural networks.
NOTE: it turns out similar ideas have already recently been investigated:
NB After having a look at the above, and playing around with the
initialisation methods. Their results usually beat the experiments below!
Experiments based on normalising neural network activations (or weights)
in a pretraining step, using random input noise.
The network is trained to achieve a target mean and standard deviation after each layer,
based on this normally distributed noise.
Results show that this method can improve training speed and
performance using the same number of weights.
