Neural networks are notoriously difficult to train. They can completely diverge on one set of parameters and get state-of-the-art results on a slightly different set of parameters. Over the years I’ve learned several tricks to training neural networks. A lot of these tricks are well kept secrets in industry and academics and I think it’s time the rest of the world learned how neural networks are really trained. So at the risk of being excommunicated from the deep learning community, here is my write up of the peculiarities of training neural networks.