Eigenvalue Decay is a new option of weight regularizer to the deep learning practitioner that aims at maximum-margin learning.