Soft Sign Activation Function with Tensorflow [ Manual Back Prop with TF ]

GIF from this website

So the paper “Quadratic Features and Deep Architectures for Chunking” have been around since 2009, however I recently found about the activation function introduced in that paper while reading “Understanding the difficulty of training deep feed-forward neural networks” (I also did a paper summary on that matter, if anyone is interested in that please click here.)