Published in

Sci-Net

# Objective Functions in Deep Learning

In this report, I shall summarize the objective functions ( loss functions ) most commonly used in Machine Learning & Deep Learning. I have given a priority to loss functions implemented in both Keras and PyTorch since it sounds like a good reflection of popularity and wide adoption. For each loss function, I shall provide the formula, the pros, and the cons. Loss functions can be grouped into two categories based on the inference task; Regression and Classifications. However, it is not exclusive since some loss functions can be used with slight or no modification with one task or the other.

Regression Objective Functions:

• Mean Absolute Error ( MAE )

It is a quite simple objective function, but it lacks stability and robustness.

• Mean Square Error ( MSE )

MSE penalizes the errors by squaring them which acts as an exaggeration factor in order to force the model to learn from previous mistakes. However, MSE is highly sensitive to outliers and there are some modifications that suggest a solution to outliers.

• Huber

This method is less sensitive to outliers since it only squares the difference if it’s below a predefined threshold delta.

• Poisson
• Cosine

This is very similar to cosine similarity and works by minimizing the dot product between the output vector and the ground truth vector. The value 0 indicates maximum similarity or parallel vectors and the value +1 indicates maximum dissimilarity or orthogonal vectors.

Classification Objective Functions:

• Hinge

Most commonly used for optimizing Support Vector Machine ( SVM ) models, but it suffers from the fact that its derivative is discontinuous at j = y_{i}, and that’s a why a variant was introduced that squares the difference in order to introduce a continuous derivative.

• Binary Cross Entropy

The special case of log loss when the target is a binary classification value. It’s based on cross-entropy between two probabilities. A large cross entropy indicates divergence and a small cross entropy indicates similarity. It converges much faster than MSE and its derivative has favorable properties such as being either to compute and use with non-linear activations.

• Categorical Cross Entropy

The extension of cross-entropy to the multi-label classification problem.

• Kullback-Leibler

It minimizes the distance between two probability distribution and it’s mostly used in Generative Adversarial Networks ( GANs ). It converges by making the probability distribution of the predicted output and the ground truth very close to each other. It’s based on concepts from Shanon information theory and cross entropy.

• Negative Logarithmic Likelihood

It’s very similar to log loss ( cross entropy ) and it is equivalent to maximizing the probability that a given sample is generated from the target class distribution. It’s also equivalent to Maximum A Posteriori MAP estimation in Bayesian inference.

• Cauchy-Schwarz

According to On Loss Functions for Deep Neural Networks in Classification, Katarzyna Janocha et al., 2017, it was found that Cauchy-Schwarz divergence, when used as an objective function on both MNIST and CIFAR-10, performs better than Log Loss.

Conclusion

The choice of the objective function has an impact on the speed of learning and the overall inference accuracy. Each learning task requires an adapted objective function such as regression, classification, super-resolution, style transfer, or generation. In practice, one should benchmark results obtained from different objective functions and fine-tune the hyper-parameters that best suit the task at hand.

According to Support Vector Machines by Andreas ChristmannIngo Steinwart the loss functions can be categorized as margin-based or distance-based losses. It dives deeper into the convexity, concavity, and continuity properties of each objective function and how it affects the statistical learning process.

Check out my course on PacktPub:
Machine Learning for Algorithmic Trading Bots with Python

Support Vector Machines, Steinwart et al.:
https://www.springer.com/gp/book/9780387772417

On Loss Functions for Deep Neural Networks in Classification, Janocha et al., 2017:
https://arxiv.org/abs/1702.05659

--

--

## More from Sci-Net

Digital Image Processing, Machine Learning, Deep Learning, and Computer Vision