Loss functions in deep learning

Demystifying Focal Loss I: A More Focused Cross Entropy Loss

Rethinking cross entropy loss for semantic segmentation tasks

Inokashira park in Tokyo

In computer vision, cross entropy is a widely used loss item in classification problems. In semantic segmentation, we use it for each pixel. However, this per-pixel fashion may cause problems in some scenarios.

Figure 1: a sparse binary image

Imagine we have a binary image like Figure 1, in which six pixels are labeled to 1 (blue) and all other pixels labeled to 0 (white). Training a model to segment it will be a pixel level binary classification problem. Ordinarily, you forward the image through an encoder-decoder style network, after which a sigmoid function is used to convert the prediction to a probability value with range 0 ~ 1. Finally, binary cross entropy loss is used for backpropagation.

The loss function looks good. In general, for binary classification, cross entropy is a standard loss. However in this case, since the blue areas are sparse and small, the loss will be overwhelmed by white areas. As the loss is back propagated as a whole, it will be difficult for the…