Loss functions in TensorFlow 2.0
By Amanpreet SinghIn this tutorial, we will learn about several types of loss functions and their implementation in Python using TensorFlow.
What is the loss function?
The loss function is often referred to as the error function or the cost function. These are errors caused by machines while training data, and by utilizing an optimizer and modifying weight machines, you may limit the loss and accurately predict results.
The formula for calculating loss :
Loss= abs(Y_pred – Y_actual)
Inputs required by the loss function are as follows :
- y_true (true label): Can either be 0 or 1.
- y_pred (predicted value): It is the predicted label from the model i.e, a single floating-point value which represents either a logit, (i.e, value that ranges from [-inf to inf] when from_logits=True) or a probability (i.e, value range in between [0., 1.] when from_logits=False).
Keras and TensorFlow both provide loss function documentation, but with the same code, which you can get here:
Types of Loss Function
Binary Cross-Entropy loss
The cross-entropy between true labels and anticipated outputs is calculated using binary cross-entropy. It is used when the two-class issues arise like (yes or no, failure or success,0 or 1, head or tail when tossing a random coin).
First of all Importing TensorFlow library to calculate the different types of loss.
Check the TensorFlow version.
import tensorflow as tf print(tf.__version__)
Output
2.7.0
Let’s see how to implement binary cross-entropy in Python.
# Input values y_true=[[0.,1.],[1.,0.]] # Predicted y_pred=[[0.4,0.5],[0.3,0.6]] #Binary-Cross-entropy binary_cross=tf.keras.losses.BinaryCrossentropy() binary_cross(y_true=y_true,y_pred=y_pred).numpy()
In the above code, we take batch size = 2 i.e y_true, and number of samples = 4 i.e y_pred
Output
0.83105886
This is how we can compute binary cross-entropy.
Categorical Crossentropy
To compute the loss between true and predicted labels, the categorical cross-entropy loss function is utilized. It’s mostly used to solve problems with multiclass categorization. For example Image Classification of human body parts like ear, nose, knee, belly, etc.
Let’s see how to implement Categorical Crossentropy.
# Actual y_true = [[0, 0, 1], [0, 1, 0]] #Predicted y_pred = [[0.04, 0.90, 0.50], [0.1, 0.4, 0.1]] # Cat cross loss cat_cross_entropy = tf.keras.losses.CategoricalCrossentropy() cat_cross_entropy(y_true=y_true,y_pred=y_pred).numpy()
In the above code, we take batch size=2 i.e y_true and num_of_samples=8 i.e y_pred
Output
1.1308815
That’s how we can calculate categorical cross-entropy.
Sparse Categorical Crossentropy loss
When there are two or more classes in our classification work, Sparse Categorical Crossentropy is employed. However, there is one distinction between categorical and sparse categorical cross-entropy: sparse categorical cross-entropy labels are supposed to be supplied in integers.
Let’s see the implementation of sparse categorical cross-entropy.
#iActual label
y_true = [1, 2]
#Predicted Labl
y_pred = [[0.07, 0.90, 0],
[0.1, 0.6, 0.1]]
#Sparse Categorical-Crossentropy
tf.keras.losses.sparse_categorical_crossentropy(y_true,y_pred).numpy()Output
array([0.07490139, 2.0794415 ], dtype=float32)
That’s how we can calculate sparse categorical cross-entropy.
Poisson Loss
It is the average of Tensors elements. Poisson loss can be calculated using the formula:
y_pred – y_true*log(y_true)
Let’s see how to implement Poisson loss in python :
y_true = [[1., 0.], [0., 0.]] #Predicted y_pred = [[1., 0.], [0., 0.]] # Poisson loss pois = tf.keras.losses.Poisson() pois(y_true, y_pred).numpy()
Output
0.24999997
Kullback-Leibler Divergence Loss
It’s also known as KL divergence, and it’s determined by taking the negative sum of each event’s probability P and multiplying it by the log of the likelihood of that event.
Formula
Let’s see how to implement KL Divergence loss using TensorFlow.
y_true = [[0, 1], [0, 0]] y_pred = [[0.4, 0.6], [0.6, 0.4]] # The reduction type 'auto'/'sum over batch size' is being used. kl = tf.keras.losses.KLDivergence() kl(y_true, y_pred).numpy()
Output
0.2554105
Mean Squared Error (MSE)
The MSE value indicates how near a regression line is to the anticipated points. And it’s as simple as squaring the distance between the spot and the regression line. We employ square to solve the problem of the negative sign.
Formula
MSE value lies in between 0 to infinity. A smaller value of MSE indicates the best performance of the model.
MSE is sensitive to outliers.
Implementation for MSE:
#Actual
y_true = [[9., 9.],
[0., 0.]]
#Predicted
y_pred = [[9., 9.],
[1., 0.]]
#MSE
mse = tf.keras.losses.MeanSquaredError()
mse(y_true, y_pred).numpy()Output
0.25
Mean Absolute Error(MAE)
The MAE is calculated by calculating the distance between a point and the regression line. MAE is also sensitive to outliers just like MSE, always remove all outliers in data when you use MAE.
Formula
Implementation for MAE:
# Input values
y_true = [[10., 15.],
[20., 25.]]
# Predicted values
y_pred = [[10., 15.],
[20., 0.]]
#MAE calculation
mae = tf.keras.losses.MeanAbsoluteError()
mae(y_true, y_pred).numpy()Output
6.25
Huber Loss
The Huber loss is the mixture of ( MSE+MAE). Huber loss is robust to outliers.
For small values, the Huber loss function is quadratic, whereas, for bigger values, it is linear.
Formula
Let’s see the implementation of Huber loss in python:
# Actual values
y_true = [[15., 20.],
[25., 30.]]
# Predicted value
y_pred = [[15., 20.],
[25., 0.]]
#Huber loss
h_loss = tf.keras.losses.Huber()
h_loss(y_true, y_pred).numpy()Output
7.375
Hinge Loss
Hinge loss is used by Support Vector Machines(SVM) to solve problems like “maximum margin” classification.
The value of y_true in Hinge loss is [-1 or 1].In Hinge loss binary value i.e 0 or 1 converted into -1 or 1.
Formula
Implementation of Hinge loss using TensorFlow:
#Actual value y_true = [[1., 0.], [1., 0.]] #Predicted value y_pred = [[0.4, 0.5], [0.5, 0.4]] #Hinge loss h_loss = tf.keras.losses.Hinge() h_loss(y_true, y_pred).numpy()
Output
1.0
Leave a Reply