IOS>> Loss-Functions-Package-Tensorflow-Keras-PyTorch>> 返回
项目作者： Mr-TalhaIlyas

项目描述 ：
Loss function Package Tensorflow Keras PyTOrch
高级语言：
项目地址: git://github.com/Mr-TalhaIlyas/Loss-Functions-Package-Tensorflow-Keras-PyTorch.git
创建时间： 2021-03-13T05:24:24Z
项目社区：https://github.com/Mr-TalhaIlyas/Loss-Functions-Package-Tensorflow-Keras-PyTorch
开源协议：Apache License 2.0
下载


License
PyTorch Keras TensorFlow Hits

Loss-Functions-Package-Tensorflow-Keras-PyTorch

This rope implements some popular Loass/Cost/Objective Functions that you can use to train your Deep Learning models.

With multi-class classification or segmentation, we sometimes use loss functions that calculate the average loss for each class, rather than calculating loss from the prediction tensor as a whole. This kernel is meant as a template reference for the basic code, so all examples calculate loss on the entire tensor, but it should be trivial for you to modify it for multi-class averaging.

I have provided the implementations in three popular libraries i.e. tensorflow keras and pytorch. Lets get started.

These functions cannot simply be written in NumPy, as they must operate on tensors that also have gradient parameters which need to be calculated throughout the model during backpropagation. According, loss functions must be written using backend functions from the respective model library.

With multi-class classification or segmentation, we sometimes use loss functions that calculate the average loss for each class, rather than calculating loss from the prediction tensor as a whole. This kernel is meant as a template reference for the basic code, so all examples calculate loss on the entire tensor, but it should be trivial for you to modify it for multi-class averaging.

For Learning-Rate-Schedulers-Packege-Tensorflow-PyTorch-Keras

For Evaluation-Metrics-Package-Tensorflow-PyTorch-Keras

Necessary Imports

You can import some necessary packages as follows

  1. # PyTorch
  2. import torch
  3. import torch.nn as nn
  4. import torch.nn.functional as F
  5. # Keras
  6. import keras
  7. import keras.backend as K
  8. # Tensorflow
  9. form tensorflow import keras
  10. from tensorflow import keras.backend as K

Weighted Catagorical Cross Entropy Loss

  1. # Tensorflow/Keras
  2. def weighted_categorical_crossentropy(weights):
  3.     """
  4.     A weighted version of keras.objectives.categorical_crossentropy
  6.     weights: numpy array of shape (C,) where C is the number of classes
  7.              np.array([0.5,2,10]) # Class one at 0.5, class 2 twice the normal weights, class 3 10x.
  8.     """
  9.     weights = K.variable(weights)
  11.     def loss(y_true, y_pred, from_logits=False):
  12.         if from_logits:
  13.         y_pred = tf.keras.activations.softmax(y_pred, axis=-1)
  14.             #y_pred /= K.sum(y_pred, axis=-1, keepdims=True)
  15.         # clip to prevent NaN's and Inf's
  16.         y_pred = K.clip(y_pred, K.epsilon(), 1 - K.epsilon())
  17.         # calc
  18.         loss = y_true * K.log(y_pred) * weights
  19.         loss = -K.sum(loss, -1)
  20.         return loss
  22.     return loss

Or you can check the weighted focal loss below and set the gamma=0 and alpha=1 and it’ll work same as weighted catagorical cross entropy

Dice Loss

The Dice coefficient, or Dice-Sørensen coefficient, is a common metric for pixel segmentation that can also be modified to act as a loss function:

  1. #PyTorch
  2. class DiceLoss(nn.Module):
  3.     def __init__(self, weight=None, size_average=True):
  4.         super(DiceLoss, self).__init__()
  6.     def forward(self, inputs, targets, smooth=1):
  8.         #comment out if your model contains a sigmoid or equivalent activation layer
  9.         inputs = F.sigmoid(inputs)       
  11.         #flatten label and prediction tensors
  12.         inputs = inputs.view(-1)
  13.         targets = targets.view(-1)
  15.         intersection = (inputs * targets).sum()                            
  16.         dice = (2.*intersection + smooth)/(inputs.sum() + targets.sum() + smooth)  
  18.         return 1 - dice
  1. def DiceLoss(y_true, y_pred, smooth=1e-6):
  3.     # if you are using this loss for multi-class segmentation then uncomment 
  4.     # following lines
  5.     # if y_pred.shape[-1] <= 1:
  6.     #     # activate logits
  7.     #     y_pred = tf.keras.activations.sigmoid(y_pred)
  8.     # elif y_pred.shape[-1] >= 2:
  9.     #     # activate logits
  10.     #     y_pred = tf.keras.activations.softmax(y_pred, axis=-1)
  11.     #     # convert the tensor to one-hot for multi-class segmentation
  12.     #     y_true = K.squeeze(y_true, 3)
  13.     #     y_true = tf.cast(y_true, "int32")
  14.     #     y_true = tf.one_hot(y_true, num_class, axis=-1)
  16.     # cast to float32 datatype
  17.     y_true = K.cast(y_true, 'float32')
  18.     y_pred = K.cast(y_pred, 'float32')
  20.     #flatten label and prediction tensors
  21.     inputs = K.flatten(y_pred)
  22.     targets = K.flatten(y_true)
  24.     intersection = K.sum(K.dot(targets, inputs))
  25.     dice = (2*intersection + smooth) / (K.sum(targets) + K.sum(inputs) + smooth)
  26.     return 1 - dice

BCE-Dice Loss

This loss combines Dice loss with the standard binary cross-entropy (BCE) loss that is generally the default for segmentation models. Combining the two methods allows for some diversity in the loss, while benefitting from the stability of BCE. The equation for multi-class BCE by itself will be familiar to anyone who has studied logistic regression:

  1. #PyTorch
  2. class DiceBCELoss(nn.Module):
  3.     def __init__(self, weight=None, size_average=True):
  4.         super(DiceBCELoss, self).__init__()
  6.     def forward(self, inputs, targets, smooth=1):
  8.         #comment out if your model contains a sigmoid or equivalent activation layer
  9.         inputs = F.sigmoid(inputs)       
  11.         #flatten label and prediction tensors
  12.         inputs = inputs.view(-1)
  13.         targets = targets.view(-1)
  15.         intersection = (inputs * targets).sum()                            
  16.         dice_loss = 1 - (2.*intersection + smooth)/(inputs.sum() + targets.sum() + smooth)  
  17.         BCE = F.binary_cross_entropy(inputs, targets, reduction='mean')
  18.         Dice_BCE = BCE + dice_loss
  20.         return Dice_BCE
  1. class DiceLossMulticlass(nn.Module):
  2.     def __init__(self, weights=None, size_average=False):
  3.         super(mIoULoss, self).__init__()
  5.     def forward(self, inputs, targets, smooth=1):
  6.         if self.weights is not None:
  7.             assert self.weights.shape == (targets.shape[1], )
  9.         # make a copy not to change the default weights in the instance of DiceLossMulticlass
  10.         weights = self.weights.copy()
  12.         #comment out if your model contains a sigmoid or equivalent activation layer
  13.         inputs = F.sigmoid(inputs)
  15.         # flatten label and prediction images, leave BATCH and NUM_CLASSES
  16.         # (BATCH, NUM_CLASSES, H, W) -> (BATCH, NUM_CLASSES, H * W)
  17.         inputs = inputs.view(inputs.shape[0],inputs.shape[1],-1)
  18.         targets = targets.view(targets.shape[0],targets.shape[1],-1)
  20.         #intersection = (inputs * targets).sum()
  21.         intersection = (inputs * targets).sum(0).sum(1)
  22.         #dice = (2.*intersection + smooth)/(inputs.sum() + targets.sum() + smooth)
  23.         dice = (2.*intersection + smooth)/(inputs.sum(0).sum(1) + targets.sum(0).sum(1) + smooth)
  25.         if (weights is None) and self.size_average==True:
  26.             weights = (targets == 1).sum(0).sum(1)
  27.             weights /= weights.sum() # so they sum up to 1
  29.         if weights is not None:
  30.             return 1 - (dice*weights).mean()
  31.         else:
  32.             return 1 - weights.mean()
  1. #Tensorflow / Keras
  2. def DiceBCELoss(y_true, y_pred, smooth=1e-6):    
  4.     # if you are using this loss for multi-class segmentation then uncomment 
  5.     # following lines
  6.     # if y_pred.shape[-1] <= 1:
  7.     #     # activate logits
  8.     #     y_pred = tf.keras.activations.sigmoid(y_pred)
  9.     # elif y_pred.shape[-1] >= 2:
  10.     #     # activate logits
  11.     #     y_pred = tf.keras.activations.softmax(y_pred, axis=-1)
  12.     #     # convert the tensor to one-hot for multi-class segmentation
  13.     #     y_true = K.squeeze(y_true, 3)
  14.     #     y_true = tf.cast(y_true, "int32")
  15.     #     y_true = tf.one_hot(y_true, num_class, axis=-1)
  17.     # cast to float32 datatype
  18.     y_true = K.cast(y_true, 'float32')
  19.     y_pred = K.cast(y_pred, 'float32')
  21.     #flatten label and prediction tensors
  22.     inputs = K.flatten(y_pred)
  23.     targets = K.flatten(y_true)
  25.     BCE =  binary_crossentropy(targets, inputs)
  26.     intersection = K.sum(K.dot(targets, inputs))    
  27.     dice_loss = 1 - (2*intersection + smooth) / (K.sum(targets) + K.sum(inputs) + smooth)
  28.     Dice_BCE = BCE + dice_loss
  30.     return Dice_BCE

Weighted BCE and Dice Loss

Combines BCE and Dice loss

  1. # Keras/ Tensorflow
  2. def Weighted_BCEnDice_loss(y_true, y_pred):
  4.     # if you are using this loss for multi-class segmentation then uncomment 
  5.     # following lines
  6.     # if y_pred.shape[-1] <= 1:
  7.     #     # activate logits
  8.     #     y_pred = tf.keras.activations.sigmoid(y_pred)
  9.     # elif y_pred.shape[-1] >= 2:
  10.     #     # activate logits
  11.     #     y_pred = tf.keras.activations.softmax(y_pred, axis=-1)
  12.     #     # convert the tensor to one-hot for multi-class segmentation
  13.     #     y_true = K.squeeze(y_true, 3)
  14.     #     y_true = tf.cast(y_true, "int32")
  15.     #     y_true = tf.one_hot(y_true, num_class, axis=-1)
  18.     y_true = K.cast(y_true, 'float32')
  19.     y_pred = K.cast(y_pred, 'float32')
  20.     # if we want to get same size of output, kernel size must be odd number
  21.     averaged_mask = K.pool2d(
  22.             y_true, pool_size=(11, 11), strides=(1, 1), padding='same', pool_mode='avg')
  23.     border = K.cast(K.greater(averaged_mask, 0.005), 'float32') * K.cast(K.less(averaged_mask, 0.995), 'float32')
  24.     weight = K.ones_like(averaged_mask)
  25.     w0 = K.sum(weight)
  26.     weight += border * 2
  27.     w1 = K.sum(weight)
  28.     weight *= (w0 / w1)
  29.     loss =  weighted_dice_loss(y_true, y_pred, weight) + weighted_bce_loss(y_true, y_pred, weight) 
  30.     return loss
  32. def weighted_bce_loss(y_true, y_pred, weight):
  33.     # avoiding overflow
  34.     epsilon = 1e-7
  35.     y_pred = K.clip(y_pred, epsilon, 1. - epsilon)
  36.     logit_y_pred = K.log(y_pred / (1. - y_pred))
  37.     #logit_y_pred = y_pred
  39.     loss = (1. - y_true) * logit_y_pred + (1. + (weight - 1.) * y_true) * \
  40.     (K.log(1. + K.exp(-K.abs(logit_y_pred))) + K.maximum(-logit_y_pred, 0.))
  41.     return K.sum(loss) / K.sum(weight)
  43. def weighted_dice_loss(y_true, y_pred, weight):
  44.     smooth = 1.
  45.     w, m1, m2 = weight * weight, y_true, y_pred
  46.     intersection = (m1 * m2)
  47.     score = (2. * K.sum(w * intersection) + smooth) / (K.sum(w * (m1**2)) + K.sum(w * (m2**2)) + smooth) # Uptill here is Dice Loss with squared
  48.     loss = 1. - K.sum(score)  #Soft Dice Loss
  49.     return loss

HED Loss

I was introduced in holistic edge detector to detect edges/boundaries of objects in https://arxiv.org/pdf/1504.06375.pdf.

  1. # Keras/ Tensorflow
  2. def HED_loss(y_true, y_pred):
  4.     #y_true = y_true * 255 # b/c keras generator normalizes images
  5.     if y_pred.shape[-1] <= 1:
  6.         y_true = y_true[:,:,:,0:1]
  7.     elif y_pred.shape[-1] >= 2:
  8.         y_true = K.squeeze(y_true, 3)
  9.         y_true = tf.cast(y_true, "int32")
  10.         y_true = tf.one_hot(y_true, num_class, axis=-1)
  12.     y_true = K.cast(y_true, 'float32')
  13.     y_pred = K.cast(y_pred, 'float32')
  15.     loss = sigmoid_cross_entropy_balanced(y_pred, y_true) 
  16.     return loss
  18. def sigmoid_cross_entropy_balanced(logits, label, name='cross_entropy_loss'):
  19.     """
  20.     From:
  22.     https://github.com/moabitcoin/holy-edge/blob/master/hed/losses.py
  24.     Implements Equation [2] in https://arxiv.org/pdf/1504.06375.pdf
  25.     Compute edge pixels for each training sample and set as pos_weights to
  26.     tf.nn.weighted_cross_entropy_with_logits
  27.     """
  28.     y = tf.cast(label, tf.float32)
  30.     count_neg = tf.reduce_sum(1. - y)
  31.     count_pos = tf.reduce_sum(y)
  33.     # Equation [2]
  34.     beta = count_neg / (count_neg + count_pos)
  36.     # Equation [2] divide by 1 - beta
  37.     pos_weight = beta / (1 - beta)
  38.     if int(str(tf.__version__)[0]) == 1:
  39.         cost = tf.nn.weighted_cross_entropy_with_logits(logits=logits, targets=y, pos_weight=pos_weight)
  40.     if int(str(tf.__version__)[0]) == 2:
  41.         cost = tf.nn.weighted_cross_entropy_with_logits(logits=logits, labels=y, pos_weight=pos_weight)
  43.     # Multiply by 1 - beta
  44.     cost = tf.reduce_mean(cost * (1 - beta))
  46.     # check if image has no edge pixels return 0 else return complete error function
  47.     return tf.where(tf.equal(count_pos, 0.0), 0.0, cost, name=name)

Jaccard/Intersection over Union (IoU) Loss

The IoU metric, or Jaccard Index, is similar to the Dice metric and is calculated as the ratio between the overlap of the positive instances between two sets, and their mutual combined values:

Like the Dice metric, it is a common means of evaluating the performance of pixel segmentation models.

  1. #PyTorch
  2. class IoULoss(nn.Module):
  3.     def __init__(self, weight=None, size_average=True):
  4.         super(IoULoss, self).__init__()
  6.     def forward(self, inputs, targets, smooth=1):
  8.         #comment out if your model contains a sigmoid or equivalent activation layer
  9.         inputs = F.sigmoid(inputs)       
  11.         #flatten label and prediction tensors
  12.         inputs = inputs.view(-1)
  13.         targets = targets.view(-1)
  15.         #intersection is equivalent to True Positive count
  16.         #union is the mutually inclusive area of all labels & predictions 
  17.         intersection = (inputs * targets).sum()
  18.         total = (inputs + targets).sum()
  19.         union = total - intersection 
  21.         IoU = (intersection + smooth)/(union + smooth)
  23.         return 1 - IoU
  1. #Tensorflow / Keras 
  2. def IoULoss(y_true, y_pred, smooth=1e-6):
  4.     # if you are using this loss for multi-class segmentation then uncomment 
  5.     # following lines
  6.     # if y_pred.shape[-1] <= 1:
  7.     #     # activate logits
  8.     #     y_pred = tf.keras.activations.sigmoid(y_pred)
  9.     # elif y_pred.shape[-1] >= 2:
  10.     #     # activate logits
  11.     #     y_pred = tf.keras.activations.softmax(y_pred, axis=-1)
  12.     #     # convert the tensor to one-hot for multi-class segmentation
  13.     #     y_true = K.squeeze(y_true, 3)
  14.     #     y_true = tf.cast(y_true, "int32")
  15.     #     y_true = tf.one_hot(y_true, num_class, axis=-1)
  17.     # cast to float32 datatype
  18.     y_true = K.cast(y_true, 'float32')
  19.     y_pred = K.cast(y_pred, 'float32')
  21.     #flatten label and prediction tensors
  22.     inputs = K.flatten(y_pred)
  23.     targets = K.flatten(y_true)
  25.     intersection = K.sum(K.dot(targets, inputs))
  26.     total = K.sum(targets) + K.sum(inputs)
  27.     union = total - intersection
  29.     IoU = (intersection + smooth) / (union + smooth)
  30.     return 1 - IoU

Focal Loss

Focal Loss was introduced by Lin et al of Facebook AI Research in 2017 as a means of combatting extremely imbalanced datasets where positive cases were relatively rare. Their paper “Focal Loss for Dense Object Detection” is retrievable here: https://arxiv.org/abs/1708.02002. In practice, the researchers used an alpha-modified version of the function so I have included it in this implementation.

  1. #PyTorch
  2. ALPHA = 0.8
  3. GAMMA = 2
  5. class FocalLoss(nn.Module):
  6.     def __init__(self, weight=None, size_average=True):
  7.         super(FocalLoss, self).__init__()
  9.     def forward(self, inputs, targets, alpha=ALPHA, gamma=GAMMA, smooth=1):
  11.         #comment out if your model contains a sigmoid or equivalent activation layer
  12.         inputs = F.sigmoid(inputs)       
  14.         #flatten label and prediction tensors
  15.         inputs = inputs.view(-1)
  16.         targets = targets.view(-1)
  18.         #first compute binary cross-entropy 
  19.         BCE = F.binary_cross_entropy(inputs, targets, reduction='mean')
  20.         BCE_EXP = torch.exp(-BCE)
  21.         focal_loss = alpha * (1-BCE_EXP)**gamma * BCE
  23.         return focal_loss
  1. #Tensorflow / Keras
  3. def FocalLoss(y_true, y_pred):   
  5.     alpha = 0.8
  6.     gamma = 2
  7.     # if you are using this loss for multi-class segmentation then uncomment 
  8.     # following lines
  9.     # if y_pred.shape[-1] <= 1:
  10.     #     # activate logits
  11.     #     y_pred = tf.keras.activations.sigmoid(y_pred)
  12.     # elif y_pred.shape[-1] >= 2:
  13.     #     # activate logits
  14.     #     y_pred = tf.keras.activations.softmax(y_pred, axis=-1)
  15.     #     # convert the tensor to one-hot for multi-class segmentation
  16.     #     y_true = K.squeeze(y_true, 3)
  17.     #     y_true = tf.cast(y_true, "int32")
  18.     #     y_true = tf.one_hot(y_true, num_class, axis=-1)
  20.     # cast to float32 datatype
  21.     y_true = K.cast(y_true, 'float32')
  22.     y_pred = K.cast(y_pred, 'float32')
  24.     inputs = K.flatten(inputs)
  25.     targets = K.flatten(targets)
  27.     BCE = K.binary_crossentropy(targets, inputs)
  28.     BCE_EXP = K.exp(-BCE)
  29.     focal_loss = K.mean(alpha * K.pow((1-BCE_EXP), gamma) * BCE)
  31.     return focal_loss

Weighted Focal Loss

in developement

  1. # TensorFlow/Keras
  2. class WFL():
  3.     '''
  4.     Weighted Focal loss
  5.     '''
  6.     def __init__(self, alpha=0.25, gamma=2, class_weights=None, from_logits=False):
  7.         self.class_weights = class_weights
  8.         self.from_logits = from_logits
  9.         self.alpha = alpha
  10.         self.gamma = gamma
  12.     def __call__(self, y_true, y_pred):
  14.         if self.from_logits:
  15.             y_pred = tf.keras.activations.softmax(y_pred, axis=-1)
  17.         y_pred = K.clip(y_pred, K.epsilon(), 1. - K.epsilon())
  19.         # cast to float32 datatype
  20.         y_true = K.cast(y_true, 'float32')
  21.         y_pred = K.cast(y_pred, 'float32')
  23.         WCCE = y_true * K.log(y_pred) * self.class_weights
  24.         WFL = (self.alpha * K.pow((1-y_pred), self.gamma)) * WCCE
  25.         # reduce sum -> reduces the loss over number of batches by simply taking sum over all samples
  26.         # reduce mean -> reduces the loss ove number of batches by taking mean of all samples
  27.         # if axis=-1 is given input batch is like B * C then loss will have shape B * 1
  28.         # if axis is None then only 1 scaler value is output
  30.         return -tf.math.reduce_sum(WFL, -1) #use this for custom training loop and dviding by global batch size. * (1/GB)
  31.         #return -tf.reduce_mean(WFL, -1) # use this for complie fit keras API

Tversky Loss

This loss was introduced in “Tversky loss function for image segmentationusing 3D fully convolutional deep networks”, retrievable here: https://arxiv.org/abs/1706.05721. It was designed to optimise segmentation on imbalanced medical datasets by utilising constants that can adjust how harshly different types of error are penalised in the loss function. From the paper:
… in the case of α=β=0.5 the Tversky index simplifies to be the same as the Dice coefficient, which is also equal to the F1 score. With α=β=1, Equation 2 produces Tanimoto coefficient, and setting α+β=1 produces the set of Fβ scores. Larger βs weigh recall higher than precision (by placing more emphasis on false negatives).
To summarise, this loss function is weighted by the constants ‘alpha’ and ‘beta’ that penalise false positives and false negatives respectively to a higher degree in the loss function as their value is increased. The beta constant in particular has applications in situations where models can obtain misleadingly positive performance via highly conservative prediction. You may want to experiment with different values to find the optimum. With alpha==beta==0.5, this loss becomes equivalent to Dice Loss.

  1. #PyTorch
  2. ALPHA = 0.5
  3. BETA = 0.5
  5. class TverskyLoss(nn.Module):
  6.     def __init__(self, weight=None, size_average=True):
  7.         super(TverskyLoss, self).__init__()
  9.     def forward(self, inputs, targets, smooth=1, alpha=ALPHA, beta=BETA):
  11.         #comment out if your model contains a sigmoid or equivalent activation layer
  12.         inputs = F.sigmoid(inputs)       
  14.         #flatten label and prediction tensors
  15.         inputs = inputs.view(-1)
  16.         targets = targets.view(-1)
  18.         #True Positives, False Positives & False Negatives
  19.         TP = (inputs * targets).sum()    
  20.         FP = ((1-targets) * inputs).sum()
  21.         FN = (targets * (1-inputs)).sum()
  23.         Tversky = (TP + smooth) / (TP + alpha*FP + beta*FN + smooth)  
  25.         return 1 - Tversky
  1. #Tensorflow / Keras
  2. def TverskyLoss(y_true, y_pred, smooth=1e-6):
  4.         if y_pred.shape[-1] <= 1:
  5.             alpha = 0.3
  6.             beta = 0.7
  7.             gamma = 4/3 #5.
  8.             y_pred = tf.keras.activations.sigmoid(y_pred)
  9.             #y_true = y_true[:,:,:,0:1]
  10.         elif y_pred.shape[-1] >= 2:
  11.             alpha = 0.3
  12.             beta = 0.7
  13.             gamma = 4/3 #3.
  14.             y_pred = tf.keras.activations.softmax(y_pred, axis=-1)
  15.             y_true = K.squeeze(y_true, 3)
  16.             y_true = tf.cast(y_true, "int32")
  17.             y_true = tf.one_hot(y_true, num_class, axis=-1)
  19.         y_true = K.cast(y_true, 'float32')
  20.         y_pred = K.cast(y_pred, 'float32')
  21.         #flatten label and prediction tensors
  22.         inputs = K.flatten(y_pred)
  23.         targets = K.flatten(y_true)
  25.         #True Positives, False Positives & False Negatives
  26.         TP = K.sum((inputs * targets))
  27.         FP = K.sum(((1-targets) * inputs))
  28.         FN = K.sum((targets * (1-inputs)))
  30.         Tversky = (TP + smooth) / (TP + alpha*FP + beta*FN + smooth)  
  32.         return 1 - Tversky

Focal Tversky Loss

A variant on the Tversky loss that also includes the gamma modifier from Focal Loss.

  1. #PyTorch
  2. ALPHA = 0.5
  3. BETA = 0.5
  4. GAMMA = 1
  6. class FocalTverskyLoss(nn.Module):
  7.     def __init__(self, weight=None, size_average=True):
  8.         super(FocalTverskyLoss, self).__init__()
  10.     def forward(self, inputs, targets, smooth=1, alpha=ALPHA, beta=BETA, gamma=GAMMA):
  12.         #comment out if your model contains a sigmoid or equivalent activation layer
  13.         inputs = F.sigmoid(inputs)       
  15.         #flatten label and prediction tensors
  16.         inputs = inputs.view(-1)
  17.         targets = targets.view(-1)
  19.         #True Positives, False Positives & False Negatives
  20.         TP = (inputs * targets).sum()    
  21.         FP = ((1-targets) * inputs).sum()
  22.         FN = (targets * (1-inputs)).sum()
  24.         Tversky = (TP + smooth) / (TP + alpha*FP + beta*FN + smooth)  
  25.         FocalTversky = (1 - Tversky)**gamma
  27.         return FocalTversky
  1. #Tensorflow / Keras
  2. def FocalTverskyLoss(y_true, y_pred, smooth=1e-6):
  5.         if y_pred.shape[-1] <= 1:
  6.             alpha = 0.3
  7.             beta = 0.7
  8.             gamma = 4/3 #5.
  9.             y_pred = tf.keras.activations.sigmoid(y_pred)
  10.             #y_true = y_true[:,:,:,0:1]
  11.         elif y_pred.shape[-1] >= 2:
  12.             alpha = 0.3
  13.             beta = 0.7
  14.             gamma = 4/3 #3.
  15.             y_pred = tf.keras.activations.softmax(y_pred, axis=-1)
  16.             y_true = K.squeeze(y_true, 3)
  17.             y_true = tf.cast(y_true, "int32")
  18.             y_true = tf.one_hot(y_true, num_class, axis=-1)
  21.         y_true = K.cast(y_true, 'float32')
  22.         y_pred = K.cast(y_pred, 'float32')
  23.         #flatten label and prediction tensors
  24.         inputs = K.flatten(y_pred)
  25.         targets = K.flatten(y_true)
  27.         #True Positives, False Positives & False Negatives
  28.         TP = K.sum((inputs * targets))
  29.         FP = K.sum(((1-targets) * inputs))
  30.         FN = K.sum((targets * (1-inputs)))
  32.         Tversky = (TP + smooth) / (TP + alpha*FP + beta*FN + smooth)  
  33.         FocalTversky = K.pow((1 - Tversky), gamma)
  35.         return FocalTversky

Lovasz Hinge Loss

This complex loss function was introduced by Berman, Triki and Blaschko in their paper “The Lovasz-Softmax loss: A tractable surrogate for the optimization of the intersection-over-union measure in neural networks”, retrievable here: https://arxiv.org/abs/1705.08790. It is designed to optimise the Intersection over Union score for semantic segmentation, particularly for multi-class instances. Specifically, it sorts predictions by their error before calculating cumulatively how each error affects the IoU score. This gradient vector is then multiplied with the initial error vector to penalise most strongly the predictions that decreased the IoU score the most. This procedure is detailed by jeandebleu in his excellent summary here.

This code is taken directly from the author’s github repo here: https://github.com/bermanmaxim/LovaszSoftmax and all credit is to them.

In this kernel I have implemented the flat variant that uses reshaped rank-1 tensors as inputs for PyTorch. You can modify it accordingly with the dimensions and class number of your data as needed. This code takes raw logits so ensure your model does not contain an activation layer prior to the loss calculation.

I have hidden the researchers’ own code below for brevity; simply load it into your kernel for the losses to function. In the case of their tensorflow implementation, I am still working to make it compatible with Keras. There are differences between the Tensorflow and Keras function libraries that complicate this.

  1. #PyTorch
  2. class LovaszSoftmax(nn.Module):
  3.     def __init__(self, classes='present', per_image=False, ignore=None):
  4.         super(LovaszSoftmax, self).__init__()
  5.         self.classes = classes
  6.         self.per_image = per_image
  7.         self.ignore = ignore
  9.     def forward(self, inputs, targets):
  10.         probas = F.softmax(inputs, dim=1) # B*C*H*W -> from logits to probabilities
  11.         return lovasz_softmax(probas, targets, self.classes, self.per_image, self.ignore)
  13. def lovasz_softmax(probas, labels, classes='present', per_image=False, ignore=None):
  14.     """
  15.     Multi-class Lovasz-Softmax loss
  16.       probas: [B, C, H, W] Variable, class probabilities at each prediction (between 0 and 1).
  17.               Interpreted as binary (sigmoid) output with outputs of size [B, H, W].
  18.       labels: [B, H, W] Tensor, ground truth labels (between 0 and C - 1)
  19.       classes: 'all' for all, 'present' for classes present in labels, or a list of classes to average.
  20.       per_image: compute the loss per image instead of per batch
  21.       ignore: void class labels
  22.     """
  23.     if per_image:
  24.         loss = mean(lovasz_softmax_flat(*flatten_probas(prob.unsqueeze(0), lab.unsqueeze(0), ignore), classes=classes)
  25.                           for prob, lab in zip(probas, labels))
  26.     else:
  27.         loss = lovasz_softmax_flat(*flatten_probas(probas, labels, ignore), classes=classes)
  28.     return loss
  31. def lovasz_softmax_flat(probas, labels, classes='present'):
  32.     """
  33.     Multi-class Lovasz-Softmax loss
  34.       probas: [P, C] Variable, class probabilities at each prediction (between 0 and 1)
  35.       labels: [P] Tensor, ground truth labels (between 0 and C - 1)
  36.       classes: 'all' for all, 'present' for classes present in labels, or a list of classes to average.
  37.     """
  38.     if probas.numel() == 0:
  39.         # only void pixels, the gradients should be 0
  40.         return probas * 0.
  41.     C = probas.size(1)
  42.     losses = []
  43.     class_to_sum = list(range(C)) if classes in ['all', 'present'] else classes
  44.     for c in class_to_sum:
  45.         fg = (labels == c).float() # foreground for class c
  46.         if (classes is 'present' and fg.sum() == 0):
  47.             continue
  48.         if C == 1:
  49.             if len(classes) > 1:
  50.                 raise ValueError('Sigmoid output possible only with 1 class')
  51.             class_pred = probas[:, 0]
  52.         else:
  53.             class_pred = probas[:, c]
  54.         errors = (Variable(fg) - class_pred).abs()
  55.         errors_sorted, perm = torch.sort(errors, 0, descending=True)
  56.         perm = perm.data
  57.         fg_sorted = fg[perm]
  58.         losses.append(torch.dot(errors_sorted, Variable(lovasz_grad(fg_sorted))))
  59.     return mean(losses)
  62. def flatten_probas(probas, labels, ignore=None):
  63.     """
  64.     Flattens predictions in the batch
  65.     """
  66.     if probas.dim() == 3:
  67.         # assumes output of a sigmoid layer
  68.         B, H, W = probas.size()
  69.         probas = probas.view(B, 1, H, W)
  70.     B, C, H, W = probas.size()
  71.     probas = probas.permute(0, 2, 3, 1).contiguous().view(-1, C)  # B * H * W, C = P, C
  72.     labels = labels.view(-1)
  73.     if ignore is None:
  74.         return probas, labels
  75.     valid = (labels != ignore)
  76.     vprobas = probas[valid.nonzero().squeeze()]
  77.     vlabels = labels[valid]
  78.     return vprobas, vlabels
  80. def xloss(logits, labels, ignore=None):
  81.     """
  82.     Cross entropy loss
  83.     """
  84.     return F.cross_entropy(logits, Variable(labels), ignore_index=255)
  86. # --------------------------- HELPER FUNCTIONS ---------------------------
  87. def isnan(x):
  88.     return x != x
  91. def mean(l, ignore_nan=False, empty=0):
  92.     """
  93.     nanmean compatible with generators.
  94.     """
  95.     l = iter(l)
  96.     if ignore_nan:
  97.         l = ifilterfalse(isnan, l)
  98.     try:
  99.         n = 1
  100.         acc = next(l)
  101.     except StopIteration:
  102.         if empty == 'raise':
  103.             raise ValueError('Empty mean')
  104.         return empty
  105.     for n, v in enumerate(l, 2):
  106.         acc += v
  107.     if n == 1:
  108.         return acc
  109.     return acc / n
  112. def lovasz_grad(gt_sorted):
  113.     """
  114.     Computes gradient of the Lovasz extension w.r.t sorted errors
  115.     See Alg. 1 in paper
  116.     """
  117.     p = len(gt_sorted)
  118.     gts = gt_sorted.sum()
  119.     intersection = gts - gt_sorted.float().cumsum(0)
  120.     union = gts + (1 - gt_sorted).float().cumsum(0)
  121.     jaccard = 1. - intersection / union
  122.     if p > 1: # cover 1-pixel case
  123.         jaccard[1:p] = jaccard[1:p] - jaccard[0:-1]
  124.     return jaccard
  1. #Keras
  2. # not working yet
  3. # def LovaszHingeLoss(inputs, targets):
  4. #     return lovasz_hinge_loss(inputs, targets)

Combo Loss

This loss was introduced by Taghanaki et al in their paper “Combo loss: Handling input and output imbalance in multi-organ segmentation”, retrievable here: https://arxiv.org/abs/1805.02798. Combo loss is a combination of Dice Loss and a modified Cross-Entropy function that, like Tversky loss, has additional constants which penalise either false positives or false negatives more respectively.

  1. #PyTorch
  2. ALPHA = 0.5 # < 0.5 penalises FP more, > 0.5 penalises FN more
  3. CE_RATIO = 0.5 #weighted contribution of modified CE loss compared to Dice loss
  5. class ComboLoss(nn.Module):
  6.     def __init__(self, weight=None, size_average=True):
  7.         super(ComboLoss, self).__init__()
  9.     def forward(self, inputs, targets, smooth=1, alpha=ALPHA, beta=BETA):
  11.         #flatten label and prediction tensors
  12.         inputs = inputs.view(-1)
  13.         targets = targets.view(-1)
  15.         #True Positives, False Positives & False Negatives
  16.         intersection = (inputs * targets).sum()    
  17.         dice = (2. * intersection + smooth) / (inputs.sum() + targets.sum() + smooth)
  19.         inputs = torch.clamp(inputs, e, 1.0 - e)       
  20.         out = - (ALPHA * ((targets * torch.log(inputs)) + ((1 - ALPHA) * (1.0 - targets) * torch.log(1.0 - inputs))))
  21.         weighted_ce = out.mean(-1)
  22.         combo = (CE_RATIO * weighted_ce) - ((1 - CE_RATIO) * dice)
  24.         return combo
  1. #Tensorflow / Keras
  2. def Combo_loss(y_true, y_pred, smooth=1):
  4.   e = K.epsilon()
  5.   if y_pred.shape[-1] <= 1:
  6.     ALPHA = 0.8    # < 0.5 penalises FP more, > 0.5 penalises FN more
  7.     CE_RATIO = 0.5 # weighted contribution of modified CE loss compared to Dice loss
  8.     y_pred = tf.keras.activations.sigmoid(y_pred)
  9.   elif y_pred.shape[-1] >= 2:
  10.     ALPHA = 0.3    # < 0.5 penalises FP more, > 0.5 penalises FN more
  11.     CE_RATIO = 0.7 # weighted contribution of modified CE loss compared to Dice loss
  12.     y_pred = tf.keras.activations.softmax(y_pred, axis=-1)
  13.     y_true = K.squeeze(y_true, 3)
  14.     y_true = tf.cast(y_true, "int32")
  15.     y_true = tf.one_hot(y_true, num_class, axis=-1)
  17.   # cast to float32 datatype
  18.   y_true = K.cast(y_true, 'float32')
  19.   y_pred = K.cast(y_pred, 'float32')
  21.   targets = K.flatten(y_true)
  22.   inputs = K.flatten(y_pred)
  24.   intersection = K.sum(targets * inputs)
  25.   dice = (2. * intersection + smooth) / (K.sum(targets) + K.sum(inputs) + smooth)
  26.   inputs = K.clip(inputs, e, 1.0 - e)
  27.   out = - (ALPHA * ((targets * K.log(inputs)) + ((1 - ALPHA) * (1.0 - targets) * K.log(1.0 - inputs))))
  28.   weighted_ce = K.mean(out, axis=-1)
  29.   combo = (CE_RATIO * weighted_ce) - ((1 - CE_RATIO) * dice)
  31.   return combo

Usage

Some tips

  • Tversky and Focal-Tversky loss benefit from very low learning rates, of the order 5e-5 to 1e-4. They would not see much improvement in my kernels until around 7-10 epochs, upon which performance would improve significantly.

  • In general, if a loss function does not appear to be working well (or at all), experiment with modifying the learning rate before moving on to other options.

  • You can easily create your own loss functions by combining any of the above with Binary Cross-Entropy or any combination of other losses. Bear in mind that loss is calculated for every batch, so more complex losses will increase runtime.

  • Care must be taken when writing loss functions for PyTorch. If you call a function to modify the inputs that doesn’t entirely use PyTorch’s numerical methods, the tensor will ‘detach’ from the the graph that maps it back through the neural network for the purposes of backpropagation, making the loss function unusable. Discussion of this is available here.

Refernces

RNA Kaggle