What is the derivative of the ReLU activation function? [Including Python function]

by keshav


der_ReLU.PNG

In this article, we will see what is the formula to calculate the derivative of the ReLU activation Function. The python code to calculate the derivative of the ReLU function is also included.

The rectified linear activation function (RELU) is a piecewise linear function that, if the input is positive say x, the output will be x. otherwise, it outputs zero.

 

The mathematical representation of the ReLU function is,

ReLU

    The coding logic for the ReLU function is simple,

    if input_value > 0:
      return input_value
    else:
      return 0

    A simple python function to mimic a ReLU function is as follows,

    def ReLU(x):
      data = [max(0,value) for value in x]
      return np.array(data, dtype=float)

     

    The derivative of ReLU is,

    derivative ReLU

    A simple python function to mimic the derivative of the ReLU function is as follows,

    def der_ReLU(x):
      data = [1 if value>0 else 0 for value in x]
      return np.array(data, dtype=float)

     

    ReLU is used widely nowadays, but it has some problems. let's say if we have input less than 0, then it outputs zero, and the neural network can't continue the backpropagation algorithm. This problem is commonly known as Dying ReLU. To get rid of this problem we use an improvised version of ReLU, called Leaky ReLU

    GRAPH:

    ReLU activation Function and Derivative


    No Comments


    Post a Comment