Leaky Relu Derivative Python Implementation with Explanation

Leaky Relu Derivative Python Implementation with Explanation

GET FREE AMZAON AUDIOBOOKS

Leaky Relu solves the problem of dead neurons. Because it is not zero even in the case of negative values. Let’s see leaky Relu derivative python.

Leaky Relu & derivative –

Let’s see the mathematical expression for Leaky Relu.
x>0 then f(x)=x
x<0 then f(x)=x*constant

Here we have seen that the value of Leky Relu function is always the same as the variable if the value is greater than zero. It will be a constant in the place of zero. Now let’s see the mathematical expression of the derivative of Leky Relu.

x>0 then f(x)=1
x<0 then f(x)=constant

This constant is the only difference between Leaky Relu and Relu. In the below section, we will see its importance.

Leaky Relu derivative python Implementation –

In the above section, We have seen the mathematical expression. Now let’s see leaky Relu derivative python Implementation

def leaky_Relu(x):
  return x*0.01 if x < 0 else x

def  leaky_Relu_Derivative(x):
  return 0.01 if x < 0 else 1

Graphical Representation of Leaky Relu-

In the earlier section, We have seen the mathematical expression and python code for Leaky Relu. Now let’s see the graph of Leaky Relu.

leaky relu
leaky relu[1]

Graphical Representation of Leaky Relu Derivative-

Let’s see the Graphical Representation of Leaky Relu Derivative. Here we need to be careful that it looks approximate to zero in negative values. But actually, it is not zero.

leaky relu derivative python graph
leaky relu derivative python graph [2]

Conclusion-

Leaky Relu is a Revolution in Neural Network. It solves the problem of Vanishing Gradient Descent in RNNs. That is a clear reason for rising in the Deep Learning journey. Actually, Sigmoid Function’s derivative has a range between (0,0.25) which tends to zero because of the chain rule. This causes the deactivation of neurons.

I hope now must have understood an intuition of Leaky Relu. How its existence helps neural networks. Still, If you have any doubt on Leaky Relu, Please comment below in the comment box. We also appreciate you for reading the whole article till the end.

Thanks 

Data Science Learner Team

Reference

1 .https://paperswithcode.com/method/leaky-relu

2. https://www.analyticsvidhya.com/blog/2020/01/fundamentals-deep-learning-activation-functions-when-to-use-them/

Join our list

Subscribe to our mailing list and get interesting stuff and updates to your email inbox.

Thank you for signup. A Confirmation Email has been sent to your Email Address.

Something went wrong.

 
Thank you For sharing.We appreciate your support. Don't Forget to LIKE and FOLLOW our SITE to keep UPDATED with Data Science Learner