Softmax Activation Function in Neural Network [formula included]

by keshav

---

---

The softmax activation function is the generalized form of the sigmoid function for multiple dimensions. It is the mathematical function that converts the vector of numbers into the vector of the probabilities. The softmax activation function is commonly used as an activation function in the case of multi-class classification problems in machine learning. The output of the softmax is interpreted as the probability of getting each class.

 

The mathematical expression of softmax activation function is,

Let’s learn about the working of the softmax function with an example. Let’s consider a neural network that classifies a given image, whether it is of cat, dog, tiger, or none. Let X is the feature vector (i.e. X = [x1, x2, x3, x4]).

Also Read:

• Numpy Tutorials [beginners to Intermediate]
• Sigmoid(Logistic) Activation Function ( with python code)
• Hyperbolic Tangent (tanh) Activation Function [with python code]
• ReLU Activation Function [with python code]
• Leaky ReLU Activation Function [with python code]

 

We normally use a softmax activation function in the last layer of a neural network as shown in the figure above.

 

In the neural network shown above, we have

 

Where,

 

   

, calculated values at layer (L-1),

 

 

is the weight matrix. 

 

m = total nodes in layer L-1 and n = nodes in output layer L.

For this example, m = 3, n = 4.

 

And,

 

 is the bias matrix, n = 4 in this example.

 

Now we calculate the exponential values of elements of matrix Z ^[L].

 

 

And,

The probabilities of being indifferent classes given input X are calculated as follows.

 

On the basis of this probability distribution, our neural network classifies whether the given image is of cat, dog, tiger, or none.

 

Now let’s make it clearer by taking some numeric values.

 

Suppose we input an image and obtained these values

 

 

 

 

Then,

 

 

 

 

 

 

 

 

And,

 

 

 

 

Therefore the probability distribution is calculated as,

 

 

 

 

 

 

 

 

 

 

By observing the probability distribution, we can say that the supplied image is of a dog.

The advantages of using the softmax activation function are:

• It can be used for multiclass classification
• It normalizes the outputs for each class between 0 and 1, and divides by their sum, giving the probability of the input value being in a specific class.
• For neural networks that need to categorize inputs into numerous categories, Softmax is often employed exclusively for the output layer.

---