The softmax activation function is the generalized form of the sigmoid function for multiple dimensions. It is the mathematical function that converts the vector of numbers into the vector of the probabilities. The softmax activation function is commonly used as an activation function in the case of multi-class classification problems in machine learning. The output of the softmax is interpreted as the probability of getting each class.
The mathematical expression of softmax activation function is,
Let’s learn about the working of the softmax function with an example. Let’s consider a neural network that classifies a given image, whether it is of cat, dog, tiger, or none. Let X is the feature vector (i.e. X = [x1, x2, x3, x4]).
We normally use a softmax activation function in the last layer of a neural network as shown in the figure above.
In the neural network shown above, we have
, calculated values at layer (L-1),
is the weight matrix.
m = total nodes in layer L-1 and n = nodes in output layer L.
For this example, m = 3, n = 4.
is the bias matrix, n = 4 in this example.
Now we calculate the exponential values of elements of matrix Z [L].
The probabilities of being indifferent classes given input X are calculated as follows.
On the basis of this probability distribution, our neural network classifies whether the given image is of cat, dog, tiger, or none.
Now let’s make it clearer by taking some numeric values.
Suppose we input an image and obtained these values
Therefore the probability distribution is calculated as,
By observing the probability distribution, we can say that the supplied image is of a dog.
The advantages of using the softmax activation function are: