Neural Network – Activation Function

In this post we look deep into activation function.

If you remember from my previous blog post. Activation function receives a weighted sum of weights from the nodes and it decides if the node will get activated or not.

To add activation to our NN its very simple

from keras.layers import Activation

model.add(Activation('relu'))

model.summary()

It is very simple to implement activation function using Keras as we see above, but its important that we understand in detail different kind of activation layers and their pro/cons

https://colab.research.google.com/drive/1IOjSB1PF85t9qEhCWgDbWxuCRSDz71H6

Activation Functions

This is a blog which post which explores in detail pro/cons of different activation functions

https://medium.com/the-theory-of-everything/understanding-activation-functions-in-neural-networks-9491262884e0

It’s highly recommended to read this.

In summary Relu is a very simple and efficient activation function which solves the “varnishing gradient problem” effectively. You can use this as a default activation function in most cases and also experiment with sigmoid/tahn in case of logical regression problems.

Further most read links to understand the above

https://deeplizard.com/learn/video/m0pIlLfpXWE

https://machinelearningmastery.com/rectified-linear-activation-function-for-deep-learning-neural-networks/

https://machinelearningmastery.com/how-to-fix-vanishing-gradients-using-the-rectified-linear-activation-function/

All these blogs are quite technical and explain activation functions in depth.

Keras Activation Functions

Keras supports many different activation functions out of the box which you can see here https://keras.io/activations/