Supervised Learning - Classification Week 9 Challenge

1. What is a Neural Network?

Neural Network reflects the behaviour of the Human Brain, allowing computer to recognize patterns and solve common problems in the fields of AI, Machine Learning, and Deep Learning.

Neural Networks, also known as Artificial Neural Networks (ANN) or simulated neual Networks (SNN), are a subset of machine learning and are at the heart of deep learning algorithms. Their name and structure are inspired by the human brain, mimicking the way that biological neurons signal to one another.

Artificial Neural Networks (ANNs) are comprised of a node layers, containing an

1.Input layer,

2.One or more hidden layers

3.Output layer.

Each node, or artificial neuron, connects to another and has an associated weight and threshold. If the output of any individual node is above the specified threshold value, that node is activated, sending data to the next layer of the network. Else, no data will be passed to the next layer of the network.

Neural networks rely on training data to learn and improve their accuracy over time. However, once these learning algorithms are fine-tuned for accuracy, they are powerful tools in computer science and AI, allowing us to classify and cluster data at a high velocity. Tasks in speech recognition or image recognition can take minutes versus hours when compared to the manual identification by human experts. One of the most well-known neural networks is Google's search algorithm.

2. What is Deep Learning?

Deep Learning is a subset of Machine Learning, which is essentially a neural network with three or more layers. These neural networks attempt to simulate the behaviour of the human brain - albeit far from matching its ability - allowing it to "learn" from large amounts of data. While a neural network with a singlelayer can still make approx predictions, additional hiddem layers can help to optimize and refine for accuracy.

Applications:

1. Deep Learning drive many AI applications and services that improve automaton, performing analytical and physical tasks without human intervention.

2. Deep Learning technology lies behind everyday products and services and some of the examples are:

1. Digital assistants

2. Voice-enabled TV remotes

3. Credit card fraud detection

4. Anamoly detection

5. Self driving cars

3. What is the role of activation function?

Activation function decides, whether a neuron should be activated or not by calculating weighted sum and further adding bias with it.

The main purpose of the activation function is to introduce non-linearity into the output of the neuron.

A neural Network without activation function is essentially just a linear regression model. The activation function does the non-linear transformation to the input making it capable to learn and perform more complex tasks.

4. What are the different types of activation function?

1) Linear Function

Equation : y = ax - Linear function has the equation similar to as of a straight line

No matter how many layers we have, if all are linear in nature, the final activation function of last layer is nothing but just a linear function of the output of first layer.

Range: -inf to +inf

Advantage - Linear activation function is used at just one plae i.e output layer

disadvantage - If we differentiate the linear function to bring non-linearity, result will no more depend on input "x" and function will become constant, it won't introduce any ground-breaking behaviour to our algorithm

2) Sigmoid Function

• It is a function which is plotted as ‘S’ shaped graph.
• Equation :
  A = 1/(1 + e^-x)
• Nature : Non-linear. Notice that X values lies between -2 to 2, Y values are very steep. This means, small changes in x would also bring about large changes in the value of Y.
• Value Range : 0 to 1
• Uses : Usually used in output layer of a binary classification, where result is either 0 or 1, as value for sigmoid function lies between 0 and 1 only so, result can be predicted easily to be 1 if value is greater than 0.5 and 0 otherwise.

3) tanh Function

The activation that works almost always better than sigmoid function is Tanh function also knows as Tangent Hyperbolic function. It’s actually mathematically shifted version of the sigmoid function. Both are similar and can be derived from each other.

• Equation :-

f(x) = tanh(x) = 2/(1 + e-2x) - 1
OR
tanh(x) = 2 * sigmoid(2x) - 1 

• Value Range :- -1 to +1
• Nature :- non-linear
• Uses :- Usually used in hidden layers of a neural network as it’s values lies between -1 to 1 hence the mean for the hidden layer comes out be 0 or very close to it, hence helps in centering the data by bringing mean close to 0. This makes learning for the next layer much easier.

4) RELU - Rectified Linear Unit

Stands for Rectified linear unit. It is the most widely used activation function. Chiefly implemented in hidden layers of Neural network.

• Equation :- A(x) = max(0,x). It gives an output x if x is positive and 0 otherwise.
• Value Range :- [0, inf)
• Nature :- non-linear, which means we can easily backpropagate the errors and have multiple layers of neurons being activated by the ReLU function.
• Uses :- ReLu is less computationally expensive than tanh and sigmoid because it involves simpler mathematical operations. At a time only a few neurons are activated making the network sparse making it efficient and easy for computation.

In simple words, RELU learns much faster than sigmoid and Tanh function.

5) Softmax Function

The softmax function is also a type of sigmoid function but is handy when we are trying to handle classification problems.

• Nature :- non-linear
• Uses :- Usually used when trying to handle multiple classes. The softmax function would squeeze the outputs for each class between 0 and 1 and would also divide by the sum of the outputs.
• Output:- The softmax function is ideally used in the output layer of the classifier where we are actually trying to attain the probabilities to define the class of each input.

Reference link - Link

5. What do you undertand by Back propogation?

In the backward propogation, based on the error on the output layer, the weights and bias of the neuron are updated and based on each step, the weights and bias will be updated until the error becomes minimum or zero. This process is Back propogation.

Activation functions make the back-propogation possible since the gradients are supplied along with the error to update the weights and bias.