Supervised Learning - Classification Week 9 Challenge

Q.1 What is a Neural Network?

• Neural Networks is a computational learning system that uses a network of functions to understand and translate a data input of one form into a desired output, usually in another form.
• The concept of the artificial neural network was inspired by human biology and the way neurons of the human brain function together to understand inputs from human senses.
• In simple words, Neural Networks are a set of algorithms that tries to recognize the patterns, relationships, and information from the data through the process which is inspired by and works like the human brain/biology.
• Simple Neuron network consists of three components.
• Input layer:-
• It is also known as inputs nodes.It takes the information from outside the world and provide to model so that model can learn and derive conclusion from the it.Input nodes pass the information to the next layer i.e Hidden layer.
• Hidden layer:-
• Hidden layer is the set of neurons where all the computations are performed on the input data.
• There can be any number of hidden layers in a neural network.
• The simplest network consists of a single hidden layer.
• Output layer:-
• The output layer is the output/conclusions of the model derived from all the computations performed.
• There can be single or multiple nodes in the output layer.
• If we have a binary classification problem the output node is 1 but in the case of multi-class classification, the output nodes can be more than 1.
• 
• Let's undertstand single layer & multi layer neural network.
• Perceptron: It is a simple form of Neural Network and consists of a single layer where all the mathematical computations are performed.
• 

• Multilayer Perceptron :It also known as Artificial Neural Networks consists of more than one perception which is grouped together to form a multiple layer neural network.

• 
• If we see in above artificial neural network, there are four layers and which are interconnected with each other.
• Input layer with six inputs nodes
• first Hidden layer ,with 4 hidden nodes/4 perceptrons
• second Hidden layer, with 4 hidden nodes/4 perceptrons
• Output layer with 1 output node
• Working of neural network in brief:
• 
• In the above image x₁,x₂,x₃,….x_n are the inputs , w1,w2,w3...wn are the weights of connection and b is the bias.
• In the first step, Input units are passed i.e data is passed with some weights attached to it to the hidden layer.

• Each hidden layer consists of neurons. All the inputs are connected to each neuron.

• After passing on the inputs, all the computation is performed in the hidden layer (Blue oval in the picture)

• In Hidden layer,first of all, all the inputs are multiplied by their weights. then bias is added this is called weighted sum.
• Weighhted sum
• `Z_1 = W_1 times X_1 +W_2 times X_2+W_3 times X_3+....+W_n times X_n + b`
• Then in the second step, the activation function is applied to the linear equation Z1.
• After performing all the computation in each hidden layer,we move to the last layer i.e our output layer which gives us the final output.

Q.2 What is deep learning?

• Deep learning is a subset of machine learning in artificial intelligence, i.e., based upon artificial neural network and representation learning, as it is capable of implementing a function that is used to mimic the functionality of the brain by creating patterns and processing data.
• Deep Learning is also used for decision-making in fields like driverless cars ( to detect pedestrians, street lights, other cars, etc.), speech recognition, image analysis( e.g., Identifying cancer in blood and tumors), smart TV, etc.
• Researchers tried to mimic the working of the human brain and replicated it into the machine making machines capable of thinking and solving complex problems. Deep Learning (DL) is a subset of Machine Learning (ML) that allows us to train a model using a set of inputs and then predict output based.
• Like the human brain, the model consists of a set of neurons that can be grouped into 3 layers:

1. Input Layer
2. Hidden Layers
3. Output Layer

• 
• We have already seen in above question how the deep neural network works.
• Let's talk about other details

• Deep learning is a subset of machine learning. Machine learning is a subset of artificial intelligence. Said another way — all deep learning algorithms are machine learning algorithms, but many machine learning algorithms do not use deep learning.
• 
• deep learning and machine learning are the subsets of artificial intelligence but deep learning technology represents the next evolution of machine learning.
• Machine learning will work based on algorithms and programs developed by humans, whereas deep learning learns through a neural network model which acts similar to humans and allows machines or computers to analyze the data in a similar way as humans do.
• And this becomes possible because we train the neural network models with a huge amount of data as data is the fuel or food for neural network models.
• Real world applications
• Computer Vision: Computer vision deals with algorithms for computers to understand the world using the image and video data and tasks such as image recognition, image classification, object detection, image segmentation, image restoration, etc.
• Autonomous Vehicles: Deep learning models are trained with a huge amount of data for identifying street signs; some models specialize in identifying pedestrians, identifying humans, etc., for driverless cars while driving.
• Speech and Natural Language Processing: Natural language processing deals with algorithms for computers to understand, interpret, and manipulate human language.

Q.3 What is the role of the activation function?

• The activation function is a mathematical “gate” in between the input feeding the current neuron and its output going to the next layer. They basically decide whether the neuron should be activated or not.
• 
• Each neuron has an activation function that performs computation. Different layers can have different activation functions but neurons belonging to one layer have the same activation function.
• In deep neural network, a weighted sum of input is calculated based on weights and inputs provided. Then, the activation function comes into the picture that works on weighted sum and converts it into output.
• Activation functions help model learn complex relationship that exists within the dataset.
• If we do not use the activation function in neurons and give weighted sum as output, in that case, computations will be difficult as there is no specific range for weighted sum.So, the activation function helps to keep output in a particular range.
• Secondly, the non-linear activation function is always preferred as it adds non-linearity to the dataset which otherwise would form a simple linear regression model incapable of taking the benefit of hidden layers.
• Activation function help in normalizing the output between 0 to 1 or -1 to 1. It helps in the process of backpropagation due to their differentiable property. During backpropagation,loss function gets updated, and activation function helps the gradient descent curves to achieve their local minima.

Q.4 What are the different types of activation functions?

• Types of activation functions:

1. Linear or Identity Activation Function
2. Non-linear Activation Functions

1.Linear or Identity Activation Function

• It takes the inputs, multiplied by the weights for each neuron, and creates an output signal proportional to the input.
• 
• eqaution f(x) = x
• `f^'(x) =1`
• Its ranges is from `-infty`to`+infty`
• This type of function is best suited to for simple regression problems, maybe housing price prediction.
• Demerits

1. Back-propagation is not possible — The derivative of the function is a constant, and has no relation to the input, X. So it’s not possible to go back and understand which weights in the input neurons can provide a better prediction.
2. All layers of the neural network collapse into one — with linear activation functions, no matter how many layers in the neural network, the last layer will be a linear function of the first layer

2.Non-linear activation function

• Modern neural network models use non-linear activation functions. They allow the model to create complex mappings between the network’s inputs and outputs, which are essential for learning and modeling complex data, such as images, video, audio, and data sets which are non-linear or have high dimensionality.
• Non-linear functions address the problems of a linear activation function
• They allow back-propagation because they have a derivative function which is related to the inputs.
• They allow “stacking” of multiple layers of neurons to create a deep neural network. Multiple hidden layers of neurons are needed to learn complex data sets with high levels of accuracy.
• some commonly used non-linear activation functions are:

1.Sigmoid/Logistic:

• Sigmoid function gives an ‘S’ shaped curve. In order to map predicted values to probabilities, we use the sigmoid function. The function maps any real value into another value between 0 and 1.
• 
• eqaution = `f(x)=sigma(x) =1/(1+e^-x)`
• derivative = `f^'(x) =sigma^'(x) = sigma(x) *(1-sigma(x))`
• Its Range is (0,1)
• The function is differentiable.That means, we can find the slope of the sigmoid curve at any two points.
• Output values bound between 0 and 1, normalizing the output of each neuron.

2.Tan_h/Hyperbolic tangent:

• 
• Eqaution:`f(x)=a=tanh(x)=(e^x-e^-x)/(e^x+e^-x)`
• Derivative :`1-a^2`
• Range:(-1,1)
• Zero centered — making it easier to model inputs that have strongly negative, neutral, and strongly positive values.
• The function and its derivative both are monotonic.
• It works better than sigmoid function

3.ReLU(Rectified Linear Unit)

• 
• Eqaution:`f(x)=a=max(0,x)`
• Derivative :`f’(x) = { 1 ; if z>0, 0; if z<0 and undefined if z=0 }`
• Range:`(0,+infty)`
• Advantages:
• Computationally efficient — allows the network to converge very quickly
• Non-linear — although it looks like a linear function, ReLU has a derivative function and allows for back-propagation

5.Softmax

• Softmax function calculates the probabilities distribution of the event over ’n’ different events.
• 
• 
• Advantages:
• Useful for output neurons- typically Softmax is used only for the output layer, for neural networks that need to classify inputs into multiple categories.

Q.5 What do you understand by backpropagation?

• Backpropagation is a supervised learning algorithm, for training Multi-layer Perceptrons (Artificial Neural Networks).
• It is an essential mechanism by which neural networks get trained. It is a mechanism used to fine-tune the weights of a neural network .
• The main goal of any machine learning algorithm is to minimize error. For this, we can update the learning parameters like weights and biases using gradient descent. This method of updating the parameters for each layer in a neural network is done using back-propagation. It’s called back-propagation because we update the parameters starting from the last layer and moving backward to the first layer.
• Neural network
• 
• Backpropagation in neural networks is about the transmisssion of information and relating this information to the error generated by the model when a guess was made. This method seeks to reduce the error, which is otherwise referred to as the loss function
• Let's understand how does it work
• As we can see in the above image, the inputs are nothing but features. In other words, we can say this as columns of our dataset.
• ‘W’, the real weights are used to model the input. The values of W are randomly allotted.
• The output for every neuron is calculated through forwarding propagation – the input layer, hidden layer, and output layer.
• For Hidden Layer
• 
• a1 = g(z1), a2 = g(z2), a3 = g(z3), where g is the activation function.
• After calculating a1, a2, a3 take them as input features to the Output layer.
• Apply same procedure for output layer i.e take weighted sum and pass to the activation function we will get outputs.
• Then error is calculated at the outputs using the equation Propagating backward again through output and hidden layers, weights are adjusted to reduce the error.

• Again propagate forward to calculate the output and error. If the error is minimized, this process ends, or else propagates backward and adjusts the values of weight.