Supervised Learning - Classification Week 9 Challenge

Neural networks rely on training data to learn and improve their accuracy over time. However, once these learning algorithms are fine-tuned for accuracy, they are powerful tools in computer science and artificial intelligence, allowing us to classify and cluster data at a high velocity. Tasks in speech recognition or image recognition can take minutes versus hours when compared to manual identification by human experts. One of the most well-known neural networks is Google’s search algorithm.

Deep learning is a subset of machine learning, which is essentially a neural network with three or more layers. These neural networks attempt to simulate the behavior of the human brain—albeit far from matching its ability—allowing it to “learn” from large amounts of data. While a neural network with a single layer can still make approximate predictions, additional hidden layers can help to optimize and refine for accuracy.

Deep learning drives many artificial intelligence applications and services that improve automation, performing analytical and physical tasks without human intervention. Deep learning technology lies behind everyday products and services (such as digital assistants, voice-enabled TV remotes, and credit card fraud detection) as well as emerging technologies (such as self-driving cars).

Real world examples:

• Law enforcement
• Financial services
• Customer service
• Health care

3) What is the role of the activation function?

An activation function is a function that is added to an artificial neural network (ANN) in order to help the network learn complex patterns in the data. When compared with a neuron-based model that is in our brains, the activation function is at the end deciding what is to be fired to the next neuron. That is exactly what an activation function does in an ANN as well. It takes in the output signal from the previous cell and converts it into some form that can be taken as input to the next cell.

Activation functions are mathematical equations that determine the output of a neural network model. Activation functions also have a major effect on the neural netwoork's ability to converge and the convergence speed, or in some cases, activation functions might prevent neural networks from converging in the first place. Activation function also helps to normalize the output of any input in the range between 1 to -1 or 0 to 1.

Activation function must be efficient and it should reduce the computation time because the neural network sometimes trained on millions of data points.

Properties of activation functions

1. Non Linearity
2. Continuously differentiable
3. Range
4. Monotonic
5. Approximates identity near the origin

4) What are the different types of activation functions?

The activation functions can be broadly classified into 2 categories.

1. Binary Step Function
2. Linear Activation Function

Binary Step Function:

A binary step function is generally used in the Perceptron linear classifier. It thresholds the input values to 1 and 0, if they are greater or less than zero, respectively.

Linear Activation function:

The equation for Linear activation function is:

f(x) = a.x

When a = 1 then f(x) = x and this is a special case known as identity.

Properties:

1. Range is -infinity to +infinity
2. Provides a convex error surface so optimisation can be achieved faster
3. df(x)/dx = a which is constant. So cannot be optimised with gradient descent

Limitations:

1. Since the derivative is constant, the gradient has no relation with input
2. Back propagation is constant as the change is delta x

Non Linear Activation functions:

Sigmoid

Hyperbolic Tangent function or Tanh

Softmax

Softsign

Rectified Linear Unit (ReLU)

Exponential Linear units (ELU)

Softplus or SmoothRelu

Swish

1. Sigmoid function

In an ANN, the sigmoid function is a non-linear AF used primarily in feedforward neural networks. It is a differentiable real function, defined for real input values, and contains positive derivatives everywhere with a specific degree of smoothness. The sigmoid function appears in the output layer of the deep learning models and is used for predicting probability-based outputs.

2. Hyperbolic Tangent function ( Tanh )

The hyperbolic tangent function, a.k.a., the tanh function, is another type of AF. It is a smoother, zero-centered function having a range between -1 to 1.

Y = tanh(x)
tanh(x) = (e^x – e^-x) / (e^x + e^-x)

3. SoftMax function

The softmax function is another type of AF used in neural networks to compute probability distribution from a vector of real numbers. This function generates an output that ranges between values 0 and 1 and with the sum of the probabilities being equal to 1.

4. Softsign function

The soft sign function is another AF that is used in neural network computing. Although it is primarily in regression computation problems, nowadays it is also used in DL-based text-to-speech applications. It is a quadratic polynomial.

5. Rectified Linear Unit (ReLU) function

One of the most popular AFs in DL models, the rectified linear unit (ReLU) function, is a fast-learning AF that promises to deliver state-of-the-art performance with stellar results. Compared to other AFs like the sigmoid and tanh functions, the ReLU function offers much better performance and generalization in deep learning. The function is a nearly linear function that retains the properties of linear models, which makes them easy to optimize with gradient-descent methods.

The ReLU function performs a threshold operation on each input element where all values less than zero are set to zero.

ReLU(x) = max(0,x)

Though it looks like a linear function, it’s not. ReLU has a derivative function and allows for backpropagation.

There is one problem with ReLU. Let’s suppose most of the input values are negative or 0, the ReLU produces the output as 0 and the neural network can’t perform the back propagation. This is called the Dying ReLU problem. Also, ReLU is an unbounded function which means there is no maximum value.

Pros:

1. Less time and space complexity
2. Avoids the vanishing gradient problem.

Cons:

1. Introduces the dead relu problem.
2. Does not avoid the exploding gradient problem.

6. Exponential Linear Units (ELUs) function

The exponential linear units (ELUs) function is an AF that is also used to speed up the training of neural networks (just like ReLU function). The biggest advantage of the ELU function is that it can eliminate the vanishing gradient problem by using identity for positive values and by improving the learning characteristics of the model.

ELUs have negative values that push the mean unit activation closer to zero, thereby reducing computational complexity and improving the learning speed. The ELU is an excellent alternative to the ReLU – it decreases bias shifts by pushing mean activation towards zero during the training process.

The derivative or gradient of the ELU equation

7. Softplus or SmoothRelu

The derivative of the softplus function is the logistic function.

The mathematical expression is:

And the derivative of softplus is:

8. Swish function

The Swish function was developed by Google, and it has superior performance with the same level of computational efficiency as the ReLU function. ReLU still plays an important role in deep learning studies even for today. But experiments show that this new activation function overperforms ReLU for deeper networks

The mathematical expression for Swish Function is:

The modified version of swish function is:

5) What do you understand by backpropagation?

We can define the Backpropagation algorithm as an algorithm that trains some given feed-forward Neural Network for a given input pattern where the classifications are known to us. At the point when every passage of the example set is exhibited to the network, the network looks at its yield reaction to the example input pattern. After that, the comparison done between output response and expected output with the error value is measured. Later, we adjust the connection weight based upon the error value measured.

1. The input layer receives x
2. Input is modeled using weights w
3. Each hidden layer calculates the output and data is ready at the output layer
4. The difference between actual output and the desired output is known as the error
5. Go back to the hidden layers and adjust the weights so that this error is reduced in future runs