Supervised Learning - Classification Week 9 Challenge

1.

Neural networks, also known as artificial neural networks (ANNs) or simulated neural networks (SNNs), are a subset of machine learning and are at the heart of deep learning algorithms. Their name and structure are inspired by the human brain, mimicking the way that biological neurons signal to one another.

Artificial neural networks (ANNs) are comprised of a node layers, containing an input layer, one or more hidden layers, and an output layer. Each node, or artificial neuron, connects to another and has an associated weight and threshold. If the output of any individual node is above the specified threshold value, that node is activated, sending data to the next layer of the network. Otherwise, no data is passed along to the next layer of the network.



2.

Deep learning is related to machine learning based on algorithms inspired by the brain's neural networks. Though it sounds almost like science fiction, it is an integral part of the rise in artificial intelligence (AI). Machine learning uses data reprocessing driven by algorithms, but deep learning strives to mimic the human brain by clustering data to produce startlingly accurate predictions.

Examples of deep learning
Deep learning is generating a lot of conversation about the future of machine learning. Technology is rapidly evolving, generating both fear and excitement. While most people understand machine learning and AI, deep learning is the "new kid on the block" in tech circles and generates both anxiety and excitement.

Deep learning is also known as neural organized learning and happens when artificial neural networks learn from large volumes of data. Deep learning algorithms perform tasks repeatedly, tweaking them each time to improve the outcome. The algorithms depend on vast amounts of data to drive "learning."

Current estimates predict that 1.145 trillion MB of data is produced every day, and it is this staggering amount of data production that makes deep learning possible . The vast increase in data creation is the driving force behind the rise in deep learning capabilities. Though deep learning can sound mysterious, the truth is that most of us are already using deep learning processes in our everyday lives.

4.

Formulae for some Activation Functions
ReLU Function Formula
 

There are a number of widely used activation functions in deep learning today. One of the simplest is the rectified linear unit, or ReLU function, which is a piecewise linear function that outputs zero if its input is negative, and directly outputs the input otherwise:

 



Mathematical definition of the ReLU Function




Graph of the ReLU function, showing its flat gradient for negative x.

ReLU Function Derivative
It is also instructive to calculate the gradient of the ReLU function, which is mathematically undefined at x = 0 but which is still extremely useful in neural networks.



The derivative of the ReLU function. In practice the derivative at x = 0 can be set to either 0 or 1. 

The zero derivative for negative x

 can give rise to problems when training a neural network, since a neuron can become 'trapped' in the zero region and backpropagation will never change its weights.

 

PReLU Function, a Variation on the ReLU
Because of the zero gradient problem faced by the ReLU, it is sometimes common to use an adjusted ReLU function called the parametric rectified linear unit, or PReLU:



This has the advantage that the gradient is nonzero at all points (except 0 where it is undefined).

The PReLU function with α set to 0.1 avoids the zero gradient problem.

Logistic Sigmoid Function Formula
Another well-known activation function is the logistic sigmoid function:



Mathematical definition of the Logistic Sigmoid Function

The logistic sigmoid function has the useful property that its gradient is defined everywhere, and that its output is conveniently between 0 and 1 for all x. The logistic sigmoid function is easier to work with mathematically, but the exponential functions make it computationally intensive to compute in practice and so simpler functions such as ReLU are often preferred.


Graph showing the characteristic S-shape of the logistic sigmoid function

Logistic Sigmoid Function Derivative
The derivative of the logistic sigmoid function can be expressed in terms of the function itself:

The derivative of the logistic sigmoid function.




Graph of the derivative of the logistic sigmoid function

 

The derivative of the logistic sigmoid function is nonzero at all points, which is an advantage for use in the backpropagation algorithm, although the function is intensive to compute, and the gradient becomes very small for large absolute x, giving rise to the vanishing gradient problem.

 

Because the derivative contains exponentials, it is computationally expensive to calculate. The backpropagation algorithm requires the derivative of all operations in a neural network to be calculated, and so the sigmoid function is not well suited for use in neural networks in practice due to the complexity of calculating its derivative repeatedly.

 

3.

An activation function is a function used in artificial neural networks which outputs a small value for small inputs, and a larger value if its inputs exceed a threshold. If the inputs are large enough, the activation function "fires", otherwise it does nothing. In other words, an activation function is like a gate that checks that an incoming value is greater than a critical number.

 

Activation functions are useful because they add non-linearities into neural networks, allowing the neural networks to learn powerful operations. If the activation functions were to be removed from a feedforward neural network, the entire network could be re-factored to a simple linear operation or matrix transformation on its input, and it would no longer be capable of performing complex tasks such as image recognition.

 

Well-known activation functions used in data science include the rectified linear unit (ReLU) function, and the family of sigmoid functions such as the logistic sigmoid function, the hyperbolic tangent, and the arctangent function.

 4.

Backpropagation is the essence of neural network training. It is the method of fine-tuning the weights of a neural network based on the error rate obtained in the previous epoch (i.e., iteration). Proper tuning of the weights allows you to reduce error rates and make the model reliable by increasing its generalization.

Backpropagation in neural network is a short form for “backward propagation of errors.” It is a standard method of training artificial neural networks. This method helps calculate the gradient of a loss function with respect to all the weights in the network.

The Back propagation algorithm in neural network computes the gradient of the loss function for a single weight by the chain rule. It efficiently computes one layer at a time, unlike a native direct computation. It computes the gradient, but it does not define how the gradient is used. It generalizes the computation in the delta rule.