Supervised Learning - Classification Week 9 Challenge

Ans 1. A neural network is a computing system inspired by the way biological neural networks in the human brain process information. These networks consist of interconnected nodes or "neurons" organised into layers: an input layer, one or more hidden layers, and an output layer. Each connection between neuron has a weight, which is adjusted during training to minimize the difference between neurons has a weight, which is adjusted during training to reduce the difference between the network's predictions and the actual outcomes.

Ans 2. Deep learning is a subset of machine learning that focuses on neural networks with many layers (hence"deep"). It enables computers to learn and make decisions from large amounts of data by mimicking the way humans learn from experience. Deep learning models are compelling for tasks involving image and speech recognition, natural language processing, and other complex data-intensive process.

Key characteristics of deep learning include:

1. Multiple layers: Deep learning models use neural networks with several hidden layers, allowing them to learn hierarchical representations of data.

2. Large datasets: Deep learning models often require large amounts of data to achieve high performance.

3. High computational Power: Deep learning algorithms typically demand significant computational resources, often utilizing GPUs or TPUs for training.

4. Feature Extraction: Unlike traditional machine learning, deep learning models automatically extract relevant features from raw data, reducing the need for manual feature engineering.

Ans 3. The activation function is a crucial component of a neural network. It determines whether a neuron should be activated or not by introducing non-linearity into the network. Without activation functions, a neural network would only be able to learn and represent linear relationships, making it less powerful and less capable of handling complex data.

Here are the primary roles of the activation function:

1. Introduce Non-Linearity: The activation function enables neural networks to learn complex patterns and relationships in the data by introducing non-linear transformations. This allows the network to approximate non-linear functions and solve more complex problems.

2. Control Output Range: Activation functions can constrain the output of neurons to a specific range, which can help stabilize the training process and improve the network's performance.

3. Enable Gradient-Based optimization: Activation functions enable the use of gradient-based optimization techniques, such as backpropagation, by providing a differentiable function. This allows the network to update weights effectively during training.

Ans 4. Activation functions are a critical component of neural networks, and there are several types, each with its own unique characteristics and use cases.

1. Sigmoid Function:

• Formula:$$\sigma(x) = \frac{1}{1 +e^{-x}}$$
• Output Range : (0,1)
• Characteristics: S-shaped curve, smooth gradient, but can suffer from vanishing gradients.

2. Tanh(Hyperbolic Tangent) Function:

• Formula : $$\tanh(x)=\frac{e^x-e^{-x}}{e x +e^{-x}}$$
• Output Range: (-1,1)
• Characteristics: Zero-centered output, smooth gradient, can still suffer from vanishing gradient but less than sigmoid.

3. ReLU( Rectified Linear unit) Function:

• Formula: f(x) = max(0,x)
• output range: [0, infinity)
• Characteristics: Computationally efficient, helps mitigate vanishing gradient problem. However, it can suffer from "dying ReLUs" where neurons become inactive for all inputs.

4. Leaky ReLU Function:

• Formula: f(x)= max(0.01x, x)
• Output range: (-inf,inf)
• Characteristics: Addresses the dying ReLU problem by allowing a small, non-zero gradient when input is negative.

5. Softmax Function:

• Formula: $$\text{softmax}(x_i)=\frac{e^{x_i}{\sum_{j}e^{x_j}}$$
• Output Range: (0,1), output sum to 1
• Characteristics: Typically used in the output layer for classification problems to generate probability distributions over classes.

6. Swish function:

• Formula: f(x) = x.sigma(x)
• output Range: (-inf, inf)
• Characteristics: Combines properties of ReLU and sigmoid. Often leads to better performance in deep networks.

Ans 5: Backpropagation, short for " backward propagation of errors", is a fundamental algorithm used for training neural networks. It helps the network learn by adjusting the weights of connections between neurons to minimize the error between the network's predictions and the actual outcomes. The process involves two main steps: Forward pass and backward pass.

1. Forward pass:

• During the forward pass, the input data is fed through the network and the output is calculated using the current weights and the activation functions.
• The network's predictions are compared to the actual target values and the error(loss) is computed using a loss function.( e.g., mean squared error for regression, cross-entropy for classification)

2. Backward pass:

• In the backward pass, the error is propagated backwards through the network to update the weights.
• This is done using the chain rule of calculus to compute the gradient of the loss function with respect to each weight.
• The gradients indicate the direction and magnitude of the change needed for each weight to reduce the error.
• The weights are then updated using an optimization algorithm, such as gradient descent(SGD), by subtracting a fraction of the gradient (learning rate) from current weights.