Supervised Learning - Classification Week 9 Challenge

1. What is a Neural Network?

A Neural Network is a computational model inspired by the human brain. It consists of layers of artificial neurons that process input data to make predictions or classifications. It is commonly used in machine learning for tasks like image recognition, speech processing, and pattern detection. Neural networks learn by adjusting weights and biases through training on data.

2. What is Deep Learning?

Deep Learning is a subset of machine learning that involves neural networks with multiple hidden layers (deep neural networks). These deep structures allow the model to learn complex patterns and representations from raw data, making it highly effective for tasks like natural language processing, autonomous driving, and medical diagnosis.

3. What is the Role of the Activation Function?

An activation function determines whether a neuron should be activated or not by applying a transformation to the input. It introduces non-linearity, allowing neural networks to learn complex patterns instead of just simple linear relationships.

What are the Different Types of Activation Functions?

1. Sigmoid Function:

   • Output: $0$0 to $1$1

   • Used for binary classification

   • Formula: $\sigma(x) = \frac{1}{1+e^{-x}}$σ(x)=1+e−x1​

   • Disadvantage: Vanishing gradient problem

2. Tanh (Hyperbolic Tangent) Function:

   • Output: $-1$−1 to $1$1

   • Used in hidden layers for zero-centered activation

   • Formula: $\tanh(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}}$tanh(x)=ex+e−xex−e−x​

   • Disadvantage: Still suffers from vanishing gradients

3. ReLU (Rectified Linear Unit):

   • Output: $0$0 for negative inputs, $x$x for positive inputs

   • Used in most deep networks

   • Formula: $f(x) = \max(0, x)$f(x)=max(0,x)

   • Advantage: Avoids vanishing gradient issue for positive values

   • Disadvantage: "Dying ReLU" problem (neurons can become inactive permanently)

4. Leaky ReLU:

   • Allows small negative values instead of zero

   • Formula: $f(x) = x$f(x)=x if $x > 0$x>0, else $\alpha x$αx where $\alpha$α is a small value (e.g., 0.01)

   • Helps prevent "dying ReLU" problem

5. Softmax Function:

   • Used for multi-class classification

   • Converts logits into probabilities

   • Formula: $S(x_i) = \frac{e^{x_i}}{\sum_{j} e^{x_j}}$S(xi​)=∑j​exj​exi​​

4. What Do You Understand by Backpropagation?

Backpropagation (Backward Propagation of Errors) is an algorithm used to train neural networks by adjusting weights and biases. It works in two main steps:

1. Forward Propagation: Compute the output using current weights.

2. Backward Propagation: Calculate the error (difference between actual and predicted output) and update the weights using Gradient Descent.

Key Concepts in Backpropagation:

• Uses the chain rule to compute gradients

• Updates weights using learning rate

• Helps neural networks minimize loss function efficiently