Supervised Learning - Prediction Week 3 Challenge

Perform Gradient Descent in Python with any Loss Function

Gradient descent is an optimization algorithm used to minimize a loss function by iteratively adjusting the model parameters. Below is an example of implementing Gradient Descent for Linear Regression with Mean Squared Error (MSE) Loss Function.

Python Implementation

import numpy as np

import matplotlib.pyplot as plt

# Generate synthetic data

np.random.seed(42)

X = 2 * np.random.rand(100, 1) # Independent variable

y = 4 + 3 * X + np.random.randn(100, 1) # Dependent variable with some noise

# Gradient Descent Implementation

def compute_cost(X, y, theta):

"""Compute Mean Squared Error Loss"""

m = len(y)

predictions = X.dot(theta)

cost = (1 / (2 * m)) * np.sum((predictions - y) ** 2)

return cost

def gradient_descent(X, y, theta, learning_rate=0.1, iterations=1000):

"""Perform gradient descent to optimize theta"""

m = len(y)

cost_history = []

for _ in range(iterations):

gradients = (1/m) * X.T.dot(X.dot(theta) - y) # Compute gradients

theta -= learning_rate * gradients # Update weights

cost_history.append(compute_cost(X, y, theta)) # Track cost

return theta, cost_history

# Prepare data for Linear Regression

X_b = np.c_[np.ones((100, 1)), X] # Add bias term (intercept)

theta_init = np.random.randn(2, 1) # Initialize weights

# Run gradient descent

theta_opt, cost_history = gradient_descent(X_b, y, theta_init)

print(f"Optimized Weights (Theta): \n{theta_opt}")

# Plot the cost function reduction

plt.plot(cost_history)

plt.xlabel("Iterations")

plt.ylabel("Cost (MSE)")

plt.title("Cost Function Convergence")

plt.show()

Output:

• Optimized weights $\theta$θ will be close to $[4, 3]$[4,3] since the true function is $y = 4 + 3X$.
• A decreasing cost function plot indicating convergence.

---

Difference Between L1 & L2 Gradient Descent Method

L1 and L2 regularization methods are used to prevent overfitting and improve generalization by adding penalties to the cost function.

Feature	L1 Regularization (Lasso)	L2 Regularization (Ridge)
Penalty Term	( \lambda \sum	w
Weight Reduction	Some weights become exactly zero (feature selection)	Shrinks weights gradually towards zero
Computational Cost	Slower, involves absolute value derivative	Faster, involves squared terms
Use Case	Feature selection in sparse models	Multicollinearity problems

Formula for Loss Function

• L1 Regularization:

  $$Loss = MSE + \lambda \sum |w|$$
  

• L2 Regularization:

  $$Loss = MSE + \lambda \sum w^2$$
  

---

Different Loss Functions for Regression

Loss functions measure the difference between predicted and actual values.

Common Regression Loss Functions

1. Mean Squared Error (MSE)

   $$MSE = \frac{1}{n} \sum (y_i - \hat{y_i})^2$$
   
2. • Most common, penalizes large errors.
   • Sensitive to outliers.

3. Mean Absolute Error (MAE)

   $$MAE = \frac{1}{n} \sum |y_i - \hat{y_i}|$$
   
   • Less sensitive to outliers than MSE.
   • Linear penalty.

4. Huber Loss

   $$L_{\delta}(a) = \begin{cases} \frac{1}{2} a^2 & \text{for } |a| \leq \delta \\ \delta (|a| - \frac{1}{2} \delta) & \text{for } |a| > \delta \end{cases}$$​
   • Combination of MSE and MAE.
   • Robust to outliers.

5. Log-Cosh Loss

   $$L(y, \hat{y}) = \sum \log(\cosh(y - \hat{y}))$$
   
   • Similar to Huber but smoother.
   • Best for handling outliers.

---

Importance of Learning Rate in Gradient Descent

The learning rate ($\alpha$) controls how fast the model updates parameters.

Effect of Different Learning Rates

Learning Rate	Effect
Too Small ($\alpha$)	Very slow convergence, takes too many iterations.
Too Large ($\alpha$)	Model may overshoot the optimal solution and diverge.
Optimal ($\alpha$)	Ensures steady convergence to the minimum.

Learning Rate Visualization

• High Learning Rate → Diverging loss.
• Low Learning Rate → Slow convergence.
• Optimal Learning Rate → Smooth convergence.

To find the best learning rate, you can use:

from tensorflow.keras.callbacks import LearningRateScheduler

How to Evaluate Linear Regression

To evaluate a Linear Regression Model, the following metrics are used:

1. R-squared (R²)

$$R^2 = 1 - \frac{SS_{res}}{SS_{tot}}$$​​

• Measures the proportion of variance explained by the model.
• Higher R² (closer to 1) = Better model.

2. Mean Squared Error (MSE)

$$MSE = \frac{1}{n} \sum (y_i - \hat{y_i})^2$$

• Measures average squared error.

3. Root Mean Squared Error (RMSE)

$$RMSE = \sqrt{MSE}$$​

• Similar to MSE but in the same units as the target.

4. Mean Absolute Error (MAE)

$$MAE = \frac{1}{n} \sum |y_i - \hat{y_i}|$$

• Measures absolute error, less sensitive to outliers.

Example Code for Evaluation

from sklearn.metrics import mean_squared_error, r2_score

mse = mean_squared_error(y_test, y_pred)

rmse = np.sqrt(mse)

r2 = r2_score(y_test, y_pred)

print(f"MSE: {mse}")

print(f"RMSE: {rmse}")

print(f"R² Score: {r2}")

Difference Between Multiple and Adjusted Coefficient of Determination

Metric	Formula	Interpretation
Multiple R²	$R^2 = 1 - \frac{SS_{res}}{SS_{tot}}$​​	Measures how well the model fits the data.
Adjusted R²	$R^2_{adj} = 1 - (1 - R^2) \frac{n-1}{n-p-1}$​	Accounts for number of predictors in the model.

Key Differences:

• R² always increases when adding more features, even if they don’t improve the model.
• Adjusted R² penalizes for adding unnecessary features, making it a better metric.

Example:

• Multiple R² = 0.85, meaning 85% of variance is explained.
• Adjusted R² = 0.78, meaning adding features may not be significantly improving the model.

---

Final Summary

1. Gradient Descent Implementation

✅ Performed gradient descent using Mean Squared Error (MSE).
✅ Visualized convergence using a cost function plot.

2. Difference Between L1 & L2

✅ L1 (Lasso) shrinks some weights to zero (good for feature selection).
✅ L2 (Ridge) shrinks all weights towards zero but does not make them exactly zero.

3. Loss Functions for Regression

✅ MSE, MAE, Huber, Log-Cosh used for different scenarios.

4. Learning Rate

✅ Too high = Divergence
✅ Too low = Slow convergence
✅ Optimal = Efficient learning

5. Evaluation of Linear Regression

✅ Used R², MSE, RMSE, MAE for model evaluation.

6. Multiple vs. Adjusted R²

✅ Adjusted R² accounts for feature count, making it more reliable