Supervised Learning - Prediction Week 3 Challenge
Ans1
import numpy as np
def loss_function(x):
return x**2
def gradient(x):
return 2 * x
# Hyperparameters
learning_rate = 0.1
num_iterations = 100
initial_x = 4.0
# Gradient Descent
x = initial_x
for i in range(num_iterations):
gradient_x = gradient(x)
x -= learning_rate * gradient_x
loss = loss_function(x)
print(f"Iteration {i+1}: x = {x:.4f}, Loss = {loss:.4f}")
print(f"Optimal x: {x:.4f}")
print(f"Minimum Loss: {loss:.4f}")
===================================================================================
Ans2.
L1 and L2 are two different regularization techniques used in gradient descent to prevent overfitting in machine learning models. They affect how the gradient is computed and, consequently, the way the model parameters are updated during training.
L1 Regularization (Lasso):
L1 regularization adds a penalty term to the loss function that is proportional to the absolute values of the model parameters (L1 norm).
The L1 penalty encourages sparsity in the model, meaning that it tries to drive some of the model parameters to exactly zero.
L1 regularization can be used for feature selection, as it tends to set some feature weights to zero, effectively eliminating those features from the model.
The gradient update for L1 regularization is a constant multiplied by the sign of the parameter (positive or negative), which means that the parameter is pushed towards zero.
Mathematically, the L1 regularization term added to the loss function is: λ * Σ|θ_i|, where θ_i is a model parameter.
L2 Regularization (Ridge):
L2 regularization adds a penalty term that is proportional to the square of the model parameters (L2 norm).
The L2 penalty discourages extremely large parameter values, leading to a smoother, more well-behaved loss landscape.
L2 regularization does not encourage sparsity as strongly as L1. It will not force parameters to be exactly zero, but it will make them very small.
The gradient update for L2 regularization is a constant multiplied by the parameter value itself, which pushes the parameter towards zero but not as aggressively as L1.
Mathematically, the L2 regularization term added to the loss function is: λ * Σθ_i^2, where θ_i is a model parameter.
====================================================================================
Ans3.
1-:Mean sqared error(MSE):
MSE is one of the most widely used loss functions for regression.
It measures the average squared difference between the predicted and actual values.
MSE is sensitive to outliers and punishes large errors more heavily.
Mathematically, MSE is defined as: MSE = (1/n) * Σ(y_i - ŷ_i)^2, where y_i is the actual target, ŷ_i is the predicted value, and n is the number of data points.
2-:Root Mean Squared Error (RMSE):
RMSE is the square root of the MSE and has the advantage of being in the same unit as the target variable.
It is also sensitive to outliers like MSE.
Mathematically, RMSE is defined as: RMSE = √((1/n) * Σ(y_i - ŷ_i)^2).
3-:Mean Absolute Error (MAE):
MAE measures the average absolute difference between the predicted and actual values.
MAE is less sensitive to outliers compared to MSE and RMSE.
Mathematically, MAE is defined as: MAE = (1/n) * Σ|y_i - ŷ_i|.
=================================================================================
Ans4.
The learning rate is a critical hyperparameter in the training of machine learning models, particularly in optimization algorithms like gradient descent. Its importance lies in its impact on the convergence, stability, and overall success of the training process. Here are some key reasons why the learning rate is important:
Coverence speed-: The learning rate controls the step size taken during each iteration of the optimization algorithm. A well-chosen learning rate can help the model converge faster, meaning it reaches a good solution with fewer iterations. Too high a learning rate may lead to divergence, while too low a learning rate can result in slow convergence.
Stability-:An appropriate learning rate ensures the stability of the optimization process. It helps prevent the model from oscillating around the optimal solution or overshooting it. A learning rate that's too high can lead to overshooting, while a learning rate that's too low can make the optimization process get stuck in local minima.
Quality of solution-:The learning rate affects the quality of the solution obtained. A well-chosen learning rate can help the model converge to a better minimum of the loss function. However, if the learning rate is too high, it may converge to a suboptimal solution, or if it's too low, it may get trapped in a local minimum
===================================================================================
Ans5.
Mean Squared Error (MSE)-:MSE measures the average squared difference between the actual and predicted values. It quantifies the overall quality of the model's predictions.
Lower MSE values indicate better model performance.
Root Mean Sqaured Error(RMSE)-:RMSE is the square root of the MSE and is in the same unit as the target variable.
It provides a more interpretable measure of prediction error.
Mean Absolute Error(MAE)-:MAE measures the average absolute difference between the actual and predicted values.
It is less sensitive to outliers compared to MSE.
===================================================================================
Ans6.
The coefficient of determination, denoted as R-squared (R²), is a statistical measure used to assess the goodness of fit of a regression model to the data. Both the multiple (or standard) R-squared and the adjusted R-squared are used in this context, but they serve slightly different purposes.
Multiple (Standard) R-squared (R²):
Multiple R-squared is a measure of the proportion of the variance in the dependent variable (the target) that is explained by the independent variables (predictors) included in the regression model.
It provides an indication of how well the model fits the data.
Multiple R-squared ranges from 0 to 1, where 0 means that the model explains none of the variance, and 1 means that the model explains all of the variance.
Mathematically, for a linear regression model, it is calculated as:
R² = 1 - (SSE / SST)
where SSE (Sum of Squared Errors) is the sum of the squared differences between the actual and predicted values, and SST (Total Sum of Squares) is the sum of the squared differences between the actual values and their mean.
Adjusted R-squared:
Adjusted R-squared is a modification of the multiple R-squared that takes into account the number of independent variables in the model.
It penalizes the inclusion of additional variables that do not significantly improve the model's performance.
As more variables are added to the model, the multiple R-squared will tend to increase, even if the added variables do not contribute much to explaining the variance in the dependent variable. The adjusted R-squared corrects for this by introducing a penalty term for adding more predictors.
Mathematically, it is calculated as:
Adjusted R² = 1 - [(1 - R²) * (n - 1) / (n - k - 1)]
where:
n is the number of data points.
k is the number of independent variables (predictors) in the model.