Supervised Learning - Prediction Week 3 Challenge
1. Python
import numpy as np
def gradient_descent (X, y, learning_rate=0.01, iterations=1000, loss_func
"""
Performs Gradient Descent optimization.
Args:
X: Input features (numpy array).
y: Target values (numpy array).
learning_rate : Learning rate.
iterations : Number of iterations.
loss_func : Loss function to use.
grad_func : Gradient function of the loss function.
Returns:
theta : Optimized parameters (numpy array).
"""
m, n = X.shape
theta = np.zeros(n)
for_in range(iterations):
predictions = np.dot(X, theta)
error = predictions - y
gradient = grad_func(X, error)
theta -= learning_rate * gradient
return theta
# Example usage with Mean squared Error (MSE)
def mse_loss(y_true, y_pred):
return np.mean (y_true - y_pred)**2)
def mse_gradient(X, error):
return (1/m) * np.dot(X,T, error)
# Create sample data
X = np.array ([[1 , 2] , [3 , 4] , [5 , 6]])
y = np.array ([1, 2, 3])
# Perform gradient descent with MSE loss
theta = gradient_descent(X,y, loss_func=mse_loss, grad_func = mse_gradient
print (theta)
3. Different loss function as follows
1 Mean Squared Error (MSE)
MSE = (1/n) * E(y_i - y_i)^2
Calculates the average squared difference between predicted and actual values.
General regression tasks where large errors are penalized heavily.
2. Root Mean Squared Error (RMSE)
RMSE = sqrt (MSE)
Calculates the square root of the mean squared error.
Similar to MSE, but provides error in the same units as the target variable
3. Mean Absolute Error (MAE)
Formulae
MAE = (1/n) * E/y_i - y_i l
calculates the average absolute difference between predicted and actual values.
Tasks where outliers should not have a significant impact on the loss.
4. Mean Absolute percentage Error (MAPE)
MAPE = (1/n) * E( ly_i - y_i l / ly_i l) * 100%
Calculates the average percentage error between perdicted and actual values.
Tasks where relatives error is important
2. Difference between L1 & L2
L1
Absolute value of coefficients
Encourages sparsity( many coefficients become zero)
Implicit feature selection
More robust to outliers
L2
Square of coefficients
Encourage small coefficients
No explicit feature selection
less robust to outliers
4. Importance of learning Rate
1. Convergence Speed
High Learning Rate
Faster Convergence initally
Risk of overshooting the minimum and diverging
Low Learning Rate
Slower Convergence
More stable and less likely to overshoot
2. Avoiding Local Minima
A well tuned learning rate can help the algorithm escape shallow local minima and find the global minimum
3. Preventing Oscillations
A high learning rate can lead to oscillations around the minimum, hindering convergence
5. Evaluate linear regression
1. Coefficient of Determination ( R-squared)
Interpretation : Measures the proportion of the variance in the dependent variable this is explained by the independent variables.
Range : 0 to 1
Higher is better : A higher R-squared value indicates a better fit
2. Mean Squared Error (MSE)
Interpretation : Calculates the average squared difference between the predicted and actual values.
Lower is better: A lower MSE indicates a better fit.
3. Root Mean Squared Error (RMSE)
Interpretation: Calculates the square root of the MSE, providing the error in the same units as the dependent variable
Lower is better : A lower RMSE indicates a better fit
4. Mean Absolute Error (MAE)
Interpretation: Calculates the average absolute difference between the predicted and actual values.
Lower is Better : A lower MAE indicates a better fit.
5. Residual Analysis
Interpretation: Analyzing the residuals (difference between predicted and actual values) can help identify patterns, outliers, and potential model issues.
6. Multiple R squared
Measures the proportion of the variance in the dependent variable that is explained by the independent variables.
Always increase as more independent variables are added to the model, even if they are not significant.
Adjusted R squared
A modified version of R squared that penalizes the addition of unnecessary independent variables.
It adjusts for the number of predictions in the model, providing a more accurate measure of the model's fit.
It can decrease if adding a new variable doen't significantly improve the model's predictive power.