Supervised Learning - Prediction Week 3 Challenge

Q2. 1. L1 Regularization (Lasso):

- L1 regularization adds a penalty term to the loss function based on the absolute value of the coefficients.

- It encourages sparsity by pushing some coefficients to exactly zero.

- Effect on Model Weights:

- L1 regularization tends to shrink coefficients towards zero.

- It results in sparse models where some features are completely ignored.

- Useful for feature selection because it can eliminate irrelevant features.

- Use Cases:

- When you suspect that only a few features are truly important.

- In scenarios where interpretability matters (e.g., identifying key predictors).

2. L2 Regularization (Ridge):

- L2 regularization adds a penalty term to the loss function based on the squared magnitude of the coefficients.

- It discourages large weights and promotes more evenly distributed weights.

- Effect on Model Weights:

- L2 regularization shrinks all coefficients proportionally.

- It doesn't force any coefficients to exactly zero.

- Useful for improving generalization by preventing overfitting.

- Use Cases:

- When you want to prevent collinearity (correlation) among features.

- In most scenarios where robustness and generalization are important.

Q3. Some commonly used loss functions for regression:

1. Mean Squared Error (MSE): MSE calculates the average of the squared differences between predicted values and actual observations. Widely used in linear regression and other regression models. Penalizes large errors more heavily due to the squaring operation.

2. Mean Absolute Error (MAE): MAE computes the average of the absolute differences between predicted values and actual observations. Robust to outliers and suitable when you want to minimize the impact of extreme errors. Treats all errors equally without squaring them.

3. Huber Loss: Huber loss combines the benefits of both MSE and MAE. It behaves like MSE for small errors and like MAE for large errors. Robust to outliers and balances the trade-off between MSE and MAE. Quadratic loss for small errors and linear loss for large errors.

4. Root Mean Squared Error (RMSE): RMSE is the square root of MSE. Commonly used for evaluating model performance. Provides an interpretable metric in the same units as the target variable.

5. Log-Cosh Loss: Log-Cosh loss is a smooth approximation of the Huber loss. Balances robustness and smoothness. Similar to Huber loss but with a logarithmic term.

Q4. The learning rate is a crucial hyperparameter in the gradient descent algorithm and other optimization algorithms used in machine learning. It determines the size of the steps that the algorithm takes to reach a minimum of the cost function.

Q5. Key steps and methods to evaluate a linear regression model:

1. Visual Inspection

•Scatter Plot of Actual vs. Predicted Values: Plot the actual values against the predicted values. Ideally, the points should be close to a 45-degree line, indicating that the predictions are close to the actual values.

• Residual Plot: Plot the residuals (errors) against the predicted values. The residuals should be randomly distributed without any clear pattern, indicating that the model is capturing the underlying trend in the data.

2. Statistical Metrics

• R-squared (Coefficient of Determination): Measures the proportion of the variance in the dependent variable that is predictable from the independent variables. Values range from 0 to 1, with higher values indicating better fit.

• Adjusted R-squared: Similar to R-squared but adjusts for the number of predictors in the model. It is useful when comparing models with different numbers of predictors.

• Mean Absolute Error (MAE): The average of the absolute errors between the predicted and actual values.

• Mean Squared Error (MSE): The average of the squared errors between the predicted and actual values. It gives more weight to larger errors.

• Root Mean Squared Error (RMSE): The square root of the MSE, providing error units in the same scale as the dependent variable.

3. Assumption Checks

• Linearity: Check if the relationship between the independent and dependent variables is linear.

• Independence: Ensure that the residuals are independent of each other. This can be checked using the Durbin-Watson test.

• Homoscedasticity: Check that the residuals have constant variance across all levels of the independent variables. This can be assessed using residual plots.

• Normality: Verify that the residuals are normally distributed. This can be checked using histograms, Q-Q plots, or statistical tests like the Shapiro-Wilk test.

4. Cross-Validation

• K-Fold Cross-Validation: Split the data into k subsets, train the model on k-1 subsets, and validate it on the remaining subset. Repeat this k times and calculate the average performance metric.

5. Model Comparison

• F-test: Compare the fits of different linear models. It tests whether a complex model provides a significantly better fit than a simpler model.

Q6. Sensitivity to Predictors:

Multiple R^2 : Always increases or remains the same when more predictors are added, even if those predictors are irrelevant.

Adjusted R^2 : Can decrease if the added predictors do not improve the model sufficiently, providing a penalty for adding irrelevant predictors.

Use Case:

Multiple 𝑅^2 : Suitable for understanding the overall proportion of variance explained by the model but can be misleading in models with many predictors.

Adjusted 𝑅^2 : More reliable for comparing models with different numbers of predictors and ensuring that only relevant predictors are included.