Supervised Learning - Prediction Week 3 Challenge
- Perform Gradient Descent in Python with any loss function
- Code Attached
- Difference between L1 & L2 Gradient descent method:
- Only the loss function and it's corresponding derivative change when L1 loss is replaced by L2 loss in gradient descent method.
- What are the different loss functions for regression:
- L1: Sum of absolute difference between y_true and y_predicted = Sum(|y_true - y_predicted|)
- L2: Sum of squared difference between y_true and y_predicted = Sum((y_true - y_predicted) * (y_true - y_predicted))
- y_true and y_predicted are true values and predicted values for a set of training samples
- L1 is robust but unstable and L2 is not that robust but stable. In practice many use L2. Only when outliers are more L1 is preferred
- What is the importance of learning rate:
- Learning rate is the hyper parameter in gradient descent which determines how the weights change w.r.t each iteration. When the learning rate is high the weights change fast so that the weights reach the optimum sooner. When the learning rate is low the weights change slowly and so the training may take more time. When the learning rate is high there is a chance of missing the global mimimum optimum and the gradient descent may not converge. When the learning rate is low then there is a chance of gradient descent stuck in local minima. In practice many use adaptive learning rate to avoid both the issues.
- How to evaluate linear regression:
- Mean Absolute Error or L1 error
- Mean sqaure loss or L2 error
- Mean absolute percentage error: 100 * mean(abs(y_true - y_predicted) / y_true)
- Mean Sqaure logarithmic error: mean(square(log(y_true + 1) - log(y_predicted + 1)))
- Cosine Similarity: -sum(l2_norm(y_true) * l2_norm(y_predicted))
- What is the difference between multiple and adjusted coefficient of determination:
- R-squared or multiple coefficient of determination is used to check how closely the model has been fit to the original data. More the R-sqaure better the fit.
- R-square = 1 - (SSR/SST); It is ratio of Sum of squared residual to sum of squared residual from the average value.
- When this ratio is low it is a good fit which makes R-square = 1. When the ratio is high the model is not a good fit and R-square value
- decreases. The drawback with R-square is the value increases when new independent variables even though if it is not a good fit.
- So we use Adjusted R-square which is penalizes the value when more independent variables are added and so it is a better parameter
- to evaluate the model.