Supervised Learning - Prediction Week 3 Challenge

Question 1-  Perform Gradient Descent in Python with any loss function

Answer-

Answer in .ipynb file and in screenshots

Question 2- Difference between L1 & L2 Gradient descent method

Answer- 

->L1 Regularization, also called a lasso regression, adds the “absolute value of magnitude” of the coefficient as a penalty term to the loss function.

->L2 Regularization, also called a ridge regression, adds the “squared magnitude” of the coefficient as the penalty term to the loss function.

Question 3 - What are the different loss functions for regression

Answer-

Loss Functions used in Regression:

Regression or Regression Analysis is a supervised learning technique that can be defined as statistical techniques that are used to model the relationship between the dependent real numbered variable “y” and the independent variables “xi”. Based on different scenarios and their importance, we use different types of regression methods. Let’s understand the various possible loss functions that can be used for improving the performance evaluation in regression analysis.

1-Mean Squared Error / L2 Loss:

This is the most used loss function as it is very easy to understand and implement. It works well with almost all regression problems. As the name says, “the mean squared error is the mean of the squared errors”. Here error is the difference between the model-predicted value “y_hat” and the actual value “y_real”.

-> Squaring is to magnify the large errors and also to observe the variance of our losses.

1-Mean Absolute Error / L1 Loss:

Mean absolute error is the mean of the absolute difference between the model-predicted value and the actual value. Mean absolute error is more robust in the case of outliers because it’s not good to square the difference as it will be too huge.

3- Root Mean Squared Error:

Root Mean Squared Error is the square root of Mean square error. As we are applying square root to MSE, RMSE comes down to the same unit as the target variable. The loss function of RMSE is exactly the same as MSE, we just brought down the order from 2 to 1 so that correlation can be easily done. The correlation becomes difficult in MSE because it severely penalizes large errors.

 

4- Mean Bias Error:

Mean Bias Error is the exact difference between the predicted value and the actual value without any math function like absolute or square root applied to it. “The major limitation that MBE has is positive and negative errors have a chance of canceling out”. That is why this is the rarely used and less popular loss function.

5- Huber Loss:

As we already discussed, there are drawbacks in both MSE and MAE, so here is where Huber's loss comes into the picture. It takes only beneficial things from both MSE and MAE and comes them into a single loss function.

Question 4- What is the importance of learning rate

Answer:

This parameter determines how fast or slow we will move towards the optimal weights. If the learning rate is very large we will skip the optimal solution. If it is too small we will need too many iterations to converge to the best values. So using a good learning rate is crucial.

Question 5- How to evaluate linear regression

Answer: