Supervised Learning - Prediction Week 3 Challenge

Q1.

in ipython file

Q2.

The key difference between these techniques is that the less important feature’s coefficient to zero thus, removing some feature altogether. So, this works well for feature selection in case we have a huge number of features.

• L1 regularization penalizes the sum of absolute values of the weights, whereas L2 regularization penalizes the sum of squares of the weights
• The L1 regularization solution is sparse. The L2 regularization solution is non-sparse
• L2 regularization doesn’t perform feature selection since weights are only reduced to values near 0 instead of 0. L1 regularization has built-in feature selection
• L1 regularization is robust to outliers, L2 regularization is not

L2 regularization, or the L2 norm, or Ridge , combats overfitting by forcing weights to be small, but not making them exactly 0.
So, if we’re predicting house prices again, this means the less significant features for predicting the house price would still have some influence over the final prediction, but it would only be a small influence.

Q3.

The terms cost and loss functions almost refer to the same meaning.

 But, loss function mainly applies for a single training set as compared to the

 cost function which deals with a penalty for a number of training sets or the complete batch. It is also sometimes called an error function.

The optimization strategies aim at “minimizing the cost function”.

• Mean squared error (MSE)

Mean squared error (MSE) is the most commonly used loss function for regression. The loss is the mean overseen data of the squared differences between true and predicted values, or writing it as a formula.

L(y,y^)=1N∑i=0N(y−y^i)2L(y, \hat{y}) = \frac{1}{N} \sum_{i=0}^{N}(y - {\hat{y}}_i)^2L(y,y^​)=N1​i=0∑N​(y−y^​i​)2

where ŷ is the predicted value.

• The RMSE can be calculated by taking the square root of above mentioned Mean Squared Errors (MSE) / L2 Loss. The effect of each error on RMSE is proportional to the size of the squared error. It represents the square root of the second sample moment of the deviation between predicted model values and actual values

• Mean absolute error (MAE)

  Mean absolute error (MAE) is a loss function used for regression. The loss is the mean overseen data of the absolute differences between true and predicted values

Q4.

The amount that the weights are updated during training is referred to as the step size or the “learning rate.”

• Learning rate controls how quickly or slowly a neural network model learns a problem.
• How to configure the learning rate with sensible defaults, diagnose behavior, and develop a sensitivity analysis.
• How to further improve performance with learning rate schedules, momentum, and adaptive learning rates.

Learning rate controls how quickly or slowly a neural network model learns a problem.How to configure the learning rate with sensible defaults, diagnose behavior, and develop a sensitivity analysis.How to further improve performance with learning rate schedules, momentum, and adaptive learning rates.

If the gradient is less than 0, we increase the parameters by the value of the gradient multiplied by learning rate α.

If the gradient is greater than 0, we decrease the parameters by the value of the gradient multiplied by learning rate α.

Q5.

evaluating regression Models

There are three main errors (metrics) used to evaluate models, Mean absolute error, Mean Squared error and R2 score.

Mean Squared Error

Mean Squared Error solves differentiability problem of the MAE. Consider the same diagram above. We have a line that fits those points. When we do a summation of the square of distances from the points to the line, we get Mean squared error.

R2 score

Let us take a naive approach by taking an average of all the points by thinking of a horizontal line through them. Then we can calculate the MSE for this simple model.

R2 score answers the question that if this simple model has a larger error than the linear regression model. However, it terms of metrics the answer we need is how much larger. The R2 score answers this question. R2 score is 1 — (Error from Linear Regression Model/Simple average model).

Best possible score is 1.0 and it can be negative (because the model can be arbitrarily worse). A constant model that always predicts the expected value of y, disregarding the input features, would get a R² score of 0.0.

Mean absolute error

We have a line that fits those points. When we do a summation of the absolute value distance from the points to the line, we get Mean absolute error. 

Use Mean absolute error when you are doing regression and don’t want outliers to play a big role. It can also be useful if you know that your distribution is multimodal, and it’s desirable to have predictions at one of the modes, rather than at the mean of them.

Q6.

I wont go into the real maths of it (as I don't understand it myself), but I can explain it in more general terms.

Multiple R squared is simply a measure of Rsquared for models that have multiple predictor variables. Therefore it measures the amount of variation in the response variable that can be explained by the predictor variables. The fundamental point is that when you add predictors to your model, the multiple Rsquared will always increase, as a predictor will always explain some portion of the variance.

Adjusted Rsquared controls against this increase, and adds penalties for the number of predictors in the model. Therefore it shows a balance between the most parsimonious model, and the best fitting model. Generally, if you have a large difference between your multiple and your adjusted Rsquared that indicates you may have overfit your model.