Supervised Learning - Prediction Week 3 Challenge

1. Perform Gradient Descent in Python with any loss function

import numpy as np

def gradient_descent(x,y):
m_curr = b_curr = 0
iterations=1000
n=len(x)
learning_rate=0.001

for i in range(iterations):
y_p = m_curr * x + b_curr
cost= (1/n) * sum([val**2 for val in (y - y_p)])
md = -(2/n) * sum(x * (y - y_p))
bd = -(2/n) * sum(y - y_p)
m_curr = m_curr - learning_rate * md #calculation of slope
b_curr = b_curr -learning_rate * bd #calculation of coeeficent
print(" m {} , b {} , cost {} , iterations {} , " .format(m_curr,b_curr,cost,iterations))

x = np.array([1,2,3,4,5])
y = np.array([5,7,9,11,13])

gradient_descent(x,y)

2. Difference between L1 & L2 Gradient descent method

Mean absolute error is another function used for regression models. MAE is the sum of absolute differences between our target and predicted values. It measures the average magnitude of errors in set prediction
without considering their directions

L1 regularization penalizes the sum of absolute values of the weights, whereas L2 regularization penalizes the sum of squares of the weights.
The L1 regularization solution is sparse. The L2 regularization solution is non-sparse.
L2 regularization doesn’t perform feature selection, since weights are only reduced to values near 0 instead of 0. L1 regularization has built-in feature selection.
L1 regularization is robust to outliers, L2 regularization is not.

3. What are the different loss functions for regression

The different loss functions used are as follows:

1. Mean square error
2. Mean absolute error
3. Root mean square error
4. Huber loss
5. Log cosh loss
6. Quantile loss(used to find the range of prediction)

4. What is the importance of the learning rate

Specifically, the learning rate is a configurable hyperparameter used in the training of neural networks that has a small positive value, often in the range between 0.0 and 1.0.

The learning rate controls how quickly the model is adapted to the problem. Smaller learning rates require more training epochs given the more minor changes made to the weights of each update, whereas larger learning rates result in rapid changes and require fewer training epochs.

A learning rate that is too large can cause the model to converge too quickly to a suboptimal solution, whereas a learning rate that is too small can cause the process to get stuck.

The challenge of training deep learning neural networks involves carefully selecting the learning rate. It may be the most important hyperparameter for the model.

5. How to evaluate linear regression

Linear regression can be evaluated by determining how good the fit is ie; the error between the predicted value and data set

In statistics the coefficient of determination denoted R2 or r2 pronounce "R square" the proportion of the variance in the dependent variable that is predictable from the independent variable(s)

For the data it can be found as below:

R2=SSR/SST

The Sum of square total denoted SST, is the squared differences between the observed dependent variable and its mean

The Total SS (TSS or SST) tells you how much variation there is in the dependent variable.
Total SS = Σ(Yi – mean of Y)2.

The second term is the sum of squares due to regression or SSR it is sum of the differences between the predicted value and the mean of the dependent variable.

6. What is the difference between multiple and adjusted coefficients of determination

When dependent variables depend on only one independent variable then it is known as the linear coefficient of determination, But when it depends on multiple independent variables then it is multiple coefficients of determination

The adjusted coefficient of determination is the method in which the parameters which do not contribute towards the predicted value are ignored, These parameters are called insignificant independent variables

Adding new variables to the system will affect the R-squared value and the adjusted R-square discards the infinitesimally small increments and only considers the parameters which contribute to output in a significant way

n=number of data points

k=number of independent points

Case 1: If added variables are insignificant the K means will increase with no effect on R2
Case 2: If added variables are insignificant that means k will increase with a significant increase in R2