Supervised Learning - Prediction Week 3 Challenge

Ans No 1:-

The loss function to be used is the mean square error which is defined as:

MSE=1/n∑(ya−yp)**2

n = total number of data points

yp = predicted outcome

ya = actual outcome

• Gradient descent method

Gradient descent is an optimization algorithm. It's based on a convex function and tweaks its parameters iteratively to minimize the loss function to its local minimum

Let us define the y_predicted as:

y_predicted = mx + b

where m,b are constants.

Therefore, we can rewrite MSE as:

MSE=1/n∑(ya−(mx+b))**2

To find the local minima, the gradient of MAE wrt to b and m is calculated:

∂M/∂b=2/n∑(−1)(ya−(mx+b))**2

∂M/∂m=2/n∑i=1(−x)(ya−(mx+b))**2

Using these differentials, we iterate the values of b and m as follows:

b=b−learning_rate⋅∂M/∂b

m=m−learning_rate⋅∂M/∂m

• Program algorithm:
  • Intialize arrays x and y
  • Assume and initialize b and m
  • Define tolerance. Since the given data set is linear, we can consider a tolerance of 1e-6
  • initialize learning rate as 0.001
  • Using while loop, perform gradient descent steps till MSE is less than tolerance value
  • Print m and b

Ans No 2:- Difference between L1 & L2 Gradient descent method

Mean Absollute Error, L1-(MAE) is another loss function used for regression models. MAE is the sum of absolute differences between our target and predicted variables. So it measures the average magnitude of errors in a set of predictions, without considering their directions. (If we consider directions also, that would be called Mean Bias Error (MBE), which is a sum of residuals/errors). The range is also 0 to ∞.

MAE=(1/n)∑|yi-yi**p|">MAE=(1/n)∑∣∣yi−yi**p∣

Mean Squared Error, L2-(MSE) is the most commonly used regression loss function. MSE is the sum of squared distances between our target variable and predicted values.

MSE=(1/n)∑(yi-yi**p)**2">MSE=(1/n)∑(yi−yi**p)**2

Since MSE squares the error (y — y_predicted = e), the value of error (e) increases a lot if e > 1. If we have an outlier in our data, the value of e will be high and e² will be >> |e|. This will make the model with MSE loss give more weight to outliers than a model with MAE loss.

Ans No 3:- What are the different loss functions for regression

The different loss functions used are as follows:

1. Mean Square Error
2. Mean Absolute Error
3. Root Mean Square Error
4. Huber loss
5. Log-cosh loss

Ans No 4:- What is the importance of learning rate

The learning rate is a configurable hyperparameter used in the training of neural networks that has a small positive value, often in the range between 0.0 and 1.0.

The learning rate controls how quickly the model is adapted to the problem. Smaller learning rates require more training epochs given the smaller changes made to the weights each update, whereas larger learning rates result in rapid changes and require fewer training epochs.

A learning rate that is too large can cause the model to converge too quickly to a suboptimal solution, whereas a learning rate that is too small can cause the process to get stuck.

Ans No 5:- How to evaluate linear regression

Linear regression can be evaluated by determining how good the fit is i.e., the error between the predicted value and the data set.

In statistics, the coefficient of determination, denoted R² or r² and pronounced "R squared", is the proportion of the variance in the dependent variable that is predictable from the independent variable(s).

For the data it can be found as below:

R2=SSR/SST">R**2=SSR/SST

The sum of squares total, denoted SST, is the squared differences between the observed dependent variable and its mean.

SST=∑(yi-y¯)**2">SST=∑(yi−¯y)**2

The second term is the sum of squares due to regression, or SSR. It is the sum of the differences between the predicted value and the mean of the dependent variable.

SSR=∑(yiP-y¯)**2">SSR=∑(yi**P−¯y)**2

Ans No6:- What is the difference between multiple and adjusted coefficient of determination

When the dependent variable depends on only one independent variable then it is known as the linear coefficient of determination, but when it depends on multiple independent variables then it is multiple coefficient of determination.

The adjusted coefficient of determination is the method in which the parameters which do not contribute towards the predicted value are ignored. These parameters are called insignificant independent variables.

Adding new variables to the system will affect the R-squared value and the adjusted R-Squared discards the infinitesimly small increments and only considers the parameters which contribute to the output in a significant way.

Radj**2=(1-(1-R2)⋅(n-1))/(n-k-1)">Radj**2=1−((1−R2)⋅(n−1)/(n−k−1)

n = number of data points

k = number of independent points

Case 1: If added variables are insignificant, that means k will increase with little or no effect on R2">R**2, hence R2_(adj)">R**2_(adj) will decrease.

Case 2: If added variables are insignificant, that means k will increase with a significant increase on R2">R**2, hence R2_(adj)">R**2_(adj) will increase.