Implementing Gradient Descent in Python with MSE loss function
1.
Aim: To perform Gradient Descent in Python with any loss function.
Objective: To perform gradient descent in python using mean square error as loss function on the following data set:
- y = [1,2,3,4,5]
x = [0,1,2,3,4]
Theory:
Loss function:
J = `(1/n)sum(y(i) - (mx(i)+b))^2`, where
x(i) are the input
m and b are the weights
mx(i)+b is the predicted output and y(i) is the actual value
Differentiating the loss function J with respect to m and b, we get
dJ/dm = `(2/n)sum-x(i)(y(i)-(mx(i)+b))`
dJ/db = `(2/n)sum-(y(i)-(mx(i)+b))`
m and b values are then optimized
m = m - alpha*dJ/dm
b = b - alpha*dJ/db
alpha is the learning rate, i.e. how large a step the gradient takes.
Python code:
import numpy as np
import matplotlib.pyplot as plt
from scipy.optimize import curve_fit
from statistics import mean
#Initialising data
y = [1,2,3,4,5]
x = [0,1,2,3,4]
#fit function y = mx+b
n = len(y) #number of elements
#Initialising parameters
m = 0
b = 0
#Learning rate alpha
alp = 0.0008
#Cost function
J = 9e9
tol = 1e-4 #Error tolerance
iter = 1 #Iterations
while (J>tol): #If error is greater than tolerance while loop is executed
term1 = []; #Empty array initialised
d_m = []; #Gradient of parameter m
d_b = []; #Gradient of parameter b
for i in range(0,n):
term1.append(pow(y[i]-m*x[i]-b,2))
d_m.append(-x[i]*(y[i]-m*x[i]-b))
d_b.append(-y[i]+m*x[i]+b)
print("nNumber of iterations:",iter)
iter = iter+1
J = (1/n)*sum(term1)
print("nError:",J)
print("nParameters:")
print("nm value:",m)
print("nb value:",b)
m = m - alp*(2/n)*sum(d_m)
b = b - alp*(2/n)*sum(d_b)
Code Explanation:
Necessary libraries are imported to execute the program:
Initial data and parameters are initialized:
Main body:
If the condition J>tol is true, then the while loop will be executed.
Inside the while loop, there is a for loop which fetches each value present in x and y and calculates the error and change in the loss function with respect to the parameters m and b
After exiting the for loop, the sum of the terms are calculated and this process keeps repeating till the error is lesser than the given tolerance value
Result:
For a learning rate of 0.0008, it takes 7352 iterations for the error to be lesser than the tolerance and the m and b values are found to be 1.006 and 0.982 which are mathematically correct.
2.
L1 Mean absolute error: finds the mean of the absolute difference between the predicted and actual value. It is good for data set with little or no outliers. However, it needs more complicated tools such as linear programming to calculate the gradients of the parameters.
L2 Mean squared error: Finds the mean of the square of the difference between the predicted and actual value. It is good for data set with outliers, however returns an error in different units. Due to its mathematical properties, it is easier to calculate the gradients of the parameters. The squaring means that larger mistakes result in more error than smaller mistakes, meaning that the model is punished for making larger mistakes.
Finding the root of the mean squared error, i.e. using root mean square error bring the error term to match with the data set in terms of units.
3. What are the different loss functions for regression
Root mean square error
Mean square error
Mean absolute error
4. What is the importance of learning rate
A.
Let F be a function that determines the error between the data set and the predicted values. The predicted values depend on two parameters m, and c. Gradient descent is a method to determine the values of the parameters for which F returns the least value.
Learning rate is an important factor that determines how fast the function parameter approaches a nominal value. Gradient descent is a method to determine the values of the function parameters for which the function returns the minimum error.
Giving a high learning rate will cause the function to overshoot beyond its lowest value and will never reach its lowest value or will oscillate before arriving at one.
Giving a low learning rate will cause the function to approach the lowest value very slowly leading to a high number of iterations.
5. How to evaluate linear regression
Linear regression can be evaluated by determining how good the fit is, i.e. what is the error between the predicted value and the data set.
R-squared :
Assume function f(x) to satisfy the data points (x1,y1), (x2,y2).. and so on.
Sum of squared error SSE = `sum_(i=1)^N(y(i)-f(x(i)))^2`
If the proposed equation is an nth degree polynomial
`f(x) = A1+ A2*x+A3*x^2+..+underset(" "n)(A)*x^(n-1)`
Differentiating the SSE with each coefficient and equating it to zero will return us the value of the coefficient for which the error is minimum.
Solving the multiple systems of equations we can get the value of the coefficients.
This is the least square error method.
R-square:
`"R square" = (SSR)/(SST)`, where
`SST = SSE + SSR`
SSR stands for the sum of the square of the regression and is equal to the square of the difference between the value of the fitting function at the given point and the mean of the data set.
SSR = `sum_(i=1)^N(f(x(i))-M)^2`, where M is the arithmetic mean of the given data set.
6. What is the difference between multiple and adjusted coefficient of determination?
When the dependent variable depends on only one independent variable then it is known as the linear coefficient of determination, but when it depends on multiple Independent variables then it is multiple coefficient of determination.
The Adjusted coefficient of determination is the method in which the parameters which do not contribute towards the predicted value are ignored. These parameters are called insignificant independent variables.
Adding new variables to the system will affect the R-squared value and the adjusted R-squared disregards the infinitesimally small increments and only considers the parameters which contribute to the output in a significant way.
`underset(" "adj)(R^2) = 1 - ((1-R^2)(n-1))/(n-k-1)`, where
n = number of data points
k = number of independent variables
Case 1: If added variables are insignificant, that means k will increase with little or no effect on `R^2`, hence `underset(" "adj)(R^2)` will decrease.
Case 2: If added variables are significant, that means k will increase with a significant increase in `R^2`, hence `underset(" "adj)(R^2)` will also increase.
https://skill-lync.com/projects/Curve-Fitting-and-criterion-to-choose-best-fit-41860