Supervised Learning - Prediction Week 3 Challenge
Ques 1 : Gradient Descent is an optimization algorithm in machine learning used to minimize a function by iteratively moving towards the minimum value of the function. We use the algorithm when we have to find the least possible values that can satisfy a given cost function. In machine learning, more often than not we try to minimize loss functions. By minimizing the loss function, we can improve our model, and Gradient Descent is one of the most popular algorithms used for this purpose.
x = [1,2,3,4,5]
y = [20,40,60,80,100]
n=len(x)
#y_predicted = mx + b
#Assume m and b
m= 0
b = 0
#tolerance
tol=1e-6
#loss function
mse= 1e9
learning_rate = 0.001
iteration = 1
while mse>tol :
square_error = 0
d_dm = 0
d_db = 0
for i in range(0,n):
square_error = square_error + pow((y[i] - (m*x[i]+b)),2)
d_dm = d_dm + 2/n*(-x[i]*(y[i] - (m*x[i]+b)))
d_db = d_db + 2/n*(-1*(y[i] - (m*x[i]+b)))
mse = square_error/n
print("Iteration ="+ str(iteration) + " Mean Square Error="+str(mse))
print("m = "+ str(m) + " b = " + str(b) )
b= b - learning_rate*d_db
m = m - learning_rate*d_dm
iteration = iteration + 1
print("The parameters for the best fit line are:" + "b = " + str(b) + " m = "+ str(m))
Output : Iteration =22753 Mean Square Error=1.0005611871180232e-06
m = 19.99935058863212 b = 0.00234458109512576
Iteration =22754 Mean Square Error=9.998847295998687e-07
m = 19.99935080819564 b = 0.002343788401142791
The parameters for the best fit line are:b = 0.0023429959751666628 m = 19.99935102768493
Ques 2 : Mean Absolute Error, L1-(MAE) is another loss function used for regression models. MAE is the sum of absolute differences between our target and predicted variables. So it measures the average magnitude of errors in a set of predictions, without considering their directions. The range is also 0 to ∞.
MAE=(1n)n∑i=1∣∣yi−ypi∣∣
Mean Squared Error, L2-(MSE) is the most commonly used regression loss function. MSE is the sum of squared distances between our target variable and predicted values.
MSE=(1n)n∑i=1(yi−ypi)2
Since MSE squares the error (y — y_predicted = e), the value of error (e) increases a lot if e > 1. If we have an outlier in our data, the value of e will be high and e² will be >> |e|. This will make the model with MSE loss give more weight to outliers than a model with MAE loss.
Ques 3 : The different loss functions used are as follows:
- Mean Square Error
- Mean Absolute Error
- Root Mean Square Error
- Huber loss
- Log-cosh loss
- Quantile loss (this is used to find range of predictions)
Ques 4 : The learning rate is a configurable hyperparameter used in the training of neural networks that has a small positive value, often in the range between 0.0 and 1.0.
The learning rate controls how quickly the model is adapted to the problem. Smaller learning rates require more training epochs given the smaller changes made to the weights each update, whereas larger learning rates result in rapid changes and require fewer training epochs.
A learning rate that is too large can cause the model to converge too quickly to a suboptimal solution, whereas a learning rate that is too small can cause the process to get stuck.
The challenge of training deep learning neural networks involves carefully selecting the learning rate. It may be the most important hyperparameter for the model.
Ques 5 : Linear regression can be evaluated by determining how good the fit is i.e., the error between the predicted value and the data set.
In statistics, the coefficient of determination, denoted R2 or r2 and pronounced "R squared", is the proportion of the variance in the dependent variable that is predictable from the independent variable(s).
For the data it can be found as below:
R2 = SSR/SST
The sum of squares total, denoted SST, is the squared differences between the observed dependent variable and its mean.
SST = n∑i=1(yi−¯y)2
The second term is the sum of squares due to regression, or SSR. It is the sum of the differences between the predicted value and the mean of the dependent variable.
SSR = n∑i=1(yPi−¯y)2