Supervised Learning - Prediction Week 3 Challenge

Prediction Week 3 Challenge

1. Perform Gradient Descent in Python with any loss function.

It is an iterative first-order optimisation algorithm used to find a local minimum of a given function,until the function is close to or equal to zero, the model will continue to adjust its parameters to yield the smallest possible error. This method is commonly used in machine learning and deep learning to minimise a cost/loss function in a linear regression.

The starting point is just an arbitrary point for us to evaluate the performance. From that starting point, we will find the derivative (or slope), and from there, we can use a tangent line to observe the steepness of the slope. The slope will inform the updates to the parameters—i.e. the weights and bias. The slope at the starting point will be steeper, but as new parameters are generated, the steepness should gradually reduce until it reaches the lowest point on the curve, known as the point of convergence.

Similar to finding the line of best fit in linear regression, the goal of gradient descent is to minimize the cost function, or the error between predicted and actual y. In order to do this, it requires two data points—a direction and a learning rate. These factors determine the partial derivative calculations of future iterations, allowing it to gradually arrive at the local or global minimum (i.e. point of convergence).

• Learning rate (also referred to as step size or the alpha) is the size of the steps that are taken to reach the minimum. This is typically a small value, and it is evaluated and updated based on the behavior of the cost function. High learning rates result in larger steps but risks overshooting the minimum. Conversely, a low learning rate has small step sizes. While it has the advantage of more precision, the number of iterations compromises overall efficiency as this takes more time and computations to reach the minimum.

• The cost (or loss) function measures the difference, or error, between actual y and predicted y at its current position. This improves the machine learning model's efficacy by providing feedback to the model so that it can adjust the parameters to minimize the error and find the local or global minimum. It continuously iterates, moving along the direction of steepest descent (or the negative gradient) until the cost function is close to or at zero. At this point, the model will stop learning. Additionally, while the terms, cost function and loss function, are considered synonymous, there is a slight difference between them. It’s worth noting that a loss function refers to the error of one training example, while a cost function calculates the average error across an entire training set.

Attached the python file below, from the data set the best fit line with the minimum cost is 180.86 and corresponding m is 14.4 and b is 69.1 with learing rate 0.003 reached with iteration 6676 after this value of the cost funtion remains same.

2. Difference between L1 & L2 Gradient descent method.

Mean Absolute Error (L1) :

Mean Absolute Error(MAE) is the mean size of the mistakes in collected predictions. We know that an error basically is the absolute difference between the actual values and the values that are predicted. The absolute difference means that if the result has a negative sign, it is ignored.

Hence, MAE = Actual values – Predicted values

MAE = (1/n) * Σ|actual – Predicted |

MAE takes the average of this error from every sample in a dataset and gives the output.

Mean Squared Error (L2):

The Mean Absolute Error is the squared mean of the difference between the actual values and predictable values.

Lesser the MSE => Smaller is the error => Better the estimator.

The Mean Squared Error is calculated as:

MSE = (1/n) * Σ(actual – Predicted )2

L1 Regularization	L2 Regularization
Sum of absolute value of weights	Sum of squares of wieghts.
Sparse solution is the outcome	Non sparse solution
Multiple solution are possible	Only one soution in possible
Robust to outliers	Not so robust to outlier due to square term
Used in dataset with sparse features	Used in data set with complex features

3.What are the different loss functions for regression.

The loss functions are a measure of a machine learning model’s accuracy in predicting the predicted outcome. The cost function and loss function both relate to the same thing: the training process that employs backpropagation to reduce the difference between the actual and anticipated outcome A loss function measures how well a given machine learning model fits the specific data set. if predictions of model are perfect then loss would be very less otherwise the loss would be greater. Regardless of the problem statement, there should always be performance evaluation for comparing the predicted values with the actual values, aim would be to minimize the cost function.

Different loss function are

• Mean Absolute Error
• Mean Squared Error
• Root Mean Squared Error
• Huber Loss
• Log-loss
• Quantile Loss

Mean Absolute Error :

Mean Absolute Error(MAE) is the mean size of the mistakes in collected predictions. We know that an error basically is the absolute difference between the actual values and the values that are predicted. The absolute difference means that if the result has a negative sign, it is ignored. Even for small error there is a drastic change in slope of the regression line.

Hence, MAE = Actual values – Predicted values

MAE = (1/n) * Σ|actual – Predicted |

MAE takes the average of this error from every sample in a dataset and gives the output.

Mean Squared Error :

The Mean Absolute Error is the squared mean of the difference between the actual values and predictable values. Magnifies errors and disturbs loss funtions in case of outliers.

Lesser the MSE => Smaller is the error => Better the estimator.

The Mean Squared Error is calculated as:

MSE = (1/n) * Σ(actual – Predicted )2

Root Mean Squared Error:

Root Mean Squared Error is the square root of Mean square error. As we are applying square root to MSE, RMSE comes down to the same unit as the target variable. The loss function of RMSE is exactly the same as MSE, we just brought down the order from 2 to 1 so that correlation can be easily done. The correlation becomes difficult in MSE because it severely penalizes large errors.

RMSE = √( (1/n) * Σ(actual – Predicted )2)

Huber Loss:

Huber Loss combines the robustness of L1 with the stability of L2, essentially the best of L1 and L2 losses. For huge errors, it is linear and for small errors, it is quadratic in nature. Huber loss approaches MSE for small error and approaches MAE in case of outliers.

Log Loss:

Logarithmic loss indicates how close a prediction probability comes to the actual/corresponding true value.Log Loss is the most important classification metric based on probabilities. It’s hard to interpret raw log-loss values, but log-loss is still a good metric for comparing models. For any given problem, a lower log loss value means better predictions.

Quantile Loss:

A quantile is a value below which a fraction of samples in a group falls. Machine learning models work by minimizing (or maximizing) an objective function. As the name suggests, we apply the quantile regression loss function to predict quantiles. For a set of predictions, the loss will be its average.Quantile loss function turns out to be useful when we’re interested in predicting an interval instead of only point predictions.

4.What is the importance of learning rate.