Supervised Learning - Prediction Week 3 Challenge
Q1.
Gradient Descent is an optimization algorithm in machine learning used to minimize a function by iteratively moving towards the minimum value of the function. We use the algorithm when we have to find the least possible values that can satisfy a given cost function. In machine learning, more often than not we try to minimize loss functions. By minimizing the loss function, we can improve our model, and Gradient Descent is one of the most popular algorithms used for this purpose.
We first take a point in the cost function and start moving in steps towards the minimum point. The size of that step, or how quickly we have to converge to the minimum point is defined by Learning Rate. We can cover more areas with higher learning rates but at the risk of overshooting the minima. On the other hand, small steps/smaller learning rates will consume a lot of time to reach the lowest point.
Q2.
Mean Absollute Error, L1-(MAE) is another loss function used for regression models. MAE is the sum of absolute differences between our target and predicted variables. So it measures the average magnitude of errors in a set of predictions, without considering their directions. (If we consider directions also, that would be called Mean Bias Error (MBE), which is a sum of residuals/errors). The range is also 0 to ∞.
MAE :`frac{1}{n} \sum_{i=1}^{n} |y_i - y_{i}^p|`
Mean Squared Error, L2-(MSE) is the most commonly used regression loss function. MSE is the sum of squared distances between our target variable and predicted values.
`MSE : frac{1}{n} \sum_{i=1}^{n} (y_i - y_{i}^p)^2`
Since MSE squares the error (y — y_predicted = e), the value of error (e) increases a lot if e > 1. If we have an outlier in our data, the value of e will be high and e² will be >> |e|. This will make the model with MSE loss give more weight to outliers than a model with MAE loss.
Q3.
The choice of the loss function depends on the nature of the problem, the characteristics of the data, and the desired behavior of the regression model. It's important to choose a loss function that aligns with the specific goals and requirements
Loss functions are used in regression to measure the error or the discrepancy between the predicted values and the actual target values. The choice of the loss function can affect the performance of your regression model.
Mean Squared Error (MSE):
- Formula: MSE = (1/n) * Σ(y_i - ŷ_i)^2
- It's the most commonly used loss function for regression. It penalizes larger errors heavily.
Root Mean Squared Error (RMSE):
- Formula: RMSE = √MSE
- RMSE is the square root of the MSE. It's in the same unit as the target variable, making it more interpretable.
Mean Absolute Error (MAE):
- Formula: MAE = (1/n) * Σ|y_i - ŷ_i|
- MAE is less sensitive to outliers compared to MSE because it doesn't square the errors.
Huber Loss:
- Huber loss is a combination of MSE and MAE. It is less sensitive to outliers than MSE and has some properties of MAE.
- It uses a threshold (δ) to switch between the quadratic loss of MSE and the linear loss of MAE.
Quantile Loss (Quantile Regression):
- Formula: Quantile Loss = Σ(r_i * |y_i - ŷ_i|)
- This loss function is used in quantile regression, where you estimate quantiles of the target variable. "r_i" is a function of the quantile of interest.
Log-Cosh Loss:
- Formula: Log-Cosh Loss = Σlog(cosh(y_i - ŷ_i))
- Log-Cosh loss is smooth and behaves similar to MAE but is differentiable.
Pseudo-Huber Loss:
- Formula: Pseudo-Huber Loss = δ^2 * (sqrt(1 + (ε/δ)^2) - 1)
- This loss function is another alternative to Huber loss. It is smooth like MSE near the origin and MAE for large values.
Tukey's Biweight Loss:
- Tukey's loss function is similar to Huber loss but uses a different weighting function for the errors.
Cosine Proximity Loss:
- This loss function measures the cosine similarity between predicted and actual values.
Kullback-Leibler Divergence Loss (KLD Loss):
- It measures the divergence between the predicted distribution and the actual distribution. It is often used in probabilistic regression models.
Q4.
The learning rate is a configurable hyperparameter used in the training of neural networks that has a small positive value, often in the range between 0.0 and 1.0.
The learning rate controls how quickly the model is adapted to the problem. Smaller learning rates require more training epochs given the smaller changes made to the weights each update, whereas larger learning rates result in rapid changes and require fewer training epochs
The learning rate controls how quickly the model is adapted to the problem. Smaller learning rates require more training epochs given the smaller changes made to the weights each update, whereas larger learning rates result in rapid changes and require fewer training epochs.
A learning rate that is too large can cause the model to converge too quickly to a suboptimal solution, whereas a learning rate that is too small can cause the process to get stuck.
The learning rate is a crucial hyperparameter in training machine learning and deep learning models. It plays a significant role in determining how quickly or slowly a neural network or machine learning algorithm learns. Here's the importance of the learning rate in the context of training models:
Control of Model Convergence:
Learning rate controls the step size taken during each iteration of optimization. A too-large learning rate can lead to overshooting the optimal solution, causing the model to fail to converge. A too-small learning rate can result in extremely slow convergence or getting stuck in local minima.
Stability and Robustness:
The right learning rate helps in stabilizing the training process. It prevents large, erratic updates in the model's parameters, which can lead to instability and poor convergence.
Speed of Convergence:
An appropriately chosen learning rate can speed up training. If the learning rate is set optimally, the model can converge to a good solution faster.
Avoiding Plateaus:
In deep learning, when training large neural networks, setting the right learning rate can help avoid getting stuck on plateaus, where the loss function doesn't change significantly. Plateaus can slow down or halt training.
Hyperparameter Tuning:
Learning rate is an important hyperparameter that often requires tuning to achieve the best performance for a specific problem. Experimenting with different learning rates is a common part of model development.
Q5.
Coefficient of Determination (R-squared):
R-squared is a statistical measure that represents the proportion of the variance in the dependent variable (target) that is explained by the independent variables (features) in the model. It ranges from 0 to 1, where 1 indicates a perfect fit.
A higher R-squared value typically suggests a better fit to the data, but it doesn't provide information about the goodness of fit on its own.
Mean Squared Error (MSE):
MSE measures the average squared difference between the actual target values and the predicted values. It penalizes larger errors more heavily.
Lower MSE values indicate a better fit, and it is often used to compare models or choose the best model.
Root Mean Squared Error (RMSE):
RMSE is the square root of the MSE and is in the same unit as the target variable. It provides a more interpretable measure of the model's performance.
Mean Absolute Error (MAE):
MAE measures the average absolute difference between the actual target values and the predicted values. It is less sensitive to outliers compared to MSE.
MAE is also a commonly used evaluation metric, especially when outliers are present in the data.
Q6.
When the dependent variable depends on only one independent variable then it is known as the linear coefficient of determination, but when it depends on multiple independent variables then it is multiple coefficient of determination.
The adjusted coefficient of determination is the method in which the parameters which do not contribute towards the predicted value are ignored. These parameters are called insignificant independent variables.
Adding new variables to the system will affect the R-squared value and the adjusted R-Squared discards the infinitesimly small increments and only considers the parameters which contribute to the output in a significant way.
`R^2_adj = 1 - (1 - R^2) * (n - 1) / (n - k - 1)`
n = number of data points
k = number of independent points
Case 1: If added variables are insignificant, that means k will increase with little or no effect on
Case 2: If added variables are insignificant, that means k will increase with a significant increase on
R^2 measures the proportion of the total variation in Y explained by the regression model.
Here are some key points about R^2:
- It is a non-negative quantity with a range 0 ≤ R^2 ≤ 1
- R^2 = 0 implies that the regression line does not fit the data at all.
- R^2 = 1 implies that the regression line is a perfect fit.
Adjusted R^2 is a modified version of R^2 adjusted with the number of predictors. It penalizes for adding unnecessary features and allows a comparison of regression models with a different number of predictors. The adjusted R-squared increases only if the new term improves the model more than would be expected by chance. It decreases when a predictor improves the model by less than expected by chance.
Obtaining a negative value for Adjusted R^2 can indicate few or all of the following:
- The linear model is a poor fit for the data
- The number of predictors is large
- The number of samples is small