Supervised Learning - Prediction Week 3 Challenge

import numpy as np

import random

# m denotes the number of examples here.

def gradientDescent(x, y, theta, alpha, m, numIterations):

xTrans = x.transpose()

for i in range(0, numIterations):

hypothesis = np.dot(x, theta)

loss = hypothesis - y

# avg cost per example (the 2 in 2*m doesn't really matter here.

# But to be consistent with the gradient, I include it)

cost = np.sum(loss ** 2) / (2 * m)

print("Iteration %d | Cost: %f" % (i, cost))

# avg gradient per example

gradient = np.dot(xTrans, loss) / m

# update

theta = theta - alpha * gradient

return theta

def genData(numPoints, bias, variance):

x = np.zeros(shape=(numPoints, 2))

y = np.zeros(shape=numPoints)

# basically a straight line

for i in range(0, numPoints):

# bias feature

x[i][0] = 1

x[i][1] = i

# our target variable

y[i] = (i + bias) + random.uniform(0, 1) * variance

return x, y

1. Calculate the hypothesis h = X * theta
2. Calculate the loss = h - y and maybe the squared cost (loss^2)/2m
3. Calculate the gradient = X' * loss / m
4. Update the parameters theta = theta - alpha * gradient
5. Here m denotes the number of examples in our training set.

2. Difference between L1 & L2 Gradient descent method

Solutions:

L1 regularization, also known as L1 norm or Lasso, it actually combats overfitting by shrinking the parameters towards zero. This in turn makes some features obsolete. It’s kind of a form of feature selection, because when we assign a feature with a zero weight, we’re multiplying the feature values by zero which returns zero, eradicating the significance of that feature. If the input features of our model have weights closer to zero, our L1 norm would be sparse. A selection of the input features would have weights equal to zero, and the rest would be non-zero.

For instance, imagine we want to predict housing pricing using ML. Let's consider the following features:

Street – road access,
Neighborhood – property location,
Accessibility – transport access,
Year Built – year the house was built in,
Rooms – number of rooms,
Kitchens – number of kitchens,
Fireplaces – number of fireplaces in the house.

When predicting the value of a house, our generic sense tells us that different input features won’t have the same influence on the price. And it’s highly likely that the neighborhood or the number of rooms have a higher influence on the price of the property than the number of fireplaces. So, our L1 regularization technique would assign the fireplaces feature with a zero weight, because it doesn’t have a significant effect on the pricing. We can expect the neighborhood and the number of rooms to be assigned non-zero weights, because these features influence the pricing of a property significantly. Basically, when we use L1 regularization, we are penalizing the absolute value of the weights. In a real time environment, we usually have features that are highly correlated. So, the year our home was built and the number of rooms in the home may have a high correlation. Something we should consider while using L1 regularization is that when we have highly correlated features, the L1 norm would select only 1 of the features from the group of correlated features in an arbitrary nature, which is something that we might not want. Nonetheless, for our example regression problem, Lasso regression would produce a model that is highly interpretable, and only uses a subset of input features, thus reducing the complexity of the model.

L2 regularization or the L2 norm, or Ridge, combats overfitting by forcing weights to be small, but not making them exactly zero. So, if we’re predicting house pricing again, this would mean that the lesser significant features for predicting the house pricing would still have some influence over the final prediction, but it would only be a small influence. The regularization term that we add to the loss function when performing L2 regularization is the sum of squares of all of the feature weights.

So, L2 regularization returns a non-sparse solution since the weights will be non-zero, although some may be close to zero. A major dilema to consider when using L2 regularization is that it’s not robust to outliers. The squared terms will blow up the differences in the error of the outliers. The regularization would then attempt to fix this by penalizing the weights. So to breifly list down the differences between L1 and L2 regularization:
L1 regularization penalizes the sum of absolute values of the weights, whereas L2 regularization penalizes the sum of squares of the weights.
The L1 regularization solution is sparse. The L2 regularization solution is non-sparse.
L2 regularization doesn’t perform feature selection, since weights are only reduced to values near 0 instead of 0. L1 regularization has built-in feature selection.
L1 regularization is robust to outliers, L2 regularization is not.

3. What are the different loss functions for regression?

Solution:

When we are dealing with numeric variables, we have to measure the losses numerically, meaning, just knowing if the predicted value is wrong is not enough, we have to calculate the amount of deviation of our prediction from the actual value, so we can train our network accordingly.

The different loss functions for this are :

• Mean Absolute Error (MAE).
• Mean Absolute Percentage Error (MAPE).
• Mean Squared Error (MSE).
• Root Mean Squared Error (RMSE).
• Huber Loss.
• Log-Cosh Loss.

4. What is the importance of learning rate?

Solution:

Learning rate gives the rate of speed where the gradient moves during gradient descent. Setting it too high would make your path instable, too low would make convergence slow. Put it to zero means your model isn't learning anything from the gradients. Generally, a large learning rate allows the model to learn faster, at the cost of arriving on a sub-optimal final set of weights. A smaller learning rate may allow the model to learn a more optimal or even globally optimal set of weights but may take significantly longer to train.

5. How to evaluate linear regression?

Solution:

A common question for regression predictive modeling projects is: How do we calculate accuracy for my regression model? Accuracy is definitely a measure for classification but not regression. But we cannot calculate accuracy for a regression model. The requisite skill or performance of a regression model must be reported as an error in those predictions. So, this will make sense if you think about it. If you are predicting a numeric value like a height or a dollar amount, you don’t want to know if the model predicted the value exactly (this might be intractably difficult in practice); instead, we want to know how close the predictions were to the expected values.
Error addresses exactly this and summarizes on average how close predictions were to their expected values. There are three error metrics that are commonly used for evaluating and reporting the performance of a regression model; they are:

Mean Squared Error (MSE).
Root Mean Squared Error (RMSE).
Mean Absolute Error (MAE)

There are many other metrics for regression, although these are the most commonly used.

6. What is the difference between multiple and adjusted coefficient of determination.

Solutions:

The fundamental point is that when you add predictors to your model, the multiple R-squared will always increase, as a predictor will always explain some portion of the variance. Adjusted R-squared controls against this increase, and adds penalties for the number of predictors in the model. The most vital difference between adjusted R-squared and R-squared is simply that adjusted R-squared considers and tests different independent variables against the model and R-squared does not. Also most obvious difference between adjusted R-squared and R-squared is simply that adjusted R-squared considers and tests different independent variables against the stock index and R-squared does not. Because of this, many fund managers like using adjusted R-squared because it has the potential to be more accurate. If we dig a little deeper from the perspective of a fund manager can gain additional information about what is affecting a stock by testing various independent variables using the adjusted R-squared model. R-squared, on the other hand, does have its limitations. One of the most essential limits to using this model is that R-squared cannot be used to determine whether or not the coefficient estimates and predictions are biased. Also, in multiple linear regression, the R-squared can not tell us which regression variable is more important than the other.