Project 1

1. Explain the data. Check if it is supervised or unsupervised. Demonstrate:

• Explanation: The data contains information about individuals, including:

  • Age, Sex, BMI, Children, Smoker, and Region as independent variables (predictors).

  • Expenses as the dependent variable (target), which represents the insurance charges.

• Supervised or Unsupervised: This is a supervised learning dataset because the goal is to predict expenses (charges) using the independent variables. The dataset has both input variables (features) and a target variable (expenses).

2. Estimate the minimum sample size n$n$ to get the 99% accurate predictions (precision = 0.02):

We use the formula for sample size estimation in a proportion, assuming we have no prior estimate for the proportion, so we take p=0.5$p = 0.5$ for maximum variance:

n=Z2⋅p⋅(1−p)E2$$n = \frac{Z^2 \cdot p \cdot (1 - p)}{E^2}$$

Where:

• Z=2.576$Z = 2.576$ for a 99% confidence level

• p=0.5$p = 0.5$ (since we don't have prior data on the proportion)

• E=0.02$E = 0.02$

n=2.5762⋅0.5⋅(1−0.5)0.022=6.635⋅0.250.0004=4156.25$$n = \frac{2.576^2 \cdot 0.5 \cdot (1 - 0.5)}{0.02^2} = \frac{6.635 \cdot 0.25}{0.0004} = 4156.25$$

So, you would need around 4157 samples to ensure 99% accurate predictions with a precision of 0.02.

3. Check if the data is cleaned or not. If not, then clean it (Null values, Row/Column Duplicates, Outliers, Change the string into numbers):

• Null values: Check if there are any missing values in the dataset.

  df.isnull().sum()
  

  In this case, the dataset doesn't have missing values.

• Duplicates: Check if there are any duplicate rows.

  df.duplicated().sum()
  

  If duplicates exist, remove them using df.drop_duplicates().

• Outliers: Use IQR or Z-score to check for outliers in numerical columns such as age, BMI, and expenses.

• String to Numbers: Convert categorical variables like Sex, Smoker, and Region into numerical values (encoding).

  • For Sex, use Label Encoding (0 for Male, 1 for Female).

  • For Smoker, encode as 0 for No and 1 for Yes.

  • Region can be One-Hot Encoded.

4. Check that sex and smoking are statistically independent or not:

Perform a Chi-square test to check if sex and smoking are statistically independent.

from scipy.stats import chi2_contingency
contingency_table = pd.crosstab(df['sex'], df['smoker'])
chi2, p, dof, expected = chi2_contingency(contingency_table)

• If the p-value is less than 0.05, reject the null hypothesis, indicating that sex and smoking are dependent.

5. Check that all regressor variables (independent variables) are independent of each other or not:

Use Correlation Matrix and VIF (Variance Inflation Factor) to check for multicollinearity.

• Correlation Matrix:

  df.corr()
  

  If the correlation between any two independent variables is high (above 0.8), they are not independent.

• VIF:

  from statsmodels.stats.outliers_influence import variance_inflation_factor
  

6. Check the dependency between response and regressors:

You can check the correlation between the response (expenses) and the regressors (independent variables such as age, sex, BMI, etc.).

df.corr()['expenses']

This will give you the correlation values between each independent variable and the dependent variable, showing which ones are most strongly correlated with the target (expenses).

7. Predict the regression line to predict the charges for insurance using independent variables:

Use Linear Regression to predict expenses based on the independent variables.

from sklearn.linear_model import LinearRegression

# Prepare features (X) and target (y)
X = df[['age', 'sex', 'bmi', 'children', 'smoker', 'region']]
y = df['expenses']

# Encode categorical variables (sex, smoker, region)
X = pd.get_dummies(X, columns=['sex', 'smoker', 'region'], drop_first=True)

# Split the data into training and test sets
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Create and train the model
model = LinearRegression()
model.fit(X_train, y_train)

# Coefficients (regression line)
model.coef_, model.intercept_

8. Predict the accuracy of the regression model:

You can evaluate the model's accuracy using R-squared and Mean Absolute Error (MAE):

from sklearn.metrics import mean_absolute_error, r2_score

y_pred = model.predict(X_test)
r2 = r2_score(y_test, y_pred)
mae = mean_absolute_error(y_test, y_pred)

The R-squared value indicates how well the independent variables explain the variation in the target variable.

9. Predict insurance charge for Age = 29, Sex = F, BMI = 28, Children = 1, Smoker = Yes, Region = Southeast:

Encode the new data point and use the model to predict:

new_data = pd.DataFrame([[29, 'female', 28, 1, 'yes', 'southeast']],
                        columns=['age', 'sex', 'bmi', 'children', 'smoker', 'region'])

# Encode the new data point
new_data = pd.get_dummies(new_data, columns=['sex', 'smoker', 'region'], drop_first=True)

# Predict
new_expense = model.predict(new_data)
new_expense

10. Give the percentage of error in regression model:

Use Mean Absolute Percentage Error (MAPE):

mape = np.mean(np.abs((y_test - y_pred) / y_test)) * 100

11. Give the 95% confidence interval for average charge insurance:

To calculate the 95% confidence interval for the average charge, you can use the formula:

CI=mean±Z×σ√n$$CI = \text{mean} \pm Z \times \frac{\sigma}{\sqrt{n}}$$

Where:

• Z=1.96$Z = 1.96$ for a 95% confidence level

• σ$\sigma$ is the standard deviation of the errors

• n$n$ is the sample size (10 in this case)

You can use statsmodels to calculate this:

import statsmodels.api as sm
model = sm.OLS(y_train, sm.add_constant(X_train)).fit()
model.conf_int(alpha=0.05)