Project 1

Q1

The data consists of insurance charges for various individuals and their demographic and lifestyle information. The variables in the dataset are:

age: age of the primary beneficiary

sex: gender of the primary beneficiary (male or female)

bmi: body mass index of the primary beneficiary

children: number of children covered by the insurance plan

smoker: whether the primary beneficiary is a smoker or not (yes or no)

region: the beneficiary's residential area in the US (northeast, northwest, southeast, southwest)

charges: individual medical costs billed by health insurance

This is a supervised learning dataset because it has a target variable (charges) that needs to be predicted using the other variables as predictors

Q2.

The minimum sample size n required for 99% accurate predictions can be calculated using the formula:

n = (Z^2 * p * (1-p)) / e^2

where Z is the Z-score corresponding to the desired confidence level (for 99% confidence, Z = 2.576), p is the estimated proportion of the population with a particular characteristic (e.g. income, age, etc.), and e is the desired precision or margin of error (e.g. 0.02).

Since the proportion p of the population is unknown, it can be estimated as 0.5. With a precision of 0.02 and Z = 2.576, we get:

n = (2.576^2 * 0.5 * (1-0.5)) / 0.02^2 = 165.67

Therefore, the minimum sample size required for 99% accurate predictions with a precision of 0.02 is approximately 326 individuals.

Q3.

insurance.isnull().sum()

insurance.duplicated

duplicate_rows = insurance[insurance.duplicated()]

num_duplicates = len(duplicate_rows)

# Print the duplicate rows and the count of duplicates

print("Duplicate rows:")

print(duplicate_rows)

print("Number of duplicates:", num_duplicates)

df=insurance.drop_duplicates()

insurance.loc[insurance['smoker']=='yes','smoker']=1

insurance.loc[insurance['smoker']=='no','smoker']=0

insurance.loc[insurance['sex']=='female','sex']=1

insurance.loc[insurance['sex']=='male','sex']=0

insurance.loc[insurance['region']=='southwest','region']=1

insurance.loc[insurance['region']=='northwest','region']=2

insurance.loc[insurance['region']=='southeast','region']=3

insurance.loc[insurance['region']=='northeast','region']=4

print(insurance)

# Create a box plot

plt.boxplot(insurance)

# Show the plot

plt.show()

Q1 = np.percentile(insurance, 25)

Q3 = np.percentile(insurance, 75)

IQR = Q3 - Q1

# Set a threshold for outliers

threshold = 1.5

# Identify outliers

lower_bound = Q1 - threshold * IQR

upper_bound = Q3 + threshold * IQR

outliers = insurance[(insurance < lower_bound) | (insurance > upper_bound)]

# Remove outliers

insurance[(insurance >= lower_bound) & (insurance <= upper_bound)]

Q4

from scipy.stats import chi2_contingency

# Create a contingency table of the two variables

contingency_table = pd.crosstab(insurance['sex'], insurance['smoker'])

# Perform the Chi-Square test of independence

chi2, p_value, dof, expected = chi2_contingency(contingency_table)

# Print the test result

print("Chi-Square test statistic:", chi2)

print("p-value:", p_value)

print("Degrees of freedom:", dof)

Q5.

import seaborn as sns

# Select the regressor variables

regressor_variables = ['age', 'sex', 'bmi', 'children', 'smoker', 'region']

# Create a correlation matrix

correlation_matrix = insurance[regressor_variables].corr()

# Visualize the correlation matrix as a heatmap

sns.heatmap(correlation_matrix, annot=True, cmap='coolwarm')

# Show the plot

plt.show()

Q6.

import statsmodels.api as sm

response_variable = 'expenses'

regressor_variables = ['age', 'sex', 'bmi', 'children', 'smoker', 'region']

# Create the design matrix

X = insurance[regressor_variables]

X = sm.add_constant(X)

# Create the response variable vector

y = insurance[response_variable]

# Fit the ordinary least squares (OLS) model

model = sm.OLS(y, X).fit()

# Print the summary of the model

print(model.summary())

Q7

A linear regression analysis models the relationship between the response and one or more independent variables (regressors). It provides an equation that describes the relationship between the response and the regressors, and allows you to make predictions about the response based on the values of the regressors. This analysis can also provide information about the significance of each regressor in explaining the variation in the response.

Q8.

from sklearn.model_selection import train_test_split

from sklearn.linear_model import LinearRegression

from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score

# Split the dataset into features (X) and target variable (y)

X = insurance[['age', 'sex', 'bmi', 'children', 'smoker', 'region']]

y = insurance['expenses']

# Split the data into training and testing sets

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Initialize and train the regression model

model = LinearRegression()

model.fit(X_train, y_train)

# Make predictions on the testing set

y_pred = model.predict(X_test)

# Evaluate the accuracy of the regression model

mse = mean_squared_error(y_test, y_pred)

rmse = mean_squared_error(y_test, y_pred, squared=False)

mae = mean_absolute_error(y_test, y_pred)

r2 = r2_score(y_test, y_pred)

# Print the evaluation metrics

print("Mean Squared Error (MSE):", mse)

print("Root Mean Squared Error (RMSE):", rmse)

print("Mean Absolute Error (MAE):", mae)

print("R-squared (R2) Score:", r2)

Q9.New Data Prediction: We used the trained model to predict insurance charges for a new data point (Age = 29, Sex = Female, BMI = 28, Children = 1, Smoker = Yes, Region = Southeast)

Q10

# Calculate the absolute difference between predicted and actual values

absolute_difference = np.abs(y_test - y_pred)

# Calculate the percentage difference

percentage_difference = absolute_difference / y_test

# Calculate the mean absolute percentage error (MAPE)

mape = np.mean(percentage_difference) * 100

# Print the percentage of error

print("Mean Absolute Percentage Error (MAPE):", mape)

# Calculate the difference between predicted and actual values

difference = y_test - y_pred

# Calculate the percentage difference

percentage_difference = difference / y_test

# Calculate the mean percentage error (MPE)

mpe = np.mean(percentage_difference) * 100

# Print the percentage of error

print("Mean Percentage Error (MPE):", mpe)

Q11

import numpy as np

from scipy.stats import t

# Define the insurance charges dataset

charges = [1000, 1200, 1500, 1300, 1100, 1600, 1400, 1700, 1900, 1800]

# Calculate the sample mean and sample standard deviation

x_bar = np.mean(charges)

s = np.std(charges, ddof=1)

# Calculate the sample size

n = len(charges)

# Calculate the critical value (t) for the desired confidence level and degrees of freedom

confidence_level = 0.95

df = n - 1

t_value = t.ppf(confidence_level, df)

# Calculate the standard error

SE = s / np.sqrt(n)

# Calculate the margin of error

ME = t_value * SE

# Calculate the lower and upper bounds of the confidence interval

LB = x_bar - ME

UB = x_bar + ME

# Print the confidence interval

print("95% Confidence Interval for Average Charge Insurance:")

print("Lower Bound:", LB)

print("Upper Bound:", UB)