Project 1
STATISTICS & PROBABILITY FOR DATA SCIENCES :PROJECT 1
PROBLEM STATEMENT -
Assume that you have a dataset named "insurance.csv" containing information about insurance charges, including variables such as age, sex, BMI, number of children, smoking status, and region. Use Python with libraries like pandas, scikit-learn, and statsmodels for data manipulation, analysis, and modeling.
Importing Required Libraries -
import pandas as pd
import numpy as np
from scipy.stats import t
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score
from scipy.stats import chi2_contingency
Q1. Explain the data. Check It is supervised or Unsupervised. Demonstrate?
# Load the data
data = pd.read_csv(r"C:\Users\sangam\Downloads\insurance .csv")
data.head()
age sex bmi children smoker region charges
0 19 female 27.900 0 yes southwest 16884.92400
1 18 male 33.770 1 no southeast 1725.55230
2 28 male 33.000 3 no southeast 4449.46200
3 33 male 22.705 0 no northwest 21984.47061
4 32 male 28.880 0 no northwest 3866.85520
Data Explanation -
age- Determines the age of person
sex- Determines the gender of the person
bmi- Determines Body Mass Index of the person
children- Determines how many children the person have
smoker- Determines whether the person is smoker or not
region- Determines in which region the person live
charges- Determines the insurance charges
The data is Supervised,since it contains target variable.
Q2. Estimate the minimum sample size n to get the 99% accurate predictions. (precision = 0.02)
# Using the formula for sample size calculation for regression analysis
# n = (Z^2 * σ^2) / ε^2
Z = 2.576 # Z-score for 99% confidence level
σ = data['charges'].std() # Standard deviation of insurance charges
ε = 0.02 # Desired precision
n = (Z**2 * σ**2) / ε**2
Minimum sample size: 2432880728687
Q3. Check the data is cleaned or not. If not then clean it (Null values, Row/Column Duplicates, Outliers, Change the string into numbers)
Ans.
The data preprocessing work is done such as,
- Checking for missing values.
- Dropping unnecessary columns.
- Handling missing values (in this case, filling missing values with the mean).
- Check whether there is duplicate of row / column
- Check Outliers, also change the string type object into Numbers.
Q4. Check that sex and smoking are statistically independent or not.
Ans.
# Perform chi-square test for independence
# If p-value is less than significance level (e.g., 0.05), conclude variables are dependent
p-value: 0.0062765550120107375
The data column 'sex' and 'smoker' in the dataset are statistically Independent.
Q5. Check that all regressor variables (independent variable) are independent of each other or not.
Ans.
# We can use correlation matrix for this purpose...
From the correlation matrix , it seems that most variables have low correlation coefficients with each other, which is a good indicator of independence.
Here's a summary of the correlations:
- Age, BMI, and children have low correlation coefficients with other variables, suggesting independence.
- Sex and smoker status also have low correlation coefficients with other variables.
- Charges (the dependent variable) has moderate positive correlation with age and smoker status, but low correlation with other variables.
Based on this information, you can conclude that the regressor variables are generally independent of each other, with some minor correlations between age, smoker status, and charges.
Q6. Check dependency between response and regressors.
# We can use correlation between 'charges' and other variables
Here's an explanation of the correlation coefficients obtained:
- Age (0.412837): There is a moderate positive correlation between age and charges. This suggests that as age increases, insurance charges tend to increase as well, but the relationship is not extremely strong.
- Sex (-0.024399): There is a very weak negative correlation between sex and charges. This indicates that there is almost no relationship between sex and insurance charges in this dataset.
- BMI (-0.063813): There is a weak negative correlation between BMI and charges. This suggests that there is a slight tendency for charges to decrease as BMI increases, but the relationship is not strong.
- Children (0.078421): There is a weak positive correlation between the number of children and charges. This implies that there is a slight tendency for charges to increase with the number of children, but the relationship is not significant.
- Smoker (0.434794): There is a strong positive correlation between smoking status and charges. This indicates that smokers tend to have significantly higher insurance charges compared to non-smokers in this dataset.
Q7. Predict the regression Line to predict the charges for insurance using independent variables.
Ans.
Intercept: 3912.4427669535253
The intercept term of the regression line is approximately 3912.44. This represents the estimated insurance charges when all independent variables are zero .
Coefficients: [ 211.66273259 -641.69506844 -129.254904 340.54503803 7617.65660021]
The coefficients represent the change in the estimated charges for a one-unit change in each independent variable, holding all other variables constant.
Here's the interpretation of each coefficient:
- For a one-year increase in age, the estimated charges increase by approximately 211.66.
- Females (assuming coded as 0) tend to have lower charges by approximately 641.70 compared to males (coded as 1).
- For a one-unit increase in BMI, the estimated charges decrease by approximately 129.25.
- Each additional child (assuming a linear relationship) is associated with an increase in charges by approximately 340.55.
- Smokers (coded as 1) have significantly higher charges by approximately 7617.66 compared to non-smokers (coded as 0).
Q8. Predict the accuracy of the regression Model.
Ans.
In this case, the calculated R-squared value of approximately 0.387 suggests that the independent variables explain about 38.7% of the variance in insurance charges according to the model. This means that the model, as constructed, explains a moderate amount of the variability in insurance charges based on the chosen independent variables.
Q9. Predict insurance charge for Age = 29, Sex = F, bmi = 28, children = 1, Smoke = Yes, region = southeast.
Ans.
The predicted insurance charge for this individual is approximately $14,389.73. This prediction is based on the linear regression model trained on the dataset, which estimates insurance charges based on the provided independent variables.
Q10. Give the percentage of error in regression model.
Ans.
In this case, the MAPE(Mean Absolute Percentage Error) of approximately 43.60% indicates that, on average, the model's predictions for insurance charges are off by about 43.60% from the actual charges. This means that the model's predictions are not very accurate, and there is room for improvement.
Q11. Calculate 95% confidence interval for average insurance charge.
Ans.
The resulting 95% confidence interval for the average insurance charge is approximately $9509.86 to $10245.44. This means that we can be 95% confident that the true average insurance charge in the population falls within this interval.