Project 1

Data cleaning on Automobile 1985 dataset and perform descriptive analytics

1. Data Importing:

# "Importing the Required libraries"
import numpy as np
import pandas as pd

# 'Importing the Dataset using Pandas'
df_auto = pd.read_csv("auto.csv", header = None)

# 'Printing the first 5 rows in the Data Frame'
df_auto.head()

Out[1]:

5 rows × 26 columns

# 'Fetching the correct names using file'
headers = ['symboling','normalized-losses','make','fuel-type','aspiration','num-of-doors','body-style',\
           'drive-wheels','engine-location','wheel-base','length','width','height','curb-weight','engine-type', \
           'num-of-cylinders','engine-size','fuel-system','bore','stroke','compression-ratio','horsepower', \
           'peak-rpm','city-mpg','highway-mpg','price']
# Assigning Column names to the Data Frame
df_auto.columns = headers
df_auto.columns

# 'Printing the first 5 rows in the Data Frame'
df_auto.head()

Out[2]:

5 rows × 26 columns

# 'Saving the CSV to Local'
df_auto.to_csv('autoclean_df.csv', index = False)

The new dataset file is saved to local system.

2. Data Cleaning:

Importing the new data set to identify and fix missing values.

# 'Importing the required Libraries'
import pandas as pd
import numpy as np
import matplotlib.pylab as plt

# 'Importing the new Dataset using Pandas'
df_auto1 = pd.read_csv("autoclean_df.csv", header = None)

# 'Printing the first 5 rows in the Data Frame'
df_auto1.head()

Out[3]:

5 rows × 26 columns

# 'Identifying and Handling Missing Values'
df_auto1.replace('?', np.nan, inplace = True)
df_auto1.head()

Out[4]:

5 rows × 26 columns

# 'Check for missing Data in Data Frame'
missing_data = df_auto1.isnull()
missing_data.head()

Out[5]:

5 rows × 26 columns

# 'Counting Missing Data in Each Column'
for col in missing_data.columns.values.tolist():
    print('Column: ', col)
    print(missing_data[col].value_counts())
    print('----------')

Out[6]:

Column:  symboling
False    205
Name: symboling, dtype: int64
----------
Column:  normalized-losses
False    164
True      41
Name: normalized-losses, dtype: int64
----------
Column:  make
False    205
Name: make, dtype: int64
----------
Column:  fuel-type
False    205
Name: fuel-type, dtype: int64
----------
Column:  aspiration
False    205
Name: aspiration, dtype: int64
----------
Column:  num-of-doors
False    203
True       2
Name: num-of-doors, dtype: int64
----------
Column:  body-style
False    205
Name: body-style, dtype: int64
----------
Column:  drive-wheels
False    205
Name: drive-wheels, dtype: int64
----------
Column:  engine-location
False    205
Name: engine-location, dtype: int64
----------
Column:  wheel-base
False    205
Name: wheel-base, dtype: int64
----------
Column:  length
False    205
Name: length, dtype: int64
----------
Column:  width
False    205
Name: width, dtype: int64
----------
Column:  height
False    205
Name: height, dtype: int64
----------
Column:  curb-weight
False    205
Name: curb-weight, dtype: int64
----------
Column:  engine-type
False    205
Name: engine-type, dtype: int64
----------
Column:  num-of-cylinders
False    205
Name: num-of-cylinders, dtype: int64
----------
Column:  engine-size
False    205
Name: engine-size, dtype: int64
----------
Column:  fuel-system
False    205
Name: fuel-system, dtype: int64
----------
Column:  bore
False    201
True       4
Name: bore, dtype: int64
----------
Column:  stroke
False    201
True       4
Name: stroke, dtype: int64
----------
Column:  compression-ratio
False    205
Name: compression-ratio, dtype: int64
----------
Column:  horsepower
False    203
True       2
Name: horsepower, dtype: int64
----------
Column:  peak-rpm
False    203
True       2
Name: peak-rpm, dtype: int64
----------
Column:  city-mpg
False    205
Name: city-mpg, dtype: int64
----------
Column:  highway-mpg
False    205
Name: highway-mpg, dtype: int64
----------
Column:  price
False    201
True       4
Name: price, dtype: int64
----------

From the output it is clear that the following missing values in each column:

1. normalized-losses (41 Records)
2. num-of-doors (2 records)
3. bore (4 records)
4. stroke (4 records)
5. horsepower (2 records)
6. peak-rpm (2 records)
7. price (4 records)

# 'Identify which method to apply to which data'
df_auto1[['normalized-losses', 'bore', 'stroke', 'horsepower', 'peak-rpm', 'num-of-doors', 'price']].head()

Out[7]:

After finding the missing values we need to fix the values by following method :

Replace by Mean
- normalized-losses (41 Records)
- bore (4 records)
- stroke (4 records)
- horsepower (2 records)
- peak-rpm (2 records)

Replace by Frequency
- num-of-doors (2 records)

Drop the Entire row
- price (4 records)

# 'Fixing Price Data (Drop the Entire row)'
df_auto1.dropna(subset = ['price'], axis = 0, inplace = True)
df_auto1.reset_index(drop=True, inplace=True)
df_auto1.shape

Out[8]:

(201, 26)

# 'Replace by Frequency'
    # Fixing num-of-doors variable
print(df_auto1['num-of-doors'].value_counts())

    # 'Printing what value occures the most'
print('\nTop Freq: ', df_auto1['num-of-doors'].value_counts().idxmax())

    # 'Replacing the NaN values with 'four''
df_auto1['num-of-doors'].replace(np.nan,"four", inplace = True)

    # 'Verifying if NaN removed'
df_auto1['num-of-doors'].isnull().value_counts()

Out[9]:

# Replace missing values by Mean
#- normalized-losses (41 Records)  --> Object type --> Convert to Float
#- bore (4 records)
#- stroke (4 records)
#- horsepower (2 records)
#- peak-rpm (2 records)

replace_by_mean  = ['normalized-losses', 'bore', 'stroke', 'horsepower', 'peak-rpm']
for i in replace_by_mean:
    mean_value = df_auto1[i].astype('float').mean()
    df_auto1[i].replace(np.nan, mean_value, inplace = True)

# Validating if the values are replaced
test_data = df_auto1[['normalized-losses', 'bore', 'stroke', 'horsepower', 'peak-rpm']].isnull()

for i in replace_by_mean:
    print(test_data[i].value_counts())

df_auto1.head()

Out[10]:

5 rows × 26 columns

# 'Converting the above Object types to Float'
for i in replace_by_mean:
    df_auto1[[i]] = df_auto1[[i]].astype('float')
    
# 'Coverting the Price Valriable'
df_auto1[['price']] = df_auto1[['price']].astype('float')

# 'Verifying the Data types in Data Frame'
print(df_auto1.dtypes)
df_auto1.head(3)

Out[11]:

3 rows × 26 columns

# Creating Dummy Variable for Fuel Type and adding it back to the Dataset
fuel_dummy = pd.get_dummies(df_auto1['fuel-type'])
fuel_dummy.head()

Out[12]:

# 'Renaming the dummy variables to fuel-type-diesel and fuel-type-gas'
fuel_dummy.rename(columns = {'diesel': 'fuel-type-diesel', 'gas': 'fuel-type-gas'}, inplace = True)
fuel_dummy.head()

Out[13]:

# 'Saving the CSV to Local'
df_auto1.to_csv('autoclean1_df.csv', index = False)

The new dataset file is saved to local system.

3. Exploratory Data Analysis:

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

# 'Importing the new Dataset using Pandas'
df_auto2 = pd.read_csv("autoclean1_df.csv")

df_auto2.corr()

Out[14]:

let's find out the variable that influence the price

# Engine size as potential predictor variable of price
sns.regplot(x="engine-size", y="price", data=df_auto2)
plt.ylim(0,)
plt.show()

Out[15]:

(0.0, 53546.217412832855)

# 'Findng the  corelation with engine size an price'
df_auto2[["engine-size", "price"]].corr()

Out[16]:

The Corelation between the engine size and price is about 0.87.

let see some other variables also

# stroke as potential predictor variable of price
sns.regplot(x="stroke", y="price", data=df_auto2)
plt.ylim(0,)
plt.show()
# 'Findng the corelation with stroke and price'
df_auto2[["stroke", "price"]].corr()

Out[17]:

The Corelation between the stroke and price is about 0.082.

# drive-wheels,engine-location,body-style as potential predictor variable of price
sns.boxplot(x="drive-wheels", y="price", data=df_auto2)
plt.show()
sns.boxplot(x="engine-location", y="price", data=df_auto2)
plt.show()
sns.boxplot(x="body-style", y="price", data=df_auto2)
plt.show()

Out[18]:

From the above results the drive-wheels might be the potential predictor for price because the distribution of price differ along the catagories.

4. Descriptive Statistical Analysis:

df_auto2.describe()
df_auto2.describe(include=['object'])
df_auto2['drive-wheels'].value_counts().to_frame()

Out[19]:

# 'grouping results'
df_gptest = df_auto2[['drive-wheels','body-style','price']]
grouped_test1 = df_gptest.groupby(['drive-wheels','body-style'],as_index=False).mean()
grouped_test1# grouping results
df_gptest = df_auto2[['drive-wheels','body-style','price']]
grouped_test1 = df_gptest.groupby(['drive-wheels','body-style'],as_index=False).mean()
grouped_test1

df_auto2.corr()

Out[20]:

df_gptest

Out[21]:

201 rows × 3 columns

5. Conclusion:

From the we have found out the which variable is used to predict the output.

The Variables are listed down as follows:

Continuous numerical variables:
1. Length
2. Width
3. Curb-weight
4. Engine-size
5. Horsepower
6. City-mpg
7. Highway-mpg
8. Wheel-base
9. Bore
Categorical variables:
1. Drive-wheels