Week 6 - Data analysis

DATA VISUALIZER TOOL:

AIM:

1. Plot the data between any of the columns in the given dataset and saving the plotted images in the respective names of the fields based on which they are plotted.
2. In this case, we are going to extract the data between columns 2(INDEX=1) and 8(INDEX=7) which is going to deliver the user with the plot of Pressure vs Volume or P-V diagram.
3. Calculate the area under the P-V diagram.
4. Calculate the power output of this engine.
5. Calculate the specific fuel consumption.

PROCEDURE:

STEP-WISE CODE EXPLANATION:

Library import:

• Importing essential Libraries such as numpy, pandas,xlsxwriter along with matplotlib.
• Numpy library is highly essential for numerical calcuklations and dealing with numbers.
• Pandas is essential for dealing with large amount of DataSets and generate them in DataFrames.
• xlsxwriter is used to write the data to excel. In our case , I have used it to write the specific two column for which we are plotting the relationship so that the correspong excel sheet is generated for those columns of data alone.
• matplotlib is used for plotting the data.
• sys library is imported for the requirement for the cprint.This is highly useful for printing a particular statement in highlighted manner so that the user can understand the primpt clearly.

Function 1 (skip_rows(file_name)):

• The above function in the code is deployed to skip the rows that contains symbol "#"
• The function when called, returns the value of count of consecutive lines that contain "#"
• This can be utilized and this function is called in the parameter skip_rows in pandas DataFrame.
• Eg: df=pd.read_csv(file,sep=',',skiprows=skip_rows(file),names=headers,delim_whitespace=True)
• See bold text in the above function.

TEMPLATE FUNCTION (write_output_to_excel(dataframe, filename)):

• This function is responsible for creating excel templates with particular format in order to write our output.
• This function in the below following line creates format for headers and also for each cell.Eg:
• header_format = workbook.add_format(
  {
  "bold": True,
  "font_color": "#ffffff",
  "bg_color": "#31869b",
  "border": True,
  "font_size": "10.5",
  "align": "center",
  "valign": "vcenter",
  }
  )
  cellFormat = workbook.add_format({"border": True, "font_size": "10.5"})
  worksheet.set_column(0, len(dataframe.columns),30)
• This enables us to present the userend with clean and structured excel sheet of specific column for which X and Y variables are plotted.

Function 3 (fetch_column(dataset)):

• This function gets the input from the user for the column number of the variables to be plotted.
• In this case Pressure is column_no 2 and Volume is column_no 8.
• The uniqueness of the function is it validates the user input with conditional statements.
• If the user input is a string or float or negative number it prompts the user with the statement 'Invalid input !!!. Please check the datatype'. This keeps on continuing until the user gives the proper value for the column number within the range of of number of column with a positive integer.
• Exceptions are handled to check Value Errors
• The parameter that has to be pased in this function is the dataset from which we have to fetch the columns and process them for further calculations.
• input1=input1-1 The following variable is declared as the user input of column number can be converted to column index, as in python dataframes indices begin from 0 for both rows and columns

Function 4 (area_under_curve(x,y)):

• This function is used to calculate the area under the curve for P_V diagram.
• Hence integration is carried out to calculate the area.
• In this case area calculated is the work done by the gas.
• In our case our curve is not a exact square or rectangle to to find the area.
• Hence we need to find areas in form small minimal trapezoidal regions and finally integrate them to get the actual value.

def area_under_curve(x,y):
area=0
n=len(x)
dx=x[1]-x[0]
for i in range(1,n):
area+=(y[i-1]+y[i])*dx/2
return area

Note:There are various methods to calculating the area under a curve, for example, Rectangle Method, Trapezoidal Rule and Simpson's Rule. In this case we have used the above method.

Function 4 (data_visualizer()):

• This function is a heart of all as it deleivers the final code.
• This plays an important role in file_handling.
• We introduce a new variable called headers which stores the name of all parameters in our data.
• This variable headers is later passed in the dataframe function in the parameter 'names'
• df=pd.read_csv(file,sep=',',skiprows=skip_rows(file),names=headers,delim_whitespace=True)
• Xvariable,Yvariable=fetch_column(df). The fetch_column function returns the indices of two columns.These 2 values are assigned to variables Xvariable and Yvariable.
• x=df.iloc[ : , Xvariable ]
  y=df.iloc[ : , Yvariable ]
• The iloc method is used to select specifically those 2 columns which user has given as input.
• files = list(filter(lambda x: '.out' in x, os.listdir(input_path))). This is declared in order to filter the files in our current working directory(cwd) based on the extension and it consider only .out format files.
• In the case of non existence of such files the code will exit gracefully raising the statement "File not recognized. Please provide a valid CONVERGE output file"
• label1=df.columns[Xvariable]
  label2=df.columns[Yvariable]
  plt.xlabel(label1)
  plt.ylabel(label2)
  fileName=label1 + ' vs ' + label2
  plt.savefig(fileName)
  plt.show()
  data=pd.DataFrame([df[label1],df[label2]]).T
  write_output_to_excel(data, fileName)
  area=area_under_curve(x,y)

In the above lines of the code we can clearly see we have not deployed any case of manual entry whereas from the label of the axis till the file name is taken automatically from the user input which is dynamic in nature. There is no hardcoding done. The name are subjected to change if the user selects different columns.

• a=round(area*10**6,4)
  output_power=a*1500/60000
  output_power=round(output_power,5)
  fuelConsumption=(20*(10**-6))
  sfc=round(fuelConsumption,9)
  cprint('The area under the curve P-V diagram is: {0} {1}'.format(a,'m^2'),color='green',attrs=['bold'],file=sys.stderr)
  cprint('Power ouput of the engine is: {0} {1}'.format(output_power,'kW'),color='green',attrs=['bold'],file=sys.stderr)
  cprint('Power ouput of the engine is: {0} {1}'.format(sfc,'g/kWh'),color='green',attrs=['bold'],file=sys.stderr)
• The above lines of code gives the final output of calculations made.

Output ScreenShots:

We can see the advantage of using cprint in the above image as we can see our output in a more convenient manner.

Alternate_Output_ScreenShots:

In the case of non existense of the file, the output will be like as follows

Case of Invalid Inputs:

Suppose if the user gives an invalid input for the column numner, our code has been designed to manage that so like, The script will continue until the user gives a valid input.

__author__ = "Sandeep Ilangovan"

# Library Imports
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import sys
from termcolor import cprint
import os          # in order to get the files in the cwd
import xlsxwriter  # in order to write the outpu of selected columns to excel.

# Variable: To get the current working directory of our path
input_path = os.path.join(os.getcwd())
# Function to identify the unwanted columns and give its count
def skip_rows(file_name):
    line_count=0
    string='#'
    for line in open(file_name):
        if string in line:
            line_count=line_count+1
        else:
            pass
    return(line_count)


# Function for standard excel template
def write_output_to_excel(dataframe, filename):
	workbook = xlsxwriter.Workbook(filename+'.xlsx')
	worksheet = workbook.add_worksheet("Sheet1")
	header_format = workbook.add_format(
		{
			"bold": True,
			"font_color": "#ffffff",
			"bg_color": "#31869b",
			"border": True,
			"font_size": "10.5",
			"align": "center",
			"valign": "vcenter",
		}
	)
	cellFormat = workbook.add_format({"border": True, "font_size": "10.5"})
	worksheet.set_column(0, len(dataframe.columns),30)
	worksheet.freeze_panes(1, 0)
	worksheet.hide_gridlines(2)
	for col_num, value in enumerate(dataframe.columns.values):
		value = value if str(value) != "nan" else None
		worksheet.write(0, col_num, value, header_format)
	row = 1
	for rows in dataframe.values:
		for col_num, value in enumerate(rows):
			value = value if str(value) != "nan" else None
			worksheet.write(row, col_num, value, cellFormat)
		row += 1
	workbook.close()

# Function to provide the user to select the required two columns
def fetch_column(dataset):
    maxValue = len(dataset.columns)
    while True:
        try:
            input1=int(input('Enter the column_number of X-axis variable: '))
            input1=input1-1
            input2=int(input('Enter the column_number of Y-axis variable: '))
            input2=input2-1
        except ValueError:
            cprint('Invalid input !!!. Please check the datatype', 'red', attrs=['bold'], file=sys.stderr)
            continue
        else:
            if input1 and input2 < 1 or input1 and input2 > maxValue:
                cprint('Column index out of range!!! Please enter the column index within the range','green', attrs=['bold'], file=sys.stderr)
                continue
            else:
                return (input1,input2)

#Function to calculate the area under the curve.
def area_under_curve(x,y):
    area=0
    n=len(x)
    dx=x[1]-x[0]
    for i in range(1,n):
        area+=(y[i-1]+y[i])*dx/2
    return area


# Core Function
def data_visualizer():
    
	headers=['Crank','Pressure','Max_Pres','Min_Pres','Mean_Temp','Max_Temp','Min_Temp','Volume','Mass','Density','Integrated_HR','HR_Rate','C_p','C_v','Gamma','Kin_Visc','Dyn_Visc']
	files = list(filter(lambda x: '.out' in x, os.listdir(input_path)))
	if len(files)<1:
			print("File not recognized. Please provide a valid CONVERGE output file")
			sys.exit()
	for file in files:
		file = os.path.join(input_path,file)
		print(file)
		fileprefix, file_extension = os.path.splitext(file)

		if file_extension == '.out':
			df=pd.read_csv(file,sep=',',skiprows=skip_rows(file),names=headers,delim_whitespace=True)
			Xvariable,Yvariable=fetch_column(df)  
			x=df.iloc[ : , Xvariable ]
			y=df.iloc[ : , Yvariable ]
			plt.plot(x,y,linewidth=3,color='red') 
			label1=df.columns[Xvariable]
			label2=df.columns[Yvariable]
			plt.xlabel(label1)
			plt.ylabel(label2)
			fileName=label1 + ' vs ' + label2     
			plt.savefig(fileName)
			plt.show()
			data=pd.DataFrame([df[label1],df[label2]]).T
			write_output_to_excel(data, fileName)
			area=area_under_curve(x,y)
			a=round(area*10**6,4)
			output_power=a*1500/60000
			output_power=round(output_power,5)
			fuelConsumption=(20*(10**-6))
			sfc=round(fuelConsumption,9)
			cprint('The area under the curve P-V diagram is: {0} {1}'.format(a,'m^2'),color='green',attrs=['bold'],file=sys.stderr)
			cprint('Power ouput of the engine is: {0} {1}'.format(output_power,'kW'),color='green',attrs=['bold'],file=sys.stderr)
			cprint('Specific_Fuel_Combustion of the engine is: {0} {1}'.format(sfc,'g/kWh'),color='green',attrs=['bold'],file=sys.stderr)            
		else:
			print("File not recognized. Please provide a valid CONVERGE output file")

# Calling the function:
data_visualizer()

Results:

The required output has been derived successfully through the Data_Visualiser Tool.

The area under the curve P-V diagram is: 505.537 m^2
Power ouput of the engine is: 12.63842 kW
Specific_Fuel_Consumption of the engine is: 2e-05 g/kWh

as from the output obtained.