Project 2

First KMeans (Initial Run)

• Objective: To apply KMeans clustering on the car dataset (likely based on features like fuel type, city mpg, highway mpg, etc.), without much prior tuning or pre-processing.

• Steps:

  • Standard KMeans clustering applied using default parameters (e.g., random initialization, fixed number of clusters).

  • Expected Result: A basic clustering result with the initial cluster separation, where the Silhouette Score and Inertia would give insights into the clustering quality.

  • Metrics:

    • Inertia: Measures the sum of squared distances of samples to their closest cluster center.

    • Silhouette Score: Evaluates how similar an object is to its own cluster versus other clusters.

Second KMeans (Tuning/Refinement)

• Objective: This run of KMeans could involve hyperparameter tuning (e.g., changing the number of clusters, using different initializations) and/or applying scaling to the features to improve the clustering result.

• Steps:

  • Tune the number of clusters (e.g., via elbow method or silhouette score optimization).

  • Apply feature scaling (e.g., StandardScaler or MinMaxScaler) to ensure that each feature (like mpg) contributes equally to the distance metric.

  • Expected Result: Improved clustering with better separation between clusters and potentially higher Silhouette Scores and lower Inertia values.

  • Metrics:

    • Inertia: It should ideally decrease, showing that the clustering is tighter.

    • Silhouette Score: Should improve compared to the first run, indicating better cluster separation.

Third Run (Using Agglomerative Clustering or DBSCAN)

• Objective: To see if a different clustering method such as Agglomerative Clustering (hierarchical) or DBSCAN (density-based) performs better or differently compared to KMeans.

  Agglomerative Clustering:

  • Objective: Hierarchical clustering method that builds a tree of clusters.

  • Steps:

    • It does not require the number of clusters to be specified upfront.

    • The algorithm will merge clusters based on proximity until a stopping criterion is met (e.g., desired number of clusters or distance threshold).

  • Expected Result: It may offer better results for data with irregular shapes (unlike KMeans, which assumes spherical clusters).

  • Metrics:

    • Silhouette Score: A higher score would indicate better clustering.

    • Dendrogram: Used to visualize how clusters are merged step by step.

  DBSCAN (Density-Based Spatial Clustering of Applications with Noise):

  • Objective: A density-based algorithm that groups points that are closely packed together and marks points that lie alone in low-density regions as outliers.

  • Steps:

    • Key parameters: eps (epsilon) (distance threshold) and min_samples (minimum number of points to form a cluster).

    • DBSCAN can detect outliers, so some points may not belong to any cluster (labeled as noise).

  • Expected Result: DBSCAN may create a different set of clusters, especially if the data has noise or irregularly shaped clusters. It is less sensitive to the initial selection of cluster count.

  • Metrics:

    • Silhouette Score: A high score would indicate well-defined clusters.

    • Number of clusters and noise points: Number of detected clusters and how many points are labeled as noise.

Clustering MethodNumber of ClustersInertia (SSE)Silhouette ScoreNotes

First KMeans

e.g., 4

Initial value

Initial score

Default settings, no scaling or tuning

Second KMeans

e.g., 4-6 (tuned)

Lower than first

Higher than first

Hyperparameter tuning and feature scaling

Agglomerative

Variable (depends on linkage)

N/A

Typically high

Hierarchical, no need to specify k

DBSCAN

Variable (depends on eps and min_samples)

N/A

Could vary

Detects noise, no predefined number of clusters

Model	Metric	0	1	2	3	4	5	6	Accuracy	Macro Avg	Weighted Avg	R² Score	Adjusted Rand Index (ARI)	Silhouette Score
Decision Tree	Precision	1.00	1.00	1.00	1.00	1.00	0.67	0.86	0.95	0.93	0.96	0.95	0.9487	0.3113
	Recall	1.00	1.00	1.00	0.67	0.67	1.00	1.00
	F1-Score	1.00	1.00	1.00	0.80	0.80	0.80	0.92
Random Forest	Precision	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.00	1.0	1.0
	Recall	1.00	1.00	1.00	1.00	1.00	1.00	1.00
	F1-Score	1.00	1.00	1.00	1.00	1.00	1.00	1.00
KMeans	Inertia	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	N/A	0.3113	0.5434
	Silhouette Score	N/A	N/A	N/A	N/A	N/A	N/A	N/A

Key Insights:

1. Decision Tree:

   • Precision, Recall, and F1-Score are high for most classes, except for classes 3, 4, and 5, where recall is lower, indicating misclassifications in those classes. Despite this, the overall accuracy of 0.95 is still quite good.

   • R² Score for Decision Tree is 0.9487, indicating that the model explains around 95% of the variance in the data.

   • The Adjusted Rand Index (ARI) for KMeans is 0.3113, showing a modest level of agreement between the clusters found by KMeans and any ground truth (if available).

2. Random Forest:

   • Precision, Recall, and F1-Score are perfect (1.00) across all classes, reflecting the model's excellent performance.

   • R² Score is 1.0, meaning the model perfectly fits the data and predicts the outcomes with no error.

   • Random Forest achieved perfect performance in both accuracy and all class-specific metrics.

3. KMeans:

• Silhouette Score for KMeans is 0.5434, indicating that the clusters are moderately well-separated. This is a reasonable value for unsupervised learning, but there may still be room for improvement in clustering quality.

• Inertia (SSE) was not provided, but typically, lower inertia means better clustering.

• The Adjusted Rand Index (ARI) indicates how well the clusters align with actual labels, and 0.3113 is a modest value. This suggests some clustering is happening, but it's far from perfect.

Metric	Initial KMeans (Model 1)	Improved KMeans (Model 2)	New KMeans (Model 3)
Inertia (SSE)	High	Lower	Lowest
Silhouette Score	Low	Higher	Highest
Homogeneity	Low	Moderate	High
Completeness	Low	Moderate	High
V-Measure	Low	Moderate	High

Interpretation and Improvements:

• Initial KMeans likely had high inertia and poor cluster separation, meaning the clusters weren't well-defined.

• Improved KMeans showed better results with feature scaling and optimization of the number of clusters.

• New KMeans would have achieved the best performance, with tighter and well-separated clusters, likely thanks to further improvements like KMeans++ or better initialization.