CSE

Uploaded on

30 Dec 2022

Skill-Lync

While using classification algorithms, two kinds of outputs are generated. In one of the types, the output is class, while in another, the probability is the output.

It is a method for summarising a classification algorithm's performance. If your dataset has more than two classes or if each class has an unequal amount of observations, classification accuracy alone may be deceiving. You can acquire a better understanding of the categorization model's successes and failures by calculating a confusion matrix.

Let us consider an example to understand it much better. Suppose imagine that we create an algorithm to predict whether an item belongs to class A or B. Algorithms such as K Nearest Neighbors (KNN) or Support Vector Machine (SVM) create class output, while Random forest, Gradient Boost and Ada Boost give the probability.

In the former set of algorithms, the model predicts whether an item belongs to class A or class B, while in the latter set, the model gives you the probability that it would belong to Class A or B. So each output in this case will have two probability values, one for an item belonging to class A and one for belonging to class B.

The advantage of probability-based output is that it gives flexibility to the user to set a threshold based on which decisions about class type can be made.

To understand its power, let us understand a bit more about the confusion matrix.

A confusion matrix is a 2x2 table in case of a two-class problem. This gives you the details of correct classifications and misclassifications. Suppose imagine we have two classes, A and B. The labels are given to us. We construct a model and use the model to make some predictions. We compare our predictions with the actual label.

We have the following scenario. We may have a situation where our actual class is A, and our model predicted it as A. In this case, we call it a True positive.

In the second case, we have the actual class as A, but our model has classified it as B (not A). In this case, it is called a False Negative.

In the third case, we have the actual class as B, but our model classified it as A. This is called the False Positive.

Finally, we have a scenario where our item belongs to B, and our model classifies it as B. This is called a True Negative.

These four values are used to construct the confusion matrix.

**TP: True Positive**

- FP: False Positive

- TN: True Negative

**FN: False Negative**

Classification models that give probability as the output requires a threshold from the user to make a judgement.

For instance, let us take an example wherein a model has given the probability value of [0.4,0.6] for item I.

This means that item I belongs to A with a probability of 0.4 (call it p_a) and belongs to B with a probability of 0.6 (call it p_b). This is just an example. The values can be anything.

Here we can set a threshold like, say, p=0.7 (it can again be any value). Now our model compares the first value of probability with the threshold p.

In this case, 0.4 is compared with p=0.7.

Let us assume we have a condition that if p_a>p, then assign A. You can see that 0.4 is less than 0.7, so by default, the item is assigned to class B. If our p were 0.3, then the same item would have been assigned to A.

This way, the model can be tuned to give varied output.

For instance, if we require absolute pure true positives only, we will set the value of p to be higher. Similarly, if we want pure, true negatives, we will set the p-value to be low.

Here is a list of the following scenarios where our priorities are different.

In some of the problems, our main criteria would be to reduce the false positive rate. The cost of dealing with false positives could be more devastating.

For example, let us imagine, the government distributes relief packages of about 1 crore rupees (10 million) package to all covid positive patients. In this scenario, you can see having a false positive is a big loss to the Government. So to reduce this, the threshold value is reduced so that we are certain of what is positive.

This is a measure of model FNR. It measures the degree of missing rate and is a perfect measure when dealing with a scenario where the cost of missing a true positive is higher. For instance, if there is a fraudulent transaction, we need to find it with the highest accuracy; missing it would result in tremendous loss.

This is also called specificity. Here we are more interested in being 100% sure that something is truly negative. For instance, during the tests for covid, those who tested negative are allowed to move freely, while those who tested positive quarantine themselves. Here we want to make sure that those who tested negative are truly negative.

This is an important metric when it is inexpensive to check everyone but expensive when even one right one is missed. This is also termed as catching all thieves. So it is easier to check everyone going into the airport, but even leaving one bad guy can be dangerous. In those scenarios, TPR is used to evaluate the model.

This is more important when you are dealing with medical data. Here you want to be certain about the negative predicted value. If a person is negative for a disease, the model should rightly predict all who had tested negative. This is because if the model misses and end up treating a healthy person, it will affect the individual adversely and even lead to the wastage of resources.

This is more important when you are dealing with medical data. Here you want to be certain about the positive predicted value. If a person is positive for a disease, the model should not miss him from being labeled positive. This is because if the model misses and if we fail to treat the patient, it can be a life-threatening situation for them.

This metric is important when false discovery is devastating. For instance, a false alarm regarding a bomb threat leads to complete evacuation of the area, scanning of the area and public panic. So we do not want to have many false discovery scenarios.

We use this when all classes are important.

Apart from this, we also have the following metrics, which are useful for checking our model's performance.

**Cohen-Kapp metric.**This is an alternative to accuracy and works well with imbalanced datasets. Imbalanced datasets are those where one output is more than another. For instance, if we try to create a model which will predict if a given soldier is a male or female. You will notice that by default, more members are male than female in the military. Therefore if we were to take the entire data of all military personnel, there would be an imbalance already in the data. While for balanced data, the numbers in each class are the same.**Matthews correlation coefficient**: for imbalanced dataset**ROC curve**: for a balanced dataset**Precision recall curve****F-Beta score**: Similar to accuracy, with weights to the precision and recall**ROC-AUC**: a graph that plots between sensitivity and specificity for various values of threshold probability.**PR AUC score**: a graph that plots between sensitivity and specificity for various values of threshold probability for an imbalanced dataset.

Author

Navin Baskar

Author

Skill-Lync

Continue Reading

**Related Blogs**

What is Confusion Matrix?

While using classification algorithms, two kinds of outputs are generated. In one of the types, the output is class while in another probability is the output.

CSE

19 May 2023

Real-Time Applications of Python You Need to Know

Since 1991 when the Python language was developed, it has been used for various applications. Due to its simplicity and versatile nature, Python codes can help developers to complete the process of software development without much hassle.

CSE

16 May 2023

A Brief Introduction to Python: Its Features and Different IDEs

Python is an open-source programming language which means it is available on the official website, and anyone can make use of this technology free of cost. Since it is open-source, this means that the source code is also available to the public.

CSE

15 May 2023

Cybersecurity in Telecom: Protecting Networks & Data from Cyber Threats

Telecommunications networks support our digital society. They are, therefore, a top target for cyberattacks.

CSE

15 Apr 2023

Everything you Need to Know About Full-Stack Web Development

Are you interested in becoming a web developer? If so, you've come to the right place! This comprehensive guide to full-stack web development will give you all the information you need to start.

CSE

13 Apr 2023

Author

Skill-Lync

Continue Reading

**Related Blogs**

What is Confusion Matrix?

While using classification algorithms, two kinds of outputs are generated. In one of the types, the output is class while in another probability is the output.

CSE

19 May 2023

Real-Time Applications of Python You Need to Know

Since 1991 when the Python language was developed, it has been used for various applications. Due to its simplicity and versatile nature, Python codes can help developers to complete the process of software development without much hassle.

CSE

16 May 2023

A Brief Introduction to Python: Its Features and Different IDEs

Python is an open-source programming language which means it is available on the official website, and anyone can make use of this technology free of cost. Since it is open-source, this means that the source code is also available to the public.

CSE

15 May 2023

Cybersecurity in Telecom: Protecting Networks & Data from Cyber Threats

Telecommunications networks support our digital society. They are, therefore, a top target for cyberattacks.

CSE

15 Apr 2023

Everything you Need to Know About Full-Stack Web Development

Are you interested in becoming a web developer? If so, you've come to the right place! This comprehensive guide to full-stack web development will give you all the information you need to start.

CSE

13 Apr 2023

Book a Free Demo, now!

Related Courses

4.8

23 Hours of content

Electrical Domain

4.8

29 Hours of content

Embedded Domain

Showing 1 of 4 courses

Try our top engineering courses, projects & workshops today!Book a Live Demo