Confusion Matrix — Case Study

Prathamesh Mistry
Jun 6 · 4 min read

The following article will help you get over the confusion created due to Confusion Matix in a Classification Problem.

Nowadays machine learning and AI are used in almost every domain in the industry. Evaluating our models is as important as building them. Let’s try to understand the confusion matrix — an evaluation method and its attributes from a perspective of a Cybersecurity Risk Analyst.

Consider a scenario in which we have to detect the anonymous system which is hitting our private servers. We have put up a Machine Learning system in between the firewall and the server. So basically we are using machine learning to detect the malicious system that is able to pass through the firewall(s), that is we ought to classify whether the system passing through the firewall is malicious or non-malicious.

CASE SCENARIO

Thus, the scenario question is whether any anonymous and potentially malicious system is able to breach the system.

Let’s put up a null hypothesis —

Breach is detected into the server

Now we have to determine whether this hypothesis is to be rejected or are we failing to reject this hypothesis.

Suppose there are 10,000 total hits on the server in a day including all the employees from various departments of our company and the malicious users who are trying to steal your data or hack into your systems.

Our machine learning model has classified the following: —

Total hits to the system: 10000

  • Hits classified as Non-Malicious: 9465
  • Hits classified as Malicious: 535

After Further Investigation into the matter, we found that —

  • Out of those 9465 hits which were classified as non-malicious, 8809 of them were actually non-malicious.
  • Out of those 535 hits which were classified as malicious, 437 of them were actually malicious

This is known as the ground truth.

Now, will be evaluating our Model’s Predictions with respect to its Ground Truth. Our machine learning model is performing binary classification and we could use a number of metrics and plots to gather insights on the performance of our model. One of these evaluation metrics is the Confusion Matrix.

A confusion matrix is a table that allows visualization of the performance of an algorithm. Let’s look at what a confusion matrix looks like—

CONFUSION MATRIX

It’s a 2X2 Grid. The Columns represent the classes of the Ground Truth whereas the Rows represent the classes of the Predictions. A confusion matrix is a technique for summarizing the performance of a classification algorithm. There are 4 cells — TP, FP, FN, TN.

Let’s relate this table with our application…

Let’s dig into this matrix…

  • Out of 10,000 records, our model predicted 535 records as malicious ( class 1) out of which 437 were actually malicious (class 1).
  • Out of 10,000 records, our model predicted 9465 as non-malicious (class 0) out of which 8809 were actually non-malicious (class 0).

TP — True Positives

True Positive is the total number of records that were correctly classified as malicious (class 1)

TN— True Negatives

True Negative is the number of total records that were correctly classified as non-malicious (class 0)

FP — False Positives

False Positives is the number of total records that were actually malicious (class 1) but were classified as non-malicious (class 0) by the model.

This is known as Type 1 error —

A Type I error is the rejection of the true null hypothesis. This means we reject the hypothesis “Breach is Detected” even there is an actual breach on the server.

FN — False Negatives

False Negatives is the number of total records that were actually non-malicious (class 0) but were classified as malicious (class 1) by the model.

This is known as Type 2 error

A Type II error is the non-rejection of the false null hypothesis. This means we failed to reject the hypothesis “Breach is Detected” even though there was no breach on the server.

Which error is potentially more dangerous and why?

Let’s look at what potential threats the two of these posses…

In the Type II error, we classified a system (non-malicious) of the company to be malicious. In this case, we could further investigate the user of this system to find out that this system was actually non-malicious and we could potentially drop this concern.

But in the case of Type I error, we classified a malicious system as non-malicious. This means this malicious system has bypassed our security and could cause data leaks and breach in the system. Thus, it is very crucial to keep this error as low as possible as it’s very dangerous.

That’s it for this one! Thank You!

Analytics Vidhya

Analytics Vidhya is a community of Analytics and Data…

Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore

If you have a story to tell, knowledge to share, or a perspective to offer — welcome home. It’s easy and free to post your thinking on any topic. Write on Medium

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store