What is a Confusion Matrix in Machine Learning?

March 30, 2018

What is Confusion Matrix?

In the field of machine learning and specifically the problem of statistical classification, a confusion matrix, also known as an error matrix. A confusion matrix represents information about actual and classified cases produced by a classification system. Performance of such a system is commonly evaluated by demonstrating the correct and incorrect patterns classification. A confusion matrix is a table that is often used to describe the performance of a classification model (or “classifier”) on a set of test data for which the true values are known. It allows the visualization of the performance of an algorithm.

A confusion matrix (Kohavi and Provost, 1998) contains information about actual and predicted classifications done by a classification system. Performance of such systems is commonly evaluated using the data in the matrix. The following table shows the confusion matrix for a two class classifier. It allows easy identification of confusion between classes e.g. one class is commonly mislabeled as the other. Most performance measures are computed from the confusion matrix.

The entries in the confusion matrix have the following meaning in the context of our study:

• TN is the number of correct predictions that an instance is negative,
• FP is the number of incorrect predictions that an instance is positive,
• FN is the number of incorrect of predictions that an instance negative, and
• TP is the number of correct predictions that an instance is positive.

Several standard terms have been defined for the 2 class matrix

The accuracy (AC) is the proportion of the total number of predictions that were correct. It is determined using the equation:

$$AC= (TN+TP)/(TN+FP+FN+TP)$$

The recall or true positive rate (TP) is the proportion of positive cases that were correctly identified, as calculated using the equation:

$$TP= TP/(FN+TP)$$

Finally, precision (P) is the proportion of the predicted positive cases that were correct, as calculated using the equation:

$$P= TP/(FP+TP)$$

How to Calculate a Confusion Matrix

Below is the process for calculating a confusion Matrix.

• We need a test dataset or a validation dataset with expected outcome values.
• Make a prediction for each row in test dataset.
• From the expected outcomes and predictions count:
• The number of correct predictions for each class.
• The number of incorrect predictions for each class, organized by the class that was predicted.

These numbers are then organized into a table, or a matrix as follows:

• Expected down the side: Each row of the matrix corresponds to a predicted class.
• Predicted across the top: Each column of the matrix corresponds to an actual class

The counts of correct and incorrect classification are then filled into the table. The total number of correct predictions for a class go into the expected row for that class value and the predicted column for that class value. In the same way, the total number of incorrect predictions for a class go into the expected row for that class value and the predicted column for that class value.

Example confusion matrix for a binary classifier

For the simplification of the above confusion matrix we have added all the terms like TP, FP, etc. and the row and column totals in the following image.

Accuracy: Overall, how often is the classifier correct?

$$AC= (TN+TP)/(TN+FP+FN+TP) = (100+50)/(100+5+10+50)$$

Recall: Recall gives us an idea about when it’s actually yes, how often does it predict yes.

$$TP= TP/(FN+TP) = 100/(100+5)=0.95$$

Precision: Precision tells us about when it predicts yes, how often is it correct.

$$P= TP/(FP+TP)= 100/(100+10)=0.91$$

References

[1] Provost, Foster, and Ron Kohavi. “Guest editors’ introduction: On applied research in machine learning.” Machine learning 30, no. 2 (1998): 127-132.

[2] “Confusion Matrix”, available online at: http://www2.cs.uregina.ca/~dbd/cs831/notes/confusion_matrix/confusion_matrix.html

[3] Jason Brownlee, “What is a Confusion Matrix in Machine Learning”, available online at: https://machinelearningmastery.com/confusion-matrix-machine-learning/

[4] “Simple guide to confusion matrix terminology”, available online at: http://www.dataschool.io/simple-guide-to-confusion-matrix-terminology/

$${}$$