Explaining precision and recall
The first days and weeks of getting into NLP, I had a hard time grasping the concepts of precision, recall and F1-score. Accuracy is also a metric which is tied to these, as well as micro-precision and macro-precision. These metrics are important in general machine learning and deep learning as well.
However one of my colleagues (Thanks Marci!) explained it in an excellent way in which I’d thought I’d share. Many of you have probably already seen this, but for me it was a revelation in its simplicity. Credit is also due to “Foundations of Statistical Natural Language Processing” the book where I think this first surfaced.
Low recall, low precision
Let’s explain the meaning of “True negatives”, “False negatives”, “True positives” and “False positives”
- True negatives (denoted tn) samples in your data, which you classified as not belonging to your class correctly. Eg. your “hot dog” vs “not hot dog” image classifier correctly classified your image of a car as not being a “hot dog”.
- False negatives (denoted fn) samples in your data, which you classified as not belonging to your class, incorrectly. Eg. your “hot dog” vs “not hot dog” image classifier incorrectly classified an image of a messed up “hot dog” as not being a “hot dog”.
- True positives (denoted tp) samples in your data, which you classifed as belonging to your class correctly. Eg. your “hot dog” vs “not hot dog” classifier correctly classifies a “hot dog” as being a “hot dog”.
- False positives (denoted fp) samples in your data, which you classified as belonging to your class incorrectly. Eg. your “hot dog” vs “not hot dog” classifier incorrectly classifies a hamburger as being a “hot dog”.
How would the charts look in other scenarios?
High recall, low precision
Our classifier thinks a lot of things are “hot dogs”; legs on beaches, fries and whatnot. However it also thinks a lot of “hot dogs” are “hot dogs”. So from our set of images we got a lot of images classified as “hot dogs”, many of them was in the set of actual “hot dogs”, however a lot of them were also “not hot dogs”.
Low recall, high precision
Our classifier is very picky, and does not think many things are hot dogs. All the images it thinks are “hot dogs”, are really “hot dogs”. However it also misses a lot of actual “hot dogs”, because it is so very picky. We have low recall, but a very high precision.
High recall, high precision
Our classifier is very good, it is very picky, but still it gets almost all of the images of “hot dogs” which are “hot dogs” correct. We are happy!
There is of course also a mathematical explanation to these different definitions:
Precision = tp÷(tp+fp)
The relation between true positives and the total number of true positives and false positives.
Recall = tp ÷ (tp+fn)
The relation between true positives to the total number of true positives and false negatives. [https://en.wikipedia.org/wiki/Precision_and_recall]
There are a lot more to say about precision/recall, but this will hopefully provide a good introduction.
In general one take away when building machine learning applications for the real world. Is that its not always necessary to get to the holy grail of having a very high Precision and Recall (F-score is another metric commonly used to convey these both metric into one coherent metric [https://en.wikipedia.org/wiki/F1_score]). Some times its more important to correctly classify your samples than getting all of them (high precision). Sometimes its more important to get all of them (high recall) than getting all of them correct.
An example of this compromise exists in our domain, media monitoring (at least traditionally), many customers expect an almost perfect recall. They never want to miss an article about the subjects they are interested in. However precision is not as important (albeit highly desirable), if you get a bit of noise in the article feed, you are usually fine.
Let me know in the comments what you think!
I work for Meltwater where we do this stuff all day, feel free to check it out www.meltwater.com. Credit to Norah Sakal for creating the explanatory images.