How does the algorithm measure this?
Mathew Lowry
11

To calculate idf, we divide the total number of documents in the corpus by the number of documents containing that word and take the log of this quotient. This statistic will be large for words that are rare and small for words that are common.