Does this curve make my test look inaccurate?
The new applications for ROC curve analysis in psychological testing
If you’re reading this, you’re probably a seasoned test-taker. From pre-K to grade 12, the average student has taken 112 standardised tests. Factor in the other tests we take outside of school — personality inventories, aptitude tests for job interviews, college placement exams, psychiatric assessments — and suddenly the number becomes unfathomably large. Tests are heavily involved in our lives — how can we ensure that they provide us with useful information? In short, adopt smarter test design strategies.
If a test is ideally designed, it is predictively valid, and its results tell us something meaningful. The problem is that not all tests are created equally. An informative test needs to be reliable, sensitive, specific, accurate, and predictively valid. If any criterion is compromised, the test results are virtually meaningless. This has grave implications for the clinical applications of testing; since tests are so often used to diagnose psychiatric disorders, a healthy person could be wrongly diagnosed due to a false positive result, or an affected person could miss a diagnosis due to a false negative result. As it so happens, this proves to be a very frequent occurrence — an alarming 12 million Americans in outpatient clinics are misdiagnosed every year!
So why is it so difficult to create an informative (read: accurate) test? As it turns out, addressing the issues of specificity and sensitivity are no easy task. Thresholds must be chosen carefully. Too low and the test will yield too many false positives; too high and the test will yield too many false negatives. Using sensitivity and specificity as measures of accuracy often comes at a compromise to the informative value of the test.
In recent years, ROC curve analysis has been used by clinicians and psychologists as an alternative solution to the specificity/sensitivity problem. Short for Receiver Operating Characteristic, the ROC curve is a function that represents the true positive (correctly diagnosed, affected individual) rate over the false positive (misdiagnosed) frequencies. In a nutshell, curves are constructed based on cumulative probabilities. The more leftward the curve appears in the upper field of the graph, the more accurate the test.
The more leftward the curve appears in the upper field of the graph, the more accurate the test. The area under the curve (AUC) is represented as a point value corresponding to accuracy. The closer the AUC is to 1.0, the more accurate the test is, and values below 0.7 reflect poor accuracy, meaning that the test does not discriminate well between false and true positives.
ROC curve analysis allows you to compare tests to see which ones have greater AUC values. In terms of test design, this is an incredibly useful feature, because often when comparing tests side-by-side for accuracy, sensitivity and specificity values alone are often insufficient to make a judgment — ROC curve analysis provides a great solution to the accuracy problem in psychological testing.
