Ten Fatal Flaws in Data Analysis
Things that will make you redo your entire analysis … if you recognize them.
Mistakes happen, especially in work as complicated as analyzing data. But there’s a difference between minor lapses and fundamental flaws. If you can fix a mistake without redoing your entire analysis, it’s not a fatal flaw. Here are ten examples of real fatal flaws.
1. Where’s the Beef?
In a way, the worst flaw a data analysis can have is no analysis at all. Instead, you get data lists, sorts and queries, and maybe some simple descriptive statistics but nothing that addresses objectives, answers questions, or tells a story. If that’s all you want, that’s fine. But, a data report is not a data analysis. Reports provide information; analyses provide knowledge. It’s like with your bank account. Sometimes you just want a quick report of your balance. That information has to be readily available whenever you might need it and both you and the bank have to be working with exactly the same data. If you want to assess patterns in your spending, though, you have to conduct an analysis. Say you want to figure out how much more you’re spending on commuting over the past five years, you’ll have to compile the data and scrub out anomalies, like the cross-country driving you did on vacation, to look for patterns. Analyses involve much more than a glance. They take time, sometimes, a lot of time. To make sure you’re getting what you need, look beyond the data tables for models, findings, conclusions, and recommendations. If they’re not there, you didn’t get an analysis.
2. Phantom Populations
If there were to be a fatal flaw in an analysis, it would probably involve how well the samples represent the population. Sometimes data analysts don’t give enough thought to the populations they want to analyze. They use observations to make inferences to a population that doesn’t exist. Populations must be based on some identifiable commonalities that would meaningfully affect some characteristic. A group of anomalies would not be a population. Opinion polls sometimes suffer from phantom populations. Say you surveyed people wearing red shirts. Could you then generalize to everyone who wears red shirts? Canadian researchers found one such phantom population when they tried to create a control group of men who had not been exposed to pornography. Make sure the population being analyzed is more than an illusion.
3. Wow, Sham Samples
Sometimes the population is real and well defined, but the samples don’t represent it adequately. This is a common criticism of opinion polls, especially election polls. It was the reason cited for why exit polls during the presidential election of 2004 indicated that John Kerry won many precincts that ballot counts later awarded to George Bush. Medical and sociological studies may have sham samples because it is often difficult to select subjects to match some target demographic. Likewise, environmental studies can suffer from inconsistencies between soil types or aquifers. To identify sham samples, look for three things: (1) a clear definition of a real population, (2) a description of how samples were selected so that they represent the population, and (3) information about any changes that occurred during sampling, such as subjects being dropped or samples moved.
4. Enough Is Enough
The number of samples always seems to be an issue in statistical studies. When there appears to be too few samples, question confidence and power; when there appears to be too many samples, question meaningfulness. Usually analysts are ready for this question but beware if they cite the old familiar fable about using 30 samples. It may indicate their understanding of statistics is not as formidable as you supposed. Also, if they appear to be using a reasonable number of samples but then break out categories for further analysis, make sure each category has an appropriate number of samples for the analysis they are doing.
5. Indulging Variance
Most people don’t acknowledge variance. They don’t even know it’s there. If their candidate for office is up by two percentage points in a poll, they figure the election is in the bag. Even professionals like scientists, engineers, and doctors don’t want to deal with it. They ignore it whenever they can and just address the average or most common case. Business people talk about variances all the time, only they mean differences rather than statistical dispersion. Baseball players thrive on variance. Where else can you have two failures out of every three chances and still be considered a star? Data analysts have to understand variance and address it at every step of a project. Look for how variance will be controlled in study plans. Look for variance to be reported with results. And most importantly, look for some assessment of how uncertainty affects any decisions made from the analysis.
6. Madness to the Methods
NASA uses checklists to ensure that every astronaut does things correctly, completely, and consistently. Make sure the analysis you are doing or reviewing takes the same care. If there are multiple data collection points or times, be sure there is a standard protocol or script for generating the data. Be especially concerned if the data are collected over multiple years. Better and cheaper methods and equipment are continuously being developed so be sure they are compatible. Be sure the data had been scrubbed adequately of errors, censored and missing data, replicates, and outliers. Finally, be sure the data analysis method is appropriate for the numbers and natures of the variables and samples.
7. Torrents of Tests
If a statistical test is conducted in a study, false positives and false negatives can be controlled, or at least, evaluated. But if there are many tests, you can bet there will be false results just because of Mother Nature’s sense of humor. In groundwater testing, for example, there may be a test for every combination of well, analytes, and sampling rounds, resulting in literally hundreds of tests. There are strategies for dealing with this type of situation, such as hierarchical testing and the use of special tests (search for the term Bonferroni). Be careful of bad decisions based on a small proportion of the tests being (apparently) significant.
8. Significant Insignificance and Insignificant Significance
Here’s where you have to use your gut feel. If a test is statistically significant and you don’t believe it should be, ask about the confidence level and whether the size of the difference is meaningful. Just as correlation doesn’t necessarily imply causation, significance doesn’tnecessarily imply meaningfulness. If something is not statistically significant and you believe it should be, ask about the power of the test and the size of the difference the test should have detected. Be sure the study looked at violations of assumptions. Also, look for what’s not there. Sometimes studies do not report non-significant results. Such results could be exactly what you’re looking for.
9. Extrapolation Intoxication
Make sure the data spans the parts of the variable scales about which you want to make predictions. For example, if a study collects test data at ambient indoor temperatures, beware of predictions made for freezing conditions. Similarly, be careful of tests performed on rabbits that are extrapolated to humans, maps showing information beyond the borders observed, surveys of one demographic extrapolated to another, and the like. Perhaps the only example of extrapolation that is even grudgingly accepted by statisticians is time-series analysis. You have to extrapolate to predict the future. The issue is how far into the future is reasonable, which will depend on the degree of autocorrelation, the stability of the data, and the model.
10. Misdirected Models
Models are great tools for helping you understand your data. Statistical models are based on data. Deterministic models, though, rely on theories, mainly the theories believed by the researcher using the model. But deterministic models are no better than the theories on which they are based. Misdirected models involve researchers creating models based on biased or mistaken theories, and then using the model to explain data or observed phenomena in a way that fits the researchers preconceived notions. This flaw is more common in areas that tend to be more observational than experimental.
Have you seen any of these fatal flaws?
Originally published at http://statswithcats.net on November 7, 2010.