Notes from the Field
The Significance of Statistical Insignificance
A Story of Birds, Graphs, and the Scientific Process
It is early morning when I lean back from my desk with a sigh and rub my temples. My computer is displaying a string of numbers that I have been trying to get for over a month. Weeks of scouring trail camera photos and manually inputting data line by line has finally led me to the numbers staring back at me on my screen. I pinch the bridge of my nose and let out another tired sigh.
This string of numbers, a statistical test for comparing the data, confirms my fears. The comparison I have been researching is statistically insignificant, the data does not support my hypothesis. I shake my head, close my laptop and go to sleep thinking this is a problem for future me. I am wrong about there being a problem though. What I have just experienced and what I am sure many of us have encountered before is simply a part of the scientific process.
What is Statistical Significance?
In a scientific sense significance is the statistical ruler by which we measure two or more subjects. When we measure these subjects, we are looking to find distinct relationships or comparisons between them. In many cases, these relationships are visible to us:
This plant grows because of the presence of sunlight.
or
This plant grows better in direct sunlight than this other plant.
Sometimes however, the significance isn’t so easily visible. To examine these topics, scientists rely on a series of statistical tests to analyze data and determine if the relationships or comparisons are strong. The results of these tests produce something called a p-value. The lower this number, the more statistically significant the relationship or comparison being analyzed.
Typically we look for a p-value lower than 0.05 before we consider the statistical existence of a relationship or difference between the subjects studied. As mentioned before my particular research just underwent these tests and revealed a p-value much higher than what I was hoping for.
Seeking out Significance
Since October, I had been comparing bird activity at three bird feeders. I was trying to answer how seed-eating birds selected their forage sites. My first hypothesis was that birds used surrounding color to determine potential foraging sites. My second was that individual species would prefer different colors than other species in the area.
To test this, I had set up three feeders and game cameras along with them. For the next month I watched these three feeders, identical in all but color. In that time, I received 2,387 visits from four species of birds and a staggering 11,000 visits from squirrels, deer, raccoons, and people. After I manually submitted all this data however, the analysis did not match what I had been hoping for.
My p-values were too high, meaning color was statistically insignificant when birds were choosing which feeder to eat from.
The disappointing truth is that sometimes our hypotheses are not supported by the data. My educated guess was wrong. Even after sorting through nearly 14,000 images, my hypothesis could still be wrong. After putting all that effort in, it hurt to realize this. Frustrated, I spoke with my friends doing similar projects and found it is not uncommon to feel defeated when this happens or even feel like you failed completely.
But neither of those feelings are true. They are the result of becoming too attached to the hypothesis we made. In my case, it took me several days of processing the results in my own head and talking with other colleagues to understand this. I was not defeated, and I had certainly not failed. I was instead on the cusp of deeper understanding; I was at the crossroads of the scientific process.
Thanks to the advice and perspective of my colleagues and project mentor I was reminded that science is not about proving my own theories right. Science is about learning and understanding. We use hypotheses to build a framework by which we can measure the data we collect. The solution we come to is valuable regardless of whether is supported or failed to support our initial guesses. The only time we can truly fail in these situations is by refusing the results we get. We can certainly retest them, repeatability is a backbone of the scientific method after all, but refusing repeated results only serves to reveal our own stubbornness. Instead, when the results do not match our expectations, we need to look at those results with the intend to understand why we got them. I could either give up or refuse these results or try to understand them better.
The Significance of Insignificant Findings
For my study, I had focused on the two species that visited the feeders most often: the Steller’s Jay and the Red-Breasted Nuthatch. The Steller’s Jays are a gregarious cousin to the Blue Jay. Found throughout the mountainous areas of the western United States, these birds are known for being active, inquisitive, territorial, and noisy. Red-Breasted Nuthatches on the other hand are tiny forest dwelling birds known for their energy, unique calls, and tendency to climb vertically up and down trees.
After compiling my data, I needed to understand why the Steller’s Jays visited so often but so irregularly each day. I began researching their behavior and what I found was fascinating. Steller’s Jays are well observed caching birds which explained why they made so many repeated trips to my feeders each day.
What was even more interesting was a study I found detailing their distaste for competition from other jays and general disturbances to feeding sites. Both details shed significant light on why they behaved like the did at my feeders. Furthermore, it showed me that the placement of each feeder nearby and the 11,000 disturbances I recorded at the feeder may have had unforeseen impacts on Jay behavior that could have limited their display of preference in my test.
The Nuthatches posed a different problem. Visibly there appeared to be a preference for the red and silver feeder over the green one. But, statistically, I knew this comparison was insignificant. In researching their behavior, I was not able to find much that could help explain this. I found a few recorded instances of similar species using color to select the best fruits to consume, but nothing related to these Nuthatches. By the end of my search, I was frustrated more than defeated.
It was about this time my mentor on the project forwarded an article by Dr. Valentin Amrhein titled Retire Statistical Significance. I will spare you the detailed review of it as you can read it yourself. Simply put, the article argues that there tends to be an over-reliance on statistical significance in many scientific publications to the point where visible significance like the one found with my nuthatches are being overlooked and ignored because they do not meet a standardized measure that somehow applies to every study topic and statistical analysis.
Acknowledging What We See
This is not to say statistical significance thresholds are unimportant, but rather that statistically insignificant results do not immediately mean there is not possibly an existing or visibly obvious relationship. This article helped breathe some new life into my project which had stalled in the search for nuthatch behaviors. With this new understanding I could safely say that the nuthatches were displaying a preference towards red and silver over green. I could not say it was distinct and significant preference, but I also did not need to ignore what I saw.
Finally I was able to finish the paper tied to this project. While I know it won’t be published or peer reviewed, I’m not worried because I learned a lot about the process of scientific understanding. If by chance you want to know more about the project itself, I’ve added a link to my Instagram highlights which details the project in more depth. Please check it out, I’m a firm believer that science should be shared and communicated, even when the results are like mine.
Ultimately, neither of my hypotheses were supported by the data I collected. When comparing all the birds I studied, they did not show any preference towards any one color. When I looked at each species individually however, I was able to find my results supporting existing observations of Steller’s Jay behavior, potential flaws in my study methods, and the possibility of some color preference in Red-Breasted Nuthatches. These are not the statements of failed science, they are opportunities to further research, repeated studies, and future discoveries. All of which I can attribute to my unsupported hypotheses, supportive colleagues, and distinctively insignificant results.
