The Scientific Method
and the importance of responsible writing
Any longer-term readers of Pragmatic Life may have noticed that my publish rate has drastically slowed over the course of this blog’s life. Contrary to what may be expected, this is not in any way a result of me being distracted, losing focus, or running out of ideas. It is in reality due to my own research and thinking about what comprises responsible writing.
You see, when something is written and released to the public, and some number of people read and appreciate it, it makes an influence in their lives. It could be a tiny influence, or a complete lifestyle change, and I’m sure readers will be able to recall occasions in which the result of reading a piece has resulted in either one of these effects. But that is powerful! The saying “the pen is more powerful than the sword” isn’t well-known for no reason. On many occasions throughout history, literature and other creative works have driven and helped to inspire wide-scale changes in society. Changes like ending slavery, improving treatment of orphans, and decreasing the horrors of war, among many others. Writing is powerful, and with great power comes great responsibility.
The more I dwell on this thought, and the more of others’ work I read, both amazing and terrible, the more I feel a strong sense of responsibility to write carefully, responsibly, and ensure that the things that I say are well-researched and carry the mission of inspiring something positive in everyone who reads them. And writing a piece like this simply takes a lot more effort than spilling your brain, looking it over for edits, and hitting the publish button.
An Example
The other day, I came across an article written by Ev Williams, a founder of Twitter and Medium. This is a man who is brilliant, and highly admired and respected by many. Needless to say, anything he writes will get a lot of attention and respect. In case you are curious to read the full piece, you can check it out here.
The article was about working parents. Essentially, Ev was curious about the working hours of “highly successful” parents, so he sent out an informal survey to some of his friends and connections, collected the results, and published them as statistics for everyone else to see.
He did note at one point in the article that the results were from “a very non-scientific, selected-with-my-bias group of folks”, but then noted that nonetheless, “the survey response rate was high enough to draw some interesting conclusions”.
Was it though? We’ll get back to this later.
Drawing Conclusions
Drawing conclusions about things is really hard. There is simply no absolute truth, so any conclusion that you draw about anything could be wrong. However, as human beings, we really enjoy drawing conclusions about things, and in fact rely on the conclusions we draw in order to guide our lives, our societies, etc. We’ve discussed how your own opinions and life experience are not legitimate enough sources to be trusted when it comes to building important values for your life. So what is?
It turns out this is not a unique line of questioning, and people have been asking this for hundreds of years. And indeed, hundreds of years ago, human beings invented a system for evaluating conclusions that, in the opinion of the intellectual community, is the best we can do — it gets us as close to the truth as we can manage. And on top of that, when it comes to numbers, with math and statistics we have ways to calculate and show directly the likelihood of the results being a mistake or coincidence.
Truly, this was an incredible breakthrough for human beings. This standardized way of investigating a hypothesis (thought about something that might be true) to evaluate it’s true-ness was deemed the scientific method, and and entire field of study developed around it called science. Advancements within science spurred incredible innovation and growth in human society, and pushed us far beyond the level that we ever would have naturally come to. In short, it was revolutionary for our entire race, and continues to be to this day.
The Science Bubble
So you might think that science would become the standard upon which we base many, if not all of our conclusions. But for some reason, this is not at all what has happened. Science tends to be contained to its own field. You can take classes in science, you can become a researcher or scientist, and to all other fields of study or lines of work, it’s irrelevant. Conclusions are drawn arbitrarily just as before.
Think about it. Politicians tend to be uneducated in how science works, and frequently make claims that fly directly in the face of it, such as denying climate change, and this is seen as perfectly acceptable. People write articles on the internet and in newspapers where they make claims about what people think, how they work, and display graphs without any statistics or scientific backing whatsoever. Presenting information that is not backed by science outside of the field of science research is not only considered perfectly valid and acceptable, it’s the norm.
But at the same time, presenting research or opinions on the way things are that have not been investigated scientifically means that they are more likely to be wrong. Much more likely, in fact. On top of that, while science requires you to discuss the limitations of your research and the likelihood that you could be wrong, this is entirely unnecessary and in fact frowned upon outside the field. So every time you read something that has not been investigated scientifically, there’s a good chance it could be bullshit, and on top of that you will never know how high that chance is. That’s scary.
It Could Be Bullshit
Within the scientific community, there is still a whole lot of bullshit published, even through the filter of scientific rigor, peer review, and such. For example, check out this science paper that somewhat ironically presents the fact that most science papers are wrong, just because they do not carefully and rigorously follow the rules set forward by statistics and the scientific method. Or this article, which breaks down in detail how easy it is to fiddle around with numbers and produce a valid p-value for an arbitrary relationship. Or this article, which discusses how scientific-sounding pseudoscience is frequently used to dismiss legitimate science, or argue against the principles upon which the scientific method itself was founded.
I could keep going. The number of people writing about how much bullshit is published as science is enormous, because there’s a lot of it happening. The entire “vaccination causes autism” trend was based on an invalid yet still published science paper, in fact.
But despite this, we are still talking about what has been agreed upon by humans as the most reliable method for getting as close to the truth as possible.
So now imagine how much bullshit is published outside the field of science, where there are no rules, no standards, and no review.
Take a moment, just consider it.
The Harm of Irresponsible Writing
I consider publishing numbers, graphs, or grand conclusions that are not backed by science to be vastly irresponsible, because of the staggeringly high chance that they are bullshit. Therefore, I consider writing that presents unscientific conclusions to be also irresponsible. Well, unless it includes giant caveat stating that there is an enormous probability that this is not correct, and readers are encouraged to do independent science research in order to validate the points (which not only never happens, but most people don’t know how to do).
But why is it irresponsible? Freedom of speech, people can say whatever they want, right? You can always ignore it, right?
Not exactly. When people read something coming from a source that seems authoritative, most of the time, they simply believe it. Numerous studies have shown that people blindly trust authority, most notably the famous Milgram experiment. So if you write something up that seems credible, don’t back it with any science, and publish it, people will probably read it and believe it despite the caveat of it having a staggeringly high chance of being bullshit.
Then, if they take that knowledge, and apply it to their lives, they are now basing a part of their lives on a fact that has a very high chance of being incorrect. How many people do you know that say MSG is bad for you, in fact? Have you asked if they ever tried googling it?
Back to the Start
So let’s return to Ev Williams’ article. This article is a perfect example of publishing pseudoscience results in an authoritative way, and such that it directly affects the lives of many people, in what I see to be a negative manner.
Here it is again, in case you didn’t get a chance to read or skim it before:
https://medium.com/working-parents-in-america/growing-kids-and-growing-companies-8657049a057c
The topic we are discussing here is work-life balance for parents. In other words, how much time you spend with your children vs on your work. This is not a light topic at all. This is a very heavy topic, and small changes in this balance can produce large changes in both the parents’ and children’ lives. It must be approached very carefully.
But Ev jumps right in with no shame. He says that he took an informal survey of highly successful people with children to ask a few questions about their work life balance, in order to figure out what’s normal. Now we layer up this sensitive topic with a couple more very dangerous lines of thinking.
First, successful is presented without any additional qualification. What is “highly successful” for Ev? Is it someone who has a lot of money? Holds a high position in a big company? The truth is that everyone has different definitions of success. Some people think it would be great success if they were a CEO of a big business making millions of dollars and driving expensive cars. Others are disgusted by this idea and see success in owning a small farm and being surrounded by family and friends, having enough to live comfortably but not much more. And there’s a whole range in between. By presenting “highly successful” without any qualifications, Ev implies that success for the reader is the same as what it is to him. And being a well known and obviously very smart person, he is looked up to by many, and that assumption subtly sinks in to the back of their minds.
Second, he is trying to figure out what’s normal for parents who are also “successful” in their careers. However, I’d like to present the counter argument of who cares what’s normal? You should not be deciding how much time you spend with your own children based on “what’s normal”. You should especially not be deciding how much time to spend with your children based on what is normal for people that Ev Williams considers to be successful. But yet, Ev published this information for a reason, and it wasn’t for people to read it and say “eh, who cares”. It was for them to read and base their own values around it. The direct implication is “Successful people spend an average of this amount of time on work and this amount on kids. Therefore, you should have around the same split if you want to be successful.” While this is already a cause and effect fallacy (or cum hoc ergo propter hoc if you want to be a rationality hipster), it’s a very direct statement to working parents, and will doubtlessly make an impact on the lifestyle choices of readers. And the “normal” recommendation from the article is working ~60 hours each week, which, if you work 5 days a week, is about 12 hours a day. This is a very high number, especially when you have children to care for.
Now on top of all that, he presents data that is entirely unscientific. There are been an enormous number of surveys taken and published under scientific standards, and it’s not hard to figure out how to select a group for your survey and run the numbers to show how accurate it might be. We all probably learned this in basic science and/or statistics classes. And the consequences of not doing so are vast. As discussed above, they increase the chance of the results being bullshit by an enormous amount. And yet Ev dives directly into this very touchy subject, knowing that he is an influential person and the purpose of the article is to influence people, and presents data that is very likely to be inaccurate.
Responsibility & Speed
I have slowed down significantly with my writing here on Pragmatic Life, and this is part of the reason. I write a lot about big lifestyle changes and values, and I want to make sure that what I’m saying is as close to the truth as I am able to get. But it’s a lot more work being responsible. Doing scientific research is not a quick or easy task, and on top of that, after doing research it’s not infrequent that I find that my ideas or thoughts were actually wrong, and have to go back to the drawing board.
I hope that my adherence to these principles improves the quality of the pieces here, and that any readers who got this far will start to be a little more skeptical of reading things with no citations or backing, no matter where they are found, or who wrote them.
Photo is the first one in an article I haven’t taken (gasp!). Taken by NASA, an organization that does a whole lot of science and math before making any decisions. You’d better believe nobody is making arbitrary calls about space flight!
If you liked this article, please “recommend” it by clicking the little heart below. Thank you!
