Pseudo-Therapy Apps: The Fad Diet Of Mental Health

By Irene Yancher, PsyD. Originally posted on TechCrunch.

There’s a life-threatening condition that prevents people from feeling pain. An afflicted person may leave their hand on a hot stove without realizing it. There are only 20 reported cases, because the genes responsible have been weeded out by evolution. Pain is adaptive.

Like a tea kettle whistling louder and louder, pain implores action — move your hand! In more complex cases, pain’s message and origin may be less evident, but equally important to understand. Although you may take aspirin to relieve daily headaches, it’s vital to determine the symptom’s cause.

The same is true for emotional pain. Sometimes it’s easy to understand and alleviate an unpleasant emotion. Perhaps you feel afraid walking alone on a deserted street, and fear is telling you to take a different route. Other times, emotional pain is more complex. We may find ourselves feeling sad every day and not know why.

The sadness could be telling you that something in your life needs to change. Psychotherapists help people discover the root cause of symptoms like anxiety and depression by bringing attention to issues and patterns that may lie outside of awareness. Perhaps you are in an unhappy relationship or governed by rigid rules, such as “telling others what I want is selfish.” These things may be difficult to recognize or acknowledge on your own. Psychotherapy helps you question the status quo and, if you want to, change it.

However, psychotherapy takes work — it’s not a quick fix. In many ways, it is analogous to physical exercise. Just like finding a workout that’s effective for you, finding a psychotherapist can be a laborious process. And even after a psychotherapist or an effective workout is found, you must invest time and energy into the process. In some sessions/workouts, you may realize great progress, while in others, you may simply feel pain. And just like workouts, psychotherapy has to be done on a regular basis before results are evident. The commitment that both psychotherapy and physical exercise require leave the door open for alternatives to enter the marketplace.

With more than 40 million people suffering from mental health disorders, there is a large opportunity to profit by promising quick fix remedies to emotional pain. Leveraging ubiquitous mobile devices, a single app can deliver these quick fix remedies in a highly scalable way. Joyable, Happify, SuperBetter and Personal Zen are examples of pseudo-therapy apps that have recently entered the mental health field.

What are their claims?

• “90% of our clients see their social anxiety decline.” — Joyable
• “Our online CBT program is proven to reduce social anxiety in just 12 weeks.” — Joyable
• “86% of frequent users get happier in 2 months.” — Happify
• “Playing SuperBetter for 30 days improves mood, reduces symptoms of anxiety & depression and increases self-confidence.” — SuperBetter
• “Personal Zen Scientifically Reduces Anxiety in 25 Minutes” — Personal Zen

As a psychotherapist, I am troubled by this misguided, quick fix approach to mental health. We’re promised relief in 3 months, 2 months, 30 days, 25 minutes.

As a (former) statistician, I am appalled by the misused statistics and flimsy science underlying these assertions. Marketing ploys that use bloated claims exploit society’s Achilles’ heel: a quick fix (backed by statistics).

What are we to believe about the efficacy of this approach to mental health if we can’t trust the numbers? Do the scientifically backed games and exercises become just games and exercises? Similar marketing ploys were used with brain games like Lumosity. Once the “science” of Lumosity was debunked, it became clear that the only thing Lumosity made you better at was Lumosity itself.

In 2005, John Ioannidis, a professor of statistics at Stanford, published a paper called “Why Most Published Research Findings Are False.” It has become the most downloaded technical paper in the history of the PLOS Medicine journal. Ioannidis has been quoted summarizing his findings with this explanation: “It feels good to validate a hypothesis. It feels even better when you’ve got a financial interest in the idea or your career depends upon it.”

This observation manifests itself in the way an experiment is designed and in how results are interpreted. Even when an experimenter has the best intentions at heart, the unconscious desire to have a specific result can influence subtle behaviors and decisions.

In the case of Happify and Joyable, the statistics are self-reported and calculated by their respective teams using their own data sets. The researcher who conducted the Personal Zen study was also its founder, and the researchers who conducted the SuperBetter study include the company’s co-founder and Science Advisor. Let’s explore the selective reporting and methodological shortcomings for these pseudo-therapy apps.

Joyable and Happify’s reported success rates employ a common statistical fallacy known as survival bias. Consider how Happify’s 86 percent was calculated. By ignoring non-frequent users and any users who dropped Happify within 2 months, Happify systematically excluded people with one common trait: they quit Happify (perhaps it wasn’t effective for them).

Similarly, in the SuperBetter study, 74 percent and 82 percent of users in the study dropped out by weeks 4 and 6, respectively. When a large group that’s churned is excluded from analysis, no logical claim can be made about success. Joyable and Happify have not responded to questions about the number of people who quit (i.e., user churn).

The reported success rates are analogous to asking people at a Nickelback concert whether they like Nickelback. An honest statistic would tell us the number of people who benefited from each pseudo-therapy app out of all people who gave it a shot.

How Happify excluded users to come up with an 86 percent success rate. 86 percent = (Frequent users for 2+ months and reported happier) / (Frequent users for 2+ months).

The Personal Zen and SuperBetter studies have other significant methodological problems:

• Sampling Bias: Personal Zen and SuperBetter participants were recruited from an intro psychology course and a self-help website, respectively. Whenever a population for testing is not selected at random, but rather from a specific group, results cannot be generalized to the rest of the population. Participants from a psychology course or a self-help website may have several traits in common that influence how a pseudo-therapy app affects them. They are also more likely to understand the intention of an app’s exercise, and may thus report a positive effect. Sampling bias is especially significant in studies with small sample sizes.
• Small Samples Sizes: Small sample sizes make it more likely to observe an effect that’s not really there. Imagine standing on one leg and flipping a coin and getting 4 heads in a row. That result may statistically imply that standing on one leg is correlated with flipping heads, but the low sample size means you can’t make any meaningful conclusions. A larger sample size of coin flips would clearly eliminate the spurious correlation.
• Multiple Tests (and the criteria for statistical significance was not adjusted to take this into account): This point is best exemplified by the website Spurious Correlations. In it, completely random events are found to have very high correlations, such as “the per capita consumption of Mozzarella” has a 96 percent correlation with “the number of civil engineering PhDs awarded.” The more tests conducted, the more likely you are to identify a statistically significant correlation.
• Survival Bias (SuperBetter only): With 74 percent of participants churning after week 4, it’s impossible to draw any conclusions from the study. These 74 percent may have one thing in common: SuperBetter was ineffective for them.
• Placebo Effect (SuperBetter only): In pharmaceutical experiments, the control group receives a placebo pill but isn’t told it’s a placebo. This is done to distill the true impact of the medication, since just believing that you are receiving treatment is enough to induce positive physical and psychological effects. The placebo effect is significant — “in several recent studies, placebos have performed as well as drugs that Americans spend millions of dollars on each year.” With apps, there is no equivalent of a placebo except perhaps a similar game. However, in the SuperBetter study, the control group was on a waitlist and received no treatment (i.e., they did not play any game). Knowing you are in the control group can impact both actual and self-reported mood.

After presenting these methodological critiques to the Personal Zen and SuperBetter researchers, I was not surprised to receive acknowledgements of each respective study’s shortcomings. Personal Zen’s founder, wrote: “I think your critiques are fair.” SuperBetter’s Science Advisor wrote: “In response to the points you noted: It’s certainly true that this study, like all studies, is limited.” Full email responses from Personal Zen and SuperBetter can be found HERE and HERE, respectively.

Designing and implementing truly rigorous tests is difficult, expensive and time-consuming; moreover, the results may be unfavorable. However, barriers to conducting rigorous research does not justify exaggerating claims of success.

Louis CK has a joke about how we overuse the word “amazing” by using it to describe a burger at Wendy’s. We can’t allow the word “science” to lose its meaning when it’s related to people’s health. Flimsy science used for promotional purposes trivializes the significance of mental health issues.

Even the self-reported science behind the pseudo-therapy apps is notably silent in regard to long-term effectiveness. The marketing angle is centered on quick relief. This is because apps cannot provide a personalized approach. For instance, if you are sad because of dissatisfaction with your partner, an app will not know whether you need to change your view on relationships, perception of yourself, approach to conflicts or maybe even change your partner. This type of individualized approach does not scale. Yet without a holistic understanding of yourself and how you navigate the world, long-term mental health is difficult to attain.

As a psychotherapist, I can empathize with people who look for relief with pseudo-therapy apps. Oftentimes, anxiety or depression can be so severe that it is hard to pursue any meaningful change, and understanding the source takes time. A quick fix can be tempting.

In this way, pseudo-therapy apps are strikingly similar to fad diets. A sustainable, healthy diet coupled with exercise is the tried and true method for losing weight and keeping it off. In spite of this common knowledge, smart people fall prey to the latest weight-loss fad because their hope overwhelms their common sense. Every decade there is a new fad diet with brazen claims for rapid weight loss. In 1903, chewing each bite of food 32 times was in vogue and in 1932, the cigarette diet made its debut. More recently, Kellogg’s claimed that you could lose 6 pounds in just 2 weeks if you replaced 2 meals with Special K.

It is tempting to believe that you can lose weight by replacing meals with sugary cereal. It is much easier to do that than commit to a healthy lifestyle. Similarly, it is tempting to believe that you can conquer negative thoughts by slinging cupcakes at little monsters named “sadness” (this is a Happify game).

People who spent 2 weeks eating Special K also wasted 2 weeks hoping this diet would work. Similarly, people who spend months using pseudo-therapy apps may delay thinking about how to gradually achieve long-term happiness. That’s the best-case scenario. In the worst-case scenario, these apps are used in isolation to silence an important message your emotions are trying to send. As a result, an opportunity for meaningful change may vanish.

The history of fad diets and brain games is repeating itself with mental health, but it’s not too late to lift the veneer. Let’s not be fooled by “scientific” claims backed by misleading statistics. In the case of pseudo-therapy apps, if we strip away the marketing ploy, we’re left with an Angry Birds knock-off masquerading as science.

Screenshot from a Happify game resembling Angry Birds.

Thanks to Alex Yancher for helping me research and write this article.

Featured Image: Russell Werges

Irene Yancher, PsyD is a psychotherapist practicing in San Francisco. If you are interested in seeing Irene for psychotherapy, please visit Liv Psychotherapy.