ideal study participants, F. Ulman

I Too Am Looking For Evidence that Therapy Works

Frances Ulman, PhD
Jun 14, 2013 · 10 min read

Reading Harriet Brown’s critique of my profession felt like someone was pouring salt on a wound I didn’t know I had. I recently went into private practice as a therapist, leaving behind the world of research in clinical psychology with mixed emotions of elation (no more bosses!) and grief (goodbye dear research). It’s been a tough transition and Ms. Brown’s epic failure to understand the challenges faced by therapists is both hurtful to me and harmful to those who might take her critiques at face value.

Ms. Brown is concerned that patients are not getting the therapy they need largely because we therapists choose to disregard science and instead like to think of therapy as an art. Now I wholeheartedly agree that not all therapists are competent (some astonishingly so). But this common critique of my profession is filled with misunderstandings, errors, and straw men that lead the people away from a deeper understanding of the complexities that exist in the development and delivery of evidence-based mental-health care. Let’s take a critical look at some reasons why patients often do not receive therapy based solely on what the science says to do.

Truth, Science, and Purebred Dogs.

Let’s talk about what science is not. Science is not the same as truth. Science is just a method that moves us towards truth. The amount of effective interventions we have yet to uncover through research in clinical psychology will eventually dwarf what we have figured out thus far. The scientific studies that do exist are full of limitations. Nothing wrong there. It’s good science to acknowledge the limits of your particular study when publishing – sample size, drop out rates, etc. Then there are the bigger Limitations Whose Existence Shall Not Be Named. Many good ideas for therapeutic interventions never have a chance to be evaluated. Let’s look at some of the less controversial reasons why this is so. Evidence-based treatments come from clinical studies using a randomized, controlled design. Such studies require an astounding amount of time, funding, and person-power. These studies are the product of the science industrial complex where there is not equal access to largely federal funding.

The funding process is driven by very human biases about which grants deserve funding. With limited money and a lot on the line, grants that are most likely to yield positive results are preferred. And how is that likelihood assessed? Usually by comparing a proposed study to similar studies that have already yielded statistically significant (and hopefully clinical useful) results. Given the limited funds available for research I can’t say I entirely disagree with giving money to studies that are most likely to yield positive findings. But patterns in decision-making can lead to unintended consequences. With the pressure to get positive results, innovation is rarely rewarded. One cognitive-behavioral therapy (CBT) study spawns another… And another. And another. Meanwhile other potential avenues of exploration never see the light of day.

There is another unintended consequence that emerges from this system which is equally if not more serious. The selection process for funding research grants is not like the natural laws of evolution, complete with a feedback loop from therapist in the community back to those making funding decisions for the next iterations of studies. The selection process is more similar to the breeding of purebred dogs, often selecting studies based on some preferred characteristics not accounting for how other characteristics may develop. Survival no longer goes only to the fittest. Soon enough we have Labrador Retrievers with progressive retinal atrophy, and evidence-based treatments that are often not robust enough to hold up in the community therapy room.

We Can’t All Look Like Models

Does it seem like a stretch to say that evidence-based treatments – the sine qua non of therapy – aren’t the ideal choice for a therapist? Perhaps. At first I jolted at writing such a strong statement. But perhaps not. Here is a problem with evidence-based treatments that I encounter all the time as a therapist outside of academia. Research studies want to isolate variables of interest. That’s just good science. If there is a multi-million dollar, multi-site study of an intervention for anorexia nervosa they had better be able to say that any changes seen were the result of an intervention acting on the symptoms of anorexia. It is introducing confounding variables to have participants in the study who have anorexia and, say, autism, or bi-polar disorder. Confounding variables make it harder to understand the data. So participation in a treatment study for anorexia will be limited to folks who meet for anorexia but do not have a history of many other common psychological disorders (although most studies will allow for a history of anxiety or depression). Ruling out participants who have other disorders is good science. But good science is not synonymous with useful science.

As my graduate advisor used to say, having one disorder is not protective against having another. In fact, the opposite is true. Comorbidity of psychological disorders occurs at above predicted rates based on probability alone. In other words, having one disorder puts you at increased risk to have another. Let’s look at some numbers in two of my areas of specialization. In a sample of adolescent females being hospitalized for anorexia, 73% had at least one other psychiatric disorder including mood disorders (60.4%), OCD (16.8%), and substance use disorders (7.9%). In a sample of children with an autism spectrum disorder, 70% had at least one other disorder, and 41% had two or more. There is even preliminary evidence (although further studies are needed) suggesting individuals with anorexia or bulimia are more likely to have an autism spectrum disorder than those without an eating disorder. None of the evidence-based studies I know of are generalizable to such folks who have to endure living with more than one diagnosis.

So what are we community therapists to do when we have a client before us with anorexia nervosa and autism? (Which is not a theoretical question for me). Am I to adhere 100% to the CBT interventions shown to be “effective” for the treatment of anorexia in individuals who do not have autism? That option is just not viable for me. Am I to solemnly explain to my client that we don’t yet have an evidence-based treatment protocol for her? Please come back a few decades later? Obviously not. And yet when I use my clinical expertise on the treatment of autism and eating disorders to tailor a treatment plan for the suffering human before me I am accused of using a “dim-sum approach” by Ms. Brown and told that I don’t know the difference between art and science. I wish I were surprised that I even have to acknowledge such an inaccurate claim with a response. Yet I know this failure to understand what I do continues to be propagated by well-meaning people and makes what I do even harder.

I Do Not Think It Means What You Think It Means

Speaking of “effective,” just what is meant by this term? Effective is a dangerous misnomer. Ms. Brown repeatedly refers to “effective” treatments without unpacking that term, inferring that we have a proven method for curing conditions like post-traumatic stress disorder, eating disorders, schizophrenia, and depression.

Here is the scary truth: we do not know how to cure the vast majority of psychological disorders.

We are making progress in being able to manage symptoms for many disorders. Many individuals do get better. But we are a long way from having a surefire “cure” for everyone. As an example from my area, a peer reviewed journal article from 2010 on the current state of evidence-based treatments for eating disorders makes clear that our attempts to develop “effective” treatments are a mess. This article found only four randomized-controlled trials for the treatment of anorexia in adults, all conducted during the acute phase of illness. Two small, randomized-controlled studies found no difference between CBT and behavior therapy. One study found nonspecific supportive clinical management to be better than CBT and interpersonal therapy. Finally, one other study comparing CBT and cognitive therapy to dietary and nutritional counseling had such a high dropout rate that no comparisons could be made. This review article ends on a sobering note. “In conclusion, evidence for treatments in anorexia remains limited despite decades of randomized-controlled trials, and there are no definitive treatments that work best for the majority of patients.”

If my sunscreen were as effective as these evidence-based treatments, an aisle of tomato sauce would be my perfect hideout spot.

The Elephant in the Research Lab

I have saved this point for last because it is uncomfortable for me to admit this one openly. I know some scientists – even some of my mentors – will disagree with the following. But times are changing and I think we need to start making friends with an elephant that has been ignored long enough in psychology’s attempts to gain legitimacy through the objective lens of the medical model. It is time we admit that there are aspects of the healing and recovery process that most evidence-based treatment studies haven’t addressed quantitatively.

We must consider the role of the relationship between patient and therapist as a potential variable that influences outcomes in therapy. When Ms. Brown discusses relationships in her article she put parentheses around “therapeutic alliance” as if putting on gloves to throw out the trash. She continues on with this disregard for the relationship in therapy by including the following quote from a psychology professor: “No one believes it’s a good idea to have a bad relationship with your client… The argument is really more, ‘Is a good relationship all we need to help a patient?’ ”

Pump. The. Brakes.

It’s a straw man argument to say that anyone thinks a good relationship is all that is needed. No therapist is sitting around drinking bubble tea and making friendship bracelets with their clients, fun as that might be. But a good therapeutic relationship is a necessary ingredient and, I believe sometimes, a key ingredient that helps some people get better. And the cold hard truth is that variability exists in therapists’ abilities to create a “therapeutic alliance.” I can’t help but wonder if there may be a correlation between those therapists who assume a good relationship with a client just magically happens, and those who may benefit from some training in relational skills.

Here is what I have learned from the science of attachment and relationships. We are born wired to connect with others. Just as babies are born with the beginning of eyesight and then need environmental input for vision to be fully functional, babies are born with the capacity to connect but need environmental input for relational skills to mature. This drive to connect can be seen within hours of birth when infants are already capable of imitating adults tongue movements. Within hours of birth! Think about the software that must already exist in an infant’s head for this to be true - it’s mind boggling! We are the only species that has the ability to imitate others, and as this system slowly learns and grows in complexity we grow our capacities to empathize and connect with others.

People coming in for therapy have often experienced childhood disruptions in the development of this capacity to be in relationships with others. Sometimes after having been hurt we develop strategies to keep us from this natural impulse to connect with others. These strategies often become intricately linked with the psychopathology that brings someone to therapy. A skilled therapist can help create a safe place to rebuild one’s capacity to be in connection with others. I would never use the relationship as the primary focus of therapy, but it’s often a vital adjunct to evidence-based treatments. I am not talking about an art. I am talking about applying the science of relationships we have now, in even if it hasn’t yet made it through a randomized, controlled clinical trial that yielded an evidence-based treatment option.

I wonder if we’ll ever be able to measure how the therapeutic relationship contributes to the healing process. I think we will, but what if we never can? Does that mean it’s therefore not part of what makes therapy useful? Where is it written that we will be able to isolate and measure every nuanced variable at play in complex ways during the therapeutic experience? We are limiting our capacity to think clearly when we unquestioningly submit to science, confusing it with truth, without critically thinking about the strengths and limits that are present in this powerful way of knowing. (Oh jeez, grad school. Pretty please do not take back my degree for this confession.)

Present company included

The points I’ve made here are just a handful from my bucket of responses. If I had more time to write and you had more time to read I would discuss how an overly narrow definition of recovery inflates the interpretation of outcome data in randomized-controlled trials, the remarkable subjectivity that actually exists in cognitive behavioral therapy, the limited applicability of treatments that last 20 weeks to patients that are in therapy for much longer, and the concerns I have that the author does not put enough trust in the clients to know when therapy is benefiting them. But I’ve babbled enough, and I’m sure you have emails piling up.

I understand the author’s frustration with the problems in disseminating research from academia to clinical practice. Like Ms. Brown, I also worry about the quality of care provided by some therapists. Evidence-based treatments are absolutely the backbone of what we in the community should be using to help our clients. But Ms. Brown’s critique of therapists is ultimately a very odoriferous red herring, pulling our noses away from the real reasons why mental-health care has a long way to go before it can reliably help everyone. To act is if the problems with mental-health care come down to a bunch of undisciplined therapists who don’t know the difference between art and science is a significant failure to understand what is really going on.

I am heartened to know that my colleagues in academia continue to work tirelessly to find evidence-based treatments for those seeking help with a psychological disorder. We need those scientists working for us all, and science will certainly contribute to our deepening understanding of psychological function and dysfunction. But right now, scary as it is to admit, we just don’t have all the answers.

And so for now I will continue to work tirelessly in the community to help the people who are before me now, using everything I have in my arsenal now. That includes evidence-based practices, empathy, and staying with someone when we are both scared and don’t have all the answers but continue to have faith that maybe between the two of us we can slowly move towards healing and growth.

    Frances Ulman, PhD

    Written by

    A clinical psychologist with a healthy skepticism of the norm. www.francesulmanphd.com