Blog

Barkham, M. & Lambert, M.J. (2021). The efficacy and effectiveness of psychological therapies. In Barkham, W. Lutz, and L.G. Castonguay (Eds.) Bergin and Garfield’s Handbook of Psychotherapy and Behavior Change (7^th ed.). Wiley. Chapter 5.

Meta-analyses are not perfect, but they are the best way to summarize findings from a research field. In psychotherapy research, for example, meta-analyses aggregate effect sizes across a number of studies by producing a mean effect compared to a control condition. In this part of the chapter, Barkham and Lambert review the research on the efficacy of psychotherapies. The very first meta-analysis of psychotherapies was conducted by Smith and Glass in 1977 in which they aggregated effects of 475 studies that compared treated versus untreated groups. They reported a standard mean effect size of 0.85 in favor of psychotherapy, which is equivalent to a treated person being better off than 80% of untreated people. That sounds impressive, but it turned out to be an over-estimate of the effects of psychotherapy. Later studies took a more conservative approach by removing lower quality studies (small samples, poorly designed), which resulted in an average effect size of 0.67 when psychotherapy was compared to control conditions. Some meta-analyses went even further by excluding studies with wait-list control groups. It turns out that using a wait-list control (people who get no treatment) may inflate the effects of therapy because people waiting for treatment sometimes get worse, which by comparison makes the patients who get therapy look even better. These meta-analyses also controlled for publication bias (the likelihood that some negative or unflattering studies were never published). By taking all these possible sources of bias into account, the overall effect size of psychotherapy drops to about 0.31 (95%CI [0.24, 0.38]). Nevertheless, even when taking such a conservative approach to the research findings, one can conclude with some certainty that psychological therapies are effective to improve mental health conditions.

Practice Implications

The research over the years has demonstrated two important things about psychotherapy. First, psychotherapy is effective for many mental health problems, even if the effects are smaller than we once thought. The success rate for psychotherapy in treated persons is about 60% compared to 40% for untreated people (or to put it another way, therapists must treat 3 to 5 patients for one to recover). And these effects of psychotherapy are as large as one gets from many common medical interventions. Second, the effects of psychotherapy have not changed in the past 50 years. That is, new developments in psychological therapies and technologies have not moved the needle on patient outcomes since the 1970s.

Courtois, C. A. & Brown, L. S. (2019). Guideline orthodoxy and resulting limitations of the American Psychological Association’s Clinical Practice Guideline for the Treatment of PTSD in Adults. Psychotherapy, 56(3), 329-339.

Recently the American Psychological Association (APA) published clinical practice guidelines for the treatment of post-traumatic stress disorder (PTSD). The reaction from the clinical community that treats those with PTSD, client groups, and from many academic and research quarters was swift and negative. APA received almost 900 comments in their public consultations from many who felt the document was overly prescriptive, overly symptom-focused, and narrow in its recommendations. In this interesting inside look at the process, the Chair of the PTSD Practice Guidelines Committee (Christine Courtois) and a senior member of the Committee (Laura Brown) wrote a scathing commentary of the process imposed on them by APA that constrained the Committee’s access to information which affected their decisions. The Committee was bound by APA’s use of the Institute of Medicine (IOM) rules for developing practice guidelines. In other words, a psychological organization (APA) used a biomedical model to define what is relevant research, how to define treatment, what is an appropriate outcome, and how to decide on recommendations. As a result, the APA Committee reached several conclusions/decisions that were biased or premature. First, they defined PTSD only by its symptom presentation and not for the complex disorder that it is. In other words, PTSD was viewed almost exclusively from within a framework that defined it as only a fear-based response to a stressor. Such an approach downplays any developmental or attachment-related factors in the genesis or maintenance of PTSD. Second, the Committee was instructed to ignore a vast array of research on therapist factors, relationship factors, and client factors in psychotherapy. This runs counter to many clinicians’ views that one cannot engage in technical interventions related to PTSD symptoms without the patient experiencing a heightened sense of security in their relationship with the therapist. This also meant that the Committee largely ignored cultural and diversity factors. Third, the treatment recommendations focused on time-limited exposure-based interventions – which is a natural outcome of the first two decisions (i.e., seeing PTSD as only fear-based, ignoring issues of development, and ignoring relational factors in the treatment context). The authors were also disappointed that the APA ignored its own policy on evidence-based practice that puts equal weight on research, clinician expertise, and client factors when making clinical decisions. In the end the authors clearly were not confident in the narrow focus of the Clinical Practice Guideline, and they were concerned that clinicians, researchers, policy makers, and third party funders could misuse the Guideline to limit research, theory, and funding.

Practice Implications

In this extraordinary piece, the Chair and a senior committee member of the PTSD Practice Guideline Committee were highly critical of the process and outcome of APA’s effort to develop clinical practice guidelines for PTSD. The authors did not diminish the importance of exposure-based interventions for PTSD, however they did argue that these interventions must be offered only after clinicians take a sufficient amount of time to create a clinical context characterized by clients experiencing heightened safety in the therapeutic relationship, and to into account client preferences and culture. Further, clinicians should be highly sensitive to attachment-related insecurities and developmental traumas that may lengthen the treatment and that may have a complicating impact on the therapeutic relationship.

Norcross, J. C., & Wampold, B. E. (2019). Relationships and responsiveness in the psychological treatment of trauma: The tragedy of the APA Clinical Practice Guideline. Psychotherapy, 56(3), 391-399.

The American Psychological Association’s (APA) Clinical Practice Guideline for Posttraumatic Stress Disorder in Adults published in 2017 was met with a great deal of concern and criticism by the community of scholars and practitioners working with patients with PTSD. A key concern was that the APA used a biomedical model and not a psychological or contextual model in guiding their understanding of PTSD, their approach to what constitutes evidence, and to decisions about recommended treatments. In particular, the biomedical approach focuses almost exclusively on treatment methods, and down-plays the context of treatment (i.e., the relationship, patient factors, and therapist responsiveness). In this critique, Norcross and Wampold highlight the flaws in the APA Clinical Practice Guideline for PTSD, and the authors focus specifically on those variables that are known to predict patient outcomes but that were ignored by the Guideline. Norcross and Wampold highlighted that there exists numerous meta analyses that demonstrate that all bona fide psychotherapies work about equally well for trauma, and that the particular treatment method has little impact on PTSD outcomes. Yet, the restrictive review process undertaken by APA all but ignored this well-established finding. Also ignored was the research on the importance of the therapeutic relationship in the treatment of trauma. One review outlined nineteen studies that found that the therapeutic alliance was associated with or predicted reduction in PTSD symptoms. This is consistent with the general psychotherapy research literature, in which the alliance is the most researched and most reliable factor related to patient outcomes. Also missing from the PTSD Guideline was reference to a large body of research on therapist responsiveness to patient characteristics. Patients are more likely to improve if their therapists can adapt to the patient’s coping style, culture, preferences, level of resistance, and stage of change. In one study of cognitive-processing therapy (CPT; a treatment recommended by the APA Guideline), there were substantial differences between therapists in their patient’s PTSD symptom outcomes. That is, some therapists reliably were more effective than others, even though all therapists were trained in and supervised in providing the same manualized evidence-based treatment. Among the identified skills of the most effective CPT therapists were: a flexible interpersonal style, and an ability to develop and maintain a good therapeutic alliance across patients.

Practice Implications

There is growing consensus that the APA Clinical Practice Guideline for PTSD are based on dubious methodology and are of limited use to therapists and their patients with PTSD. Psychotherapists should practice a bona-fide therapy for PTSD, but should do so by taking into account the treatment context. In other words, more effective therapists are good at developing, maintaining, and repairing the therapeutic alliance across a range of patients. Effective therapists can also respond and adapt to patient characteristics such as level of resistance, coping style, culture, and stage of change. And so, even when providing a treatment based on the APA Guideline, therapists should nurture trust in the therapeutic relationship and be adaptive to their patients’ characteristics.

Sakaluk, J. K., Williams, A. J., Kilshaw, R. E., & Rhyner, K. T. (2019). Evaluating the evidential value of empirically supported psychological treatments (ESTs): A meta-scientific review. Journal of Abnormal Psychology, 128(6), 500-509.

In the 1990s the Clinical Division of the American Psychological Association commissioned a Task Force to identify “Empirically Supported Treatments” (EST). The Task Force decided that psychotherapies that repeatedly showed statistically significant improvements over no treatment, placebos, or another treatment would be designated as “Strongly” supported. They also designated some treatments as “Modestly” supported or with “Controversial” support. The EST movement continues to have a great impact on the practice, research, and funding of psychotherapy. Time-limited, diagnosis-focused therapies, tested in randomized controlled trials became the “gold standard”. Clinicians are expected to practice these ESTs, research agencies focus funding on these models, and some governments and insurance companies provide reimbursements only for these types of therapy. The Empirically Supported Treatments (EST) movement redefined the practice of psychotherapy as short-term, symptom-focused, technically-oriented, and mostly cognitive-behavioral. In this meta-scientific review Sakaluk and colleagues asked: how good is the evidence for the ESTs? The authors were particularly concerned with the quality of the studies from a methodological and statistical point of view: how likely was it that these findings could be replicated, or how reliable were the findings? The good news is that there were few instances (about 10%) of research supporting ESTs in which researchers mis-reported the statistics (i.e., error in the reporting of statistical findings). This is quite a bit lower than previously identified mis-reporting rates (about 50%) in psychological research in general. However, only about 19% of ESTs were supported consistently by high quality studies. Over half of ESTs were supported consistently by poor quality studies. Most of the studies supporting ESTs were not sufficiently powered to detect differences between treatments or conditions. That is, often the sample sizes of patients in the studies were too small, and so the significant results were not likely reliable or perhaps not plausible. Also, those therapies that the EST list defined as having “Strong” support were not backed by more higher quality research compared to therapies considered to have “Moderate” support. In other words, the decision to designate treatments as “Strongly” or “Moderately” supported appears to have almost no relationship with the quality of the research.

Practice Implications

Embedded in this dense methodological paper are some troubling findings and important practice implications. The authors suggested that there are a number treatments on the EST list that have dubious research support because the studies of those treatments may not stand up to replication (a critical test in scientific research). It is not clear that ESTs are any more effective than other bona-fide psychotherapies that are not on the list. (Bona-fide psychotherapies are those that are based on a psychological theory, delivered by trained therapists, and in which the patient and therapist develop a relationship). The findings question whether dissemination of and training in ESTs to the exclusion of other psychotherapies can be justified given the quality of the evidence. In other words, it is possible that other bona-fide psychotherapies that are not on the EST list may be just as effective. This does not imply that psychotherapy is not effective or that anything goes when it comes to the practice of psychotherapy. Evidence-based practice in psychotherapy should guide psychotherapists’ clinical choices. However, the EST list is not the final word on what constitutes “evidence-based” practice in psychotherapy, or on what treatments should be researched and funded.

Wampold, B.E., Flückiger, C., Del Re, A.C., Yulish, N.E., Frost, N.D., …Hilsenroth, M. (2017) In pursuit of truth: A critical examination of meta-analyses of cognitive behavior therapy, Psychotherapy Research, 27, 14-32.

The vast majority of meta-analyses of studies that compare different brands of psychotherapy for any particular disorder indicate that differences between treatments are quite small and clinically trivial. Meta-analyses are an important way of aggregating effect sizes across studies and of providing reliable estimates of the state of a research field. But meta-analyses are not perfect - they rely on judgements made by the researchers that may bias findings. Despite a large body of evidence to the contrary, three meta-analyses in particular have purported to demonstrate that cognitive-behavioral therapy (CBT) is superior to other therapies for some specific disorders. In this paper, Wampold and colleagues critically review these three meta analyses to see if in fact CBT is superior to other psychotherapies. A meta-analysis by Tolin that reported that CBT was more efficacious than other therapies for anxiety and depression was surprising given that it contradicted 5 previous meta-analyses. It turns out that Tolin misclassified some treatments as CBT (including eye movement desensitization and reprocessing [EMDR] and present-centred therapy [PCT]). Further, Tolin made a critical computational error with one of the studies that when corrected wiped out any superiority for CBT. A second meta-analysis by Marcus and colleagues reported small differences in favor of CBT for primary (i.e., target symptoms) outcomes at post-treatment but not at follow up. Wampold and colleagues reported that the small difference at post-treatment was unduly affected by one study in the meta-analysis that showed unusually large effect in favor of CBT (i.e., the study was likely unreliable because its results were so much out of line with all other studies). Further, the purported superiority of CBT disappeared in the longer term. Finally, a meta-analysis by Mayo-Wilson and colleagues published in the prestigious journal Lancet Psychiatry used a network meta-analysis to compare treatments, and reported that CBT was more effective than other psychotherapies. Network meta-analysis relies heavily on indirect comparisons rather than including only studies that directly compared two therapy modalities. For example, if there are only a few studies that compare treatment A to treatment B (AB), one could look at studies of treatment A versus treatment C (AC), and studies of treatment B versus treatment C (BC), and then use the transitive property (remember high school math?) to estimate the effect of AB indirectly from the studies of AC and BC. It turns out that this practice in the context of meta-analysis is unreliable and can grossly over-estimate differences between treatments.

Practice Implication

The vast majority of meta-analyses show that bona-fide psychotherapies are effective, and one therapeutic orientation does not seem to be superior to another. The three meta-analyses that run counter to this conclusion are deeply flawed. To claim that one treatment is more effective than another will limit patients’ access to other treatments. This is concerning, since most time-limited treatments result in about half of patients recovering from their mental health problems. And so many patients and their therapists need more therapeutic options to draw upon. Falsely claiming that one treatment is more effective than others may lead insurance companies and government policy makers to make erroneous decisions to fund only one type of therapy.

Huey, S. J. & Tilley, J. L. (2018). Effects of mental health interventions with Asian Americans: A review and meta-analysis. Journal of Consulting and Clinical Psychology, 86, 915-930.

Do existing mental health interventions work well for patients of Asian descent? Interventions delivered in the typical way in which they were devised may not be as effective as intended when it comes to culturally diverse groups like Asian Americans. The clinical trials in which the treatments were developed typically are almost exclusively made up of White participants, and most evidence-based treatments do not consider cultural considerations. Culturally responsive psychotherapies that are consistent with the cultural norms, values, and expectations of patients may be more effective. That is, if an evidence-based treatment is not culture specific, it may not be as effective as intended. Even when culture is taken into account in evidence-based treatments, the accommodation tends to be for African American or Hispanic/Latino patients, and not for Asian American patients. Asian American and East Asian heritage is often influenced by Confucian values that emphasize interpersonal harmony, mutual obligations, and respect for hierarchy in relationships. This may mean that patients of Asian descent may be less committed to personal choice, more attuned to others, and more socially conforming. This may lead to cultural differences in cognitive processing and emotional reactions to interpersonal contexts. In this meta-analysis, Huey and colleagues assessed if the effects of evidence-based treatments will be bigger if the treatments were specifically tailored for Asian Americans. Their review included 18 studies with 6,377 participants. Samples included Chinese Americans, Cambodian Americans, Korean Americans, Vietnamese Americans, and other Asian groups. Problems treated included depression, PTSD, smoking, and other concerns. About half of the studies were of CBT, and most (91%) were culturally tailored in some way either for an Asian subgroup or tailored for minorities in general. The mean effect size for evidence-based treatments versus control groups was d = .75, SE = .14, p < .001, indicating a moderate to large effect. Treatments tailored specifically for Asian subgroups (e.g., Chinese Americans) showed the largest effects (d = 1.10), whereas treatment with no cultural tailoring or non-Asian tailoring showed the smallest effects (d = .25).

Practice Implications

Existing psychological treatments are efficacious for Asian Americans, with moderate effects. However, treatments specifically adapted for Asian American subgroups showed the largest effects, indicating that specific cultural adaptations could substantially improve the effectiveness of psychotherapy. Asian Americans face challenges in terms of using and engaging in treatments. Developing culturally specific interventions to improve acceptability of treatment may be one way to make the most therapeutic impact on one of the largest growing racial groups in North America.

Author email: hueyjr@usc.edu