Blog

Sakaluk, J. K., Williams, A. J., Kilshaw, R. E., & Rhyner, K. T. (2019). Evaluating the evidential value of empirically supported psychological treatments (ESTs): A meta-scientific review. Journal of Abnormal Psychology, 128(6), 500-509.

In the 1990s the Clinical Division of the American Psychological Association commissioned a Task Force to identify “Empirically Supported Treatments” (EST). The Task Force decided that psychotherapies that repeatedly showed statistically significant improvements over no treatment, placebos, or another treatment would be designated as “Strongly” supported. They also designated some treatments as “Modestly” supported or with “Controversial” support. The EST movement continues to have a great impact on the practice, research, and funding of psychotherapy. Time-limited, diagnosis-focused therapies, tested in randomized controlled trials became the “gold standard”. Clinicians are expected to practice these ESTs, research agencies focus funding on these models, and some governments and insurance companies provide reimbursements only for these types of therapy. The Empirically Supported Treatments (EST) movement redefined the practice of psychotherapy as short-term, symptom-focused, technically-oriented, and mostly cognitive-behavioral. In this meta-scientific review Sakaluk and colleagues asked: how good is the evidence for the ESTs? The authors were particularly concerned with the quality of the studies from a methodological and statistical point of view: how likely was it that these findings could be replicated, or how reliable were the findings? The good news is that there were few instances (about 10%) of research supporting ESTs in which researchers mis-reported the statistics (i.e., error in the reporting of statistical findings). This is quite a bit lower than previously identified mis-reporting rates (about 50%) in psychological research in general. However, only about 19% of ESTs were supported consistently by high quality studies. Over half of ESTs were supported consistently by poor quality studies. Most of the studies supporting ESTs were not sufficiently powered to detect differences between treatments or conditions. That is, often the sample sizes of patients in the studies were too small, and so the significant results were not likely reliable or perhaps not plausible. Also, those therapies that the EST list defined as having “Strong” support were not backed by more higher quality research compared to therapies considered to have “Moderate” support. In other words, the decision to designate treatments as “Strongly” or “Moderately” supported appears to have almost no relationship with the quality of the research.

Practice Implications

Embedded in this dense methodological paper are some troubling findings and important practice implications. The authors suggested that there are a number treatments on the EST list that have dubious research support because the studies of those treatments may not stand up to replication (a critical test in scientific research). It is not clear that ESTs are any more effective than other bona-fide psychotherapies that are not on the list. (Bona-fide psychotherapies are those that are based on a psychological theory, delivered by trained therapists, and in which the patient and therapist develop a relationship). The findings question whether dissemination of and training in ESTs to the exclusion of other psychotherapies can be justified given the quality of the evidence. In other words, it is possible that other bona-fide psychotherapies that are not on the EST list may be just as effective. This does not imply that psychotherapy is not effective or that anything goes when it comes to the practice of psychotherapy. Evidence-based practice in psychotherapy should guide psychotherapists’ clinical choices. However, the EST list is not the final word on what constitutes “evidence-based” practice in psychotherapy, or on what treatments should be researched and funded.

Wampold, B.E., Flückiger, C., Del Re, A.C., Yulish, N.E., Frost, N.D., …Hilsenroth, M. (2017) In pursuit of truth: A critical examination of meta-analyses of cognitive behavior therapy, Psychotherapy Research, 27, 14-32.

The vast majority of meta-analyses of studies that compare different brands of psychotherapy for any particular disorder indicate that differences between treatments are quite small and clinically trivial. Meta-analyses are an important way of aggregating effect sizes across studies and of providing reliable estimates of the state of a research field. But meta-analyses are not perfect - they rely on judgements made by the researchers that may bias findings. Despite a large body of evidence to the contrary, three meta-analyses in particular have purported to demonstrate that cognitive-behavioral therapy (CBT) is superior to other therapies for some specific disorders. In this paper, Wampold and colleagues critically review these three meta analyses to see if in fact CBT is superior to other psychotherapies. A meta-analysis by Tolin that reported that CBT was more efficacious than other therapies for anxiety and depression was surprising given that it contradicted 5 previous meta-analyses. It turns out that Tolin misclassified some treatments as CBT (including eye movement desensitization and reprocessing [EMDR] and present-centred therapy [PCT]). Further, Tolin made a critical computational error with one of the studies that when corrected wiped out any superiority for CBT. A second meta-analysis by Marcus and colleagues reported small differences in favor of CBT for primary (i.e., target symptoms) outcomes at post-treatment but not at follow up. Wampold and colleagues reported that the small difference at post-treatment was unduly affected by one study in the meta-analysis that showed unusually large effect in favor of CBT (i.e., the study was likely unreliable because its results were so much out of line with all other studies). Further, the purported superiority of CBT disappeared in the longer term. Finally, a meta-analysis by Mayo-Wilson and colleagues published in the prestigious journal Lancet Psychiatry used a network meta-analysis to compare treatments, and reported that CBT was more effective than other psychotherapies. Network meta-analysis relies heavily on indirect comparisons rather than including only studies that directly compared two therapy modalities. For example, if there are only a few studies that compare treatment A to treatment B (AB), one could look at studies of treatment A versus treatment C (AC), and studies of treatment B versus treatment C (BC), and then use the transitive property (remember high school math?) to estimate the effect of AB indirectly from the studies of AC and BC. It turns out that this practice in the context of meta-analysis is unreliable and can grossly over-estimate differences between treatments.

Practice Implication

The vast majority of meta-analyses show that bona-fide psychotherapies are effective, and one therapeutic orientation does not seem to be superior to another. The three meta-analyses that run counter to this conclusion are deeply flawed. To claim that one treatment is more effective than another will limit patients’ access to other treatments. This is concerning, since most time-limited treatments result in about half of patients recovering from their mental health problems. And so many patients and their therapists need more therapeutic options to draw upon. Falsely claiming that one treatment is more effective than others may lead insurance companies and government policy makers to make erroneous decisions to fund only one type of therapy.

Nehrig, N., Prout, T.A., & Aafjes-van Doorn, K. (2019). Whose anxiety are we treating, anyway? Journal of Clinical Psychology. Online first publication.

Evidence-based practice (EBP) in psychotherapy is defined by the American Psychological Association as the deliberate integration of: (1) the research evidence, (2) clinician expertise in making treatment decisions, and (3) client characteristics, preferences, and culture. The EBP statement was meant to supplant an older model of prescriptive psychotherapy practice that resulted in the creation of lists of empirically-supported treatments (EST). The ESTs were defined as: (1) manualized therapies, (2) shown to be efficacious in randomized controlled trials, (3) for patients with a specific diagnosed mental disorder. However, manualized therapies are not necessarily more effective than non-manualized treatments, and patients in randomized controlled trials may not represent those typically seen by therapists in everyday practice. Although EBPs are the current standard by which psychotherapists should practice, many therapists and organizations focus almost exclusively on the first of the EBP criteria (the research evidence of ESTs) to the exclusion of the second and third criteria (clinician expertise, and patient characteristics, preferences, and culture). In this review article, Nehrig and colleagues speculated about why this is the case by asking: “whose anxiety are we treating?” They argued that manualized therapies identified as ESTs reduce therapists’ anxiety caused by: uncertainty about treatment outcomes, the emotional toll of providing psychotherapy to people who are suffering, and the negative emotions (anxiety, despair, cynicism) that sometimes arises in therapists from the work. Nehrig and colleagues argued that ESTs provide therapists with a sense of control and certainty, while limiting therapists’ attention on relational challenges in the work of therapy. However, this emphasis on ESTs comes at a cost for therapists and patients. Therapists may not focus on developing skills to manage the relational challenges inherent in providing psychotherapy, greater certainty may reduce therapists’ engagement in sufficient self-reflection, and therapists may attend only to patients’ symptoms and not to the patient as a whole person. Nehrig and colleagues also discuss the preference for ESTs among institutions, insurance companies, and government funders of psychotherapy. ESTs reduce anxiety in these contexts because ESTs are seen by managers as methods to enhance accountability and standardization of treatment, to uphold standards of care, and to reduce potential liability. The short-term nature of most ESTs also assuages economic concerns for institutions and funders who wish to manage costs. However, this emphasis on short term manualized treatment also reduces psychotherapy from a complex interpersonal process with inherent uncertainty to one that resembles a clear-cut medical procedure that encourages top-down decision-making about clinical practice.

Practice Implications

Anxiety about the complexity of psychotherapy can cause therapists, institutional managers, and government funders to place greater value on ESTs rather than on clinical expertise of the therapist and patient characteristics. Patient characteristics, preferences, and culture are related to developing the therapeutic alliance and to patient outcomes. Astute therapists can learn to adjust their interventions to these patient characteristics, which may mean using clinical judgement to alter or deviate from a prescriptive manual. An EBP approach that integrates research, clinical expertise, and patient characteristics allows therapists to take into account transtheoretical factors known to affect outcomes like the therapeutic alliance, repairing alliance ruptures, empathy, and to use their clinical expertise to adjust their interpersonal stances to relevant patient characteristics, preferences, and culture.

Ravitz, P., Watson, P., Lawson, A., Constantino, M.J., Bernecker, S., Park, J., & Swartz, H.A. (2019). Interpersonal Psychotherapy: A scoping review and historical perspective (1974-2017). Harvard Review of Psychiatry, 27, 165-179.

Interpersonal Psychotherapy (IPT) focuses on relationships and emotions, and emphasizes stressful interpersonal loss, life changes, relationship disputes, and social isolation as the causes and maintenance factors related to depression and other disorders. The IPT is predicated on the importance of relationships for survival and the bidirectional links between depression and problems with relationships and social support. IPT was first manualized in 1974 and used as a comparison condition in studies of pharmacotherapy for depression. Contrary to expectations, IPT did just as well as antidepressant medication in early treatment trials, thus giving rise to an important new psychological treatment. This scoping review by Ravitz and colleagues summarizes the development of IPT over the past 40 years. The review identified over 1000 articles of IPT, 133 of which were randomized controlled trials. Following the initial trials in the 1970s, IPT was included in the Treatment of Depression Collaborative Research Program in the 1980s, funded by the National Institute of Mental Health in the United States. This was the largest trial of its kind whose results indicated that patients with high baseline depression did best with medications followed by IPT, whereas CBT did not show significant advantage over the placebo condition. The 1990s to the mid-2000s saw a precipitous increase in randomized controlled trials of IPT in which IPT was: compared to other therapies, compared to medications, and/or provided in combination with medications. In addition, treatment trials of group IPT were conducted in low- and middle-income countries. These studies led the World Health Organization to publish and disseminate a group IPT manual. More recent research in the past decade has seen IPT offered in different formats (individual, telephone, group, internet), for different populations (adolescents, perinatal women, late-life), and in a variety of low- and high-income countries. Currently, there is good research support for the efficacy of IPT for depression, eating disorders, bipolar disorder, PTSD, and anxiety.

Practice Implications

Relationships play an important role in determining health, disease, vulnerability, recovery, and resilience. Because of the universal importance of relationships, IPT is relevant to mental health care across cultures and populations. Therapists should consider the importance of relationship loss and grief, role transitions throughout the lifespan, persistent conflicts in relationships, and social isolation when treating patients with depression and other mental disorders.

Huey, S. J. & Tilley, J. L. (2018). Effects of mental health interventions with Asian Americans: A review and meta-analysis. Journal of Consulting and Clinical Psychology, 86, 915-930.

Do existing mental health interventions work well for patients of Asian descent? Interventions delivered in the typical way in which they were devised may not be as effective as intended when it comes to culturally diverse groups like Asian Americans. The clinical trials in which the treatments were developed typically are almost exclusively made up of White participants, and most evidence-based treatments do not consider cultural considerations. Culturally responsive psychotherapies that are consistent with the cultural norms, values, and expectations of patients may be more effective. That is, if an evidence-based treatment is not culture specific, it may not be as effective as intended. Even when culture is taken into account in evidence-based treatments, the accommodation tends to be for African American or Hispanic/Latino patients, and not for Asian American patients. Asian American and East Asian heritage is often influenced by Confucian values that emphasize interpersonal harmony, mutual obligations, and respect for hierarchy in relationships. This may mean that patients of Asian descent may be less committed to personal choice, more attuned to others, and more socially conforming. This may lead to cultural differences in cognitive processing and emotional reactions to interpersonal contexts. In this meta-analysis, Huey and colleagues assessed if the effects of evidence-based treatments will be bigger if the treatments were specifically tailored for Asian Americans. Their review included 18 studies with 6,377 participants. Samples included Chinese Americans, Cambodian Americans, Korean Americans, Vietnamese Americans, and other Asian groups. Problems treated included depression, PTSD, smoking, and other concerns. About half of the studies were of CBT, and most (91%) were culturally tailored in some way either for an Asian subgroup or tailored for minorities in general. The mean effect size for evidence-based treatments versus control groups was d = .75, SE = .14, p < .001, indicating a moderate to large effect. Treatments tailored specifically for Asian subgroups (e.g., Chinese Americans) showed the largest effects (d = 1.10), whereas treatment with no cultural tailoring or non-Asian tailoring showed the smallest effects (d = .25).

Practice Implications

Existing psychological treatments are efficacious for Asian Americans, with moderate effects. However, treatments specifically adapted for Asian American subgroups showed the largest effects, indicating that specific cultural adaptations could substantially improve the effectiveness of psychotherapy. Asian Americans face challenges in terms of using and engaging in treatments. Developing culturally specific interventions to improve acceptability of treatment may be one way to make the most therapeutic impact on one of the largest growing racial groups in North America.

Author email: hueyjr@usc.edu

Truijens, F., Zühlke‐van Hulzen, L., & Vanheule, S. (2018). To manualize, or not to manualize: Is that still the question? A systematic review of empirical evidence for manual superiority in psychological treatment. Journal of Clinical Psychology. Advance online publication. 

In 2010 Webb and colleagues published a meta-analysis in which they showed that the association between adherence to a psychotherapy manual and treatment outcome was close to zero. The same was true for therapist competence in delivering the manualized psychotherapy – almost no relationship to client outcome. Psychotherapy manuals typically specify the theoretical basis for an intervention, the number and sequencing of treatment sessions, the content and objective of sessions, and the procedures of each session. National institutes in the US and the UK have promoted manuals as a means to define what is evidence-based psychotherapy. By doing so these institutes assume that psychotherapy that is manualized is more effective that non-manualized treatment. However, detractors have argued that: (1) strict adherence to manuals may reduce therapists’ ability to individualize treatment to client needs and characteristics; (2) manuals are often designed for single disorders but clients tend to have many comorbid conditions; and (3) it is impossible for clinicians to gain competence in all different manuals for the various client conditions they may encounter. In this systematic review, Truijens and colleagues ask: does the use of manuals increase therapy effectiveness? To answer this question they conducted three different systematic reviews. First, they reviewed six studies that directly compared manualized versus non-manualized versions of a psychotherapy within the same study. One study showed manuals were superior, three showed no difference, and two studies showed that non-manualized therapies were more effective. Second, they reviewed eight meta-analyses that compared the pre- to post-treatment effect sizes of manualized therapies and of non-manualized therapies versus no-treatment control conditions. Three meta-analyses concluded that manualized therapies were superior, four meta-analyses did not find differences, and one observed non-manualized treatments to be superior. Third, the authors reviewed 15 additional studies to those reviewed by Webb and colleagues in their original meta-analysis. Overall, Truijens found similar results that support the conclusion that the level of adherence to psychotherapy manuals is not substantially related to better treatment outcomes.

Practice Implications

Although treatment manuals may be helpful for training purposes and to ensure validity in psychotherapy research, there is actually little consistent evidence that adhering to a manual results in better client outcomes. Some have argued that rigid adherence to a treatment manual can be harmful to clients. Therapists may need to take a flexible stance when applying research-supported therapeutic principles and interventions. Such a stance adjusts therapy to take into account client characteristics like level of resistance, coping style, attachment style, and others. Truly evidence-informed approaches incorporate what we know about client characteristics, therapeutic relationship factors, and therapist factors to promote positive outcomes in psychotherapy clients.