Blog

Owen, J., Drinane, J. M., Idigo, K. C., & Valentine, J. C. (2015). Psychotherapist effects in meta-analyses: How accurate are treatment effects? Psychotherapy, 52(3), 321-328.

One of the ongoing debates in the psychotherapy research literature has to do with the relative efficacy of psychotherapies. Is psychotherapy brand A (CBT, for example) more effective than psychotherapy brand B (psychodynamic therapy, for example)? The most common way to test this question is with randomized controlled trials (RCTs), in which clients are randomly assigned to treatment condition (brand A or B). This study design controls for systematic bias in the results that may be caused by differences between clients. But what about therapists? We know for example that therapist effects (i.e., differences between therapists) account for approximately 5% to 10% of client outcomes. Therapist effects are often larger than the effect of the empirically supported treatment that is being offered. Yet it is almost unheard of for therapists to be randomized to treatments, so therapist effects are not controlled in most psychotherapy trials. As a result the effects of the differences between therapists get statistically rolled into the treatment effects. As Owen and colleagues point out, the impact of not controlling for therapist effects is that some differences between treatments in an RCT will appear statistically significant when in fact they are not. One can control for the effect of therapist differences, thus providing a more accurate estimate of treatment effects, but this is rarely done in published RCTs. So, when these RCTs are summarized in a meta analysis, the meta analysis results are also affected by ignoring therapist effects. In their study, Owen colleagues did something very clever. They took data from 17 recent meta analyses of RCTs that found differences between two interventions. These included meta analyses of studies comparing: CBT vs alternative treatments, bona fide treatments vs non-bona fide treatments, culturally adapted treatments vs those that were not adapted, etc. There are many other meta analyses that show no differences between treatments, but the authors wanted to focus specifically on the 17 that did show differences. Owen and colleagues statistically estimated what would happen to the original study findings of significant differences between treatments if therapist effects on patient outcomes were controlled. They controlled for three different sizes of therapist effects that accounted for: 5% (small), 10% (medium), or 20% (large) of patient outcomes. Even small therapist effects (5%) reduced the number of significant differences between treatments from 100% to 80%. When psychotherapist effects were estimated to be medium (10% - which is the best estimate based on research), the number of significant differences between treatments dropped to 65%. For large therapist effects (20%), the number of significant treatment differences was only 35%.

Practice Implications

I have argued previously that the psychotherapist matters. Placing more time and effort in developing good reflective practice based on quality information and developing therapist skills like empathy, progress monitoring, and identifying and repairing alliance ruptures will result in better patient outcomes. As Owen and colleagues note, when reading an RCT that claims to find significant differences between psychotherapies, ask yourself if they took into account the effects of differences between therapists.

The Great Psychotherapy Debate: Since in April, 2015 I review parts of The Great Psychotherapy Debate (Wampold & Imel, 2015) in the PPRNet Blog. This is the second edition of a landmark, and sometimes controversial, book that surveys the evidence for what makes psychotherapy work. You can view parts of the book in Google Books.

Wampold, B.E. & Imel, Z.E. (2015). The great psychotherapy debate: The evidence for what makes psychotherapy work (2^nd edition). New York: Routledge.

Some therapists achieve better patient outcomes than others. This seems obvious on the surface and yet few people talk about it, and the research literature seems to downplay or ignore this fact. To illustrate the differing outcomes achieved by therapists, I reviewed a unique study in the September 2013 PPRNet Blog. In that study, 10 cases were randomly selected from 700 therapists (N = 7000 patients), and therapist outcomes were assessed by averaging their patient outcomes. Depending on the presenting problem, as many as 67% of therapists were reliably effective, but as many as 16% were reliably harmful. Clearly therapists differ. Yet psychotherapy research typically treats therapists as if they are uniformly effective. In their chapter, Wampold and Imel review some of the research that estimates the therapist’s effect on outcomes. In other words, what is the impact of the particular therapist on the patient outcomes? Even in studies in which: (a) therapists are selected as experts to provide a specific type of therapy (i.e., CBT, psychodynamic, interpersonal, etc.), (b) therapists are highly trained to be adherent to a manual with repeated supervision, and (c) patients are randomly assigned to treatments, there remains a significant amount of variability in therapist outcomes. Indeed in many studies the therapist effect is as large or larger than the effect of the intervention that is being delivered. In other words, which therapist a patient gets in a treatment study matters just as much or more than what type of therapy they receive. This is also true in medication trials. Better psychiatrists (i.e., those with overall better patient outcomes) who gave a placebo had better patient outcomes than poorer psychiatrists who gave the active medication. A recent large meta analysis found that about 5% of patient outcomes in controlled psychotherapy trials was attributable to the therapist, and the effect is as high as 7% in naturalistic settings. For treatment of PTSD, therapist effects are as high as 12%. On the surface these look like small effects, but in reality they can have a big cumulative impact. Therapists with the best and worst outcomes differ dramatically. For example in one large study, the best performing therapists had a patient response rate of 80% compared to the worst performing therapists who had only 20% of their patients improve. Which therapist would you want a loved one to see?

Practice Implications

Wampold and Imel reported that that therapist effects generally exceed the effects of the specific treatment that is being tested or provided. Some therapists consistently achieve better patient outcomes than others. What are the characteristics and actions of effective therapists? Factors like therapist allegiance to the therapy, empathy, and the ability to form and maintain an alliance with clients appear to differentiate therapists who consistently have good patient outcomes versus those whose patients tend to have poor outcomes.

The Great Psychotherapy Debate: Since in April, 2015 I review parts of The Great Psychotherapy Debate (Wampold & Imel, 2015) in the PPRNet Blog. This is the second edition of a landmark, and sometimes controversial, book that surveys the evidence for what makes psychotherapy work. You can view parts of the book in Google Books.

Wampold, B.E. & Imel, Z.E. (2015). The great psychotherapy debate: The evidence for what makes psychotherapy work (2^nd edition). New York: Routledge.

Why We Should Care About Allegiance Effects in Psychotherapy Research

Allegiance in psychotherapy refers to the degree to which a researcher or therapist believes that the therapy they are studying or delivering is effective. Clients have an expectation that therapists have an explanation for their disorder and that the therapy used to address that explanation will lead to improvements. On their part, psychotherapists choose a therapeutic approach that is consistent with their understanding of psychological distress. Wampold and Imel argue that therapist allegiance is a common factor across therapies that contributes to good patient outcomes. Although allegiance is an important therapeutic factor, it complicates the conduct of psychotherapy research. In a trial comparing two treatments, for example, researchers and therapists tend to be affiliated with one of the treatments and so they believe in the effectiveness of their treatment. They often do not feel the same way about the comparison treatment, or they may desire that their preferred treatment be more effective than the comparison. In medication trials, this allegiance effect can be controlled by a double blind placebo controlled design in which both therapist and patient are not aware of who is receiving which active medication, or who is receiving a placebo. It is impossible to blind therapists in psychotherapy trials – therapists have to know what treatment they are providing. When doing a meta analytic review of psychotherapy trials, it is possible and relatively easy to identify the allegiance of the researchers in a particular study by looking at their past publications, and by reading what they say about the therapies they are comparing. Often, the developer of a treatment manual is a co-author of the trial. Wampold and Imel review several meta analyses that assess the allegiance effects. In three meta analyses from published in 1980, 1999, and 2013 the correlation between ratings of researcher allegiance and effects of psychotherapy on patient outcomes ranged from moderate to large (r = .26 to r = .85). One interesting meta analysis illustrates the magnitude of this effect. The reviewers looked at 69 studies on self statement monitoring (SSM), a type of cognitive therapy developed by Meichenbaum. The average effect of SSM compared to controls in all studies was d = .53 to d = .74, which is moderate. However, effect sizes found in the studies co-authored by Meichenbaum were nearly twice as large, d = d = 1.23. Being a co-investigator in a study of a therapy that one develops, apparently doubles the effect of the treatment on patient outcomes.

Practice Implications

Therapist allegiance to a treatment is important to the effectiveness of the treatment in that therapist allegiance increases the therapist’s confidence in the treatment’s effectiveness and increases a patient’s expectation of getting better. However, when interpreting psychotherapy trials, especially those that pit one type of therapy against another, it is important to keep in mind the researchers’ allegiance. It is rare to see trials that compare two interventions in which the research team is made of up proponents of the two interventions. However such trials are important and necessary.

Laska, K. M., Gurman, A. S., & Wampold, B. E. (2014). Expanding the lens of evidence-based practice in psychotherapy: A common factors perspective. Psychotherapy, 51(4), 467-481.

In this wide ranging review of the Common Factors (CF) perspective in psychotherapy, Laska and colleagues tackle the complex issues of defining CF and describing the evidence. The authors argue that CF in psychotherapy are not a vague set of ideas that fit under the label of “non-specific factors” or “relationship factors”. They also state that there is an unnecessary dichotomy between the concepts of empirically supported treatments (EST) and CF. In EST, specific and brief manualized therapies for specific disorders are tested in highly controlled randomized trials. ESTs purport that efficacious psychotherapies contain specific techniques based on an articulated theory of the disorder, and a specific mechanism of change for that disorder (e.g., depression is partly caused by depressogenic beliefs and so CBT for depression specifically targets cognitive distortions). There are published lists of ESTs for many disorders. However, Laska and colleagues argue that there is little evidence of the specificity of these treatments. For example, in dismantling studies an intervention like CBT for depression is compared to a dismantled version that removes an “active ingredient” [e.g., by providing only behavioral activation as an intervention], with little difference in patient outcomes between the full and dismantled versions. Further, for a number of disorders, several therapies based on very different theories of the disorder and of change are equally effective. In contrast to the EST approach, Laska and colleagues describe the CF approach which focuses on factors that are necessary and sufficient for patient change across psychotherapies, such as: (1) an emotional bond between client and therapist, (2) a healing setting for therapy, (3) a therapist who provides a theoretically and culturally relevant explanation for emotional distress, (4) an adaptive explanation that is acceptable to clients, and (5) procedures that lead clients to do something that is positive and helpful. Nevertheless, CF does not provide therapists with a license to do whatever they want without considering the evidence of a therapy’s efficacy. Rather CF does encourage therapists to make use of specific factors found in ESTs and to practice with a purpose. In support of the importance of CF, Laska and colleagues review the evidence from a number of meta analyses that show that CF (i.e., alliance, empathy, collaboration, positive regard, genuineness, therapist effects) each account for 5% to 11.5% of patient outcomes. These are moderate effects. Specific ingredients of psychotherapies or differences between ESTs account for 0% to 1% of patient outcomes, which represent small effects.

Practice Implications

An excessive focus or reliance on empirically supported therapies (EST) may unnecessarily limit what the profession and funders consider to be evidence-based practice. A common factors (CF) approach provides scientific evidence for effective therapeutic practices that are necessary in addition to the specific treatments found in lists of ESTs. To be effective, therapists should be able to: (1) develop a therapeutic alliance and repair ruptures to the alliance, (2) provide a safe context for the therapy, (3) be able to communicate sound psychological theory for the client’s distress based on evidence, (4) suggest a course of action that is based on evidence, and (5) conduct therapy based on established theories of distress and healing. Laska and colleagues argue that systematic patient progress monitoring and ongoing monitoring of the therapeutic alliance may be an effective method of quality improvement of therapists’ outcomes. Progress monitoring may provide therapists with information about areas for continuing education to improve their patients’ outcomes.

Bell, E. C., Marcus, D. K., & Goodlad, J. K. (2013). Are the parts as good as the whole? A meta-analysis of component treatment studies. Journal of Consulting and Clinical Psychology, 81, 722-736.

Component studies (i.e., dismantling treatments or adding to existing treatments) may provide a method for identifying whether specific active ingredients in psychotherapy contribute to client outcomes. In a dismantling design, at least one element of the treatment is removed and the full treatment is compared to this dismantled version. In additive designs, an additional component is added to an existing treatment to examine whether the addition improves client outcomes. If the dismantled or added component is an active ingredient, then the condition with fewer components should yield less improvement. Among other things, results from dismantling or additive design studies can help clinicians make decisions about which components of treatments to add or remove with some clients who are not responding. For example, Jacobson and colleagues (1996) conducted a dismantling study of cognitive-behavioral therapy (CBT) for depression. They compared: (1) the full package of CBT, (2) behavioral activation (BA) plus CBT modification of automatic thoughts, and (3) BA alone. This study failed to find differences among the three treatment conditions. These findings were interpreted to indicate that BA was as effective as CBT, and there followed an increased interest in behavioral treatments for depression. However, relying on a single study to influence practice is risky because single studies are often statistically underpowered and their results are not as reliable as the collective body of research. One way to evaluate the collective research is by meta analysis, which allows one to assess an overall effect size in the available literature (see my November, 2013 blog on why clinicians should rely on meta analyses). In their meta analysis, Bell and colleagues (2013) collected 66 component studies from 1980 to 2010. For the dismantling studies, there were no significant differences between the full treatments and the dismantled treatments. For the additive studies, the treatment with the added component yielded a small but significant effect at treatment completion and at follow-up. These effects were only found for the specific problems that were targeted by the treatment. Effects were smaller and non-non-significant for other outcomes such as quality of life.

Practice Implications

Psychotherapists are sometimes faced with a decision about whether to supplement current treatments with an added component, or whether to remove a component that may not be helping. Adding components to existing treatments leads to modestly improved outcomes at least with regard to targeted symptoms. Removing components appears not to have an impact on outcomes. The findings of Bell and colleagues’ (2013) meta analysis suggest that specific components or active ingredients of current treatments’ have a significant but small effect on outcomes. Some writers, such as Wampold, have argued that the small effects of specific components highlight the greater importance of common factors in psychotherapy (i.e., therapeutic alliance, client expectations, therapist empathy, etc.). This may be especially the case when it comes to improving a patient’s quality of life.

Author email: david.marcus@wsu.edu

Larsson, B. P., Broberg, A. G., & Kaldo, V. (2013). Do psychotherapists with different theoretical orientations stereotype or prejudge each other? Journal of Contemporary Psychotherapy, 1-10.

A remarkable difference between the field of psychotherapy and other health care or scientific areas is that psychotherapy is organized in different and somewhat competing theoretical orientations or schools. Leading thinkers of psychotherapy integration, have emphasized how this division presents an obstacle to integration and therefore to progress within the practice and science of psychotherapy. One of these obstacles could be persistent stereotypes that psychotherapists might have about other therapists who practice from a different theoretical orientation. Social psychologists have long known that people in one group (e.g., an in-group) may misjudge or stereotype people in other groups (e.g., out-groups). Stereotypes may be negative if members of an in-group hold a positive bias toward their in-group coupled with antagonism toward members of an out-group. Do psychotherapists stereotype other therapists who practice from a different theoretical orientation? A recent study by Larsson and colleagues addressed this question. They surveyed 416 therapists divided into four ‘pure’ self-reported schools: 161 psychodynamic therapists, 93 cognitive therapists, 95 behavioural therapists, and 67 integrative/eclectic therapists. Most were women (76%), mean age was in the mid 50s, mean experience was 5 to 10 years, and they represented a variety of disciplines including psychology, psychiatry, social work, and nursing. In the first section of the survey, therapists indicated what focus they deemed most important to their own psychotherapeutic work, including: (1) therapeutic relationship, (2) patient’s thoughts, (3) patient’s feelings, (4) patient’s behaviour, or (5) connection between the patient’s thoughts, feelings, and behaviors. Therapists then estimated how they thought psychotherapists from other orientations would rate each of these foci. In the second section of the survey, therapists completed scales about what they deemed were important aspects of psychodynamic, cognitive, behavioral, and eclectic/integrative therapy, respectively. Once again, they rated how they thought therapists from the other orientations would respond. Self-ratings of therapists within each orientation indicated the ‘true’ (i.e., prototypical) opinions of each orientation. The differences between ‘true’ opinions of the in-group versus the in-group’s ratings of therapists from other orientations (i.e. of the out-group) indicated the level of misjudgement or stereotyping. Of the 18 areas on which out-groups were rated, 11 were significantly misjudged by the in-group. Eclectic/integrative therapists were much less likely to stereotype therapists of cognitive or psychodynamic orientations, who were equally likely to stereotype others. The belief that one’s own orientation compared to others is better characterized as an applied science (a belief endorsed most often by cognitive therapists) was a statistically stronger predictor of stereotyping than orientation per se.

Practice Implications

Some researchers argue that different orientations are more similar in their practice of psychotherapy than theory would predict. Furthermore, research about common factors in psychotherapy suggests that these factors may be more important than techniques specific to a school of psychotherapy. However, as long as there are different therapeutic orientations there will likely remain a tendency among some psychotherapists to search for differences rather than to look for similarities between their own and other orientations. This may lead to stereotyping (i.e., an inaccurate opinion about therapists of other orientations), and perhaps negative stereotyping. Psychotherapists and researchers may want to keep in mind the tendency to stereotype clinicians from other orientations when talking to or about other psychotherapists. Such stereotyping is likely an impediment to good client care and research.

Author email: billy.larsson@psy.gu.se