Blog

Leichsenring, F. Abbass, A., Hilsenroth, M.J., Leweke, F., Luyten, P., ….Steinert, C. (2016). Bias in research: Risk factors for non-replicability in psychotherapy and pharmacotherapy research. Psychological Medicine, doi:10.1017/S003329171600324X.

An important feature of research is that it should be replicable. That is, another researcher should be able to obtain the same finding as the original study as a pre-requisite for the validity of the conclusions. A recent estimate for cognitive and social psychology research is that only about 36% to 47% of studies are successfully replicated. Another study showed similar low replicability of psychotherapy and pharmacotherapy research. Results that are neither replicable nor valid can lead to improper treatment recommendations. Leichsenring and colleagues review several research biases that affect the replicability of findings in psychotherapy and pharmacotherapy research, and they discuss how to limit these biases. Psychotherapy trials often involve an established treatment approach that is pit against a comparison treatment in a head to head contest. Below I list some of the biases detailed by Leichsenring and colleagues that may affect the validity of psychotherapy trials. First, in psychotherapy trials a large proportion of the differences in outcomes between a treatment and a comparison may be due to the researcher’s allegiance to a particular therapy modality. This may be expressed unconsciously by selecting outcome measures that are more sensitive to the effects of one type of treatment versus another. For example the Beck Depression Inventory (BDI) is particularly sensitive to changes in cognitions, whereas the Hamilton Depression Rating Scale (HDRS) is particularly sensitive to physiological side effects related to antidepressant medications. One way to deal with researcher allegiance effects is to include researchers and therapists who have an allegiance to both of the treatments that are under study. Second, the integrity of the comparison treatment may be impaired. That is the comparison treatment may not be carried out exactly as originally intended. This could occur in pharmacological trials in which doses do not match clinical practice, or in psychotherapy trials in which therapists in the comparison treatment may be told to ignore key symptoms. Properly training and supervising therapists and not constraining them by the study protocol is important to avoid this type of bias. Third, some studies make a lot of noise about small effects that are statistically significant. When two bona-fide psychotherapies are compared the differences tend to be small – this is a common finding. Small differences, even if statistically significant, often turn out to be random, unimportant, and of little clinical significance. Concurrent with this problem is that sometimes researchers will use multiple outcome measures, find significant differences only with some, and report these as meaningful. This refers to selectively emphasizing a small number of findings among a larger number of analyses, which are likely due to chance variation and therefore not replicable.

Practice Implications

What should a clinician do when reading a comparative outcome study of psychotherapy? There are some technical red flags for research bias that require specialized knowledge (e.g., small sample sizes and their effect on reliability, over-interpreting statistical significance in the context of small effects, and non-registration of a trial). But there are a few less technical things to look for. First, I suggest that clinicians focus primarily on meta-analyses and not on single research studies. Although not perfect, meta-analyses review a whole body of literature, and are more likely to give a reliable estimate of the state of the research in a particular area. Second, clinicians should ask some important questions about the particular study: (a) are the results unusual (i.e., when comparing 2 bona-fide treatments, is one “significantly” better; or are the results spectacular); (b) does the research team represent only one treatment orientation; and (c) do the researchers reduce the integrity of the comparison treatment in some way (e.g., by not training and supervising therapists properly, by unreasonably limiting what therapists can do)?  

Waldinger, R.J. & Schulz, M.S. (2016). The long reach of nurturing family environments: Links with midlife emotion-regulatory styles and late-life security in intimate relationships. Psychological Science. DOI: 10.1177/0956797616661556.

Although, not a psychotherapy study, this research has important implications for psychological treatment of adults, including older adults. This research, drawn from the original Grant study, is extraordinary because the sample is from a 78-year long study of 81 men. The original cohort of over 200 men were first assessed as adolescents and young adults between 1939 and 1942. At that time, the original authors conducted intensive interviews of the adolescents` family experiences and current life situations. These men were re-interviewed in mid-life in the 1960s (aged between 45 and 50 years), which included interviews and assessments of challenges in relationships, work functioning, and physical health. Waldinger and Schulz recently re-interviewed these men and their current partner in late-life (aged between 75 and 85 years), with interviews focusing on their current partner relationship. Raters reviewed audio recordings and notes from all the interviews and coded for: (a) quality of family environment in childhood (distant, hostile vs cohesive, warm) - taken from the first interview; (b) style of regulating emotions (suppressive, maladaptive vs engaged, adaptive) – taken from the midlife interview; and (c) security of attachment with their current partner (secure, comforting vs insecure, anxious) – taken from the late-life interview. The authors found that more nurturing early family environments were significantly linked with late-life attachment security with a partner (r = .23, 95% CI = .01, .45), and early family environment was significantly related to midlife adaptive emotion regulation strategies (r = .29, 95% CI = .06, .50). Also, adaptive emotion regulation strategies in midlife were significantly correlated with greater late-life attachment security (r = .23, 95% CI = .05, .51). These are medium-sized correlations, but they are remarkable because they represent associations between variables that were assessed decades apart. Through a statistical mediation analysis, the authors also reported that adaptiveness of emotion-regulation strategies partially explained why positive childhood family environments may lead to late-life attachment security (accounting for 6% of the variance).

Practice Implications

This compelling study adds to the argument that early family environment shapes the way adults regulate their emotions, which in turn affects how they experience relationships in old age. More securely attached adults were better able to meet two challenges associated with aging: accepting vulnerability in depending on a partner, and accepting the responsibility of being depended upon by that partner. The early family environment indeed has a long reach. Psychotherapy directed at reducing the effects of childhood adversity takes on a heightened meaning in the context of these findings. Treatment for adults who struggle with the consequences of non-nurturing early environments should include improving emotion regulation strategies.

Cuijpers, P., Cristea, I.A., Karyotaki, E., Reijnders, M., Huibers, M.J.J. (2016). How effective are cognitive behavior therapies for major depression and anxiety disorders? A meta-analytic update of the evidence. World Psychiatry, 15, 245-258.

You might think that an esoteric topic like study quality should not really be of interest or concern to clinicians – but it is an important topic with practice implications. In this meta analysis Pim Cuijpers and his research group updated the meta analytic evidence for the efficacy of cognitive behavioral therapy (CBT) for a variety of disorders (major depressive disorder [MDD], generalized anxiety disorder [GAD], panic disorder [PAD], and social anxiety disorder [SAD]). The important thing about meta analyses is that the method combines the effect sizes from all relevant studies into a single metric – an average effect size. These average effect sizes are much more reliable than findings from any one single study. In fact, whenever possible, clinical decision-making should be based on a meta analysis and systematic review and not on a single study. Meta analyses also allow one to give more weight to those studies that have larger sample sizes, and that employ better methodologies. Even more, meta analytic techniques allow one to adjust the averaged effect size by taking into account publication bias (i.e., an indication of the effects from studies that might have been completed but were never published, likely because they had unfavorable findings). Usually, average effect sizes are lower when they are adjusted for study quality and publication bias. Cuijpers and colleagues’ meta analyses found that the unadjusted average effects of CBT were large for each of the disorders (ranging from g = .75 to .88 [confidence intervals not reported]). However adjusting for publication bias reduced the effects to medium-sized for MDD (g = .65) and GAD (g = .59). Only 17.4% of the individual studies of CBT were considered to be of “high quality” (i.e., studies that use the best methodology to reduce bias, like random allocation, blinding, using all the available data, etc.). After adjusting for study quality, the effects of CBT for SAD (g = .61) and PAD (g = .76) were also reduced to medium-sized. Not surprisingly, the effects of CBT were largest when the treatment was compared to a wait-list no-treatment control group. The effects were small to moderate when CBT was compared to treatment as usual or to a placebo.

Practice Implications

Even when adjusting for study quality and publication bias, the average effects of CBT were medium-sized for a variety of common disorders compared to control conditions. Unfortunately, the quality of the studies was not high for most trials, reducing the effect sizes and lowering our confidence in the efficacy of the treatment. Nevertheless, the findings of this meta analysis suggest that CBT will likely have moderate effects for the average patient with MDD, SAD, PAD, and GAD.

Saxon, D., Barkham, M., Foster, A., & Parry, G. (2016). The contribution of therapist effects to patient dropout and deterioration in the psychological therapies. Clinical Psychology and Psychotherapy. Advanced online publication, DOI: 10.1002/cpp.2028.

Outcomes for patients receiving psychotherapy are generally positive, but not always. For example, patients might drop out of therapy (i.e., unilaterally end therapy). In clinical trials, the average drop out rate is somewhere between 17% and 26% of patients. Also, patients might deteriorate during therapy (i.e., show a reliable negative change in symptoms from pre- to post-therapy).  On average, about 8.2% of patients show a reliable deterioration after therapy. In this large study from a practice-based research network in the UK, Saxon and colleagues were interested in estimating the effect that therapists had on patient drop out and deterioration. Therapist effects refer to differences between therapists and the effects of this difference on patient outcomes. The authors were also interested in whether therapist effects predicted negative outcomes after controlling for therapist case-mix (i.e., patient variables like severity of symptoms, risk of self harm). Their study included 85 therapists who treated more than 10,000 adult patients over a 10-year period. Each therapist saw between 30 and 468 patients at one of 14 sites in the UK. About half of patients had moderate to severe depressive symptoms, and/or moderate to severe anxiety symptoms prior to starting therapy. Outcomes were measured with a reliable and valid psychometric instrument at pre- and post-treatment. The proportion of patients who dropped out of therapy was 33.8%. Patients who dropped out attended an average of 2.8 sessions (SD = 1.91), whereas treatment completers attended an average of 6.1 sessions (SD = 2.68). About 23.5% of therapists had drop out rates that were significantly worse than average.  These below average therapists (n = 13) had 49% of their patients drop out, whereas above average therapists (n = 20) had only 12% of their patients drop out. Most patients who completed therapy improved (72.2%), but about 7.2% of patients deteriorated to some degree. The average therapist (i.e., 74% of therapists) had 4.6% of their patients who got worse, whereas below average therapists (i.e., 4.7% of therapists) had up to 14.9% of their patients who got worse. That is, almost 3 times as many patients deteriorated with below average therapists.

Practice Implications

We know from previous studies that the type and amount of therapist training or theoretical orientation are not predictive of patient outcomes. However, previous research does suggest that therapists’ lack of empathy, negative countertransference, over-use of transference interpretations, and disagreement with patients about therapy process was associated with negative outcomes. Patient safety concerns might necessitate below average therapists to be identified and provided with greater support, supervision, and training.

Farrell, N.R. & Deacon, B.J. (2015). The relative importance of relational and scientific characteristics of psychotherapy: Perceptions of community members vs. therapists. Journal of Behavior Therapy and Experimental Psychiatry. DOI: 10.1016/j.jbtep.2015.08.004

The American Psychological Association defines evidence-based practice (EBP) in psychotherapy as based on: (a) research evidence, (b) clinical expertise, and (c) client characteristics and preferences. We know for example, that clients who receive their preferred treatments better engage with therapy, drop out at a lower rate, and achieve better symptom outcomes. However, we know very little about clients’ preferences for the relative importance of the therapeutic relationship with an empathic therapist versus the scientific merit of the treatment they receive. We do know that therapists generally prefer research on the therapeutic relationship, and that therapists may place greater value on relationship issues versus research support for the treatments they provide. In this study Farrell and Deacon sample 200 members of the community about the relative importance of the relationship with a therapist versus the scientific basis of the treatment. The authors also surveyed a similar number of therapists about what therapists thought clients would prefer (relationship vs research evidence) in psychotherapy. Not surprisingly, community members rated both the therapeutic relationship and research evidence highly when indicating what they preferred should they receive psychotherapy. However, the authors found that members of the community rated the therapeutic relationship much more highly than they rated research evidence (d = 1.24). But the difference shrank (d = .24) when it came to treating panic disorder or obsessive compulsive disorder. Therapists tended to under-estimate the importance of community members’ preferences for scientific evidence for psychotherapy. The under-estimation was greater for therapists who placed less value on research. In other words, therapists who valued research less in their own practice were more likely to underestimate the importance of scientific credibility to members of the general public.

Practice Implications

This is by no means a perfect study. As readers of this blog know, I prefer to write about meta analyses, which are much more reliable than findings from a single study. However, it is quite rare to have a study on a large sample of members of the community, let alone one that asks about their perceptions and preferences about psychotherapy. The findings from this study suggest that members of the community highly value the therapeutic relationship and factors like therapist empathy. However, members of the community also place much faith in the scientific evidence that supports the use of psychotherapy. The preference for both a good therapeutic relationship coupled with research evidence may be very important to most people who may seek therapy. Therapists, particularly those who place less weight on research, should keep in mind that clients value the scientific evidence for psychotherapy.

Author email: bdeacon@uow.edu.au

The Great Psychotherapy Debate: Since in April, 2015 I review parts of The Great Psychotherapy Debate (Wampold & Imel, 2015) in the PPRNet Blog. This is the second edition of a landmark, and sometimes controversial, book that surveys the evidence for what makes psychotherapy work. You can view parts of the book in Google Books.

In this chapter, Wampold and Imel contrast the Medical Model to a Contextual Model of psychotherapy. The Medical Model argues that there is a biological explanation for a disorder and that the basis for treatment is to address the biological system causing the disorder. Understanding the cause of the disorder (e.g., excess stomach acid) leads to an explanation of the mechanism of the disorder and of change (e.g., reduce stomach acid), which in turn leads to specific interventions (e.g., administer an antacid). Key to this model is specificity: that is, a specific disorder can be explained by a specific mechanism, and a specific treatment based on this explanation will result in alleviation of the disorder. In psychotherapy, for example, one could argue that PTSD symptoms are caused by maladaptive avoidance of traumatic memories, which can be successfully treated by repeated exposure to the traumatic memories to reduce symptoms. Exposure is the specific intervention indicated by the purported mechanism or cause of PTSD. By contrast, Wampold and Imel discuss a Contextual Model of psychotherapy. This model combines the elements of common factors in psychotherapy (e.g., the bond, real relationship, therapeutic alliance) with specific factors of interventions. The Contextual Model indicates that people are fundamentally social animals that require relationships with others to survive and to heal. That is, the initial therapeutic bond between client and therapist is the basis of psychotherapeutic practice. There are three elements to the Contextual Model. (1) The real relationship – which is based on genuineness [openness and honesty] and therapist empathy. Both genuineness and empathy are related to client outcomes. (2) Expectations – which, like the placebo effect, increases one’s readiness to benefit from treatment, is related to greater hope of improvement, and belief in the treatment. The placebo effect is associated with improvements in a pharmacological treatments of depression, and client expectations are related to psychotherapy outcomes. (3) Specific ingredients – as indicated in the Medical Model refers to psychotherapeutic interventions based on a psychological theory of a disorder. But unlike in the Medical Model, the Contextual Model sees the key element of specific ingredients as the agreement between client and therapist on the explanation for the disorder and on the treatment. In other words, a therapeutic alliance in part depends on clients and therapists agreeing on the specific tasks and goals of therapy.

Practice Implications

Patient outcomes are enhanced by a positive therapeutic bond, genuineness and congruence in the way a therapist interacts with a client, and a therapeutic alliance in which clients and therapists agree on tasks and goals of therapy. Typically, these occur within a context in which therapists use some specific techniques of therapy to which he or she feels an allegiance. It is also important that the client agree with the explanation for their disorder provided by this specific treatment model and that the client expects that this treatment will provide them with relief. Although common and specific factors of psychotherapy have been seen as having entirely separate roles in client outcomes, Wampold and Imel argue that within the Contextual Model, they are intimately tied to each other.