Blog

Miller, D.J., Spengler, E.S., & Spengler, P.M. (2015). A meta-analysis of confidence and judgement accuracy in clinical decision making. Journal of Counseling Psychology, 62, 553-567.

People can make errors in judgements based on decision making rules that are biased. Clinicians also may be prone to making such errors. In their Nobel Prize winning work, Kahneman and Tversky outlined a number of heuristics (i.e., mental shortcuts) that lead to cognitive biases, which in turn affect accuracy of decisions. For example, when making a differential diagnosis clinicians may: rely too heavily on only one piece of information which may be the most available (e.g., “I vividly remember a patient with conversion disorder who had the same history”); or ignore that a particular event (e.g., conversion disorder) is very rare; or seek confirming rather than disconfirming evidence (e.g., the patient has PTSD symptoms that can explain some symptoms). Complicating these biases is the tendency for clinicians to be over-confident. For example, in one study the average psychotherapist rated their performance as better than 80% of their peers, and no therapist rated him or herself in the lower 50^th percentile among peers. In their meta analysis, Miller and colleagues reviewed 36 studies of the relationship between clinician confidence ratings and accuracy of decisions among 1,485 clinicians. The authors were particularly interested in the overconfidence bias, which occurs when individuals report higher confidence in their judgments than is warranted by their actual accuracy. For example, studies have assessed the impact of clinician confidence on clinical accuracy in: detecting random responding on a psychological test, diagnosing a brain disorder verified by medical test using neuropsychological test data, predicting future violence and recidivism in offenders, and patient progress in psychotherapy. Most studies find that clinicians are quite confident in their judgments. But, is this confidence warranted? Miller and colleagues’ meta analysis found a significant but small (r = .15) association between confidence and accuracy. This suggests that clinician confidence is only slightly indicative of decision-making accuracy. The effect was a little larger for more experienced clinicians (r = .25), indicating that more experience and training resulted in somewhat more consistency between a clinician’s confidence and their clinical accuracy. Further, higher confidence leads to poorer accuracy when clinicians have to make repeated decisions without feedback, when feedback is not written, and when an event is rare.

Practice Implications

Clinicians, like everyone else, are sometimes subject to making errors when they only look at confirming evidence, when they rely only on their own memory rather than objective data, and when they are over-confident. Accuracy can be increased when clinicians use decision-making aids that provide quality corrective feedback. Aids to help in decision making might include the use of: objective standardized test data, repeated measurements with feedback to assess patient progress in psychotherapy, and actively looking for disconfirming evidence before making a clinical judgement. As the authors conclude, confidence is not a good substitute for accuracy.

Owen, J., Drinane, J. M., Idigo, K. C., & Valentine, J. C. (2015). Psychotherapist effects in meta-analyses: How accurate are treatment effects? Psychotherapy, 52(3), 321-328.

One of the ongoing debates in the psychotherapy research literature has to do with the relative efficacy of psychotherapies. Is psychotherapy brand A (CBT, for example) more effective than psychotherapy brand B (psychodynamic therapy, for example)? The most common way to test this question is with randomized controlled trials (RCTs), in which clients are randomly assigned to treatment condition (brand A or B). This study design controls for systematic bias in the results that may be caused by differences between clients. But what about therapists? We know for example that therapist effects (i.e., differences between therapists) account for approximately 5% to 10% of client outcomes. Therapist effects are often larger than the effect of the empirically supported treatment that is being offered. Yet it is almost unheard of for therapists to be randomized to treatments, so therapist effects are not controlled in most psychotherapy trials. As a result the effects of the differences between therapists get statistically rolled into the treatment effects. As Owen and colleagues point out, the impact of not controlling for therapist effects is that some differences between treatments in an RCT will appear statistically significant when in fact they are not. One can control for the effect of therapist differences, thus providing a more accurate estimate of treatment effects, but this is rarely done in published RCTs. So, when these RCTs are summarized in a meta analysis, the meta analysis results are also affected by ignoring therapist effects. In their study, Owen colleagues did something very clever. They took data from 17 recent meta analyses of RCTs that found differences between two interventions. These included meta analyses of studies comparing: CBT vs alternative treatments, bona fide treatments vs non-bona fide treatments, culturally adapted treatments vs those that were not adapted, etc. There are many other meta analyses that show no differences between treatments, but the authors wanted to focus specifically on the 17 that did show differences. Owen and colleagues statistically estimated what would happen to the original study findings of significant differences between treatments if therapist effects on patient outcomes were controlled. They controlled for three different sizes of therapist effects that accounted for: 5% (small), 10% (medium), or 20% (large) of patient outcomes. Even small therapist effects (5%) reduced the number of significant differences between treatments from 100% to 80%. When psychotherapist effects were estimated to be medium (10% - which is the best estimate based on research), the number of significant differences between treatments dropped to 65%. For large therapist effects (20%), the number of significant treatment differences was only 35%.

Practice Implications

I have argued previously that the psychotherapist matters. Placing more time and effort in developing good reflective practice based on quality information and developing therapist skills like empathy, progress monitoring, and identifying and repairing alliance ruptures will result in better patient outcomes. As Owen and colleagues note, when reading an RCT that claims to find significant differences between psychotherapies, ask yourself if they took into account the effects of differences between therapists.

Fernandez, E., Salem, D., Swift, J. K., & Ramtahal, N. (2015, August 24). Meta-analysis of dropout from cognitive behavioral therapy: Magnitude, timing, and moderators. Journal of Consulting and Clinical Psychology. Advance online publication.

“Dropping out” refers to clients who discontinue therapy prematurely and against professional advice. In contrast, “refusing” refers to clients who do not start a therapy that is made available to them. Together, both dropping out and refusing are referred to as “attrition” from therapy. Attrition is a problem for clinicians because of loss of revenue and time, and a problem for clients because their mental health needs remain unmet. In a previous meta analysis that included 669 studies, Swift and Greenberg (2012) reported that the average drop out rate across all therapies was 19.7%. In this meta analysis, Fernandez and colleagues looked specifically at drop outs and refusers in cognitive behavioral therapy (CBT). The authors reviewed 115 studies that reported drop outs, 36 of which also reported on the number of participants who refused treatment before starting. The average percent of patients who refused CBT prior to starting treatment was 15.9%, and the average percent of patients who dropped out after starting CBT was 26.2%. So the total average attrition rate was 42.1%. Compared to any other disorder, patients with depression were significantly more likely to refuse CBT (21.6%) or to drop out (36.4%). It is possible that depressed patients have a harder time summoning the energy to participate in therapy, and experience lower hope, greater social withdrawal, and lower motivation once they initiate CBT. For those receiving e-therapies (e.g., internet, phone, and CD-based treatments), pre-treatment refusal rates were 10% to 15% higher than individual or group CBT, and drop outs from e-therapies were 10% higher compared to individual or group CBT. Those offered e-therapy might be ambivalent about its utility, the therapeutic alliance might be limited, and they might have a lower sense of engagement in the therapeutic process. Finally, a greater number of planned therapy sessions was related to lower attrition rates. Perhaps the promise of more sessions raised clients’ hopes of achieving better outcomes.

Practice Implications

These findings suggest that engaging and encouraging clients to participate in the therapy may have to start even before therapy begins. This may involve enhancing readiness by means of motivational interviewing, for example. Clients who are depressed are particularly likely to refuse treatment or drop out, and so clinicians must pay particular attention to the level of motivation and engagement of depressed clients. Although e-therapies are promising in that they may allow a therapist or agency to reach more people including those who live in remote areas, the attrition rate of e-therapies may be unacceptably high. Attrition may lead to demoralization and lowered expectations for treatment among these patients, which may negatively impact future treatment. Perhaps e-therapies should not be considered as a first-line treatment for those who can easily access individual or group therapy. Alternatively, the high attrition rates of e-therapies may be reduced by supplementing the intervention with some in-person therapy sessions to enhance engagement and a therapeutic alliance.

Weitz, E.S., Hollon, S.D., Twisk, J., van Straten, A., Huibers, M.J.H., David, D., …. Cuijpers, P. (2015). Baseline depression severity as moderator of depression outcomes between cognitive behavioral therapy vs pharmacotherapy: An individual patient data meta-analysis. JAMA Psychiatry. doi:10.1001/jamapsychiatry.2015.1516.

The American Psychiatric Association guidelines for the treatment of depression indicates that although psychotherapy is adequate for mild to moderate depression, anti-depressant medications are indicated for the treatment of severe depression in major depressive disorder. These recommendations are mainly based on the findings of the National Institute of Mental Health Treatment of Depression Collaborative Research Program that was published in the mid 1990s. Several authors since then have disputed this claim, but no meta-analyses have been done on the studies of head-to-head patient-level comparisons of psychotherapy vs antidepressant medications for the purpose of evaluating their relative efficacy for severity of depression. In this meta analysis, Weitz and colleagues look at medications vs psychotherapy for depression and then evaluate if initial severity of depressive symptoms helped to explain any differences. The authors looked at all studies that compared cognitive behavioral therapy (CBT) against antidepressant medications for depression. They focused on CBT because it was the most often studied of the psychotherapies in this context. A systematic review turned up 24 studies, and they were able to get original patient-level data from the authors of 16 of the 24 studies. This represented over 1,700 participants with major depression. These 16 studies were no different from the 8 studies that did not provide original data. Between 17% and 54% of the 1,700 depressed participants met criteria for severe depression at pre-treatment. There were no significant differences between antidepressant medications and CBT on clinically relevant outcomes in terms of “response” (i.e., improvement) or “remission” (i.e., symptom-free). In total, 63% of patients in the antidepressant medication condition and 58% of patients in the CBT condition responded to treatment, and 51% of patients in the antidepressant medication condition and 47% of patients in the CBT condition met criteria for remission. Most importantly, the effects of CBT and antidepressant medications on response to treatment or remission did not differ based on initial severity of depressive symptoms.

Practice Implications

Patients with severe depression were no more likely to require medication to get better than patients with less severe depression. This meta analysis that included the majority of studies that exist on the topic found no evidence to support the guidelines that severe depression should be treated with antidepressant medications over psychotherapy. The authors conclude that CBT may also be a first-line treatment for severe depression.

Tao, K. W., Owen, J., Pace, B. T., & Imel, Z. E. (2015). A meta-analysis of multicultural competencies and psychotherapy process and outcome. Journal of Counseling Psychology, 62(3), 337-350.

Cultural factors shape health-related beliefs, behaviors and values. For decades, many have argued that therapist multicultural competence shapes the therapy process and affects patient outcomes. Some therapists have poorer outcomes with patients of racial/ethnic minorities compared to White patients. Multicultural competence refers to the ability to work effectively across many groups including minority groups. In 2008, an American Psychological Association Task Force detailed recommendations for multicultural competencies. Multiculturally competent providers are those who: expand their knowledge of their client’s background, use culturally relevant interventions, and gain awareness of their own assumptions and the impact of these on their therapeutic work. In this meta analysis, Tao and colleagues aimed to assess the relationship between multicultural competence in therapists with therapy processes and client outcomes. They reviewed 18 studies that included over 1600 clients, the vast majority of whom identified as a racial/ethnic minority. Therapist multicultural competence was assessed by client self report. Therapist multicultural competence was highly correlated with therapy processes like: therapeutic alliance (r = .61), client satisfaction (r = .72), and session depth (r = .58). The association between therapist multicultural competence and client symptom outcomes were moderate in size but significant (r = .29). A separate analysis showed that the relationship between multicultural competence and therapy process variables (alliance, satisfaction, depth) were significantly larger that associations with client outcomes.

Practice Implications

Therapists’ abilities to integrate aspects of their client’s cultural narrative into their interventions significantly accounted for difference in outcomes. In other words, clients who perceived their therapist as more culturally sensitive had better outcomes. This was likely related to more positive therapeutic processes (i.e., alliance, satisfaction, session depth) between clients and therapist dyads, within which clients perceived the therapist as multiculturally sensitive. A provider’s ability to recognize how their own personal backgrounds influence their own and clients’ behaviors will result in better therapy processes and improved client outcomes.

Author email: k.tao@utah.edu

Clarke, K., Mayo-Wilson, E., Kenny, J., & Phillig, S. (2015). Can non-pharmacological interventions prevent relapse in adults who have recovered from depression? A systematic review and meta-analysis of randomised controlled trials. Clinical Psychology Review, 39, 58-70.

According to the American Psychiatric Association, the risk for relapse from depression can be as high as 60% for those who had one episode, 70% for those who had two episodes, and 90% for those who had three previous episodes. Intervening after recovery from an episode of depression might prevent relapse. A relapse is defined as any significant deterioration in depression following a period of clear improvement. We know that relapse after discontinuing antidepressant treatment is greater than relapse after discontinuing psychotherapy, likely because psychotherapy and not medications result in the patient acquiring new coping skills and strengths. Clarke and colleagues conducted a meta analysis of psychological interventions that were designed or adapted in order to reduce relapse after the acute phase depression. These include mindfulness based therapy (MBT) which helps individuals process experience without judgment by using mindfulness techniques; cognitive behavioral therapy (CBT) which helps to modify thoughts and behaviors key to depression; and interpersonal psychotherapy (IPT) which focuses on helping to deal with interpersonal and social role problems related to depression. Clarke and colleagues reviewed 29 studies that included 4216 participants who had at least one episode of depression, had recovered after treatment, and who received either MBT, CBT, or IPT to prevent relapse. These were compared to control conditions that included wait-lists, treatment as usual, or some other active intervention. Compared to all of the controls, MBT, CBT, and IPT reduced relapse rates from 21% to 25% among patients one year post acute treatment. The effects for CBT were maintained up to two years post treatment. There were no differences between psychotherapies and control conditions in drop out rates.

Practice Implications

Psychotherapies (e.g., MBT, CBT, and IPT) reduce relapse from depression by about 22%. up to one year post recovery. Practitioners should consider offering MBT, CBT, or IPT as a form of booster sessions to reduce the likelihood of relapse from a previous episode of depression. Such interventions are important given the increasing relapse rates for each subsequent episode of depression.