Blog

Miller, D.J., Spengler, E.S., & Spengler, P.M. (2015). A meta-analysis of confidence and judgement accuracy in clinical decision making. Journal of Counseling Psychology, 62, 553-567.

People can make errors in judgements based on decision making rules that are biased. Clinicians also may be prone to making such errors. In their Nobel Prize winning work, Kahneman and Tversky outlined a number of heuristics (i.e., mental shortcuts) that lead to cognitive biases, which in turn affect accuracy of decisions. For example, when making a differential diagnosis clinicians may: rely too heavily on only one piece of information which may be the most available (e.g., “I vividly remember a patient with conversion disorder who had the same history”); or ignore that a particular event (e.g., conversion disorder) is very rare; or seek confirming rather than disconfirming evidence (e.g., the patient has PTSD symptoms that can explain some symptoms). Complicating these biases is the tendency for clinicians to be over-confident. For example, in one study the average psychotherapist rated their performance as better than 80% of their peers, and no therapist rated him or herself in the lower 50^th percentile among peers. In their meta analysis, Miller and colleagues reviewed 36 studies of the relationship between clinician confidence ratings and accuracy of decisions among 1,485 clinicians. The authors were particularly interested in the overconfidence bias, which occurs when individuals report higher confidence in their judgments than is warranted by their actual accuracy. For example, studies have assessed the impact of clinician confidence on clinical accuracy in: detecting random responding on a psychological test, diagnosing a brain disorder verified by medical test using neuropsychological test data, predicting future violence and recidivism in offenders, and patient progress in psychotherapy. Most studies find that clinicians are quite confident in their judgments. But, is this confidence warranted? Miller and colleagues’ meta analysis found a significant but small (r = .15) association between confidence and accuracy. This suggests that clinician confidence is only slightly indicative of decision-making accuracy. The effect was a little larger for more experienced clinicians (r = .25), indicating that more experience and training resulted in somewhat more consistency between a clinician’s confidence and their clinical accuracy. Further, higher confidence leads to poorer accuracy when clinicians have to make repeated decisions without feedback, when feedback is not written, and when an event is rare.

Practice Implications

Clinicians, like everyone else, are sometimes subject to making errors when they only look at confirming evidence, when they rely only on their own memory rather than objective data, and when they are over-confident. Accuracy can be increased when clinicians use decision-making aids that provide quality corrective feedback. Aids to help in decision making might include the use of: objective standardized test data, repeated measurements with feedback to assess patient progress in psychotherapy, and actively looking for disconfirming evidence before making a clinical judgement. As the authors conclude, confidence is not a good substitute for accuracy.

Owen, J., Drinane, J. M., Idigo, K. C., & Valentine, J. C. (2015). Psychotherapist effects in meta-analyses: How accurate are treatment effects? Psychotherapy, 52(3), 321-328.

One of the ongoing debates in the psychotherapy research literature has to do with the relative efficacy of psychotherapies. Is psychotherapy brand A (CBT, for example) more effective than psychotherapy brand B (psychodynamic therapy, for example)? The most common way to test this question is with randomized controlled trials (RCTs), in which clients are randomly assigned to treatment condition (brand A or B). This study design controls for systematic bias in the results that may be caused by differences between clients. But what about therapists? We know for example that therapist effects (i.e., differences between therapists) account for approximately 5% to 10% of client outcomes. Therapist effects are often larger than the effect of the empirically supported treatment that is being offered. Yet it is almost unheard of for therapists to be randomized to treatments, so therapist effects are not controlled in most psychotherapy trials. As a result the effects of the differences between therapists get statistically rolled into the treatment effects. As Owen and colleagues point out, the impact of not controlling for therapist effects is that some differences between treatments in an RCT will appear statistically significant when in fact they are not. One can control for the effect of therapist differences, thus providing a more accurate estimate of treatment effects, but this is rarely done in published RCTs. So, when these RCTs are summarized in a meta analysis, the meta analysis results are also affected by ignoring therapist effects. In their study, Owen colleagues did something very clever. They took data from 17 recent meta analyses of RCTs that found differences between two interventions. These included meta analyses of studies comparing: CBT vs alternative treatments, bona fide treatments vs non-bona fide treatments, culturally adapted treatments vs those that were not adapted, etc. There are many other meta analyses that show no differences between treatments, but the authors wanted to focus specifically on the 17 that did show differences. Owen and colleagues statistically estimated what would happen to the original study findings of significant differences between treatments if therapist effects on patient outcomes were controlled. They controlled for three different sizes of therapist effects that accounted for: 5% (small), 10% (medium), or 20% (large) of patient outcomes. Even small therapist effects (5%) reduced the number of significant differences between treatments from 100% to 80%. When psychotherapist effects were estimated to be medium (10% - which is the best estimate based on research), the number of significant differences between treatments dropped to 65%. For large therapist effects (20%), the number of significant treatment differences was only 35%.

Practice Implications

I have argued previously that the psychotherapist matters. Placing more time and effort in developing good reflective practice based on quality information and developing therapist skills like empathy, progress monitoring, and identifying and repairing alliance ruptures will result in better patient outcomes. As Owen and colleagues note, when reading an RCT that claims to find significant differences between psychotherapies, ask yourself if they took into account the effects of differences between therapists.

Elliott, R., Bohart, A. C., Watson, J. C., & Greenberg, L. S. (2011). Empathy. Psychotherapy, 48(1), 43.

There has been a recent upsurge in interest in empathy in psychotherapy following scientific studies in the field of social neuroscience. This research has focused on activation in areas of the brain associated with emotional stimulation, perspective taking, and emotion regulation. Conceptualizations of the role of empathy in psychotherapy have a rich history in both client-centered and psychodynamic traditions. Carl Rogers defined empathy in part as “...the therapist’s sensitive ability and willingness to understand the client...from the client’s point of view.” Elliott and colleagues indicate three main modes of expressing therapeutic empathy: empathic rapport (compassionate understanding of the client’s experience); communicative attunement (ongoing effort to stay attuned with the client’s experience); and person empathy (experience-near understanding of the client’s world). In this meta-analysis of research on therapeutic empathy, Elliott and colleagues were interested in the strength of the relationship between therapist empathy and client outcome, and factors that might determine this relationship. Their meta analysis included 57 different studies of 3,599 clients. The relationship between therapist empathy and client outcome was medium-sized (r = .31), and in the same order of magnitude as the alliance-outcome relationship. There were no differences between theoretical orientations in the size of the empathy-outcome relationship – in other words, empathy was equally important across types of therapy. Client measures of therapist empathy had the largest relationship to client outcome, whereas therapist ratings of empathy had the smallest association with client outcomes. In other words, if you are interested in therapist empathy, best to ask the client. Also, the empathy-outcome relationship was larger for less experienced (vs more experienced) therapists and for more severely (vs less severely) distressed clients. That is, empathy likely is most important for newer therapists and more distressed clients.

Practice Implications

Therapist empathy is essential to any psychotherapy regardless of orientation. Empathic attunement and expression is particularly important for clients of newer therapists, and for more distressed clients. Elliott and colleagues suggest that the empathic therapist’s primary goal is to understand the client’s experience and to communicate this understanding to the client. This can be done through: empathic affirmations (i.e., validating the client’s perspective); empathic evocations (bringing the client’s experience to life with rich, evocative, and concrete language); and empathic conjectures (making explicit what is implicit in the client’s narrative). Empathy can deepen client’s experiences, but therapeutic empathy also involves individualizing responses to the client. For example, some fragile patients may find typical expressions of empathy as too intrusive, whereas other clients may find therapeutic empathy to be too directive or too foreign. Being attuned to the client’s receptiveness to empathy is an important therapeutic skill. Elliott and colleagues emphasize that empathy should be grounded in authentic caring for the client and as part of a healthy therapeutic relationship.

Del Re, A.C., Fluckiger, C., Horvath, A.O., Symonds, D., & Wampold, B.E. (2012). Therapist effects in the therapeutic alliance-outcome relationship: A restricted-maximum likelihood meta-analysis. Clinical Psychology Review, 32, 642-649.

The therapeutic alliance, defined as the agreement on tasks and goals and the bond between therapist and patient, is one of the most researched concepts in psychotherapy. A meta-analysis of over 200 studies showed that the association between the therapeutic alliance and patient outcomes is moderate but robust (i.e., consistent across studies, patient types, and therapy types). Some have stated that the importance of the therapeutic alliance as reported in studies is an under-estimate of its real impact on patient outcomes. Del Re and colleagues argue that the main reason for this underestimation is that while the therapist’s effect on the alliance-outcome relationship might be large, the client’s effect might be quite small, and so the average of these two effects (which is what most studies report) will be diminished. Del Re and colleagues conducted the first meta analysis to assess the relative size of therapist versus client effects across many studies. Their strategy was clever. They looked at the ratio of the number of patients to therapists (PTR) within a study as a “predictor” of the alliance-outcome relationship across studies. This allowed them to examine the relative contribution of therapists and clients to the alliance-outcome relationship. Two extreme examples illustrate this ratio. (1) In one study, many patients might have been seen by only one therapist, in which case the alliance-outcome correlation could only be attributed to differences between clients since there was only one therapist. (2) In another study, each client might have been seen by a different therapist (i.e., there were as many therapists as clients), in which case the alliance-outcome correlation could only be attributed to differences between the therapists; that is, there are no differences between clients seen by the same therapist as this did not occur. The patient to therapist ratio (PTR) captures the variability between these two extreme examples across studies. Del Re and colleagues included 69 studies that provided enough information about the number of patients and therapists. The overall correlation between alliance and outcome was moderate, r = .27, which was very similar to what was found in a previous large meta-analysis. PTR was significantly associated with the alliance-outcome relationship even after controlling for a number of possible confounding variables. Patients accounted for almost 0% of the alliance-outcome relationship, whereas the effect of therapists was substantially larger, r = .40, accounting for 16% of the alliance-outcome association.

Practice Implications

Therapists’ capacity to develop an alliance with their patients is associated with outcomes. We also know that some therapists demonstrate better patient outcomes than others. So, therapists who consistently are better at forming alliances with patients likely have patients with better treatment outcomes. The quality of the alliance between patients and therapists appears to be the result of what therapists do or bring to the therapy. And so, on average, the therapist’s role in the alliance is most important for achieving good patient outcomes. Del Re and colleagues note that they were not able to look at the interaction between therapist and patient factors. For example, it may be possible that some therapists might form better alliances some types of patients, but not others. Integrating feedback systems so therapists can monitor the therapeutic alliance and patient outcomes may help therapists identify areas in which they need more training or supervision.

Beutler, L.E., Harwood, T.M., Michelson, A., Song, X., & Holman, J. (2011). Resistance/Reactance level. Journal of Clinical Psychology, 67, 133-142.

Patient resistance to psychotherapy is a persistent and perplexing problem. Resistance can be defined as patient behavior that is directly or indirectly contrary to therapist recommendations or to the health of the patient. However, the label “resistance” implies that the problem lies entirely within the patient, i.e., that the patient is the problem. Beutler and colleagues (2011) argue that it is more accurate to define the problem as “reactance”, which refers to the relational or co-constructed nature of psychotherapy. The notion of reactance (instead of resistance) suggests that the therapist also plays a role in the resistance, since the therapist is also responsible to create a context within which highly ambivalent clients do or do not thrive. Failure to thrive could be viewed as a poor fit between patient and therapy. Using social psychological theory, Beutler and colleagues conceptualized reactance as a state of mind aroused in the patients when he or she perceives their freedom to be limited by the therapy. A therapist may elicit resistant behavior from a patient by assuming more control of the patient’s behavior within and outside of the therapy sessions than is tolerable, by using confrontational techniques, and by creating and failing to repair alliance ruptures. Beutler and colleagues argued that therapist directiveness was a key factor in determining reactance in the therapy. Therapist directiveness refers to the extent to which a therapist dictates the pace and direction of therapy. Beutler and colleagues conducted a meta analysis to assess if therapist directiveness was associated with poorer outcome in patients who were more resistant in therapy. The meta analysis included 12 studies with 1,103 patients. They found that higher patient resistance was related to poorer outcomes, and the effect was moderate. The interaction between therapist directiveness and patient level of resistance directly affected outcomes, and this effect was significant and large. That is, greater therapist directiveness with patients who were more resistant resulted in poorer outcomes. Conversely, patients who were low in resistance responded well to more directive therapy.

Practice Implications

Therapists should view some manifestations of client resistance as a signal that they are using ineffective methods. A therapist’s response to resistant states in a patient requires: acknowledgement and reflection of the patient’s concerns; discussion of the therapeutic relationship; and renegotiation of the therapeutic contract regarding goals and therapeutic roles. These therapist responses are designed to provide the patient with a greater sense of control over the process. High reactance indicates that a treatment should: de-emphasize therapist authority and guidance, employ tasks that are designed to provide the patient with control and self-direction, and de-emphasize the use of rigid homework assignments. As Beutler and colleagues indicate, resistance is best characterized as a problem of the therapy relationship (not of the patient) and as such, becomes a problem for the therapist and patient to solve. The skilled therapist can find a way to stimulate change and reduce a patient’s fear of losing control or freedom.

Handbook of Psychotherapy and Behavior Change: Starting in March 2013 I will review one chapter a month from the Handbook of Psychotherapy and Behavior Change in addition to reviewing psychotherapy research articles. Book chapters have more restrictive copy right rules than journal articles, so I will not provide author email addresses for these chapters. If you are interested, the Handbook table of content and sections of the book can be read on Google Books.

Baldwin, S. & Imel, Z.E. (2013). Therapist effects. In M.E. Lambert (Ed.), Bergin and Garfield’s Handbook of Psychotherapy and Behavior Change, 6^th Edition (pp. 258-297). New York: Wiley.

Does it matter that some therapists are more effective than others? Can less effective therapists be trained to improve their outcomes and relationship quality with patients?  These are important questions not only for our patients’ well-being but also for the long term survival of psychotherapy as a health enterprise. If we do not measure outcomes and help therapists who are less effective, stakeholders (i.e., clients, families, agencies, insurance companies) may stop paying for the services. In the September 2013 blog I discussed a large study that showed that a few therapists were reliably harmful and some therapists were reliably helpful to their patients. That study also reported that most therapists were effective in 5 of 12 problem domains for which their patients sought help. What these findings and the Handbook chapter by Baldwin and Imel (2013) show is that there are significant between-therapist effects (i.e., therapists differed from each other on patient outcomes) and within-therapist effects (i.e., therapist outcomes within their own caseload differed based on the patients’ problems). Baldwin and Imel (2013) reported on their meta analysis in which between-therapist differences accounted for 5% of the outcome variance. That seems small, but it’s not. One study, for example, estimated that for each 100 patients that would be treated, the worst therapist compared to the best therapist would have 6 more patients who deteriorated. I would prefer my loved ones to be seen by the best therapist, even if the difference between best and worst is only 5%. Nevertheless, 95% of the variance in outcomes is within the therapist’s caseload. That is, the patient, other contextual variables, and the therapist-patient relationship are by far the biggest contributors to outcome. As Baldwin and Imel point out, not only are some therapists are more effective for some patients and not others, but also some therapists are better at developing a therapeutic relationship with some patients than with others. Baldwin and Imel reported that, on average, 9% of the variance in the quality of the therapeutic alliance is associated with the therapist – that’s a clinically meaningful effect.

Practice Implications

As Baldwin and Imel (2013) state, ignoring therapist accountability is detrimental to patients and to the mental health field in general. If stakeholders do not see evidence of positive outcomes, then they will withdraw funding, and patients will have even less access to services. Therapists differ in their outcomes, and outcomes also differ within each therapist’s caseload. If a primary goal is to improve therapist performance and patient outcomes, then therapists need to measure outcomes and therapeutic relationship quality. This knowledge about performance with specific patients can help therapists seek continuing education and training to improve outcomes and therapeutic alliances with specific patients for whom the therapist is less effective. This may require continuous outcome monitoring and real-time feedback to therapists regarding their patients’ outcomes (see my September 2013 blog in identifying clients who might deteriorate).