Blog

Imel, Z. E., Pace, B. T., Soma, C. S., Tanana, M., Hirsch, T., Gibson, J., Georgiou, P., Narayanan, S., & Atkins, D. C. (2019). Design feasibility of an automated, machine-learning based feedback system for motivational interviewing. Psychotherapy, 56(2), 318–328.

I do not mean to conjure up the image of a dystopian future, but I could not resist the pithy title for this blog. Ideally, psychotherapists in training or those who seek professional development would receive high quality accurate feedback about their behavior (e.g., about interpersonal skills, empathy, vocal tone, body language) and competence (e.g., regarding specific interventions) in real time. This would allow psychotherapists and trainees can make fine-tuned adjustments to their behaviors and interventions that match or complement the specific patient with which they are working. But, given the current technology, this is impossible. Instead psychotherapy training and feedback to practicing clinicians is slow, cumbersome, and imprecise. Current supervision and consultation practices rely on giving feedback based on the clinician’s verbal case report or, at best, based on viewing video recordings. There are systems that provide feedback on patient outcomes that may alert psychotherapists to something going amiss in for the patient. But such feedback occurs post-session, is based on patient self-report, and does not inform immediate in-session therapist behaviors. In this study, Imel and colleagues evaluated an initial proof of concept of an automated feedback system that generated quality metrics about specific therapist interventions and about therapist skills like empathy. They used computer technology based on natural language processing to take conversational data from video of psychotherapy sessions in order to answer questions like: “what did the therapist and patient talk about during the session?”, “how empathic was the therapist?”, and “how often did the therapist use reflections versus closed questions in the session?” The authors developed a machine learning tool to transcribe, code, and rapidly generate feedback to 21 experienced and novice therapists who recorded a 10-minute session with a standardized patient (a standardized patient is an actor who loosely follows a script). The machine learning technology was accurate at defining or coding a “closed question” by a therapist (e.g., a question with a yes/no answer; inter rater agreement with a human coder ICC = .80), but not as accurate at defining or coding a therapist empathic statement (inter rater agreement with a human coder ICC = .23). The system provided immediate feedback the therapists about their behaviors during the session using graphics and text (fidelity to specific interventions, counseling style, empathy, percent open/closed questions, percent reflections). All therapists rated the tool as “easy to use”, 86% strongly agreed that the feedback was representative of their performance, 90% agreed that if the tool was available, they would use it in their clinical practice.

Practice Implications

Typically, professional consultation or supervision involves a consultant giving the therapist feedback based on imprecise descriptions of events in a therapy session that occurred at some point in the recent past. This method of training and consultation in psychotherapy has not changed much in the past 60 years. One key drawback of current methods of training and consultation is that they do not make use of real-time feedback to help therapist adjust behaviors to the specific patient or context. It is possible that in the near future with rapid advances in artificial intelligence and machine learning a therapist will be able to finish a session with a patient and receive an immediate feedback report about the previous hour. The feedback might include metrics on empathy, the percent of questions vs reflections, competence in specific interventions, among other personalize ratings. This future might also have novice trainees receive immediate real-time in-session feedback about behaviors of interest that need to be adjusted, or for which more training is necessary. For some, this might be a vision of a dystopian future, for others it may represent a way forward in which therapists achieve more refined skills and better patient outcomes.

Okiishi, J., Lambert, M. J., Nielsen, S. L., & Ogles, B. M. (2003). Waiting for supershrink: An empirical analysis of therapist effects. Clinical Psychology & Psychotherapy, 10(6), 361-373.

Much of psychotherapy research has focused on searching for effective psychotherapies rather than focusing on effective psychotherapists. Research on psychotherapies generally assumes that therapists are equally effective or relatively less important to patient outcomes than the interventions themselves. Therapists in clinical trials are trained to follow a manual in an attempt to reduce the therapists’ impact on patient outcomes, and to focus the study on the specific ingredients of the therapy itself. However, research indicates that the degree to which a therapist follows a manual has little bearing on patient outcomes, and that therapists do differ in terms of their patients’ outcomes. In one large study, between 33% and 65% of therapists was ineffective or harmful. Okiishi and colleagues asked if it is possible to identify highly effective therapists (“super-shrinks”) and highly ineffective therapists (“pseudo-shrinks”) based on their patients’ outcomes. The therapists were 56 men and women who treated 1779 clients in a university counselling centre. Each therapist saw at least 15 clients, so that there was a good sampling of therapists’ outcomes across a variety of clients. Therapists had a range of experience, training, and theoretical orientations. Clients were adults who had moderate to severe problems with anxiety, depression, or adjustment. Outcomes were measured after every session, and the average number of sessions was 5.16 (SD = 7.20). On average clients improved so that their level of distress significantly declined. Therapist characteristics (sex, experience, training background, theoretical orientation) did not predict patient outcomes. However, client change varied significantly, so that some clients improved at a faster rate than others, some did not change, and some got worse. There were no differences between therapists in their clients’ level of distress, so therapists had equivalent caseloads in terms of client initial distress. However, therapists significantly differed from each other in terms of their clients’ outcomes. For example, the top 3 therapists consistently had clients who got better (super-shrinks), and the bottom 3 therapists consistently had clients who got worse (pseudo-shrinks).

Practice Implications

One would hope that a loved one would get to see a “super-shrink” therapist, since these therapists seem to consistently have clients who do well in therapy. But what about the average or “pseudo-shrink” therapist– what can be done to elevate their skills and their patients’ outcomes? We’ve discussed in this blog several things therapists can do to improve their outcomes, including: using progress monitoring in their practice, receiving training focused on deliberate practice, and seeking out specific continuing education around developing, maintaining, and repairing the therapeutic alliance.

Reese, R. J., Duncan, B. L., Bohanske, R. T., Owen, J. J., & Minami, T. (2014). Benchmarking outcomes in a public behavioral health setting: Feedback as a quality improvement strategy. Journal of Consulting and Clinical Psychology, 82(4), 731-742.

Psychotherapy has demonstrated its efficacy in randomized controlled trials. But do these findings in highly controlled studies translate to everyday practice in publicly funded agencies that treat low income clients? Previous research in the US showed that outcomes of treatment-as-usual in public behavioural health agencies are generally not positive, so that only 20 to 35% of clients reliably improved. One approach to improving outcomes is to transport specific evidence-based treatments into practice settings. For example, research on applying CBT for panic and depression in a publicly funded agency resulted in similar outcomes to those achieved in randomized controlled trials. However, an alternative strategy of improving outcomes is to use continuous outcome monitoring, which involves repeated (weekly) measurement of client outcomes with reliable scales, and feedback to therapists on the client’s status relative to previous sessions and relative to other similar clients. Research has demonstrated that this strategy improves client outcomes and reduces the number of clients who deteriorate. In this study, Reese and colleagues examined the outcomes of a large public behavioural health service in the U.S. that treats low-income individuals. The service implemented repeated outcome monitoring of clients with feedback to therapists. Over 5,000 clients mainly with depression, mood, and anxiety disorders were treated by 84 therapists who were licensed at the masters degree or higher. The clients completed the Outcome Rating Scale (a measure of symptom outcome) prior to each session, and the Session Rating Scale (a measure of the therapeutic alliance) after each session. Therapists received two days of training on how to use these measures and on the continuous feedback they were provided in order to improve their treatment of clients and their outcomes. Outcomes from this public behavioural health service were compared to previous large studies in publicly funded settings that implemented specific evidence-based treatments. The findings were similar, with about 42% showing reliable pre- to post-treatment improvement. The results of implementing continuous outcome monitoring with feedback for depressive symptoms were also large and positive (d = 1.34). These effects were similar to benchmarks established in randomized controlled trials of specific psychotherapies.

Practice Implications

Continuous outcome feedback enables therapists to identify clients who are not benefiting
from a given treatment, so that clinicians may collaboratively design different interventions or change their interpersonal stances. The inclusion of outcome monitoring and feedback in this publicly funded psychotherapy system, resulted in outcomes that were: better than what is often seen in such public service settings, equivalent to those public systems that implemented specific evidence based treatments, and similar to those reported in highly controlled randomized trials. The authors concluded that adding routine outcome monitoring and feedback is a viable alternative to transporting specific evidence based treatments to publicly funded psychotherapy programs. The measures used in this study are available free for individuals to use at: betteroutcomesnow.com.

Author email: jeff.reese@uky.edu

Boswell, J. F., Constantino, M. J., Oswald, J. M., Bugatti, M., Goodwin, B., & Yucel, R. (2018). Mental health care consumers’ relative valuing of clinician performance information. Journal of Consulting and Clinical Psychology, 86(4), 301-308.

Research has shown that some therapists are more effective than others both in terms of their overall effectiveness and in terms of their effectiveness with specific patient problems. Further, despite advances in medicine on this topic, there is little or no information provided to patients about a therapist’s track record on overall effectiveness. In any case, little is known about what patients value in psychotherapists and how much they are willing to give up in order to get what they value. For example, do patients prefer therapists who are highly effective for most problems, and would they be willing to tolerate a poorer therapeutic relationship in order to work with such a highly effective therapist? In this study, Boswell and colleagues employed a relative valuing procedure often used in economics to assess the relative value to patients of different therapist characteristics and performance. Patients were asked how much they were willing to give up on one therapist characteristic (therapist’s overall effectiveness with clients [i.e., overall track record]) in order to receive more of some other characteristic (therapist specific effectiveness in a problem domain, a better therapeutic alliance, lower cost of therapy). The study included 403 patients treated in mental health clinics in the U.S. Patient characteristics were typical of those seen in such clinics – predominantly they had problems with depression or anxiety, were 41 years old on average, mostly women (68.5%), and receiving individual psychotherapy (89.3%). In general, patients highly valued a therapist with a track record of general overall effectiveness. However, patients were willing to give up more of their therapists overall effectiveness if the therapist had a track record of successfully treating their specific problem (e.g., therapist A has lower general efficacy but has demonstrated greater specific efficacy for depression). Patients were also willing to sacrifice therapist general effectiveness in order to pay less for therapy (vs paying a higher fee for a more effective therapist), and in order to work with a provider with whom they would have a better therapeutic alliance (vs a lower alliance with a more generally effective therapist). Surprisingly, patients placed a lower value on factors like therapist gender and race. Younger patients put greater value on therapist performance data (i.e., their track record data), suggesting a generational effect in which younger clients tend to prefer to make decisions based on available data.

Practice Implications

Patients were willing to give up some therapist general effectiveness in order to work with someone who has a track record of being effective for their specific problem, who costs less, and with whom they could have a better therapeutic alliance. Fortunately, therapist general efficacy and domain specific efficacy tend to be highly correlated, and so patients may not have to choose between these. The findings also suggest that patients may be willing to see a therapist who is less generally effective if it meant they could have a good relational experience with the therapist. Research indicates that therapists are able to improve their outcomes and therapeutic alliances with additional training and deliberate practice.

Schermuly-Haupt, M. L., Linden, M., & Rush, A. J. (2018). Unwanted events and side effects in cognitive behavior therapy. Cognitive Therapy and Research, 42(3), 219-229.

Unwanted events are negative consequences for clients that may or may not be related to treatment (i.e., events outside of therapy or inside of therapy that may negatively affect clients).  These might include: occupational problems, stigmatization, strains in personal relationships, changes in the social network, patients feeling overwhelmed, undermined self-efficacy, deterioration of symptoms, emergence of new symptoms, suicidality, and others. Side effects refer to negative reactions in clients directly related to appropriately delivered therapy. Research estimates that between 5% and 20% of patients report side effects of psychotherapy. One could argue that side effects may be inevitable even in well-delivered therapy, and therapists who are aware of the potential for side effects may be better equipped to help clients to manage. In this study, Schermuly-Haupt, interviewed 100 psychotherapists who provided CBT in outpatient clinics in Germany about side effects among their clients. All therapists were supervised as part of their work and so the authors assumed the therapy was appropriately delivered. Therapists had on average 5 years of experience and were trained to provide CBT. The interview asked therapists about their most recent treatment case in which the client attended at least 10 sessions. Clients typically had major depression, an anxiety disorder, or a personality disorder, and had attended 28 sessions of therapy on average. During the interview, therapists identified if an unwanted event occurred for a client from a standardized list, and then rated the duration and severity of the effects. They also rated the degree to which the unwanted event was directly related to therapy (i.e., a side effect). Prior to the interview, only 26% of therapists reported their client experienced side effects. However, the interview process found that almost all clients experienced an unwanted event (98%) that may or may have been related to therapy, and 43% experienced at least one side effect that was at least somewhat related to treatment. The most frequent side effects were: “negative wellbeing/distress” (27% of clients), “deterioration of symptoms” (9% of clients) and “strains in family relations” (6% of clients). Of the therapists, 46% rated the side effects as at least moderately severe, and 8.8% of side effects were rated as persistent (lasting more than a month).

Practice Implications

Unwanted events outside of therapy are very common among our clients, but so are side effects from appropriately delivered treatment. Psychotherapy is not always harmless, and it may be best to acknowledge and prepare both clients and therapists for side effects. These may represent ruptures in the alliance that can be managed through alliance-focused therapy, for example. That is, side effects may be caused a mismatch between the goals of a therapist and client, or a disagreement on how to proceed in therapy given what a client needs at the time. Goals and tasks of therapy may need to be renegotiated following the experience of a side effect.

Love, M. & Farber, B.A. (2018). Honesty in psychotherapy: Results of an online survey comparing high vs. low self-concealers, Psychotherapy Research, DOI: 10.1080/10503307.2017.1417652.

An important task of psychotherapy is for therapists to provide a context within which clients feel comfortable disclosing difficult feelings, thoughts, and other experiences in their lives. Self-disclosure likely improves the therapeutic alliance (agreeing on tasks and goals, and an emotional bond between therapist and client), which is necessary for good outcomes. In fact, research indicates that client self-disclosure is generally associated with positive outcomes in therapy. And yet a number of surveys report that clients keep secrets or lie to their therapists. Clients appear to struggle between being honest and self-disclosing versus the fear or anxiety related to doing so. Research indicates that one can describe individuals as high self-concealers in most relationships in their lives. Such individuals consistently conceal negative aspects of themselves from others to help manage their anxiety in relationships in the short term. However, in the long term, high levels of self-concealment increases rumination and anxiety and reduces coping. In this study, Love and Farber conducted an online survey of 572 participants who were currently in therapy or were in therapy in the past year. The sample characteristics and the type of therapy they received were surprisingly similar to a nationally representative sample of clients who seek treatment, though this online survey sample was somewhat younger. Over 84% of clients in this survey reported being dishonest about at least one topic with their therapist. Most frequent topics for being dishonest included: details of sex life (33.9%), suicidal thoughts (21.9%), self-harm (14.5%), real reactions to therapist comments (18.9%), whether therapy was helping (15.7%), and family secrets (16.3%). The most predominant motive for dishonesty was embarrassment or shame (63.6%), followed by doubts that the therapist would understand (27.0%), fear of overwhelming emotions (18.1%), and disappointing or hurting the therapist (16.4%). Not surprisingly, clients who tended to conceal their experiences reported disclosing less distressful information and also reported a lower therapeutic alliance with their therapists. Almost half of high self-concealers reported that dishonesty hurt their therapeutic progress.

Practice Implications

Topics like suicidal ideation and sex are particularly difficult to speak about honestly in therapy, especially for those who are uncomfortable with disclosing in general. Most clients are willing to discuss difficult topics with therapists if the therapist inquires sensitively and directly. High self-concealers are highly attuned to how therapists might react, and these clients anticipate shame or judgement. Therapists need to monitor the state of the therapeutic relationship with each client, especially the client’s perception of therapist warmth and trustworthiness. This could include monitoring for any ruptures in the therapeutic alliance. Further, therapists may need to communicate that self-concealment serves a short term purpose to reduce anxiety, but has a long term cost in terms of amplifying distress.