Blog

Delgadillo, J., Deisenhofer, A.-K., Probst, T., Shimokawa, K., Lambert, M. J., & Kleinstäuber, M. (2022). Progress feedback narrows the gap between more and less effective therapists: A therapist effects meta-analysis of clinical trials. Journal of Consulting and Clinical Psychology, 90, 559–567.

Some therapists are more effective than others. This is often referred to as the therapist effect. Somewhere between 1% and 29% of patient outcomes can be attributed to which therapist the patient receives. In general, therapists with high facilitative interpersonal skills, high humility, and an ability to withstand difficulties in practice (i.e., ruptures, burnout) may be more clinically effective. To improve outcomes in therapy, some have suggested using routine outcome monitoring and progress feedback. This involves regularly measuring and tracking patient progress with standardized self-report scales throughout treatment and providing the clinician with this information during therapy. Progress feedback allows the therapist to compare their patient’s progress against norms and against the patient’s own progress in preceding sessions. If the patient is not progressing or is deteriorating, then the therapist is alerted to address the issue. Research indicates that progress feedback makes therapy more effective. Less is known about how progress feedback leads to better outcomes. In this meta-analysis, Delgadillo and colleagues assessed the impact of progress feedback on the therapist effect – that is, does progress feedback improve the outcomes of less effective therapists? The meta-analysis was of six clinical trials with data from 4,549 patients and 131 therapists who were randomly assigned to a progress feedback condition or to a control condition without progress feedback. The variability between therapists (ICC = .011) suggested that 1.1% of the overall variance in patient outcomes was due to therapist effects. However, feedback was associated with a significant reduction in the therapist effect (ICC = .009) by 18.2%. A closer look at the data indicated that progress feedback narrowed the gap between more and less effective therapists, such that patients of less effective therapists benefitted the most from their therapist receiving feedback.

Practice Implications

In this meta-analysis conducted on data from controlled studies, there were few under-performing therapists. However, implementing progress feedback was clinically important to achieve better outcomes among some of these therapists. That is, even a single underperforming therapist could attain relatively poor outcomes with dozens or even hundreds of patients. Who the therapist is matters – and some therapists (and their patients) can benefit from supplementing clinical judgement with reliable feedback about patient progress throughout the course of psychotherapy.

Lutz, W., de Jong, K., Rubel, J.A., & Delgadillo, J. (2021). Measuring, predicting, and tracking change in psychotherapy. In M. Barkham, W. Lutz, and L.G. Castonguay (Eds.) Bergin and Garfield’s Handbook of Psychotherapy and Behavior Change (7^th ed.). Wiley. Chapter 4.

Routine outcome monitoring is also known as progress monitoring and feedback. This involves regularly assessing patients with a psychometrically reliable scale before a therapy session and providing the therapist with feedback on the patient’s progress. The feedback includes how the patient is doing relative to the average patient, and how the patient is doing in this session relative to their own scores in previous sessions. By doing this, therapists can get regular and reliable information about their patient’s progress and be alerted to when the patient is not on track to improve or is getting worse. Decisions about patient improvement on a session-to-session basis are complex because they involve knowing how the patient is doing regarding symptoms, quality of life, and relationship functioning relative to other patients and relative to their own functioning in the past. No wonder therapists often mis-judge when a patient is getting worse. Routine outcome monitoring involves decision-making tools to enhance a clinician’s decisions – like the way a physician uses a blood test or x-ray to enhance their clinical observations. In this part of the chapter, Lutz and colleagues review the research over the past 50 years on outcome monitoring and feedback. The effects of psychotherapy with feedback compared to psychotherapy without feedback ranges from small (g = .07) to medium (d = .40) in size. These effects seem small, but the authors remind us that feedback is a relatively simple clinical tool provided in addition to psychotherapy, and so these positive effects occur are over and above the general effectiveness of psychotherapy. Highest effect sizes are achieved for clients who are not on track (likely to get worse) such that feedback compared to no feedback in these not-on-track patients result in effect sizes ranging from 0.36 to 0.53, indicating a moderate to large effect. Further, when feedback was provided, patient dropout was reduced by 20% compared to when feedback was not provided.

Practice Implications

Although routine outcome monitoring is relatively easy to use, there are barriers to their implementation. Organizational cultures are difficult to change, and resources must be assigned to implement these strategies. Clinicians must have the technology, some training, and funds to purchase the psychometric scales which may be a challenge for some. And attitudinal barriers are a problem if managers or clinicians do not value outcome measurement. Nevertheless, patient reported outcomes with psychometrically valid scales should be central to ensure good patient-centered care. Psychotherapists can benefit from quality information to help their clinical decision making, especially when it comes to identifying patients who might not be benefitting. Such feedback about patients who are at risk of getting worse may help clinicians to adjusting treatment and their interpersonal stances to these patients.

Muir, H. J., Coyne, A. E., Morrison, N. R., Boswell, J. F., & Constantino, M. J. (2019). Ethical implications of routine outcomes monitoring for patients, psychotherapists, and mental health care systems. Psychotherapy, 56(4), 459–469.

On average, psychotherapy is effective for a variety of disorders, however about two thirds of patients in regular clinical practice do not improve or recover. Even more problematic is the 5% to 10% of patients who get worse during psychotherapy. Research has repeatedly demonstrated that psychotherapists are not capable of identifying patients who get worse, and therapists regularly over-estimate their effectiveness. This is not surprising because the information therapists need to make these determinations about patients is complex. Therapists, like most humans, have a difficult time integrating complex information to make nuanced decisions about how to act. Therapists need help in the form of reliable and accurate information to make complex clinical decisions. One approach to addressing these concerns is to use routine outcome monitoring (ROM), which involves assessing patient progress by reliable means as patients progress through therapy. ROM also involves feeding that information back to therapists on a regular basis so that they can make the best determinations. In this narrative review, Muir and colleagues discuss the ethical implications of the use of ROM given the research support. The research indicates that ROM feedback leads better outcomes for patients than treatment as usual, and most importantly ROM feedback to therapists reduces patient deterioration in half. Qualitative research also finds that patients generally view ROM feedback as valuable, and patients prefer to use it. Given these findings, implementing ROM represents one way to help therapists to meet ethical aspirations of beneficence, and to fulfill ethical obligations of forestalling harm. The research also indicates that therapists differ in their outcomes. One large study in clinical practices indicated that above average therapists were two times more effective than other clinicians. Individual psychotherapists may not be aware of their effectiveness and so do not know if they need more training or need to focus on certain types of patients for whom they are particularly effective. Ethically, ROM may be one means by which therapists can assess the boundaries of and areas competence, and those areas for which they may need more training and supervision.

Practice Implications

ROM may be a means for psychotherapists to practice ethically by knowing their patients better and by knowing themselves better. Basing one’s perception of one’s own competence fulfills the ethical requirement to ground clinical practice within the scientific knowledge of the field. Consistent and reliable information may allow therapists to know their own general effectiveness, and also to know for which patients they are more or less effective. This might result in therapists focusing their practices on those patient problems for which they are effective, or to seek further training and supervision in those areas in which they can improve.

Cuijpers, P. (2020) Measuring success in the treatment of depression: What is most important to patients? Expert Review of Neurotherapeutics, 20, 123-125.

There is lots of evidence now that psychotherapies of various types are efficacious for the treatment of depression. Psychotherapy trials focus largely on depressive symptoms, and define major depression according to psychiatric diagnostic manuals. However, the diagnosis of major depression, for example, is not a unitary construct. That is, it is simply a collection of symptoms and signs that are purported to make up a category of disorder. In fact, people with major depression are quite varied on a whole range of things, like severity, coping style, motivation, attachment style, personality, and extent of comorbidity with other diagnoses. This means that many psychotherapy studies may be focusing on patient outcomes (i.e., reduction of depressive symptoms) that may or may not be important to patients. In this paper, Cuijpers reviews the literature on what patients want from psychotherapy. He found that while symptom reduction was important to patients with depressive disorders, it was not the only outcome they wanted from psychotherapy. Patients also want to have a more fulfilling lives, to return to productive work, to solve conflicts with close loved ones, to learn to live with a chronic disability or disease, to learn to handle the effects of trauma, and other quality of life issues. Fortunately, some studies do report the effects of psychotherapy on quality of life, social functioning, anxiety, hopelessness, and interpersonal problems. However, even these studies treat such outcomes as if they were uniformly important to all patients in the study. Very few studies take a personalized approach to patient outcomes, in which the outcomes of interest are those determined by each patient specific to their own circumstances and wishes.

Practice Implications

Psychotherapists who practice from an evidence-informed perspective often try to measure outcomes in their own practices using reliable measurements. However, many of these measurements may be too general for any specific patient, or they may represent outcomes that do not align with what the individual patient wants. Practicing clinicians who assess outcomes in their own practices, may want to consider supplementing standard symptom outcome measures with more personalized assessments for patients.

Imel, Z. E., Pace, B. T., Soma, C. S., Tanana, M., Hirsch, T., Gibson, J., Georgiou, P., Narayanan, S., & Atkins, D. C. (2019). Design feasibility of an automated, machine-learning based feedback system for motivational interviewing. Psychotherapy, 56(2), 318–328.

I do not mean to conjure up the image of a dystopian future, but I could not resist the pithy title for this blog. Ideally, psychotherapists in training or those who seek professional development would receive high quality accurate feedback about their behavior (e.g., about interpersonal skills, empathy, vocal tone, body language) and competence (e.g., regarding specific interventions) in real time. This would allow psychotherapists and trainees can make fine-tuned adjustments to their behaviors and interventions that match or complement the specific patient with which they are working. But, given the current technology, this is impossible. Instead psychotherapy training and feedback to practicing clinicians is slow, cumbersome, and imprecise. Current supervision and consultation practices rely on giving feedback based on the clinician’s verbal case report or, at best, based on viewing video recordings. There are systems that provide feedback on patient outcomes that may alert psychotherapists to something going amiss in for the patient. But such feedback occurs post-session, is based on patient self-report, and does not inform immediate in-session therapist behaviors. In this study, Imel and colleagues evaluated an initial proof of concept of an automated feedback system that generated quality metrics about specific therapist interventions and about therapist skills like empathy. They used computer technology based on natural language processing to take conversational data from video of psychotherapy sessions in order to answer questions like: “what did the therapist and patient talk about during the session?”, “how empathic was the therapist?”, and “how often did the therapist use reflections versus closed questions in the session?” The authors developed a machine learning tool to transcribe, code, and rapidly generate feedback to 21 experienced and novice therapists who recorded a 10-minute session with a standardized patient (a standardized patient is an actor who loosely follows a script). The machine learning technology was accurate at defining or coding a “closed question” by a therapist (e.g., a question with a yes/no answer; inter rater agreement with a human coder ICC = .80), but not as accurate at defining or coding a therapist empathic statement (inter rater agreement with a human coder ICC = .23). The system provided immediate feedback the therapists about their behaviors during the session using graphics and text (fidelity to specific interventions, counseling style, empathy, percent open/closed questions, percent reflections). All therapists rated the tool as “easy to use”, 86% strongly agreed that the feedback was representative of their performance, 90% agreed that if the tool was available, they would use it in their clinical practice.

Practice Implications

Typically, professional consultation or supervision involves a consultant giving the therapist feedback based on imprecise descriptions of events in a therapy session that occurred at some point in the recent past. This method of training and consultation in psychotherapy has not changed much in the past 60 years. One key drawback of current methods of training and consultation is that they do not make use of real-time feedback to help therapist adjust behaviors to the specific patient or context. It is possible that in the near future with rapid advances in artificial intelligence and machine learning a therapist will be able to finish a session with a patient and receive an immediate feedback report about the previous hour. The feedback might include metrics on empathy, the percent of questions vs reflections, competence in specific interventions, among other personalize ratings. This future might also have novice trainees receive immediate real-time in-session feedback about behaviors of interest that need to be adjusted, or for which more training is necessary. For some, this might be a vision of a dystopian future, for others it may represent a way forward in which therapists achieve more refined skills and better patient outcomes.

Lambert, M. J., Whipple, J. L., & Kleinstäuber, M. (2018). Collecting and delivering progress feedback: A meta-analysis of routine outcome monitoring. Psychotherapy, 55(4), 520-537.

Somewhere between 5% and 10% of adult clients in clinical trials of psychotherapy get worse, and the numbers are likely higher in regular clinical practice. In addition, some therapists are more effective than others, so that some therapists have few clients who get worse whereas others consistently have high rates of poor client outcomes. Unfortunately, therapists have a difficult time assessing their client outcomes. Many therapists are overly optimistic about their clients’ outcomes, and clinicians frequently do not identify when clients get worse. One likely reason for this erroneous assessment of client outcomes is that typically psychotherapists do not have quality information in order to make accurate decisions and predictions. Assessing client outcomes on a regular basis throughout treatment is a difficult and complicated endeavour, and one that is beyond the capacity of most people. So, like other professionals (pilots, air traffic controllers, engineers) psychotherapists can improve their predictions and decision-making if they have access to quality information about their clients’ functioning. One source of such information for psychotherapists could be from the use of routine outcome monitoring. Routine outcome monitoring involves assessing client mental health functioning with reliable psychometric scales throughout the course of treatment, and feeding this information back to therapists who can use the data to adjust what they are doing if necessary. The two most commonly used outcome monitoring tools are the Outcome Questionnaire-45 (OQ-45) which is part of the OQ Analyst Feedback System, and the Outcome Rating Scale (ORS) and Session Rating Scale (SRS) which are part of the Partners for Change Outcome Management System (PCOMS). In this meta-analysis, Lambert and colleagues assessed the effect of regular outcome monitoring with the OQ-45 and the ORS to improve client outcomes. In 15 studies with almost 8,500 participants, the OQ-45 outperformed treatment as usual but with a small effect (SMD = .14, 95% CI [.08, .21]). However, the positive effect of using the OQ-45 with feedback was larger for the 31.2% of clients who were not doing well in therapy (SMD = .33, 95% CI [.25, .41]). Among those studies that used the OQ standardized feedback system that provides recommendations to therapists, the effects were even larger (SMD = .49, 95% CI [.25, .73]). Similarly, in nine studies with over 2,000 participants, the effects of using the PCOMS system had a small to moderate positive effects on client outcomes (SMD = .40, 95% CI [.29, .51]).

Practice Implications

The research evidence supports the use of routine outcome monitoring with the OQ-45 or the PCOMS to improve client outcomes. Quality information that is fed back to clinicians can compensate for the limited capacity that any clinician has to accurately detect a client that is worsening in psychotherapy. The information provided to therapists with these feedback systems can highlight potential problems in the client and identify strain in the therapeutic alliance. This information can sensitise therapists to at-risk clients and situations, and encourage therapists to adjust their interventions or interpersonal stances accordingly.

Author email: lambert.michaelphd@gmail.com