Blog

Lutz, W., de Jong, K., Rubel, J.A., & Delgadillo, J. (2021). Measuring, predicting, and tracking change in psychotherapy. In M. Barkham, W. Lutz, and L.G. Castonguay (Eds.) Bergin and Garfield’s Handbook of Psychotherapy and Behavior Change (7^th ed.). Wiley. Chapter 4.

Routine outcome monitoring is also known as progress monitoring and feedback. This involves regularly assessing patients with a psychometrically reliable scale before a therapy session and providing the therapist with feedback on the patient’s progress. The feedback includes how the patient is doing relative to the average patient, and how the patient is doing in this session relative to their own scores in previous sessions. By doing this, therapists can get regular and reliable information about their patient’s progress and be alerted to when the patient is not on track to improve or is getting worse. Decisions about patient improvement on a session-to-session basis are complex because they involve knowing how the patient is doing regarding symptoms, quality of life, and relationship functioning relative to other patients and relative to their own functioning in the past. No wonder therapists often mis-judge when a patient is getting worse. Routine outcome monitoring involves decision-making tools to enhance a clinician’s decisions – like the way a physician uses a blood test or x-ray to enhance their clinical observations. In this part of the chapter, Lutz and colleagues review the research over the past 50 years on outcome monitoring and feedback. The effects of psychotherapy with feedback compared to psychotherapy without feedback ranges from small (g = .07) to medium (d = .40) in size. These effects seem small, but the authors remind us that feedback is a relatively simple clinical tool provided in addition to psychotherapy, and so these positive effects occur are over and above the general effectiveness of psychotherapy. Highest effect sizes are achieved for clients who are not on track (likely to get worse) such that feedback compared to no feedback in these not-on-track patients result in effect sizes ranging from 0.36 to 0.53, indicating a moderate to large effect. Further, when feedback was provided, patient dropout was reduced by 20% compared to when feedback was not provided.

Practice Implications

Although routine outcome monitoring is relatively easy to use, there are barriers to their implementation. Organizational cultures are difficult to change, and resources must be assigned to implement these strategies. Clinicians must have the technology, some training, and funds to purchase the psychometric scales which may be a challenge for some. And attitudinal barriers are a problem if managers or clinicians do not value outcome measurement. Nevertheless, patient reported outcomes with psychometrically valid scales should be central to ensure good patient-centered care. Psychotherapists can benefit from quality information to help their clinical decision making, especially when it comes to identifying patients who might not be benefitting. Such feedback about patients who are at risk of getting worse may help clinicians to adjusting treatment and their interpersonal stances to these patients.

Muir, H. J., Coyne, A. E., Morrison, N. R., Boswell, J. F., & Constantino, M. J. (2019). Ethical implications of routine outcomes monitoring for patients, psychotherapists, and mental health care systems. Psychotherapy, 56(4), 459–469.

On average, psychotherapy is effective for a variety of disorders, however about two thirds of patients in regular clinical practice do not improve or recover. Even more problematic is the 5% to 10% of patients who get worse during psychotherapy. Research has repeatedly demonstrated that psychotherapists are not capable of identifying patients who get worse, and therapists regularly over-estimate their effectiveness. This is not surprising because the information therapists need to make these determinations about patients is complex. Therapists, like most humans, have a difficult time integrating complex information to make nuanced decisions about how to act. Therapists need help in the form of reliable and accurate information to make complex clinical decisions. One approach to addressing these concerns is to use routine outcome monitoring (ROM), which involves assessing patient progress by reliable means as patients progress through therapy. ROM also involves feeding that information back to therapists on a regular basis so that they can make the best determinations. In this narrative review, Muir and colleagues discuss the ethical implications of the use of ROM given the research support. The research indicates that ROM feedback leads better outcomes for patients than treatment as usual, and most importantly ROM feedback to therapists reduces patient deterioration in half. Qualitative research also finds that patients generally view ROM feedback as valuable, and patients prefer to use it. Given these findings, implementing ROM represents one way to help therapists to meet ethical aspirations of beneficence, and to fulfill ethical obligations of forestalling harm. The research also indicates that therapists differ in their outcomes. One large study in clinical practices indicated that above average therapists were two times more effective than other clinicians. Individual psychotherapists may not be aware of their effectiveness and so do not know if they need more training or need to focus on certain types of patients for whom they are particularly effective. Ethically, ROM may be one means by which therapists can assess the boundaries of and areas competence, and those areas for which they may need more training and supervision.

Practice Implications

ROM may be a means for psychotherapists to practice ethically by knowing their patients better and by knowing themselves better. Basing one’s perception of one’s own competence fulfills the ethical requirement to ground clinical practice within the scientific knowledge of the field. Consistent and reliable information may allow therapists to know their own general effectiveness, and also to know for which patients they are more or less effective. This might result in therapists focusing their practices on those patient problems for which they are effective, or to seek further training and supervision in those areas in which they can improve.

Cuijpers, P. (2020) Measuring success in the treatment of depression: What is most important to patients? Expert Review of Neurotherapeutics, 20, 123-125.

There is lots of evidence now that psychotherapies of various types are efficacious for the treatment of depression. Psychotherapy trials focus largely on depressive symptoms, and define major depression according to psychiatric diagnostic manuals. However, the diagnosis of major depression, for example, is not a unitary construct. That is, it is simply a collection of symptoms and signs that are purported to make up a category of disorder. In fact, people with major depression are quite varied on a whole range of things, like severity, coping style, motivation, attachment style, personality, and extent of comorbidity with other diagnoses. This means that many psychotherapy studies may be focusing on patient outcomes (i.e., reduction of depressive symptoms) that may or may not be important to patients. In this paper, Cuijpers reviews the literature on what patients want from psychotherapy. He found that while symptom reduction was important to patients with depressive disorders, it was not the only outcome they wanted from psychotherapy. Patients also want to have a more fulfilling lives, to return to productive work, to solve conflicts with close loved ones, to learn to live with a chronic disability or disease, to learn to handle the effects of trauma, and other quality of life issues. Fortunately, some studies do report the effects of psychotherapy on quality of life, social functioning, anxiety, hopelessness, and interpersonal problems. However, even these studies treat such outcomes as if they were uniformly important to all patients in the study. Very few studies take a personalized approach to patient outcomes, in which the outcomes of interest are those determined by each patient specific to their own circumstances and wishes.

Practice Implications

Psychotherapists who practice from an evidence-informed perspective often try to measure outcomes in their own practices using reliable measurements. However, many of these measurements may be too general for any specific patient, or they may represent outcomes that do not align with what the individual patient wants. Practicing clinicians who assess outcomes in their own practices, may want to consider supplementing standard symptom outcome measures with more personalized assessments for patients.

Imel, Z. E., Pace, B. T., Soma, C. S., Tanana, M., Hirsch, T., Gibson, J., Georgiou, P., Narayanan, S., & Atkins, D. C. (2019). Design feasibility of an automated, machine-learning based feedback system for motivational interviewing. Psychotherapy, 56(2), 318–328.

I do not mean to conjure up the image of a dystopian future, but I could not resist the pithy title for this blog. Ideally, psychotherapists in training or those who seek professional development would receive high quality accurate feedback about their behavior (e.g., about interpersonal skills, empathy, vocal tone, body language) and competence (e.g., regarding specific interventions) in real time. This would allow psychotherapists and trainees can make fine-tuned adjustments to their behaviors and interventions that match or complement the specific patient with which they are working. But, given the current technology, this is impossible. Instead psychotherapy training and feedback to practicing clinicians is slow, cumbersome, and imprecise. Current supervision and consultation practices rely on giving feedback based on the clinician’s verbal case report or, at best, based on viewing video recordings. There are systems that provide feedback on patient outcomes that may alert psychotherapists to something going amiss in for the patient. But such feedback occurs post-session, is based on patient self-report, and does not inform immediate in-session therapist behaviors. In this study, Imel and colleagues evaluated an initial proof of concept of an automated feedback system that generated quality metrics about specific therapist interventions and about therapist skills like empathy. They used computer technology based on natural language processing to take conversational data from video of psychotherapy sessions in order to answer questions like: “what did the therapist and patient talk about during the session?”, “how empathic was the therapist?”, and “how often did the therapist use reflections versus closed questions in the session?” The authors developed a machine learning tool to transcribe, code, and rapidly generate feedback to 21 experienced and novice therapists who recorded a 10-minute session with a standardized patient (a standardized patient is an actor who loosely follows a script). The machine learning technology was accurate at defining or coding a “closed question” by a therapist (e.g., a question with a yes/no answer; inter rater agreement with a human coder ICC = .80), but not as accurate at defining or coding a therapist empathic statement (inter rater agreement with a human coder ICC = .23). The system provided immediate feedback the therapists about their behaviors during the session using graphics and text (fidelity to specific interventions, counseling style, empathy, percent open/closed questions, percent reflections). All therapists rated the tool as “easy to use”, 86% strongly agreed that the feedback was representative of their performance, 90% agreed that if the tool was available, they would use it in their clinical practice.

Practice Implications

Typically, professional consultation or supervision involves a consultant giving the therapist feedback based on imprecise descriptions of events in a therapy session that occurred at some point in the recent past. This method of training and consultation in psychotherapy has not changed much in the past 60 years. One key drawback of current methods of training and consultation is that they do not make use of real-time feedback to help therapist adjust behaviors to the specific patient or context. It is possible that in the near future with rapid advances in artificial intelligence and machine learning a therapist will be able to finish a session with a patient and receive an immediate feedback report about the previous hour. The feedback might include metrics on empathy, the percent of questions vs reflections, competence in specific interventions, among other personalize ratings. This future might also have novice trainees receive immediate real-time in-session feedback about behaviors of interest that need to be adjusted, or for which more training is necessary. For some, this might be a vision of a dystopian future, for others it may represent a way forward in which therapists achieve more refined skills and better patient outcomes.

Lambert, M. J., Whipple, J. L., & Kleinstäuber, M. (2018). Collecting and delivering progress feedback: A meta-analysis of routine outcome monitoring. Psychotherapy, 55(4), 520-537.

Somewhere between 5% and 10% of adult clients in clinical trials of psychotherapy get worse, and the numbers are likely higher in regular clinical practice. In addition, some therapists are more effective than others, so that some therapists have few clients who get worse whereas others consistently have high rates of poor client outcomes. Unfortunately, therapists have a difficult time assessing their client outcomes. Many therapists are overly optimistic about their clients’ outcomes, and clinicians frequently do not identify when clients get worse. One likely reason for this erroneous assessment of client outcomes is that typically psychotherapists do not have quality information in order to make accurate decisions and predictions. Assessing client outcomes on a regular basis throughout treatment is a difficult and complicated endeavour, and one that is beyond the capacity of most people. So, like other professionals (pilots, air traffic controllers, engineers) psychotherapists can improve their predictions and decision-making if they have access to quality information about their clients’ functioning. One source of such information for psychotherapists could be from the use of routine outcome monitoring. Routine outcome monitoring involves assessing client mental health functioning with reliable psychometric scales throughout the course of treatment, and feeding this information back to therapists who can use the data to adjust what they are doing if necessary. The two most commonly used outcome monitoring tools are the Outcome Questionnaire-45 (OQ-45) which is part of the OQ Analyst Feedback System, and the Outcome Rating Scale (ORS) and Session Rating Scale (SRS) which are part of the Partners for Change Outcome Management System (PCOMS). In this meta-analysis, Lambert and colleagues assessed the effect of regular outcome monitoring with the OQ-45 and the ORS to improve client outcomes. In 15 studies with almost 8,500 participants, the OQ-45 outperformed treatment as usual but with a small effect (SMD = .14, 95% CI [.08, .21]). However, the positive effect of using the OQ-45 with feedback was larger for the 31.2% of clients who were not doing well in therapy (SMD = .33, 95% CI [.25, .41]). Among those studies that used the OQ standardized feedback system that provides recommendations to therapists, the effects were even larger (SMD = .49, 95% CI [.25, .73]). Similarly, in nine studies with over 2,000 participants, the effects of using the PCOMS system had a small to moderate positive effects on client outcomes (SMD = .40, 95% CI [.29, .51]).

Practice Implications

The research evidence supports the use of routine outcome monitoring with the OQ-45 or the PCOMS to improve client outcomes. Quality information that is fed back to clinicians can compensate for the limited capacity that any clinician has to accurately detect a client that is worsening in psychotherapy. The information provided to therapists with these feedback systems can highlight potential problems in the client and identify strain in the therapeutic alliance. This information can sensitise therapists to at-risk clients and situations, and encourage therapists to adjust their interventions or interpersonal stances accordingly.

Author email: lambert.michaelphd@gmail.com

Norcross, J. C., & Lambert, M. J. (2018). Psychotherapy relationships that work III. Psychotherapy, 55(4), 303-315.

Relationship factors in psychotherapy are some of the most important predictors of patient outcomes. They outweigh factors like the type of therapy provided in determining whether patients get better after psychotherapy. In this second overview article, Norcross and Lambert provide a review of 17 meta-analyses of relationship factors in psychotherapy that contribute to positive outcomes. Like the review of patient factors also found in this blog and E-Newsletter, this article briefly outlines those evidence-based relationship factors that reliably predict patient outcomes in psychotherapy. The therapeutic relationship refers to how the therapist and patient relate to each other, or their interpersonal behaviors. By contrast, techniques or interventions refer to what is done by the therapist. Practice guidelines typically focus on interventions or therapeutic orientation. As the authors argue, what is missing from treatment guidelines are the person of the therapist and the therapeutic relationship – evidence for which is backed up by 5 decades of research. Even in studies of highly structured manualized psychotherapy for a specific disorder in which efforts were made to reduce the effect of individual therapist, up to 18% of outcomes (a moderate to large effect) could be attributed to the person of the therapist. By contrast somewhere between 0% and 10% of outcomes (a small to moderate effect) is attributable to specific treatment methods. So, which therapeutic relationship factors are reliably related to patient outcomes? These include: the therapeutic alliance in individual therapy (306 studies, g = .57) couple therapy (40 studies, g = .62), and adolescent psychotherapy (43 studies, g = .40), collaboration (53 studies, g = .61) and goal consensus (54 studies, g = .49), cohesion in group therapy (55 studies, g = .56), therapist empathy (82 studies, g = .58), collecting and delivering client feedback or progress monitoring (24 studies, g = .14 to .49), managing countertransference (9 studies, g = .84), and repairing therapeutic alliance ruptures (11 studies, g = .62) among others. Over the next few months, I will be reviewing these meta analyses in more detail to discuss how therapists can use this evidence base to improve their patients’ outcomes.

Practice Implications

The research as a whole indicates that therapists should make the creation and cultivation of the therapeutic relationship a primary goal of therapy. Factors such as managing the therapeutic alliance, repairing alliance ruptures, engaging in ongoing progress monitoring, managing countertransference and others should be used to modify treatments and interpersonal stances in order to maximize outcomes. When seeking out professional development and training, practitioners should focus on evidence-based relationship factors (managing the alliance, judicious self disclosure, managing emotional expression, promoting credibility of the treatment, collecting formal feedback, managing countertransference) in addition to focusing on evidence-based treatments.