Blog

Reese, R. J., Duncan, B. L., Bohanske, R. T., Owen, J. J., & Minami, T. (2014). Benchmarking outcomes in a public behavioral health setting: Feedback as a quality improvement strategy. Journal of Consulting and Clinical Psychology, 82(4), 731-742.

Psychotherapy has demonstrated its efficacy in randomized controlled trials. But do these findings in highly controlled studies translate to everyday practice in publicly funded agencies that treat low income clients? Previous research in the US showed that outcomes of treatment-as-usual in public behavioural health agencies are generally not positive, so that only 20 to 35% of clients reliably improved. One approach to improving outcomes is to transport specific evidence-based treatments into practice settings. For example, research on applying CBT for panic and depression in a publicly funded agency resulted in similar outcomes to those achieved in randomized controlled trials. However, an alternative strategy of improving outcomes is to use continuous outcome monitoring, which involves repeated (weekly) measurement of client outcomes with reliable scales, and feedback to therapists on the client’s status relative to previous sessions and relative to other similar clients. Research has demonstrated that this strategy improves client outcomes and reduces the number of clients who deteriorate. In this study, Reese and colleagues examined the outcomes of a large public behavioural health service in the U.S. that treats low-income individuals. The service implemented repeated outcome monitoring of clients with feedback to therapists. Over 5,000 clients mainly with depression, mood, and anxiety disorders were treated by 84 therapists who were licensed at the masters degree or higher. The clients completed the Outcome Rating Scale (a measure of symptom outcome) prior to each session, and the Session Rating Scale (a measure of the therapeutic alliance) after each session. Therapists received two days of training on how to use these measures and on the continuous feedback they were provided in order to improve their treatment of clients and their outcomes. Outcomes from this public behavioural health service were compared to previous large studies in publicly funded settings that implemented specific evidence-based treatments. The findings were similar, with about 42% showing reliable pre- to post-treatment improvement. The results of implementing continuous outcome monitoring with feedback for depressive symptoms were also large and positive (d = 1.34). These effects were similar to benchmarks established in randomized controlled trials of specific psychotherapies.

Practice Implications

Continuous outcome feedback enables therapists to identify clients who are not benefiting
from a given treatment, so that clinicians may collaboratively design different interventions or change their interpersonal stances. The inclusion of outcome monitoring and feedback in this publicly funded psychotherapy system, resulted in outcomes that were: better than what is often seen in such public service settings, equivalent to those public systems that implemented specific evidence based treatments, and similar to those reported in highly controlled randomized trials. The authors concluded that adding routine outcome monitoring and feedback is a viable alternative to transporting specific evidence based treatments to publicly funded psychotherapy programs. The measures used in this study are available free for individuals to use at: betteroutcomesnow.com.

Author email: jeff.reese@uky.edu

Boswell, J. F., Constantino, M. J., Oswald, J. M., Bugatti, M., Goodwin, B., & Yucel, R. (2018). Mental health care consumers’ relative valuing of clinician performance information. Journal of Consulting and Clinical Psychology, 86(4), 301-308.

Research has shown that some therapists are more effective than others both in terms of their overall effectiveness and in terms of their effectiveness with specific patient problems. Further, despite advances in medicine on this topic, there is little or no information provided to patients about a therapist’s track record on overall effectiveness. In any case, little is known about what patients value in psychotherapists and how much they are willing to give up in order to get what they value. For example, do patients prefer therapists who are highly effective for most problems, and would they be willing to tolerate a poorer therapeutic relationship in order to work with such a highly effective therapist? In this study, Boswell and colleagues employed a relative valuing procedure often used in economics to assess the relative value to patients of different therapist characteristics and performance. Patients were asked how much they were willing to give up on one therapist characteristic (therapist’s overall effectiveness with clients [i.e., overall track record]) in order to receive more of some other characteristic (therapist specific effectiveness in a problem domain, a better therapeutic alliance, lower cost of therapy). The study included 403 patients treated in mental health clinics in the U.S. Patient characteristics were typical of those seen in such clinics – predominantly they had problems with depression or anxiety, were 41 years old on average, mostly women (68.5%), and receiving individual psychotherapy (89.3%). In general, patients highly valued a therapist with a track record of general overall effectiveness. However, patients were willing to give up more of their therapists overall effectiveness if the therapist had a track record of successfully treating their specific problem (e.g., therapist A has lower general efficacy but has demonstrated greater specific efficacy for depression). Patients were also willing to sacrifice therapist general effectiveness in order to pay less for therapy (vs paying a higher fee for a more effective therapist), and in order to work with a provider with whom they would have a better therapeutic alliance (vs a lower alliance with a more generally effective therapist). Surprisingly, patients placed a lower value on factors like therapist gender and race. Younger patients put greater value on therapist performance data (i.e., their track record data), suggesting a generational effect in which younger clients tend to prefer to make decisions based on available data.

Practice Implications

Patients were willing to give up some therapist general effectiveness in order to work with someone who has a track record of being effective for their specific problem, who costs less, and with whom they could have a better therapeutic alliance. Fortunately, therapist general efficacy and domain specific efficacy tend to be highly correlated, and so patients may not have to choose between these. The findings also suggest that patients may be willing to see a therapist who is less generally effective if it meant they could have a good relational experience with the therapist. Research indicates that therapists are able to improve their outcomes and therapeutic alliances with additional training and deliberate practice.

Schermuly-Haupt, M. L., Linden, M., & Rush, A. J. (2018). Unwanted events and side effects in cognitive behavior therapy. Cognitive Therapy and Research, 42(3), 219-229.

Unwanted events are negative consequences for clients that may or may not be related to treatment (i.e., events outside of therapy or inside of therapy that may negatively affect clients).  These might include: occupational problems, stigmatization, strains in personal relationships, changes in the social network, patients feeling overwhelmed, undermined self-efficacy, deterioration of symptoms, emergence of new symptoms, suicidality, and others. Side effects refer to negative reactions in clients directly related to appropriately delivered therapy. Research estimates that between 5% and 20% of patients report side effects of psychotherapy. One could argue that side effects may be inevitable even in well-delivered therapy, and therapists who are aware of the potential for side effects may be better equipped to help clients to manage. In this study, Schermuly-Haupt, interviewed 100 psychotherapists who provided CBT in outpatient clinics in Germany about side effects among their clients. All therapists were supervised as part of their work and so the authors assumed the therapy was appropriately delivered. Therapists had on average 5 years of experience and were trained to provide CBT. The interview asked therapists about their most recent treatment case in which the client attended at least 10 sessions. Clients typically had major depression, an anxiety disorder, or a personality disorder, and had attended 28 sessions of therapy on average. During the interview, therapists identified if an unwanted event occurred for a client from a standardized list, and then rated the duration and severity of the effects. They also rated the degree to which the unwanted event was directly related to therapy (i.e., a side effect). Prior to the interview, only 26% of therapists reported their client experienced side effects. However, the interview process found that almost all clients experienced an unwanted event (98%) that may or may have been related to therapy, and 43% experienced at least one side effect that was at least somewhat related to treatment. The most frequent side effects were: “negative wellbeing/distress” (27% of clients), “deterioration of symptoms” (9% of clients) and “strains in family relations” (6% of clients). Of the therapists, 46% rated the side effects as at least moderately severe, and 8.8% of side effects were rated as persistent (lasting more than a month).

Practice Implications

Unwanted events outside of therapy are very common among our clients, but so are side effects from appropriately delivered treatment. Psychotherapy is not always harmless, and it may be best to acknowledge and prepare both clients and therapists for side effects. These may represent ruptures in the alliance that can be managed through alliance-focused therapy, for example. That is, side effects may be caused a mismatch between the goals of a therapist and client, or a disagreement on how to proceed in therapy given what a client needs at the time. Goals and tasks of therapy may need to be renegotiated following the experience of a side effect.

Pybis, J., Saxon, D., Hill, A., & Barkham, M. (2017). The comparative effectiveness and efficiency of cognitive behaviour therapy and generic counselling in the treatment of depression: Evidence from the 2nd UK National Audit of psychological therapies. BMC Psychiatry, 17, 215. DOI 10.1186/s12888-017-1370-7

Over a decade ago the United Kingdom (UK) invested large sums of public dollars to fund the Increasing Access to Psychotherapy (IAPT) program. In IAPT, most patients receive cognitive behavioral therapy (CBT) as first-line treatment for depression or anxiety, and may receive generic counseling as second line treatment. One of the admirable aspects of IAPT is that the program consistently assesses outcomes, makes its data available for analyses, and publishes yearly reports on their outcomes. In this very large study, Pybis and colleagues assess whether CBT and generic counseling have different outcomes for patients with depression or anxiety. Over 33,000 patients who received treatment at one of 103 sites were in the study. Most patients (about 23,000) receiving CBT, and the others (about 10,000) receiving generic counseling. Two-thirds of the patients were female, most (84%) were white British, and the mean age was 41 (SD = 13.86). CBT focused on changing negative thoughts and behaviors in order to improve depressive symptoms. Generic counselling was harder to define, though the authors described these therapists as practicing in an integrative manner by bringing skills from training in different forms of psychotherapy. Generic counseling therapists did not focus on giving advice or opinions, but rather on helping clients understand themselves better. Pre- to post-treatment effect sizes for CBT (0.94 [0.92, 0.95]) and generic counseling (0.95 [0.92, 0.98]) were equivalent for depression outcomes. In CBT 50.4% of patients reliably improved, whereas 49.6% reliably improved if they received generic counseling. The average number of sessions attended by patients in the two treatments (CBT = 8.9 [6.34]; counseling = 7.5 [5.54]) were also equivalent. However, there were significant site effects. That is, a moderate and significant amount of patient outcomes (15%) could be accounted for by the site at which they received treatment (i.e., some sites or clinics had better outcomes than others).

Practice Implications

Generic counseling as provided in the IAPT in the UK was as effective as structured CBT for reducing symptoms of depression. However, almost half of patients did not improve in either treatment. Generic counseling was likely a label used to describe integrative psychotherapy that followed principles from a variety of psychotherapies that were based on psychological principles. There were much larger site/clinic effects than treatment modality effects, so that clients in some clinics had better than clients who received treatment in other clinics. This is consistent with research on therapist effects that show that some therapists are more effective than others, regardless of their orientation. This research suggests that training therapists to be more effective by improving their facilitative interpersonal skills may yield better outcomes for clients.

Leichsenring, F. Abbass, A., Hilsenroth, M.J., Leweke, F., Luyten, P., ….Steinert, C. (2016). Bias in research: Risk factors for non-replicability in psychotherapy and pharmacotherapy research. Psychological Medicine, doi:10.1017/S003329171600324X.

An important feature of research is that it should be replicable. That is, another researcher should be able to obtain the same finding as the original study as a pre-requisite for the validity of the conclusions. A recent estimate for cognitive and social psychology research is that only about 36% to 47% of studies are successfully replicated. Another study showed similar low replicability of psychotherapy and pharmacotherapy research. Results that are neither replicable nor valid can lead to improper treatment recommendations. Leichsenring and colleagues review several research biases that affect the replicability of findings in psychotherapy and pharmacotherapy research, and they discuss how to limit these biases. Psychotherapy trials often involve an established treatment approach that is pit against a comparison treatment in a head to head contest. Below I list some of the biases detailed by Leichsenring and colleagues that may affect the validity of psychotherapy trials. First, in psychotherapy trials a large proportion of the differences in outcomes between a treatment and a comparison may be due to the researcher’s allegiance to a particular therapy modality. This may be expressed unconsciously by selecting outcome measures that are more sensitive to the effects of one type of treatment versus another. For example the Beck Depression Inventory (BDI) is particularly sensitive to changes in cognitions, whereas the Hamilton Depression Rating Scale (HDRS) is particularly sensitive to physiological side effects related to antidepressant medications. One way to deal with researcher allegiance effects is to include researchers and therapists who have an allegiance to both of the treatments that are under study. Second, the integrity of the comparison treatment may be impaired. That is the comparison treatment may not be carried out exactly as originally intended. This could occur in pharmacological trials in which doses do not match clinical practice, or in psychotherapy trials in which therapists in the comparison treatment may be told to ignore key symptoms. Properly training and supervising therapists and not constraining them by the study protocol is important to avoid this type of bias. Third, some studies make a lot of noise about small effects that are statistically significant. When two bona-fide psychotherapies are compared the differences tend to be small – this is a common finding. Small differences, even if statistically significant, often turn out to be random, unimportant, and of little clinical significance. Concurrent with this problem is that sometimes researchers will use multiple outcome measures, find significant differences only with some, and report these as meaningful. This refers to selectively emphasizing a small number of findings among a larger number of analyses, which are likely due to chance variation and therefore not replicable.

Practice Implications

What should a clinician do when reading a comparative outcome study of psychotherapy? There are some technical red flags for research bias that require specialized knowledge (e.g., small sample sizes and their effect on reliability, over-interpreting statistical significance in the context of small effects, and non-registration of a trial). But there are a few less technical things to look for. First, I suggest that clinicians focus primarily on meta-analyses and not on single research studies. Although not perfect, meta-analyses review a whole body of literature, and are more likely to give a reliable estimate of the state of the research in a particular area. Second, clinicians should ask some important questions about the particular study: (a) are the results unusual (i.e., when comparing 2 bona-fide treatments, is one “significantly” better; or are the results spectacular); (b) does the research team represent only one treatment orientation; and (c) do the researchers reduce the integrity of the comparison treatment in some way (e.g., by not training and supervising therapists properly, by unreasonably limiting what therapists can do)?  

Waldinger, R.J. & Schulz, M.S. (2016). The long reach of nurturing family environments: Links with midlife emotion-regulatory styles and late-life security in intimate relationships. Psychological Science. DOI: 10.1177/0956797616661556.

Although, not a psychotherapy study, this research has important implications for psychological treatment of adults, including older adults. This research, drawn from the original Grant study, is extraordinary because the sample is from a 78-year long study of 81 men. The original cohort of over 200 men were first assessed as adolescents and young adults between 1939 and 1942. At that time, the original authors conducted intensive interviews of the adolescents` family experiences and current life situations. These men were re-interviewed in mid-life in the 1960s (aged between 45 and 50 years), which included interviews and assessments of challenges in relationships, work functioning, and physical health. Waldinger and Schulz recently re-interviewed these men and their current partner in late-life (aged between 75 and 85 years), with interviews focusing on their current partner relationship. Raters reviewed audio recordings and notes from all the interviews and coded for: (a) quality of family environment in childhood (distant, hostile vs cohesive, warm) - taken from the first interview; (b) style of regulating emotions (suppressive, maladaptive vs engaged, adaptive) – taken from the midlife interview; and (c) security of attachment with their current partner (secure, comforting vs insecure, anxious) – taken from the late-life interview. The authors found that more nurturing early family environments were significantly linked with late-life attachment security with a partner (r = .23, 95% CI = .01, .45), and early family environment was significantly related to midlife adaptive emotion regulation strategies (r = .29, 95% CI = .06, .50). Also, adaptive emotion regulation strategies in midlife were significantly correlated with greater late-life attachment security (r = .23, 95% CI = .05, .51). These are medium-sized correlations, but they are remarkable because they represent associations between variables that were assessed decades apart. Through a statistical mediation analysis, the authors also reported that adaptiveness of emotion-regulation strategies partially explained why positive childhood family environments may lead to late-life attachment security (accounting for 6% of the variance).

Practice Implications

This compelling study adds to the argument that early family environment shapes the way adults regulate their emotions, which in turn affects how they experience relationships in old age. More securely attached adults were better able to meet two challenges associated with aging: accepting vulnerability in depending on a partner, and accepting the responsibility of being depended upon by that partner. The early family environment indeed has a long reach. Psychotherapy directed at reducing the effects of childhood adversity takes on a heightened meaning in the context of these findings. Treatment for adults who struggle with the consequences of non-nurturing early environments should include improving emotion regulation strategies.