Wellness Recovery Action Plan (WRAP) Lacks Independent Verification and is not Evidence Based

WRAP labeled as 'evidence based' by SAMHSA. 

But is it?

When we ask “is a program evidence-based’, at Mental Illness Policy Org we break that question down into three components:

1.  Does it help people with serious mental illness (ex. schizophrenia, bipolar, major depressive disorders) or is it a program designed to improve the mental health of anyone who feels their mental health can be improved. 
2. Does the program improve a meaningful outcome? To be evidence based, we require it to improve a meaningful independent measure such as reducing suicide, homelessness, hospitalization, violence, substance abuse, arrest, incarceration, etc. Self reports of greatier happiness (“improved mental health”) are not sufficient or unique to WRAP
3. What is the quality and independence of the research.

WRAP (Wellness Recovery Action Plan) has been certified by SAMHSA National Registry of Evidence Based Practices and Programs (NREPP) as an evidence-based intervention. This certification encourages states to implement it. But the evidence is not clear that WRAP improves any meaningful measure like decreasing hospital days, decreasing incarceration, decreasing homelessness or that it is for people with serious mental illness. The certification of unproven programs leads states to waste money.

WRAP was invented by and is the property of Mary E. Copeland. WRAP is a workshop run by a paid WRAP-trained facilitator who is a “peer” stable in recovery. (In the mental health industry, the word "peer" refers to someone who has had experience in the mental health system and may or may not believe mental illness exists). There is a manual for the instructor. The workshop is presented to groups of 8-12 individuals 26-55 years old with mental illness over eight weekly two-hour sessions. The purpose of WRAP is to help participants develop a wellness plan and

• Teach participants how to implement the key concepts of recovery (hope, personal responsibility, education, self-advocacy, and support) in their day-to-day lives.
• Help participants organize a list of their wellness tools--activities they can use to help themselves feel better when they are experiencing mental health difficulties and to prevent these difficulties from arising
• Assist each participant in creating an advance directive that guides the involvement of family members or supporters when he or she can no longer take appropriate actions on his or her own behalf
• Help each participant develop an individualized post-crisis plan for use as the mental health difficulty subsides, to promote a return to wellness

SAMHSA says their review found WRAP had a positive impact on
1: Symptoms of mental illness
2: Hopefulness
3: Recovery from mental illness
4: Self-advocacy
5: Physical and mental health

100% of the studies on WRAP used by SAMHSA to certify it were by WRAP’s founder and owner: Mary E. Copeland.

SAMHSA’s Review was extremely faulty

• Review based on a single test: SAMHSA’s certification took place in 2010, and they claimed to have used two studies to prove the intervention is evidence based.
• While eventually there would be two different ‘studies’, they were both based on the exact same test on the exact same people at the exact same time. Colloquially, they were two reports on the exact same single test population.
• One was a preliminary brief study that was published prior to SAMHSA approving the program in 2010.(FN1)
• The second study was more complete but was not published until 2011 and therefore not likely peer reviewed before SAMHSA relied on it (FN2).

• Study had Conflict of Interest:  Mary Ann Copeland, who created the program and two others who work on it were authors on both studies.

• No review of long-term efficacy – The studies SAMHSA relied on were 6 month studies.

• Important outcomes not measured – WRAP does not claim to reduce hospitalization, violence, incarceration, homlesseness, awareness of illness and other important outcomes. So these were not measured.

• SAMHSA certified the program in spite of negative findings
• No statistically significant differences were found between the two groups across time on somatization, obsessive-compulsive, hostility, and psychoticism subscales.
• No statistically significant difference was found between the two groups across time on the subscale for ability to generate routes by which goals may be reached.
• ‘Participation in WRAP may have had negative effects on empowerment

• SAMHSA still reports findings that are not likely true

SAMHSA, relying on the non peer-reviewed study still say

“WRAP participants had a significantly greater reduction in the severity and number of symptoms across time (from baseline to posttest to 6-month follow-up) relative to control group participants, as indicated by scores on the BSI Global Severity Index (p = .023); Positive Symptom Total (p = .027); and subscales measuring interpersonal sensitivity (p = .023), depression (p = .022), anxiety (p = .022), phobic anxiety (p = .034), and paranoid ideation (p = .009). No statistically significant differences were found between the two groups across time on somatization, obsessive-compulsive, hostility, and psychoticism subscales.”

None of those claims remained in the study after it underwent peer review.

Impact 
In the United States, local and regional WRAP programs sponsored by mental health agencies and peer-run centers exist in every State, and over 25 States have integrated statewide WRAP initiatives. (Per SAMHSA)

Spending money on WRAP
In 2002, SAMHSA gave a Community Action Grant to the Minnesota Consumer/Survivor Network to deliver WRAP education and educator training to people in recovery around the state and to engage in a two-year consensus building process to encourage the adoption of WRAP. SAMHSA is also gave $70K to Vermont Psychiatric Survivors who’s director Linda Correy works to spread WRAP and promote it.

Conclusion

WRAP likely makes people who go through it feel better as do most interventions that involve and recognize the suffering of people with mental illness.   But clearly this has not been rigorously evaluated. The desired outcomes may or may not be the most important. The results are self-reported and not independently verified, there is no data showing efficacy beyond six months, no data comparing it to other interventions (say, non-formalized support or horse riding).  There is not information about which mental illnesses it helps more than others.

Other SAMHSA-funded Copeland Initiatives

            SAMHSA contracted with WRAP owner, Mary Ann Copeland to write, so SAMHSA could publish and distribute the following brochures. We were unable to identify any evidence that any of them are evidence based or help improve the lives of people with serious mental illness:  Building Self-Esteem, Making and Keeping Friends, Speaking Out for Yourself, Developing a Recovery and Wellness Lifestyle, Recovering Your Mental Health, Action Planning for Prevention and Recovery, Dealing With the Effects of Trauma, Alternative Therapies.

POTENTIAL LIMITATIONS OF THE TWO STUDIES SAMHSA USED TO CERTIFY WRAP AS “EVIDENCE BASED”

Following are details on the two studies SAMHSA used to certify WRAP as evidence based 

SAMHSA certified WRAP (Wellness Recovery Action Plan) as an evidence-based intervention.

Mental Illness Policy Org did not call on any scientists to review the studies. However, following are concerns we have about the two studies that warrant closer scrutiny.

1. Initial Outcomes of a Mental Illness Self-Management Program Based on Wellness Recovery Action Planning 

This is the only study that had been published prior to SAMHSA certifying WRAP as being evidenced. Three of the authors of the study are associated with the developing and distribution of the program and therefore may lack the independence to accurately review the efficacy. The study found

“Those who attended the groups “revealed significant improvement in self-reported symptoms, recovery, hopefulness, self-advocacy, and physical health; empowerment decreased significantly and no significant changes were observed in social support.”

This could have been a placebo effect. The study did not use potentially more valuable measures incarceration rates, hospitalization rates, homeless rates, etc. It only looked for results one month after intervention. That was clearly too short a period for SAMHSA to rely on. The only measurements of progress were self-measurements by the patients that were not independently verified.
The measuring tools may have been inadequate.

• Brief Symptom Inventory
• Recovery (as defined by Illinois Office of Mental Health)
• Hopefulness as defined in the Journal of Personality)
• Self-advocacy and Empowerment (As co-developed by anti treatment advocate Judi Chambelin
• Social support
• Perceived physical health.

The study claimed

“A statistically significant decrease was found in global symptom severity. Scores on several symptom subscales—psychoticism, depression, phobic anxiety, obsessive-compulsive, interpersonal sensitivity, paranoid ideation, and general anxiety— also decreased significantly, indicating improvement.”

The sentence above is included on the SAMHSA site. But when the second study was published (a more detailed description of the same population as the first study (that underwent extensive peer review) that sentence does not appear. (See below).

2. Results of a Randomized Controlled Trial of Mental Illness Self-management Using Wellness Recovery Action Planning

This study had been submitted to SAMHSA in pre-publication (pre-review?) form and used by SAMHSA to evaluate WRAP in CA 2009. However it was not peer reviewed and published until June 2012. The delay between 2009 and 2011 may suggest reviewers had problem with the version presented to SAMHSA. Following are statements in this study that raise concern or deserve scrutiny.

• Three of the 11 authors had conflicts of interest in that they were connected with developing and distributing WRAP.
• Twice as many who received WRAP were lost to follow up compared to the control group: Of the 276 experimental subjects, 233 (84%) received the intervention and 43 (16%) did not. Eleven control subjects and 25 intervention subjects were lost to follow-up with reasons including death or ill health, moving away from the area, and formal withdrawal
• The study contains no data after 6 months
• The WRAP program served people with diverse diagnosis but did the study did not report results on efficacy by diagnosis (ex. 21% had SZ 38% were BP, etc.)
• As time went on the WRAP made less of a difference in symptom severity:  “On the other hand, the experimental vs control differences in symptom severity were larger between T(ime)1  and T2 and seemed to attenuate over the long term, even though WRAP participants were still doing better at T3 in the multivariate analysis.”
• The control group also improved: “Also noted in these results was improvement among control-condition subjects on all 3 outcomes.”
• Changes in hopefulness were modest: “Another finding inviting further explication is that regarding participants’ degree of hopefulness given that ob- served changes in raw scores were relatively modest.”
• “There were no differences by study condition in subjects’ self-perceived ability to construct successful plans of action, as measured by the ‘‘pathways’’ subscale.”
• “Regarding QOL (Quality of Life), again changes in raw scores were somewhat modest.”

Study limitations the researchers reported did not prevent SAMHSA from relying on results. Researchers reported:
“There are a number of study limitations that should be considered when interpreting these results.

• The first major caveat to our findings is that the study’s subjects were not drawn from a national probability sample of individuals with severe and persistent mental illness, which limits the generalizability of our results.
• A second caveat is the fact that all subjects came from a single Midwestern state, preventing an assessment of potential US regional variations in WRAP implementation and outcomes.
• A third caveat concerns the design of the study using a waitlist control condition. Use of an attention-control placebo would have allowed us to assess whether 8 weeks of peer interaction alone, and not the specific features of the WRAP intervention, caused the observed outcomes.
• A fourth caveat is that the study relied on participant self-report data that were uncorroborated by clinicians or objective observers such as research staff.” “Future studies using external raters and attention-control placebo interventions will offer a more rigorous evaluation of WRAP’s efficacy.
• A fifth caveat is that fidelity assessment was limited to WRAP facilitator self-report, while the additional use of direct observation to verify the validity of self-reports would have added credibility to fidelity assessment.
• Another potential confound is the high level of study subjects’ participation in peer-led programs and support groups, which may have exposed control-condition subjects to some of the same active ingredients as those contained in the WRAP intervention. As a result, the study may have underestimated the effects of WRAP relative to its impact in communities with low levels of peer support, as is typical in many areas of the United States.
• Finally, a longer time period of data collection might have revealed different findings than those attained at the end of the 8 months tracked in this study. All these limitations suggest that caution should be applied to interpretations from study results.”
• SAMHSA partially Funded the study: Cooperative Agreement H133B050003 and H133B100028.

NOTES ON ONE MORE STUDY

EFFECT OF WELLNESS RECOVERY ACTION PLAN PARTICIPATION ON PSYCHIATRIC SYMPTOMS, SENSE OF HOPE AND RECOVERY

Psychiatric Rehabilitation Journal, 2011, Vol 34., # 3, 214-222 Someone sent us the above study and suggested it shows that WRAP is a an evidence-based program. We disagree.When we ask “is a program evidence-based’, at Mental Illness Policy Org we break that question down into three components:1. Does it help people with serious mental illness (ex. schizophrenia, bipolar, major depressive disorders) or is it a program designed to improve the mental health of anyone who feels their mental health can be improved. 2. Does the program improve a meaningful outcome? To be evidence based, we require it to improve a meaningful independent measure such as reducing suicide, homelessness, hospitalization, violence, substance abuse, arrest, incarceration, etc. Self reports of greatier happiness (“improved mental health”) are not sufficient or unique to WRAP3. What is the quality or independence of the research?

Our findings

`1. We did a quick review of this WRAP study and were impressed that 40% had schizophrenia or a related disorder. 

2. We did not find that it improved any meaningful outcome 

The authors main finding was that participants self-reported they felt more hopeful. This is a common finding of any activity that adds the word ‘therapy’ to it in order to make it eligible for government funding. Among “therapies” that make people more hopeful are Dog Therapy, Horse Therapy, Art Therapy, Group Therapy, Shopping Therapy, Yoga Therapy, Dance Therapy and it’s subset, Tango Therapy (Dawn.com 2013). There’s Tai-Chi Therapy (Karpova 2011) and (Abbott and Lavretsky 2013), Surfing Therapy (Moore 2011, Wade n.d.) Meatloaf and Mashed Potato Therapy (United Press International, Inc. 2011) Cleaning Therapy, (BBC 2008) Remarriage Therapy (Daily Mail, UK 2011), Induced After Death Communication Therapy (L, Hogan and Moody 2005), Paintball Therapy (Tomcat 2011) and every kind of talk, meditation, massage, acupressure counseling and introspection. The authors did not measure meaningful outcomes, only self-reported self-perceptions. As the authors admitted, “We have a lack of formal and informal service usage information on the participants”. The authors reported no results on suicide, homelessness, arrest, violence, incarceration, substance abuse, or on any other more meaningful measure.

3. Poor Study design.

Our quick review found the same issue affect this study as appear to affect all WRAP studies. The studies only show, at best, that those who take the course self-report that they feel better, have more hope, feel empowered, etc. 

There is no evidence that WRAP works, because consumers are involved in the delivery of it. To quote the authors: “The groups were facilitated by one staff person and one peer worker….Each group incorporat(ed) both a peer WRAP educator and a community mental health center (CMHC) psychosocial rehabilitation leader. Peer educators who were not employed at the CMHC were compenstated $250 for the planning, preparation and teaching of the group sessions. The CHMC co-facilitators were not compensated outside their agency salary.” Results might have been as good or better, had WRAP been delivered by professionals, bartenders, parents or others. There is no evidence that it was half peer taught affected anything.

All subjects were already receiving services at a CHMC. All were voluntary (i.e, therefore had insight into illness). It seems like they may have had a history of hospitalization and/or inability to work, but no evidence they had history of homelessness, suicide attempts, dangerous behavior, arrest, or incarceration.

Only 33% (56) of the 170 people identified as eligible for being in the control group were included in the control group. 

The results of the test were self-reported and not verified. It is merely people saying they do or don’t hope more. (I am growing as a person” “I have a reason to get up in morning” “I have trusted people I can turn to for help”)

The researchers do not appear to have been blinded. Those in the experimental group reported their responses in face to face interviews while those in the control group mailed in their responses. As the researcher put it “The different data collection methods (face to fave, vs. mail using self report measures) used in our study might have caused a response bias (including social desirability bias and inverviewer bias) for the significant improvement of outcomes for the experimental group, while improvement was not observed for the control group.

Conclusion: The study shows those who participated felt good about it. It does not show that they achieved higher recovery, less hospitalization, fewer days homeless, shorter hospitalization, less arrest, less incarceration, more time employed, etc. Given limited dollars, and the availabilitly of programs that do improve important measurable outcomes (ACT, ICM, AOT, etc), we suggest spending the funds on those programs.

FN 1: Initial outcomes of a mental illness self-management program based on Wellness Recovery Action Planning Cook, J. A., Copeland, M. E., Hamilton, M. M., Jonikas, J. A., Razzano, L. A., Floyd, C. B., et al. (2009).. Psychiatric Services, 60(2), 246-249 available at  http://www.illinoismentalhealthcollaborative.com/news/WRAP_Initial_Research_Findings.pdf

FN 2: Results of a Randomized Controlled Trial of Mental Illness Self-management Using Wellness Recovery Action Planning Judith A. Cook, Mary Ellen Copeland, Jessica A. Jonikas, Marie M. Hamilton, Lisa A. Razzano, Dennis D. Grey, Carol B. Floyd, Walter B. Hudson, Rachel T. Macfarlane, Tina M. Carter1, and Sherry Boyd available at http://www.mentalhealthrecovery.org/wrap/documents/SchizophrBull-2011-Cook-schbul_sbr012.pdf

Wellness Recovery Action Plan ® and WRAP® are registered trademarks.