MA Wraparound Fidelity Assessment System:

MA Wraparound Fidelity Assessment System: Promoting Positive Outcomes through Fidelity Monitoring September 16, 2016 With special thanks to the Wraparound Evaluation Team at University of Washington: Eric Bruns, April Sather, and Alyssa Hook 1 Agenda Agenda: 1.Quick Recap What is Wraparound Fidelity? Why spend valuable time measuring fidelity? New this year 2.WFAS TOM/TOM 2.0 and WFI/WFI EZ Statistics What are the tools? How is statewide data collected? How do our FY 2016 scores compare to that of other states? How do our FY 2016 scores compare to last years results? How do we make practical sense of the scores? Looking ahead to FY17

1. TOM 2.0 2. WFI EZ 2 What is Wraparound Fidelity? Fidelity is the degree to which a program is implemented as intended by its developers. Wraparound fidelity, as measured by the MA Wraparound Fidelity Assessment System, is defined as the degree to which intensive care coordination teams adhere to the principles of quality wraparound and carry out the basic activities of facilitating a wraparound process. 3 Why Measure Fidelity? Research has linked high fidelity scores with better outcomes for youth and families: Improved functioning in school and community

Safe, stable, home-like environment Improved resilience and quality of life Improved mental health outcomes. It also provides a vehicle for comparing our experiences with peers who are promoting and implementing Wraparound here and in other states. Walter UM and Petr CG. 2011. Best Practices in Wraparound: A Multidimensional View of the Evidence. Social Work 56(1): 73-80 Bruns EJ, Suter JC, Force MM and Burchard JD. 2005 Adherence to wraparound principles and association with outcomes. Journal of Child and Family Studies 14:521-534 4 New This Year Majority of CSAs scored at or above the national mean in FY 2015, suggesting scores are experiencing a ceiling effect Pilot of two new tools the WFI EZ and TOM 2.0 to evaluate whether more information could be gathered that was useful in measuring Fidelity. Question content for both tools is no longer correlated to Wraparound principles or by phase but is driven by five key elements (WFI EZ) plus two additional elements for the TOM 2.0.

Effective Teamwork Determined by Families Based on Priority Needs Use of Natural and Community Supports Outcomes-Based Process Attendance (TOM 2.0) Facilitation skills (TOM 2.0) 5 New This Year (continued) 1. WFI EZ Reduced to 25 process-based questions; added four satisfaction and nine outcomes-based questions Greater consistency in scores and higher response rates due to ease of the survey. Survey has three options for administration electronic, paper (submitted via mail), and interview via phone 2. TOM 2.0 Revised to be more streamlined, easier to administer, and more practice-oriented

Reduction in number of indicators from 71 (TOM) to 40 (TOM 2.0) Companion tool to the WFI EZ 6 New This Year (continued) CSA WFI EZ Pilot Selection: CSAs with the highest and lowest five-year average scores 18 CSAs selected to participate CSA TOM 2.0 Pilot Selection: One CSA piloting the WFI EZ per region, selected at random 14 CSAs completed the WFI-4 and TOM 13 CSAs completed the WFI-EZ and TOM 5 CSAs completed the WFI-EZ and TOM 2.0 7 WFI-4 and TOM Total Scores TOM NM: 87 WFI-4 NM: 81 8 Team Observation Measure

(TOM) 9 What is the MA TOM? Team Observation Measure (MA TOM) Supervisors observe care planning team meetings to assess adherence to standards of high-quality wraparound Tool consists of 20 items, two items linked to each of the 10 principles of Wraparound Each item consists of three to five indicators of highquality wraparound practice as expressed during a care planning team meeting. July 1, 2015 through June 30, 2016 data collection period Total of 571 assessments completed and entered into Wraparound Online Data Entry and Reporting System 10 TOM Total Fidelity Scores Total Score FVC

TB NS COL CB CC INDIV SB PER OB MA 2011 85% 94% 85% 51%

92% 91% 93% 86% 90% 92% 78% MA 2012 87% 97% 84% 51% 93% 93%

95% 90% 93% 93% 85% MA 2013 88% 94% 83% 52% 92% 92% 93% 89%

92% 92% 86% MA 2014 90% 99% 86% 58% 96% 96% 97% 95% 95% 96%

91% MA 2015 90% 95% 84% 59% 93% 94% 94% 92% 93% 94% 89% MA 2016

91% 98% 87% 60% 96% 98% 98% 96% 98% 97% 93% National Avg. 87% 95%

88% 65% 87% 93% 93% 89% 89% 93% 80% 2012-2013 Change -- -- --

-- -- -- -- -- 2013-2014 Change -- -- -- --

-- -- -- -- 2014-2015 Change -- -- -- -- -- --

-- -- -- -- -- 2015-2016 Change -- -- -- -- -- -- -- --

-- -- -- Key: Family Voice and Choice (FVC) Team-Based (TB) Natural Supports (NS) Collaboration (Col) Community-Based (CB) Culturally Competent (CC) Individualized (Indiv) Strengths-Based (SB) Persistence (Per) Outcome-Based (OB) 11 TOM: Areas of Strength and Areas for Improvement Large Increase in Strengths-Based Team utilizes familys perspectives to identify functional strengths Team understands how strengths contribute to team mission and goals.

Four-point increase in Community-Based, Culturally Competent, and Individualized Increase of Natural Supports present at team meetings Natural Support score is below the National Mean Lowest item across all six years that teams continue to struggle with Youth is present at the team meetings less than half of the time (43 percent) 12 Team Observation Measure, Version 2 (TOM 2.0) 13 What is the MA Tom 2.0 ? Team Observation Measure 2 (MA TOM) Supervisors observe care planning team meetings to assess adherence to standards of high-quality wraparound

Tool consists of 41 indicator across eight subscales Six subscales are dedicated to the Key Elements, one evaluates meeting attendance, and one assesses facilitation skills. July 1, 2015 through June 30, 2016 data collection period Total of 154 assessments completed and entered into Wraparound Online Data Entry and Reporting System. 14 TOM 2.0: Scores by Subscale 15 TOM 2.0: Areas of strengths and areas for improvement Strong scores in Determined by Families Parent/caregiver(s) are given an opportunity to share their opinions and ideas. Strong scores in Driven by Strengths

Strong facilitation skills observed in practice Facilitators are dynamically engaged in team meetings. Planning for Transition Discuss how teams will know when needs have been sufficiently met to warrant a transition out of Wraparound services 16 TOM 2.0: Areas of strengths and areas for improvement (continued) Low scores in Full Meeting Attendance The majority of team meetings did not have a natural support present, and when appropriate, the youth was present at only approximately half of the meetings. Lack of progress monitoring in meetings Approximately 15 percent of observed meetings reported teams did not review how close they were to achieving the family vision or team mission. 17 Wraparound Fidelity Index Version 4 (WFI -4)

18 What is the MA WFI-4? Wraparound Fidelity Index, Version 4 (MA WFI-4) Intended to assess both conformance to the Wraparound practice model and adherence to the principles of Wraparound in service delivery. Brief, confidential interviews completed via telephone or with caregivers and a Demographic Form. Tool consists of 40 items; four items are linked to each of the 10 principles of Wraparound. 19 How was MA WFI-4 Data Collected? MA WFI-4: Conducting the Interviews Collaborative Quality Improvements (CQI), a mental health research and evaluation organization, implements the MA WFI-4. CQI trains interviewers (primarily parents of youth with

SED) to conduct the interviews and provides ongoing supervision to interviewers to ensure inter-rater reliability. CQI currently has four interviewers and capacity for three languages: English, Spanish, and Haitian Creole. The goal is to complete 20 interviews with caregivers of youth enrolled at each of the CSAs. CQI completed 285 interviews during FY 2016. 20 How was MA WFI-4 Data Collected? MA WFI-4: Collecting the Data CSA Staff Responsibilities: Inform caregivers of the interview and evaluation process Seek consent from all eligible caregivers, who should have signed a consent indicating whether they chose to participate or not Make sure a call information sheet was completed for each caregiver Fax signed consents along with the call information sheets to Melissa Goodman/CQI Information from the call information sheet was entered into a call contact database which provided interviewers with an updated listing of those caregivers who were eligible to be interviewed. Eligibility was defined as anyone (with an enrolled child under the age of 18) enrolled in ICC between January 1 and December 31, 2015. Caregivers were

eligible to be interviewed if they had been enrolled in ICC for three or more months. 21 How was MA WFI-4 Data Collected? MA WFI-4: Collecting the Data CQI Tasks: Review call information sheets for any missing or inaccurate information and follow up with CSA Enter call contact data into database Contact caregivers who were eligible to participate and schedule interview time Conduct phone interview and complete WFI scoring Enter completed interview data (scores) into WrapTrack Routinely send reports to MBHP: (# of interviews completed at each CSA, # of consents received from each CSA, total # of attempted and refused calls for the week, total # of calls made and interviews completed since the project began) Interviews averaged 30 to 45 minutes.

Caregivers received a $15 check for their participation. Addresses are confirmed with caregiver before completing the call. 22 How was MA WFI-4 Data Collected? MA WFI-4: Challenges Consent Process 1. Incomplete, inaccurate, ineligible consents 2. Varying levels of awareness of caregivers (getting better) both of the evaluation and description/terminology for ICC Difficulty Reaching Caregivers 1. Dont return messages 2. Frequent phone number changes 3. Several repeated no-shows with caregivers 23 WFI-4 Fidelity by Wraparound Phase Total

Phase Mean Overall Engagement Planning Implementation Transition MA 2011 77 85 82 77 64 MA 2012 79

88 84 79 67 MA 2013 78 86 83 77 66 MA 2014 82 90 86

83 71 MA 2015 83 91 88 83 68 MA 2016 82 91 86 85 63

National Mean (CG 81 82 81 85 73 only) 24 WFI-4: Areas of Strength and Areas for Improvement Significant increase in Natural and Community Supports from 2015 Strong scores in Collaboration An encouraging jump in Team-Based item 1.4 Did you select the people on your team? from 1.46 to 1.65 Historically highest scores for Strengths-Based

Significant decrease in Community-Based Significant decrease in Cultural Competence Significant decrease in Individualized implementation Significant decrease in Team-Based planning and implementation 25 WFI-4 Total and Principle Scores Total Score FVC TB NS COL CB CC INDIV

SB PER OB MA 2011 77% 85% 82% 52% 89% 74% 93% 73% 79% 78%

65% MA 2012 79% 89% 86% 55% 91% 72% 94% 75% 81% 82% 66% MA 2013

78% 87% 84% 52% 88% 73% 93% 71% 81% 78% 70% MA 2014 82% 91%

85% 64% 91% 74% 95% 80% 86% 83% 75% MA 2015 83% 92% 85% 61%

92% 72% 95% 85% 86% 84% 74% MA 2016 82% 92% 84% 65% 94% 65%

93% 82% 87% 84% 71% National Avg. 81% 90% 75% 66% 90% 78% 94% 71%

85% 85% 72% 2012-2013 Change -- -- -- -- -- -- --

2013-2014 Change -- --

2014-2015 Change -- -- -- -- -- -- -- -- -- 2015-2016 Change*

-- -- -- -- -- -- -- -- Key: Family Voice and Choice (FVC) Team-Based (TB) Natural Supports (NS) Collaboration (Col) Community-Based (CB)

Culturally Competent (CC) Individualized (Indiv) Strengths-Based (SB) Persistence (Per) Outcome-Based (OB) 26 Length of Time in Services The length of time in services is defined by the amount of days between the enrollment date and when the WFI-4 was administered. Note: For 2014 there is a total of 629 forms analyzed, due to incomplete data. Some WFI-4 forms did not have an interview date, or the enrollment date and interview date were entered as the same date. All forms for 2015 and 2016 had complete dates. Data from 2016 only include the 14 sites using the WFI-4. 27 Fidelity by Length of Time in Services Fidelity was analyzed based on length of time in service when the interview was conducted, comparing two groups: 1) Fidelity scores for youth enrolled for 0120 days when the interview was conducted, and 2) Fidelity for youth enrolled for greater than 120 days when the interview was conducted 28

Fidelity by Length of Time in Services Fidelity was analyzed based on length of time in service when the interview was conducted, comparing four groups: 1) Fidelity scores for youth enrolled for 90-179 days when the interview was conducted, 2) 180-269 days, 3) 270-364 days, and 4) greater than 365 days. 29 MA TOM and WFI-4 Scores Compared to National Mean 30 Wraparound Fidelity Index Short Form (WFI-EZ) 31 What is the MA WFI-EZ? Wraparound Fidelity Index, Short Form (MA WFI-EZ) Intended to assess both conformance to the Wraparound practice model and adherence to the principles of Wraparound in service delivery Brief, confidential survey completed via telephone, email, or mail with caregivers and a Demographic Form

Tool consists of 42 items; four linked to Basic Characteristics of Wraparound, 25 linked to Key Elements, four linked to Satisfaction, four linked to Outcomes, and five linked to Functional Outcomes 32 How was the WFI-EZ data collected? MA WFI EZ: Conducting the Interviews Collaborative Quality Improvements (CQI), a mental health research and evaluation organization, implements the MA WFIEZ. Caregivers can complete the survey on their own by mail, online, or via a phone interview with a CQI interviewer. CQI trains interviewers (primarily parents of youth with SED) conduct the interviews and provide ongoing supervision to interviewers to ensure inter-rater reliability. CQI currently has four interviewers and capacity for three languages: English, Spanish, and Haitian Creole. The goal is to complete 20 surveys of caregivers of youth enrolled at each of the CSAs.

CQI completed 346 surveys during FY 2016. 33 How was WFI EZ data collected? MA WFI EZ: Collecting the Data CSA Staff Responsibilities: Inform caregivers of the survey and evaluation process, including options for completing the survey Seek consent from all eligible caregivers, who should have signed a consent indicating whether they chose to participate or not Make sure a call information sheet was completed for each caregiver, including preferred method of completion Eligibility was defined as anyone (with an enrolled child under the age of 18) enrolled in ICC between January 1 and December 31, 2015. Caregivers were eligible to be interviewed if they had been enrolled in ICC for three or more months. 34 How was WFI EZ data collected? Fax signed consents along with the call information sheets to CQI. Information from the call information sheet was entered into two databases; one database for those who indicated their preferred method as mail or email. While those who indicated their preferred method as

phone interview or did not provide a preference were entered into a call contact database which provided interviewers with an updated listing of those caregivers who were eligible to be interviewed. 35 How was WFI EZ data collected? MA WFI EZ: Collecting the Data CQI Tasks: Review call information sheets for any missing or inaccurate information and follow up with CSA Enter contact information into one of two databases, depending on preferred survey method and contact information provided Contact caregivers who were eligible to participate; either through a mailed letter, emailed instructions for online completion, or via the phone to schedule an interview time Track mail/email responses and cross walk with phone lists; track total number completed per CSA to determine necessary follow-up methods Conduct phone interviews as indicated/needed Review and enter surveys completed by mail and email into WrapTrack Enter completed interview data (scores) into WrapTrack Routinely send reports to MBHP: number of interviews completed at each CSA, number of consents received from each CSA, total number of attempted and refused calls for the week, and total number of calls made and interviews completed since the project began 36

How was WFI EZ data collected? Interviews averaged 15-20 minutes. Caregivers received a $15 check for their participation. Addresses are confirmed with caregiver before completing the call. 37 How was the MA WFI EZ collected? MA WFI EZ: Challenges Consent Process Incomplete, inaccurate, ineligible consents; sending wrong consent (WFI rather than EZ or old form of EZ) Preferred completion method not being indicated; preferred method indicated but did not include necessary contact information Difficulty Reaching Caregivers Dont return messages No way of knowing if email address is correct/reaching respondent Frequent phone number/address changes 38 Key differences WFI 4 vs. WFI EZ WFI 4

Items are based on principles and core activities of Wraparound and are organized by the four phases. Responses are scored 0-2 (No/Sometimes/Yes/NA) and scaled 0-4. Survey completed by telephonic interview with caregiver Survey completion takes approximately 40 minutes. WFI EZ Items on Fidelity are based on Wraparound involvement and key elements; self-administered survey also includes sections on satisfaction and outcomes. Responses are scored on scale of Strongly agree to Strongly disagree. The scale is converted to -2 (Strongly disagree) to 2 (Agree), so some responses are in the negative. The survey is completed via telephonic interview with caregiver. or selfadministered via email or mail. Survey completion takes approximately 15-20 minutes 39 WFI EZ: Fidelity Scores by Key Element 40 WFI EZ: Basic Characteristics Most respondents report basic characteristics of Wraparound occurred during services 41

WFI EZ: Satisfaction 42 WFI EZ: Satisfaction 43 WFI EZ: Outcomes 44 WFI EZ: Functioning Outcomes Very much Not at all 45 WFI EZ: Areas of Strengths and Areas for Improvement

Strong scores in Effective Teamwork Team members follow-through on tasks they are assigned. Team consists of the right individuals for the youth and family Because of Wraparound, families know what to do to handle a crisis Families are reporting lower than average negative contact with police since starting Wraparound. Natural and Community Supports score below the national mean. Lower scores in Strength and Family Driven Key Element Overall satisfaction with Wraparound Services is lower than the National Mean. 46 WFI-EZ and TOM 2.0 Score Comparison 47 Score Comparison Across Tools 48

WFI EZ Looking Ahead to FY 2017 Signed consent forms are sent to CQI on a rolling basis - please make sure to use the correct consent form (in packet). Eligible caregivers include those of youth under the age of 18 enrolled in ICC between January 1 and December 31, 2016 with signed consent forms TOM 2.0 Complete TOMs and enter them into WrapTrack on a rolling basis as they are conducted. The data collection period is July 1, 2016 through June 30, 2017. When to complete a TOM? Existing ICC Staff: Each ICC staff must have two TOMs completed per year of employment. New ICC Staff - New ICC staff must have two TOMs completed within months four and six from the date of hire. This allows adequate training of staff before utilizing the TOMs. Transition Continue to focus on improving practice in this area 49

Recently Viewed Presentations

  • Who are the real first offenders? Hollie Wilson,

    Who are the real first offenders? Hollie Wilson,

    Acknowledgements. Supervisory team. Em Prof. Mary Sheehan. Dr. Gavan Palk. Funding. Australian Transport Safety Bureau (ATSB) Centre for Accident Research & Road Safety - Queensland (CARRS-Q)
  • Board Game Counter - Digital

    Board Game Counter - Digital

    The schematic diagram for the sequential logic section can be divided into two parts. The first is a 3-Bit counter that is constructed from two Divide-By-Two sections.
  • ASE 2020 GCSE science networking hubs - filestore.aqa.org.uk

    ASE 2020 GCSE science networking hubs - filestore.aqa.org.uk

    ASE Annual Conference: The GCSE Science Hub network. Jane Bryant. Spring 2020. 11/29/2019
  • The Federal Bureaucracy - Sara Russo- Stratford High School

    The Federal Bureaucracy - Sara Russo- Stratford High School

    Understanding Bureaucracies Understanding Bureaucracies Bureaucracy and the Scope of Government The size of federal bureaucracy is an example of a government out of control. Even though the size of the bureaucracy has shrunk Some agencies don't have enough resources to...
  • On Chip Communication Architectures

    On Chip Communication Architectures

    FLIT Transmit Access routing tables Assemble packets Split into flits Receive Synchronize Drop routing information * Bidirectional Link 2 * 32-bits data links Asynchronous, credit based flow control Easy floorplan routing & timing in DSM process * 36 bits 36...
  • EVM tools at CERN - Agenda (Indico)

    EVM tools at CERN - Agenda (Indico)

    The AIS group provides CERN with a set of integrated and reliable Corporate Information Systems. The group ensures coherent, reliable, comprehensive and accurate management information across CERN. The group provides optimal support for all business processes at CERN. ... EDH....
  • ATMOSPHERIC EMITTED RADIANCE INTERFEROMETER - CURRENT AND FUTURE

    ATMOSPHERIC EMITTED RADIANCE INTERFEROMETER - CURRENT AND FUTURE

    In Preparation Applications of High Temporal Resolution AERI Data: Boundary Layer Roll Detection during CRYSTAL Updated AERIplus retrieval algorithm implemented at for DOE ARM at PNNL, all AERI data from SGP with be reprocess with RUC analysis at first guess...
  • Inside the black box: Raising standards ... - Dylan Wiliam

    Inside the black box: Raising standards ... - Dylan Wiliam

    Assessment for learning: why, what, and how? Dylan Wiliam www.dylanwiliam.net Overview of presentation Why raising achievement is important Why investing in teachers is the answer Why formative assessment should be the focus Why teacher learning communities should be the mechanism...