Breaking Barriers in San Diego Evaluation Narrative

Transcription

1 Breaking Barriers in San Diego Evaluation Narrative Contents Description of Evaluation Plan... 1 Factor 1 Evaluation Design... 1 Factor 2 Description of Evaluation Components... 3 The implementation study... 7 Cost-Effectiveness Analysis... 9 Factor 3 Evaluation Timeline and Schedule Contribution to Evidence Base Factor 1 Building on Evidence Base Factor 2 Contributing to the Knowledge Base of the Broader Workforce System Evaluator Procurement and Deliverables Plan Factor 1 Process for Procuring Evaluator Factor 2 Strategy for Estimating Level of Effort and Conveying Requirements to Potential Bidders

2 Description of Evaluation Plan Factor 1 Evaluation Design: Breaking Barriers in San Diego seeks to replicate and extend the findings from the Families Achieving Success Today (FAST) pilot in Ramsey County, Minnesota. The FAST researchers identified this pilot as the first to apply the evidenced-based IPS model in the service delivery context of TANF recipients with disabilities. 1 Given the small sample size and exploratory nature of this pilot, it is surprising that a significant, positive effect of the treatment was identified. The researchers caution against interpreting these findings without additional investigation to determine if the effect can be replicated with a larger sample size and in a variety of contexts. We echo these reservations and propose that they be addressed by replicating this experiment through a randomized controlled trial with enough participants to conduct a full impact evaluation. The Ramsey County pilot generated compelling support for the hypothesis that the Supported Employment model is appropriate for families receiving TANF and that this approach to increasing families earnings and economic stability warrants further study. While promising, these findings were not large (in magnitude or statistical power). The average duration of employment for the treatment group was 1-2 months longer than for the control group, with average earnings for the year around $1,235 higher. As mentioned above, it is surprising that any statistically significant effect was identified, given the preliminary nature of the Ramsey County pilot and its small sample size. Further study is needed to determine the extent and nature of these effects. 1 Farrell et al. (2013). The TANF/SSI Disability Transition Project: Innovative Strategies for Serving TANF Recipients with Disabilities. OPRE Report

3 With the FAST program having identified a significant effect of the intervention, but lacking the statistical power for more detailed interpretation of findings, SDWP intends to expand upon these efforts. By implementing a larger randomized controlled trial and collecting data on a wider variety of participant characteristics and outcomes, the San Diego project seeks to determine: (1) If the effects identified in Ramsey County are replicable amidst a larger sample size in a different county; (2) How participants barriers to employment are related to success in the program; and (3) What types of employment outcomes can be expected when TANF recipients with a disability exemption receive the IPS model? As we investigate how the IPS model addresses the needs of TANF families with disabilityrelated exemptions from work participation requirements, we will also rely on the substantial body of literature evaluating supported employment and the IPS model. 2 Dartmouth researchers presented findings from 16 Randomized Controlled Trials (RCTs) that have investigated the outcomes of supported employment programs among individuals with severe mental illness, all finding higher employment rates among participants receiving supported employment services than those enrolled in a control condition. A 2011 meta-analysis compared results of four RCTs that enrolled participants with severe mental illness in either an IPS program or a control group receiving industry-standard vocational programs. They found strong support for the IPS model when looking at outcomes including employment attainment and retention (with 70% of effect sizes exceeding 0.70). 3 The literature on supported employment primarily addresses its application in the context of severe mental illness; this is 2 Bond, G.R., Drake, R.E., & Becker, D.R. (2008). An update on randomized controlled trials of evidencebased supported employment. Psychiatric Rehabilitation Journal, 31, Campbell, K., Bond, G.R., & Drake, R.E. (2011). Who Benefits from Supported Employment: A Meta-Analytic Study. Schizophrenia Bulletin, 37:2,

4 especially true when searching for impact evaluations or rigorous analyses. There is a clear need for more research to investigate whether the model is equally promising for other populations, including low-income families with disabilities. While San Diego s program s primary goal is to determine whether the Ramsey County pilot s outcomes are replicated in a larger randomized controlled trial, we also seek to better understand the needs and circumstances of TANF families with disability exemptions and to investigate whether there are additional needs that the current systems are unable to meet. This approach will allow us to build upon the supported employment work that SDWP has undertaken in partnership with the County of San Diego. By bringing these endeavors together with the Common Customer approach to serving TANF recipients participating in the Health Profession Opportunity Grant (HPOG) program, we can work from the strength that have already been identified at both service delivery and system alignment levels. These collaborative efforts will provide an incredibly strong foundation for Breaking Barriers San Diego. Factor 2 Description of Evaluation Components: Breaking Barriers San Diego project proposes to undertake a rigorous Type B project, including extensive impact evaluation, an implementation study, and a Cost-Effectiveness Analysis. The overarching aims of this research include (1) comparing the treatment and control group on employment-related outcomes and (2) assessing the role of participants barriers to employment. These avenues of inquiry relate to all three components of the evaluation plan: if there is a significant, positive effect of receiving the treatment, policymakers should consider both the effect sizes of any significant findings (from the impact evaluation) and the cost per unit to achieve the outcome (from the 3

5 cost effectiveness analysis). The implementation study will shed light on the context of these findings and identify limitations or best practices for further replications. Together these evaluation components will contribute to the national conversation around the utility of providing supported employment services to low-income families dealing with multiple barriers to employment. Each component is described in greater detail below. Impact evaluation: This Breaking Barriers program design and evaluation plan were largely driven by the intention to test our replication and extension of the FAST program with an (RCT) design. As SDWP is responsible for developing both the program and the evaluation concurrently, we are committed to ensuring the program services and performance metrics are complementary to the outcomes and research questions around which the evaluation is designed (something that can be difficult to achieve if the evaluation is orchestrated by an outside organization or is not planned concurrently with the program). Survey materials will be selected on the basis of their appropriateness for the participant population. SDWP will work with the evaluation team to assess whether materials have been validated for use among participants with varying education and comprehension levels as well as a variety of cultural backgrounds and primary languages. In the event that materials need to be developed or validated for the purposes of this study, SDWP will work with its evaluator to ensure reliability and validity are appropriately assessed. Data collection can be categorized into 4 major phases: (1) at intake; (2) while assigned to the treatment or control condition; (3) at the termination of services (in the case of the control group, after a comparable amount of time has passed); and (4) during the 12 months of followup after program completion. The entire sample of 1,000 participants will go through a process 4

6 of orientation, eligibility, and informed consent before being randomly assigned between treatment and control groups. As such, some data will be collected concurrently with the random assignment process, minimizing differences in the experiences of the treatment and control groups. This data will include (a) demographic information and other family characteristics; (b) employment history; and (c) an assessment of barriers to employment using either the Minnesota employability measure 4 or a comparable inventory 5 ; Following random assignment, much of the data collected on the treatment group will relate to programmatic activities (e.g., type and intensity of services received, frequency of contact with case manager, and number of days between intake and first employer interaction). SDWP s performance policies require case managers to record such client services in the MIS within three business days of their occurrence. These same requirements would likely be implemented for the Breaking Barriers program, with performance metrics set for the most important program outputs to ensure complete and accurate data collection. SDWP will work with its evaluator to identify or develop appropriate survey materials for assessing whether control group participants are accessing employment services from other sources during the established time period after random assignment. 6 These materials will 4 Minnesota Department of Human Services Employability Measure and User s Guide. St. Paul, MN: Minnesota Department of Human Services. 5 The state of Minnesota has an employability measure that is administered to TANF families every six months. The inventory assesses participant strengths and barriers, returning a score between 1 and 5 on each of 11 domains related to employability. This measure or a comparable assessment will be employed in our study at random assignment and at least one additional time point. 6 The goal will be to begin collecting follow-up data on treatment and control group participants after comparable amounts of time following random assignment (e.g., an average of 12 months for both groups), but the time-unlimited principle of supported employment complicates establishing an arbitrary period of time after which control group participants are considered in follow-up. SDWP will work with its evaluator to develop a strategy for making this determination. 5

7 include questions about whether they are voluntarily participating in Welfare to Work activities, have enrolled in an occupational skills training program, or have engaged in job search activities. This data collection phase would also include revisiting assessments of employability and barriers to employment. In collecting exit and follow-up data, some identical survey elements will be administered to both treatment and control groups (i.e., those measuring the key program outcomes of interest). Additional data will be collected on the treatment group, both through the MIS and perhaps as an additional component of follow-up survey materials. While this study is designed with a clear sense of the baseline, employability and outcome measures of interest, not all aspects of exit and follow-up data collection have been developed, in recognition that data priorities and needs will evolve as the program is implemented and preliminary analyses are conducted. For example, we cannot know at this time what the breakdown of qualifying disabilities will look like among our participants. 7 If several major categories of disabilities emerge (e.g., if medical/physical, mental health, and child disability are the three main groups that enroll), we may want to augment our data collection with group- or disability-specific questions. SDWP will work closely with its third party evaluator and the National Evaluation Coordinator (NEC) to revise the study s methodology as appropriate. Finally, it is understood that any participants who are veterans must be excluded from the evaluation. With a full 500 treatment group members, it is not expected that veteran participation will threaten statistical power to identify a treatment effect where one exists. We have taken additional considerations regarding the need to balance the risks of Type I and Type 7 In CA, participation is voluntary for TANF families with an exemption status. Therefore, we cannot know which families will choose to enroll in the Breaking Barriers program. 6

8 II errors that is, the more care we take not to mistakenly conclude a difference exists (where it does not), the greater the risk that we fail to notice a difference where it actually exists. 8 In comparing two models for social service delivery (in this case, supported employment for the treatment group and case management as usual for the control group), we are able to pose the research question of whether more services and increased spending on programmatic costs delivers better outcomes for participants. In other words, if supported employment results in more participants getting jobs, but not many more or the jobs don t last very long then it falls to evaluators and practitioners to decide whether the impact is worth what it costs compared to the current standard of services. With this type of evaluation, we feel it is essential for the program s impact evaluation to be accompanied by a cost-effectiveness analysis. We are committed to ensuring that any statistically significant finding is presented and interpreted in the context of its effect size and the results of a cost-effectiveness analysis. The implementation study will look at the degree to which the program, as implemented, reflects the program design, including the process, level of participant engagement/participation, and the types of services received. With the intention to replicate the FAST model as closely as possible, we will assess program fidelity to both the IPS model and the aspects of the Minnesota study that differed from the traditional supported employment model. Measures of participation will include duration, intensity and nature of services received to assess the extent to which specific program components were implemented as intended. Duration will be measured by the number of days between enrollment and client-driven 8 By performing multiple regression analyses in our impact evaluation, we reduce concerns about inflated alpha levels compared to conducting a series of t-tests or ANOVAs. 7

9 (voluntary) exit from the program (or a client-initiated lack of contact that becomes permanent). The intensity and nature of services each participant receives will be evaluated using data from the MIS and individual case files. Dichotomous variables will be developed from questions such as (a) was client referred for child/family therapy?; or (b) if more than 30 days after intake and client has not begun intensive job search, have they been referred for vocational assessment? Additional quantitative measures will include (a) the number and type of interactions each participant has with the case manager and employment specialist; (b) number of days between enrollment and job search; and (c) number of days between employment placement and voluntary program exit. Evaluators qualitative data collection and analysis will deepen understanding of how program design manifested in implementation. In-person interviews will take place 3-5 times over the life of the grant, at least once per year for the three years of program activities. Program staff, both at SDWP and those who provide direct service to the clients, will be interviewed one on one or in small groups. In general, interviewees will be asked questions to supplement evaluators understanding of the quantitative data that is available, along with any trends or issues that might not otherwise be captured. Expected topics of discussion include the context in which program activities occur, any communication or system alignment issues that might affect service delivery, and how data observed in the MIS and case notes relates to the realities of program activities. While the third party evaluator is a vendor of SDWP, the quality of the implementation study requires that we grant the evaluator considerable autonomy in developing the interviews and maintaining the confidentiality of staff interviewed. We will also work with the third party 8

10 evaluator to develop a survey for a small sample of treatment group participants regarding their experience in (and perception of) the program. Cost-Effectiveness Analysis: Understanding the cost of delivering this intervention will inform decisions about the efficacy of providing supported employment services to low-income families with disabilities. The Cost-Effectiveness Analysis component of the Breaking Barriers in San Diego evaluation will allow us to identify unit cost for the study s results. Our preliminary evaluation plan identifies the following outcomes of interest for the cost-effectiveness study: (1) Ever-employed rate; (2) Days of (consecutive) employment; and (3) Reduction in number of barriers to employment. The primary costs associated with this intervention will stem from staffing time in providing the IPS model of supported employment. We will track programmatic spending through staff hours worked as well as spending on other essential components of the program, including software or applications that involve licensing fees per use (e.g., the CareerScope assessment program and the income calculator for benefits counseling) and the costs associated with business services endeavors that increase employment opportunities for participants (i.e., employer socials and customized recruitments). As the control group will not receive a structured program for which we could estimate the cost of service provision, these analyses will focus on substantive services on which the treatment and control groups differ thus excluding costs associated with random assignment or data collection activities that pertain to both conditions. Cost data is expected to be obtained from two main sources: SDWP s budget for the project and financial/accounting records indicating direct service providers spending on these needs and SDWP s investment in software, equipment and supplies. Outcome data will be obtained 9

11 from the program s MIS and evaluation data, which will contain information about clients employment placements and changes in employability scores between intake and program completion. The primary innovation being tested here is the provision of supported employment services to TANF recipients with disabilities. This model differs from traditional workforce development programs in its emphasis on combining employment services with counseling (or other relevant rehabilitative services) and beginning job search and employer contact shortly after enrollment in the program. Tracking program activities will contribute to the implementation study. The impact evaluation will define success in terms of both program outputs (e.g., participated in job search activities or received child/family therapy) and outcomes (i.e., employmentrelated measures). This approach is in line with the zero-exclusion principle of the supported employment model: we recognize that not all participants will be able to obtain employment, but we will strive to reduce barriers to employment among all who want to work. The primary outcomes of interest are as follows: (1) Employment placement (worked at least one day between enrollment and completion dates); (2) Duration of employment; (3) Reduced barriers to employment. Factor 3 Evaluation Timeline and Schedule: All aspects of this program design have been empirically oriented and focused on complimentarily between service delivery and the evaluation plan. SDWP looks forward to partaking in all evaluation-related activities, including working with DOL/ETA s NEC and technical assistance providers and collaborating with the third party evaluator we procure. We are prepared to travel to Washington, D.C. or another location 10

12 for training and other activities upon receipt of the grant award and on an annual basis thereafter. The tentative (overall) evaluation timeline and schedule is as follows: Evaluation activities Evaluation component & description Schedule Implementation year: October 1, 2014 September 30, 2015 Grant award notification 9/30/2014 Refine and solidify program design Establish workgroup/advisory committee of local experts to advise on program design, implementation details and region-specific needs Consulting and training from IPS experts Procure third party evaluator By March 2015 Work with third party evaluator on program and evaluation design; collaborate on experimental design and project management plan Develop Initial Evaluation Design Report June 2015 (within 9 months of grant award) Develop survey materials for participant intake and pre-program assessment July 2015 September 2015 Work with third party evaluator and NEC to revise Initial Evaluation Design Report and prepare Final Evaluation Design Report August 2015 (within 11 months of grant award) Program Year 1 Begin random assignment 10/1/2015 Administer pre-program assessment survey to treatment and control 10/1/2015 until group participants complete Contribution to Evidence Base Factor 1 Building on Evidence Base: As mentioned above, supported employment is an evidence-based model that has received national attention for its success increasing employment rates among adults with serious mental illness (SMI). While several studies have piloted applying this model for other populations that face difficulty obtaining employment, the FAST project in Minnesota was the first to bring supported employment into the context of TANF families with a disability exemption from the federal work participation requirement. While supported employment and the IPS model have both been rigorously evaluated in RCT settings, there is little evidence (beyond the FAST results) regarding the appropriateness of IPS 11

13 for TANF families with disabilities. Findings from the FAST pilot suggest the IPS model is a promising approach, but it has yet to be determined whether these findings are replicable among larger samples and in different contexts. With the FAST pilot having been a small-scale random assignment study, the next step to build upon this research base is to replicate the FAST pilot in an RCT with enough participants to draw statistical inferences. Enrolling more than twice as many participants through an RCT design will generate sufficient statistical power to measure the size of an effect if one is found to be present. The San Diego project will substantially augment the evidence base regarding supported employment and IPS model interventions among the TANF disability population. From an evaluation perspective, this project s primary goal is to determine whether the integrated services that participants receive yield a significant, positive impact compared to the control group s experience. Additionally, we intend to collect data on a broader variety of measures than were assessed in the Ramsey County pilot. The larger sample size and broader range of variables measured will also allow us to investigate whether the entire treatment group benefitted from the intervention. It is possible that the IPS model is more appropriate for some participants than others; should such a within-group difference exist, a larger group would have to receive the intervention for it to be identifiable. With a full-scale replication of this study, the San Diego project would pursue insight into such uncertainties in two ways: (1) by conducting cross-sectional analyses to better understand who is in the treatment group; what sorts of barriers to employment they are experiencing at intake, while participating, and after completing the program; and if there are needs identified at intake that the present model is unable to meet; and (2) by performing a series of linear 12

14 regression analyses to investigate which characteristics and experiences predict programmatic impacts or employment placement. If it is found that the intervention is more effective for some participants than others, such data will allow us to better understand for whom the IPS model is appropriate. Benefits to the region include better understanding the needs of a population that is largely underserved through federal resources. Factor 2 Contributing to the Knowledge Base of the Broader Workforce System: As the supported employment model originated as an alternative to day treatment centers for individuals with severe mental illness, it is largely untested in the field of workforce development. The ten-year MDRC study Enhanced Services for the Hard-to-Employ (HtE) Demonstration and Evaluation Project found mixed results for several innovative workforce strategies intended to improve employment outcomes for groups that struggle to find work in the competitive marketplace. 9 Two of the sites studied involved TANF recipient families, but neither intervention yielded lasting effects. The evaluators suggested that future research combine job search and employment services with the relevant rehabilitation-oriented supports, specifically highlighting IPS as a model that could address this need. San Diego could be one of the first counties to join the FAST project in piloting IPS for the workforce development context. There are some differences between the traditional IPS and workforce development approaches: IPS prioritizes rapid entry into job search over the pursuit of training and emphasizes co-location with mental health and other services, among other distinctions. This 9 Butler et al., (2012). Enhanced Services for the Hard-to-Employ Demonstration and Evaluation Project: Final Results of the Hard-to- Employ Demonstration and Evaluation Project and Selected Sites from the Employment Retention and Advancement Project. OPRE Report

15 distance between the two models warrants further study, as there may be supported employment principles that could enrich the workforce system at large, or populations seen as hard to serve under the WIA model that would see better outcomes if engaged in a supported employment approach. Breaking Barriers in San Diego stands to contribute to this knowledge base in multiple ways. We expect some of our control group participants to utilize other services in the AJCC system, whether they be training funds or subsidized employment opportunities. While we do not seek to influence the activities of the control group, the study will capture data on any alternate activities or services that control group participants report. Depending on how many control group participants utilize the AJCC system, we may be able to conduct exploratory analyses into how randomly assigned participants fared between supported employment and WIA services. This possibility aside, the results of this study will help us to better understand the circumstances and needs of TANF families with disabilities. Evaluator Procurement and Deliverables Plan Factor 1 Process for Procuring Evaluator: Immediately following grant award, SDWP will issue a Request for Proposals (RFP) using a competitive procurement to secure the third party evaluation team. SDWP procurement policies require open and free competition to the extent possible. Factor 2 Strategy for Estimating Level of Effort and Conveying Requirements to Potential Bidders: SDWP feels that a critical component of conveying the evaluation needs to potential bidders is having staff members who understand research design, program evaluation and statistical analysis. Current SDWP staff includes a Research Manager and Performance 14

16 Management Analyst, both of whom have training in these areas. SDWP also has robust experience with collaboration and project management on evaluation projects of similar scope. As mentioned above, the HPOG took over 1,000 participants through the random assignment process. Our navigators (case managers) were responsible for the outreach and intake activities, a task that became more difficult during the random assignment period when participants knew they had a 50% chance of not receiving services through the HPOG program. Nonetheless, the random assignment process was completed within 18 months. With two program years remaining after random assignment was completed, evaluators were able to follow those randomly assigned for a longer period of participation than was attainable for other grantee sites. SDWP has also worked closely with the team evaluating our HPOG grant for the ISIS study. The evaluation team makes site visits 1-2 times per year, and phone meetings take place at least once per month. While we did not procure the evaluator for this project (as the ISIS study involves multiple HPOG grantees and several other programs), Manager of Special Projects and Performance Management Analyst at SDWP have both worked closely with the evaluation team. We have come to appreciate the importance of facilitating communication between the evaluators and our direct service providers, to ensure we all operate from common definitions of various program components. 15