DEI INTERIM SYNTHESIS REPORT FOR YEAR 4

Transcription

1 Social Dynamics, LLC 481 North Frederick Ave., Suite 410 Gaithersburg, MD Phone: (301) DEI INTERIM SYNTHESIS REPORT FOR YEAR 4 AUTHORS Social Dynamics, LLC Robert Bleimann, Ph.D. Douglas Klayman, Ph.D. Mathematica Policy Research Heinrich Hock, Ph.D. David Stapleton, Ph.D. CONTACT INFORMATION Douglas Klayman, Ph.D., Project Director dklayman@socialdynamicsllc.com x105 August 1, 2016 PREPARED FOR: U.S. Department of Labor Office of Disability Employment Policy 200 Constitution Avenue, NW Washington, DC PREPARED BY: Social Dynamics, LLC 481 North Frederick Ave., Suite 410 Gaithersburg, MD Office: Fax:

2 Table of Contents Introduction... 7 Section 1.0 Background... 8 Section 2.0 Status of DEI Program Requirements Key DEI Requirements Select Youth or Adult Focus Hire DEI State Lead and Disability Resource Coordinators Ensure American Job Center Accessibility Participate in Ticket to Work as an Employment Network Implement Two of the Six Service Delivery Strategies Selection of Service Delivery Strategies by DEI Rounds 1 4 Grantees...14 Section 3.0 DEI Evaluation DEI Evaluation Logic Model Random Assignment of LWIAs Data Sources Reliance on WIASRD Evaluating Systems Change...25 Section 4.0 DEI Ticket to Work Ticket to Work TTW and the DEI TTW Implementation Challenges TTW Improvements Identifying and Recruiting TTW Customers Screening TTW Beneficiaries DEI Grant Timing and TTW Program Self-Sufficiency at the AJC SVRC and Partnership Plus

3 Table of Contents (continued) 4.9 TTW (Ticket to Work) Promising Practices Expedited Ticket (Tennessee) Sharing Revenue with Employers (Washington) SSA e-processes (Iowa) State ENs as Learning Opportunities for AJCs Sustaining DRCs through Ticket Revenue and LWIA Resources...45 Section 5.0 Implementation of the DEI Service Delivery Strategies by Round 1 4 Grantees Customized Employment Self-Employment Guideposts for Success Asset Development Integrated Resources Teams Blending and Braiding Resources...65 Section 6.0 Disability Resource Coordinator Roles and Responsibilities of DRCs (Disability Resource Coordinators) in Rounds 2 and Section 7.0 Benefits Planning Section 8.0 Other Staff Serving DEI Customers Section 9.0 DEI Participant Characteristics and Customer Outcomes Description of the Population of DEI Customers in the Pilot LWIAs Methodology Description of DEI Customers in the Pilot Sites Number of DEI Adult and Youth Customers Distribution on DEI Customers, by State Entry Flow of DEI Customers Over the Grant Operation Period DEI Customers Exit Status at the End of Grant Operations

4 Table of Contents (continued) 9.2 Creation of the RCT Analysis Sample Adjustments Due to the RCT Design and Structure of the WIASRD Extracts Adjustment Based on Random Assignment of LWIAs Adjustments Based on the Structure of the WIASRD Consequences of Adjustments for Analysis Sample of DEI Customers Including Customers in the RCT Control Group Distribution of the RCT Analysis Sample Among DEI States Pre-Enrollment Characteristics of DEI Customers and Treatment-Control Differences in the RCT Analysis Sample Methodology Pre-Enrollment Characteristics of Adults Pre-Enrollment Characteristics of Youth Exit Outcomes for Adults and Youth Methodology Exit and Employment Outcomes for Adult Entrants Exit and Employment/Education Outcomes for Youth Entrants Employment, Employment Retention, and Earnings Outcomes for Adult Exiters Methodology Common Measures Outcomes for Adult Exiters Placement in Employment/Education, Degree Attainment, and Literacy/Numeracy Outcomes for Youth Exiters Methodology Common Measures Outcomes for Youth Exiters Growth in the Number of JSWDs Receiving Intensive Services or Training Methodology Percentage Growth from Pre-DEI Year in Adult States Comparison of Pre- and Post-DEI Entrants Exit Rates Methodology Exit Rates of Pre- and Post-DEI Entrants in Adult States

5 Table of Contents (continued) List of Exhibits Exhibit 1: Grantees Implementation Timeline... 7 Exhibit 2: DEI Grantees by Round...10 Exhibit 3: Map of DEI States...10 Exhibit 4: Selection of DEI Service Delivery Strategies by Grant Round...15 Exhibit 5: DEI Logic Model...22 Exhibit 6: System Change Indicators...25 Exhibit 7: Program Maturation and Systems Change Model...27 Exhibit 8: List of Work Incentives Available to SSI and SSDI Beneficiaries...28 Exhibit 9: TTW Detailed Milestone Payment Structure (2015 Figures)...35 Exhibit 10: Self-Reported Ticket Assignments...38 Exhibit 11: Type of EN Used by Round 1-3 DEI Grantees...44 Exhibit 12: Use of Disability Resource Coordinators across Grantees...76 Exhibit 13: Key Responsibilities of DRCs by Type and State...77 Exhibit 14: Rounds 1-3 Grantees with Certified Benefits Planners on Staff and/or Through Local Providers...82 Exhibit 15: Number of DEI Participants in the Pilot LWIAs...89 Exhibit 16: Distribution of DEI Customers, by Adult State...90 Exhibit 17: Distribution of DEI Customers, by Youth State...90 Exhibit 18: Entry Flow of DEI Customers, by Grant Operation Quarter (Adult States)...91 Exhibit 19: Entry Flow of DEI Customers, by Grant Operation Quarter (Youth States)...92 Exhibit 20: Number of Active and Exiter DEI Customers at the End of the Grant Operation Period (Adult States)...93 Exhibit 21: Number of Active and Exiter DEI Customers at the End of the Grant Operation Period (Youth States)...93 Exhibit 22: Distribution of Active and Exiter DEI Customers at the End of the Grant Operation Period, by Entry Quarter (Adult States)...94 Exhibit 23: Distribution of Active and Exiter DEI Customers at the End of the Grant Operation Period, by Entry Quarter (Youth States)...94 Exhibit 24: Summary of Adjustments to Create the RCT Analysis Sample, by Focus and Round...98 Exhibit 25: Number of Customers in the RCT Treatment and Control Groups, by Focus and Round

6 Table of Contents (continued) List of Appendices Exhibit 26: Distribution of Treatment and Control Customers in Adult States Exhibit 27: Distribution of Treatment and Control Customers in Youth States Exhibit 28: Pre-Enrollment Characteristics of Customers in Adult States Exhibit 29: Pre-Enrollment Characteristics of Customers in Youth States Exhibit 30: Cumulative Percentage of Adults Exiting from WIA, by Quarter After Entry Exhibit 31: Cumulative Percentage of Adults Exiting to Employment, by Quarter After Entry Exhibit 32: Cumulative Percentage of Youth Exiting from WIA, by Quarter After Entry Exhibit 33: Cumulative Percentage of Youth Exiting to Employment or Placement in Education, by Quarter After Entry Exhibit 34: Employment, Retention, and Earnings Outcomes for Adult Exiters Exhibit 35: Employment/Enrollment and Degree Attainment Outcomes for Youth Exiters Exhibit 36: Growth from Pre-DEI Year in the Number of Adults Receiving Intensive Services or Training, by DEI Grant Operation Quarter Exhibit 37: Exit Rates of Adults Enrolling in a WIA Program Before and After DEI Grant Operations Began, by Quarter After Entry Appendix A: Appendix B: Customer Characteristics and Outcomes by Round and Focus...A-1 Data and Methods for Quantitative Analyses...B-1 6

7 Introduction The purpose of the Disability Employment Initiative (DEI) Evaluation is to report to the U.S. Department of Labor s (DOL) Office of Disability Employment Policy (ODEP) and Employment and Training Administration (ETA) on the implementation, outcomes, and impact of the DEI. While the main research questions of the DEI Evaluation concern employment outcomes and the impact of the program on job seekers with disabilities (JSWDs), other questions focus on how specific service delivery strategies and grant requirements were embedded within the workforce investment systems of participating Local Workforce Investment Areas (LWIAs) in 26 states and 31 grantees, with the support of both public and private sector partners and service providers. The DEI is designed to increase access to training and employment opportunities and improve outcomes for youth and adults with disabilities who are unemployed, underemployed, and/or receiving Social Security disability benefits. ODEP jointly funds and administers the DEI with ETA. The DEI Evaluation includes process/implementation, outcome, and impact analyses. It also includes the collection of qualitative information designed to identify systems change and measure program fidelity in participating LWIAs. This fourth-year report provides findings on the implementation, outcomes, and impact of the first (Alaska, Arkansas, Delaware, Illinois, Kansas, Maine, New Jersey, New York, and Virginia) and second (California, Hawaii, Ohio, Washington, Tennessee, South Dakota, and Wisconsin) rounds of DEI grantees. Round 1 began in October 2010 and ended in September 2013 (Exhibit 1), while Round 2 began in October 2011 and ended in September Preliminary information on Round 3 and 4 grantees is also discussed. Round 3 grantees completed their implementation year in September 2013 and began full operations in October Round 4 grantees began their implementation year in October 2013 and began operations in October See Grantee Profiles for Rounds 1, 2, 3, and 4 in Appendix A. Exhibit 1: Grantee Implementation Timeline Grant Period Oct10 Oct11 Oct12 Sept13 Oct13 Sept14 Oct14 Sep15 Oct15 Mar17 Round 1 (2010) Round 2 (2011) Round 3 (2012) Round 4* (2013) Implementation Year Operation Years Completion * Round 4 grantees have an 18-month implementation period. Rounds 1 3 have a 12-month implementation period. 7

8 Section Background Recent figures from the U.S. Bureau of Labor Statistics indicate that the aggregate unemployment rate for JSWDs reached 10.2 percent in August 2015, more than twice that of the rate for individuals without disabilities (5.0%). 1 Meanwhile, the labor force participation rate among people with disabilities (19.8%) is less than one-third the rate of individuals without disabilities (68.8%). The DEI is DOL s latest effort to ensure that American Job Center (AJC) staff has the training and resources to serve JSWDs. They currently serve thousands of JSWDs through core services and Workforce Investment Act- (WIA) funded staff-assisted core, intensive, and training services, which include skills assessments, job searches, job skills training, and related services. The precursor to the DEI was the Disability Program Navigator (DPN) Initiative, which operated from in 42 states, the District of Columbia, Puerto Rico, Guam, and the U.S. Virgin Islands. The DPN Initiative focused on developing new and ongoing partnerships to achieve seamless, comprehensive, and integrated access to services; creating systems change; and expanding the workforce development system s capacity to serve JSWDs and employers. DEI Funding In August 2010, ETA and ODEP joined their efforts and released a Solicitation for Grant Applications (SGA) that made provisions for approximately $21,276,575 million in funds for cooperative agreements to go to the first round of state WIA entities. This funding was a product of the Consolidated Appropriations Act of 2010, 2 which allocated one-half of these funds to ETA and one-half to ODEP for the two agencies to develop and implement an approach that would increase the effective and meaningful participation of JSWDs in the workforce. 3 To accomplish this goal, ODEP and ETA designed the DEI to (a) refine and verify service delivery strategies for employing JSWDs that have proven effective or promising, and (b) foster the replication of those strategies across public workforce development systems. DEI is designed to help JSWDs achieve self-sustaining employment through model service delivery. 4 Nine states were awarded grants in 2010 to implement Round 1 DEI projects. In 2011 and 2012, ETA awarded $21,166,560 and $20,654,352 to 14 new states, and in 2013 a total of $18,597,758 was awarded to five Round 1 states, which received Round 4 continuation grants, and three new DEI states. 5 In total, ETA and U.S. Department of Labor, Bureau of Labor Statistics. (2015). Economic news release: Table A-6. Employment status of the civilian population by sex, age, and disability status, not seasonally adjusted. Retrieved from Pub. L. No U.S. Department of Labor, Office of Disability Employment Policy. (2010). Disability Employment Initiative fact sheet October Retrieved from U.S. Department of Labor, Office of Disability Employment Policy. (2010). Disability Employment Initiative fact sheet October Retrieved from See Methodology Section for a description of the random assignment of LWIAs to either pilot/treatment or control groups. In the five states that received both Round 1 and Round 4 funding, LWIAs that served as control sites during Round 1 were eligible for Round 4 random assignment, as were LWIAs that were not involved in the Round 1 DEI grant. 8

9 ODEP have allocated $81,695,245 in funds to state LWIA entities to fund systems change efforts for the employment of JSWDs in local areas around the country. Section Status of DEI Program Requirements The DEI is designed to improve coordination and collaboration among employment and training and asset development programs implemented at the state and local levels, including the Ticket to Work [TTW] Program. It is also designed to build effective community partnerships that leverage public and private resources to better serve individuals with disabilities and improve employment outcomes. 6 DEI grants are three years in duration. During the first year, grantees and their LWIAs focus on program design and implementation. In the second and third years, the grants must be operational and evaluation data is collected to measure implementation, outcomes, and program impact. 7 While grantees and participating LWIAs have the autonomy to configure their DEI programs in accordance with the needs of their participating LWIAs and JSWDs, they must meet eight program requirements: (1) select either a youth or adult focus; (2) hire a DEI State Lead; (3) hire Disability Resource Coordinators (DRCs); (4) ensure AJC accessibility for JSWDs; (5) implement TTW and establish an Employment Network (EN) at the LWIA and/or state levels; and (6) implement two of the six DEI service delivery strategies. In addition, DEI grantees are required to (7) have a plan for sustaining DEI activities after the grant period and (8) participate in the DEI Evaluation. DEI grantees have access to the National Disability Institute (NDI) Technical Assistance (TA) Center. The receipt of TA is not a requirement of the DEI grant and TA is provided on an as-requested basis. Across all four DEI rounds (a total of 31 DEI grantees), 24 selected an adult focus and 7 selected a youth focus (Round 1: Arkansas, Delaware, New Jersey; Round 2: South Dakota; Round 3: Minnesota; Round 4: Alabama, Idaho; see Exhibit 2). One Round 1 youth grantee (New Jersey) implemented a hybrid (youth/adult) program. It incorporated events and services for youth years of age through their local AJCs, schools, juvenile justice, and/or community-based agencies, while at the same time using DEI funds to meet the requirement of implementing the TTW Program, which is only available to those 18 years of age or older. This age range was a variation on the standard WIA/Workforce Innovation and Opportunity Act- (WIOA) recognized definition of youth. This approach was not used by Round 2 and 3 youth states. Exhibit 3 provides a map of DEI grantees in Rounds U.S. Department of Labor, Employment and Training Administration. (2010). Ticket to Work & Social Security disability benefits TOOLKIT for One-Stop customers who are ready to work. Retrieved from Two Round 1 states (Maine and Virginia) received authorization to use a small amount of DEI grant resources as a flexible spending fund. In 2013, the Round 4 SGA allowed up to 15 percent of grant funds for flexible spending purposes to assure availability of training and employment services for individual job seekers, or other innovative approaches to meet the unique needs of an individual participant (DEI SGA, 2013). 9

10 Exhibit 2: DEI Grantees by Round N = 31 Nine Round 1 States: Alaska Arkansas* Delaware* Illinois Kansas Maine New York New Jersey* Virginia Seven Round 2 States: California South Dakota* Wisconsin Hawaii Ohio Tennessee Washington Seven Round 3 States: Florida Massachusetts Iowa Indiana Louisiana Minnesota* Rhode Island Eight Round 4 States: Alaska** New York** Connecticut Illinois** Virginia** Idaho* Maine** Alabama* * Youth state ** Round 1 state continuing in Round 4 Exhibit 3: Map of DEI States 10

11 2.1 Key DEI Requirements The DEI grants are awarded to state WIA administrative agencies. WIA offers a range of services for adult and youth job seekers, dislocated workers, veterans, and JSWDs, including job searches, occupational skills development, training programs, and employer outreach. The DEI is an intervention designed to support JSWDs seeking employment. It does this through the implementation of service delivery strategies identified by ETA and ODEP that improve coordination and collaboration among employment and training programs implemented at the state and local levels to better serve individuals with disabilities and improve employment outcomes. The following subsections will describe the key programmatic requirements of the DEI Select Youth or Adult Focus To meet the first requirement, DEI grantees must choose to focus on either adults or youth with disabilities. The adult focus includes customers 21 years of age or older, while the age range for youth is As shown in Exhibit 2, among Round 1 grantees, Alaska, Illinois, Kansas, Maine, New York, and Virginia selected to focus on adult job seekers, while Arkansas, Delaware, and New Jersey selected to focus on school-age youth (14 18 years of age). Six Round 2 states focused on adults (California, Hawaii, Ohio, Tennessee, Washington, and Wisconsin) and one grantee, South Dakota, selected youth ages Among Round 3 states, six (Florida, Indiana, Iowa, Louisiana, Massachusetts, and Rhode Island) selected an adult focus and one, Minnesota, selected youth years of age. In Round 4, two states, Alabama (youth ages 19 24) and Idaho (youth ages 14 24), chose to focus on youth, while Alaska, Connecticut, Illinois, Maine, New York, and Virginia focus on adults Hire DEI State Lead and Disability Resource Coordinators DEI grantees must also hire or designate a DEI State Lead and DRCs. The DEI State Lead provides leadership and administrative oversight. Some also provide TA to participating LWIAs in the program design and implementation of DEI requirements and promising practices. The DRCs provide disability employment counseling and coordination with LWIAs and JSWDs. DRCs also coordinate TTW activities and trainings for AJC staff. DEI grantees have been encouraged to consider former DPNs as a resource to fill [the DRC] position in participating LWIAs Ensure American Job Center Accessibility The accessibility of the AJCs is a requirement under WIA. WIA reformed Federal employment, training, adult education, and vocational rehabilitation (VR) programs by creating an integrated system of training and education services for adults, dislocated workers, and youth. All services provided by AJCs must comply with Federal regulations that define accessibility. These include Section 504 of the Rehabilitation Act, Title II of the Americans with Disabilities Act (ADA), and 8 DEI Rounds 1 4, p.15 SGAs. 11

12 Section 188 of WIA. 9 To meet this requirement, LWIAs must demonstrate that they have a systematic accessibility monitoring system in place that assesses physical, programmatic, and communications accessibility Participate in Ticket to Work as an Employment Network Grantees are required to implement the TTW Program and become a local EN at the Local Workforce Investment Board (LWIB) level or have access to a state-level administrative EN. DRCs and AJC counselors located in LWIA treatment sites are required to recruit Ticket holders, allow Tickets to be assigned to their LWIAs, and provide employment and training services to TTW customers Implement Two of the Six Service Delivery Strategies DEI grantees are required to select at least two of the following service delivery strategies: (1) Integrated Resource Teams (IRTs); (2) Blending and Braiding Funds/Leveraging Resources; (3) Customized Employment; (4) Self-Employment; (5) Guideposts for Success; and (6) Asset Development Strategies Integrated Resource Teams: IRTs originated in the DPN Initiative when DPNs were charged with bringing together various partners from different agencies to collaborate in serving job seekers both in service delivery and in leveraging funds. According to NDI, the national TA provider for DEI, an IRT brings together private and public sector representatives at the local, AJC community level. It improves communication and collaboration which results in enhanced coordination of services and supports for an individual job seeker with a disability. 11 The focus on the individual is paramount and differentiates IRTs from interagency committees, which collaborate on systems-level improvements. 2. Blending and Braiding Funds/Leveraging Resources: This requirement refers to the contribution of funds from two or more state and/or Federal agencies toward the job seeker s goals in education, training, and/or job placement. Blending refers to arrangements that pool funds from multiple sources and make the funding streams indistinguishable. Braiding also refers to pulling together resources from different sources, but keeps the funding streams clearly separated. The use of funds from multiple C.F.R. 37; U.S. Department of Labor. (n.d.). Equal employment opportunity: Disability. Retrieved from Partnerships and Collaboration was included in the list of service delivery strategies in for Round 1 states. In the , , and SGAs, this strategy was made a requirement. Partnerships and Collaboration means the coordination of various partners who are involved in the employment outcomes of JSWDs and are successfully participating in the grantee activities at any or all levels of the workforce development system. The use of partnerships is important, as together, government agencies, community organizations, and nonprofits can better assist in improving the job and earnings outcomes for JSWDs seeking services. Collaboration is the more intensive form of partnership, as it includes joint planning and shared resources, as well as shared funding and accountability. NDI Consulting, Inc. (2011). Integrated resource teams frequently asked questions (FAQs): Strategic service delivery component Disability Employment Initiative [PowerPoint slides, pg. 4]. 12

13 sources is believed to provide more effective and personalized services to the job seeker, as different funding streams can address the differing needs of JSWDs. 3. Customized Employment: Customized Employment uses a flexible and individualized process for matching employer job descriptions to job seekers, one job seeker and one employer at a time. The foundation of Customized Employment is built around job seeker exploration commonly known as Discovery, leading to the development of an individualized employment profile. Among the components of Customized Employment, ODEP has highlighted task reassignment, job carving, job sharing, and self-employment. Task Reassignment allows for certain job tasks to be reassigned so that a new employee can focus on primary job responsibilities germane to her/his skill set. Job Carving is when there is a modification in the job description, thus reducing the number of responsibilities from an existing job description. Job Sharing allows two or more workers to share the responsibilities of one job based on each worker s strengths. 12 Customized Employment practices have been shown to lead to the successful employment of individuals with significant disabilities and multiple barriers to employment, although it is not only beneficial to these groups. 4. Self-Employment: Self-Employment, often used as a form of Customized Employment, allows individuals more flexibility in terms of when they work, where they work, and what type of work they do. Over the years, the workforce development system has introduced resources to train and assist individuals in starting their own businesses, and DOL has supported Self-Employment grants. ODEP funded Self-Employment programs in four states called START-UP/USA. Self-Employment may be particularly beneficial to individuals with significant disabilities, such as Social Security Administration (SSA) disability beneficiaries. This is because of the availability of work incentives and the ability to retain benefits, including health care benefits, while obtaining income through Self-Employment. 5. Guideposts for Success (Required for all Youth Grantees): All DEI grantees with a youth focus must include one or more of the Guideposts for Success in their grant implementation plan. The Guideposts include research-based activities in education and career development that can have a positive impact on all youth, including youth with disabilities. 13 The Guideposts are based on the following principles: 1. High expectations for all youth, including youth with disabilities; 2. Equality of opportunity for everyone, including nondiscrimination, individualization, inclusion, and integration; 3. Full participation through self-determination, informed choice, and participation in decision-making; U.S. Department of Labor, Office of Disability Employment Policy. (n.d.). What is customized employment? Retrieved from U.S. Department of Labor, Office of Disability Employment Policy (n.d.). Youth: the Guideposts for Success. Retrieved from 13

14 4. Independent living, including skills development and long-term supports and services; 5. Competitive employment and economic self-sufficiency, which may include supports; and 6. Individualized, person-driven, and culturally and linguistically appropriate transition planning Asset Development Strategies: Asset Development services can help JSWDs navigate and take full advantage of a variety of benefits, programs, and incentives that are available to them. JSWDs can benefit from SSA work and tax incentives that reduce some disability and work-related costs, or even incentives provided to JSWDs to start their own businesses. 15 Programs include housing, childcare assistance, health care, nutrition, and other areas not directly related to employment. Asset Development programs are not funded through WIA funds, but instead by a variety of sources at different levels of government or other organizations. 2.2 Selection of Service Delivery Strategies by DEI Rounds 1 4 Grantees As noted, DEI grantees are required to select a minimum of two service delivery strategies. As shown in Exhibit 4, all grantees selected IRTs and Blending and Braiding/Integrated Resources, as these service delivery strategies are foundational to the DEI. The percentage of grantees that selected Customized Employment varied from round to round. More than three-quarters (78%) of Round 1, none of Round 3, and less than two-thirds (57%) and one-quarter (25%) of Round 2 and Round 4 grantees, respectively, selected Customized Employment. Selection of the Self- Employment service delivery strategy followed a similar pattern of high interest among grantees in earlier rounds, but decreased interest in later rounds. In Round 1, 56 percent of states selected Self-Employment, while 86 percent of Round 2 grantees selecting this strategy. This is compared to 17 percent of Round 3 and 25 percent of Round 4 grantees selecting this strategy. The Guideposts for Success are required of youth grantees; the selection of this service delivery strategy simply mirrors the number of youth grantees in each round. Asset Development took many forms across the four DEI rounds and, along with benefits planning, was seen as an integral part of recruiting Ticket holders and assigning Tickets. Two-thirds (67%) of Round 1, 86 percent of Round 2, 33 percent of Round 3, and 75 percent of Round 4 grantees selected Asset Development as one of their DEI service delivery strategies National Collaborative on Workforce and Disability for Youth. (n.d.). Key principles to remember. Retrieved from U.S. Department of Labor, Office of Disability Employment Policy. (n.d.). Financial education, asset development, and work and tax incentives. Retrieved from 14

15 Exhibit 4: Selection of DEI Service Delivery Strategies by Grant Round 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% 100% 100% 100% 100% 100% 100% 100% 100% IRTs Blending and Bracing 78% 57% 0% Customized Emp. 56% 86% 25% 25% 33% 29% 17% 17% 13% 67% 86% 33% 75% Self-Emp. Guideposts Asset Dev. R1 R2 R3 R4 Section DEI Evaluation The DEI Evaluation is a comprehensive implementation/process, outcome, and impact study. The implementation component includes the collection of information on systems change and the strategies implemented by each of the DEI grantees. The outcome evaluation provides findings on customer-level outcomes, while the impact evaluation focuses on comparing results between DEI sites and control sites. The implementation/process evaluation also documents the strategies implemented by treatment LWIAs and obtains stakeholder assessments about their value. The DEI Evaluation is designed to address a variety of research questions pertaining to the process and outcomes of the DEI intervention. The report examines various quantitative and qualitative research questions, which apply to participants in the DEI adult states and participants in the DEI youth states. The list of qualitative research questions for the DEI Year 4 Interim Synthesis Report includes the following: To date, Social Dynamics has prepared four DEI Interim Synthesis Reports. Each report includes qualitative information on program start-up and implementation while emphasizing the key challenges, promising practices, and events of the most recently completed DEI grantees. For example, the DEI Year 1 (2011) and DEI Year 2 (2012) Interim Synthesis Reports focused exclusively on program implementation. At the time, the Office of Management and Budget (OMB) package for the evaluation had not been cleared, and Social Dynamics was required to collect only information that would document the implementation of DEI requirements, service delivery strategies, and the roles and structures of state and LWIA activities. The OMB package cleared in February This allowed for the collection of Workforce Investment Act Standardized Record Data (WIASRD) and Wagner-Peyser (WP) data and more extensive interviews with DEI and AJC staff and partners. The DEI Year 3 Interim Synthesis Report (2013) also included quantitative process and outcome data for Round 1 15

16 1. What challenges did the grantees encounter implementing TTW? How were these issues resolved? 2. What TTW promising practices have the DEI grantees identified? 3. What are the roles and responsibilities of the DRC? 4. How have benefits planning services played a role in the employment preparation of DEI customers? 5. What service delivery strategies were selected by the grantees to achieve the objectives of the DEI (i.e., IRTs; Blending and Braiding Resources; Customized Employment; Self- Employment; Guideposts for Success; Asset Development; and Partnerships and Collaboration)? In addition to the five questions above, the list of qualitative research questions for the final DEI report, due to be completed in , will include the following: 6. What are the challenges that need to be addressed to improve the employment outcomes of individuals with disabilities? 7. What system changes need to be made to overcome these challenges? 8. What are the specific challenges grantees encounter serving JSWDs? 9. How has having DEI funds and programmatic requirements for integrated services affected the level of services or systems integration? The list of quantitative research questions is grouped into the following topics: 17 Adult and Youth Processes and Outcomes Utilization of AJCs by DEI customers. How many DEI customers utilize job search and skill development services funded by WIA? What percentage exits the program? What percentage is still active in the program? Entry rate of JSWDs into the WIA program. Has DEI affected the entry rate of JSWDs into the WIA program? Exiting from WIA. How many DEI customers exit from the WIA program and when did they exit? What percentage of customers exit and are subsequently employed or (for youth only) placed in an educational program? Does the exit rate of DEI customers differ significantly between LWIAs randomly assigned to the evaluation s treatment and control groups? 17 grantees. The current report is the DEI Year 4 Interim Synthesis Report. It includes qualitative findings as well as pooled process and outcome data. There are some additional research questions pertaining to subgroups of JSWDs who receive SSA disability insurance (e.g., Supplemental Security Income and Social Security Disability Insurance [SSDI and SSI]). These questions pertain to the types of disabilities among the DEI customer population, the various disability benefits received by these individuals, and the extent to which JSWDs registering at AJCs self-identify as having a disability. At this time, we will not include these SSA-related issues in our list of active research questions because they require data from SSA, which will not be made available to the evaluation team until the final year of the evaluation project. 16

17 Adult Outcomes Employment rate after exiting from WIA. What percentage of DEI customers become employed in their first quarter after exit from WIA services? Do treatment and control sites differ in the share of exiters who obtain employment? Employment retention rate. What percentage of DEI customers employed in the first quarter after exit continues to be employed in the third quarter after exit? Does the percentage of exiters with employment retention differ between DEI treatment and control sites? Earnings. Among JSWDs who become employed, what are their average earnings during the first, second, and third quarters after exit? Do the average earnings in these quarters after exit differ between the DEI treatment and control sites? Youth Outcomes Placement in employment or education. What percentage of youth obtains employment or is enrolled in postsecondary education, advanced training, or occupational skills training in the first quarter after exit? Does the placement rate differ between the DEI treatment and control sites? Attainment of a degree or certificate. Of the youth enrolled in education, what percentage attains a diploma, GED, or certificate by the end of the third quarter after the exit quarter? Does the degree or certification completion rate differ between the DEI treatment and control sites? Literacy and numeracy gains. Of the out-of-school youth who are deficient in basic skills, what percentage advances one or more educational functioning levels? Do the literacy and numeracy gains differ between the DEI treatment and control sites? This report addresses implementation evaluation research questions 1 through 5, relying on extensive qualitative interviews with Round 1 and Round 2 grantees at state and local levels. Additionally, we address the quantitative research questions through process and outcome analyses using WIASRD extracts. The quantitative results in the main text of this report are based on pooled analyses of grantees from Rounds 1 and 2; separate appendices present findings for Round 1 grantees and for Round 2 grantees. Although Round 3 ended in September 2015, data for those grantees are not yet fully available; Round 4 will end in The final impact evaluation will be conducted in early Qualitative Methodology The DEI Evaluation includes two types of qualitative data collection activities: (1) annual site visits to a random sample of DEI grantees and (2) telephone interviews with DEI grantees not included in the annual site visit sample. Annual Site Visits and Telephone Interviews Beginning in 2010 through 2015, Social Dynamics conducted site visits to a random sample of DEI grantees. Grantees not included in the site visit sample participated in telephone interviews. 17

18 In each of the randomly assigned LWIAs participating in the DEI Evaluation, either on-site or telephone interviews were conducted with the DEI State Lead, DRCs, Workforce Investment Board (WIB) directors, AJC managers, AJC staff members, and agency partners, providers, and employers. The site visits were tailored to fit each grantee as well as each DEI or control site. Due to variations across sites, the number and types of individuals interviewed varied. The general domains that were investigated through collecting and analyzing site visit data included: the status at baseline and the systems changes in grantees workforce development systems at follow-up; implementation of DEI grant requirements; implementation of grantees selected DEI service delivery strategies; and program implementation challenges. The study team developed respondent-specific questions that were asked in the interviews. The site visits included three- to four-person site visit teams that were on-site for three to five days. The first step in preparing for the site visits was to review the existing evaluation instrument and site visit plan that were designed as part of previous ODEP work on customized employment, along with site visit protocols developed by the study team for other past work involving site visits to LWIBs and AJCs focused on serving JSWDs, and adapting them to the specific evaluation questions associated with the DEI Evaluation. Other relevant materials were also analyzed and reviewed to prepare for the individual site visits, including grantee applications, program descriptions, and current reporting instruments and reports. Based on this review and in consultation with the Contracting Officer s Representative (COR) and other key ODEP/ETA staff, the team created a set of site visit protocols specific to the proposed study. Quantitative Methodology The quantitative analyses found in this report are based on an analysis of quarterly WIASRD extracts obtained from the various DEI states. The current report analyzes data obtained from the Round 1 and Round 2 DEI states covering the period from October 1, 2011, through September 30, For the final report, Social Dynamics will pool data from Rounds 1 4 of the DEI and will conduct analyses of various programmatic outcomes and impacts. Our evaluation approach incorporates a longitudinal design in which we identify groups of individuals in the treatment and control LWIAs, track their program-related activities and strategies, and compare their outcomes to assess the DEI s impacts. Random assignment of LWIAs in Rounds 1 2 was conducted in 2010 and 2011, respectively. Random assignment increases the likelihood that on average, treatment and control sites (LWIAs) would have been similar in the absence of the DEI. This, in turn, makes it more likely that any significant differences in system- and customer-level outcomes can be attributed to the new service delivery approaches used in the DEI treatment sites and not in the control sites, rather than to preexisting differences between LWIAs. The DEI was implemented in four rounds. Each yearly synthesis report builds on the previous reports until the final synthesis report (due in 2018). The final synthesis report will include pooled data from all DEI grantees. It is important to keep in mind which states are included in each yearly report and the restrictions in the data, which lead to exclusions from certain types of analyses. These are discussed below: 18

19 DEI states included in the Year 4 Interim Synthesis Report. This report analyzes data from 16 states, 9 of which were funded in Round 1 and 7 of which were funded in Round 2 of the DEI. Of these 16 states, 12 targeted an adult population, and 4 states focused on serving a population of youth ages 14 to 21 years. DEI states excluded from the outcome and impact analyses. For the reasons discussed above, the DEI project uses a randomized controlled trial (RCT) evaluation design to assess the impacts of the initiative. However, situations arose whereby some states were unable or unwilling to randomly assign LWIAs to these two groups. Because this would have violated one of the central requirements of the evaluation design, Social Dynamics decided that data from the affected states would be excluded from the main analyses of outcomes and impacts. This exclusion applies to three states: Delaware and Alaska from Round 1, and South Dakota from Round 2. In addition, a small set of LWIAs in Maine and New York was excluded because the LWIAs were purposively selected to be DEI pilot sites. The following considerations informed the data analyses in this report: Pooled data. The tables in the main body of the report involve data from WIASRD that are pooled for all Round 1 and Round 2 states. In the next report, we will add data from Round 3 states to the total. Therefore, the results in this report must be considered as preliminary because subsequent reports will add more data to the analytic file. Stratified random assignment. To maximize the comparability of the treatment and control LWIAs, prior to random assignment for each state we divided the state s LWIA into groups, or strata, according to major characteristics (for example, rural versus suburban versus urban). As described in more detail later, implementation of stratification for each state varied, depending upon characteristics or administrative issues of particular relevance to the state. Once we had defined the strata, we randomly assigned half of the LWIA within each stratum to the treatment group and the others to the control group. In our analysis, we use weights to account for differences between groups in the distribution of customers across strata within each state, which arose from chance variation in the sizes of the specific LWIAs that were randomly selected to be in the treatment group. In addition, we account for the clustered design of the evaluation, which arises from fact that LWIAs rather than customers were randomized, when assessing the statistical precision of estimates for the treatment and control groups. JSWDs included in the analysis. The analyses in this report focus on DEI customers in the pilot sites for the initiative and DEI-like customers in control sites who enrolled in WIA during each state s two-year DEI grant operation period. Both groups are defined as job seekers who participated in WIA programs for more than one day and self-declared as having a disability. In states with an adult focus, the analysis also concentrates on WIA customers who received intensive services or training. A select set of analyses also uses data on customers who enrolled in WIA before the start of 19

20 each state s DEI grant operation period began to better understand changes that occurred during the grant period. Longitudinal analysis. We use the quarterly WIASRD snapshots provided by states to track the service receipt and outcomes of customers longitudinally over time after they entered the WIA program. For example, we analyze the percentages of customers who exited (or exited to employment) in each of several quarters after entry. Because the last WIASRD extract provides a snapshot of customers participation data near the end of the grant operation period, we have progressively less follow-up information about customers entering later during that period. We address this in our longitudinal analysis of outcomes by using statistical methods that account for such censoring of follow-up data. In addition, we focus the impact analysis on customers entering during the first DEI operation year, for whom we can consistently observe at least a few quarters of post-entry exit outcomes. Analysis of post-exit outcomes. In some analyses, we focus on exiters only and examine employment retention and earnings outcomes during the first quarters after they exit. Given the available data, the sample for these analyses consists of customers who enrolled in the first four quarters of the DEI grant operation period and also exited during that period. As discussed more extensively in Section 9, because these results are based only on the subgroup of early exiters (rather than all entrants) they are not based directly on the randomized evaluation design. Therefore, comparisons of exiters outcomes might not reflect unbiased impacts of the DEI. Nonetheless, this is a valuable exploratory analysis because it provides tracking information about JSWDs who are exiters from WIA, which aligns broadly with DOL s Common Measures. 3.1 DEI Evaluation Logic Model The DEI Evaluation logic model was developed in five phases, beginning with the first iteration in fall 2010, followed by subsequent revisions in 2011, 2012, 2013, and As Social Dynamics and its partners collected information through site visits, interviews, and focus groups, as well as through the collection of program documentation and WIASRD and WP data, changes to the logic model were made to illustrate the program s design and its evolution. The logic model (Exhibit 5) specifies relationships among situations/priorities, inputs, outputs, and outcomes at the individual (job seeker) and systems levels, and the challenges and facilitators of program implementation. It has five components: 1. Inputs: Investments in the DEI, including staff knowledge, experience, and skills developed through training and TA (T&TA), DEI grant resources and requirements, including grant-funded positions, partnerships, providers, TTW and EN activities, and quality of WIASRD and WP data. 2. Outputs: Products and services provided to JSWDs. Individual-level outputs include: DEI service delivery strategies, activities/leadership of the DEI State Lead and DRCs, TTW activities, and JSWD and staff trainings. Systems-level outputs include: local LWIA engagement, outreach to JSWDs and employers, and partnerships. 20

21 3. Outcomes: Changes in the circumstances (e.g., employment, wages, job retention) of JSWDs that can be attributed to the DEI. Individual-level outcomes include increases in: the number of JSWDs served by AJCs, the employment rate of JSWDs, the disability self-disclosure rate, SSI/SSDI beneficiary registration rate, SSI/SSDI beneficiary public assistance termination rate that is replaced by employment, degree/training completion, and wages. Systems-level outcomes include: increases in partner, provider, and employer engagement, and systems change through successful integration of resources through Blending and Braiding Funds. 4. Program impact: A primary goal of the DEI Evaluation is to identify the impact of DEI activities on employment-related and other outcomes for JSWDs. To accomplish this, a clustered randomized selection procedure is used to assign sites/lwias to serve as treatment sites, which will implement the DEI strategies, or as control sites, which will continue their normal operations without the additional services. Administrative data (WIASRD and WP) and the DEI Data Elements are used to measure program impact Challenges and facilitators: These can be either exogenous or endogenous factors that affect program inputs, outputs, and outcomes. Exogenous factors are external to the DEI. For example, a state hiring freeze that delays the hiring of the DEI State Lead and/or DRCs is an exogenous factor. One of the most important exogenous factors is the capacity of participating LWIAs to implement the DEI. Capacity is defined as having the staff expertise, coordination, and management systems in place at the time of the commencement of the DEI. Other exogenous factors include: quality of evaluation data submitted by grantees, LWIA unemployment rates, types of industries located in the LWIA, quality of local transportation systems and availability of other support services, population density, local politics, and institutions. Conversely, endogenous factors are internal to the DEI program. These include the quality and effectiveness of DEI State Lead and DRC leadership, staff turnover, LWIA engagement, and prior knowledge of and experience with the TTW Program and EN application process. In addition, JSWD characteristics are likely to influence evaluation findings. These include: educational attainment, work experience, vocational skills, age, and disability. 18 Social Dynamics collects the same information that ETA receives on a quarterly basis from the WIASRD and WP data systems in DEI states. It was decided in 2010 by DOL that the DEI Evaluation will focus only on customers who received WIA intensive and/or training services or WP intensive services and self-disclosed a disability. Social Dynamics does not have a separate data system but rather uses the data ETA uses to measure WIA outcomes and to collect information on the WP program. 21

22 Exhibit 5: DEI Logic Model 3.2 Random Assignment of LWIAs LWIAs selected by DEI Rounds 1 4 grantees were randomly assigned to the treatment and control groups on an annual basis from 2010 through 2014 based on a methodology approved by DOL in The DEI Evaluation is designed to measure the impact of the DEI on outcomes for JSWDs, including (but not limited to) take-up of services implemented through the DEI and other WIA-funded services, employment, wages, and employment retention. Random assignment will tend to yield treatment and control sites that would, on average, have been similar in the absence of the DEI. This increases the likelihood that differences in system- and customer-level outcomes between the sites with and without DEI services can be attributed to the DEI itself, rather than other factors that systematically differed between the two types of LWIAs. To improve the odds of such factors balancing out across groups, participating LWIAs in most states were divided into one or more strata, each containing two or more sites that were characterized as similar. In states where this was feasible, LWIAs within each stratum were then divided between the treatment and control groups by using random assignment. 22

23 Each grantee initially determined which of its LWIAs would be included in the pool of sites to be randomized for the DEI Evaluation. To minimize the potential for chance imbalances between LWIAs randomly assigned to the treatment and control groups, we (1) divided these LWIAs into more homogeneous strata defined by LWIA characteristics thought to be associated with substantially different customer outcomes, if possible, and (2) then randomly assigned LWIA within each stratum to treatment or control with approximately equal probability. This approach ensured that treatment and control LWIAs would be balanced across the key characteristics that define strata. This reduces the potential for chance differences between DEI treatment and control sites, thereby increasing the expected precision of contrasts between customers across the two sets of LWIAs. We determined the strata for each state following discussions with each grantee. Due to substantial demographic, geographic, and programmatic differences across states, the parameters of stratification varied by state. Many state representatives highlighted the substantial differences between LWIAs in primarily urban areas and those in primarily rural areas. In these states, we divided LWIAs into an urban stratum and a rural stratum. In states where the greatest differences were believed to be across regional lines, the regions became the strata. Although the selection of strata is necessarily a judgment call made jointly by the evaluation team and state representatives, random assignment within each stratum facilitates an impact analysis whose estimates can be causally attributed to the intervention. To conduct random assignment, we used the Stata random number generation function to assign each LWIA a random number between 0 and 1. Then, we used a rule that assigned to the treatment group all LWIAs whose random number was greater than or equal to the median in its stratum. In sites with an even number of LWIAs, this ensured an equal allocation of sites to the treatment and control groups. In sites with an odd number of LWIAs, the randomization rule resulted in one extra treatment site, as compared to the number of control sites. This guaranteed that each site in the stratum had an approximately equal probability of being selected as a treatment site or control site. Not all LWIAs were randomly assigned, however, because DOL decided that certain states with only one LWIA (e.g., Alaska, Delaware, and South Dakota) would be exempt from the evaluation s requirement of random assignment. However, Idaho, another state with one LWIA, demarcated the state into two geographic areas, one of which was then randomly assigned to be a treatment site. Consequently, customers from Alaska, Delaware, and South Dakota will be excluded from the impact analysis, while customers from Idaho will be included. In addition, the impact analysis will exclude data from two LWIAs that were purposively assigned by Maine and New York to be DEI pilot sites, but include customers from the other LWIAs put forward by those states to be randomly assigned. 3.3 Data Sources The DEI Evaluation collects qualitative and quantitative data from participating state agencies and LWIAs. Qualitative data includes program documentation, site visits and observations of meetings and trainings, interviews with stakeholders, and focus groups with JSWDs. Customer 23

24 focus groups were conducted in AJCs in Delaware, Maine, New Jersey, Rhode Island, and Virginia. Quantitative data includes quarterly uploads from state WIASRD and WP data systems and the DEI Data Elements, which are collected through two web portals certified by the National Institute of Standards and Technology. From 2011 to 2014, the DEI Evaluation team conducted hundreds of telephone and site visit interviews. Site visit interviews took place in LWIAs and AJCs, as well as in state agency and community-based provider offices. Three types of quantitative data were collected about the experiences and outcomes of customers in DEI treatment and control LWIAs: WIASRD, WP, and the DEI Data Elements (although not all states collected these data elements for all LWIAs). Collectively, we refer to this as the DEI Data System. WIASRD and WP include data on employment and earnings that are provided through each state s unemployment insurance data system. Both are person-level data systems required under WIA and the Wagner-Peyser Act of 1933 (amended in 1998). Grantees are required to submit quarterly WIASRD Report 9090 and Wagner-Peyser Report 9002 A-E, in addition to the DEI Data Elements designed specifically for this evaluation. These two datasets contain information that can be used to measure utilization of DEI strategies, program outcomes, and program impact. WIASRD and WP data include detailed information on services provided, as well as individual outcome measures such as employment and earnings. The DEI Data Elements include information on JSWD work history, participation in SSA benefits programs, self-described barriers to employment, and other factors that are associated with individual employment outcomes Reliance on WIASRD Quantitative analyses in the current interim report were based only on records drawn from the WIASRD extracts supplied by Round 1 and Round 2 grantees. Although these grantees also provided WP data, we did not use it in our analysis for the following two reasons: 1. As noted above, the evaluation focuses only on AJC customers who received intensive services or training. Although intensive services could be provided through WP, discussions between Social Dynamics staff and grantees indicated that they primarily relied on WIA funding to deliver such services. This appeared to be borne out by a preliminary assessment made by Social Dynamics to determine the overlap between WIASRD and WP data files provided by Round 1 grantees. Among individuals whose WP records contained a valid WIB identifier (see below), this investigation revealed that a large proportion of the records in the WP system have a duplicate WIASRD record. 2. The data provided by several states contained limited or inconsistent information about the WIB through which customers received WP-funded services. The WP data files include a field for WIB names, rather than a numeric code like what is used for WIASRD 19 Social Dynamics currently has data sharing agreements (DSAs) with the 31 grantees that have been awarded DEI grants; five states have been awarded two grants. The DSAs include grantee reporting and security requirements and specify parameters for accessing confidential and personally identifiable information (PII). The DSAs do not expire. They continue in perpetuity or until the data is returned to each state and destroyed or Social Dynamics and its partners destroy the data and provide evidence of such to each state. 24

25 reporting. Several states do not fill in this field at all, and some do not fill it in consistently. A separate analysis by Mathematica Policy Research also found substantial rates of missing WIB information in the WP data of several states, including at least one state that received a DEI grant in each round. 20 As a result of these problems in how WIB names are recorded, the WP records cannot be used to augment WIASRD records in a way that would yield a study population of customers that is consistently defined across states. Although we will continue receiving WP data from states, the study population for our analysis of customer outcomes will be based exclusively on WIARSD records. 3.5 Evaluating Systems Change DEI systems change represents an adjustment in the way LWIAs coordinate and allocate resources and revise policies that resolve inefficiencies and sustain promising practices after the grant period. The DEI Evaluation team developed the Systems Change Coding Scheme (SCCS) in to structure the analysis of systems change activities implemented by DEI grantees. The DEI Evaluation team, in conjunction with ODEP and ETA, identified seven systems change domains and indicators that align with the goals and objectives of the DEI (Exhibit 6): 1. Capacity to achieve positive employment outcomes for JSWDs; 2. Coordination and integration of services; 3. Customer choice; 4. Employer support and employer partnerships; 5. Development of new or enhancement of existing practices; 6. Dissemination of effective practices and outreach to the disability and employer communities; and 7. Sustainability. Exhibit 6: System Change Indicators System Change Domains 1. Capacity to achieve positive employment outcomes for JSWDs Definitions This includes the development of functional IRTs and resources, and workforce development systems that are inclusive and change perceptions, attitudes, and understanding of the issues related to disability, as well as improve access and availability of JSWDs to different types of resources (e.g., funds, designated staff for JSWDs, resources, policies, procedures) and services. Designated DEI State Lead and DRC(s) are tasked with managing/coordinating the DEI and fully accessible AJCs. AJCs are TTW ENs. Key Areas of Measurement Accessible AJCs DEI State Lead DRC(s) EN/TTW link IRTs & resources 20 Maxwell, N., Anand, P., & Francis, C. M. (2015). Understanding adult subpopulations served by workforce investment programs. Retrieved from 25

26 System Change Domains 2. Coordination & integration of services Definitions Coordination of employment services for JSWDs that are integrated with existing workforce development systems. Indicators include number and nature of partnerships, availability of cross-agency training, creation of interagency partnerships, shared resources, employer cooperation, access to partner services, IRTs, and innovative approaches to Blending and Braiding Funds/Leveraging Resources. 3. Customer choice Customization of products and services that assist customers in obtaining employment; customer involvement in the design of products and services, the use of existing state financial assistance (e.g., SSI, Medicaid, VR), targeted training, and satisfaction with products and services. 4a. Employer support 4b. Employer partnerships 5. Development of new or enhancement of existing practices 6a. Dissemination of effective practices & outreach to the disability communities 6b. Employer & JSWD outreach Facilitate the recruitment and hiring of JSWDs. Support for employers in forums to discuss hiring needs and the job candidate pool, developing position announcements, pay scales for available employment opportunities, and apprenticeship opportunities. Opportunities for partnerships with local employers that take advantage of available workforce development system services. Identifying, developing, or adapting innovative practices and approaches to the use of IRTs; Integrated Resources; Customized Employment; Self- Employment; Guideposts for Success; Asset Development; and Partnerships and Collaboration. Do these new and existing practices have a positive impact on customer outcomes? Knowledge dissemination and transfer of best practices to employers and workforce development system partners/personnel through website content/ training. Formalized methods of outreach/communication to the disability (youth and/or adult) and employer communities. Communication with employers about AJC services, employing JSWDs, workplace accessibility, and accommodation issues. Opportunities for employers to meet with workforce development staff to learn about arranging apprenticeship opportunities, subsidized employment, and Customized Employment, in addition to various training opportunities. Key Areas of Measurement Partnerships/Collaboration Blending/Braiding of Funds IRTs Shared resources Employer outreach Customer choice Services supported by system Existing subsidies/benefits used efficiently Facilitate recruitment & hiring Forums/opportunities to discuss hiring needs Support in developing position announcements, apprenticeships, & other supportive employment opportunities Arranging apprenticeship & other experiential programs for AJC customers Involvement in LWIB strategic planning Communication protocol for reaching out to employers across a range of industries IRTs; Integrated Resources; Customized Employment; Self- Employment; Guideposts for Success; Asset Development; & Partnerships & Collaboration Communication strategies targeting key audiences: adults with disabilities, youth with disabilities, agency partners, service providers Communication activities Content of communication Opportunities for employers to learn about workforce development services 26

27 System Change Definitions Domains 7. Sustainability Sustainability achieved through system members developing access to alternative sources of funding through interagency partnerships, grants, and legislation. Funding through existing workforce system programs; policy development and policy change that lead to the sustainability of DEI strategies and activities. Key Areas of Measurement Evidence of plans to sustain DEI strategies & activities: Formal agreements, Memorandums of Understanding (MOUs), identified sources of funding, new grants, legislation Exhibit 7 presents the SCCS and Program Maturation Coding Scheme rubrics. The SCCS Rubric is a four-level ordinal scale: (1) Not Implemented, (2) Exploring, (3) Partially Implemented, and (4) Fully Implemented. Each of the seven systems change domains is defined by detailed indicators that are used to assist researchers in determining the level of systems change achieved from levels 1 4. Meanwhile, the Program Maturation Rubric also has four levels: (1) Start-Up, (2) Implementation, (3) Operational, and (4) Sustained (see lower section of Exhibit 7). The purpose of the Program Maturation Rubric is to assess the extent to which DEI grantees met the requirements of the grant and were implemented as they were designed and presented in each state s grant application. Exhibit 7: Program Maturation and Systems Change Model Systems Change Coding Scheme Rubric Not Implemented (1) Exploring (2) Partially Implemented (3) Fully Implemented (4) No. No evidence that this indicator is being met. Not yet. Some evidence that an effort is underway to meet this indicator; this LWIA is actively moving forward on the implementation of this indicator. Yes. Evidence that implementation of this indicator has begun or is partially in place in parts of the LWIA. Yes. This indicator has been fully implemented throughout the LWIA. Program Maturation Coding Scheme Rubric Start-Up: Hiring of State Lead, DRCs. Structure of program confirmed. Evaluation requirements are in place. Service delivery strategies are designed. Outreach to local areas and JSWDs begins. Information on program distributed to stakeholders, including JSWDs. Implementation: Program customers engage in strategizing and defining their own goals and objectives. Service delivery strategies are implemented. All DEI requirements are implemented with fidelity. Operational: LWIAs and DRCs have implemented all program requirements and have a clearly defined agenda that engages the employer community, JSWD community, and entire workforce investment system. Sustained: LWIAs engage the workforce investment system in carrying forward the program created under their DEI grant. They have facilitated strong relationships with partners and have a realistic sustainability plan in place. They have also been able to resolve various challenges that hinder their progress to implementation. Promising practices are sustained after the grant period. 27

28 Section DEI Ticket to Work The section describes the challenges and successes of Rounds 1 2 in addition to information on implementation of Rounds 3 4, including TTW implementation and challenges; design and implementation of the DRC and DEI State Lead positions; and the implementation of the DEI service delivery strategies. 4.1 Ticket to Work TTW is a national program that provides SSA beneficiaries (Ticket holders) with services designed to help them rejoin the labor market and have access to certain work incentives. TTW employment services are provided to Ticket holders by assigning their Tickets to an SSAapproved EN. Ticket holders choose which EN they would like to work with. Once a Ticket is assigned to an EN, the Ticket holder works with an employment counselor to complete an Individual Work Plan (IWP). The IWP states the responsibilities of the EN to provide relevant employment/job readiness services and the Ticket holder to participate in all relevant service and training opportunities, respectively. The IWP includes the Ticket holder s employment goals, services to be provided by the EN to the Ticket holder, the process of withdrawing a Ticket from the EN if the Ticket holder is not satisfied with the services provided, and making changes to the IWP once it is signed by the Ticket holder and EN. A key challenge for many DEI grantees is marketing their EN services to Ticket holders. TTW requires that ENs compete for Ticket assignments as would any commercial business entity by selling products (e.g., information on job availability) and/or services (e.g., skills development training). Exhibit 8 lists the 19 work incentives that are available to SSI and SSDI beneficiaries. 21 Exhibit 8: List of Work Incentives Available to SSI and SSDI Beneficiaries 1. Impairment-Related Work Expenses X X 2. Subsidies and Special Conditions X X 3. Unincurred Business Expenses X X 4. Unsuccessful Work Attempt X X 5. Continued Payment under a VR Program X X 7. Trial Work Period X 8. Extended Eligibility Period X 9. Continuation of Medicare Coverage X 10. Medicare for People with Disabilities who Work X 11. Earned Income Exclusion X 12. Student-Earned Income Exclusion X 13. Blind Work Expenses X SSI SSDI 21 For more information on SSA work incentives, go to: 28

29 14. Plan to Achieve Self-Support X 15. Property Essential to Self-Support X 16. Special SSI Payments for People who Work X 17. Continued Medicaid Eligibility (Section 1619 (b)) X 18. Special Benefits for People Eligible Under Section 1619 (a) or (b) who enter a Medical Treatment Facility 19. Reinstating Eligibility without a New Application X SSI X SSDI 4.2 TTW and the DEI In the DEI Year 3 Interim Synthesis Report, Social Dynamics reported that providing TTW and WIA services within the same AJC was a challenge due to the lack of monetary incentives for participating LWIAs to provide Ticket services and the difference in the level of case management intensity between the two programs. TTW requires a much longer term and is intensive. The time it takes for outreach and to get a Ticket assigned to an AJC, provide benefits planning information, and prepare a Ticket holder for employment is significantly greater than the time it takes to devise an IWP and arrange services for WIA customers. As a result, some stakeholders mentioned that the lengthy process could have an adverse effect on LWIA common measures performance. 4.3 TTW Implementation Challenges In addition to the need for more intensive case management services for Ticket holders, several grantees expressed concerns about the EN application process. Round 1 and 2 grantees experience considerable delays due to changes in the EN application process that added invasive security clearance requirements (i.e., fingerprinting, background checks, collection of Social Security numbers of AJC/EN staff), lost EN applications that were submitted and verified by LWIA staff, and erroneous EN identification numbers provided to LWIAs. As a result, some LWIAs reported having to resubmit their EN application a second time. As a result, Round 1 3 grantee EN applications took on average 1.5 years to complete. A new SSA requirement for the collection of Social Security numbers of staff with access to Ticket holders PII and an extensive background check and fingerprinting requirement created challenges for several treatment sites. For example, one treatment area in New Jersey (Round 1) stopped the application process when informed of these new requirements, which they termed an invasive process; this requirement was discontinued one year later. In California (Round 2), a respondent disapproved of what he felt was a requirement only for AJC staff and did not complete the suitability requirement, while in Louisiana (Round 3) stakeholders reported that the EN application and suitability determination processes were challenging: The entire process of the suitability determination [was] frustrating. When we did the EN application, we had to get the Assistant Director to sign off on it. We did this in November Finally, SSA decided that they re ready to get the ball rolling and they 29

30 sent the information to the Assistant Director instead of myself. The Assistant Director didn t remember anything that was going on, and that caused a two-week setback. Once we got past that, the e-qip application [was] very tedious. 22 We [were] being asked to provide basically a life history of our agency. We got through that, but in order for the process to continue, we had to do fingerprinting. No one informed us that we have to do to get that through SSA or Maximus. It was just a challenging process something we thought was going to be real simple turned into months of headaches. All of this is holding us up. In Massachusetts (Round 3), LWIAs implemented TTW but reportedly got locked in a void by the feds and weren t told about it until September. We were in a category whereby SSA was revamping the program [and could not complete our application], so we are behind seven months into the grant. Communicating with Maximus and SSA was a challenge for most DEI grantees: The SSA office doesn t readily work with us. I ve tried to schedule meetings with my local office and they re not really welcoming us. Locally, I ve found that the SSA office doesn t really know about work incentives. If we worked more closely with them, we could reach our program goals. Another DEI stakeholder commented that some of the SSA staff were not familiar with the basic components of TTW and provided incorrect information to beneficiaries: It s not going to be the program it is now [when the grant is over]. Because we ve done an incredible job of creating a new system, we re trying to drive that home as being sustainable. We have two LWIAs using the cradle-to-grave process for TTW. [One LWIA] is using [TTW] extensively. They love it. It works for a smaller area. The EN process has been difficult. We have had lots of difficulties working with SSA. The local areas do not feel that Ticket is the be-all/end-all that will allow them to [sustain the DRCs and service delivery strategies]. The extra revenue is nice, but it s not going to make this thing sustainable. Several Round 2 stakeholders reported having to submit their applications multiple times to SSA. Three months after submitting the first EN application, one state was told that SSA changed the application form and required them to start over and complete the new application. While the state submitted the new application that was approved three months later, during the same period the suitability determination process had also changed and was not completed by SSA during the same time period. The state inquired about the delay in the suitability determination process and was told by an SSA representative that the process was revised to require that EN staff that would have access to PII would need to submit their Social Security numbers to SSA. Due to this change in protocol, the DEI State Lead decided to withdraw the state s EN application. The result of the delay and change in EN application and suitability determination requirements resulted in two of three DEI treatment LWIAs in the state not being able to implement TTW. The state s third treatment site completed its own EN application and suitability determination and accepted 76 Tickets during the grant period. 22 e-qip is a web-based system used for conducting background investigations for Federal security, suitability, fitness, and credentialing purposes: 30

31 4.4 TTW Improvements In 2013, we reported that the lack of financial incentives (for the treatment LWIAs) for implementing TTW was an obstacle to full implementation of the program in the AJCs. At the time, in the case of the state EN model, in which a state agency provides EN administrative services for treatment LWIAs, payments to AJCs for meeting milestones and outcomes were being retained by the state rather than used as a financial incentive for participating treatment LWIAs. The state EN model offers the advantage of reducing the administrative burden of processing Ticket revenue, but it also restricts treatment LWIAs from receiving Ticket revenue, as there was uncertainty about whether or how the TTW milestone payments would be provided to the LWIAs that serve Ticket holders. In 2011, with the exception of Virginia and New York (Round 1), Ticket revenue remained with the state EN. By 2012, DEI State Leads and DRCs recognized the need to distribute Ticket revenue to the LWIAs as an incentive for implementing the program. By the end of the Round 1 grant period, all state ENs had made arrangements for the distribution of Ticket revenue to their treatment LWIAs. By 2014, some state ENs distributed all Ticket revenue to treatment LWIAs, while others set aside a processing fee of 1 5 percent to cover the administrative costs of operating the EN and transferred the remaining funds to treatment LWIAs. In addition, it appears that there has been a significant cultural shift in many of the treatment LWIAs toward the acceptance of TTW as a secondary source of revenue for the local areas and as an opportunity to serve customers with significant disabilities. In 2010, we reported that the structure and culture of the AJCs, their historical reliance on WIA staff-assisted core, intensive, and training services, and limited experience with incentive-based employment programs would make the implementation of TTW challenging. We also reported mixed perceptions of TTW (among DEI stakeholders) and moderate success with beneficiary outreach, Ticket assignment, and job placement. By 2012, treatment LWIAs had begun to show marked improvement in terms of TTW implementation and their understanding of the program, beneficiary recruitment activities, and revenue generation. When asked about the implementation of TTW in 2014, the final year of the Round 3 grant period, Indiana DEI stakeholders were optimistic about the program: We re absolutely, positively thriving in one of our three treatment LWIAs. In that one, we have a hefty program in operation. This is the LWIA that became its own EN. They ll sustain their DRC with Ticket revenue. Among Round 3 grantees, four DEI states whose LWIAs had not yet implemented TTW (Indiana, Iowa, Louisiana, and Massachusetts) implemented the program during the first year of their DEI grants; Florida treatment LWIAs became ENs prior to the implementation of DEI. 4.5 Identifying and Recruiting TTW Customers Identifying and recruiting TTW beneficiaries is a key part of the program, and being able to attract Ticket-eligible customers to an AJC, when most Centers had not previously provided outreach services, requires thoughtful and deliberate planning. 23 All DEI grantees use mailers 23 Youth states in general did not focus on the TTW Program, as their target population was oftentimes not Ticket eligible (under the age of 18). 31

32 sent to TTW beneficiaries in their local areas. AJC mailers are used by ENs to inform beneficiaries that the Centers serve Ticket customers and invite them to benefits and work incentives events and provide information about job availability and local industries. The information describes the program and the EN services available. Many LWIAs also use SSA Beneficiary Referral CDs (BRCDs), which include data on local Ticket holders that is encrypted and provided to ENs upon request. The BRCD includes state VR agency (SVRA) service information, Tickets available for assignment by ZIP Code, and information on EN services. Despite their efforts, most DEI stakeholders reported mixed results with mass mailers. In some LWIAs, Ticket holders responded to the mailers by visiting their local AJCs, but overall the numbers of customers following up in this manner were low as no LWIA reported a large number of Ticket assignments following the mailers. A DRC commented, We have put up banners, we have workshops, and we do blasts. We can send the to 800 people and have four people show up. One Round 2 site relayed the situation thusly: In our area, we just don t have the volume. We had over 3,000 beneficiaries on our monthly beneficiary CD. I filtered by earnings, by those aged 18 65, and got 208. Out of the 208, I got just five calls. It is very discouraging. And those who do call aren t interested in working full-time and ending their disability benefits. They want to work part-time and keep their Social Security benefits. Individuals who are on SSDI and disability benefits they probably could survive on disability benefits out here we have a very low cost of living. They re just not interested in working full-time and getting off benefits. This could be unique to our area and population. In addition to the BRCD, many LWIAs are now using SSA s e-processes, including e-data Share, which searches for customers who have self-disclosed a disability during AJC enrollment and have an unassigned Ticket. E-Ticket assignment allows ENs that have completed an IWP with one or more Ticket beneficiaries to automatically have their Tickets assigned to an AJC on the 15 th and 30 th day of each month. Automated payment to Ticket beneficiaries is accomplished through e-pay, which creates a WIB payment file with the Social Security numbers of Ticket beneficiaries who have quarterly earnings at a trial work level or above. E-Pay automatically pays beneficiaries with additional paperwork submitted by the EN. Alaska (Rounds 1 and 4), Massachusetts (Round 3), New York (Rounds 1 and 4), Virginia (Rounds 1 and 4), Iowa (Round 3), and Connecticut (Round 4) have either begun piloting or are/were using one or more of the e- Processes to identify Ticket beneficiaries, automatically assign Tickets, and/or pay Ticket beneficiaries through e-pay. Collaboration with organizations in the community that serve individuals with disabilities was an avenue pursued in multiple sites to get the word out that the AJCs offered TTW services to disability beneficiaries. Following state grant leadership orientation, New York DRCs incorporated outreach to disability-focused organizations into their day-to-day schedules. The DRCs often had prior experience working with individuals with disabilities and were familiar with the local organizations. The goal of this outreach was to educate the community about the TTW Program and make it known that the AJCs were now offering EN services to TTW 32

33 customers. These organizations, both governmental and non-governmental, would have the AJC in mind if they assisted someone on the Ticket seeking to get back to work. For example, Rhode Island (Round 3) had success with an intensive outreach initiative that targeted community-based organizations. The Supervisory Lead DRC did presentations to 15 communitybased agencies to discuss services to be provided under DEI, including TTW. The state started working with Ticket holders before finalizing the EN application/suitability determination process so that it could assign Tickets on the first day the EN application was approved. In the first year of the grant period, Rhode Island assigned 95 Tickets that they attributed to the continuous building of relationships in the community and word of mouth. A common challenge that has been reported by DEI stakeholders in previous DEI interim synthesis reports is the lack of understanding and commitment encountered by many Ticket holders to go back to work. While the AJCs can help these individuals find employment, they might not be good candidates for TTW and its associated milestone payments. DRCs learned during the implementation of TTW in their LWIA that: Outreach [to public and community-based agencies] is so important because if [other] agencies don't know how TTW works they can t effectively refer to people [the AJC]. We do get a lot of referrals and walk-ins due to word of mouth. But we also see customers who are not on SSI/SSDI [and many who are Ticket holders but] aren t workready. We still refer them to the resources they need. Not everyone who comes in here is someone who doesn t want to work and thinks they can get something for free. We re getting a better pool of candidates who want to work. Over half of our Ticket enrollees are co-enrolled with WIA. They ll be going into sustainable work and come off their benefits [at some point]. The outreach we do to TTW beneficiaries is important because it helps them understand disability and how going to work affects benefits. DEI grantees also use WISE (Work Incentive Seminar Events) so people [can] hear it straight from the SSA representatives. Also, they can say something about disability determination and DEI why we re here and our goals in the community. However, numerous DEI stakeholders have reported that outreach for WISE events has resulted in very low attendance and that outreach to local service providers and automatic Ticket assignments are more likely to boost Ticket assignment rates than traditional marketing strategies. Tennessee (Round 2) used the BRCD to do a mass mailing and planned to cross-reference the beneficiary information with the WIA data system. The DRCs hit a snag when they could not access the DOL data system since they are not DOL employees but rather SVRA employees. In Wisconsin (Round 2), the state received feedback from the AJCs indicating that JSWDs targeted by the mailers seemed to have mischaracterized the program. Their understanding was that TTW would compensate them for their lack of job readiness with an additional financial subsidy. Because mailers cast a wide net on a population that may not be seeking immediate employment, more targeted outreach efforts, or the use of filters and screens for the BRCD, were used by some LWIAs to good effect. For example, some sites contacted beneficiaries by telephone to discuss how TTW would affect their SSA benefits. 33

34 We have also seen a pattern of suburban and urban LWIAs reportedly having vibrant TTW programs, while those in rural areas struggle to recruit and register Ticket holders. One Indiana (Round 3) stakeholder reported having assigned 20 Tickets, but some individuals who have shown interest in [TTW] are either not eligible or are not ready Out of about 18 people who have attended the workshops, about three or four went online and had already started learning about TTW. Iowa (Round 3), a generally rural state, appears to be an anomaly. Iowa has done well with its TTW Program in its first year of operation, as two treatment LWIAs implemented automatic Ticket assignments, which generate lists of Ticket holders and match them to the workforce data available at the AJC. Through this process, JSWDs can automatically have their Tickets assigned to an LWIA, or a Ticket can be transferred if requested by a beneficiary. Iowa also implemented an integrated WIA membership system that asks each customer to become a member of the AJC through an online registration process. As a member, Ticket and non-ticket holders can gain access to AJC, TTW, and SVRA services through the AJC; Iowa also promotes the National Career Readiness Certificate, which reportedly 85 percent of AJC customers complete. Several LWIAs screen Ticket holders to determine if they are interested in going back to work. California (Round 2) reported that [each customer] gets an overview of all our [EN] services and completes a prescreening form. On the prescreening form it asks if they are receiving SSI or SSDI. The prescreening form helps us determine funding streams and identify who is workready. Alaska (Round 1) opted to be more selective in terms of the Tickets they accept in order to ensure that their investment in case management and job readiness services would eventually generate revenue for the LWIA. In two areas in Ohio (Round 2), staff indicated that a screening process was necessary, as a number of individuals were assigned Tickets and received services but did not appear to be willing to go back to work. Another Ohio LWIA stakeholder mentioned having placed an individual three times with different jobs only to have them quit each job within a short period of time. Stakeholders in Massachusetts (Round 3) indicated that while it was fine to use some discretion in screening customers, the fact that the AJC could discontinue TTW made intensive screening unnecessary. If the Ticket holder was not following through with the plan agreed upon in the beginning, the AJC could drop him/her. 4.6 Screening TTW Beneficiaries More experienced EN staff, including a few who worked at AJCs prior to DEI, reported that one of the lessons they learned about running the TTW Program is that not every Ticket-eligible customer who enters the AJC should have his/her Ticket assigned. Some Ticket beneficiaries are reluctant to return to work at all, and many are reluctant to work full-time due to the risk of losing their medical benefits. Although most DEI stakeholders support the use of TTW and several sites have been moderately to very successful in terms of implementing the program, some have indicated that Ticket beneficiaries who want part-time employment may not be appropriate for TTW because of the cost-to-benefit of providing EN services that only generate relatively small milestone payments. In order to gauge the level of interest of Ticket beneficiaries who assigned their Tickets to an LWIA EN, several Round 1 and Round 2 sites (Round 1: Alaska, Illinois, New York, Virginia; Round 2: California, Ohio, Washington, Wisconsin) developed TTW screening forms designed to determine if an individual was committed to the goals of TTW (long-term, full-time employment). 34

35 The reticence on the part of LWIA staff to work with all Ticket holders is due in large part to the lack of work readiness of Ticket holders and the length of time it takes to generate milestone and outcome payments. The four phase 1 milestones can be achieved through part-time work (about 26.5 hours per week) at a minimum wage of $8.00 per hour. 24 The first milestone requires beneficiary gross earnings to be greater than or equal to $780 for one calendar month. To achieve this milestone, a newly re-employed beneficiary would need to work about 106 hours in one month or 26.5 hours per week to earn enough to generate a milestone payment to an EN. Milestones 2 4 are similarly structured but extend the length of the employment period: milestone 2 requires about 318 hours in 3 months within a 6-month period; milestone 3 requires 637 hours in 6 months within a 12-month period; and milestone 4 requires 955 hours in 9 months within an 18-month period. All four milestones generate about $848 per month and for each milestone achieved, an EN receives $1,377, for a total of $5, Payments to ENs continue in phases 2 and 3 when beneficiaries work at the Substantial Gainful Activity (SGFA) (or full-time) level; SGFA earning levels in 2015 are as follows: $1,090/month for Ticket holders with disabilities other than blindness and $1,820 for blind individuals. 26 Phase 2 EN maximum payments are $4,543 under SSI and $4,284 under SSDI. EN payments increase with outcome payments that total $14,868 and $14,280 for SSI and SSDI, respectively (see Exhibit 9). Exhibit 9: TTW Detailed Milestone Payment Structure (2015 Figures) 27 Performance Payment to EN PHASE 1 SSDI beneficiary SSI beneficiary Milestone 1: Beneficiary gross earnings greater than or equal to Trial Work Level ($780/mo in 2015) for 1 calendar month Milestone 2: Beneficiary gross earnings greater than or equal to $780/mo for 3 months within 6 months Milestone 3: Beneficiary gross earnings greater than or equal to $780/mo for 6 months within 12 months Milestone 4: Beneficiary gross earnings greater than or equal to $780/mo for 9 months within 18 months $1,377 $1,377 $1,377 $1,377 $1,377 $1,377 $1,377 $1,377 TOTAL Potential Phase 1 Payments $5,508 $5, U.S. Department of Labor, Wage and Hour Division. (2015, February 24). Minimum wage laws in the states - January 1, Retrieved from U.S. Social Security Administration, Ticket to Work. (n.d.). The milestone/outcome payment system. Retrieved from U.S. Social Security Administration, Ticket to Work. (n.d.). Outcome payments under the milestone-outcome payment system. Retrieved from 27 Ticket to Work. (n.d.). The Milestone/Outcome Payment System. Retrieved from 35

36 PHASE 2 A beneficiary must earn at or above the applicable SGFA level ($1,800 for Blind and $1,070 for non- Blind) Performance Payment to EN SSDI beneficiary An EN can qualify for up to 11 months of payments at the rate of $413/month SSI beneficiary An EN can qualify for up to 18 months of payments at the rate of $238/month TOTAL Potential Phase 2 Payments $4,543 $4,284 OUTCOME PAYMENTS Available when a beneficiary has earnings above SGFA and the disability benefits cease due to work and earnings An EN can qualify for up to 36 months of outcome payments at the rate of $407/month An EN can qualify for up to 60 months of outcome payments at the rate of $234/month TOTAL Potential Outcome Payments $14,868 $14,280 Total Potential Payout Once Ticket holder has satisfied all milestones and phases $24,919 $24, DEI Grant Timing and TTW Program Self-Sufficiency at the AJC The time frame of the DEI grant poses significant obstacles to the treatment LWIAs in terms of sustaining the program after the grant period. The grant period for Rounds 1 3 is 36 months, including a one-year implementation period, while the Round 4 grant period is 36 months with a six-month implementation period. Both of these time frames, according to several DEI stakeholders, are too short a period of time to allow for the development of a self-sustaining pipeline of Ticket holders and EN payments that can sustain TTW. Several stakeholders expressed concern about the enormous efforts made to get TTW off the ground for limited recompense. Because payments to ENs are structured around the achievement of certain milestones (i.e., phase 2 and outcome payments) rather than on a fee-for-service basis, DRCs reported delays as long as several months before receipt of payments for their services. The four phase 1 TTW milestones combined can be achieved in as little as 19 months or as many as 36 months, yet all four milestone payments generate a total of just $5,508 for the EN. 28 Payments to ENs may continue in phase 2 if an SSDI beneficiary works 11 months at the SGFA (full-time) level. Completion of phase 2 generates $4,543 for the EN. For an SSI beneficiary the work requirement is 18 months at the SGA level, which generates $4,284 for the EN. At this point, an SSDI beneficiary would have had to work a minimum of 30 months (19 part-time plus 11 at the SGFA level) for the EN to receive a phase 2 payment. An SSI beneficiary would have had to work at least 37 months (19 part-time plus 18 at the SGFA level) to generate a phase 2 payment. 28 U.S. Social Security Administration, Ticket to Work. (n.d.). The milestone/outcome payment system. Retrieved from 36

37 After phase 2 is completed, outcome payments are available to the EN if beneficiaries work at the SGFA for a sustained period of time. An SSDI beneficiary would have to work consistently at the SGFA level for another 36 months to generate an outcome payment for the EN of $14,868, while an SSI beneficiary would have to work 60 months for a slightly lower payment to the EN of $14,280. The total time from Ticket assignment to the phase 2 EN outcome payment for SSDI beneficiaries ranges from (19 Milestone + 11 Phase Outcome ) 66 months to (36 Milestone + 11 Phase Outcome ) to 83 months. For SSI beneficiaries, the phase 2 EN outcome payment time line ranges from (19 Milestone + 18 Phase Outcome ) 97 months to (36 Milestone + 18 Phase Outcome ) 114 months. The DEI treatment LWIAs have made considerable progress implementing TTW in a very short period of time. Yet, several stakeholders have expressed concern about the cost-to-benefit ratio of operating TTW. This is largely due to the amount of time DRCs spend providing assistance in the completion of the IWP, benefits planning, career planning, identifying job leads, securing a job placement, and providing ongoing employment support services. DRCs in several states were able to recruit Ticket holders and assign Tickets to their ENs, but the time between completion of milestones and EN payments were significant. One Round 3 DRC reported that: There are 19 Ticket holders and five are working right now. It takes a bit of extra time getting these clients through the intake process. The AJC has a couple of people who have hit milestones 1 and 2, but they have not received any funding yet. It can take up to nine months to get reimbursement. That is a challenge.... The AJC is leveraging funding from other sources to serve these customers, which is probably why there are not many ENs that can use TTW to sustain a position. In order for an LWIA to be successful and self-sufficient using TTW, they would need a steady flow of both incoming Ticket holders and Tickets holders nearing completion of their last milestone or outcome payment. To reach self-sufficiency, the program may need an initial funding source to create and maintain the technical staff needed to run the operation; getting to a self-sustaining level of Ticket revenue takes longer than the 24- (Rounds 1 3) or 30- (Round 4) month operational periods if the LWIA has not completed the EN application process prior to award of the DEI grant. While some of the Round 1 and 2 LWIAs planned to try to continue TTW in the treatment LWIAs, several reported that the amount of time needed to maintain its administrative components and provide intensive case management services to Ticket holders, combined with the potential for long delays in receiving milestone payments, may make it less likely that they will be able to continue with the program after the grant period. In the final months of their grant period, some LWIAs actually scaled back from conducting outreach and Ticket assignment activities due to the uncertainty that the LWIA will continue to provide Ticket services after the end of the grant period. A few DRCs mentioned that they felt uncomfortable accepting new Ticket assignments as the grant entered its final months. As one DRC put it: I didn t want a situation where I recruited a bunch of people and then had to leave them high and dry in the middle of the process with the grant ending soon. I didn t want to leave anybody in a hard spot or add to any negative consumer perceptions about TTW. 37

38 While the DEI grant period may not have given LWIAs enough time to implement and operate Ticket at sustainable levels, interviews with DEI stakeholders who have operated ENs for several years indicated that the program workload decreases as the program matures, especially as the pipeline of Ticket holders reaches a saturation point when the community knows that Tickets holders are served in a certain location that has a reputation for successfully placing them in good jobs. The relatively short time frame of the DEI grant and the multiple delays incurred by Rounds 1 3 due to the EN application and suitability determination processes led one DEI State Lead to advise that future grantees start the EN paperwork before applying for the grant; one can always stop the application if the grant is not awarded. A more realistic time line for reaching TTW self-sufficiency would likely involve a grant operational period of a minimum of 48 or, preferably, 60 months. For Round 4 grantees, ETA/ODEP added six months to the implementation period, which should make a difference in terms of completing the EN application and suitability determination processes before grant operations, but grantees need much more than 24 months of grant operations to develop sustainable TTW programs within the AJCs (See Exhibit 10). Exhibit 10: Self-Reported Ticket Assignments State No. of Tickets Assigned to Treatment LWIAs During Final Quarter ROUND 1 (Qtr ending 6/30/2013) Alaska 14 Arkansas 0 Delaware 1 Illinois 48 Kansas 9 Maine 58 New Jersey 5 New York 145 Virginia 149 Tickets Reported 429 ROUND 2 (Qtr ending 9/30/14) California 170 Ohio 28 Hawaii 8 South Dakota 0 Tennessee 220 Washington 55 Wisconsin 61 Tickets Reported

39 State No. of Tickets Assigned to Treatment LWIAs During Final Quarter ROUND 3 (Qtr ending 3/31/15) Florida 963 Indiana 74 Iowa 901 Massachusetts 251 Louisiana 133 Minnesota 78 Rhode Island 79 Tickets Reported 2, SVRA and Partnership Plus In previous DEI interim synthesis reports, we explored Partnership Plus, which is an arrangement whereby LWIAs, acting as ENs, collaborate with their local SVRA office to provide post-placement job retention services to Ticket holders. SVRA receives the TTW payments associated with job training and placement services, and the AJC receives TTW payments associated with job retention services. Originally described by one DEI stakeholder as the gateway to both WIA and Ticket services for JSWDs, Partnership Plus is still underutilized by DEI treatment sites. When an SVRA customer is placed in a job, there is a 90-day waiting period that begins immediately after job placement. During this time period, SVRA is still the responsible agency for the customer, and the LWIAs/AJCs are not authorized to provide services to SVRA clients before the waiting period has elapsed. As a result, many Ticket holders struggle to make it through the 90-day waiting period; one DRC described the need for support services to continue through the waiting period in order to build a relationship with the Ticket holder: It has to do with the relationship between the AJC and the local SVRA office. If a participant receives services from SVRA they have to wait 90 days post-job placement to transfer a Ticket to an AJC. The local SVRA office can decide to hand over responsibility to the AJC prior to the completion of the 90-day waiting period, but that doesn t happen often. We know they want to see SVRA customers succeed. It takes a relationship between the AJC and the local SVRA office to make this happen. It s about instilling in the DRCs the importance of working with SVRA and its clients so they can meet with SVRA clients during the 90-day waiting period There is an advantage to a beneficiary that assigns his/her Ticket to an EN within 90 days, as doing so means that the Ticket holder will not be subject to the Continuing Disability Review (CDR) process if she/he is working at the SGA level in phase 2. Ticket beneficiaries who are not enrolled in Partnership Plus can also avoid the CDR process if their Ticket is assigned to an EN before a CDR notice is received and timely 39

40 Regarding of the relationship between LWIAs/AJCs and SVRA, numerous DEI stakeholders commented that they want that relationship to be better. I think the challenge stems from the Ticket Program and sharing [those resources]. [SVRA] doesn t want to share the [milestone payments] and will serve the customers themselves and not utilize us [to do that]. But you can t get around not utilizing us because the customers themselves will choose to come to us when they are ready to work. When I originally started the [Ticket] Program [in our LWIA], I went to SVRA myself and I was told to my face, no, we have everything we need here. So I thought we could partner, and they said, no, we don t need a partnership. But you can t get around looking at what the customer s needs are. It s supposed to be about the customer, not the money. Numerous DEI stakeholders commented that the success of Partnership Plus is driven primarily by local relationships forged between LWIA/AJC staff and personnel at the local SVRA office. Development of a solid working relationship with SVRA reduces the need for extensive outreach to Ticket holders and can also reduce the reluctance on the part of SVRA to send all or most of their transitioning customers to an AJC rather than a community-based EN. The collaboration between SVRA and the AJCs was at first difficult to pull off for many LWIAs. DEI grantees that were successful in this regard began by forging relationships with SVRA staff early in the grant period while working on their EN application packages. In some LWIAs, DEI stakeholders reported that SVRA at first resented the AJC focus on TTW, as they envisioned their partnership with DEI to cover a broader group of SVRA customers, not just TTW beneficiaries. SVRA also envisioned working with a broad array of ENs, not just LWIA/AJC ENs. Several Round 1 3 grantees, such as California (Round 2), Indiana (Round 3), Iowa (Round 3), Massachusetts (Round 3), New York (Round 1), Tennessee (Round 2), Virginia (Round 1), Washington (Round 2), and Wisconsin (Round 2), developed productive relationships with their local SVRAs. For other DEI grantees, even though state-level MOUs between the workforce systems and SVRAs were created locally, not all of the SVRA/LWIA partnerships came to fruition. As a DEI stakeholder from one Round 3 site reported: Partnership Plus has been brought up often. [Our situation] is a work in progress. Nothing has been done yet. Becoming an EN didn t really impact our relationship with SVRA. There s still tension between us. But once they come to the table and understand the partnership, the tension will abate. But that can only happen one [local SVRA] office at a time, so we are working hard to make the relationship more amicable. To date, five Round 1 (Alaska, Maine, New Jersey, New York, and Virginia), four Round 2 (California, Ohio, Tennessee, and Wisconsin), and four Round 3 (Florida, Indiana, Iowa, and Massachusetts) grantees have implemented Partnership Plus by establishing nonexclusive agreements with SVRA. Although many Round 1 3 LWIAs expressed interest in exclusive Partnership Plus agreements, SVRA s preference was to maintain competition from the private progress toward SGA employment is being made. If a Ticket is assigned after receipt of a CDR notice, the CDR process will continue regardless of employment status until the next CDR. 40

41 sector EN community. In Ohio (Round 2), a nonexclusive agreement was created between the LWIA and SVRA that states that all ENs, including the LWIA, may compete for Ticket assignments in order to maintain competition among all ENs in the area. In Wisconsin (Round 2), DEI stakeholders are working with SVRA to get them to give us customers who are done with their services. But right now, all they will do is mail out a flyer telling [Ticket holders] about our services and the services of all other ENs in the area. According to a majority of DEI stakeholders, SVRA is not in favor of exclusive Partnership Plus agreements with LWIA ENs because it would create an unfair advantage by excluding other ENs from competing for Ticket assignments. In Alaska (Round 1), the SVRA/AJC relationship was solidified in an MOU between the LWIA and SVRA stating that when an SVRA case is closed, the customer would receive a mailer with contact information on all available national and local ENs. But in a state with fewer EN options than most, there is limited competition for Ticket customers; during the Round 1 grant period, Alaska s MOU with SVRA led to seven Partnership Plus cases. As a result, All SVRA counselors know who is on their caseload so they start having conversations about them about what is next after closure, and then we developed some materials to give to customers. Some offices have the DRC come to sit in on some meetings in VR when a customer is nearing closing so then it is more of a warm hand-off for the customer and establishes trust. Then, the Job Center can help with the employment process. In New Jersey, the competition from private sector ENs is much stronger but, in part because the host agency for the DEI grant is the SVRA, there is a preference for the LWIA EN to handle Partnership Plus cases. For example, one New Jersey DRC explained that: We can help Ticket holders with training programs and additional funding or whatever they need and then they work with SVRA and wait for the 90 days for [the case] to close and for us to then pick them up. We re lucky because I m [co-located with] the [local] SVRA office we work hand-in-hand. When people come in here the client has a choice about which [EN] to go to. Clients come in for us or SVRA and we ll partner with them. We don t have many SSI/SSDI clients, and [SVRA doesn t] want to do Ticket and [is] afraid of reaching the trial period. It wasn t worth it for them. California s (Round 2) DEI leadership instructed AJC staff to keep a relationship going with their local SVRA office to increase the likelihood of Partnership Plus cases being assigned to LWIA ENs. AJC staff provides initial core and WIA services as needed and refers eligible customers to their local SVRA office on a regular basis for additional training, assistive technology, and job readiness services that are unavailable at the AJCs. According to several California stakeholders, through this process a relationship between AJC staff, SVRA staff, and each client is created. Having received AJC services and support from SVRA, the client is more likely to return to the AJC after their case is closed with SVRA for job retention services and monitoring through WIA. Wisconsin (Round 2) had success with one site that became an EN early on in the grant period. 41

42 According to one DRC, marketing EN services to SVRA resulted in a consistent flow of Partnership Plus referrals: Marketing seems to be a difficult issue for all ENs. I can t stress [its importance] enough. When I work with Partnership Plus and a DEI EN, marketing [becomes] a huge issue.... They don t know what services they can provide, how to provide them, and how to market them. That seems to be hard for any [EN] in the TTW Program. Another Wisconsin DRC stated that: I go through a list of [Ticket holders] from the prior week that closed out and are working at or above SGA and make contact with them. I look at the county they are in and their particular disability and encourage them to take their Ticket to the [AJC] in their area. I tell them about the website for yourtickettowork.com... [and] about Partnership Plus and what we have to offer. In another Wisconsin site, an expedited Ticket pilot was implemented. This approach allowed an EN to work with a Ticket holder during the 90-day waiting period to maintain the continuity of support services and eventually transfer the Ticket to another EN. One Wisconsin DEI stakeholder described the expedited Ticket pilot in the following way: Lots of times they don t make contact [with the Ticket holder] until after file closure. And with expedited Ticket, they re making contact before file closure and Maximus is aware they re getting the Ticket before file closure. So it s about a week. Without expedited, it could be 1 2 months before Ticket closure, which could delay payment to the EN. We re hoping this will increase Tickets to the ENs and payments to the EN. With expedited Ticket, an EN can provide support services, but the actual transferal of the Ticket to an EN happens after the 90-day period. Among Round 3 states, Florida, Indiana, Iowa, and Massachusetts have reported implementing Partnership Plus. In Florida, one treatment site applied for Partnership Plus soon after the grant started: They had a lot difficulty with the process. The other regions were looking to them as an example so we ve had Maximus and NDI work with us and SVRA to see if we can t smooth out the rough edges around this. That first region got their agreement done, and other regions are following their lead. Getting through the SVRA process is a little rough. Indiana has begun the process of implementing Partnership Plus. In its first year of operations, one site in Indiana was working on a Partnership Plus agreement. The DEI State Lead organized meetings with their local SVRA staff and trainings on TTW and the use of Partnership Plus, but like Massachusetts had not received any hand-offs from SVRA. Iowa has a state-level administrative EN that keeps 5 percent of Ticket revenue as a processing fee. The remainder of the revenue goes to the LWIAs. The state also has an automatic Ticket assignment system that checks Ticket eligibility and assigns Tickets automatically. Treatment LWIAs have an agreement 42

43 with SVRA to transfer Tickets to an LWIA after the 90-day waiting period. This process reportedly does not always function smoothly because LWIAs still must compete to get the Ticket since customers may shop around their Ticket to community-based providers: It doesn t always function that smoothly. At the state level, there s a good agreement but regionally I think we are both competing for funding so it becomes difficult. A majority of people are working with SVRA. [For those who aren t,] only if they remain employed through the last milestone will we see the Ticket assignment return. 4.9 TTW Promising Practices Expedited Ticket (Tennessee) We have identified several promising practices related to the implementation and operation of TTW. A few treatment LWIAs have been using Ticket revenue as an incentive for frontline workers. In one Tennessee (Round 2) treatment area, the staff member with the highest placement rates decided how they wanted to use TTW revenue. They may have used it for staff training on disability-related issues or to provide training for customers. Wisconsin s (Round 2) pilot test of its expedited Ticket operated by having SVRA automatically release Tickets prior to the completion of the 90-day waiting period so that job retention services could be provided soon after job placement. All Ticket holders received a letter that presented the expedited Ticket assignment process. The letter listed all Partnership Plus ENs available to them. SVRA presented the EN options to each client and talked about the yourtickettowork.com website, how the program worked, and what Partnership Plus had to offer through EN job retention services. The client benefited from this process because they did not have to wait the 90-day period to reassign their Ticket and receive EN services. This allowed for more prompt payments to the EN, as the Tickets were reassigned within one week. In addition, because the EN reported client earnings to SSA automatically without action on the part of the Ticket holder, the latter would not be found in an overpayment situation when a portion of benefits had to be paid back to SSA Sharing Revenue with Employers (Washington) A number of grantees reported that the types of long-term, follow-along services included in the TTW did not fit well with the typical services provided at the AJCs. In some instances, in an attempt to provide better services to these customers, a few grantees considered splitting up the responsibilities for serving the Ticket holder so that the AJC would provide employment services and training and another EN would then get the Ticket and provide follow-along services after employment was achieved. In the case of Washington (Round 2), the state explored a partnership with the chamber of commerce. They envisioned a partnership in which the Ticket holder received employment preparation services and training from the AJC. When ready for job placement, the chamber would receive the Ticket and use its connections with employers to get the job-ready individual into an employment situation. According to the plan, the chamber would have had to apply to become an EN and the payments received for milestones would have been given to employers. In Washington, the chamber started the EN application process, but it took much longer than anticipated. By the time the chamber had been certified as an EN two years later its leadership had changed and the project was abandoned. While this initiative never 43

44 came to fruition, the state continued to pursue options that would split the Ticket services to allow the AJCs to provide employment readiness services and then transfer the Ticket for placement and job retention services to another organization SSA e-processes (Iowa) Iowa (Round 3) implemented SSA e-processes to automate Ticket assignments and payments to ENs and Ticket holders, allowing LWIAs to assign Tickets quickly and efficiently. As stated in Section 4.5, SSA s e-processes expedite Ticket assignments and EN and beneficiary payments. Alaska (Round 1), Massachusetts (Round 3), New York (Round 1), Virginia (Round 1), Iowa (Round 3), and Connecticut (Round 4) have either begun piloting or were/are using one or more of the SSA e-processes. In Iowa, after Ticket assignment, the DRCs contact each Ticket holder to offer them the option of opting out. Most Ticket holders agree to work with the EN, and this practice has led to a rapid increase in the number of Tickets assigned and Ticket holders working with the AJC. In the final year of the Round 3 grant period, we will examine whether Iowa and other states that use automatic Ticket assignment are satisfied with the performance of the Ticket beneficiaries assigned in this manner State ENs as Learning Opportunities for AJCs Seventeen DEI states created a state EN, which provided LWIAs an opportunity to learn the ropes about running TTW, assigning Tickets, and working with Ticket holders. Eight commercial ENs set up treatment LWIAs after having apprenticed with a state EN. With the exception of Hawaii (Round 2), which put all Ticket revenue into a general workforce investment fund, state administrative ENs distribute between percent of TTW revenue to the local areas. Commercial ENs created by treatment LWIAs receive 100 percent of the Ticket revenue and are responsible for implementing the required administrative services associated with TTW through SSA s e-processes (See Exhibit 11). Exhibit 11: Type of EN Used by Round 1 3 DEI Grantees STATE LWIB ROUND 1 Alaska Arkansas None 30 None Delaware None None Illinois Kansas Maine New Jersey New York Virginia 30 These states did not create state or LWIA ENs during the grant period. 44

45 ROUND 2 STATE LWIB California Hawaii Ohio South Dakota Tennessee Wisconsin Washington ROUND 3 Florida Iowa Indiana Massachusetts Minnesota Louisiana Rhode Island 4.10 Sustaining DRCs through Ticket Revenue and LWIA Resources Several Round 1 and 2 DRCs were retained through LWIA Ticket revenue, although it appears that most Round 1 states did not generate as much revenue from TTW as Round 2. Round 2 showed greater numbers of Ticket assignments and milestone payments, as more Round 2 than Round 1 DRCs were retained after the grant period. In many cases, these individuals successfully implemented TTW in their LWIAs, were certified benefits planners, and had developed extensive networks of providers, employers, and Ticket holders that made them indispensable to the workforce investment system. Although LWIA and AJC stakeholders overwhelmingly felt that it was important to keep DRCs on staff due to their experience working with individuals with disabilities, many were rehired in a different capacity (e.g., as WIA counselors) but still provide benefits planning, outreach, and TTW management activities. In one site in Virginia (Round 1), one DRC was recruited after the end of the DEI grant period to work in a DEI control site in the same role, and others found jobs in non-dei LWIAs by taking on a variety of responsibilities including working with individuals who do not have disabilities. Among Round 1 states that were awarded Round 4 grants (Alaska, Illinois, New York, Maine, and Virginia), several DRCs were transferred to new treatment sites. For instance, in Maine, where TTW was operated by a state-level TTW coordinator, Ticket processing and direct Ticket holder support continued after the grant period through the existing DRCs. In Alaska, the state planned to use funds from the Mental Health Trust Authority to sustain two DRCs, including one 45

46 who was a certified benefits counselor. Alaska s Round 1 grant used a cyclical DRC training program that allowed AJC employment counselors to become Alaska-certified DRCs and then return to their original positions. The Round 4 grant allows the state to continue this process. New York s vibrant TTW Program was able to sustain most of its DRCs, all of whom are certified benefits planners, although many were transferred to control or non-dei sites/ajcs; some were sustained through the state s Round 4 grant. Illinois reported sustaining two DRCs through TTW revenue. Kansas struggled throughout the Round 1 grant period with the DEI State Lead position turning over three times; Kansas did not report sustaining any DRCs. Two of the three Round 1 youth states, Arkansas and Delaware, did not focus on the implementation of TTW because they served younger youth (14 18 years of age) through their DEI grants. These states appeared to have ignored the requirement to implement TTW because they focused on a population that was ineligible for the program. Arkansas work with local school systems made it possible for their DRCs to offer WIA services to youth with disabilities in transition, but reportedly little was accomplished due to the challenges of collaborating with local school districts. Delaware also did not implement TTW during the grant period, as staff focused on career counseling, school visits to discuss WIA services, and job fairs at communitybased agencies and correctional facilities. Finally, despite being a youth DEI grantee, New Jersey made a concerted effort to implement TTW in two DEI sites but reported that the program would not continue after the grant period due to a lack of resources to sustain the DRCs. 31 In Ohio (Round 2), several DEI stakeholders expressed the concern that the case management demands of TTW were too high for the DRCs given their responsibilities. Only one of the three Ohio treatment sites received TTW revenue. While the TTW activities in Tennessee (Round 2) went well, a couple of the local areas started the EN process late and experienced a funding lag in which EN services were being provided but no revenue was coming into the LWIA. DEI personnel in Tennessee said that TTW did not produce a lot of revenue, even for those LWIAs that moved through the EN process quickly. Tennessee reported that all areas kept their DRCs mostly through LWIA revenue, as the LWIAs valued their expertise. Washington (Round 2) was seeking to develop a partnership with an organization to provide follow-along services to Ticket customers the AJCs placed in employment. According to one Washington DRC, Once the [LWIA] realizes the profits from Tickets, that will help institutionalize the program. Staff is trained, comfortable, and confident working with JSWDs. Hopefully it will keep going. Wisconsin (Round 2) reported sustaining two DRCs through Ticket revenue. South Dakota (Round 2 youth state), which had not assigned any Tickets during the grant period, and Hawaii (Round 2) did not report sustaining any of their DRCs with Ticket revenue. Section Implementation of the DEI Service Delivery Strategies by Round 1 4 Grantees In the following sections, we discuss the implementation of each service delivery strategy selected by Round 1 4 DEI grantees. 31 One treatment site in New Jersey never completed the TTW suitability determination process. 46

47 5.1 Customized Employment Round 1 Round 2 Round 3 Round 4 Alaska California Not selected by any Alaska * Arkansas Hawaii Round 3 grantees Alabama Illinois Kansas Maine New York * Round 1 continuation state Ohio Tennessee Washington Customized Employment is an expanding practice for providing services to JSWDs as evidenced by its inclusion in the recently passed WIOA. It is comprised of several steps (discovery, negotiation with employers regarding task assignment and employment supports, job carving, job sharing options, and non-employment-related supports). Six grantees from Round 1, five from Round 2, and two from Round 4 selected Customized Employment as one of their DEI strategies. The foundation of this strategy is built around nontraditional assessment, job seeker exploration, and/or discovery, leading to the development of an IWP. Among the varieties of Customized Employment, individual components of this strategy (task reassignment, job carving, job sharing, and Self-Employment) can be used separately or combined depending on the needs of each JSWD. Initially, Round 1 grantees Alaska and Illinois reported having difficulty implementing Customized Employment due to limited knowledge among AJC staff of this strategy and the perception that few JSWDs who visited AJCs would be interested and/or need it. The other Round 1 grantees never actually implemented Customized Employment and felt that it was time-consuming for the fast-paced environment of the AJC. However, two Round 2 states received TA from the LEAD Center and the NDI TA Center that explained the various components of Customized Employment and how they could be used alone or in combination to increase the likelihood of several successful job placements. In addition to innovative adaptations to Customized Employment, such as group discovery, an activity introduced by trainers affiliated with the LEAD Center, Tennessee utilized job carving and task reassignment in its Customized Employment activities. The state s DRCs and employment counselors were trained by Michael Callahan of Marc Gold & Associates, a provider affiliated with the LEAD Center. Several Tennessee stakeholders reported that the hands-on training they received led to the frequent use of Customized Employment throughout the state s treatment LWIAs. Funding for additional TA was provided by Tennessee s VR agency. Although designated for employees of the Department of Intellectual and Developmental Disabilities and DRCs at two treatment sites, the training was made available to other staff through a partnership with the state s other Federal grants. This included the Employment First State Leadership Mentoring Program (EFSLMP) and funding from the Administration on Intellectual and Developmental Disabilities (AIDD), which focus on employment for JSWDs. 47

48 One Tennessee DRC spoke about applying what she learned through Customized Employment training with a specific client who was ineligible for services through the state VR agency. She discussed job carving options with an employer, and her efforts led to a placement with the insurance company AFLAC: We were able to place a client who couldn't be helped by VR and through the discovery process we found skills and talents that he had and placed him in a job that matched his desires. Another Tennessee DRC stated that Our job developer really uses Customized Employment when she is approaching employers about internships and jobs. She does discovery to pitch to the customer.... We see employers who had never hired a person with a disability now hiring them. Every participant has Customized Employment activities. We have pulled in partners to get them the services and supports they need (job development, transportation, on-the-jobtraining). Tennessee DRC Ohio received T&TA from Griffin-Hammis Associates. Through this training, several DRCs and employment counselors received their Association of Community Rehabilitation Educators National Basic Certificate of Achievement in Community Employment. The state reported that 26 DRCs and AJC staff members had participated in Customized Employment training by the end of the grant period. As a result, one treatment site reached its goal of using Customized Employment with 10 JSWDs, while another began to use Customized Employment in its AJCs for the first time on a regular basis in the final year of the Round 2 grant period. A third Ohio treatment site had job developers who were already trained in Customized Employment and had been providing Customized Employment services, while another LWIA employment counselor certified in Customized Employment worked with the state s developmental disabilities agency to connect the AJC and the agency to create a partnership through which they would share promising practices related to the use of Customized Employment. The AJC staff felt that this partnership increased the rate of referrals of people with developmental disabilities to the AJC: We get through all of the Customized Employment stages; we even do the job carving part. The job services reps from one area did the training, and they do the Customized Employment piece instead of the DRC. From another area, she does the discovery herself. We partnered with AFLAC; they needed telemarketers for employers. A lot of our clients like that type of job because they can work from home. We used the DEI funds to do the training and pay for the insurance license. Once they get that, they get hired by AFLAC. We were able to place a client who couldn t be helped by SVRA and through the discovery process we found skills and talents that he had and placed him in a job that matched his desires. Ohio also developed a Customized Employment handbook entitled Tips for Customized Employment Contracting, which provides a framework for the development of Request for Proposals for Customized Employment services contracting. This framework includes Key Definitions, Introduction and Purpose, Target Population, Overview, Anticipated Goals and Outcomes, Scope of Work, Deliverables, Vendor Requirements, Reporting, and Funding and Payments. The document also includes detailed instructions on discovery, job search planning, job development/negotiation, and post-placement support services. In California, several DRCs had received Customized Employment training but were not as successful in implementing it. 48

49 Stakeholders attributed the limited progress to the complexity of the program and limited access to the job developers needed to move beyond the discovery stage of the process. Iowa, in spite of not selecting Customized Employment as a DEI strategy, implemented it at several treatment sites. State officials described why this has occurred: We re very interested in Customized Employment and we ve done it [before]. It s a major focus for us in two related projects Iowa Coalition for Integrated Employment and the Employment First [EFSLMP] project. The pilots they ve developed: Some are co-located with our [DEI treatment LWIAs]. So we ve looked at that as a natural way for IRTs to occur and for regions to experience Customized Employment as a tool. People are looking at Customized Employment differently including job coaching, discovery, etc. One LWIA has DEI-approved flex funding for the discovery process and for Customized Employment. We hope that will happen in all five regions. The impact of this effort has been a larger role for Customized Employment in Iowa s SVRA practices and, due to partnerships with the AJCs, a larger role for it in the state s DEI activities. As one SVRA official stated: SVRA has integrated a discovery piece in our services. We ve had several joint trainings talking about what Customized Employment is, [and] what discovery is. We have a monthly webinar that brings together all interested parties to talk about Customized Employment. The first two months we shared examples of projects and experiences in Customized Employment. Through Employment First, we have six pilot projects that perform Customized Employment, and now we ve rolled that out to providers in the state. With the Customized Employment, the benefits planning piece is a crucial effort, and that s been really woven in through the DEI. DRCs in Iowa are expecting to see more integration of Customized Employment into their work with JSWDs due to changes in state policy that were developed through the state s EFSLMP activities. One DRC summarized these changes by saying: We have a huge push to use Customized Employment instead of sheltered workshops. I like that we re seeing that. I do think [Customized Employment] is going to be more and more necessary. I expect in our state a big push for more employment specialists so that we can move more people into community jobs. Though none of the Round 3 states selected Customized Employment, in Round 4 both Alabama and Alaska selected it. Several Alabama DRCs indicated that they were familiar with Customized Employment and had participated in a training session conducted by NDI after learning they were awarded their DEI grant. Many DRCs viewed Customized Employment as a broad category of employment strategies that includes Self-Employment. Others shared the concern that AJC employment counselors may not have the skill set or time to use Customized Employment effectively without the assistance of SVRA counselors or additional AJC staff. One DRC, who described the time commitment that may be necessary to use Customized Employment, stated that: 49

50 I am trying to use [Customized Employment]. I have a customer now and just mentioned it to them; we are doing some discovery with her, and I suggested Self-Employment but she is not into it. We might go down the Customized Employment road. She has been having some difficulty staying with a job because of her needs [related to employment readiness]. [Customized Employment] has been a nice tool to have in my pocket. It is a challenge to have longevity with this process, but I have been trying to look at it as a modified approach and customize it to the customer. [For example,] this customer will have a modified discovery process, not the whole thing. Sometimes because it takes a long time to get through the process, people get impatient. I think it is a great thing and makes sense but have not gotten all the way through it yet. 5.2 Self-Employment Round 1 Round 2 Round 3 Round 4 Alaska California Louisiana Alaska * Delaware Kansas New Jersey * Round 1 continuation state Washington Through Self-Employment training, which often includes a multi-week curriculum, an individual can develop the skills and knowledge necessary to choose the environment, number of hours, and form of work they wish to do. Components of this strategy commonly include identification of the type of business that suits the job seeker, development of a business plan, acquisition of startup resources, and marketing and skill development through career technical education or shortterm technical training. Self-Employment may include components of Customized Employment, particularly discovery, as well as an exploration of the supports necessary for success. Among the initial Round 1 grantees that selected Self-Employment, none sought to integrate Customized Employment components into this service delivery strategy. Alaska and Kansas worked closely with partner agencies, such as SVRA, the U.S. Small Business Administration, and community colleges, to develop agreements that would support Self-Employment training. Alaska and New York developed several businesses with participants of its START-UP/USA grant funded by ODEP prior to the award of their Round 1 DEI grants. But as a DEI grantee, Alaska stakeholders reported limited interest among DEI participants and knowledge of how to implement Self-Employment among AJC staff. As part of its Round 4 continuation grant, some of Alaska s DRCs completed training on how to work with JSWDs to help them become selfemployed, and they reportedly plan to revisit it as the need arises. New Jersey s instituted LearnDoEarn, an afterschool program for youth DEI participants age designed to build the academic and behavioral foundations [youth] will need for success in college and/or work. LearnDoEarn utilizes a curriculum, games, and guided activities to help youth understand what academic and behavioral skills are needed to succeed in college and employment. While not a Self-Employment program per se, this effort touched upon the broader topic of entrepreneurship, 50

51 development of professional support systems, basic group process and employment issues, and finance and collaboration. Two Round 2 grantees, California and Washington, also selected Self-Employment. In an approach similar to that used by Alaska and Kansas, California utilized strategic partnerships with local small business development centers (SBDCs) to obtain support and guidance for Self-Employment and reported three viable small businesses being formed through the DEI. This included one supported through a successful partnership between SVRA and a DEI treatment site. One California state official described these outcomes as a consequence of the state s Customized Employment efforts when interests in Self-Employment were identified during discovery. Louisiana, a Round 3 grantee, also developed partnerships between its treatment LWIAs and SBDCs and the locally based Operation Hope Center, which provides financial literacy and Asset Development training. In addition, a partnership between the Louisiana DEI and a community college has paved the way for career technical education opportunities for DEI participants interested in Self-Employment. This also supports a range of other employment and training options. Barriers to increased use of Self-Employment include limited interest among customers and the cost of training programs available through partnering organizations, with the exception of the one-day class available through the SBDCs. One DRC remarked that when working with the SBDC, the only challenge is that they give you one class for free then they charge the clients to continue the classes. Similar to the approach being taken with Customized Employment, Iowa has begun to build its capacity to provide Self-Employment training in spite of not having selected it as one of its DEI service delivery strategies. Motivation for the use of Self- Employment in Iowa is due to the success of the local SVRA, where 55 case closures have been attributed to Self-Employment training alone. DRCs have received training on how to use Self- Employment with JSWDs, though interest among DEI JSWDs has been limited. One DRC described these early steps thusly: In November, we did a two-day Self-Employment training for Ticket beneficiaries and discussed the benefits of a Self-Employment plan, brainstorming, how to launch business ideas, etc. It included benefits planning and helping [Ticket beneficiaries] understand how their benefits are affected by employment. The NDI TA providers came to provide training. We did outreach and promotion of the event for the community. This year, we re hosting a Self-Employment seminar for community agencies. That s still in the works, but the basic guideline is helping partners understand that if they have clients looking for Self-Employment, how to help them with that process and be an advocate for them in that direction. 5.3 Guideposts for Success Round 1 Round 2 Round 3 Round 4 Arkansas South Dakota Minnesota Idaho Delaware Alabama New Jersey 51

52 Developed in partnership with ODEP, the National Collaborative on Workforce and Disability for Youth identified a series of thematic findings from the research literature focusing on youth transitioning into adulthood. These research-based practices and strategies, the Guideposts for Success, focus on five areas of youth development: (1) School-Based Preparatory Experiences; (2) Career Preparation and Work-Based Learning Experiences; (3) Youth Development and Leadership; (4) Connecting Activities; and (5) Family Involvement and Supports. Each of the youth states is required to select the Guideposts for Success as one of their DEI strategies. Variations in the age range focused upon were components of several states DEI programs. For example, Arkansas, Delaware, and Idaho each focused on youth aged in their initial grant applications. New Jersey, South Dakota, and Minnesota focused on youth aged Alabama focused on youth aged In practice, states that focused upon youth aged placed greater emphasis on school-age (14 18) youth. Round 1 2 grantees, in practice, placed particular emphasis on one Guidepost rather than all five. According to DEI Round 1 grantees that selected a youth focus, this was to programmatically align an existing youth developmental practice already in place in the state or treatment LWIAs. This alignment contributed to an expansion of these practices to better include youth with disabilities, as well as sustainability by aligning with already-established state initiatives. Arkansas, a Round 1 youth state, emphasized Youth Development and Leadership in its DEI program. Unlike the other youth states, Arkansas utilized DEI resources to initiate a new program for youth with disabilities. Planning began during the second year of funding to conduct a series of Youth Leadership Summits in the four DEI pilot regions. Modeled on California s Youth Leadership Forum for Students with Disabilities, these events focused on introducing youth and school staff to resources available for individuals with disabilities and provided seminars focusing on career preparation and self-advocacy. Four Youth Summits occurred, with strengthened local connections to the area high schools being an additional impact. These connections, and the access they provided, allowed for the formation of Job Clubs, a program to reinforce the ideas presented at the summits, at several participating high schools. Although the Arkansas Youth Leadership Summits were reportedly well-attended, there was only moderate follow-up with the youth participating in the summits and there is no evidence that this practice will continue after the grant period. (This finding was reported in a previous DEI Synthesis Report for Year Three June 20, 2014, Section Guideposts for Success, pp ). New Jersey focused on the Guidepost Career Preparation and Work-Based Learning Experiences, which aligns with the state s LearnDoEarn curriculum. Arkansas selected the Guidepost Youth Development and Leadership. Planning began during the second year of funding to conduct a series of Youth Leadership Summits in the four DEI treatment regions. Modeled on California s Youth Leadership Forum for Students with Disabilities, these events focused on introducing youth and school staff to resources available for individuals with disabilities and provided seminars focusing on career preparation and self-advocacy. 52

53 Delaware, also a Round 1 youth state, selected Connecting Activities, which examines barriers to employment and community inclusion, placing particular emphasis on youth involved in the state juvenile justice and mental health systems. These youth often encounter significant barriers focusing on housing, health care, employability, and the potential impact of their criminal history on their employability. A record expungement event was held at the Delaware s juvenile detention centers. Although both the Arkansas Youth Leadership Summits and Delaware record expungement events were reportedly well attended, only moderate follow-up with the youth participating in these events occurred. No evidence arose that these practices would continue after the grant period or if any of the youth involved were connected to the local workforce investment system in their states. DEI staff utilized their existing contacts in the Delaware service community to link JSWDs to job opportunities, particularly in the areas of health care and housing. To identify youth who might benefit from these services staff liaised with community workers at the state pediatric hospital, alternative schools for at-risk youth, and Delaware s juvenile justice facilities. Youth within the juvenile justice facilities were a population of particular focus due to additional barriers to employment, which frequently included limited family support and hiring restrictions due to criminal history. DEI staff, in addition to health care and housing support, provided instruction in financial literacy, conducted interview preparation and work expectation coaching, and explored the criminal record expungement process with applicable youth. (This finding was reported in a previous DEI Synthesis Report for Year Three June , Section Guideposts for Success, p. 63). South Dakota, the only Round 2 state focusing on youth, built upon its previous efforts in developing School-Based Preparatory Experiences. This work included using the ACT National Career Readiness Certificate program as a preparatory tool for students preparing to transition out of special education services. One South Dakota stakeholder discussed the influence that the Guideposts had over the way they served youth with disabilities in transition: We ve taken a lot from the Guideposts, which say that all youth need certain things and specifically youth with disabilities also need specific things. Resources can be used by any job seeker and also by youth with disabilities... We ve used the Guideposts extensively. We used them as a guide for developing the teacher toolkit [and] also for SD Resources 4 You, which is a search engine developed to be able to search various local resources. This work also consisted of training events for school staff about DEI and the workforce system, as well as attendance at students IEP meetings, either in person or through the use of phone or video conference technology. The goal of these conversations with schools, students, and their families was to link certain credentials to DEI and WIA-sponsored employment and training experiences. (This finding was reported in a previous DEI Synthesis Report for Year Three June 20, 2014, Section Guideposts for Success, p. 63) 53

54 South Dakota also actively promoted the National Career Readiness Certificate within high schools. The certificate rates employability skills and offers skill remediation plans for areas of weakness. A pilot was recently conducted and up to 2,000 assessments for the certificates were offered to South Dakota youth. South Dakota also contacted students with disabilities who had left the school without a diploma or certificate of completion to link them with service providers early on that could assist in reconnecting them with services to support job readiness and employment. South Dakota incorporated each Guidepost for Success through the development of an online teacher toolkit that provides information on job seeker services for youth, a video on selfdisclosure in the workplace, staff training videos, information on the state s career development forum, workforce training assistance, youth job search, Asset Development, disability disclosure and social networking, as well as fact sheets from the U.S. Business Leadership Network and Chamber of Commerce, and information on person-first language, assistive technology, and employer hiring incentives. From the South Dakota DEI website: Every job seeker with a disability is faced with the same decision: Should I or shouldn t I disclose my disability? This decision may be framed differently depending upon whether you have a visible disability or a non-visible disability. Ultimately, the decision of whether to disclose is entirely up to you. Below is a video developed by South Dakota Disability Employment Initiative with personal testimonies by South Dakotans on their choice to disclose or not disclose their disability in the education and workplace setting. 32 The South Dakota DEI website also includes videos on the development and application of IRTs and the process of bringing organizations and individuals together through public and private sector partnerships to coordinate services and supports for a youth with disabilities. Minnesota, a Round 3 grantee, through its long-standing partnership with the PACER Center, has embedded the Guideposts for Success into its youth transition practices. As such, the model has been added to the standard practices of the state s Community Transition Interagency Committees (CTICs), which are regional groups in Minnesota that are staffed by teachers, parents, youth in transition, county representatives and other community members. The goal of the CTICs is to create awareness of transitional supports to young adults with disabilities. Working in collaboration with the CTICs, DEI has been able to enhance the School-Based Preparatory Experiences and Career Preparation and Work-Based Learning Experiences available to participating youth by providing additional support through WIA and DEI-sponsored activities for skills development, career counseling, and job readiness. (This finding was reported in a previous DEI Interim Synthesis Report for Year Four June 10, 2015, See Section Guideposts for Success, p. 47). Minnesota provided an explanation of its partnership with VR agencies: The [Minnesota] DEI Project allowed us to partner with VRS on a whole other level. After the grant was awarded but before we received DEI funds, we held a meeting of all area vocational rehabilitation as well as independent living center staff and the youth program staff of our centers to introduce them to each other and to brainstorm how to work with each other. Prior to this meeting we did not have a history of working closely 32 South Dakota Disability Employment Initiative. (2014). Home. Retrieved from 54

55 together. A lot of times we were not familiar with each other and how each other worked. VRS didn t know what WIA youth programs provided and vice versa. Right off the bat, this meeting was a game changer. It was the beginning of new relationships between organizations. This meeting opened the doors to future collaboration and led to referrals to one another s programs and the blending and braiding of resources to best meet the needs of each individual. 33 In addition to helping counselors assess youth needs, the Guideposts for Success framework helped engage youth in goal setting. Minnesota reported that they will continue to make this information available for the youth staff and will introduce new youth counselors to the Guideposts for Success framework during their orientation in the AJCs. Minnesota also implemented a policy for serving youth and young adults, including those with disabilities, which incorporates the IRT and Guideposts for Success permanently into their service delivery structure under WIOA. This policy requires the creation of an Individual Service Strategy based on the Guideposts for Success, setting goals for younger youth and the use of the Guideposts for Success whereby each of the Guideposts are represented by staff who can assist with strategies and services to meet the needs of each youth. Minnesota DRCs coordinated implementation of DEI activities in each treatment site, building partnerships and leveraging resources to expand state, local agency, and community-based agencies with support from the National Collaborative on Workforce and Disability for Youth. The Guideposts for Success was used as framework for youth assessment and coordinated service delivery. Minnesota also implemented person-centered IRTs. Ultimately, Minnesota seeks to have state agency leaders assigned to each of the Guideposts for Success to organize T&TA, policy, and implementation of evidence-based practices associated with each one. As of the completion of the Minnesota Rounds 3 DEI grant, the state was in the process of selecting state agency personnel to participate in the Guideposts initiative. Idaho, a Round 4 grantee, shares some structural similarities with South Dakota and Minnesota. Like South Dakota, Idaho has strong linkages between its school districts and workforce investment systems. Similar to Minnesota, the state has developed regional structures that provide support for student with disabilities engaged in the transition process. Idaho selected three Guideposts: School-Based Preparatory Experiences, Career Preparation and Work-Based Learning Experiences, and Connecting Activities. These Guideposts reflect Idaho s efforts to connect with local school systems, DRC involvement in Individualized Education Program (IEP) meetings, and enrollment in WIA and connecting school-based education with WIA services. Successful systems change has resulted in DRCs active participation in students IEP meetings. These linkages were developed during Idaho s DPN grant; as one DRC stated: Some of the similarities are that when we were in the DPN, we were going out to schools and working with counselors and special education teachers and those students, we were doing disability mentoring days, we were collaborating with many different agencies on 33 Minnesota Department of Employment and Economic Development. (n.d.). MN partners for youth: Lessons learned. Retrieved from This information was provided by the state of Minnesota. It was not collected by the DEI Evaluation team. 55

56 committees to make those mentoring days and different events happen. We were working with youth even though the DPN wasn t specific to youth. Similar to Minnesota, the state-sponsored Idaho Interagency Council on Secondary Transition has developed close relationships with its LWIAs and local school districts. Though a statewide initiative, rather than multiple regional groups such as Minnesota s CTICs, it serves a similar function in working to identify transition resources for youth with disabilities. Early DEI activities linked to the Guideposts for Success have included sponsored field trips to local employers funded through the WIA youth program with the purpose of providing information about local employers and the skills needed to work in different kinds of industries. Youth can also obtain advice and guidance from the employers and develop relationships with other youth and industry leaders. (This finding was reported in a previous DEI Interim Synthesis Report for Year Four June 10, 2015, Section Guideposts for Success, pp ). Alabama, which received training in the Guideposts for Success, described difficulties in utilizing the strategy related to their focus on youth older than the traditional high school ages of Youth who were out of school and not currently participating in any form of training lacked many of the resources aligned with School-Based Preparatory Experiences, Career Preparation and Work-Based Learning Experiences, Youth Development and Leadership, and Connecting Activities frequently available during active school or training enrollment. Early activities to address these challenges have included the development of a Resource Map, the identification of potential partners for Connecting Activities, and the development of presentations for youth and their families. 5.4 Asset Development Round 1 Round 2 Round 3 Round 4 Alaska California Florida Alaska * Illinois Hawaii Indiana Connecticut Kansas Ohio Iowa Illinois * Maine South Dakota Louisiana Maine * New York Tennessee New York * Virginia Wisconsin Virginia * * Round 1 continuation state Asset Development consists of services designed to help individuals navigate the various financial benefits, programs, and work incentives that encourage employment. DEI grantees focus on tax preparation services, financial literacy, banking, benefits planning, and development of T&TA materials for JSWDs on a range of financial topics. Seventeen states, including all five Round 4 continuation states, selected Asset Development as one of their DEI strategies and 56

57 placed significant focus on financial literacy and tax preparation, including the availability of tax credits on SSI/SSDI benefits. Several DEI states also developed partnerships and/or used DEI resources to expand their capacity for providing Asset Development services to JSWDs. Examples of this included financial literacy forums, often cosponsored with local banks, the local IRS offices, or the Federal Deposit Insurance Corporation. This also included services such as Money Smart and My Money Works curricula, the Volunteer Income Tax Assistance (VITA) program, and MyFreeTaxes. Illinois and Hawaii provided tax preparation services to 262 and 100 JSWDs, respectively, during the third year of their DEI grant periods. Several Round 2 grantees placed particular emphasis on the financial literacy aspects of Asset Development. Partnerships with outside agencies supported these efforts at several treatment sites. Other grantees emphasized the use of Individual Development Accounts for customers, assisting them in putting money aside for needed tools or training to enhance their employability. Interest in attending Asset Development events among JSWDs was mixed. According to several DEI stakeholders, the focus of Round 2 LWIAs on Asset Development strategies resulted in an increase in their capacity to provide information that had broad applicability to JSWDs. Examples of Round 2 Asset Development activities included state-level DEI personnel in Ohio developing a guide for the local areas about Asset Development activities and motivating them to develop action plans that included time lines for tax preparation, regular deposits in savings and investment accounts, and learning about Ticket and other work incentive options. One LWIA in Ohio relied heavily on a partner agency to provide financial fitness fairs during which JSWDs would provide information to financial experts and receive advice about their financial situations. Unfortunately, these, like most of the Asset Development events held by DEI grantees, were sparsely attended. California DEI leadership staff expressed that many treatment sites enthusiastically supported Asset Development as a worthwhile strategy for addressing financial barriers to work participation: We ve embraced Asset Development close to the heart. Each area has realized that helping an individual who has been long-term unemployed or never employed and never had a paycheck that lasted to the next paycheck, giving them info on how they can actually start saving, budgeting, dealing with bank accounts, fraud issues, filing taxes, different tax breaks, has been a real help to establish relationships with the individual and having them feel like the case manager truly cares about them as a whole. LWIAs in both California and Tennessee participated in the IRS VITA program, which provides free tax preparation services. They have also developed partnerships with local banks to provide financial literacy seminars. JSWDs who completed a series of trainings received incentives from Wells Fargo Bank and a certificate of completion from the AJC. This particular site, located in California, expanded the program to include incarcerated individuals at a nearby prison. Washington placed significant emphasis on Individual Training Accounts. Using a local provider, one Washington LWIA enrolled 36 participants. As part of this program, $50,000 was available for participant matches that could be used for purchases that mitigated barriers to employment. These included transportation, computers, uniforms, and assistive technology to 57

58 improve access and performance in essential job tasks. Participants were required to save half of the money required for the purchase in a custodial bank account while the purchase was made by the AJC upon reaching half the value of the equipment or service. During their enrollment, participants participated in financial literacy seminars for a minimum of three months before matching funds were made available. Hawaii described financial literacy instruction as a necessary component of benefits planning, and stated, If people want to go back to work and they have a goal, money comes into it somehow. This is part of a larger discussion around benefits what happens when money starts coming in the door. To further develop their capacity to implement the Asset Development strategy, Hawaii LWIAs partnered with Hawaii Community Assets, a nonprofit organization that provides housing and financial education, to fund specialized training from Michael Raff of the LEAD Center, including financial planning and counseling and making investment decisions. Wisconsin, perhaps due to repeated state leadership turnover during their Round 2 grant, described difficulty in enhancing or extending the Asset Development services it possessed prior to DEI. Examples of this included delays in rolling out their Asset Development strategy, including one DRC describing just beginning the design of an Asset Development training and guidebook late in the third year of their grant. Efforts to introduce financial literacy services into the AJCs were hindered by low participation where five of the six [DRCs] said it was difficult to do this and hard to get customers to show up for Asset Development events. One theory suggested for the low participation in Asset Development and benefits planning events was the distance necessary to travel to the AJCs. Among the Round 3 grantees, both Louisiana and Indiana described delays in instituting Asset Development services during their grant period, but progress was clearly made in implementing the strategy during their second year of the grant. Louisiana, in particular, placed significant emphasis on developing its capacity to enhance self-sufficiency through TTW. This has largely been accomplished through the development of strategic partnerships that have led to the availability of benefits counseling services being provided by local providers while DRCs receive training. The DRC and director at one site have been conducting TTW orientation sessions with potential DEI customers, placing great emphasis on the goal of reducing dependency on SSI/SSDI. As they stated: Part of the TTW orientation is to explain to individuals that our EN is designed to work with individuals who want to limit their dependency on SSI/SSDI with the goal of becoming self-sufficient. If their goal is not to decrease their dependency on SSI/SSDI, we may not be the agency for them. Indiana DRCs reported that initial Asset Development activities were hindered by a limited understanding of the strategy and interest among DEI customers. State leadership provided technical support in this area and a series of regional Asset Development fairs were offered. These fairs, though open to all AJC customers, focused on connecting low-income JSWDs to services that could enhance economic security. Viewing programs focusing on financial literacy as being too narrow, Indiana LWIAs invited representatives from subsidized housing agencies and low-cost medical providers in addition to staff with expertise in the TTW Program. One 58

59 LWIA, in the process of implementing an Asset Development event, estimated that exhibitors would be in attendance. In contrast, Florida began implementing Asset Development soon after they received their DEI grant. In the first year of implementation, Florida DEI staff attended a two-day workshop conducted by NDI on this strategy, followed by individual TA meetings with each region to gather information on the existing Asset Development strategies and planning for future activities. During their second year, Florida developed outreach materials for employers describing available tax incentives for hiring JSWDs, instituted financial literacy and goal surveys into their intake procedures, and developed weekly Asset Development tip sheets for DEI customers on Money Mondays. Financial tutorial services, through programs such as Money Smart, as well as forums on tax preparation, budgeting, and Asset Development were at varied stages of planning. One LWIA in the state was planning to integrate Asset Development strategies into their AJCs so that they could be maintained after the DEI grant period. Iowa conducted significant outreach and marketing of its Asset Development services. In a partnership with United Way, an advertising campaign was created to generate interest in financial literacy and Asset Development strategies; they were shown in AJCs along with trainings focusing on the Earned Income Tax Credit and general assistance with preparing and filing taxes. Specific methods varied by LWIA. For example, one LWIA offered group and individual trainings to AJC staff so they could counsel JSWDs on a range of Asset Development topics. Another LWIA offered a monthly Money Smart class for DEI customers. All AJCs in Iowa also offered monthly financial literacy trainings. Seven Round 4 states selected Asset Development as a DEI strategy. Current Connecticut plans for this strategy include the development of financial literacy and tax credit assistance services. Both Connecticut treatment sites obtained print, online, and partner resources during their first year of DEI funding. Idaho is working with the VITA program and Ticket holders to help with Asset Development and tax preparation. Illinois is using a coordinated case management approach that includes an Asset Development questionnaire, which was developed in Round 1. The questionnaire includes the state s Health Benefits for Workers with Disabilities (Medicaid Buy-In) program; work incentives counseling for SSI/SSDI beneficiaries through Work Incentives Planning and Assistance (WIPA); and Plan to Achieve Self-Support and TTW EN activities located in treatment LWIAs. New York has taken a similar approach to Asset Development as the state is working on tax preparation services, SSA work incentives counseling, and promoting the Earned Income Tax Credit, Individual Development Accounts, and basic principles of accounting, including profit and loss projections. Idaho (Round 3) worked with the VITA program and Ticket holders to help with Asset Development and tax preparation services for youth DEI participants. (This finding was reported in a previous DEI Interim Synthesis Report for Year Four June 10, 2015, Section Asset Development, p. 51) 59

60 5.5 Integrated Resource Teams Round 1 Round 2 Round 3 Round 4 Alaska California Florida Alabama Arkansas Hawaii Indiana Alaska* Delaware Ohio Iowa Connecticut Illinois South Dakota Louisiana Idaho Kansas Tennessee Massachusetts Illinois* Maine Washington Minnesota Maine* New Jersey Wisconsin Rhode Island New York* New York Virginia * Round 1 continuation state Virginia* An IRT is designed to bring representatives of different agencies and services systems together in order to determine the needs and availability of resources and services for an individual JSWD. Through the IRT, appropriate services and funding streams are identified and coordinated to address those needs. The use of IRTs began during the DPN Initiative in 2003 with the development of a customer-focused process to assist workforce systems and WIA-mandated and non-mandated partners to identify and obtain needed employment and related resources, expertise, and services relevant to JSWDs. IRTs are significantly influenced by context, with variability introduced by the needs and roles of their members, the physical location of those partners, and the time commitment given to the process. Ideally, the IRT process is led by the individual JSWD, but typically coordinated by the DRC. It is their responsibility to bring IRT partners, such as parents and family members, educational providers, government agencies, and community providers, together to identify needs and potential supportive resources. IRT members also leverage resources from participating agencies to address barriers to employment. DEI stakeholders have described challenges to implementing IRTs related to determining whether each JSWD should receive an IRT, the immediacy of an employment placement, scheduling meetings with partners, and the extent of the involvement of partners and the individual JSWD. Grantees from later rounds described, generally, more positive experiences with the IRT process when compared to the Round 1 states. Statements from stakeholders in New Jersey, a Round 1 grantee, attributed some of the early difficulties to a lack of understanding of the purpose and structure of IRTs and the value of meetings with numerous partners. Others expressed hesitation concerning which agency should initiate services due to requirements to be the funder of last resort. Grantees such as Virginia and Maine, through initiatives such as First Dollar Down and Flexible Spending Funds, allowed DEI resources to be used to initiate services, allowing other agencies to then provide support or address gaps in service not covered by the multiple 60

61 funding streams brought together by the IRT. Although IRTs were designed to bring all partners to the table, in most cases Round 1 DRCs sought to expedite the IRT process by contacting certain partners by telephone and/or to discuss the employment challenges of individual JSWDs rather than bringing the entire team together. This approach worked well in highpopulation areas where efficiency was valued over group process. Among the Round 2 grantees, IRTs were seen as valuable to the process of arranging funding and services, but, as reported by one Wisconsin DRC, while most of them have been extremely beneficial,... there are a few that maybe could have worked out better maybe in one form or another rather than having to bring together all of the members of the IRT. An approach used by several DRCs in that state, as well as by some stakeholders in Tennessee and Hawaii, was to rebrand the practice to make it appear less formal to encourage partners from other agencies to participate: I love the concept but putting the IRT label on it didn t work. I found people reluctant to join another team. They were stretched too thin and didn t want to be on yet another team or be responsible for the IRT. So I called informal meetings, not calling them IRTs. We were kind of doing them already anyway. I did them all and documented them. Two additional DRCs from Wisconsin, describing similar strategies, stated that this informal approach was more likely to lead to greater participation among partners, and especially WIA and SVRA participation. Other Round 2 state stakeholders, though not abandoning the IRT label, also believed that the concept had not yet reached maturation in their treatment sites. One State Lead summarized the belief held by many that the IRT thing hasn t realized its potential just yet, while Ohio, Washington, and Wisconsin attributed this to a lack of knowledge early on concerning what constituted an IRT and what the goals for the practice were. This delayed roll out in several treatment sites. Confusion on this topic was resolved in time, with one Washington DRC crediting the ODEP/ETA meeting in 2012 as the source of clarification. Hesitation concerning the goals and purposes of IRTs also seemed to be shared by the JSWDs themselves. In Hawaii, significant efforts were made to communicate that the true leaders of the process were the customers themselves. At one site, explanation of the strategy was made during a 12-week job readiness program in which they explored both individual goals through traditionally defined IRTs, with significant support from SVRA, and developed peer support through sharing these goals in group IRT sessions: For the last job readiness class we held a group IRT, which was interesting. It was more of a sharing session. We talked about everyone s short- and long term goals and how AJC employment counselors would be achieving them through different support networks. We tried to explain to them that the responsibility lies with them. They are the IRT leader and they would need to be the driver and have [the DRCs] be the support to their goals. [The DRCs] would just provide them with advice and guidance along the way. That s basically how we did the individual ones, too. 61

62 As IRTs became more embedded in the AJCs, WIA case managers began to lead some of the meetings in Washington and South Dakota, where state officials reported that WIA personnel in their state had adopted many of the practices recommended by the NDI TA Center. Some Round 2 grantees whose treatment sites were located in less densely populated areas embraced nontraditional IRTs to accommodate the locations and scheduling requirements of partners. Technology was the most common tool used to accomplish this, as South Dakota, whose treatment region encompassed its rural western expanse, used several means to staff its IRTs, with one stakeholder stating: It s a combination of all methods of communication. Face-to-face time is going to be the number one priority. Most often we can do that with the initial meeting and then later on when trust is developed it s easier to communicate electronically or by phone. Ohio and South Dakota used Skype and/or other video conferencing technologies to accommodate JSWDs, family members, or agency partners in distant locations. These states also recorded trainings on the use of IRTs and saved them in an archive that could be accessed by AJC staff. One of the sites in Ohio, in recognition of the difficulty associated with scheduling meetings for each IRT, modified the structure so that much of the work could be done asynchronously. Another site indicated that the needs of particular clients, in addition to staff availability, influenced the depth and frequency of IRT meetings: We have a meeting [when] we have new customers. They re in-person meetings. Depending on the customer we ll determine how often they need an IRT [meeting]. If they have barriers or challenges [that come up periodically] we ll come back to the table and brainstorm to problem solve. Customers have had positive feedback about the IRTs. We [have a standing] meet[ing] once a week and we have the new customer come to talk to the partners. We have the same core people at the meetings, and sometimes we ll bring in someone for a particular customer. It can be difficult to schedule the non-co-located partners to attend the standing meeting. If a client can t attend the meeting we involve them in other ways. The majority of the Round 3 grantees demonstrated an ability to develop a more thorough understanding of the IRT strategy more quickly than the Round 1 and 2 grantees by building on the experiences of previous rounds and TA from NDI. For example, the Rhode Island State Lead mentioned that using IRTs allows the DRC and AJC staff to provide additional services [to JSWDs] without referring [to community providers]. We re using the IRT to bring the partners into the Center to meet with the customer instead of making a referral [outside of the AJC]. This process often brings resources into the AJC by blending and braiding funds across partner agencies. Other grantees recognized similarities in practice between the DEI IRT strategy and other initiatives, such as veterans and special education initiatives occurring at some of the DEI treatment sites as a second stage of case management. Examples of this include a DRC in Louisiana who remarked that [between] the mental health center, the veterans representatives, and SVRA there is a lot of overlap [among existing initiatives]. Minnesota, a state with 62

63 significant history of collaboration between its workforce and educational systems, saw the IRT process as parallel to the IEP process that students with disabilities participate in. These meetings were frequently extended so that the IRT could be incorporated into the process. This finding was reported in a previous DEI Interim Synthesis Report for Year Four June 10, 2015, Section Integrated Resource Teams, p. 54). One Minnesota DRC, describing the process, stated that: They ve all been able to meet together. A lot of times it ll be at the IEP meeting and include a [Southeast Minnesota for Independent Living] representative, an SVRA representative, and our [AJC] representative, along with the youth and parents, and we ll go through our youth action plan, that includes the five Guideposts, and... talk about how we can work together to achieve the youth s goals. Interviews also showed that Round 3 treatment sites have included various agencies in addition to WIA-mandated partners in their IRTs and are influenced strongly by the needs and interests of their JSWDs. One stakeholder in Rhode Island, describing the process at their site, stated that: I believe the goal [of the IRT] is the best interest of the customer. I believe that part of that is bringing together everyone who has an interest in that customer other agencies, social workers, business services, and obviously the customer. The customer is making the decisions ultimately. We have an IRT coming up next week. We had a previous meeting in January and this is a follow-up meeting to review his progress in the program. We are bringing in everyone who is involved with him, and we offer direction, but it s on the customer ownership and responsibility. It s very customer-driven. Similarly, a DRC in Florida stated that: The goal of an IRT is to make sure that resources are available for all customers. You want to make sure that you have all the information you need to service a customer and the only way to do that is to build teams and keep communication open. We use the WIA program, so we get the customer the resources they need [to] succeed in the program. IRTs depend on everyone s schedule but [include] conference calls, s, going over and meeting with different agencies one-on-one. Anything you need to do. The customer is the main point of the IRT, so we make sure we communicate with them what their needs are and ask them to follow through. Round 3 stakeholders also described some of the challenges they have had implementing IRTs. Similar to the descriptions provided by several of the Round 1 and 2 grantees, Round 3 grantees expressed some concern about bringing all partners to the table in-person for every meeting. As one State Lead said: They don t look like the YouTube video with people around a table for a long time. No one has that kind of time. So it s a matter of pulling people in as they re needed and they may never be in the same room, might just talk on the phone. It s very individualized. They re supposed to be customer-driven and I m sure that happens to a certain degree 63

64 depending on the individual. Some people feel completely lost in the job search market so they need more hand holding. A DRC in Massachusetts stated that: I haven t found that IRTs have been utilized yet my customers are complicated but those [JSWDs] who have multiple agencies haven t wanted that type of collaboration. It comes back to case management implementing that is pretty time-intensive for drawing everyone together. I work with my customers and it s hard for me to spend additional time on their case outside of that. Iowa stakeholders shared that there has been uneven development in their capacity to offer IRTs. Though DRCs at two AJCs reported that they had conducted approximately 12 IRTs each, others commented that there was limited interest, often compounded by limited resources, amongst the partners and, as a result, some were reluctant to participant in IRT meetings. One state leader summarized this issue in the following way: We have the infrastructure [for implementing IRTs] but as far as really achieving a functional strategy of IRTs that works how we want it to, it s not quite happening yet in all regions. We feel the need to continue the focus on local leadership teams on what the value added is explore the opportunity together to help the customer. If staff had worked with just one agency and those services don t work for a customer, it s about not giving up on that customer and instead looking for other resources. One region is good with IRTs and another is doing fine, but the other three think of IRTs as referrals backand-forth between agencies. It s a partnership issue, less of a DRC issue. They think of it as referring a customer for a service instead of collaborating for the customer. A referral might be because of paying for a particular service. Quality of IRTs is different in different areas. One area considers it an IRT if they make a referral. That s why we need to improve that strategy. Early interviews from Round 4 grantees, which are in their third year of the grant period, show that they are implementing IRTs but have yet to see the results of their efforts. One Alabama DRC, demonstrating their understanding of the strategy, stated that: The goal of an IRT is to improve employment outcomes by bringing resources together, which is a reason why I ve been reaching out to local groups in case we or SVRA can t pay for something. That s why I m looking for community resources to break down any barriers there might be for someone going back to work. IRTs might change depending on who we are dealing with. I d prefer face-to-face, but if that s not possible, conference calls. If that s not possible, individual calls. Whatever it takes; I will do whatever it takes. We will include as many people as it takes for the person to reach employment. The customer would have to be very involved. Their wishes and desires would be very important. 64

65 A Connecticut DRC, whose site had been using IRTs since the DPN grant, recognized that IRTs often result in bringing resources together that support JSWD services and employment outcomes: We ve been doing IRTs for the past five years. It just made sense. We have been doing it as a best practice. I just didn t know the official name was IRT. We ve had a lot of success. No one agency can fulfill all your needs. I don t have the funding that SVRA has for example for on-the-job-training. When you present an IRT to the customer, it looks very inviting. It s the menu. I m so invested in this model because I believe in it, but you have to have someone at the helm. It s very important that there is a good leader involved. When we come to the table, it s consumer-driven and we each bring our piece of the pie. Let us do what we re experts in. 5.6 Blending and Braiding Resources Round 1 Round 2 Round 3 Round 4 Alaska California Florida Alabama Arkansas Hawaii Indiana Alaska* Delaware Ohio Iowa Connecticut Illinois South Dakota Louisiana Idaho Kansas Tennessee Massachusetts Illinois* Maine Washington Minnesota Maine* New Jersey Wisconsin Rhode Island New York* New York Virginia* Virginia * Round 1 continuation state Blending and Braiding Resources, also known as Leveraging Resources, refers to the use of multiple funding streams, often Federal, to better support a job seeker by addressing service needs and filling the gaps presented by partner agencies as they develop a plan to transition an individual into employment. Ideally, these resources are brought together by working with the customer through an IRT. For grantees that have developed Blending and Braiding MOUs or less formal agreements, this DEI strategy has been used to compensate for resource scarcity, initiate service use by addressing funder of last resort rules, and support the training of staff across agencies. DOL guidance and NDI TA place emphasis on the leveraging of resources from Federal, state, and local agencies and many treatment sites defined this strategy as an extension of their partnership and collaboration activities. Treatment sites were less likely to pool or blend resources, but rather braid resources from multiple agencies. This approach was a reaction to the restrictions associated with the scheduling, eligibility qualifications, and funding parameters 65

66 intrinsic to Federal, state, and local programs. Ohio, for example, used DEI funds to support JSWDs who had resource needs that were not covered by WIA using a flexible fund out of their DEI resources. The DRC meets with the customers individually to find out what help is needed and if it is a legitimate need. She has used funds to help people get textbooks, gas cards so they can get to training, bus passes for customers to be able to get to interviews or job shadowing or training and employment needs. In Iowa, one treatment site described supporting two JSWDs who encountered multiple barriers to training and employment. DEI funds were used to supplement what was available from WIA. We have someone we re serving who needs some skill upgrading. She wants to go to college but one of her concerns is managing her home and children. Someone contacted me from SVRA for an IRT and we were able to leverage funding with WIA for her tuition costs and SVRA could purchase assistive technology equipment for her computer, as well as ongoing supports, through braided funding. As far as employment, we have a client who we could provide services for and SVRA could help purchase [job] interview clothing and a hearing aid through the [DEI] flexible fund. He s now making a high hourly wage and doing well. Job coaching and transportation have used [DEI flexible] funding as well. SVRA has a strong partnership with a local medical center and we had a client who needed a job readiness program for a placement with this medical center, but she has severe seizures so we partnered with SVRA and Goodwill to negotiate a rate to fund part of her transportation for the six-week job readiness program and we picked up the rest. When she completes it, SVRA will place her in a work experience program at the medical center for direct hire. It was a good IRT because everyone came together and picked up different pieces. States such as New Jersey, Iowa, and Wisconsin have successfully merged funding streams from Federal programs. New Jersey and Iowa received funding from the state s Medicaid Infrastructure Grant for training and employment readiness activities; New Jersey also received a grant from the Kessler Foundation to support job training activities. In Wisconsin, a DRC had begun to strengthen a partnership developed during the course of their grant: Another [DRC] IRT connection is the TANF program in our Job Centers. I am now trying to get them some vocational funding. A lot of JSWDs were told they can t work, and the TANF case managers were unclear about this, so they came to me for clarification. Their clients were scared of losing SSDI benefits if they went to work. So let s talk to people about SMART work placement, 34 enabling JSWDs to work to their capacity with an employer who knows and is prepared for their levels of ability. Other grantees viewed their DEI funds as flexible, allowing them discretion to use them to address service gaps and, in some cases, enhance the effectiveness of supports provided by partner agencies. Blending and Braiding allowed IRTs to begin services outside of WIA and 34 Science, Mathematics, and Research for Transformation (SMART). 66

67 reduce the impact of having to use multiple funding streams and their restrictions. Hawaii utilized its DEI flexible fund to address service gaps and to purchase equipment such as assistive technology. Similarly, one California site described its use of flexible funds in the following way: [The DEI flexible fund can] be used for supportive services like Fast Passes and holding events [at the AJC]. For our TTW outreach events, we were able to purchase food that was huge. Some of the money was used to pay for training [AJC staff]. Typically, for WIA-funded training, the provider needs to be on the eligibility list; with this money we didn t have to follow those guidelines. Like I said, that was huge. Sharing resources, particularly training opportunities, was a common way for agencies to support each other without having to develop formal funding agreements. States such as Illinois, Tennessee, and California utilized this approach for both customers (e.g., job readiness workshops and WIA-funded orientation and training) as well as for staff. Tennessee was able to obtain Customized Employment training for its DRCs, which was supplemented through their EFSLMP and AIDD grants. One California site described the use of their DEI funding to provide training across their LWIA: In the 18 Centers we have EmployAbility Partnerships where every Center has a Disability Coordinator for our customers to see on an as-needed basis. We meet quarterly with our [WIA] mandatory and other partners and do training. Under the DEI grant we made it bimonthly because we had to get a lot of people up to speed. Without the grant, we d be going back to quarterly [trainings], but we ll continue. There s also LEGACY Training a disability sensitivity training through which we trained 130 people this year. We use the [DEI] flex funding to pay for the training. We re having one last class that will be funded from DEI. In spite of these successful examples of how DEI grantees use integrated resources, it continues to be a challenge to implement across systems due to numerous restrictions associated with the scheduling, eligibility qualifications, and funding parameters inherent in Federal, state, and local programs, which hinder full implementation of this DEI strategy. One DRC summarized these difficulties by stating that: Money is [always] a challenge when you talk about someone else s funding. There are difficulties with identifying resources and funding with changing allocations and different fiscal year calendars. With Blending and Braiding [of resources], some agencies are willing to try [to collaborate], but sometimes the money isn t there. There s also a misunderstanding that SVRA and [the AJC] have all this money. It s a misunderstanding of how much money we have, not so much [that we have a] negative attitude [about the use of integrated resources]. In spite of the challenge that sharing resources across agencies and systems represents, many states have shown that braiding, which keeps resources from different agencies separate, is much easier, particularly when partnerships are developed at the state and/or local level and there is a level of trust among the staff of each agency. Braiding, as the result of partnerships and 67

68 collaboration, may be evolving into a single practice that LWIAs and their AJCs can use to support JSWDs Disability Resource Coordinators The DRC position continued to be a critical component of the DEI. Qualitative data from this year s data collection activities indicates that the range of DRC activities and responsibilities vary among grantees and even between treatment areas within states. During the fourth year of the program, DEI stakeholders continued to make comparisons to the DPN Initiative when speaking about DRCs, as they saw the latter in a more supervisory and TA role rather than as a case manager for JSWDs. One DRC spoke about the differences between the DEI and the DPN grants, saying, with DPN we focused on providing AJCs with the tools to provide the services. DEI actually has job placement goals [and a focus on the customer] we ve been able to do a lot more hands-on service [through DEI]. 6.1 Roles and Responsibilities of DRCs in Rounds 2 and 3 As with the Round 1 DRCs, two overarching factors influenced the roles and activities of the Round 2 DRCs. First was the type of managerial authority of the DRC. At the local level, some Round 2 DRCs were employees of the state, while others were employed by the LWIA. Those who worked for the LWIA tended to be more involved in the day-to-day operations of the AJC. This arrangement had advantages in that it had the effect of integrating the DRC into the AJC environment. It also resulted in DRCs taking on certain administrative tasks within the AJC. On the other hand, DRCs who were employed by the state were less likely to have additional administrative responsibilities, allowing them to focus on TTW beneficiaries, forging collaborations and partnerships with other agencies, and/or implementing the DEI service delivery strategies. The downside of this arrangement was that as state employees, DRCs were somewhat less likely to be as fully integrated into the AJC environment as their locally hired counterparts. The second factor was the two models of implementation of the DRC that were seen across the DEI grantees. Variations in the DRC typology presented in the DEI Year 3 Interim Synthesis Report were not identified among Round 2 DRCs. Among Round 1 and 2 DRCs, most states employed the State Lead Supervisory DRC. In three Round 1 (Alaska, Delaware, and Maine) and four Round 2 (Tennessee, Hawaii, 35 South Dakota, and Ohio) states, the DEI State Lead played a role we refer to as a State Lead DRC by engaging with local areas to support program implementation and strategic planning. In this capacity, these DEI State Leads closely supervised Local-Area-Only DRCs. For example, the DEI State Lead in Tennessee (Round 2) met monthly in person or by conference call with the Local-Area-Only DRCs in each treatment LWIA to discuss partnerships and collaborations, the implementation of the DEI service delivery 35 The first DEI State Lead position in Hawaii provided extensive support to local pilot areas. When this position turned over due to staff reductions, the new DEI State Lead was unable to provide the same level of support and engagement with the local areas. 68

69 strategies, TTW, and resource needs. Alaska s Round 1 DEI State Lead spearheaded the development of a DRC training curriculum that effectively upgraded the skills and knowledge of existing AJC employment counselors to that of the Local-Area-DRC position. Some Round 1 (Illinois, New York, Virginia) grantees and one Round 2 (California) grantee created a statelevel DEI-funded DRC position we refer to as the State Supervisor DRC (SSDRC). This position is also referred to by DEI grantees as the Supervisory DRC or the Lead DRC. Five Round 3 states utilize the SSDRC approach: Indiana, Iowa, Louisiana, Minnesota, and Rhode Island. In Florida, responsibility for monitoring and engaging with Florida DEI treatment sites lies with the regional LWIA personnel. LWIA Leads serve as the SSDRCs while the DEI State Lead provides administrative support to the grant. The role of the DRC has evolved from Round 1 to Round 3 to emphasize TTW implementation and related services. Although this change has led to some confusion about whether or not JSWDs who are not Ticket beneficiaries can participate in the DEI, it has resulted in a marked increase in understanding of TTW, including the use of SSA s e-processes, including automatic Ticket assignment, as well as Partnership Plus and expedited Ticket transfers. One Round 1 (Virginia) and another Round 2 (California) state utilize the SSDRC position. These SSDRCs support and oversee Local-Area-Only DRCs in the DEI treatment LWIAs, providing T&TA on program implementation and use of service delivery strategies and TTW. The Local-Area-Only DRCs are located in treatment LWIAs and work with one or more AJCs. While the activities of Local-Area-Only DRCs vary across grantees and LWIAs, they typically provide case management to JSWDs and TTW beneficiaries, implement selected DEI service delivery strategies, and forge local area partnerships. In three Round 2 states (Ohio, South Dakota, and Tennessee) DEI State Leads played the State Lead Supervisory DRC role, as they were all engaged with local-level planning and implementation throughout the entire grant period. Only one Round 2 grantee (Wisconsin) used the Local-Area-Only DRC approach. In states that funded a separate SSDRC position, the SSDRC was a full-time employee of the state. By comparison, DEI State Leads who assumed the role of the State Lead Supervisory DRC spent less time on typical DRC responsibilities, such as meeting with participants and creating local area partnerships, than the DEI-funded SSDRCs due to their responsibilities as senior/executive leaders in the state s host agency. As with Round 1, Round 2 states varied in terms of the number of DRCs in each LWIA and their roles and responsibilities. Due to the variability in the size and population density of participating LWIAs, some treatment sites had one DRC. Some areas had two DRCs, while others had a DRC and a Disability Resource Specialist (DRS). The DRS was typically also funded by the DEI and they often served as an assistant to the DRC and performed many of the same functions as a DRC. In LWIAs where there were two DRCs or a DRC and a DRS, the division of responsibilities ranged from sharing all DEI-related activities, to dividing up responsibilities for specific DEI tasks. In some instances, one DRC oversaw all TTW activities, while the other was responsible for all other DEI activities and service delivery strategies. In Tennessee (Round 2), two of the treatment LWIAs had a DRS in addition to a DRC. Each area developed its own division of responsibilities between the DRC and DRS. In one area, the DRS 69

70 was in charge of job development, reportedly the most time-intensive AJC activity. As a part of this, they worked with JSWDs to develop or update their résumés and communicate with employers to match JSWDs to available positions. In another Tennessee LWIA, the DRS focused mostly on clerical work but also provided job coach services and assisted the DRC on other activities, including arranging trainings, developing partnerships and collaborations with public and private service providers, and implementing the DEI service delivery strategies. Population density is the determining factor when staffing LWIA AJCs. In most cases, having two DRCs or a DRC and a DRS occurs in areas with above average population densities. The DEI State Lead in Tennessee communicated directly with local areas and played the role of the SSD. She monitored the implementation of the DEI service delivery strategies, provided TA to AJC staff, and engaged with LWIA and WIB personnel. One Tennessee Local-Area DRC described her role as facilitating training and employment with JSWDs, working with community organizations, leveraging funds.... I makes sure we market the program, basically being involved in the lives of JSWDs to get them employment and training, and also employment with self-sufficient wages. In California (Round 2), the SSDRC had direct contact with local area staff on a regular basis. He conducted site visits to the local areas, provided and/or arranged T&TA for quarterly DEI planning committee meetings, and met regularly with DRCs and other DEI stakeholders. At the beginning of the grant, the SSDRC conducted site visits to local areas, which helped AJC staff to become more knowledgeable about the DEI, including the service delivery strategies and implementation of TTW. The California SSDRC emphasized that the local areas know what is best for them so my intent is not to tell them how to implement the grant, but to support them and help them examine what is and is not working for their local area. DEI state-level staff in California spoke about the variation in DRC responsibilities and said that no matter what the specific role of the DRC is in an area, the DEI team has worked to ensure that all AJC staff receives training related to JSWDs. This allowed JSWDs to be served by anyone in the AJC, rather than waiting to work with one designated person. Our role is to train and educate. In Hawaii (Round 2; a Local-Area-Only DRC state), the DRCs had been providing information to AJC staff and service providers to participate in strategic planning meetings. One DRC in Hawaii described her responsibilities as including outreach, setting up IRTs, and working with case managers on an as-needed basis to provide services to JSWDs. Another DRC explained that if a customer disclosed a disability, they worked with the DRC to complete an intake assessment and determine if they were TTW eligible. If they were not Ticket eligible, the AJC would continue to work with them on core services, including access to labor market information and job search and placement assistance. The first DEI State Lead in Hawaii engaged with local areas on the implementation of DEI grant activities. She left the position in 2013 and subsequent DEI State Leads were less engaged with local area activities. DRCs in South Dakota (Round 2; an SSDRC youth state) worked not only with customers in the AJCs, but also with teachers, students, and families through the local public schools. They attended IEP meetings and provided T&TA to special education teachers related to the services available for students at the AJC and how to enroll students in WIA. They were also involved in events such as career development forums for year-olds, the development of resources such as a teacher toolkit and an online resource of service providers, and disseminating 70

71 information about services provided by local agencies for IRTs. The state s career development forums covered topics including self-advocacy, the development of employment goals and a career plan, employer expectations, and disability disclosure in the workplace. The teacher toolkit was aimed at high school special education students as well as teachers and counselors and provided ideas and materials for classroom lessons. The toolkit included resources such as Skills to Pay the Bills, 411 on Disability Disclosure, a WIA registration form and consideration checklist, and an interactive Prezi lesson for teachers to use to teach their students about registering and using SDWorks (South Dakota s online job search system) and other AJC services. One DEI stakeholder in South Dakota described the role of DRC someone who brings employers and additional services into the AJC, coordinates cross-training of team staff, and provides training to the employees of the AJC. DRCs in Washington (Round 2; a Local-Area-Only DRC state) provided case management services to JSWDs. DRCs in Washington explained that customers who disclosed a disability were directed to them first for intake and to begin the case management process. They met with each customer to discuss AJC resource availability and assign a WIA case manager. If the JSWD required resources from another service provider, such as VR or private sector service providers, the DRC would create an IRT and/or make referrals as necessary. DRCs also supported AJC staff by training them on how to serve JSWDs. These trainings included customer selfdisclosure, availability of support services for JSWDs in LWIAs, opportunities for Blending and Braiding resources, and information on the implementation of DEI service delivery strategies. Washington AJCs also had monthly trainings on TTW. Wisconsin (Round 2) was a Local-Area-Only DRC state with support from state-level staff, due in large part to turnover in the DEI State Lead position. DRCs in Wisconsin spoke about employer outreach, IRTs, and benefits planning in relation to the role of the DRC. DRCs in Wisconsin worked with more than DEI treatment LWIAs to provide benefits counseling, assess the accessibility of DRCs, and provide training to LWIA staff. One DRC in Wisconsin was responsible for conducting Section 188 compliance inspections and provided training to AJC staff on assistive technology. Another was a certified benefits specialist. He visited AJCs monthly to meet with Ticket holders to discuss how employment would affect their SSA subsidies. He was available to any DEI treatment site on an as-needed basis. All JSWDs were not automatically referred to him. If someone at a Job Center just sent a JSWD away, they would be fired. Depending on the individual, the Job Centers have all kinds of programs available; it just depends on the particular individual. The resource specialist at each of our Job Centers basically finds out exactly what they're looking for and if it s something specific, such as a benefits analysis, then they would be referred to me. Most of the time they are eligible for other programs right there in the Job Center; they may be eligible for a WIA program, for example. Round 3 states, which are in their final year of operation, all utilize the DEI State Lead SSDRC model, with Local-Area-Only DRCs providing support services to JSWDs. Florida is focusing on the implementation of TTW. All DRCs focus on identifying, recruiting, counseling, and supporting TTW beneficiaries in gainful employment situations. The five DRCs in Florida are 71

72 overseen by Regional Leads, which are similar to DEI State Leads and function as State Lead Supervisory DRCs for the LWIAs in their regions. Local-Area-Only DRCs work directly with JSWDs, connecting them to resources, implementing the DEI service delivery strategies, and providing case management services. Florida DRCs are also responsible for the implementation of the regions DEI strategic plans and are the conduits for the design and implementation of the Asset Development service delivery strategy, which is a key feature of the Florida DEI. Florida DRCs were highly involved in the planning and implementation of Asset Development Summits in their regions, including arranging and facilitating meetings with local employers on Asset Development services and activities. One DRC commented on a business-as-usual approach as all of the LWIAs in Florida have been operating TTW for several years: To be honest, the WIA staff has not seen many differences in how we work with people with disabilities because we really have always done this, other than the fact that we can now accept Tickets. There has not been a huge change in how we do business, which I think is a good thing. There are five DRCs in Indiana (Round 3), which is a Local-Area-Only DRC state. Indiana DEI stakeholders reported initial challenges related to the role of the DRCs versus existing AJC employment counselors. In the second year of the Round 3 grant period, several Indiana DEI stakeholders raised the concern that the responsibilities of the DRC position may overlap with those of the management and business services specialists working in the AJCs and therefore reduce the latter s workload and/or role in the AJC. However, since the decision by state leadership to have the DRCs primarily work with TTW beneficiaries, the demarcation between the two roles has become more defined and as a result, a DRC/AJC employment counselor partnership has formed. According to one Indiana DEI stakeholder: [The DRCs] are not employment counselors. They serve as a resource for the Ticket beneficiary and the AJC employment counselor. The AJC employment counselor is invited to all DRC-JSWD meetings so that the IWP and employment and related services are coordinated according to customer needs. With the exception of one rural LWIA in the state, most Indiana DRCs have had some success with the implementation of TTW. According to one DEI stakeholder, The DRC is very good and has had great leaders/guidance [from the LWIA]. She has 14 assigned Tickets now and [has existing relationships] with the staff [at the AJC]. Another Indiana DRC reports that: [We have] assigned three Tickets and another [will be assigned] next week. We are dealing right now with [Ticket beneficiaries]. Out of about 18 people who have attended the [latest TTW and work incentives] workshop, I ve run across about three or four who went online and had already started learning about [TTW]. According to another Indiana DRC: It s been a busy year. We ve assigned three Tickets at this point, and another one next week. We are dealing right now with individuals. Some individuals who have shown interest [in Ticket] are not eligible due to [their age]. We have [also] developed a 72

73 monthly TTW orientation at WorkOne centers and case managers are made aware of when those meetings are and are doing a pretty good job of getting the SSI/SSDI individuals registered for in those orientations. Two Indiana LWIAs have had turnover in their DRC staff during the second year of the grant. The Indiana DEI State Lead provides direct oversight to all of the DRCs. There are five Local-Area DRCs in Iowa (Round 3) who work closely with an engaged DEI State Lead who functions as the SSDRC. 36 State-level DEI staff emphasized that DRCs are not the only AJC staff working with JSWDs. DRCs provide training and assistance to AJC employment counselors and outreach to employers and other agencies. Training is provided by DRCs on Asset Development, DEI service delivery strategies, and TTW. State DEI leaders described an evolution of the DRC role since the beginning of the grant: When the DRC is new, they focus more on direct services to customers and once they blaze the trail [and] establish expertise in the role they can step back and start looking at the [broader systemic features] of their job by connect[ing] the dots and organizing support services and systems such as IRTs, Blending and Braiding resources, and enrolling customers in TTW. Iowa DRCs also do a lot of training with local partners; one noted 30 agencies on their LWIA leadership team, which meets quarterly to discuss benefits counseling/training, recruiting job candidates from AJCs, and training employers and local government agencies and providers on TTW and LWIA employment resources. In Louisiana, the DEI State Lead works closely with Local-Area-Only DRCs. Louisiana has been slowed by turnover of the DRCs in three of the five treatment LWIAs. The DRC role was described as a mix of direct service/referral and training/capacity-building roles were described. DEI leadership has placed emphasis on creating balance in the responsibilities of the DRC so that the latter has time to provide coaching to employment counselors and other AJC staff to their capacity-building efforts and outreach to community-based providers and employers. According to one Louisiana DRC, The first step [was to begin] educating our AJC staff on assisting people with disabilities. We have [created] a resource guide that [is] utilized in the AJC.... For example, if a client is seeking transportation services, that resource guide provides information on where to obtain that service for a client....eventually, we may look into getting [this] statewide bible out across Louisiana. Louisiana has a membership program in all of its AJCs to engage both customers and employers in their work and mission. Member customers receive an array of job preparation services such as soft skills, résumé writing, and job search support; employers receive assistance from the AJC on recruitment and screening job candidates. Louisiana DRCs conduct needs assessments to collect information on AJC staff members knowledge of TTW, working with JSWDs, and community resources. WIB staff with expertise in working with JSWDs also support DRCs and other AJC staff with T&TA on self-disclosure and identification of JSWDs. Louisiana reports 12 Tickets assigned to its DEI treatment sites. 36 The DEI State Lead left the position in January A new DEI State Lead was assigned in February

74 I m learning that a DRC requires wearing different hats at different times. I have to become familiar with every program and service of the EN. Round 3 DRC (2014) In Massachusetts, the DEI State Lead works closely with a state-level advisory team that provides strategic support to the state s DEI activities. The team consists of the DEI State Lead and members of the Institute for Community Inclusion (ICI), community-based providers, and SVRA personnel. ICI provides T&TA to each DEI treatment site designed to optimize the customer flow of JSWDs and Ticket beneficiaries. ICI has provided T&TA on the use and implementation of SSA s e-processes, including e-data Share to check the Ticket assignability of current and new JSWDs, e-ticket assignment to automatically assign multiple Tickets to the AJC and its EN, and e-pay to automatically pay AJC ENs without submission of proof of earnings documentation. Four Local-Area-Only DRCs coordinate resources for individual JSWDs as well as AJC employment counselors and community-based providers for one treatment LWIA. In this role, they create, manage, and coordinate IRTs and work closely with local Community Work Incentives Coordinators (CWICs) to help JSWDs understand the effects of employment on their SSDI benefits. Massachusetts DRCs request JSWDs who are Ticket beneficiaries to assign their Ticket to their local AJC. They then utilize an eco-map approach, which is similar to discovery, to identify social and job-related supports, how employment may affect their SSA benefits, and how work incentives can be used to maximize employment income. DRCs prepare an IWP, or Career Action Plan, as it is known in Massachusetts, with each Ticket beneficiary once the Ticket has been assigned. One Massachusetts DRC emphasized the difficulty and time-consuming process that JSWDs have negotiating complex systems of services: What I have learned is how difficult negotiating the system as a person with a disability [is], which is a large part of what they have to spend their energy [on]. It takes away from energy they could use being employed. The time and energy it takes to gather everything for disability status and reporting consumes an enormous amount of time. There are other barriers too, such as Internet access, transportation, and having limited [income]. Most customers need a case manager to help them get their life together enough to accept these challenges. In Minnesota, the DEI State Lead plays the role of the SSDRC and overseas three Local-Area- Only DRCs. The main role of the DRC is to conduct outreach to youth with disabilities, SVRA, and service providers in the community. Several stakeholders, including DRCs, DEI State officials, and affiliated providers, remarked that the role DEI has played within the Minnesota public workforce investment system has changed since the grant was awarded. State officials reported that they have really started to gain traction in the local areas and that: It was a little hard in the beginning [of the DEI grant] because we were told [DEI] was not a program, but was meant to enhance other programs. I think the DEI [service delivery strategies and requirements] have become more [salient] to everyone when they 74

75 work with youth with disabilities. That has been a change people are calling me rather than me calling them [for information and resources]. According to several Minnesota stakeholders, there is a greater awareness among parents, youth, and providers of the availability of services for youth with disabilities through the AJCs. A similar viewpoint was shared by another Minnesota DRC, who stated that: As far as my role [is concerned], it s kind of hard to pinpoint because things just kind of evolved, but looking back, I think of all the connections I ve made, those that included many different organizations and school and youth counselors, and helping them form IRTs to collaborate were the most helpful. I guess I ve become more involved in making connections and reaching out. It s been our biggest step; connecting them with our youth counselors to create a better team for each of the youth we serve is what I ve been able to do well. SVRA staff members reported that they have found the collaborative nature of the DEI to be useful in terms of arranging services for youth with disabilities. One SVRA Youth Counselor, when asked what gaps in services DEI has addressed, stated that DEI has helped youth with disabilities [gain access to] training programs and related supports [such as] job coaching. For one youth who is deaf and had never had a job, through DEI we were able to get interpreters [so he can attend]... training; having DEI means getting services that they couldn t before. Rhode Island s one DRC functions as the statewide coordinator of DEI and oversees five DEIfunded Customer Services Representatives and two Business Services Specialists located in the state s treatment LWIA. In the role of SSDRC, the DRC works with the DEI State Lead to develop MOUs that formalize partnership arrangements across agencies. The Rhode Island DRC also coordinates TTW. He completed the EN application and is training DEI staff on a range of issues including self-disclosure, identification of JSWDs, Ticket assignment, and AJC customer flow protocols. The Rhode Island DRC also provides trainings for AJC staff on disability etiquette and techniques for presenting information to potential community partners. In order to strengthen existing relationships and establish new partnerships, the DRC conducts partner outreach presentations during which he presents detailed information on the DEI, including specifics on TTW and opportunities for partnering on job retention and support services. In regular monthly meetings with the Customer Services Representatives and Business Services Specialists, the DRC coordinates staff caseloads, customer progress, challenges, and potential solutions to barriers to providing needed services, developing IWPs, integrating resources, and Ticket assignment. Exhibit 12 lists the states that use the SSD and/or the Local-Area-Only DRC models. See Exhibit 13: Key Responsibilities of DRCs by Type and State 75

76 Exhibit 12: Use of Disability Resource Coordinators Across Grantees N = 24 State Grantees State-Level Supervisory DRC Alaska Delaware* Illinois**** Maine* New York California Ohio Tennessee* South Dakota* Rhode Island Florida*** Minnesota* Iowa* *Connecticut *Idaho **Virginia Round 1 (N = 9) Round 2 (N = 7) Round 3 (N = 7) Round 4 (N = 7) Local-Area-Only DRCs Arkansas Kansas New Jersey Virginia** Hawaii Washington Wisconsin Indiana Louisiana Massachusetts***** Alabama Alaska Maine New York * In Connecticut, Delaware, Idaho, Iowa, Maine, Minnesota, South Dakota, and Tennessee, the State Lead the plays the role of the SSDRC. ** Virginia (Round 1 and 4) has an SSDRC and a statewide planning committee to steer activities under DEI. *** Florida, Illinois, and Massachusetts have state-level teams that share the supervisory DEI role. **** Illinois Round 1 team included a state-level DRC Coordinator. ***** Massachusetts has a state-level advisory team for its DEI grant. 76

77 Exhibit 13: Key Responsibilities of DRCs by Type and State SSDRC states Local Area DRC states SSDRC states Local Area DRC states SSDRC states Local Area DRC states SSDRC states Local Area DRC states Round 1 Managerial Authority Programmatic Design Case Management TTW Implementation Benefits Counseling* Service Delivery Strategies T&TA to AJC Staff/ Partners Outreach Employers JSWDs Partners Alaska State Delaware State Illinois State Maine State New York LWIA Virginia LWIA Arkansas State Kansas LWIA New Jersey LWIA/State Round 2 California LWIA/State? Ohio LWIA S. Dakota State Tennessee LWIA Hawaii State Washington LWIA Wisconsin LWIA Round 3 Rhode Island State Minnesota LWIA/State Iowa Massachusetts LWIA Florida LWIA Indiana LWIA Louisiana LWIA/State Round 4 Connecticut State Idaho State Virginia LWIA/State Alabama State Alaska State Maine State/LWIA New York State/LWIA Managerial Authority agency-level responsible for hiring Local Area DRCs. Programmatic Design participates in designing the program features associated with DEI at the state and/or local level, including service to customers. Case Management works with job seekers to develop an IWP and directs them to the services provided at AJCs. Implementation of TTW Implementation of Service Delivery Strategies T&TA to Local Area DRCs provides direct support and direction to local DRCs T&TA to AJC Staff and Partners provides direct support and direction to staff and partners, commonly on issues related to working with JSWDs. 77

78 Outreach to Employers seeks employers to promote employment of JSWDs directly or in job fairs. Outreach to JSWDs seeks to disseminate information and publicize services to JSWDs available at AJCs. Outreach to Partners seeks out partners who may come in contact with JSWDs to promote services at AJCs and/or to provide T&TA on how to best serve JSWDs. * In states that do have DRCs that provide benefits counseling, not all DRCs perform this task Benefits Planning Benefits planning continued to be a critical component of the DEI in Rounds 2 and 3. According to numerous DEI stakeholders, the availability and use of benefits planning services alleviated the concerns many TTW beneficiaries have about the potential impact of employment on the availability of their SSDI benefits. To address these concerns, many ENs provide benefits planning services through WIPA (i.e., CWICs). WIPA is designed to assist SSA beneficiaries with disabilities to make informed choices about work, and to support working beneficiaries to make a successful transition to self-sufficiency. 37 CWICs not only provide individual counseling on the impact of employment on benefits, but also conduct outreach to SSA beneficiaries with disabilities who may be eligible for work incentive programs. The first challenge is... getting beneficiaries to recognize that if they're able to work, they are better off working. That s why having DRCs all [become] CWICs was one of the core priorities. There is also the challenge of overcoming the fears of employers about accommodations. That's one reason why the use of temporary job placements is appealing so employers can see how it goes. LWIA Partner DEI benefits planning activities during the Round 1 grant period took place within the context of a changing, and at times uncertain, SSA WIPA environment. WIPA was discontinued shortly after the Round 1 DEI grants were funded in September At the time, Round 1 grantees had just completed their first year and were about to begin recruiting and providing services to TTW beneficiaries. This left Round 1 grantees without benefits planning services that they expected to be available through WIPA. The program was reinstated in August 2013, which left Round 1 grantees with only one month left in its grant period. Round 2 was in its second year and Round 3, which began in 2013, was about to begin its first year of grant operations (second grant year); the temporary loss of CWIC services impacted Ticket holder recruitment and TTW enrollment for grantees in Rounds 1, 2, and 3. However, the temporary loss of the CWIC program spurred a movement among DEI grantees to make concerted efforts to certify their DRCs as benefits planners through the Cornell University and/or Virginia Commonwealth University benefits planning courses. Soon after WIPA was terminated, Round 1 states such as Illinois, New York, and New Jersey chose to provide benefits counseling training to their DRCs. Alaska relied on funding from the Alaska Mental Health Trust Authority to provide benefits planning. Other states, such as Delaware and Kansas, provided some training to their DRCs, but also referred customers to certified benefits counselors rather than certify DRCs. Similar to Round 1, State Leads in Rounds 2 and 3 moved quickly to increase 37 U.S. Social Security Administration. (n.d.). Work Incentives Planning and Assistance. Retrieved from 78

79 the number of DRCs certified as benefits planners, while others established key partnerships with local service providers with similar expertise. Some AJCs in DEI treatment areas are still served by CWICs, while others use providers for benefits planning services. In Hawaii (Round 2), the Center for Disability Services provided benefits planning training for the DRCs to encourage them to develop spending plans for Ticket holders and discuss their eligibility for disability benefits if they engage in SGA. 38 According to one Hawaii DRC, If people want to go back to work and they have a goal, money comes into it somehow. This is part of a larger discussion around benefits what happens when money starts coming in the door. Especially with folks with mental health disabilities on fixed incomes who are sometimes uncomfortable when they start earning more money they don t know what to do with it. One Hawaii DRC completed the Center for Disability Services benefits planning training at Cornell University and another is in the process of completing the final requirement for her certification, a case study based on an actual DEI customer. The AJC needs to have [benefits planning expertise] so we can help Ticket clients right away and save them from navigating the SSA bureaucracy. In one Washington (Round 2) DEI treatment site, benefits planning were provided by a local provider (Community-Minded Enterprises), but DRCs also discuss benefits with Ticket holders during the case management process. One Washington DRC explained the information she collects to provide customized information for each Ticket holder on the impact of employment on SSDI benefits: We have our own intake packet that asks about job goals, transportation, demographics, Social Security releases, and releases to determine state benefits they might be receiving. We do a Benefits Summary Analysis where we verify actual benefits received and how earning will interact with each of the benefits. We can also do a basic benefit consult a more informal meeting relying on information provided by clients and a lot of hypotheticals. We provide general information to ease their minds that they won t be cut off from benefits. Similar to Washington, Ohio (Round 2) DRCs work with customers on benefits planning and have a contract with a local provider that provides benefits analysis and planning services to figure out what s needed and to avoid sending people on a wild goose chase related to the potential decrease or loss of SSA subsidies due to work. Tennessee (Round 2) made a decision early on in their grant period not to train DRCs to provide benefits counseling for JSWDs. Some LWIAs worked with banks that provide information on asset development options but did not focus specifically on benefits as they relate to Ticket holders. One Tennessee stakeholder commented that after WIPA initially ended we thought about getting DRCs training through Cornell s program but decided against it because they were not hired to do the benefits planning in the first place. It would have been too much to do benefits planning on top of everything else. 38 A Ticket holder can earn up to $1,090 per month in 2015 (for non-blind individuals; net of impairment-related work expenses) and can engage in SGA. 79

80 Tennessee, instead, contracted with local providers and local CWICs to provide benefits planning to Ticket holders. DEI stakeholders in Rhode Island (Round 3) spoke of the importance of benefits planning as a way to alleviate the concerns of Ticket holders seeking employment: You have Ticket holders trying to become employed and not knowing how it will affect their benefits. So to have a benefits specialist is an advantage. The challenge was with the AJC customers understanding how their benefits would be affected. It didn t affect how we run our program, but it s a matter of how the participant perceives the program. Rhode Island contracts with a local benefits planning provider to provide services to Ticket holders. In Florida (Round 3), the DEI State Lead became a CWIC. She is available by telephone, , and in-person (by appointment). Treatment LWIAs in the state have the option of contracting out for benefits planning services. One Florida DEI stakeholder stated that a former CWIC tells me to call her even though she doesn t have the WIPA grant anymore. There s also someone from the state s Agency for Persons with Disabilities who I can call who s certified as a CWIC. Massachusetts (Round 3) subcontracts its benefits planning to Work without Limits, a local provider, and some CWICs working for VR are co-located at DEI treatment sites and are there on a regular basis. They re available to DRCs and AJC employment counselors. In Massachusetts treatment sites, CWICS attend staff meetings and are in touch with the DRCs. They re part of the community partnership collaborative group. The DRCs refer to our CWICs and vice versa. It s constant communication. We re really part of the One-Stop treatment sites under the DEI, almost treated as staff members. Another DRC in Massachusetts spoke about the challenges of CWIC training and the value of benefits planning information: I took advantage of the training that was available through the DEI grant. I went to a benefits planning course last year. I didn t pass the first time, but I will take the test again. I learned a lot. We also have a CWIC we work with who comes into the Center. During our biweekly orientations, we have a CWIC come in to do an orientation, which gives the customer a lot of valuable information. In Louisiana (Round 3), DEI staff described benefits counseling as a gap in the available AJC services: clients who have SSI/SSDI are concerned about losing their benefits if they have employment. This is a big challenge in talking with the clients they need assurances that getting employed will help them: that they will be able to improve their lives. Most treatment LWIAs in Louisiana refer Ticket holders to local providers for benefits planning services. 80

81 Three out of five DRCs in Indiana have pursued benefits counseling certification. One DRC shared that she went through the Cornell University Work Incentive Practitioner certification and was credentialed in January. She reported that she: Touched on the topic of benefits planning with approximately 20 customers, but [has] not necessarily [had] a full in-depth benefits review. A lot of the people who come in with concerns are in the pre-contemplation phase and not actually ready to take the action step of getting a job. A lot of people don t want to know everything at once; they want to know a little at a time. The biggest selling point to getting people to take that step toward a job is the money in their pocket. I haven t had a person yet who was better off not working. Everyone I meet with needs more money every month. That s why they want to go back to work. Another Indiana DRC who went through the same certification process spoke about the time commitment required to provide benefits planning to Ticket holders: when you do benefits planning with an individual, it occupies a lot of time... we have figured it takes between 5 and 12 hours per person to do benefits planning properly. To alleviate the burden on DRCs, one local area in Indiana created an MOU with a local CWIC and uses a local community-based provider. One DEI treatment site in Minnesota already had two staff members trained as CWICs. They were more familiar with TTW and work incentives, which allowed them to be further along in their ability to serve Ticket holders than the other DEI sites. Another DEI treatment site in Minnesota had recently begun the CWIC process during the 2014 DEI Evaluation interviews. All of Iowa s (Round 3) DRCs in the five treatment LWIAs are certified benefits planners, which was a priority set by the DEI State Lead during the implementation stage of the grant with the goal of ensuring that we re all informed when talking with beneficiaries and can speak with one voice with regard to the impact of employment on SSA benefits. One Iowa DRC who received her CWIC certification prior to the DEI has also been participating in other trainings provided by the state and the Iowa Association for Workforce Partners, including Asset Development and financial planning using the Money Smart curriculum, Self-Employment services for individuals with disabilities, marketing TTW to Ticket holders, providing workforce services, and strategies for WIA enrollment for individuals with disabilities (Exhibit 14). 81

82 Exhibit 14: Rounds 1-3 Grantees with Certified Benefits Planners on Staff and/or Through Local Providers Alaska Arkansas Delaware Illinois Kansas Maine New Jersey New York Virginia STAFF ROUND 1 PROVIDERS ROUND 2 California Hawaii Ohio South Dakota - - Tennessee Wisconsin Washington ROUND 3 Florida Iowa Indiana Massachusetts Minnesota Louisiana Rhode Island The fear of losing disability benefits, including health insurance, has been recognized as a leading impediment to getting individuals with disabilities to return to work. DRCs reported that disability beneficiaries, their families, and various organizations with which they interact have historically lacked an understanding of the work incentives available to SSA beneficiaries. Across the board, DEI sites expressed the challenge of helping Ticket beneficiaries understand how the TTW Program operates and how it impacts SSDI benefits. Successful recruiting of Ticket holders requires AJC staff to be able to accurately determine the impact of earnings on disability benefits. LWIA staff has recognized that having access to trained benefits planners is an indispensable component of operating an EN. One Round 1 DRC described the challenges of recruiting Ticket beneficiaries who are reluctant to return to work full-time due to the concern that they will lose their medical benefits: 82

83 Anyone on Social Security is going to have anxiety about being self-sufficient, but you [have to] go over the [TTW] briefing with them and make them feel more comfortable. This could go on for a year, and then eventually maybe some will try part-time work. The issue is finding people who are willing to assign their Ticket. Most people want to work part-time because there will always be some fear about losing medical benefits. They re not willing to give that up because they expect it will be difficult to get medical benefits again. Most jobs they find don t offer benefits. Many people want to know about the program but are skeptical about assigning their Tickets. The practice of training DRCs as benefits specialists became widespread but not the unanimous choice across all DEI grantees. Some AJCs counted on CWICs on-site and, therefore, did not feel the need to invest in training the DRCs for that purpose. They reasoned that the benefits counseling training comes at a cost and can be very time-consuming. For instance, we spoke to DRCs who had yet to pass the exam at the end of the grant period to become certified after having invested the time to participate in all the training. Other states felt it would be too taxing on DRCs and would take attention away from other activities. Some sites had contracted with local agencies that did have CWICs and they felt that referring the client to the agency for benefits counseling worked well. New York (Round 1) had planned to focus on TTW from the beginning and therefore planned to have all DRCs trained in benefits counseling from the state of the grant period. Providing benefits counseling on the spot for Ticket holders interested in work was considered important to TTW success. Other Round 1 grantees also used DEI funds to pay for benefits specialist trainings for their DRCs or other AJC staff, including Virginia and Illinois. In a few cases, the DRCs hired already had the CWIC certification. Kansas (Round 1) connected Ticket customers to benefits specialists in partner organizations and the state noted that the CWICs in the state were spread thin. In Maine (Round 1), the DRCs did not do benefits counseling. The state used local CWICs. Maine s DEI leadership felt that it would be too taxing on DRCs and would take attention away from other activities. Round 2 states also provided benefit planning services by DRCs and/or contracting these services out. In Washington, all DRCs had completed the Cornell University benefits training. DEI personnel in Tennessee discussed the difficulty in serving customers who need benefits trainings after the SSA-funded benefits counselors (CWICs) were all released. LWIAs addressed this in different ways. Some partnered with banks, while others contracted with former CWICs to provide services to customers. The state-level DEI personnel considered providing benefits counseling training to the DRCs. They decided against it because providing benefits planning on top of all of their other responsibilities would have been too much. One Tennessee DRC described providing benefits planning along with paid work experiences in order to help address customers fears of losing their benefits if they took a paid position. In California, all treatment sites sent their DRCs to become certified benefit planners, although several used former CWICs from WIPA early on the grant period. In Ohio, the AJCs relied on community partners to provide benefits counseling to ticket customers. In Round 3 states Florida, Louisiana, Rhode Island, Massachusetts, and Minnesota, benefits counseling was being done by external service providers. In Louisiana, this was a stop-gap 83

84 measure until the DRCs finished the training in benefits counseling. Iowa has made benefits planning services a priority statewide. Currently, all of Iowa s DRCs are now certified benefits planner and independently of its DEI grant activities, Iowa is also working on developing a network of benefit planners across the state to be supported through Medicaid waivers. In Massachusetts, DRCs had some benefits counseling lessons but they also secured CWICs to refer to. In one site, the CWICs participated in IRTs. In Indiana, most sites have a certified benefits planner. As one DRC reported, due to the heavy focus on benefits planning within the TTW Program, I chose to pursue certification as a benefits planner. Another Indiana DRC who became a certified benefits planner through the DEI commented that benefits planning is a preliminary step that some TTW beneficiaries need before going back to work. Two DRCs in Florida were already certified benefits planners through the CWIC program and provide services to TTW beneficiaries in their sites. Other Florida sites contract with local providers for benefits planning services. One Florida DRC, a former Ticket holder, commented that his biggest challenge is convincing Ticket holders that what they are doing is in their best interest: They don t like the idea of having to go full-time and losing their benefits. So we send out a representative to do benefits trainings. I personally know how the program works, and benefits planning has become a tool for me to convince my job seekers that it is okay to move forward Other Staff Serving DEI Customers During each round of the DEI, some grantees have chosen to divide up the responsibilities for local DEI activities rather than assigning all tasks to a DRC. Some LWIAs chose to hire two DRCs or one DRC and one DRS, while others sought the help of community-based providers for benefits planning services, Self-Employment, or Customized Employment services. For example, Maine (Round 1) hired a statewide TTW coordinator to implement the program in DEI treatment sites rather than utilize DRCs. In one New Jersey (Round 1) LWIA, the responsibilities of one DRC were divided among AJC staff who received training to prepare them for specific DRC roles (e.g., employment counselor for JSWDs, benefits planning, employment engagement). During the DEI grant period, each DRC had the opportunity to experience each role to develop the LWIA s overall capacity to serve JSWDs. Delaware (Round 1) hired two part-time Employment Specialists to work with DEI customers on placement activities once the DRCs felt the customers were ready to begin a job search. One local area in Ohio (Round 2) relied on a Business Services team to do outreach to employers about hiring JSWDs. A treatment area in Tennessee used DEI funds to hire one staff member to recruit Ticket holders and other JSWDs and oversee work-based training opportunities such as apprenticeships, internships, job shadowing, and integrated employment opportunities. In Hawaii (Round 2), one treatment LWIA hired a Business Resources Inclusion Coordinator who assisted the AJCs in communicating with employers, provided work-based training opportunities, and 84

85 provided a range of DEI service delivery strategies, such as IRTs and Asset Development services, and TTW recruitment and benefits counseling. In Round 3, some grantees are also using a variety of DEI-funded positions to serve JSWDs. In Rhode Island, a DEI-dedicated Business Services Specialist was assigned to work closely with both JSWDs and employers to identify job opportunities. The Business Services Specialist provides soft skills trainings that target specific job placements and job readiness skills, including development of résumés and training on effective interviewing. The Business Services Specialist also conducts outreach to employers, providing information on job matching and résumé review services. Along with the Business Services Specialist, Rhode Island assigned a Customer Resource Specialist to work with JSWDs. The Customer Resource Specialist works directly with both Ticket and non-ticket JSWDs, providing basic benefits planning information. The Business Services Specialist and Customer Resource Specialist work closely with JSWDs and DRCs to provide information during the regular DEI team meetings and receive feedback from the DRC on potential courses of action for particular JSWDs. In Florida (Round 3), some treatment LWIAs hired or reassigned up to four additional staff members to support systems change efforts. Funding for other staff comes from various sources. Some areas have given staff specialized roles on the team, while others just have two or more DRCs who essentially do the same thing. Additional staff titles in various areas include: Disability Program Success Coaches, Job Developers, Account Managers, and Business Development Managers. There are general staff who work directly with customers, those who work with businesses, and those hired for DEI-specific administrative purposes such as WIA enrollment and TTW. Many DEI regions in Florida also have Business Services Representatives who conduct outreach to employers, review résumés, provide job matching information to JSWDs and DRCs, and monitor job placement through job retention services DEI Participant Characteristics and Customer Outcomes The DEI Evaluation is designed to address a variety of research questions pertaining to the process and outcomes of the DEI intervention. This section of the report lists the various quantitative research questions that apply to participants in the DEI adult and youth states. Each of these research questions will be discussed in greater detail in the subsections that follow. We have divided the list below into research questions that involve (1) adults and youth, (2) adults only, and (3) youth only. The list of research questions are grouped into the following topics: There are some additional research questions pertaining to subgroups of JSWDs in the treatment and control sites who receive SSA disability insurance (for example, SSDI and SSI). These questions pertain to the types of disabilities among the DEI customer population, the disability benefits received by these individuals, and the extent to which JSWDs who register at AJCs self-identify as having a disability. At this time, we will not include these SSA-related issues in our list of active research questions because they require data from SSA, which will not be made available to the evaluation team until the final year of the evaluation project. 85

86 Adult and Youth Processes and Outcomes Utilization of AJCs by DEI customers. How many DEI customers utilize job search and skill development services funded by WIA? What percentage exits the program? What percentage is still active in the program? Entry rate of JSWDs into the WIA program. Has DEI affected the entry rate of JSWDs into the WIA program? Exiting from WIA. How many DEI customers exit from the WIA program and when did they exit? What percentage of customers exit and are subsequently employed or (for youth only) placed in an educational program? Does the exit rate of DEI customers differ significantly between LWIAs randomly assigned to the evaluation s treatment and control groups? 40 Adult Outcomes Employment rate after exiting from WIA. What percentage of DEI customers becomes employed in the first quarter after exit from WIA services? Do treatment and control sites differ in the share of exiters who obtain employment? Employment retention rate. What percentage of DEI customers employed in the first quarter after exit continues to be employed in the third quarter after exit? Does the percentage of exiters with employment retention differ between DEI treatment and control sites? Earnings. Among JSWDs who become employed, what are their average earnings during the first, second, and third quarters after exit? Do the average earnings in these quarters after exit differ between the DEI treatment and control sites? Youth Outcomes Placement in employment or education. What percentage of youth obtains employment or enrolls in postsecondary education, advanced training, or occupational skills training in the first quarter after exit? Does the placement rate differ between the DEI treatment and control sites? Attainment of a degree or certificate. Of the youth who are enrolled in education, what percentage attains a diploma, GED, or certificate by the end of the third quarter after the exit quarter? Does the degree or certification completion rate differ between the DEI treatment and control sites? Literacy and numeracy gains. Of the out-of-school youth who are deficient in basic skills, what percentage advances one or more educational functioning levels? Do the literacy and numeracy gains differ between the DEI treatment and control sites? The first set of research questions about overall utilization of DEI services and exits from the WIA program involve important process and descriptive indicators. These are derived directly from the requirements of the outcome evaluation. The remaining research questions stem mainly 40 Throughout this chapter, we use pilot sites to refer to the full set of LWIAs in which the DEI was implemented and treatment sites to refer to the subset of pilot sites chosen through random assignment. 86

87 from the core indicators of performance for the WIA adult, dislocated worker, and youth programs. The specifics of these indicators, which are called Common Measures, are promulgated in TEGL (2005). 41 Social Dynamics and Mathematica Policy Research decided to utilize Common Measures outcomes in our analysis because they are frequently used by DOL and they are readily understood by DOL stakeholders. However, we adapted some of these measures to meet the needs of the DEI Evaluation because the Common Measures were developed to assess WIA performance using snapshots of information available in a given calendar quarter. 42 In contrast, the evaluation requires tracking participants longitudinally to measure programmatic outcomes and impacts for a fixed cohort of DEI customers who applied for and received WIA services during the DEI grant operation period for each state. This allows us to establish the direct relationship between enrollment in WIA, exiting WIA, and obtaining employment needed to assess the likely effects of the DEI intervention. Given that the longitudinal analysis focuses on WIA entrants, differences between the randomly selected treatment and control groups could potentially reflect impacts of the DEI on program participants or differences in the types of customers entering as a result of the DEI. Although we cannot rule out the latter possibility, as discussed in Section 9.3, we found small and statistically insignificant treatment-control differences in the pre-wia characteristics of customers included in the analysis for adult states. However, when conducting a similar analysis of customers in youth states, we found large and significant differences in some of these pre-wia characteristics. 9.1 Description of the Population of DEI Customers in the Pilot LWIAs Who participated in the DEI in the pilot LWIAs where it was implemented? How many participants enrolled in WIA programs and either exited or remained active in WIA at the end of the demonstration period? What was their pattern of participation across the states? These descriptive questions will be addressed in this section. DEI grants operated in Round 1 states between October 1, 2011, and September 30, 2013, and in Round 2 states between October 1, 2012, and September 30, The states that applied for a DEI grant were required to determine which LWIAs would be potential pilot sites for the initiative. Most states participated in a random assignment process whereby a subset of potential sites was randomly selected to be in the treatment group of LWIAs for the evaluation s RCT. In WIOA changed some of the Common Measures, aligned the performance indicators for core programs, and added new ones related to services to employers and postsecondary credential attainment. These new requirements became effective in July Because this evaluation largely covers JSWDs who entered WIA programs prior that point, however, we use the WIA requirements. The specific outcome measures used in this report improve in two ways upon what was presented in the Year 3 Synthesis Report (February 18, 2014), which covered DEI participants and outcomes based on data from Round 1 states. First, the current report uses updated definitions, based on discussions with Employment and Training Administration staff that we understand to be better aligned with DOL s objectives for learning about the DEI. Second, in response to DOL comments from an earlier draft, we carefully reviewed and, if applicable, revised the analysis measures and methods to better account for the evaluation s clustered random assignment design. 87

88 these states, the other LWIAs not randomly selected to have DEI services constituted the control group. In this section we present descriptive information about all pilot LWIAs both those in the treatment group and those that were selected non-randomly in a subset of states. This descriptive information is based on data contained in WIASRD extracts obtained from each of the Round 1 and Round 2 states Methodology Data Source The primary data source for the DEI evaluation is the WIASRD, which consists of quarterly records on WIA customers characteristics, activities, and outcomes. These standardized records are maintained by state workforce investment agencies for all individuals who receive services or benefits from programs funded by WIA Title IB. States track the types and duration of WIA services received, as well as participants outcomes after exiting from the program. Each state is required to submit quarterly extracts of the data to DOL, ETA. Social Dynamics completed DSAs with the various DEI grantees, which required the states to submit quarterly WIASRD extracts to the evaluation contractor for the duration of the DEI grant operation period. These extracts constituted a series of cross-sectional snapshots of the services received by WIA customers and the outcomes of exiters in each quarter. Based on the quarterly data, we developed a longitudinal analysis dataset with a single record per WIA entrant containing the most up-to-date information about service receipt and outcomes. There were no exclusions in the dataset, which is the baseline from which all of the study s analysis samples are derived. The descriptive tables in this section are based on a universe of adult and youth customers with disabilities who received services in pilot LWIAs during the full two years of DEI grant operations. Definitions An important first question is: who are DEI customers? Throughout the report, we use the following definitions of DEI customers (whom we also refer to as DEI participants ). Adult DEI customer. For those DEI states that selected an adult focus, we define a DEI adult customer for the purposes of this evaluation as an individual who: o Enrolled at an AJC in a DEI pilot LWIA and participated in (1) WIA Adult and/or Dislocated Worker Program activities and/or (2) WIA Youth Program activities and was at least 19 years old at the time of enrollment o Started receiving WIA services during the eight quarters of the DEI operation period for each grantee o Self-identified as having a disability (as reflected in the WIASRD system) o Received WIA intensive services and/or training 88

89 o Participated in a WIA program for more than one day Youth DEI customer. For those states that selected a youth focus, we define a DEI youth customer for this evaluation as an individual who: o Enrolled at an AJC in a DEI pilot site and participated in WIA Youth Program activities o Started receiving WIA services during the eight quarters of the DEI operation period for each grantee o Self-identified as having a disability (as reflected in the WIASRD system) o Participated in a WIA program for more than one day Description of DEI Customers in the Pilot Sites Exhibits 15 to 23 describe the distribution of DEI customers in the various pilot sites in order to provide a comprehensive snapshot of the universe of participants for all Round 1 and Round 2 states for the whole of the DEI grant operation period Number of DEI Adult and Youth Customers How many DEI customers are there in the Round 1 and Round 2 states? Exhibit 15 provides the count of DEI customers in the adult and youth states, broken down by DEI round. In the 12 DEI adult grantees in Rounds 1 and 2, there were 48 pilot LWIAs and the universe of adult DEI customers was 7450 participants. The breakdown of participants by DEI round was 71.3 percent in Round 1 states and 28.7 percent in Round 2 states. In the four youth states participating in Rounds 1 and 2 of the DEI, there were nine pilot LWIAs and the universe of youth DEI customers was 636 participants. There is a major imbalance in the number of participants by DEI round, with 98.9 percent of participants coming from the Round 1 programs in Arkansas, Delaware, and New Jersey. South Dakota is the only youth state in Round 2, and there were only 7 participants enrolled in WIA. Exhibit 15: Number of DEI Participants in the Pilot LWIAs DEI Focus Round 1 Round 2 Total Adult States Youth States % 28.7% 100.0% % 1.1% 100.0% Distribution of DEI Customers, by State What is the distribution of DEI customers across the various DEI states? Exhibits 16 and 17 present data on the percentage of total DEI customers, by state, for adult and youth states, respectively. The states are sorted from high to low in descending order, with the state on the left representing the highest percentage of total DEI customers. 89

90 In Exhibit 16, which describes the distribution of DEI adult customers among the states with an adult focus, we can see that New York, for instance, has 65.5 percent of adult DEI participants, followed by California with 16.6 percent of adult DEI participants. The remaining 10 DEI adult states account for 18 percent of adult DEI participants, with state-specific percentages ranging from a high of 5 percent in Wisconsin to a low of 0.2 percent in Hawaii. Exhibit 16: Distribution of DEI Customers, by Adult State Exhibit 17 displays the distribution of DEI youth customers among states with a youth focus. New Jersey contained 67.3 percent of all DEI youth participants, followed by Arkansas (20.4 percent) and Delaware (11.2 percent). South Dakota accounted for just 1.1 percent of total youth participants. Exhibit 17: Distribution of DEI Customers, by Youth State 90

91 Entry Flow of DEI Customers Over the Grant Operation Period When do DEI customers enroll in WIA and to what extent does the distribution of enrollments differ across all quarters? Was there seasonality in when customers started participating in WIAfunded programs, and were there changes in entry patterns over the grant operation periods? Exhibits 18 and 19 provide information to answer these questions by depicting the number of DEI customers entering WIA programs during each grant operation quarter of the adult and youth states, respectively. The entry quarters are standardized for all DEI customers in relation to the start of each state s grant period. 43 This will enable us to assess how the flow of WIA enrollments evolved over the DEI grant operations period. In addition, these graphs include all DEI customers, regardless of their exit status at the end of the initiative. Exhibit 18 is a bar graph displaying the number of DEI customers first enrolling in a WIA program during each of the eight quarters of the grant operation period in states with an adult focus. The graph shows that 901 to 910 adult DEI customers entered in each of the first three grant operation quarters, and then there was a modest increase to 977 in quarter 4. The flow of entry in the second grant operation year was not quite as uniform as the first year. Only 811 DEI customers entered in quarter 5, followed by 1030 in quarter 6. The share of entrants in quarters 7 and 8 dipped to 919 and 922, respectively. There does not seem to be any clear pattern in the entry timing of the DEI participants over the duration of the initiative. Exhibit 18: Entry Flow of DEI Customers, by Grant Operation Quarter (Adult States) 43 Because the operational periods of each DEI grant round spanned different two-year intervals, we used a standardized measure of entry timing that allowed us to pool information across multiple rounds. Specifically, we measured the customers quarter of first enrollment in a WIA program relative to the start of the grant operations period. For example, customers entering during the first quarter of a grantee s operation period were assigned a standardized entry quarter of 1, those entering in the following quarter were assigned a standardized entry quarter of 2, and so on. Because the grant operation period for both Round 1 and Round 2 grantees began in October, standardized entry quarters for each round correspond to the same season of the calendar year. 91

92 Exhibit 19 shows the percentage of DEI customers in youth states first enrolling in a WIA program, by grant operation quarter. The graph shows that a relatively large number of DEI youth customers entered during the last quarters of each grant operation year, with 186 first enrolling in the 4th quarter and 162 enrolling in the 8th quarter. The number of DEI youth customers who enrolled in the remaining entry quarters varied from a low of 27 in the 6th quarter to 73 in the 1st quarter of the grant operation period. Thus, it appears that enrollment in youth programs is highly seasonal, with the highest enrollment occurring from July to September. This pattern is very different from that exhibited in the adult states, but accords with our understanding of how intake into the youth training and workforce program fluctuates in relation to the school year. Exhibit 19: Entry Flow of DEI Customers, by Grant Operation Quarter (Youth States) DEI Customers Exit Status at the End of Grant Operations What is the programmatic status of DEI customers? In other words, how many customers exited the program and how many were still active in the program when the states grants ended? Exhibits 20 and 21 describe the program status of DEI customers in the adult and youth states, respectively. DEI exiters are those customers who exited the WIA program at some time during the DEI operation period, as recorded in WIASRD. Customers without a recorded exit continue to receive WIA services or training and are considered to be active in the program. This distinction is important because several outcome indicators for the DEI program only involve WIA exiters. Exhibit 20 shows the status of DEI customers in the adult states at the end of the DEI operation period. From this table, we can see that a total of 7450 adults in the pilot LWIAs received WIA services at an AJC during the DEI period. Overall, 54 percent of adult customers had exited and 46 percent were active. 92

93 Exhibit 20: Number of Active and Exiter DEI Customers at the End of the Grant Operation Period (Adult States) DEI Status Active Exiter Total % 54.0% 100.0% Exhibit 21 shows the status of DEI customers in the youth states at the end of the grant operation period, by round. From this table, we can see that a total of 636 youth in the pilot LWIAs received WIA services at an AJC during the DEI period. Overall, the percentage of exiters was 27.7 percent and the percentage of actives was 72.3 percent. Exhibit 21: Number of Active and Exiter DEI Customers at the End of the Grant Operation Period (Youth States) DEI Status Active Exiter Total % 27.7% 100.0% Exhibits 22 and 23 continue our analysis of active and exiter DEI customers from the perspective of the customer s entry quarter in the adult and youth states, respectively. These exhibits show conclusively that individuals entering earlier in the grant operations period are more likely to have exited by the end, a result that follows from the structure of the WIASRD extracts available for this study. Customers exit from WIA programs after enough time has elapsed for them to meet their goals for participation. Ideally, we would track each customer for an equal and sufficiently large amount of time to consistently observe their exit patterns. However, the state WIASRD extracts available for the evaluation included information that was updated only through the end of the DEI (i.e., quarter 8 of grant operations). The result is that we have progressively fewer quarters of post-entry information on customers who entered later during the grant operation period. This means that there is simply less of an opportunity to observe later entrants eventual exits. In the next section, we will discuss the issues of data censoring in greater detail and how we accounted for it in our analysis. 93

94 Exhibit 22: Distribution of Active and Exiter DEI Customers at the End of the Grant Operation Period, by Entry Quarter (Adult States) Exhibit 23: Distribution of Active and Exiter DEI Customers at the End of the Grant Operation Period, by Entry Quarter (Youth States) 9.2 Creation of the RCT Analysis Sample This section of the report discusses how we created our main working sample for the RCT analysis of the DEI evaluation. This RCT analysis sample includes (1) a treatment group, consisting of DEI customers in one or more pilot LWIAs that states selected through random assignment, and (2) a control group of DEI-like customers in one or more randomly assigned 94