To: U.S. Department of Labor and U.S. Department of Education From: Workforce Data Quality Campaign Re: Workforce Innovation and Opportunity Act (WIOA) Notice of Proposed Rulemaking Date: June 12, 2015 Workforce Data Quality Campaign (WDQC) is a non-profit initiative that promotes inclusive, aligned and market-relevant education and workforce data. Guided by a diverse group of national partners and state officials representing stakeholders across the education/workforce spectrum, WDQC encourages the use of data to ensure that all of our nation s education and training programs are preparing students and workers to succeed in a changing economy. The following are WDQC s comments in response to the Workforce Innovation and Opportunity Act (WIOA) Notice of Proposed Rulemaking (NPRM) RIN 1205-AB73 (Docket No. ETA-2015-0001), implementing Title I and Title III of WIOA, and NPRM RIN 1205-AB74 (Docket No. ETA-2015-0002), Joint Rule for Unified and Combined State Plans, Performance Accountability, and the OneStop System Joint Provisions. We appreciate the opportunity to offer recommendations relating to performance reporting, accountability, and data management to the U.S. Departments of Labor and Education (the Departments). WIOA contains a number of changes from WIA that will improve cross-program data and performance measurement to help create a system of shared accountability across core programs, and potentially other education and workforce services. These changes include establishing common performance indicators for the core programs and for training providers, common methods for establishing adjusted levels of performance, common requirements for performance reports, and steps to facilitate common data collection and linking. WDQC supports this approach, which incentivizes programs to work together to achieve positive results for individuals and employers. We commend the Departments for collaborating to produce regulations that will enable WIOA programs to work in partnership towards common goals and strengthen the data infrastructure required to measure success. Wage record access and use WDQC applauds the Departments for 20 CFR 677.175, which acknowledges Unemployment Insurance (UI) wage records as the best source for WIOA reporting. We agree, but note that there are gaps in wage records, including those who are self-employed. ACTION: We urge the Departments to issue additional guidance, as was done under WIA, on acceptable ways to track employment outcomes for participants for whom wage record matching is not a viable solution. Furthermore, the indicator outcomes shown on required performance reports should incorporate information from allowable alternatives to wage records.
WDQC also appreciates the commitment to renegotiate the Wage Record Interchange System (WRIS) agreement to allow all WIOA core programs to use the system for performance reporting. ACTION: We encourage the Departments to also make clear that all the core programs may use the Federal Employment Data Exchange System (FEDES) for WIOA performance reporting. Guidance to state agencies could assist in raising awareness of FEDES. Eligible Training Provider (ETP) performance reporting We applaud the Departments for proposing regulations that provide significant improvements to the data infrastructure needed for ETP performance reporting. The proposed language appropriately describes the Governor s role, including designating a state agency or entity to assist the Governor and coordinate UI wage record matching. We are especially pleased to see the clarification of the public official and official duties definitions for disclosure of confidential UI information. Allowing public education entities to receive UI wage records to conduct matching with participant data will give states critical flexibility in setting up reporting processes and may alleviate burden on state UI agencies. Overall, the proposed regulations facilitate the use of UI wage records for reporting, research, and evaluation while preserving important privacy and security protections for personally identifiable information. ACTION: To reduce burden on training providers in complying with performance requirements, WDQC recommends that ETP reporting and data submitted for ETP eligibility determination should, to the extent possible, be identical. Furthermore, required elements should align with reporting that community colleges already do for state education authorities, so that they do not have to create new reports. The regulations do not allow access to individual UI wage records for non-public training providers, i.e. for-profit colleges and community-based organizations. It will be challenging and potentially expensive for many of these providers to set up initial data sharing agreements with an appropriate statewide entity that can collect and match their data. WDQC proposes a way to facilitate non-public providers engaging with the WIOA system and setting up the required performance reporting. Training providers want access to individual-level wage data so they can follow up with past participants who do not appear in the wage records, both to enhance their performance levels and to offer additional services to those who may still be facing unemployment. ACTION: For non-public training providers, who cannot access individual records, we recommend that regulations provide that the data these entities submitted for training participants not found in the UI wage records be returned to the provider, indicating that the records do not match UI records. This would allow ETPs to focus their follow-up efforts on participants who do not appear in UI wage records, relieving data collection burdens on the individual participant and the non-public training providers. Returning data as proposed does not disclose any sensitive confidential information to non-public providers. Regulations should specify that knowledge of whether or not an individual was found in the UI wage records should only be used by non-public training providers to audit performance levels and may not be shared with any other entities. The data from all providers, both public and non-public, will be most useful to a variety of stakeholders if it can be kept longitudinally by the state and utilized not only for year-to-year aggregate performance reporting, but also for longitudinal research and evaluation to improve the mix and targeting of 2 of 6 pages
program services. ACTION: The Departments should present a united vision through appropriate guidance and technical assistance of how data shared for WIOA performance reporting may be incorporated into state longitudinal data systems in compliance with both UI confidentiality provisions and the Family Educational Rights and Privacy Act (FERPA). Finally, although the federal regulations lay a solid foundation for reporting, states will need technical assistance, resources, and time to set up the processes required for WIOA performance reporting. Implementation will be particularly challenging where state laws include additional restrictions on data sharing beyond the federal rules. The Departments should consider how regulations and guidance can provide a path to allow states to interpret (or revise) their own laws to allow greater access to data for strategic planning and evaluation purposes. Defining performance indicators The proposed definition of exit is critical for WIOA program performance. The proposed language for 20 CFR 677.150 defines exit as ending services within an individual program, but the Departments note that they are considering a common exit definition that would require individuals to exit all core program services before they would show up in performance indicators for any single program. ACTION: WDQC recommends a common exit definition, which would support career pathways and other cross-program participation that can benefit participants. In order to properly implement such a definition, it is crucial for states to have performance management systems that can accurately track co-enrollment. ACTION: WDQC recommends that the Departments give further consideration to defining exit and participation for the purposes of required eligible training provider reporting on all students. While the proposed definitions are appropriate for WIOA accountability purposes and determining which populations should be counted as having received services under the Act, they are not applicable to the entire population of students in a training program who must be reported on in ETP reports. The definitions of participant and exit are not appropriate for the way that most students enroll in and progress through postsecondary education programs at institutions of higher education. For instance, the 90-days-to-exit provision could be the equivalent of a student not taking classes for a summer term. The exit definition also raises questions for student transfer between institutions or programs. In addition, postsecondary students participate in a variety of ways that may not be captured by the participant definition. For example, many students never officially enroll in a program of study, instead taking courses for a period of time before deciding on a particular degree pathway. It is currently unclear what population of students should be included in reporting, and additional guidance is necessary. When possible, this guidance should seek to utilize existing data sources and definitions to streamline data collection and burden on institutions. The Departments request comments on using the performance indicators identified in 20 CFR 677.155 for additional programs beyond the core programs. WDQC supports the use of common metrics across education and workforce programs whenever possible and appropriate, recognizing that decisions about metrics should carefully consider differing program goals and structures. The WIOA primary indicators would work well for a variety of programs, such as H-1B training grants and veterans state grants. 3 of 6 pages
Proposed 20 CFR 677.155 defines the performance indicators. For the second indicator, the Departments are considering calculating an employment retention rate as was done under WIA to supplement the statutorily required employment rate measured four quarters after exit. ACTION: We recommend against calculating an employment retention rate because: This will be a confusing data point when viewed side-by-side with the statutory longer-term employment rate indicator. It does not add significantly to our understanding of how programs are serving people. As implemented under WIA, this was not a true retention measure. It looked at whether exiters who got employed remained employed, but those in the measure were not necessarily with the same employer. The required employment rates taken for all exiters at the second quarter and the fourth quarter will tell us basically the same thing: whether people can find jobs after exiting and maintain employability over a period of time. A true retention measure (i.e. one that matches employer-specific codes) would not necessarily show program success, since people may benefit by switching jobs to earn higher wages. The Departments also request comments on whether and how to collect information on the quality of employment. ACTION: The Departments could explore options for monitoring investments in training and education that help participants access jobs offering wages that : 1) are above the average for entry level in an targeted sector s growth occupations; 2) provide benefits beyond direct wages (such as health, retirement, or child care benefits); and/or 3) offer wages sufficient to sustain the individual worker and his/her immediate dependents (based on calculated self-sufficiency wages that recognize regional cost differences). Quality of employment is a complicated thing to measure and would be brand new for the workforce system, so it may be most appropriate to avoid defining these metrics in regulation and instead consult with stakeholders about possibly developing them at a later time. Measuring employer effectiveness ACTION: The measure of effectiveness in serving employers, Proposed 20 CFR 677.155(a)(1)(vi), should be a shared or common measure or measures across WIOA. A program-by-program approach could lead to competition and duplication among programs seeking employer engagement. ACTION: The Departments should look to the employer community to lead efforts on defining and operationalizing these measures. Over time, the Departments should continue to engage the employer community to discover better methods of measuring effectiveness in serving employers, perhaps due to improvements in technology. Adjusting performance targets We appreciate the Departments commitment to using statistical modeling to adjust performance levels both in advance of the program year to assist with negotiating ambitious but realistic performance targets that allow service to those most in need, and following the program year to account for actual economic conditions and participant characteristics. Modeling will develop different 4 of 6 pages
adjustment coefficients for each program and each indicator, but different types of participants within a program (e.g. training recipients vs. non-training recipients) should not be modeled separately. ACTION: We recommend regular updates to the adjustment models that use additional, more timely program records to re-calculate the adjustment coefficients at least every three years. It is important to use multiple years of data because larger sample sizes over time will result in more accurately predictive coefficients. Because WIOA does not allow a federal longitudinal database of participant data, the Departments could take advantage of evolving state longitudinal data systems to collect records from multiple states to create a sample which can be reasonably expected to be nationally representative. Workforce Data Quality Campaign would be happy to facilitate conversations with leading states about how this might be done. The Departments request comment on whether any additional factors beyond those in the statute and in 20 CFR 677.170 should be considered in developing the models. ACTION: The Departments could consider including race and Hispanic ethnicity as additional factors. State regression analysis has found these factors to be correlated with some performance outcomes. Unfortunately, due to discrimination, there are barriers to education and employment associated with race and ethnicity that may be independent of the other factors. The Departments could analyze WIOA data to test collinearity and see whether race/ethnicity might contribute in a statistically significant way to explaining outcomes. Performance scorecards We look forward to providing feedback on the required reporting templates through the Paperwork Reduction Act (PRA) process. ACTION: As the Departments work on template design, WDQC suggests that they also make arrangements to post comprehensive aggregate outcome data, which does not disclose any personally identifiable information, in an easily downloadable format so that states and entrepreneurs may also develop tools to better reach particular stakeholders. Credential reporting WIOA presents a great opportunity to learn more about the credentials being earned by participants in the workforce system. The credential attainment measure includes a variety of credentials, including licenses and certifications, which are often challenging to track but are important to acknowledge, as they appear to have significant value in the labor market. Federal regulations and guidance on credential attainment reporting should strike a balance between incentivizing collection of better data and unfairly penalizing states that do not have the ability to reliably measure attainment of all types of credentials. ACTION: The Departments could consider a phased approach for making licenses and certifications part of performance levels, but in all cases, programs should have to start reporting for informational purposes on all types of credentials by type (i.e. degree, certificate, license, or certification) received by participants. This would build system capacity and provide valuable information about how different types of credentials correlate with labor market outcomes. ACTION: The Departments should collaborate with employers and business associations to offer clear guidance and technical assistance to help programs determine what credentials are industry- 5 of 6 pages
recognized, and support efforts by employers, industry associations, and other organizations to establish criteria to verify credential quality. Workforce and Labor Market Information WDQC applauds the proposed language in 20 CFR 652.300 for codifying the workforce and labor market information requirements in WIOA and creating a platform for their implementation. However, we suggest small additions to the definition of workforce and labor market information and proposed in 20 CFR 651. ACTION: In the definition s first sentence, it would be useful to specify that the decisions are being made by students, workers, employers, and educators. In addition, we suggest that the list of examples in the definition also include: occupational employment statistics and the Standard Occupational Classification system occupation-specific requirements in terms of education, training, skills, knowledge, and experience self-employment data credential attainment ACTION: For the definition of workforce and labor market information system also proposed in 20 CFR 651, we suggest adding "Federal-state cooperative" before "system" and indicating the particular partners at the federal and state levels, including Bureau of Labor Statistics, Employment and Training Administration, Census Bureau, National Center for Education Statistics, National Center for Science and Engineering Statistics, state labor market information agencies, and state educational agencies. Finally, WDQC supports proposed language in 20 CFR 652.302 that recognizes wage records as a critical part of the labor market information system and directs the Department of Labor, in consultation with other federal agencies, states, and the Workforce Information Advisory Council, to develop standard definitions for wage records and help improve their collection and reporting. Standard definitions will help wage records be more consistent across states, thus aiding in stronger reporting of workforce program outcomes. Conclusion WDQC thanks the Departments for the opportunity to comment, and for their ongoing work to improve federal programs that help America s students, workers, and businesses meet their full potential. Our comments were informed by conversations with diverse stakeholders from around the nation, and are supported by: Association for Career and Technical Education Center for Law and Social Policy (CLASP) Council for Community and Economic Research National Association of State Directors of Career Technical Education Consortium National Skills Coalition New America U.S. Chamber of Commerce 6 of 6 pages