NIRS Data Coordinators Meeting

Transcription

1 2016 NAVIGATING CHANGE: Building Our Future Together NIRS Data Coordinators Meeting Welcome! AUCD 2016 CONFERENCE

2 2016 NAVIGATING CHANGE: Building Our Future Together UCEDD Topics Overview of the Program Performance Reports (PPR) data Social media reporting Discussion of Common Topics Custom Reports Data Entry Questions Roundtable Discussions Multiple surveys completed by one Trainee Collecting gender and other ethnographic data on less frequently contacted Trainees Categorizing an activity MCH Topics Overview of the prior year s Program Performance Reports Reporting to MCHB New MCHB reporting requirements NIRS and Reporting to MCHB NIRS to EHB reporting Having demographic data match across NIRS and EHB NIRS Projects for the Current Fiscal Year AUCD 2016 CONFERENCE Agenda

3 Annual Report- PPR Data UCEDD Data Coordinators Meeting December 4, 2016

4 Quality Review System- A 3 Tiered Model Tier I - Annual PPR Desk Review (compliance and outcomes) Tier 2 - Periodic in-depth Compliance & Outcome Review (Peer review 5year Core Grant Applications /Close out Report & site visit) Technical Assistance Tier 3 - On-site review for red flag programs

5 Background Why Collect Data? 1. Required by DD Act- According to Section 154(e) of the DD Act, entities designated as UCEDDs must submit an Annual Report that provides information for the AIDD system of program accountability. This system is designed to: Monitor entities that received funds under the DD Act of 2000 to carry out its activities; Determine the extent to which the entities have been responsive to the purpose of the DD Act of 2000; and Determine the extent to which the entities have taken actions consistent with the policy described in Section 101(c) of the DD Act of Required by the 5 year UCEDD Core Grant Funding Opportunity Announcement- Application

6 Why Collect Data? Annual Report-PPR The AIDD system of program accountability Is comprised of three parts: (1) the UCEDD reporting requirements, (2) Indicators of Progress for the National Network of UCEDDs, and (3) Government Performance and Results Act (GPRA) measures. UCEDD Annual Report template or Program Performance Report (PPR) requires that grantees submit data on progress made in the previous year towards achieving the projected goals for the UCEDD, including

7 Program Performance Reports Detailed Work Plan Progress Report Each UCEDD has a work plan for the 5-year plan of the 5 year application. Guidance requires the UCEDD to provide quantitative monthly or quarterly projections of the accomplishments to be achieved for each function or activity in such terms as the number of people to be served and the number of activities accomplished. Logic Model Evaluation Plan

8 Why Collect Data? Budget Justification Departmental Budget Justification- Congress FY 2017 ACL Congressional Justification UCEDD- P179 AIDD Biannual Report to Congress Congressional Inquiries Inquiries from public Identifies Areas for T/ TA Marketing Tool

9 Why Collect Data?

10 ACL Website Posting UCEDD Data ACL webpage on Data and Evaluations will have UCEDD data posted: Provide data on the UCEDDs- a short resource document on UCEDDS and the PPR data 1

11

12

13 Program Progress Reports-Update We have finished our review of the PPRs! Grantee Notifications We also have finished a preliminary review of the data- and found 2 outliers that we re following up on

14 FY 2015 Preliminary PPR Results UCEDD's FY 2015 Progress Measures ( Outputs) Summary Progress Measures: FY 2014 FY 2015 Variance Participants in Community Training Programs By Areas Of Emphasis 158, ,711 20% Continuing Education Programs By Areas Of Emphasis 179, ,518-39% Interdisciplinary Pre-Services Trainings Program 53,164 57,553 8% Technical Assistance Programs BY Areas of Emphasis 442, ,920 6% Participants in Model Services - Specialized Direct Services Offered by Areas of Emphasis- enhance well being.status of recipient Participants in Other Model Services - Specialized Direct Services Offered by Areas of Emphasis Participants in Demonstration Services Provided By Areas of Emphasis field tested- promising best practices 90, ,607 12% 75,139 82,871 10% 9,184 12,588 37% Participants in Research and Evaluation Activities BY Areas of Emphasis Information Dissemination Products Created By UCEDDS- Product type - 7% were webbsased 396,449 95,396-76% 10,309 9, %

15 Customer Survey Data 1. Helps AIDD in compliance reviews of annual reports- Does the UCEDD Have an accessible website? Survey at least 50 people? 2. Helps AIDD and AUCD identify trends and how responsive the UCEDDs are to the needs of their user community including local and state community of PWDD and their families across the network. 3. Helps AIDD isolate outliers and pinpoint areas where TA may be needed -

16 Customer Survey Data FY2015 Total Surveyed 13,452 Total Responses 4,899 37% Percentage of total responses: How satisfied were individuals with the information on UCEDD s website 3,125 65% Satisfied 1,354 28% Not Satisfied 342 7% Not Satisfied 73% 1%

17 FY 2015 Preliminary PPR Results Community Training Percent Reporting on Increase in Knowledge Gained # of Community Training Events: 478 Total Surveys 20,678 Total Responses 14,207 Response Rate 69% Highly Satisfied # 10,166 % 72% Satisfied # 3,824 % 27% Somewhat Satisfied # 141 % 1% Not Satisfied # 76 % 1%

18 FY 2015 Preliminary PPR Results Community Education # of Community Education Events: 810 Percent reporting on Increase in Knowledge Gained Total Surveys 13,980 Total Response 12,077 Response Rate 86% Highly Satisfied # 9,525 % 79% Satisfied # 2,399 % 20% Somewhat Satisfied # 105 % 1% Not Satisfied # 48 % 0%

19 FY 2015 Preliminary PPR Results Technical Assistance Total number of technical assistance events: 2,777 Total number of technical assistance hours: 276,064 Percent Reporting on Increase in Knowledge Gained Total # of Surveys 48,535 Survey Responses 33,089 Response Rate 68% Strongly Agree # % 79% Agree # 6,201 % 19% Disagree # 535 % 2% Strongly Disagree # 110 % 0%

20 FY 2015 Preliminary PPR Results Model Services Number of model services : 890 Participants in Model Services - Specialized Direct Services Offered by Areas of Emphasis Areas of Emphasis Number of participants Percentage Childcare 363 0% Education 11,520 11% Employment 107 0% Health 82,752 82% Housing 696 1% Quality Assurance 1,407 1% Quality of Life 1,634 2% Recreation 279 0% Other 755 1% Other - Assistive Technology 993 1% Other - Cultural Diversity 33 0% Other - Leadership 68 0% Total FY , %

21 FY 2015 Preliminary PPR Results Interdisciplinary Pre-Services Trainings Number of Interdisciplinary Pre-Services Trainings : 2,540 Participants in Interdisciplinary Pre-Services Trainings Program By Areas of Emphasis Areas of Emphasis # of Participants Percentage Childcare 143 0% Education 10,832 19% Employment 713 1% Health 26,524 46% Housing 206 0% Other 2,288 4% Quality Assurance 1,713 3% Quality of Life 2,535 4% Recreation 296 1% Other - Assistive Technology 752 1% Other - Cultural Diversity 1,465 3% Other - Leadership 10,086 18% Total FY , %

22 1, ,371 FY 2015 Preliminary PPR Results Leadership Setting - Identified as in leadership position UCEDD Trainee Survey Data - FY 2015 Trainees Surveyed at 1,5, and 10 Year Trainees in Leadership Position Count By Title # of individuals with DD receiving direct services from your activities Percentage Academic ,270 18% Academic, Advocacy 62 7,662 7% Academic, Clinical ,821 14% Academic,Clinical,Advocacy 21 6,613 6% Academic,Clinical,Public Health % Academic,Clinical,Public Health, % Advocacy Academic, None 1 0 0% Academic, Public Health 22 1,523 1% Academic, Public Health, % Advocacy Advocacy 76 29,105 28% Clinical ,553 16% Clinical, Advocacy 14 1,402 1% Clinical, Public Health 17 1,183 1% Clinical, Public Health, Advocacy % Public Health 55 1,790 2% Public Health, Advocacy 5 2,034 2%

23 FY 2015 Preliminary PPR Results UCEDD FY2015 GPRA Results GPRA Reported in Annual Reports GPRA Outcomes FY 2015 FY2014 Measure 1: What is the number of individuals with developmental disabilities who are receiving direct services through activities in which you are involved? 739, ,368 Measure 2: Are you in a leadership position in the field of developmental disabilities? Yes 1,221 1,200 Percentage of Yes respondents 35% 31%

24 FY 2015 Preliminary PPR Results UCEDD's FY 2015 GPRA Outcomes and Outputs Calculated Outcomes Measure FY D Increase % of individuals w DD receiving the benefit of services through activities in which professionals were involved who completed University Centers of Excellence in Developmental Disabilities (UCEDDs) training within the past 10 years. (Outcome) Target: 41.88% Target: 42.42% (Target Exceeded) 8viii: Number of professionals trained by UCEDDs. (Output) 5,546 8ix: Number of people reached through UCEDD community training and technical assistance activities. (Output) 8x: Number of people receiving direct or model demonstration services from UCEDDs. (Output) actual 770,149 actual 100,607 actual

25 PPR near future Look to include diversity data on leadership, faculty and staff in PPR New Evaluation Measures for the 5 Year Close-out report in the QRS Tier 2 Review Process.

26 FY2016 Social Media Survey Results Does your Center use social media? 35 What type(s) of social media does your Center use? 33 3% 30 97% Yes No Number of Centers Facebook Twitter LinkedIn YouTube Instagram Flickr Google+ Other Examples of social media use: promotional sharing of Center news, events and accomplishments information dissemination, including products networking with trainees, professionals and industry leaders recruitment educational videos and modules

27 FY2016 Social Media Survey Results How often does your Center use social media? Does your Center have policies regarding social media use? 9% 12% 37% 24% 18% Multiple times a day Once daily Every few days Weekly Other 52% 48% Yes No Summary of policy descriptions: some Centers restrict who can post on social media most Centers follow Hospital or University (even Walmart s) policy regarding marketing and/or social media common guidelines include posting nothing negative and no private information about patients

28 FY2016 Social Media Survey Results How do you manage incorporating social media at your Center: departments/projects decide what they want to post more AUCD national office support is needed to encourage use of social media time and labor intensive there are several ways that social media is managed: staff person administration social media coordinator information dissemination coordinator PR person media specialist

29 FY2016 Social Media Survey Results 120% Please indicate below who you target when using social media Frequency 100% 80% 60% 40% 20% 0% 13% 58% 29% 20% 63% 17% 13% 87% 47% 47% 53% 53% 31% 69% 31% 26% 58% 66% 16% 3% 75% 25% Never Sometimes Often Target Category

30 FY2016 Social Media Survey Results Do you capture analytics for your social media use? 30% Yes No 70% Examples of analytics captured: Facebook Insights: posts, likes, visits, shares, reactions, comments, etc. timing of posts Google analytics Twitter analytics YouTube analytics

31 FY2016 Social Media Survey Results If you have been reporting social media, please indicate what product type(s) you have used in NIRS Number of Centers Blog 1 News Aggregator 0 0 Podcast RSS Feeds Social Networking Sites Web based video clips Center website Other Summary of other suggestions for social media reporting: electronic products have become quite common, and producing them takes considerable work dissemination is a huge effort, and it is currently underreported maybe we could have a list of all Centers social media work

32 2016 NAVIGATING CHANGE: Building Our Future Together Selected Fields Custom Reports Selection Criteria - Data to be included in the report - Condition(s) used to filter records - Fields available are dataset specific - Format: the name of the dataset is displayed in CAPS, then the name of the field, and finally the type of field is displayed in parenthesis; for example: TRAINEE First Name (T) - Independent of Selected Fields - Think creatively when using these criteria: Match with a certain value Non-match with a certain value Non-blank fields Analyze data request to decide which fields to include in the report, and which to use as selection criteria AUCD 2016 CONFERENCE

33 2016 NAVIGATING CHANGE: Building Our Future Together Custom Reports, cont d Data Request: Identify the Long-Term LEND Trainees, enrolled in a Masters or Doctoral Program, who speak a foreign language. Selected Fields: TRAINEE First Name TRAINEE Last Name Run the report with the above selections. Selection Criteria: YEAR LEND Trainee = Yes YEAR Trainee Type = Long-term YEAR Academic List in the list MS, DR TRAINEE Language other than English not blank Add field TRAINEE Language other than English to list of Selected Fields to see what foreign languages the trainees speak. Re-run the report. If needed, copy and paste everything to an Excel file. AUCD 2016 CONFERENCE

34 2016 NAVIGATING CHANGE: Building Our Future Together Custom Reports, cont d Data Request: Identify how many participants received Continuing Education/Community Training between FY 2012 and FY Include staff involved in each activity. Selected Fields: ACTIVITY Activity Title ACTIVITY Staff Involvement ACTIVITY Partic. Total Selection Criteria: ACTIVITY Fiscal Year >= 2012 ACTIVITY Fiscal Year <= 2016 ACTIVITY Core Function = Continuing Education/Community Training Run the report with the above selections. Sort and group records by ACTIVITY Staff Involvement. Re-run the report. Add field ACTIVITY Fiscal Year. Re-run the report. If needed, copy and paste everything to an Excel file. AUCD 2016 CONFERENCE

35 2016 NAVIGATING CHANGE: Building Our Future Together Data Entry Questions Fields in Activities that aren't used in the UCEDD PPR report but show up in the NIRS forms, for example, the number of participants on the Research/Evaluation form. This doesn't show up in the report, should we be inputting this information? Data are reported to AIDD in one of two ways: the UCEDD PPR, and a full database export. So AIDD look both at what is reported in the PPR, and the data entered in the NIRS datasets. Some data are not required in the data entry forms, so can be skipped. Example: the number of participants on the Research/Evaluation form. AUCD 2016 CONFERENCE

36 2016 NAVIGATING CHANGE: Building Our Future Together Data Entry Questions, cont d Can a trainee be counted in Activities if they are just attending a conference or do they need to be presenting a poster or topic? Reporting data in NIRS should start with checking the core function definitions. If an activity does not fit any of the core function definitions, then it should not be reported. Conference participation: - As an attendee: there is no Center involvement in the activity, which is the Conference, therefore it does not need to be reported. - As organizer/presenter: there is definite Center involvement in the activity, and it can be reported as such. Conference presentations can also be reported as Products. AUCD 2016 CONFERENCE

37 2016 NAVIGATING CHANGE: Building Our Future Together Data Entry Questions, cont d We often have student employees (research assistants) and I am entering them as Trainees and Employees in the staff directory. Does this "double-count" the same person? Does it matter? Data entry is duplicative if the same data are collected in two or more records. But if different types of data are captured in multiple records, then it is not duplication of data. AUCD 2016 CONFERENCE

38 2016 NAVIGATING CHANGE: Building Our Future Together Data Entry Questions, cont d Other Questions? AUCD 2016 CONFERENCE

39 2016 NAVIGATING CHANGE: Building Our Future Together Break AUCD 2016 CONFERENCE

40 2016 NAVIGATING CHANGE: Building Our Future Together Roundtable Discussions Multiple surveys completed by one Trainee For centers who have multiple training programs, which run on different schedules, and have to survey trainees of both types of training programs - how do you inform the trainees of the need to fill out multiple surveys? How do you reliably pull their survey data from NIRS? Collecting gender and other ethnographic data on less frequently contacted Trainees Do other Centers have hints as to how to collect gender and any other ethnographic data on less frequently contacted medium and short term Trainees? Or any other data about Trainees that is hard to obtain? Categorizing an activity Share "case studies" where you present an activity and brainstorm as a group on how to categorize it. Focus on aspects such as core function, area of emphasis, topic of continuing education, and collaborating agencies. AUCD 2016 CONFERENCE

41 Division of MCH Workforce Development: Performance Reporting December 4, 2016 Michelle Tissue, MPH Senior Public Health Analyst, Division of MCH Workforce Development Maternal and Child Health Bureau (MCHB) Health Resources and Services Administration (HRSA)

42 42 Presentation Outline LEND data snapshot Reporting requirements and use of data Performance measure revision timeline and updates

43 LEND Data Snapshot (FY 14) Trainee Category Number of LEND Trainees Short-Term Trainees 12,547 Medium-Term Trainees 3,703 Long-Term Trainees 1,297 Continuing Education Participants 161,764 Technical Assistance Recipients 81,369 FY 2014 DGIS data 43

44 LEND Long-Term Trainee Diversity Percent of LEND long-term trainees 30.0% 25.0% 20.0% 15.0% 10.0% 5.0% 0.0% The percentage of participants in LEND long-term training programs who are from underrepresented racial and ethnic groups 25.0% 21.4% Underrepresented Racial Group 9.5% 16.3% Underrepresented Ethnic Group LEND Trainees US Population FY 2014 DGIS data Grantees Reporting: 43 44

45 Former LEND Trainees are Leaders 16.40% 83.60% Demonstrate Leadership Do Not Demonstrate Leadership 45 FY 2014 DGIS data

46 Where Are LEND Trainees Now? 30% Working in Public Health 84% Working with MCH Populations 76% Working with Underserved or Vulnerable Populations 46 FY 2014 DGIS data

47 Performance Reporting Requirements Preliminary Performance Report Annual Performance Report Project Period End Performance Report Summary of reporting requirements available at: 47

48 How is LEND data used? Public facing documents Speeches and information requests MCHB program planning documents Government Accountability Office (GAO) engagements Program evaluation Publications 48

49 Performance Measure Revision Timeline Where we have been: Activity DMCHWD Performance Measure Workgroup MCHB Submit Draft Measures to Office of Management and Budget (OMB) Public Comment Period and Town Hall Webinar Timeline December 2014 August 2015 Fall 2015/Winter 2016 Submission of Final OMB Package Winter 2016 Final approval of OMB package November

50 Performance Measure Revision Timeline Next Steps: Activity TA provided to MCH Training Programs on new performance measures LEND Programs begin data collection for new performance measures Timeline Winter/Spring 2017 July 1, 2017 **REMINDER: Data collection on current performance measures is ongoing through June 30,

51 Key Reporting Timeframes: Reporting Period 7/1/2016 6/30/2017: - CURRENT performance measures - Report due Fall 2017 Reporting Period 7/1/2017 6/30/2018: - NEW performance measures - Report due Fall

52 Final OMB Package Approved MCHB performance measures available to view at: ry_grant_information_system_performance_measure_update.pd f 52

53 53 Contact Information Michelle Tissue, MPH Public Health Analyst, Division of MCH Workforce Development Maternal and Child Health Bureau (MCHB) Health Resources and Services Administration (HRSA) Phone: (301) Web: mchb.hrsa.gov Twitter: twitter.com/hrsagov Facebook: facebook.com/hhs.hrsa

54 2016 NAVIGATING CHANGE: Building Our Future Together Relationship between NIRS and EHB NIRS to EHB Reporting Data entered in NIRS Data exported from NIRS to EHB Data uploaded to forms in your Performance Report in the EHB AUCD 2016 CONFERENCE

55 2016 NAVIGATING CHANGE: Building Our Future Together NIRS to EHB Reporting, cont d Data Entered in NIRS Trainees Dataset: Long-term trainees Former trainee surveys Activities Dataset: Continuing Education Technical Assistance Products Dataset: Products and publications Directory Dataset: Faculty and staff AUCD 2016 CONFERENCE

56 2016 NAVIGATING CHANGE: Building Our Future Together NIRS Long-term trainees Former trainee surveys Products Dataset Directory Dataset NIRS to EHB Reporting, cont d Data Exported from NIRS/Uploaded to EHB Trainee Data EHB Long Term Trainee Information form Former Trainee Information form Products and Publications Products, Publications and Submissions Data Collection Form Faculty and Staff Faculty and Staff Information form AUCD 2016 CONFERENCE

57 2016 NAVIGATING CHANGE: Building Our Future Together Reporting Process NIRS to EHB Reporting, cont d Data entered in NIRS throughout the entire fiscal year Fiscal Year: July 1 June 30 Data entry for each year starts in August End of June Early July: data export review Export files ed to Center Director and Data Coordinator Resource: Data export from NIRS to the EHB August September: Final data sent from NIRS and uploaded into the specified forms in the EHB AUCD 2016 CONFERENCE

58 2016 NAVIGATING CHANGE: Building Our Future Together Reporting Process, cont d NIRS to EHB Reporting, cont d Complete Performance Report in EHB Final review of data uploaded For CE, TA and other Forms for which data entered in NIRS, use standard reports in NIRS to obtain information to enter in the EHB Resource: MCH Training Programs Performance Measure and Data Reporting At-A-Glance Note: AUCD has no access to EHB or your Performance Report. Support provided for this is extremely limited EHB Help Desk for technical issues MCHB Project Officer for programmatic questions AUCD 2016 CONFERENCE

59 2016 NAVIGATING CHANGE: Building Our Future Together Data Matching NIRS and EHB NIRS vs. EHB age breakdown of individuals seen in clinical services Difference stems from distinct AIDD and MCHB reporting requirements To be looked into this year to see if age breakdown can be modified in NIRS to match EHB If change can be made, it will be announced, but might not happen until next FY AUCD 2016 CONFERENCE

60 2016 NAVIGATING CHANGE: Building Our Future Together NIRS Projects for the Current Fiscal Year Tribal Collaborations Reporting Currently cannot be properly captured in NIRS Call was held with Data Coordinators and then recommendations were discussed with AIDD It has been decided to add an option to the list of Agencies Collaborating to indicate Tribal entities: Activities and Services with Tribal populations When change has been made, announcement will be sent out AUCD 2016 CONFERENCE

61 2016 NAVIGATING CHANGE: Building Our Future Together NIRS Projects for the Current Fiscal Year Trainees & AUCD list serves When trainees are entered in NIRS, should they be automatically added to AUCD list serves? If so, which ones? AUCD 2016 CONFERENCE

62 2016 NAVIGATING CHANGE: Building Our Future Together NIRS Projects for the Current Fiscal Year NIRS Quarterly Calls Schedule included in NIRS calendar for next year Next call should be on January 23 rd Is that too soon? Should we cancel and have the next call in April? NIRS Orientation Next one scheduled for February AUCD 2016 CONFERENCE

63 2016 NAVIGATING CHANGE: Building Our Future Together NIRS Projects for the Current Fiscal Year Suggestions for changes Small improvements to make data entry easier? Language clarifications? Anything that needs to be fixed? AUCD 2016 CONFERENCE

64 2016 NAVIGATING CHANGE: Building Our Future Together Thank you for your participation! See you next year! AUCD 2016 CONFERENCE