Getting Results for Our Youth: Evaluating Programs for Continuous Improvement will begin momentarily

Transcription

1 Thank you for joining! Today s webinar Getting Results for Our Youth: Evaluating Programs for Continuous Improvement will begin momentarily Webinar Logistics: 1. You will be joining the webinar in listen only mode 2. Ensure you have selected the appropriate Audio Setting on your control panel To join using your computer, select Mic & Speakers To join using your telephone, select Telephone and dial in using the information provided 3. To submit questions, please do so via the Questions Box located on the control panel. We will do our best to answer all questions during the moderated Q&A throughout the webinar.

2 November 18, 2016 Dr. Nancy Deutsch, University of Virginia Dr. Samuel McQuillin, University of South Carolina Getting Results for Our Youth: Evaluating Programs for Continuous Improvement

3 My Brother s Keeper Alliance Webinar Series Webinar 1: Cultural and Class Conflicts in Mentor-Mentee Matches Webinar 2: Overview of Mentoring Boys and Men of Color and Pathways to the Mentoring Effect Webinar 3: Review of the Elements of Effective Practice for Mentoring Boys and Men of Color Virtual Trainings Access these resources on MENTOR s website: 3

4 Participation in Today s Webinar All attendees muted for best sound Type questions and comments in the question box Respond to polls Who is with us today? 4

5 Today s webinar Landscape of evaluations Logic models Process evaluations Q&A 5

6 Dr. Nancy Deutsch Associate Professor at the University of Virginia s Curry School of Education and a faculty affiliate of Youth-Nex: the UVA Center to Promote Effective Youth Development Research focuses on out-of-school settings for adolescents and the role of youth-adult relationships in positive youth development Author of numerous peer-review manuscripts and book chapters as well as two books 6

7 Dr. Samuel McQuillin Assistant Professor at the University of South Carolina s Department of Psychology Studies how schools and communities can work together to promote emotional, behavioral, and academic wellness in children who are environmentally or developmentally at-risk Creator and director of the University of Houston and University of South Carolina s AMPED mentoring program 7

8 Landscape of Evaluations Types and Purposes Cultural Competence

9 Program evaluation is the application of systematic methods to address questions about program operations and results. It may include ongoing monitoring of a program as well as one-shot studies of program processes or program impact. The approaches used are based on social science research methodologies and professional standards. Although accountability continues to be an important goal of program evaluation, the major goal should be to improve program performance, thereby giving the public and funders better value for money. - Newcomer, Hatry & Wholey (2010). Handbook of Practical Program Evaluation 9

10 5 Domains Addressed by Evaluation Need for program Program design Implementation and service delivery Impact or outcomes Efficiency Rossi, Lipsey & Freeman (2004). Evaluation: A systematic approach, 7 th ed. 10

11 Two over-arching types Formative Summative Aimed at providing real-time feedback for program improvement Provides after-the-fact information about program performance Goal more likely to be: Program development and improvement Goal more likely to be: Accountability and decision making 11

12 What type of evaluation answers what types of questions? Needs Assessment What s the matter? Assessment of Program Theory How do we think we should fix it? Assessment of Program Process Is our program working like we think it should? Impact Assessment Are we fixing the problem? Efficiency Assessment Could we do it better? 12

13 As presented in Rossi, P.H., Lipsey, M.W., Freeman, H.E. (2003). Evaluation: a systematic approach, 7 th edition. Thousand Oaks, CA: Sage.

14 Evaluation types and activities Needs Assessment Program Process Theory Blueprint of program Program Process Evaluation Formative, Implementation Evaluation Outcome Evaluation Monitoring outcomes, Summative 14

15 Types of Outcome Studies Experimental Random assignment to control groups Quasi-experimental Non-randomized comparison groups Haphazard assignment Cohort controls Internal controls External controls Population norms Secondary data Recurring Institutional Cycles Group A: X 0 Group B: 0 X 0 Non-experimental No comparison group Pre-post or post-test only 15

16 Cultural Responsiveness Williams & Deutsch (2015). Beyond Between group differences: Considering race, ethnicity, and culture in research on positive youth development programs. Applied Developmental Science. 16

17 Logic Models How and why a program operates

18 Example Logic Model Inputs Activities Expected Outputs Outcomes Half-Time Coordinator Teachers School staff 6 Local business partners 2 Hours of available contact per student per week Non-profit software system Program curriculum Volunteers from local businesses School space School performance records Administrative School planning meetings Referral process Mentor schedule coordination Mentor recruitment Mentor training End of year celebration Program Board games Sports and recreational activities Mindset sparks curriculum Career education Study skills training Goal setting Administrative 45 mentors recruited 45 mentors completed 2hrs of training 45 students referred 45 mentees matched At least 40 relationship satisfaction surveys returned Program 1 hour of mentoring per session Two mentoring meetings per week Mentors spend %50 of time on fun and recreational activities Mentors spend %50 of time on curriculum Short-Term Positive relationships with mentors Conflict resolution knowledge Intermediate Reduced disruptive behavior Strong motivation to attend college Better study skills Long-Term School matriculation College attendance Reduced community crime Enhanced social mobility Improved community engagement in civic matters Assumptions IF mentees and mentors meet at least two times a week, THEN they will be able to develop a rapport and a stronger relationship.

19 Review of Logic Model Elements Inputs Resources that are available to a program/agency and demand for agency service. Examples Community concern about problem State designated critical issue (e.g. crime) Funding Staff Leadership Software Expert consultants Curriculum Partnerships 19

20 Example Logic Model Inputs Activities Expected Outputs Outcomes Half-Time Coordinator Teachers School staff 6 Local business partners 2 Hours of available contact per student per week Non-profit software system Program curriculum Volunteers from local businesses School space School performance records Administrative School planning meetings Referral process Mentor schedule coordination Mentor recruitment Mentor training End of year celebration Program Board games Sports and recreational activities Mindset sparks curriculum Career education Study skills training Goal setting Administrative 45 mentors recruited 45 mentors completed 2hrs of training 45 students referred 45 mentees matched At least 40 relationship satisfaction surveys returned Program 1 hour of mentoring per session Two mentoring meetings per week Mentors spend %50 of time on fun and recreational activities Mentors spend %50 of time on curriculum Short-Term Positive relationships with mentors Conflict resolution knowledge Intermediate Reduced disruptive behavior Strong motivation to attend college Better study skills Long-Term School matriculation College attendance Reduced community crime Enhanced social mobility Improved community engagement in civic matters Assumptions IF mentees and mentors meet at least two times a week, THEN they will be able to develop a rapport and a stronger relationship.

21 Review of Logic Model Elements Activities What programs do with resources to bring about desired goals. Both program-related (direct benefit to youth) and administrative (required to make program activities possible) Examples: Recruitment Training Mentoring meetings Partnership developments Referral services Prevention services 21

22 Example Logic Model Inputs Activities Expected Outputs Outcomes Half-Time Coordinator Teachers School staff 6 Local business partners 2 Hours of available contact per student per week Non-profit software system Program curriculum Volunteers from local businesses School space School performance records Administrative School planning meetings Referral process Mentor schedule coordination Mentor recruitment Mentor training End of year celebration Program Board games Sports and recreational activities Mindset sparks curriculum Career education Study skills training Goal setting Administrative 45 mentors recruited 45 mentors completed 2hrs of training 45 students referred 45 mentees matched At least 40 relationship satisfaction surveys returned Program 1 hour of mentoring per session Two mentoring meetings per week Mentors spend %50 of time on fun and recreational activities Mentors spend %50 of time on curriculum Short-Term Positive relationships with mentors Conflict resolution knowledge Intermediate Reduced disruptive behavior Strong motivation to attend college Better study skills Long-Term School matriculation College attendance Reduced community crime Enhanced social mobility Improved community engagement in civic matters Assumptions IF mentees and mentors meet at least two times a week, THEN they will be able to develop a rapport and a stronger relationship.

23 Review of Logic Model Elements Outputs The direct products or services that are produced by the activities - both administrative and program-related. Amount of work performed, units of service provided, units of information collected. Examples: # of mentees being mentored # of mentors recruited Amount of mentor supervision provided % of surveys completed Frequency of supervisor contact with mentors Match meeting frequency/duration Hours of prevention activities provided 23

24 Example Logic Model Inputs Activities Expected Outputs Outcomes Half-Time Coordinator Teachers School staff 6 Local business partners 2 Hours of available contact per student per week Non-profit software system Program curriculum Volunteers from local businesses School space School performance records Administrative School planning meetings Referral process Mentor schedule coordination Mentor recruitment Mentor training End of year celebration Program Board games Sports and recreational activities Mindset sparks curriculum Career education Study skills training Goal setting Administrative 45 mentors recruited 45 mentors completed 2hrs of training 45 students referred 45 mentees matched At least 40 relationship satisfaction surveys returned Program 1 hour of mentoring per session Two mentoring meetings per week Mentors spend %50 of time on fun and recreational activities Mentors spend %50 of time on curriculum Short-Term Positive relationships with mentors Conflict resolution knowledge Intermediate Reduced disruptive behavior Strong motivation to attend college Better study skills Long-Term School matriculation College attendance Reduced community crime Enhanced social mobility Improved community engagement in civic matters Assumptions IF mentees and mentors meet at least two times a week, THEN they will be able to develop a rapport and a stronger relationship.

25 Review of Logic Model Elements Outcomes The changes or results produced in program participants by the outputs. As a general rule, these are things that are not part of the program itself but rather experienced as a result of being in the program. A good check in this regard is whether the outcome could be experienced by someone not in the program -- if not, it probably isn t an outcome but rather an input Short-term, intermediate, and long-term 25

26 Outcome Examples for Participating Youth: Short-Term Outcomes Intermediate Outcomes Long-Term Outcomes Changes in academic motivation Presence of at least one experienced and important nonparental adult Stronger relationships with nonparental adults Decrease in school disciplinary infractions Decreased crime involvement Participation in higher education Decreased rate of homelessness 26

27 Process Evaluation Implementation and Fidelity

28 Process Evaluation Answers questions related to the implementation of the program and how it is experienced by participants. In terms of your program s logic model. 28

29 Goals of Process Evaluation Determine whether program components are happening as intended Identify strengths and weaknesses of implementation Identify the feasibility of specific practices and procedures Identify and areas for improvement 29

30 Why does understanding program theory and implementation matter? 3 program failings (Rossi, 1985, quoted in Shadish, Cook & Leviton, 1991) 1. Failure in our understandings of processes that give rise to the program to which the program is addressed 2. Failure in understanding of how to translate problem theory into specific programs 3. Failure to properly implement programs 30

31 Why evaluate implementation? Implementation and evaluation are two sides of the same coin, implementation providing the experience that evaluation interrogates and evaluation providing the intelligence to make sense out of what is happening. Pressman & Wildavsky, 1984, Implementation: How great expectations in Washington are dashed in Oakland 31

32 The Black Box of Program Evaluation Targets Outcomes Program 32

33 Love, 2004 (in Handbook of Practical Program Evaluation, Wholey, Hatry & Newcomer)

34 Questions Process Evaluations Ask Who is and is not receiving services? How aware is the population of the program? How are the program functions performing? Staffing and organization of the program Resources Compliance with requirements Differences between sites

35 Reasons for doing a process evaluation Accountability Program Improvement Testing your Program Theory of Change/Logic Model Real-Time Program Adjustments Optimizing Experiences of Individual Participants 35

36 Components of a process evaluation Consistency (Adaptation) Are program components being implemented as intended? Adaptations may be useful and beneficial. Participation To what extent does each youth take part in or receive intended activities or experiences? Addresses how much, how frequently, when, and where each activity/experience is received. Quality How well is the program delivered? Is the program being implemented to a standard intended? 36

37 Assessing program process Program records implementation checklists staff logs mentor logs attendance sheets case files Program registration forms Direct observation of program activities match interactions group activities match support mentor training Surveys and interviews Focus groups 37

38 Operating at Peak Efficiency 38

39 Outcome Monitoring Keeping track of your goals

40 Outcome Monitoring What are the immediate measureable outcomes that we expect our program/training/event to accomplish? Short-term or intermediate knowledge, attitudes, beliefs or behaviors that can be measured concurrently with program implementation. Periodic assessment of the *effects* produced by activities/outputs. Mentee intentions, mentor knowledge, mentee school absences, etc. 40

41 Outcome Monitoring If a program s long-term outcome is college acceptance, what might be some outcomes we can monitor as the program is being implemented? 41

42 Outcome Monitoring Researchers have found several factors that predict college acceptance, that mentors could presumably influence. Programs could monitor some of these that they believe mentoring will directly influence: GPA Absences Practice SAT scores Intentions to apply to college College applications submitted Etc. 42

43 Acceptability & Usability How to measure and improve program operations

44 Basic Definitions: Usability & Acceptability Usability: The user experience of some system, technology, curriculum, or process. Users may be mentors, mentees, administrators, supervisors, etc. Ease of use, difficulty to learn, time efficient, error free, pleasant to use 44

45 Basic Definitions: Usability & Acceptability Acceptability: The user s willingness to use the technology, curriculum, or process within their specific system Excitement, interest, positive attitudes, motivation, perceived effectiveness, fair, self-benefiting 45

46 Measures of Usability/Acceptability Usage Rating Profile-Intervention Developed by Sandra Chafouleas, Amy Briesch, Chris Riley- Tillman, and Betsy McCoach. 35 item scale used to assess acceptability and usage of interventions for children. Ratings made on a 6 point likert scale: strongly disagree to strongly agree 46

47 Measures of Usability/Acceptability The principles of measures like the URP-I can be extended to focus groups, other questionnaires, or casual conversation with mentors/administrators. 47

48 Examples of Acceptability From URP-I Example Questions of Acceptability I would be excited to use this (program, software, training). Overall, this (program) is beneficial to the child. This (training) would save time spent (preparing our mentors). This is a good work flow for checking in with mentors. I would NOT be interested in using this (technology, training, etc.) 48

49 Examples of Acceptability From URP-I Examples of non-scale extensions from this measure: Focus groups with questions like: Let s talk about some things we are excited about with this intervention, and then some things we are dreading, or not so excited about. Tell me some things you think might be helpful for your mentee with this curriculum, and then some things that might not be so helpful. What were the biggest frustrations in implementing this case management software? 49

50 Some dimensions of acceptability & usability Understanding: The end-user understands, and has the skills to implement, the technology, curriculum, or process. 50

51 Examples of Understanding From URP-I Example Questions of Understanding I understand how to use this (technology). I would know what to do if I was asked to (coordinate this training). I have NO IDEA how to (discuss these topics with my mentee). I have the skills needed to implement this (work flow). The requirements for implementing this (training) are UNCLEAR. 51

52 Examples of Understanding From URP-I Example non-scale extensions of understanding: Create a quiz that tests user knowledge. Have competency tests to assess skill. Ask focus group questions like: Tell me some things that are confusing about the work-flow process right now. 52

53 Example From USC-AMPED School-Based Program: Goals of the program: Motivate students to succeed in school and life. Equip students with skills that are necessary to succeed in the 21 st century. Support students as they set and pursue specific goals in school. Mentors use Motivational Interviewing, which is an approach to having conversations about change. 53

54 Example from USC-AMPED Motivational Interviewing is not easy. Strategically using open ended questions, affirmations, and reflections to encourage discussions about behavior change. All mentors attended the trainings, positive ratings from training experience, etc. Trainings were inadequate to actually change mentors knowledge or skills. They could not perform the basic MI skills. Training needed to be revised. 54

55 Some dimensions of acceptability & usability Feasibility: The user-demands to operate or participate in the technology, curriculum, or process are reasonable. User Demands: resources like time, cost, energy, ability, etc. 55

56 Examples of Feasibility From URP-I Example Questions of Feasibility The amount of time required for record keeping (with mentor notes) is reasonable. The (program) could be implemented in its entirety with our timeline. The (training model) could be implemented with the level of intensity as it is prescribed (e.g. all quizzes, feedback, and routines, etc.) The (process) would not be disruptive to our workflow. The program could be implemented *exactly* as prescribed. 56

57 Some dimensions of acceptability & usability System Support: The amount of extra help the technology curriculum or process would require. Extra Help: Support from administrators, staff, consultants, or technical assistance 57

58 Examples of Systems Support From URP-I Example Questions of Systems Support Conducting these (trainings) would require the support of my coworkers. I would need support from my administrators to (process these applications). Parental collaboration would be required in order for this (program) to work. I would need consultative support to implement this (program). 58

59 Q&A Type your questions in the question box: 59

60 Additional Resources Request no-cost help for your mentoring program at an online resource funded by the Office of Juvenile Justice and Delinquency Prevention and facilitated in partnership with MENTOR Measurement Guidance Toolkit for Mentoring Programs Key Evaluation Considerations Recruit mentors by submitting your program to the Mentoring Connector 60

61 My Brother s Keeper Alliance Webinar Series Webinars Virtual Trainings Access these resources on MENTOR s website: 61

62 Thank you for participating in today s webinar! Be on the lookout for a survey about your experience. As always, your feedback is greatly appreciated as we work to provide technical assistance that is valuable for your organization and community For more information, please contact info@mentoring.org or info@mbkalliance.org